# StarRocks Documentation > StarRocks is an open-source, high-performance OLAP database for real-time analytics at scale. It supports Standard SQL, materialized views, data lakes (Iceberg, Delta Lake, Hudi), stream ingestion (Kafka, Flink), and cloud-native deployment. This documentation covers SQL reference, table design, data loading, query acceleration, administration, and release notes. ## Administration ### Configuration Configuration parameters for FE and BE nodes. #### [🗃️ FE Configuration](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md) [4 items](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md) #### [🗃️ BE Configuration](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md) [4 items](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md) --- ### BE Configuration Some BE configuration items are dynamic parameters which you can set interactively when BE nodes are online. The rest of them are static parameters. You can only set the static parameters of a BE node by changing them in the corresponding configuration file **be.conf** and restarting the BE node to allow the change to take effect. #### View BE configuration items[​](#view-be-configuration-items "Direct link to View BE configuration items") You can view the BE configuration items using the following command: ```sql SELECT * FROM information_schema.be_configs WHERE NAME LIKE "%%" ``` #### Configure BE parameters[​](#configure-be-parameters "Direct link to Configure BE parameters") ##### Configure BE dynamic parameters[​](#configure-be-dynamic-parameters "Direct link to Configure BE dynamic parameters") You can configure a dynamic parameter of a BE node by updating the value in `information_schema.be_configs`. warning Setting an invalid value may cause unknown behaviors. Check twice before you run the command to update the configuration. ```sql -- Replace with the key of the configuration and with the value. UPDATE information_schema.be_configs SET VALUE = WHERE name = ""; ``` ##### Configure BE static parameters[​](#configure-be-static-parameters "Direct link to Configure BE static parameters") You can only set the static parameters of a BE by changing them in the corresponding configuration file **be.conf**, and restarting the BE to allow the changes to take effect. #### Parameter groups[​](#parameter-groups "Direct link to Parameter groups") The parameters are grouped in these categories: * [Logging](https://docs.starrocks.io/docs/administration/configuration/BE_parameters/log_server_meta.md) * [Server](https://docs.starrocks.io/docs/administration/configuration/BE_parameters/log_server_meta.md) * [Metadata and Cluster management](https://docs.starrocks.io/docs/administration/configuration/BE_parameters/log_server_meta.md) * [Query engine](https://docs.starrocks.io/docs/administration/configuration/BE_parameters/query_loading.md) * [Loading and unloading](https://docs.starrocks.io/docs/administration/configuration/BE_parameters/query_loading.md) * [Statistic report](https://docs.starrocks.io/docs/administration/configuration/BE_parameters/stats_storage.md) * [Storage](https://docs.starrocks.io/docs/administration/configuration/BE_parameters/stats_storage.md) * [Shared-data](https://docs.starrocks.io/docs/administration/configuration/BE_parameters/shared_lake_other.md) * [Data Lake](https://docs.starrocks.io/docs/administration/configuration/BE_parameters/shared_lake_other.md) * [Other](https://docs.starrocks.io/docs/administration/configuration/BE_parameters/shared_lake_other.md) --- ### BE Configuration - Logging, Server, and Metadata Some BE configuration items are dynamic parameters which you can set interactively when BE nodes are online. The rest of them are static parameters. You can only set the static parameters of a BE node by changing them in the corresponding configuration file **be.conf** and restarting the BE node to allow the change to take effect. #### View BE configuration items[​](#view-be-configuration-items "Direct link to View BE configuration items") You can view the BE configuration items using the following command: ```sql SELECT * FROM information_schema.be_configs [WHERE NAME LIKE "%%"] ``` #### Configure BE parameters[​](#configure-be-parameters "Direct link to Configure BE parameters") ##### Configure BE dynamic parameters[​](#configure-be-dynamic-parameters "Direct link to Configure BE dynamic parameters") You can configure a dynamic parameter of a BE node by updating the value in `information_schema.be_configs`. warning Setting an invalid value may cause unknown behaviors. Check twice before you run the command to update the configuration. ```sql -- Replace with the key of the configuration and with the value. UPDATE information_schema.be_configs SET VALUE = WHERE name = ""; ``` ##### Configure BE static parameters[​](#configure-be-static-parameters "Direct link to Configure BE static parameters") You can only set the static parameters of a BE by changing them in the corresponding configuration file **be.conf**, and restarting the BE to allow the changes to take effect. *** This topic introduces the following types of BE configurations: * [Logging](#logging) * [Server](#server) * [Metadata and Cluster Management](#metadata-and-cluster-management) #### Logging[​](#logging "Direct link to Logging") ##### diagnose\_stack\_trace\_interval\_ms[​](#diagnose_stack_trace_interval_ms "Direct link to diagnose_stack_trace_interval_ms") * Default: 1800000 (30 minutes) * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Controls the minimum time gap between successive stack-trace diagnostics performed by DiagnoseDaemon for `STACK_TRACE` requests. When a diagnose request arrives, the daemon skips collecting and logging stack traces if the last collection happened less than `diagnose_stack_trace_interval_ms` milliseconds ago. Increase this value to reduce CPU overhead and log volume from frequent stack dumps; decrease it to capture more frequent traces to debug transient issues (for example, in load fail-point simulations of long `TabletsChannel::add_chunk` blocking). * Introduced in: v3.5.0 ##### lake\_replication\_slow\_log\_ms[​](#lake_replication_slow_log_ms "Direct link to lake_replication_slow_log_ms") * Default: 30000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Threshold for emitting slow-log entries during lake replication. After each file copy the code measures elapsed time in microseconds and marks the operation as slow when elapsed time is greater than or equal to `lake_replication_slow_log_ms * 1000`. When triggered, StarRocks writes an INFO log with file size, cost and trace metrics for that replicated file. Increase the value to reduce noisy slow logs for large/slow transfers; decrease it to detect and surface smaller slow-copy events sooner. * Introduced in: - ##### load\_rpc\_slow\_log\_frequency\_threshold\_seconds[​](#load_rpc_slow_log_frequency_threshold_seconds "Direct link to load_rpc_slow_log_frequency_threshold_seconds") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Controls how frequently the system prints slow-log entries for load RPCs that exceed their configured RPC timeout. The slow-log also includes the load channel runtime profile. Setting this value to 0 causes per-timeout logging in practice. * Introduced in: v3.4.3, v3.5.0 ##### log\_buffer\_level[​](#log_buffer_level "Direct link to log_buffer_level") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The strategy for flushing logs. The default value indicates that logs are buffered in memory. Valid values are `-1` and `0`. `-1` indicates that logs are not buffered in memory. * Introduced in: - ##### pprof\_profile\_dir[​](#pprof_profile_dir "Direct link to pprof_profile_dir") * Default: `${STARROCKS_HOME}/log` * Type: String * Unit: - * Is mutable: No * Description: Directory path where StarRocks writes pprof artifacts (Jemalloc heap snapshots and gperftools CPU profiles). * Introduced in: v3.2.0 ##### sys\_log\_dir[​](#sys_log_dir "Direct link to sys_log_dir") * Default: `${STARROCKS_HOME}/log` * Type: String * Unit: - * Is mutable: No * Description: The directory that stores system logs (including INFO, WARNING, ERROR, and FATAL). * Introduced in: - ##### sys\_log\_level[​](#sys_log_level "Direct link to sys_log_level") * Default: INFO * Type: String * Unit: - * Is mutable: Yes (from v3.3.0, v3.2.7, and v3.1.12) * Description: The severity levels into which system log entries are classified. Valid values: INFO, WARNING, ERROR, and FATAL. This item was changed to a dynamic configuration from v3.3.0, v3.2.7, and v3.1.12 onwards. * Introduced in: - ##### sys\_log\_roll\_mode[​](#sys_log_roll_mode "Direct link to sys_log_roll_mode") * Default: SIZE-MB-1024 * Type: String * Unit: - * Is mutable: No * Description: The mode in which system logs are segmented into log rolls. Valid values include `TIME-DAY`, `TIME-HOUR`, and `SIZE-MB-`size. The default value indicates that logs are segmented into rolls, each of which is 1 GB. * Introduced in: - ##### sys\_log\_roll\_num[​](#sys_log_roll_num "Direct link to sys_log_roll_num") * Default: 10 * Type: Int * Unit: - * Is mutable: No * Description: The number of log rolls to reserve. * Introduced in: - ##### sys\_log\_timezone[​](#sys_log_timezone "Direct link to sys_log_timezone") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to show timezone information in the log prefix. `true` indicates to show timezone information, `false` indicates not to show. * Introduced in: - ##### sys\_log\_verbose\_level[​](#sys_log_verbose_level "Direct link to sys_log_verbose_level") * Default: 10 * Type: Int * Unit: - * Is mutable: No * Description: The level of the logs to be printed. This configuration item is used to control the output of logs initiated with VLOG in codes. * Introduced in: - ##### sys\_log\_verbose\_modules[​](#sys_log_verbose_modules "Direct link to sys_log_verbose_modules") * Default: * Type: Strings * Unit: - * Is mutable: No * Description: Specifies the file names (without extensions) or file name wildcards for which VLOG logs should be printed. Multiple file names can be separated by commas. For example, if you set this configuration item to `storage_engine,tablet_manager`, StarRocks prints VLOG logs from the storage\_engine.cpp and tablet\_manager.cpp files. You can also use wildcards, e.g., set to `*` to print VLOG logs from all files. The VLOG log printing level is controlled by the `sys_log_verbose_level` parameter. * Introduced in: - #### Server[​](#server "Direct link to Server") ##### abort\_on\_large\_memory\_allocation[​](#abort_on_large_memory_allocation "Direct link to abort_on_large_memory_allocation") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When a single allocation request exceeds the configured large-allocation threshold (g\_large\_memory\_alloc\_failure\_threshold `>` 0 and requested size `>` threshold), this flag controls how the process responds. If true, StarRocks calls std::abort() immediately (hard crash) when such a large allocation is detected. If false, the allocation is blocked and the allocator returns failure (nullptr or ENOMEM) so callers can handle the error. This check only takes effect for allocations that are not wrapped with the TRY\_CATCH\_BAD\_ALLOC path (the mem hook uses a different flow when bad-alloc is being caught). Enable for fail-fast debugging of unexpected huge allocations; keep disabled in production unless you want an immediate process abort on over-large allocation attempts. * Introduced in: v3.4.3, 3.5.0, 4.0.0 ##### arrow\_flight\_port[​](#arrow_flight_port "Direct link to arrow_flight_port") * Default: -1 * Type: Int * Unit: - * Is mutable: No * Description: TCP port for the BE Arrow Flight SQL server. `-1` indicaes to disable the Arrow Flight service. On non-macOS builds, BE invokes Arrow Flight SQL Server with this port during startup; if the port is unavailable, the server startup fails and the BE process exits. The configured port is reported to the FE in the heartbeat payload. * Introduced in: v3.4.0, v3.5.0 ##### be\_exit\_after\_disk\_write\_hang\_second[​](#be_exit_after_disk_write_hang_second "Direct link to be_exit_after_disk_write_hang_second") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: No * Description: The length of time that the BE waits to exit after the disk hangs. * Introduced in: - ##### be\_http\_num\_workers[​](#be_http_num_workers "Direct link to be_http_num_workers") * Default: 48 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads used by the HTTP server. * Introduced in: - ##### be\_http\_port[​](#be_http_port "Direct link to be_http_port") * Default: 8040 * Type: Int * Unit: - * Is mutable: No * Description: The BE HTTP server port. * Introduced in: - ##### enable\_http\_auth[​](#enable_http_auth "Direct link to enable_http_auth") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Introduced in: v4.2.0 * Description: When true, most external BE HTTP endpoints require HTTP Basic Auth. Credentials are verified by RPC to the FE leader using the `checkAuth` Thrift method, so the user/password store on the FE side (including LDAP / security-integration) is the source of truth. The following are exempt: * Token-gated internal transport (used by FE/BE for tablet clone and load-error file fetch): `/api/_tablet/_download`, `/api/_download_load`. These remain protected by their own token check; setting `enable_http_auth=true` does **not** compensate for `enable_token_check=false`. * Stream Load and transaction endpoints that authenticate inside the handler using the load label + table grants: `/api/{db}/{table}/_stream_load`, `/api/transaction/{txn_op}`, `/api/transaction/load`. Privileged endpoints additionally require a SYSTEM-level RBAC privilege (`OPERATE` or `NODE`) that is **active** in the session — use `SET DEFAULT ROLE TO ;` or set `activate_all_roles_on_login=true` if the role is granted but not default. LDAP / security-integration group → role mappings activate automatically. ##### be\_port[​](#be_port "Direct link to be_port") * Default: 9060 * Type: Int * Unit: - * Is mutable: No * Description: The BE thrift server port, which is used to receive requests from FEs. * Introduced in: - ##### be\_service\_threads[​](#be_service_threads "Direct link to be_service_threads") * Default: 64 * Type: Int * Unit: Threads * Is mutable: No * Description: Number of worker threads the BE Thrift server uses to serve backend RPC/execution requests. This value is passed to ThriftServer when creating the BackendService and controls how many concurrent request handlers are available; requests are queued when all worker threads are busy. Tune based on expected concurrent RPC load and available CPU/memory: increasing it raises concurrency but also per-thread memory and context-switch cost, decreasing it limits parallel handling and may increase request latency. * Introduced in: v3.2.0 ##### brpc\_connection\_type[​](#brpc_connection_type "Direct link to brpc_connection_type") * Default: `"single"` * Type: string * Unit: - * Is mutable: No * Description: The bRPC channel connection mode. Valid values: * `"single"` (Default): One persistent TCP connection for each channel. * `"pooled"`: A pool of persistent connections for higher concurrency at the cost of more sockets/file descriptors. * `"short"`: Short‑lived connections created per RPC to reduce persistent resource usage but with higher latency. The choice affects per-socket buffering behavior and can influence `Socket.Write` failures (EOVERCROWDED) when unwritten bytes exceed socket limits. * Introduced in: v3.2.5 ##### brpc\_max\_body\_size[​](#brpc_max_body_size "Direct link to brpc_max_body_size") * Default: 2147483648 * Type: Int * Unit: Bytes * Is mutable: No * Description: The maximum body size of a bRPC. * Introduced in: - ##### brpc\_max\_connections\_per\_server[​](#brpc_max_connections_per_server "Direct link to brpc_max_connections_per_server") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of persistent bRPC connections the client keeps for each remote server endpoint. For each endpoint `BrpcStubCache` creates a `StubPool` whose `_stubs` vector is reserved to this size. On first accesses, new stubs are created until the limit is reached. After that, existing stubs are returned in a round‑robin fashion. Increasing this value raises per‑endpoint concurrency (reduces contention on a single channel) at the cost of more file descriptors, memory, and channels. * Introduced in: v3.2.0 ##### brpc\_num\_threads[​](#brpc_num_threads "Direct link to brpc_num_threads") * Default: -1 * Type: Int * Unit: - * Is mutable: No * Description: The number of bthreads of a bRPC. The value `-1` indicates the same number with the CPU threads. * Introduced in: - ##### brpc\_port[​](#brpc_port "Direct link to brpc_port") * Default: 8060 * Type: Int * Unit: - * Is mutable: No * Description: The BE bRPC port, which is used to view the network statistics of bRPCs. * Introduced in: - ##### brpc\_socket\_max\_unwritten\_bytes[​](#brpc_socket_max_unwritten_bytes "Direct link to brpc_socket_max_unwritten_bytes") * Default: 1073741824 * Type: Int * Unit: Bytes * Is mutable: No * Description: Sets the per-socket limit for unwritten outbound bytes in the bRPC server. When the amount of buffered, not-yet-written data on a socket reaches this limit, subsequent `Socket.Write` calls fail with EOVERCROWDED. This prevents unbounded per-connection memory growth but can cause RPC send failures for very large messages or slow peers. Align this value with `brpc_max_body_size` to ensure single-message bodies are not larger than the allowed unwritten buffer. Increasing the value raises memory usage per connection. * Introduced in: v3.2.0 ##### brpc\_stub\_expire\_s[​](#brpc_stub_expire_s "Direct link to brpc_stub_expire_s") * Default: 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The expire time of bRPC stub cache. The default value is 60 minutes. * Introduced in: - ##### compress\_rowbatches[​](#compress_rowbatches "Direct link to compress_rowbatches") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: A boolean value to control whether to compress the row batches in RPCs between BEs. `true` indicates compressing the row batches, and `false` indicates not compressing them. * Introduced in: - ##### consistency\_max\_memory\_limit\_percent[​](#consistency_max_memory_limit_percent "Direct link to consistency_max_memory_limit_percent") * Default: 20 * Type: Int * Unit: - * Is mutable: No * Description: Percentage cap used to compute the memory budget for consistency-related tasks. During BE startup, the final consistency limit is computed as the minimum of the value parsed from `consistency_max_memory_limit` (bytes) and (`process_mem_limit * consistency_max_memory_limit_percent / 100`). If `process_mem_limit` is unset (-1), consistency memory is considered unlimited. For `consistency_max_memory_limit_percent`, values less than 0 or greater than 100 are treated as 100. Adjusting this value increases or decreases memory reserved for consistency operations and therefore affects memory available for queries and other services. * Introduced in: v3.2.0 ##### delete\_worker\_count\_normal\_priority[​](#delete_worker_count_normal_priority "Direct link to delete_worker_count_normal_priority") * Default: 2 * Type: Int * Unit: Threads * Is mutable: No * Description: Number of normal-priority worker threads dedicated to handling delete (REALTIME\_PUSH with DELETE) tasks on the BE agent. At startup this value is added to delete\_worker\_count\_high\_priority to size the DeleteTaskWorkerPool (see agent\_server.cpp). The pool assigns the first delete\_worker\_count\_high\_priority threads as HIGH priority and the rest as NORMAL; normal-priority threads process standard delete tasks and contribute to overall delete throughput. Increase to raise concurrent delete capacity (higher CPU/IO usage); decrease to reduce resource contention. * Introduced in: v3.2.0 ##### disable\_mem\_pools[​](#disable_mem_pools "Direct link to disable_mem_pools") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to disable MemPool. When this item is set to `true`, the MemPool chunk pooling is disabled so each allocation gets its own sized chunk instead of reusing or increasing pooled chunks. Disabling pooling reduces long-lived retained buffer memory at the cost of more frequent allocations, increased number of chunks, and skipped integrity checks (which are avoided because of the large chunk count). Keep `disable_mem_pools` as `false` (default) to benefit from allocation reuse and fewer system calls. Set it to `true` only when you must avoid large pooled memory retention (for example, low-memory environments or diagnostic runs). * Introduced in: v3.2.0 ##### enable\_https[​](#enable_https "Direct link to enable_https") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: When this item is set to `true`, the BE's bRPC server is configured to use TLS: `ServerOptions.ssl_options` will be populated with the certificate and private key specified by `ssl_certificate_path` and `ssl_private_key_path` at BE startup. This enables HTTPS/TLS for incoming bRPC connections; clients must connect using TLS. Ensure the certificate and key files exist, are accessible to the BE process, and match bRPC/SSL expectations. * Introduced in: v4.0.0 ##### enable\_jemalloc\_memory\_tracker[​](#enable_jemalloc_memory_tracker "Direct link to enable_jemalloc_memory_tracker") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: When this item is set to `true`, the BE starts a background thread (jemalloc\_tracker\_daemon) that polls jemalloc statistics (once per second) and updates the GlobalEnv jemalloc metadata MemTracker with the jemalloc "stats.metadata" value. This ensures jemalloc metadata consumption is included in StarRocks process memory accounting and prevents under‑reporting of memory used by jemalloc internals. The tracker is only compiled/started on non‑macOS builds (#ifndef **APPLE**) and runs as a daemon thread named "jemalloc\_tracker\_daemon". Because this setting affects startup behaviour and threads that maintain MemTracker state, changing it requires a restart. Disable only if jemalloc is not used or when jemalloc tracking is intentionally managed differently; otherwise keep enabled to maintain accurate memory accounting and allocation safeguards. * Introduced in: v3.2.12 ##### enable\_jvm\_memory\_tracker[​](#enable_jvm_memory_tracker "Direct link to enable_jvm_memory_tracker") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to count the memory used by the JVM toward the BE memory limit (`mem_limit`). The JVM maps its heap and non-heap memory (such as Metaspace and the code cache) itself, outside the BE memory allocator, so this memory is otherwise not counted. When this item is `true`, each time the JVM metrics are collected (every 15 seconds when `enable_metric_calculator` is `true`), BE sets the `jvm` memory tracker, under the `process` tracker, to the committed heap plus non-heap memory of the JVM. The `jvm` tracker is shown on the `/mem_tracker` page. When this item is changed to `false`, the next collection resets the `jvm` tracker to 0. This item takes effect only when `enable_jvm_metrics` is `true`. Before a JVM is created in the BE process, the `jvm` tracker is 0. * Introduced in: - ##### enable\_jvm\_metrics[​](#enable_jvm_metrics "Direct link to enable_jvm_metrics") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to collect JVM memory metrics on the BE. The metrics are `jvm_heap_size_bytes` and `jvm_non_heap_size_bytes` for heap and non-heap memory, and `jvm_young_size_bytes`, `jvm_survivor_size_bytes`, `jvm_old_size_bytes`, and `jvm_perm_size_bytes` (Metaspace) for memory pools. The `type` label of each metric is `used`, `committed`, `max`, `peak_used`, or `peak_max`. Enabling this item does not start a JVM. The BE starts collecting the metrics only after a JVM is created in the BE process by a feature that uses Java, such as HDFS access, JNI-based readers, or Java UDFs. Until then, the metrics are 0. The metrics are refreshed every 15 seconds when `enable_metric_calculator` is `true`. When `enable_jvm_memory_tracker` is `true`, each refresh also counts the memory of the JVM toward the BE memory limit. See `enable_jvm_memory_tracker`. If `JAVA_HOME` is not set, the metrics are not registered. A `max` or `peak_max` value of -1 means that the memory pool has no defined maximum. With the default G1 garbage collector, this applies to `jvm_young_size_bytes` and `jvm_survivor_size_bytes` (the Eden and Survivor spaces), and to `jvm_perm_size_bytes` (Metaspace) unless `MaxMetaspaceSize` is set. The memory pool metrics are populated only for the Serial, Parallel, and G1 garbage collectors. Use `enable_system_metrics` to control system-level metric collection. * Introduced in: v4.0.0 ##### get\_pindex\_worker\_count[​](#get_pindex_worker_count "Direct link to get_pindex_worker_count") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: Sets the number of worker threads for the "get\_pindex" thread pool in UpdateManager, which is used to load / fetch persistent index data (used when applying rowsets for primary-key tables). At runtime, a config update will adjust the pool's maximum threads: if `>0` that value is applied; if 0 the runtime callback uses the number of CPU cores (CpuInfo::num\_cores()). On initialization the pool's max threads is computed as max(get\_pindex\_worker\_count, max\_apply\_thread\_cnt \* 2) where max\_apply\_thread\_cnt is the apply-thread pool maximum. Increase to raise parallelism for pindex loading; lowering reduces concurrency and memory/CPU usage. * Introduced in: v3.2.0 ##### heartbeat\_service\_port[​](#heartbeat_service_port "Direct link to heartbeat_service_port") * Default: 9050 * Type: Int * Unit: - * Is mutable: No * Description: The BE heartbeat service port, which is used to receive heartbeats from FEs. * Introduced in: - ##### heartbeat\_service\_thread\_count[​](#heartbeat_service_thread_count "Direct link to heartbeat_service_thread_count") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: The thread count of the BE heartbeat service. * Introduced in: - ##### jemalloc\_conf[​](#jemalloc_conf "Direct link to jemalloc_conf") * Default: `percpu_arena:percpu,oversize_threshold:134217728,muzzy_decay_ms:5000,dirty_decay_ms:5000,metadata_thp:auto,background_thread:true,prof:true,prof_active:false` * Type: string * Unit: - * Is mutable: Yes (only some options) * Description: The jemalloc runtime options that `bin/start_backend.sh` exports as the `JEMALLOC_CONF` environment variable when the BE is started in the normal mode. This item is ignored if `JEMALLOC_CONF` is already set in the environment, and if the BE is started with `--jemalloc_debug` or `--check_mem_leak`, because those modes force their own option string. jemalloc reads `JEMALLOC_CONF` before the BE parses its configuration, so modifying this item at runtime only re-applies the options that jemalloc itself allows to be changed after initialization: `dirty_decay_ms`, `muzzy_decay_ms`, and `prof_active`. Adding, removing, or changing any other option is rejected with an error and the configuration value is rolled back, which means those options require restarting the BE. `prof_active` can only be changed if the BE was started with `prof:true`. Setting `dirty_decay_ms` or `muzzy_decay_ms` to `0` purges all unused pages synchronously and can therefore make the modification take a while, and setting it to `-1` disables purging. * Introduced in: v4.0.15, v4.1.3 ##### local\_library\_dir[​](#local_library_dir "Direct link to local_library_dir") * Default: `${UDF_RUNTIME_DIR}` * Type: string * Unit: - * Is mutable: No * Description: Local directory on the BE where UDF (user-defined function) libraries are staged and where Python UDF worker processes operate. StarRocks copies UDF libraries from HDFS into this path, creates per-worker Unix domain sockets at `/pyworker_`, and chdirs Python worker processes into this directory before exec. The directory must exist, be writable by the BE process, and reside on a filesystem that supports Unix domain sockets (i.e., a local filesystem). Because this config is immutable at runtime, set it before startup and ensure adequate permissions and disk space on each BE. * Introduced in: v3.2.0 ##### max\_transmit\_batched\_bytes[​](#max_transmit_batched_bytes "Direct link to max_transmit_batched_bytes") * Default: 262144 * Type: Int * Unit: Bytes * Is mutable: No * Description: Maximum number of serialized bytes to accumulate in a single transmit request before it is flushed to the network. Sender implementations add serialized ChunkPB payloads into a PTransmitChunkParams request and send the request once the accumulated bytes exceed `max_transmit_batched_bytes` or when EOS is reached. Increase this value to reduce RPC frequency and improve throughput at the cost of higher per-request latency and memory use; reduce it to lower latency and memory but increase RPC rate. * Introduced in: v3.2.0 ##### mem\_limit[​](#mem_limit "Direct link to mem_limit") * Default: 90% * Type: String * Unit: - * Is mutable: No * Description: BE process memory upper limit. You can set it as a percentage ("80%") or a physical limit ("100G"). The default hard limit is 90% of the server's memory size, and the soft limit is 80%. You need to configure this parameter if you want to deploy StarRocks with other memory-intensive services on a same server. * Introduced in: - ##### memory\_max\_alignment[​](#memory_max_alignment "Direct link to memory_max_alignment") * Default: 16 * Type: Int * Unit: Bytes * Is mutable: No * Description: Sets the maximum byte alignment that MemPool will accept for aligned allocations. Increase this value only when callers require larger alignment (for SIMD, device buffers, or ABI constraints). Larger values increase per-allocation padding and reserved memory waste and must remain within what the system allocator and platform support. * Introduced in: v3.2.0 ##### memory\_urgent\_level[​](#memory_urgent_level "Direct link to memory_urgent_level") * Default: 85 * Type: long * Unit: Percentage (0-100) * Is mutable: Yes * Description: The emergency memory water‑level expressed as a percentage of the process memory limit. When process memory consumption exceeds `(limit * memory_urgent_level / 100)`, BE triggers immediate memory reclamation, which forces data cache shrinkage, evicts update caches, and causes persistent/lake MemTables to be treated as "full" so they will be flushed/compacted soon. The code validates that this setting must be greater than `memory_high_level`, and `memory_high_level` must be greater or equal to `1`, and less thant or equal to `100`). A lower value causes more aggressive, earlier reclamation, that is, more frequent cache evictions and flushes. A higher value delays reclamation and risks OOM if too close to 100. Tune this item together with `memory_high_level` and Data Cache-related auto‑adjust settings. * Introduced in: v3.2.0 ##### net\_use\_ipv6\_when\_priority\_networks\_empty[​](#net_use_ipv6_when_priority_networks_empty "Direct link to net_use_ipv6_when_priority_networks_empty") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: A boolean value to control whether to use IPv6 addresses preferentially when `priority_networks` is not specified. `true` indicates to allow the system to use an IPv6 address preferentially when the server that hosts the node has both IPv4 and IPv6 addresses and `priority_networks` is not specified. * Introduced in: v3.3.0 ##### num\_cores[​](#num_cores "Direct link to num_cores") * Default: 0 * Type: Int * Unit: Cores * Is mutable: No * Description: Controls the number of CPU cores the system will use for CPU-aware decisions (for example, thread-pool sizing and runtime scheduling). A value of 0 enables auto-detection: the system reads `/proc/cpuinfo` and uses all available cores. If set to a positive integer, that value overrides the detected core count and becomes the effective core count. When running inside containers, cgroup cpuset or cpu quota settings can further restrict usable cores; `CpuInfo` also respects those cgroup limits. * Introduced in: v3.2.0 ##### plugin\_path[​](#plugin_path "Direct link to plugin_path") * Default: `${STARROCKS_HOME}/plugin` * Type: String * Unit: - * Is mutable: No * Description: Filesystem directory where StarRocks loads external plugins (dynamic libraries, connector artifacts, UDF binaries, etc.). `plugin_path` should point to a directory accessible by the BE process (read and execute permissions) and must exist before plugins are loaded. Ensure correct ownership and that plugin files use the platform's native binary extension (for example, .so on Linux). * Introduced in: v3.2.0 ##### priority\_networks[​](#priority_networks "Direct link to priority_networks") * Default: An empty string * Type: String * Unit: - * Is mutable: No * Description: Declares a selection strategy for servers that have multiple IP addresses. Note that at most one IP address must match the list specified by this parameter. The value of this parameter is a list that consists of entries, which are separated with semicolons (;) in CIDR notation, such as `10.10.10.0/24`. If no IP address matches the entries in this list, an available IP address of the server will be randomly selected. From v3.3.0, StarRocks supports deployment based on IPv6. If the server has both IPv4 and IPv6 addresses, and this parameter is not specified, the system uses an IPv4 address by default. You can change this behavior by setting `net_use_ipv6_when_priority_networks_empty` to `true`. * Introduced in: - ##### process\_force\_exit\_after\_crash\_handler\_hang\_second[​](#process_force_exit_after_crash_handler_hang_second "Direct link to process_force_exit_after_crash_handler_hang_second") * Default: 0 * Type: Int * Unit: Seconds * Is mutable: No * Description: If the fatal-signal (crash) handler hangs, for example a jemalloc deadlock while releasing resources before the core dump, the BE/CN forces the process to exit after this many seconds so that an orchestrator can restart it. The crash flag is still set first, so the FE keeps seeing `SHUTDOWN` heartbeats during the grace window; this only bounds how long a crashing process can linger while alive. Disabled by default (`0`) so that an upgrade keeps the existing crash and core dump behavior unchanged; set a positive value (and restart the node) to opt in, choosing a timeout suited to your environment. The value is read once at startup, and the watchdog thread is only launched when it is positive; a value of `0` or less keeps the watchdog disabled. * Introduced in: v4.0.15, v4.1.5 ##### rpc\_compress\_ratio\_threshold[​](#rpc_compress_ratio_threshold "Direct link to rpc_compress_ratio_threshold") * Default: 1.1 * Type: Double * Unit: - * Is mutable: Yes * Description: Threshold (uncompressed\_size / compressed\_size) used when deciding whether to send serialized row-batches over the network in compressed form. When compression is attempted (e.g., in DataStreamSender, exchange sink, tablet sink index channel, dictionary cache writer), StarRocks computes compress\_ratio = uncompressed\_size / compressed\_size; it uses the compressed payload only if compress\_ratio `>` rpc\_compress\_ratio\_threshold. With the default 1.1, compressed data must be at least ~9.1% smaller than uncompressed to be used. Lower the value to prefer compression (more CPU for smaller bandwidth savings); raise it to avoid compression overhead unless it yields larger size reductions. Note: this applies to RPC/shuffle serialization and is effective only when row-batch compression is enabled (compress\_rowbatches). * Introduced in: v3.2.0 ##### enable\_threadpool\_catch\_task\_exception[​](#enable_threadpool_catch_task_exception "Direct link to enable_threadpool_catch_task_exception") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether a ThreadPool worker swallows an exception thrown by a task and continues with the next task. When set to `false` (default), there is no catch clause enclosing the task body, so an exception that escapes the task finds no handler and terminates the BE process at the throw point. When set to `true`, the exception is logged at the ERROR level, the process-wide metric [`threadpool_task_exception_total`](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/t-z.md#threadpool_task_exception_total) is incremented, and the worker proceeds to the next task, which keeps the process alive. Note that keeping the worker alive does not make the task exception-safe: if a task signals its completion from a `DeferOp` or from its destructor while the statements that record its result are skipped, the waiter reads the task as successful, because an unset `Status` reads as OK. The failure then produces wrong results instead of an error. Set this item to `true` only to mitigate a crash loop, and expect the affected failures to become silent. * Introduced in: - ##### ssl\_private\_key\_path[​](#ssl_private_key_path "Direct link to ssl_private_key_path") * Default: An empty string * Type: String * Unit: - * Is mutable: No * Description: File system path to the TLS/SSL private key (PEM) that the BE's brpc server uses as the private key for the default certificate. When `enable_https` is set to `true`, the system sets `brpc::ServerOptions::ssl_options().default_cert.private_key` to this path at process start. The file must be accessible by the BE process and must match the certificate provided by `ssl_certificate_path`. If this value is not set or the file is missing or unaccessible, HTTPS will not be configured and the bRPC server may fail to start. Protect this file with restrictive filesystem permissions (for example, 600). * Introduced in: v4.0.0 ##### thrift\_client\_retry\_interval\_ms[​](#thrift_client_retry_interval_ms "Direct link to thrift_client_retry_interval_ms") * Default: 100 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The base retry interval for BE Thrift RPCs. Before each retry, the client waits a uniformly random duration from 0 to `min(1000, base * 2^retry_index)` milliseconds, where `base` is this value clamped to \[0, 1000] and `retry_index` starts at 0. The value is read once per RPC. Zero or negative values disable the wait. No wait is added after the final failed attempt. * Introduced in: - ##### thrift\_connect\_timeout\_seconds[​](#thrift_connect_timeout_seconds "Direct link to thrift_connect_timeout_seconds") * Default: 3 * Type: Int * Unit: Seconds * Is mutable: No * Description: Connection timeout (in seconds) used when creating Thrift clients. ClientCacheHelper::\_create\_client multiplies this value by 1000 and passes it to ThriftClientImpl::set\_conn\_timeout(), so it controls the TCP/connect handshake timeout for new Thrift connections opened by the BE client cache. This setting affects only connection establishment; send/receive timeouts are configured separately. Very small values can cause spurious connection failures on high-latency networks, while large values delay detection of unreachable peers. * Introduced in: v3.2.0 ##### thrift\_port[​](#thrift_port "Direct link to thrift_port") * Default: 0 * Type: Int * Unit: - * Is mutable: No * Description: Port used to export the internal Thrift-based BackendService. When the process runs as a Compute Node and this item is set to a non-zero value, it overrides `be_port` and the Thrift server binds to this value; otherwise `be_port` is used. This configuration is deprecated — setting a non-zero `thrift_port` logs a warning advising to use `be_port` instead. * Introduced in: v3.2.0 ##### thrift\_rpc\_connection\_max\_valid\_time\_ms[​](#thrift_rpc_connection_max_valid_time_ms "Direct link to thrift_rpc_connection_max_valid_time_ms") * Default: 5000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: Maximum valid time for a thrift RPC connection. A connection will be closed if it has existed in the connection pool for longer than this value. It must be set consistent with FE configuration `thrift_client_timeout_ms`. * Introduced in: - ##### thrift\_rpc\_max\_body\_size[​](#thrift_rpc_max_body_size "Direct link to thrift_rpc_max_body_size") * Default: 0 * Type: Int * Unit: * Is mutable: No * Description: The maximum string body size of RPC. `0` indicates the size is unlimited. * Introduced in: - ##### thrift\_rpc\_strict\_mode[​](#thrift_rpc_strict_mode "Direct link to thrift_rpc_strict_mode") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether thrift's strict execution mode is enabled. For more information on thrift strict mode, see [Thrift Binary protocol encoding](https://github.com/apache/thrift/blob/master/doc/specs/thrift-binary-protocol.md). * Introduced in: - ##### thrift\_rpc\_timeout\_ms[​](#thrift_rpc_timeout_ms "Direct link to thrift_rpc_timeout_ms") * Default: 5000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The timeout for a thrift RPC. * Introduced in: - ##### transaction\_apply\_thread\_pool\_num\_min[​](#transaction_apply_thread_pool_num_min "Direct link to transaction_apply_thread_pool_num_min") * Default: 0 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Sets the minimum number of threads for the "update\_apply" thread pool in BE's UpdateManager — the pool that applies rowsets for primary-key tables. A value of 0 disables a fixed minimum (no enforced lower bound); when transaction\_apply\_worker\_count is also 0 the pool's max threads defaults to the number of CPU cores, so effective worker capacity equals CPU cores. You can raise this to guarantee a baseline concurrency for applying transactions; setting it too high may increase CPU contention. Changes are applied at runtime via the update\_config HTTP handler (it calls update\_min\_threads on the apply thread pool). * Introduced in: v3.2.11 ##### transaction\_publish\_version\_thread\_pool\_num\_min[​](#transaction_publish_version_thread_pool_num_min "Direct link to transaction_publish_version_thread_pool_num_min") * Default: 0 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Sets the minimum number of threads reserved in the AgentServer "publish\_version" dynamic thread pool (used to publish transaction versions / handle TTaskType::PUBLISH\_VERSION tasks). At startup the pool is created with min = max(config value, MIN\_TRANSACTION\_PUBLISH\_WORKER\_COUNT) (MIN\_TRANSACTION\_PUBLISH\_WORKER\_COUNT = 1), so the default 0 results in a minimum of 1 thread. Changing this value at runtime invokes the update callback to call ThreadPool::update\_min\_threads, raising or lowering the pool's guaranteed minimum (but not below the enforced minimum of 1). Coordinate with transaction\_publish\_version\_worker\_count (max threads) and transaction\_publish\_version\_thread\_pool\_idle\_time\_ms (idle timeout). * Introduced in: v3.2.11 ##### use\_mmap\_allocate\_chunk[​](#use_mmap_allocate_chunk "Direct link to use_mmap_allocate_chunk") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: When this item is set to `true`, the system allocates chunks using anonymous private mmap mappings (MAP\_ANONYMOUS | MAP\_PRIVATE) and frees them with munmap. Enabling this may create many virtual memory mappings, thus you must raise the kernel limit (as root user, running `sysctl -w vm.max_map_count=262144` or `echo 262144 > /proc/sys/vm/max_map_count`), and set `chunk_reserved_bytes_limit` to a relatively large value. Otherwise, enabling mmap can cause very poor performance due to frequent mapping/unmapping. * Introduced in: v3.2.0 ##### ssl\_certificate\_path[​](#ssl_certificate_path "Direct link to ssl_certificate_path") * Default: * Type: String * Unit: - * Is mutable: No * Description: Absolute path to the TLS/SSL certificate file (PEM) that the BE's brpc server will use when enable\_https is true. At BE startup this value is copied into `brpc::ServerOptions::ssl_options().default_cert.certificate`; you must also set `ssl_private_key_path` to the matching private key. Provide the server certificate and any intermediate certificates in PEM format (certificate chain) if required by your CA. The file must be readable by the StarRocks BE process and is applied only at startup. If unset or invalid while enable\_https is enabled, brpc TLS setup may fail and prevent the server from starting correctly. * Introduced in: v4.0.0 #### Metadata and cluster management[​](#metadata-and-cluster-management "Direct link to Metadata and cluster management") ##### cluster\_id[​](#cluster_id "Direct link to cluster_id") * Default: -1 * Type: Int * Unit: - * Is mutable: No * Description: Global cluster identifier for this StarRocks backend. At startup StorageEngine reads config::cluster\_id into its effective cluster id and verifies that all data root paths contain the same cluster id (see StorageEngine::\_check\_all\_root\_path\_cluster\_id). A value of -1 means "unset" — the engine may derive the effective id from existing data directories or from master heartbeats. If a non‑negative id is configured, any mismatch between configured id and ids stored in data directories will cause startup verification to fail (Status::Corruption). When some roots lack an id and the engine is allowed to write ids (options.need\_write\_cluster\_id), it will persist the effective id into those roots. * Introduced in: v3.2.0 ##### consistency\_max\_memory\_limit[​](#consistency_max_memory_limit "Direct link to consistency_max_memory_limit") * Default: 10G * Type: String * Unit: - * Is mutable: No * Description: Memory size specification for the CONSISTENCY memory tracker. * Introduced in: v3.2.0 ##### make\_snapshot\_rpc\_timeout\_ms[​](#make_snapshot_rpc_timeout_ms "Direct link to make_snapshot_rpc_timeout_ms") * Default: 20000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: Sets the Thrift RPC timeout in milliseconds used when making a snapshot on a remote BE. Increase this value when remote snapshot creation regularly exceeds the default timeout; reduce it to fail faster on unresponsive BEs. Note other timeouts may affect end-to-end operations (for example the effective tablet-writer open timeout can relate to `tablet_writer_open_rpc_timeout_sec` and `load_timeout_sec`). * Introduced in: v3.2.0 ##### metadata\_cache\_memory\_limit\_percent[​](#metadata_cache_memory_limit_percent "Direct link to metadata_cache_memory_limit_percent") * Default: 30 * Type: Int * Unit: Percent * Is mutable: Yes * Description: Sets the metadata LRU cache size as a percentage of the process memory limit. At startup StarRocks computes cache bytes as (process\_mem\_limit \* metadata\_cache\_memory\_limit\_percent / 100) and passes that to the metadata cache allocator. The cache is only used for non-PRIMARY\_KEYS rowsets (PK tables are not supported) and is enabled only when `metadata_cache_memory_limit_percent > 0`; set it to `<= 0` to disable the metadata cache. Increasing this value raises metadata cache capacity but reduces memory available to other components; tune based on workload and system memory. Not active in BE\_TEST builds. * Introduced in: v3.2.10 ##### retry\_apply\_interval\_second[​](#retry_apply_interval_second "Direct link to retry_apply_interval_second") * Default: 30 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Base interval (in seconds) used when scheduling retries of failed tablet apply operations. It is used directly to schedule a retry after a submission failure and as the base multiplier for backoff: the next retry delay is calculated as min(600, `retry_apply_interval_second` \* failed\_attempts). The code also uses `retry_apply_interval_second` to compute the cumulative retry duration (arithmetic-series sum) which is compared against `retry_apply_timeout_second` to decide whether to keep retrying. Effective only when `enable_retry_apply` is true. Increasing this value lengthens both individual retry delays and the cumulative time spent retrying; decreasing it makes retries more frequent and may increase the number of attempts before reaching `retry_apply_timeout_second`. * Introduced in: v3.2.9 ##### retry\_apply\_timeout\_second[​](#retry_apply_timeout_second "Direct link to retry_apply_timeout_second") * Default: 7200 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Maximum cumulative retry time (in seconds) allowed for applying a pending version before the apply process gives up and the tablet enters an error state. The apply logic accumulates exponential/backoff intervals based on `retry_apply_interval_second` and compares the total duration against `retry_apply_timeout_second`. If `enable_retry_apply` is true and the error is considered retryable, apply attempts will be rescheduled until the accumulated backoff exceeds `retry_apply_timeout_second`; then apply stops and the tablet transitions to error. Explicitly non-retryable errors (e.g., Corruption) are not retried regardless of this setting. Tune this value to control how long StarRocks will keep retrying apply operations (default 7200s = 2 hours). * Introduced in: v3.3.13, v3.4.3, v3.5.0 ##### stream\_load\_thrift\_rpc\_timeout\_ms[​](#stream_load_thrift_rpc_timeout_ms "Direct link to stream_load_thrift_rpc_timeout_ms") * Default: 60000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Maximum allowed lifetime (in milliseconds) for Thrift RPC connections used by BE stream-load and transaction commit calls. StarRocks sets this value as the `thrift_rpc_timeout_ms` on requests sent to FE (used in stream\_load planning, loadTxnBegin/loadTxnPrepare/loadTxnCommit, and getLoadTxnStatus). If a connection has been pooled longer than this value it will be closed. When a per-request timeout (`ctx->timeout_second`) is provided, the BE computes the RPC timeout as rpc\_timeout\_ms = max(ctx*1000/4, min(ctx*1000/2, stream\_load\_thrift\_rpc\_timeout\_ms)), so the effective RPC timeout is bounded by the context and this configuration. Keep this consistent with FE's `thrift_client_timeout_ms` to avoid mismatched timeouts. The legacy name `txn_commit_rpc_timeout_ms` is still accepted as a backward-compatible alias. * Introduced in: v3.2.0 ##### txn\_map\_shard\_size[​](#txn_map_shard_size "Direct link to txn_map_shard_size") * Default: 128 * Type: Int * Unit: - * Is mutable: No * Description: Number of lock-map shards used by the transaction manager to partition transaction locks and reduce contention. Its value should be a power of two (2^n); increasing it augments concurrency and reduces lock contention at the cost of additional memory and marginal bookkeeping overhead. Choose a shard count sized for expected concurrent transactions and available memory. * Introduced in: v3.2.0 ##### txn\_shard\_size[​](#txn_shard_size "Direct link to txn_shard_size") * Default: 1024 * Type: Int * Unit: - * Is mutable: No * Description: Controls the number of lock shards used by the transaction manager. This value determines the shard size for txn locks. It must be a power of two; Setting it to a larger value reduces lock contention and improves concurrent COMMIT/PUBLISH throughput at the expense of additional memory and finer-grained internal bookkeeping. * Introduced in: v3.2.0 ##### update\_schema\_worker\_count[​](#update_schema_worker_count "Direct link to update_schema_worker_count") * Default: 3 * Type: Int * Unit: Threads * Is mutable: No * Description: Sets the maximum number of worker threads in the backend's "update\_schema" dynamic ThreadPool that processes TTaskType::UPDATE\_SCHEMA tasks. The ThreadPool is created in agent\_server during startup with a minimum of 0 threads (it can scale down to zero when idle) and a max equal to this setting; the pool uses the default idle timeout and an effectively unlimited queue. Increase this value to allow more concurrent schema-update tasks (higher CPU and memory usage), or lower it to limit parallel schema operations. * Introduced in: v3.2.3 ##### update\_tablet\_meta\_info\_worker\_count[​](#update_tablet_meta_info_worker_count "Direct link to update_tablet_meta_info_worker_count") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: Sets the maximum number of worker threads in the backend thread pool that handles tablet metadata update tasks. The thread pool is created during backend startup with a minimum of 0 threads (it can scale down to zero when idle) and a max equal to this setting (clamped to at least 1). Updating this value at runtime adjusts the pool's max threads. Increase it to allow more concurrent metadata-update tasks, or lower it to limit concurrency. * Introduced in: v4.1.0, v4.0.6, v3.5.13 --- ### BE Configuration - Query and Loading Some BE configuration items are dynamic parameters which you can set interactively when BE nodes are online. The rest of them are static parameters. You can only set the static parameters of a BE node by changing them in the corresponding configuration file **be.conf** and restarting the BE node to allow the change to take effect. #### View BE configuration items[​](#view-be-configuration-items "Direct link to View BE configuration items") You can view the BE configuration items using the following command: ```sql SELECT * FROM information_schema.be_configs [WHERE NAME LIKE "%%"] ``` #### Configure BE parameters[​](#configure-be-parameters "Direct link to Configure BE parameters") ##### Configure BE dynamic parameters[​](#configure-be-dynamic-parameters "Direct link to Configure BE dynamic parameters") You can configure a dynamic parameter of a BE node by updating the value in `information_schema.be_configs`. warning Setting an invalid value may cause unknown behaviors. Check twice before you run the command to update the configuration. ```sql -- Replace with the key of the configuration and with the value. UPDATE information_schema.be_configs SET VALUE = WHERE name = ""; ``` ##### Configure BE static parameters[​](#configure-be-static-parameters "Direct link to Configure BE static parameters") You can only set the static parameters of a BE by changing them in the corresponding configuration file **be.conf**, and restarting the BE to allow the changes to take effect. *** This topic introduces the following types of BE configurations: * [Query](#query) * [Loading and unloading](#loading-and-unloading) #### Query[​](#query "Direct link to Query") ##### agg\_hash\_map\_prefetch\_dist[​](#agg_hash_map_prefetch_dist "Direct link to agg_hash_map_prefetch_dist") * Default: 16 * Type: Int * Unit: Rows * Is mutable: Yes * Description: Software prefetch distance (in rows) for the aggregation hash-map / hash-set probe loop. While building the aggregation hash table, the loop prefetches the bucket for the row this many positions ahead of the one it is currently processing, hiding memory latency on large tables. Setting it to `0` disables software prefetch. The value is read once per chunk, so changes take effect on the next chunk. The default 16 is empirical for L3-resident tables; raise it for DRAM-resident workloads and lower it for cache-resident ones. Prefetch is additionally gated by `agg_prefetch_l2_ratio`: regardless of this distance, no prefetch is issued while the hash table still fits in L2. * Introduced in: - ##### agg\_prefetch\_l2\_ratio[​](#agg_prefetch_l2_ratio "Direct link to agg_prefetch_l2_ratio") * Default: 1.0 * Type: Double * Unit: - * Is mutable: Yes * Description: Gates aggregation hash-table software prefetch on L2 residency. Prefetch is enabled only once the bucket array spills L2, that is, when `bucket_count * slot_bytes >= L2_size * agg_prefetch_l2_ratio`, where the L2 size is detected at runtime (falling back to 1 MiB if detection fails). Below this point the table is L2-resident and prefetching is a net loss. Lower the ratio on contended deployments that run many drivers per core, where the effective per-table share of L2 is smaller than the nominal per-core size; raising it above 1.0 delays prefetch until the table is well past L2. See also `agg_hash_map_prefetch_dist`. * Introduced in: - ##### clear\_udf\_cache\_when\_start[​](#clear_udf_cache_when_start "Direct link to clear_udf_cache_when_start") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: When enabled, the BE's UserFunctionCache will clear all locally cached user function libraries on startup. During UserFunctionCache::init, the code calls \_reset\_cache\_dir(), which removes UDF files from the configured UDF library directory (organized into kLibShardNum subdirectories) and deletes files with Java/Python UDF suffixes (.jar/.py). When disabled (default), the BE loads existing cached UDF files instead of deleting them. Enabling this forces UDF binaries to be re-downloaded on first use after restart (increasing network traffic and first-use latency). * Introduced in: v4.0.0 ##### dictionary\_speculate\_min\_chunk\_size[​](#dictionary_speculate_min_chunk_size "Direct link to dictionary_speculate_min_chunk_size") * Default: 10000 * Type: Int * Unit: Rows * Is mutable: No * Description: Minimum number of rows (chunk size) used by StringColumnWriter and DictColumnWriter to trigger dictionary-encoding speculation. If an incoming column (or the accumulated buffer plus incoming rows) has size larger than or equal `dictionary_speculate_min_chunk_size` the writer will run speculation immediately and set an encoding (DICT, PLAIN or BIT\_SHUFFLE) rather than buffering more rows. Speculation uses `dictionary_encoding_ratio` for string columns and `dictionary_encoding_ratio_for_non_string_column` for numeric/non-string columns to decide whether dictionary encoding is beneficial. Also, a large column byte\_size (larger than or equal to UINT32\_MAX) forces immediate speculation to avoid `BinaryColumn` overflow. * Introduced in: v3.2.0 ##### disable\_storage\_page\_cache[​](#disable_storage_page_cache "Direct link to disable_storage_page_cache") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: A boolean value to control whether to disable PageCache. * When PageCache is enabled, StarRocks caches the recently scanned data. * PageCache can significantly improve the query performance when similar queries are repeated frequently. * `true` indicates disabling PageCache. * The default value of this item has been changed from `true` to `false` since StarRocks v2.4. * Introduced in: - ##### enable\_bitmap\_index\_memory\_page\_cache[​](#enable_bitmap_index_memory_page_cache "Direct link to enable_bitmap_index_memory_page_cache") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable memory cache for Bitmap index. Memory cache is recommended if you want to use Bitmap indexes to accelerate point queries. * Introduced in: v3.1 ##### enable\_compaction\_flat\_json[​](#enable_compaction_flat_json "Direct link to enable_compaction_flat_json") * Default: True * Type: Boolean * Unit: * Is mutable: Yes * Description: Whether to enable compaction for Flat JSON data. * Introduced in: v3.3.3 ##### enable\_json\_flat[​](#enable_json_flat "Direct link to enable_json_flat") * Default: false * Type: Boolean * Unit: * Is mutable: Yes * Description: Whether to enable the Flat JSON feature. After this feature is enabled, newly loaded JSON data will be automatically flattened, improving JSON query performance. * Introduced in: v3.3.0 ##### enable\_lazy\_dynamic\_flat\_json[​](#enable_lazy_dynamic_flat_json "Direct link to enable_lazy_dynamic_flat_json") * Default: True * Type: Boolean * Unit: * Is mutable: Yes * Description: Whether to enable Lazy Dyamic Flat JSON when a query misses Flat JSON schema in read process. When this item is set to `true`, StarRocks will postpone the Flat JSON operation to calculation process instead of read process. * Introduced in: v3.3.3 ##### enable\_ordinal\_index\_memory\_page\_cache[​](#enable_ordinal_index_memory_page_cache "Direct link to enable_ordinal_index_memory_page_cache") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable memory cache for ordinal index. Ordinal index is a mapping from row IDs to data page positions, and it can be used to accelerate scans. * Introduced in: - ##### enable\_string\_prefix\_zonemap[​](#enable_string_prefix_zonemap "Direct link to enable_string_prefix_zonemap") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable ZoneMap for string (CHAR/VARCHAR) columns using prefix-based min/max. For non-key string columns, the min/max values are truncated to a fixed prefix length configured by `string_prefix_zonemap_prefix_len`. * Introduced in: - ##### enable\_zonemap\_index\_memory\_page\_cache[​](#enable_zonemap_index_memory_page_cache "Direct link to enable_zonemap_index_memory_page_cache") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable memory cache for zonemap index. Memory cache is recommended if you want to use zonemap indexes to accelerate scan. * Introduced in: - ##### exchg\_node\_buffer\_size\_bytes[​](#exchg_node_buffer_size_bytes "Direct link to exchg_node_buffer_size_bytes") * Default: 10485760 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The maximum buffer size on the receiver end of an exchange node for each query. This configuration item is a soft limit. A backpressure is triggered when data is sent to the receiver end with an excessive speed. * Introduced in: - ##### exec\_state\_report\_max\_threads[​](#exec_state_report_max_threads "Direct link to exec_state_report_max_threads") * Default: 2 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Maximum number of threads for the exec-state-report thread pool. This pool is used by `ExecStateReporter` to asynchronously send non-priority execution status reports (such as fragment completion and error status) from BE to FE via RPC. The actual pool size at startup is `max(1, exec_state_report_max_threads)`. Changing this config at runtime triggers `update_max_threads` on the pool in every executor set (shared and exclusive). The pool has a fixed task queue size of 1000; report submissions are silently dropped when all threads are busy and the queue is full. Paired with `priority_exec_state_report_max_threads` for the high-priority pool. Increase this value when delayed or dropped exec-state reports are observed under high query concurrency. * Introduced in: v4.1.0, v4.0.8, v3.5.15 ##### file\_descriptor\_cache\_capacity[​](#file_descriptor_cache_capacity "Direct link to file_descriptor_cache_capacity") * Default: 16384 * Type: Int * Unit: - * Is mutable: No * Description: The number of file descriptors that can be cached. * Introduced in: - ##### flamegraph\_tool\_dir[​](#flamegraph_tool_dir "Direct link to flamegraph_tool_dir") * Default: `${STARROCKS_HOME}/bin/flamegraph` * Type: String * Unit: - * Is mutable: No * Description: Directory of the flamegraph tool, which should contain pprof, stackcollapse-go.pl, and flamegraph.pl scripts for generating flame graphs from profile data. * Introduced in: - ##### fragment\_pool\_queue\_size[​](#fragment_pool_queue_size "Direct link to fragment_pool_queue_size") * Default: 2048 * Type: Int * Unit: - * Is mutable: No * Description: The upper limit of the query number that can be processed on each BE node. * Introduced in: - ##### fragment\_pool\_thread\_num\_max[​](#fragment_pool_thread_num_max "Direct link to fragment_pool_thread_num_max") * Default: 4096 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of threads used for query. * Introduced in: - ##### fragment\_pool\_thread\_num\_min[​](#fragment_pool_thread_num_min "Direct link to fragment_pool_thread_num_min") * Default: 64 * Type: Int * Unit: Minutes - * Is mutable: No * Description: The minimum number of threads used for query. * Introduced in: - ##### hdfs\_client\_enable\_hedged\_read[​](#hdfs_client_enable_hedged_read "Direct link to hdfs_client_enable_hedged_read") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Specifies whether to enable the hedged read feature. * Introduced in: v3.0 ##### hdfs\_client\_hedged\_read\_threadpool\_size[​](#hdfs_client_hedged_read_threadpool_size "Direct link to hdfs_client_hedged_read_threadpool_size") * Default: 128 * Type: Int * Unit: - * Is mutable: No * Description: Specifies the size of the Hedged Read thread pool on your HDFS client. The thread pool size limits the number of threads to dedicate to the running of hedged reads in your HDFS client. It is equivalent to the `dfs.client.hedged.read.threadpool.size` parameter in the **hdfs-site.xml** file of your HDFS cluster. * Introduced in: v3.0 ##### hdfs\_client\_hedged\_read\_threshold\_millis[​](#hdfs_client_hedged_read_threshold_millis "Direct link to hdfs_client_hedged_read_threshold_millis") * Default: 2500 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: Specifies the number of milliseconds to wait before starting up a hedged read. For example, you have set this parameter to `30`. In this situation, if a read from a block has not returned within 30 milliseconds, your HDFS client immediately starts up a new read against a different block replica. It is equivalent to the `dfs.client.hedged.read.threshold.millis` parameter in the **hdfs-site.xml** file of your HDFS cluster. * Introduced in: v3.0 ##### io\_coalesce\_adaptive\_lazy\_active[​](#io_coalesce_adaptive_lazy_active "Direct link to io_coalesce_adaptive_lazy_active") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Based on the selectivity of predicates, adaptively determines whether to combine the I/O of predicate columns and non-predicate columns. * Introduced in: v3.2 ##### jit\_lru\_cache\_size[​](#jit_lru_cache_size "Direct link to jit_lru_cache_size") * Default: 0 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The LRU cache size for JIT compilation. It represents the actual size of the cache if it is set to greater than 0. If it is set to less than or equal to 0, the system will adaptively set the cache using the formula `jit_lru_cache_size = min(mem_limit*0.01, 1GB)` (while `mem_limit` of the node must be greater or equal to 16 GB). * Introduced in: - ##### json\_flat\_column\_max[​](#json_flat_column_max "Direct link to json_flat_column_max") * Default: 100 * Type: Int * Unit: * Is mutable: Yes * Description: The maximum number of sub-fields that can be extracted by Flat JSON. This parameter takes effect only when `enable_json_flat` is set to `true`. * Introduced in: v3.3.0 ##### json\_flat\_create\_zonemap[​](#json_flat_create_zonemap "Direct link to json_flat_create_zonemap") * Default: true * Type: Boolean * Unit: * Is mutable: Yes * Description: Whether to create ZoneMaps for flattened JSON sub-columns during write. This parameter takes effect only when `enable_json_flat` is set to `true`. * Introduced in: - ##### json\_flat\_null\_factor[​](#json_flat_null_factor "Direct link to json_flat_null_factor") * Default: 0.3 * Type: Double * Unit: * Is mutable: Yes * Description: The proportion of NULL values in the column to extract for Flat JSON. A column will not be extracted if its proportion of NULL value is higher than this threshold. This parameter takes effect only when `enable_json_flat` is set to `true`. * Introduced in: v3.3.0 ##### json\_flat\_sparsity\_factor[​](#json_flat_sparsity_factor "Direct link to json_flat_sparsity_factor") * Default: 0.3 * Type: Double * Unit: * Is mutable: Yes * Description: The proportion of columns with the same name for Flat JSON. Extraction is not performed if the proportion of columns with the same name is lower than this value. This parameter takes effect only when `enable_json_flat` is set to `true`. * Introduced in: v3.3.0 ##### lake\_tablet\_ignore\_invalid\_delete\_predicate[​](#lake_tablet_ignore_invalid_delete_predicate "Direct link to lake_tablet_ignore_invalid_delete_predicate") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: A boolean value to control whether ignore invalid delete predicates in tablet rowset metadata which may be introduced by logic deletion to a duplicate key table after the column name renamed. * Introduced in: v4.0 ##### late\_materialization\_ratio[​](#late_materialization_ratio "Direct link to late_materialization_ratio") * Default: 10 * Type: Int * Unit: - * Is mutable: No * Description: Integer ratio in range \[0-1000] that controls the use of late materialization in the SegmentIterator (vector query engine). A value of `0` (or ≤ 0) disables late materialization; `1000` (or ≥ 1000) forces late materialization for all reads. Values > 0 and < 1000 enable a conditional strategy where both late and early materialization contexts are prepared and the iterator selects behavior based on predicate filter ratios (higher values favor late materialization). When a segment contains complex metric types, StarRocks uses `metric_late_materialization_ratio` instead. If `lake_io_opts.cache_file_only` is set, late materialization is disabled. * Introduced in: v3.2.0 ##### max\_hdfs\_file\_handle[​](#max_hdfs_file_handle "Direct link to max_hdfs_file_handle") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of HDFS file descriptors that can be opened. * Introduced in: - ##### max\_hdfs\_scanner\_num[​](#max_hdfs_scanner_num "Direct link to max_hdfs_scanner_num") * Default: 50 * Type: Int * Unit: - * Is mutable: No * Description: Maximum number of concurrent remote scanners (HDFS, object storage, etc.) that ConnectorScanNode can run simultaneously. This value caps estimated concurrency at startup and also limits pending-scanner scheduling at runtime, controlling thread, memory, and file-handle pressure. * Introduced in: v3.2.0 ##### max\_memory\_sink\_batch\_count[​](#max_memory_sink_batch_count "Direct link to max_memory_sink_batch_count") * Default: 20 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of Scan Cache batches. * Introduced in: - ##### max\_pushdown\_conditions\_per\_column[​](#max_pushdown_conditions_per_column "Direct link to max_pushdown_conditions_per_column") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of conditions that allow pushdown in each column. If the number of conditions exceeds this limit, the predicates are not pushed down to the storage layer. * Introduced in: - ##### max\_scan\_key\_num[​](#max_scan_key_num "Direct link to max_scan_key_num") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of scan keys segmented by each query. * Introduced in: - ##### metric\_late\_materialization\_ratio[​](#metric_late_materialization_ratio "Direct link to metric_late_materialization_ratio") * Default: 1000 * Type: Int * Unit: - * Is mutable: No * Description: Controls when the late-materialization row access strategy is used for reads that include complex metric columns. Valid range: \[0-1000]. `0` disables late materialization; `1000` forces late materialization for all applicable reads. Values 1–999 enable a conditional strategy where both late and early materialization contexts are prepared and chosen at runtime based on predicate/selectivity. When complex metric types exist, `metric_late_materialization_ratio` overrides the general `late_materialization_ratio`. Note: `cache_file_only` I/O mode will cause late materialization to be disabled regardless of this setting. * Introduced in: v3.2.0 ##### min\_file\_descriptor\_number[​](#min_file_descriptor_number "Direct link to min_file_descriptor_number") * Default: 60000 * Type: Int * Unit: - * Is mutable: No * Description: The minimum number of file descriptors in the BE process. * Introduced in: - ##### object\_storage\_client\_cache\_size[​](#object_storage_client_cache_size "Direct link to object_storage_client_cache_size") * Default: 8 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of object storage clients (S3-compatible and Azure Blob) cached per client factory. The value is read on each client creation, so lowering it takes effect gradually as cached clients are evicted during subsequent creations. Values below `1` are treated as `1`. * Introduced in: v4.1.4, v4.0.14 ##### object\_storage\_connect\_timeout\_ms[​](#object_storage_connect_timeout_ms "Direct link to object_storage_connect_timeout_ms") * Default: -1 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: Timeout duration to establish socket connections with object storage. `-1` indicates to use the default timeout duration of the SDK configurations. * Introduced in: v3.0.9 ##### enable\_poco\_client\_for\_aws\_sdk[​](#enable_poco_client_for_aws_sdk "Direct link to enable_poco_client_for_aws_sdk") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to use the Poco HTTP client for the AWS SDK. `true` replaces the AWS SDK's default curl HTTP client with Poco. `false` uses the default curl client. * Introduced in: - ##### object\_storage\_request\_timeout\_ms[​](#object_storage_request_timeout_ms "Direct link to object_storage_request_timeout_ms") * Default: 10000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: How long a request to object storage may stay stalled before the client aborts and retries. This is not a deadline on the request: for the curl client it is the low speed time, the time a transfer may stay below 1 byte/s, and for the Poco client it is the socket send/receive timeout. A transfer that keeps making progress is never cut off, however long it runs. `0` disables the check; a negative value leaves the client on its own default, which on the Poco path is 60 seconds. * Introduced in: v3.0.9 ##### parquet\_late\_materialization\_enable[​](#parquet_late_materialization_enable "Direct link to parquet_late_materialization_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: A boolean value to control whether to enable the late materialization of Parquet reader to improve performance. `true` indicates enabling late materialization, and `false` indicates disabling it. * Introduced in: - ##### parquet\_page\_index\_enable[​](#parquet_page_index_enable "Direct link to parquet_page_index_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: A boolean value to control whether to enable the pageindex of Parquet file to improve performance. `true` indicates enabling pageindex, and `false` indicates disabling it. * Introduced in: v3.3 ##### parquet\_reader\_bloom\_filter\_enable[​](#parquet_reader_bloom_filter_enable "Direct link to parquet_reader_bloom_filter_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: A boolean value to control whether to enable the bloom filter of Parquet file to improve performance. `true` indicates enabling the bloom filter, and `false` indicates disabling it. You can also control this behavior on session level using the system variable `enable_parquet_reader_bloom_filter`. Bloom filters in Parquet are maintained **at the column level within each row group**. If a Parquet file contains bloom filters for certain columns, queries can use predicates on those columns to efficiently skip row groups. * Introduced in: v3.5 ##### path\_gc\_check\_step[​](#path_gc_check_step "Direct link to path_gc_check_step") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of files that can be scanned continuously each time. * Introduced in: - ##### path\_gc\_check\_step\_interval\_ms[​](#path_gc_check_step_interval_ms "Direct link to path_gc_check_step_interval_ms") * Default: 10 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The time interval between file scans. * Introduced in: - ##### path\_scan\_interval\_second[​](#path_scan_interval_second "Direct link to path_scan_interval_second") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which GC cleans expired data. * Introduced in: - ##### pipeline\_connector\_scan\_thread\_num\_per\_cpu[​](#pipeline_connector_scan_thread_num_per_cpu "Direct link to pipeline_connector_scan_thread_num_per_cpu") * Default: 8 * Type: Double * Unit: - * Is mutable: Yes * Description: The number of scan threads assigned to Pipeline Connector per CPU core in the BE node. This configuration is changed to dynamic from v3.1.7 onwards. * Introduced in: - ##### pipeline\_enable\_large\_column\_checker[​](#pipeline_enable_large_column_checker "Direct link to pipeline_enable_large_column_checker") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable large column detection in the pipeline execution framework. When enabled, queries fail with a capacity limit error if an intermediate column reaches the chunk capacity limit in pipeline execution or spill serialization. * Introduced in: v4.0.0 ##### pipeline\_poller\_timeout\_guard\_ms[​](#pipeline_poller_timeout_guard_ms "Direct link to pipeline_poller_timeout_guard_ms") * Default: -1 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: When this item is set to greater than `0`, if a driver takes longer than `pipeline_poller_timeout_guard_ms` for a single dispatch in the poller, then the information of the driver and operator is printed. * Introduced in: - ##### load\_sink\_close\_thread\_pool\_num\_max[​](#load_sink_close_thread_pool_num_max "Direct link to load_sink_close_thread_pool_num_max") * Default: 64 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads that run the final close of load sinks. In a shared-data cluster this close writes the combined transaction log to object storage; running it in this pool keeps a slow write from blocking the pipeline poller and the pipeline execution threads. When all threads are busy, further closes wait in the queue of the pool, which has no size limit, so no close is dropped. Values must be greater than 0. This limit can be changed at runtime without restarting the BE or CN. Lowering the limit does not interrupt closes already running. One worker is prestarted and retained while idle; additional workers can exit after 500 ms of idle time. * Introduced in: - ##### pipeline\_prepare\_thread\_pool\_queue\_size[​](#pipeline_prepare_thread_pool_queue_size "Direct link to pipeline_prepare_thread_pool_queue_size") * Default: 102400 * Type: Int * Unit: - * Is mutable: No * Description: The maximum queue lenggth of PREPARE fragment thread pool for Pipeline execution engine. * Introduced in: - ##### pipeline\_prepare\_thread\_pool\_thread\_num[​](#pipeline_prepare_thread_pool_thread_num "Direct link to pipeline_prepare_thread_pool_thread_num") * Default: 0 * Type: Int * Unit: - * Is mutable: No * Description: Number of threads in the pipeline execution engine PREPARE fragment thread pool. `0` indicates the value is equal to the number of system VCPU core number. * Introduced in: - ##### pipeline\_prepare\_timeout\_guard\_ms[​](#pipeline_prepare_timeout_guard_ms "Direct link to pipeline_prepare_timeout_guard_ms") * Default: -1 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: When this item is set to greater than `0`, if a plan fragment exceeds `pipeline_prepare_timeout_guard_ms` during the PREPARE process, a stack trace of the plan fragment is printed. * Introduced in: - ##### pipeline\_scan\_thread\_pool\_queue\_size[​](#pipeline_scan_thread_pool_queue_size "Direct link to pipeline_scan_thread_pool_queue_size") * Default: 102400 * Type: Int * Unit: - * Is mutable: No * Description: The maximum task queue length of SCAN thread pool for Pipeline execution engine. * Introduced in: - ##### case\_when\_selective\_eval\_ratio[​](#case_when_selective_eval_ratio "Direct link to case_when_selective_eval_ratio") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls whether each `THEN` branch of a searched `CASE WHEN` is evaluated only on the rows that branch owns, instead of being evaluated over the whole chunk and having its rows picked afterwards. It only applies when the `CASE` returns a collection (ARRAY/MAP/STRUCT) type, where materializing a row is expensive enough to pay for compacting the branch's input rows into a sub-chunk. A branch is compacted only when `owned_rows * case_when_selective_eval_ratio < chunk_rows`, that is, when the branch owns less than `1 / case_when_selective_eval_ratio` of the chunk; a branch above the threshold is still evaluated over the whole chunk, because compacting its input would copy more than the skipped evaluation saves. Setting this item to `1` compacts every branch that does not own the whole chunk. Setting it to `0` or a negative value turns the optimization off entirely and restores the previous behavior, including the behavior change it carries: a `THEN` or `ELSE` that would raise an error is no longer evaluated when no row selects it. * Introduced in: - ##### pk\_index\_parallel\_get\_threadpool\_size[​](#pk_index_parallel_get_threadpool_size "Direct link to pk_index_parallel_get_threadpool_size") * Default: 1048576 * Type: Int * Unit: - * Is mutable: Yes * Description: Sets the maximum queue size (number of pending tasks) for the "cloud\_native\_pk\_index\_get" thread pool used by PK index parallel get operations in shared-data (cloud-native/lake) mode. The actual thread count for that pool is controlled by `pk_index_parallel_get_threadpool_max_threads`; this setting only limits how many tasks may be queued awaiting execution. The very large default (2^20) effectively makes the queue unbounded; lowering it prevents excessive memory growth from queued tasks but may cause task submissions to block or fail when the queue is full. Tune together with `pk_index_parallel_get_threadpool_max_threads` based on workload concurrency and memory constraints. * Introduced in: - ##### priority\_exec\_state\_report\_max\_threads[​](#priority_exec_state_report_max_threads "Direct link to priority_exec_state_report_max_threads") * Default: 2 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Maximum number of threads for the high-priority exec-state-report thread pool. This pool is used by `ExecStateReporter` to asynchronously send high-priority execution status reports (such as urgent fragment failures) from BE to FE via RPC. Unlike the normal exec-state-report pool, this pool has an unbounded task queue. The actual pool size at startup is `max(1, priority_exec_state_report_max_threads)`. Changing this config at runtime triggers `update_max_threads` on the priority pool in every executor set (shared and exclusive). Paired with `exec_state_report_max_threads` for the normal pool. Increase this value when high-priority reports are delayed under heavy concurrent query loads. * Introduced in: v4.1.0, v4.0.8, v3.5.15 ##### priority\_queue\_remaining\_tasks\_increased\_frequency[​](#priority_queue_remaining_tasks_increased_frequency "Direct link to priority_queue_remaining_tasks_increased_frequency") * Default: 512 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls how often the BlockingPriorityQueue increases ("ages") the priority of all remaining tasks to avoid starvation. Each successful get/pop increments an internal `_upgrade_counter`; when `_upgrade_counter` exceeds `priority_queue_remaining_tasks_increased_frequency`, the queue increments every element's priority, rebuilds the heap, and resets the counter. Lower values cause more frequent priority aging (reducing starvation but increasing CPU cost due to iterating and re-heapifying); higher values reduce that overhead but delay priority adjustments. The value is a simple operation count threshold, not a time duration. * Introduced in: v3.2.0 ##### query\_cache\_capacity[​](#query_cache_capacity "Direct link to query_cache_capacity") * Default: 536870912 * Type: Int * Unit: Bytes * Is mutable: No * Description: The size of the query cache in the BE. The default size is 512 MB. The size cannot be less than 4 MB. If the memory capacity of the BE is insufficient to provision your expected query cache size, you can increase the memory capacity of the BE. * Introduced in: - ##### query\_pool\_spill\_mem\_limit\_threshold[​](#query_pool_spill_mem_limit_threshold "Direct link to query_pool_spill_mem_limit_threshold") * Default: 1.0 * Type: Double * Unit: - * Is mutable: No * Description: If automatic spilling is enabled, when the memory usage of all queries exceeds `query_pool memory limit * query_pool_spill_mem_limit_threshold`, intermediate result spilling will be triggered. * Introduced in: v3.2.7 ##### query\_scratch\_dirs[​](#query_scratch_dirs "Direct link to query_scratch_dirs") * Default: `${STARROCKS_HOME}` * Type: string * Unit: - * Is mutable: No * Description: Comma-separated list of writable scratch directories used by query execution to spill intermediate data (for example, external sorts, hash joins, and other operators). Specify one or more paths separated by `;` (e.g. `/mnt/ssd1/tmp;/mnt/ssd2/tmp`). Directories should be accessible and writable by the BE process and have sufficient free space; StarRocks will pick among them to distribute spill I/O. Changes require a restart to take effect. If a directory is missing, not writable, or full, spilling may fail or degrade query performance. * Introduced in: v3.2.0 ##### result\_buffer\_cancelled\_interval\_time[​](#result_buffer_cancelled_interval_time "Direct link to result_buffer_cancelled_interval_time") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The wait time before BufferControlBlock releases data. * Introduced in: - ##### scan\_context\_gc\_interval\_min[​](#scan_context_gc_interval_min "Direct link to scan_context_gc_interval_min") * Default: 5 * Type: Int * Unit: Minutes * Is mutable: Yes * Description: The time interval at which to clean the Scan Context. * Introduced in: - ##### scanner\_row\_num[​](#scanner_row_num "Direct link to scanner_row_num") * Default: 16384 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum row count returned by each scan thread in a scan. * Introduced in: - ##### scanner\_thread\_pool\_queue\_size[​](#scanner_thread_pool_queue_size "Direct link to scanner_thread_pool_queue_size") * Default: 102400 * Type: Int * Unit: - * Is mutable: No * Description: The number of scan tasks supported by the storage engine. * Introduced in: - ##### scanner\_thread\_pool\_thread\_num[​](#scanner_thread_pool_thread_num "Direct link to scanner_thread_pool_thread_num") * Default: 48 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of threads which the storage engine used for concurrent storage volume scanning. All threads are managed in the thread pool. * Introduced in: - ##### spill\_max\_dir\_bytes\_ratio[​](#spill_max_dir_bytes_ratio "Direct link to spill_max_dir_bytes_ratio") * Default: 0.5 * Type: Double * Unit: - * Is mutable: Yes * Description: The maximum proportion of disk capacity that a spill directory (`spill_local_storage_dir`) can use when it is on the same disk as a storage path (`storage_root_path`). The default value `0.5` means intermediate result spilling can use up to 50% of that disk's capacity. A spill directory on a separate disk is not subject to this limit. This value is read when the BE starts, so a change takes effect only after a restart. * Introduced in: v3.2.0 ##### string\_prefix\_zonemap\_prefix\_len[​](#string_prefix_zonemap_prefix_len "Direct link to string_prefix_zonemap_prefix_len") * Default: 16 * Type: Int * Unit: - * Is mutable: Yes * Description: Prefix length used for string ZoneMap min/max when `enable_string_prefix_zonemap` is enabled. * Introduced in: - ##### udf\_thread\_pool\_size[​](#udf_thread_pool_size "Direct link to udf_thread_pool_size") * Default: 1 * Type: Int * Unit: Threads * Is mutable: No * Description: Sets the size of the UDF call PriorityThreadPool created in ExecEnv (used for executing user-defined functions / UDF-related tasks). The value is used as the pool thread count and also as the pool queue capacity when constructing the thread pool (PriorityThreadPool("udf", thread\_num, queue\_size)). Increase to allow more concurrent UDF executions; keep small to avoid excessive CPU and memory contention. * Introduced in: v3.2.0 ##### update\_memory\_limit\_percent[​](#update_memory_limit_percent "Direct link to update_memory_limit_percent") * Default: 60 * Type: Int * Unit: Percent * Is mutable: Yes * Description: Fraction of the BE process memory reserved for update-related memory and caches. During startup `GlobalEnv` computes the `MemTracker` for updates as process\_mem\_limit \* clamp(update\_memory\_limit\_percent, 0, 100) / 100. `UpdateManager` also uses this percentage to size its primary-index/index-cache capacity (index cache capacity = GlobalEnv::process\_mem\_limit \* update\_memory\_limit\_percent / 100). When this value is changed at runtime, the update `MemTracker` limit, the primary index cache capacity, and the derived limits of the update state, compaction state, and primary index block caches are recalculated. Decreasing this value evicts unused primary index cache entries immediately, but memory that is in use is released only after the running loads and compactions finish. Increasing this value gives more memory to update/primary-index paths (reducing memory available for other pools); decreasing it reduces update memory and cache capacity. Values are clamped to the range 0–100. * Introduced in: v3.2.0 ##### vector\_chunk\_size[​](#vector_chunk_size "Direct link to vector_chunk_size") * Default: 4096 * Type: Int * Unit: Rows * Is mutable: No * Description: The number of rows per vectorized chunk (batch) used throughout the execution and storage code paths. This value controls Chunk and RuntimeState batch\_size creation, affects operator throughput, memory footprint per operator, spill and sort buffer sizing, and I/O heuristics (for example, ORC writer natural write size). Increasing it can improve CPU and I/O efficiency for wide/CPU-bound workloads but raises peak memory usage and can increase latency for small-result queries. Tune only when profiling shows batch-size is a bottleneck; otherwise keep the default for balanced memory and performance. * Introduced in: v3.2.0 #### Loading and unloading[​](#loading-and-unloading "Direct link to Loading and unloading") ##### clear\_transaction\_task\_worker\_count[​](#clear_transaction_task_worker_count "Direct link to clear_transaction_task_worker_count") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads used for clearing transaction. * Introduced in: - ##### column\_mode\_partial\_update\_insert\_batch\_size[​](#column_mode_partial_update_insert_batch_size "Direct link to column_mode_partial_update_insert_batch_size") * Default: 4096 * Type: Int * Unit: - * Is mutable: Yes * Description: Batch size for column mode partial update when processing inserted rows. If this item is set to `0` or negative, it will be clamped to `1` to avoid infinite loop. This item controls the number of newly inserted rows processed in each batch. Larger values can improve write performance but will consume more memory. * Introduced in: v3.5.10, v4.0.2 ##### partial\_update\_memory\_limit\_per\_worker[​](#partial_update_memory_limit_per_worker "Direct link to partial_update_memory_limit_per_worker") * Default: 2147483648 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: Maximum memory per worker thread for partial update operations. Controls the memory footprint of individual worker threads when processing partial updates. * Introduced in: - ##### enable\_load\_spill\_parallel\_merge[​](#enable_load_spill_parallel_merge "Direct link to enable_load_spill_parallel_merge") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Specifies whether to enable parallel spill merge within a single tablet. Enabling this can improve the performance of spill merge during data loading. * Introduced in: - ##### enable\_parallel\_memtable\_finalize[​](#enable_parallel_memtable_finalize "Direct link to enable_parallel_memtable_finalize") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Specifies whether to enable parallel memtable finalize when loading data to lake tables (shared-data mode). When enabled, the memtable finalize operation (sort/aggregate) is moved from the write thread to the flush thread, allowing the write thread to continue inserting data into a new memtable while the previous one is being finalized and flushed in parallel. This can significantly improve load throughput by overlapping CPU-intensive finalize operations with I/O-bound flush operations. Note that this optimization is automatically disabled when auto-increment columns need to be filled, as auto-increment ID assignment must happen before the memtable is submitted for flush. * Introduced in: - ##### allow\_list\_object\_for\_random\_bucketing\_on\_cache\_miss[​](#allow_list_object_for_random_bucketing_on_cache_miss "Direct link to allow_list_object_for_random_bucketing_on_cache_miss") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls whether to allow object-storage LIST fallback when lake metadata cache misses during random bucketing size checks. `true` means fallback to LIST metadata files to compute base size (historical behavior, more accurate size estimation). `false` means skip LIST and use `base_size = 0`, which reduces LIST object requests but may delay immutable marking due to less accurate size estimation. * Introduced in: 4.1.0, 4.0.7, 3.5.15 ##### enable\_stream\_load\_verbose\_log[​](#enable_stream_load_verbose_log "Direct link to enable_stream_load_verbose_log") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Specifies whether to log the HTTP requests and responses for Stream Load jobs. * Introduced in: v2.5.17, v3.0.9, v3.1.6, v3.2.1 ##### flush\_thread\_num\_per\_store[​](#flush_thread_num_per_store "Direct link to flush_thread_num_per_store") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: Number of threads that are used for flushing MemTable in each store. * Introduced in: - ##### lake\_flush\_thread\_num\_per\_store[​](#lake_flush_thread_num_per_store "Direct link to lake_flush_thread_num_per_store") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: Number of threads that are used for flushing MemTable in each store in a shared-data cluster. When this value is set to `0`, the system uses twice of the CPU core count as the value. When this value is set to less than `0`, the system uses the product of its absolute value and the CPU core count as the value. * Introduced in: v3.1.12, 3.2.7 ##### load\_data\_reserve\_hours[​](#load_data_reserve_hours "Direct link to load_data_reserve_hours") * Default: 4 * Type: Int * Unit: Hours * Is mutable: No * Description: The reservation time for the files produced by small-scale loadings. * Introduced in: - ##### load\_error\_log\_reserve\_hours[​](#load_error_log_reserve_hours "Direct link to load_error_log_reserve_hours") * Default: 48 * Type: Int * Unit: Hours * Is mutable: Yes * Description: The time for which data loading logs are reserved. * Introduced in: - ##### load\_process\_max\_memory\_limit\_bytes[​](#load_process_max_memory_limit_bytes "Direct link to load_process_max_memory_limit_bytes") * Default: 107374182400 * Type: Int * Unit: Bytes * Is mutable: No * Description: The maximum size limit of memory resources that can be taken up by all load processes on a BE node. * Introduced in: - ##### load\_spill\_memory\_usage\_per\_merge[​](#load_spill_memory_usage_per_merge "Direct link to load_spill_memory_usage_per_merge") * Default: 1073741824 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The maximum memory usage per merge operation during spill merge. Default is 1 GB (1073741824 bytes). This parameter controls the memory consumption of individual merge tasks during data loading spill merge to prevent excessive memory usage. * Introduced in: - ##### max\_consumer\_num\_per\_group[​](#max_consumer_num_per_group "Direct link to max_consumer_num_per_group") * Default: 3 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of consumers in a consumer group of Routine Load. * Introduced in: - ##### max\_runnings\_transactions\_per\_txn\_map[​](#max_runnings_transactions_per_txn_map "Direct link to max_runnings_transactions_per_txn_map") * Default: 100 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of transactions that can run concurrently in each partition. * Introduced in: - ##### number\_tablet\_writer\_threads[​](#number_tablet_writer_threads "Direct link to number_tablet_writer_threads") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of tablet writer threads used in ingestion, such as Stream Load, Broker Load and Insert. When the parameter is set to less than or equal to 0, the system uses half of the number of CPU cores, with a minimum of 16. When the parameter is set to greater than 0, the system uses that value. This configuration is changed to dynamic from v3.1.7 onwards. * Introduced in: - ##### push\_worker\_count\_high\_priority[​](#push_worker_count_high_priority "Direct link to push_worker_count_high_priority") * Default: 3 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads used to handle a load task with HIGH priority. * Introduced in: - ##### push\_worker\_count\_normal\_priority[​](#push_worker_count_normal_priority "Direct link to push_worker_count_normal_priority") * Default: 3 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads used to handle a load task with NORMAL priority. * Introduced in: - ##### streaming\_load\_max\_batch\_size\_mb[​](#streaming_load_max_batch_size_mb "Direct link to streaming_load_max_batch_size_mb") * Default: 100 * Type: Int * Unit: MB * Is mutable: Yes * Description: The maximum size of a JSON file that can be streamed into StarRocks. * Introduced in: - ##### streaming\_load\_max\_mb[​](#streaming_load_max_mb "Direct link to streaming_load_max_mb") * Default: 102400 * Type: Int * Unit: MB * Is mutable: Yes * Description: The maximum size of a file that can be streamed into StarRocks. From v3.0, the default value has been changed from `10240` to `102400`. * Introduced in: - ##### streaming\_load\_rpc\_max\_alive\_time\_sec[​](#streaming_load_rpc_max_alive_time_sec "Direct link to streaming_load_rpc_max_alive_time_sec") * Default: 1200 * Type: Int * Unit: Seconds * Is mutable: No * Description: The RPC timeout for Stream Load. * Introduced in: - ##### transaction\_publish\_version\_thread\_pool\_num\_min[​](#transaction_publish_version_thread_pool_num_min "Direct link to transaction_publish_version_thread_pool_num_min") * Default: 0 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Minimum number of threads in the Publish Version thread pool. The pool can shrink to this value when idle. `0` means no fixed lower bound. * Introduced in: - ##### transaction\_publish\_version\_thread\_pool\_idle\_time\_ms[​](#transaction_publish_version_thread_pool_idle_time_ms "Direct link to transaction_publish_version_thread_pool_idle_time_ms") * Default: 60000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: The idle time before a thread is reclaimed by the Publish Version thread pool. * Introduced in: - ##### transaction\_publish\_version\_worker\_count[​](#transaction_publish_version_worker_count "Direct link to transaction_publish_version_worker_count") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads used to publish a version. When this value is set to less than or equal to `0`, the system uses the CPU core count as the value, so as to avoid insufficient thread resources when import concurrency is high but only a fixed number of threads are used. From v2.5, the default value has been changed from `8` to `0`. * Introduced in: - ##### use\_mmap\_allocate\_chunk[​](#use_mmap_allocate_chunk "Direct link to use_mmap_allocate_chunk") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to use anonymous mmap (`MAP_ANONYMOUS | MAP_PRIVATE`) for chunk allocation. When enabled, many VM mappings are created; you must raise `vm.max_map_count` (e.g., `echo 262144 > /proc/sys/vm/max_map_count`) and set a large `chunk_reserved_bytes_limit`, otherwise frequent map/unmap operations will cause severe performance degradation. * Introduced in: v3.2.0 ##### write\_buffer\_size[​](#write_buffer_size "Direct link to write_buffer_size") * Default: 104857600 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The buffer size of MemTable in the memory. This configuration item is the threshold to trigger a flush. * Introduced in: - ##### broker\_write\_timeout\_seconds[​](#broker_write_timeout_seconds "Direct link to broker_write_timeout_seconds") * Default: 30 * Type: int * Unit: Seconds * Is mutable: No * Description: Timeout (in seconds) used by backend broker operations for write/IO RPCs. The value is multiplied by 1000 to produce millisecond timeouts and is passed as the default timeout\_ms to BrokerFileSystem and BrokerServiceConnection instances (e.g., file export and snapshot upload/download). Increase this when brokers or network are slow or when transferring large files to avoid premature timeouts; decreasing it may cause broker RPCs to fail earlier. This value is defined in common/config and is applied at process start (not dynamically reloadable). * Introduced in: v3.2.0 ##### enable\_load\_channel\_rpc\_async[​](#enable_load_channel_rpc_async "Direct link to enable_load_channel_rpc_async") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, handling of load-channel open RPCs (for example, `PTabletWriterOpen`) is offloaded from the BRPC worker to a dedicated thread pool: the request handler creates a `ChannelOpenTask` and submits it to the internal `_async_rpc_pool` instead of running `LoadChannelMgr::_open` inline. This reduces work and blocking inside BRPC threads and allows tuning concurrency via `load_channel_rpc_thread_pool_num` and `load_channel_rpc_thread_pool_queue_size`. If the thread pool submission fails (when pool is full or shut down), the request is canceled and an error status is returned. The pool is shut down on `LoadChannelMgr::close()`, so consider capacity and lifecycle when you want to enable this feature so as to avoid request rejections or delayed processing. * Introduced in: v3.5.0 ##### enable\_load\_diagnose[​](#enable_load_diagnose "Direct link to enable_load_diagnose") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, StarRocks will attempt an automated load diagnosis from BE OlapTableSink/NodeChannel after a brpc timeout matching "\[E1008]Reached timeout". The code creates a `PLoadDiagnoseRequest` and sends an RPC to the remote LoadChannel to collect a profile and/or stack trace (controlled by `load_diagnose_rpc_timeout_profile_threshold_ms` and `load_diagnose_rpc_timeout_stack_trace_threshold_ms`). The diagnose RPC uses `load_diagnose_send_rpc_timeout_ms` as its timeout. Diagnosis is skipped if a diagnose request is already in progress. Enabling this produces additional RPCs and profiling work on target nodes; disable on sensitive production workloads to avoid extra overhead. * Introduced in: v3.5.0 ##### enable\_load\_segment\_parallel[​](#enable_load_segment_parallel "Direct link to enable_load_segment_parallel") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: When enabled, rowset segment loading and rowset-level reads are performed concurrently using StarRocks background thread pools (ExecEnv::load\_segment\_thread\_pool and ExecEnv::load\_rowset\_thread\_pool). Rowset::load\_segments and TabletReader::get\_segment\_iterators submit per-segment or per-rowset tasks to these pools, falling back to serial loading and logging a warning if submission fails. Enable this to reduce read/load latency for large rowsets at the cost of increased CPU/IO concurrency and memory pressure. Note: parallel loading can change the load completion order of segments and therefore prevents partial compaction (code checks `_parallel_load` and disables partial compaction when enabled); consider implications for operations that rely on segment order. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### enable\_streaming\_load\_thread\_pool[​](#enable_streaming_load_thread_pool "Direct link to enable_streaming_load_thread_pool") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls whether streaming load scanners are submitted to the dedicated streaming load thread pool. When enabled and a query is a LOAD with `TLoadJobType::STREAM_LOAD`, ConnectorScanNode submits scanner tasks to the `streaming_load_thread_pool` (which is configured with INT32\_MAX threads and queue sizes, i.e. effectively unbounded). When disabled, scanners use the general `thread_pool` and its `PriorityThreadPool` submission logic (priority computation, try\_offer/offer behavior). Enabling isolates streaming-load work from regular query execution to reduce interference; however, because the dedicated pool is effectively unbounded, enabling may increase concurrent threads and resource usage under heavy streaming-load traffic. This option is on by default and typically does not require modification. * Introduced in: v3.2.0 ##### es\_http\_timeout\_ms[​](#es_http_timeout_ms "Direct link to es_http_timeout_ms") * Default: 5000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: HTTP connection timeout (in milliseconds) used by the ES network client in ESScanReader for Elasticsearch scroll requests. This value is applied via network\_client.set\_timeout\_ms() before sending subsequent scroll POSTs and controls how long the client waits for an ES response during scrolling. Increase this value for slow networks or large queries to avoid premature timeouts; decrease to fail faster on unresponsive ES nodes. This setting complements `es_scroll_keepalive`, which controls the scroll context keep-alive duration. * Introduced in: v3.2.0 ##### es\_index\_max\_result\_window[​](#es_index_max_result_window "Direct link to es_index_max_result_window") * Default: 10000 * Type: Int * Unit: - * Is mutable: No * Description: Limits the maximum number of documents StarRocks will request from Elasticsearch in a single batch. StarRocks sets the ES request batch size to min(`es_index_max_result_window`, `chunk_size`) when building `KEY_BATCH_SIZE` for the ES reader. If an ES request exceeds the Elasticsearch index setting `index.max_result_window`, Elasticsearch returns HTTP 400 (Bad Request). Adjust this value when scanning large indexes or increase the ES `index.max_result_window` on the Elasticsearch side to permit larger single requests. * Introduced in: v3.2.0 ##### ignore\_load\_tablet\_failure[​](#ignore_load_tablet_failure "Direct link to ignore_load_tablet_failure") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: When this item is set to `false`, the system will treat any tablet header load failures (non-NotFound and non-AlreadyExist errors) as fatal: the code logs the error and calls LOG(FATAL) to stop the BE process. When it is set to `true`, the BE continues startup despite such per-tablet load errors — failed tablet IDs are recorded and skipped while successful tablets are still loaded. Note that this parameter does NOT suppress fatal errors from the RocksDB meta scan itself, which always cause the process to quit. * Introduced in: v3.2.0 ##### load\_channel\_abort\_clean\_up\_delay\_seconds[​](#load_channel_abort_clean_up_delay_seconds "Direct link to load_channel_abort_clean_up_delay_seconds") * Default: 600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Controls how long (in seconds) the system keeps the load IDs of aborted load channels before removing them from `_aborted_load_channels`. When a load job is cancelled or fails, the load ID stays recorded so any late-arriving load RPCs can be rejected immediately; once the delay expires, the entry is cleaned during the periodic background sweep (minimum sweep interval is 60 seconds). Setting the delay too low risks accepting stray RPCs after an abort, while setting it too high may retain state and consume resources longer than necessary. Tune this to balance correctness of late-request rejection and resource retention for aborted loads. * Introduced in: v3.5.11, v4.0.4 ##### load\_channel\_rpc\_thread\_pool\_num[​](#load_channel_rpc_thread_pool_num "Direct link to load_channel_rpc_thread_pool_num") * Default: -1 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Maximum number of threads for the load-channel async RPC thread pool. When set to less than or equal to 0 (default `-1`) the pool size is auto-set to the number of CPU cores (`CpuInfo::num_cores()`). The configured value is used as ThreadPoolBuilder's max threads and the pool's min threads is set to min(5, max\_threads). The pool queue size is controlled separately by `load_channel_rpc_thread_pool_queue_size`. This setting was introduced to align the async RPC pool size with brpc workers' default (`brpc_num_threads`) so behavior remains compatible after switching load RPC handling from synchronous to asynchronous. Changing this config at runtime triggers `ExecEnv::GetInstance()->load_channel_mgr()->async_rpc_pool()->update_max_threads(...)`. * Introduced in: v3.5.0 ##### load\_channel\_rpc\_thread\_pool\_queue\_size[​](#load_channel_rpc_thread_pool_queue_size "Direct link to load_channel_rpc_thread_pool_queue_size") * Default: 1024000 * Type: int * Unit: Count * Is mutable: No * Description: Sets the maximum pending-task queue size for the Load channel RPC thread pool created by LoadChannelMgr. This thread pool executes asynchronous `open` requests when `enable_load_channel_rpc_async` is enabled; the pool size is paired with `load_channel_rpc_thread_pool_num`. The large default (1024000) aligns with brpc workers' defaults to preserve behavior after switching from synchronous to asynchronous handling. If the queue is full, ThreadPool::submit() will fail and the incoming open RPC is cancelled with an error, causing the caller to receive a rejection. Increase this value to buffer larger bursts of concurrent `open` requests; reducing it tightens backpressure but may cause more rejections under load. * Introduced in: v3.5.0 ##### load\_diagnose\_rpc\_timeout\_profile\_threshold\_ms[​](#load_diagnose_rpc_timeout_profile_threshold_ms "Direct link to load_diagnose_rpc_timeout_profile_threshold_ms") * Default: 60000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: When a load RPC times out (error contains "\[E1008]Reached timeout") and `enable_load_diagnose` is true, this threshold controls whether a full profiling diagnose is requested. If the request-level RPC timeout `_rpc_timeout_ms` is greater than `load_diagnose_rpc_timeout_profile_threshold_ms`, profiling is enabled for that diagnose. For smaller `_rpc_timeout_ms` values, profiling is sampled once every 20 timeouts to avoid frequent heavy diagnostics for real-time/short-timeout loads. This value affects the `profile` flag in the `PLoadDiagnoseRequest` sent; stack-trace behavior is controlled separately by `load_diagnose_rpc_timeout_stack_trace_threshold_ms` and send timeout by `load_diagnose_send_rpc_timeout_ms`. * Introduced in: v3.5.0 ##### load\_diagnose\_rpc\_timeout\_stack\_trace\_threshold\_ms[​](#load_diagnose_rpc_timeout_stack_trace_threshold_ms "Direct link to load_diagnose_rpc_timeout_stack_trace_threshold_ms") * Default: 600000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Threshold (in ms) used to decide when to request remote stack traces for long-running load RPCs. When a load RPC times out with a timeout error and the effective RPC timeout (\_rpc\_timeout\_ms) exceeds this value, `OlapTableSink`/`NodeChannel` will include `stack_trace=true` in a `load_diagnose` RPC to the target BE so the BE can return stack traces for debugging. `LocalTabletsChannel::SecondaryReplicasWaiter` also triggers a best-effort stack-trace diagnose from the primary if waiting for secondary replicas exceeds this interval. This behavior requires `enable_load_diagnose` and uses `load_diagnose_send_rpc_timeout_ms` for the diagnose RPC timeout; profiling is gated separately by `load_diagnose_rpc_timeout_profile_threshold_ms`. Lowering this value increases how aggressively stack traces are requested. * Introduced in: v3.5.0 ##### load\_diagnose\_send\_rpc\_timeout\_ms[​](#load_diagnose_send_rpc_timeout_ms "Direct link to load_diagnose_send_rpc_timeout_ms") * Default: 2000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Timeout (in milliseconds) applied to diagnosis-related brpc calls initiated by BE load paths. It is used to set the controller timeout for `load_diagnose` RPCs (sent by NodeChannel/OlapTableSink when a LoadChannel brpc call times out) and for replica-status queries (used by SecondaryReplicasWaiter / LocalTabletsChannel when checking primary replica state). Choose a value high enough to allow the remote side to respond with profile or stack-trace data, but not so high that failure handling is delayed. This parameter works together with `enable_load_diagnose`, `load_diagnose_rpc_timeout_profile_threshold_ms`, and `load_diagnose_rpc_timeout_stack_trace_threshold_ms` which control when and what diagnostic information is requested. * Introduced in: v3.5.0 ##### load\_fp\_brpc\_timeout\_ms[​](#load_fp_brpc_timeout_ms "Direct link to load_fp_brpc_timeout_ms") * Default: -1 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Overrides the per-channel brpc RPC timeout used by OlapTableSink when the `node_channel_set_brpc_timeout` fail point is triggered. If set to a positive value, NodeChannel will set its internal `_rpc_timeout_ms` to this value (in milliseconds) causing open/add-chunk/cancel RPCs to use the shorter timeout and enabling simulation of brpc timeouts that produce the "\[E1008]Reached timeout" error. Default (`-1`) disables the override. Changing this value is intended for testing and fault injection; small values may produce false timeouts and trigger load diagnostics (see `enable_load_diagnose`, `load_diagnose_rpc_timeout_profile_threshold_ms`, `load_diagnose_rpc_timeout_stack_trace_threshold_ms`, and `load_diagnose_send_rpc_timeout_ms`). * Introduced in: v3.5.0 ##### load\_fp\_tablets\_channel\_add\_chunk\_block\_ms[​](#load_fp_tablets_channel_add_chunk_block_ms "Direct link to load_fp_tablets_channel_add_chunk_block_ms") * Default: -1 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: When enabled (set to a positive milliseconds value) this fail-point configuration makes TabletsChannel::add\_chunk sleep for the specified time during load processing. It is used to simulate BRPC timeout errors (e.g., "\[E1008]Reached timeout") and to emulate an expensive add\_chunk operation that increases load latency. A value less than or equal to 0 (default `-1`) disables the injection. Intended for testing fault handling, timeouts, and replica synchronization behavior — do not enable in normal production workloads as it delays write completion and can trigger upstream timeouts or replica aborts. * Introduced in: v3.5.0 ##### load\_segment\_thread\_pool\_num\_max[​](#load_segment_thread_pool_num_max "Direct link to load_segment_thread_pool_num_max") * Default: 128 * Type: Int * Unit: - * Is mutable: No * Description: Sets the maximum number of worker threads for BE load-related thread pools. This value is used by ThreadPoolBuilder to limit threads for both `load_rowset_pool` and `load_segment_pool` in exec\_env.cpp, controlling concurrency for processing loaded rowsets and segments (e.g., decoding, indexing, writing) during streaming and batch loads. Increasing this value raises parallelism and can improve load throughput but also increases CPU, memory usage, and potential contention; decreasing it limits concurrent load processing and may reduce throughput. Tune together with `load_segment_thread_pool_queue_size` and `streaming_load_thread_pool_idle_time_ms`. Change requires BE restart. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### load\_segment\_thread\_pool\_queue\_size[​](#load_segment_thread_pool_queue_size "Direct link to load_segment_thread_pool_queue_size") * Default: 10240 * Type: Int * Unit: Tasks * Is mutable: No * Description: Sets the maximum queue length (number of pending tasks) for the load-related thread pools created as "load\_rowset\_pool" and "load\_segment\_pool". These pools use `load_segment_thread_pool_num_max` for their max thread count and this configuration controls how many load segment/rowset tasks can be buffered before the ThreadPool's overflow policy takes effect (further submissions may be rejected or blocked depending on the ThreadPool implementation). Increase to allow more pending load work (uses more memory and can raise latency); decrease to limit buffered load concurrency and reduce memory usage. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### max\_pulsar\_consumer\_num\_per\_group[​](#max_pulsar_consumer_num_per_group "Direct link to max_pulsar_consumer_num_per_group") * Default: 10 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls the maximum number of Pulsar consumers that may be created in a single data consumer group for routine load on a BE. Because cumulative acknowledge is not supported for multi-topic subscriptions, each consumer subscribes exactly one topic/partition; if the number of partitions in `pulsar_info->partitions` exceeds this value, group creation fails with an error advising to increase `max_pulsar_consumer_num_per_group` on the BE or add more BEs. This limit is enforced when constructing a PulsarDataConsumerGroup and prevents a BE from hosting more than this many consumers for one routine load group. For Kafka routine load, `max_consumer_num_per_group` is used instead. * Introduced in: v3.2.0 ##### pull\_load\_task\_dir[​](#pull_load_task_dir "Direct link to pull_load_task_dir") * Default: `${STARROCKS_HOME}/var/pull_load` * Type: string * Unit: - * Is mutable: No * Description: Filesystem path where the BE stores data and working files for "pull load" tasks (downloaded source files, task state, temporary output, etc.). The directory must be writable by the BE process and have sufficient disk space for incoming loads. The default is relative to STARROCKS\_HOME; tests create and expect this directory to exist (see test configuration). * Introduced in: v3.2.0 ##### routine\_load\_kafka\_timeout\_second[​](#routine_load_kafka_timeout_second "Direct link to routine_load_kafka_timeout_second") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: No * Description: Timeout in seconds used for Kafka-related routine load operations. When a client request does not specify a timeout, `routine_load_kafka_timeout_second` is used as the default RPC timeout (converted to milliseconds) for `get_info`. It is also used as the per-call consume poll timeout for the librdkafka consumer (converted to milliseconds and capped by remaining runtime). Note: the internal `get_info` path reduces this value to 80% before passing it to librdkafka to avoid FE-side timeout races. Set this to a value that balances timely failure reporting and sufficient time for network/broker responses; changes require a restart because the setting is not mutable. * Introduced in: v3.2.0 ##### routine\_load\_pulsar\_timeout\_second[​](#routine_load_pulsar_timeout_second "Direct link to routine_load_pulsar_timeout_second") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: No * Description: Default timeout (in seconds) that the BE uses for Pulsar-related routine load operations when the request does not supply an explicit timeout. Specifically, `PInternalServiceImplBase::get_pulsar_info` multiplies this value by 1000 to form the millisecond timeout passed to the routine load task executor methods that fetch Pulsar partition metadata and backlog. Increase to allow slower Pulsar responses at the cost of longer failure detection; decrease to fail faster on slow brokers. Analogous to `routine_load_kafka_timeout_second` used for Kafka. * Introduced in: v3.2.0 ##### streaming\_load\_thread\_pool\_idle\_time\_ms[​](#streaming_load_thread_pool_idle_time_ms "Direct link to streaming_load_thread_pool_idle_time_ms") * Default: 2000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: Sets the thread idle timeout (in milliseconds) for streaming-load related thread pools. The value is used as the idle timeout passed to ThreadPoolBuilder for the `stream_load_io` pool and also for `load_rowset_pool` and `load_segment_pool`. Threads in these pools are reclaimed when idle for this duration; lower values reduce idle resource usage but increase thread creation overhead, while higher values keep threads alive longer. The `stream_load_io` pool is used when `enable_streaming_load_thread_pool` is enabled. * Introduced in: v3.2.0 ##### streaming\_load\_thread\_pool\_num\_min[​](#streaming_load_thread_pool_num_min "Direct link to streaming_load_thread_pool_num_min") * Default: 0 * Type: Int * Unit: - * Is mutable: No * Description: Minimum number of threads for the streaming load IO thread pool ("stream\_load\_io") created during ExecEnv initialization. The pool is built with `set_max_threads(INT32_MAX)` and `set_max_queue_size(INT32_MAX)` so it is effectively unbounded to avoid deadlocks for concurrent streaming loads. A value of 0 lets the pool start with no threads and grow on demand; setting a positive value reserves that many threads at startup. This pool is used when `enable_streaming_load_thread_pool` is true and its idle timeout is controlled by `streaming_load_thread_pool_idle_time_ms`. Overall concurrency is still constrained by `fragment_pool_thread_num_max` and `webserver_num_workers`; changing this value is rarely necessary and may increase resource usage if set too high. * Introduced in: v3.2.0 --- ### BE Configuration - Shared-data, Data Lake, and Others Some BE configuration items are dynamic parameters which you can set interactively when BE nodes are online. The rest of them are static parameters. You can only set the static parameters of a BE node by changing them in the corresponding configuration file **be.conf** and restarting the BE node to allow the change to take effect. #### View BE configuration items[​](#view-be-configuration-items "Direct link to View BE configuration items") You can view the BE configuration items using the following command: ```sql SELECT * FROM information_schema.be_configs [WHERE NAME LIKE "%%"] ``` #### Configure BE parameters[​](#configure-be-parameters "Direct link to Configure BE parameters") ##### Configure BE dynamic parameters[​](#configure-be-dynamic-parameters "Direct link to Configure BE dynamic parameters") You can configure a dynamic parameter of a BE node by updating the value in `information_schema.be_configs`. warning Setting an invalid value may cause unknown behaviors. Check twice before you run the command to update the configuration. ```sql -- Replace with the key of the configuration and with the value. UPDATE information_schema.be_configs SET VALUE = WHERE name = ""; ``` ##### Configure BE static parameters[​](#configure-be-static-parameters "Direct link to Configure BE static parameters") You can only set the static parameters of a BE by changing them in the corresponding configuration file **be.conf**, and restarting the BE to allow the changes to take effect. *** This topic introduces the following types of BE configurations: * [Shared-data](#shared-data) * [Data Lake](#data-lake) * [Other](#other) #### Shared-data[​](#shared-data "Direct link to Shared-data") ##### cloud\_native\_pk\_index\_rebuild\_files\_threshold[​](#cloud_native_pk_index_rebuild_files_threshold "Direct link to cloud_native_pk_index_rebuild_files_threshold") * Default: 50 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of segment files that need to be rebuilt in cloud-native Primary Key index. If the number of files that need to be rebuilt during index recovery exceeds this threshold, StarRocks will flush the in-memory MemTable immediately to reduce the number of segments that must be replayed. Set to `0` to disable this early-flush strategy. * Introduced in: - ##### cloud\_native\_pk\_index\_rebuild\_rows\_threshold[​](#cloud_native_pk_index_rebuild_rows_threshold "Direct link to cloud_native_pk_index_rebuild_rows_threshold") * Default: 10000000 * Type: Long * Unit: Rows * Is mutable: Yes * Description: The maximum number of rows that need to be rebuilt in cloud-native Primary Key index. If the number of rows that need to be rebuilt during index recovery exceeds this threshold, StarRocks will flush the in-memory MemTable immediately to reduce the rebuild overhead. Set to `0` to disable this early-flush strategy. Works in conjunction with `cloud_native_pk_index_rebuild_files_threshold`; a flush is triggered if either threshold is exceeded. The row count includes segment rows plus the tombstone (delete) rows recorded in del files, so a delete-heavy workload that produces a few large del files also counts toward this threshold; del files written by older versions that did not record a row count contribute 0. * Introduced in: - ##### download\_buffer\_size[​](#download_buffer_size "Direct link to download_buffer_size") * Default: 4194304 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: Size (in bytes) of the in-memory copy buffer used when downloading snapshot files. SnapshotLoader::download passes this value to fs::copy as the per-transfer chunk size when reading from the remote sequential file into the local writable file. Larger values can improve throughput on high-bandwidth links by reducing syscall/IO overhead; smaller values reduce peak memory use per active transfer. Note: this parameter controls buffer size per stream, not the number of download threads—total memory consumption = download\_buffer\_size \* number\_of\_concurrent\_downloads. * Introduced in: v3.2.13 ##### graceful\_exit\_wait\_for\_frontend\_heartbeat[​](#graceful_exit_wait_for_frontend_heartbeat "Direct link to graceful_exit_wait_for_frontend_heartbeat") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Determines whether to await at least one frontend heartbeat response indicating SHUTDOWN status before completing graceful exit. When enabled, the graceful shutdown process remains active until a SHUTDOWN confirmation is responded via heartbeat RPC, ensuring the frontend has sufficient time to detect the termination state between two regular heartbeat intervals. * Introduced in: v3.4.5 ##### lake\_compaction\_stream\_buffer\_size\_bytes[​](#lake_compaction_stream_buffer_size_bytes "Direct link to lake_compaction_stream_buffer_size_bytes") * Default: 1048576 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The reader's remote I/O buffer size for cloud-native table compaction in a shared-data cluster. The default value is 1MB. You can increase this value to accelerate compaction process. * Introduced in: v3.2.3 ##### lake\_dump\_tablet\_metadata\_per\_request\_memory\_limit\_bytes[​](#lake_dump_tablet_metadata_per_request_memory_limit_bytes "Direct link to lake_dump_tablet_metadata_per_request_memory_limit_bytes") * Default: 268435456 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: The tracked memory budget for synchronous allocations made while processing one `/api/cloudnative/dump_tablet_metadata` request on a CN. It uses the standard MemTracker accounting granularity rather than enforcing a byte-exact ceiling. It covers metadata inspection, sensitive-field redaction, and JSON serialization, but not the metadata already stored in the CN metadata cache or the asynchronous HTTP output buffer. Each request uses the value captured when it is admitted. If this value is less than or equal to `0`, new requests fail closed. * Introduced in: v4.1.7 ##### lake\_dump\_tablet\_metadata\_per\_request\_json\_size\_limit\_bytes[​](#lake_dump_tablet_metadata_per_request_json_size_limit_bytes "Direct link to lake_dump_tablet_metadata_per_request_json_size_limit_bytes") * Default: 33554432 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: The maximum size of the complete JSON response for one `/api/cloudnative/dump_tablet_metadata` request. The size is checked before the response is handed to the HTTP output buffer. Each request uses the value captured when it is admitted. If this value is less than or equal to `0`, new requests fail closed. * Introduced in: v4.1.7 ##### lake\_dump\_tablet\_metadata\_max\_concurrency[​](#lake_dump_tablet_metadata_max_concurrency "Direct link to lake_dump_tablet_metadata_max_concurrency") * Default: 1 * Type: Int * Unit: Requests * Is mutable: Yes * Description: The maximum number of `/api/cloudnative/dump_tablet_metadata` requests admitted concurrently on one CN. A request continues to occupy one slot until its HTTP request is released. Increasing the value applies immediately to new requests. Decreasing it does not cancel admitted requests, and new requests are rejected until the active count is below the new limit. If this value is less than or equal to `0`, new requests fail closed. * Introduced in: v4.1.7 ##### lake\_enable\_del\_file\_crc\_check[​](#lake_enable_del_file_crc_check "Direct link to lake_enable_del_file_crc_check") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to verify a Primary Key delete file (`.del`) of a shared-data cluster against the CRC32C recorded in its metadata when the file is read back during transaction publish or Primary Key index rebuild. When a mismatch is detected, the operation fails with a corruption error instead of erasing the wrong primary keys. Delete files written before the checksum existed carry none and are always accepted, so this item has no effect on data written by an earlier version. Writing the checksum is not controlled by this item; this item only controls verification. * Introduced in: v4.2 ##### lake\_enable\_pk\_preserve\_txn\_delete\_order[​](#lake_enable_pk_preserve_txn_delete_order "Direct link to lake_enable_pk_preserve_txn_delete_order") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to preserve the in-transaction upsert/delete order for Primary Key tables in a shared-data cluster. When a single load transaction contains both a `DELETE` and a later re-`UPSERT` of the same key, enabling this makes the re-upsert win (consistent with shared-nothing clusters). It is enabled by default. For downgrade safety, set it to `false` before rolling back to (or running a mixed cluster with) a BE version without this fix: when enabled, a load can persist on-disk metadata that a pre-fix BE would misinterpret, potentially producing duplicate primary keys. When disabled, deletes fall back to the legacy behavior (applied after all upserts in the transaction). * Introduced in: v4.1.4 ##### lake\_enable\_protobuf\_file\_checksum[​](#lake_enable_protobuf_file_checksum "Direct link to lake_enable_protobuf_file_checksum") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to write tablet metadata and transaction log files of a shared-data cluster with an Adler-32 checksum, so that corruption of these files can be detected when they are read. Regardless of this item, readers always detect and verify the checksum automatically when it is present; this item only controls the write format. Set it to `false` only while the cluster may still be downgraded to a version that predates the checksummed format. During a rolling upgrade or a downgrade, an earlier BE or CN uses the legacy reader and cannot parse files written in the new format. * Introduced in: v4.2 ##### lake\_pk\_compaction\_max\_input\_rowsets[​](#lake_pk_compaction_max_input_rowsets "Direct link to lake_pk_compaction_max_input_rowsets") * Default: 500 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of input rowsets allowed in a Primary Key table compaction task in a shared-data cluster. The default value of this parameter is changed from `5` to `1000` since v3.2.4 and v3.1.10, and to `500` since v3.3.1 and v3.2.9. After the Sized-tiered Compaction policy is enabled for Primary Key tables (by setting `enable_pk_size_tiered_compaction_strategy` to `true`), StarRocks does not need to limit the number of rowsets for each compaction to reduce write amplification. Therefore, the default value of this parameter is increased. * Introduced in: v3.1.8, v3.2.3 ##### lake\_pk\_compaction\_base\_enable\_delete\_trigger[​](#lake_pk_compaction_base_enable_delete_trigger "Direct link to lake_pk_compaction_base_enable_delete_trigger") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether a Primary Key tablet in a shared-data cluster switches from cumulative compaction to base compaction on its own when its deletes reach `lake_pk_compaction_base_delete_ratio_threshold` or `lake_pk_compaction_base_delete_rows_threshold`. It is disabled by default because `lake_pk_compaction_base_delete_rows_threshold` is an absolute count: on a large table whose existing rows keep being updated or deleted, a tablet reaches it again soon after each base compaction, and each base compaction rewrites up to `update_compaction_ratio_threshold` (0.5 by default) of the tablet's data. A manual `ALTER TABLE ... COMPACT` runs base compaction regardless of this item. * Introduced in: v4.1.7 ##### lake\_pk\_compaction\_base\_delete\_ratio\_threshold[​](#lake_pk_compaction_base_delete_ratio_threshold "Direct link to lake_pk_compaction_base_delete_ratio_threshold") * Default: 0.5 * Type: Double * Unit: - * Is mutable: Yes * Description: One of two triggers that switch a Primary Key tablet in a shared-data cluster from cumulative compaction (size-tiered small-file merges) to base compaction, which rewrites the delete-bearing rowsets (the ones with the most deleted rows first) to drop deleted rows and shrink their delete vectors. When `lake_pk_compaction_base_enable_delete_trigger` is `true`, base compaction runs when the tablet's aggregate delete ratio (`sum(num_dels) / sum(num_rows)` across rowsets) reaches this value or when its absolute delete-row count reaches `lake_pk_compaction_base_delete_rows_threshold`. A manual `ALTER TABLE ... COMPACT` forces a base compaction regardless of these items. * Introduced in: v4.1.4 ##### lake\_pk\_compaction\_base\_delete\_rows\_threshold[​](#lake_pk_compaction_base_delete_rows_threshold "Direct link to lake_pk_compaction_base_delete_rows_threshold") * Default: 10000000 * Type: Int * Unit: - * Is mutable: Yes * Description: One of two triggers for Primary Key base compaction in a shared-data cluster (see `lake_pk_compaction_base_delete_ratio_threshold`). When `lake_pk_compaction_base_enable_delete_trigger` is `true`, base compaction runs when a tablet's absolute delete-row count (`sum(num_dels)` across rowsets) reaches this value. This absolute-count trigger complements the ratio trigger: on hot update/delete tables the delete vectors bloat and space is wasted while the aggregate delete ratio stays low (diluted by many mostly-live rowsets), so the ratio trigger alone would not fire. Raise it to make base compaction less frequent, or lower it to reclaim delete vectors sooner. * Introduced in: v4.1.4 ##### lake\_put\_txn\_log\_timeout\_guard\_ms[​](#lake_put_txn_log_timeout_guard_ms "Direct link to lake_put_txn_log_timeout_guard_ms") * Default: -1 * Type: Int64 * Unit: Milliseconds * Is mutable: Yes * Description: Timeout guard for writing a transaction log to object storage in a shared-data cluster (the `put_txn_log` and `put_combined_txn_log` paths). If writing a transaction log takes longer than this value, StarRocks dumps the stack trace of the slow thread to the BE log to help diagnose slow object-storage writes. Disabled by default (a value less than or equal to `0` disables the guard); set it to a positive value such as `4000` (4 seconds) to enable. * Introduced in: - ##### lake\_rows\_mapper\_read\_parallelism[​](#lake_rows_mapper_read_parallelism "Direct link to lake_rows_mapper_read_parallelism") * Default: 32 * Type: Int * Unit: sub-chunks * Is mutable: Yes * Description: Maximum number of in-flight `.lcrm` (lake compaction rows-mapper) sub-chunk reads kept by `RowsMapperIterator` during light Primary Key compaction publish in a shared-data cluster. Each sub-chunk is `lake_rows_mapper_sub_chunk_bytes` in size and never crosses a segment boundary; the iterator submits up to this many reads to the PK index execution thread pool and pipelines them against the caller's per-segment processing. Memory bound is `lake_rows_mapper_read_parallelism * lake_rows_mapper_sub_chunk_bytes`. Set to `1` to disable pipelining and fall back to sequential reads. ##### lake\_rows\_mapper\_sub\_chunk\_bytes[​](#lake_rows_mapper_sub_chunk_bytes "Direct link to lake_rows_mapper_sub_chunk_bytes") * Default: 4194304 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: Sub-chunk granularity for `RowsMapperIterator` pipelined reads of `.lcrm` files during light Primary Key compaction publish in a shared-data cluster. Each output segment is split into `ceil(segment_bytes / lake_rows_mapper_sub_chunk_bytes)` sub-chunks pipelined independently. Smaller values raise the achievable parallelism for few-but-large output segments at the cost of more range reads and an extra memcpy on consume. Defaults to 4 MiB to align with the starcache disk-tier block size. ##### lake\_vacuum\_enable\_task\_timeout[​](#lake_vacuum_enable_task_timeout "Direct link to lake_vacuum_enable_task_timeout") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether Vacuum tasks honor the timeout carried in the request (`VacuumRequest.timeout_ms`) and abort themselves once it elapses. Set this item to `false` to let Vacuum tasks always run to completion no matter how long the FE caller waits. ##### lake\_vacuum\_min\_batch\_delete\_size[​](#lake_vacuum_min_batch_delete_size "Direct link to lake_vacuum_min_batch_delete_size") * Default: 200 * Type: Int64 * Unit: Number of files * Is mutable: Yes * Description: The number of stale files Vacuum batches into a single `DeleteObjects` request on a shared-data cluster. A larger batch amortizes per-call HTTP / auth / signing overhead and reduces the prefix-level request rate against the object store, at the cost of higher single-call latency and a larger replay cost when a transient error retries. Users running on AWS S3 are encouraged to raise this further (up to the protocol cap of 1000) where per-request server time is nearly batch-size insensitive. ##### loop\_count\_wait\_fragments\_finish[​](#loop_count_wait_fragments_finish "Direct link to loop_count_wait_fragments_finish") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of loops to be waited when the BE/CN process exits. Each loop is a fixed interval of 10 seconds. You can set it to `0` to disable the loop wait. From v3.4 onwards, this item is changed to mutable and its default value is changed from `0` to `2`. * Introduced in: v2.5 ##### max\_client\_cache\_size\_per\_host[​](#max_client_cache_size_per_host "Direct link to max_client_cache_size_per_host") * Default: 10 * Type: Int * Unit: entries (cached client instances) per host * Is mutable: No * Description: The maximum number of cached client instances retained for each remote host by BE-wide client caches. This single setting is used when creating BackendServiceClientCache, FrontendServiceClientCache, and BrokerServiceClientCache during ExecEnv initialization, so it limits the number of client stubs/connections kept per host across those caches. Raising this value reduces reconnects and stub creation overhead at the cost of increased memory and file-descriptor usage; lowering it saves resources but may increase connection churn. The value is read at startup and cannot be changed at runtime. Currently one shared setting controls all client cache types; separate per-cache configuration may be introduced later. * Introduced in: v3.2.0 ##### primary\_key\_index\_memory\_high\_percent[​](#primary_key_index_memory_high_percent "Direct link to primary_key_index_memory_high_percent") * Default: 75 * Type: Int * Unit: - * Is mutable: Yes * Description: The level to which a background thread periodically shrinks the primary key index cache in a shared-data cluster, expressed as a percentage of the cache capacity. The cache capacity equals the memory limit for updates (see `update_memory_limit_percent`). When the cache holds more than this percentage, the least recently used primary key indexes that are not in use are evicted until the cache is within this percentage. Indexes in use are never evicted. Set it to a value lower than `primary_key_index_memory_urgent_percent`. * Introduced in: - ##### primary\_key\_index\_memory\_middle\_percent[​](#primary_key_index_memory_middle_percent "Direct link to primary_key_index_memory_middle_percent") * Default: 50 * Type: Int * Unit: - * Is mutable: Yes * Description: The level, expressed as a percentage of the cache capacity, to which the primary key index cache is shrunk when a publish of a Primary Key table in a shared-data cluster fails because it exceeds the memory limit. The primary key indexes that are not in use are evicted until the cache is within this percentage, which frees memory for the retried publish. Set it to a value lower than `primary_key_index_memory_high_percent`. * Introduced in: - ##### primary\_key\_index\_memory\_urgent\_percent[​](#primary_key_index_memory_urgent_percent "Direct link to primary_key_index_memory_urgent_percent") * Default: 85 * Type: Int * Unit: - * Is mutable: Yes * Description: The upper limit of the primary key index cache in a shared-data cluster, expressed as a percentage of the cache capacity. It is checked after every publish that does not fail because it exceeds the memory limit: when the cache holds more than this percentage, the least recently used primary key indexes that are not in use are evicted until the cache is within this percentage. It is a backstop for when the cache grows faster than the background thread brings it down to `primary_key_index_memory_high_percent`, so set it to a value higher than that item. * Introduced in: - ##### starlet\_filesystem\_instance\_cache\_capacity[​](#starlet_filesystem_instance_cache_capacity "Direct link to starlet_filesystem_instance_cache_capacity") * Default: 10000 * Type: Int * Unit: - * Is mutable: Yes * Description: The cache capacity of starlet filesystem instances. * Introduced in: v3.2.16, v3.3.11, v3.4.1 ##### starlet\_filesystem\_instance\_cache\_ttl\_sec[​](#starlet_filesystem_instance_cache_ttl_sec "Direct link to starlet_filesystem_instance_cache_ttl_sec") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The cache expiration time of starlet filesystem instances. * Introduced in: v3.3.15, 3.4.5 ##### starlet\_fslib\_azure\_storage\_max\_single\_part\_size[​](#starlet_fslib_azure_storage_max_single_part_size "Direct link to starlet_fslib_azure_storage_max_single_part_size") * Default: 104857600 * Type: Int64 * Unit: Bytes * Is mutable: Yes * Description: In a shared-data cluster, the object size above which files written to Azure Blob Storage or ADLS Gen2 use a multipart upload instead of a single upload request. Both Azure filesystems share this item. BE buffers up to this many bytes in memory per output stream before switching to multipart, so raising it raises memory usage, multiplied by the number of concurrent output streams. Raising it high enough to avoid multipart entirely can also fail uploads outright if it exceeds the object store's single-request upload limit; the upload then fails with the backend's own error. Changing it takes effect on uploads already in progress. The value must be greater than 0; a non-positive value is rejected. A valid value here overrides an equivalent `--fslib_*` gflag passed on the BE command line. A value rejected at BE startup is not applied at all: BE keeps the previously effective value and logs a warning, so this item can report a value that is not the one in effect. * Introduced in: v4.1.5, v4.2 ##### starlet\_fslib\_azure\_storage\_min\_upload\_part\_size[​](#starlet_fslib_azure_storage_min_upload_part_size "Direct link to starlet_fslib_azure_storage_min_upload_part_size") * Default: 5242880 * Type: Int64 * Unit: Bytes * Is mutable: Yes * Description: In a shared-data cluster, the block size used for multipart uploads to Azure Blob Storage or ADLS Gen2. Both Azure filesystems share this item. BE buffers up to this many bytes in memory per output stream between block uploads, so raising it raises memory usage, multiplied by the number of concurrent output streams. Setting it very low increases request count and can exceed the block limits of the storage service. Changing it takes effect on uploads already in progress. The value must be greater than 0; a non-positive value is rejected. A valid value here overrides an equivalent `--fslib_*` gflag passed on the BE command line. A value rejected at BE startup is not applied at all: BE keeps the previously effective value and logs a warning, so this item can report a value that is not the one in effect. * Introduced in: v4.1.5, v4.2 ##### starlet\_fslib\_gcs\_max\_single\_part\_size[​](#starlet_fslib_gcs_max_single_part_size "Direct link to starlet_fslib_gcs_max_single_part_size") * Default: 104857600 * Type: Int64 * Unit: Bytes * Is mutable: Yes * Description: In a shared-data cluster, the object size above which files written to Google Cloud Storage use a streaming upload instead of a single insert request. BE buffers up to this many bytes in memory per output stream before switching to streaming, so raising it raises memory usage, multiplied by the number of concurrent output streams. Raising it high enough to avoid streaming entirely can also fail uploads outright if it exceeds the object store's single-request upload limit; the upload then fails with the backend's own error. Changing it takes effect on uploads already in progress. The value must be greater than 0; a non-positive value is rejected. A valid value here overrides an equivalent `--fslib_*` gflag passed on the BE command line. A value rejected at BE startup is not applied at all: BE keeps the previously effective value and logs a warning, so this item can report a value that is not the one in effect. * Introduced in: v4.1.5, v4.2 ##### starlet\_fslib\_s3\_max\_single\_part\_size[​](#starlet_fslib_s3_max_single_part_size "Direct link to starlet_fslib_s3_max_single_part_size") * Default: 104857600 * Type: Int64 * Unit: Bytes * Is mutable: Yes * Description: In a shared-data cluster, the object size above which files written to S3 or an S3-compatible object store use a multipart upload instead of a single `PutObject` request. BE buffers up to this many bytes in memory per output stream before switching to multipart, so raising it raises memory usage, multiplied by the number of concurrent output streams. Raising it too high to avoid multipart entirely can also fail uploads outright: AWS S3 rejects a single `PutObject` larger than 5 GiB, and the limits of S3-compatible object stores differ. Changing it takes effect on uploads already in progress. The value must be greater than 0; a non-positive value is rejected. A valid value here overrides an equivalent `--fslib_*` gflag passed on the BE command line. A value rejected at BE startup is not applied at all: BE keeps the previously effective value and logs a warning, so this item can report a value that is not the one in effect. This item controls starlet uploads in a shared-data cluster and is unrelated to the `experimental_s3_*` items. * Introduced in: v4.1.5, v4.2 ##### starlet\_fslib\_s3\_min\_upload\_part\_size[​](#starlet_fslib_s3_min_upload_part_size "Direct link to starlet_fslib_s3_min_upload_part_size") * Default: 5242880 * Type: Int64 * Unit: Bytes * Is mutable: Yes * Description: In a shared-data cluster, the part size used for multipart uploads to S3 or an S3-compatible object store. BE buffers up to this many bytes in memory per output stream between part uploads, so raising it raises memory usage, multiplied by the number of concurrent output streams. Setting it very low increases request count and can exceed the 10,000-part limit or fall below the 5 MB minimum part size that AWS S3 enforces; the limits of other S3-compatible stores differ. Changing it takes effect on uploads already in progress. The value must be greater than 0; a non-positive value is rejected. A valid value here overrides an equivalent `--fslib_*` gflag passed on the BE command line. A value rejected at BE startup is not applied at all: BE keeps the previously effective value and logs a warning, so this item can report a value that is not the one in effect. This item controls starlet uploads in a shared-data cluster and is unrelated to the `experimental_s3_*` items. * Introduced in: v4.1.5, v4.2 ##### starlet\_port[​](#starlet_port "Direct link to starlet_port") * Default: 9070 * Type: Int * Unit: - * Is mutable: No * Description: An extra agent service port for BE and CN. * Introduced in: - ##### starlet\_star\_cache\_disk\_size\_percent[​](#starlet_star_cache_disk_size_percent "Direct link to starlet_star_cache_disk_size_percent") * Default: 80 * Type: Int * Unit: - * Is mutable: No * Description: The percentage of disk capacity that Data Cache can use at most in a shared-data cluster. Only takes effect when `datacache_unified_instance_enable` is `false`. * Introduced in: v3.1 ##### starlet\_use\_star\_cache[​](#starlet_use_star_cache "Direct link to starlet_use_star_cache") * Default: false in v3.1 and true from v3.2.3 * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Data Cache in a shared-data cluster. `true` indicates enabling this feature and `false` indicates disabling it. The default value is set from `false` to `true` from v3.2.3 onwards. * Introduced in: v3.1 ##### starlet\_write\_file\_with\_tag[​](#starlet_write_file_with_tag "Direct link to starlet_write_file_with_tag") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: In a shared-data cluster, whether to tag files written to object storage with object storage tags for convenient custom file management. * Introduced in: v3.5.3 ##### table\_schema\_service\_max\_retries[​](#table_schema_service_max_retries "Direct link to table_schema_service_max_retries") * Default: 3 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of retries for Table Schema Service requests. * Introduced in: v4.1 #### Data Lake[​](#data-lake "Direct link to Data Lake") ##### datacache\_block\_buffer\_enable[​](#datacache_block_buffer_enable "Direct link to datacache_block_buffer_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to enable Block Buffer to optimize Data Cache efficiency. When Block Buffer is enabled, the system reads the Block data from the Data Cache and caches it in a temporary buffer, thus reducing the extra overhead caused by frequent cache reads. * Introduced in: v3.2.0 ##### datacache\_disk\_adjust\_interval\_seconds[​](#datacache_disk_adjust_interval_seconds "Direct link to datacache_disk_adjust_interval_seconds") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The interval of Data Cache automatic capacity scaling. At regular intervals, the system checks the cache disk usage, and triggers Automatic Scaling when necessary. * Introduced in: v3.3.0 ##### datacache\_disk\_idle\_seconds\_for\_expansion[​](#datacache_disk_idle_seconds_for_expansion "Direct link to datacache_disk_idle_seconds_for_expansion") * Default: 7200 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The minimum wait time for Data Cache automatic expansion. Automatic scaling up is triggered only if the disk usage remains below `datacache_disk_low_level` for longer than this duration. * Introduced in: v3.3.0 ##### datacache\_disk\_size[​](#datacache_disk_size "Direct link to datacache_disk_size") * Default: 0 * Type: String * Unit: - * Is mutable: Yes * Description: The maximum amount of data that can be cached on a single disk. You can set it as a percentage (for example, `80%`) or a physical limit (for example, `2T`, `500G`). For example, if you use two disks and set the value of the `datacache_disk_size` parameter as `21474836480` (20 GB), a maximum of 40 GB data can be cached on these two disks. The default value is `0`, which indicates that only memory is used to cache data. * Introduced in: - ##### datacache\_enable[​](#datacache_enable "Direct link to datacache_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to enable Data Cache. `true` indicates Data Cache is enabled, and `false` indicates Data Cache is disabled. The default value is changed to `true` from v3.3. * Introduced in: - ##### datacache\_eviction\_policy[​](#datacache_eviction_policy "Direct link to datacache_eviction_policy") * Default: slru * Type: String * Unit: - * Is mutable: No * Description: The eviction policy of Data Cache. Valid values: `lru` (least recently used) and `slru` (Segmented LRU). * Introduced in: v3.4.0 ##### datacache\_inline\_item\_count\_limit[​](#datacache_inline_item_count_limit "Direct link to datacache_inline_item_count_limit") * Default: 130172 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of inline cache items in Data Cache. For some particularly small cache blocks, Data Cache stores them in `inline` mode, which caches the block data and metadata together in memory. * Introduced in: v3.4.0 ##### datacache\_mem\_size[​](#datacache_mem_size "Direct link to datacache_mem_size") * Default: 0 * Type: String * Unit: - * Is mutable: Yes * Description: The maximum amount of data that can be cached in memory. You can set it as a percentage (for example, `10%`) or a physical limit (for example, `10G`, `21474836480`). * Introduced in: - ##### datacache\_min\_disk\_quota\_for\_adjustment[​](#datacache_min_disk_quota_for_adjustment "Direct link to datacache_min_disk_quota_for_adjustment") * Default: 10737418240 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The minimum effective capacity for Data Cache Automatic Scaling. If the system tries to adjust the cache capacity to less than this value, the cache capacity will be directly set to `0` to prevent suboptimal performance caused by frequent cache fills and evictions due to insufficient cache capacity. * Introduced in: v3.3.0 ##### disk\_high\_level[​](#disk_high_level "Direct link to disk_high_level") * Default: 90 * Type: Int * Unit: - * Is mutable: Yes * Description: The upper limit of disk usage (in percentage) that triggers the automatic scaling up of the cache capacity. When the disk usage exceeds this value, the system automatically evicts cache data from the Data Cache. From v3.4.0 onwards, the default value is changed from `80` to `90`. This item is renamed from `datacache_disk_high_level` to `disk_high_level` from v4.0 onwards. * Introduced in: v3.3.0 ##### disk\_low\_level[​](#disk_low_level "Direct link to disk_low_level") * Default: 60 * Type: Int * Unit: - * Is mutable: Yes * Description: The lower limit of disk usage (in percentage) that triggers the automatic scaling down of the cache capacity. When the disk usage remains below this value for the period specified in `datacache_disk_idle_seconds_for_expansion`, and the space allocated for Data Cache is fully utilized, the system will automatically expand the cache capacity by increasing the upper limit. This item is renamed from `datacache_disk_low_level` to `disk_low_level` from v4.0 onwards. * Introduced in: v3.3.0 ##### disk\_safe\_level[​](#disk_safe_level "Direct link to disk_safe_level") * Default: 80 * Type: Int * Unit: - * Is mutable: Yes * Description: The safe level of disk usage (in percentage) for Data Cache. When Data Cache performs automatic scaling, the system adjusts the cache capacity with the goal of maintaining disk usage as close to this value as possible. From v3.4.0 onwards, the default value is changed from `70` to `80`. This item is renamed from `datacache_disk_safe_level` to `disk_safe_level` from v4.0 onwards. * Introduced in: v3.3.0 ##### enable\_connector\_sink\_spill[​](#enable_connector_sink_spill "Direct link to enable_connector_sink_spill") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Spilling for writes to external tables. Enabling this feature prevents the generation of a large number of small files as a result of writing to an external table when memory is insufficient. Currently, this feature only supports writing to Iceberg tables. * Introduced in: v4.0.0 ##### enable\_datacache\_disk\_auto\_adjust[​](#enable_datacache_disk_auto_adjust "Direct link to enable_datacache_disk_auto_adjust") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Automatic Scaling for Data Cache disk capacity. When it is enabled, the system dynamically adjusts the cache capacity based on the current disk usage rate. This item is renamed from `datacache_auto_adjust_enable` to `enable_datacache_disk_auto_adjust` from v4.0 onwards. * Introduced in: v3.3.0 ##### datacache\_unified\_instance\_enable[​](#datacache_unified_instance_enable "Direct link to datacache_unified_instance_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to use a unified Data Cache instance to manage data caching for both internal catalog and external catalog in a shared-data cluster. * Introduced in: v3.4.0 ##### jdbc\_connection\_idle\_timeout\_ms[​](#jdbc_connection_idle_timeout_ms "Direct link to jdbc_connection_idle_timeout_ms") * Default: 600000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: The length of time after which an idle connection in the JDBC connection pool expires. If the connection idle time in the JDBC connection pool exceeds this value, the connection pool closes idle connections beyond the number specified in the configuration item `jdbc_minimum_idle_connections`. * Introduced in: - ##### jdbc\_connection\_pool\_size[​](#jdbc_connection_pool_size "Direct link to jdbc_connection_pool_size") * Default: 8 * Type: Int * Unit: - * Is mutable: No * Description: The JDBC connection pool size. On each BE node, queries that access the external table with the same `jdbc_url` share the same connection pool. * Introduced in: - ##### jdbc\_minimum\_idle\_connections[​](#jdbc_minimum_idle_connections "Direct link to jdbc_minimum_idle_connections") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: The minimum number of idle connections in the JDBC connection pool. * Introduced in: - ##### jdbc\_connection\_max\_lifetime\_ms[​](#jdbc_connection_max_lifetime_ms "Direct link to jdbc_connection_max_lifetime_ms") * Default: 300000 * Type: Long * Unit: Milliseconds * Is mutable: No * Description: Maximum lifetime of a connection in the JDBC connection pool. Connections are recycled before this timeout to prevent stale connections. Minimum allowed value is 30000 (30 seconds). * Introduced in: - ##### jdbc\_connection\_keepalive\_time\_ms[​](#jdbc_connection_keepalive_time_ms "Direct link to jdbc_connection_keepalive_time_ms") * Default: 30000 * Type: Long * Unit: Milliseconds * Is mutable: No * Description: Keepalive interval for idle JDBC connections. Idle connections are tested at this interval to detect stale connections proactively. Set to 0 to disable keepalive probing. When enabled, must be >= 30000 and less than `jdbc_connection_max_lifetime_ms`. Invalid enabled values are silently disabled (reset to 0). * Introduced in: - ##### lake\_clear\_corrupted\_cache\_data[​](#lake_clear_corrupted_cache_data "Direct link to lake_clear_corrupted_cache_data") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow the system to clear the corrupted data cache in a shared-data cluster. * Introduced in: v3.4 ##### lake\_pk\_index\_sst\_verify\_checksum[​](#lake_pk_index_sst_verify_checksum "Direct link to lake_pk_index_sst_verify_checksum") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to verify sstable block checksums when reading the cloud-native Primary Key index (point lookups and compaction merges) in a shared-data cluster. When enabled, corrupted bytes (usually from a corrupted local cache copy) deterministically fail as a Corruption error and trigger the corrupted-cache cleanup, instead of being misparsed or silently returning wrong index values. Note that the automatic cleanup of the corrupted cache additionally requires `lake_clear_corrupted_cache_data` to be enabled. * Introduced in: v4.1 ##### lake\_clear\_corrupted\_cache\_meta[​](#lake_clear_corrupted_cache_meta "Direct link to lake_clear_corrupted_cache_meta") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow the system to clear the corrupted metadata cache in a shared-data cluster. This covers the local cache of tablet metadata files and transaction log files. When a read of such a file fails with a corruption error, the system drops the cached copy and reads the file again from remote storage once. * Introduced in: v3.3 ##### lake\_enable\_vertical\_compaction\_fill\_data\_cache[​](#lake_enable_vertical_compaction_fill_data_cache "Direct link to lake_enable_vertical_compaction_fill_data_cache") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow vertical compaction tasks to cache data on local disks in a shared-data cluster. * Introduced in: v3.1.7, v3.2.3 ##### lake\_replication\_read\_buffer\_size[​](#lake_replication_read_buffer_size "Direct link to lake_replication_read_buffer_size") * Default: 16777216 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: The read buffer size used when downloading lake segment files during lake replication. This value determines the per-read allocation for reading remote files; the implementation uses the larger of this setting and a 1 MB minimum. A larger value reduces the number of read calls and can improve throughput but increases memory used per concurrent download; a smaller value lowers memory usage at the cost of more I/O calls. Tune according to network bandwidth, storage I/O characteristics, and the number of parallel replication threads. * Introduced in: v4.1.2 ##### lake\_replication\_max\_file\_copy\_retry[​](#lake_replication_max_file_copy_retry "Direct link to lake_replication_max_file_copy_retry") * Default: 3 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of retry attempts for non-segment file copy (`.sst`, `.delvec`, `.del`, `.cols`) during lake-to-lake (shared-data) cross-cluster replication. Each attempt verifies the copied file size matches the source to detect truncated copies caused by transient object storage issues. Increase this value if experiencing intermittent file corruption during replication over unreliable storage. * Introduced in: v4.1.2 ##### lake\_replication\_file\_copy\_threads[​](#lake_replication_file_copy_threads "Direct link to lake_replication_file_copy_threads") * Default: 16 * Type: Int * Unit: - * Is mutable: No * Description: Number of threads in the CN-wide dedicated thread pool used by lake-to-lake (shared-data) cross-cluster replication for per-file copy. The default caps concurrent copy tasks and their read buffers independently of CPU count. `0` means `cpu_cores * 4`; negative values mean `-value * cpu_cores`. The pool is built once at startup, so CN restart is required to change its size. * Introduced in: v4.1.2 ##### lake\_replication\_max\_parallel\_files\_per\_tablet[​](#lake_replication_max_parallel_files_per_tablet "Direct link to lake_replication_max_parallel_files_per_tablet") * Default: 4 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of files copied concurrently for one tablet during lake-to-lake (shared-data) cross-cluster replication. All tablets share the CN-wide `lake_replication_file_copy_threads` pool, so this setting prevents one tablet from monopolizing the pool. Values less than or equal to `1` serialize file copy within each tablet. * Introduced in: v4.1.4 ##### lake\_replication\_parallel\_copy\_min\_file\_count[​](#lake_replication_parallel_copy_min_file_count "Direct link to lake_replication_parallel_copy_min_file_count") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: Minimum number of files in a tablet required to copy that tablet's files concurrently during lake-to-lake (shared-data) cross-cluster replication. Set this parameter to `0` to disable the dedicated file copy pool and execute file copy in the calling replication task. * Introduced in: v4.1.2 ##### lake\_service\_max\_concurrency[​](#lake_service_max_concurrency "Direct link to lake_service_max_concurrency") * Default: 0 * Type: Int * Unit: - * Is mutable: No * Description: The maximum concurrency of RPC requests in a shared-data cluster. Incoming requests will be rejected when this threshold is reached. When this item is set to `0`, no limit is imposed on the concurrency. * Introduced in: - ##### max\_hdfs\_scanner\_num[​](#max_hdfs_scanner_num "Direct link to max_hdfs_scanner_num") * Default: 50 * Type: Int * Unit: - * Is mutable: No * Description: Limits the maximum number of concurrently running connector (HDFS/remote) scanners that a ConnectorScanNode can have. During scan startup the node computes an estimated concurrency (based on memory, chunk size and scanner\_row\_num) and then caps it with this value to determine how many scanners and chunks to reserve and how many scanner threads to start. It is also consulted when scheduling pending scanners at runtime (to avoid oversubscription) and when deciding how many pending scanners can be re-submitted considering file-handle limits. Lowering this reduces threads, memory and open-file pressure at the cost of potential throughput; increasing it raises concurrency and resource usage. * Introduced in: v3.2.0 ##### query\_max\_memory\_limit\_percent[​](#query_max_memory_limit_percent "Direct link to query_max_memory_limit_percent") * Default: 90 * Type: Int * Unit: - * Is mutable: No * Description: The maximum memory that the Query Pool can use. It is expressed as a percentage of the Process memory limit. * Introduced in: v3.1.0 ##### rocksdb\_max\_write\_buffer\_memory\_bytes[​](#rocksdb_max_write_buffer_memory_bytes "Direct link to rocksdb_max_write_buffer_memory_bytes") * Default: 1073741824 * Type: Int64 * Unit: - * Is mutable: No * Description: It is the max size of the write buffer for meta in rocksdb. Default is 1GB. * Introduced in: v3.5.0 ##### rocksdb\_write\_buffer\_memory\_percent[​](#rocksdb_write_buffer_memory_percent "Direct link to rocksdb_write_buffer_memory_percent") * Default: 5 * Type: Int64 * Unit: - * Is mutable: No * Description: It is the memory percent of write buffer for meta in rocksdb. default is 5% of system memory. However, aside from this, the final calculated size of the write buffer memory will not be less than 64MB nor exceed 1G (rocksdb\_max\_write\_buffer\_memory\_bytes) * Introduced in: v3.5.0 #### Other[​](#other "Direct link to Other") ##### default\_mv\_resource\_group\_concurrency\_limit[​](#default_mv_resource_group_concurrency_limit "Direct link to default_mv_resource_group_concurrency_limit") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum concurrency (per BE node) of the materialized view refresh tasks in the resource group `default_mv_wg`. The default value `0` indicates no limits. * Introduced in: v3.1 ##### default\_mv\_resource\_group\_cpu\_limit[​](#default_mv_resource_group_cpu_limit "Direct link to default_mv_resource_group_cpu_limit") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of CPU cores (per BE node) that can be used by the materialized view refresh tasks in the resource group `default_mv_wg`. * Introduced in: v3.1 ##### default\_mv\_resource\_group\_memory\_limit[​](#default_mv_resource_group_memory_limit "Direct link to default_mv_resource_group_memory_limit") * Default: 0.8 * Type: Double * Unit: * Is mutable: Yes * Description: The maximum memory proportion (per BE node) that can be used by the materialized view refresh tasks in the resource group `default_mv_wg`. The default value indicates 80% of the memory. * Introduced in: v3.1 ##### default\_mv\_resource\_group\_spill\_mem\_limit\_threshold[​](#default_mv_resource_group_spill_mem_limit_threshold "Direct link to default_mv_resource_group_spill_mem_limit_threshold") * Default: 0.8 * Type: Double * Unit: - * Is mutable: Yes * Description: The memory usage threshold before a materialized view refresh task in the resource group `default_mv_wg` triggers intermediate result spilling. The default value indicates 80% of the memory. * Introduced in: v3.1 ##### enable\_resolve\_hostname\_to\_ip\_in\_load\_error\_url[​](#enable_resolve_hostname_to_ip_in_load_error_url "Direct link to enable_resolve_hostname_to_ip_in_load_error_url") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: For `error_urls` debugging, whether to allow operators to choose between using original hostnames from FE heartbeat or forcing resolution to IP addresses based on their environment needs. * `true`: Resolve hostnames to IPs. * `false` (Default): Keeps the original hostname in the error URL. * Introduced in: v4.0.1 ##### enable\_retry\_apply[​](#enable_retry_apply "Direct link to enable_retry_apply") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, Tablet apply failures that are classified as retryable (for example transient memory-limit errors) are rescheduled for retry instead of immediately marking the tablet in error. The retry path in TabletUpdates schedules the next attempt using `retry_apply_interval_second` multiplied by the current failure count and clamped to a 600s maximum, so backoff grows with successive failures. Explicitly non-retryable errors (for example corruption) bypass retries and cause the apply process to enter the error state immediately. Retries continue until an overall timeout/terminal condition is reached, after which the apply will enter the error state. Turning this off disables automatic rescheduling of failed apply tasks and causes failed applies to transition to error state without retries. * Introduced in: v3.2.9 ##### enable\_token\_check[​](#enable_token_check "Direct link to enable_token_check") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: A boolean value to control whether to enable the token check. `true` indicates enabling the token check, and `false` indicates disabling it. * Introduced in: - ##### es\_scroll\_keepalive[​](#es_scroll_keepalive "Direct link to es_scroll_keepalive") * Default: 5m * Type: String * Unit: Minutes (string with suffix, e.g. "5m") * Is mutable: No * Description: The keep-alive duration sent to Elasticsearch for scroll search contexts. The value is used verbatim (for example "5m") when building the initial scroll URL (`?scroll=`) and when sending subsequent scroll requests (via ESScrollQueryBuilder). This controls how long the ES search context is retained before garbage collection on the ES side; setting it longer keeps scroll contexts alive for more time but prolongs resource usage on the ES cluster. The value is read at startup by the ES scan reader and is not changeable at runtime. * Introduced in: v3.2.0 ##### load\_replica\_status\_check\_interval\_ms\_on\_failure[​](#load_replica_status_check_interval_ms_on_failure "Direct link to load_replica_status_check_interval_ms_on_failure") * Default: 2000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The interval that the secondary replica checks it's status on the primary replica if the last check rpc fails. * Introduced in: v3.5.1 ##### load\_replica\_status\_check\_interval\_ms\_on\_success[​](#load_replica_status_check_interval_ms_on_success "Direct link to load_replica_status_check_interval_ms_on_success") * Default: 15000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The interval that the secondary replica checks it's status on the primary replica if the last check rpc successes. * Introduced in: v3.5.1 ##### max\_length\_for\_bitmap\_function[​](#max_length_for_bitmap_function "Direct link to max_length_for_bitmap_function") * Default: 1000000 * Type: Int * Unit: Bytes * Is mutable: No * Description: The maximum length of input values for bitmap functions. * Introduced in: - ##### max\_length\_for\_to\_base64[​](#max_length_for_to_base64 "Direct link to max_length_for_to_base64") * Default: 200000 * Type: Int * Unit: Bytes * Is mutable: No * Description: The maximum length of input values for the to\_base64() function. * Introduced in: - ##### memory\_high\_level[​](#memory_high_level "Direct link to memory_high_level") * Default: 75 * Type: Long * Unit: Percent * Is mutable: Yes * Description: High water memory threshold expressed as a percentage of the process memory limit. When total memory consumption rises above this percentage, BE begins to free memory gradually (currently by evicting data cache and update cache) to relieve pressure. The monitor uses this value to compute memory\_high = mem\_limit \* memory\_high\_level / 100 and, if consumption `>` memory\_high, performs controlled eviction guided by the GC advisor; if consumption exceeds memory\_urgent\_level (a separate config), more aggressive immediate reductions occur. This value is also consulted to disable certain memory‑intensive operations (for example, primary-key preload) when the threshold is exceeded. Must satisfy validation with memory\_urgent\_level (memory\_urgent\_level `>` memory\_high\_level, memory\_high\_level `>=` 1, memory\_urgent\_level `<=` 100). * Introduced in: v3.2.0 ##### report\_exec\_rpc\_request\_retry\_num[​](#report_exec_rpc_request_retry_num "Direct link to report_exec_rpc_request_retry_num") * Default: 10 * Type: Int * Unit: - * Is mutable: Yes * Description: The retry times of rpc request to report exec rpc request to FE. The default value is 10, which means that the rpc request will be retried 10 times if it fails only if it's fragment instatnce finish rpc. Report exec rpc request is important for load job, if one fragment instance finish report failed, the load job will be hang until timeout. * Introduced in: - ##### sleep\_one\_second[​](#sleep_one_second "Direct link to sleep_one_second") * Default: 1 * Type: Int * Unit: Seconds * Is mutable: No * Description: A small, global sleep interval (in seconds) used by BE agent worker threads as a one-second pause when the master address/heartbeat is not yet available or when a short retry/backoff is needed. In the codebase it is referenced by several report worker pools (e.g., ReportDiskStateTaskWorkerPool, ReportOlapTableTaskWorkerPool, ReportWorkgroupTaskWorkerPool) to avoid busy-waiting and reduce CPU consumption while retrying. Increasing this value slows the retry frequency and responsiveness to master availability; reducing it increases polling rate and CPU usage. Adjust only with awareness of the trade-off between responsiveness and resource use. * Introduced in: v3.2.0 ##### small\_file\_dir[​](#small_file_dir "Direct link to small_file_dir") * Default: `${STARROCKS_HOME}/lib/small_file/` * Type: String * Unit: - * Is mutable: No * Description: The directory used to store the files downloaded by the file manager. * Introduced in: - ##### upload\_buffer\_size[​](#upload_buffer_size "Direct link to upload_buffer_size") * Default: 4194304 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: Buffer size (in bytes) used by file copy operations when uploading snapshot files to remote storage (broker or direct FileSystem). In the upload path (snapshot\_loader.cpp) this value is passed to fs::copy as the read/write chunk size for each upload stream. The default is 4 MiB. Increasing this value can improve throughput on high-latency or high-bandwidth links but increases memory usage per concurrent upload; decreasing it reduces per-stream memory but may reduce transfer efficiency. Tune together with upload\_worker\_count and overall available memory. * Introduced in: v3.2.13 ##### user\_function\_dir[​](#user_function_dir "Direct link to user_function_dir") * Default: `${STARROCKS_HOME}/lib/udf` * Type: String * Unit: - * Is mutable: No * Description: The directory used to store User-defined Functions (UDFs). * Introduced in: - ##### web\_log\_bytes[​](#web_log_bytes "Direct link to web_log_bytes") * Default: 1048576 (1 MB) * Type: long * Unit: Bytes * Is mutable: No * Description: Maximum number of bytes to read from the INFO logfile and show on the BE debug webserver's log page. The handler uses this value to compute a seek offset (showing the last N bytes) to avoid reading or serving very large log files. If the logfile is smaller than this value the whole file is shown. Note: in the current implementation the code that reads and serves the INFO log is commented out and the handler reports that the INFO log file couldn't be opened, so this parameter may have no effect unless the log-serving code is enabled. * Introduced in: v3.2.0 --- ### BE Configuration - Statistics and Storage Some BE configuration items are dynamic parameters which you can set interactively when BE nodes are online. The rest of them are static parameters. You can only set the static parameters of a BE node by changing them in the corresponding configuration file **be.conf** and restarting the BE node to allow the change to take effect. #### View BE configuration items[​](#view-be-configuration-items "Direct link to View BE configuration items") You can view the BE configuration items using the following command: ```sql SELECT * FROM information_schema.be_configs [WHERE NAME LIKE "%%"] ``` #### Configure BE parameters[​](#configure-be-parameters "Direct link to Configure BE parameters") ##### Configure BE dynamic parameters[​](#configure-be-dynamic-parameters "Direct link to Configure BE dynamic parameters") You can configure a dynamic parameter of a BE node by updating the value in `information_schema.be_configs`. warning Setting an invalid value may cause unknown behaviors. Check twice before you run the command to update the configuration. ```sql -- Replace with the key of the configuration and with the value. UPDATE information_schema.be_configs SET VALUE = WHERE name = ""; ``` ##### Configure BE static parameters[​](#configure-be-static-parameters "Direct link to Configure BE static parameters") You can only set the static parameters of a BE by changing them in the corresponding configuration file **be.conf**, and restarting the BE to allow the changes to take effect. *** This topic introduces the following types of BE configurations: * [Statistic report](#statistic-report) * [Storage](#storage) #### Statistic report[​](#statistic-report "Direct link to Statistic report") ##### enable\_metric\_calculator[​](#enable_metric_calculator "Direct link to enable_metric_calculator") * Default: true * Type: boolean * Unit: - * Is mutable: No * Description: When true, the BE process launches a background "metrics\_daemon" thread (started in Daemon::init on non-Apple platforms) that runs every ~15 seconds to invoke `StarRocksMetrics::instance()->metrics()->trigger_hook()` and compute derived/system metrics (e.g., push/query bytes/sec, max disk I/O util, max network send/receive rates), log memory breakdowns and run table metrics cleanup. When false, those hooks are executed synchronously inside `MetricRegistry::collect` at metric collection time, which can increase metric-scrape latency. Requires process restart to take effect. * Introduced in: v3.2.0 ##### enable\_system\_metrics[​](#enable_system_metrics "Direct link to enable_system_metrics") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: When true, StarRocks initializes system-level monitoring during startup: it discovers disk devices from the configured store paths and enumerates network interfaces, then passes this information into the metrics subsystem to enable collection of disk I/O, network traffic and memory-related system metrics. If device or interface discovery fails, initialization logs a warning and aborts system metrics setup. This flag only controls whether system metrics are initialized; periodic metric aggregation threads are controlled separately by `enable_metric_calculator`, and JVM metrics initialization is controlled by `enable_jvm_metrics`. Changing this value requires a restart. * Introduced in: v3.2.0 ##### profile\_report\_interval[​](#profile_report_interval "Direct link to profile_report_interval") * Default: 30 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Interval in seconds that the ProfileReportWorker uses to (1) decide when to report per-fragment profile information for LOAD queries and (2) sleep between reporting cycles. The worker compares current time against each task's last\_report\_time using (profile\_report\_interval \* 1000) ms to determine if a profile should be re-reported for both non-pipeline and pipeline load tasks. At each loop the worker reads the current value (mutable at runtime); if the configured value is less than or euqual to 0 the worker forces it to 1 and emits a warning. Changing this value affects the next reporting decision and sleep duration. * Introduced in: v3.2.0 ##### report\_disk\_state\_interval\_seconds[​](#report_disk_state_interval_seconds "Direct link to report_disk_state_interval_seconds") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to report the storage volume state, which includes the size of data within the volume. * Introduced in: - ##### report\_resource\_usage\_interval\_ms[​](#report_resource_usage_interval_ms "Direct link to report_resource_usage_interval_ms") * Default: 1000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Interval, in milliseconds, between periodic resource-usage reports sent by the BE agent to the FE (master). The agent worker thread collects TResourceUsage (number of running queries, memory used/limit, CPU used permille, and resource-group usages) and calls report\_task, then sleeps for this configured interval (see task\_worker\_pool). Lower values increase reporting timeliness but raise CPU, network, and master load; higher values reduce overhead but make resource information less current. The reporting updates related metrics (report\_resource\_usage\_requests\_total, report\_resource\_usage\_requests\_failed). Tune according to cluster scale and FE load. * Introduced in: v3.2.0 ##### report\_tablet\_interval\_seconds[​](#report_tablet_interval_seconds "Direct link to report_tablet_interval_seconds") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to report the most updated version of all tablets. * Introduced in: - ##### report\_task\_interval\_seconds[​](#report_task_interval_seconds "Direct link to report_task_interval_seconds") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to report the state of a task. A task can be creating a table, dropping a table, loading data, or changing a table schema. * Introduced in: - ##### report\_workgroup\_interval\_seconds[​](#report_workgroup_interval_seconds "Direct link to report_workgroup_interval_seconds") * Default: 5 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to report the most updated version of all workgroups. * Introduced in: - #### Storage[​](#storage "Direct link to Storage") ##### alter\_tablet\_worker\_count[​](#alter_tablet_worker_count "Direct link to alter_tablet_worker_count") * Default: 3 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of threads used for Schema Change. * Introduced in: - ##### automatic\_partition\_thread\_pool\_thread\_num[​](#automatic_partition_thread_pool_thread_num "Direct link to automatic_partition_thread_pool_thread_num") * Default: 1000 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads in the automatic partition thread pool used for automatic partition creation during loading. The queue size of the pool is automatically set to 10 times the thread count. * Introduced in: - ##### avro\_ignore\_union\_type\_tag[​](#avro_ignore_union_type_tag "Direct link to avro_ignore_union_type_tag") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to strip the type tag from the JSON string serialized from the Avro Union data type. * Introduced in: v3.3.7, v3.4 ##### base\_compaction\_check\_interval\_seconds[​](#base_compaction_check_interval_seconds "Direct link to base_compaction_check_interval_seconds") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval of thread polling for a Base Compaction. * Introduced in: - ##### base\_compaction\_interval\_seconds\_since\_last\_operation[​](#base_compaction_interval_seconds_since_last_operation "Direct link to base_compaction_interval_seconds_since_last_operation") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval since the last Base Compaction. This configuration item is one of the conditions that trigger a Base Compaction. * Introduced in: - ##### base\_compaction\_num\_threads\_per\_disk[​](#base_compaction_num_threads_per_disk "Direct link to base_compaction_num_threads_per_disk") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads used for Base Compaction on each storage volume. * Introduced in: - ##### base\_cumulative\_delta\_ratio[​](#base_cumulative_delta_ratio "Direct link to base_cumulative_delta_ratio") * Default: 0.3 * Type: Double * Unit: - * Is mutable: Yes * Description: The ratio of cumulative file size to base file size. The ratio reaching this value is one of the conditions that trigger the Base Compaction. * Introduced in: - ##### chaos\_test\_enable\_random\_compaction\_strategy[​](#chaos_test_enable_random_compaction_strategy "Direct link to chaos_test_enable_random_compaction_strategy") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When this item is set to `true`, TabletUpdates::compaction() uses the random compaction strategy (compaction\_random) intended for chaos engineering tests. This flag forces compaction to follow a nondeterministic/random policy instead of normal strategies (e.g., size-tiered compaction), and takes precedence during compaction selection for the tablet. It is intended only for controlled testing: enabling it can produce unpredictable compaction order, increased I/O/CPU, and test flakiness. Do not enable in production; use only for fault-injection or chaos-test scenarios. * Introduced in: v3.3.12, 3.4.2, 3.5.0, 4.0.0 ##### check\_consistency\_worker\_count[​](#check_consistency_worker_count "Direct link to check_consistency_worker_count") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads used for checking the consistency of tablets. * Introduced in: - ##### clear\_expired\_replication\_snapshots\_interval\_seconds[​](#clear_expired_replication_snapshots_interval_seconds "Direct link to clear_expired_replication_snapshots_interval_seconds") * Default: 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which the system clears the expired snapshots left by abnormal replications. * Introduced in: v3.3.5 ##### compact\_threads[​](#compact_threads "Direct link to compact_threads") * Default: 4 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads used for concurrent compaction tasks. This configuration is changed to dynamic from v3.1.7 and v3.2.2 onwards. * Introduced in: v3.0.0 ##### compaction\_max\_memory\_limit[​](#compaction_max_memory_limit "Direct link to compaction_max_memory_limit") * Default: -1 * Type: Long * Unit: Bytes * Is mutable: No * Description: Global upper bound (in bytes) for memory available to compaction tasks on this BE. During BE initialization the final compaction memory limit is computed as min(`compaction_max_memory_limit`, process\_mem\_limit \* `compaction_max_memory_limit_percent` / 100). If `compaction_max_memory_limit` is negative (default `-1`) it falls back to the BE process memory limit derived from `mem_limit`. The percent value is clamped to \[0,100]. If the process memory limit is not set (negative) compaction memory remains unlimited (`-1`). This computed value is used to initialize the `_compaction_mem_tracker`. See also `compaction_max_memory_limit_percent` and `compaction_memory_limit_per_worker`. * Introduced in: v3.2.0 ##### compaction\_max\_memory\_limit\_percent[​](#compaction_max_memory_limit_percent "Direct link to compaction_max_memory_limit_percent") * Default: 100 * Type: Int * Unit: Percent * Is mutable: No * Description: Percentage of the BE process memory that may be used for compaction. The BE computes the compaction memory cap as the minimum of `compaction_max_memory_limit` and (process memory limit × this percent / 100). If this value is < 0 or > 100 it is treated as 100. If `compaction_max_memory_limit` < 0 the process memory limit is used instead. The calculation also considers the BE process memory derived from `mem_limit`. Combined with `compaction_memory_limit_per_worker` (per-worker cap), this setting controls total compaction memory available and therefore affects compaction concurrency and OOM risk. * Introduced in: v3.2.0 ##### compaction\_memory\_limit\_per\_worker[​](#compaction_memory_limit_per_worker "Direct link to compaction_memory_limit_per_worker") * Default: 2147483648 * Type: Int * Unit: Bytes * Is mutable: No * Description: The maximum memory size allowed for each Compaction thread. * Introduced in: - ##### compaction\_chunk\_reset\_memory\_tracker\_threshold\_percent[​](#compaction_chunk_reset_memory_tracker_threshold_percent "Direct link to compaction_chunk_reset_memory_tracker_threshold_percent") * Default: -1 * Type: Int * Unit: Percent * Is mutable: Yes * Description: Controls when compaction releases retained internal chunk capacity. Currently, this parameter takes effect only for Primary Key table compaction in shared-nothing clusters. When the current compaction task memory tracker consumption exceeds `compaction_memory_limit_per_worker * compaction_chunk_reset_memory_tracker_threshold_percent / 100`, StarRocks releases retained chunk capacity while resetting internal chunks. A negative value disables this behavior. * Introduced in: - ##### compaction\_trace\_threshold[​](#compaction_trace_threshold "Direct link to compaction_trace_threshold") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time threshold for each compaction. If a compaction takes more time than the time threshold, StarRocks prints the corresponding trace. * Introduced in: - ##### create\_tablet\_worker\_count[​](#create_tablet_worker_count "Direct link to create_tablet_worker_count") * Default: 3 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Sets the maximum number of worker threads in the AgentServer thread pool that process TTaskType::CREATE (create-tablet) tasks submitted by FE. At BE startup this value is used as the thread-pool max (the pool is created with min threads = 1 and max queue size = unlimited), and changing it at runtime triggers `ExecEnv::agent_server()->get_thread_pool(TTaskType::CREATE)->update_max_threads(...)`. Increase this to raise concurrent tablet creation throughput (useful during bulk load or partition creation); decreasing it throttles concurrent create operations. Raising the value increases CPU, memory and I/O concurrency and may cause contention; the thread pool enforces at least one thread, so values less than 1 have no practical effect. * Introduced in: v3.2.0 ##### cumulative\_compaction\_check\_interval\_seconds[​](#cumulative_compaction_check_interval_seconds "Direct link to cumulative_compaction_check_interval_seconds") * Default: 1 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval of thread polling for a Cumulative Compaction. * Introduced in: - ##### cumulative\_compaction\_num\_threads\_per\_disk[​](#cumulative_compaction_num_threads_per_disk "Direct link to cumulative_compaction_num_threads_per_disk") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: The number of Cumulative Compaction threads per disk. * Introduced in: - ##### data\_page\_size[​](#data_page_size "Direct link to data_page_size") * Default: 65536 * Type: Int * Unit: Bytes * Is mutable: No * Description: Target uncompressed page size (in bytes) used when building column data and index pages. This value is copied into ColumnWriterOptions.data\_page\_size and IndexedColumnWriterOptions.index\_page\_size and is consulted by page builders (e.g., BinaryPlainPageBuilder::is\_page\_full and buffer reservation logic) to decide when to finish a page and how much memory to reserve. A value of 0 disables the page-size limit in builders. Changing this value affects page count, metadata overhead, memory reservation and I/O/compression trade-offs (smaller pages → more pages and metadata; larger pages → fewer pages, potentially better compression but larger memory spikes). * Introduced in: v3.2.4 ##### default\_num\_rows\_per\_column\_file\_block[​](#default_num_rows_per_column_file_block "Direct link to default_num_rows_per_column_file_block") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of rows that can be stored in each row block. * Introduced in: - ##### delete\_worker\_count\_high\_priority[​](#delete_worker_count_high_priority "Direct link to delete_worker_count_high_priority") * Default: 1 * Type: Int * Unit: Threads * Is mutable: No * Description: Number of worker threads in the DeleteTaskWorkerPool that are allocated as HIGH-priority delete threads. On startup AgentServer creates the delete pool with total threads = delete\_worker\_count\_normal\_priority + delete\_worker\_count\_high\_priority; the first delete\_worker\_count\_high\_priority threads are marked to exclusively try to pop TPriority::HIGH tasks (they poll for high-priority delete tasks and sleep/loop if none are available). Increasing this value increases concurrency for high-priority delete requests; decreasing it reduces dedicated capacity and may increase latency for high-priority deletes. * Introduced in: v3.2.0 ##### dictionary\_encoding\_ratio[​](#dictionary_encoding_ratio "Direct link to dictionary_encoding_ratio") * Default: 0.7 * Type: Double * Unit: - * Is mutable: No * Description: Fraction (0.0–1.0) used by StringColumnWriter during the encode-speculation phase to decide between dictionary (DICT\_ENCODING) and plain (PLAIN\_ENCODING) encoding for a chunk. The code computes max\_card = row\_count \* `dictionary_encoding_ratio` and scans the chunk’s distinct key count; if the distinct count exceeds max\_card the writer chooses PLAIN\_ENCODING. The check is performed only when the chunk size passes `dictionary_speculate_min_chunk_size` (and when row\_count > dictionary\_min\_rowcount). Setting the value higher favors dictionary encoding (tolerates more distinct keys); setting it lower causes earlier fallback to plain encoding. A value of 1.0 effectively forces dictionary encoding (distinct count can never exceed row\_count). * Introduced in: v3.2.0 ##### dictionary\_encoding\_ratio\_for\_non\_string\_column[​](#dictionary_encoding_ratio_for_non_string_column "Direct link to dictionary_encoding_ratio_for_non_string_column") * Default: 0 * Type: double * Unit: - * Is mutable: No * Description: Ratio threshold used to decide whether to use dictionary encoding for non-string columns (numeric, date/time, decimal types). When enabled (value > 0.0001) the writer computes max\_card = row\_count \* dictionary\_encoding\_ratio\_for\_non\_string\_column and, for samples with row\_count > `dictionary_min_rowcount`, chooses DICT\_ENCODING only if distinct\_count ≤ max\_card; otherwise it falls back to BIT\_SHUFFLE. A value of `0` (default) disables non-string dictionary encoding. This parameter is analogous to `dictionary_encoding_ratio` but applies to non-string columns. Use values in (0,1] — smaller values restrict dictionary encoding to lower-cardinality columns and reduce dictionary memory/IO overhead. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### dictionary\_page\_size[​](#dictionary_page_size "Direct link to dictionary_page_size") * Default: 1048576 * Type: Int * Unit: Bytes * Is mutable: No * Description: Size in bytes of dictionary pages used when building rowset segments. This value is read into `PageBuilderOptions::dict_page_size` in the BE rowset code and controls how many dictionary entries can be stored in a single dictionary page. Increasing this value can improve compression ratio for dictionary-encoded columns by allowing larger dictionaries, but larger pages consume more memory during write/encode and can increase I/O and latency when reading or materializing pages. Set conservatively for large-memory, write-heavy workloads and avoid excessively large values to prevent runtime performance degradation. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### disk\_stat\_monitor\_interval[​](#disk_stat_monitor_interval "Direct link to disk_stat_monitor_interval") * Default: 5 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to monitor health status of disks. * Introduced in: - ##### download\_low\_speed\_limit\_kbps[​](#download_low_speed_limit_kbps "Direct link to download_low_speed_limit_kbps") * Default: 50 * Type: Int * Unit: KB/Second * Is mutable: Yes * Description: The download speed lower limit of each HTTP request. An HTTP request aborts when it constantly runs with a lower speed than this value within the time span specified in the configuration item `download_low_speed_time`. * Introduced in: - ##### download\_low\_speed\_time[​](#download_low_speed_time "Direct link to download_low_speed_time") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum time that an HTTP request can run with a download speed lower than the limit. An HTTP request aborts when it constantly runs with a lower speed than the value of `download_low_speed_limit_kbps` within the time span specified in this configuration item. * Introduced in: - ##### download\_worker\_count[​](#download_worker_count "Direct link to download_worker_count") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads for the download tasks of restore jobs on a BE node. `0` indicates setting the value to the number of CPU cores on the machine where the BE resides. * Introduced in: - ##### drop\_tablet\_worker\_count[​](#drop_tablet_worker_count "Direct link to drop_tablet_worker_count") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of threads used to drop a tablet. `0` indicates half of the CPU cores in the node. * Introduced in: - ##### enable\_check\_string\_lengths[​](#enable_check_string_lengths "Direct link to enable_check_string_lengths") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to check the data length during loading to solve compaction failures caused by out-of-bound VARCHAR data. * Introduced in: - ##### enable\_event\_based\_compaction\_framework[​](#enable_event_based_compaction_framework "Direct link to enable_event_based_compaction_framework") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to enable the Event-based Compaction Framework. `true` indicates Event-based Compaction Framework is enabled, and `false` indicates it is disabled. Enabling Event-based Compaction Framework can greatly reduce the overhead of compaction in scenarios where there are many tablets or a single tablet has a large amount of data. * Introduced in: - ##### enable\_lazy\_delta\_column\_compaction[​](#enable_lazy_delta_column_compaction "Direct link to enable_lazy_delta_column_compaction") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, compaction will prefer a "lazy" strategy for delta columns produced by partial column updates: StarRocks will avoid eagerly merging delta-column files back into their main segment files to save compaction I/O. In practice the compaction selection code checks for partial column-update rowsets and multiple candidates; if found and this flag is true, the engine will either stop adding further inputs to the compaction or only merge empty rowsets (level -1), leaving delta columns separate. This reduces immediate I/O and CPU during compaction at the cost of delayed consolidation (potentially more segments and temporary storage overhead). Correctness and query semantics are unchanged. * Introduced in: v3.2.3 ##### enable\_new\_load\_on\_memory\_limit\_exceeded[​](#enable_new_load_on_memory_limit_exceeded "Direct link to enable_new_load_on_memory_limit_exceeded") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow new loading processes when the hard memory resource limit is reached. `true` indicates new loading processes will be allowed, and `false` indicates they will be rejected. * Introduced in: v3.3.2 ##### enable\_pk\_index\_parallel\_compaction[​](#enable_pk_index_parallel_compaction "Direct link to enable_pk_index_parallel_compaction") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable parallel Compaction for Primary Key index in a shared-data cluster. * Introduced in: - ##### enable\_pk\_index\_parallel\_execution[​](#enable_pk_index_parallel_execution "Direct link to enable_pk_index_parallel_execution") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable parallel execution for Primary Key index operations in a shared-data cluster. When enabled, the system uses a thread pool to process segments concurrently during publish operations, significantly improving performance for large tablets. * Introduced in: - ##### enable\_pk\_size\_tiered\_compaction\_strategy[​](#enable_pk_size_tiered_compaction_strategy "Direct link to enable_pk_size_tiered_compaction_strategy") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to enable the Size-tiered Compaction policy for Primary Key tables. `true` indicates the Size-tiered Compaction strategy is enabled, and `false` indicates it is disabled. * Introduced in: This item takes effect for shared-data clusters from v3.2.4 and v3.1.10 onwards, and for shared-nothing clusters from v3.2.5 and v3.1.10 onwards. ##### enable\_rowset\_verify[​](#enable_rowset_verify "Direct link to enable_rowset_verify") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to verify the correctness of generated rowsets. When enabled, the correctness of the generated rowsets will be checked after Compaction and Schema Change. * Introduced in: - ##### enable\_size\_tiered\_compaction\_strategy[​](#enable_size_tiered_compaction_strategy "Direct link to enable_size_tiered_compaction_strategy") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to enable the Size-tiered Compaction policy (excluding Primary Key tables). `true` indicates the Size-tiered Compaction strategy is enabled, and `false` indicates it is disabled. * Introduced in: - ##### enable\_strict\_delvec\_crc\_check[​](#enable_strict_delvec_crc_check "Direct link to enable_strict_delvec_crc_check") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enable\_strict\_delvec\_crc\_check is set to true, we will perform a strict CRC32 check on the delete vector, and if a mismatch is detected, a failure will be returned. * Introduced in: - ##### enable\_transparent\_data\_encryption[​](#enable_transparent_data_encryption "Direct link to enable_transparent_data_encryption") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: When enabled, StarRocks will create encrypted on‑disk artifacts for newly written storage objects (segment files, delete/update files, rowset segments, lake SSTs, persistent index files, etc.). Writers (RowsetWriter/SegmentWriter, lake UpdateManager/LakePersistentIndex and related code paths) will request encryption info from the KeyCache, attach encryption\_info to writable files and persist encryption\_meta into rowset / segment / sstable metadata (segment\_encryption\_metas, delete/update encryption metadata). The Frontend and Backend/CN encryption flags must match — a mismatch causes the BE to abort on heartbeat (LOG(FATAL)). This flag is not runtime‑mutable; enable it before deployment and ensure key management (KEK) and KeyCache are properly configured and synchronized across the cluster. * Introduced in: v3.3.1, 3.4.0, 3.5.0, 4.0.0 ##### enable\_zero\_copy\_from\_page\_cache[​](#enable_zero_copy_from_page_cache "Direct link to enable_zero_copy_from_page_cache") * Default: true * Type: boolean * Unit: - * Is mutable: Yes * Description: When enabled, FixedLengthColumnBase may avoid copying bytes when appending data that originates from a page-cache-backed buffer. In append\_numbers the code will acquire the incoming ContainerResource and set the column's internal resource pointer (zero-copy) if all conditions are met: the config is true, the incoming resource is owned, the resource memory is aligned for the column element type, the column is empty, and the resource length is a multiple of the element size. Enabling this reduces CPU and memory-copy overhead and can improve ingestion/scan throughput. Drawbacks: it couples the column lifetime to the acquired buffer and relies on correct ownership/alignment; disable to force safe copying. * Introduced in: - ##### file\_descriptor\_cache\_clean\_interval[​](#file_descriptor_cache_clean_interval "Direct link to file_descriptor_cache_clean_interval") * Default: 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to clean file descriptors that have not been used for a certain period of time. * Introduced in: - ##### ignore\_broken\_disk[​](#ignore_broken_disk "Direct link to ignore_broken_disk") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Controls startup behavior when configured storage paths fail read/write checks or fail to parse. When `false` (default), BE treats any broken entry in `storage_root_path` or `spill_local_storage_dir` as fatal and will abort startup. When `true`, StarRocks will skip (log a warning and remove) any storage path that fails `check_datapath_rw` or fails parsing so the BE can continue starting with the remaining healthy paths. Note: if all configured paths are removed, BE will still exit. Enabling this can mask misconfigured or failed disks and cause data on ignored paths to be unavailable; monitor logs and disk health accordingly. * Introduced in: v3.2.0 ##### inc\_rowset\_expired\_sec[​](#inc_rowset_expired_sec "Direct link to inc_rowset_expired_sec") * Default: 1800 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The expiration time of the incoming data. This configuration item is used in incremental clone. * Introduced in: - ##### load\_process\_max\_memory\_hard\_limit\_ratio[​](#load_process_max_memory_hard_limit_ratio "Direct link to load_process_max_memory_hard_limit_ratio") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: The hard limit (ratio) of memory resources that can be taken up by all load processes on a BE node. When `enable_new_load_on_memory_limit_exceeded` is set to `false`, and the memory consumption of all loading processes exceeds `load_process_max_memory_limit_percent * load_process_max_memory_hard_limit_ratio`, new loading processes will be rejected. * Introduced in: v3.3.2 ##### load\_process\_max\_memory\_limit\_percent[​](#load_process_max_memory_limit_percent "Direct link to load_process_max_memory_limit_percent") * Default: 30 * Type: Int * Unit: - * Is mutable: No * Description: The soft limit (in percentage) of memory resources that can be taken up by all load processes on a BE node. * Introduced in: - ##### lz4\_acceleration[​](#lz4_acceleration "Direct link to lz4_acceleration") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls the LZ4 "acceleration" parameter used by the built-in LZ4 compressor (passed to LZ4\_compress\_fast\_continue). Higher values prioritize compression speed at the cost of compression ratio; lower values (1) produce better compression but are slower. Valid range: MIN=1, MAX=65537. This setting affects all LZ4-based codecs in BlockCompression (e.g., LZ4 and Hadoop-LZ4) and only changes how compression is performed — it does not change the LZ4 format or decompression compatibility. Tune upward (e.g., 4, 8, ...) for CPU-bound or low-latency workloads where larger output is acceptable; keep at 1 for storage- or IO-sensitive workloads. Test with representative data before changing, since throughput vs. size trade-offs are highly data-dependent. * Introduced in: v3.4.1, 3.5.0, 4.0.0 ##### lz4\_expected\_compression\_ratio[​](#lz4_expected_compression_ratio "Direct link to lz4_expected_compression_ratio") * Default: 2.1 * Type: double * Unit: Dimensionless (compression ratio) * Is mutable: Yes * Description: Threshold used by the serialization compression strategy to judge whether observed LZ4 compression is "good". In compress\_strategy.cpp this value divides the observed compress\_ratio when computing a reward metric together with lz4\_expected\_compression\_speed\_mbps; if the combined reward `>` 1.0 the strategy records positive feedback. Increasing this value raises the expected compression ratio (making the condition harder to satisfy), while lowering it makes it easier for observed compression to be considered satisfactory. Tune to match typical data compressibility. Valid range: MIN=1, MAX=65537. * Introduced in: v3.4.1, 3.5.0, 4.0.0 ##### lz4\_expected\_compression\_speed\_mbps[​](#lz4_expected_compression_speed_mbps "Direct link to lz4_expected_compression_speed_mbps") * Default: 600 * Type: double * Unit: MB/s * Is mutable: Yes * Description: Expected LZ4 compression throughput in megabytes per second used by the adaptive compression policy (CompressStrategy). The feedback routine computes a reward\_ratio = (observed\_compression\_ratio / lz4\_expected\_compression\_ratio) \* (observed\_speed / lz4\_expected\_compression\_speed\_mbps). A reward\_ratio `>` 1.0 increments the positive counter (alpha), otherwise the negative counter (beta); this influences whether future data will be compressed. Tune this value to reflect typical LZ4 throughput on your hardware — raising it makes the policy harder to classify a run as "good" (requires higher observed speed), lowering it makes classification easier. Must be a positive finite number. * Introduced in: v3.4.1, 3.5.0, 4.0.0 ##### make\_snapshot\_worker\_count[​](#make_snapshot_worker_count "Direct link to make_snapshot_worker_count") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads for the make snapshot tasks on a BE node. * Introduced in: - ##### manual\_compaction\_threads[​](#manual_compaction_threads "Direct link to manual_compaction_threads") * Default: 4 * Type: Int * Unit: - * Is mutable: No * Description: Number of threads for Manual Compaction. * Introduced in: - ##### max\_base\_compaction\_num\_singleton\_deltas[​](#max_base_compaction_num_singleton_deltas "Direct link to max_base_compaction_num_singleton_deltas") * Default: 100 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of segments that can be compacted in each Base Compaction. * Introduced in: - ##### max\_compaction\_candidate\_num[​](#max_compaction_candidate_num "Direct link to max_compaction_candidate_num") * Default: 40960 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of candidate tablets for compaction. If the value is too large, it will cause high memory usage and high CPU load. * Introduced in: - ##### max\_compaction\_concurrency[​](#max_compaction_concurrency "Direct link to max_compaction_concurrency") * Default: -1 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum concurrency of compactions (including both Base Compaction and Cumulative Compaction). The value `-1` indicates that no limit is imposed on the concurrency. `0` indicates disabling compaction. This parameter is mutable when the Event-based Compaction Framework is enabled. * Introduced in: - ##### max\_cumulative\_compaction\_num\_singleton\_deltas[​](#max_cumulative_compaction_num_singleton_deltas "Direct link to max_cumulative_compaction_num_singleton_deltas") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of segments that can be merged in a single Cumulative Compaction. You can reduce this value if OOM occurs during compaction. * Introduced in: - ##### max\_download\_speed\_kbps[​](#max_download_speed_kbps "Direct link to max_download_speed_kbps") * Default: 50000 * Type: Int * Unit: KB/Second * Is mutable: Yes * Description: The maximum download speed of each HTTP request. This value affects the performance of data replica synchronization across BE nodes. * Introduced in: - ##### max\_garbage\_sweep\_interval[​](#max_garbage_sweep_interval "Direct link to max_garbage_sweep_interval") * Default: 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum time interval for garbage collection on storage volumes. This configuration is changed to dynamic from v3.0 onwards. * Introduced in: - ##### max\_percentage\_of\_error\_disk[​](#max_percentage_of_error_disk "Direct link to max_percentage_of_error_disk") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum percentage of error that is tolerable in a storage volume before the corresponding BE node quits. * Introduced in: - ##### max\_queueing\_memtable\_per\_tablet[​](#max_queueing_memtable_per_tablet "Direct link to max_queueing_memtable_per_tablet") * Default: 2 * Type: Long * Unit: Count * Is mutable: Yes * Description: Controls per-tablet backpressure for write paths: when a tablet's number of queueing (not-yet-flushing) memtables reaches or exceeds `max_queueing_memtable_per_tablet`, writers in `LocalTabletsChannel` and `LakeTabletsChannel` will block (sleep/retry) before submitting more write work. This reduces simultaneous memtable flush concurrency and peak memory use at the cost of increased latency or RPC timeouts for heavy load. Set higher to allow more concurrent memtables (more memory and I/O burst); set lower to limit memory pressure and increase write throttling. * Introduced in: v3.2.0 ##### max\_row\_source\_mask\_memory\_bytes[​](#max_row_source_mask_memory_bytes "Direct link to max_row_source_mask_memory_bytes") * Default: 209715200 * Type: Int * Unit: Bytes * Is mutable: No * Description: The maximum memory size of the row source mask buffer. When the buffer is larger than this value, data will be persisted to a temporary file on the disk. This value should be set lower than the value of `compaction_memory_limit_per_worker`. * Introduced in: - ##### max\_tablet\_write\_chunk\_bytes[​](#max_tablet_write_chunk_bytes "Direct link to max_tablet_write_chunk_bytes") * Default: 536870912 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: Maximum allowed memory (in bytes) for the current in-memory tablet write chunk before it is treated as full and enqueued for sending. Increase this value to reduce the frequency of RPCs when loading wide tables (many columns), which can improve throughput at the cost of higher memory usage and larger RPC payloads. Tune to balance fewer RPCs against memory and serialization/BRPC limits. * Introduced in: v3.2.12 ##### max\_update\_compaction\_num\_singleton\_deltas[​](#max_update_compaction_num_singleton_deltas "Direct link to max_update_compaction_num_singleton_deltas") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of rowsets that can be merged in a single Compaction for Primary Key tables. * Introduced in: - ##### memory\_limitation\_per\_thread\_for\_schema\_change[​](#memory_limitation_per_thread_for_schema_change "Direct link to memory_limitation_per_thread_for_schema_change") * Default: 2 * Type: Int * Unit: GB * Is mutable: Yes * Description: The maximum memory size allowed for each schema change task. * Introduced in: - ##### memory\_ratio\_for\_sorting\_schema\_change[​](#memory_ratio_for_sorting_schema_change "Direct link to memory_ratio_for_sorting_schema_change") * Default: 0.8 * Type: Double * Unit: - (unitless ratio) * Is mutable: Yes * Description: Fraction of the per-thread schema-change memory limit used as the memtable maximum buffer size during sorting schema-change operations. The ratio is multiplied by memory\_limitation\_per\_thread\_for\_schema\_change (configured in GB and converted to bytes) to compute max\_buffer\_size, and that result is capped at 4GB. Used by SchemaChangeWithSorting and SortedSchemaChange when creating MemTable/DeltaWriter. Increasing this ratio allows larger in-memory buffers (fewer flushes/merges) but raises risk of memory pressure; reducing it causes more frequent flushes and higher I/O/merge overhead. * Introduced in: v3.2.0 ##### min\_base\_compaction\_num\_singleton\_deltas[​](#min_base_compaction_num_singleton_deltas "Direct link to min_base_compaction_num_singleton_deltas") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The minimum number of segments that trigger a Base Compaction. * Introduced in: - ##### min\_compaction\_failure\_interval\_sec[​](#min_compaction_failure_interval_sec "Direct link to min_compaction_failure_interval_sec") * Default: 120 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The minimum time interval at which a tablet compaction can be scheduled since the previous compaction failure. * Introduced in: - ##### min\_cumulative\_compaction\_failure\_interval\_sec[​](#min_cumulative_compaction_failure_interval_sec "Direct link to min_cumulative_compaction_failure_interval_sec") * Default: 30 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The minimum time interval at which Cumulative Compaction retries upon failures. * Introduced in: - ##### min\_cumulative\_compaction\_num\_singleton\_deltas[​](#min_cumulative_compaction_num_singleton_deltas "Direct link to min_cumulative_compaction_num_singleton_deltas") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The minimum number of segments to trigger Cumulative Compaction. * Introduced in: - ##### min\_garbage\_sweep\_interval[​](#min_garbage_sweep_interval "Direct link to min_garbage_sweep_interval") * Default: 180 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The minimum time interval for garbage collection on storage volumes. This configuration is changed to dynamic from v3.0 onwards. * Introduced in: - ##### parallel\_clone\_task\_per\_path[​](#parallel_clone_task_per_path "Direct link to parallel_clone_task_per_path") * Default: 8 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Number of parallel clone worker threads allocated per storage path on a BE. At BE startup the clone thread-pool max threads is computed as max(number\_of\_store\_paths \* parallel\_clone\_task\_per\_path, MIN\_CLONE\_TASK\_THREADS\_IN\_POOL). For example, with 4 storage paths and default=8 the clone pool max = 32. This setting directly controls concurrency of CLONE tasks (tablet replica copies) handled by the BE: increasing it raises parallel clone throughput but also increases CPU, disk and network contention; decreasing it limits simultaneous clone tasks and can throttle FE-scheduled clone operations. The value is applied to the dynamic clone thread pool and can be changed at runtime via the update-config path (causes agent\_server to update the clone pool max threads). * Introduced in: v3.2.0 ##### partial\_update\_memory\_limit\_per\_worker[​](#partial_update_memory_limit_per_worker "Direct link to partial_update_memory_limit_per_worker") * Default: 2147483648 * Type: long * Unit: Bytes * Is mutable: Yes * Description: Maximum memory (in bytes) a single worker may use for assembling a source chunk when performing partial column updates (used in compaction / rowset update processing). The reader estimates per-row update memory (total\_update\_row\_size / num\_rows\_upt) and multiplies it by the number of rows read; when that product exceeds this limit the current chunk is flushed and processed to avoid additional memory growth. Set this to match the available memory per update worker—too low increases I/O/processing overhead (many small chunks); too high risks memory pressure or OOM. If the per-row estimate is zero (legacy rowsets), this config does not impose a byte-based limit (only the INT32\_MAX row count limit applies). * Introduced in: v3.2.10 ##### path\_gc\_check[​](#path_gc_check "Direct link to path_gc_check") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: When enabled, StorageEngine starts per-data-dir background threads that perform periodic path scanning and garbage collection. On startup `start_bg_threads()` spawns `_path_scan_thread_callback` (calls `DataDir::perform_path_scan` and `perform_tmp_path_scan`) and `_path_gc_thread_callback` (calls `DataDir::perform_path_gc_by_tablet`, `DataDir::perform_path_gc_by_rowsetid`, `DataDir::perform_delta_column_files_gc`, and `DataDir::perform_crm_gc`). The scan and GC intervals are controlled by `path_scan_interval_second` and `path_gc_check_interval_second`; CRM file cleanup uses `unused_crm_file_threshold_second`. Disable this to prevent automatic path-level cleanup (you must then manage orphaned/temp files manually). Changing this flag requires restarting the process. * Introduced in: v3.2.0 ##### path\_gc\_check\_interval\_second[​](#path_gc_check_interval_second "Direct link to path_gc_check_interval_second") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: No * Description: Interval in seconds between runs of the storage engine's path garbage-collection background thread. Each wake triggers DataDir to perform path GC by tablet, by rowset id, delta column file GC and CRM GC (the CRM GC call uses `unused_crm_file_threshold_second`). If set to a non-positive value the code forces the interval to 1800 seconds (half hour) and emits a warning. Tune this to control how frequently on-disk temporary or downloaded files are scanned and removed. * Introduced in: v3.2.0 ##### pending\_data\_expire\_time\_sec[​](#pending_data_expire_time_sec "Direct link to pending_data_expire_time_sec") * Default: 1800 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The expiration time of the pending data in the storage engine. * Introduced in: - ##### pindex\_major\_compaction\_limit\_per\_disk[​](#pindex_major_compaction_limit_per_disk "Direct link to pindex_major_compaction_limit_per_disk") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. * Introduced in: v3.0.9 ##### pk\_index\_compaction\_score\_ratio[​](#pk_index_compaction_score_ratio "Direct link to pk_index_compaction_score_ratio") * Default: 1.5 * Type: Double * Unit: - * Is mutable: Yes * Description: Compaction score ratio for Primary Key index in a shared-data cluster. For example, if there are N filesets, the Compaction score will be `N * pk_index_compaction_score_ratio`. * Introduced in: - ##### pk\_index\_early\_sst\_compaction\_threshold[​](#pk_index_early_sst_compaction_threshold "Direct link to pk_index_early_sst_compaction_threshold") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: early sst compaction threshold for primary key index in a shared-data cluster. * Introduced in: - ##### pk\_index\_map\_shard\_size[​](#pk_index_map_shard_size "Direct link to pk_index_map_shard_size") * Default: 4096 * Type: Int * Unit: - * Is mutable: No * Description: Number of shards used by the Primary Key index shard map in the lake UpdateManager. UpdateManager allocates a vector of `PkIndexShard` of this size and maps a tablet ID to a shard via a bitmask. Increasing this value reduces lock contention among tablets that would otherwise share the same shard, at the cost of more mutex objects and slightly higher memory usage. The value must be a power of two because the code relies on bitmask indexing. For sizing guidance see `tablet_map_shard_size` heuristic: `total_num_of_tablets_in_BE / 512`. * Introduced in: v3.2.0 ##### pk\_index\_memtable\_flush\_threadpool\_max\_threads[​](#pk_index_memtable_flush_threadpool_max_threads "Direct link to pk_index_memtable_flush_threadpool_max_threads") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads in the thread pool for Primary Key index MemTable flush in a shared-data cluster. `0` means automatically set to half of the number of CPU cores. * Introduced in: - ##### pk\_index\_memtable\_flush\_threadpool\_size[​](#pk_index_memtable_flush_threadpool_size "Direct link to pk_index_memtable_flush_threadpool_size") * Default: 1048576 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls the maximum queue size (number of pending tasks) for the Primary Key index memtable flush thread pool used in shared-data (cloud-native / lake) mode. The thread pool is created as "cloud\_native\_pk\_index\_flush" in ExecEnv; its max thread count is governed by `pk_index_memtable_flush_threadpool_max_threads`. Increasing this value permits more memtable flush tasks to be buffered before execution, which can reduce immediate backpressure but increases memory consumed by queued task objects. Decreasing it limits buffered tasks and can cause earlier backpressure or task rejections depending on thread-pool behavior. Tune according to available memory and expected concurrent flush workload. * Introduced in: - ##### pk\_index\_memtable\_max\_count[​](#pk_index_memtable_max_count "Direct link to pk_index_memtable_max_count") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of MemTables for Primary Key index in a shared-data cluster. * Introduced in: - ##### pk\_index\_memtable\_max\_wait\_flush\_timeout\_ms[​](#pk_index_memtable_max_wait_flush_timeout_ms "Direct link to pk_index_memtable_max_wait_flush_timeout_ms") * Default: 30000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The maximum timeout for waiting for Primary Key index MemTable flush completion in a shared-data cluster. When synchronously flushing all MemTables (for example, before an ingest SST operation), the system waits up to this timeout. The default is 30 seconds. * Introduced in: - ##### pk\_index\_parallel\_compaction\_task\_split\_threshold\_bytes[​](#pk_index_parallel_compaction_task_split_threshold_bytes "Direct link to pk_index_parallel_compaction_task_split_threshold_bytes") * Default: 33554432 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The splitting threshold for Primary Key index Compaction tasks. When the total size of the files involved in a task is smaller than this threshold, the task will not be split. * Introduced in: - ##### pk\_index\_parallel\_compaction\_threadpool\_max\_threads[​](#pk_index_parallel_compaction_threadpool_max_threads "Direct link to pk_index_parallel_compaction_threadpool_max_threads") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads in the thread pool for cloud native Primary Key index parallel Compaction in a shared-data cluster. `0` means automatically set to half of the number of CPU cores. * Introduced in: - ##### pk\_index\_parallel\_compaction\_threadpool\_size[​](#pk_index_parallel_compaction_threadpool_size "Direct link to pk_index_parallel_compaction_threadpool_size") * Default: 1048576 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum queue size (number of pending tasks) for the thread pool used by cloud-native Primary Key index parallel compaction in shared-data mode. This setting controls how many Compaction tasks can be enqueued before the thread pool rejects new submissions. The effective parallelism is bounded by `pk_index_parallel_compaction_threadpool_max_threads`; increase this value to avoid task rejections when you expect many concurrent Compaction tasks, but be aware larger queues can increase memory and latency for queued work. * Introduced in: - ##### pk\_index\_parallel\_execution\_min\_rows[​](#pk_index_parallel_execution_min_rows "Direct link to pk_index_parallel_execution_min_rows") * Default: 16384 * Type: Int * Unit: - * Is mutable: Yes * Description: The minimum rows threshold to enable parallel execution for Primary Key index operations in a shared-data cluster. * Introduced in: - ##### pk\_index\_parallel\_execution\_threadpool\_max\_threads[​](#pk_index_parallel_execution_threadpool_max_threads "Direct link to pk_index_parallel_execution_threadpool_max_threads") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads in the thread pool for Primary Key index parallel execution in a shared-data cluster. `0` means automatically set to the number of CPU cores, with at least one thread. Setting this parameter to `0` at runtime restores this automatic sizing. * Introduced in: - ##### pk\_index\_parallel\_rebuild\_mem\_ratio[​](#pk_index_parallel_rebuild_mem_ratio "Direct link to pk_index_parallel_rebuild_mem_ratio") * Default: 50 * Type: Int * Unit: percent (0-100) * Is mutable: Yes * Description: In a shared-data cluster, the memory-pressure gate for the parallel prefetch paths used while rebuilding the Primary Key index. When the update mem tracker is already past this percent of its limit, the rebuild falls back to a single-pass loop that holds only one decoded column at a time, trading the cold-start latency win for bounded peak memory. It gates parallel reads of delete files, segment files, and other files during the rebuild. Set to a higher value to allow the optimization under more memory pressure; set to `100` to disable the memory gate (always run the parallel path when `enable_pk_index_parallel_execution=true`). * Introduced in: - ##### lake\_partial\_update\_thread\_pool\_max\_threads[​](#lake_partial_update_thread_pool_max_threads "Direct link to lake_partial_update_thread_pool_max_threads") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads in the thread pool for lake partial update segment-level parallelism in a shared-data cluster. This thread pool is used by both row-mode and column-mode partial updates to parallelize I/O-heavy segment operations (load\_segment + rewrite\_segment for row-mode, DCG generation for column-mode). `0` means automatically set to half of the number of CPU cores. Runtime on/off is controlled by `enable_pk_index_parallel_execution`. * Introduced in: v4.1 ##### lake\_partial\_update\_thread\_pool\_queue\_size[​](#lake_partial_update_thread_pool_queue_size "Direct link to lake_partial_update_thread_pool_queue_size") * Default: 2048 * Type: Int * Unit: - * Is mutable: Yes * Description: The task queue size for the lake partial update thread pool. * Introduced in: v4.1 ##### pk\_index\_size\_tiered\_level\_multiplier[​](#pk_index_size_tiered_level_multiplier "Direct link to pk_index_size_tiered_level_multiplier") * Default: 10 * Type: Int * Unit: - * Is mutable: Yes * Description: The level multiplier parameter for Primary Key index size-tiered Compaction strategy. * Introduced in: - ##### pk\_index\_size\_tiered\_max\_level[​](#pk_index_size_tiered_max_level "Direct link to pk_index_size_tiered_max_level") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum level for Primary Key index size-tiered Compaction strategy. * Introduced in: - ##### pk\_index\_size\_tiered\_min\_level\_size[​](#pk_index_size_tiered_min_level_size "Direct link to pk_index_size_tiered_min_level_size") * Default: 131072 * Type: Int * Unit: - * Is mutable: Yes * Description: The minimum level for Primary Key index size-tiered Compaction strategy. * Introduced in: - ##### pk\_index\_sstable\_sample\_interval\_bytes[​](#pk_index_sstable_sample_interval_bytes "Direct link to pk_index_sstable_sample_interval_bytes") * Default: 16777216 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The sampling interval size for SSTable files in a shared-data cluster. When the size of an SSTable file exceeds this threshold, the system samples keys from the SSTable at this interval to optimize the boundary partitioning of Compaction tasks. For SSTables smaller than this threshold, only the start key is used as the boundary key. The default is 16 MB. * Introduced in: - ##### pk\_index\_target\_file\_size[​](#pk_index_target_file_size "Direct link to pk_index_target_file_size") * Default: 67108864 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The target file size for Primary Key index in a shared-data cluster. * Introduced in: - ##### pk\_index\_eager\_build\_threshold\_bytes[​](#pk_index_eager_build_threshold_bytes "Direct link to pk_index_eager_build_threshold_bytes") * Default: 104857600 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The minimum size of data generated during import or compaction for the system to eagerly build PK index files. Default is 100MB. * Introduced in: - ##### primary\_key\_limit\_size[​](#primary_key_limit_size "Direct link to primary_key_limit_size") * Default: 128 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The maximum size of a key column in Primary Key tables. * Introduced in: v2.5 ##### release\_snapshot\_worker\_count[​](#release_snapshot_worker_count "Direct link to release_snapshot_worker_count") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads for the release snapshot tasks on a BE node. * Introduced in: - ##### repair\_compaction\_interval\_seconds[​](#repair_compaction_interval_seconds "Direct link to repair_compaction_interval_seconds") * Default: 600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval to poll Repair Compaction threads. * Introduced in: - ##### replication\_max\_speed\_limit\_kbps[​](#replication_max_speed_limit_kbps "Direct link to replication_max_speed_limit_kbps") * Default: 50000 * Type: Int * Unit: KB/s * Is mutable: Yes * Description: The maximum speed of each replication thread. * Introduced in: v3.3.5 ##### replication\_min\_speed\_limit\_kbps[​](#replication_min_speed_limit_kbps "Direct link to replication_min_speed_limit_kbps") * Default: 50 * Type: Int * Unit: KB/s * Is mutable: Yes * Description: The minimum speed of each replication thread. * Introduced in: v3.3.5 ##### replication\_min\_speed\_time\_seconds[​](#replication_min_speed_time_seconds "Direct link to replication_min_speed_time_seconds") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time duration allowed for a replication thread to be under the minimum speed. Replication will fail if the time when the actual speed is lower than `replication_min_speed_limit_kbps` exceeds this value. * Introduced in: v3.3.5 ##### replication\_threads[​](#replication_threads "Direct link to replication_threads") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads used for replication. `0` indicates setting the thread number to four times the BE CPU core count. * Introduced in: v3.3.5 ##### size\_tiered\_level\_multiple[​](#size_tiered_level_multiple "Direct link to size_tiered_level_multiple") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The multiple of data size between two contiguous levels in the Size-tiered Compaction policy. * Introduced in: - ##### size\_tiered\_level\_multiple\_dupkey[​](#size_tiered_level_multiple_dupkey "Direct link to size_tiered_level_multiple_dupkey") * Default: 10 * Type: Int * Unit: - * Is mutable: Yes * Description: In the Size-tiered Compaction policy, the multiple of the data amount difference between two adjacent levels for Duplicate Key tables. * Introduced in: - ##### size\_tiered\_level\_num[​](#size_tiered_level_num "Direct link to size_tiered_level_num") * Default: 7 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of levels for the Size-tiered Compaction policy. At most one rowset is reserved for each level. Therefore, under a stable condition, there are, at most, as many rowsets as the level number specified in this configuration item. * Introduced in: - ##### size\_tiered\_max\_compaction\_level[​](#size_tiered_max_compaction_level "Direct link to size_tiered_max_compaction_level") * Default: 3 * Type: Int * Unit: Levels * Is mutable: Yes * Description: Limits how many size-tiered levels may be merged into a single primary-key real-time compaction task. During the PK size-tiered compaction selection, StarRocks builds ordered "levels" of rowsets by size and will add successive levels into the chosen compaction input until this limit is reached (the code uses compaction\_level `<=` size\_tiered\_max\_compaction\_level). The value is inclusive and counts the number of distinct size tiers merged (the top level is counted as 1). Effective only when the PK size-tiered compaction strategy is enabled; raising it lets a compaction task include more levels (larger, more I/O- and CPU-intensive merges, potential higher write amplification), while lowering it restricts merges and reduces task size and resource usage. * Introduced in: v4.0.0 ##### size\_tiered\_min\_level\_size[​](#size_tiered_min_level_size "Direct link to size_tiered_min_level_size") * Default: 131072 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The data size of the minimum level in the Size-tiered Compaction policy. Rowsets smaller than this value immediately trigger the data compaction. * Introduced in: - ##### skip\_pindex\_l1\_l2\_check\_in\_insert[​](#skip_pindex_l1_l2_check_in_insert "Direct link to skip_pindex_l1_l2_check_in_insert") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to skip duplicate-key checks in immutable L1/L2 files when inserting into the local persistent primary key index in shared-nothing clusters. The default `true` checks only the L0 memtable, avoiding disk reads for these checks and aligning with the cloud-native primary key index in shared-data clusters. Set this to `false` to restore immutable-index checks. This controls an internal index consistency check, not the update semantics of SQL INSERT statements on Primary Key tables. * Introduced in: - ##### small\_dictionary\_page\_size[​](#small_dictionary_page_size "Direct link to small_dictionary_page_size") * Default: 4096 * Type: Int * Unit: Bytes * Is mutable: No * Description: Threshold (in bytes) used by BinaryPlainPageDecoder to decide whether to eagerly parse a dictionary (binary/plain) page. If a page's encoded size is < `small_dictionary_page_size`, the decoder pre-parses all string entries into an in-memory vector (`_parsed_datas`) to accelerate random access and batch reads. Raising this value causes more pages to be pre-parsed (which can reduce per-access decoding overhead and may increase effective compression for larger dictionaries) but increases memory usage and CPU spent parsing; excessively large values can degrade overall performance. Tune only after measuring memory and access-latency trade-offs. * Introduced in: v3.4.1, v3.5.0 ##### snapshot\_expire\_time\_sec[​](#snapshot_expire_time_sec "Direct link to snapshot_expire_time_sec") * Default: 172800 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The expiration time of snapshot files. * Introduced in: - ##### sort\_key\_max\_samples\_per\_tablet[​](#sort_key_max_samples_per_tablet "Direct link to sort_key_max_samples_per_tablet") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of sort key samples taken from one tablet when computing tablet split boundaries. Samples come from a segment's short key index when that index already encodes the whole sort key, and otherwise from a bounded number of the segment's data pages, so this value also bounds that read amplification. The default matches the FE configuration `tablet_reshard_max_split_count`, because splitting a tablet into K ranges requires K-1 interior boundary points. Setting this to `0` disables sampling, and split boundaries are then derived only from each segment's minimum and maximum sort key, which is coarser but still correct. * Introduced in: - ##### stale\_memtable\_flush\_time\_sec[​](#stale_memtable_flush_time_sec "Direct link to stale_memtable_flush_time_sec") * Default: 0 * Type: long * Unit: Seconds * Is mutable: Yes * Description: When a sender job's memory usage is high, memtables that have not been updated for longer than `stale_memtable_flush_time_sec` seconds will be flushed to reduce memory pressure. This behavior is only considered when memory limits are approaching (`limit_exceeded_by_ratio(70)` or higher). In `LocalTabletsChannel`, an additional path at very high memory usage (`limit_exceeded_by_ratio(95)`) may flush memtables whose size exceeds `write_buffer_size / 4`. A value of `0` disables this age-based stale-memtable flushing (immutable-partition memtables still flush immediately when idle or on high memory). * Introduced in: v3.2.0 ##### storage\_flood\_stage\_left\_capacity\_bytes[​](#storage_flood_stage_left_capacity_bytes "Direct link to storage_flood_stage_left_capacity_bytes") * Default: 107374182400 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: Hard limit of the remaining storage space in all BE directories. If the remaining storage space of the BE storage directory is less than this value and the storage usage (in percentage) exceeds `storage_flood_stage_usage_percent`, Load and Restore jobs are rejected. You need to set this item together with the FE configuration item `storage_usage_hard_limit_reserve_bytes` to allow the configurations to take effect. * Introduced in: - ##### storage\_flood\_stage\_usage\_percent[​](#storage_flood_stage_usage_percent "Direct link to storage_flood_stage_usage_percent") * Default: 95 * Type: Int * Unit: - * Is mutable: Yes * Description: Hard limit of the storage usage percentage in all BE directories. If the storage usage (in percentage) of the BE storage directory exceeds this value and the remaining storage space is less than `storage_flood_stage_left_capacity_bytes`, Load and Restore jobs are rejected. You need to set this item together with the FE configuration item `storage_usage_hard_limit_percent` to allow the configurations to take effect. * Introduced in: - ##### storage\_high\_usage\_disk\_protect\_ratio[​](#storage_high_usage_disk_protect_ratio "Direct link to storage_high_usage_disk_protect_ratio") * Default: 0.1 * Type: double * Unit: - * Is mutable: Yes * Description: When selecting a storage root for tablet creation, StorageEngine sorts candidate disks by `disk_usage(0)` and computes the average usage. Any disk whose usage is greater than (average usage + `storage_high_usage_disk_protect_ratio`) is excluded from the preferential selection pool (it will not participate in the randomized, preferrential shuffle and thus is deferred from being chosen initially). Set to 0 to disable this protection. Values are fractional (typical range 0.0–1.0); larger values make the scheduler more tolerant of higher-than-average disks. * Introduced in: v3.2.0 ##### storage\_medium\_migrate\_count[​](#storage_medium_migrate_count "Direct link to storage_medium_migrate_count") * Default: 3 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads used for storage medium migration (from SATA to SSD). * Introduced in: - ##### storage\_root\_path[​](#storage_root_path "Direct link to storage_root_path") * Default: `${STARROCKS_HOME}/storage` * Type: String * Unit: - * Is mutable: No * Description: The directory and medium of the storage volume. Example: `/data1,medium:hdd;/data2,medium:ssd`. * Multiple volumes are separated by semicolons (`;`). * If the storage medium is SSD, add `,medium:ssd` at the end of the directory. * If the storage medium is HDD, add `,medium:hdd` at the end of the directory. * Introduced in: - ##### sync\_tablet\_meta[​](#sync_tablet_meta "Direct link to sync_tablet_meta") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: A boolean value to control whether to enable the synchronization of the tablet metadata. `true` indicates enabling synchronization, and `false` indicates disabling it. * Introduced in: - ##### tablet\_map\_shard\_size[​](#tablet_map_shard_size "Direct link to tablet_map_shard_size") * Default: 1024 * Type: Int * Unit: - * Is mutable: No * Description: The tablet map shard size. The value must be a power of two. * Introduced in: - ##### tablet\_max\_pending\_versions[​](#tablet_max_pending_versions "Direct link to tablet_max_pending_versions") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of pending versions that are tolerable on a Primary Key tablet. Pending versions refer to versions that are committed but not applied yet. * Introduced in: - ##### tablet\_max\_versions[​](#tablet_max_versions "Direct link to tablet_max_versions") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of versions allowed on a tablet. If the number of versions exceeds this value, new write requests will fail. * Introduced in: - ##### tablet\_meta\_checkpoint\_min\_interval\_secs[​](#tablet_meta_checkpoint_min_interval_secs "Direct link to tablet_meta_checkpoint_min_interval_secs") * Default: 600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval of thread polling for a TabletMeta Checkpoint. * Introduced in: - ##### tablet\_meta\_checkpoint\_min\_new\_rowsets\_num[​](#tablet_meta_checkpoint_min_new_rowsets_num "Direct link to tablet_meta_checkpoint_min_new_rowsets_num") * Default: 10 * Type: Int * Unit: - * Is mutable: Yes * Description: The minimum number of rowsets to create since the last TabletMeta Checkpoint. * Introduced in: - ##### tablet\_rowset\_stale\_sweep\_time\_sec[​](#tablet_rowset_stale_sweep_time_sec "Direct link to tablet_rowset_stale_sweep_time_sec") * Default: 1800 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to sweep the stale rowsets in tablets. * Introduced in: - ##### tablet\_stat\_cache\_update\_interval\_second[​](#tablet_stat_cache_update_interval_second "Direct link to tablet_stat_cache_update_interval_second") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: 是 * Description: The time interval at which Tablet Stat Cache updates. * Introduced in: - ##### lake\_enable\_accurate\_pk\_row\_count[​](#lake_enable_accurate_pk_row_count "Direct link to lake_enable_accurate_pk_row_count") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to use accurate row counts for lake primary-key tablets. When enabled, StarRocks reads each rowset's delete vector from object storage and subtracts deleted rows, producing more accurate stats but potentially increasing `get_tablet_stats` RPC overhead. When disabled, StarRocks uses the approximate `num_dels` value in rowset metadata to avoid remote I/O, which may slightly overcount rows that were deleted but not yet compacted. * Introduced in: - ##### lake\_tablet\_stat\_slow\_log\_ms[​](#lake_tablet_stat_slow_log_ms "Direct link to lake_tablet_stat_slow_log_ms") * Default: 300000 * Type: Int64 * Unit: Milliseconds * Is mutable: Yes * Description: Threshold (in milliseconds) for logging slow tablet-stat collection tasks. If a single tablet stat task exceeds this value, StarRocks emits a warning log with diagnostics such as `tablet_id`, version, rowset count, accurate mode, and elapsed time. * Introduced in: - ##### lake\_metadata\_fetch\_thread\_count[​](#lake_metadata_fetch_thread_count "Direct link to lake_metadata_fetch_thread_count") * Default: 3 * Type: Int * Unit: - * Is mutable: Yes * Description: The count of threads for shared-data table tablet metadata fetch operations (e.g., `get_tablet_stats`, `get_tablet_metadatas`). * Introduced in: v3.5.16, v4.0.9 ##### tablet\_writer\_open\_rpc\_timeout\_sec[​](#tablet_writer_open_rpc_timeout_sec "Direct link to tablet_writer_open_rpc_timeout_sec") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Timeout (in seconds) for the RPC that opens a tablet writer on a remote BE. The value is converted to milliseconds and applied to both the request timeout and the brpc control timeout when issuing the open call. The runtime uses the effective timeout as the minimum of `tablet_writer_open_rpc_timeout_sec` and half of the overall load timeout (i.e., min(`tablet_writer_open_rpc_timeout_sec`, `load_timeout_sec` / 2)). Set this to balance timely failure detection (too small may cause premature open failures) and giving BEs enough time to initialize writers (too large delays error handling). * Introduced in: v3.2.0 ##### transaction\_apply\_worker\_count[​](#transaction_apply_worker_count "Direct link to transaction_apply_worker_count") * Default: 0 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Controls the maximum number of worker threads used by the UpdateManager's "update\_apply" thread pool — the pool that applies rowsets for transactions (notably for primary-key tables). A value `>0` sets a fixed maximum thread count; 0 (the default) makes the pool size equal to the number of CPU cores. The configured value is applied at startup (UpdateManager::init) and can be changed at runtime via the update-config HTTP action, which updates the pool's max threads. Tune this to increase apply concurrency (throughput) or limit CPU/memory contention; min threads and idle timeout are governed by transaction\_apply\_thread\_pool\_num\_min and transaction\_apply\_worker\_idle\_time\_ms respectively. * Introduced in: v3.2.0 ##### transaction\_apply\_worker\_idle\_time\_ms[​](#transaction_apply_worker_idle_time_ms "Direct link to transaction_apply_worker_idle_time_ms") * Default: 500 * Type: int * Unit: Milliseconds * Is mutable: No * Description: Sets the idle timeout (in milliseconds) for the UpdateManager's "update\_apply" thread pool used to apply transactions/updates. The value is passed to ThreadPoolBuilder::set\_idle\_timeout via MonoDelta::FromMilliseconds, so worker threads that remain idle longer than this timeout may be terminated (subject to the pool's configured minimum thread count and max threads). Lower values free resources faster but increase thread creation/teardown overhead under bursty load; higher values keep workers warm for short bursts at the cost of higher baseline resource usage. * Introduced in: v3.2.11 ##### trash\_file\_expire\_time\_sec[​](#trash_file_expire_time_sec "Direct link to trash_file_expire_time_sec") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to clean trash files. The default value has been changed from 259,200 to 86,400 since v2.5.17, v3.0.9, and v3.1.6. * Introduced in: - ##### unused\_rowset\_monitor\_interval[​](#unused_rowset_monitor_interval "Direct link to unused_rowset_monitor_interval") * Default: 30 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to clean the expired rowsets. * Introduced in: - ##### update\_cache\_expire\_sec[​](#update_cache_expire_sec "Direct link to update_cache_expire_sec") * Default: 360 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The expiration time of Update Cache. * Introduced in: - ##### update\_compaction\_check\_interval\_seconds[​](#update_compaction_check_interval_seconds "Direct link to update_compaction_check_interval_seconds") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to check compaction for Primary Key tables. * Introduced in: - ##### update\_compaction\_delvec\_file\_io\_amp\_ratio[​](#update_compaction_delvec_file_io_amp_ratio "Direct link to update_compaction_delvec_file_io_amp_ratio") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: Used to control the priority of compaction for rowsets that contain Delvec files in Primary Key tables. The larger the value, the higher the priority. * Introduced in: - ##### update\_compaction\_num\_threads\_per\_disk[​](#update_compaction_num_threads_per_disk "Direct link to update_compaction_num_threads_per_disk") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of Compaction threads per disk for Primary Key tables. * Introduced in: - ##### update\_compaction\_per\_tablet\_min\_interval\_seconds[​](#update_compaction_per_tablet_min_interval_seconds "Direct link to update_compaction_per_tablet_min_interval_seconds") * Default: 120 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The minimum time interval at which compaction is triggered for each tablet in a Primary Key table. * Introduced in: - ##### update\_compaction\_ratio\_threshold[​](#update_compaction_ratio_threshold "Direct link to update_compaction_ratio_threshold") * Default: 0.5 * Type: Double * Unit: - * Is mutable: Yes * Description: The maximum proportion of data that a compaction can merge for a Primary Key table in a shared-data cluster. It is recommended to shrink this value if a single tablet becomes excessively large. * Introduced in: v3.1.5 ##### update\_compaction\_result\_bytes[​](#update_compaction_result_bytes "Direct link to update_compaction_result_bytes") * Default: 1073741824 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The maximum result size of a single compaction for Primary Key tables. * Introduced in: - ##### update\_compaction\_size\_threshold[​](#update_compaction_size_threshold "Direct link to update_compaction_size_threshold") * Default: 268435456 * Type: Int * Unit: - * Is mutable: Yes * Description: The Compaction Score of Primary Key tables is calculated based on the file size, which is different from other table types. This parameter can be used to make the Compaction Score of Primary Key tables similar to that of other table types, making it easier for users to understand. * Introduced in: - ##### upload\_worker\_count[​](#upload_worker_count "Direct link to upload_worker_count") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads for the upload tasks of backup jobs on a BE node. `0` indicates setting the value to the number of CPU cores on the machine where the BE resides. * Introduced in: - ##### vertical\_compaction\_max\_columns\_per\_group[​](#vertical_compaction_max_columns_per_group "Direct link to vertical_compaction_max_columns_per_group") * Default: 5 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of columns per group of Vertical Compactions. * Introduced in: - --- ### FE Configuration FE parameters are classified into dynamic parameters and static parameters. * Dynamic parameters can be configured and adjusted by running SQL commands, which is very convenient. But the configurations become invalid if you restart your FE. Therefore, we recommend that you also modify the configuration items in the **fe.conf** file to prevent the loss of modifications. * Static parameters can only be configured and adjusted in the FE configuration file **fe.conf**. **After you modify this file, you must restart your FE for the changes to take effect.** Whether a parameter is a dynamic parameter is indicated by the `IsMutable` column in the output of [ADMIN SHOW CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). `TRUE` indicates a dynamic parameter. Note that both dynamic and static FE parameters can be configured in the **fe.conf** file. #### View FE configuration items[​](#view-fe-configuration-items "Direct link to View FE configuration items") After your FE is started, you can run the ADMIN SHOW FRONTEND CONFIG command on your MySQL client to check the parameter configurations. If you want to query the configuration of a specific parameter, run the following command: ```sql ADMIN SHOW FRONTEND CONFIG [LIKE "pattern"]; ``` For detailed description of the returned fields, see [`ADMIN SHOW CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). note You must have administrator privileges to run cluster administration-related commands. #### Configure FE parameters[​](#configure-fe-parameters "Direct link to Configure FE parameters") ##### Configure FE dynamic parameters[​](#configure-fe-dynamic-parameters "Direct link to Configure FE dynamic parameters") You can configure or modify the settings of FE dynamic parameters using [`ADMIN SET FRONTEND CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md). ```sql ADMIN SET FRONTEND CONFIG ("key" = "value"); ``` note The configuration changes made with `ADMIN SET FRONTEND` will be restored to the default values in the `fe.conf` file after the FE restarts. Therefore, we recommend that you also modify the configuration items in `fe.conf` if you want the changes to be permanent. ##### Configure FE static parameters[​](#configure-fe-static-parameters "Direct link to Configure FE static parameters") note Static parameters of an FE are set by changing them in the configuration file **fe.conf** and restarting the FE to allow the changes to take effect. #### Parameter groups[​](#parameter-groups "Direct link to Parameter groups") The parameters are grouped in these categories: * [Logging](https://docs.starrocks.io/docs/administration/configuration/FE_parameters/log_server_meta.md) * [Server](https://docs.starrocks.io/docs/administration/configuration/FE_parameters/log_server_meta.md) * [Metadata and Cluster management](https://docs.starrocks.io/docs/administration/configuration/FE_parameters/log_server_meta.md) * [User, role, and privilege](https://docs.starrocks.io/docs/administration/configuration/FE_parameters/user_query_loading.md) * [Query engine](https://docs.starrocks.io/docs/administration/configuration/FE_parameters/user_query_loading.md) * [Loading and unloading](https://docs.starrocks.io/docs/administration/configuration/FE_parameters/user_query_loading.md) * [Statistic report](https://docs.starrocks.io/docs/administration/configuration/FE_parameters/stats_storage.md) * [Storage](https://docs.starrocks.io/docs/administration/configuration/FE_parameters/stats_storage.md) * [Shared-data](https://docs.starrocks.io/docs/administration/configuration/FE_parameters/shared_lake_other.md) * [Data Lake](https://docs.starrocks.io/docs/administration/configuration/FE_parameters/shared_lake_other.md) * [Other](https://docs.starrocks.io/docs/administration/configuration/FE_parameters/shared_lake_other.md) --- ### FE Configuration - Logging, Server, and Metadata FE parameters are classified into dynamic parameters and static parameters. * Dynamic parameters can be configured and adjusted by running SQL commands, which is very convenient. But the configurations become invalid if you restart your FE. Therefore, we recommend that you also modify the configuration items in the **fe.conf** file to prevent the loss of modifications. * Static parameters can only be configured and adjusted in the FE configuration file **fe.conf**. **After you modify this file, you must restart your FE for the changes to take effect.** Whether a parameter is a dynamic parameter is indicated by the `IsMutable` column in the output of [ADMIN SHOW CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). `TRUE` indicates a dynamic parameter. Note that both dynamic and static FE parameters can be configured in the **fe.conf** file. #### View FE configuration items[​](#view-fe-configuration-items "Direct link to View FE configuration items") After your FE is started, you can run the ADMIN SHOW FRONTEND CONFIG command on your MySQL client to check the parameter configurations. If you want to query the configuration of a specific parameter, run the following command: ```sql ADMIN SHOW FRONTEND CONFIG [LIKE "pattern"]; ``` For detailed description of the returned fields, see [`ADMIN SHOW CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). note You must have administrator privileges to run cluster administration-related commands. #### Configure FE parameters[​](#configure-fe-parameters "Direct link to Configure FE parameters") ##### Configure FE dynamic parameters[​](#configure-fe-dynamic-parameters "Direct link to Configure FE dynamic parameters") You can configure or modify the settings of FE dynamic parameters using [`ADMIN SET FRONTEND CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md). ```sql ADMIN SET FRONTEND CONFIG ("key" = "value"); ``` note The configuration changes made with `ADMIN SET FRONTEND` will be restored to the default values in the `fe.conf` file after the FE restarts. Therefore, we recommend that you also modify the configuration items in `fe.conf` if you want the changes to be permanent. ##### Configure FE static parameters[​](#configure-fe-static-parameters "Direct link to Configure FE static parameters") note Static parameters of an FE are set by changing them in the configuration file **fe.conf** and restarting the FE to allow the changes to take effect. *** This topic introduces the following types of FE configurations: * [Logging](#logging) * [Server](#server) * [Metadata and Cluster Management](#metadata-and-cluster-management) #### Logging[​](#logging "Direct link to Logging") ##### `audit_log_delete_age`[​](#audit_log_delete_age "Direct link to audit_log_delete_age") * Default: 30d * Type: String * Unit: - * Is mutable: No * Description: The retention period of audit log files. The default value `30d` specifies that each audit log file can be retained for 30 days. StarRocks checks each audit log file and deletes those that were generated 30 days ago. * Introduced in: - ##### `audit_log_dir`[​](#audit_log_dir "Direct link to audit_log_dir") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/log" * Type: String * Unit: - * Is mutable: No * Description: The directory that stores audit log files. * Introduced in: - ##### `audit_log_enable_compress`[​](#audit_log_enable_compress "Direct link to audit_log_enable_compress") * Default: false * Type: Boolean * Unit: N/A * Is mutable: No * Description: When true, the generated Log4j2 configuration appends a ".gz" postfix to rotated audit log filenames (fe.audit.log.\*) so that Log4j2 will produce compressed (.gz) archived audit log files on rollover. The setting is read during FE startup in Log4jConfig.initLogging and is applied to the RollingFile appender for audit logs; it only affects rotated/archived files, not the active audit log. Because the value is initialized at startup, changing it requires restarting the FE to take effect. Use alongside audit log rotation settings (`audit_log_dir`, `audit_log_roll_interval`, `audit_roll_maxsize`, `audit_log_roll_num`). * Introduced in: 3.2.12 ##### `audit_log_error_message_max_length`[​](#audit_log_error_message_max_length "Direct link to audit_log_error_message_max_length") * Default: 1024 * Type: Int * Unit: Characters * Is mutable: Yes * Description: The maximum length of the `ErrorMessage` field recorded in the audit log for a failed statement, counted in UTF-16 code units, so a character outside the Basic Multilingual Plane such as an emoji counts as two. A longer message is truncated and suffixed with `... /* truncated, audit_log_error_message_max_length= */`; the suffix is not counted against the limit. Set it to `0` to stop recording error messages altogether, in which case the field is omitted from the audit record. Error messages are also omitted when `enable_sql_desensitize_in_log` is `true` or when `enable_audit_sql` is `false`, because an error message quotes the statement that failed, down to the values and the offending token. * Introduced in: 4.2.0 ##### `audit_log_json_format`[​](#audit_log_json_format "Direct link to audit_log_json_format") * Default: false * Type: Boolean * Unit: N/A * Is mutable: Yes * Description: When true, FE audit events are emitted as structured JSON (Jackson ObjectMapper serializing a Map of annotated AuditEvent fields) instead of the default pipe-separated "key=value" string. The setting affects all built-in audit sinks handled by AuditLogBuilder: connection audit, query audit, big-query audit (big-query threshold fields are added to the JSON when the event qualifies), and slow-audit output. Fields annotated for big-query thresholds and the "features" field are treated specially (excluded from normal audit entries; included in big-query or feature logs as applicable). Enable this to make logs machine-parsable for log collectors or SIEMs; note it changes the log format and may require updating any existing parsers that expect the legacy pipe-separated format. * Introduced in: 3.2.7 ##### `audit_log_modules`[​](#audit_log_modules "Direct link to audit_log_modules") * Default: `slow_query`, query * Type: String\[] * Unit: - * Is mutable: No * Description: The modules for which StarRocks generates audit log entries. By default, StarRocks generates audit logs for the `slow_query` module and the `query` module. The `connection` module is supported from v3.0. Separate the module names with a comma (,) and a space. * Introduced in: - ##### `audit_log_roll_interval`[​](#audit_log_roll_interval "Direct link to audit_log_roll_interval") * Default: DAY * Type: String * Unit: - * Is mutable: No * Description: The time interval at which StarRocks rotates audit log entries. Valid values: `DAY` and `HOUR`. * If this parameter is set to `DAY`, a suffix in the `yyyyMMdd` format is added to the names of audit log files. * If this parameter is set to `HOUR`, a suffix in the `yyyyMMddHH` format is added to the names of audit log files. * Introduced in: - ##### `audit_log_roll_num`[​](#audit_log_roll_num "Direct link to audit_log_roll_num") * Default: 90 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of audit log files that can be retained within each retention period specified by the `audit_log_roll_interval` parameter. * Introduced in: - ##### `audit_stmt_before_execute`[​](#audit_stmt_before_execute "Direct link to audit_stmt_before_execute") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls whether FE emits a `BEFORE_QUERY` audit event before statement execution. When enabled, ConnectProcessor writes one audit record before execution and still writes the normal `AFTER_QUERY` audit record after execution. For multi-statement requests in this branch, audit records remain command-level rather than per sub-statement. * Introduced in: - ##### `bdbje_log_level`[​](#bdbje_log_level "Direct link to bdbje_log_level") * Default: INFO * Type: String * Unit: - * Is mutable: No * Description: Controls the logging level used by Berkeley DB Java Edition (BDB JE) in StarRocks. During BDB environment initialization BDBEnvironment.initConfigs() applies this value to the Java logger for the `com.sleepycat.je` package and to the BDB JE environment file logging level (`EnvironmentConfig.FILE_LOGGING_LEVEL`). Accepts standard java.util.logging.Level names such as SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, ALL, OFF. Setting to ALL enables all log messages. Increasing verbosity will raise log volume and may impact disk I/O and performance; the value is read when the BDB environment is initialized, so it takes effect only after environment (re)initialization. * Introduced in: v3.2.0 ##### `big_query_log_delete_age`[​](#big_query_log_delete_age "Direct link to big_query_log_delete_age") * Default: 7d * Type: String * Unit: - * Is mutable: No * Description: Controls how long FE big query log files (`fe.big_query.log.*`) are retained before automatic deletion. The value is passed to Log4j's deletion policy as the IfLastModified age — any rotated big query log whose last-modified time is older than this value will be removed. Supports suffixes include `d` (day), `h` (hour), `m` (minute), and `s` (second). Example: `7d` (7 days), `10h` (10 hours), `60m` (60 minutes), and `120s` (120 seconds). This item works together with `big_query_log_roll_interval` and `big_query_log_roll_num` to determine which files are kept or purged. * Introduced in: v3.2.0 ##### `big_query_log_dir`[​](#big_query_log_dir "Direct link to big_query_log_dir") * Default: `Config.STARROCKS_HOME_DIR + "/log"` * Type: String * Unit: - * Is mutable: No * Description: Directory where the FE writes big query dump logs (`fe.big_query.log.*`). The Log4j configuration uses this path to create a RollingFile appender for `fe.big_query.log` and its rotated files. Rotation and retention are governed by `big_query_log_roll_interval` (time-based suffix), `log_roll_size_mb` (size trigger), `big_query_log_roll_num` (max files), and `big_query_log_delete_age` (age-based deletion). Big query records are logged for queries that exceed user-defined thresholds such as `big_query_log_cpu_second_threshold`, `big_query_log_scan_rows_threshold`, or `big_query_log_scan_bytes_threshold`. Use `big_query_log_modules` to control which modules log to this file. * Introduced in: v3.2.0 ##### `big_query_log_modules`[​](#big_query_log_modules "Direct link to big_query_log_modules") * Default: `{"query"}` * Type: String\[] * Unit: - * Is mutable: No * Description: List of module name suffixes that enable per-module big query logging. Typical values are logical component names. For example, the default `query` produces `big_query.query`. * Introduced in: v3.2.0 ##### `big_query_log_roll_interval`[​](#big_query_log_roll_interval "Direct link to big_query_log_roll_interval") * Default: `"DAY"` * Type: String * Unit: - * Is mutable: No * Description: Specifies the time interval used to construct the date component of the rolling file name for the `big_query` log appender. Valid values (case-insensitive) are `DAY` (default) and `HOUR`. `DAY` produces a daily pattern (`"%d{yyyyMMdd}"`) and `HOUR` produces an hourly pattern (`"%d{yyyyMMddHH}"`). The value is combined with size-based rollover (`big_query_roll_maxsize`) and index-based rollover (`big_query_log_roll_num`) to form the RollingFile filePattern. An invalid value causes log configuration generation to fail (IOException) and may prevent log initialization or reconfiguration. Use alongside `big_query_log_dir`, `big_query_roll_maxsize`, `big_query_log_roll_num`, and `big_query_log_delete_age`. * Introduced in: v3.2.0 ##### `big_query_log_roll_num`[​](#big_query_log_roll_num "Direct link to big_query_log_roll_num") * Default: 10 * Type: Int * Unit: - * Is mutable: No * Description: Maximum number of rotated FE big query log files to retain per `big_query_log_roll_interval`. This value is bound to the RollingFile appender's DefaultRolloverStrategy `max` attribute for `fe.big_query.log`; when logs roll (by time or by `log_roll_size_mb`), StarRocks keeps up to `big_query_log_roll_num` indexed files (filePattern uses a time suffix plus index). Files older than this count may be removed by rollover, and `big_query_log_delete_age` can additionally delete files by last-modified age. * Introduced in: v3.2.0 ##### `dump_log_delete_age`[​](#dump_log_delete_age "Direct link to dump_log_delete_age") * Default: 7d * Type: String * Unit: - * Is mutable: No * Description: The retention period of dump log files. The default value `7d` specifies that each dump log file can be retained for 7 days. StarRocks checks each dump log file and deletes those that were generated 7 days ago. * Introduced in: - ##### `dump_log_dir`[​](#dump_log_dir "Direct link to dump_log_dir") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/log" * Type: String * Unit: - * Is mutable: No * Description: The directory that stores dump log files. * Introduced in: - ##### `dump_log_modules`[​](#dump_log_modules "Direct link to dump_log_modules") * Default: query * Type: String\[] * Unit: - * Is mutable: No * Description: The modules for which StarRocks generates dump log entries. By default, StarRocks generates dump logs for the query module. Separate the module names with a comma (,) and a space. * Introduced in: - ##### `dump_log_roll_interval`[​](#dump_log_roll_interval "Direct link to dump_log_roll_interval") * Default: DAY * Type: String * Unit: - * Is mutable: No * Description: The time interval at which StarRocks rotates dump log entries. Valid values: `DAY` and `HOUR`. * If this parameter is set to `DAY`, a suffix in the `yyyyMMdd` format is added to the names of dump log files. * If this parameter is set to `HOUR`, a suffix in the `yyyyMMddHH` format is added to the names of dump log files. * Introduced in: - ##### `dump_log_roll_num`[​](#dump_log_roll_num "Direct link to dump_log_roll_num") * Default: 10 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of dump log files that can be retained within each retention period specified by the `dump_log_roll_interval` parameter. * Introduced in: - ##### `edit_log_write_slow_log_threshold_ms`[​](#edit_log_write_slow_log_threshold_ms "Direct link to edit_log_write_slow_log_threshold_ms") * Default: 2000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Threshold (in ms) used by JournalWriter to detect and log slow edit-log batch writes. After a batch commit, if the batch duration exceeds this value, JournalWriter emits a WARN with batch size, duration and current journal queue size (rate-limited to once every ~2s). This setting only controls logging/alerts for potential IO or replication latency on the FE leader; it does not change commit or roll behavior (see `edit_log_roll_num` and commit-related settings). Metric updates still occur regardless of this threshold. * Introduced in: v3.2.3 ##### `enable_audit_sql`[​](#enable_audit_sql "Direct link to enable_audit_sql") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: When this item is set to `true`, the FE audit subsystem records the SQL text of statements into FE audit logs (`fe.audit.log`) processed by ConnectProcessor. The stored statement respects other controls: encrypted statements are redacted (`AuditEncryptionChecker`), sensitive credentials may be redacted or desensitized if `enable_sql_desensitize_in_log` is set, and digest recording is controlled by `enable_sql_digest`. When it is set to `false`, ConnectProcessor replaces the statement text with "?" in audit events — other audit fields (user, host, duration, status, slow-query detection via `qe_slow_log_ms`, and metrics) are still recorded. Enabling SQL audit increases forensic and troubleshooting visibility but may expose sensitive SQL content and increase log volume and I/O; disabling it improves privacy at the cost of losing full-statement visibility in audit logs. * Introduced in: - ##### `enable_print_load_profile_to_log`[​](#enable_print_load_profile_to_log "Direct link to enable_print_load_profile_to_log") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When set to `true`, load profiles (such as Stream Load, Routine Load, Broker Load, and Merge Commit) are additionally written to the profile log (`fe.profile.log`) at INFO level when they are pushed to `ProfileManager`, as a single-line JSON record in the same format as the query profile log. This makes load profiles recoverable from the log even after they are evicted from `ProfileManager` due to the `profile_info_reserved_num` limit. The profile log is used (rather than `fe.log`) because its JSON layout caps strings at `sys_log_json_profile_max_string_length` instead of the much smaller `sys_log_json_max_string_length`, so large load profiles are not truncated; the file is rotated and retained by the `profile_log_*` parameters. Only profiles whose query type is `Load` are printed; query profiles are not affected. A load profile is printed only when it is actually collected (for example, when `enable_profile` is enabled or the load exceeds the big-load profile threshold). * Introduced in: - ##### `enable_profile_log`[​](#enable_profile_log "Direct link to enable_profile_log") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable profile logging. When this feature is enabled, the FE writes per-query profile logs (the serialized `queryDetail` JSON produced by `ProfileManager`) to the profile log sink. This logging is performed only if `enable_collect_query_detail_info` is also enabled; when `enable_profile_log_compress` is enabled, the JSON may be gzipped before logging. Profile log files are managed by `profile_log_dir`, `profile_log_roll_num`, `profile_log_roll_interval` and rotated/deleted according to `profile_log_delete_age` (supports formats like `7d`, `10h`, `60m`, `120s`). Disabling this feature stops writing profile logs (reducing disk I/O, compression CPU and storage usage). Which queries are logged can be further filtered by `profile_log_latency_threshold_ms`. * Introduced in: v3.2.5 ##### `enable_qe_slow_log`[​](#enable_qe_slow_log "Direct link to enable_qe_slow_log") * Default: true * Type: Boolean * Unit: N/A * Is mutable: Yes * Description: When enabled, the FE builtin audit plugin (AuditLogBuilder) will write query events whose measured execution time ("Time" field) exceeds the threshold configured by `qe_slow_log_ms` into the slow-query audit log (AuditLog.getSlowAudit). If disabled, those slow-query entries are suppressed (regular query and connection audit logs are unaffected). The slow-audit entries follow the global `audit_log_json_format` setting (JSON vs. plain string). Use this flag to control generation of slow-query audit volume independently of regular audit logging; turning it off may reduce log I/O when `qe_slow_log_ms` is low or workloads produce many long-running queries. * Introduced in: 3.2.11 ##### `enable_sql_desensitize_in_log`[​](#enable_sql_desensitize_in_log "Direct link to enable_sql_desensitize_in_log") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: When this item is set to `true`, the system replaces or hides sensitive SQL content before it is written to logs, query-detail records, and query profiles. Code paths that honor this configuration include ConnectProcessor.formatStmt (audit logs), StmtExecutor.addRunningQueryDetail (query details), SimpleExecutor.formatSQL (internal executor logs), and StmtExecutor.buildTopLevelProfile / processProfileAsync (the `Sql Statement` and `ExplainPlan` info-strings stored in a profile's `Summary` section). With the feature enabled, invalid SQLs may be replaced with a fixed desensitized message, credentials (user/password) are hidden, and the SQL formatter is required to produce a sanitized representation (it can also enable digest-style output). For the `ExplainPlan` field added by the `enable_explain_in_profile` session variable, this config also forces literal-digest rendering of the embedded `EXPLAIN COSTS` text, so the profile does not leak the literals that the persisted `Sql Statement` would have hidden. This reduces leakage of sensitive literals and credentials in audit/internal logs and profiles, but also means logs, query details, and profiles no longer contain the original full SQL text (which can affect replay or debugging). * Introduced in: - ##### `internal_log_delete_age`[​](#internal_log_delete_age "Direct link to internal_log_delete_age") * Default: 7d * Type: String * Unit: - * Is mutable: No * Description: Specifies the retention period for FE internal log files (written to `internal_log_dir`). The value is a duration string. Supported suffixes: `d` (day), `h` (hour), `m` (minute), `s` (second). Examples: `7d` (7 days), `10h` (10 hours), `60m` (60 minutes), `120s` (120 seconds). This item is substituted into the log4j configuration as the `` predicate used by the RollingFile Delete policy. Files whose last-modified time is earlier than this duration will be removed during log rollover. Increase this value to free disk space sooner, or decrease it to retain internal materialized view or statistics logs longer. * Introduced in: v3.2.4 ##### `internal_log_dir`[​](#internal_log_dir "Direct link to internal_log_dir") * Default: `Config.STARROCKS_HOME_DIR` + "/log" * Type: String * Unit: - * Is mutable: No * Description: Directory used by the FE logging subsystem for storing internal logs (`fe.internal.log`). This configuration is substituted into the Log4j configuration and determines where the InternalFile appender writes internal/materialized view/statistics logs and where per-module loggers under `internal.` place their files. Ensure the directory exists, is writable, and has sufficient disk space. Log rotation and retention for files in this directory are controlled by `log_roll_size_mb`, `internal_log_roll_num`, `internal_log_delete_age`, and `internal_log_roll_interval`. If `sys_log_to_console` is enabled, internal logs may be written to console instead of this directory. * Introduced in: v3.2.4 ##### `internal_log_json_format`[​](#internal_log_json_format "Direct link to internal_log_json_format") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When this item is set to `true`, internal statistic/audit entries are written as compact JSON objects to the statistic audit logger. The JSON contains keys "executeType" (InternalType: QUERY or DML), "queryId", "sql", and "time" (elapsed milliseconds). When it is set to `false`, the same information is logged as a single formatted text line ("statistic execute: ... | QueryId: \[...] | SQL: ..."). Enabling JSON improves machine parsing and integration with log processors but also causes raw SQL text to be included in logs, which may expose sensitive information and increase log size. * Introduced in: - ##### `internal_log_modules`[​](#internal_log_modules "Direct link to internal_log_modules") * Default: `{"base", "statistic"}` * Type: String\[] * Unit: - * Is mutable: No * Description: A list of module identifiers that will receive dedicated internal logging. For each entry X, Log4j creates a logger named `internal.` with level INFO and additivity="false". Those loggers are routed to the internal appender (written to `fe.internal.log`) or to console when `sys_log_to_console` is enabled. Use short names or package fragments as needed — the exact logger name becomes `internal.` + the configured string. Internal log file rotation and retention follow `internal_log_dir`, `internal_log_roll_num`, `internal_log_delete_age`, `internal_log_roll_interval`, and `log_roll_size_mb`. Adding a module causes its runtime messages to be separated into the internal logger stream for easier debugging and audit. * Introduced in: v3.2.4 ##### `internal_log_roll_interval`[​](#internal_log_roll_interval "Direct link to internal_log_roll_interval") * Default: DAY * Type: String * Unit: - * Is mutable: No * Description: Controls the time-based roll interval for the FE internal log appender. Accepted values (case-insensitive) are `HOUR` and `DAY`. `HOUR` produces an hourly file pattern (`"%d{yyyyMMddHH}"`) and `DAY` produces a daily file pattern (`"%d{yyyyMMdd}"`), which are used by the RollingFile TimeBasedTriggeringPolicy to name rotated `fe.internal.log` files. An invalid value causes initialization to fail (an IOException is thrown when building the active Log4j configuration). Roll behavior also depends on related settings such as `internal_log_dir`, `internal_roll_maxsize`, `internal_log_roll_num`, and `internal_log_delete_age`. * Introduced in: v3.2.4 ##### `internal_log_roll_num`[​](#internal_log_roll_num "Direct link to internal_log_roll_num") * Default: 90 * Type: Int * Unit: - * Is mutable: No * Description: Maximum number of rolled internal FE log files to retain for the internal appender (`fe.internal.log`). This value is used as the Log4j DefaultRolloverStrategy `max` attribute; when rollovers occur, StarRocks keeps up to `internal_log_roll_num` archived files and removes older ones (also governed by `internal_log_delete_age`). A lower value reduces disk usage but shortens log history; a higher value preserves more historical internal logs. This item works together with `internal_log_dir`, `internal_log_roll_interval`, and `internal_roll_maxsize`. * Introduced in: v3.2.4 ##### `log_cleaner_audit_log_min_retention_days`[​](#log_cleaner_audit_log_min_retention_days "Direct link to log_cleaner_audit_log_min_retention_days") * Default: 3 * Type: Int * Unit: Days * Is mutable: Yes * Description: Minimum retention days for audit log files. Audit log files newer than this will not be deleted even if disk usage is high. This ensures that audit logs are preserved for compliance and troubleshooting purposes. * Introduced in: - ##### `log_cleaner_check_interval_second`[​](#log_cleaner_check_interval_second "Direct link to log_cleaner_check_interval_second") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Interval in seconds to check disk usage and clean logs. The cleaner periodically checks each log directory's disk usage and triggers cleaning when necessary. Default is 300 seconds (5 minutes). * Introduced in: - ##### `log_cleaner_disk_usage_target`[​](#log_cleaner_disk_usage_target "Direct link to log_cleaner_disk_usage_target") * Default: 60 * Type: Int * Unit: Percentage * Is mutable: Yes * Description: Target disk usage (percentage) after log cleaning. Log cleaning will continue until disk usage drops below this threshold. The cleaner deletes the oldest log files one by one until the target is reached. * Introduced in: - ##### `log_cleaner_disk_usage_threshold`[​](#log_cleaner_disk_usage_threshold "Direct link to log_cleaner_disk_usage_threshold") * Default: 80 * Type: Int * Unit: Percentage * Is mutable: Yes * Description: Disk usage threshold (percentage) to trigger log cleaning. When disk usage exceeds this threshold, log cleaning will start. The cleaner checks each configured log directory independently and processes directories that exceed this threshold. * Introduced in: - ##### `log_cleaner_disk_util_based_enable`[​](#log_cleaner_disk_util_based_enable "Direct link to log_cleaner_disk_util_based_enable") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Enable automatic log cleaning based on disk usage. When enabled, logs will be cleaned when disk usage exceeds the threshold. The log cleaner runs as a background daemon on the FE node and helps prevent disk space exhaustion from log file accumulation. * Introduced in: - ##### `log_plan_cancelled_by_crash_be`[​](#log_plan_cancelled_by_crash_be "Direct link to log_plan_cancelled_by_crash_be") * Default: true * Type: boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the query execution plan logging when a query is cancelled due to BE crash or an RPC exception. When this feature is enabled, StarRocks logs the query execution plan (at `TExplainLevel.COSTS`) as a WARN entry when a query is cancelled due to BE crash or an `RpcException`. The log entry includes QueryId, SQL and the COSTS plan; in the ExecuteExceptionHandler path, the exception stacktrace is also logged. The logging is skipped when `enable_collect_query_detail_info` is enabled (the plan is then stored in the query detail) — in code paths, the check is performed by verifying the query detail is null. Note that, in ExecuteExceptionHandler, the plan is logged only on the first retry (`retryTime == 0`). Enabling this may increase log volume because full COSTS plans can be large. * Introduced in: v3.2.0 ##### `log_register_and_unregister_query_id`[​](#log_register_and_unregister_query_id "Direct link to log_register_and_unregister_query_id") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow FE to log query registration and deregistration messages (e.g., `"register query id = {}"` and `"deregister query id = {}"`) from QeProcessorImpl. The log is emitted only when the query has a non-null ConnectContext and either the command is not `COM_STMT_EXECUTE` or the session variable `isAuditExecuteStmt()` is true. Because these messages are written for every query lifecycle event, enabling this feature can produce high log volume and become a throughput bottleneck in high concurrency environments. Enable it for debugging or auditing; and disable it to reduce logging overhead and improve performance. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### `log_roll_size_mb`[​](#log_roll_size_mb "Direct link to log_roll_size_mb") * Default: 1024 * Type: Int * Unit: MB * Is mutable: No * Description: The maximum size of a system log file or an audit log file. * Introduced in: - ##### `proc_profile_file_retained_days`[​](#proc_profile_file_retained_days "Direct link to proc_profile_file_retained_days") * Default: 1 * Type: Int * Unit: Days * Is mutable: Yes * Description: Number of days to retain process profiling files (CPU and memory) generated under `sys_log_dir/proc_profile`. The ProcProfileCollector computes a cutoff by subtracting `proc_profile_file_retained_days` days from the current time (formatted as yyyyMMdd-HHmmss) and deletes profile files whose timestamp portion is lexicographically earlier than that cutoff (that is, `timePart.compareTo(timeToDelete) < 0`). File deletion also respects the size-based cutoff controlled by `proc_profile_file_retained_size_bytes`. Profile files use the prefixes `cpu-profile-` and `mem-profile-` and are compressed after collection. * Introduced in: v3.2.12 ##### `proc_profile_file_retained_size_bytes`[​](#proc_profile_file_retained_size_bytes "Direct link to proc_profile_file_retained_size_bytes") * Default: 2L \* 1024 \* 1024 \* 1024 (2147483648) * Type: Long * Unit: Bytes * Is mutable: Yes * Description: Maximum total bytes of collected CPU and memory profile files (files named with prefixes `cpu-profile-` and `mem-profile-`) to keep under the profile directory. When the sum of valid profile files exceeds `proc_profile_file_retained_size_bytes`, the collector deletes the oldest profile files until the remaining total size is less than or equal to `proc_profile_file_retained_size_bytes`. Files older than `proc_profile_file_retained_days` are also removed regardless of size. This setting controls disk usage for profile archives and interacts with `proc_profile_file_retained_days` to determine deletion order and retention. * Introduced in: v3.2.12 ##### `profile_log_delete_age`[​](#profile_log_delete_age "Direct link to profile_log_delete_age") * Default: 1d * Type: String * Unit: - * Is mutable: No * Description: Controls how long FE profile log files are retained before they are eligible for deletion. The value is injected into Log4j's `` policy (via `Log4jConfig`) and is applied together with rotation settings such as `profile_log_roll_interval` and `profile_log_roll_num`. Supported suffixes: `d` (day), `h` (hour), `m` (minute), `s` (second). For example: `7d` (7 days), `10h` (10 hours), `60m` (60 minutes), `120s` (120 seconds). * Introduced in: v3.2.5 ##### `profile_log_dir`[​](#profile_log_dir "Direct link to profile_log_dir") * Default: `Config.STARROCKS_HOME_DIR` + "/log" * Type: String * Unit: - * Is mutable: No * Description: Directory where FE profile logs are written. Log4jConfig uses this value to place profile-related appenders (creates files like `fe.profile.log` under this directory). Rotation and retention for these files are governed by `profile_log_roll_size_mb`, `profile_log_roll_num` and `profile_log_delete_age`; the timestamp suffix format is controlled by `profile_log_roll_interval` (supports DAY or HOUR). Because the default directory is under `STARROCKS_HOME_DIR`, ensure the FE process has write and rotation/delete permissions on this directory. * Introduced in: v3.2.5 ##### `profile_log_latency_threshold_ms`[​](#profile_log_latency_threshold_ms "Direct link to profile_log_latency_threshold_ms") * Default: 0 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: Minimum query latency (in milliseconds) for a profile to be written to `fe.profile.log`. Only queries whose execution time is greater than or equal to this value are logged. Set to 0 to log all profiles (no threshold). Use a positive value to reduce log volume by logging only slower queries. * Introduced in: - ##### `profile_log_roll_interval`[​](#profile_log_roll_interval "Direct link to profile_log_roll_interval") * Default: DAY * Type: String * Unit: - * Is mutable: No * Description: Controls the time granularity used to generate the date part of profile log filenames. Valid values (case-insensitive) are `HOUR` and `DAY`. `HOUR` produces a pattern of `"%d{yyyyMMddHH}"` (hourly time bucket) and `DAY` produces `"%d{yyyyMMdd}"` (daily time bucket). This value is used when computing `profile_file_pattern` in the Log4j configuration and only affects the time-based component of rollover file names; size-based rollover is still controlled by `profile_log_roll_size_mb` and retention by `profile_log_roll_num` / `profile_log_delete_age`. Invalid values cause an IOException during logging initialization (error message: `"profile_log_roll_interval config error: "`). Choose `HOUR` for high-volume profiling to limit per-file size per hour, or `DAY` for daily aggregation. * Introduced in: v3.2.5 ##### `profile_log_roll_num`[​](#profile_log_roll_num "Direct link to profile_log_roll_num") * Default: 5 * Type: Int * Unit: - * Is mutable: No * Description: Specifies the maximum number of rotated profile log files retained by Log4j's DefaultRolloverStrategy for the profile logger. This value is injected into the logging XML as `${profile_log_roll_num}` (e.g. ``). Rotations are triggered by `profile_log_roll_size_mb` or `profile_log_roll_interval`; when rotation occurs, Log4j keeps at most these indexed files and older index files become eligible for removal. Actual retention on disk is also affected by `profile_log_delete_age` and the `profile_log_dir` location. Lower values reduce disk usage but limit retained history; higher values preserve more historical profile logs. * Introduced in: v3.2.5 ##### `profile_log_roll_size_mb`[​](#profile_log_roll_size_mb "Direct link to profile_log_roll_size_mb") * Default: 1024 * Type: Int * Unit: MB * Is mutable: No * Description: Sets the size threshold (in megabytes) that triggers a size-based rollover of the FE profile log file. This value is used by the Log4j RollingFile SizeBasedTriggeringPolicy for the `ProfileFile` appender; when a profile log exceeds `profile_log_roll_size_mb` it will be rotated. Rotation can also occur by time when `profile_log_roll_interval` is reached — either condition will trigger rollover. Combined with `profile_log_roll_num` and `profile_log_delete_age`, this item controls how many historical profile files are retained and when old files are deleted. Compression of rotated files is controlled by `enable_profile_log_compress`. * Introduced in: v3.2.5 ##### `qe_slow_log_ms`[​](#qe_slow_log_ms "Direct link to qe_slow_log_ms") * Default: 5000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The threshold used to determine whether a query is a slow query. If the response time of a query exceeds this threshold, it is recorded as a slow query in **fe.audit.log**. * Introduced in: - ##### `slow_lock_log_l2_info_interval_ms`[​](#slow_lock_log_l2_info_interval_ms "Direct link to slow_lock_log_l2_info_interval_ms") * Default: 3000L * Type: Long * Unit: Milliseconds * Is mutable: Yes * Alias: `slow_lock_log_every_ms` (the original name, retained for backward compatibility — both names refer to the same parameter). * Description: Minimum interval (in ms) for the **L2** slow-lock log tier — a full lock-info JSON line **without** stack traces. Slow-lock logging degrades across three tiers, throttled progressively (strictest first): **L1** = full info + stacks (`slow_lock_log_l1_stack_interval_ms`), **L2** = full info, no stacks (this parameter), **L3** = a plain-text brief line (`slow_lock_log_l3_brief_interval_ms`). For one slow-lock event the richest tier whose throttle currently admits is emitted; choosing a higher tier also consumes the looser tiers' windows, so total log volume never exceeds the loosest admitted tier's rate. The throttle scope depends on the emitting layer: **GLOBAL** in `LockManager.logSlowLockTrace` (one static gate across all rids), **per-instance** in `QueryableReentrantReadWriteLock` (each lock object — e.g. each `RoutineLoadJob` — has its own gate), and **per-Database** in the legacy `LockUtils` path. Set to `0` (or negative) to disable the L2 gate (always admit). Use a larger value to reduce log volume during prolonged contention or a smaller value for more frequent full-info diagnostics. * Introduced in: v3.2.0 (as `slow_lock_log_every_ms`); renamed to `slow_lock_log_l2_info_interval_ms` in v4.1. ##### `slow_lock_print_stack`[​](#slow_lock_print_stack "Direct link to slow_lock_print_stack") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Master switch for capturing owner / current-thread stack traces inside slow-lock warnings. Applies to both `LockManager.logSlowLockTrace` (per-owner `"stack"` field) and `QueryableReentrantReadWriteLock.getLockInfoToJson` (owner / oldest-reader / current-thread `"stack"` field used by the legacy db-lock path and `RoutineLoadJob`'s per-job lock). Enabling this feature helps debugging by giving precise thread stacks that hold the lock; disabling it reduces log volume and CPU/memory overhead caused by capturing and serializing stack traces in high concurrency environments. When enabled, the capture frequency is additionally rate-limited by `slow_lock_log_l1_stack_interval_ms`. * Introduced in: v3.3.16, v3.4.5, v3.5.1 ##### `slow_lock_log_l1_stack_interval_ms`[​](#slow_lock_log_l1_stack_interval_ms "Direct link to slow_lock_log_l1_stack_interval_ms") * Default: 30000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: Minimum interval between stack-trace captures across slow-lock log events. Only applies when `slow_lock_print_stack` is `true`. The throttle scope depends on the layer: **GLOBAL** in `LockManager.logSlowLockTrace` (one static gate across all rids) and **per-instance** in `QueryableReentrantReadWriteLock.getLockInfoToJson` (each lock object has its own gate). When the switch is on but the interval has not elapsed since the last capture, the `"stack"` field is replaced with the marker `"throttled"` (LockManager path) or omitted (QueryableReentrantReadWriteLock path), and the rest of the warn log (rid, owners, waiters, queryIds, timings) is still emitted if the outer event gate (`slow_lock_log_l2_info_interval_ms`) lets it through. Set to `0` (or negative) to disable rate limiting and restore the prior behavior of capturing stacks on every slow-lock event. `Thread.getStackTrace` triggers a JVM safepoint that becomes expensive in large clusters where slow-lock events are frequent — this gate caps that cost without suppressing the diagnostic log itself. * Introduced in: v4.1 ##### `slow_lock_max_waiter_count_to_log`[​](#slow_lock_max_waiter_count_to_log "Direct link to slow_lock_max_waiter_count_to_log") * Default: 30 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of waiter entries serialized into a single slow-lock log event. Applies to both `LockManager.logSlowLockTrace` (the `"waiter"` array) and `QueryableReentrantReadWriteLock.getLockInfoToJson` (the `"queuedReaders"` / `"queuedWriters"` arrays consumed by the legacy db-lock path and `RoutineLoadJob`'s per-job lock). When the actual waiter count exceeds this cap, the first N waiters are listed individually and the remainder is summarized as a single trailer entry `{"omitted": "remain M waiters omitted"}` appended to the array. Bounds Gson serialization cost and log-line size under extreme contention without losing the count diagnostic. Set to `0` (or negative) to disable the cap and serialize every waiter. * Introduced in: v4.1 ##### `slow_lock_log_l3_brief_interval_ms`[​](#slow_lock_log_l3_brief_interval_ms "Direct link to slow_lock_log_l3_brief_interval_ms") * Default: 1000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: Minimum interval for the **L3** slow-lock log tier — a single plain-text warn line (no JSON, no stacks) emitted when the richer tiers (`slow_lock_log_l1_stack_interval_ms` for L1, `slow_lock_log_l2_info_interval_ms` for L2) are throttled. This is the loosest of the three tiers. The brief line is emitted **at most once per this interval**: slow-lock events that arrive while the L3 gate is still closed are suppressed (no line). It does **not** guarantee a log line per event — it bounds the worst-case silence to one brief interval during sustained contention. Tune it smaller than the other two: `slow_lock_log_l3_brief_interval_ms < slow_lock_log_l2_info_interval_ms < slow_lock_log_l1_stack_interval_ms`. Set to `0` (or negative) to make the brief line unthrottled — then every otherwise-throttled event leaves a line (predictable but potentially many per second under a storm). Same scope rules as the other slow-lock throttles (GLOBAL in `LockManager`, per-instance in `QueryableReentrantReadWriteLock`). * Introduced in: v4.1 ##### `slow_lock_threshold_ms`[​](#slow_lock_threshold_ms "Direct link to slow_lock_threshold_ms") * Default: 3000L * Type: long * Unit: Milliseconds * Is mutable: Yes * Description: Threshold (in ms) used to classify a lock operation or a held lock as "slow". When the elapsed wait or hold time for a lock exceeds this value, StarRocks will (depending on context) emit diagnostic logs, include stack traces or waiter/owner info, and—in LockManager—start deadlock detection after this delay. It's used by LockUtils (slow-lock logging), QueryableReentrantReadWriteLock (filtering slow readers), LockManager (deadlock-detection delay and slow-lock trace), LockChecker (periodic slow-lock detection), and other callers (e.g., DiskAndTabletLoadReBalancer logging). Lowering the value increases sensitivity and logging/diagnostic overhead; setting it to 0 or negative disables the initial wait-based deadlock-detection delay behavior. Tune together with `slow_lock_log_l2_info_interval_ms`, `slow_lock_print_stack`, and `slow_lock_stack_trace_reserve_levels`. * Introduced in: 3.2.0 ##### `sys_log_delete_age`[​](#sys_log_delete_age "Direct link to sys_log_delete_age") * Default: 7d * Type: String * Unit: - * Is mutable: No * Description: The retention period of system log files. The default value `7d` specifies that each system log file can be retained for 7 days. StarRocks checks each system log file and deletes those that were generated 7 days ago. * Introduced in: - ##### `sys_log_dir`[​](#sys_log_dir "Direct link to sys_log_dir") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/log" * Type: String * Unit: - * Is mutable: No * Description: The directory that stores system log files. * Introduced in: - ##### `sys_log_enable_compress`[​](#sys_log_enable_compress "Direct link to sys_log_enable_compress") * Default: false * Type: boolean * Unit: - * Is mutable: No * Description: When this item is set to `true`, the system appends a ".gz" postfix to rotated system log filenames so Log4j will produce gzip-compressed rotated FE system logs (for example, fe.log.\*). This value is read during Log4j configuration generation (Log4jConfig.initLogging / generateActiveLog4jXmlConfig) and controls the `sys_file_postfix` property used in the RollingFile filePattern. Enabling this feature reduces disk usage for retained logs but increases CPU and I/O during rollovers and changes log filenames, so that tools or scripts that read logs must be able to handle .gz files. Note that audit logs use a separate configuration for compression, that is, `audit_log_enable_compress`. * Introduced in: v3.2.12 ##### `sys_log_format`[​](#sys_log_format "Direct link to sys_log_format") * Default: "plaintext" * Type: String * Unit: - * Is mutable: No * Description: Selects the Log4j layout used for FE logs. Valid values: `"plaintext"` (Default) and `"json"`. The values are case-insensitive. `"plaintext"` configures PatternLayout with human-readable timestamps, level, thread, class.method :line and stack traces for WARN/ERROR. `"json"` configures JsonTemplateLayout and emits structured JSON events (UTC timestamps, level, thread id/name, source file/method/line, message, exception stackTrace) suitable for log aggregators (ELK, Splunk). JSON output abides by `sys_log_json_max_string_length` and `sys_log_json_profile_max_string_length` for maximum string lengths. * Introduced in: v3.2.10 ##### `sys_log_json_max_string_length`[​](#sys_log_json_max_string_length "Direct link to sys_log_json_max_string_length") * Default: 1048576 * Type: Int * Unit: Bytes * Is mutable: No * Description: Sets the JsonTemplateLayout "maxStringLength" value used for the JSON-formatted system logs. When `sys_log_format` is set to `"json"`, string-valued fields (for example "message" and stringified exception stack traces) are truncated if their length exceeds this limit. The value is injected into the generated Log4j XML in `Log4jConfig.generateActiveLog4jXmlConfig()`, and is applied to default, warning, audit, dump and bigquery layouts. The profile layout uses a separate configuration (`sys_log_json_profile_max_string_length`). Lowering this value reduces log size but can truncate useful information. * Introduced in: 3.2.11 ##### `sys_log_json_profile_max_string_length`[​](#sys_log_json_profile_max_string_length "Direct link to sys_log_json_profile_max_string_length") * Default: 104857600 (100 MB) * Type: Int * Unit: Bytes * Is mutable: No * Description: Sets the maxStringLength of JsonTemplateLayout for profile (and related feature) log appenders when `sys_log_format` is "json". String field values in JSON-formatted profile logs will be truncated to this byte length; non-string fields are unaffected. This item is applied in Log4jConfig `JsonTemplateLayout maxStringLength` and is ignored when `plaintext` logging is used. Keep the value large enough for full messages you need, but note larger values increase log size and I/O. * Introduced in: v3.2.11 ##### `sys_log_level`[​](#sys_log_level "Direct link to sys_log_level") * Default: INFO * Type: String * Unit: - * Is mutable: No * Description: The severity levels into which system log entries are classified. Valid values: `INFO`, `WARN`, `ERROR`, and `FATAL`. * Introduced in: - ##### `sys_log_roll_interval`[​](#sys_log_roll_interval "Direct link to sys_log_roll_interval") * Default: DAY * Type: String * Unit: - * Is mutable: No * Description: The time interval at which StarRocks rotates system log entries. Valid values: `DAY` and `HOUR`. * If this parameter is set to `DAY`, a suffix in the `yyyyMMdd` format is added to the names of system log files. * If this parameter is set to `HOUR`, a suffix in the `yyyyMMddHH` format is added to the names of system log files. * Introduced in: - ##### `sys_log_roll_num`[​](#sys_log_roll_num "Direct link to sys_log_roll_num") * Default: 10 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of system log files that can be retained within each retention period specified by the `sys_log_roll_interval` parameter. * Introduced in: - ##### `sys_log_to_console`[​](#sys_log_to_console "Direct link to sys_log_to_console") * Default: false (unless the environment variable `SYS_LOG_TO_CONSOLE` is set to "1") * Type: Boolean * Unit: - * Is mutable: No * Description: When this item is set to `true`, the system configures Log4j to send all logs to the console (ConsoleErr appender) instead of the file-based appenders. This value is read when generating the active Log4j XML configuration (which affects the root logger and per-module logger appender selection). Its value is captured from the `SYS_LOG_TO_CONSOLE` environment variable at process startup. Changing it at runtime has no effect. This configuration is commonly used in containerized or CI environments where stdout/stderr log collection is preferred over writing log files. * Introduced in: v3.2.0 ##### `sys_log_verbose_modules`[​](#sys_log_verbose_modules "Direct link to sys_log_verbose_modules") * Default: Empty string * Type: String\[] * Unit: - * Is mutable: No * Description: The modules for which StarRocks generates system logs. If this parameter is set to `org.apache.starrocks.catalog`, StarRocks generates system logs only for the catalog module. Separate the module names with a comma (,) and a space. * Introduced in: - ##### `sys_log_warn_modules`[​](#sys_log_warn_modules "Direct link to sys_log_warn_modules") * Default: * Type: String\[] * Unit: - * Is mutable: No * Description: A list of logger names or package prefixes that the system will configure at startup as WARN-level loggers and route to the warning appender (SysWF) — the `fe.warn.log` file. Entries are inserted into the generated Log4j configuration (alongside builtin warn modules such as org.apache.kafka, org.apache.hudi, and org.apache.hadoop.io.compress) and produce logger elements like ``. Fully-qualified package and class prefixes (for example, "com.example.lib") are recommended to suppress noisy INFO/DEBUG output into the regular log and to allow warnings to be captured separately. * Introduced in: v3.2.13 #### Server[​](#server "Direct link to Server") ###### enable\_auth\_check[​](#enable_auth_check "Direct link to enable_auth_check") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Specifies whether to enable the authentication check feature. Valid values: `TRUE` and `FALSE`. `TRUE` specifies to enable this feature, and `FALSE` specifies to disable this feature. * Introduced in: - ##### `brpc_idle_wait_max_time`[​](#brpc_idle_wait_max_time "Direct link to brpc_idle_wait_max_time") * Default: 10000 * Type: Int * Unit: ms * Is mutable: No * Description: The maximum length of time for which bRPC clients wait as in the idle state. * Introduced in: - ##### `brpc_inner_reuse_pool`[​](#brpc_inner_reuse_pool "Direct link to brpc_inner_reuse_pool") * Default: true * Type: boolean * Unit: - * Is mutable: No * Description: Controls whether the underlying BRPC client uses an internal shared reuse pool for connections/channels. StarRocks reads `brpc_inner_reuse_pool` in BrpcProxy when constructing RpcClientOptions (via `rpcOptions.setInnerResuePool(...)`). When enabled (true) the RPC client reuses internal pools to reduce per-call connection creation, lowering connection churn, memory and file-descriptor usage for FE-to-BE / LakeService RPCs. When disabled (false) the client may create more isolated pools (increasing concurrency isolation at the cost of higher resource usage). Changing this value requires restarting the process to take effect. * Introduced in: v3.3.11, v3.4.1, v3.5.0 ##### `brpc_min_evictable_idle_time_ms`[​](#brpc_min_evictable_idle_time_ms "Direct link to brpc_min_evictable_idle_time_ms") * Default: 120000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: Time in milliseconds that an idle BRPC connection must remain in the connection pool before it becomes eligible for eviction. Applied to the RpcClientOptions used by `BrpcProxy` (via RpcClientOptions.setMinEvictableIdleTime). Raise this value to keep idle connections longer (reducing reconnect churn); lower it to free unused sockets faster (reducing resource usage). Tune together with `brpc_connection_pool_size` and `brpc_idle_wait_max_time` to balance connection reuse, pool growth, and eviction behavior. * Introduced in: v3.3.11, v3.4.1, v3.5.0 ##### `brpc_reuse_addr`[​](#brpc_reuse_addr "Direct link to brpc_reuse_addr") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: When true, StarRocks sets the socket option to allow local address reuse for client sockets created by the brpc RpcClient (via RpcClientOptions.setReuseAddress). Enabling this reduces bind failures and allows faster rebinding of local ports after sockets are closed, which is helpful for high-rate connection churn or rapid restarts. When false, address/port reuse is disabled, which can reduce the chance of unintended port sharing but may increase transient bind errors. This option interacts with connection behavior configured by `brpc_connection_pool_size` and `brpc_short_connection` because it affects how rapidly client sockets can be rebound and reused. * Introduced in: v3.3.11, v3.4.1, v3.5.0 ###### `brpc_connection_pool_retry_wait_time_ms`[​](#brpc_connection_pool_retry_wait_time_ms "Direct link to brpc_connection_pool_retry_wait_time_ms") * Default: 10 * Type: Int * Unit: ms * Is mutable: Yes * Description: The wait time before retrying when a bRPC connection pool exception occurs (e.g. SYN packet loss during TCP handshake). When `ChannelPool.getChannel()` throws a `NoSuchElementException` (directly or wrapped in a `RuntimeException`), the retry logic sleeps for this duration before attempting to reconnect. * Introduced in: - ##### `cluster_name`[​](#cluster_name "Direct link to cluster_name") * Default: StarRocks Cluster * Type: String * Unit: - * Is mutable: No * Description: The name of the StarRocks cluster to which the FE belongs. The cluster name is displayed for `Title` on the web page. * Introduced in: - ##### `dns_cache_ttl_seconds`[​](#dns_cache_ttl_seconds "Direct link to dns_cache_ttl_seconds") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: No * Description: DNS cache TTL (Time-To-Live) in seconds for successful DNS lookups. This sets the Java security property `networkaddress.cache.ttl` which controls how long the JVM caches successful DNS lookups. Set this item to `-1` to allow the system to always cache the infomration, or `0` to disable caching. This is particularly useful in environments where IP addresses change frequently, such as Kubernetes deployments or when dynamic DNS is used. * Introduced in: v3.5.11, v4.0.4 ##### `enable_http_async_handler`[​](#enable_http_async_handler "Direct link to enable_http_async_handler") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow the system to process HTTP requests asynchronously. If this feature is enabled, an HTTP request received by Netty worker threads will then be submitted to a separate thread pool for service logic handling to avoid blocking the HTTP server. If disabled, Netty workers will handle the service logic. * Introduced in: 4.0.0 ##### `enable_http_validate_headers`[​](#enable_http_validate_headers "Direct link to enable_http_validate_headers") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Controls whether Netty's HttpServerCodec performs strict HTTP header validation. The value is passed to HttpServerCodec when the HTTP pipeline is initialized in `HttpServer` (see UseLocations). Default is false for backward compatibility because newer netty versions enforce stricter header rules (). Set to true to enforce RFC-compliant header checks; doing so may cause malformed or nonconforming requests from legacy clients or proxies to be rejected. Change requires a restart of the HTTP server to take effect. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### `enable_https`[​](#enable_https "Direct link to enable_https") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to enable HTTPS server alongside HTTP server in FE nodes. * Introduced in: v4.0 ##### `frontend_address`[​](#frontend_address "Direct link to frontend_address") * Default: 0.0.0.0 * Type: String * Unit: - * Is mutable: No * Description: The IP address of the FE node. * Introduced in: - ##### `http_async_threads_num`[​](#http_async_threads_num "Direct link to http_async_threads_num") * Default: 4096 * Type: Int * Unit: - * Is mutable: Yes * Description: Size of the thread pool for asynchronous HTTP request processing. The alias is `max_http_sql_service_task_threads_num`. * Introduced in: 4.0.0 ##### `http_backlog_num`[​](#http_backlog_num "Direct link to http_backlog_num") * Default: 1024 * Type: Int * Unit: - * Is mutable: No * Description: The length of the backlog queue held by the HTTP server in the FE node. * Introduced in: - ##### `http_max_chunk_size`[​](#http_max_chunk_size "Direct link to http_max_chunk_size") * Default: 8192 * Type: Int * Unit: Bytes * Is mutable: No * Description: Sets the maximum allowed size (in bytes) of a single HTTP chunk handled by Netty's HttpServerCodec in the FE HTTP server. It is passed as the third argument to HttpServerCodec and limits the length of chunks during chunked transfer or streaming requests/responses. If an incoming chunk exceeds this value, Netty will raise a frame-too-large error (e.g., TooLongFrameException) and the request may be rejected. Increase this for legitimate large chunked uploads; keep it small to reduce memory pressure and surface area for DoS attacks. This setting is used alongside `http_max_initial_line_length`, `http_max_header_size`, and `enable_http_validate_headers`. * Introduced in: v3.2.0 ##### `http_max_header_size`[​](#http_max_header_size "Direct link to http_max_header_size") * Default: 32768 * Type: Int * Unit: Bytes * Is mutable: No * Description: Maximum allowed size in bytes for the HTTP request header block parsed by Netty's `HttpServerCodec`. StarRocks passes this value to `HttpServerCodec` (as `Config.http_max_header_size`); if an incoming request's headers (names and values combined) exceed this limit, the codec will reject the request (decoder exception) and the connection/request will fail. Increase only when clients legitimately send very large headers (large cookies or many custom headers); larger values increase per-connection memory use. Tune in conjunction with `http_max_initial_line_length` and `http_max_chunk_size`. Changes require FE restart. * Introduced in: v3.2.0 ##### `http_max_initial_line_length`[​](#http_max_initial_line_length "Direct link to http_max_initial_line_length") * Default: 4096 * Type: Int * Unit: Bytes * Is mutable: No * Description: Sets the maximum allowed length (in bytes) of the HTTP initial request line (method + request-target + HTTP version) accepted by the Netty `HttpServerCodec` used in HttpServer. The value is passed to Netty's decoder and requests with an initial line longer than this will be rejected (TooLongFrameException). Increase this only when you must support very long request URIs; larger values increase memory use and may raise exposure to malformed/request-abuse. Tune together with `http_max_header_size` and `http_max_chunk_size`. * Introduced in: v3.2.0 ##### `http_port`[​](#http_port "Direct link to http_port") * Default: 8030 * Type: Int * Unit: - * Is mutable: No * Description: The port on which the HTTP server in the FE node listens. * Introduced in: - ##### `enable_http_auth`[​](#enable_http_auth "Direct link to enable_http_auth") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Introduced in: v4.2.0 * Description: When true, most external FE HTTP endpoints require HTTP Basic Auth. Credentials are validated against the user store via `AuthenticationHandler.authenticate()`, so LDAP / security-integration login works on the HTTP path the same way it does for the MySQL protocol. The following are exempt: * Public probes / observability: `/api/bootstrap`, `/api/oauth2`. * Peer-FE / control-plane paths that are IP-whitelisted or token-gated inside the handler: `/image`, `/check`, `/journal_id`, `/info`, `/role`, `/static`, `/api/get_small_file`. `/dump`, `/dump_starmgr`, and `/service_id` are not exempt: regardless of this setting, they accept only an authenticated admin. The caller can authenticate with HTTP Basic Auth or with the session cookie obtained by logging in to the FE web console. A user is treated as an admin if any of the following holds: the user has the SYSTEM-level `NODE` privilege through a role that is active at login, for example the built-in `cluster_admin` role set as a default role of the user (when `activate_all_roles_on_login` is `true`, any granted role counts); the user is `root` or has the `root` role; the user has both the `db_admin` and `user_admin` roles. `/api/_meta_replay_state` is not exempt either: regardless of this setting, it requires HTTP Basic Auth of a user who is `root`, has the `root` role, or has both the `db_admin` and `user_admin` roles. The `NODE` privilege alone is not enough. Other privileged endpoints that this setting governs (this does not apply to `/dump`, `/dump_starmgr`, `/service_id`, and `/api/_meta_replay_state`) additionally require a SYSTEM-level RBAC privilege (`OPERATE` or `NODE`) that is **active** in the caller's session. If the granting role is not the user's default, run `SET DEFAULT ROLE TO ;` or set the global variable `activate_all_roles_on_login=true` so the roles activate at login. LDAP / security-integration group → role mappings activate automatically. ##### `http_web_page_display_hardware`[​](#http_web_page_display_hardware "Direct link to http_web_page_display_hardware") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When true, the HTTP index page (/index) will include a hardware information section populated via the oshi library (CPU, memory, processes, disks, filesystems, network, etc.). oshi may invoke system utilities or read system files indirectly (for example, it can execute commands such as `getent passwd`), which can surface sensitive system data. If you require stricter security or want to avoid executing those indirect commands on the host, set this configuration to false to disable collection and display of hardware details on the web UI. * Introduced in: v3.2.0 ##### `http_worker_threads_num`[​](#http_worker_threads_num "Direct link to http_worker_threads_num") * Default: 0 * Type: Int * Unit: - * Is mutable: No * Description: Number of worker threads for http server to deal with http requests. For a negative or 0 value, the number of threads will be twice the number of cpu cores. * Introduced in: v2.5.18, v3.0.10, v3.1.7, v3.2.2 ##### `https_port`[​](#https_port "Direct link to https_port") * Default: 8443 * Type: Int * Unit: - * Is mutable: No * Description: The port on which the HTTPS server in the FE node listens. * Introduced in: v4.0 ##### `max_mysql_service_task_threads_num`[​](#max_mysql_service_task_threads_num "Direct link to max_mysql_service_task_threads_num") * Default: 4096 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of threads that can be run by the MySQL server in the FE node to process tasks. * Introduced in: - ##### `max_task_runs_threads_num`[​](#max_task_runs_threads_num "Direct link to max_task_runs_threads_num") * Default: 512 * Type: Int * Unit: Threads * Is mutable: No * Description: Controls the maximum number of threads in the task-run executor thread pool. This value is the upper bound of concurrent task-run executions; increasing it raises parallelism but also increases CPU, memory, and network usage, while reducing it can cause task-run backlog and higher latency. Tune this value according to expected concurrent scheduled jobs and available system resources. * Introduced in: v3.2.0 ##### `memory_tracker_enable`[​](#memory_tracker_enable "Direct link to memory_tracker_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Enables the FE memory tracker subsystem. When `memory_tracker_enable` is set to `true`, `MemoryUsageTracker` periodically scans registered metadata modules, updates the in-memory `MemoryUsageTracker.MEMORY_USAGE` map, logs totals, and causes `MetricRepo` to expose memory usage and object-count gauges in metrics output. Use `memory_tracker_interval_seconds` to control the sampling interval. Enabling this feature helps monitoring and debugging memory consumption but introduces CPU and I/O overhead and additional metric cardinality. * Introduced in: v3.2.4 ##### `memory_tracker_interval_seconds`[​](#memory_tracker_interval_seconds "Direct link to memory_tracker_interval_seconds") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Interval in seconds for the FE `MemoryUsageTracker` daemon to poll and record memory usage of the FE process and registered `MemoryTrackable` modules. When `memory_tracker_enable` is set to `true`, the tracker runs on this cadence, updates `MEMORY_USAGE`, and logs aggregated JVM and tracked-module usage. * Introduced in: v3.2.4 ##### `mysql_nio_backlog_num`[​](#mysql_nio_backlog_num "Direct link to mysql_nio_backlog_num") * Default: 1024 * Type: Int * Unit: - * Is mutable: No * Description: The length of the backlog queue held by the MySQL server in the FE node. * Introduced in: - ##### `mysql_send_packet_timeout_ms`[​](#mysql_send_packet_timeout_ms "Direct link to mysql_send_packet_timeout_ms") * Default: 60000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: Per-packet write timeout for the MySQL protocol channel. Bounds how long the FE worker can wait for a slow client's TCP recv buffer to drain when sending result rows. Without this bound the worker can block in `Selector.select()` indefinitely and the query becomes unkillable via `KILL QUERY`. Set to `0` to disable (legacy unbounded wait). * Introduced in: v4.1 ##### `mysql_server_version`[​](#mysql_server_version "Direct link to mysql_server_version") * Default: 8.0.33 * Type: String * Unit: - * Is mutable: Yes * Description: The MySQL server version returned to the client. Modifying this parameter will affect the version information in the following situations: 1. `select version();` 2. Handshake packet version 3. Value of the global variable `version` (`show variables like 'version';`) * Introduced in: - ##### `mysql_service_io_threads_num`[​](#mysql_service_io_threads_num "Direct link to mysql_service_io_threads_num") * Default: 4 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of threads that can be run by the MySQL server in the FE node to process I/O events. * Introduced in: - ##### `mysql_service_kill_after_disconnect`[​](#mysql_service_kill_after_disconnect "Direct link to mysql_service_kill_after_disconnect") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Controls how the server handles the session when the MySQL TCP connection is detected closed (EOF on read). If it is set to `true`, the server immediately kills any running query for that connection and performs immediate cleanup. If it is `false`, the server does not kill running queries on disconnection and only performs cleanup when there are no pending request tasks, allowing long-running queries to continue after client disconnects. Note: despite a brief comment suggesting TCP keep‑alive, this parameter specifically governs post-disconnection killing behavior and should be set according to whether you want orphaned queries terminated (recommended behind unreliable/load‑balanced clients) or allowed to finish. * Introduced in: - ##### `mysql_service_nio_enable_keep_alive`[​](#mysql_service_nio_enable_keep_alive "Direct link to mysql_service_nio_enable_keep_alive") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Enable TCP Keep-Alive for MySQL connections. Useful for long-idled connections behind load balancers. * Introduced in: - ##### `net_use_ipv6_when_priority_networks_empty`[​](#net_use_ipv6_when_priority_networks_empty "Direct link to net_use_ipv6_when_priority_networks_empty") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: A boolean value to control whether to use IPv6 addresses preferentially when `priority_networks` is not specified. `true` indicates to allow the system to use an IPv6 address preferentially when the server that hosts the node has both IPv4 and IPv6 addresses and `priority_networks` is not specified. * Introduced in: v3.3.0 ##### `priority_networks`[​](#priority_networks "Direct link to priority_networks") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: Declares a selection strategy for servers that have multiple IP addresses. Note that at most one IP address must match the list specified by this parameter. The value of this parameter is a list that consists of entries, which are separated with semicolons (;) in CIDR notation, such as 10.10.10.0/24. If no IP address matches the entries in this list, an available IP address of the server will be randomly selected. From v3.3.0, StarRocks supports deployment based on IPv6. If the server has both IPv4 and IPv6 addresses, and this parameter is not specified, the system uses an IPv4 address by default. You can change this behavior by setting `net_use_ipv6_when_priority_networks_empty` to `true`. * Introduced in: - ##### `proc_profile_cpu_enable`[​](#proc_profile_cpu_enable "Direct link to proc_profile_cpu_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When this item is set to `true`, the background `ProcProfileCollector` will collect CPU profiles using `AsyncProfiler` and write HTML reports under `sys_log_dir/proc_profile`. Each collection run records CPU stacks for the duration configured by `proc_profile_collect_time_s` and uses `proc_profile_jstack_depth` for Java stack depth. Generated profiles are compressed and old files are pruned according to `proc_profile_file_retained_days` and `proc_profile_file_retained_size_bytes`. `AsyncProfiler` requires the native library (`libasyncProfiler.so`); `one.profiler.extractPath` is set to `STARROCKS_HOME_DIR/bin` to avoid noexec issues on `/tmp`. * Introduced in: v3.2.12 ##### `qe_max_connection`[​](#qe_max_connection "Direct link to qe_max_connection") * Default: 4096 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of connections that can be established by all users to the FE node. From v3.1.12 and v3.2.7 onwards, the default value has been changed from `1024` to `4096`. * Introduced in: - ##### `query_port`[​](#query_port "Direct link to query_port") * Default: 9030 * Type: Int * Unit: - * Is mutable: No * Description: The port on which the MySQL server in the FE node listens. * Introduced in: - ##### `rpc_port`[​](#rpc_port "Direct link to rpc_port") * Default: 9020 * Type: Int * Unit: - * Is mutable: No * Description: The port on which the Thrift server in the FE node listens. * Introduced in: - ##### `slow_lock_stack_trace_reserve_levels`[​](#slow_lock_stack_trace_reserve_levels "Direct link to slow_lock_stack_trace_reserve_levels") * Default: 15 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls how many stack-trace frames are captured and emitted when StarRocks dumps lock debug information for slow or held locks. This value is passed to `LogUtil.getStackTraceToJsonArray` by `QueryableReentrantReadWriteLock` when producing JSON for the exclusive lock owner, current thread, and oldest/shared readers. Increasing this value provides more context for diagnosing slow-lock or deadlock issues at the cost of larger JSON payloads and slightly higher CPU/memory for stack capture; decreasing it reduces overhead. Note: this cap applies only to the `QueryableReentrantReadWriteLock` stack-dump path; the `LockManager` slow-lock path captures full stack depth and is not bounded by this value. Reader entries can be filtered by `slow_lock_threshold_ms` when only logging slow locks. * Introduced in: v3.4.0, v3.5.0 ##### `ssl_cipher_blacklist`[​](#ssl_cipher_blacklist "Direct link to ssl_cipher_blacklist") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: Comma separated list, with regex support to blacklist ssl cipher suites by IANA names. If both whitelist and blacklist are set, blacklist takes precedence. * Introduced in: v4.0 ##### `ssl_cipher_whitelist`[​](#ssl_cipher_whitelist "Direct link to ssl_cipher_whitelist") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: Comma separated list, with regex support to whitelist ssl cipher suites by IANA names. If both whitelist and blacklist are set, blacklist takes precedence. * Introduced in: v4.0 ##### `task_runs_concurrency`[​](#task_runs_concurrency "Direct link to task_runs_concurrency") * Default: 4 * Type: Int * Unit: - * Is mutable: Yes * Description: Global limit of concurrently running TaskRun instances. `TaskRunScheduler` stops scheduling new runs when current running count is greater than or equal to `task_runs_concurrency`, so this value caps parallel TaskRun execution across the scheduler. It is also used by `MVPCTRefreshPartitioner` to compute per-TaskRun partition refresh granularity. Increasing the value raises parallelism and resource usage; decreasing it reduces concurrency and makes partition refreshes larger per run. Do not set to 0 or negative unless intentionally disabling scheduling: 0 (or negative) will effectively prevent new TaskRuns from being scheduled by `TaskRunScheduler`. * Introduced in: v3.2.0 ##### `task_runs_queue_length`[​](#task_runs_queue_length "Direct link to task_runs_queue_length") * Default: 500 * Type: Int * Unit: - * Is mutable: Yes * Description: Limits the maximum number of pending TaskRun items kept in the pending queue. `TaskRunManager` checks the current pending count and rejects new submissions when valid pending TaskRun count is greater than or equal to `task_runs_queue_length`. The same limit is rechecked before merged/accepted TaskRuns are added. Tune this value to balance memory and scheduling backlog: set higher for large bursty workloads to avoid rejects, or lower to bound memory and reduce pending backlog. * Introduced in: v3.2.0 ##### `thrift_backlog_num`[​](#thrift_backlog_num "Direct link to thrift_backlog_num") * Default: 1024 * Type: Int * Unit: - * Is mutable: No * Description: The length of the backlog queue held by the Thrift server in the FE node. * Introduced in: - ##### `thrift_client_timeout_ms`[​](#thrift_client_timeout_ms "Direct link to thrift_client_timeout_ms") * Default: 5000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: The length of time after which idle client connections time out. The FE Thrift server closes a client connection once it has been idle for this long. An FE therefore replaces its own pooled connections to other FEs once they have been idle for 80% of this value, instead of sending a request on a connection that the other FE may already have closed. Set the same value on all FEs. * Introduced in: - ##### `thrift_rpc_max_body_size`[​](#thrift_rpc_max_body_size "Direct link to thrift_rpc_max_body_size") * Default: -1 * Type: Int * Unit: Bytes * Is mutable: No * Description: Controls the maximum allowed Thrift RPC message body size (in bytes) used when constructing the server's Thrift protocol (passed to TBinaryProtocol.Factory in `ThriftServer`). A value of `-1` disables the limit (unbounded). Setting a positive value enforces an upper bound so that messages larger than this are rejected by the Thrift layer, which helps limit memory usage and mitigate oversized-request or DoS risks. Set this to a size large enough for expected payloads (large structs or batched data) to avoid rejecting legitimate requests. * Introduced in: v3.2.0 ##### `thrift_server_max_worker_threads`[​](#thrift_server_max_worker_threads "Direct link to thrift_server_max_worker_threads") * Default: 4096 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of worker threads that are supported by the Thrift server in the FE node. * Introduced in: - ##### `thrift_server_queue_size`[​](#thrift_server_queue_size "Direct link to thrift_server_queue_size") * Default: 4096 * Type: Int * Unit: - * Is mutable: No * Description: How many connections can wait for a worker thread while all `thrift_server_max_worker_threads` workers are busy. This bounds what the pending queue can hold, not what it will serve: a connection arriving once the queue is full is closed immediately instead of being retried, and a queued connection that waits longer than `thrift_server_queue_timeout_ms` is closed without being served once a worker reaches it. The two cases are counted separately, by `starrocks_fe_thrift_server_rejected_connections_total` and `starrocks_fe_thrift_server_expired_connections_total`. * Introduced in: - ##### `thrift_server_queue_timeout_ms`[​](#thrift_server_queue_timeout_ms "Direct link to thrift_server_queue_timeout_ms") * Default: 0 * Type: Long * Unit: ms * Is mutable: Yes * Description: The maximum time a connection may wait in the Thrift server pending queue before a worker thread picks it up. A connection that has waited longer is closed without being served, on the assumption that its caller has abandoned it and serving it only holds a worker away from work someone is still waiting for. When armed, this also bounds how long the queue takes to drain after a burst, however deep it grew. `0`, the default, disables the check: every queued connection is served no matter how long it waited. The check is off by default because that assumption does not hold for every caller, and a queued connection carries no deadline the server can read. The deadlines callers actually use span four orders of magnitude. Plain FE-internal RPCs give up after `thrift_rpc_timeout_ms` (10 seconds), but a statement forwarded from a follower to the leader waits the session's execution timeout plus `thrift_rpc_timeout_ms` -- about 310 seconds at the default `query_timeout`, and about four hours at the default `insert_timeout` -- and BE stream-load and transaction-commit RPCs wait `stream_load_thrift_rpc_timeout_ms` (60 seconds) or a quarter of the load's own `timeout_second`, whichever is larger. Because `query_timeout`, `insert_timeout` and `timeout_second` are all user-settable and unbounded, no fixed value is safe for every cluster: a timeout short enough to speed up recovery can also cut off a forwarded statement whose client is still waiting, during exactly the saturation it was meant to shorten. Arm it per cluster, at a value above the largest deadline that cluster's clients rely on -- derive it from that cluster's `query_timeout` and `insert_timeout`. Each drop is counted in `starrocks_fe_thrift_server_expired_connections_total` and reported by a rate-limited WARN naming the peer. `starrocks_fe_thrift_server_queue_wait_ms` reports queue waits whether or not the check is armed, so use it to size the value before arming it. * Introduced in: v4.2.0 #### Metadata and cluster management[​](#metadata-and-cluster-management "Direct link to Metadata and cluster management") ##### `alter_max_worker_queue_size`[​](#alter_max_worker_queue_size "Direct link to alter_max_worker_queue_size") * Default: 4096 * Type: Int * Unit: Tasks * Is mutable: No * Description: Controls the capacity of the internal worker thread pool queue used by the alter subsystem. It is passed to `ThreadPoolManager.newDaemonCacheThreadPool` in `AlterHandler` together with `alter_max_worker_threads`. When the number of pending alter tasks exceeds `alter_max_worker_queue_size`, new submissions will be rejected and a `RejectedExecutionException` can be thrown (see `AlterHandler.handleFinishAlterTask`). Tune this value to balance memory usage and the amount of backlog you permit for concurrent alter tasks. * Introduced in: v3.2.0 ##### `alter_max_worker_threads`[​](#alter_max_worker_threads "Direct link to alter_max_worker_threads") * Default: 4 * Type: Int * Unit: Threads * Is mutable: No * Description: Sets the maximum number of worker threads in the AlterHandler's thread pool. The AlterHandler constructs the executor with this value to run and finalize alter-related tasks (e.g., submitting `AlterReplicaTask` via handleFinishAlterTask). This value bounds concurrent execution of alter operations; raising it increases parallelism and resource usage, lowering it limits concurrent alters and may become a bottleneck. The executor is created together with `alter_max_worker_queue_size`, and the handler scheduling uses `alter_scheduler_interval_millisecond`. * Introduced in: v3.2.0 ##### `automated_cluster_snapshot_interval_seconds`[​](#automated_cluster_snapshot_interval_seconds "Direct link to automated_cluster_snapshot_interval_seconds") * Default: 600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The interval at which the Automated Cluster Snapshot tasks are triggered. * Introduced in: v3.4.2 ##### `background_refresh_metadata_interval_millis`[​](#background_refresh_metadata_interval_millis "Direct link to background_refresh_metadata_interval_millis") * Default: 600000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The interval between two consecutive Hive metadata cache refreshes. * Introduced in: v2.5.5 ##### `background_refresh_metadata_time_secs_since_last_access_secs`[​](#background_refresh_metadata_time_secs_since_last_access_secs "Direct link to background_refresh_metadata_time_secs_since_last_access_secs") * Default: 3600 \* 24 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The expiration time of a Hive metadata cache refresh task. For the Hive catalog that has been accessed, if it has not been accessed for more than the specified time, StarRocks stops refreshing its cached metadata. For the Hive catalog that has not been accessed, StarRocks will not refresh its cached metadata. * Introduced in: v2.5.5 ##### `bdbje_cleaner_threads`[​](#bdbje_cleaner_threads "Direct link to bdbje_cleaner_threads") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: Number of background cleaner threads for the Berkeley DB Java Edition (JE) environment used by StarRocks journal. This value is read during environment initialization in `BDBEnvironment.initConfigs` and applied to `EnvironmentConfig.CLEANER_THREADS` using `Config.bdbje_cleaner_threads`. It controls parallelism for JE log cleaning and space reclamation; increasing it can speed up cleaning at the cost of additional CPU and I/O interference with foreground operations. Changes take effect only when the BDB environment is (re)initialized, so a frontend restart is required to apply a new value. * Introduced in: v3.2.0 ##### `bdbje_heartbeat_timeout_second`[​](#bdbje_heartbeat_timeout_second "Direct link to bdbje_heartbeat_timeout_second") * Default: 30 * Type: Int * Unit: Seconds * Is mutable: No * Description: The amount of time after which the heartbeats among the leader, follower, and observer FEs in the StarRocks cluster time out. * Introduced in: - ##### `bdbje_lock_timeout_second`[​](#bdbje_lock_timeout_second "Direct link to bdbje_lock_timeout_second") * Default: 1 * Type: Int * Unit: Seconds * Is mutable: No * Description: The amount of time after which a lock in the BDB JE-based FE times out. * Introduced in: - ##### `bdbje_replay_cost_percent`[​](#bdbje_replay_cost_percent "Direct link to bdbje_replay_cost_percent") * Default: 150 * Type: Int * Unit: Percent * Is mutable: No * Description: Sets the relative cost (as a percentage) of replaying transactions from a BDB JE log versus obtaining the same data via a network restore. The value is supplied to the underlying JE replication parameter `REPLAY_COST_PERCENT` and is typically `>100` to indicate that replay is usually more expensive than a network restore. When deciding whether to retain cleaned log files for potential replay, the system compares replay cost multiplied by log size against the cost of a network restore; files will be removed if network restore is judged more efficient. A value of 0 disables retention based on this cost comparison. Log files required for replicas within `REP_STREAM_TIMEOUT` or for any active replication are always retained. * Introduced in: v3.2.0 ##### `bdbje_replica_ack_timeout_second`[​](#bdbje_replica_ack_timeout_second "Direct link to bdbje_replica_ack_timeout_second") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: No * Description: The maximum amount of time for which the leader FE can wait for ACK messages from a specified number of follower FEs when metadata is written from the leader FE to the follower FEs. Unit: second. If a large amount of metadata is being written, the follower FEs require a long time before they can return ACK messages to the leader FE, causing ACK timeout. In this situation, metadata writes fail, and the FE process exits. We recommend that you increase the value of this parameter to prevent this situation. * Introduced in: - ##### `bdbje_reserved_disk_size`[​](#bdbje_reserved_disk_size "Direct link to bdbje_reserved_disk_size") * Default: 512 \* 1024 \* 1024 (536870912) * Type: Long * Unit: Bytes * Is mutable: No * Description: Limits the number of bytes Berkeley DB JE will reserve as "unprotected" (deletable) log/data files. StarRocks passes this value to JE via `EnvironmentConfig.RESERVED_DISK` in BDBEnvironment; JE's built-in default is 0 (unlimited). The StarRocks default (512 MiB) prevents JE from reserving excessive disk space for unprotected files while allowing safe cleanup of obsolete files. Tune this value on disk-constrained systems: decreasing it lets JE free more files sooner, increasing it lets JE retain more reserved space. Changes require restarting the process to take effect. * Introduced in: v3.2.0 ##### `bdbje_reset_election_group`[​](#bdbje_reset_election_group "Direct link to bdbje_reset_election_group") * Default: false * Type: String * Unit: - * Is mutable: No * Description: Whether to reset the BDBJE replication group. If this parameter is set to `TRUE`, the FE will reset the BDBJE replication group (that is, remove the information of all electable FE nodes) and start as the leader FE. After the reset, this FE will be the only member in the cluster, and other FEs can rejoin this cluster by using `ALTER SYSTEM ADD/DROP FOLLOWER/OBSERVER 'xxx'`. Use this setting only when no leader FE can be elected because the data of most follower FEs have been damaged. `reset_election_group` is used to replace `metadata_failure_recovery`. * Introduced in: - ##### `black_host_connect_failures_within_time`[​](#black_host_connect_failures_within_time "Direct link to black_host_connect_failures_within_time") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The threshold of connection failures allowed for a blacklisted BE node. If a BE node is added to the BE Blacklist automatically, StarRocks will assess its connectivity and judge whether it can be removed from the BE Blacklist. Within `black_host_history_sec`, only if a blacklisted BE node has fewer connection failures than the threshold set in `black_host_connect_failures_within_time`, it can be removed from the BE Blacklist. * Introduced in: v3.3.0 ##### `black_host_history_sec`[​](#black_host_history_sec "Direct link to black_host_history_sec") * Default: 2 \* 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time duration for retaining historical connection failures of BE nodes in the BE Blacklist. If a BE node is added to the BE Blacklist automatically, StarRocks will assess its connectivity and judge whether it can be removed from the BE Blacklist. Within `black_host_history_sec`, only if a blacklisted BE node has fewer connection failures than the threshold set in `black_host_connect_failures_within_time`, it can be removed from the BE Blacklist. * Introduced in: v3.3.0 ##### `brpc_connection_pool_size`[​](#brpc_connection_pool_size "Direct link to brpc_connection_pool_size") * Default: 16 * Type: Int * Unit: Connections * Is mutable: No * Description: The maximum number of pooled BRPC connections per endpoint used by the FE's BrpcProxy. This value is applied to RpcClientOptions via `setMaxTotoal` and `setMaxIdleSize`, so it directly limits concurrent outgoing BRPC requests because each request must borrow a connection from the pool. In high concurrency scenarios increase this to avoid request queuing; increasing it raises socket and memory usage and may increase remote server load. When tuning, consider related settings such as `brpc_idle_wait_max_time`, `brpc_short_connection`, `brpc_inner_reuse_pool`, `brpc_reuse_addr`, and `brpc_min_evictable_idle_time_ms`. Changing this value is not hot-reloadable and requires a restart. * Introduced in: v3.2.0 ##### `brpc_short_connection`[​](#brpc_short_connection "Direct link to brpc_short_connection") * Default: false * Type: boolean * Unit: - * Is mutable: No * Description: Controls whether the underlying brpc RpcClient uses short-lived connections. When enabled (`true`), RpcClientOptions.setShortConnection is set and connections are closed after a request completes, reducing the number of long-lived sockets at the cost of higher connection setup overhead and increased latency. When disabled (`false`, the default) persistent connections and connection pooling are used. Enabling this option affects connection-pool behavior and should be considered together with `brpc_connection_pool_size`, `brpc_idle_wait_max_time`, `brpc_min_evictable_idle_time_ms`, `brpc_reuse_addr`, and `brpc_inner_reuse_pool`. Keep it disabled for typical high-throughput deployments; enable only to limit socket lifetime or when short connections are required by network policy. * Introduced in: v3.3.11, v3.4.1, v3.5.0 ##### `catalog_try_lock_timeout_ms`[​](#catalog_try_lock_timeout_ms "Direct link to catalog_try_lock_timeout_ms") * Default: 5000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The timeout duration to obtain the global lock. * Introduced in: - ##### `checkpoint_only_on_leader`[​](#checkpoint_only_on_leader "Direct link to checkpoint_only_on_leader") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When `true`, the CheckpointController will only select the leader FE as the checkpoint worker; when `false`, the controller may pick any frontend and prefers nodes with lower heap usage. With `false`, workers are sorted by recent failure time and `heapUsedPercent` (the leader is treated as having infinite usage to avoid selecting it). For operations that require cluster snapshot metadata, the controller already forces leader selection regardless of this flag. Enabling `true` centralizes checkpoint work on the leader (simpler but increases leader CPU/memory and network load); keeping it `false` distributes checkpoint load to less-loaded FEs. This setting affects worker selection and interaction with timeouts such as `checkpoint_timeout_seconds` and RPC settings like `thrift_rpc_timeout_ms`. * Introduced in: v3.4.0, v3.5.0 ##### `checkpoint_timeout_seconds`[​](#checkpoint_timeout_seconds "Direct link to checkpoint_timeout_seconds") * Default: 24 \* 3600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: Maximum time (in seconds) the leader's CheckpointController will wait for a checkpoint worker to complete a checkpoint. The controller converts this value to nanoseconds and polls the worker result queue; if no successful completion is received within this timeout the checkpoint is treated as failed and createImage returns failure. Increasing this value accommodates longer-running checkpoints but delays failure detection and subsequent image propagation; decreasing it causes faster failover/retries but can produce false timeouts for slow workers. This setting only controls the waiting period in `CheckpointController` during checkpoint creation and does not change the worker's internal checkpointing behavior. * Introduced in: v3.4.0, v3.5.0 ##### `db_used_data_quota_update_interval_secs`[​](#db_used_data_quota_update_interval_secs "Direct link to db_used_data_quota_update_interval_secs") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The interval at which the database used data quota is updated. StarRocks periodically updates the used data quota for all databases to track storage consumption. This value is used for quota enforcement and metrics collection. The minimum allowed interval is 30 seconds to prevent excessive system load. A value less than 30 will be rejected. * Introduced in: - ##### `drop_backend_after_decommission`[​](#drop_backend_after_decommission "Direct link to drop_backend_after_decommission") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to delete a BE after the BE is decommissioned. `TRUE` indicates that the BE is deleted immediately after it is decommissioned. `FALSE` indicates that the BE is not deleted after it is decommissioned. * Introduced in: - ##### `edit_log_port`[​](#edit_log_port "Direct link to edit_log_port") * Default: 9010 * Type: Int * Unit: - * Is mutable: No * Description: The port that is used for communication among the Leader, Follower, and Observer FEs in the cluster. * Introduced in: - ##### `edit_log_roll_num`[​](#edit_log_roll_num "Direct link to edit_log_roll_num") * Default: 50000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of metadata log entries that can be written before a log file is created for these log entries. This parameter is used to control the size of log files. The new log file is written to the BDBJE database. * Introduced in: - ##### `edit_log_type`[​](#edit_log_type "Direct link to edit_log_type") * Default: BDB * Type: String * Unit: - * Is mutable: No * Description: The type of edit log that can be generated. Set the value to `BDB`. * Introduced in: - ##### `enable_background_refresh_connector_metadata`[​](#enable_background_refresh_connector_metadata "Direct link to enable_background_refresh_connector_metadata") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the periodic Hive metadata cache refresh. After it is enabled, StarRocks polls the metastore (Hive Metastore or AWS Glue) of your Hive cluster, and refreshes the cached metadata of the frequently accessed Hive catalogs to perceive data changes. `true` indicates to enable the Hive metadata cache refresh, and `false` indicates to disable it. * Introduced in: v2.5.5 ##### `refresh_other_fe_dispatch_executor_thread_num`[​](#refresh_other_fe_dispatch_executor_thread_num "Direct link to refresh_other_fe_dispatch_executor_thread_num") * Default: 4 * Type: Integer * Unit: - * Is mutable: Yes * Description: The number of threads in the FE-global dispatch executor for asynchronous "refresh other FE" jobs. These threads only schedule background refresh tasks from connector write paths. They do not send peer FE refresh RPCs directly. Changes take effect on running FEs without restart. * Introduced in: - ##### `refresh_other_fe_rpc_executor_thread_num`[​](#refresh_other_fe_rpc_executor_thread_num "Direct link to refresh_other_fe_rpc_executor_thread_num") * Default: 4 * Type: Integer * Unit: - * Is mutable: Yes * Description: The number of threads in the FE-global RPC executor for "refresh other FE" fan-out. This executor bounds the number of concurrent refresh RPCs sent to peer FEs for both synchronous and asynchronous external table refresh flows. Changes take effect on running FEs without restart. * Introduced in: - ##### `enable_collect_query_detail_info`[​](#enable_collect_query_detail_info "Direct link to enable_collect_query_detail_info") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to collect the profile of a query. If this parameter is set to `TRUE`, the system collects the profile of the query. If this parameter is set to `FALSE`, the system does not collect the profile of the query. * Introduced in: - ##### `enable_create_partial_partition_in_batch`[​](#enable_create_partial_partition_in_batch "Direct link to enable_create_partial_partition_in_batch") * Default: false * Type: boolean * Unit: - * Is mutable: Yes * Description: When this item is set to `false` (default), StarRocks enforces that batch-created range partitions align to the standard time unit boundaries. It will reject non‑aligned ranges to avoid creating holes. Setting this item to `true` disables that alignment check and allows creating partial (non‑standard) partitions in batch, which can produce gaps or misaligned partition ranges. You should only set it to `true` when you intentionally need partial batch partitions and accept the associated risks. * Introduced in: v3.2.0 ##### `enable_internal_sql`[​](#enable_internal_sql "Direct link to enable_internal_sql") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: When this item is set to `true`, internal SQL statements executed by internal components (for example, SimpleExecutor) are preserved and written into internal audit or log messages (and can be further desensitized if `enable_sql_desensitize_in_log` is set). When it is set to `false`, internal SQL text is suppressed: formatting code (SimpleExecutor.formatSQL) returns "?" and the actual statement is not emitted to internal audit or log messages. This configuration does not change execution semantics of internal statements — it only controls logging and visibility of internal SQL for privacy or security. * Introduced in: - ##### `enable_legacy_compatibility_for_replication`[​](#enable_legacy_compatibility_for_replication "Direct link to enable_legacy_compatibility_for_replication") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the Legacy Compatibility for Replication. StarRocks may behave differently between the old and new versions, causing problems during cross-cluster data migration. Therefore, you must enable Legacy Compatibility for the target cluster before data migration and disable it after data migration is completed. `true` indicates enabling this mode. * Introduced in: v3.1.10, v3.2.6 ##### `enable_show_materialized_views_include_all_task_runs`[​](#enable_show_materialized_views_include_all_task_runs "Direct link to enable_show_materialized_views_include_all_task_runs") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls how TaskRuns are returned to the SHOW MATERIALIZED VIEWS command. When this item is set to `false`, StarRocks returns only the newest TaskRun per task (legacy behavior for compatibility). When it is set to `true` (default), `TaskManager` may include additional TaskRuns for the same task only when they share the same start TaskRun ID (for example, belong to the same job), preventing unrelated duplicate runs from appearing while allowing multiple statuses tied to one job to be shown. Set this item to `false` to restore single-run output or to surface multi-run job history for debugging and monitoring. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### `enable_statistics_collect_profile`[​](#enable_statistics_collect_profile "Direct link to enable_statistics_collect_profile") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to generate profiles for statistics queries. You can set this item to `true` to allow StarRocks to generate query profiles for queries on system statistics. * Introduced in: v3.1.5 ##### `enable_table_name_case_insensitive`[​](#enable_table_name_case_insensitive "Direct link to enable_table_name_case_insensitive") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to enable case-insensitive processing on catalog names, database names, table names, view names, and materialized view names. By default, this feature is disabled and these names are case-sensitive. When enabled, StarRocks stores these names in lowercase and forcibly converts every such name to lowercase during **both query and write (DDL/DML) processing**. This feature can be enabled only when creating a cluster. **We strongly recommend that you keep it disabled unless you have a specific, well-understood reason to enable it**, for the following reasons: * **It can make external tables and external catalogs unusable.** Different external catalog services follow different naming and case-sensitivity conventions. If an external schema, database, or table name is not already in lowercase, StarRocks lowercases the name in your SQL before passing it to the connector and then looks up a name that does not exist in the source, so the query fails with a "not found" error. * **It cannot be changed after the cluster is created.** After the cluster is started, the value cannot be modified by any means; any attempt to modify it results in an error, and FE fails to start if the value is inconsistent with the value used when the cluster was first started. * Only enable this feature on a new cluster where you are certain that all object names — including those in every external data source you plan to access — are already in lowercase. * Introduced in: v4.0 ##### `enable_task_history_archive`[​](#enable_task_history_archive "Direct link to enable_task_history_archive") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, finished task-run records are archived to the persistent task-run history table and recorded to the edit log so lookups (e.g., `lookupHistory`, `lookupHistoryByTaskNames`, `lookupLastJobOfTasks`) include archived results. Archiving is performed by the FE leader and is skipped during unit tests (`FeConstants.runningUnitTest`). When enabled, in-memory expiration and forced-GC paths are bypassed (the code returns early from `removeExpiredRuns` and `forceGC`), so retention/eviction is handled by the persistent archive instead of `task_runs_ttl_second` and `task_runs_max_history_number`. When disabled, history stays in memory and is pruned by those configurations. * Introduced in: v3.3.1, v3.4.0, v3.5.0 ##### `enable_task_run_fe_evaluation`[​](#enable_task_run_fe_evaluation "Direct link to enable_task_run_fe_evaluation") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, the FE will perform local evaluation for the system table `task_runs` in `TaskRunsSystemTable.supportFeEvaluation`. FE-side evaluation is only allowed for conjunctive equality predicates comparing a column to a constant and is limited to the columns `QUERY_ID` and `TASK_NAME`. Enabling this improves performance for targeted lookups by avoiding broader scans or additional remote processing; disabling it forces the planner to skip FE evaluation for `task_runs`, which may reduce predicate pruning and affect query latency for those filters. * Introduced in: v3.3.13, v3.4.3, v3.5.0 ##### `heartbeat_mgr_blocking_queue_size`[​](#heartbeat_mgr_blocking_queue_size "Direct link to heartbeat_mgr_blocking_queue_size") * Default: 1024 * Type: Int * Unit: - * Is mutable: No * Description: The size of the blocking queue that stores heartbeat tasks run by the Heartbeat Manager. * Introduced in: - ##### `heartbeat_mgr_threads_num`[​](#heartbeat_mgr_threads_num "Direct link to heartbeat_mgr_threads_num") * Default: 8 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads that can be run by the Heartbeat Manager to run heartbeat tasks. * Introduced in: - ##### `ignore_materialized_view_error`[​](#ignore_materialized_view_error "Direct link to ignore_materialized_view_error") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether FE ignores the metadata exception caused by materialized view errors. If FE fails to start due to the metadata exception caused by materialized view errors, you can set this parameter to `true` to allow FE to ignore the exception. * Introduced in: v2.5.10 ##### `ignore_meta_check`[​](#ignore_meta_check "Direct link to ignore_meta_check") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether non-Leader FEs ignore the metadata gap from the Leader FE. If the value is TRUE, non-Leader FEs ignore the metadata gap from the Leader FE and continue providing data reading services. This parameter ensures continuous data reading services even when you stop the Leader FE for a long period of time. If the value is FALSE, non-Leader FEs do not ignore the metadata gap from the Leader FE and stop providing data reading services. * Introduced in: - ##### `ignore_task_run_history_replay_error`[​](#ignore_task_run_history_replay_error "Direct link to ignore_task_run_history_replay_error") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When StarRocks deserializes TaskRun history rows for `information_schema.task_runs`, a corrupted or invalid JSON row will normally cause deserialization to log a warning and throw a RuntimeException. If this item is set to `true`, the system will catch deserialization errors, skip the malformed record, and continue processing remaining rows instead of failing the query. This will make `information_schema.task_runs` queries tolerant of bad entries in the `_statistics_.task_run_history` table. Note that enabling it will silently drop corrupted history records (potential data loss) instead of surfacing an explicit error. * Introduced in: v3.3.3, v3.4.0, v3.5.0 ##### `lock_checker_interval_second`[​](#lock_checker_interval_second "Direct link to lock_checker_interval_second") * Default: 30 * Type: long * Unit: Seconds * Is mutable: Yes * Description: Interval, in seconds, between executions of the LockChecker frontend daemon (named "deadlock-checker"). The daemon performs deadlock detection and slow-lock scanning; the configured value is multiplied by 1000 to set the timer in milliseconds. Decreasing this value reduces detection latency but increases scheduling and CPU overhead; increasing it reduces overhead but delays detection and slow-lock reporting. Changes take effect at runtime because the daemon resets its interval each run. This setting interacts with `lock_checker_enable_deadlock_check` (enables deadlock checks) and `slow_lock_threshold_ms` (defines what constitutes a slow lock). * Introduced in: v3.2.0 ##### `lock_manager_lock_table_num`[​](#lock_manager_lock_table_num "Direct link to lock_manager_lock_table_num") * Default: 256 * Type: Int * Unit: - * Is mutable: No * Description: The number of hash buckets, that is, lock tables, across which Lock Manager shards metadata locks. Each resource is mapped to one bucket by its resource ID, and each bucket is guarded by its own mutex, so a larger value reduces contention between lockers that work on unrelated resources, at the cost of slightly more memory. The value is read once when Lock Manager is created, so a change takes effect only after the FE restarts. The default value is changed from `32` to `256` from v4.1.0, v4.0.1, and v3.5.8 onwards. * Introduced in: v3.3.0 ##### `master_sync_policy`[​](#master_sync_policy "Direct link to master_sync_policy") * Default: SYNC * Type: String * Unit: - * Is mutable: No * Description: The policy based on which the leader FE flushes logs to disk. This parameter is valid only when the current FE is a leader FE. Valid values: * `SYNC`: When a transaction is committed, a log entry is generated and flushed to disk simultaneously. * `NO_SYNC`: The generation and flushing of a log entry do not occur at the same time when a transaction is committed. * `WRITE_NO_SYNC`: When a transaction is committed, a log entry is generated simultaneously but is not flushed to disk. If you have deployed only one follower FE, we recommend that you set this parameter to `SYNC`. If you have deployed three or more follower FEs, we recommend that you set this parameter and the `replica_sync_policy` both to `WRITE_NO_SYNC`. * Introduced in: - ##### `max_bdbje_clock_delta_ms`[​](#max_bdbje_clock_delta_ms "Direct link to max_bdbje_clock_delta_ms") * Default: 5000 * Type: Long * Unit: Milliseconds * Is mutable: No * Description: The maximum clock offset that is allowed between the leader FE and the follower or observer FEs in the StarRocks cluster. * Introduced in: - ##### `meta_delay_toleration_second`[​](#meta_delay_toleration_second "Direct link to meta_delay_toleration_second") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum duration by which the metadata on the follower and observer FEs can lag behind that on the leader FE. Unit: seconds. If this duration is exceeded, the non-leader FEs stops providing services. * Introduced in: - ##### `meta_freshness_check_interval_ms`[​](#meta_freshness_check_interval_ms "Direct link to meta_freshness_check_interval_ms") * Default: 1000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The interval at which a Follower or Observer FE re-evaluates whether its metadata is still fresh enough to serve reads, as defined by `meta_delay_toleration_second`. Values outside the range \[10, 5000] are clamped. This check runs on a dedicated thread rather than on the metadata replay thread, so that the node still stops serving stale metadata while a single journal entry is taking a long time to apply, for example when a replay operation is waiting for a database lock. * Introduced in: v4.2 ##### `metadata_replay_stuck_warn_threshold_second`[​](#metadata_replay_stuck_warn_threshold_second "Direct link to metadata_replay_stuck_warn_threshold_second") * Default: 30 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: A metadata journal entry that has been applying for longer than this duration is reported in **fe.log** as a stuck replay, together with the stack of the replay thread, so that you can identify what is blocking metadata replay. Set this item to `0` to disable the report. * Introduced in: v4.2 ##### `meta_dir`[​](#meta_dir "Direct link to meta_dir") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/meta" * Type: String * Unit: - * Is mutable: No * Description: The directory that stores metadata. * Introduced in: - ##### `metadata_ignore_unknown_operation_type`[​](#metadata_ignore_unknown_operation_type "Direct link to metadata_ignore_unknown_operation_type") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to ignore an unknown log ID. When an FE is rolled back, the FEs of the earlier version may be unable to recognize some log IDs. If the value is `TRUE`, the FE ignores unknown log IDs. If the value is `FALSE`, the FE exits. * Introduced in: - ##### `profile_info_format`[​](#profile_info_format "Direct link to profile_info_format") * Default: default * Type: String * Unit: - * Is mutable: Yes * Description: The format of the Profile output by the system. Valid values: `default` and `json`. When set to `default`, Profile is of the default format. When set to `json`, the system outputs Profile in JSON format. * Introduced in: v2.5 ##### `replica_ack_policy`[​](#replica_ack_policy "Direct link to replica_ack_policy") * Default: `SIMPLE_MAJORITY` * Type: String * Unit: - * Is mutable: No * Description: The policy based on which a log entry is considered valid. The default value `SIMPLE_MAJORITY` specifies that a log entry is considered valid if a majority of follower FEs return ACK messages. * Introduced in: - ##### `replica_sync_policy`[​](#replica_sync_policy "Direct link to replica_sync_policy") * Default: SYNC * Type: String * Unit: - * Is mutable: No * Description: The policy based on which the follower FE flushes logs to disk. This parameter is valid only when the current FE is a follower FE. Valid values: * `SYNC`: When a transaction is committed, a log entry is generated and flushed to disk simultaneously. * `NO_SYNC`: The generation and flushing of a log entry do not occur at the same time when a transaction is committed. * `WRITE_NO_SYNC`: When a transaction is committed, a log entry is generated simultaneously but is not flushed to disk. * Introduced in: - ##### `start_with_incomplete_meta`[​](#start_with_incomplete_meta "Direct link to start_with_incomplete_meta") * Default: false * Type: boolean * Unit: - * Is mutable: No * Description: When true, the FE will allow startup when image data exists but Berkeley DB JE (BDB) log files are missing or corrupted. `MetaHelper.checkMetaDir()` uses this flag to bypass the safety check that otherwise prevents starting from an image without corresponding BDB logs; starting this way can produce stale or inconsistent metadata and should only be used for emergency recovery. `RestoreClusterSnapshotMgr` temporarily sets this flag to true while restoring a cluster snapshot and then rolls it back; that component also toggles `bdbje_reset_election_group` during restore. Do not enable in normal operation — enable only when recovering from corrupted BDB data or when explicitly restoring an image-based snapshot. * Introduced in: v3.2.0 ##### `table_keeper_interval_second`[​](#table_keeper_interval_second "Direct link to table_keeper_interval_second") * Default: 30 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Interval, in seconds, between executions of the TableKeeper daemon. The TableKeeperDaemon uses this value (multiplied by 1000) to set its internal timer and periodically runs keeper tasks that ensure history tables exist, correct table properties (replication number) and update partition TTLs. The daemon only performs work on the leader node and updates its runtime interval via setInterval when `table_keeper_interval_second` changes. Increase to reduce scheduling frequency and load; decrease for faster reaction to missing or stale history tables. * Introduced in: v3.3.1, v3.4.0, v3.5.0 ##### `task_runs_ttl_second`[​](#task_runs_ttl_second "Direct link to task_runs_ttl_second") * Default: 7 \* 24 \* 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Controls the time-to-live (TTL) for task run history. Lowering this value shortens history retention and reduces memory/disk usage; raising it keeps histories longer but increases resource usage. Adjust together with `task_runs_max_history_number` and `enable_task_history_archive` for predictable retention and storage behavior. * Introduced in: v3.2.0 ##### `task_ttl_second`[​](#task_ttl_second "Direct link to task_ttl_second") * Default: 24 \* 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Time-to-live (TTL) for tasks. For manual tasks (when no schedule is set), TaskBuilder uses this value to compute the task's `expireTime` (`expireTime = now + task_ttl_second * 1000L`). TaskRun also uses this value as an upper bound when computing a run's execute timeout — the effective execute timeout is `min(task_runs_timeout_second, task_runs_ttl_second, task_ttl_second)`. Adjusting this value changes how long manually created tasks remain valid and can indirectly limit the maximum allowed execution time of task runs. * Introduced in: v3.2.0 ##### `thrift_rpc_retry_times`[​](#thrift_rpc_retry_times "Direct link to thrift_rpc_retry_times") * Default: 3 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls the total number of attempts a Thrift RPC call will make. This value is used by `ThriftRPCRequestExecutor` (and callers such as `NodeMgr` and `VariableMgr`) as the loop count for retries — i.e., a value of 3 allows up to three attempts including the initial try. On `TTransportException` the executor will try to reopen the connection and retry up to this count; it will not retry when the cause is a `SocketTimeoutException` or when reopen fails. Before each retry, the executor waits a random interval with exponential backoff (100 to 200 ms before the first retry, with both bounds doubling for each further retry until the wait reaches 500 to 1000 ms) and only then reopens the connection, so that callers that lost their connections at the same moment do not all reconnect at the same moment. Each attempt is subject to the per-attempt timeout configured by `thrift_rpc_timeout_ms`. Increasing this value improves resilience to transient connection failures but can increase overall RPC latency and resource usage. * Introduced in: v3.2.0 ##### `thrift_rpc_strict_mode`[​](#thrift_rpc_strict_mode "Direct link to thrift_rpc_strict_mode") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Controls the TBinaryProtocol "strict read" mode used by the Thrift server. This value is passed as the first argument to org.apache.thrift.protocol.TBinaryProtocol.Factory in the Thrift server stack and affects how incoming Thrift messages are parsed and validated. When `true` (default), the server enforces strict Thrift encoding/version checks and honors the configured `thrift_rpc_max_body_size` limit; when `false`, the server accepts non-strict (legacy/lenient) message formats, which can improve compatibility with older clients but may bypass some protocol validations. Use caution changing this on a running cluster because it is not mutable and affects interoperability and parsing safety. * Introduced in: v3.2.0 ##### `thrift_rpc_timeout_ms`[​](#thrift_rpc_timeout_ms "Direct link to thrift_rpc_timeout_ms") * Default: 10000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Timeout (in milliseconds) used as the default network/socket timeout for Thrift RPC calls. It is passed to TSocket when creating Thrift clients in `ThriftConnectionPool` (used by the frontend and backend pools) and is also added to an operation's execution timeout (e.g., ExecTimeout\*1000 + `thrift_rpc_timeout_ms`) when computing RPC call timeouts in places such as `ConfigBase`, `LeaderOpExecutor`, `GlobalStateMgr`, `NodeMgr`, `VariableMgr`, and `CheckpointWorker`. Increasing this value makes RPC calls tolerate longer network or remote processing delays; decreasing it causes faster failover on slow networks. Changing this value affects connection creation and request deadlines across the FE code paths that perform Thrift RPCs. * Introduced in: v3.2.0 ##### `txn_latency_metric_report_groups`[​](#txn_latency_metric_report_groups "Direct link to txn_latency_metric_report_groups") * Default: An empty string * Type: String * Unit: - * Is mutable: Yes * Description: A comma-separated list of transaction latency metric groups to report. Load types are categorized into logical groups for monitoring. When a group is enabled, its name is added as a 'type' label to transaction metrics. Valid values: `stream_load`, `routine_load`, `broker_load`, `insert`, and `compaction` (available only for shared-data clusters). Example: `"stream_load,routine_load"`. * Introduced in: v4.0 ##### `txn_rollback_limit`[​](#txn_rollback_limit "Direct link to txn_rollback_limit") * Default: 100 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of transactions that can be rolled back. * Introduced in: - --- ### FE Configuration - Shared-data, Data Lake, and Others FE parameters are classified into dynamic parameters and static parameters. * Dynamic parameters can be configured and adjusted by running SQL commands, which is very convenient. But the configurations become invalid if you restart your FE. Therefore, we recommend that you also modify the configuration items in the **fe.conf** file to prevent the loss of modifications. * Static parameters can only be configured and adjusted in the FE configuration file **fe.conf**. **After you modify this file, you must restart your FE for the changes to take effect.** Whether a parameter is a dynamic parameter is indicated by the `IsMutable` column in the output of [ADMIN SHOW CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). `TRUE` indicates a dynamic parameter. Note that both dynamic and static FE parameters can be configured in the **fe.conf** file. #### View FE configuration items[​](#view-fe-configuration-items "Direct link to View FE configuration items") After your FE is started, you can run the ADMIN SHOW FRONTEND CONFIG command on your MySQL client to check the parameter configurations. If you want to query the configuration of a specific parameter, run the following command: ```sql ADMIN SHOW FRONTEND CONFIG [LIKE "pattern"]; ``` For detailed description of the returned fields, see [`ADMIN SHOW CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). note You must have administrator privileges to run cluster administration-related commands. #### Configure FE parameters[​](#configure-fe-parameters "Direct link to Configure FE parameters") ##### Configure FE dynamic parameters[​](#configure-fe-dynamic-parameters "Direct link to Configure FE dynamic parameters") You can configure or modify the settings of FE dynamic parameters using [`ADMIN SET FRONTEND CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md). ```sql ADMIN SET FRONTEND CONFIG ("key" = "value"); ``` note The configuration changes made with `ADMIN SET FRONTEND` will be restored to the default values in the `fe.conf` file after the FE restarts. Therefore, we recommend that you also modify the configuration items in `fe.conf` if you want the changes to be permanent. ##### Configure FE static parameters[​](#configure-fe-static-parameters "Direct link to Configure FE static parameters") note Static parameters of an FE are set by changing them in the configuration file **fe.conf** and restarting the FE to allow the changes to take effect. *** This topic introduces the following types of FE configurations: * [Shared-data](#shared-data) * [Data Lake](#data-lake) * [Other](#other) #### Shared-data[​](#shared-data "Direct link to Shared-data") ##### `aws_s3_access_key`[​](#aws_s3_access_key "Direct link to aws_s3_access_key") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Access Key ID used to access your S3 bucket. * Introduced in: v3.0 ##### `aws_s3_endpoint`[​](#aws_s3_endpoint "Direct link to aws_s3_endpoint") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The endpoint used to access your S3 bucket, for example, `https://s3.us-west-2.amazonaws.com`. * Introduced in: v3.0 ##### `aws_s3_external_id`[​](#aws_s3_external_id "Direct link to aws_s3_external_id") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The external ID of the AWS account that is used for cross-account access to your S3 bucket. * Introduced in: v3.0 ##### `aws_s3_iam_role_arn`[​](#aws_s3_iam_role_arn "Direct link to aws_s3_iam_role_arn") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The ARN of the IAM role that has privileges on your S3 bucket in which your data files are stored. * Introduced in: v3.0 ##### `aws_s3_path`[​](#aws_s3_path "Direct link to aws_s3_path") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The S3 path used to store data. It consists of the name of your S3 bucket and the sub-path (if any) under it, for example, `testbucket/subpath`. * Introduced in: v3.0 ##### `aws_s3_region`[​](#aws_s3_region "Direct link to aws_s3_region") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The region in which your S3 bucket resides, for example, `us-west-2`. * Introduced in: v3.0 ##### `aws_s3_secret_key`[​](#aws_s3_secret_key "Direct link to aws_s3_secret_key") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Secret Access Key used to access your S3 bucket. * Introduced in: v3.0 ##### `aws_s3_use_aws_sdk_default_behavior`[​](#aws_s3_use_aws_sdk_default_behavior "Direct link to aws_s3_use_aws_sdk_default_behavior") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to use the default authentication credential of AWS SDK. Valid values: true and false (Default). * Introduced in: v3.0 ##### `aws_s3_use_instance_profile`[​](#aws_s3_use_instance_profile "Direct link to aws_s3_use_instance_profile") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to use Instance Profile and Assumed Role as credential methods for accessing S3. Valid values: true and false (Default). * If you use IAM user-based credential (Access Key and Secret Key) to access S3, you must specify this item as `false`, and specify `aws_s3_access_key` and `aws_s3_secret_key`. * If you use Instance Profile to access S3, you must specify this item as `true`. * If you use Assumed Role to access S3, you must specify this item as `true`, and specify `aws_s3_iam_role_arn`. * And if you use an external AWS account, you must also specify `aws_s3_external_id`. * Introduced in: v3.0 ##### `azure_adls2_endpoint`[​](#azure_adls2_endpoint "Direct link to azure_adls2_endpoint") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The endpoint of your Azure Data Lake Storage Gen2 Account, for example, `https://test.dfs.core.windows.net`. * Introduced in: v3.4.1 ##### `azure_adls2_oauth2_client_endpoint`[​](#azure_adls2_oauth2_client_endpoint "Direct link to azure_adls2_oauth2_client_endpoint") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The OAuth 2.0 token endpoint of the Managed Identity used to authorize requests for your Azure Data Lake Storage Gen2. Before v3.5.19, v4.0.12, and v4.1.2, this item was named `azure_adls2_oauth2_oauth2_client_endpoint`. The former name is still accepted as an alias. * Introduced in: v3.4.4 ##### `azure_adls2_oauth2_client_id`[​](#azure_adls2_oauth2_client_id "Direct link to azure_adls2_oauth2_client_id") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Client ID of the Managed Identity used to authorize requests for your Azure Data Lake Storage Gen2. * Introduced in: v3.4.4 ##### `azure_adls2_oauth2_tenant_id`[​](#azure_adls2_oauth2_tenant_id "Direct link to azure_adls2_oauth2_tenant_id") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Tenant ID of the Managed Identity used to authorize requests for your Azure Data Lake Storage Gen2. * Introduced in: v3.4.4 ##### `azure_adls2_oauth2_token_file`[​](#azure_adls2_oauth2_token_file "Direct link to azure_adls2_oauth2_token_file") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: Path to the federated token file used for Workload Identity authentication when creating the built-in ADLS2 storage volume. Set it together with `azure_adls2_oauth2_tenant_id` and `azure_adls2_oauth2_client_id`, and keep `azure_adls2_oauth2_use_managed_identity` as `false`. Mount the file at the same readable path on every FE and CN. An empty string disables token-file authentication. Restart the FE after changing this configuration. ##### `azure_adls2_oauth2_use_managed_identity`[​](#azure_adls2_oauth2_use_managed_identity "Direct link to azure_adls2_oauth2_use_managed_identity") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to use Managed Identity to authorize requests for your Azure Data Lake Storage Gen2. * Introduced in: v3.4.4 ##### `azure_adls2_path`[​](#azure_adls2_path "Direct link to azure_adls2_path") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Azure Data Lake Storage Gen2 path used to store data. It consists of the file system name and the directory name, for example, `testfilesystem/starrocks`. * Introduced in: v3.4.1 ##### `azure_adls2_sas_token`[​](#azure_adls2_sas_token "Direct link to azure_adls2_sas_token") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The shared access signatures (SAS) used to authorize requests for your Azure Data Lake Storage Gen2. * Introduced in: v3.4.1 ##### `azure_adls2_shared_key`[​](#azure_adls2_shared_key "Direct link to azure_adls2_shared_key") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Shared Key used to authorize requests for your Azure Data Lake Storage Gen2. * Introduced in: v3.4.1 ##### `azure_blob_endpoint`[​](#azure_blob_endpoint "Direct link to azure_blob_endpoint") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The endpoint of your Azure Blob Storage Account, for example, `https://test.blob.core.windows.net`. * Introduced in: v3.1 ##### `azure_blob_path`[​](#azure_blob_path "Direct link to azure_blob_path") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Azure Blob Storage path used to store data. It consists of the name of the container within your storage account and the sub-path (if any) under the container, for example, `testcontainer/subpath`. * Introduced in: v3.1 ##### `azure_blob_sas_token`[​](#azure_blob_sas_token "Direct link to azure_blob_sas_token") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The shared access signatures (SAS) used to authorize requests for your Azure Blob Storage. * Introduced in: v3.1 ##### `azure_blob_shared_key`[​](#azure_blob_shared_key "Direct link to azure_blob_shared_key") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Shared Key used to authorize requests for your Azure Blob Storage. * Introduced in: v3.1 ##### `azure_use_native_sdk`[​](#azure_use_native_sdk "Direct link to azure_use_native_sdk") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to use the native SDK to access Azure Blob Storage, thus allowing authentication with Managed Identities and Service Principals. If this item is set to `false`, only authentication with Shared Key and SAS Token is allowed. * Introduced in: v3.4.4 ##### `s3_use_native_sdk_for_glob`[​](#s3_use_native_sdk_for_glob "Direct link to s3_use_native_sdk_for_glob") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to use the native AWS S3 SDK to resolve glob paths in the FILES() table function for S3 and S3-compatible object stores (`s3`, `s3a`, `s3n`, `oss`, `cosn`, `ks3`, `obs`, and `tos`). When this item is set to `true`, the longest literal prefix of a wildcard is pushed down to S3 `ListObjectsV2` instead of listing the whole parent prefix through Hadoop `globStatus`, which is much faster when the prefix holds many objects but few of them match. Set this item to `false` to fall back to the Hadoop `globStatus` path. * Introduced in: v4.1.4 ##### `cloud_native_hdfs_url`[​](#cloud_native_hdfs_url "Direct link to cloud_native_hdfs_url") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The URL of the HDFS storage, for example, `hdfs://127.0.0.1:9000/user/xxx/starrocks/`. * Introduced in: - ##### `cloud_native_meta_port`[​](#cloud_native_meta_port "Direct link to cloud_native_meta_port") * Default: 6090 * Type: Int * Unit: - * Is mutable: No * Description: FE cloud-native metadata server RPC listen port. * Introduced in: - ##### `cloud_native_storage_type`[​](#cloud_native_storage_type "Direct link to cloud_native_storage_type") * Default: S3 * Type: String * Unit: - * Is mutable: No * Description: The type of object storage you use. In shared-data mode, StarRocks supports storing data in HDFS, Azure Blob (supported from v3.1.1 onwards), Azure Data Lake Storage Gen2 (supported from v3.4.1 onwards), Google Storage (with native SDK, supported from v3.5.1 onwards), and object storage systems that are compatible with the S3 protocol (such as AWS S3, and MinIO). Valid value: `S3` (Default), `HDFS`, `AZBLOB`, `ADLS2`, and `GS`. If you specify this parameter as `S3`, you must add the parameters prefixed by `aws_s3`. If you specify this parameter as `AZBLOB`, you must add the parameters prefixed by `azure_blob`. If you specify this parameter as `ADLS2`, you must add the parameters prefixed by `azure_adls2`. If you specify this parameter as `GS`, you must add the parameters prefixed by `gcp_gcs`. If you specify this parameter as `HDFS`, you only need to specify `cloud_native_hdfs_url`. * Introduced in: - ##### `enable_admin_skip_committed_txn`[​](#enable_admin_skip_committed_txn "Direct link to enable_admin_skip_committed_txn") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the `ADMIN SKIP COMMITTED TRANSACTION` statement. When `false`, the statement is rejected with an error. This is an operator-only escape hatch for unblocking a publish-stuck `COMMITTED` transaction on a shared-data (lake) table; the stuck transaction's data contribution is discarded, while the partition's visible version still advances via a "no-op publish" (a new metadata file is written that carries no data changes from this transaction). Only supports lake tables with `file_bundling=true`; load and lake-compaction transaction types only (alter / schema-change are not yet supported). Enable only when an operator needs to manually unblock a stuck transaction, and disable again afterwards to prevent accidental use. * Introduced in: - ##### `enable_load_volume_from_conf`[​](#enable_load_volume_from_conf "Direct link to enable_load_volume_from_conf") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to allow StarRocks to create the built-in storage volume by using the object storage-related properties specified in the FE configuration file. The default value is changed from `true` to `false` from v3.4.1 onwards. * Introduced in: v3.1.0 ##### `gcp_gcs_impersonation_service_account`[​](#gcp_gcs_impersonation_service_account "Direct link to gcp_gcs_impersonation_service_account") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Service Account that you want to impersonate if you use the impersonation-based authentication to access Google Storage. * Introduced in: v3.5.1 ##### `gcp_gcs_path`[​](#gcp_gcs_path "Direct link to gcp_gcs_path") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Google Cloud path used to store data. It consists of the name of your Google Cloud bucket and the sub-path (if any) under it, for example, `testbucket/subpath`. * Introduced in: v3.5.1 ##### `gcp_gcs_service_account_email`[​](#gcp_gcs_service_account_email "Direct link to gcp_gcs_service_account_email") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The email address in the JSON file generated at the creation of the Service Account, for example, `user@hello.iam.gserviceaccount.com`. * Introduced in: v3.5.1 ##### `gcp_gcs_service_account_private_key`[​](#gcp_gcs_service_account_private_key "Direct link to gcp_gcs_service_account_private_key") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Private Key in the JSON file generated at the creation of the Service Account, for example, `-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n`. * Introduced in: v3.5.1 ##### `gcp_gcs_service_account_private_key_id`[​](#gcp_gcs_service_account_private_key_id "Direct link to gcp_gcs_service_account_private_key_id") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Private Key ID in the JSON file generated at the creation of the Service Account. * Introduced in: v3.5.1 ##### `gcp_gcs_use_compute_engine_service_account`[​](#gcp_gcs_use_compute_engine_service_account "Direct link to gcp_gcs_use_compute_engine_service_account") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to use the Service Account that is bound to your Compute Engine. * Introduced in: v3.5.1 ##### `group_provider`[​](#group_provider "Direct link to group_provider") * Default: Empty * Type: String\[] * Unit: - * Is mutable: Yes * Description: The cluster-wide default list of Group Providers, separated by commas. It applies to natively authenticated users, and from v4.2 also to a security integration that sets no `group_provider` property of its own - earlier versions consulted no Group Provider at all in that case. See [Authenticate User Groups](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). * Introduced in: v3.5 ##### `group_provider_http_connect_timeout_ms`[​](#group_provider_http_connect_timeout_ms "Direct link to group_provider_http_connect_timeout_ms") * Default: 5000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Connect timeout for a File Group Provider whose `group_file_url` is an `http://` or `https://` URL. Must be positive. If the server does not accept the connection within this time, the CREATE or ALTER GROUP PROVIDER statement fails and the group provider keeps its previous configuration. Without a bound, a server that never answers would block not only the statement but also journal replay on every FE. * Introduced in: v4.2 ##### `group_provider_http_read_timeout_ms`[​](#group_provider_http_read_timeout_ms "Direct link to group_provider_http_read_timeout_ms") * Default: 30000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Read timeout for a File Group Provider whose `group_file_url` is an `http://` or `https://` URL. Must be positive. Bounds how long a connected server may stay silent before the fetch fails; see `group_provider_http_connect_timeout_ms` for the failure behavior. * Introduced in: v4.2 ##### `hdfs_file_system_expire_seconds`[​](#hdfs_file_system_expire_seconds "Direct link to hdfs_file_system_expire_seconds") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Time-to-live in seconds for an unused cached HDFS/ObjectStore FileSystem managed by HdfsFsManager. The FileSystemExpirationChecker (runs every 60s) calls each HdfsFs.isExpired(...) using this value; when expired the manager closes the underlying FileSystem and removes it from the cache. Accessor methods (for example `HdfsFs.getDFSFileSystem`, `getUserName`, `getConfiguration`) update the last-access timestamp, so expiry is based on inactivity. Lower values reduce idle resource holding but increase reopen overhead; higher values keep handles longer and may consume more resources. * Introduced in: v3.2.0 ##### `lake_autovacuum_grace_period_minutes`[​](#lake_autovacuum_grace_period_minutes "Direct link to lake_autovacuum_grace_period_minutes") * Default: 30 * Type: Long * Unit: Minutes * Is mutable: Yes * Description: The time range for retaining historical data versions in a shared-data cluster. Historical data versions within this time range are not automatically cleaned via AutoVacuum after Compactions. You need to set this value greater than the maximum query time to avoid that the data accessed by running queries get deleted before the queries finish. The default value has been changed from `5` to `30` since v3.3.0, v3.2.5, and v3.1.10. * Introduced in: v3.1.0 ##### `lake_autovacuum_parallel_partitions`[​](#lake_autovacuum_parallel_partitions "Direct link to lake_autovacuum_parallel_partitions") * Default: 8 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of partitions that can undergo AutoVacuum simultaneously in a shared-data cluster. AutoVacuum is the Garbage Collection after Compactions. * Introduced in: v3.1.0 ##### `lake_autovacuum_partition_naptime_seconds`[​](#lake_autovacuum_partition_naptime_seconds "Direct link to lake_autovacuum_partition_naptime_seconds") * Default: 180 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The minimum interval between AutoVacuum operations on the same partition in a shared-data cluster. * Introduced in: v3.1.0 ##### `lake_autovacuum_stale_partition_threshold`[​](#lake_autovacuum_stale_partition_threshold "Direct link to lake_autovacuum_stale_partition_threshold") * Default: 12 * Type: Long * Unit: Hours * Is mutable: Yes * Description: If a partition has no updates (loading, DELETE, or Compactions) within this time range, the system will not perform AutoVacuum on this partition. * Introduced in: v3.1.0 ##### `lake_compaction_allow_partial_success`[​](#lake_compaction_allow_partial_success "Direct link to lake_compaction_allow_partial_success") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: If this item is set to `true`, the system will consider the Compaction operation in a shared-data cluster as successful when one of the sub-tasks succeeds. * Introduced in: v3.5.2 ##### `lake_compaction_disable_ids`[​](#lake_compaction_disable_ids "Direct link to lake_compaction_disable_ids") * Default: "" * Type: String * Unit: - * Is mutable: Yes * Description: The table or partition list of which compaction is disabled in shared-data mode. The format is `tableId1;partitionId2`, seperated by semicolon, for example, `12345;98765`. * Introduced in: v3.4.4 ##### `lake_compaction_history_size`[​](#lake_compaction_history_size "Direct link to lake_compaction_history_size") * Default: 20 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of recent successful Compaction task records to keep in the memory of the Leader FE node in a shared-data cluster. You can view recent successful Compaction task records using the `SHOW PROC '/compactions'` command. Note that the Compaction history is stored in the FE process memory, and it will be lost if the FE process is restarted. * Introduced in: v3.1.0 ##### `lake_compaction_max_parallel_default`[​](#lake_compaction_max_parallel_default "Direct link to lake_compaction_max_parallel_default") * Default: 3 * Type: Int * Unit: - * Is mutable: Yes * Description: Default max parallel compaction subtasks per tablet when `lake_compaction_max_parallel` is not specified in table properties. `0` means disable parallel compaction. This config is used as the default value for the table property `lake_compaction_max_parallel`. ##### `lake_compaction_max_tasks`[​](#lake_compaction_max_tasks "Direct link to lake_compaction_max_tasks") * Default: -1 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of concurrent Compaction tasks allowed in a shared-data cluster. Setting this item to `-1` indicates to calculate the concurrent task number in an adaptive manner. Setting this value to `0` will disable compaction. * Introduced in: v3.1.0 ##### `lake_compaction_score_selector_min_score`[​](#lake_compaction_score_selector_min_score "Direct link to lake_compaction_score_selector_min_score") * Default: 10.0 * Type: Double * Unit: - * Is mutable: Yes * Description: The Compaction Score threshold that triggers Compaction operations in a shared-data cluster. When the Compaction Score of a partition is greater than or equal to this value, the system performs Compaction on that partition. * Introduced in: v3.1.0 ##### `lake_compaction_score_upper_bound`[​](#lake_compaction_score_upper_bound "Direct link to lake_compaction_score_upper_bound") * Default: 2000 * Type: Long * Unit: - * Is mutable: Yes * Description: The upper limit of the Compaction Score for a partition in a shared-data cluster. `0` indicates no upper limit. This item only takes effect when `lake_enable_ingest_slowdown` is set to `true`. When the Compaction Score of a partition reaches or exceeds this upper limit, incoming loading tasks will be rejected. From v3.3.6 onwards, the default value is changed from `0` to `2000`. * Introduced in: v3.2.0 ##### `lake_compaction_interval_ms_on_success`[​](#lake_compaction_interval_ms_on_success "Direct link to lake_compaction_interval_ms_on_success") * Default: 10000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The interval before triggering the next Compaction for a partition in a shared-data cluster after a successful Compaction on that partition. The alias is `lake_min_compaction_interval_ms_on_success`. * Introduced in: v3.2.0 ##### `lake_enable_balance_tablets_between_workers`[​](#lake_enable_balance_tablets_between_workers "Direct link to lake_enable_balance_tablets_between_workers") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to balance the number of tablets among Compute Nodes during the tablet migration of cloud-native tables in a shared-data cluster. `true` indicates to balance the tablets among Compute Nodes, and `false` indicates to disabling this feature. * Introduced in: v3.3.4 ##### `lake_enable_ingest_slowdown`[​](#lake_enable_ingest_slowdown "Direct link to lake_enable_ingest_slowdown") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Data Ingestion Slowdown in a shared-data cluster. When Data Ingestion Slowdown is enabled, if the Compaction Score of a partition exceeds `lake_ingest_slowdown_threshold`, loading tasks on that partition will be throttled down. This configuration only takes effect when `run_mode` is set to `shared_data`. From v3.3.6 onwards, the default value is chenged from `false` to `true`. * Introduced in: v3.2.0 ##### `lake_ingest_slowdown_threshold`[​](#lake_ingest_slowdown_threshold "Direct link to lake_ingest_slowdown_threshold") * Default: 100 * Type: Long * Unit: - * Is mutable: Yes * Description: The Compaction Score threshold that triggers Data Ingestion Slowdown in a shared-data cluster. This configuration only takes effect when `lake_enable_ingest_slowdown` is set to `true`. * Introduced in: v3.2.0 ##### `lake_publish_version_max_threads`[​](#lake_publish_version_max_threads "Direct link to lake_publish_version_max_threads") * Default: 512 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads for Version Publish tasks in a shared-data cluster. * Introduced in: v3.2.0 ##### `slow_publish_partition_log_threshold_ms`[​](#slow_publish_partition_log_threshold_ms "Direct link to slow_publish_partition_log_threshold_ms") * Default: 3000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The threshold above which `PublishVersionDaemon` logs a per-phase breakdown (`executor_queue` + `db_lock_wait` + `fe_prep` + `rpc`) of a slow partition publish at the WARN level. Lower this value to capture sub-second jitter when investigating publish latency on a live cluster; raise it to silence routine slow-but-acceptable publishes. There is no behavior change at the default value. * Introduced in: v4.2 ##### `meta_sync_force_delete_shard_meta`[​](#meta_sync_force_delete_shard_meta "Direct link to meta_sync_force_delete_shard_meta") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow deleting the metadata of the shared-data cluster directly, bypassing cleaning the remote storage files. It is recommended to set this item to `true` only when there is an excessive number of shards to be cleaned, which leads to extreme memory pressure on the FE JVM. Note that the data files belonging to the shards or tablets cannot be automatically cleaned after this feature is enabled. * Introduced in: v3.2.10, v3.3.3 ##### `run_mode`[​](#run_mode "Direct link to run_mode") * Default: `shared_nothing` * Type: String * Unit: - * Is mutable: No * Description: The running mode of the StarRocks cluster. Valid values: `shared_data` and `shared_nothing` (Default). * `shared_data` indicates running StarRocks in shared-data mode. * `shared_nothing` indicates running StarRocks in shared-nothing mode. > **CAUTION** > > * You cannot adopt the `shared_data` and `shared_nothing` modes simultaneously for a StarRocks cluster. Mixed deployment is not supported. > * DO NOT change `run_mode` after the cluster is deployed. Otherwise, the cluster fails to restart. The transformation from a shared-nothing cluster to a shared-data cluster or vice versa is not supported. * Introduced in: - ##### `shard_group_clean_threshold_sec`[​](#shard_group_clean_threshold_sec "Direct link to shard_group_clean_threshold_sec") * Default: 3600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The time before FE cleans the unused tablet and shard groups in a shared-data cluster. Tablets and shard groups created within this threshold will not be cleaned. * Introduced in: - ##### `star_mgr_meta_sync_interval_sec`[​](#star_mgr_meta_sync_interval_sec "Direct link to star_mgr_meta_sync_interval_sec") * Default: 600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The interval at which FE runs the periodical metadata synchronization with StarMgr in a shared-data cluster. * Introduced in: - ##### `starmgr_grpc_server_max_worker_threads`[​](#starmgr_grpc_server_max_worker_threads "Direct link to starmgr_grpc_server_max_worker_threads") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of worker threads that are used by the grpc server in the FE starmgr module. * Introduced in: v4.0.0, v3.5.8 ##### `starmgr_grpc_timeout_seconds`[​](#starmgr_grpc_timeout_seconds "Direct link to starmgr_grpc_timeout_seconds") * Default: 5 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: * Introduced in: - ##### `star_client_get_shard_info_batch_size`[​](#star_client_get_shard_info_batch_size "Direct link to star_client_get_shard_info_batch_size") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of tablets that the FE looks up in one StarMgr request when it resolves the Compute Nodes of many tablets at once in a shared-data cluster, for example when it plans a query or a load on a large table, or when AutoVacuum cleans up a partition or the FE deletes tablets. A larger lookup is split into consecutive requests of at most this many tablets. StarMgr holds a lock on its tablet metadata while it answers one request, which blocks StarMgr metadata changes and, on Follower FEs, the replay of StarMgr metadata. A request for all tablets of a large table can therefore stall them for seconds. A smaller value shortens each stall at the cost of more requests. Values less than `1` are treated as `1`. * Introduced in: v4.2.0 ##### `lake_scheduler_enable_colocate_group_sample`[​](#lake_scheduler_enable_colocate_group_sample "Direct link to lake_scheduler_enable_colocate_group_sample") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether the tablet scheduler estimates a large colocate group's replica distribution by sampling its tablets, instead of scanning every tablet in the group. When a tablet that belongs to a colocate group is scheduled, the scheduler builds a per-Compute Node replica histogram over the whole group to decide where the new replica goes. For a group with tens of thousands of tablets, that full scan dominates the scheduling cost, even though a healthy colocate group is uniformly placed and every tablet in it reports the same signal. When this item is set to `true`, the scheduler instead draws `lake_scheduler_colocate_group_sample_size` random tablets from any group larger than `lake_scheduler_colocate_group_sample_threshold` and scales the sampled per-Compute Node counts up to the full group size. A healthy, fully placed colocate group has all of its tablets on the same Compute Nodes, so the sample reproduces the true distribution exactly; a group that is mid-rebalance incurs a bounded error that at worst produces a sub-optimal (never invalid) placement, which the background tablet balancer later reconciles. Only colocate groups are sampled. Set this item to `false` to always scan every tablet. * Introduced in: v4.1.5 ##### `lake_scheduler_colocate_group_sample_threshold`[​](#lake_scheduler_colocate_group_sample_threshold "Direct link to lake_scheduler_colocate_group_sample_threshold") * Default: 256 * Type: Int * Unit: Count * Is mutable: Yes * Description: The minimum number of tablets a colocate group must exceed before the scheduler samples it. Groups at or below this size are always scanned in full, because the full scan is already cheap and sampling would only add error. This item takes effect only when `lake_scheduler_enable_colocate_group_sample` is set to `true`. * Introduced in: v4.1.5 ##### `lake_scheduler_colocate_group_sample_size`[​](#lake_scheduler_colocate_group_sample_size "Direct link to lake_scheduler_colocate_group_sample_size") * Default: 128 * Type: Int * Unit: Count * Is mutable: Yes * Description: The number of tablets sampled when a colocate group exceeds `lake_scheduler_colocate_group_sample_threshold`. A larger sample reduces the estimation error for a skewed (mid-rebalance) group but costs more per scheduling decision, so it trades scheduling latency for placement precision. The value should stay well below `lake_scheduler_colocate_group_sample_threshold`, otherwise sampling saves little over a full scan. This item takes effect only when `lake_scheduler_enable_colocate_group_sample` is set to `true`. * Introduced in: v4.1.5 ##### `lake_scheduler_colocate_group_sample_empty_fallback_percent`[​](#lake_scheduler_colocate_group_sample_empty_fallback_percent "Direct link to lake_scheduler_colocate_group_sample_empty_fallback_percent") * Default: 40 * Type: Int * Unit: Percent * Is mutable: Yes * Description: The density guard for colocate group sampling, expressed as the maximum tolerated percentage of empty draws. A sampled tablet is an empty draw when it holds no replica on a candidate Compute Node, that is, it is not placed yet or is not on that Compute Node. If more than this percentage of the sample is empty, the group is too sparsely placed for the sample to represent its true distribution, which is what happens while a group is still bulk filling from empty, so the scheduler discards the sample and falls back to a full scan. Equivalently, the sample is trusted only when at least (100 - this value)% of the sampled tablets are placed on a candidate Compute Node, so a lower value is more conservative and demands a denser group before sampling. A stable, fully placed group yields close to 0% empty draws and always takes the fast sampled path regardless of this value, so this item only governs the bulk-fill transient. Set it to `100` to never fall back. This item takes effect only when `lake_scheduler_enable_colocate_group_sample` is set to `true`. * Introduced in: v4.1.5 #### Data Lake[​](#data-lake "Direct link to Data Lake") ##### `files_enable_insert_push_down_column_type`[​](#files_enable_insert_push_down_column_type "Direct link to files_enable_insert_push_down_column_type") * Default: true * Alias: `files_enable_insert_push_down_schema` * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, StarRocks pushes target table column types down to the `files()` table function for `INSERT INTO target_table SELECT ... FROM files()` operations. Only the types of columns already inferred from the files are rewritten; no columns are added or removed. Complex types are skipped. This reduces type-mismatch errors caused by imprecise file-based type inference. For full schema push-down (column names and types), use the INSERT property `enable_push_down_schema`. * Introduced in: v3.4.0, v3.5.0 ##### `hdfs_read_buffer_size_kb`[​](#hdfs_read_buffer_size_kb "Direct link to hdfs_read_buffer_size_kb") * Default: 8192 * Type: Int * Unit: Kilobytes * Is mutable: Yes * Description: Size of the HDFS read buffer in kilobytes. StarRocks converts this value to bytes (`<< 10`) and uses it to initialize HDFS read buffers in `HdfsFsManager` and to populate the thrift field `hdfs_read_buffer_size_kb` sent to BE tasks (e.g., `TBrokerScanRangeParams`, `TDownloadReq`) when broker access is not used. Increasing `hdfs_read_buffer_size_kb` can improve sequential read throughput and reduce syscall overhead at the cost of higher per-stream memory usage; decreasing it reduces memory footprint but may lower IO efficiency. Consider workload (many small streams vs. few large sequential reads) when tuning. * Introduced in: v3.2.0 ##### `hdfs_write_buffer_size_kb`[​](#hdfs_write_buffer_size_kb "Direct link to hdfs_write_buffer_size_kb") * Default: 1024 * Type: Int * Unit: Kilobytes * Is mutable: Yes * Description: Sets the HDFS write buffer size (in KB) used for direct writes to HDFS or object stores when not using a broker. The FE converts this value to bytes (`<< 10`) and initializes the local write buffer in HdfsFsManager, and it is propagated in Thrift requests (e.g., TUploadReq, TExportSink, sink options) so backends/agents use the same buffer size. Increasing this value can improve throughput for large sequential writes at the cost of more memory per writer; decreasing it reduces per-stream memory usage and may lower latency for small writes. Tune alongside `hdfs_read_buffer_size_kb` and consider available memory and concurrent writers. * Introduced in: v3.2.0 ##### `lake_batch_publish_max_version_num`[​](#lake_batch_publish_max_version_num "Direct link to lake_batch_publish_max_version_num") * Default: 10 * Type: Int * Unit: Count * Is mutable: Yes * Description: Sets the upper bound on how many consecutive transaction versions may be grouped together when building a publish batch for lake (cloud‑native) tables. The value is passed to the transaction graph batching routine (see getReadyToPublishTxnListBatch) and works together with `lake_batch_publish_min_version_num` to determine the candidate range size for a TransactionStateBatch. Larger values can increase publish throughput by batching more commits, but increase the scope of an atomic publish (longer visibility latency and larger rollback surface) and may be limited at runtime when versions are not consecutive. Tune according to workload and visibility/latency requirements. * Introduced in: v3.2.0 ##### `lake_batch_publish_min_version_num`[​](#lake_batch_publish_min_version_num "Direct link to lake_batch_publish_min_version_num") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: Sets the minimum number of consecutive transaction versions required to form a publish batch for lake tables. DatabaseTransactionMgr.getReadyToPublishTxnListBatch passes this value to transactionGraph.getTxnsWithTxnDependencyBatch together with `lake_batch_publish_max_version_num` to select dependent transactions. A value of `1` allows single-transaction publishes (no batching). Values `>1` require at least that many consecutively-versioned, single-table, non-replication transactions to be available; batching is aborted if versions are non-consecutive, a replication transaction appears, or a schema change consumes a version. Increasing this value can improve publish throughput by grouping commits but may delay publishing while waiting for enough consecutive transactions. * Introduced in: v3.2.0 ##### `lake_enable_batch_publish_version`[​](#lake_enable_batch_publish_version "Direct link to lake_enable_batch_publish_version") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, PublishVersionDaemon batches ready transactions for the same Lake (shared-data) table/partition and publishes their versions together instead of issuing per-transaction publishes. In RunMode shared-data, the daemon calls getReadyPublishTransactionsBatch() and uses publishVersionForLakeTableBatch(...) to perform grouped publish operations (reducing RPCs and improving throughput). When disabled, the daemon falls back to per-transaction publishing via publishVersionForLakeTable(...). The implementation coordinates in-flight work using internal sets to avoid duplicate publishes when the switch is toggled and is affected by the thread pool sizing via `lake_publish_version_max_threads`. * Introduced in: v3.2.0 ##### `lake_enable_batch_publish_multi_table`[​](#lake_enable_batch_publish_multi_table "Direct link to lake_enable_batch_publish_multi_table") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow batch publish to group consecutive multi-table transactions into one publish operation. This benefits workloads that commit small atomic transactions spanning the same group of tables at a high rate (for example, CDC pipelines fanning out to multiple tables), where per-transaction publishing serializes on the shared table dependency chain and inflates the commit-to-visible latency. Effective only when `lake_enable_batch_publish_version` is `true`. Enable this parameter only after all FE nodes are upgraded to a version that supports it: an FE follower running an older version that replays a multi-table transaction batch applies the visible log of the first table only. * Introduced in: v4.1.5 ##### `lake_enable_tablet_creation_optimization`[​](#lake_enable_tablet_creation_optimization "Direct link to lake_enable_tablet_creation_optimization") * Default: false * Type: boolean * Unit: - * Is mutable: Yes * Description: When enabled, StarRocks optimizes tablet creation for cloud-native tables and materialized views in shared-data mode by creating a single shared tablet metadata for all tablets under a physical partition instead of distinct metadata per tablet. This reduces the number of tablet creation tasks and metadata/files produced during table creation, rollup, and schema-change jobs. The optimization is applied only for cloud-native tables/materialized views and is combined with `file_bundling` (the latter reuses the same optimization logic). Note: schema-change and rollup jobs explicitly disable the optimization for tables using `file_bundling` to avoid overwriting files with identical names. Tables that use range distribution never use the optimization: each of their tablets carries its own key range, so its metadata is always created per tablet. Enable cautiously — it changes the granularity of created tablet metadata and can affect how replica creation and file naming behave. * Introduced in: v3.3.1, v3.4.0, v3.5.0 ##### `lake_create_tablet_max_retries`[​](#lake_create_tablet_max_retries "Direct link to lake_create_tablet_max_retries") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of retry attempts for failed create-tablet tasks in shared-data mode. When a CN is unreachable or down during table creation, the failed tasks are retried on an alternative alive CN. Only send-phase failures (RPC errors, node down) are retried; CN-reported errors and timeouts are not retried. Set to `0` to disable retry. * Introduced in: v4.1 ##### `lake_use_combined_txn_log`[​](#lake_use_combined_txn_log "Direct link to lake_use_combined_txn_log") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When this item is set to `true`, the system allows Lake tables to use the combined transaction log path for relevant transactions. Available for shared-data clusters only. * Introduced in: v3.3.7, v3.4.0, v3.5.0 ##### `lake_repair_metadata_fetch_max_version_batch_size`[​](#lake_repair_metadata_fetch_max_version_batch_size "Direct link to lake_repair_metadata_fetch_max_version_batch_size") * Default: 160 * Type: Long * Unit: - * Is mutable: Yes * Description: The maximum batch size for version scanning when fetching tablet metadata during lake tablet repair. The batch size starts at 5 and grows exponentially (doubling each iteration) up to this maximum. A larger value allows more versions to be fetched in a single batch, which can improve repair efficiency by leveraging file existence caching across versions. If set to a value less than 5, it will be clamped to 5 at runtime. * Introduced in: v3.5.16, v4.0.9 ##### `lake_enable_drop_tablet_cache`[​](#lake_enable_drop_tablet_cache "Direct link to lake_enable_drop_tablet_cache") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: In shared-data mode, clears the cache on BE/CN before the underlying data is actually deleted. * Introduced in: v4.0 ##### `enable_iceberg_commit_queue`[​](#enable_iceberg_commit_queue "Direct link to enable_iceberg_commit_queue") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable commit queue for Iceberg tables to avoid concurrent commit conflicts. Iceberg uses optimistic concurrency control (OCC) for metadata commits. When multiple threads concurrently commit to the same table, conflicts can occur with errors like: "Cannot commit: Base metadata location is not same as the current table metadata location". When enabled, each Iceberg table has its own single-threaded executor for commit operations, ensuring that commits to the same table are serialized and preventing OCC conflicts. Different tables can commit concurrently, maintaining overall throughput. This is a system-level optimization to improve reliability and should be enabled by default. If disabled, concurrent commits may fail due to optimistic locking conflicts. * Introduced in: v4.1.0 ##### `iceberg_commit_queue_timeout_seconds`[​](#iceberg_commit_queue_timeout_seconds "Direct link to iceberg_commit_queue_timeout_seconds") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout in seconds for waiting for an Iceberg commit operation to complete. When using the commit queue (`enable_iceberg_commit_queue=true`), each commit operation must complete within this timeout. If a commit takes longer than this timeout, it will be cancelled and an error will be raised. Factors that affect commit time include: number of data files being committed, metadata size of the table, performance of the underlying storage (e.g., S3, HDFS). * Introduced in: v4.1.0 ##### `iceberg_commit_queue_max_size`[​](#iceberg_commit_queue_max_size "Direct link to iceberg_commit_queue_max_size") * Default: 1000 * Type: Int * Unit: Count * Is mutable: No * Description: The maximum number of pending commit operations per Iceberg table. When using the commit queue (`enable_iceberg_commit_queue=true`), this limits the number of commit operations that can be queued for a single table. When the limit is reached, additional commit operations will execute in the caller thread (blocking until capacity available). This configuration is read at FE startup and applies to newly created table executors. Requires FE restart to take effect. Increase this value if you expect many concurrent commits to the same table. If this value is too low, commits may block in the caller thread during high concurrency. * Introduced in: v4.1.0 ##### `iceberg_remove_orphan_files_min_retention_seconds`[​](#iceberg_remove_orphan_files_min_retention_seconds "Direct link to iceberg_remove_orphan_files_min_retention_seconds") * Default: 86400 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The `older_than` argument of the `remove_orphan_files` procedure must be earlier than `current time - this value`. A later `older_than` is rejected, because deleting files that recent can remove data that a concurrent write has not committed yet and leave the table unreadable. Only an explicit `older_than` is bounded; omitting it keeps the procedure's own 7-day default. Lower this value only when nothing writes to the table while the procedure runs. * Introduced in: v4.1.5 ###### lake\_balance\_tablets\_threshold[​](#lake_balance_tablets_threshold "Direct link to lake_balance_tablets_threshold") * Default: 0.15 * Type: Double * Unit: - * Is mutable: Yes * Description: The threshold the system used to judge the tablet balance among workers in a shared-data cluster, The imbalance factor is calculated as `f = (MAX(tablets) - MIN(tablets)) / AVERAGE(tablets)`. If the factor is greater than `lake_balance_tablets_threshold`, a tablet balance will be triggered. This item takes effect only when `lake_enable_balance_tablets_between_workers` is set to `true`. * Introduced in: v3.3.4 #### Other[​](#other "Direct link to Other") ##### `agent_task_resend_wait_time_ms`[​](#agent_task_resend_wait_time_ms "Direct link to agent_task_resend_wait_time_ms") * Default: 5000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The duration the FE must wait before it can resend an agent task. An agent task can be resent only when the gap between the task creation time and the current time exceeds the value of this parameter. This parameter is used to prevent repetitive sending of agent tasks. * Introduced in: - ##### `allow_system_reserved_names`[​](#allow_system_reserved_names "Direct link to allow_system_reserved_names") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow users to create columns whose names are initiated with `__op` and `__row`. To enable this feature, set this parameter to `TRUE`. Please note that these name formats are reserved for special purposes in StarRocks and creating such columns may result in undefined behavior. Therefore this feature is disabled by default. * Introduced in: v3.2.0 ##### `auth_token`[​](#auth_token "Direct link to auth_token") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The token that is used for identity authentication within the StarRocks cluster to which the FE belongs. If this parameter is left unspecified, StarRocks generates a random token for the cluster at the time when the leader FE of the cluster is started for the first time. * Introduced in: - ##### `authentication_failure_cache_capacity`[​](#authentication_failure_cache_capacity "Direct link to authentication_failure_cache_capacity") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: How many rejected credentials `authentication_failure_cache_ttl_second` remembers at most. It bounds the memory a client - or an attacker cycling usernames - can occupy; the oldest entries are evicted first. * Introduced in: v4.2, v4.1.6 ##### `authentication_failure_cache_ttl_second`[​](#authentication_failure_cache_ttl_second "Direct link to authentication_failure_cache_ttl_second") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: How long a credential rejected by the security integration chain is remembered, so that a client retrying the same wrong password does not produce one LDAP bind per attempt - which is what drives Active Directory's `badPwdCount` toward locking the account out. The cache key includes a hash of the credential, so correcting the password takes effect immediately instead of after the TTL, and a failure caused by the directory being unreachable is never cached. Set to `0` to disable. * Introduced in: v4.2, v4.1.6 ##### `authentication_ldap_case_insensitive`[​](#authentication_ldap_case_insensitive "Direct link to authentication_ldap_case_insensitive") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether the StarRocks-side matching of an LDAP/AD user name is relaxed to ignore case. When this item is set to `true`, a login whose name differs only in case from a user created with `AUTHENTICATION_LDAP_SIMPLE` still resolves to that user, so its per-user DN and the roles granted to it apply; and a login authenticated by an LDAP security integration takes its session identity from the name held by the directory rather than the one typed by the client. Because this both widens which stored user a login may resolve to and changes the value reported by `current_user()` and `SHOW PROCESSLIST`, it is disabled by default. Users authenticated by native password, JWT, or OAuth2 are never affected. If two users created with `AUTHENTICATION_LDAP_SIMPLE` have names that differ only in case, a login matching both of them is refused rather than resolved to either. Group names are matched without regard to case independently of this item. * Introduced in: v4.2.0 ##### `authentication_ldap_simple_bind_base_dn`[​](#authentication_ldap_simple_bind_base_dn "Direct link to authentication_ldap_simple_bind_base_dn") * Default: Empty string * Type: String * Unit: - * Is mutable: Yes * Description: The base DN, which is the point from which the LDAP server starts to search for users' authentication information. * Introduced in: - ##### `authentication_ldap_simple_bind_dn_pattern`[​](#authentication_ldap_simple_bind_dn_pattern "Direct link to authentication_ldap_simple_bind_dn_pattern") * Default: Empty string * Type: String * Unit: - * Is mutable: Yes * Description: The DN pattern for direct bind authentication. Use `${USER}` as a placeholder for the username. The pattern must produce a valid LDAP Distinguished Name (DN); UPN-style patterns like `${USER}@domain` are not supported. For example, `uid=${USER},ou=People,dc=example,dc=com`. Multiple patterns can be separated by semicolons, and the system will try each pattern in order until one succeeds. When set, the search step is skipped and the system binds directly with the constructed DN. ##### `authentication_ldap_simple_bind_root_dn`[​](#authentication_ldap_simple_bind_root_dn "Direct link to authentication_ldap_simple_bind_root_dn") * Default: Empty string * Type: String * Unit: - * Is mutable: Yes * Description: The administrator DN used to search for users' authentication information. * Introduced in: - ##### `authentication_ldap_simple_bind_root_pwd`[​](#authentication_ldap_simple_bind_root_pwd "Direct link to authentication_ldap_simple_bind_root_pwd") * Default: Empty string * Type: String * Unit: - * Is mutable: Yes * Description: The password of the administrator used to search for users' authentication information. * Introduced in: - ##### `authentication_ldap_simple_conn_read_timeout_ms`[​](#authentication_ldap_simple_conn_read_timeout_ms "Direct link to authentication_ldap_simple_conn_read_timeout_ms") * Default: 30000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Socket read timeout for the LDAP bind performed by `authentication_ldap_simple`. See `authentication_ldap_simple_conn_timeout_ms`. * Introduced in: v4.2, v4.1.6 ##### `authentication_ldap_simple_conn_timeout_ms`[​](#authentication_ldap_simple_conn_timeout_ms "Direct link to authentication_ldap_simple_conn_timeout_ms") * Default: 30000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: TCP connect timeout for the LDAP bind performed by `authentication_ldap_simple`. Without it the bind falls back to the operating system's TCP timeout, which can hold the thread serving the request - an HTTP worker or a Thrift handler, now that a security integration also authenticates those channels - for minutes when the directory is unreachable. Lower it if authentication has to fail fast. * Introduced in: v4.2, v4.1.6 ##### `authentication_ldap_simple_group_source`[​](#authentication_ldap_simple_group_source "Direct link to authentication_ldap_simple_group_source") * Default: group\_provider * Type: String * Unit: - * Is mutable: Yes * Description: Cluster-wide default for where the groups of an LDAP-authenticated user come from. Valid values: `group_provider` (only the configured group providers, the behavior of earlier versions), `memberof` (only the group membership attribute of the user's own LDAP entry), `both` (the union of the two). A security integration property of the same name overrides this value. An illegal value is treated as `group_provider` and logged at ERROR level, so that a typo cannot lock every LDAP user out. * Introduced in: v4.2 ##### `authentication_ldap_simple_memberof_attr`[​](#authentication_ldap_simple_memberof_attr "Direct link to authentication_ldap_simple_memberof_attr") * Default: memberOf * Type: String * Unit: - * Is mutable: Yes * Description: Cluster-wide default for the name of the attribute on the user entry that carries its group membership, used when `authentication_ldap_simple_group_source` is `memberof` or `both`. `memberOf` fits Active Directory and OpenLDAP with the `memberof` overlay installed; Oracle Directory Server and 389 Directory Server use `isMemberOf`. A security integration property of the same name overrides this value. * Introduced in: v4.2 ##### `authentication_ldap_simple_server_host`[​](#authentication_ldap_simple_server_host "Direct link to authentication_ldap_simple_server_host") * Default: Empty string * Type: String * Unit: - * Is mutable: Yes * Description: The host on which the LDAP server runs. * Introduced in: - ##### `authentication_ldap_simple_server_port`[​](#authentication_ldap_simple_server_port "Direct link to authentication_ldap_simple_server_port") * Default: 389 * Type: Int * Unit: - * Is mutable: Yes * Description: The port of the LDAP server. * Introduced in: - ##### `authentication_ldap_simple_user_search_attr`[​](#authentication_ldap_simple_user_search_attr "Direct link to authentication_ldap_simple_user_search_attr") * Default: uid * Type: String * Unit: - * Is mutable: Yes * Description: The name of the attribute that carries the login name on a user entry, used to build the search filter of search-and-bind mode. `uid` suits OpenLDAP; on Active Directory set it to `sAMAccountName`, because an AD entry's RDN is the display name and its `uid` is empty unless an administrator populated it. That choice also rules out `authentication_ldap_simple_bind_dn_pattern` - see the property of the same name in [Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md). * Introduced in: - ##### `backup_job_default_timeout_ms`[​](#backup_job_default_timeout_ms "Direct link to backup_job_default_timeout_ms") * Default: 86400 \* 1000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The timeout duration of a backup job. If this value is exceeded, the backup job fails. * Introduced in: - ##### `enable_collect_tablet_num_in_show_proc_backend_disk_path`[​](#enable_collect_tablet_num_in_show_proc_backend_disk_path "Direct link to enable_collect_tablet_num_in_show_proc_backend_disk_path") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the collection of tablet numbers for each disk in the `SHOW PROC /BACKENDS/{id}` command * Introduced in: v4.0.1, v3.5.8 ##### `enable_colocate_restore`[​](#enable_colocate_restore "Direct link to enable_colocate_restore") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Backup and Restore for Colocate Tables. `true` indicates enabling Backup and Restore for Colocate Tables and `false` indicates disabling it. * Introduced in: v3.2.10, v3.3.3 ##### `enable_external_catalog_information_schema_tables_access_full_metadata`[​](#enable_external_catalog_information_schema_tables_access_full_metadata "Direct link to enable_external_catalog_information_schema_tables_access_full_metadata") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls whether `information_schema.tables` is allowed to access external metadata services when resolving tables in external catalogs (such as Hive, Iceberg, JDBC). When set to `false` (default), columns like `TABLE_COMMENT` may be empty for external tables but the query is fast and avoids remote calls. When set to `true`, the FE contacts the corresponding external metadata service and can populate fields like `TABLE_COMMENT` at the cost of additional latency and remote calls per table. * Introduced in: - ##### `enable_materialized_view_concurrent_prepare`[​](#enable_materialized_view_concurrent_prepare "Direct link to enable_materialized_view_concurrent_prepare") * Default: true * Type: Boolean * Unit: * Is mutable: Yes * Description: Whether to prepare materialized view concurrently to improve performance. * Introduced in: v3.4.4 ##### `enable_metric_calculator`[​](#enable_metric_calculator "Direct link to enable_metric_calculator") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Specifies whether to enable the feature that is used to periodically collect metrics. Valid values: `TRUE` and `FALSE`. `TRUE` specifies to enable this feature, and `FALSE` specifies to disable this feature. * Introduced in: - ##### `enable_table_metrics_collect`[​](#enable_table_metrics_collect "Direct link to enable_table_metrics_collect") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to export table-level metrics in FE. When disabled, FE will skip exporting table metrics (such as table scan/load counters and table size metrics), but still records the counters in memory. * Introduced in: - ##### `enable_mv_post_image_reload_cache`[​](#enable_mv_post_image_reload_cache "Direct link to enable_mv_post_image_reload_cache") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to perform reload flag check after FE loaded an image. If the check is performed for a base materialized view, it is not needed for other materialized views that related to it. * Introduced in: v3.5.0 ##### `enable_mv_query_context_cache`[​](#enable_mv_query_context_cache "Direct link to enable_mv_query_context_cache") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable query-level materialized view rewrite cache to improve query rewrite performance. * Introduced in: v3.3 ##### `enable_mv_refresh_collect_profile`[​](#enable_mv_refresh_collect_profile "Direct link to enable_mv_refresh_collect_profile") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable profile in refreshing materialized view by default for all materialized views. * Introduced in: v3.3.0 ##### `enable_mv_refresh_extra_prefix_logging`[​](#enable_mv_refresh_extra_prefix_logging "Direct link to enable_mv_refresh_extra_prefix_logging") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable prefixes with materialized view names in logs for better debug. * Introduced in: v3.4.0 ##### `enable_mv_refresh_query_rewrite`[​](#enable_mv_refresh_query_rewrite "Direct link to enable_mv_refresh_query_rewrite") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable rewrite query during materialized view refresh so that the query can use the rewritten mv directly rather than the base table to improve query performance. * Introduced in: v3.3 ##### `enable_trace_historical_node`[​](#enable_trace_historical_node "Direct link to enable_trace_historical_node") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow the system to trace the historical nodes. By setting this item to `true`, you can enable the Cache Sharing feature and allow the system to choose the right cache nodes during elastic scaling. * Introduced in: v3.5.1 ##### `es_state_sync_interval_second`[​](#es_state_sync_interval_second "Direct link to es_state_sync_interval_second") * Default: 10 * Type: Long * Unit: Seconds * Is mutable: No * Description: The time interval at which the FE obtains Elasticsearch indexes and synchronizes the metadata of StarRocks external tables. * Introduced in: - ##### `hive_meta_cache_refresh_interval_s`[​](#hive_meta_cache_refresh_interval_s "Direct link to hive_meta_cache_refresh_interval_s") * Default: 60 * Type: Long * Unit: Seconds * Is mutable: No * Description: The time interval at which the cached metadata of Hive external tables is updated. * Introduced in: - ##### `hive_meta_store_timeout_s`[​](#hive_meta_store_timeout_s "Direct link to hive_meta_store_timeout_s") * Default: 10 * Type: Long * Unit: Seconds * Is mutable: No * Description: The amount of time after which a connection to a Hive metastore times out. * Introduced in: - ##### `jdbc_connection_idle_timeout_ms`[​](#jdbc_connection_idle_timeout_ms "Direct link to jdbc_connection_idle_timeout_ms") * Default: 600000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: The maximum amount of time after which a connection for accessing a JDBC catalog times out. Timed-out connections are considered idle. * Introduced in: - ##### `jdbc_connection_timeout_ms`[​](#jdbc_connection_timeout_ms "Direct link to jdbc_connection_timeout_ms") * Default: 10000 * Type: Long * Unit: Milliseconds * Is mutable: No * Description: The timeout in milliseconds for HikariCP connection pool to acquire a connection. If a connection cannot be acquired from the pool within this time, the operation will fail. * Introduced in: v3.5.13 ##### `jdbc_query_timeout_ms`[​](#jdbc_query_timeout_ms "Direct link to jdbc_query_timeout_ms") * Default: 30000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The timeout in milliseconds for JDBC statement query execution. This timeout is applied to all SQL queries executed through JDBC catalogs (e.g., partition metadata queries). The value is converted to seconds when passed to the JDBC driver. * Introduced in: v3.5.13 ##### `jdbc_network_timeout_ms`[​](#jdbc_network_timeout_ms "Direct link to jdbc_network_timeout_ms") * Default: 30000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The timeout in milliseconds for JDBC network operations (socket read). This timeout applies to database metadata calls (e.g., getSchemas(), getTables(), getColumns()) to prevent indefinite blocking when the external database is unresponsive. * Introduced in: v3.5.13 ##### `jdbc_connection_max_lifetime_ms`[​](#jdbc_connection_max_lifetime_ms "Direct link to jdbc_connection_max_lifetime_ms") * Default: 300000 * Type: Long * Unit: Milliseconds * Is mutable: No * Description: Maximum lifetime of a connection in the JDBC connection pool. Connections are recycled before this timeout to prevent stale connections. Should be shorter than the external database's connection timeout. Minimum allowed value is 30000 (30 seconds). * Introduced in: - ##### `jdbc_connection_keepalive_time_ms`[​](#jdbc_connection_keepalive_time_ms "Direct link to jdbc_connection_keepalive_time_ms") * Default: 30000 * Type: Long * Unit: Milliseconds * Is mutable: No * Description: Keepalive interval for idle JDBC connections. Idle connections are tested at this interval to detect stale connections proactively. Set to 0 to disable keepalive probing. When enabled, must be >= 30000 and less than `jdbc_connection_max_lifetime_ms`. Invalid enabled values are silently disabled (reset to 0). * Introduced in: - ##### `jdbc_connection_leak_detection_threshold_ms`[​](#jdbc_connection_leak_detection_threshold_ms "Direct link to jdbc_connection_leak_detection_threshold_ms") * Default: 0 * Type: Long * Unit: Milliseconds * Is mutable: No * Description: Threshold for JDBC connection leak detection. If a connection is held longer than this, a warning is logged. Set to 0 to disable. This is a debugging aid for identifying code paths that hold connections too long. * Introduced in: - ##### `jdbc_connection_pool_size`[​](#jdbc_connection_pool_size "Direct link to jdbc_connection_pool_size") * Default: 8 * Type: Int * Unit: - * Is mutable: No * Description: The maximum capacity of the JDBC connection pool for accessing JDBC catalogs. * Introduced in: - ##### `jdbc_meta_default_cache_enable`[​](#jdbc_meta_default_cache_enable "Direct link to jdbc_meta_default_cache_enable") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: The default value for whether the JDBC Catalog metadata cache is enabled. When set to True, newly created JDBC Catalogs will default to metadata caching enabled. * Introduced in: - ##### `jdbc_meta_default_cache_expire_sec`[​](#jdbc_meta_default_cache_expire_sec "Direct link to jdbc_meta_default_cache_expire_sec") * Default: 600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The default expiration time for the JDBC Catalog metadata cache. When `jdbc_meta_default_cache_enable` is set to true, newly created JDBC Catalogs will default to setting the expiration time of the metadata cache. * Introduced in: - ##### `jdbc_row_count_cache_refresh_sec`[​](#jdbc_row_count_cache_refresh_sec "Direct link to jdbc_row_count_cache_refresh_sec") * Default: 600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: Background refresh interval for the JDBC table row-count cache. After this interval, the stale value is returned immediately while a reload runs asynchronously in the background. Overridable per-catalog via the catalog property `jdbc_row_count_cache_refresh_sec`. * Introduced in: - ##### `jdbc_row_count_cache_expire_sec`[​](#jdbc_row_count_cache_expire_sec "Direct link to jdbc_row_count_cache_expire_sec") * Default: 1200 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: Hard eviction TTL for JDBC table row-count cache entries. Entries not accessed within this window are evicted. Must be greater than `jdbc_row_count_cache_refresh_sec`. Overridable per-catalog via the catalog property `jdbc_row_count_cache_expire_sec`. * Introduced in: - ##### `jdbc_row_count_cache_max_size`[​](#jdbc_row_count_cache_max_size "Direct link to jdbc_row_count_cache_max_size") * Default: 10000 * Type: Long * Unit: - * Is mutable: Yes * Description: Maximum number of entries in the JDBC table row-count cache per catalog. Limits memory growth for catalogs with large numbers of tables. Overridable per-catalog via the catalog property `jdbc_row_count_cache_max_size`. * Introduced in: - ##### `jdbc_minimum_idle_connections`[​](#jdbc_minimum_idle_connections "Direct link to jdbc_minimum_idle_connections") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: The minimum number of idle connections in the JDBC connection pool for accessing JDBC catalogs. * Introduced in: - ##### `jwt_jwks_url`[​](#jwt_jwks_url "Direct link to jwt_jwks_url") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The URL to the JSON Web Key Set (JWKS) service or the path to the public key local file under the `fe/conf` directory. * Introduced in: v3.5.0 ##### `jwt_principal_field`[​](#jwt_principal_field "Direct link to jwt_principal_field") * Default: sub * Type: String * Unit: - * Is mutable: No * Description: The string used to identify the field that indicates the subject (`sub`) in the JWT. The default value is `sub`. The value of this field must be identical with the username for logging in to StarRocks. * Introduced in: v3.5.0 ##### `jwt_required_audience`[​](#jwt_required_audience "Direct link to jwt_required_audience") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The list of strings used to identify the audience (`aud`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT audience. * Introduced in: v3.5.0 ##### `jwt_required_issuer`[​](#jwt_required_issuer "Direct link to jwt_required_issuer") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The list of strings used to identify the issuers (`iss`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT issuer. * Introduced in: v3.5.0 ##### locale[​](#locale "Direct link to locale") * Default: `zh_CN.UTF-8` * Type: String * Unit: - * Is mutable: No * Description: The character set that is used by the FE. * Introduced in: - ##### `max_agent_task_threads_num`[​](#max_agent_task_threads_num "Direct link to max_agent_task_threads_num") * Default: 4096 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of threads that are allowed in the agent task thread pool. * Introduced in: - ##### `max_download_task_per_be`[​](#max_download_task_per_be "Direct link to max_download_task_per_be") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: In each RESTORE operation, the maximum number of download tasks StarRocks assigned to a BE node. When this item is set to less than or equal to 0, no limit is imposed on the task number. * Introduced in: v3.1.0 ##### `max_mv_check_base_table_change_retry_times`[​](#max_mv_check_base_table_change_retry_times "Direct link to max_mv_check_base_table_change_retry_times") * Default: 10 * Type: - * Unit: - * Is mutable: Yes * Description: The maximum retry times for detecting base table change when refreshing materialized views. * Introduced in: v3.3.0 ##### `max_mv_refresh_failure_retry_times`[​](#max_mv_refresh_failure_retry_times "Direct link to max_mv_refresh_failure_retry_times") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum retry times when materialized view fails to refresh. * Introduced in: v3.3.0 ##### `max_mv_refresh_try_lock_failure_retry_times`[​](#max_mv_refresh_try_lock_failure_retry_times "Direct link to max_mv_refresh_try_lock_failure_retry_times") * Default: 3 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum retry times of try lock when materialized view fails to refresh. * Introduced in: v3.3.0 ##### `max_small_file_number`[​](#max_small_file_number "Direct link to max_small_file_number") * Default: 100 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of small files that can be stored on an FE directory. * Introduced in: - ##### `max_small_file_size_bytes`[​](#max_small_file_size_bytes "Direct link to max_small_file_size_bytes") * Default: 1024 \* 1024 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The maximum size of a small file. * Introduced in: - ##### `max_upload_task_per_be`[​](#max_upload_task_per_be "Direct link to max_upload_task_per_be") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: In each BACKUP operation, the maximum number of upload tasks StarRocks assigned to a BE node. When this item is set to less than or equal to 0, no limit is imposed on the task number. * Introduced in: v3.1.0 ##### `mv_create_partition_batch_interval_ms`[​](#mv_create_partition_batch_interval_ms "Direct link to mv_create_partition_batch_interval_ms") * Default: 1000 * Type: Int * Unit: ms * Is mutable: Yes * Description: During materialized view refresh, if multiple partitions need to be created in bulk, the system divides them into batches of 64 partitions each. To reduce the risk of failures caused by frequent partition creation, a default interval (in milliseconds) is set between each batch to control the creation frequency. * Introduced in: v3.3 ##### `mv_plan_cache_max_size`[​](#mv_plan_cache_max_size "Direct link to mv_plan_cache_max_size") * Default: 1000 * Type: Long * Unit: * Is mutable: Yes * Description: The maximum size of materialized view plan cache (which is used for materialized view rewrite). If there are many materialized views used for transparent query rewrite, you may increase this value. * Introduced in: v3.2 ##### `mv_plan_cache_thread_pool_size`[​](#mv_plan_cache_thread_pool_size "Direct link to mv_plan_cache_thread_pool_size") * Default: 8 * Type: Int * Unit: - * Is mutable: Yes * Description: The default thread pool size of materialized view plan cache (which is used for materialized view rewrite). * Introduced in: v3.2 ##### `mv_refresh_default_planner_optimize_timeout`[​](#mv_refresh_default_planner_optimize_timeout "Direct link to mv_refresh_default_planner_optimize_timeout") * Default: 30000 * Type: - * Unit: - * Is mutable: Yes * Description: The default timeout for the planning phase of the optimizer when refresh materialized views. * Introduced in: v3.3.0 ##### `mv_refresh_fail_on_filter_data`[​](#mv_refresh_fail_on_filter_data "Direct link to mv_refresh_fail_on_filter_data") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Mv refresh fails if there is filtered data in refreshing, true by default, otherwise return success by ignoring the filtered data. * Introduced in: - ##### `mv_refresh_try_lock_timeout_ms`[​](#mv_refresh_try_lock_timeout_ms "Direct link to mv_refresh_try_lock_timeout_ms") * Default: 30000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The default try lock timeout for materialized view refresh to try the DB lock of its base table/materialized view. * Introduced in: v3.3.0 ##### `oauth2_auth_server_url`[​](#oauth2_auth_server_url "Direct link to oauth2_auth_server_url") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The authorization URL. The URL to which the users’ browser will be redirected in order to begin the OAuth 2.0 authorization process. * Introduced in: v3.5.0 ##### `oauth2_client_id`[​](#oauth2_client_id "Direct link to oauth2_client_id") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The public identifier of the StarRocks client. * Introduced in: v3.5.0 ##### `oauth2_client_secret`[​](#oauth2_client_secret "Direct link to oauth2_client_secret") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The secret used to authorize StarRocks client with the authorization server. * Introduced in: v3.5.0 ##### `oauth2_jwks_url`[​](#oauth2_jwks_url "Direct link to oauth2_jwks_url") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The URL to the JSON Web Key Set (JWKS) service or the path to the local file under the `conf` directory. * Introduced in: v3.5.0 ##### `oauth2_principal_field`[​](#oauth2_principal_field "Direct link to oauth2_principal_field") * Default: sub * Type: String * Unit: - * Is mutable: No * Description: The string used to identify the field that indicates the subject (`sub`) in the JWT. The default value is `sub`. The value of this field must be identical with the username for logging in to StarRocks. * Introduced in: v3.5.0 ##### `oauth2_redirect_url`[​](#oauth2_redirect_url "Direct link to oauth2_redirect_url") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The URL to which the users’ browser will be redirected after the OAuth 2.0 authentication succeeds. The authorization code will be sent to this URL. In most cases, it need to be configured as `http://:/api/oauth2`. * Introduced in: v3.5.0 ##### `oauth2_required_audience`[​](#oauth2_required_audience "Direct link to oauth2_required_audience") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The list of strings used to identify the audience (`aud`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT audience. * Introduced in: v3.5.0 ##### `oauth2_required_issuer`[​](#oauth2_required_issuer "Direct link to oauth2_required_issuer") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The list of strings used to identify the issuers (`iss`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT issuer. * Introduced in: v3.5.0 ##### `oauth2_token_server_url`[​](#oauth2_token_server_url "Direct link to oauth2_token_server_url") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The URL of the endpoint on the authorization server from which StarRocks obtains the access token. * Introduced in: v3.5.0 ##### `plugin_dir`[​](#plugin_dir "Direct link to plugin_dir") * Default: `System.getenv("STARROCKS_HOME")` + "/plugins" * Type: String * Unit: - * Is mutable: No * Description: The directory that stores plugin installation packages. * Introduced in: - ##### `plugin_enable`[​](#plugin_enable "Direct link to plugin_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether plugins can be installed on FEs. Plugins can be installed or uninstalled only on the Leader FE. * Introduced in: - ##### `proc_profile_jstack_depth`[​](#proc_profile_jstack_depth "Direct link to proc_profile_jstack_depth") * Default: 128 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum Java stack depth when the system collects CPU and memory profiles. This value controls how many Java stack frames are captured for each sampled stack: larger values increase trace detail and output size and may add profiling overhead, while smaller values reduce details. This setting is used when the profiler is started for both CPU and memory profiling, so adjust it to balance diagnostic needs and performance impact. * Introduced in: - ##### `proc_profile_mem_enable`[​](#proc_profile_mem_enable "Direct link to proc_profile_mem_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable collection of process memory allocation profiles. When this item is set to `true`, the system generates an HTML profile named `mem-profile-.html` under `sys_log_dir/proc_profile`, sleeps for `proc_profile_collect_time_s` seconds while sampling, and uses `proc_profile_jstack_depth` for Java stack depth. Generated files are compressed and purged according to `proc_profile_file_retained_days` and `proc_profile_file_retained_size_bytes`. The native extraction path uses `STARROCKS_HOME_DIR` to avoid `/tmp` noexec issues. This item is intended for troubleshooting memory-allocation hotspots. Enabling it increases CPU, I/O and disk usage and may produce large files. * Introduced in: v3.2.12 ##### `query_detail_explain_level`[​](#query_detail_explain_level "Direct link to query_detail_explain_level") * Default: COSTS * Type: String * Unit: - * Is mutable: true * Description: The detail level of query plan returned by the EXPLAIN statement. Valid values: COSTS, NORMAL, VERBOSE. * Introduced in: v3.2.12, v3.3.5 ##### `replication_interval_ms`[​](#replication_interval_ms "Direct link to replication_interval_ms") * Default: 100 * Type: Int * Unit: - * Is mutable: No * Description: The minimum time interval at which the replication tasks are scheduled. * Introduced in: v3.3.5 ##### `replication_max_parallel_data_size_mb`[​](#replication_max_parallel_data_size_mb "Direct link to replication_max_parallel_data_size_mb") * Default: 1048576 * Type: Int * Unit: MB * Is mutable: Yes * Description: The maximum size of data allowed for concurrent synchronization. * Introduced in: v3.3.5 ##### `replication_max_parallel_replica_count`[​](#replication_max_parallel_replica_count "Direct link to replication_max_parallel_replica_count") * Default: 10240 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of tablet replicas allowed for concurrent synchronization. * Introduced in: v3.3.5 ##### `replication_max_parallel_table_count`[​](#replication_max_parallel_table_count "Direct link to replication_max_parallel_table_count") * Default: 100 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of concurrent data synchronization tasks allowed. StarRocks creates one synchronization task for each table. * Introduced in: v3.3.5 ##### `replication_transaction_timeout_sec`[​](#replication_transaction_timeout_sec "Direct link to replication_transaction_timeout_sec") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for synchronization tasks. * Introduced in: v3.3.5 ##### `skip_whole_phase_lock_mv_limit`[​](#skip_whole_phase_lock_mv_limit "Direct link to skip_whole_phase_lock_mv_limit") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls when StarRocks applies the "non-lock" optimization for tables that have related materialized views. When this item is set to less than 0, the system always applies non-lock optimization and does not copy related materialized views for queries (FE memory usage and metadata copy/lock contention is reduced but risk of metadata concurrency issues can be increased). When it is set to 0, non-lock optimization is applied only for tables that carry no related materialized view. When it is set to greater than 0, non-lock optimization is applied only for tables whose number of related materialized views is less than or equal to the configured threshold. This threshold is not applied to a statement that also reads a table in an external catalog: holding the lock for the whole planning phase would bind it to partition, statistics, and file-list requests to a system outside the FE's control, while that lock protects nothing on the external side, so such a statement uses the non-lock path however many related materialized views the internal table carries. Additionally, when the value is greater than and equal to 0, the planner records query OLAP tables into the optimizer context to enable materialized view-related rewrite paths; when it is less than 0, this step is skipped. * Introduced in: v3.2.1 ##### `small_file_dir`[​](#small_file_dir "Direct link to small_file_dir") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/small\_files" * Type: String * Unit: - * Is mutable: No * Description: The root directory of small files. * Introduced in: - ##### `task_runs_max_history_number`[​](#task_runs_max_history_number "Direct link to task_runs_max_history_number") * Default: 10000 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of task run records to retain in memory and to use as a default LIMIT when querying archived task-run history. When `enable_task_history_archive` is false, this value bounds in-memory history: Force GC trims older entries so only the newest `task_runs_max_history_number` remain. When archive history is queried (and no explicit LIMIT is provided), `TaskRunHistoryTable.lookup` uses `"ORDER BY create_time DESC LIMIT "` if this value is greater than 0. Note: setting this to 0 disables the query-side LIMIT (no cap) but will cause in-memory history to be truncated to zero (unless archiving is enabled). * Introduced in: v3.2.0 ##### `tmp_dir`[​](#tmp_dir "Direct link to tmp_dir") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/temp\_dir" * Type: String * Unit: - * Is mutable: No * Description: The directory that stores temporary files such as files generated during backup and restore procedures. After these procedures finish, the generated temporary files are deleted. * Introduced in: - ##### `transform_type_prefer_string_for_varchar`[​](#transform_type_prefer_string_for_varchar "Direct link to transform_type_prefer_string_for_varchar") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to prefer string type for fixed length char/varchar columns in materialized view creation. * Introduced in: v4.0.0 --- ### FE Configuration - Statistics and Storage FE parameters are classified into dynamic parameters and static parameters. * Dynamic parameters can be configured and adjusted by running SQL commands, which is very convenient. But the configurations become invalid if you restart your FE. Therefore, we recommend that you also modify the configuration items in the **fe.conf** file to prevent the loss of modifications. * Static parameters can only be configured and adjusted in the FE configuration file **fe.conf**. **After you modify this file, you must restart your FE for the changes to take effect.** Whether a parameter is a dynamic parameter is indicated by the `IsMutable` column in the output of [ADMIN SHOW CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). `TRUE` indicates a dynamic parameter. Note that both dynamic and static FE parameters can be configured in the **fe.conf** file. #### View FE configuration items[​](#view-fe-configuration-items "Direct link to View FE configuration items") After your FE is started, you can run the ADMIN SHOW FRONTEND CONFIG command on your MySQL client to check the parameter configurations. If you want to query the configuration of a specific parameter, run the following command: ```sql ADMIN SHOW FRONTEND CONFIG [LIKE "pattern"]; ``` For detailed description of the returned fields, see [`ADMIN SHOW CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). note You must have administrator privileges to run cluster administration-related commands. #### Configure FE parameters[​](#configure-fe-parameters "Direct link to Configure FE parameters") ##### Configure FE dynamic parameters[​](#configure-fe-dynamic-parameters "Direct link to Configure FE dynamic parameters") You can configure or modify the settings of FE dynamic parameters using [`ADMIN SET FRONTEND CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md). ```sql ADMIN SET FRONTEND CONFIG ("key" = "value"); ``` note The configuration changes made with `ADMIN SET FRONTEND` will be restored to the default values in the `fe.conf` file after the FE restarts. Therefore, we recommend that you also modify the configuration items in `fe.conf` if you want the changes to be permanent. ##### Configure FE static parameters[​](#configure-fe-static-parameters "Direct link to Configure FE static parameters") note Static parameters of an FE are set by changing them in the configuration file **fe.conf** and restarting the FE to allow the changes to take effect. *** This topic introduces the following types of FE configurations: * [Statistic report](#statistic-report) * [Storage](#storage) #### Statistic report[​](#statistic-report "Direct link to Statistic report") ##### `enable_collect_warehouse_metrics`[​](#enable_collect_warehouse_metrics "Direct link to enable_collect_warehouse_metrics") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When this item is set to `true`, the system will collect and export per-warehouse metrics. Enabling it adds warehouse-level metrics (slot/usage/availability) to the metric output and increases metric cardinality and collection overhead. Disable it to omit warehouse-specific metrics and reduce CPU/network and monitoring storage cost. * Introduced in: v3.5.0 ##### `enable_http_detail_metrics`[​](#enable_http_detail_metrics "Direct link to enable_http_detail_metrics") * Default: false * Type: boolean * Unit: - * Is mutable: Yes * Description: When true, the HTTP server computes and exposes detailed HTTP worker metrics (notably the `HTTP_WORKER_PENDING_TASKS_NUM` gauge). Enabling this causes the server to iterate over Netty worker executors and call `pendingTasks()` on each `NioEventLoop` to sum pending task counts; when disabled the gauge returns 0 to avoid that cost. This extra collection can be CPU- and latency-sensitive — enable only for debugging or detailed investigation. * Introduced in: v3.2.3 ##### `proc_profile_collect_time_s`[​](#proc_profile_collect_time_s "Direct link to proc_profile_collect_time_s") * Default: 120 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Duration in seconds for a single process profile collection. When `proc_profile_cpu_enable` or `proc_profile_mem_enable` is set to `true`, AsyncProfiler is started, the collector thread sleeps for this duration, then the profiler is stopped and the profile is written. Larger values increase sample coverage and file size but prolong profiler runtime and delay subsequent collections; smaller values reduce overhead but may produce insufficient samples. Ensure this value aligns with retention settings such as `proc_profile_file_retained_days` and `proc_profile_file_retained_size_bytes`. * Introduced in: v3.2.12 ##### `enable_dict_thrash_guard`[​](#enable_dict_thrash_guard "Direct link to enable_dict_thrash_guard") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the global-dictionary thrash guard. A "rolling" low-cardinality column—whose instantaneous distinct value count stays below the dictionary threshold but whose value set keeps rotating (for example, a daily-partitioned column that is reloaded with fresh values)—never trips the cardinality blacklist, yet every load introduces values missing from the current global dictionary and invalidates it. Each invalidation forces a full-table dictionary re-collection, which wastes IO and, in shared-data clusters, contends heavily on the segment metadata cache lock. When this guard is enabled, StarRocks counts how often each column's dictionary is invalidated within `dict_thrash_guard_window_sec`; once a column reaches `dict_thrash_guard_threshold` invalidations, StarRocks forbids collecting that column's global dictionary. The forbid takes effect immediately and is persisted as the table's `no_dict_columns` property, so it survives FE restart and leader failover. To re-enable dictionary collection for a column, run `ALTER TABLE ... ENABLE DICTIONARY (column)`. * Introduced in: v4.2.0 ##### `dict_thrash_guard_window_sec`[​](#dict_thrash_guard_window_sec "Direct link to dict_thrash_guard_window_sec") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The length of the time window (in seconds) over which the global-dictionary thrash guard counts how often a column's dictionary is invalidated. This parameter takes effect only when `enable_dict_thrash_guard` is set to `true`. * Introduced in: v4.2.0 ##### `dict_thrash_guard_threshold`[​](#dict_thrash_guard_threshold "Direct link to dict_thrash_guard_threshold") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of dictionary invalidations within `dict_thrash_guard_window_sec` at which the global-dictionary thrash guard forbids collecting a column's global dictionary. Set to `0` to disable the count check while keeping the guard enabled (no column is automatically forbidden). This parameter takes effect only when `enable_dict_thrash_guard` is set to `true`. * Introduced in: v4.2.0 ##### `min_max_stats_collect_interval_sec`[​](#min_max_stats_collect_interval_sec "Direct link to min_max_stats_collect_interval_sec") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Minimum interval between two min/max statistics collections for the same column. Min/max stats (used to constant-fold `min()`/`max()` and to build compressed group-by keys) are collected on demand via a `[_META_]` MetaScan that reads every segment's zone-map metadata; without throttling a frequently loaded column re-scans on every load, contending on the segment metadata cache the same way global-dictionary re-collection does. Within the interval the min/max optimization is skipped rather than re-collected -- a stale value is never served, so this only affects the optimization, never correctness. Set to `0` to disable throttling. * Introduced in: v4.2.0 ##### `enable_external_predicate_columns_collection`[​](#enable_external_predicate_columns_collection "Direct link to enable_external_predicate_columns_collection") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to record predicate column usage (columns used in WHERE/JOIN/GROUP BY) for external (non-native) tables during query optimization. StarRocks uses this usage information to narrow down which columns ANALYZE collects statistics for on wide external tables. When disabled, external table predicate columns are not recorded, and ANALYZE falls back to collecting statistics for all columns. * Introduced in: v4.2.0 ##### `statistic_external_predicate_columns_ttl_hours`[​](#statistic_external_predicate_columns_ttl_hours "Direct link to statistic_external_predicate_columns_ttl_hours") * Default: 168 * Type: Long * Unit: Hours * Is mutable: Yes * Description: The time-to-live (TTL) of recorded external table predicate column usage. Entries whose `last_used` timestamp is older than this value are removed by the periodic vacuum job. Set to a negative value (e.g. -1) to disable vacuum. Defaults to a week because external table ANALYZE runs far less frequently than for internal tables, so a short TTL (matching the internal table's 24-hour default) would evict usage information between two collections. * Introduced in: v4.2.0 ##### `statistic_external_predicate_columns_cache_ttl_sec`[​](#statistic_external_predicate_columns_cache_ttl_sec "Direct link to statistic_external_predicate_columns_cache_ttl_sec") * Default: 300 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The TTL of the in-memory cache that serves external table predicate column queries (for example, during automatic ANALYZE column selection). A shorter value makes newly recorded usage visible sooner but increases the query load on the underlying storage table; a longer value reduces that load at the cost of staleness. * Introduced in: v4.2.0 ##### `enable_temporary_table_statistic_collect`[​](#enable_temporary_table_statistic_collect "Direct link to enable_temporary_table_statistic_collect") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether analyze jobs collect statistics for temporary tables. It applies to the collection jobs built from analyze job definitions, that is, the automatic analyze jobs and the jobs created with CREATE ANALYZE. When this item is set to `false`, temporary tables are skipped while those jobs are built, and only non-temporary tables are analyzed. A one-off `ANALYZE TABLE` statement is not affected and still collects statistics for a temporary table. Set it to `false` when short-lived temporary tables generate collection work whose cost outweighs the value of their statistics. * Introduced in: v3.4.0 #### Storage[​](#storage "Direct link to Storage") ##### `allow_implicit_key_column_in_agg_add_column`[​](#allow_implicit_key_column_in_agg_add_column "Direct link to allow_implicit_key_column_in_agg_add_column") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether `ALTER TABLE ... ADD COLUMN` on an Aggregate table may create a key column when the new column specifies neither an aggregate function nor the `KEY` keyword. Such a statement is ambiguous, and creating a key column changes the table's aggregation key and rewrites existing data. When set to `true`, the column is created as a key column, which is the behavior in earlier versions. Set to `false` to reject the statement instead, so that the error names both options. This item is mutable but is not persisted across a restart unless it is set with `WITH PERSISTENT`. * Introduced in: v4.1.5 ##### `alter_table_timeout_second`[​](#alter_table_timeout_second "Direct link to alter_table_timeout_second") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for the schema change operation (ALTER TABLE). * Introduced in: - ##### `capacity_used_percent_high_water`[​](#capacity_used_percent_high_water "Direct link to capacity_used_percent_high_water") * Default: 0.75 * Type: double * Unit: Fraction (0.0–1.0) * Is mutable: Yes * Description: The high-water threshold of disk capacity used percent (fraction of total capacity) used when computing backend load scores. `BackendLoadStatistic.calcSore` uses `capacity_used_percent_high_water` to set `LoadScore.capacityCoefficient`: if a backend's used percent less than 0.5 the coefficient equal to 0.5; if used percent `>` `capacity_used_percent_high_water` the coefficient = 1.0; otherwise the coefficient transitions linearly with used percent via (2 \* usedPercent - 0.5). When the coefficient is 1.0, the load score is driven entirely by capacity proportion; lower values increase the weight of replica count. Adjusting this value changes how aggressively the balancer penalizes backends with high disk utilization. * Introduced in: v3.2.0 ##### `catalog_trash_expire_second`[​](#catalog_trash_expire_second "Direct link to catalog_trash_expire_second") * Default: 86400 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The longest duration the metadata can be retained after a database, table, or partition is dropped. If this duration expires, the data will be deleted and cannot be recovered through the RECOVER command. * Introduced in: - ##### `catalog_recycle_bin_erase_min_latency_ms`[​](#catalog_recycle_bin_erase_min_latency_ms "Direct link to catalog_recycle_bin_erase_min_latency_ms") * Default: 600000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The minimum delay in milliseconds before the metadata is erased when a database, table, or partition is dropped. This avoids the erase log being written ahead of the drop log. * Introduced in: - ##### `catalog_recycle_bin_erase_max_operations_per_cycle`[​](#catalog_recycle_bin_erase_max_operations_per_cycle "Direct link to catalog_recycle_bin_erase_max_operations_per_cycle") * Default: 500 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of erase operations per cycle for actually deleting databases, tables, or partitions from the recycle bin. The erase operation holds a lock, so one batch should not be too large. * Introduced in: - ##### `catalog_recycle_bin_erase_fail_retry_interval_ms`[​](#catalog_recycle_bin_erase_fail_retry_interval_ms "Direct link to catalog_recycle_bin_erase_fail_retry_interval_ms") * Default: 60000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The retry interval in milliseconds when an erase operation in the recycle bin fails. * Introduced in: - ##### `check_consistency_default_timeout_second`[​](#check_consistency_default_timeout_second "Direct link to check_consistency_default_timeout_second") * Default: 600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for a replica consistency check. You can set this parameter based on the size of your tablet. * Introduced in: - ##### `consistency_check_cooldown_time_second`[​](#consistency_check_cooldown_time_second "Direct link to consistency_check_cooldown_time_second") * Default: 24 \* 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Controls the minimal interval (in seconds) required between consistency checks of the same tablet. During tablet selection, a tablet is considered eligible only if `tablet.getLastCheckTime()` is less than `(currentTimeMillis - consistency_check_cooldown_time_second * 1000)`. The default value (24 \* 3600) enforces roughly one check per tablet per day to reduce backend disk I/O. Lowering this value increases check frequency and resource usage; raising it reduces I/O at the cost of slower detection of inconsistencies. The value is applied globally when filtering cooldowned tablets from an index's tablet list. * Introduced in: v3.5.5 ##### `consistency_check_end_time`[​](#consistency_check_end_time "Direct link to consistency_check_end_time") * Default: "4" * Type: String * Unit: Hour of day (0-23) * Is mutable: No * Description: Specifies the end hour (hour-of-day) of the ConsistencyChecker work window. The value is parsed with SimpleDateFormat("HH") in the system time zone and accepted as 0–23 (single or two-digit). StarRocks uses it with `consistency_check_start_time` to decide when to schedule and add consistency-check jobs. When `consistency_check_start_time` is greater than `consistency_check_end_time`, the window spans midnight (for example, default is `consistency_check_start_time` = "23" to `consistency_check_end_time` = "4"). When `consistency_check_start_time` is equal to `consistency_check_end_time`, the checker never runs. Parsing failure will cause FE startup to log an error and exit, so provide a valid hour string. * Introduced in: v3.2.0 ##### `consistency_check_start_time`[​](#consistency_check_start_time "Direct link to consistency_check_start_time") * Default: "23" * Type: String * Unit: Hour of day (00-23) * Is mutable: No * Description: Specifies the start hour (hour-of-day) of the ConsistencyChecker work window. The value is parsed with SimpleDateFormat("HH") in the system time zone and accepted as 0–23 (single or two-digit). StarRocks uses it with `consistency_check_end_time` to decide when to schedule and add consistency-check jobs. When `consistency_check_start_time` is greater than `consistency_check_end_time`, the window spans midnight (for example, default is `consistency_check_start_time` = "23" to `consistency_check_end_time` = "4"). When `consistency_check_start_time` is equal to `consistency_check_end_time`, the checker never runs. Parsing failure will cause FE startup to log an error and exit, so provide a valid hour string. * Introduced in: v3.2.0 ##### `consistency_tablet_meta_check_interval_ms`[​](#consistency_tablet_meta_check_interval_ms "Direct link to consistency_tablet_meta_check_interval_ms") * Default: 2 \* 3600 \* 1000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Interval used by the ConsistencyChecker to run a full tablet-meta consistency scan between `TabletInvertedIndex` and `LocalMetastore`. The daemon in `runAfterCatalogReady` triggers checkTabletMetaConsistency when `current time - lastTabletMetaCheckTime` exceeds this value. When an invalid tablet is first detected, its `toBeCleanedTime` is set to `now + (consistency_tablet_meta_check_interval_ms / 2)` so actual deletion is delayed until a subsequent scan. Increase this value to reduce scan frequency and load (slower cleanup); decrease it to detect and remove stale tablets faster (higher overhead). * Introduced in: v3.2.0 ##### `default_replication_num`[​](#default_replication_num "Direct link to default_replication_num") * Default: 3 * Type: Short * Unit: - * Is mutable: Yes * Description: Sets the default number of replicas for each data partition when creating a table in StarRocks. This setting can be overridden when creating a table by specifying `replication_num=x` in the CREATE TABLE DDL. * Introduced in: - ##### `enable_auto_tablet_distribution`[​](#enable_auto_tablet_distribution "Direct link to enable_auto_tablet_distribution") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to automatically set the number of buckets. * If this parameter is set to `TRUE`, you don't need to specify the number of buckets when you create a table or add a partition. StarRocks automatically determines the number of buckets. * If this parameter is set to `FALSE`, you need to manually specify the number of buckets when you create a table or add a partition. If you do not specify the bucket count when adding a new partition to a table, the new partition inherits the bucket count set at the creation of the table. However, you can also manually specify the number of buckets for the new partition. * Introduced in: v2.5.7 ##### `enable_experimental_rowstore`[​](#enable_experimental_rowstore "Direct link to enable_experimental_rowstore") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the hybrid row-column storage feature. * Introduced in: v3.2.3 ##### `enable_fast_schema_evolution`[​](#enable_fast_schema_evolution "Direct link to enable_fast_schema_evolution") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Fast Schema Evolution for all tables within the StarRocks cluster. Valid values are `TRUE` and `FALSE` (default). Enabling Fast Schema Evolution can increase the speed of schema changes and reduce resource usage when columns are added or dropped. * Introduced in: v3.2.0 > **NOTE** > > * StarRocks shared-data clusters supports this parameter from v3.3.0. > * If you need to configure the Fast Schema Evolution for a specific table, such as disabling Fast Schema Evolution for a specific table, you can set the table property `fast_schema_evolution` at table creation. ##### `enable_online_optimize_table`[​](#enable_online_optimize_table "Direct link to enable_online_optimize_table") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls whether StarRocks will use the non-blocking online optimization path when creating an optimize job. When `enable_online_optimize_table` is true and the target table meets compatibility checks (no partition/keys/sort specification, distribution is not `RandomDistributionDesc`, storage type is not `COLUMN_WITH_ROW`, replicated storage enabled, and the table is not a cloud-native table or materialized view), the planner creates an `OnlineOptimizeJobV2` to perform optimization without blocking writes. If false or any compatibility condition fails, StarRocks falls back to `OptimizeJobV2`, which may block write operations during optimization. * Introduced in: v3.3.3, v3.4.0, v3.5.0 ##### `enable_strict_storage_medium_check`[​](#enable_strict_storage_medium_check "Direct link to enable_strict_storage_medium_check") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether the FE strictly checks the storage medium of BEs when users create tables. If this parameter is set to `TRUE`, the FE checks the storage medium of BEs when users create tables and returns an error if the storage medium of the BE is different from the `storage_medium` parameter specified in the CREATE TABLE statement. For example, the storage medium specified in the CREATE TABLE statement is SSD but the actual storage medium of BEs is HDD. As a result, the table creation fails. If this parameter is `FALSE`, the FE does not check the storage medium of BEs when users create a table. * Introduced in: - ##### `max_bucket_number_per_partition`[​](#max_bucket_number_per_partition "Direct link to max_bucket_number_per_partition") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of buckets can be created in a partition. * Introduced in: v3.3.2 ##### `max_column_number_per_table`[​](#max_column_number_per_table "Direct link to max_column_number_per_table") * Default: 10000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of columns can be created in a table. * Introduced in: v3.3.2 ##### `max_dynamic_partition_num`[​](#max_dynamic_partition_num "Direct link to max_dynamic_partition_num") * Default: 500 * Type: Int * Unit: - * Is mutable: Yes * Description: Limits the maximum number of partitions that can be created at once when analyzing or creating a dynamic-partitioned table. During dynamic partition property validation, the `systemtask_runs_max_history_number` computes expected partitions (end offset + history partition number) and throws a DDL error if that total exceeds `max_dynamic_partition_num`. Raise this value only when you expect legitimately large partition ranges; increasing it allows more partitions to be created but can increase metadata size, scheduling work, and operational complexity. * Introduced in: v3.2.0 ##### `max_partition_number_per_table`[​](#max_partition_number_per_table "Direct link to max_partition_number_per_table") * Default: 100000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of partitions can be created in a table. * Introduced in: v3.3.2 ##### `max_task_consecutive_fail_count`[​](#max_task_consecutive_fail_count "Direct link to max_task_consecutive_fail_count") * Default: 10 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of consecutive failures a task may have before the scheduler automatically suspends it. When `TaskSource.MV.equals(task.getSource())` and `max_task_consecutive_fail_count` are greater than 0, if a task's consecutive failure counter reaches or exceeds `max_task_consecutive_fail_count`, the task is suspended via the TaskManager and, for materialized-view tasks, the materialized view is inactivated. An exception is thrown indicating suspension and how to reactivate (for example, `ALTER MATERIALIZED VIEW ACTIVE`). Set this item to 0 or a negative value to disable automatic suspension. * Introduced in: - ##### `partition_recycle_retention_period_secs`[​](#partition_recycle_retention_period_secs "Direct link to partition_recycle_retention_period_secs") * Default: 1800 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The metadata retention time for the partition that is dropped by INSERT OVERWRITE or materialized view refresh operations. Note that such metadata cannot be recovered by executing RECOVER. * Introduced in: v3.5.9 ##### `recover_with_empty_tablet`[​](#recover_with_empty_tablet "Direct link to recover_with_empty_tablet") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to replace a lost or corrupted tablet replica with an empty one. If a tablet replica is lost or corrupted, data queries on this tablet or other healthy tablets may fail. Replacing the lost or corrupted tablet replica with an empty tablet ensures that the query can still be executed. However, the result may be incorrect because data is lost. The default value is `FALSE`, which means lost or corrupted tablet replicas are not replaced with empty ones, and the query fails. * Introduced in: - ##### `storage_usage_hard_limit_percent`[​](#storage_usage_hard_limit_percent "Direct link to storage_usage_hard_limit_percent") * Default: 95 * Alias: `storage_flood_stage_usage_percent` * Type: Int * Unit: - * Is mutable: Yes * Description: Hard limit of the storage usage percentage in a BE directory. If the storage usage (in percentage) of the BE storage directory exceeds this value and the remaining storage space is less than `storage_usage_hard_limit_reserve_bytes`, Load and Restore jobs are rejected. You need to set this item together with the BE configuration item `storage_flood_stage_usage_percent` to allow the configurations to take effect. * Introduced in: - ##### `storage_usage_hard_limit_reserve_bytes`[​](#storage_usage_hard_limit_reserve_bytes "Direct link to storage_usage_hard_limit_reserve_bytes") * Default: 100 \* 1024 \* 1024 \* 1024 * Alias: `storage_flood_stage_left_capacity_bytes` * Type: Long * Unit: Bytes * Is mutable: Yes * Description: Hard limit of the remaining storage space in a BE directory. If the remaining storage space in the BE storage directory is less than this value and the storage usage (in percentage) exceeds `storage_usage_hard_limit_percent`, Load and Restore jobs are rejected. You need to set this item together with the BE configuration item `storage_flood_stage_left_capacity_bytes` to allow the configurations to take effect. * Introduced in: - ##### `storage_usage_soft_limit_percent`[​](#storage_usage_soft_limit_percent "Direct link to storage_usage_soft_limit_percent") * Default: 90 * Alias: `storage_high_watermark_usage_percent` * Type: Int * Unit: - * Is mutable: Yes * Description: Soft limit of the storage usage percentage in a BE directory. If the storage usage (in percentage) of the BE storage directory exceeds this value and the remaining storage space is less than `storage_usage_soft_limit_reserve_bytes`, tablets cannot be cloned into this directory. * Introduced in: - ##### `storage_usage_soft_limit_reserve_bytes`[​](#storage_usage_soft_limit_reserve_bytes "Direct link to storage_usage_soft_limit_reserve_bytes") * Default: 200 \* 1024 \* 1024 \* 1024 * Alias: `storage_min_left_capacity_bytes` * Type: Long * Unit: Bytes * Is mutable: Yes * Description: Soft limit of the remaining storage space in a BE directory. If the remaining storage space in the BE storage directory is less than this value and the storage usage (in percentage) exceeds `storage_usage_soft_limit_percent`, tablets cannot be cloned into this directory. * Introduced in: - ##### `tablet_checker_lock_time_per_cycle_ms`[​](#tablet_checker_lock_time_per_cycle_ms "Direct link to tablet_checker_lock_time_per_cycle_ms") * Default: 1000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The maximum lock hold time per cycle for tablet checker before releasing and reacquiring the table lock. Values less than 100 will be treated as 100. * Introduced in: v3.5.9, v4.0.2 ##### `tablet_create_timeout_second`[​](#tablet_create_timeout_second "Direct link to tablet_create_timeout_second") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for creating a tablet. The default value is changed from 1 to 10 from v3.1 onwards. * Introduced in: - ##### `tablet_delete_timeout_second`[​](#tablet_delete_timeout_second "Direct link to tablet_delete_timeout_second") * Default: 2 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for deleting a tablet. * Introduced in: - ##### `tablet_sched_balance_load_disk_safe_threshold`[​](#tablet_sched_balance_load_disk_safe_threshold "Direct link to tablet_sched_balance_load_disk_safe_threshold") * Default: 0.5 * Alias: `balance_load_disk_safe_threshold` * Type: Double * Unit: - * Is mutable: Yes * Description: The percentage threshold for determining whether the disk usage of BEs is balanced. If the disk usage of all BEs is lower than this value, it is considered balanced. If the disk usage is greater than this value and the difference between the highest and lowest BE disk usage is greater than 10%, the disk usage is considered unbalanced and a tablet re-balancing is triggered. * Introduced in: - ##### `tablet_sched_balance_load_score_threshold`[​](#tablet_sched_balance_load_score_threshold "Direct link to tablet_sched_balance_load_score_threshold") * Default: 0.1 * Alias: `balance_load_score_threshold` * Type: Double * Unit: - * Is mutable: Yes * Description: The percentage threshold for determining whether the load of a BE is balanced. If a BE has a lower load than the average load of all BEs and the difference is greater than this value, this BE is in a low load state. On the contrary, if a BE has a higher load than the average load and the difference is greater than this value, this BE is in a high load state. * Introduced in: - ##### `tablet_sched_be_down_tolerate_time_s`[​](#tablet_sched_be_down_tolerate_time_s "Direct link to tablet_sched_be_down_tolerate_time_s") * Default: 900 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The maximum duration the scheduler allows for a BE node to remain inactive. After the time threshold is reached, tablets on that BE node will be migrated to other active BE nodes. * Introduced in: v2.5.7 ##### `tablet_sched_disable_balance`[​](#tablet_sched_disable_balance "Direct link to tablet_sched_disable_balance") * Default: false * Alias: `disable_balance` * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to disable tablet balancing. `TRUE` indicates that tablet balancing is disabled. `FALSE` indicates that tablet balancing is enabled. * Introduced in: - ##### `tablet_sched_disable_colocate_balance`[​](#tablet_sched_disable_colocate_balance "Direct link to tablet_sched_disable_colocate_balance") * Default: false * Alias: `disable_colocate_balance` * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to disable replica balancing for Colocate Table. `TRUE` indicates replica balancing is disabled. `FALSE` indicates replica balancing is enabled. * Introduced in: - ##### `tablet_sched_max_balancing_tablets`[​](#tablet_sched_max_balancing_tablets "Direct link to tablet_sched_max_balancing_tablets") * Default: 500 * Alias: `max_balancing_tablets` * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of tablets that can be balanced at the same time. If this value is exceeded, tablet re-balancing will be skipped. * Introduced in: - ##### `tablet_sched_max_clone_task_timeout_sec`[​](#tablet_sched_max_clone_task_timeout_sec "Direct link to tablet_sched_max_clone_task_timeout_sec") * Default: 2 \* 60 \* 60 * Alias: `max_clone_task_timeout_sec` * Type: Long * Unit: Seconds * Is mutable: Yes * Description :The maximum timeout duration for cloning a tablet. * Introduced in: - ##### `tablet_sched_max_not_being_scheduled_interval_ms`[​](#tablet_sched_max_not_being_scheduled_interval_ms "Direct link to tablet_sched_max_not_being_scheduled_interval_ms") * Default: 15 \* 60 \* 1000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: When the tablet clone tasks are being scheduled, if a tablet has not been scheduled for the specified time in this parameter, StarRocks gives it a higher priority to schedule it as soon as possible. * Introduced in: - ##### `tablet_sched_max_scheduling_tablets`[​](#tablet_sched_max_scheduling_tablets "Direct link to tablet_sched_max_scheduling_tablets") * Default: 10000 * Alias: `max_scheduling_tablets` * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of tablets that can be scheduled at the same time. If the value is exceeded, tablet balancing and repair checks will be skipped. * Introduced in: - ##### `tablet_sched_min_clone_task_timeout_sec`[​](#tablet_sched_min_clone_task_timeout_sec "Direct link to tablet_sched_min_clone_task_timeout_sec") * Default: 3 \* 60 * Alias: `min_clone_task_timeout_sec` * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The minimum timeout duration for cloning a tablet. * Introduced in: - ##### `tablet_sched_num_based_balance_threshold_ratio`[​](#tablet_sched_num_based_balance_threshold_ratio "Direct link to tablet_sched_num_based_balance_threshold_ratio") * Default: 0.5 * Alias: - * Type: Double * Unit: - * Is mutable: Yes * Description: Doing num based balance may break the disk size balance, but the maximum gap between disks cannot exceed `tablet_sched_num_based_balance_threshold_ratio` \* `tablet_sched_balance_load_score_threshold`. If there are tablets in the cluster that are constantly balancing from A to B and B to A, reduce this value. If you want the tablet distribution to be more balanced, increase this value. * Introduced in: - 3.1 ##### `tablet_sched_repair_delay_factor_second`[​](#tablet_sched_repair_delay_factor_second "Direct link to tablet_sched_repair_delay_factor_second") * Default: 60 * Alias: `tablet_repair_delay_factor_second` * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The interval at which replicas are repaired, in seconds. * Introduced in: - ##### `tablet_sched_slot_num_per_path`[​](#tablet_sched_slot_num_per_path "Direct link to tablet_sched_slot_num_per_path") * Default: 8 * Alias: `schedule_slot_num_per_path` * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of tablet-related tasks that can run concurrently in a BE storage directory. From v2.5 onwards, the default value of this parameter is changed from `4` to `8`. * Introduced in: - ##### `tablet_sched_storage_cooldown_second`[​](#tablet_sched_storage_cooldown_second "Direct link to tablet_sched_storage_cooldown_second") * Default: -1 * Alias: `storage_cooldown_second` * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The latency of automatic cooling starting from the time of table creation. The default value `-1` specifies that automatic cooling is disabled. If you want to enable automatic cooling, set this parameter to a value greater than `-1`. * Introduced in: - ##### `tablet_stat_update_interval_second`[​](#tablet_stat_update_interval_second "Direct link to tablet_stat_update_interval_second") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which the FE retrieves tablet statistics from each BE. * Introduced in: - ##### `enable_range_distribution`[​](#enable_range_distribution "Direct link to enable_range_distribution") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the Range-based Distribution semantic for table creation. * Introduced in: v4.1.0 ##### `tablet_reshard_max_parallel_tablets`[​](#tablet_reshard_max_parallel_tablets "Direct link to tablet_reshard_max_parallel_tablets") * Default: 10240 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of tablets that can be split or merged in parallel. * Introduced in: v4.1.0 ##### `tablet_reshard_target_size`[​](#tablet_reshard_target_size "Direct link to tablet_reshard_target_size") * Default: 10737418240 (10 GB) * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The target size of the tablets after the SPLIT or MERGE operation. * Introduced in: v4.1.0 ##### `tablet_reshard_max_split_count`[​](#tablet_reshard_max_split_count "Direct link to tablet_reshard_max_split_count") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of new tablets that an old tablet can be split into. * Introduced in: v4.1.0 ##### `tablet_reshard_min_split_size`[​](#tablet_reshard_min_split_size "Direct link to tablet_reshard_min_split_size") * Default: 2147483648 (2 GB) * Type: Long * Unit: Bytes * Is mutable: Yes * Description: The minimum size of a tablet produced by tablet pre-split. It bounds compute-node alignment during pre-split so that a small load on a large cluster is not split into many tiny tablets. Should be no larger than `tablet_reshard_target_size`. * Introduced in: v4.1.0 ##### `tablet_reshard_history_job_max_keep_ms`[​](#tablet_reshard_history_job_max_keep_ms "Direct link to tablet_reshard_history_job_max_keep_ms") * Default: 259200000 (72 hours) * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The maximum retention time of historical tablet SPLIT/MERGE jobs. * Introduced in: v4.1.0 ##### `tablet_reshard_colocate_checker_membership_batch_size`[​](#tablet_reshard_colocate_checker_membership_batch_size "Direct link to tablet_reshard_colocate_checker_membership_batch_size") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of tablets that the range-colocate checker sends to StarOS in a single `getShardInfo` membership-read batch RPC on shared-data clusters. Values less than `1` are treated as `1`. * Introduced in: v4.1.3 ##### `tablet_reshard_colocate_checker_convergence_batch_size`[​](#tablet_reshard_colocate_checker_convergence_batch_size "Direct link to tablet_reshard_colocate_checker_convergence_batch_size") * Default: 64 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of PACK shard groups that the range-colocate checker sends to StarOS in a single `queryShardGroupStable` placement-convergence batch RPC. Each group's stability check is computed server-side, so a smaller batch bounds per-RPC latency, and the full result is assembled across repeated calls. Values less than `1` are treated as `1`. * Introduced in: v4.1.3 ##### `tablet_reshard_colocate_checker_convergence_cache_ttl_ms`[​](#tablet_reshard_colocate_checker_convergence_cache_ttl_ms "Direct link to tablet_reshard_colocate_checker_convergence_cache_ttl_ms") * Default: 1000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The TTL of the range-colocate checker's placement-convergence negative cache. Within this window, a PACK shard group that StarOS last reported as not yet converged is not re-queried, which throttles the per-tick `queryShardGroupStable` load while the group is still migrating. Only not-converged results are cached, so a stale entry can only delay the group's flip to stable by up to this window, never cause a premature flip. Values less than or equal to `0` disable the cache. * Introduced in: v4.1.3 ##### `enable_tablet_pre_split_for_insert_from_files`[​](#enable_tablet_pre_split_for_insert_from_files "Direct link to enable_tablet_pre_split_for_insert_from_files") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Sample-Based Tablet Pre-Split for `INSERT INTO ... SELECT FROM FILES()` loads. On by default as of v4.1.0. Set to `false` to disable cluster-wide. The session variable `enable_tablet_pre_split` must also be `true` for pre-split to run. * Introduced in: v4.1.0 ##### `enable_tablet_pre_split_for_broker_load`[​](#enable_tablet_pre_split_for_broker_load "Direct link to enable_tablet_pre_split_for_broker_load") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Sample-Based Tablet Pre-Split for Broker Load. On by default as of v4.1.0. Set to `false` to disable cluster-wide. The session variable `enable_tablet_pre_split` must also be `true` for pre-split to run. * Introduced in: v4.1.0 ##### `enable_tablet_pre_split_for_insert_from_table`[​](#enable_tablet_pre_split_for_insert_from_table "Direct link to enable_tablet_pre_split_for_insert_from_table") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Sample-Based Tablet Pre-Split for `INSERT INTO ... SELECT FROM ` loads whose source is an internal OLAP table or a table in an external catalog, such as Hive, Iceberg, Paimon, Delta Lake, Hudi, JDBC, or Elasticsearch. Views are not supported as a source. When an external source cannot be sized from its table statistics, the load skips pre-split. The feature supports automatic range-partition targets, including explicitly named real or temporary partitions and both static and dynamic `INSERT OVERWRITE`. For `INSERT INTO`, it also supports manually range-partitioned targets, whose existing empty partitions are split without creating any partition. On by default as of v4.1.0. Set to `false` to disable cluster-wide. The session variable `enable_tablet_pre_split` must also be `true` for pre-split to run. To roll back, set to `false`; new INSERT-from-table loads will skip pre-split immediately. * Introduced in: v4.1.0 ##### `tablet_pre_split_pre_submit_timeout_seconds`[​](#tablet_pre_split_pre_submit_timeout_seconds "Direct link to tablet_pre_split_pre_submit_timeout_seconds") * Default: 300 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: Wall-clock budget for the pre-submit phase of Sample-Based Tablet Pre-Split (sample + plan boundaries + build reshard job). On expiry the coordinator skips pre-split and the load proceeds against the original single tablet. Default 300s: the data-tier sampler can take tens of seconds on large datasets / slow object storage (a ~40GB many-file Parquet load sampled in ~78s in testing), and this budget mainly bites large loads — exactly the ones pre-split benefits; small loads sample in well under a second regardless. The load stays `PENDING` for at most this long during sampling, so keep it below the load's own timeout. * Introduced in: v4.1.0 ##### `tablet_pre_split_post_submit_wait_seconds`[​](#tablet_pre_split_post_submit_wait_seconds "Direct link to tablet_pre_split_post_submit_wait_seconds") * Default: 300 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: Maximum time the coordinator will wait for an admitted Sample-Based Tablet Pre-Split reshard job to reach `FINISHED`. Both INSERT-from-FILES and Broker Load synchronously wait and on expiry **proceed without abort** — the load then plans against the currently visible tablet layout (still the original layout if the daemon hasn't transitioned, or partially / fully post-split if the daemon raced past the wait); the `tablet_pre_split_post_submit_hard_cap` counter records the timeout. The strict `runPreSplit` wrapper used by tests aborts the calling load via `PreSplitPostSubmitTimeoutException`. For Broker Load the wait runs after the broker pending task resolves file statuses but before `beginTxn` opens `T_load` — it occupies a `pending_load_task_scheduler` thread for at most this many seconds per table, so size `max_broker_load_job_concurrency` accordingly when many concurrent Broker Loads target a pre-splittable layout. **Operator note:** the Broker Load remains `PENDING` in `SHOW LOAD` during the wait and is still subject to its own `timeoutSecond` — set this well below the smallest Broker Load timeout in normal use. * Introduced in: v4.1.0 ##### `tablet_pre_split_sample_byte_limit`[​](#tablet_pre_split_sample_byte_limit "Direct link to tablet_pre_split_sample_byte_limit") * Default: 16777216 (16 MiB) * Type: Long * Unit: Bytes * Is mutable: Yes * Description: Soft byte cap on the FE-side accumulation buffer of the data-tier reservoir sampler used by Sample-Based Tablet Pre-Split. The sampler stops reading once accumulated values exceed this limit. The first row is always admitted so an oversize row still produces a non-empty sample. * Introduced in: v4.1.0 ##### `tablet_pre_split_data_tier_scan_byte_limit`[​](#tablet_pre_split_data_tier_scan_byte_limit "Direct link to tablet_pre_split_data_tier_scan_byte_limit") * Default: 4294967296 (4 GiB) * Type: Long * Unit: Bytes * Is mutable: Yes * Description: Soft limit on the source-file bytes the data tier of Sample-Based Tablet Pre-Split scans for a Broker Load or `INSERT INTO ... SELECT FROM FILES()` load. When the input is larger, the sample reads a subset of the files instead of every file, so sampling time no longer grows with the input. The exception is a partition column read from the file data rather than from the path, or path and literal partition columns mixed: every file is then still scanned, because a subset could miss whole partitions. Files are sorted by path and picked at even byte intervals, which favors larger files. When the partition column comes from the file path (`COLUMNS FROM PATH` or `columns_from_path`), each partition gets a share of the limit in proportion to its bytes and at least one file (unless the load has more partitions than `tablet_pre_split_data_tier_max_scan_files`), and each partition's size is taken from the bytes of all of its files. Files are taken whole, so the scan can exceed the limit, for example when a single file is larger than it. The tablet count is still sized from the whole input. Set to `0` to scan every file. ##### `tablet_pre_split_data_tier_min_scan_files`[​](#tablet_pre_split_data_tier_min_scan_files "Direct link to tablet_pre_split_data_tier_min_scan_files") * Default: 64 * Type: Int * Unit: - * Is mutable: Yes * Description: Minimum number of files the data tier of Sample-Based Tablet Pre-Split scans once it samples a subset of a load's files (see `tablet_pre_split_data_tier_scan_byte_limit`), so that the sample spans enough independent files even when each file is large or holds only a narrow range of the sort key. When the partition column comes from the file path, the minimum is split across partitions in proportion to their bytes. Meeting it can take the scan above `tablet_pre_split_data_tier_scan_byte_limit`, but no files are added for it once the scan reaches four times that limit, so large files do not multiply the scan time. A positive `tablet_pre_split_data_tier_max_scan_files` takes precedence over this minimum. ##### `tablet_pre_split_data_tier_max_scan_files`[​](#tablet_pre_split_data_tier_max_scan_files "Direct link to tablet_pre_split_data_tier_max_scan_files") * Default: 512 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of files the data tier of Sample-Based Tablet Pre-Split scans once it samples a subset of a load's files (see `tablet_pre_split_data_tier_scan_byte_limit`). Every file in the subset costs one metadata lookup before any data is read, so without a cap a subset of many small files could spend minutes on lookups alone. When the partition column comes from the file path, the cap is split across partitions in proportion to their bytes; every partition still keeps at least one file, so the cap can be exceeded by up to one file per partition. When there are more partitions than the cap, only the heaviest partitions are sampled, one file each. The cap takes precedence over `tablet_pre_split_data_tier_min_scan_files`. Set to `0` or a negative value to remove the cap. ##### `tablet_pre_split_meta_tier_overlap_threshold`[​](#tablet_pre_split_meta_tier_overlap_threshold "Direct link to tablet_pre_split_meta_tier_overlap_threshold") * Default: 0.3 * Type: Double * Unit: - * Is mutable: Yes * Description: Maximum overlap fraction tolerated when Sample-Based Tablet Pre-Split's meta tier (Parquet/ORC row-group metadata) computes boundaries. Above this threshold the cumulative-row count stops being monotone in sorted-min order so meta tier falls back to data tier (row sampling). * Introduced in: v4.1.0 ##### `tablet_pre_split_meta_tier_footer_read_parallelism`[​](#tablet_pre_split_meta_tier_footer_read_parallelism "Direct link to tablet_pre_split_meta_tier_footer_read_parallelism") * Default: 16 * Type: Int * Unit: - * Is mutable: Yes * Description: Number of Parquet/ORC footers the Sample-Based Tablet Pre-Split meta tier reads concurrently from a file-backed load source (`FILES()` or Broker Load). Footer reads are independent per file and the sampler sorts the aggregated statistics, so concurrency only cuts the wall time of the pre-split hook (each footer is a remote round-trip; a many-file source otherwise serializes hundreds of round-trips). Set to `1` to disable concurrency. * Introduced in: v4.1.0 ##### `tablet_pre_split_max_partitions_per_load`[​](#tablet_pre_split_max_partitions_per_load "Direct link to tablet_pre_split_max_partitions_per_load") * Default: 32 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of predicted target partitions a single Sample-Based Tablet Pre-Split invocation will operate on. Excess predicted partitions (the lightest ones: fewest input bytes when the data tier sampled a subset of files whose partition values come from the file paths, otherwise fewest sampled rows) are dropped and fall back to runtime auto-create with no pre-split. Bounds hook latency on pathological multi-partition loads. Set to zero or a negative value to disable the cap. * Introduced in: v4.1.0 ##### `tablet_pre_split_target_size`[​](#tablet_pre_split_target_size "Direct link to tablet_pre_split_target_size") * Default: 0 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: Target tablet size that Sample-Based Tablet Pre-Split sizes a load's split count against. `0` (the default) inherits `tablet_reshard_target_size`. Lower it to give a load more write parallelism without shrinking every tablet in the cluster: the background tablet split/merge daemon keeps measuring against `tablet_reshard_target_size`, so it merges the finer tablets back together after the load finishes. This matters most for a load that writes brand-new range-distributed partitions (for example the replacement partitions of an `INSERT OVERWRITE`), which start from a single catch-all tablet and would otherwise be written by a single backend. ###### Rolling back Sample-Based Tablet Pre-Split[​](#rolling-back-sample-based-tablet-pre-split "Direct link to Rolling back Sample-Based Tablet Pre-Split") To disable the feature safely before a downgrade or during a production rollback: 1. Set all three pre-split flags to `false`: `enable_tablet_pre_split_for_insert_from_files`, `enable_tablet_pre_split_for_broker_load`, and `enable_tablet_pre_split_for_insert_from_table`. New loads will skip pre-split immediately. 2. Wait for in-flight reshard jobs created by pre-split to drain. Monitor them with the following query: ```sql SELECT DB_NAME, TABLE_NAME, JOB_TYPE, JOB_STATE FROM information_schema.tablet_reshard_jobs WHERE JOB_STATE NOT IN ('FINISHED', 'ABORTED'); ``` The rollback is complete once this query returns no rows. A job in any non-final state, including `PENDING`, `PREPARING`, `RUNNING`, `CLEANING`, and `ABORTING`, is still in flight; only `FINISHED` and `ABORTED` are final. 3. Proceed with the downgrade. The substrate (External-Boundaries Tablet Split) remains available regardless of the pre-split feature flag. ###### Behavioral notes for multi-partition Sample-Based Tablet Pre-Split (P2-a)[​](#behavioral-notes-for-multi-partition-sample-based-tablet-pre-split-p2-a "Direct link to Behavioral notes for multi-partition Sample-Based Tablet Pre-Split (P2-a)") The multi-partition path extends Sample-Based Tablet Pre-Split to loads that target many partitions in one statement. Three operational caveats apply: * **Broker Load triggering-load asymmetry.** The multi-partition pre-split hook fires from `BrokerLoadJob.createLoadingTask` **after** `task.prepare()` has built the load's sink plan against the catalog as it existed at that moment. For Broker Load, pre-created partitions and the post-reshard tablet layout are therefore only visible to **subsequent** loads on the same table — the triggering Broker Load itself runs against the original layout and uses BE runtime auto-create for any partitions it touches. INSERT-from-FILES (where the hook fires before `StatementPlanner.plan()`) is unaffected and benefits in the same load. * **Pre-created partition leak on subsequent INSERT failure.** When pre-create succeeds and the triggering INSERT later fails for unrelated reasons (FILES schema mismatch, BE crash, load timeout, etc.), the empty pre-created partitions remain in the catalog. This matches the semantics of `ALTER TABLE ADD PARTITION`, which also leaves a partition behind on subsequent failure. Operators who care can drop the empty partitions manually with `ALTER TABLE ... DROP PARTITION`; in practice empty partitions are cheap and the next retry of the load will reuse them. * **Manually range-partitioned targets.** For a table with user-declared RANGE partitions, pre-split never creates a partition. Each sampled row is mapped onto the existing partition whose range contains it, and a value outside every declared range is dropped from the plan. Only partitions that are still empty and hold a single tablet are split, typically a partition just added with `ALTER TABLE ... ADD PARTITION`; a load whose target partitions all hold data already skips pre-split without sampling the source. `INSERT OVERWRITE`, LIST partitioning, and expression partitioning are not covered for manually partitioned tables. ###### Production deployment guidance[​](#production-deployment-guidance "Direct link to Production deployment guidance") Set `enable_execute_script_on_frontend = false` in production. Sample-Based Tablet Pre-Split exposes no SQL surface that depends on FE-side script execution; the production code paths sample through the connector + planner directly. Leaving `enable_execute_script_on_frontend = true` widens the FE attack surface without enabling any pre-split functionality, so the safe default for production clusters is to keep it off. --- ### FE Configuration - Authentication, Query, and Loading FE parameters are classified into dynamic parameters and static parameters. * Dynamic parameters can be configured and adjusted by running SQL commands, which is very convenient. But the configurations become invalid if you restart your FE. Therefore, we recommend that you also modify the configuration items in the **fe.conf** file to prevent the loss of modifications. * Static parameters can only be configured and adjusted in the FE configuration file **fe.conf**. **After you modify this file, you must restart your FE for the changes to take effect.** Whether a parameter is a dynamic parameter is indicated by the `IsMutable` column in the output of [ADMIN SHOW CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). `TRUE` indicates a dynamic parameter. Note that both dynamic and static FE parameters can be configured in the **fe.conf** file. #### View FE configuration items[​](#view-fe-configuration-items "Direct link to View FE configuration items") After your FE is started, you can run the ADMIN SHOW FRONTEND CONFIG command on your MySQL client to check the parameter configurations. If you want to query the configuration of a specific parameter, run the following command: ```sql ADMIN SHOW FRONTEND CONFIG [LIKE "pattern"]; ``` For detailed description of the returned fields, see [`ADMIN SHOW CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). note You must have administrator privileges to run cluster administration-related commands. #### Configure FE parameters[​](#configure-fe-parameters "Direct link to Configure FE parameters") ##### Configure FE dynamic parameters[​](#configure-fe-dynamic-parameters "Direct link to Configure FE dynamic parameters") You can configure or modify the settings of FE dynamic parameters using [`ADMIN SET FRONTEND CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md). ```sql ADMIN SET FRONTEND CONFIG ("key" = "value"); ``` note The configuration changes made with `ADMIN SET FRONTEND` will be restored to the default values in the `fe.conf` file after the FE restarts. Therefore, we recommend that you also modify the configuration items in `fe.conf` if you want the changes to be permanent. ##### Configure FE static parameters[​](#configure-fe-static-parameters "Direct link to Configure FE static parameters") note Static parameters of an FE are set by changing them in the configuration file **fe.conf** and restarting the FE to allow the changes to take effect. *** This topic introduces the following types of FE configurations: * [User, role, and privilege](#user-role-and-privilege) * [Query](#query-engine) * [Loading and unloading](#loading-and-unloading) #### User, role, and privilege[​](#user-role-and-privilege "Direct link to User, role, and privilege") ##### `authorization_enable_query_profile_access_check`[​](#authorization_enable_query_profile_access_check "Direct link to authorization_enable_query_profile_access_check") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to restrict who can read cached query profiles. When set to `true`, SHOW PROFILELIST, ANALYZE PROFILE, the `get_query_profile` function, the `/api/profile` and `/api/query/progress` HTTP endpoints, and the `/query` and `/query_profile` Web UI pages return a profile only to the user who ran the query or to a user with the SYSTEM-level OPERATE privilege. Profiles that record no user (for example, EXPORT jobs and stream loads) are readable only with OPERATE. The `/api/query_detail` and `/api/v2/query_detail` endpoints keep listing every query but omit the `profile` field, and the `explain` field when it holds a plan rendered from that profile, under the same rule. When `authorization_enable_admin_user_protection` is also enabled, profiles of queries run by `root` are readable only by `root`. When set to `false` (the default), every authenticated user can read every profile, which is the behavior of earlier versions. Each FE enforces the check for the profiles it holds, so to restrict access cluster-wide, set this item in `fe.conf` on every FE after all FEs have been upgraded to a version that supports it. Enabling it has two further consequences to plan for: `/api/query/progress` stops accepting anonymous requests, because the rule needs a caller identity, so an existing anonymous poller of that endpoint starts receiving `401`; and `get_query_profile()` requires the OPERATE privilege for a `query_id` whose profile is not cached on the FE the session is connected to, because the RPC that fetches it from the other FEs carries no caller identity to authorize there, which restricts the cross-FE lookup to OPERATE holders while the check is on. * Introduced in: v4.1 ##### `enable_task_info_mask_credential`[​](#enable_task_info_mask_credential "Direct link to enable_task_info_mask_credential") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When true, StarRocks redacts credentials from task SQL definitions before returning them in `information_schema.tasks` and `information_schema.task_runs` by applying SqlCredentialRedactor.redact to the DEFINITION column. In `information_schema.task_runs` the same redaction is applied whether the definition comes from the task run status or, when empty, from the task definition lookup. The same redaction is applied before finished task runs are archived to `_statistics_.task_run_history`, so the archived definition does not contain credentials in clear text. When false, raw task definitions are returned and archived (may expose credentials). Masking is CPU/string-processing work and can be time-consuming when the number of tasks or `task_runs` is large; disable only if you need unredacted definitions and accept the security risk. * Introduced in: v3.5.6 ##### `privilege_max_role_depth`[​](#privilege_max_role_depth "Direct link to privilege_max_role_depth") * Default: 16 * Type: Int * Unit: * Is mutable: Yes * Description: The maximum role depth (level of inheritance) of a role. * Introduced in: v3.0.0 ##### `privilege_max_total_roles_per_user`[​](#privilege_max_total_roles_per_user "Direct link to privilege_max_total_roles_per_user") * Default: 64 * Type: Int * Unit: * Is mutable: Yes * Description: The maximum number of roles a user can have. * Introduced in: v3.0.0 #### Query engine[​](#query-engine "Direct link to Query engine") ##### `brpc_send_plan_fragment_timeout_ms`[​](#brpc_send_plan_fragment_timeout_ms "Direct link to brpc_send_plan_fragment_timeout_ms") * Default: 60000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Timeout in milliseconds applied to the BRPC TalkTimeoutController before sending a plan fragment. `BackendServiceClient.sendPlanFragmentAsync` sets this value prior to calling the backend `execPlanFragmentAsync`. It governs how long BRPC will wait when borrowing an idle connection from the connection pool and while performing the send; if exceeded, the RPC will fail and may trigger the method's retry logic. Set this lower to fail fast under contention, or raise it to tolerate transient pool exhaustion or slow networks. Be cautious: very large values can delay failure detection and block request threads. * Introduced in: v3.3.11, v3.4.1, v3.5.0 ##### `connector_row_size_estimate_bytes`[​](#connector_row_size_estimate_bytes "Direct link to connector_row_size_estimate_bytes") * Default: 256 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: The estimated average row size in bytes used by the optimizer to estimate row counts for external file tables (FILES() and ENGINE=file tables) when the storage format is unknown or the column schema is not available. The row count is estimated as `total_file_bytes / connector_row_size_estimate_bytes`. A smaller value produces a higher row count estimate and may affect join ordering decisions. * Introduced in: v3.4 ##### `connector_table_analyze_scan_bytes_cap`[​](#connector_table_analyze_scan_bytes_cap "Direct link to connector_table_analyze_scan_bytes_cap") * Default: 2147483648 (2 GB) * Type: Long * Unit: Bytes * Is mutable: Yes * Description: The primary per-scan byte budget for external-table statistics collection (Iceberg). Each per-(partition, column) statistics scan accumulates the byte size of the splits it opens and stops early once this budget is reached (a soft cap: the last split may overshoot), so an oversized single partition or unpartitioned table is collected as a bounded, degraded sample instead of failing or timing out. A value of `0` or less means this dimension is unlimited. Calibrate against the number of columns collected, because a partition runs one independent scan per column. Set this and `connector_table_analyze_scan_files_cap` and `connector_table_analyze_scan_rows_cap` all to `0` or less to disable bounded-cost collection entirely and fall back to full scans. Can be overridden per statement with `ANALYZE TABLE ... PROPERTIES("scan_bytes_cap" = "...")`. * Introduced in: v4.1 ##### `connector_table_analyze_scan_files_cap`[​](#connector_table_analyze_scan_files_cap "Direct link to connector_table_analyze_scan_files_cap") * Default: 1000 * Type: Long * Unit: - * Is mutable: Yes * Description: The secondary per-scan file-count budget for external-table statistics collection (Iceberg). A statistics scan stops early once it has opened this many files, which caps the cost of a partition made up of a very large number of small files. A value of `0` or less means this dimension is unlimited. Can be overridden per statement with `ANALYZE TABLE ... PROPERTIES("scan_files_cap" = "...")`. * Introduced in: v4.1 ##### `connector_table_analyze_scan_rows_cap`[​](#connector_table_analyze_scan_rows_cap "Direct link to connector_table_analyze_scan_rows_cap") * Default: 10000000 * Type: Long * Unit: - * Is mutable: Yes * Description: The auxiliary per-scan estimated-row budget for external-table statistics collection (Iceberg). A statistics scan stops early once the estimated number of rows it has scanned reaches this budget. Because per-split row counts can only be estimated (the record count is recorded per file, not per split), this is an auxiliary soft budget rather than the primary control. The default aligns with `connector_table_query_trigger_analyze_small_table_rows`. A value of `0` or less means this dimension is unlimited. Can be overridden per statement with `ANALYZE TABLE ... PROPERTIES("scan_rows_cap" = "...")`. * Introduced in: v4.1 ##### `connector_table_analyze_query_timeout`[​](#connector_table_analyze_query_timeout "Direct link to connector_table_analyze_query_timeout") * Default: 360 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The maximum time a single external-table statistics collection query may run. Statistics collection issues one query per (partition, column group); each of them reads one partition under the scan caps above into fixed-size sketches and normally returns in well under a second. Without this ceiling every such query inherits whatever is left of `statistic_collect_query_timeout`, which is the budget for the whole collection job, so one query stuck behind an overloaded backend can consume the entire budget and leave every remaining partition uncollected. Raise this for cold or distant object storage where a capped scan can legitimately take longer. A value of `0` or less removes the separate ceiling, so a query may again use the job's whole remaining budget. The effective timeout is always the smaller of this value and the job's remaining budget. * Introduced in: v4.1 ##### `connector_table_query_trigger_analyze_large_table_interval`[​](#connector_table_query_trigger_analyze_large_table_interval "Direct link to connector_table_query_trigger_analyze_large_table_interval") * Default: 12 \* 3600 * Type: Int * Unit: Second * Is mutable: Yes * Description: The interval for query-trigger ANALYZE tasks of large tables. * Introduced in: v3.4.0 ##### `connector_table_query_trigger_analyze_max_pending_task_num`[​](#connector_table_query_trigger_analyze_max_pending_task_num "Direct link to connector_table_query_trigger_analyze_max_pending_task_num") * Default: 100 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of query-trigger ANALYZE tasks that are in Pending state on the FE. * Introduced in: v3.4.0 ##### `connector_table_query_trigger_analyze_max_running_task_num`[​](#connector_table_query_trigger_analyze_max_running_task_num "Direct link to connector_table_query_trigger_analyze_max_running_task_num") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of query-trigger ANALYZE tasks that are in Running state on the FE. * Introduced in: v3.4.0 ##### `connector_table_query_trigger_analyze_small_table_interval`[​](#connector_table_query_trigger_analyze_small_table_interval "Direct link to connector_table_query_trigger_analyze_small_table_interval") * Default: 2 \* 3600 * Type: Int * Unit: Second * Is mutable: Yes * Description: The interval for query-trigger ANALYZE tasks of small tables. * Introduced in: v3.4.0 ##### `connector_table_query_trigger_analyze_small_table_rows`[​](#connector_table_query_trigger_analyze_small_table_rows "Direct link to connector_table_query_trigger_analyze_small_table_rows") * Default: 10000000 * Type: Int * Unit: - * Is mutable: Yes * Description: The threshold for determining whether a table is a small table for query-trigger ANALYZE tasks. * Introduced in: v3.4.0 ##### `connector_table_query_trigger_task_schedule_interval`[​](#connector_table_query_trigger_task_schedule_interval "Direct link to connector_table_query_trigger_task_schedule_interval") * Default: 30 * Type: Int * Unit: Second * Is mutable: Yes * Description: The interval at which the Scheduler thread schedules the query-trigger background tasks. This item is to replace `connector_table_query_trigger_analyze_schedule_interval` introduced in v3.4.0. Here, the background tasks refer `ANALYZE` tasks in v3.4，and the collection task of low-cardinality columns' dictionary in versions later than v3.4. * Introduced in: v3.4.2 ##### `create_table_max_serial_replicas`[​](#create_table_max_serial_replicas "Direct link to create_table_max_serial_replicas") * Default: 128 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of replicas to create serially. If actual replica count exceeds this value, replicas will be created concurrently. Try to reduce this value if table creation is taking a long time to complete. * Introduced in: - ##### `default_mv_partition_refresh_number`[​](#default_mv_partition_refresh_number "Direct link to default_mv_partition_refresh_number") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: When a materialized view refresh involves multiple partitions, this parameter controls how many partitions are refreshed in a single batch by default. Starting from version 3.3.0, the system defaults to refreshing one partition at a time to avoid potential out-of-memory (OOM) issues. In earlier versions, all partitions were refreshed at once by default, which could lead to memory exhaustion and task failure. However, note that when a materialized view refresh involves a large number of partitions, refreshing only one partition at a time may lead to excessive scheduling overhead, longer overall refresh time, and a large number of refresh records. In such cases, it is recommended to adjust this parameter appropriately to improve refresh efficiency and reduce scheduling costs. * Introduced in: v3.3.0 ##### `default_mv_refresh_immediate`[​](#default_mv_refresh_immediate "Direct link to default_mv_refresh_immediate") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to refresh an asynchronous materialized view immediately after creation. When this item is set to `true`, newly created materialized view will be refreshed immediately. * Introduced in: v3.2.3 ##### `dynamic_partition_check_interval_seconds`[​](#dynamic_partition_check_interval_seconds "Direct link to dynamic_partition_check_interval_seconds") * Default: 600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The interval at which new data is checked. If new data is detected, StarRocks automatically creates partitions for the data. * Introduced in: - ##### `dynamic_partition_enable`[​](#dynamic_partition_enable "Direct link to dynamic_partition_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the dynamic partitioning feature. When this feature is enabled, StarRocks dynamically creates partitions for new data and automatically deletes expired partitions to ensure the freshness of data. * Introduced in: - ##### `enable_active_materialized_view_schema_strict_check`[​](#enable_active_materialized_view_schema_strict_check "Direct link to enable_active_materialized_view_schema_strict_check") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to strictly check the length consistency of data types when activating an inactive materialized view. When this item is set to `false`, the activation of the materialized view is not affected if the length of the data types has changed in the base table. * Introduced in: v3.3.4 ##### `mv_fast_schema_change_mode`[​](#mv_fast_schema_change_mode "Direct link to mv_fast_schema_change_mode") * Default: strict * Type: String * Unit: - * Is mutable: Yes * Description: Controls the behavior of Materialized View (MV) Fast Schema Evolution (FSE). Valid values are: `strict` (default) - only allow FSE when `isSupportFastSchemaEvolutionInDanger` is true and clear affected partition entries from the version map; `force` - allow FSE even when `isSupportFastSchemaEvolutionInDanger` is false and clear affected partition entries to trigger recomputation on refresh; `force_no_clear` - allow FSE even when `isSupportFastSchemaEvolutionInDanger` is false but do not clear partition entries. * Introduced in: v4.1.0 ##### `enable_auto_collect_array_ndv`[​](#enable_auto_collect_array_ndv "Direct link to enable_auto_collect_array_ndv") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable automatic collection for the NDV information of the ARRAY type. * Introduced in: v4.1 ##### `enable_backup_materialized_view`[​](#enable_backup_materialized_view "Direct link to enable_backup_materialized_view") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the BACKUP and RESTORE of asynchronous materialized views when backing up or restoring a specific database. If this item is set to `false`, StarRocks will skip backing up asynchronous materialized views. * Introduced in: v3.2.0 ##### `enable_collect_full_statistic`[​](#enable_collect_full_statistic "Direct link to enable_collect_full_statistic") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable automatic full statistics collection. This feature is enabled by default. * Introduced in: - ##### `enable_colocate_mv_index`[​](#enable_colocate_mv_index "Direct link to enable_colocate_mv_index") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to support colocating the synchronous materialized view index with the base table when creating a synchronous materialized view. If this item is set to `true`, tablet sink will speed up the write performance of synchronous materialized views. * Introduced in: v3.2.0 ##### `enable_decimal_v3`[​](#enable_decimal_v3 "Direct link to enable_decimal_v3") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to support the DECIMAL V3 data type. * Introduced in: - ##### `enable_experimental_mv`[​](#enable_experimental_mv "Direct link to enable_experimental_mv") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the asynchronous materialized view feature. TRUE indicates this feature is enabled. From v2.5.2 onwards, this feature is enabled by default. For versions earlier than v2.5.2, this feature is disabled by default. * Introduced in: v2.4 ##### `enable_local_replica_selection`[​](#enable_local_replica_selection "Direct link to enable_local_replica_selection") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to select local replicas for queries. Local replicas reduce the network transmission cost. If this parameter is set to TRUE, the CBO preferentially selects tablet replicas on BEs that have the same IP address as the current FE. If this parameter is set to `FALSE`, both local replicas and non-local replicas can be selected. * Introduced in: - ##### `enable_manual_collect_array_ndv`[​](#enable_manual_collect_array_ndv "Direct link to enable_manual_collect_array_ndv") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable manual collection for the NDV information of the ARRAY type. * Introduced in: v4.1 ##### `enable_materialized_view`[​](#enable_materialized_view "Direct link to enable_materialized_view") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the creation of materialized views. * Introduced in: - ##### `enable_materialized_view_external_table_precise_refresh`[​](#enable_materialized_view_external_table_precise_refresh "Direct link to enable_materialized_view_external_table_precise_refresh") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Set this item to `true` to enable an internal optimization for materialized view refresh when a base table is an external (non-cloud-native) table. When enabled, the materialized view refresh processor computes candidate partitions and refreshes only the affected base-table partitions instead of all partitions, reducing I/O and refresh cost. Set it to `false` to force full-partition refresh of external tables. * Introduced in: v3.2.9 ##### `enable_materialized_view_metrics_collect`[​](#enable_materialized_view_metrics_collect "Direct link to enable_materialized_view_metrics_collect") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to collect monitoring metrics for asynchronous materialized views by default. * Introduced in: v3.1.11, v3.2.5 ##### `enable_materialized_view_spill`[​](#enable_materialized_view_spill "Direct link to enable_materialized_view_spill") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Intermediate Result Spilling for materialized view refresh tasks. * Introduced in: v3.1.1 ##### `enable_materialized_view_text_based_rewrite`[​](#enable_materialized_view_text_based_rewrite "Direct link to enable_materialized_view_text_based_rewrite") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable text-based query rewrite by default. If this item is set to `true`, the system builds the abstract syntax tree while creating an asynchronous materialized view. * Introduced in: v3.2.5 ##### `enable_mv_automatic_active_check`[​](#enable_mv_automatic_active_check "Direct link to enable_mv_automatic_active_check") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the system to automatically check and re-activate the asynchronous materialized views that are set inactive because their base tables (views) had undergone Schema Change or had been dropped and re-created. Please note that this feature will not re-activate the materialized views that are manually set inactive by users. * Introduced in: v3.1.6 ##### `enable_mv_automatic_repairing_for_broken_base_tables`[​](#enable_mv_automatic_repairing_for_broken_base_tables "Direct link to enable_mv_automatic_repairing_for_broken_base_tables") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When this item is set to `true`, StarRocks will attempt to automatically repair materialized view base-table metadata when a base external table is dropped and recreated or its table identifier changes. The repair flow can update the materialized view's base table information, collect partition-level repair information for external table partitions, and drive partition refresh decisions for async auto-refresh materialized views while honoring `autoRefreshPartitionsLimit`. Currently the automated repair supports Hive external tables; unsupported table types will cause the materialized view to be set inactive and a repair exception. Partition information collection is non-blocking and failures are logged. * Introduced in: v3.3.19, v3.4.8, v3.5.6 ##### `enable_predicate_columns_collection`[​](#enable_predicate_columns_collection "Direct link to enable_predicate_columns_collection") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable predicate columns collection. If disabled, predicate columns will not be recorded during query optimization. * Introduced in: - ##### `push_down_non_grouped_aggregate_below_union`[​](#push_down_non_grouped_aggregate_below_union "Direct link to push_down_non_grouped_aggregate_below_union") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to push down non-grouped aggregations below Union in the physical plan. * Introduced in: - ##### `enable_query_queue_v2`[​](#enable_query_queue_v2 "Direct link to enable_query_queue_v2") * Default: true * Type: boolean * Unit: - * Is mutable: No * Description: When true, switches the FE slot-based query scheduler to Query Queue V2. The flag is read by the slot manager and trackers (for example, `BaseSlotManager.isEnableQueryQueueV2` and `SlotTracker#createSlotSelectionStrategy`) to choose `SlotSelectionStrategyV2` instead of the legacy strategy. `query_queue_v2_xxx` configuration options and `QueryQueueOptions` take effect only when this flag is enabled. From v4.1 onwards, the default value is changed from `false` to `true`. * Introduced in: v3.3.4, v3.4.0, v3.5.0 ##### `enable_sql_blacklist`[​](#enable_sql_blacklist "Direct link to enable_sql_blacklist") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable blacklist check for SQL queries. When this feature is enabled, queries in the blacklist cannot be executed. * Introduced in: - ##### `enable_statistic_collect`[​](#enable_statistic_collect "Direct link to enable_statistic_collect") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to collect statistics for the CBO. This feature is enabled by default. * Introduced in: - ##### `enable_statistic_collect_on_first_load`[​](#enable_statistic_collect_on_first_load "Direct link to enable_statistic_collect_on_first_load") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls automatic statistics collection and maintenance triggered by data loading operations. This includes: * Statistics collection when data is first loaded into a partition (partition version equals 2). * Statistics collection when data is loaded into empty partitions of multi-partition tables. * Statistics copying and updating for INSERT OVERWRITE operations. **Decision Policy for Statistics Collection Type:** * For INSERT OVERWRITE: `deltaRatio = |targetRows - sourceRows| / (sourceRows + 1)` * If `deltaRatio < statistic_sample_collect_ratio_threshold_of_first_load` (Default: 0.1), statistics collection will not be performed. Only the existing statistics will be copied. * Else, if `targetRows > statistic_sample_collect_rows` (Default: 200000), SAMPLE statistics collection is used. * Else, FULL statistics collection is used. * For First Load: `deltaRatio = loadRows / (totalRows + 1)` * If `deltaRatio < statistic_sample_collect_ratio_threshold_of_first_load` (Default: 0.1), statistics collection will not be performed. * Else, if `loadRows > statistic_sample_collect_rows` (Default: 200000), SAMPLE statistics collection is used. * Else, FULL statistics collection is used. **Synchronization Behavior:** * For DML statements (INSERT INTO/INSERT OVERWRITE): Synchronous mode with table lock. The load operation waits for statistics collection to complete (up to `semi_sync_collect_statistic_await_seconds`). * For Stream Load and Broker Load: Asynchronous mode without lock. Statistics collection runs in background without blocking the load operation. note Disabling this configuration will prevent all loading-triggered statistics operations, including statistics maintenance for INSERT OVERWRITE, which may result in tables lacking statistics. If new tables are frequently created and data is frequently loaded, enabling this feature will increase memory and CPU overhead. * Introduced in: v3.1 ##### `enable_statistic_collect_on_update`[​](#enable_statistic_collect_on_update "Direct link to enable_statistic_collect_on_update") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls whether UPDATE statements can trigger automatic statistics collection. When enabled, UPDATE operations that modify table data may schedule statistics collection through the same ingestion-based statistics framework controlled by `enable_statistic_collect_on_first_load`. Disabling this configuration skips statistics collection for UPDATE statements while keeping load-triggered statistics collection behavior unchanged. * Introduced in: v3.5.11, v4.0.4 ##### `enable_udf`[​](#enable_udf "Direct link to enable_udf") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to enable UDF. * Introduced in: - ##### `expr_children_limit`[​](#expr_children_limit "Direct link to expr_children_limit") * Default: 10000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of child expressions allowed in an expression. * Introduced in: - ##### `histogram_buckets_size`[​](#histogram_buckets_size "Direct link to histogram_buckets_size") * Default: 64 * Type: Long * Unit: - * Is mutable: Yes * Description: The default bucket number for a histogram. * Introduced in: - ##### `histogram_max_sample_row_count`[​](#histogram_max_sample_row_count "Direct link to histogram_max_sample_row_count") * Default: 10000000 * Type: Long * Unit: - * Is mutable: Yes * Description: The maximum number of rows to collect for a histogram. * Introduced in: - ##### `histogram_mcv_size`[​](#histogram_mcv_size "Direct link to histogram_mcv_size") * Default: 100 * Type: Long * Unit: - * Is mutable: Yes * Description: The number of most common values (MCV) for a histogram. * Introduced in: - ##### `histogram_sample_ratio`[​](#histogram_sample_ratio "Direct link to histogram_sample_ratio") * Default: 0.1 * Type: Double * Unit: - * Is mutable: Yes * Description: The sampling ratio for a histogram. * Introduced in: - ##### `http_slow_request_threshold_ms`[​](#http_slow_request_threshold_ms "Direct link to http_slow_request_threshold_ms") * Default: 5000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: If the response time for an HTTP request exceeds the value specified by this parameter, a log is generated to track this request. * Introduced in: v2.5.15, v3.1.5 ##### `lock_checker_enable_deadlock_check`[​](#lock_checker_enable_deadlock_check "Direct link to lock_checker_enable_deadlock_check") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, the LockChecker thread performs JVM-level deadlock detection using ThreadMXBean.findDeadlockedThreads() and logs the offending threads' stack traces. The check runs inside the LockChecker daemon (whose frequency is controlled by `lock_checker_interval_second`) and writes detailed stack information to the log, which may be CPU- and I/O-intensive. Enable this option only for troubleshooting live or reproducible deadlock issues; leaving it enabled in normal operation can increase overhead and log volume. * Introduced in: v3.2.0 ##### `low_cardinality_threshold`[​](#low_cardinality_threshold "Direct link to low_cardinality_threshold") * Default: 255 * Type: Int * Unit: - * Is mutable: No * Description: Threshold of low cardinality dictionary. * Introduced in: v3.5.0 ##### `materialized_view_min_refresh_interval`[​](#materialized_view_min_refresh_interval "Direct link to materialized_view_min_refresh_interval") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The minimum allowed refresh interval (in seconds) for ASYNC materialized view schedules. When a materialized view is created with a time-based interval, the interval is converted to seconds and must not be less tham this value; otherwise the CREATE/ALTER operation fails with a DDL error. If this value is greater than 0, the check is enforced; set it to 0 or a negative value to disable the limit, which prevents excessive TaskManager scheduling and high FE memory/CPU usage from overly frequent refreshes. This item does not apply to `EVENT_TRIGGERED` refreshes. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### `materialized_view_refresh_ascending`[​](#materialized_view_refresh_ascending "Direct link to materialized_view_refresh_ascending") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When this item is set to `true`, materialized view partition refresh will iterate partitions in ascending partition-key order (oldest to newest). When it is set to `false` (default), the system iterates in descending order (newest to oldest). StarRocks uses this item in both list- and range-partitioned materialized view refresh logic to choose which partitions to process when partition refresh limits apply and to compute the next start/end partition boundaries for subsequent TaskRun executions. Changing this item alters which partitions are refreshed first and how the next partition range is derived; for range-partitioned materialized views, the scheduler validates new start/end and will raise an error if a change would create a repeated boundary (dead-loop), so set this item with care. * Introduced in: v3.3.1, v3.4.0, v3.5.0 ##### `max_allowed_in_element_num_of_delete`[​](#max_allowed_in_element_num_of_delete "Direct link to max_allowed_in_element_num_of_delete") * Default: 10000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of elements allowed for the IN predicate in a DELETE statement. * Introduced in: - ##### `max_create_table_timeout_second`[​](#max_create_table_timeout_second "Direct link to max_create_table_timeout_second") * Default: 600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum timeout duration for creating a table. * Introduced in: - ##### `max_distribution_pruner_recursion_depth`[​](#max_distribution_pruner_recursion_depth "Direct link to max_distribution_pruner_recursion_depth") * Default: 100 * Type: Int * Unit: - * Is mutable: Yes * Description:: The maximum recursion depth allowed by the partition pruner. Increasing the recursion depth can prune more elements but also increases CPU consumption. * Introduced in: - ##### `max_partitions_in_one_batch`[​](#max_partitions_in_one_batch "Direct link to max_partitions_in_one_batch") * Default: 4096 * Type: Long * Unit: - * Is mutable: Yes * Description: The maximum number of partitions that can be created when you bulk create partitions. * Introduced in: - ##### `max_planner_scalar_rewrite_num`[​](#max_planner_scalar_rewrite_num "Direct link to max_planner_scalar_rewrite_num") * Default: 100000 * Type: Long * Unit: - * Is mutable: Yes * Description: The maximum number of times that the optimizer can rewrite a scalar operator. * Introduced in: - ##### `max_query_queue_history_slots_number`[​](#max_query_queue_history_slots_number "Direct link to max_query_queue_history_slots_number") * Default: 0 * Type: Int * Unit: Slots * Is mutable: Yes * Description: Controls how many recently released (history) allocated slots are retained per query queue for monitoring and observability. When `max_query_queue_history_slots_number` is set to a value `> 0`, BaseSlotTracker keeps up to that many most-recently released LogicalSlot entries in an in-memory queue, evicting the oldest when the limit is exceeded. Enabling this causes getSlots() to include these history entries (newest first), allows BaseSlotTracker to attempt registering slots with the ConnectContext for richer ExtraMessage data, and lets LogicalSlot.ConnectContextListener attach query finish metadata to history slots. When `max_query_queue_history_slots_number` `<= 0` the history mechanism is disabled (no extra memory used). Use a reasonable value to balance observability and memory overhead. * Introduced in: v3.5.0 ##### `max_query_retry_time`[​](#max_query_retry_time "Direct link to max_query_retry_time") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of query retries on an FE. * Introduced in: - ##### `max_running_rollup_job_num_per_table`[​](#max_running_rollup_job_num_per_table "Direct link to max_running_rollup_job_num_per_table") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of rollup jobs can run in parallel for a table. * Introduced in: - ##### `max_scalar_operator_flat_children`[​](#max_scalar_operator_flat_children "Direct link to max_scalar_operator_flat_children") * Default: 10000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of flat children for ScalarOperator. You can set this limit to prevent the optimizer from using too much memory. * Introduced in: - ##### `max_scalar_operator_optimize_depth`[​](#max_scalar_operator_optimize_depth "Direct link to max_scalar_operator_optimize_depth") * Default: 256 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum depth that ScalarOperator optimization can be applied. * Introduced in: - ##### `mv_active_checker_interval_seconds`[​](#mv_active_checker_interval_seconds "Direct link to mv_active_checker_interval_seconds") * Default: 60 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: When the background `active_checker` thread is enabled, the system will periodically detect and automatically reactivate materialized views that became Inactive due to schema changes or rebuilds of their base tables (or views). This parameter controls the scheduling interval of the checker thread, in seconds. The default value is system-defined. * Introduced in: v3.1.6 ##### `mv_rewrite_consider_data_layout_mode`[​](#mv_rewrite_consider_data_layout_mode "Direct link to mv_rewrite_consider_data_layout_mode") * Default: `enable` * Type: String * Unit: - * Is mutable: Yes * Description: Controls whether materialized view rewrite should take the base table data layout into account when selecting the best materialized view. Valid values: * `disable`: Never use data-layout criteria when choosing between candidate materialized views. * `enable`: Use data-layout criteria only when the query is recognized as layout-sensitive. * `force`: Always apply data-layout criteria when selecting the best materialized view. Changing this item affects `BestMvSelector` behavior and can improve or broaden rewrite applicability depending on whether physical layout matters for plan correctness or performance. * Introduced in: - ##### `publish_version_interval_ms`[​](#publish_version_interval_ms "Direct link to publish_version_interval_ms") * Default: 10 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: The time interval at which release validation tasks are issued. * Introduced in: - ##### `query_queue_slots_estimator_strategy`[​](#query_queue_slots_estimator_strategy "Direct link to query_queue_slots_estimator_strategy") * Default: PBE * Type: String * Unit: - * Is mutable: Yes * Description: Selects the slot estimation strategy used for queue-based queries when `enable_query_queue_v2` is true. Valid values: `PBE` (parallelism-based, the default), `MBE` (memory-cost-based), and `CBE` (CPU-cost-based). PBE estimates a query's slots from scan parallelism, capped by the worker count: for OLAP tables it uses the number of scan ranges left after pruning, so only very small queries fall below the worker count; a connector/external scan is treated as a full-parallelism scan (the worker count) rather than a single-slot query. MBE estimates slots from the query's memory cost divided by `query_queue_v2_mem_bytes_per_slot`. CBE estimates slots from the plan CPU cost divided by `query_queue_v2_cpu_costs_per_slot`. MBE and CBE per-query slots are additionally capped by `number_of_workers * max(1, pipeline_dop / 2)`. The legacy values `MAX` and `MIN` are still accepted for forward compatibility and are treated as the default estimator; any other value is rejected by configuration validation. * Introduced in: v3.5.0 ##### `query_queue_v2_concurrency_level`[​](#query_queue_v2_concurrency_level "Direct link to query_queue_v2_concurrency_level") * Default: 4 * Type: Int * Unit: - * Is mutable: Yes * Description: Interpreted as a capacity level relative to the default level `4`. For the default (PBE) and CPU-cost-based (CBE) estimators, the system's total query slots are computed as `number_of_workers * cores_per_worker * (query_queue_v2_concurrency_level / 4)` (derived from BackendResourceStat). For the memory-cost-based estimator (MBE), the total slots are instead derived from the warehouse memory budget. If the configured value is non-positive it is treated as `4`. total\_slots is clamped to at least `number_of_workers`. Increasing this value raises totalSlots (and therefore concurrent query capacity); at the default `4` the total capacity equals `number_of_workers * cores_per_worker`. Set it proportional to the concurrency you want for the cluster. * Introduced in: v3.3.4, v3.4.0, v3.5.0 ##### `query_queue_v2_cpu_costs_per_slot`[​](#query_queue_v2_cpu_costs_per_slot "Direct link to query_queue_v2_cpu_costs_per_slot") * Default: 1000000000 * Type: Long * Unit: planner CPU cost units * Is mutable: Yes * Description: Per-slot CPU cost threshold used by the CPU-cost-based estimator (CBE) to estimate how many slots a query needs from its plan CPU cost. The scheduler computes slots as `ceil(plan_cpu_costs / query_queue_v2_cpu_costs_per_slot)` and clamps the result to the range `[1, min(totalSlots, number_of_workers * max(1, pipeline_dop / 2))]`. A non-positive value is normalized to `1`. Increasing this value reduces slots allocated per query (favoring fewer, larger-slot queries); decreasing it increases slots per query. * Introduced in: v3.3.4, v3.4.0, v3.5.0 ##### `query_queue_v2_mem_bytes_per_slot`[​](#query_queue_v2_mem_bytes_per_slot "Direct link to query_queue_v2_mem_bytes_per_slot") * Default: 0 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: Per-slot memory target used by the memory-cost-based estimator (MBE). When `query_queue_slots_estimator_strategy` is `MBE`, the total slots are derived from the warehouse memory budget, and a query's slots are estimated from its total memory cost divided by this value, capped by `number_of_workers * max(1, pipeline_dop / 2)`. If it is non-positive, Query Queue V2 uses the average worker memory per core. * Introduced in: - ##### `query_queue_v2_num_rows_per_slot`[​](#query_queue_v2_num_rows_per_slot "Direct link to query_queue_v2_num_rows_per_slot") * Default: 4096 * Type: Int * Unit: Rows * Is mutable: Yes * Description: Retained for backward compatibility with existing Query Queue V2 serialized and debug output. It is no longer used by the PBE, MBE, or CBE slot estimators. * Introduced in: v3.3.4, v3.4.0, v3.5.0 ##### `query_queue_v2_schedule_strategy`[​](#query_queue_v2_schedule_strategy "Direct link to query_queue_v2_schedule_strategy") * Default: SWRR * Type: String * Unit: - * Is mutable: Yes * Description: Selects the scheduling policy used by Query Queue V2 to order pending queries. Supported values (case-insensitive) are `SWRR` (Smooth Weighted Round Robin) — the default, suitable for mixed/hybrid workloads that need fair weighted sharing — and `SJF` (Short Job First + Aging) — prioritizes short jobs while using aging to avoid starvation. The value is parsed with case-insensitive enum lookup; an unrecognized value is logged as an error and the default policy is used. This configuration only affects behavior when Query Queue V2 is enabled and interacts with V2 sizing settings such as `query_queue_v2_concurrency_level`. * Introduced in: v3.3.12, v3.4.2, v3.5.0 ##### `semi_sync_collect_statistic_await_seconds`[​](#semi_sync_collect_statistic_await_seconds "Direct link to semi_sync_collect_statistic_await_seconds") * Default: 30 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Maximum wait time for semi-synchronous statistics collection during DML operations (INSERT INTO and INSERT OVERWRITE statements). Stream Load and Broker Load use asynchronous mode and are not affected by this configuration. If statistics collection time exceeds this value, the load operation continues without waiting for collection to complete. This configuration works in conjunction with `enable_statistic_collect_on_first_load`. * Introduced in: v3.1 ##### `slow_query_analyze_threshold`[​](#slow_query_analyze_threshold "Direct link to slow_query_analyze_threshold") * Default: 5000 * Type: Int * Unit: Seconds * Is mutable: Yes * Description:: The execution time threshold for queries to trigger the analysis of Query Feedback. * Introduced in: v3.4.0 ##### `statistic_analyze_status_keep_second`[​](#statistic_analyze_status_keep_second "Direct link to statistic_analyze_status_keep_second") * Default: 3 \* 24 \* 3600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The duration to retain the history of collection tasks. The default value is 3 days. * Introduced in: - ##### `statistic_auto_analyze_end_time`[​](#statistic_auto_analyze_end_time "Direct link to statistic_auto_analyze_end_time") * Default: 23:59:59 * Type: String * Unit: - * Is mutable: Yes * Description: The end time of automatic collection. Value range: `00:00:00` - `23:59:59`. * Introduced in: - ##### `statistic_auto_analyze_start_time`[​](#statistic_auto_analyze_start_time "Direct link to statistic_auto_analyze_start_time") * Default: 00:00:00 * Type: String * Unit: - * Is mutable: Yes * Description: The start time of automatic collection. Value range: `00:00:00` - `23:59:59`. * Introduced in: - ##### `statistic_auto_collect_ratio`[​](#statistic_auto_collect_ratio "Direct link to statistic_auto_collect_ratio") * Default: 0.8 * Type: Double * Unit: - * Is mutable: Yes * Description: The threshold for determining whether the statistics for automatic collection are healthy. If statistics health is below this threshold, automatic collection is triggered. * Introduced in: - ##### `statistic_auto_collect_small_table_rows`[​](#statistic_auto_collect_small_table_rows "Direct link to statistic_auto_collect_small_table_rows") * Default: 10000000 * Type: Long * Unit: - * Is mutable: Yes * Description: Threshold to determine whether a table in an external data source (Hive, Iceberg, Hudi) is a small table during automatic collection. If the table has rows less than this value, the table is considered a small table. * Introduced in: v3.2 ##### `statistic_cache_columns`[​](#statistic_cache_columns "Direct link to statistic_cache_columns") * Default: 100000 * Type: Long * Unit: - * Is mutable: No * Description: The number of rows that can be cached for the statistics table. * Introduced in: - ##### `statistic_cache_thread_pool_size`[​](#statistic_cache_thread_pool_size "Direct link to statistic_cache_thread_pool_size") * Default: 5 * Type: Int * Unit: - * Is mutable: No * Description: The size of the thread-pool which will be used to refresh statistic caches. * Introduced in: - ##### `statistic_collect_interval_sec`[​](#statistic_collect_interval_sec "Direct link to statistic_collect_interval_sec") * Default: 10 \* 60 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The interval for checking data updates during automatic collection. * Introduced in: - ##### `statistic_max_full_collect_data_size`[​](#statistic_max_full_collect_data_size "Direct link to statistic_max_full_collect_data_size") * Default: 100 \* 1024 \* 1024 \* 1024 * Type: Long * Unit: bytes * Is mutable: Yes * Description: The data size threshold for the automatic collection of statistics. If the total size exceeds this value, then sampled collection is performed instead of full. * Introduced in: - ##### `statistic_sample_collect_rows`[​](#statistic_sample_collect_rows "Direct link to statistic_sample_collect_rows") * Default: 200000 * Type: Long * Unit: - * Is mutable: Yes * Description: The row count threshold for deciding between SAMPLE and FULL statistics collection during loading-triggered statistics operations. If the number of loaded or changed rows exceeds this threshold (default 200,000), SAMPLE statistics collection is used; otherwise, FULL statistics collection is used. This setting works in conjunction with `enable_statistic_collect_on_first_load` and `statistic_sample_collect_ratio_threshold_of_first_load`. * Introduced in: - ##### `statistic_update_interval_sec`[​](#statistic_update_interval_sec "Direct link to statistic_update_interval_sec") * Default: 24 \* 60 \* 60 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The interval at which the cache of statistical information is updated. * Introduced in: - ##### `enable_external_stats_lazy_refresh_on_replay`[​](#enable_external_stats_lazy_refresh_on_replay "Direct link to enable_external_stats_lazy_refresh_on_replay") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls how followers (and restart recovery) refresh the connector (external table) statistics cache when replaying statistics journals. When set to `true`, the cache is invalidated by the table UUID persisted in the journal and reloaded lazily on the next query, which avoids resolving external table metadata (`MetadataMgr.getTable`) during replay — such resolution may block the journal replayer on the Hive Metastore or object storage. When set to `false` (default), the legacy eager refresh is used, preserving existing behavior. Statistics journals written before this UUID was persisted always fall back to eager refresh regardless of this setting. ##### `statistics_large_string_column_merge_threshold`[​](#statistics_large_string_column_merge_threshold "Direct link to statistics_large_string_column_merge_threshold") * Default: 0 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: Disabled by default (`0`). When set to a positive value, a dedicated SQL is generated during statistics collection to collect the statistics of string columns (`VARCHAR` / `CHAR`) whose declared length exceeds this threshold, instead of merging them with other columns. Both sampled and full statistics collection follow this strategy. The purpose is to bound the Exchange-stage memory peak of a single statistics SQL and prevent long string columns from further amplifying the aggregate operator state when merged with other columns. Keep it at `0` to collect all columns through the original merged-batch path. Note that `STRING` is represented internally as a maximum-length `VARCHAR`, so enabling this option with a positive threshold may also isolate `STRING` columns. * Introduced in: - ##### `task_check_interval_second`[​](#task_check_interval_second "Direct link to task_check_interval_second") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Interval between executions of task background jobs. GlobalStateMgr uses this value to schedule the TaskCleaner FrontendDaemon which invokes `doTaskBackgroundJob()`; the value is multiplied by 1000 to set the daemon interval in milliseconds. Decreasing the value makes background maintenance (task cleanup, checks) run more frequently and react faster but increases CPU/IO overhead; increasing it reduces overhead but delays cleanup and detection of stale tasks. Tune this value to balance maintenance responsiveness and resource usage. * Introduced in: v3.2.0 ##### `task_min_schedule_interval_s`[​](#task_min_schedule_interval_s "Direct link to task_min_schedule_interval_s") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Minimum allowed schedule interval (in seconds) for task schedules checked by the SQL layer. When a task is submitted, TaskAnalyzer converts the schedule period to seconds and rejects the submission with `ERR_INVALID_PARAMETER` if the period is smaller than `task_min_schedule_interval_s`. This prevents creating tasks that run too frequently and protects the scheduler from high-frequency tasks. If a schedule has no explicit start time, TaskAnalyzer sets the start time to the current epoch seconds. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### `task_runs_timeout_second`[​](#task_runs_timeout_second "Direct link to task_runs_timeout_second") * Default: 4 \* 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Default execution timeout (in seconds) for a TaskRun. This item is used by TaskRun execution as the baseline timeout. If the task run's properties include session variables `query_timeout` or `insert_timeout` with a positive integer value, the runtime uses the larger value between that session timeout and `task_runs_timeout_second`. The effective timeout is then bounded to not exceed the configured `task_runs_ttl_second` and `task_ttl_second`. Set this item to limit how long a task run may execute. Very large values may be clipped by the task/task-run TTL settings. * Introduced in: - #### Loading and unloading[​](#loading-and-unloading "Direct link to Loading and unloading") ##### `broker_load_default_timeout_second`[​](#broker_load_default_timeout_second "Direct link to broker_load_default_timeout_second") * Default: 14400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for a Broker Load job. * Introduced in: - ##### `desired_max_waiting_jobs`[​](#desired_max_waiting_jobs "Direct link to desired_max_waiting_jobs") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of pending jobs in an FE. The number refers to all jobs, such as table creation, loading, and schema change jobs. If the number of pending jobs in an FE reaches this value, the FE will reject new load requests. This parameter takes effect only for asynchronous loading. From v2.5 onwards, the default value is changed from 100 to 1024. * Introduced in: - ##### `disable_load_job`[​](#disable_load_job "Direct link to disable_load_job") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to disable loading when the cluster encounters an error. This prevents any loss caused by cluster errors. The default value is `FALSE`, indicating that loading is not disabled. `TRUE` indicates loading is disabled and the cluster is in read-only state. * Introduced in: - ##### `empty_load_as_error`[​](#empty_load_as_error "Direct link to empty_load_as_error") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to return an error message "all partitions have no load data" if no data is loaded. Valid values: * `true`: If no data is loaded, the system displays a failure message and returns an error "all partitions have no load data". * `false`: If no data is loaded, the system displays a success message and returns OK, instead of an error. * Introduced in: - ##### `enable_file_bundling`[​](#enable_file_bundling "Direct link to enable_file_bundling") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the File Bundling optimization for the cloud-native table. When this feature is enabled (set to `true`), the system automatically bundles the data files generated by loading, Compaction, or Publish operations, thereby reducing the API cost caused by high-frequency access to the external storage system. You can also control this behavior on the table level using the CREATE TABLE property `file_bundling`. For detailed instructions, see CREATE TABLE. * Introduced in: v4.1 ##### `enable_routine_load_lag_metrics`[​](#enable_routine_load_lag_metrics "Direct link to enable_routine_load_lag_metrics") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to collect Routine Load Kafka partition offset lag metrics. Please note that set this item to `true` will call the Kafka API to get the partition's latest offset. * Introduced in: - ##### `enable_sync_publish`[​](#enable_sync_publish "Direct link to enable_sync_publish") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to synchronously execute the apply task at the publish phase of a load transaction. This parameter is applicable only to Primary Key tables. Valid values: * `TRUE` (default): The apply task is synchronously executed at the publish phase of a load transaction. It means that the load transaction is reported as successful only after the apply task is completed, and the loaded data can truly be queried. When a task loads a large volume of data at a time or loads data frequently, setting this parameter to `true` can improve query performance and stability, but may increase load latency. * `FALSE`: The apply task is asynchronously executed at the publish phase of a load transaction. It means that the load transaction is reported as successful after the apply task is submitted, but the loaded data cannot be immediately queried. In this case, concurrent queries need to wait for the apply task to complete or time out before they can continue. When a task loads a large volume of data at a time or loads data frequently, setting this parameter to `false` may affect query performance and stability. * Introduced in: v3.2.0 ##### `export_checker_interval_second`[​](#export_checker_interval_second "Direct link to export_checker_interval_second") * Default: 5 * Type: Int * Unit: Seconds * Is mutable: No * Description: The time interval at which load jobs are scheduled. * Introduced in: - ##### `export_max_bytes_per_be_per_task`[​](#export_max_bytes_per_be_per_task "Direct link to export_max_bytes_per_be_per_task") * Default: 268435456 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: The maximum amount of data that can be exported from a single BE by a single data unload task. * Introduced in: - ##### `export_running_job_num_limit`[​](#export_running_job_num_limit "Direct link to export_running_job_num_limit") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of data exporting tasks that can run in parallel. * Introduced in: - ##### `export_task_default_timeout_second`[​](#export_task_default_timeout_second "Direct link to export_task_default_timeout_second") * Default: 2 \* 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for a data exporting task. * Introduced in: - ##### `export_task_pool_size`[​](#export_task_pool_size "Direct link to export_task_pool_size") * Default: 5 * Type: Int * Unit: - * Is mutable: No * Description: The size of the unload task thread pool. * Introduced in: - ##### `external_table_commit_timeout_ms`[​](#external_table_commit_timeout_ms "Direct link to external_table_commit_timeout_ms") * Default: 10000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The timeout duration for committing (publishing) a write transaction to a StarRocks external table. The default value `10000` indicates a 10-second timeout duration. * Introduced in: - ##### `finish_transaction_default_lock_timeout_ms`[​](#finish_transaction_default_lock_timeout_ms "Direct link to finish_transaction_default_lock_timeout_ms") * Default: 1000 * Type: Int * Unit: MilliSeconds * Is mutable: Yes * Description: The default timeout for acquiring the db and table lock during finishing transaction. * Introduced in: v4.0.0, v3.5.8 ##### `history_job_keep_max_num`[​](#history_job_keep_max_num "Direct link to history_job_keep_max_num") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of finished or cancelled alter jobs that the FE keeps in memory and shows in SHOW ALTER TABLE. Schema change and optimize jobs share one limit, and rollup jobs have their own. Once the limit is exceeded, the jobs that finished earliest are removed, even if they are still within the retention period set by `history_job_keep_max_second`. A job is kept while its old schema may still be used by a running load or, in a shared-data cluster, while a cluster snapshot still needs it, so the number of jobs can temporarily exceed this limit. `0` or a negative value disables the limit. * Introduced in: - ##### `history_job_keep_max_second`[​](#history_job_keep_max_second "Direct link to history_job_keep_max_second") * Default: 7 \* 24 \* 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum duration a historical job can be retained, such as schema change jobs. * Introduced in: - ##### `insert_load_default_timeout_second`[​](#insert_load_default_timeout_second "Direct link to insert_load_default_timeout_second") * Default: 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for the INSERT INTO statement that is used to load data. * Introduced in: - ##### `label_clean_interval_second`[​](#label_clean_interval_second "Direct link to label_clean_interval_second") * Default: 4 \* 3600 * Type: Int * Unit: Seconds * Is mutable: No * Description: The time interval at which labels are cleaned up. Unit: second. We recommend that you specify a short time interval to ensure that historical labels can be cleaned up in a timely manner. * Introduced in: - ##### `label_keep_max_num`[​](#label_keep_max_num "Direct link to label_keep_max_num") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of load jobs that can be retained within a period of time. If this number is exceeded, the information of historical jobs will be deleted. * Introduced in: - ##### `label_keep_max_second`[​](#label_keep_max_second "Direct link to label_keep_max_second") * Default: 3 \* 24 \* 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum duration in seconds to keep the labels of load jobs that have been completed and are in the FINISHED or CANCELLED state. The default value is 3 days. After this duration expires, the labels will be deleted. This parameter applies to all types of load jobs. A value too large consumes a lot of memory. * Introduced in: - ##### `load_checker_interval_second`[​](#load_checker_interval_second "Direct link to load_checker_interval_second") * Default: 5 * Type: Int * Unit: Seconds * Is mutable: No * Description: The time interval at which load jobs are processed on a rolling basis. * Introduced in: - ##### `load_parallel_instance_num`[​](#load_parallel_instance_num "Direct link to load_parallel_instance_num") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls the number of parallel load fragment instances created on a single host for broker and stream loads. LoadPlanner uses this value as the per-host degree of parallelism unless the session enables adaptive sink DOP; if the session variable `enable_adaptive_sink_dop` is true, the session`s `sink\_degree\_of\_parallelism\` overrides this configuration. When shuffle is required, this value is applied to fragment parallel execution (scan fragment and sink fragment parallel exec instances). When no shuffle is needed, it is used as the sink pipeline DOP. Note: loads from local files are forced to a single instance (pipeline DOP = 1, parallel exec = 1) to avoid local disk contention. Increasing this number raises per-host concurrency and throughput but may increase CPU, memory and I/O contention. * Introduced in: v3.2.0 ##### `load_straggler_wait_second`[​](#load_straggler_wait_second "Direct link to load_straggler_wait_second") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum loading lag that can be tolerated by a BE replica. If this value is exceeded, cloning is performed to clone data from other replicas. * Introduced in: - ##### `loads_history_retained_days`[​](#loads_history_retained_days "Direct link to loads_history_retained_days") * Default: 30 * Type: Int * Unit: Days * Is mutable: Yes * Description: Number of days to retain load history in the internal `_statistics_.loads_history` table. This value is used for table creation to set the table property `partition_live_number` and is passed to `TableKeeper` (clamped to a minimum of 1) to determine how many daily partitions to keep. Increasing or decreasing this value adjusts how long completed load jobs are retained in daily partitions; it affects new table creation and the keeper's pruning behavior but does not automatically recreate past partitions. The `LoadsHistorySyncer` relies on this retention when managing the loads history lifecycle; its sync cadence is controlled by `loads_history_sync_interval_second`. * Introduced in: v3.3.6, v3.4.0, v3.5.0 ##### `loads_history_sync_interval_second`[​](#loads_history_sync_interval_second "Direct link to loads_history_sync_interval_second") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Interval (in seconds) used by LoadsHistorySyncer to schedule periodic syncs of finished load jobs from `information_schema.loads` into the internal `_statistics_.loads_history` table. The value is multiplied by 1000 in the constructor to set the FrontendDaemon interval. The syncer skips the first run (to allow table creation) and only imports loads that finished more than one minute ago; small values increase DML and executor load, while larger values delay availability of historical load records. See `loads_history_retained_days` for retention/partitioning behavior of the target table. * Introduced in: v3.3.6, v3.4.0, v3.5.0 ##### `max_broker_load_job_concurrency`[​](#max_broker_load_job_concurrency "Direct link to max_broker_load_job_concurrency") * Default: 5 * Alias: `async_load_task_pool_size` * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of concurrent Broker Load jobs allowed within the StarRocks cluster. This parameter is valid only for Broker Load. The value of this parameter must be less than the value of `max_running_txn_num_per_db`. From v2.5 onwards, the default value is changed from `10` to `5`. * Introduced in: - ##### `max_get_loads_result_count`[​](#max_get_loads_result_count "Direct link to max_get_loads_result_count") * Default: 10000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of load records that FE returns in one response when a BE or CN scans `information_schema.loads`. When a scan matches more records than this, FE returns a page and a cursor, and the BE or CN requests the next page until all records are read; query results are unaffected, only the number of RPC round-trips behind one scan changes. A page is cut at a job boundary, so one load job's rows are never split across two pages and a page can slightly exceed this value. Raising it reduces round-trips but grows each response, which risks exceeding the message size limit of the BE-side RPC client; lowering it bounds each response at the cost of more round-trips. Paging applies only when the BE or CN is new enough to send the cursor; an older one still receives the whole result set in a single response. * Introduced in: - ##### `max_load_initial_open_partition_number`[​](#max_load_initial_open_partition_number "Direct link to max_load_initial_open_partition_number") * Default: 4096 * Type: Long * Unit: - * Is mutable: Yes * Description: The upper bound on how many partitions a load can open up front. The value is used as a cap in two scenarios: (1) for LIST-partitioned tables (which open all partitions by default) and (2) for RANGE-partitioned tables loaded via INSERT / Broker Load / Spark Load (which also open all partitions by default). Stream Load and Routine Load on RANGE-partitioned tables ignore this cap and keep the conservative latest-32 default. The per-table property `load_initial_open_partition_number` overrides this value, bypasses this cap, and is the highest-priority setting. From v4.0 onwards, the default value is increased from 32 to 4096. * Introduced in: - ##### `max_load_timeout_second`[​](#max_load_timeout_second "Direct link to max_load_timeout_second") * Default: 259200 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum timeout duration allowed for a load job. The load job fails if this limit is exceeded. This limit applies to all types of load jobs. * Introduced in: - ##### `max_routine_load_batch_size`[​](#max_routine_load_batch_size "Direct link to max_routine_load_batch_size") * Default: 4294967296 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: The maximum amount of data that can be loaded by a Routine Load task. * Introduced in: - ##### `max_routine_load_task_concurrent_num`[​](#max_routine_load_task_concurrent_num "Direct link to max_routine_load_task_concurrent_num") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of concurrent tasks for each Routine Load job. * Introduced in: - ##### `max_routine_load_task_num_per_be`[​](#max_routine_load_task_num_per_be "Direct link to max_routine_load_task_num_per_be") * Default: 16 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of concurrent Routine Load tasks on each BE. Since v3.1.0, the default value for this parameter is increased to 16 from 5, and no longer needs to be less than or equal to the value of BE static parameter `routine_load_thread_pool_size` (deprecated). * Introduced in: - ##### `max_running_txn_num_per_db`[​](#max_running_txn_num_per_db "Direct link to max_running_txn_num_per_db") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of load transactions allowed to be running for each database within a StarRocks cluster. The default value is `1000`. From v3.1 onwards, the default value is changed to `1000` from `100`. When the actual number of load transactions running for a database exceeds the value of this parameter, new load requests will not be processed. New requests for synchronous load jobs will be denied, and new requests for asynchronous load jobs will be placed in queue. We do not recommend you increase the value of this parameter because this will increase system load. * Introduced in: - ##### `max_stream_load_timeout_second`[​](#max_stream_load_timeout_second "Direct link to max_stream_load_timeout_second") * Default: 259200 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum allowed timeout duration for a Stream Load job. * Introduced in: - ##### `max_tolerable_backend_down_num`[​](#max_tolerable_backend_down_num "Direct link to max_tolerable_backend_down_num") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of faulty BE nodes allowed. If this number is exceeded, Routine Load jobs cannot be automatically recovered. * Introduced in: - ##### `min_bytes_per_broker_scanner`[​](#min_bytes_per_broker_scanner "Direct link to min_bytes_per_broker_scanner") * Default: 67108864 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: The minimum allowed amount of data that can be processed by a Broker Load instance. * Introduced in: - ##### `min_load_timeout_second`[​](#min_load_timeout_second "Direct link to min_load_timeout_second") * Default: 1 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The minimum timeout duration allowed for a load job. This limit applies to all types of load jobs. * Introduced in: - ##### `min_routine_load_lag_for_metrics`[​](#min_routine_load_lag_for_metrics "Direct link to min_routine_load_lag_for_metrics") * Default: 10000 * Type: INT * Unit: - * Is mutable: Yes * Description: The minimum offset lag of Routine Load jobs to be shown in monitoring metrics. Routine Load jobs whose offset lags are greater than this value will be displayed in the metrics. * Introduced in: - ##### `period_of_auto_resume_min`[​](#period_of_auto_resume_min "Direct link to period_of_auto_resume_min") * Default: 5 * Type: Int * Unit: Minutes * Is mutable: Yes * Description: The interval at which Routine Load jobs are automatically recovered. * Introduced in: - ##### `prepared_transaction_default_timeout_second`[​](#prepared_transaction_default_timeout_second "Direct link to prepared_transaction_default_timeout_second") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The default timeout duration for a prepared transaction. * Introduced in: - ##### `routine_load_task_consume_second`[​](#routine_load_task_consume_second "Direct link to routine_load_task_consume_second") * Default: 15 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The maximum time for each Routine Load task within the cluster to consume data. Since v3.1.0, Routine Load job supports a new parameter `task_consume_second` in `job_properties`. This parameter applies to individual load tasks within a Routine Load job, which is more flexible. * Introduced in: - ##### `routine_load_task_timeout_second`[​](#routine_load_task_timeout_second "Direct link to routine_load_task_timeout_second") * Default: 60 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for each Routine Load task within the cluster. Since v3.1.0, Routine Load job supports a new parameter `task_timeout_second` in `job_properties`. This parameter applies to individual load tasks within a Routine Load job, which is more flexible. * Introduced in: - ##### `routine_load_unstable_threshold_second`[​](#routine_load_unstable_threshold_second "Direct link to routine_load_unstable_threshold_second") * Default: 3600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: Routine Load job is set to the UNSTABLE state if any task within the Routine Load job lags. To be specific, the difference between the timestamp of the message being consumed and the current time exceeds this threshold, and unconsumed messages exist in the data source. * Introduced in: - ##### `spark_dpp_version`[​](#spark_dpp_version "Direct link to spark_dpp_version") * Default: 1.0.0 * Type: String * Unit: - * Is mutable: No * Description: The version of Spark Dynamic Partition Pruning (DPP) used. * Introduced in: - ##### `spark_home_default_dir`[​](#spark_home_default_dir "Direct link to spark_home_default_dir") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/lib/spark2x" * Type: String * Unit: - * Is mutable: No * Description: The root directory of a Spark client. * Introduced in: - ##### `spark_launcher_log_dir`[​](#spark_launcher_log_dir "Direct link to spark_launcher_log_dir") * Default: `sys_log_dir` + "/spark\_launcher\_log" * Type: String * Unit: - * Is mutable: No * Description: The directory that stores Spark log files. * Introduced in: - ##### `spark_load_default_timeout_second`[​](#spark_load_default_timeout_second "Direct link to spark_load_default_timeout_second") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for each Spark Load job. * Introduced in: - ##### `spark_load_submit_timeout_second`[​](#spark_load_submit_timeout_second "Direct link to spark_load_submit_timeout_second") * Default: 300 * Type: long * Unit: Seconds * Is mutable: No * Description: Maximum time in seconds to wait for a YARN response after submitting a Spark application. `SparkLauncherMonitor.LogMonitor` converts this value to milliseconds and will stop monitoring and forcibly kill the spark launcher process if the job remains in UNKNOWN/CONNECTED/SUBMITTED longer than this timeout. `SparkLoadJob` reads this configuration as the default and allows a per-load override via the `LoadStmt.SPARK_LOAD_SUBMIT_TIMEOUT` property. Set it high enough to accommodate YARN queueing delays; setting it too low may abort legitimately queued jobs, while setting it too high may delay failure handling and resource cleanup. * Introduced in: v3.2.0 ##### `spark_resource_path`[​](#spark_resource_path "Direct link to spark_resource_path") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The root directory of the Spark dependency package. * Introduced in: - ##### `stream_load_default_timeout_second`[​](#stream_load_default_timeout_second "Direct link to stream_load_default_timeout_second") * Default: 600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The default timeout duration for each Stream Load job. * Introduced in: - ##### `stream_load_max_txn_num_per_be`[​](#stream_load_max_txn_num_per_be "Direct link to stream_load_max_txn_num_per_be") * Default: -1 * Type: Int * Unit: Transactions * Is mutable: Yes * Description: Limits the number of concurrent stream-load transactions accepted from a single BE (backend) host. When set to a non-negative integer, FrontendServiceImpl checks the current transaction count for the BE (by client IP) and rejects new stream-load begin requests if the count `>=` this limit. A value of `< 0` disables the limit (unlimited). This check occurs during stream load begin and may cause a `streamload txn num per be exceeds limit` error when exceeded. Related runtime behavior uses `stream_load_default_timeout_second` for request timeout fallback. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### `stream_load_task_keep_max_num`[​](#stream_load_task_keep_max_num "Direct link to stream_load_task_keep_max_num") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of Stream Load tasks that StreamLoadMgr keeps in memory (global across all databases). When the number of tracked tasks (`idToStreamLoadTask`) exceeds this threshold, StreamLoadMgr first calls `cleanSyncStreamLoadTasks()` to remove completed synchronous stream-load tasks; if the size still remains greater than half of this threshold, it invokes `cleanOldStreamLoadTasks(true)` to force removal of older or finished tasks. Increase this value to retain more task history in memory; decrease it to reduce memory usage and make cleanup more aggressive. This value controls in-memory retention only and does not affect persisted/replayed tasks. * Introduced in: v3.2.0 ##### `stream_load_task_keep_max_second`[​](#stream_load_task_keep_max_second "Direct link to stream_load_task_keep_max_second") * Default: 3 \* 24 \* 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Retention window for finished or cancelled Stream Load tasks. After a task reaches a final state and its end timestamp is ealier than this threshold (`currentMs - endTimeMs > stream_load_task_keep_max_second * 1000`), it becomes eligible for removal by `StreamLoadMgr.cleanOldStreamLoadTasks` and is discarded when loading persisted state. Applies to both `StreamLoadTask` and `StreamLoadMultiStmtTask`. If total task count exceeds `stream_load_task_keep_max_num`, cleanup may be triggered earlier (synchronous tasks are prioritized by `cleanSyncStreamLoadTasks`). Set this to balance history/debugability against memory usage. * Introduced in: v3.2.0 ##### `transaction_clean_interval_second`[​](#transaction_clean_interval_second "Direct link to transaction_clean_interval_second") * Default: 30 * Type: Int * Unit: Seconds * Is mutable: No * Description: The time interval at which finished transactions are cleaned up. Unit: second. We recommend that you specify a short time interval to ensure that finished transactions can be cleaned up in a timely manner. * Introduced in: - ##### `transaction_stream_load_coordinator_cache_capacity`[​](#transaction_stream_load_coordinator_cache_capacity "Direct link to transaction_stream_load_coordinator_cache_capacity") * Default: 4096 * Type: Int * Unit: - * Is mutable: Yes * Description: The capacity of the cache that stores the mapping from transaction label to coordinator node. * Introduced in: - ##### `transaction_stream_load_coordinator_cache_expire_seconds`[​](#transaction_stream_load_coordinator_cache_expire_seconds "Direct link to transaction_stream_load_coordinator_cache_expire_seconds") * Default: 900 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time to keep the coordinator mapping in the cache before it's evicted(TTL). * Introduced in: - ##### `yarn_client_path`[​](#yarn_client_path "Direct link to yarn_client_path") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/lib/yarn-client/hadoop/bin/yarn" * Type: String * Unit: - * Is mutable: No * Description: The root directory of the Yarn client package. * Introduced in: - ##### `yarn_config_dir`[​](#yarn_config_dir "Direct link to yarn_config_dir") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/lib/yarn-config" * Type: String * Unit: - * Is mutable: No * Description: The directory that stores the Yarn configuration file. * Introduced in: - --- ### Cross-cluster Data Migration Tool The StarRocks Cross-cluster Data Migration Tool is provided by StarRocks Community. You can use this tool to easily migrate data from the source cluster to the target cluster. | Migration Path | Support information | | ------------------------------------- | ------------------------------ | | From Shared-nothing to Shared-nothing | From v3.1.8 and v3.2.3 onwards | | From Shared-nothing to Shared-data | From v3.1.8 and v3.2.3 onwards | | From Shared-data to Shared-data | From v4.1 onwards | | From Shared-data to Shared-nothing | Not supported | #### Preparations[​](#preparations "Direct link to Preparations") * Migrate from Shared-nothing * Migrate between Shared-data ##### On source cluster[​](#on-source-cluster "Direct link to On source cluster") You do not need to perform any preparations on the source cluster. ##### On target cluster[​](#on-target-cluster "Direct link to On target cluster") The following preparations must be performed on the target cluster for data migration. ###### Open ports[​](#open-ports "Direct link to Open ports") If you have enabled the firewall, you must open these ports: | **Component** | **Port** | **Default** | | ------------- | -------------- | ----------- | | FE | query\_port | 9030 | | FE | http\_port | 8030 | | FE | rpc\_port | 9020 | | BE/CN | be\_http\_port | 8040 | | BE/CN | be\_port | 9060 | ##### On source cluster[​](#on-source-cluster-1 "Direct link to On source cluster") During migration, the source cluster's Auto-Vacuum mechanism may delete historical data versions that the target CNs still need to read. To prevent this situation, you must extend the Auto-Vacuum grace period by dynamically setting the FE configuration item `lake_autovacuum_grace_period_minutes` to a significantly large value: ```sql ADMIN SET FRONTEND CONFIG("lake_autovacuum_grace_period_minutes"="10000000"); ``` important This setting prevents the source cluster from reclaiming stale object storage files during migration, which will cause storage amplification. It is recommended to keep the migration window as short as possible, and to reset this item to its default value `30` after migration. ```sql ADMIN SET FRONTEND CONFIG("lake_autovacuum_grace_period_minutes"="30"); ``` ##### On target cluster[​](#on-target-cluster-1 "Direct link to On target cluster") The following preparations must be performed on the target cluster for data migration. ###### Open ports[​](#open-ports-1 "Direct link to Open ports") If you have enabled the firewall, you must open these ports: | **Component** | **Port** | **Default** | | ------------- | ----------- | ----------- | | FE | query\_port | 9030 | | FE | http\_port | 8030 | | FE | rpc\_port | 9020 | ###### Disable Compaction[​](#disable-compaction "Direct link to Disable Compaction") You must disable Compaction on the target cluster during migration to prevent conflicts with incoming replication data. 1. Dynamically disable Compaction: ```sql ADMIN SET FRONTEND CONFIG("lake_compaction_max_tasks"="0"); ``` 2. To prevent Compaction from being re-enabled after a cluster restart, also add the following configuration to the FE configuration file **fe.conf**: ```properties lake_compaction_max_tasks = 0 ``` important After migration is complete, re-enable Compaction by removing the above configuration from **fe.conf**, and enable Compaction dynamically by executing: ```sql ADMIN SET FRONTEND CONFIG("lake_compaction_max_tasks"="-1"); ``` ###### Create source storage volumes on the target cluster[​](#create-source-storage-volumes-on-the-target-cluster "Direct link to Create source storage volumes on the target cluster") The Migration Tool identifies which storage volume each source table uses, and looks up a corresponding storage volume on the target cluster using the naming convention `src_`. You must pre-create these storage volumes before starting migration. important These storage volumes are used **only during migration** to give target CNs read access to the source cluster's object storage. After migration is complete, they are no longer needed and can be dropped. 1. On the **source cluster**, list all storage volumes: ```sql SHOW STORAGE VOLUMES; ``` 2. For each storage volume used by the tables you plan to migrate, describe it to get its configuration: ```sql DESCRIBE STORAGE VOLUME ; ``` Example output: ```text +---------------------+------+-----------+-------------------------------+-----------------------------+ | Name | Type | IsDefault | Location | Params | +---------------------+------+-----------+-------------------------------+-----------------------------+ | builtin_storage_vol | S3 | true | s3://my-bucket | {"aws.s3.region":"...",...} | +---------------------+------+-----------+-------------------------------+-----------------------------+ ``` 3. On the **target cluster**, create a mirrored storage volume using the same object storage credentials, but with the name prefixed by `src_`: ```sql CREATE STORAGE VOLUME src_ TYPE = S3 LOCATIONS = ("") PROPERTIES ( "enabled" = "true", "aws.s3.region" = "", "aws.s3.endpoint" = "", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.enable_partitioned_prefix" = "false" ); ``` note * Set `aws.s3.enable_partitioned_prefix` to `false` regardless of the source cluster's setting. The migration tool reads files using the source partition's full path directly, so partitioned prefix must not be applied to the mirrored volume. * Repeat this step for **each** unique storage volume used by the tables to be migrated. For example, if the source uses `builtin_storage_volume`, create `src_builtin_storage_volume` on the target cluster. * It is recommended to use temporary credentials (access key / secret key) for the source storage volume. These can be revoked after migration is complete. When transparent data encryption (TDE) is used in shared-data-to-shared-data migration, a newly copied private standalone physical file is read with its source encryption metadata and then re-encrypted under the target policy. If the source key hierarchy cannot be unwrapped, migration fails before the target publication. Newly copied shared or bundled physical files are unsupported when the source file is encrypted or target TDE is enabled. Objects that already exist on the target can be reused directly and retain their target encryption metadata. Migration from a shared-nothing source to a shared-data target fails closed when a source rowset or DCG file is encrypted; decrypting and re-encrypting those files on this path is not supported. ###### Enable Legacy Compatibility for Replication[​](#enable-legacy-compatibility-for-replication "Direct link to Enable Legacy Compatibility for Replication") StarRocks may behave differently between the old and new versions, causing problems during cross-cluster data migration. Therefore, you must enable Legacy Compatibility for the target cluster before data migration and disable it after data migration is completed. 1. You can check whether Legacy Compatibility for Replication is enabled by using the following statement: ```sql ADMIN SHOW FRONTEND CONFIG LIKE 'enable_legacy_compatibility_for_replication'; ``` If `true` is returned, it indicates that Legacy Compatibility for Replication is enabled. 2. Dynamically enable Legacy Compatibility for Replication: ```sql ADMIN SET FRONTEND CONFIG("enable_legacy_compatibility_for_replication"="true"); ``` 3. To prevent Legacy Compatibility for Replication from automatically disabling during the data migration process in case of cluster restart, you also need to add the following configuration item in the FE configuration file **fe.conf**: ```properties enable_legacy_compatibility_for_replication = true ``` important After the data migration is completed, you need to remove the configuration `enable_legacy_compatibility_for_replication = true` from the configuration file, and dynamically disable Legacy Compatibility for Replication using the following statement: ```sql ADMIN SET FRONTEND CONFIG("enable_legacy_compatibility_for_replication"="false"); ``` ###### Configure Data Migration (Optional)[​](#configure-data-migration-optional "Direct link to Configure Data Migration (Optional)") You can configure data migration operations using the following FE and BE parameters. In most cases, the default configuration can meet your needs. If you wish to use the default configuration, you can skip this step. note Please note that increasing the values of the following configuration items can accelerate migration but will also increase the load pressure on the source cluster. ###### FE Parameters[​](#fe-parameters "Direct link to FE Parameters") The following FE parameters are dynamic configuration items. Refer to [Configure FE Dynamic Parameters](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#configure-fe-dynamic-parameters) on how to modify them. | **Parameter** | **Default** | **Unit** | **Description** | | ------------------------------------------ | ----------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- | | replication\_max\_parallel\_table\_count | 100 | - | The maximum number of concurrent data synchronization tasks allowed. StarRocks creates one synchronization task for each table. | | replication\_max\_parallel\_replica\_count | 10240 | - | The maximum number of tablet replica allowed for concurrent synchronization. | | replication\_max\_parallel\_data\_size\_mb | 1048576 | MB | The maximum size of data allowed for concurrent synchronization. | | replication\_transaction\_timeout\_sec | 86400 | Seconds | The timeout duration for synchronization tasks. | ###### BE Parameters[​](#be-parameters "Direct link to BE Parameters") The following BE parameter is a dynamic configuration item. Refer to [Configure BE Dynamic Parameters](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md) on how to modify it. | **Parameter** | **Default** | **Unit** | **Description** | | -------------------- | ----------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | replication\_threads | 0 | - | The number of threads for executing synchronization tasks. `0` indicates setting the number of threads to the 4 times of number of CPU cores on the machine where the BE resides. | #### Step 1: Install the Tool[​](#step-1-install-the-tool "Direct link to Step 1: Install the Tool") It is recommended to install the migration tool on the server where the target cluster resides. 1. Launch a terminal, and download the binary package of the tool. ```bash wget https://releases.starrocks.io/starrocks/starrocks-cluster-sync.tar.gz ``` 2. Decompress the package. ```bash tar -xvzf starrocks-cluster-sync.tar.gz ``` #### Step 2: Configure the Tool[​](#step-2-configure-the-tool "Direct link to Step 2: Configure the Tool") ##### Migration-related configuration[​](#migration-related-configuration "Direct link to Migration-related configuration") Navigate to the extracted folder and modify the configuration file **conf/sync.properties**. ```bash cd starrocks-cluster-sync vi conf/sync.properties ``` The file content is as follows: ```properties # If true, all tables will be synchronized only once, and the program will exit automatically after completion. one_time_run_mode=false source_fe_host= source_fe_query_port=9030 source_cluster_user=root source_cluster_password= source_cluster_password_secret_key= # You can leave this empty or omit it if you want to migrate data between shared-data source clusters. source_cluster_token= target_fe_host= target_fe_query_port=9030 target_cluster_user=root target_cluster_password= target_cluster_password_secret_key= jdbc_connect_timeout_ms=30000 jdbc_socket_timeout_ms=60000 # Comma-separated list of database names or table names like or # example: db1,db2.tbl2,db3 # Effective order: 1. include 2. exclude include_data_list= exclude_data_list= # If there are no special requirements, please maintain the default values for the following configurations. target_cluster_storage_volume= # This configuration item is for migration between shared-data clusters only. target_cluster_use_builtin_storage_volume_only=false target_cluster_replication_num=-1 target_cluster_max_disk_used_percent=80 # To maintain consistency with the source cluster, use null. target_cluster_enable_persistent_index= max_replication_data_size_per_job_in_gb=1024 meta_job_interval_seconds=180 meta_job_threads=4 ddl_job_interval_seconds=10 ddl_job_batch_size=10 # table config ddl_job_allow_drop_target_only=false ddl_job_allow_drop_schema_change_table=true ddl_job_allow_drop_inconsistent_partition=true ddl_job_allow_drop_inconsistent_time_partition = true ddl_job_allow_drop_partition_target_only=true # index config enable_bitmap_index_sync=false ddl_job_allow_drop_inconsistent_bitmap_index=true ddl_job_allow_drop_bitmap_index_target_only=true # MV config enable_materialized_view_sync=false ddl_job_allow_drop_inconsistent_materialized_view=true ddl_job_allow_drop_materialized_view_target_only=false # View config enable_view_sync=false ddl_job_allow_drop_inconsistent_view=true ddl_job_allow_drop_view_target_only=false replication_job_interval_seconds=10 replication_job_batch_size=10 report_interval_seconds=300 enable_table_property_sync=false ``` The description of the parameters is as follows: | **Parameter** | **Description** | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | one\_time\_run\_mode | Whether to enable one-time synchronization mode. When one-time synchronization mode is enabled, the migration tool only performs full synchronization instead of incremental synchronization. | | source\_fe\_host | The IP address or FQDN (Fully Qualified Domain Name) of the source cluster's FE. | | source\_fe\_query\_port | The query port (`query_port`) of the source cluster's FE. | | source\_cluster\_user | The username used to log in to the source cluster. This user must be granted the OPERATE privilege on the SYSTEM level. | | source\_cluster\_password | The user password used to log in to the source cluster. | | source\_cluster\_password\_secret\_key | The secret key used to encrypt the password of the login user for the source cluster. The default value is an empty string, which means that the login password is not encrypted. If you want to encrypt `source_cluster_password`, you can get the encrypted `source_cluster_password` string by using SQL statement `SELECT TO_BASE64(AES_ENCRYPT('',''))`. | | source\_cluster\_token | Token of the source cluster. For information on how to obtain the cluster token, refer to [Obtain Cluster Token](#obtain-cluster-token) below.
**NOTE**
The Cluster Token is not required for migration between shared-data clusters because files are read directly from object storage. You can leave this empty or omit it if you want to migrate data between shared-data source clusters. | | target\_fe\_host | The IP address or FQDN (Fully Qualified Domain Name) of the target cluster's FE. | | target\_fe\_query\_port | The query port (`query_port`) of the target cluster's FE. | | target\_cluster\_user | The username used to log in to the target cluster. This user must be granted the OPERATE privilege on the SYSTEM level. | | target\_cluster\_password | The user password used to log in to the target cluster. | | target\_cluster\_password\_secret\_key | The secret key used to encrypt the password of the login user for the target cluster. The default value is an empty string, which means that the login password is not encrypted. If you want to encrypt `target_cluster_password`, you can get the encrypted `target_cluster_password` string by using SQL statement `SELECT TO_BASE64(AES_ENCRYPT('',''))`. | | jdbc\_connect\_timeout\_ms | JDBC connection timeout in milliseconds for FE queries. Default: `30000`. | | jdbc\_socket\_timeout\_ms | JDBC socket timeout in milliseconds for FE queries. Default: `60000`. | | include\_data\_list | The databases and tables that need to be migrated, with multiple objects separated by commas (`,`). For example: `db1, db2.tbl2, db3`. This item takes effect prior to `exclude_data_list`. If you want to migrate all databases and tables in the cluster, you do not need to configure this item. | | exclude\_data\_list | The databases and tables that do not need to be migrated, with multiple objects separated by commas (`,`). For example: `db1, db2.tbl2, db3`. `include_data_list` takes effect prior to this item. If you want to migrate all databases and tables in the cluster, you do not need to configure this item. | | target\_cluster\_storage\_volume | The storage volume used to store tables in the target cluster when the target cluster is a shared-data cluster. If you want to use the default storage volume, you do not need to specify this item. | | target\_cluster\_use\_builtin\_storage\_volume\_only | Whether to use `builtin_storage_volume` for migration in the target cluster. It is required for migration between shared-data clusters only. When this item is set to `true`, tables created in the target cluster will use `builtin_storage_volume` uniformly, instead of using the source cluster's `storage_volume` configuration. This is useful when the source cluster has multiple custom storage volumes but you want to consolidate all tables under one storage volume in the target cluster. | | target\_cluster\_replication\_num | The number of replicas specified when creating tables in the target cluster. If you want to use the same replica number as the source cluster, you do not need to specify this item. | | target\_cluster\_max\_disk\_used\_percent | Disk usage percentage threshold for BE nodes of the target cluster when the target cluster is shared-nothing. Migration is terminated when the disk usage of any BE in the target cluster exceeds this threshold. The default value is `80`, which means 80%. | | meta\_job\_interval\_seconds | The interval, in seconds, at which the migration tool retrieves metadata from the source and target clusters. You can use the default value for this item. | | meta\_job\_threads | The number of threads used by the migration tool to obtain metadata from the source and target clusters. You can use the default value for this item. | | ddl\_job\_interval\_seconds | The interval, in seconds, at which the migration tool executes DDL statements on the target cluster. You can use the default value for this item. | | ddl\_job\_batch\_size | The batch size for executing DDL statements on the target cluster. You can use the default value for this item. | | ddl\_job\_allow\_drop\_target\_only | Whether to allow the migration tool to delete databases or tables that exist only in the target cluster but not in the source cluster. The default is `false`, which means they will not be deleted. You can use the default value for this item. | | ddl\_job\_allow\_drop\_schema\_change\_table | Whether to allow the migration tool to delete tables with inconsistent schemas between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. The migration tool will automatically synchronize the deleted tables during the migration. | | ddl\_job\_allow\_drop\_inconsistent\_partition | Whether to allow the migration tool to delete partitions with inconsistent data distribution between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. The migration tool will automatically synchronize the deleted partitions during the migration. | | ddl\_job\_allow\_drop\_partition\_target\_only | Whether to allow the migration tool to delete partitions that are deleted in the source cluster to keep the partitions consistent between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. | | replication\_job\_interval\_seconds | The interval, in seconds, at which the migration tool triggers data synchronization tasks. You can use the default value for this item. | | replication\_job\_batch\_size | The batch size at which the migration tool triggers data synchronization tasks. You can use the default value for this item. | | max\_replication\_data\_size\_per\_job\_in\_gb | The data size threshold at which the migration tool triggers data synchronization tasks. Unit: GB. Multiple data synchronization tasks will be triggered if the size of the partition to be migrated exceed this value. The default value is `1024`. You can use the default value for this item. | | report\_interval\_seconds | The time interval at which the migration tool prints the progress information. Unit: Seconds. Default value: `300`. You can use the default value for this item. | | target\_cluster\_enable\_persistent\_index | Whether to enable persistent index the in the target cluster. If this item is not specified, the target cluster is consistent with the source cluster.
**NOTE**
When migrating data between two shared-data clusters, the tool automatically converts `persistent_index_type = LOCAL` to `CLOUD_NATIVE` in the CREATE TABLE statement for Primary Key tables. No manual action is needed. | | ddl\_job\_allow\_drop\_inconsistent\_time\_partition | Whether to allow the migration tool to delete partitions with inconsistent time between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. The migration tool will automatically synchronize the deleted partitions during the migration. | | enable\_bitmap\_index\_sync | Whether to enable synchronization for Bitmap indexes. | | ddl\_job\_allow\_drop\_inconsistent\_bitmap\_index | Whether to allow the migration tool to delete inconsistent Bitmap indexes between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. The migration tool will automatically synchronize the deleted indexes during the migration. | | ddl\_job\_allow\_drop\_bitmap\_index\_target\_only | Whether to allow the migration tool to delete Bitmap indexes that are deleted in the source cluster to keep the indexes consistent between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. | | enable\_materialized\_view\_sync | Whether to enable synchronization for materialized views. | | ddl\_job\_allow\_drop\_inconsistent\_materialized\_view | Whether to allow the migration tool to delete inconsistent materialized views between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. The migration tool will automatically synchronize the deleted materialized views during the migration. | | ddl\_job\_allow\_drop\_materialized\_view\_target\_only | Whether to allow the migration tool to delete materialized views that are deleted in the source cluster to keep the materialized views consistent between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. | | enable\_view\_sync | Whether to enable synchronization for views. | | ddl\_job\_allow\_drop\_inconsistent\_view | Whether to allow the migration tool to delete inconsistent views between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. The migration tool will automatically synchronize the deleted views during the migration. | | ddl\_job\_allow\_drop\_view\_target\_only | Whether to allow the migration tool to delete views that are deleted in the source cluster to keep the views consistent between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. | | enable\_table\_property\_sync | Whether to enable synchronization for table properties. | * Migrate from Shared-nothing * Migrate between Shared-data ##### Obtain Cluster Token[​](#obtain-cluster-token "Direct link to Obtain Cluster Token") note The Cluster Token is not required for migration between shared-data clusters. You can skip this step if you want to migrate data between shared-data source clusters. The Cluster Token is available in the FE metadata. Log in to the server where the FE node is located and run the following command: ```bash cat fe/meta/image/VERSION | grep token ``` Output: ```properties token=wwwwwwww-xxxx-yyyy-zzzz-uuuuuuuuuu ``` ##### Map storage volumes[​](#map-storage-volumes "Direct link to Map storage volumes") When the migration tool creates a table on the target cluster, it determines the table's storage volume as follows (in order of precedence): 1. If `target_cluster_use_builtin_storage_volume_only` is set to `true`, `builtin_storage_volume` is used for all tables. 2. If `target_cluster_storage_volume` is set to a specific storage volume, the specified storage volume is used for all tables. 3. Otherwise, by default, the source table's storage volume property is reserved. Tables from different source storage volumes are created under the corresponding storage volumes on the target cluster, provided those storage volumes exist on the target. Therefore, if you want to keep the storage volume property for each table in the source cluster, you can pre-create the same storage volumes in the target cluster. After migration, each table in the target cluster will inherit the storage volume property it had in the source cluster. Note that the `src_`-prefixed storage volumes serve a different purpose: they are used **only during migration** to give target CNs read access to the source cluster's object storage. After migration is complete, the `src_` volumes are no longer needed and can be dropped. ##### Network-related configuration (Optional)[​](#network-related-configuration-optional "Direct link to Network-related configuration (Optional)") * Migrate from Shared-nothing * Migrate between Shared-data During data migration, the migration tool needs to access **all** FE nodes of both the source and target clusters, and the target cluster needs to access **all** BE and CN nodes of the source cluster. You can obtain the network addresses of these nodes by executing the following statements on the corresponding cluster: ```sql -- Obtain the network addresses of FE nodes in a cluster. SHOW FRONTENDS; -- Obtain the network addresses of BE nodes in a cluster. SHOW BACKENDS; -- Obtain the network addresses of CN nodes in a cluster. SHOW COMPUTE NODES; ``` If these nodes use private addresses that cannot be accessed outside the cluster, such as internal network addresses within a Kubernetes cluster, you need to map these private addresses to addresses that can be accessed from outside. Navigate to the extracted folder of the tool and modify the configuration file **conf/hosts.properties**. ```bash cd starrocks-cluster-sync vi conf/hosts.properties ``` The default content of the file is as follows, describing how network address mapping is configured: ```properties # _=[;:[,:...]] ``` note The `` must match the address shown in the `IP` column returned by `SHOW FRONTENDS`, `SHOW BACKENDS`, or `SHOW COMPUTE NODES`. The following example performs these operations: 1. Map the source cluster's private network addresses `192.1.1.1` and `192.1.1.2` to `10.1.1.1` and `10.1.1.2`. 2. Map the source cluster's FE ports `8030` and `9030` to `38030` and `39030` on `10.1.1.1`. 3. Map the target cluster's private network address `fe-0.starrocks.svc.cluster.local` to `10.1.2.1` and remap port `9030`. ```properties # _=[;:[,:...]] SOURCE_192.1.1.1=10.1.1.1;8030:38030,9030:39030 SOURCE_192.1.1.2=10.1.1.2 TARGET_fe-0.starrocks.svc.cluster.local=10.1.2.1;9030:19030 ``` During data migration, the migration tool needs to access **all** FE nodes of both the source and target clusters. note Unlike migration from shared-nothing clusters, you do **not** need to configure network access from the target cluster to the source cluster's CN nodes, because data is transferred directly between object storage systems. You can obtain the FE network addresses by executing the following statement on the corresponding cluster: ```sql -- FE nodes SHOW FRONTENDS; ``` If FE nodes use private addresses that cannot be accessed outside the cluster, such as internal network addresses within a Kubernetes cluster, you need to map these private addresses to addresses that can be accessed from outside. Navigate to the extracted folder of the tool and modify the configuration file **conf/hosts.properties**. ```bash cd starrocks-cluster-sync vi conf/hosts.properties ``` The default content of the file is as follows, describing how network address mapping is configured: ```properties # _=[;:[,:...]] ``` note The `` must match the address shown in the `IP` column returned by `SHOW FRONTENDS`. The following example maps the target cluster's internal Kubernetes FQDN to a reachable IP: ```properties TARGET_frontend-0.frontend.mynamespace.svc.cluster.local=10.1.2.1;9030:19030 ``` #### Step 3: Start the Migration Tool[​](#step-3-start-the-migration-tool "Direct link to Step 3: Start the Migration Tool") After configuring the tool, start the migration tool to initiate the data migration process. ```bash ./bin/start.sh ``` note * If you are migrating data from a shared-nothing cluster, make sure that the BE nodes of the source and target clusters can properly communicate via the network. * During runtime, the migration tool regularly checks whether the data in the target cluster is lagging behind the source cluster. If there is a lag, it initiates data migration tasks. * If new data is constantly loaded into the source cluster, data synchronization will continue until the data in the target cluster is consistent with that in the source cluster. * You can query tables in the target cluster during migration, but do not load new data into the tables, as it may result in inconsistencies between the data in the target cluster and the source cluster. Currently, the migration tool does not forbid data loading into the target cluster during migration. * Note that data migration does not automatically terminate. You need to manually check and confirm the completion of migration and then stop the migration tool. #### View Migration Progress[​](#view-migration-progress "Direct link to View Migration Progress") ##### View Migration Tool logs[​](#view-migration-tool-logs "Direct link to View Migration Tool logs") You can check the migration progress through the migration tool log **log/sync.INFO.log**. Example 1: View task progress. ![img](/assets/images/data_migration_tool-1-73d324f48ca1372ae6a4c8c7499d6a15.png) The important metrics are as follows: * `Sync job progress`: The progress of data migration. The migration tool regularly checks whether the data in the target cluster is lagging behind the source cluster. Therefore, a progress of 100% only means that the data synchronization is completed within the current check interval. If new data continues to be loaded into the source cluster, the progress may decrease in the next check interval. * `total`: The total number of all types of jobs in this migration operation. * `ddlPending`: The number of DDL jobs pending to be executed. * `jobPending`: The number of pending data synchronization jobs to be executed. * `sent`: The number of data synchronization jobs sent from the source cluster but not yet started. Theoretically, this value should not be too large. If the value keeps increasing, please contact our engineers. * `running`: The number of data synchronization jobs that are currently running. * `finished`: The number of data synchronization jobs that are finished. * `failed`: The number of failed data synchronization jobs. Failed data synchronization jobs will be resent. Therefore, in most cases, you can ignore this metric. If this value is significantly large, please contact our engineers. * `unknown`: The number of jobs with an unknown status. Theoretically, this value should always be `0`. If this value is not `0`, please contact our engineers. Example 2: View the table migration progress. ![img](/assets/images/data_migration_tool-2-fc47ac5133c6a1618e5738f88f8694d4.png) * `Sync table progress`: Table migration progress, that is, the ratio of tables that have been migrated in this migration task to all the tables that need to be migrated. * `finishedTableRatio`: Ratio of tables with at least one successful synchronization task execution. * `expiredTableRatio`: Ratio of tables with expired data. * `total table`: Total number of tables involved in this data migration progress. * `finished table`: Number of tables with at least one successful synchronization task execution. * `unfinished table`: Number of tables with no synchronization task execution. * `expired table`: Number of tables with expired data. ##### View Migration Transaction Status[​](#view-migration-transaction-status "Direct link to View Migration Transaction Status") The migration tool opens a transaction for each table. You can view the status of the migration for a table by checking the status of its corresponding transaction. ```sql SHOW PROC "/transactions//running"; ``` `` is the name of the database where the table is located. ##### View Partition Data Versions[​](#view-partition-data-versions "Direct link to View Partition Data Versions") You can compare the data versions of the corresponding partitions in the source and target clusters to view the migration status of that partition. ```sql SHOW PARTITIONS FROM ; ``` `` is the name of the table to which the partition belongs. ##### View Data Volume[​](#view-data-volume "Direct link to View Data Volume") You can compare the data volumes in the source and target clusters to view the migration status. ```sql SHOW DATA; ``` ##### View Table Row Count[​](#view-table-row-count "Direct link to View Table Row Count") You can compare the row counts of tables in the source and target clusters to view the migration status of each table. ```sql SELECT TABLE_NAME, TABLE_ROWS FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' ORDER BY TABLE_NAME; ``` #### After Migration[​](#after-migration "Direct link to After Migration") When `Sync job progress` has been stable at 100% and your business is ready to switch, complete the cutover as follows: 1. Stop writes to the source cluster. 2. Verify `Sync job progress` reaches and remains at 100% after writes stop. 3. Stop the migration tool. 4. Point your applications to the target cluster address. 5. If you migrated data between shared-data clusters, restore the Auto-Vacuum setting on the source cluster: ```sql ADMIN SET FRONTEND CONFIG("lake_autovacuum_grace_period_minutes"="30"); ``` 6. If you migrated data between shared-data clusters, re-enable Compaction on the target cluster. Remove `lake_compaction_max_tasks = 0` from **fe.conf** and execute: ```sql ADMIN SET FRONTEND CONFIG("lake_compaction_max_tasks"="-1"); ``` 7. Disable Legacy Compatibility for Replication on the target cluster. Remove `enable_legacy_compatibility_for_replication = true` from **fe.conf** and execute: ```sql ADMIN SET FRONTEND CONFIG("enable_legacy_compatibility_for_replication"="false"); ``` #### Limits[​](#limits "Direct link to Limits") The list of objects that support synchronization currently is as follows (those not included indicate that synchronization is not supported): * Databases * Internal tables and their data * Materialized view schemas and their building statements (The data in the materialized view will not be synchronized. And if the base tables of the materialized view is not synchronized to the target cluster, the background refresh task of the materialized view reports an error.) * Logical views For migration between shared-data clusters: * The target cluster must be running on v4.1 or later. * Migration from a shared-data cluster to a shared-nothing target is not supported. * Each storage volume used by the source cluster's tables must have a corresponding `src_` storage volume pre-created on the target cluster. --- ### Data Recovery Recover mistakenly deleted databases/tables/partitions. After `drop table` or `drop database`, StarRocks will not physically delete the data immediately, but keep it in Trash for a period of time (1 day by default). Administrators can recover the mistakenly deleted data with the `RECOVER` command. #### Related commands[​](#related-commands "Direct link to Related commands") Syntax: ```sql -- 1) Recover database RECOVER DATABASE db_name; -- 2) Recover table RECOVER TABLE [db_name.]table_name; -- 3) Recover partition RECOVER PARTITION partition_name FROM [db_name.]table_name; ``` #### Notes[​](#notes "Direct link to Notes") 1. This operation can only restore the deleted meta information. The default time is 1 day, which can be configured by the `catalog_trash_expire_second` parameter in `fe.conf`. 2. If new meta information of the same name and type is created after the meta information is deleted, the previously deleted meta information cannot be restored. #### Examples[​](#examples "Direct link to Examples") 1. Recover the database named `example_db` ```sql RECOVER DATABASE example_db; ``` 2. Recover the table named `example_tbl` ```sql RECOVER TABLE example_db.example_tbl; ``` 3. Recover the partition named `p1` in the table `example_tbl` ```sql RECOVER PARTITION p1 FROM example_tbl; ``` --- ### HTTP Interface To facilitate the maintenance of StarRocks clusters, StarRocks provides various types of operation and query interfaces. This topic introduces these HTTP interfaces and their usage. #### FE[​](#fe "Direct link to FE") | Request Method | Request Path | Description | | -------------- | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | PUT | `/api/{db}/{table}/_stream_load` | Stream Load operation, see [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) for details. | | POST/PUT | `/api/transaction/{txn_op}` | Stream Load transaction interface, see [Stream Load Transaction Interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) for details. | | GET | `/api/{db}/_load_info` | | | GET | `/api/_set_config?config_key1=config_value1` | Update FE configuration. | | GET | `/api/_get_ddl?db={}&tbl={}` | View table DDL statement. | | GET | `/api/_migration?db={}&tbl={}` | View table tablet information. | | GET | `/api/_check_storagetype` | | | POST | `/api/{db}/{table}/_cancel?label={}` | | | GET | `/api/{db}/get_load_state` | | | GET | `/api/health` | | | GET | `/metrics?type={core/json}` | View metrics of the current FE. | | GET | `/api/show_meta_info` | | | GET | `/api/show_proc` | | | GET | `/api/show_runtime_info` | | | HEAD/GET | `/api/get_log_file` | | | GET | `/api/get_small_file` | | | GET | `/api/rowcount` | | | GET | `/api/check_decommission` | | | GET | `/api/_meta_replay_state` | | | POST | `/api/colocate/bucketseq` | | | GET | `/api/colocate` | | | POST | `/api/colocate/group_stable` | | | POST | `/api/colocate/group_unstable` | | | POST | `/api/colocate/update_group` | | | POST | `/api/global_dict/table/enable` | | | GET | `/api/profile?query_id={}` | Get profile information for the specified query ID. If the FE configuration item `authorization_enable_query_profile_access_check` is set to `true`, only the user who ran the query or a user with the SYSTEM-level OPERATE privilege can read it. | | GET | `/api/query_detail` | Get query details. See [Query detail API](https://docs.starrocks.io/docs/administration/http_interface/query_detail.md). | | GET | `/api/connection` | | | GET | `/api/show_data?db={}` | Query the size of the specified database. | | POST | `/api/query_dump` | Get query dump information, see [Query Dump](https://docs.starrocks.io/docs/faq/Dump_query.md) for details. | | GET | `/api/stop` | | | GET | `/image` | | | GET | `/info` | | | GET | `/version` | | | GET | `/put` | | | GET | `/journal_id` | | | GET | `/check` | | | GET | `/dump` | | | GET | `/role` | | | GET | `/api/{db}/{table}/_count` | | | GET | `/api/{db}/{table}/_schema` | View table schema. | | GET/POST | `/api/{db}/{table}/_query_plan` | | #### BE[​](#be "Direct link to BE") | HTTP Request Method | HTTP Request Path | Description | | ------------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | PUT | `/api/{db}/{table}/_stream_load` | Stream Load operation, see [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) for details. | | POST/PUT | `/api/transaction/{txn_op}` | Stream Load transaction interface, see [Stream Load Transaction Interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) for details. | | PUT | `/api/transaction/load` | | | HEAD/GET | `/api/_download_load` | | | HEAD/GET | `/api/_tablet/_download` | | | HEAD/GET | `/api/_load_error_log` | | | GET | `/api/health` | | | GET | `/api/_stop_be` | | | GET | `/pprof/heap` | | | GET | `/pprof/growth` | | | GET | `/pprof/profile` | | | GET | `/pprof/pmuprofile` | | | GET | `/pprof/contention` | | | GET | `/pprof/cmdline` | | | HEAD/GET/POST | `/pprof/symbol` | | | GET | `/metrics` | View metrics of the current BE. | | HEAD | `/api/meta/header/{tablet_id}` | | | GET | `/api/checksum` | | | GET | `/api/reload_tablet` | | | POST | `/api/restore_tablet` | | | GET | `/api/snapshot` | | | GET | `/api/compaction/show?tablet_id={}` | View compaction information for the specified tablet. | | POST | `/api/compact?tablet_id={}&compaction_type={base/cumulative}` | Manually perform compaction on the specified tablet. | | GET | `/api/compaction/show_repair` | | | PUT | `/api/compaction/submit_repair` | | | POST | `/api/update_config` | Update BE configuration, see [Update BE Configuration](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md) for details. | | GET/PUT | `/api/runtime_filter_cache/{action}` | | | POST | `/api/compact_rocksdb_meta` | | | GET/PUT | `/api/query_cache/{action}` | | | GET | `/api/pipeline_blocking_drivers/{action}` | | | GET | `/greplog` | | | GET | `/varz` | View current BE configuration. | #### CN[​](#cn "Direct link to CN") | Request Method | Request Path | Description | | -------------- | ------------------- | ------------------------------- | | GET | `/api/health` | | | GET | `/pprof/heap` | | | GET | `/pprof/growth` | | | GET | `/pprof/profile` | | | GET | `/pprof/pmuprofile` | | | GET | `/pprof/contention` | | | GET | `/pprof/cmdline` | | | HEAD/GET/POST | `/pprof/symbol` | | | GET | `/metrics` | View metrics of the current CN. | --- ### Query detail API The **query detail** API returns recent query execution details that are cached in FE memory. note Query detail records are collected only when the FE configuration `enable_collect_query_detail_info` is set to `true`. #### Endpoints[​](#endpoints "Direct link to Endpoints") * `GET /api/query_detail` (v1) * `GET /api/v2/query_detail` (v2, supports `is_request_all_frontend` and returns a wrapped result) #### Parameters[​](#parameters "Direct link to Parameters") | Name | Required | Default | Description | | ------------------------- | ------------ | ------- | ------------------------------------------------------------------------------------------------------------------------- | | `event_time` | Yes | - | Lower bound filter. Returns items whose `eventTime` is greater than this value. You can pass `0` to get all cached items. | | `user` | No | - | Filter by the `user` field. Case-insensitive match. | | `is_request_all_frontend` | No (v2 only) | `false` | When `true`, the current FE queries other alive FEs and merges their results. | #### Response[​](#response "Direct link to Response") * v1 returns a JSON array of `QueryDetail` objects. * v2 returns `{ "code": "0", "message": "OK", "result": [ ... ] }` where `result` is a `QueryDetail` list. #### Authentication and authorization[​](#authentication-and-authorization "Direct link to Authentication and authorization") This API requires HTTP Basic authentication. By default there is no additional privilege check beyond a successful login: any authenticated user can access the endpoint and can view all cached query details unless a `user` filter is applied. If the FE configuration item `authorization_enable_query_profile_access_check` is set to `true`, the `profile` field, and the `explain` field when it holds a plan rendered from that profile, are returned only for queries run by the current user, or for every query if the user has the SYSTEM-level OPERATE privilege. Other records are still listed, with those fields omitted. #### QueryDetail fields[​](#querydetail-fields "Direct link to QueryDetail fields") | Field | Type | Description | | ------------------- | ------- | --------------------------------------------------------------------------------------------------- | | `queryId` | string | Query ID. | | `eventTime` | long | Internal timestamp used for filtering. Monotonic nanosecond timestamp derived from wall-clock time. | | `isQuery` | boolean | Whether the statement is a query. | | `remoteIP` | string | Client IP address or `System`. | | `connId` | int | Connection ID. | | `startTime` | long | Query start time in milliseconds since epoch. | | `endTime` | long | Query end time in milliseconds since epoch. `-1` if not finished. | | `latency` | long | Query latency in milliseconds. `-1` if not finished. | | `pendingTime` | long | Pending time in milliseconds. | | `netTime` | long | Net execution time in milliseconds. | | `netComputeTime` | long | Net compute time in milliseconds. | | `state` | string | One of `RUNNING`, `FINISHED`, `FAILED`, `CANCELLED`. | | `database` | string | Current database. | | `sql` | string | SQL text (may be desensitized if configured). | | `user` | string | Login user (qualified user). | | `impersonatedUser` | string | Target user of `EXECUTE AS`. `null` if not executing as another user. | | `errorMessage` | string | Error message when failed. | | `explain` | string | Explain plan (level controlled by `query_detail_explain_level`). | | `profile` | string | Profile text if collected. | | `resourceGroupName` | string | Resource group name. | | `scanRows` | long | Scanned rows. | | `scanBytes` | long | Scanned bytes. | | `returnRows` | long | Returned rows. | | `cpuCostNs` | long | CPU cost in nanoseconds. | | `memCostBytes` | long | Memory cost in bytes. | | `spillBytes` | long | Spill bytes. | | `cacheMissRatio` | float | Cache miss ratio in percent (0-100). | | `warehouse` | string | Warehouse name. | | `digest` | string | SQL digest. | | `catalog` | string | Catalog name. | | `command` | string | MySQL command name. | | `preparedStmtId` | string | Prepared statement ID. | | `queryFeMemory` | long | FE memory allocated by the query, in bytes. | | `querySource` | string | Query source: `EXTERNAL`, `INTERNAL`, `MV`, or `TASK`. | #### Examples[​](#examples "Direct link to Examples") ##### v1[​](#v1 "Direct link to v1") ```bash curl -u root: "http://:/api/query_detail?event_time=0" ``` ##### v2[​](#v2 "Direct link to v2") ```bash curl -u root: "http://:/api/v2/query_detail?event_time=0&is_request_all_frontend=true" ``` --- ### Manage audit logs within StarRocks via AuditLoader This topic describes how to manage StarRocks audit logs within a table via the plugin - AuditLoader. StarRocks stores its audit logs in the local file **fe/log/fe.audit.log** rather than an internal database. The plugin AuditLoader allows you to manage audit logs directly within your cluster. Once installed, AuditLoader reads logs from the file, and loads them into StarRocks via HTTP PUT. You can then query the audit logs in StarRocks using SQL statements. #### Create a table to store audit logs[​](#create-a-table-to-store-audit-logs "Direct link to Create a table to store audit logs") Create a database and a table in your StarRocks cluster to store its audit logs. See [CREATE DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/CREATE_DATABASE.md) and [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) for detailed instructions. Because the fields of audit logs vary among different StarRocks versions, it is important to follow recommendations mentioned below to avoid compatibility issues during upgrade: > **CAUTION** > > * All new fields should be marked as `NULL`. > * Fields should NOT be renamed, as users may rely on them. > * Only backward compatible changes should be applied to field types, e.g. `VARCHAR(32)` -> `VARCHAR(64)`, to avoid errors during insert. > * `AuditEvent` fields are resolved by name only. The order of columns within table doesn't matter, and can be changed by user in any time. > * `AuditEvent` fields which doesn't exist in the table are ignored, so users can remove columns they don't need. ```sql CREATE DATABASE starrocks_audit_db__; CREATE TABLE starrocks_audit_db__.starrocks_audit_tbl__ ( `queryId` VARCHAR(64) COMMENT "Unique ID of the query", `timestamp` DATETIME NOT NULL COMMENT "Query start time", `queryType` VARCHAR(12) COMMENT "Query type (query, slow_query, connection)", `clientIp` VARCHAR(32) COMMENT "Client IP", `user` VARCHAR(64) COMMENT "Query username", `authorizedUser` VARCHAR(64) COMMENT "Unique identifier of the user, i.e., user_identity", `resourceGroup` VARCHAR(64) COMMENT "Resource group name", `catalog` VARCHAR(32) COMMENT "Catalog name", `db` VARCHAR(96) COMMENT "Database where the query runs", `state` VARCHAR(8) COMMENT "Query state (EOF, ERR, OK)", `errorCode` VARCHAR(512) COMMENT "Error code", `errorMessage` VARCHAR(1048576) NULL COMMENT "Error message returned to the client", `queryTime` BIGINT COMMENT "Query execution time (milliseconds)", `scanBytes` BIGINT COMMENT "Number of bytes scanned by the query", `scanRows` BIGINT COMMENT "Number of rows scanned by the query", `returnRows` BIGINT COMMENT "Number of rows returned by the query", `cpuCostNs` BIGINT COMMENT "CPU time consumed by the query (nanoseconds)", `memCostBytes` BIGINT COMMENT "Memory consumed by the query (bytes)", `stmtId` INT COMMENT "Incremental ID of the SQL statement", `isQuery` TINYINT COMMENT "Whether the SQL is a query (1 or 0)", `feIp` VARCHAR(128) COMMENT "FE IP that executed the statement", `stmt` VARCHAR(1048576) COMMENT "Original SQL statement", `digest` VARCHAR(32) COMMENT "Fingerprint of slow SQL", `planCpuCosts` DOUBLE COMMENT "CPU usage during query planning (nanoseconds)", `planMemCosts` DOUBLE COMMENT "Memory usage during query planning (bytes)", `pendingTimeMs` BIGINT COMMENT "Time the query waited in the queue (milliseconds)", `candidateMVs` VARCHAR(65533) NULL COMMENT "List of candidate materialized views", `hitMvs` VARCHAR(65533) NULL COMMENT "List of matched materialized views", `QueriedRelations` ARRAY NULL COMMENT "List of directly referenced tables and views", `warehouse` VARCHAR(32) NULL COMMENT "Warehouse name" ) ENGINE = OLAP DUPLICATE KEY (`queryId`, `timestamp`, `queryType`) COMMENT "Audit log table" PARTITION BY date_trunc('day', `timestamp`) PROPERTIES ( "replication_num" = "1", "partition_live_number" = "30" ); ``` `starrocks_audit_tbl__` is created with dynamic partitions. By default, the first dynamic partition is created 10 minutes after the table is created. Audit logs can then be loaded into the table. You can check the partitions in the table using the following statement: ```sql SHOW PARTITIONS FROM starrocks_audit_db__.starrocks_audit_tbl__; ``` After a partition is created, you can move on to the next step. note If the table already exists from an earlier version, add the columns that were introduced since, otherwise the corresponding fields are silently dropped when the audit logs are loaded. For example, `errorMessage` is introduced in v4.2.0: ```sql ALTER TABLE starrocks_audit_db__.starrocks_audit_tbl__ ADD COLUMN `errorMessage` VARCHAR(1048576) NULL COMMENT "Error message returned to the client" AFTER `errorCode`; ``` Add the column before you configure the `filter` property of AuditLoader to reference it, because that filter is evaluated against the columns of the table. #### Download and configure AuditLoader[​](#download-and-configure-auditloader "Direct link to Download and configure AuditLoader") 1. [Download](https://releases.starrocks.io/resources/auditloader.zip) the AuditLoader installation package. The package is compatible with all available versions of StarRocks. 2. Unzip the installation package. ```shell unzip auditloader.zip ``` The following files are inflated: * **auditloader.jar**: the JAR file of AuditLoader. * **plugin.properties**: the properties file of AuditLoader. You do not need to modify this file. * **plugin.conf**: the configuration file of AuditLoader. In most cases, you only need to modify the `user` and `password` fields in the file. 3. Modify **plugin.conf** to configure AuditLoader. You must configure the following items to make sure AuditLoader can work properly: * `frontend_host_port`: FE IP address and HTTP port, in the format `:`. It is recommended to set it to its default value `127.0.0.1:8030`. Each FE in StarRocks manages its own Audit Log independently, and after installing the plugin, each FE will start its own background thread to fetch and save Audit Logs, and write them via Stream Load. The `frontend_host_port` configuration item is used to provide the IP and port of the HTTP protocol for the background Stream Load task of the plug-in, and this parameter does not support multiple values. The IP part of the parameter can use the IP of any FE in the cluster, but it is not recommended because if the corresponding FE crashes, the audit log writing task in the background of other FEs will also fail due to the failure of communication. It is recommended to set it to the default value `127.0.0.1:8030`, so that each FE uses its own HTTP port to communicate, thus avoiding the impact on the communication in case of an exception of the other FEs (all the write tasks will be forwarded to the FE Leader node to be executed eventually). * `database`: name of the database you created to host audit logs. * `table`: name of the table you created to host audit logs. * `user`: your cluster username. You MUST have the privilege to load data (LOAD\_PRIV) into the table. * `password`: your user password. * `secret_key`: the key (string, must not be longer than 16 bytes) used to encrypt the password. If this parameter is not set, it indicates that the password in **plugin.conf** will not be encrypted, and you only need to specify the plaintext password in `password`. If this parameter is specified, it indicates that the password is encrypted by this key, and you need to specify the encrypted string in `password`. The encrypted password can be generated in StarRocks using the `AES_ENCRYPT` function: `SELECT TO_BASE64(AES_ENCRYPT('password','secret_key'));`. * `filter`: the filter conditions for audit log loading. This parameter is based on the [WHERE parameter](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#opt_properties) in Stream Load, i.e. `-H “where: ”`, defaults to an empty string. Example: `filter=isQuery=1 and clientIp like '127.0.0.1%' and user='root'`. 4. Zip the files back into a package. ```shell zip -q -m -r auditloader.zip auditloader.jar plugin.conf plugin.properties ``` 5. Dispatch the package to all machines that host FE nodes. Make sure all packages are stored in an identical path. Otherwise, the installation fails. Remember to copy the absolute path to the package after you dispatched the package. > **NOTE** > > You can also distribute **auditloader.zip** to an HTTP service accessible to all FEs (for example, `httpd` or `nginx`) and install it using the network. Note that in both cases the **auditloader.zip** needs to be persisted in the path after the installation is performed, and the source files should not be deleted after installation. #### Install AuditLoader[​](#install-auditloader "Direct link to Install AuditLoader") Execute the following statement along with the path you copied to install AuditLoader as a plugin in StarRocks: ```sql INSTALL PLUGIN FROM ""; ``` Example of installation from a local package: ```sql INSTALL PLUGIN FROM ""; ``` If you want install the plugin via a network path, you need to provide the md5 of the package in the properties of the INSTALL statement. Example: ```sql INSTALL PLUGIN FROM "http://xx.xx.xxx.xxx/extra/auditloader.zip" PROPERTIES("md5sum" = "3975F7B880C9490FE95F42E2B2A28E2D"); ``` See [INSTALL PLUGIN](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md) for detailed instructions. #### Verify the installation and query audit logs[​](#verify-the-installation-and-query-audit-logs "Direct link to Verify the installation and query audit logs") 1. You can check if the installation is successful via [SHOW PLUGINS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plugin/SHOW_PLUGINS.md). In the following example, the `Status` of the plugin `AuditLoader` is `INSTALLED`, meaning installation is successful. ```plain mysql> SHOW PLUGINS\G *************************** 1. row *************************** Name: __builtin_AuditLogBuilder Type: AUDIT Description: builtin audit logger Version: 0.12.0 JavaVersion: 1.8.31 ClassName: com.starrocks.qe.AuditLogBuilder SoName: NULL Sources: Builtin Status: INSTALLED Properties: {} *************************** 2. row *************************** Name: AuditLoader Type: AUDIT Description: Available for versions 3.3.11+. Load audit log to starrocks, and user can view the statistic of queries Version: 5.0.0 JavaVersion: 11 ClassName: com.starrocks.plugin.audit.AuditLoaderPlugin SoName: NULL Sources: /x/xx/xxx/xxxxx/auditloader.zip Status: INSTALLED Properties: {} 2 rows in set (0.01 sec) ``` 2. Execute some random SQLs to generate audit logs, and wait for 60 seconds (or the time you have specified in the item `max_batch_interval_sec` when you configure AuditLoader) to allow AuditLoader to load audit logs into StarRocks. 3. Check the audit logs by querying the table. ```sql SELECT * FROM starrocks_audit_db__.starrocks_audit_tbl__; ``` The following example shows when audit logs are loaded into the table successfully: ```plain mysql> SELECT * FROM starrocks_audit_db__.starrocks_audit_tbl__\G *************************** 1. row *************************** queryId: 01975a33-4129-7520-97a2-05e641cec6c9 timestamp: 2025-06-10 14:16:37 queryType: query clientIp: xxx.xx.xxx.xx:65283 user: root authorizedUser: 'root'@'%' resourceGroup: default_wg catalog: default_catalog db: state: EOF errorCode: queryTime: 3 scanBytes: 0 scanRows: 0 returnRows: 1 cpuCostNs: 33711 memCostBytes: 4200 stmtId: 102 isQuery: 1 feIp: xxx.xx.xxx.xx stmt: SELECT * FROM starrocks_audit_db__.starrocks_audit_tbl__ digest: planCpuCosts: 908 planMemCosts: 0 pendingTimeMs: -1 candidateMvs: null hitMVs: null ``` QueriedRelations: \["default\_catalog.db1.tbl1","default\_catalog.db1.view1"] ………… ````text ## Troubleshooting If no audit logs are loaded to the table after the dynamic partition is created and the plugin is installed, you can check whether **plugin.conf** is configured properly or not. To modify it, you must first uninstall the plugin: ```SQL UNINSTALL PLUGIN AuditLoader; ```` Logs of AuditLoader are printed in **fe.log**, you can retrieve them by searching the keyword `audit` in **fe.log**. After all configurations are set correctly, you can follow the above steps to install AuditLoader again. --- ### Back up and restore data This topic describes how to back up and restore data in StarRocks, or migrate data to a new StarRocks cluster. StarRocks supports backing up data as snapshots into a remote storage system and restoring the data to any StarRocks clusters. From v3.4.0 onwards, StarRocks have enhanced the functionality of BACKUP and RESTORE by supporting more objects and refactoring the syntax for better flexibility. StarRocks supports the following remote storage systems: * Apache™ Hadoop® (HDFS) cluster * AWS S3 * Google GCS * MinIO StarRocks supports backing up the following objects: * Internal databases, tables (of all types and partitioning strategies), and partitions * Metadata of external catalogs (supported from v3.4.0 onwards) * Synchronous materialized views and asynchronous materialized views * Logical views (supported from v3.4.0 onwards) * User-defined functions (supported from v3.4.0 onwards) > **NOTE** > > Shared-data StarRocks clusters do not support data BACKUP and RESTORE. #### Create a repository[​](#create-a-repository "Direct link to Create a repository") Before backing up data, you need to create a repository, which is used to store data snapshots in a remote storage system. You can create multiple repositories in a StarRocks cluster. For detailed instructions, see [CREATE REPOSITORY](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/CREATE_REPOSITORY.md). * Create a repository in HDFS The following example creates a repository named `test_repo` in an HDFS cluster. ```sql CREATE REPOSITORY test_repo WITH BROKER ON LOCATION "hdfs://:/repo_dir/backup" PROPERTIES( "username" = "", "password" = "" ); ``` * Create a repository in AWS S3 You can choose IAM user-based credential (Access Key and Secret Key), Instance Profile, or Assumed Role as the credential method for accessing AWS S3. * The following example creates a repository named `test_repo` in the AWS S3 bucket `bucket_s3` using IAM user-based credentials as the credential method. ```sql CREATE REPOSITORY test_repo WITH BROKER ON LOCATION "s3a://bucket_s3/backup" PROPERTIES( "aws.s3.access_key" = "XXXXXXXXXXXXXXXXX", "aws.s3.secret_key" = "yyyyyyyyyyyyyyyyyyyyyyyy", "aws.s3.region" = "us-east-1" ); ``` * The following example creates a repository named `test_repo` in the AWS S3 bucket `bucket_s3` using Instance Profile as the credential method. ```sql CREATE REPOSITORY test_repo WITH BROKER ON LOCATION "s3a://bucket_s3/backup" PROPERTIES( "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-east-1" ); ``` * The following example creates a repository named `test_repo` in the AWS S3 bucket `bucket_s3` using Assumed Role as the credential method. ```sql CREATE REPOSITORY test_repo WITH BROKER ON LOCATION "s3a://bucket_s3/backup" PROPERTIES( "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::xxxxxxxxxx:role/yyyyyyyy", "aws.s3.region" = "us-east-1" ); ``` > **NOTE** > > StarRocks supports creating repositories in AWS S3 only according to the S3A protocol. Therefore, when you create repositories in AWS S3, you must replace `s3://` in the S3 URI you pass as a repository location in `ON LOCATION` with `s3a://`. * Create a repository in Google GCS The following example creates a repository named `test_repo` in the Google GCS bucket `bucket_gcs`. ```sql CREATE REPOSITORY test_repo WITH BROKER ON LOCATION "s3a://bucket_gcs/backup" PROPERTIES( "fs.s3a.access.key" = "xxxxxxxxxxxxxxxxxxxx", "fs.s3a.secret.key" = "yyyyyyyyyyyyyyyyyyyy", "fs.s3a.endpoint" = "storage.googleapis.com" ); ``` > **NOTE** > > * StarRocks supports creating repositories in Google GCS only according to the S3A protocol. Therefore, when you create repositories in Google GCS, you must replace the prefix in the GCS URI you pass as a repository location in `ON LOCATION` with `s3a://`. > * Do not specify `https` in the endpoint address. * Create a repository in MinIO The following example creates a repository named `test_repo` in the MinIO bucket `bucket_minio`. ```sql CREATE REPOSITORY test_repo WITH BROKER ON LOCATION "s3://bucket_minio/backup" PROPERTIES( "aws.s3.access_key" = "XXXXXXXXXXXXXXXXX", "aws.s3.secret_key" = "yyyyyyyyyyyyyyyyy", "aws.s3.endpoint" = "http://minio:9000" ); ``` After the repository is created, you can check the repository via [SHOW REPOSITORIES](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_REPOSITORIES.md). After restoring data, you can delete the repository in StarRocks using [DROP REPOSITORY](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/DROP_REPOSITORY.md). However, data snapshots backed up in the remote storage system cannot be deleted through StarRocks. You need to delete them manually in the remote storage system. #### Back up data[​](#back-up-data "Direct link to Back up data") After the repository is created, you need to create a data snapshot and back up it in the remote repository. For detailed instructions, see [BACKUP](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/BACKUP.md). BACKUP is an asynchronous operation. You can check the status of a BACKUP job using [SHOW BACKUP](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_BACKUP.md), or cancel a BACKUP job using [CANCEL BACKUP](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/CANCEL_BACKUP.md). StarRocks supports FULL backup on the granularity level of database, table, or partition. If you have stored a large amount of data in a table, we recommend that you back up and restore data by partition. This way, you can reduce the cost of retries in case of job failures. If you need to back up incremental data on a regular basis, you can configure a [partitioning plan](https://docs.starrocks.io/docs/table_design/data_distribution.md#partitioning) for your table, and back up only new partitions each time. ##### Back up database[​](#back-up-database "Direct link to Back up database") Performing a full BACKUP on a database will back up all tables, synchronous and asynchronous materialized views, logical views, and UDFs within the database. The following examples back up the database `sr_hub` in the snapshot `sr_hub_backup` and upload the snapshot to the repository `test_repo`. ```sql -- Supported from v3.4.0 onwards. BACKUP DATABASE sr_hub SNAPSHOT sr_hub_backup TO test_repo; -- Compatible with the syntax in earlier versions. BACKUP SNAPSHOT sr_hub.sr_hub_backup TO test_repo; ``` ##### Back up table[​](#back-up-table "Direct link to Back up table") StarRocks supports backing up and restoring tables of all types and partitioning strategies. Performing a full BACKUP on a table will back up the table and the synchronous materialized views built on it. The following examples back up the table `sr_member` from the database `sr_hub` in the snapshot `sr_member_backup` and upload the snapshot to the repository `test_repo`. ```sql -- Supported from v3.4.0 onwards. BACKUP DATABASE sr_hub SNAPSHOT sr_member_backup TO test_repo ON (TABLE sr_member); -- Compatible with the syntax in earlier versions. BACKUP SNAPSHOT sr_hub.sr_member_backup TO test_repo ON (sr_member); ``` The following example backs up two tables, `sr_member` and `sr_pmc`, from the database `sr_hub` in the snapshot `sr_core_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_core_backup TO test_repo ON (TABLE sr_member, TABLE sr_pmc); ``` The following example backs up all tables from the database `sr_hub` in the snapshot `sr_all_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_all_backup TO test_repo ON (ALL TABLES); ``` ##### Back up partition[​](#back-up-partition "Direct link to Back up partition") The following examples back up the partition `p1` of the table `sr_member` from the database `sr_hub` in the snapshot `sr_par_backup` and upload the snapshot to the repository `test_repo`. ```sql -- Supported from v3.4.0 onwards. BACKUP DATABASE sr_hub SNAPSHOT sr_par_backup TO test_repo ON (TABLE sr_member PARTITION (p1)); -- Compatible with the syntax in earlier versions. BACKUP SNAPSHOT sr_hub.sr_par_backup TO test_repo ON (sr_member PARTITION (p1)); ``` You can specify multiple partition names separated by commas (`,`) to back up partitions in batch. ##### Back up materialized view[​](#back-up-materialized-view "Direct link to Back up materialized view") You do not need to manually back up synchronous materialized views because they will be backed up along with the BACKUP operation of the base table. Asynchronous materialized views can be backed up along with the BACKUP operation of the database it belongs to. You can also manually back up them. The following example backs up the materialized view `sr_mv1` from the database `sr_hub` in the snapshot `sr_mv1_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_mv1_backup TO test_repo ON (MATERIALIZED VIEW sr_mv1); ``` The following example backs up two materialized views, `sr_mv1` and `sr_mv2`, from the database `sr_hub` in the snapshot `sr_mv2_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_mv2_backup TO test_repo ON (MATERIALIZED VIEW sr_mv1, MATERIALIZED VIEW sr_mv2); ``` The following example backs up all materialized views from the database `sr_hub` in the snapshot `sr_mv3_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_mv3_backup TO test_repo ON (ALL MATERIALIZED VIEWS); ``` ##### Back up logical view[​](#back-up-logical-view "Direct link to Back up logical view") The following example backs up the logical view `sr_view1` from the database `sr_hub` in the snapshot `sr_view1_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_view1_backup TO test_repo ON (VIEW sr_view1); ``` The following example backs up two logical views, `sr_view1` and `sr_view2`, from the database `sr_hub` in the snapshot `sr_view2_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_view2_backup TO test_repo ON (VIEW sr_view1, VIEW sr_view2); ``` The following example backs up all logical views from the database `sr_hub` in the snapshot `sr_view3_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_view3_backup TO test_repo ON (ALL VIEWS); ``` ##### Back up UDF[​](#back-up-udf "Direct link to Back up UDF") The following example backs up the UDF `sr_udf1` from the database `sr_hub` in the snapshot `sr_udf1_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_udf1_backup TO test_repo ON (FUNCTION sr_udf1); ``` The following example backs up two UDFs, `sr_udf1` and `sr_udf2`, from the database `sr_hub` in the snapshot `sr_udf2_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_udf2_backup TO test_repo ON (FUNCTION sr_udf1, FUNCTION sr_udf2); ``` The following example backs up all UDFs from the database `sr_hub` in the snapshot `sr_udf3_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_udf3_backup TO test_repo ON (ALL FUNCTIONS); ``` ##### Back up metadata of external catalog[​](#back-up-metadata-of-external-catalog "Direct link to Back up metadata of external catalog") The following example backs up the metadata of the external catalog `iceberg` in the snapshot `iceberg_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP EXTERNAL CATALOG (iceberg) SNAPSHOT iceberg_backup TO test_repo; ``` The following example backs up the metadata of two external catalogs, `iceberg` and `hive`, in the snapshot `iceberg_hive_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP EXTERNAL CATALOGS (iceberg, hive) SNAPSHOT iceberg_hive_backup TO test_repo; ``` The following example backs up the metadata of all external catalogs in the snapshot `all_catalog_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP ALL EXTERNAL CATALOGS SNAPSHOT all_catalog_backup TO test_repo; ``` To cancel the BACKUP operation on external catalogs, execute the following statement: ```sql CANCEL BACKUP FOR EXTERNAL CATALOG; ``` #### Restore data[​](#restore-data "Direct link to Restore data") You can restore the data snapshot backed up in the remote storage system to the current or other StarRocks clusters to restore or migrate data. **When you restore an object from a snapshot, you must specify the timestamp of the snapshot.** Use the [RESTORE](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/RESTORE.md) statement to restore data snapshots in the remote storage system. RESTORE is an asynchronous operation. You can check the status of a RESTORE job using [SHOW RESTORE](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_RESTORE.md), or cancel a RESTORE job using [CANCEL RESTORE](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/CANCEL_RESTORE.md). ##### (Optional) Create a repository in the new cluster[​](#optional-create-a-repository-in-the-new-cluster "Direct link to (Optional) Create a repository in the new cluster") To migrate data to another StarRocks cluster, you need to create a repository with the same **repository name** and **location** in the target cluster, otherwise, you will not be able to view the previously backed-up data snapshots. See [Create a repository](#create-a-repository) for details. ##### Obtain snapshot timestamp[​](#obtain-snapshot-timestamp "Direct link to Obtain snapshot timestamp") Before restoring data, you can check the snapshots in the repository to obtain the timestamps using [SHOW SNAPSHOT](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_SNAPSHOT.md). The following example checks the snapshot information in `test_repo`. ```plain mysql> SHOW SNAPSHOT ON test_repo; +------------------+-------------------------+--------+ | Snapshot | Timestamp | Status | +------------------+-------------------------+--------+ | sr_member_backup | 2023-02-07-14-45-53-143 | OK | +------------------+-------------------------+--------+ 1 row in set (1.16 sec) ``` ##### Restore database[​](#restore-database "Direct link to Restore database") The following examples restore the database `sr_hub` in the snapshot `sr_hub_backup` to the database `sr_hub` in the target cluster. If the database does not exist in the snapshot, the system will return an error. If the database does not exist in the target cluster, the system will create it automatically. ```sql -- Supported from v3.4.0 onwards. RESTORE SNAPSHOT sr_hub_backup FROM test_repo DATABASE sr_hub PROPERTIES("backup_timestamp" = "2024-12-09-10-25-58-842"); -- Compatible with the syntax in earlier versions. RESTORE SNAPSHOT sr_hub.sr_hub_backup FROM `test_repo` PROPERTIES("backup_timestamp" = "2024-12-09-10-25-58-842"); ``` The following example restore the database `sr_hub` in the snapshot `sr_hub_backup` to the database `sr_hub_new` in the target cluster. If the database `sr_hub` does not exist in the snapshot, the system will return an error. If the database `sr_hub_new` does not exist in the target cluster, the system will create it automatically. ```sql -- Supported from v3.4.0 onwards. RESTORE SNAPSHOT sr_hub_backup FROM test_repo DATABASE sr_hub AS sr_hub_new PROPERTIES("backup_timestamp" = "2024-12-09-10-25-58-842"); ``` ##### Restore table[​](#restore-table "Direct link to Restore table") The following examples restore the table `sr_member` of the database `sr_hub` in the snapshot `sr_member_backup` to the table `sr_member` of the database `sr_hub` in the target cluster. ```sql -- Supported from v3.4.0 onwards. RESTORE SNAPSHOT sr_member_backup FROM test_repo DATABASE sr_hub ON (TABLE sr_member) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Compatible with the syntax in earlier versions. RESTORE SNAPSHOT sr_hub.sr_member_backup FROM test_repo ON (sr_member) PROPERTIES ("backup_timestamp"="2024-12-09-10-52-10-940"); ``` The following examples restore the table `sr_member` of the database `sr_hub` in the snapshot `sr_member_backup` to the table `sr_member_new` of the database `sr_hub_new` in the target cluster. ```sql RESTORE SNAPSHOT sr_member_backup FROM test_repo DATABASE sr_hub AS sr_hub_new ON (TABLE sr_member AS sr_member_new) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores two tables, `sr_member` and `sr_pmc`, of the database `sr_hub` in the snapshot `sr_core_backup` to two tables, `sr_member` and `sr_pmc`, of the database `sr_hub` in the target cluster. ```sql RESTORE SNAPSHOT sr_core_backup FROM test_repo DATABASE sr_hub ON (TABLE sr_member, TABLE sr_pmc) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores all tables from the database `sr_hub` in the snapshot `sr_all_backup`. ```sql RESTORE SNAPSHOT sr_all_backup FROM test_repo DATABASE sr_hub ON (ALL TABLES); ``` The following example restores one of all tables from the database `sr_hub` in the snapshot `sr_all_backup`. ```sql RESTORE SNAPSHOT sr_all_backup FROM test_repo DATABASE sr_hub ON (TABLE sr_member) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` ##### Restore partition[​](#restore-partition "Direct link to Restore partition") The following examples restore the partition `p1` of the table `sr_member` in the snapshot `sr_par_backup` to the partition `p1` of the table `sr_member` in the target cluster. ```sql -- Supported from v3.4.0 onwards. RESTORE SNAPSHOT sr_par_backup FROM test_repo DATABASE sr_hub ON (TABLE sr_member PARTITION (p1)) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Compatible with the syntax in earlier versions. RESTORE SNAPSHOT sr_hub.sr_par_backup FROM test_repo ON (sr_member PARTITION (p1)) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` You can specify multiple partition names separated by commas (`,`) to restore partitions in batch. ##### Restore materialized view[​](#restore-materialized-view "Direct link to Restore materialized view") The following example restores the materialized view `sr_mv1` from the database `sr_hub` in the snapshot `sr_mv1_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_mv1_backup FROM test_repo DATABASE sr_hub ON (MATERIALIZED VIEW sr_mv1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores two materialized views, `sr_mv1` and `sr_mv2`, from the database `sr_hub` in the snapshot `sr_mv2_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_mv2_backup FROM test_repo DATABASE sr_hub ON (MATERIALIZED VIEW sr_mv1, MATERIALIZED VIEW sr_mv2) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores all materialized views from the database `sr_hub` in the snapshot `sr_mv3_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_mv3_backup FROM test_repo DATABASE sr_hub ON (ALL MATERIALIZED VIEWS) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores one of the materialized views from the database `sr_hub` in the snapshot `sr_mv3_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_mv3_backup FROM test_repo DATABASE sr_hub ON (MATERIALIZED VIEW sr_mv1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` info After RESTORE, you can check the status of the materialized view using [SHOW MATERIALIZED VIEWS](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/SHOW_MATERIALIZED_VIEW.md). * If the materialized view is active, it can be used directly. * If the materialized view is inactive, it might be because its base tables are not restored. After all the base tables are restored, you can use [ALTER MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/ALTER_MATERIALIZED_VIEW.md) to re-activate the materialized view. ##### Restore logical view[​](#restore-logical-view "Direct link to Restore logical view") The following example restores the logical view `sr_view1` from the database `sr_hub` in the snapshot `sr_view1_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_view1_backup FROM test_repo DATABASE sr_hub ON (VIEW sr_view1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores two logical views, `sr_view1` and `sr_view2`, from the database `sr_hub` in the snapshot `sr_view2_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_view2_backup FROM test_repo DATABASE sr_hub ON (VIEW sr_view1, VIEW sr_view2) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores all logical views from the database `sr_hub` in the snapshot `sr_view3_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_view3_backup FROM test_repo DATABASE sr_hub ON (ALL VIEWS) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores one of all logical views from the database `sr_hub` in the snapshot `sr_view3_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_view3_backup FROM test_repo DATABASE sr_hub ON (VIEW sr_view1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` ##### Restore UDF[​](#restore-udf "Direct link to Restore UDF") The following example restores the UDF `sr_udf1` from the database `sr_hub` in the snapshot `sr_udf1_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_udf1_backup FROM test_repo DATABASE sr_hub ON (FUNCTION sr_udf1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores two UDFs, `sr_udf1` and `sr_udf2`, from the database `sr_hub` in the snapshot `sr_udf2_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_udf2_backup FROM test_repo DATABASE sr_hub ON (FUNCTION sr_udf1, FUNCTION sr_udf2) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores all UDFs from the database `sr_hub` in the snapshot `sr_udf3_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_udf3_backup FROM test_repo DATABASE sr_hub ON (ALL FUNCTIONS) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores one of all UDFs from the database `sr_hub` in the snapshot `sr_udf3_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_udf3_backup FROM test_repo DATABASE sr_hub ON (FUNCTION sr_udf1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` ##### Restore metadata of external catalog[​](#restore-metadata-of-external-catalog "Direct link to Restore metadata of external catalog") The following example restores the metadata of the external catalog `iceberg` in the snapshot `iceberg_backup` to the target cluster, and rename it as `iceberg_new`. ```sql RESTORE SNAPSHOT iceberg_backup FROM test_repo EXTERNAL CATALOG (iceberg AS iceberg_new) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores the metadata of two external catalogs, `iceberg` and `hive`, in the snapshot `iceberg_hive_backup` to the target cluster. ```sql RESTORE SNAPSHOT iceberg_hive_backup FROM test_repo EXTERNAL CATALOGS (iceberg, hive) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores the metadata of all external catalogs in the snapshot `all_catalog_backup` to the target cluster. ```sql RESTORE SNAPSHOT all_catalog_backup FROM test_repo ALL EXTERNAL CATALOGS PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` To cancel the RESTORE operation on external catalogs, execute the following statement: ```sql CANCEL RESTORE FOR EXTERNAL CATALOG; ``` #### Configure BACKUP or RESTORE jobs[​](#configure-backup-or-restore-jobs "Direct link to Configure BACKUP or RESTORE jobs") You can optimize the performance of BACKUP or RESTORE jobs by modifying the following configuration items in the BE configuration file **be.conf**: | Configuration item | Description | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | make\_snapshot\_worker\_count | The maximum number of threads for the make snapshot tasks of BACKUP jobs on a BE node. Default: `5`. Increase the value of this configuration item to increase the concurrency of the make snapshot task. | | release\_snapshot\_worker\_count | The maximum number of threads for the release snapshot tasks of failed BACKUP jobs on a BE node. Default: `5`. Increase the value of this configuration item to increase the concurrency of the release snapshot task. | | upload\_worker\_count | The maximum number of threads for the upload tasks of BACKUP jobs on a BE node. Default: `0`. `0` indicates setting the value to the number of CPU cores on the machine where the BE resides. Increase the value of this configuration item to increase the concurrency of the upload task. | | download\_worker\_count | The maximum number of threads for the download tasks of RESTORE jobs on a BE node. Default: `0`. `0` indicates setting the value to the number of CPU cores on the machine where the BE resides. Increase the value of this configuration item to increase the concurrency of the download task. | #### Usage notes[​](#usage-notes "Direct link to Usage notes") * Performing backup and restore operations on global, database, table, and partition levels requires different privileges. For detailed information, see [Customize roles based on scenarios](https://docs.starrocks.io/docs/administration/user_privs/authorization/User_privilege.md#customize-roles-based-on-scenarios). * In each database, only one running BACKUP or RESTORE job is allowed each time. Otherwise, StarRocks returns an error. * Because BACKUP and RESTORE jobs occupy many resources of your StarRocks cluster, you can back up and restore your data while your StarRocks cluster is not heavily loaded. * StarRocks does not support specifying data compression algorithms for data backup. * Because data is backed up as snapshots, the data loaded upon snapshot generation is not included in the snapshot. Therefore, if you load data into the old cluster after the snapshot is generated and before the RESTORE job is completed, you also need to load the data into the cluster that data is restored into. It is recommended that you load data into both clusters in parallel for a period of time after the data migration is complete, and then migrate your application to the new cluster after verifying the correctness of the data and services. * Before the RESTORE job is completed, you cannot operate the table to be restored. * Primary Key tables cannot be restored to a StarRocks cluster earlier than v2.5. * You do not need to create the table to be restored in the new cluster before restoring it. The RESTORE job automatically creates it. * If there is an existing table that has a duplicated name with the table to be restored, StarRocks first checks whether or not the schema of the existing table matches that of the table to be restored. If the schemas match, StarRocks overwrites the existing table with the data in the snapshot. If the schema does not match, the RESTORE job fails. You can either rename the table to be restored using the keyword `AS`, or delete the existing table before restoring data. * If the RESTORE job overwrites an existing database, table, or partition, the overwritten data cannot be restored after the job enters the COMMIT phase. If the RESTORE job fails or is canceled at this point, the data may be corrupted and inaccessible. In this case, you can only perform the RESTORE operation again and wait for the job to complete. Therefore, we recommend that you do not restore data by overwriting unless you are sure that the current data is no longer used. The overwrite operation first checks metadata consistency between the snapshot and the existing database, table, or partition. If an inconsistency is detected, the RESTORE operation cannot be performed. * Currently, StarRocks does not support backing up and restoring the configuration data related to user accounts, privileges, and resource groups. * Currently, StarRocks does not support backing up and restoring the Colocate Join relationship among tables. --- ### Manage BE and CN Blacklist From v3.3.0 onwards, StarRocks supports the BE Blacklist feature, which allows you to forbid the usage of certain BE nodes in query execution, thereby avoiding frequent query failures or other unexpected behaviors caused by the failed connections to the BE nodes. A network issue preventing connections to one or more BEs would be an example of when to use the blacklist. From v4.0 onwards, StarRocks supports adding Compute Nodes (CNs) to the Blacklist. By default, StarRocks can automatically manage the BE and CN Blacklist, adding the BE or CN nodes that have lost connection to the blacklist and removing them from the blacklist when the connection is reestablished. However, StarRocks will not remove the node from the Blacklist if it is manually blacklisted. note * Only users with the SYSTEM-level BLACKLIST privilege can use this feature. * Each FE node keeps its own BE and CN Blacklist, and will not share it with other FE nodes. #### Add a BE/CN to the blacklist[​](#add-a-becn-to-the-blacklist "Direct link to Add a BE/CN to the blacklist") You can manually add a BE/CN node to the Blacklist using [ADD BACKEND/COMPUTE NODE BLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ADD_BACKEND_BLACKLIST.md). In this statement, you must specify the ID of the BE/CN node to be blacklisted. You can obtain the BE ID by executing [SHOW BACKENDS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md) and CN ID by executing [SHOW COMPUTE NODES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md). Example: ```sql -- Obtain BE ID. SHOW BACKENDS\G *************************** 1. row *************************** BackendId: 10001 IP: xxx.xx.xx.xxx ... -- Add BE to the blacklist. ADD BACKEND BLACKLIST 10001; -- Obtain CN ID. SHOW COMPUTE NODES\G *************************** 1. row *************************** ComputeNodeId: 10005 IP: xxx.xx.xx.xxx ... -- Add CN to the blacklist. ADD COMPUTE NODE BLACKLIST 10005; ``` #### Remove a BE/CN from blacklist[​](#remove-a-becn-from-blacklist "Direct link to Remove a BE/CN from blacklist") You can manually remove a BE/CN node from the Blacklist using [DELETE BACKEND/COMPUTE NODE BLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/DELETE_BACKEND_BLACKLIST.md). In this statement, you must also specify the ID of the BE/CN node. Example: ```sql -- Remove a BE from the Blacklist. DELETE BACKEND BLACKLIST 10001; -- Remove a CN from the Blacklist. DELETE COMPUTE NODE BLACKLIST 10005; ``` #### View BE/CN Blacklist[​](#view-becn-blacklist "Direct link to View BE/CN Blacklist") You can view the BE/CN nodes in the Blacklist using [SHOW BACKEND/COMPUTE NODE BLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKEND_BLACKLIST.md). Example: ```sql -- View the BE Blacklist. SHOW BACKEND BLACKLIST; +-----------+------------------+---------------------+------------------------------+--------------------+ | BackendId | AddBlackListType | LostConnectionTime | LostConnectionNumberInPeriod | CheckTimePeriod(s) | +-----------+------------------+---------------------+------------------------------+--------------------+ | 10001 | MANUAL | 2024-04-28 11:52:09 | 0 | 5 | +-----------+------------------+---------------------+------------------------------+--------------------+ -- View the CN Blacklist. SHOW COMPUTE NODE BLACKLIST; +---------------+------------------+---------------------+------------------------------+--------------------+ | ComputeNodeId | AddBlackListType | LostConnectionTime | LostConnectionNumberInPeriod | CheckTimePeriod(s) | +---------------+------------------+---------------------+------------------------------+--------------------+ | 10005 | MANUAL | 2025-08-18 10:47:51 | 0 | 5 | +---------------+------------------+---------------------+------------------------------+--------------------+ ``` The following fields are returned: * `AddBlackListType`: How the BE/CN node was added to the blacklist. `MANUAL` indicates it is manually blacklisted by the user. `AUTO` indicates it is automatically blacklisted by StarRocks. * `LostConnectionTime`: * For the `MANUAL` type, it indicates the time when the BE/CN node was manually added to the blacklist. * For the `AUTO` type, it indicates the time when the last successful connection was established. * `LostConnectionNumberInPeriod`: The number of disconnections detected within `CheckTimePeriod(s)`, which is the interval at which StarRocks checks the connection status of the BE/CN nodes in the blacklist. * `CheckTimePeriod(s)`: The interval at which StarRocks checks the connection status of the blacklisted BE/CN nodes. Its value is evaluated to the value you specified for the FE configuration item `black_host_history_sec`. Unit: Seconds. #### Configure automatic management of BE/CN Blacklist[​](#configure-automatic-management-of-becn-blacklist "Direct link to Configure automatic management of BE/CN Blacklist") Each time a BE/CN node loses connection to the FE node, or a query fails due to timeout on a BE/CN node, the FE node adds the BE/CN node to its BE and CN Blacklist. The FE node will constantly assess the connectivity of the BE/CN node in the blacklist by counting its connection failures within a certain duration of time. StarRocks will remove a blacklisted BE/CN node only if the number of its connection failures is below a pre-specified threshold. You can configure the automatic management of the BE and CN Blacklist using the following [FE configurations](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md): * `black_host_history_sec`: The time duration for retaining historical connection failures of BE/CN nodes in the Blacklist. * `black_host_connect_failures_within_time`: The threshold of connection failures allowed for a blacklisted BE/CN node. If a BE/CN node is added to the Blacklist automatically, StarRocks will assess its connectivity and judge whether it can be removed from the Blacklist. Within `black_host_history_sec`, only if a blacklisted BE/CN node has fewer connection failures than the threshold set in `black_host_connect_failures_within_time`, it can be removed from the Blacklist. --- ### Cluster Snapshot Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) This topic describes how to use Cluster Snapshot for disaster recovery on shared-data clusters. This feature is supported from v3.4.2 onwards and only available on shared-data clusters. #### Overview[​](#overview "Direct link to Overview") The fundamental idea of disaster recovery for shared-data clusters is to ensure that the full cluster state (including data and metadata) is stored in object storage. This way, if the cluster encounters a failure, it can be restored from the object storage as long as the data and metadata remain intact. Additionally, features like backups and cross-region replication offered by cloud providers can be used to achieve remote recovery and cross-region disaster recovery. In shared-data clusters, the CN state (data) is stored in object storage, but the FE state (metadata) remains local. To ensure that object storage has all the cluster state for restoration, StarRocks now supports Cluster Snapshot for both data and metadata in object storage. ##### Workflow[​](#workflow "Direct link to Workflow") ![Workflow](/assets/images/cluster_snapshot_workflow-6a61b5e029595a53a7072a54bd5c6e4f.png) ##### Terms[​](#terms "Direct link to Terms") * **Cluster snapshot** A cluster snapshot refers to a snapshot of the cluster state at a certain moment. It contains all the objects in the cluster, such as catalogs, databases, tables, users & privileges, loading tasks, and more. It does not include all external dependent objects, such as configuration files of external catalogs, and local UDF JAR packages. * **Generating cluster snapshot** The system automatically maintains a snapshot closely following the latest cluster state. Historical snapshots will be dropped right after the latest one is created, keeping only one snapshot available all the time. * **Cluster Restore** Restore the cluster from a snapshot. #### Automated cluster snapshot[​](#automated-cluster-snapshot "Direct link to Automated cluster snapshot") Automated Cluster Snapshot is disabled by default. Use the following statement to enable this feature: Syntax: ```sql ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON [STORAGE VOLUME ] ``` Parameter: `storage_volume_name`: Specifies the storage volume used to store the snapshot. If this parameter is not specified, the default storage volume will be used. For details on creating a storage volume, see [CREATE STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md). Each time FE creates a new metadata image after completing a metadata checkpoint, it automatically creates a snapshot. The name of the snapshot is generated by the system, following the format `automated_cluster_snapshot_{timestamp}`. Metadata snapshots are stored under `/{storage_volume_locations}/{service_id}/meta/image/automated_cluster_snapshot_timestamp`. Data snapshots are stored in the same location as the original data. FE configuration item `automated_cluster_snapshot_interval_seconds` controls the snapshot automation cycle. The default value is 600 seconds (10 minutes). ##### Disable automated cluster snapshot[​](#disable-automated-cluster-snapshot "Direct link to Disable automated cluster snapshot") Use the following statement to disable automated cluster snapshot: ```sql ADMIN SET AUTOMATED CLUSTER SNAPSHOT OFF ``` Once Automated Cluster Snapshot is disabled, the system will automatically purge the historical snapshot. #### View cluster snapshot[​](#view-cluster-snapshot "Direct link to View cluster snapshot") You can query the view `information_schema.cluster_snapshots` to view the latest cluster snapshot and the snapshots yet to be dropped. ```sql SELECT * FROM information_schema.cluster_snapshots; ``` Return: | Field | Description | | -------------------- | ----------------------------------------------------------------- | | snapshot\_name | The name of the snapshot. | | snapshot\_type | The type of the snapshot. Valid values: `automated` and `manual`. | | created\_time | The time at which the snapshot was created. | | fe\_journal\_id | The ID of the FE journal. | | starmgr\_journal\_id | The ID of the StarManager journal. | | properties | Applies to a feature not yet available. | | storage\_volume | The storage volume where the snapshot is stored. | | storage\_path | The storage path under which the snapshot is stored. | #### View cluster snapshot job[​](#view-cluster-snapshot-job "Direct link to View cluster snapshot job") You can query the view `information_schema.cluster_snapshot_jobs` to view the job information of cluster snapshots. ```sql SELECT * FROM information_schema.cluster_snapshot_jobs; ``` Return: | Field | Description | | -------------- | ----------------------------------------------------------------------------------------------------------------- | | snapshot\_name | The name of the snapshot. | | job\_id | The ID of the job. | | created\_time | The time at which the job was created. | | finished\_time | The time at which the job was finished. | | state | The state of the job. Valid values: `INITIALIZING`, `SNAPSHOTING`, `FINISHED`, `EXPIRED`, `DELETED`, and `ERROR`. | | detail\_info | The specific progress information of the current execution stage. | | error\_message | The error message (if any) of the job. | #### Restore the cluster[​](#restore-the-cluster "Direct link to Restore the cluster") Follow these steps to restore the cluster with the cluster snapshot. 1. **(Optional)** If you want to perform a cross-cluster recovery within the region, you MUST modify the configuration file **cluster\_snapshot.yaml** under the directory `fe/conf` of the Leader FE node. Otherwise, you can skip this step. To restore the data in a new cluster, all files under the original storage path must be copied to the new path. You must use the template provided in [Appendix](#appendix). 2) Start the Leader FE node. ```bash ./fe/bin/start_fe.sh --cluster_snapshot --daemon ``` 3) Start other FE nodes **after cleaning the `meta` directories**. ```bash ./fe/bin/start_fe.sh --helper : --daemon ``` Add the nodes to the cluster. ```sql -- Add the Follower node: ALTER SYSTEM ADD FOLLOWER ":"; -- Add the Observer node: ALTER SYSTEM ADD OBSERVER ":"; ``` 4) Start CN nodes **after cleaning the `storage_root_path` directories**. ```bash ./be/bin/start_cn.sh --daemon ``` Add the node to the cluster. ```sql ALTER SYSTEM ADD COMPUTE NODE ":"; ``` If you have modified **cluster\_snapshot.yaml** in the step 1, the node and storage volumes will be re-configured in the new cluster according to the information in the file. #### Appendix[​](#appendix "Direct link to Appendix") Template of **cluster\_snapshot.yaml** for cross-cluster recovery: ```yaml # Information of the cluster snapshot to be downloaded for restoration. cluster_snapshot: # The URI of the snapshot. # Example 1: s3://defaultbucket/test/f7265e80-631c-44d3-a8ac-cf7cdc7adec811019/meta/image/automated_cluster_snapshot_1704038400000 # Example 2: s3://defaultbucket/test/f7265e80-631c-44d3-a8ac-cf7cdc7adec811019/meta cluster_snapshot_path: # The name of the storage volume to store the snapshot. You must define it in the `storage_volumes` section. # NOTE: It must be identical with that in the original cluster. storage_volume_name: my_s3_volume # [Optional] Node information of the new cluster where the snapshot is to be restored. # If this section is not specified, the new cluster after recovery only has the Leader FE node. # CN nodes retain the information of the original cluster. # NOTE: DO NOT include the Leader FE node in this section. frontends: # FE host. - host: xxx.xx.xx.x1 # FE edit_log_port. edit_log_port: 9010 # The FE node type. Valid values: `follower` (Default) and `observer`. type: follower - host: xxx.xx.xx.x2 edit_log_port: 9010 type: observer compute_nodes: # CN host. - host: xxx.xx.xx.x3 # CN heartbeat_service_port. heartbeat_service_port: 9050 - host: xxx.xx.xx.x4 heartbeat_service_port: 9050 # Information of the storage volume in the new cluster. It is used for restoring a cloned snapshot. # NOTE: The name of the storage volume must be identical with that in the original cluster. storage_volumes: # Example for S3-compatible storage volume. - name: my_s3_volume type: S3 location: s3://defaultbucket/test/ comment: my s3 volume properties: - key: aws.s3.region value: us-west-2 - key: aws.s3.endpoint value: https://s3.us-west-2.amazonaws.com - key: aws.s3.access_key value: xxxxxxxxxx - key: aws.s3.secret_key value: yyyyyyyyyy # Example for HDFS storage volume. - name: my_hdfs_volume type: HDFS location: hdfs://127.0.0.1:9000/sr/test/ comment: my hdfs volume properties: - key: hadoop.security.authentication value: simple - key: username value: starrocks ``` note For more information on credentials for AWS, see [Authenticate to AWS S3](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md). #### Limitations[​](#limitations "Direct link to Limitations") * Currently, standby mode is not supported. The primary and secondary clusters cannot be online simultaneously. Otherwise, the normal operation of the secondary cluster cannot be guaranteed. * Currently, only one automated cluster snapshot can be retained. --- ### Compaction for Shared-data Clusters This topic describes how to manage compaction in StarRocks shared-data clusters. #### Overview[​](#overview "Direct link to Overview") Each data loading operation in StarRocks generates a new version of data files. Compaction merges data files from different versions into larger files, reducing the number of small files and improving query efficiency. #### Compaction Score[​](#compaction-score "Direct link to Compaction Score") ##### Overview[​](#overview-1 "Direct link to Overview") The *Compaction Score* reflects the merging status of data files in a partition. A higher score indicates lower merging progress, meaning the partition has more unmerged data file versions. FE maintains Compaction Score information for each partition, including the Max Compaction Score (the highest score among all tablets in the partition). If a partition's Max Compaction Score is below the FE parameter `lake_compaction_score_selector_min_score` (default: 10), compaction for that partition is considered complete. A Max Compaction Score exceeding 100 indicates an unhealthy compaction state. When the score exceeds the FE parameter `lake_ingest_slowdown_threshold` (default: 100), the system slows down data loading transaction commits for the partition. If it surpasses `lake_compaction_score_upper_bound` (default: 2000), the system rejects import transactions for the partition. ##### Calculation Rules[​](#calculation-rules "Direct link to Calculation Rules") Typically, each data file contributes 1 to the Compaction Score. For example, if a partition has one tablet and 10 data files generated from the first loading operation, the partition’s Max Compaction Score is 10. All data files generated by a transaction within a tablet are grouped as a Rowset. During score calculation, a tablet’s Rowsets are grouped by size, and the group with the highest number of files determines the tablet’s Compaction Score. For example, a tablet undergoes 7 loading operations, generating Rowsets with sizes: 100 MB, 100 MB, 100 MB, 10 MB, 10 MB, 10 MB, and 10 MB. During calculation, the system will make three 100 MB Rowsets into one group and four 10 MB Rowsets into another. The Compaction Score is calculated based on the group with more files. In this case, the second group has bigger compaction score. The compaction prioritizes the higher-scoring group, so after the first compaction, the Rowset diatribution would be: 100 MB, 100 MB, 100 MB, and 40 MB. #### Compaction Workflow[​](#compaction-workflow "Direct link to Compaction Workflow") For shared-data clusters, StarRocks introduces a new FE-controlled compaction mechanism: 1. Score Calculation: The Leader FE node calculates and stores Compaction Scores for partitions based on transaction publish results. 2. Candidate Selection: FE selects partitions with the highest Max Compaction Scores as compaction candidates. 3. Task Generation: FE initiates compaction transactions for selected partitions, generates tablet-level subtasks, and dispatches them to Compute Nodes (CNs) until reaching the limit set by the FE parameter `lake_compaction_max_tasks`. 4. Subtask Execution: CNs execute compaction subtasks in the background. The number of concurrent subtasks per CN is controlled by the CN parameter `compact_threads`. 5. Result Collection: FE aggregates subtask results and commits the compaction transaction. 6. Publish: FE publishes the successfully committed compaction transaction. #### Manage compaction[​](#manage-compaction "Direct link to Manage compaction") ##### View compaction scores[​](#view-compaction-scores "Direct link to View compaction scores") * You can view the compaction scores of partitions in a specific table by using the SHOW PROC statement. Typically, you only need to focus on the `MaxCS` field. If `MaxCS` is below 10, compaction is considered complete. If `MaxCS` is above 100, the Compaction Score is relatively high. If `MaxCS` exceeds 500, the Compaction Score is very high and manual intervention may be required. ```plain SHOW PARTITIONS FROM SHOW PROC '/dbs///partitions' ``` Example: ```plain mysql> SHOW PROC '/dbs/load_benchmark/store_sales/partitions'; +-------------+---------------+----------------+----------------+-------------+--------+--------------+-------+------------------------------+---------+----------+-----------+----------+------------+-------+-------+-------+ | PartitionId | PartitionName | CompactVersion | VisibleVersion | NextVersion | State | PartitionKey | Range | DistributionKey | Buckets | DataSize | RowCount | CacheTTL | AsyncWrite | AvgCS | P50CS | MaxCS | +-------------+---------------+----------------+----------------+-------------+--------+--------------+-------+------------------------------+---------+----------+-----------+----------+------------+-------+-------+-------+ | 38028 | store_sales | 913 | 921 | 923 | NORMAL | | | ss_item_sk, ss_ticket_number | 64 | 15.6GB | 273857126 | 2592000 | false | 10.00 | 10.00 | 10.00 | +-------------+---------------+----------------+----------------+-------------+--------+--------------+-------+------------------------------+---------+----------+-----------+----------+------------+-------+-------+-------+ 1 row in set (0.20 sec) ``` * You can also view the partition compaction scores by querying the system-defined view `information_schema.partitions_meta`. Example: ```plain mysql> SELECT * FROM information_schema.partitions_meta ORDER BY Max_CS LIMIT 10; +--------------+----------------------------+----------------------------+--------------+-----------------+-----------------+----------------------+--------------+---------------+-----------------+-----------------------------------------+---------+-----------------+----------------+---------------------+-----------------------------+--------------+---------+-----------+------------+------------------+----------+--------+--------+-------------------------------------------------------------------+ | DB_NAME | TABLE_NAME | PARTITION_NAME | PARTITION_ID | COMPACT_VERSION | VISIBLE_VERSION | VISIBLE_VERSION_TIME | NEXT_VERSION | PARTITION_KEY | PARTITION_VALUE | DISTRIBUTION_KEY | BUCKETS | REPLICATION_NUM | STORAGE_MEDIUM | COOLDOWN_TIME | LAST_CONSISTENCY_CHECK_TIME | IS_IN_MEMORY | IS_TEMP | DATA_SIZE | ROW_COUNT | ENABLE_DATACACHE | AVG_CS | P50_CS | MAX_CS | STORAGE_PATH | +--------------+----------------------------+----------------------------+--------------+-----------------+-----------------+----------------------+--------------+---------------+-----------------+-----------------------------------------+---------+-----------------+----------------+---------------------+-----------------------------+--------------+---------+-----------+------------+------------------+----------+--------+--------+-------------------------------------------------------------------+ | tpcds_1t | call_center | call_center | 11905 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | cc_call_center_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 12.3KB | 42 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/11906/11905 | | tpcds_1t | web_returns | web_returns | 12030 | 3 | 3 | 2024-03-17 08:40:48 | 4 | | | wr_item_sk, wr_order_number | 16 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 3.5GB | 71997522 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/12031/12030 | | tpcds_1t | warehouse | warehouse | 11847 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | w_warehouse_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 4.2KB | 20 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/11848/11847 | | tpcds_1t | ship_mode | ship_mode | 11851 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | sm_ship_mode_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 1.7KB | 20 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/11852/11851 | | tpcds_1t | customer_address | customer_address | 11790 | 0 | 2 | 2024-03-17 08:32:19 | 3 | | | ca_address_sk | 16 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 120.9MB | 6000000 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/11791/11790 | | tpcds_1t | time_dim | time_dim | 11855 | 0 | 2 | 2024-03-17 08:30:48 | 3 | | | t_time_sk | 16 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 864.7KB | 86400 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/11856/11855 | | tpcds_1t | web_sales | web_sales | 12049 | 3 | 3 | 2024-03-17 10:14:20 | 4 | | | ws_item_sk, ws_order_number | 128 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 47.7GB | 720000376 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/12050/12049 | | tpcds_1t | store | store | 11901 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | s_store_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 95.6KB | 1002 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/11902/11901 | | tpcds_1t | web_site | web_site | 11928 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | web_site_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 13.4KB | 54 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/11929/11928 | | tpcds_1t | household_demographics | household_demographics | 11932 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | hd_demo_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 2.1KB | 7200 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/11933/11932 | +--------------+----------------------------+----------------------------+--------------+-----------------+-----------------+----------------------+--------------+---------------+-----------------+-----------------------------------------+---------+-----------------+----------------+---------------------+-----------------------------+--------------+---------+-----------+------------+------------------+----------+--------+--------+-------------------------------------------------------------------+ ``` ##### View compaction tasks[​](#view-compaction-tasks "Direct link to View compaction tasks") As new data is loading to the system, FE constantly schedules compaction tasks to be executed on different CN nodes. You can first view the general status of compaction tasks on FE, and then view the execution details of each tasks on CN. ###### View general status of compaction tasks[​](#view-general-status-of-compaction-tasks "Direct link to View general status of compaction tasks") You can view the general status of compaction tasks using the SHOW PROC statement. ```sql SHOW PROC '/compactions'; ``` Example: ```plain mysql> SHOW PROC '/compactions'; +---------------------+-------+---------------------+---------------------+---------------------+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Partition | TxnID | StartTime | CommitTime | FinishTime | Error | Profile | +---------------------+-------+---------------------+---------------------+---------------------+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ssb.lineorder.10081 | 15 | 2026-01-10 03:29:07 | 2026-01-10 03:29:11 | 2026-01-10 03:29:12 | NULL | {"sub_task_count":12,"read_local_sec":0,"read_local_mb":218,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":120,"write_segment_count":12,"write_segment_mb":219,"write_remote_sec":4,"in_queue_sec":18,"score_before":{"avg":10.0,"p50":10.0,"max":10.0},"score_after":{"avg":8.0,"p50":8.0,"max":8.0},"partial_success":false} | | ssb.lineorder.10068 | 16 | 2026-01-10 03:29:07 | 2026-01-10 03:29:13 | 2026-01-10 03:29:14 | NULL | {"sub_task_count":12,"read_local_sec":0,"read_local_mb":218,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":120,"write_segment_count":12,"write_segment_mb":218,"write_remote_sec":4,"in_queue_sec":38,"score_before":{"avg":10.0,"p50":10.0,"max":10.0},"score_after":{"avg":8.0,"p50":8.0,"max":8.0},"partial_success":false} | | ssb.lineorder.10055 | 20 | 2026-01-10 03:29:11 | 2026-01-10 03:29:15 | 2026-01-10 03:29:17 | NULL | {"sub_task_count":12,"read_local_sec":0,"read_local_mb":218,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":120,"write_segment_count":12,"write_segment_mb":218,"write_remote_sec":4,"in_queue_sec":23,"score_before":{"avg":10.0,"p50":10.0,"max":10.0},"score_after":{"avg":8.0,"p50":8.0,"max":8.0},"partial_success":false} | +---------------------+-------+---------------------+---------------------+---------------------+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` The following fields are returned: * `Partition`: The partition to which the compaction task belongs. * `TxnID`: The transaction ID assigned to the compaction task. * `StartTime`: The time when the compaction task starts. `NULL` indicates that the task has not yet been initiated. * `CommitTime`: The time when the compaction task commits the data. `NULL` indicates that the data has not yet been committed. * `FinishTime`: The time when the compaction task publishes the data. `NULL` indicates that the data has not yet been published. * `Error`: The error message (if any) of the compaction task. * `Profile`: (supported from v3.2.12 and v3.3.4) The Profile of the compaction task after finished. * `sub_task_count`: The number of sub-tasks (equivalent to tablets) in the partition. * `read_local_sec`: The total time consumption of all sub-tasks on reading data from the local cache. Unit: Seconds. * `read_local_mb`: The total size of data read from the local cache by all sub-tasks. Unit: MB. * `read_remote_sec`: The total time consumption of all sub-tasks on reading data from the remote storage. Unit: Seconds. * `read_remote_mb`: The total size of data read from the remote storage by all sub-tasks. Unit: MB. * `read_segment_count`: The total number of files read by all sub-tasks. * `write_segment_count`: The total number of new files generated by all sub-tasks. * `write_segment_mb`: The total size of new files generated by all sub-tasks. Unit: MB. * `write_remote_sec`: The total time consumption of all sub-tasks on writing data to the remote storage. Unit: Seconds. * `in_queue_sec`: The total time of all sub-tasks staying in the queue. Unit: Seconds. * `score_before`: The compaction score of the partition before the compaction. Includes `avg`, `p50`, and `max` fields. * `score_after`: The compaction score of the partition after the compaction. Includes `avg`, `p50`, and `max` fields. * `partial_success`: Whether the compaction job is partially successful (some tablets failed). ###### View execution details of compaction tasks[​](#view-execution-details-of-compaction-tasks "Direct link to View execution details of compaction tasks") Each compaction task is divided into multiple sub-tasks, each of which corresponds to a tablet. You can view the execution details of each sub-task by querying the system-defined view `information_schema.be_cloud_native_compactions`. Example: ```plain mysql> SELECT * FROM information_schema.be_cloud_native_compactions; +-------+--------+-----------+---------+---------+------+---------------------+-------------+----------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | BE_ID | TXN_ID | TABLET_ID | VERSION | SKIPPED | RUNS | START_TIME | FINISH_TIME | PROGRESS | STATUS | PROFILE | +-------+--------+-----------+---------+---------+------+---------------------+-------------+----------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 10001 | 51047 | 43034 | 12 | 0 | 1 | 2024-09-24 19:15:15 | NULL | 82 | | {"read_local_sec":0,"read_local_mb":31,"read_remote_sec":0,"read_remote_mb":0,"read_remote_count":0,"read_local_count":1900,"segment_init_sec":0,"column_iterator_init_sec":0,"in_queue_sec":0} | | 10001 | 51048 | 43032 | 12 | 0 | 1 | 2024-09-24 19:15:15 | NULL | 82 | | {"read_local_sec":0,"read_local_mb":32,"read_remote_sec":0,"read_remote_mb":0,"read_remote_count":0,"read_local_count":1900,"segment_init_sec":0,"column_iterator_init_sec":0,"in_queue_sec":0} | | 10001 | 51049 | 43033 | 12 | 0 | 1 | 2024-09-24 19:15:15 | NULL | 82 | | {"read_local_sec":0,"read_local_mb":31,"read_remote_sec":0,"read_remote_mb":0,"read_remote_count":0,"read_local_count":1900,"segment_init_sec":0,"column_iterator_init_sec":0,"in_queue_sec":0} | | 10001 | 51051 | 43038 | 9 | 0 | 1 | 2024-09-24 19:15:15 | NULL | 84 | | {"read_local_sec":0,"read_local_mb":31,"read_remote_sec":0,"read_remote_mb":0,"read_remote_count":0,"read_local_count":1900,"segment_init_sec":0,"column_iterator_init_sec":0,"in_queue_sec":0} | | 10001 | 51052 | 43036 | 12 | 0 | 0 | NULL | NULL | 0 | | | | 10001 | 51053 | 43035 | 12 | 0 | 1 | 2024-09-24 19:15:16 | NULL | 2 | | {"read_local_sec":0,"read_local_mb":1,"read_remote_sec":0,"read_remote_mb":0,"read_remote_count":0,"read_local_count":100,"segment_init_sec":0,"column_iterator_init_sec":0,"in_queue_sec":0} | +-------+--------+-----------+---------+---------+------+---------------------+-------------+----------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` The following fields are returned: * `BE_ID`: The ID of the CN. * `TXN_ID`: The ID of transaction to which the sub-task belongs. * `TABLET_ID`: The ID of tablet to which the sub-task belongs. * `VERSION`: The version of the tablet. * `RUNS`: The number of times the sub-task has been executed. * `START_TIME`: The time when the sub-task starts. * `FINISH_TIME`: The time when the sub-task finishes. * `PROGRESS`: The compaction progress of the tablet in percentage. * `STATUS`: The status of the sub-task. Error messages will be returned in this field if there is an error. * `PROFILE`: (supported from v3.2.12 and v3.3.4) The runtime profile of the sub-task. * `read_local_sec`: The time consumption of the sub-task on reading data from the local cache. Unit: Seconds. * `read_local_mb`: The size of data read from the local cache by the sub-task. Unit: MB. * `read_remote_sec`: The time consumption of the sub-task on reading data from the remote storage. Unit: Seconds. * `read_remote_mb`: The size of data read from the remote storage by the sub-task. Unit: MB. * `read_local_count`: The number of times the sub-task reads data from the local cache. * `read_remote_count`: The number of times the sub-task reads data from the remote storage. * `in_queue_sec`: The time of the sub-task staying in queue. Unit: Seconds. ##### Configure compaction tasks[​](#configure-compaction-tasks "Direct link to Configure compaction tasks") You can configure compaction tasks using these FE and CN (BE) parameters. ###### FE parameter[​](#fe-parameter "Direct link to FE parameter") You can configure the following FE parameter dynamically. ```sql ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1"); ``` ###### lake\_compaction\_max\_tasks[​](#lake_compaction_max_tasks "Direct link to lake_compaction_max_tasks") * Default: -1 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of concurrent Compaction tasks allowed in a shared-data cluster. Setting this item to `-1` indicates calculating the concurrent task number in an adaptive manner, that is, the number of surviving CN nodes multiplied by 16. Setting this value to `0` will disable compaction. * Introduced in: v3.1.0 ```sql ADMIN SET FRONTEND CONFIG ("lake_compaction_disable_tables" = "11111;22222"); ``` ###### lake\_compaction\_disable\_tables[​](#lake_compaction_disable_tables "Direct link to lake_compaction_disable_tables") * Default："" * Type：String * Unit：- * Is mutable：Yes * Description：Disable compaction for certain tables. This will not affect compaction that has started. The value of this item is table ID. Multiple values are separated by ';'. * Introduced in：v3.2.7 ###### CN parameters[​](#cn-parameters "Direct link to CN parameters") You can configure the following CN parameter dynamically. ```sql UPDATE information_schema.be_configs SET VALUE = 8 WHERE name = "compact_threads"; ``` ###### compact\_threads[​](#compact_threads "Direct link to compact_threads") * Default: 4 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads used for concurrent compaction tasks. This configuration is changed to dynamic from v3.1.7 and v3.2.2 onwards. * Introduced in: v3.0.0 > **NOTE** > > In production, it is recommended to set `compact_threads` to 25% of the BE/CN CPU core count. ###### max\_cumulative\_compaction\_num\_singleton\_deltas[​](#max_cumulative_compaction_num_singleton_deltas "Direct link to max_cumulative_compaction_num_singleton_deltas") * Default: 500 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of segments that can be merged in a single Cumulative Compaction. You can reduce this value if OOM occurs during compaction. * Introduced in: - > **NOTE** > > In production, it is recommended to set `max_cumulative_compaction_num_singleton_deltas` to `100` to accelerate the compaction tasks and reduce their recource consumption. ###### lake\_pk\_compaction\_max\_input\_rowsets[​](#lake_pk_compaction_max_input_rowsets "Direct link to lake_pk_compaction_max_input_rowsets") * Default: 500 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of input rowsets allowed in a Primary Key table compaction task in a shared-data cluster. The default value of this parameter is changed from `5` to `1000` since v3.2.4 and v3.1.10, and to `500` since since v3.3.1 and v3.2.9. After the Sized-tiered Compaction policy is enabled for Primary Key tables (by setting `enable_pk_size_tiered_compaction_strategy` to `true`), StarRocks does not need to limit the number of rowsets for each compaction to reduce write amplification. Therefore, the default value of this parameter is increased. * Introduced in: v3.1.8, v3.2.3 > **NOTE** > > In production, it is recommended to set `max_cumulative_compaction_num_singleton_deltas` to `100` to accelerate the Compaction tasks and reduce their resource consumption. ##### Manually trigger compaction tasks[​](#manually-trigger-compaction-tasks "Direct link to Manually trigger compaction tasks") ```sql -- Trigger compaction for the whole table. ALTER TABLE COMPACT; -- Trigger compaction for a specific partition. ALTER TABLE COMPACT ; -- Trigger compaction for multiple partitions. ALTER TABLE COMPACT (, , ...); ``` ##### Cancel compaction tasks[​](#cancel-compaction-tasks "Direct link to Cancel compaction tasks") You can manually cancel a compaction task using the transaction ID of the task. ```sql CANCEL COMPACTION WHERE TXN_ID = ; ``` > **NOTE** > > * The CANCEL COMPACTION statement must be submitted from the Leader FE node. > * The CANCEL COMPACTION statement only applies to transactions that have not committed, that is, `CommitTime` is NULL in the return of `SHOW PROC '/compactions'`. > * CANCEL COMPACTION is an asynchronous process. You can check if the task is cancelled by executing `SHOW PROC '/compactions'`. #### Best practices[​](#best-practices "Direct link to Best practices") Since Compaction is crucial for query performance, it is recommended to regularly monitor the data merging status of tables and partitions. Here are some best practices and guidelines: * Try to increase the time interval between loading (avoid scenarios with intervals less than 10 seconds) and increase the batch size per load (avoid batch sizes smaller than 100 rows of data). * Adjust the number of parallel compaction worker threads on CN to accelerate task execution. It is recommended to set `compact_threads` to 25% of the BE/CN CPU core count in a production environment. * Monitor the Compaction task status using `show proc '/compactions'` and `select * from information_schema.be_cloud_native_compactions;`. * Monitor the Compaction Score, and configure alerts based on it. StarRocks' built-in Grafana monitoring template includes this metric. * Pay attention to the resource consumption during compaction, especially memory usage. The Grafana monitoring template also includes this metric. #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") ##### Slow queries[​](#slow-queries "Direct link to Slow queries") To identify slow queries caused by untimely Compaction, you can check, in the SQL Profile, the value of `PhySegmentsCount` divided by `TabletCount` within a single Fragment. If it is an large value, such as tens or more, untimely Compaction may be the cause of the slow query. Do not use `SegmentsReadCount` for this check: it counts a segment once for every split that reads it, so it also grows when a tablet is scanned with intra-tablet parallelism. ##### High Max Compaction Score in the cluster[​](#high-max-compaction-score-in-the-cluster "Direct link to High Max Compaction Score in the cluster") 1. Check whether the Compaction-related parameters are within reasonable ranges using `ADMIN SHOW FRONTEND CONFIG LIKE "%lake_compaction%"` and `SELECT * FROM information_schema.be_configs WHERE name = "compact_threads"`. 2. Check if Compaction is stuck using `SHOW PROC '/compactions'`: * If `CommitTime` remains NULL, check the system view `information_schema.be_cloud_native_compactions` for the reason why Compaction is stuck. * If `FinishTime` remains NULL, search for the Publish failure reason in the Leader FE log using `TxnID`. 3. Check if compaction is running slowly using `SHOW PROC '/compactions'`: * If `sub_task_count` is too large (check the size of each tablet in this partition using `SHOW PARTITIONS`), the table may be created improperly. * If `read_remote_mb` is too large (more than 30% of the total read data), check the server disk size and also check the cache quota through `SHOW BACKENDS` for field `DataCacheMetrics`. * If `write_remote_sec` is too large (more than 90% of the total Compaction time), write to the remote storage may be too slow. This can be verified by checking the shared-data-specific monitoring metrics with keywords `single upload latency` and `multi upload latency`. * If `in_queue_sec` is too large (average waiting time per tablet exceeds 60 seconds), the parameter settings may be unreasonable or other running Compactions are too slow. --- ### Enable FQDN access This topic describes how to enable cluster access by using a fully qualified domain name (FQDN). An FQDN is a **complete domain name** for a specific entity that can be accessed over the Internet. The FQDN consists of two parts: the hostname and the domain name. Before 2.4, StarRocks supports access to FEs and BEs via IP address only. Even if an FQDN is used to add a node to a cluster, it is transformed into an IP address eventually. This causes a huge inconvenience for DBAs because changing the IP addresses of certain nodes in a StarRocks cluster can lead to access failures to the nodes. In version 2.4, StarRocks decouples each node from its IP address. You can now manage nodes in StarRocks solely via their FQDNs. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") To enable FQDN access for a StarRocks cluster, make sure the following requirements are satisfied: * Each machine in the cluster must have a hostname. * In the file **/etc/hosts** on each machine, you must specify the corresponding IP addresses and FQDNs of other machines in the cluster. * IP addresses in the **/etc/hosts** file must be unique. #### Set up a new cluster with FQDN access[​](#set-up-a-new-cluster-with-fqdn-access "Direct link to Set up a new cluster with FQDN access") By default, FE nodes in a new cluster are started via IP address access. To start a new cluster with FQDN access, you must start the FE nodes by running the following commands **when you start the cluster for the first time**: ```shell ./bin/start_fe.sh --host_type FQDN --daemon ``` The property `--host_type` specifies the access method that is used to start the node. Valid values include `FQDN` and `IP`. You only need to specify this property ONCE when you start the node for the first time. Each BE node identifies itself with `BE Address` defined in the FE metadata. Therefore, you DO NOT need to specify `--host_type` when you start BE nodes. If the `BE Address` defines a BE node with an FQDN, the BE node identifies itself with this FQDN. #### Enable FQDN access in an existing cluster[​](#enable-fqdn-access-in-an-existing-cluster "Direct link to Enable FQDN access in an existing cluster") To enable FQDN access in an existing cluster that was previously started via IP addresses, you must first **upgrade** StarRocks to version 2.4.0 or later. ##### Enable FQDN access for FE nodes[​](#enable-fqdn-access-for-fe-nodes "Direct link to Enable FQDN access for FE nodes") You need to enable FQDN access for all the non-Leader Follower FE nodes before enabling that for the Leader FE node. > **CAUTION** > > Make sure that the cluster has at least three Follower FE nodes before you enable FQDN access for FE nodes. ###### Enable FQDN access for non-Leader Follower FE nodes[​](#enable-fqdn-access-for-non-leader-follower-fe-nodes "Direct link to Enable FQDN access for non-Leader Follower FE nodes") 1. Navigate to the deployment directory of the FE node, and run the following command to stop the FE node: ```shell ./bin/stop_fe.sh ``` 2. Execute the following statement via your MySQL client to check the `Alive` status of the FE node that you have stopped. Wait until the `Alive` status becomes `false`. ```sql SHOW PROC '/frontends'\G ``` 3. Execute the following statement to replace the IP address with FQDN. ```sql ALTER SYSTEM MODIFY FRONTEND HOST "" TO ""; ``` 4. Run the following command to start the FE node with FQDN access. ```shell ./bin/start_fe.sh --host_type FQDN --daemon ``` The property `--host_type` specifies the access method that is used to start the node. Valid values include `FQDN` and `IP`. You only need to specify this property ONCE when you restart the node after you modify the node. 5. Check the `Alive` status of the FE node. Wait until the `Alive` status becomes `true`. ```sql SHOW PROC '/frontends'\G ``` 6. Repeat the above steps to enable FQDN access for the other non-Leader Follower FE nodes one after another when the `Alive` status of the current FE node is `true`. ###### Enable FQDN access for the Leader FE node[​](#enable-fqdn-access-for-the-leader-fe-node "Direct link to Enable FQDN access for the Leader FE node") After all the non-Leader FE nodes have been modified and restarted successfully, you can now enable FQDN access for the Leader FE node. > **NOTE** > > Before the Leader FE node is enabled with FQDN access, the FQDNs used to add nodes to a cluster are still transformed into the corresponding IP addresses. After a Leader FE node with FQDN access enabled is elected for the cluster, the FQDNs will not be transformed into IP addresses. 1. Navigate to the deployment directory of the Leader FE node, and run the following command to stop the Leader FE node. ```shell ./bin/stop_fe.sh ``` 2. Execute the following statement via your MySQL client to check whether a new Leader FE node has been elected for the cluster. ```sql SHOW PROC '/frontends'\G ``` Any FE node with status `Alive` and `isMaster` being `true` is a Leader FE that is running. 3. Execute the following statement to replace the IP address with FQDN. ```sql ALTER SYSTEM MODIFY FRONTEND HOST "" TO ""; ``` 4. Run the following command to start the FE node with FQDN access. ```shell ./bin/start_fe.sh --host_type FQDN --daemon ``` The property `--host_type` specifies the access method that is used to start the node. Valid values include `FQDN` and `IP`. You only need to specify this property ONCE when you restart the node after you modify the node. 5. Check the `Alive` status of the FE node. ```plain SHOW PROC '/frontends'\G ``` If the `Alive` status becomes `true`, the FE node is successfully modified and added to the cluster as a Follower FE node. ##### Enable FQDN access for BE nodes[​](#enable-fqdn-access-for-be-nodes "Direct link to Enable FQDN access for BE nodes") Execute the following statement via your MySQL client to replace the IP address with FQDN to enable FQDN access for the BE node. ```sql ALTER SYSTEM MODIFY BACKEND HOST "" TO ""; ``` > **NOTE** > > You DO NOT need to restart the BE node after FQDN access is enabled. #### Rollback[​](#rollback "Direct link to Rollback") To rollback an FQDN access-enabled StarRocks cluster to an earlier version that does not support FQDN access, you must first enable IP address access for all nodes in the cluster. You can refer [Enable FQDN access in an existing cluster](#enable-fqdn-access-in-an-existing-cluster) for the general guidance except that you need to change the SQL commands to the following ones: * Enable IP address access for an FE node: ```sql ALTER SYSTEM MODIFY FRONTEND HOST "" TO ""; ``` * Enable IP address access for a BE node: ```sql ALTER SYSTEM MODIFY BACKEND HOST "" TO ""; ``` The modification takes effect after your cluster is successfully restarted. #### FAQ[​](#faq "Direct link to FAQ") **Q: An error occurs when I enable FQDN access for an FE node: "required 1 replica. But none were active with this master". What should I do?** A: Make sure the cluster has at least three Follower FE nodes before you enable FQDN access for FE nodes. **Q: Can I add a new node by using IP address to a cluster with FQDN access enabled?** A: Yes. --- ### Develop Static Extensions Static Extensions are StarRocks FE’s extension modules that allow you to add new features or optimize existing functionality without modifying the core code. Compared to dynamic plugins, static extensions are automatically loaded at system startup and provide more registrable extension points, covering the system’s core modules. This feature is supported from v4.1 onwards. #### Usage[​](#usage "Direct link to Usage") The following example demonstrates how to develop a static extension. ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Prepare StarRocks FE development environment as follows: ```xml org.apache.maven.plugins maven-jar-plugin 3.3.0 default-jar package jar ${your_extension_directory}/** build-ext-jar package jar ${your_extension_name} ${your_extension_directory}/** ext ``` ##### Extension entry point[​](#extension-entry-point "Direct link to Extension entry point") ```java @SRModule(name = "extension_name") public class MultiWarehouseExtension implements StarRocksExtension { @Override public void onLoad(ExtensionContext ctx) { // Register your own class or perform other initialization ctx.register(WarehouseManager.class, new MyWarehouseManager()); ... } } ``` ##### Logs[​](#logs "Direct link to Logs") After building the extension, place the `${your_extension_name}-ext.jar` file into the `Config.ext_dir` directory (default is `FE/lib`), and then restart the FE. Example FE logs after startup are as follows: ```sh 2025-12-26 12:47:46.047+08:00 INFO (main|1) [ExtensionManager.loadExtensionsFromDir():39] start to load extensions 2025-12-26 12:47:46.152+08:00 INFO (main|1) [ExtensionManager.loadExtensions():63] Loaded extension: extension_name 2025-12-26 12:47:46.152+08:00 INFO (main|1) [ExtensionManager.loadExtensionsFromDir():42] all extensions loaded finished ``` * `start to load extensions`: FE has started scanning the extension directory. * `Loaded extension: extension_name`: The extension was successfully loaded. * `all extensions loaded finished`: All extensions in the directory have been loaded. --- ### Graceful Exit From v3.3 onwards, StarRocks supports Graceful Exit. #### Overview[​](#overview "Direct link to Overview") Graceful Exit is a mechanism designed to support **non-disruptive upgrades and restarts** of StarRocks FE, BE, and CN nodes. Its primary objective is to minimize the impact on running queries and data ingestion tasks during maintenance operations such as node restart, rolling upgrade, or cluster scaling. Graceful Exit ensures that: * The node **stops accepting new tasks** once exit begins; * Existing queries and load jobs are **allowed to complete** within a controlled time window; * System components (FE/BE/CN) **coordinate status changes** so that the cluster correctly reroutes traffic. Graceful Exit mechanisms differ between FE and BE/CN nodes, as described below. ##### FE Graceful Exit Mechanism[​](#fe-graceful-exit-mechanism "Direct link to FE Graceful Exit Mechanism") ###### Trigger Signal[​](#trigger-signal "Direct link to Trigger Signal") FE Graceful Exit is initiated via: ```bash stop_fe.sh -g ``` This sends a `SIGUSR1` signal, while the default exit (without the `-g` option) sends `SIGTERM` signal. ###### Load Balancer Awareness[​](#load-balancer-awareness "Direct link to Load Balancer Awareness") Upon receiving the signal: * FE immediately returns **HTTP 500** on the `/api/health` endpoint. * Load balancers detect the degraded state within ~15 seconds and stop routing new connections to this FE. ###### Connection Drain and Shutdown Logic[​](#connection-drain-and-shutdown-logic "Direct link to Connection Drain and Shutdown Logic") **Follower FE** * Handles read-only queries. * If the FE node has no active sessions, the connection is closed immediately. * If SQL is running, the FE node waits for execution to finish before closing the session. **Leader FE** * Read request handling is identical to the of Followers. * Write request handling requires: * Closing BDBJE. * Allowing a new Leader election to complete. * Redirecting subsequent writes to the newly elected Leader. ###### Timeout Control[​](#timeout-control "Direct link to Timeout Control") If a query runs for too long, FE forcibly exits after **60 seconds** (configurable via the `--timeout` option). ##### BE/CN Graceful Exit Mechanism[​](#becn-graceful-exit-mechanism "Direct link to BE/CN Graceful Exit Mechanism") ###### Trigger Signal[​](#trigger-signal-1 "Direct link to Trigger Signal") BE Graceful Exit is initiated via: ```bash stop_be.sh -g ``` CN Graceful Exit is initiated via: ```bash stop_cn.sh -g ``` This sends a `SIGTERM` signal, while the default exit (without the `-g` option) sends `SIGKILL` signal. ###### State Transition[​](#state-transition "Direct link to State Transition") After receiving the signal: * The BE/CN node marks itself as **exiting**. * It rejects **new query fragments** by returning `INTERNAL_ERROR`. * It continues processing existing fragments. ###### Wait Loop for In-Flight Queries[​](#wait-loop-for-in-flight-queries "Direct link to Wait Loop for In-Flight Queries") The behavior that BE/CN waits for existing fragments to finish is controlled by the BE/CN configuration `loop_count_wait_fragments_finish` (Default: 2). The actual wait duration equals `loop_count_wait_fragments_finish × 10 seconds` (that is, 20 seconds by default). If fragments remain after timeout, BE/CN proceeds with normal shutdown (closing threads, network, and other processes). ###### Improved FE Awareness[​](#improved-fe-awareness "Direct link to Improved FE Awareness") From v3.4 onwards, FE no longer marks BE/CN as `DEAD` based on heartbeat failures. It correctly recognizes the BE/CN “exiting” state, allowing significantly longer graceful-exit windows for fragments to be completed. #### Configurations[​](#configurations "Direct link to Configurations") ##### FE Configurations[​](#fe-configurations "Direct link to FE Configurations") ###### `stop_fe.sh -g --timeout`[​](#stop_fesh--g---timeout "Direct link to stop_fesh--g---timeout") * Description: Maximum waiting time before FE is force-killed. * Default: 60 (seconds) * How to apply: Specify it in the script command, for example, `--timeout 120`. ###### *Minimum LB detection time*[​](#minimum-lb-detection-time "Direct link to minimum-lb-detection-time") * Description: LB requires at least 15 seconds to detect degraded health. * Default: 15 (seconds) * How to apply: Fixed value ##### BE/CN Configurations[​](#becn-configurations "Direct link to BE/CN Configurations") ###### `loop_count_wait_fragments_finish`[​](#loop_count_wait_fragments_finish "Direct link to loop_count_wait_fragments_finish") * Description: BE/CN wait duration for existing fragments. Multiply the value with 10 seconds. * Default: 2 * How to apply: Modify it in the BE/CN configuration file or update it dynamically. ###### `graceful_exit_wait_for_frontend_heartbeat`[​](#graceful_exit_wait_for_frontend_heartbeat "Direct link to graceful_exit_wait_for_frontend_heartbeat") * Description: Whether BE/CN waits for FE to confirm **SHUTDOWN** via heartbeat. From v3.4.5 onwards. * Default: false * How to apply: Modify it in the BE/CN configuration file or update it dynamically. ###### `stop_be.sh -g --timeout`, `stop_cn.sh -g --timeout`[​](#stop_besh--g---timeout-stop_cnsh--g---timeout "Direct link to stop_besh--g---timeout-stop_cnsh--g---timeout") * Description: Maximum waiting time before BE/CN is force-killed. Set it to a value larger than `loop_count_wait_fragments_finish` \* 10 to prevent termination before the BE/CN wait duration is reached. * Default: false * How to apply: Specify it in the script command, for example, `--timeout 30`. ##### Global Switches[​](#global-switches "Direct link to Global Switches") Graceful Exit is **enabled by default** from v3.4 onwards. To disable it temporarily, set the BE/CN configuration `loop_count_wait_fragments_finish` to `0`. #### Expected Behavior During Graceful Exit[​](#expected-behavior-during-graceful-exit "Direct link to Expected Behavior During Graceful Exit") ##### Query Workloads[​](#query-workloads "Direct link to Query Workloads") | Query Type | Expected Behavior | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | **Short (less than 20 seconds)** | BE/CN waits long enough, so queries complete normally. | | **Medium (20 to 60 seconds)** | Queries completed within BE/CN wait window are returned with success; else queries are cancelled and require manual retry. | | **Long (more than 60 seconds)** | Queries are likely terminated by FE/BE/CN due to timeout and requires manual retry. | ##### Data Ingestion Tasks[​](#data-ingestion-tasks "Direct link to Data Ingestion Tasks") * **Loading tasks via Flink or Kafka Connectors** are automatically retried with no user-visible interruption. * **Stream Load (non-framework), Broker Load, and Routine Load tasks** may fail if connection breaks. Manual retry is required. * **Background tasks** are automatically re-scheduled and executed by the FE retry mechanism. ##### Upgrade and Restart Operations[​](#upgrade-and-restart-operations "Direct link to Upgrade and Restart Operations") Graceful Exit ensures: * No cluster-wide downtime; * Safe rolling upgrade by draining nodes one at a time. #### Limitations and Version Differences[​](#limitations-and-version-differences "Direct link to Limitations and Version Differences") ##### Version Behavior Differences[​](#version-behavior-differences "Direct link to Version Behavior Differences") | Version | Behavior | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | **v3.3** | BE Graceful Exit flawed: FE may prematurely mark BE/CN as `DEAD`, causing queries to get cancelled. The effective wait is limited (15 seconds by default). | | **v3.4+** | Fully supports extended wait duration; FE correctly identifies BE/CN “exiting” state. Recommended for production. | ##### Operational Limitations[​](#operational-limitations "Direct link to Operational Limitations") * In extreme cases (for example, BE/CN hangs), Graceful Exit may fail. Terminating the process requires `kill -9`, risking partial data persistence (recoverable via snapshot). #### Usage[​](#usage "Direct link to Usage") ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") **StarRocks version**: * **v3.3+**: Basic Graceful Exit support. * **v3.4+**: Enhanced status management, longer wait windows (up to several minutes). **Configuration**: * Make sure `loop_count_wait_fragments_finish` is set to a positive integer. * Set `graceful_exit_wait_for_frontend_heartbeat` to `true` allow FE to detect BE's "EXITING" state. ##### Perform FE Graceful Exit[​](#perform-fe-graceful-exit "Direct link to Perform FE Graceful Exit") ```bash ./bin/stop_fe.sh -g --timeout 60 ``` Parameters: * `--timeout`: The maximum time to wait before the FE node is force-killed. Behavior: * The system sends the `SIGUSR1` signal first. * After timeout, it falls back to `SIGKILL`. ###### Validate FE State[​](#validate-fe-state "Direct link to Validate FE State") You can check the FE health via the following API: ```text http://:8030/api/health ``` LB removes the node after receiving consecutive non-200 responses. ##### Perform BE/CN Graceful Exit[​](#perform-becn-graceful-exit "Direct link to Perform BE/CN Graceful Exit") * **For v3.3:** * BE: ```bash ./be/bin/stop_be.sh -g ``` * CN: ```bash ./be/bin/stop_cn.sh -g ``` * **For v3.4+:** * BE: ```bash ./bin/stop_be.sh -g --timeout 600 ``` * CN: ```bash ./bin/stop_cn.sh -g --timeout 600 ``` BE/CN exits immediately if no fragments remain. ###### Validate BE/CN Status[​](#validate-becn-status "Direct link to Validate BE/CN Status") Run on FE: ```sql SHOW BACKENDS; ``` `StatusCode`: * `SHUTDOWN`: BE/CN Graceful Exit in progress. * `DISCONNECTED`: BE/CN Node has fully exited. #### Rolling Upgrade Workflow[​](#rolling-upgrade-workflow "Direct link to Rolling Upgrade Workflow") ##### Procedure[​](#procedure "Direct link to Procedure") 1. Perform Graceful Exit on the node `A`. 2. Confirm the node `A` is shown as `DISCONNECTED` from the FE side. 3. Upgrade and restart the node `A`. 4. Repeat the above for remaining nodes. ##### Monitor Graceful Exit[​](#monitor-graceful-exit "Direct link to Monitor Graceful Exit") Check FE logs `fe.log`, BE logs `be.log`, or CN logs `cn.log` to make sure whether there were tasks during the exit. #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") ##### BE/CN exits by timeout[​](#becn-exits-by-timeout "Direct link to BE/CN exits by timeout") If tasks fail to complete within the Graceful Exit period, BE/CN will trigger forced termination (`SIGKILL`). Verify whether this is caused by excessively long task duration or improper configurations (for example, an overly small `--timeout` value). ##### Node status is not SHUTDOWN[​](#node-status-is-not-shutdown "Direct link to Node status is not SHUTDOWN") If the node status is not `SHUTDOWN`, verify whether `loop_count_wait_fragments_finish` is set to a positive integer, or if BE/CN reported a heartbeat before exiting (if not, set `graceful_exit_wait_for_frontend_heartbeat` to `true`). --- ### logs When deploying and operating StarRocks, understanding and properly using the logging system is critical for troubleshooting, performance analysis, and system tuning. This article provides a detailed overview of the log file types, typical content, configuration methods, and log rolling and retention strategies for both the Frontend (FE) and Backend (BE or CN) components of StarRocks. The information in this document is based on StarRocks Version 3.5.x. #### FE Logging in detail[​](#fe-logging-in-detail "Direct link to FE Logging in detail") ##### `fe.log`[​](#felog "Direct link to felog") The main FE logs include the startup process, cluster state changes, DML/DQL requests, and scheduling-related information. These logs primarily record the behavior of the FE during its runtime. ###### Configuration[​](#configuration "Direct link to Configuration") * `sys_log_dir`: Log storage directory. Default is `${STARROCKS_HOME}/log` * `sys_log_level`: Log level. Default is `INFO` * `sys_log_roll_num`: Controls the number of retained log files to prevent unlimited growth from consuming too much disk space. Default is 10 * `sys_log_roll_interval`: Specifies the rotation frequency. Default is `DAY`, meaning logs are rotated daily * `sys_log_delete_age`: Controls how long to keep old log files before deletion. Default is 7 day * `sys_log_roll_mode`: Log rotation mode. Default is `SIZE-MB-1024`, meaning a new log file will be created when the current one reaches 1024 MB. Together with sys\_log\_roll\_interval, this indicates that FE logs can be rotated either daily or based on file size * `sys_log_enable_compress`: Controls whether log compression is enabled. Default is false, meaning compression is disabled ##### `fe.warn.log`[​](#fewarnlog "Direct link to fewarnlog") `fe.warn.log` is an important log file for system monitoring and troubleshooting: * Operations monitoring–monitors system health status * Fault diagnosis–quickly locates critical issues * Performance analysis–identifies system bottlenecks and anomalies * Security auditing–records permission and access errors Compared to `fe.log`, which records logs of all levels, `fe.warn.log` focuses on warnings and errors that require attention, helping operations personnel quickly identify and address system issues. ##### `fe.gc.log`[​](#fegclog "Direct link to fegclog") `fe.gc.log` is the Java garbage collection log of the StarRocks FE, used to monitor and analyze JVM garbage collection behavior. It’s important to note that this log file uses the native JVM log rotation mechanism. For example, you can enable automatic rotation based on file size and file count with the following configuration: ```bash -Xlog:gc*:${LOG_DIR}/fe.gc.log:time,tags:filecount=7,filesize=100M ``` ##### `fe.out`[​](#feout "Direct link to feout") `fe.out` is the standard output log file of StarRocks FE. It records the content printed to standard output (stdout) and standard error (stderr) during the runtime of the FE process. The main content includes: * Console output during FE startup * JVM startup information (e.g., heap settings, GC parameters) * FE module initialization order (Catalog, Scheduler, RPC, HTTP Server, etc.) * Error stack traces from stderr * Java exceptions (Exception/StackTrace) * Uncaught errors like ClassNotFound, NullPointerException, etc. * Output not captured by other logging systems * Some third-party libraries that use `System.out.println()` or `e.printStackTrace()` You should check `fe.out` in the following scenarios: * FE fails to start: Check `fe.out` for Java exceptions or invalid parameter messages. * FE crashes unexpectedly: Look for uncaught exception stack traces. * Logging system uncertainty: If `fe.out` is the only log file with output in the FE log directory, it’s likely that `log4j` failed to initialize, or the configuration is incorrect. By default, `fe.out` does not support automatic log rotation. ##### `fe.profile.log`[​](#feprofilelog "Direct link to feprofilelog") The purpose of `fe.profile.log` is to record detailed query execution information for performance analysis. Its main functions include: * Query performance analysis: Logs detailed execution data for each query, including: * Query ID, user, database, SQL statement * Execution timing (startTime, endTime, latency) * Resource usage (CPU, memory, number/size of scanned rows) * Execution status (RUNNING, FINISHED, FAILED, CANCELLED) * Runtime metrics tracking: Captures key indicators via the QueryDetail class: * `scanRows / scanBytes`: Amount of data scanned * `returnRows`: Number of result rows returned * `cpuCostNs`: CPU time consumed (in nanoseconds) * `memCostBytes`: Memory usage * `spillBytes`: Amount of data spilled to disk * Error diagnosis: Records error messages and stack traces for failed queries * Resource group monitoring: Tracks query execution metrics across different resource groups `fe.profile.log` is stored in JSON format. ###### Configuration[​](#configuration-1 "Direct link to Configuration") * `enable_profile_log`: Whether to enable profile logging * `profile_log_dir`: Directory for storing profile logs * `profile_log_roll_size_mb`: Log rotation size (in MB) * `profile_log_roll_num`: Controls the number of retained profile log files to prevent unlimited growth and excessive disk usage. Default is 5 * `profile_log_roll_interval`: Specifies the rotation frequency. Default is DAY, meaning daily rotation. When rotation conditions are met, the latest 5 files are retained, and older files are deleted * `profile_log_delete_age`: Controls how long old files are kept before deletion. Default is 1 day ##### `fe.internal.log`[​](#feinternallog "Direct link to feinternallog") The purpose of `fe.internal.log` is to record logs dedicated to internal operations of the FE (Frontend), primarily for system-level auditing and debugging. Its main functions include: 1. Internal operation auditing: Logs system-initiated internal SQL executions separately from user queries 2. Statistics tracking: Specifically records operations related to statistics collection 3. Debugging support: Provides detailed logs for troubleshooting internal operation issues The log records include entries such as: * Statistics module (internal.statistic) * Core system module (internal.base) This log is especially useful for analyzing StarRocks’ internal statistics collection process and diagnosing issues related to internal operations. ###### Configuration[​](#configuration-2 "Direct link to Configuration") * `internal_log_dir`: Controls the storage directory for this log * `internal_log_modules`: An array configuring internal log modules, defining which internal operation modules need to be recorded in the `fe.internal.log` file. Default is `{"base", "statistic"}` * `internal_log_roll_num`: Number of files to retain. Default is 90 * `internal_log_roll_interval`: Specifies the rotation frequency. Default is DAY, meaning daily rotation. When rotation conditions are met, the latest 90 files are retained, and older files are deleted * `internal_log_delete_age`: Controls how long old files are kept before deletion. Default is 7 days ##### `fe.audit.log`[​](#feauditlog "Direct link to feauditlog") This is StarRocks’ query audit log, which records detailed information about all user queries and connections. It is used for monitoring, analysis, and auditing. Its main purposes include: 1. Query monitoring: Logs the execution status and performance metrics of all SQL queries 2. User auditing: Tracks user behavior and database access 3. Performance analysis: Provides metrics such as query execution time and resource consumption 4. Issue diagnosis: Records error statuses and error codes to facilitate troubleshooting ###### Configuration[​](#configuration-3 "Direct link to Configuration") * `audit_log_dir`: Controls the storage directory for this log * `audit_log_roll_num`: Number of files to retain. Default is 90 * `audit_log_roll_interval`: Specifies the rotation frequency. Default is DAY, meaning daily rotation. When rotation conditions are met, the latest 90 files are retained, and older files are deleted * `audit_log_delete_age`: Controls how long old files are kept before deletion. Default is 7 days * `audit_log_json_format`: Whether to log in JSON format. Default is false * `audit_log_enable_compress`: Whether compression is enabled ##### `fe.big_query.log`[​](#febig_querylog "Direct link to febig_querylog") This is StarRocks’ dedicated Big Query log file, used to monitor and analyze queries with high resource consumption. Its structure is similar to the audit log, but it includes three additional fields: * `bigQueryLogCPUSecondThreshold`: CPU time threshold * `bigQueryLogScanBytesThreshold`: Scan size (in bytes) threshold * `bigQueryLogScanRowsThreshold`: Scan row count threshold ###### Configuration[​](#configuration-4 "Direct link to Configuration") * `big_query_log_dir`: Controls the storage directory for this log * `big_query_log_roll_num`: Number of files to retain. Default is 10 * `big_query_log_modules`: Types of internal log modules. Default is query * `big_query_log_roll_interval`: Specifies the rotation frequency. Default is DAY, meaning daily rotation. When rotation conditions are met, the latest 10 files are retained, and older files are deleted * `big_query_log_delete_age`: Controls how long old files are kept before deletion. Default is 7 days ##### `fe.dump.log`[​](#fedumplog "Direct link to fedumplog") This is StarRocks’ query dump log, specifically used for detailed query debugging and issue diagnosis. Its main purposes include: * Exception debugging: Automatically records the complete query context when query execution encounters exceptions * Issue reproduction: Provides sufficiently detailed information to reproduce query problems * In-depth diagnosis: Contains debugging information such as metadata, statistics, execution plans, and more * Technical support: Provides comprehensive data for the technical support team to analyze issues It can be enabled using the following command: ```bash SET enable_query_dump = true; ``` ###### Configuration[​](#configuration-5 "Direct link to Configuration") * `dump_log_dir`: Controls the storage directory for this log * `dump_log_roll_num`: Number of files to retain. Default is 10 * `dump_log_modules`: Types of internal log modules. Default is query * `dump_log_roll_interval`: Specifies the rotation frequency. Default is DAY, meaning daily rotation. When rotation conditions are met, the latest 10 files are retained, and older files are deleted * `dump_log_delete_age`: Controls how long old files are kept before deletion. Default is 7 days ##### `fe.features.log`[​](#fefeatureslog "Direct link to fefeatureslog") This is StarRocks’ query plan feature log, used to collect and record feature information of query execution plans. It mainly serves machine learning and query optimization analysis. Key purposes include: 1. Query plan feature collection: Extracting various characteristics from query execution plans 2. Machine learning data source: Providing training data for query cost prediction models 3. Query pattern analysis: Analyzing execution patterns and feature distributions 4. Optimizer improvement: Supplying data to support enhancements in the cost-based optimizer (CBO) It can be enabled via configuration. ```bash // Enable plan feature collection enable_plan_feature_collection = false // Disabled by default // Enable query cost prediction enable_query_cost_prediction = false // Disabled by default ``` ###### Configuration[​](#configuration-6 "Direct link to Configuration") * `feature_log_dir`: Controls the storage directory for this log * `feature_log_roll_num`: Number of files to retain. Default is 5 * `feature_log_roll_interval`: Specifies the rotation frequency. Default is DAY, meaning daily rotation. When rotation conditions are met, the latest 5 files are retained, and older files are deleted * `feature_log_delete_age`: Controls how long old files are kept before deletion. Default is 3 days * `feature_log_roll_size_mb`: Log rotation size. Default is 1024 MB, meaning a new file is created every 1 GB #### BE/CN Logging in detail[​](#becn-logging-in-detail "Direct link to BE/CN Logging in detail") ##### `{be or cn}.INFO.log`[​](#be-or-cninfolog "Direct link to be-or-cninfolog") It primarily records various runtime behavior logs generated by BE/CN nodes, and these logs are at the `INFO` level. For example: * System startup information: * BE process startup and initialization * Hardware resource detection (CPU, memory, disk) * Configuration parameter loading * Query execution information: * Query reception and dispatch * Fragment execution status * Storage-related information: * Tablet loading and unloading * Compaction execution process * Data import status * Storage space management ###### Configuration[​](#configuration-7 "Direct link to Configuration") * `sys_log_level`: Log level, default is `INFO` * `sys_log_dir`: Log storage directory, default is `${STARROCKS_HOME}/log` * `sys_log_roll_mode`: Log rotation mode, default is `SIZE-MB-1024`, meaning a new log file is created when the current one reaches 1024 MB * `sys_log_roll_num`: Number of retained log files, default is 10 ##### `{be or cn}.WARN.log`[​](#be-or-cnwarnlog "Direct link to be-or-cnwarnlog") `be.WARN.log` stores log entries at WARNING level and above. Examples include Query execution warnings: * Query execution time too long * Memory allocation failure warning * Operator execution exception Storage-related warnings: * Abnormal Tablet status * Slow compaction execution * Data file corruption warning * Storage I/O exception ###### Configuration[​](#configuration-8 "Direct link to Configuration") * `sys_log_level`: Log level, default is INFO * `sys_log_dir`: Log storage directory, default is `${STARROCKS_HOME}/log` * `sys_log_roll_mode`: Log rotation mode, default is `SIZE-MB-1024`, meaning a new log file is created when the current one reaches 1024 MB * `sys_log_roll_num`: Number of retained log files, default is 10 ##### `{be or cn}.ERROR.log`[​](#be-or-cnerrorlog "Direct link to be-or-cnerrorlog") `be.ERROR.log` stores log entries at `ERROR` level and above. Typical error log contents: Query Execution Errors * Query timeout or cancellation * Query failure due to insufficient memory Data Processing Errors * Data load failures (e.g., format errors, constraint violations) * Data write failures * Data read errors (e.g., file corruption, I/O errors) Storage System Errors * Tablet load failures * Compaction execution failures * Corrupted data files * Disk I/O errors ###### Configuration[​](#configuration-9 "Direct link to Configuration") * `sys_log_level`: Log level, default is `INFO` * `sys_log_dir`: Log storage directory, default is `${STARROCKS_HOME}/log` * `sys_log_roll_mode`: Log rotation mode, default is `SIZE-MB-1024`, meaning a new log file is created when the current one reaches 1024 MB * `sys_log_roll_num`: Number of retained log files, default is 10 ##### `{be or cn}.FATAL.log`[​](#be-or-cnfatallog "Direct link to be-or-cnfatallog") It primarily records various runtime behavior logs generated by BE/CN nodes, and these logs are at the `FATAL` level. Once such a log is generated, the BE/CN node process will exit. ###### Configuration[​](#configuration-10 "Direct link to Configuration") * `sys_log_level`: Log level, default is `INFO` * `sys_log_dir`: Log storage directory, default is `${STARROCKS_HOME}/log` * `sys_log_roll_mode`: Log rotation mode, default is `SIZE-MB-1024`, meaning a new log file is created when the current one reaches 1024 MB * `sys_log_roll_num`: Number of retained log files, default is 10 ##### `error_log`[​](#error_log "Direct link to error_log") This log primarily records various errors, rejected records, and ETL issues encountered by BE/CN nodes during data import. Users can obtain the main reasons for import errors via `http://be_ip:be_port/api/get_log_file`. The log files are stored in the `${STARROCKS_HOME}/storage/error_log` directory. ###### Configuration[​](#configuration-11 "Direct link to Configuration") * `load_error_log_reserve_hours`: How long error log files are retained. The default is 48 hours, meaning the log files will be deleted 48 hours after they are created. --- ### Monitor and manage big queries This topic describes how to monitor and manage big queries in your StarRocks cluster. Big queries include queries that scan too many rows or occupy too many CPU and memory resources. They can easily exhaust cluster resources and cause system overload if no restrictions are imposed on them. To tackle this issue, StarRocks provides a series of measures to monitor and manage big queries, preventing queries from monopolizing cluster resources. The overall idea of handling big queries in StarRocks is as follows: 1. Set automatic precautions against big queries with resource groups and query queues. 2. Monitor big queries in real-time, and terminate those who bypass the precautions. 3. Analyze audit logs and Big Query Logs to study the patterns of big queries, and fine-tune the precaution mechanisms you set earlier. This feature is supported from v3.0. #### Set precautions against big queries[​](#set-precautions-against-big-queries "Direct link to Set precautions against big queries") StarRocks provides two precautionary instruments for dealing with big queries - resource groups and query queues. You can use resource groups to stop big queries from being executed. Query queues, on the other hand, can help you queue the incoming queries when the concurrency threshold or resource limit is reached, preventing system overload. ##### Filter out big queries via resource groups[​](#filter-out-big-queries-via-resource-groups "Direct link to Filter out big queries via resource groups") Resource groups can automatically identify and terminate big queries. When creating a resource group, you can specify the upper limit of CPU time, memory usage, or scan row count that a query is entitled to. Among all queries that hit the resource group, any queries that require more resources are rejected and returned with an error. For more information and instructions on resource groups, see [Resource Isolation](https://docs.starrocks.io/docs/administration/management/resource_management/resource_group.md). Before creating resource groups, you must execute the following statement to enable Pipeline Engine, on which the Resource Group feature depends: ```sql SET GLOBAL enable_pipeline_engine = true; ``` The following example creates a resource group `bigQuery` that limits the CPU time upper limit to `100` seconds, scan row count upper limit to `100000`, and memory usage upper limit to `1073741824` bytes (1 GB): ```sql CREATE RESOURCE GROUP bigQuery TO (db='sr_hub') WITH ( 'cpu_weight' = '10', 'mem_limit' = '20%', 'big_query_cpu_second_limit' = '100', 'big_query_scan_rows_limit' = '100000', 'big_query_mem_limit' = '1073741824' ); ``` If the required resources of a query exceed any of the limits, the query will not be executed and is returned with an error. The following example shows the error message returned when a query demands too many scan rows: ```plain ERROR 1064 (HY000): exceed big query scan_rows limit: current is 4 but limit is 1 ``` If it is your first time setting up resource groups, we recommend you set relatively higher limits so that they will not hinder regular queries. You can fine-tune these limits after you have a better knowledge of the big query patterns. ##### Ease system overload via query queues[​](#ease-system-overload-via-query-queues "Direct link to Ease system overload via query queues") Query queues are designed to cushion the system overload deterioration when the cluster resource occupation exceeds the prespecified thresholds. You can set thresholds for maximum concurrency, memory usage, and CPU usage. StarRocks automatically queues the incoming queries when any of these thresholds is reached. Pending queries either wait in the queue for execution or get cancelled when the prespecified resource threshold is reached. For more information, see [Query Queues](https://docs.starrocks.io/docs/administration/management/resource_management/query_queues.md). Execute the following statements to enable query queues for the SELECT queries: ```sql SET GLOBAL enable_query_queue_select = true; ``` After the query queue feature is enabled, you can then define the rules to trigger query queues. * Specify the concurrency threshold for triggering the query queue. The following example sets the concurrency threshold to `100`: ```sql SET GLOBAL query_queue_concurrency_limit = 100; ``` * Specify the memory usage ratio threshold for triggering the query queue. The following example sets the memory usage ratio threshold to `0.9`: ```sql SET GLOBAL query_queue_mem_used_pct_limit = 0.9; ``` * Specify the CPU usage ratio threshold for triggering the query queue. The following example sets the CPU usage permille (CPU usage \* 1000) threshold to `800`: ```sql SET GLOBAL query_queue_cpu_used_permille_limit = 800; ``` You can also decide how to deal with these queued queries by configuring the maximum queue length and the timeout for each pending query in the queue. * Specify the maximum query queue length. When this threshold is reached, incoming queries are rejected. The following example sets the query queue length to `100`: ```sql SET GLOBAL query_queue_max_queued_queries = 100; ``` * Specify the maximum timeout of a pending query in a queue. When this threshold is reached, the corresponding query is rejected. The following example sets the maximum timeout to `480` seconds: ```sql SET GLOBAL query_queue_pending_timeout_second = 480; ``` You can check whether a query is pending using [SHOW PROCESSLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROCESSLIST.md). ```plain mysql> SHOW PROCESSLIST; +------+------+---------------------+-------+---------+---------------------+------+-------+-------------------+-----------+ | Id | User | Host | Db | Command | ConnectionStartTime | Time | State | Info | IsPending | +------+------+---------------------+-------+---------+---------------------+------+-------+-------------------+-----------+ | 2 | root | xxx.xx.xxx.xx:xxxxx | | Query | 2022-11-24 18:08:29 | 0 | OK | SHOW PROCESSLIST | false | +------+------+---------------------+-------+---------+---------------------+------+-------+-------------------+-----------+ ``` If `IsPending` is `true`, the corresponding query is pending in the query queue. #### Monitor big queries in real-time[​](#monitor-big-queries-in-real-time "Direct link to Monitor big queries in real-time") From v3.0 onwards, StarRocks supports viewing the queries that are currently processed in the cluster and the resources they occupy. This allows you to monitor the cluster in case any big queries bypass the precautions and cause unexpected system overload. ##### Monitor via MySQL client[​](#monitor-via-mysql-client "Direct link to Monitor via MySQL client") 1. You can view the queries that are currently processed (`current_queries`) using [SHOW PROC](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md). ```sql SHOW PROC '/current_queries'; ``` StarRocks returns the query ID (`QueryId`), connection ID (`ConnectionId`), and the resource consumption of each query, including the scanned data size (`ScanBytes`), processed row count (`ProcessRows`), CPU time (`CPUCostSeconds`), memory usage (`MemoryUsageBytes`), and execution time (`ExecTime`). ```plain mysql> SHOW PROC '/current_queries'; +--------------------------------------+--------------+------------+------+-----------+----------------+----------------+------------------+----------+ | QueryId | ConnectionId | Database | User | ScanBytes | ProcessRows | CPUCostSeconds | MemoryUsageBytes | ExecTime | +--------------------------------------+--------------+------------+------+-----------+----------------+----------------+------------------+----------+ | 7c56495f-ae8b-11ed-8ebf-00163e00accc | 4 | tpcds_100g | root | 37.88 MB | 1075769 Rows | 11.13 Seconds | 146.70 MB | 3804 | | 7d543160-ae8b-11ed-8ebf-00163e00accc | 6 | tpcds_100g | root | 13.02 GB | 487873176 Rows | 81.23 Seconds | 6.37 GB | 2090 | +--------------------------------------+--------------+------------+------+-----------+----------------+----------------+------------------+----------+ 2 rows in set (0.01 sec) ``` 2. You can further examine a query's resource consumption on each BE node by specifying the query ID. ```sql SHOW PROC '/current_queries//hosts'; ``` StarRocks returns the query's scanned data size (`ScanBytes`), scanned row count (`ScanRows`), CPU time (`CPUCostSeconds`), and memory usage (`MemUsageBytes`) on each BE node. ```plain mysql> show proc '/current_queries/7c56495f-ae8b-11ed-8ebf-00163e00accc/hosts'; +--------------------+-----------+-------------+----------------+---------------+ | Host | ScanBytes | ScanRows | CpuCostSeconds | MemUsageBytes | +--------------------+-----------+-------------+----------------+---------------+ | 172.26.34.185:8060 | 11.61 MB | 356252 Rows | 52.93 Seconds | 51.14 MB | | 172.26.34.186:8060 | 14.66 MB | 362646 Rows | 52.89 Seconds | 50.44 MB | | 172.26.34.187:8060 | 11.60 MB | 356871 Rows | 52.91 Seconds | 48.95 MB | +--------------------+-----------+-------------+----------------+---------------+ 3 rows in set (0.00 sec) ``` ##### Monitor via FE console[​](#monitor-via-fe-console "Direct link to Monitor via FE console") In addition to MySQL client, you can use the FE console for visualized, interactive monitoring. 1. Navigate to the FE console in your browser using the following URL: ```bash http://:/system?path=//current_queries ``` ![FE console 1](/assets/images/console_1-cbd35b88e205f18566c1a3ecdb888f4c.png) You can view the queries that are currently processed and their resource consumption on the **System Info** page. 2. Click the **QueryID** of the query. ![FE console 2](/assets/images/console_2-b772487ce01c6612ba48da1e08fd7609.png) You can view the detailed, node-specific resource consumption information on the page that appears. ##### Manually terminate big queries[​](#manually-terminate-big-queries "Direct link to Manually terminate big queries") If any big queries bypass the precautions you have set and threaten the system availability, you can terminate them manually using the corresponding connection ID in the [KILL](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/KILL.md) statement: ```sql KILL QUERY ; ``` #### Analyze Big Query Logs[​](#analyze-big-query-logs "Direct link to Analyze Big Query Logs") From v3.0 onwards, StarRocks supports Big Query Logs, which are stored in the file **fe/log/fe.big\_query.log**. Compared to the StarRocks audit logs, Big Query Logs print additional three fields: * `bigQueryLogCPUSecondThreshold` * `bigQueryLogScanBytesThreshold` * `bigQueryLogScanRowsThreshold` These three fields correspond to the resource consumption thresholds you defined to determine whether a query is a big query. To enable Big Query Logs, execute the following statement: ```sql SET GLOBAL enable_big_query_log = true; ``` After Big Query Logs are enabled, you can then define the rules to trigger Big Query Logs. * Specify the CPU time threshold for triggering Big Query Logs. The following example sets the CPU time threshold to `600` seconds: ```sql SET GLOBAL big_query_log_cpu_second_threshold = 600; ``` * Specify the scan data size threshold for triggering Big Query Logs. The following example sets the scan data size threshold to `10737418240` bytes (10 GB): ```sql SET GLOBAL big_query_log_scan_bytes_threshold = 10737418240; ``` * Specify the scan row count threshold for triggering Big Query Logs. The following example sets the scan row count threshold to `1500000000`: ```sql SET GLOBAL big_query_log_scan_rows_threshold = 1500000000; ``` #### Fine-tune precautions[​](#fine-tune-precautions "Direct link to Fine-tune precautions") From the statistics obtained from real-time monitoring and Big Query Logs, you can study the pattern of the omitted big queries (or regular queries that are mistakenly diagnosed as big queries) in your cluster, and then optimize the settings for resource groups and the query queue. If a notable proportion of big queries conform to a certain SQL pattern, and you want to permanently forbid this SQL pattern, you can add this pattern to SQL Blacklist. StarRocks rejects all queries that match any patterns specified in SQL Blacklist, and returns an error. For more information, see [Manage SQL Blacklist](https://docs.starrocks.io/docs/administration/management/resource_management/Blacklist.md). To enable SQL Blacklist, execute the following statement: ```sql ADMIN SET FRONTEND CONFIG ("enable_sql_blacklist" = "true"); ``` Then you can add the regular expression that represents the SQL pattern to SQL Blacklist using [ADD SQLBLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/sql_blacklist/ADD_SQLBLACKLIST.md). The following example adds `COUNT(DISTINCT)` to SQL Blacklist: ```sql ADD SQLBLACKLIST "SELECT COUNT(DISTINCT .+) FROM .+"; ``` --- ### Monitor and Alert with Prometheus and Grafana StarRocks provides a monitor and alert solution by using Prometheus and Grafana. This allows you to visualize the running of your cluster, facilitating monitoring and troubleshooting. #### Overview[​](#overview "Direct link to Overview") StarRocks provides a Prometheus-compatible information collection interface. Prometheus can retrieve metric information of StarRocks by connecting to the HTTP ports of BE and FE nodes and storing the information in its own time-series database. Grafana can then use Prometheus as a data source to visualize the metric information. By using the dashboard templates provided by StarRocks, you can easily monitor your StarRocks cluster and set alerts for it with Grafana. ![MA-1](/assets/images/monitor1-c933907ca154e75553acfa00488e5904.png) Follow these steps to integrate your StarRocks cluster with Prometheus and Grafana: 1. Install necessary components - Prometheus and Grafana. 2. Understand the core monitoring metrics of StarRocks. 3. Set alert channel and alert rule. #### Step 1: Install Monitoring Components[​](#step-1-install-monitoring-components "Direct link to Step 1: Install Monitoring Components") The default ports of Prometheus and Grafana do not conflict with those of StarRocks. However, it is recommended to deploy them on a different server from that of your StarRocks clusters for production. This reduces the risk of resource conflicts and avoids potential alert failure due to the server's abnormal shutdown. Additionally, please note that Prometheus and Grafana cannot monitor their own service's availability. Therefore, in a production environment, it is recommended to use Supervisor to set up a heartbeat service for them. The following tutorial deploys monitoring components on the monitoring node (IP: 192.168.110.23) using the root OS user. They monitor the following StarRocks cluster (which uses default ports). When setting up a monitoring service for your own StarRocks cluster based on this tutorial, you only need to replace the IP addresses. | **Host** | **IP** | **OS user** | **Services** | | -------- | --------------- | ----------- | ------------ | | node01 | 192.168.110.101 | root | 1 FE + 1 BE | | node02 | 192.168.110.102 | root | 1 FE + 1 BE | | node03 | 192.168.110.103 | root | 1 FE + 1 BE | > **NOTE** > > Prometheus and Grafana can only monitor FE, BE, and CN nodes, not Broker nodes. ##### 1.1 Deploy Prometheus[​](#11-deploy-prometheus "Direct link to 1.1 Deploy Prometheus") ###### 1.1.1 Download Prometheus[​](#111-download-prometheus "Direct link to 1.1.1 Download Prometheus") For StarRocks, you only need to download the installation package of the Prometheus server. Download the package to the monitoring node. [Click here to download Prometheus](https://prometheus.io/download/). Take the LTS version v2.45.0 as an example, click the package to download it. ![MA-2](/assets/images/monitor2-c1ff652ccf35625c91e46ffcb24cd63e.png) Alternatively, you can download it using the `wget` command: ```bash # The following example downloads the LTS version v2.45.0. # You can download other versions by replacing the version number in the command. wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz ``` After the download is complete, upload or copy the installation package to the directory **/opt** on the monitoring node. ###### 1.1.2 Install Prometheus[​](#112-install-prometheus "Direct link to 1.1.2 Install Prometheus") 1. Navigate to **/opt** and decompress the Prometheus installation package. ```bash cd /opt tar xvf prometheus-2.45.0.linux-amd64.tar.gz ``` 2. For ease of management, rename the decompressed directory to **prometheus**. ```bash mv prometheus-2.45.0.linux-amd64 prometheus ``` 3. Create a data storage path for Prometheus. ```bash mkdir prometheus/data ``` 4. For ease of management, you can create a system service startup file for Prometheus. ```bash vim /etc/systemd/system/prometheus.service ``` Add the following content to the file: ```properties [Unit] Description=Prometheus service After=network.target [Service] User=root Type=simple ExecReload=/bin/sh -c "/bin/kill -1 `/usr/bin/pgrep prometheus`" ExecStop=/bin/sh -c "/bin/kill -9 `/usr/bin/pgrep prometheus`" ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --storage.tsdb.path=/opt/prometheus/data --storage.tsdb.retention.time=30d --storage.tsdb.retention.size=30GB [Install] WantedBy=multi-user.target ``` Then, save and exit the editor. > **NOTE** > > If you deploy Prometheus under a different path, please make sure to synchronize the path in the ExecStart command in the file above. Additionally, the file configures the expiration conditions for Prometheus data storage to be "30 days or more" or "greater than 30 GB". You can modify this according to your needs. 5. Modify the Prometheus configuration file **prometheus/prometheus.yml**. This file has strict requirements for the format of the content. Please pay special attention to spaces and indentation when making modifications. ```bash vim prometheus/prometheus.yml ``` Add the following content to the file: ```yaml global: scrape_interval: 15s # Set the global scrape interval to 15s. The default is 1 min. evaluation_interval: 15s # Set the global rule evaluation interval to 15s. The default is 1 min. scrape_configs: - job_name: 'StarRocks_Cluster01' # A cluster being monitored corresponds to a job. You can customize the StarRocks cluster name here. metrics_path: '/metrics' # Specify the Restful API for retrieving monitoring metrics. static_configs: # The following configuration specifies an FE group, which includes 3 FE nodes. # Here, you need to fill in the IP and HTTP ports corresponding to each FE. # If you modified the HTTP ports during cluster deployment, make sure to adjust them accordingly. - targets: ['192.168.110.101:8030','192.168.110.102:8030','192.168.110.103:8030'] labels: group: fe # The following configuration specifies a BE group, which includes 3 BE nodes. # Here, you need to fill in the IP and HTTP ports corresponding to each BE. # If you modified the HTTP ports during cluster deployment, make sure to adjust them accordingly. - targets: ['192.168.110.101:8040','192.168.110.102:8040','192.168.110.103:8040'] labels: group: be ``` note Please note that Prometheus is unable to detect the service changes (`targets`) after the cluster has been scaled in or out. For example, for clusters deployed on AWS, you can grant the EC2 instance that hosts the Prometheus service the `ec2:DescribeInstances` and `ec2:DescribeTags` permissions, and add the `ec2_sd_configs` and `relabel_configs` properties to **prometheus/prometheus.yml**. For detailed instructions, see [Appendix - Enable Service Detection for Prometheus](#enable-service-detection-for-prometheus). After you have modified the configuration file, you can use `promtool` to verify whether the modification is valid. ```bash ./prometheus/promtool check config prometheus/prometheus.yml ``` The following prompt indicates that the check has passed. You can then proceed. ```bash SUCCESS: prometheus/prometheus.yml is valid prometheus config file syntax ``` 6. Start Prometheus. ```bash systemctl daemon-reload systemctl start prometheus.service ``` 7. Check the status of Prometheus. ```bash systemctl status prometheus.service ``` If `Active: active (running)` is returned, it indicates that Prometheus has started successfully. You can also use `netstat` to check the status of the default Prometheus port (9090). ```bash netstat -nltp | grep 9090 ``` 8. Set Prometheus to start on boot. ```bash systemctl enable prometheus.service ``` **Other commands**: * Stop Prometheus. ```bash systemctl stop prometheus.service ``` * Restart Prometheus. ```bash systemctl restart prometheus.service ``` * Reload configurations on runtime. ```bash systemctl reload prometheus.service ``` * Disable start on boot. ```bash systemctl disable prometheus.service ``` ###### 1.1.3 Access Prometheus[​](#113-access-prometheus "Direct link to 1.1.3 Access Prometheus") You can access the Prometheus Web UI through a browser, and the default port is 9090. For the monitoring node in this tutorial, you need to visit `192.168.110.23:9090`. On the Prometheus homepage, navigate to **Status** --> **Targets** in the top menu. Here, you can see all the monitored nodes for each group job configured in the **prometheus.yml** file. Usually, the status of all nodes should be UP, indicating that the service communication is normal. ![MA-3](/assets/images/monitor3-0edea7a4319bf836da3e55c4a24559f9.jpeg) At this point, Prometheus is configured and set up. For more detailed information, you can refer to the [Prometheus Documentation](https://prometheus.io/docs/). ##### 1.2 Deploy Grafana[​](#12-deploy-grafana "Direct link to 1.2 Deploy Grafana") ###### 1.2.1 Download Grafana[​](#121-download-grafana "Direct link to 1.2.1 Download Grafana") [Click here to download Grafana](https://grafana.com/grafana/download). Alternatively, you can use the `wget` command to download the Grafana RPM installation package. ```bash # The following example downloads the LTS version v10.0.3. # You can download other versions by replacing the version number in the command. wget https://dl.grafana.com/enterprise/release/grafana-enterprise-10.0.3-1.x86_64.rpm ``` ###### 1.2.2 Install Grafana[​](#122-install-grafana "Direct link to 1.2.2 Install Grafana") 1. Use the `yum` command to install Grafana. This command will automatically install the dependencies required for Grafana. ```bash yum -y install grafana-enterprise-10.0.3-1.x86_64.rpm ``` 2. Start Grafana. ```bash systemctl start grafana-server.service ``` 3. Check the status of Grafana. ```bash systemctl status grafana-server.service ``` If `Active: active (running)` is returned, it indicates that Grafana has started successfully. You can also use `netstat` to check the status of the default Grafana port (3000). ```bash netstat -nltp | grep 3000 ``` 4. Set Grafana to start on boot. ```bash systemctl enable grafana-server.service ``` **Other commands**: * Stop Grafana. ```bash systemctl stop grafana-server.service ``` * Restart Grafana. ```bash systemctl restart grafana-server.service ``` * Disable start on boot. ```bash systemctl disable grafana-server.service ``` For more information, refer to the [Grafana Documentation](https://grafana.com/docs/grafana/latest/). ###### 1.2.3 Access Grafana[​](#123-access-grafana "Direct link to 1.2.3 Access Grafana") You can access the Grafana Web UI through a browser, and the default port is 3000. For the monitoring node in this tutorial, you need to visit `192.168.110.23:3000`. The default username and password required for login are both set to `admin`. Upon the initial login, Grafana will prompt you to change the default login password. If you want to skip this for now, you can click `Skip`. Then, you will be re-directed to the Grafana Web UI homepage. ![MA-4](/assets/images/monitor4-769f8f814df66a7014c2baba78a300f0.png) ###### 1.2.4 Configure data sources[​](#124-configure-data-sources "Direct link to 1.2.4 Configure data sources") Click on the menu button in the upper-left corner, expand **Administration**, and then click **Data sources**. ![MA-5](/assets/images/monitor5-ce53b5a566261841ef338f2dbd65e148.png) On the page that appears, click **Add data source**, and then choose **Prometheus**. ![MA-6](/assets/images/monitor6-12deb6c827e59ca679b17ea3823564e5.png) ![MA-7](/assets/images/monitor7-c79999ed4ef04037c3c97153a52b7f6c.png) To integrate Grafana with your Prometheus service, you need to modify the following configuration: * **Name**: The name of the data source. You can customize the name for the data source. ![MA-8](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAmwAAABoCAYAAABMm6waAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAABquSURBVHhe7d3/U1XnnQfw/glLtt1EK2C8UVtDivFLwKBGNmwhrKQkGiImBA1QFDQQQoAgguGqCGQSghZmDaQWthaiQeMAzTBki6bFYcMP5of1p+zMznZmZ2faaTc72yRtmnz283nOc+59zuUA96K5HK7vH15z7jnPc8659/LIefs85zl8K+6uuwkAAAAAvAuBDQAAAMDjENgAAAAAPA6BDQAAAMDjENgAAAAAPM4zge17319PL1S84tj29J79tHPnU45tR46cVHXNbRB9j+zIpEOHax1KSipo5b1rFXkdWp6S+ojrsQAAAGBunglsO9KzaHr6E8e2t9++oJjbpI7UNbd5QnYrDX0wRT2HU9zLY8xhDmDHj7+hlrb29m5avz5FkddmmV3X7ViLYWvpG9ReXeBaJnZXd5O/NMO1LGryGqm9uYq2upUBAMAd5c4IbGmV1HFxiianPlH7T09O0dCJMvK51XVTP6r2G6gPbqsblGONUp1dp7CfJrjO8Kns4H4xzA5i5rbQwGaWudWfXQYdaO5Wx7C1HGum4uxNLnUXBoENAACWEk8EtvgEH/X2XlChyBwWvT2BrYA6xiRc3aChMx3U2NBBXRdvqOOMvTn7BdshnMB2h4lGYLMD04r7d1BWQSP5209Tdd7tCW0IbAAAsJR4IrBVVNTTyMiv1VKC0abNW9X20MAm20ND3bzyrJ6vyd5KY3sRdX3AgevDQSpR6z7KaRilsUkJYWxinLpKH1V1rWBmGBymAXOdqSCnzzNxxgoB1n58nNfHaUL37I2dO0ppgffwKJWcnQocY+zsIA0b+8cl5FHj+WD55NgoNeb49L6LL5qBzbZp/0lqb6mnncv1tuU7aM9L7dTSavfC1VPuA8H6y3aUUe1x3UvX+gbVFuXQCl2mAltNGeVWtAfL9z9Gy3S5CmyHXqDKY/b+7VSZa4ZFH2UUNFOTPnf7sWZ6dkeiUb6JskqbyW+XH2+mfUa5dfwqqpb3Z4eyB/KD5+P6Bw7x5zUC21yfBwAAYpuHAtuHKohJOHELbNILJ2XvvPM+Xb16g4qKX3AcY1YbWmlIQs/kOPl3rXOtE3/4Mk1ynYnzXaoHrmfkJp/rGrVl302+1GxKPzGuzj10gl+nptHmjGzyX7ICmZ9fb17Dx3ENbBy0RgbJHzjmTeo7bJ0zs/2aS3lw/8Kz0gt4gwaaiiinuIuGJExODFKhfs+LreTHFVZwCGEGtlCyj9uxZnIPbHFrS+hI+2kqz7LWd1ac5qBURRmrOQgtT6U91bxet59Wq/pPUWVLNzUdzKHVyxNp9Q+rqIn3rcy1QpMKbLxeW/AYrU5MotRnmqml/STt22gdWwJVe3s7ledspRWJ6ynroAS7Ztqz1ipPekbC40kq/mESLVueRBmq/CQ9qwPjg4VS3kh7NkvI5nAn5cdrKEO9t+DxK3c/Rg8m8zHu2kr7Gjh01pXQjtU+Wr3lKSqX8BYIbHN/HgAAiG2eGRKVYCaBRcKbvd0MbPY9blJXQpvMFrXrzSetflQFMtl/euIaDZzx0y4JWao8hfxXeHugt43p8DX2Zp61Hs6Q6Cw9bI0Jurx4UL0H65jZ1DbC5ZOXqcouT7CCpWN/ozyt0M9hsoZ2Jen6i0xmgtrhzLSMw4RwK5N93I410yyB7a4CqubgV50nrzm0JKdSkoQ1u1yGENsbaXegLoe7bLs8iXY8uZ9ydySpdRXY6kt0uLPr87F3W+uqB+zgY7pMPEHlJ7vpyP5Ufm29v6Yi6z8Wlq1U3Bh8zyvuT6UH7zd6RLdXkZ8DWvHD1rocv+XQEzPLU/Q6W/Z0sxHY5v48AAAQ25bUpAMJalJHetginniwpoDqzozS8IQEKTY1RR35ckGtoT57KDREYHhywYFttnK/Nax6pYOS7XK9zd4/uVaHzKkbNHxxkNoOF4Q/SSIKJHwlcwgLZQc2t7LbG9hYYoZjSNRiB7Yk2nlIetE4WNXV04H8J2hTon0ct3vYnMdWgc1x/kTaU2Nv0+FJ9/TZdr54OnjM5am00xwSVd6gA9utujOOL2FThnvtdXtbILDN/XkAACC2eTawSU+aDJMKeW1vlzqhz2aLlK+wn8YkDI10Ubodnsb6qTAjm9INacn6vIsQ2IQvs5IazZDJ7zczUH9xLf6QaBKHHmtINMvuyXL0sFlWbH6Cdu+voVoZXmw5Sft0YPomAtvul/gc+pjq+DIkukWGO7lc9aDdSmCzzPZ5AAAgtnk2sHV2/lRNRBDy2t4udSLpXUvX94oN1BvDU3zBVZMOJi9TuT08KcOXG4L7paVnU7xd/7YHtvmGRMuo7fwoDZwqs8ru8lFJr9zjNkUdubr+Ilv8SQdWnUBvG1NDiHZg42Cz5+knaJMuCx2yDCuw3cKQ6IxAtmOewDbfkOg8nwcAAGKbZwObDHvK/WxCXtvbIw1scWl+GpAhzyn7sR6tHIasx3pMnverHq7A8OMHo9TG5f6z13hdJgjov6jA5VJ/op/3L85V2+rOSyC7QX0nWqkkk+tEFNg4EDbPNekghfe3AlpfbRGlP+unPnk0iRnwFlk0A5s81mNn6UlqaTcf65Fq3aRfX0Y77l9P3/v7EqrlQBUIbCllqjeutiBDzaRcsZnLWzhwFUrgCi+wzT/pQGaG6kkHJfL+gpMO1ISI5hrauYXfm0wg4HA155DojEkHBVRpziCd5/MAAEBs82xgk0kFsi7MCQayHlFgY/HpNc4H58p9YedaKScQfkIe6zE1RUOvVwYfwZFQRB3v67LzR61j5vfSmD7eQDXXiTCwzXisx0UJiUb5mgLHYz3kUSMdhe6zXBdDNAKbHMPm+uDc0MdgFDVaockOVdlVjsdgHCnLpyS9b1g9bCGP9Sh3nN9HGUXGPWryWI8Moxd3+WO0r47PofetLqhyzHCdGdjYPI/1mOvzAABAbFtSkw6E1Ik0sHnTo5SeblzgHbNIzXreJOHLf/wNx98KlSBhBjazTOqGH9gAAADA5JnAJn/Q3Xykh5DJBaHhLFb++Pv3G8ZpevIa9ZxopcaGLhpQEwusZ7+51fca+UPudq+ZTSYVyExQIa9DyyXIuR0LAAAA5uaZwHbHkb9kcO5a4K8gmH9dAQAAAMCEwAYAAADgcQhsAAAAAB6HwAYAAADgcQhsAAAAAB6HwAYAAADgcQhsAAAAAB6HwAYAAADgcbcc2MofWUW/b3yQqGUj20R00unrE7xkXx8Xm+lrPy/9stSatVctXx0TDwU1aY3aUctf1TKF/trArxt4eUSrd/ryFV6+kkpf1mm1IWq0l8UW+ks1L6tlqb2kVWkviocD/lypVWgvmNLoz4d5eTiNvjjEr9kX5aat9EUZL8t4edDy+QFeMlkqpdqPxTb6vMTps2KtSGynz57n5fOy1PZb/rRvO/12bxqVJq9x/TkCAACAd91yYPtd43oV1r62ndROCAlpvFRhTYLaJvpKBTZeNmuvCg5pslRhTWuy/LXRoIKaXqqgxo5YvpRlPS8VCWq85LD2ZZ2hNoX+wtSyRqTSX17mJYc1hcNawEuWP1eJLdaSw1pApVZh+UKWHNK+eIFfy5KDmnLIUP4wfc4krH2ughovDxoOGErFVvrsx7zksPaZKDEUa0VCBzYOa3+y7df2GQq303/uedj15wgAAADedcuBTfWunZLeNbuHjZfSs2b0sFnBTQvpYftK97BJaAv2sGmqd00vdQ+b1bNmLVXP2lw9bBLYbCq0pbr3rqngxkvdwzajd83oYZOwJr1samn3rs3WwyY9a0YPm2L2sOnQZgc3u4dtRu+ao4eNl9KzJj1sdu9aoIeNBXrYeCk9a0YPm+ply09z/TkCAACAd936kOj2e+l3HNqkd031tM3Vw8bMHjYrqNk9bJpLD9tXgV42HdRcetgsHMoCPWxmUDMYPWx2UHNw9LBxMKvipeph02bpYbM8rHrXVE+bWw+bCmy8NMKao4ct0MsmIY2XgaC2VfWwqaWjh01CGi/tsMakd031tDl62CSsbVNhDUOiAAAAS8+37lv7AAEAAACAd6nAds93EwEAAADAoxDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ3uOM9997v0aUIcUeLfEMkyQZYAGreL/+N2cTL+Htf2AwCwGBDY4I4iYe1rt4s0wAxxdDr+btd2BAAQbVEJbCsS76N771tHq1bfDxA10uak7Zlt8U/oUYMIfM3tZcPmNIAlZ/3GLfx7cJ3j91+sXovdfteLlavW0rqkjZT0wOaYEJXAhrAGi0XantkWzYvxG/d8m9b+7d0UdxdAUM53vuNoJ251ALxu2fJEFdrM33+xfC0O/V0vJKzdsyzB9ftZiqIS2Ny+XIBoMduidd+aZSXCGswieG9jnGs5wFIgPW3m7z+334+xxPysQnql3L6XpQqBDWKe2RbtsCbc/kEACLQTiAUIbAhsEXP7YgGixWyLuBBDONBOIBYgsCGwRcztiwWIFrMt4kIM4UA7gViAwIbAFjG3LxYgWsy2iAsxhAPtBGIBAhsCW8TcvliAaDHbIi7EEA60E4gFCGwIbBFz+2IBosVsi7gQQzjQTiAWILDdWmC7NzmL0gr/iXJe/ZjyOv9AT5/53/Cc/pTr/5GeevP3tOv1/6Yn239LT7T+B+W2fEJZr/yaUp/ppJU/+KHrOeeCwAYxz2yLuBBDONBOIBYgsC08sElQcw1j81Fh7Q8qqP2o5d/pH5tuUGbdrymj+l8UeS3bpEyCm9u5Z4PABp61fUcW5T9TTC/XvqrIa9nmVncuZluM3Quxnwamb1BXnlvZEpGQR+UNNZST4FIWZbHbTuBOgsC2sMCWUTniHsbmY4e11/6Ldjb/mwpoP8g4QL77Uyhx5SpFXss2KZM66WUXXd+DG28EtucH6er0J3T9F8fogcD2g/TWrz6h6Xdfc9aFmPf9pA207/lyerPzbTpw8CXa9dRziryWbVLmtt9szLZ42y7E9aM0PT1KdW5liyIGAltuL41N36S+Mmu9bpD//Q/6Z9aLgtvWTgAWUcSBTV+Lp23XP6ILJ4qN6/Ickiup48rH1n5XOinVrU4Ymt7l/X81SM+p9dfoAh/vQvPMem7MzyoWEtgW3LPGZBhUetYkiG0t7rOCWuJKV1ImdaRuuD1tngps09Mf07mKh/R2BLY71f6iQ9TQ2EopW9JnlMk2KZM6oWWzMdvibbsQI7B94xDYAG7NQgPbe6/l0+NPFlNF5wRd5/XRzufc65tqh7nuTQ5XxfR45lb3OmFYzMAm96y5BbGwnP5U3bMmQ53SezZXWLNJHakr+4RzT5vHAhsb79c/qJDAlvIy/cRO7+zqlbepOFnqWT/Qq++O0fiULvt5C1V3/kY1tOmpj/mHvStwrqyqQXrvmlVvenyY6rOt7eANMuQpvWhmWPuHzB8p9rqUSZ1wh0fNthjJhTg+v4uGJnRbmbpBQ6+XkY+3qyAh27SBeqt+Wmm/s357EcXrY+06c4OmL/ZTx/tSbgU9Oc5Ebz8NyD4f9NMu3haffpR6RriuPsbw2aOUpo+h3lNOKw2MGec4U6nLnYEtPr9f9VYNNGRb7yGhiNou6eOyiUu9VLgmeFxXef00MT1OXe3jNKH/bU1c6qK9/B77Au9hivpefDS4z5oy8l+cCpxn7GIH7Q0McVrvsaepN/g9jV2mqjRdrs4nn0Hq6XLFDsY+Sq8epGF734lr1GOc2+07tssiFUk7AfCqhQa2YEB6iCp+xtfd63ytVNfbWa6hIT1zV88epFXJ+VT/84/ouv7dYV5vnzsr1/IxatLnNUNa4LXbMXX92ZifVUQa2G6pd+30/6jeNbk/TYY83QKaG6kr+4TTy+apwHbhbD+N89JK887AlnV0kC68M0hNT22lDWVDqv5oZz6XWYFtenyIqgvyqaLnI/XDvf5uJ+Xz/xDaLt1Uja1azpPdTaNcNt5zhP/3cITeGuf9rnTSNvO9wKLa+2yJGvoM3SbMbVIndNtszLYY/oW4hvomOXB0l1Fygo+Sn7cCUN+LPqs8tIcto4uGJSA15apQ5yuU+hxOCq1yFSa4fKi9knIythnBjwNPbRGlb0+h+IRK6uEwMtHvp0wOU75MDmf8HiZ6K63QtYGDTOA9cfmuDhri9aGmFD6HEdjSdL0zwcBYfo7/HYz1U2Gyj+KTy6iLA9dk/9FAuSsVoPg4vTWUk7yOkvN7+TPye+aQ1nOYP+eabVTSzeFs8jKVq30eJf8VPu4Ih0HjPNNXOoxQyetjg1SeuY58qfx5P5D3UWOcLxg6Q3vY4osHVXlfrXzH6yizaZQm5Tsutn4mbt+xvW+kwm8nAN5164GNNY3xv6uP6a3n+fVc19Bmo56sl3bT+XeGqe1QJm3I7aT3rlu3PUlZWIFNlUW3h01mg7qFsXDIcKjMBpVJBeH0rtmkruwjs0fd3pPJW4Gt2U7zE3Qie64hUavMSty6h81O36ENTjUiq2E83vkbfs3H1v9T2NYm68FGA4vv5dpmdb+auW39xi2KuU3qSF1z22zMthj+hVjCxU3qKdUB7a4U2lvdSlX5Eo54PTSwcXhJd4SEAuriMDL2Zp5aV2GCg0tyoNwKJJMcxuz1eDkmh58q46b7+Bcvcyjh8/C25OZrNP3hIJXoMpFZP0gDp8r4tQ5s+UVWGOOgY/bM2eeyA1py/lFqrC5yvJ8ZdA9bY+D9+KjxIv+bvNgaDHpmyFKvp6gj267PsuW+NDuE6e+0OFiuvpexXsoJPRavOwObz/oMjpDpo6p+DqJcR7a5fccLFX47AfCu2xLY7G1H57mGhga2ECqI6eu5VwNbRI/uCCH7yqM7ZIhTTTBwCWeuuK7sI4/8cHtPJo8FNn6dfIzOqyT+9pxDosEu0vADm9VIgvtbENi8xC2wufWwffOBLYXKe6XHhkPXpcvU1VSper0C5TPuYfNRepkxJKpNnClQ5SpMhNyPJQHELg/UCQ0cqufOCjGq3AxLDlZgG77CdabGyZ/uLE8+PGgNa46NU9+ZVirJXOcod6UClHNo0Rmi7Do6ZMl3EhIo7Z5Ka9hYh0odyIT6THo4eO7AZgXg4WYdmLX0dg6xen+373ihwm8nAN51u3vY5ryGhga20CFRgcA205IObLyedcK60dH8AR/8Kf+ApectVyYlLKyHLb9bhkv5tf7fAXiPBDNvDIlafJmVVNc+SEMytDd5jdrssBEa2FTYkCHRPDVcaQeMWw5sauZkuIFNhkHLVK9TaA+bsiaXSmq7qOeS3GN2k4ZPBc/t6rYENut9fVOBLedN/iwIbACubvc9bHNeQ0MC27aWCbV+rsyagLAUetgwJMrcvliHGY3kOer4Jf/QeJv9A67+hdyLNkFtBfmUXzuk7nW72lPJZeEHtlVPv22Nv//sGOU/mU/Fr03Q1Uud9Lg6J3iB26QD2WZOMIjKpAMOao0NlZQZ2JarbmYPBKzQwCbrdvBQiiIObAsZEk0r9FNjcS6/NsKQuteNw2O9fUM+B7UG6VUL7pf5ejDo2NtmiDSwqdfzDYkuNLCFOSSKwAYQsNDAFjpLdLxbX1/nuoaGBDZr+FSe/MDHKu1W97BNc115RMgDuu65WqNsjsA22llMWY/YT5CYnflZBSYdLIDbF+swI7AFtwWGRLNb6Lzc4CjbxifoPRkuVT/8CAIbc8xwuTZBb1UFZ5CCN6jHehw95QhtNtkmZd/4Yz2yrUkEQycKrEkEmdYN/sOnsq3yfB1OclMoOdlHcWUSrG5Qz+E82pyaR+VnrZmSkQS2OD3pQG7ytyYd+KlPJiFEOumA66Y1jNPk5Dg1qhmYudQ2wmHnUgftkmHdNbnkv8hBZ6SL0u1zu4k0sNmTDvg8e/Wkg44RPo9j0kH4gW1vN5e930s5yfwd8+e1Jh3oCQ8y6aD2sqrvmHSAwAYQsNDApq6PwuU5bLNeQ0MCm+O5bFzvvV/y7wIOYsV22TCv67IL8to1sGVS0ztWvfHu4sB7mI35WUWkgQ2P9WBuXyzAXCSQSS9a6cGX6MndBYq8lm2RhDVhtsVILsShj+lwPmKjgNquWL9IrIkFPtprPP5i7HwHdUl40ZMKwgpsTB7r0Temf5Hpc6abPW6Ox3pMcaC0HjUyMwxZ4SkQltIqqct8rMfIYPBxGrOJOLCxNRzSjPPIYz2Cjw+JLLDF5XFo/lCOI8FYtsljPS7TGIdUdXx5rEe1fmyJfSwENoCAiAPbEmd+VhFpYBN4cK7LFwswHxnylPvUZHKBkNfhDoOazLaIC7GTCmA6XDncpuCzVKGdQCxAYIs8sAn8aSqARWK2RVyIIRxoJxALENgWFtgE/vg7wCIw2yIuxBAOtBOIBQhsCw9sQu5pk+Ams0cjeuSHCm1/VPe0SXCT2aPyyA95dIfMBpWgFs49a6EQ2CDmmW0RF2IIB9oJxAIEtlsLbF6DwAYxz2yLuBBDONBOIBYgsCGwRcztiwWIFrMt4kIM4UA7gViAwIbAFjG3LxYgWsy2iAsxhAPtBGIBAhsCW8TcvliAaDHbIi7EEA60E4gFCGwIbBFz+2IBosVsi7gQQzjQTiAWILAhsEXs3vvWuX65AN80aXtmW1QX4URciGFuaCew1C1bvpLWb9zi+P0Xy9fi0N/1Yl3SRrpnWYLr97MURSWwrUi8D6ENok7anLQ9sy2qC7G2c8ND6n+gAKbUTQ872olbHQCvk7C2arUzxMTqtdjtd71YuWqtCm3S0xYLohLYALziK+NCTAlxxmsAm7NduLUjAIBoQ2CDO8o/x/+d42IMMJfr8d9xbUcAANGGwAZ3nEsrzNCGXjZwE0f/Gv9t1/YDALAYENgAAAAAPA6BDQAAAMDjENgAAAAAPA6BDQAAAMDjENgAAAAAPA6BDQAAAMDjENgAAAAAPA6BDQAAAMDjENgAAAAAPA6BDQAAAMDjENgAAAAAPA6BDQAAAMDjENgAAAAAPC2R/h/myl9s2S1DBgAAAABJRU5ErkJggg==) * **Prometheus Server URL**: The URL of the Prometheus server, which, in this tutorial, is `http://192.168.110.23:9090`. ![MA-9](/assets/images/monitor9-0b3e34d7be2872123104e38932fcc362.png) After the configuration is complete, click **Save & Test** to save and test the configuration. If **Successfully queried the Prometheus API** is displayed, it means the data source is accessible. ![MA-10](/assets/images/monitor10-2d17d7d8ca6a8d25231e381d60df8f39.png) ###### 1.2.5 Configure Dashboard[​](#125-configure-dashboard "Direct link to 1.2.5 Configure Dashboard") 1. Download the corresponding Dashboard template based on your StarRocks version. * [Dashboard template for All Architecture](https://releases.starrocks.io/resources/Dashboard-All-Arch-20260113.json) * [Dashboard template for Shared-data Cluster - General](https://releases.starrocks.io/resources/Dashboard-Shared-data-General-3.5.json) * [Dashboard template for Shared-data Cluster - Starlet](https://releases.starrocks.io/resources/Dashboard-Shared-data-Starlet-3.5.json) > **NOTE** > > The template file needs to be uploaded through the Grafana Web UI. Therefore, you need to download the template file to the machine you use to access Grafana, not the monitoring node itself. 2. Configure the Dashboard template. Click on the menu button in the upper-left corner and click **Dashboards**. ![MA-11](/assets/images/monitor11-e18189f47fceec908b80c7d127a51446.png) On the page that appears, expand the **New** button and click **Import**. ![MA-12](/assets/images/monitor12-40d0bdaad5a6097aaa9edbe3e19c2093.png) On the new page, click on **Upload Dashboard JSON file** and upload the template file you downloaded earlier. ![MA-13](/assets/images/monitor13-200a76c2389b70d7ad8799f3db961757.png) After uploading the file, you can rename the Dashboard. By default, it is named `StarRocks Overview`. Then, select the data source, which is the one you created earlier (`starrocks_monitor`). Then, click **Import**. ![MA-14](/assets/images/monitor14-d5df752b2e5dd0b98b25014c1dbf2f21.png) After the import is complete, you should see the StarRocks Dashboard displayed. ![MA-15](/assets/images/monitor15-210b53b9e827c45dffd4e1c1c37e415c.png) ###### 1.2.6 Monitor StarRocks via Grafana[​](#126-monitor-starrocks-via-grafana "Direct link to 1.2.6 Monitor StarRocks via Grafana") Log in to the Grafana Web UI, click on the menu button in the upper-left corner, and click **Dashboards**. ![MA-16](/assets/images/monitor16-5a986d52c1c7f54049d8d545a6a2f4a6.png) On the page that appears, select **StarRocks Overview** from the **General** directory. ![MA-17](/assets/images/monitor17-bda14636790d5f7f78a7f5565e6f771e.png) After you enter the StarRocks monitoring Dashboard, you can manually refresh the page in the upper-right corner or set the automatic refresh interval for monitoring the StarRocks cluster status. ![MA-18](/assets/images/monitor18-9a6f4f7963351ba69d2147ee5a54d88d.png) #### Step 2: Understand the core monitoring metrics[​](#step-2-understand-the-core-monitoring-metrics "Direct link to Step 2: Understand the core monitoring metrics") To accommodate the needs of development, operations, DBA, and more, StarRocks provides a wide range of monitoring metrics. This section only introduces some important metrics commonly used in business and their alert rules. For other metric details, please refer to [Monitoring Metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics.md). ##### 2.1 Metrics for FE and BE status[​](#21-metrics-for-fe-and-be-status "Direct link to 2.1 Metrics for FE and BE status") | **Metric** | **Description** | **Alert rule** | **Note** | | ---------------- | --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | Frontends Status | FE Node Status. The status of a live node is represented by `1`, while a node that is down (DEAD) will be displayed as `0`. | The status of all FE nodes should be alive, and any FE node with a status of DEAD should trigger an alert. | The failure of any FE or BE nodes is considered critical, and it requires prompt troubleshooting to identify the cause of failure. | | Backends Status | BE Node Status. The status of a live node is represented by `1`, while a node that is down (DEAD) will be displayed as `0`. | The status of all BE nodes should be alive, and any BE node with a status of DEAD should trigger an alert. | | ##### 2.2 Metrics for query failure[​](#22-metrics-for-query-failure "Direct link to 2.2 Metrics for query failure") | **Metric** | **Description** | **Alert rule** | **Note** | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Query Error | The query failure (including timeout) rate within one minute. Its value is calculated as the number of failed queries in one minute divided by 60 seconds. | You can configure this based on the actual QPS of your business. 0.05, for example, can be used as a preliminary setting. You can adjust it later as needed. | Usually, the query failure rate should be kept low. Setting this threshold to 0.05 means allowing a maximum of 3 failed queries per minute. If you receive the alert from this item, you can check resource utilization or configure the query timeout appropriately. | ##### 2.3 Metrics for external operation failure[​](#23-metrics-for-external-operation-failure "Direct link to 2.3 Metrics for external operation failure") | **Metric** | **Description** | **Alert rule** | **Note** | | ------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Schema Change | The Schema Change operation failure rate. | Schema Change is a low-frequency operation. You can set this item to send an alert immediately upon failure. | Usually, Schema Change operations should not fail. If an alert is triggered for this item, you can consider increasing the memory limit of Schema Change operations, which is set to 2GB by default. | ##### 2.4 Metrics for internal operation failure[​](#24-metrics-for-internal-operation-failure "Direct link to 2.4 Metrics for internal operation failure") | **Metric** | **Description** | **Alert rule** | **Note** | | ------------------- | -------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | BE Compaction Score | The highest Compaction Score among all BE nodes, indicating the current compaction pressure. | In typical offline scenarios, this value is usually lower than 100. However, when there are a large number of loading tasks, the Compaction Score may increase significantly. In most cases, intervention is required when this value exceeds 800. | Usually, if the Compaction Score is greater than 1000, StarRocks will return an error "Too many versions". In such cases, you may consider reducing the loading concurrency and frequency. | | Clone | The tablet clone operation failure rate. | You can set this item to send an alert immediately upon failure. | If an alert is triggered for this item, you can check the status of BE nodes, disk status, and network status. | ##### 2.5 Metrics for service availability[​](#25-metrics-for-service-availability "Direct link to 2.5 Metrics for service availability") | **Metric** | **Description** | **Alert rule** | **Note** | | -------------- | ------------------------------------------------------ | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Meta Log Count | The number of BDB metadata log entries on the FE node. | It is recommended to configure this item to trigger an immediate alert if it exceeds 100,000. | By default, the leader FE node triggers a checkpoint to flush the log to disk when the number of logs exceeds 50,000. If this value exceeds 50,000 by a large margin, it usually indicates a checkpoint failure. You can check whether the Xmx heap memory configuration is reasonable in **fe.conf**. | ##### 2.6 Metrics for system load[​](#26-metrics-for-system-load "Direct link to 2.6 Metrics for system load") | **Metric** | **Description** | **Alert rule** | **Note** | | -------------------- | --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | BE CPU Idle | CPU idle rate of the BE node. | It is recommended to configure this item to trigger an alert if the idle rate is lower than 10% for 30 consecutive seconds. | This item is used to monitor CPU resource bottlenecks. CPU usage can fluctuate significantly, and setting a small polling interval may result in false alerts. Therefore, you need to adjust this item based on the actual business conditions. If you have multiple batch processing tasks or a large number of queries, you may consider setting a lower threshold. | | BE Mem | Memory usage for the BE node. | It is recommended to configure this item to 90% of the available memory size for each BE. | This value is equivalent to the value of Process Mem, and BE's default memory limit is 90% of the server's memory size (controlled by configuration `mem_limit` in **be.conf**). If you have deployed other services on the same server, be sure to adjust this value to avoid OOM. The alert threshold for this item should be set to 90% of BE's actual memory limit so that you can confirm whether BE memory resources have reached a bottleneck. | | Disks Avail Capacity | Available disk space ratio (percentage) of the local disks on each BE node. | It is recommended to configure this item to trigger an alert if the value is less than 20%. | It is recommended to reserve sufficient available space for StarRocks based on your business requirements. | | FE JVM Heap Stat | JVM heap memory usage percentage for each FE node in the cluster. | It is recommended to configure this item to trigger an alert if the value is greater than or equal to 80%. | If an alert is triggered for this item, it is recommended to increase the Xmx heap memory configuration in **fe.conf**; otherwise, it may affect query efficiency or lead to FE OOM issues. | #### Step 3: Configure alert via Email[​](#step-3-configure-alert-via-email "Direct link to Step 3: Configure alert via Email") ##### 3.1 Configure SMTP service[​](#31-configure-smtp-service "Direct link to 3.1 Configure SMTP service") Grafana supports various alerting solutions, such as email and webhooks. This tutorial uses email as an example. To enable email alerting, you first need to configure SMTP information in Grafana, allowing Grafana to send emails to your mailbox. Most commonly used email providers support SMTP services, and you need to enable SMTP service for your email account and obtain an authorization code. After completing these steps, modify the Grafana configuration file on the node where Grafana is deployed. ```bash vim /usr/share/grafana/conf/defaults.ini ``` Example: ```properties ###################### SMTP / Emailing ##################### [smtp] enabled = true host = user = johndoe@gmail.com # If the password contains # or ; you have to wrap it with triple quotes.Ex """#password;""" password = ABCDEFGHIJKLMNOP # The authorization password obtained after enabling SMTP. cert_file = key_file = skip_verify = true ## Verify SSL for SMTP server from_address = johndoe@gmail.com ## Address used when sending out emails. from_name = Grafana ehlo_identity = startTLS_policy = [emails] welcome_email_on_sign_up = false templates_pattern = emails/*.html, emails/*.txt content_types = text/html ``` You need to modify the following configuration items: * `enabled`: Whether to allow Grafana to send email alerts. Set this item to `true`. * `host`: The SMTP server address and port for your email, separated by a colon (`:`). Example: `smtp.gmail.com:465`. * `user`: SMTP username. * `password`: The authorization password obtained after enabling SMTP. * `skip_verify`: Whether to skip SSL verification for the SMTP server. Set this item to `true`. * `from_address`: The email address used to send alert emails. After the configuration is complete, restart Grafana. ```bash systemctl daemon-reload systemctl restart grafana-server.service ``` ##### 3.2 Create alert channel[​](#32-create-alert-channel "Direct link to 3.2 Create alert channel") You need to create an alert channel (Contact Point) in Grafana to specify how to notify contacts when an alert is triggered. 1. Log in to the Grafana Web UI, click on the menu button in the upper-left corner, expand **Alerting**, and select **Contact Points**. On the **Contact points** page, click **Add contact point** to create a new alert channel. ![MA-19](/assets/images/monitor19-6109fd0b12144f79f8f68466b5b47507.png) 2. In the **Name** field, customize the name of the contact point. Then, in the **Integration** dropdown list, select **Email**. ![MA-20](/assets/images/monitor20-8e6a36e70958a93278b465e375aea321.png) 3. In the **Addresses** field, enter the email addresses of the contacts to receive the alert. If there are multiple email addresses, separate the addresses using semicolons (`;`), commas (`,`), or line breaks. The configurations on the page can be left with their default values except for the following two items: * **Single email**: When enabled, if there are multiple contacts, the alert will be sent to them through a single email. It's recommended to enable this item. * **Disable resolved message**: By default, when the issue causing the alert is resolved, Grafana sends another notification notifying the service recovery. If you don't need this recovery notification, you can disable this item. It's not recommended to disable this option. 4. After the configuration is complete, click the **Test** button in the upper-right corner of the page. In the prompt that appears, click **Sent test notification**. If your SMTP service and address configuration are correct, the target email account should receive a test email with the subject "TestAlert Grafana". Once you confirm that you can receive the test alert email successfully, click the **Save contact point** button at the bottom of the page to complete the configuration. ![MA-21](/assets/images/monitor21-d97b1c38652ebb72514b2b5df4579fc6.png) ![MA-22](/assets/images/monitor22-4dca103ae0c8a75cfd54e43c39337abb.png) You can configure multiple notification methods for each contact point through "Add contact point integration", which will not be detailed here. For more details about Contact Points, you can refer to the [Grafana Documentation](https://grafana.com/docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/contact-points/) For subsequent demonstration, let's assume that in this step, you have created two contact points, "StarRocksDev" and "StarRocksOp", using different email addresses. ##### 3.3 Set notification policies[​](#33-set-notification-policies "Direct link to 3.3 Set notification policies") Grafana uses notification policies to associate contact points with alert rules. Notification policies use matching labels to provide a flexible way to route different alerts to different contacts, allowing for alert grouping during O\&M. 1. Log in to the Grafana Web UI, click on the menu button in the upper-left corner, expand **Alerting**, and select **Notification policies**. ![MA-23](/assets/images/monitor23-0a648b6966faea93ea94a1870594e82a.png) 2. On the **Notification policies** page, click the more (**...**) icon to the right of **Default policy** and click **Edit** to modify the Default policy. ![MA-24](/assets/images/monitor24-bf167112de0d0d807fa4e548e1d813fb.png) ![MA-25](/assets/images/monitor25-7f769039514cfe682b6360d88d18ca55.png) Notification policies use a tree-like structure, and the Default policy represents the default root policy for notification. When no other policies are set, all alert rules will default to matching this policy. It will then use the default contact point configured within it for notifications. 1. In the **Default contact point** field, select the contact point you created previously, for example, "StarRocksOp". 2. **Group by** is a key concept in Grafana Alerting, grouping alert instances with similar characteristics into a single funnel. This tutorial does not involve grouping, and you can use the default setting. ![MA-26](/assets/images/monitor26-f0f18c8a09e860caf8e1292f0a3df0ab.png) 3. Expand the **Timing options** field and configure **Group wait**, **Group interval**, and **Repeat interval**. * **Group wait**: The time waiting for the initial notification to send after the new alert creates a new group. Default 30 seconds. * **Group interval**: The interval at which alerts are sent for an existing group. Defaults to 5 minutes, which means that notifications will not be sent to this group any sooner than 5 minutes since the previous alert was sent. This means that notifications will not be sent any sooner than 5 minutes (default) since the last batch of updates were delivered, regardless of whether the alert rule interval for those alert instances was lower. Default 5 minutes. * **Repeat interval**: The waiting time to resend an alert after they have successfully been sent. The interval at which alerts are sent for an existing group. Defaults to 5 minutes, which means that notifications will not be sent to this group any sooner than 5 minutes since the previous alert was sent. You can configure the parameters as shown below so that Grafana will send the alert by these rules: 0 seconds (Group wait) after the **alert conditions are met**, Grafana will send the first alert email. After that, Grafana will re-send the alert every 1 minute (Group interval + Repeat interval). ![MA-27](/assets/images/monitor27-1555bd354163a5ba006ff518c875ff62.png) > **NOTE** > > The previous paragraph uses "meeting the alert conditions" rather than "reaching the alert threshold" to avoid false alerts. It's recommended to set the alert to be triggered a certain duration of time after the threshold has been reached. 3. After the configuration is complete, click **Update default policy**. 4. If you need to create a nested policy, click on **New nested policy** on the **Notification policies** page. Nested policies use labels to define matching rules. The labels defined in a nested policy can be used as conditions to match when configuring alert rules later. The following example configures a label as `Group=Development_team`. ![MA-28](/assets/images/monitor28-4ee3adad9577249db57b9f77193d8ea9.png) In the **Contact point** field, select "StarRocksDev". This way, when configuring alert rules with the label `Group=Development_team`, "StarRocksDev" is set to receive the alerts. You can have the nested policy inherit the timing options from the parent policy. After the configuration is complete, click **Save policy** to save the policy. ![MA-29](/assets/images/monitor29-757db2ff569cc44b57305940cc844a67.png) If you are interested in the details of notification policies or if your business has more complex alerting scenarios, you can refer to the [Grafana Documentation](https://grafana.com/docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/contact-points/) for more information. ##### 3.4 Define alert rules[​](#34-define-alert-rules "Direct link to 3.4 Define alert rules") After setting up notification policies, you also need to define alert rules for StarRocks. Log in to the Grafana Web UI, and search for and navigate to the previously configured StarRocks Overview Dashboard. ![MA-30](/assets/images/monitor30-a9c570096158e3748f7eaafb731ef8fe.png) ![MA-31](/assets/images/monitor31-39435dfb2795b8b967dbd7022ea11aee.jpeg) ###### 3.4.1 FE and BE status alert rule[​](#341-fe-and-be-status-alert-rule "Direct link to 3.4.1 FE and BE status alert rule") For a StarRocks cluster, the status of all FE and BE nodes must be alive. Any node with a status of DEAD should trigger an alert. The following example uses the Frontends Status and Backends Status metrics under StarRocks Overview to monitor FE and BE status. As you can configure multiple StarRocks clusters in Prometheus, note that the Frontends Status and Backends Status metrics are for all clusters that you have registered. ###### Configure the alert rule for FE[​](#configure-the-alert-rule-for-fe "Direct link to Configure the alert rule for FE") Follow these procedures to configure alerts for **Frontends Status**: 1. Click on the More (...) icon to the right of the **Frontends Status** monitoring item, and click **Edit**. ![MA-32](/assets/images/monitor32-ffeb20bf4d65aec43d33d6e55c77bf28.jpeg) 2. On the new page, choose **Alert**, then click **Create alert rule** from this panel to enter the rule creation page. ![MA-33](/assets/images/monitor33-2dd84ee6a971dcad5cf34fbc3289d1be.jpeg) 3. Set the rule name in the **Rule name** field. The default value is the title of the monitoring metric. If you have multiple clusters, you can add the cluster name as a prefix for differentiation, for example, "\[PROD]Frontends Status". ![MA-34](/assets/images/monitor34-462961937451b355d3dec8634360782e.jpeg) 4. Configure the alert rule as follows. 1. Choose **Grafana managed alert**. 2. For section **B**, modify the rule as `(up{group="fe"})`. 3. Click on the delete icon on the right of section **A** to remove section **A**. 4. For section **C**, modify the **Input** field to **B**. 5. For section **D**, modify the condition to `IS BELOW 1`. After completing these settings, the page will appear as shown below: ![MA-35](/assets/images/monitor35-9b9881f59ac229c3d6b340b22087cf62.jpeg) Click to view detailed instructions Configuring alert rules in Grafana typically involves three steps: 1. Retrieve the metric values from Prometheus through PromQL queries. PromQL is a data query DSL language developed by Prometheus, and it is also used in the JSON templates of Dashboards. The `expr` property of each monitoring item corresponds to the respective PromQL. You can click **Run queries** on the rule settings page to view the query results. 2. Apply functions and modes to process the result data from the above queries. Usually, you need to use the Last function to retrieve the latest value and use Strict mode to ensure that if the returned value is non-numeric data, it can be displayed as `NaN`. 3. Set rules for the processed query results. Taking FE as an example, if the FE node status is alive, the output result is `1`. If the FE node is down, the result is `0`. Therefore, you can set the rule to `IS BELOW 1`, meaning an alert will be triggered when this condition occurs. 5. Set up alert evaluation rules. According to the Grafana documentation, you need to configure the frequency for evaluating alert rules and the frequency at which their status changes. In simple terms, this involves configuring "how often to check with the alert rules" and "how long the abnormal state must persist after detection before triggering the alert (to avoid false alerts caused by transient spikes)". Each Evaluation group contains an independent evaluation interval to determine the frequency of checking the alert rules. You can create a new folder named **PROD** specifically for the StarRocks production cluster and create a new Evaluation group `01` within it. Then, configure this group to check every `10` seconds, and trigger the alert if the anomaly persists for `30` seconds. ![MA-36](/assets/images/monitor36-45dbde604bf936329236090fe7560a61.png) > **NOTE** > > The previously mentioned "Disable resolved message" option in the alert channel configuration section, which controls the timing of sending emails for cluster service recovery, is also influenced by the "Evaluate every" parameter above. In other words, when Grafana performs a new check and detects that the service has recovered, it sends an email to notify the contacts. 6. Add alert annotations. In the **Add details for your alert rule** section, click **Add annotation** to configure the content of the alert email. Please note not to modify the **Dashboard UID** and **Panel ID** fields. ![MA-37](/assets/images/monitor37-af96a30ba8b1a14ec11923250de8aa6d.jpeg) In the **Choose** drop-down list, select **Description**, and add the descriptive content for the alert email, for example, "FE node in your StarRocks production cluster failed, please check!" 7. Match notification policies. Specify the notification policy for the alert rule. By default, all alert rules match the Default policy. When the alert condition is met, Grafana will use the "StarRocksOp" contact point in the Default policy to send alert messages to the configured email group. ![MA-38](/assets/images/monitor38-249ddb362026eabbad3bdd3dba5f8f06.jpeg) If you want to use a nested policy, set the **Label** field to the corresponding nested policy, for example, `Group=Development_team`. Example: ![MA-39](/assets/images/monitor39-bcb6c9f6d736823e0dc4550a73901a1a.jpeg) When the alert condition is met, emails will be sent to "StarRocksDev" instead of "StarRocksOp" in the Default policy. 8. Once all configurations are complete, click **Save rule and exit**. ![MA-40](data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsKCwsNDhIQDQ4RDgsLEBYQERMUFRUVDA8XGBYUGBIUFRT/2wBDAQMEBAUEBQkFBQkUDQsNFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBT/wAARCACkAZcDAREAAhEBAxEB/8QAHQABAAIDAQEBAQAAAAAAAAAAAAMEBQYIBwECCf/EAEQQAAEDAgIHBAUIBwkBAAAAAAABAgMEBQYRBxIVVJGT0hQXIdMTGDFBYQgWIjJRVXOyMzQ1ZHJ0gTY3RXGDobGzwSP/xAAcAQEAAQUBAQAAAAAAAAAAAAAAAgMEBQYHAQj/xABKEQACAQMBAggJBgoLAQEAAAAAAQIDBBEhBTEGEhdBUWGR0xMVUlRVgZPR0hYiMnGSlDQ1NkVzdKGxssIHFCNCcnWCs8Hw8WLh/9oADAMBAAIRAxEAPwD+dmzKbeZeSnUXBTyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMkwPAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAb3hXQ3fMXWWG6Uk9FBTTK5GJUSPRy6qq1Vya1fei8DarDg5ebQoK4pSiovOMt8zxzJmIuNp0Lao6Uk210f+mX9XbEm+2nmyeWZH5HX/AJcO2Xwlt47t/Jl2L3j1dsSb7aebJ5Y+R1/5cO2Xwjx3b+TLsXvHq7Yk32082Tyx8jr/AMuHbL4R47t/Jl2L3j1dsSb7aebJ5Y+R1/5cO2Xwjx3b+TLsXvHq7Yk32082Tyx8jr/y4dsvhHju38mXYvePV2xJvtp5snlj5HX/AJcO2Xwjx3b+TLsXvHq7Yk32082Tyx8jr/y4dsvhHju38mXYvePV2xJvtp5snlj5HX/lw7ZfCPHdv5Muxe8ertiTfbTzZPLHyOv/AC4dsvhHju38mXYvePV2xJvtp5snlj5HX/lw7ZfCPHdv5Muxe8/EvyecSxxPe2qtkqtRVRjJpM3fBM2In+5CXBDaEU2pQfrfwnq21bN4w+xe880pqeSrqIoIm68srkYxueWaquSJ4mh1KkaUJVJvCSy/qRtNpa1r64p2lvHjVKklGK0WXJ4Sy8JZb53g3WLRHcZImvddLZEqoirG902afBco1T/cto7T2TJLN5Bf6avdHQuTLhn6OftKHen77oK/73tXGfyiXjLY/nsPs1u6HJnwz9Gy9pQ70d0Ff972rjP5Q8ZbH89h9mt3Q5M+Gfo2XtKHejugr/ve1cZ/KHjLY/nsPs1u6HJnwz9Gy9pQ70d0Ff8Ae9q4z+UPGWx/PYfZrd0OTPhn6Nl7Sh3o7oK/73tXGfyh4y2P57D7NbuhyZ8M/RsvaUO9HdBX/e9q4z+UPGWx/PYfZrd0OTPhn6Nl7Sh3o7oK/wC97Vxn8oeMtj+ew+zW7ocmfDP0bL2lDvR3QV/3vauM/lDxlsfz2H2a3dDkz4Z+jZe0od6O6Cv+97Vxn8oeMtj+ew+zW7ocmfDP0bL2lDvR3QV/3vauM/lDxlsfz2H2a3dDkz4Z+jZe0od6O6Cv+97Vxn8oeMtj+ew+zW7ocmfDP0bL2lDvR3QV/wB72rjP5Q8ZbH89h9mt3Q5M+Gfo2XtKHejugr/ve1cZ/KHjLY/nsPs1u6HJnwz9Gy9pQ70d0Ff972rjP5Q8ZbH89h9mt3Q5M+Gfo2XtKHejugr/AL3tXGfyh4y2P57D7NbuhyZ8M/RsvaUO9HdBX/e9q4z+UPGWx/PYfZrd0OTPhn6Nl7Sh3o7oK/73tXGfyh4y2P57D7NbuhyZ8M/RsvaUO9Hc9c3ZJFcrZNIq5MiY6bNyr7ETONETiVqF5s26rQt7e7hKc2opcWqstvCWXTS39JZXv9H3CvZ1rVvbuwcaVKMpyfHovEYptvCqtvCWcJN9CNDKpoBlvm3Ong6eBrve1Vd4cGgYHzcm3in4v6Qe4Hzcm3in4v6QMD5uTbxT8X9IGB83Jt4p+L+kDA+bk28U/F/SBgfNybeKfi/pAwPm5NvFPxf0gYHzcm3in4v6QMENVZJqSnfMssUjWZayMVc08cveifaDzBjwAAAAAAAAAAAdR6Fv7s7N/rf9zzunBr8VUf8AV/Ezn+1fwyfq/cjdzZzElmgttXdaj0FFSz1k+Su9FTxq92Se1ckTMpzqQprjTaS69CUYuTxFZZBJG6J7mParHtVUc1yZKi/YpNNNZR4008M/J6eAAAAAmpqSetl9HTwyTyZK7UiYrlyTxVckISlGCzJ4R6k5NJbyEmeAAAHGuHf7QWz+ai/Oh8q7Q/A63+GX7mfQPBL8otnfpqX8cTr+0fsmi/AZ+VD6H4N/iSx/RU/4Ec04cflVtX9Yrf7ki2bGaSW4bVW1FDNWRUdRLRwrqyVDInLGxfsc5EyT+pSlUhGShKSTe5Z1JqEpJtLcVCqQAAAAAAJkpJ3UzqlIZFp2uRjpkauojl9iKvsz+BDjRUuLnUkk2m0txCTIgAAAAmqqSeilWKohkp5URF1JWK12S+KLkpCMozWYvJJpreiEmRABZrrbV2uVkdZSzUkj2NlayeNWK5i+LXIip4ovuUpwqQqZcGnjTTp6CTi0k2t+q610lYqETE4g/wAN/nYv/TnPDP8ANn61R/mO2f0Yfn7/AC+5/kOOzlRiTdKj9Yl/iX/kEiMAs0tsrK6GeampJ6iKButLJFG5zY0+1yongn+Yeiy9wWrwisAAAAACRtNM+B87YnuhYqNdIjV1WqvsRV9wem8b9CMAr3L9l1f8LfztAZq4IgAAAAAAAAAA6j0Lf3Z2b/W/7nndODX4qo/6v4mc/wBq/hk/V+5G7mzmJPdNC2N6XDOje9W+rr7vgta+vY6LF1uoXTx6zWfq0ipk77XZNXPxXPw9uobXs53F3SqQjGrxU805PGcv6S5urXQzFhWjShVUsx42Fx0s46n9fabXddH81/0jXe6Y2pLZiOlhs9LVRXaGtS1UE7JFVsU9S9fpo9yNVMmoqqqfYYqlext7ONKycqcnOScWuPJNauMVuwusvXQlXrwnWfhI8TKkvm5XM5Pm6+fcQ4q0F4Wlr8RWWw0q7ansNLf7QkVW+eNPpKk8LHLl6RqoiK1zkz8Sdtti7Uada4l8yNR055STw0uK2tcNPfgjUtLfOEvp03KOHlKUW8pPn4yWmfUWaPQngWDEWLUmippaPClHRUc8dwur6Snqa6RF9LJJN4qxqL9FGtyTNMinLa99KhRlFvNaU2sR4zUI7klplvfrzEoWNCNV0qm+nCLlzJyl18yXUeRacMKYbwtimjbhitpqigq6KOpkp6WsSrZSyrmjo2y5Irm+CKir45KbPse5ubmlL+txacZNJtcVtczxzPmeDHX1GjS4kqTXzlqk84f19D957DNow0dv0nU+C4MLztkbZ1uc1e65TLrv7Kr0jazPwTW1XZ5555plkaq9o7RVjUvnWWkuKlxV5SWc/Vpu695koWtsq1pQlDPhEm3l+TLTHW1n9h+vk64MtdBhvDWJYqZzLvXsvFPNOsjlR8bIF1URqrkmS5+KIR4QXdadSvat/MioNLrckQ2TRi5Ua7XzvC49Xg5P958boN0eWixW213qutlLcayztr5LxUX30VVHM+NXt1KVU1XRJllmq5qmf2E57Y2jUq1KlvGTjCfFUVDKaTxrLemyVtZ2zo0XWa/tIptuWGs7sLnxz56zllUyVUzzy96HRzW2sPB8B4ca4d/tBbP5qL86HyrtD8Drf4ZfuZ9A8Evyi2d+mpfxxOv7R+yaL8Bn5UPofg3+JLH9FT/gRzThx+VW1f1it/uSLZsZpJ1tg7GVZeKfBeHrJfq3A+Iqa2spY8MXq2PW23ZVYq+lVzcvCTx+k5F+Hx5peWkITubmtTVam5NucZLjw6tfJ6F6+rZbW4SpUKUZunJYxp82eXo/Xu6OjpMJos0L4dvFLbocV2Cmprhd7jU0/p6m99ldkx6syo6dmayarkVF1/D/ADQuto7VuKbk7Oq3GEIvSHG3rPz5PCWV0HlG2puUpXEFlzcd/FW/DUcatp827ciphXQXhrFUVhqIY5I6S0Xiut+JZFmd9OKFHSsk9v0NZjNVdXLxUncbaubbjuX9+nGVNdDeE116vOvMQhs+nWl4CnnjRquLfTHV56sJNZ6TK4Z0QaOH4csNzu0dBDBiSWeeN1ffnUktDT+kVsbYI8l9M5qZZ66+3w95b3G09pKtUo0m80lHOIcZSljLcn/dT5scxVo29pOHh8LiylJJOTWIxeNOl8+vUeU6JsCWPEenSkwxXSJd7EtVUw+lhkVqTxsZIrHo5qp7dVF8FNl2jeV6GypXcVxanFTx0NtZWv1mMp0Kbv426fGg5Y+teoy+JMN4PvWibEGJbDh2Sy1FBeYLbCj66SoVY1Y5XOdnkmbly8MvDLwLO3r31LaFG2uKvHU4yk/mpdGEvq1LydC2lSuZQjh0+KlrvfGw361zcx7VWaHLNNZ7lge3JJbbTV4ktmv/APRXvajqP0kmSuVfFfHLP2Kpqkdq1lUp31T50owqdW6WFuMpG0hG3qUqenHjSz9bnhv/APDyvTHo8wHacGV1wsU1qtt2oK9tOyjob4twdVQqqoqyNciLHI1UzVG5plmbDsu/2hVuYQrqUoTjltw4qT36Nb092vUWF3a2tOnVUcKUGsfOy3rhprmfPp0Mi0G6NcPX7DUdyxJZYKiOtuTaGCruV67BCrck1mwMZm+WVFX2Kmr7Ez9pW2xf3NGt4G2qNNRcmow4z6m86KP7TH2NKnOE6tSOUmlq+Kt2frb/AGYMtctHGBNHVixTcLvYanEi23FbrNTRrXvp84VjRyayt9qp4+xEVVy8ci0o39/tCrb0qNRQ49PjN8VPXONMmTuLO3tFcTceMoOOFnH0lnDfr379C1jjRRgTRLTYju9wstZiWl21FbKGh7c6DszHU7J1c57Uzc5NfVRF+xM/eU7LaW0Npyo0KdRQk4OUpcVPOJOKwnpzZZCva29uqtdxzFcXEctfSjnV78cxpfysWsbplrGxo5GJQ0aNR/1kT0DMs/iZTgxl2Gu/jS/eUNrY41Hi7uJH/k9GuugrR5YLRsa5V1so7psdKxbxPfNSq7QseuiJSKmr6L3Z555cTCLbO0a1SVajGTip44qhlYTx9PfxvVgu6FjaxhSjXazOOW+NhrOdy3NLr6zWbzoxwhR6Ge8WOzVbW1lBFRwWp0kupT1qyOY+qV+eaxLq5tRVyVVyL6O0bx7S8WeEWks8bT6OM8XHldPPjUt6NrQnZ/1yUfopprXDlnClv3dK9R6Rdo7DZMOY0S42ea/OiwnaamSStuMrnvY5ckia5c1YiOTWzb9uRgIuvVrUvBTUP7aa0it+N76ejUv7LiKEHNcb+wzq+bXKXRnq3cxxqviq5Jkn2HWDUXv0MRiD/Df52L/05zwz/Nn61R/mO1/0Yfn7/L7n+Q47OVGJN0qP1iX+Jf8AkEiMA6W0i6SMWaIu7qx4ErJbZZJLHR10UdJEituVRLmsrpPD/wCiq76OqvsLupKX9fqUksxi1GMeZxwv3vOvOUYJK0hUejkm5PnTy9M//On/ACTYZ0dW3EV7xHe8cYIobLUVV6ZSLT1t62ZR0z3MR8kcLG60kkv0kdq/VTPL4FK3o0+LTpp/Sbx04UsYiv8A5emvUidarNudSX92KfQs4zmT5srXTrbIL7ovwDovs+Orjd7DV4nSy4oZaaOFbg+m1oXwq9Ee5ieOX2oiKqonsTNC3o1IulSlNZcpTi/9L346err6i5lTbq1VF6RhCX2ub9v7CxjvRFgLRHDijEVfZK3EtsZdaW32+0rXugSnbLStqHOklYms7LW1W/5eOZXklR4tOercpLPVF8y6dezUpqLqpVY6LiRljrk2tX0aftwVcEaIsDXGhxfimpoZIrRS18FDb7Riu57K9F6SNJHLLKxHKqongxE+snip6oqnSjx9W5SW/Gi6vK115ljJBPwlR8XRKKfTq3jf0aaPn3FzH2F8PYS0L6SKLC1xiuVmfd7VPEsNQlQkLnxvV0XpE+tqrmiL70yzLW4b4lGD1Uas0n0rwa1/4fWme0ceFqyW904Nrfh8d/8AvrOYCoCvcv2XV/wt/O0BmrgiAAAAAAAAAADqPQt/dnZv9b/ued04Nfiqj/q/iZz/AGr+GT9X7kbubOYk3PAml3EujqmqaS0VcLrfUvSSWhrKaOohc9EyR+q9Fyd8Uy9iGJvdl2t/JTrL5y3NNp9qLqhc1bZvwb0e9b0/UZCl0941p7/c7vNdI7hUXKJkFXDXUsU0EjGfUb6JW6qI1VzTJE/3UoPYlk6MaMYYUXlNNppve85zl85Wd/cOr4Vy1xjGFjHRjoKs+mzGtRjGhxW+9u2/Qwdlp6ttNC3UiycmrqIzUVPpO9qL7fghVjsiyjbztVT+ZJ5ay9+muc55luZCd5XqVIVZS1hu0Wm/q62U8N6VsTYVvtzu1DcEdVXRXLXsqYWTRVWs5XL6SNyK1fFVX2eGfhkVK+zbW4owoTj82H0cNprGmj3kY3daNZ11L5zznrzv03f90KGMsb3nH142le6tKqqSNsMerG2NkUbc9VjGNREa1M18EQr2lnRsafg6Cws5erbb6W3zka9xUuGnUe7RcyS9RkU0s4sTGdLixLu5t/po2QxVbYY25MazURqsRuqqavh4p4+8oeLLT+rytOJ8yTbay97eenO89lc1pSpzctYJKL00Sz73vM1W/KJ0gV1TBK+9sZ6D03oY46GnayJJWaj0aiR5ZK3w/wB/b4lnHYOzoxcfB78J/Olrh5XP0+7cXL2jcuUZcbc+MtFo8NZ3dDfbneQW/T3ja14cjs1PdWtp4oFpIah1NE6phgVMljZMrddrcvj4e7InV2LY1qzrzhq3lrLw30tZwyFG+uKEFCEt27RNr6m93/cHnpnTHgA41w7/AGgtn81F+dD5V2h+B1v8Mv3M+geCX5RbO/TUv44nX9o/ZNF+Az8qH0Pwb/Elj+ip/wACOacOPyq2r+sVv9yRbNjNJPS7N8orHdhsdPbKS7R6lLF6ClqpaSGSpp48stRkrmq5Ey/qnuyNfrbCsLiq6s4PMtWstJvpaTMjRv69CChB6Ldonj6slfDenzGuFbVTUFvukKNpJHS0tRUUcM89Ornaz0ZI9qqiOVVzT35lS42LZXNR1KkN6w0m0mlospNbuYhTva9JNRlz51SeG97Wed85hrZpRxRZ6LEVJR3Z8FPiHW2nG2KNUqM1dn7W/Qz1nfVy9uRdVNm2tXwTnDPgvo6vTGOvXct+SEbutCc6kZYc853a5z73uMnhHThjDBNljtdruELaSBzn0vaaSKd1I931nROe1VYq/AtrrZFneVPC1YvL0eG1ldDw9SVC7rW8eLB6ZzhpPD6Vnca7YMZ3rDOJ48RW2udDeWPfKlW9jZXaz0VHKqPRUVVRy+1PeZCvaULig7apH5jWMbtFu3FFV6ka3h0/nZznrMrgrS1inR7HcY7Hcm00VeqOnilp45mOemeq9GvaqI5M/ahbXezLW+4nh454u7Vp/VlNaFSldVqM5VIPWW/Ra+rcWrnpxxzeJZZKq/yPllq4K58jIIo3rPC1GRv1msRUVGoiZJ4L78yjT2NYUsKFPcmt73S3rV8//hUnfXFSLjOecpJ7v7ryux653nzGmmnFmPrS22XauhWi9Kk8sVLSRU/aJUTL0kmo1Nd3j7/D4HtpsizsqnhaMddyy28LoWXoe1b6vWg4Tej36LLxuy+c+YP0zYrwLZnWu1VsDaRJu0wJU0kU7qaZUyWSJXtXUcqeHgSutlWt5UVWrF5xjRtZXQ8PVFKhc1LeMoQ3PXVJ67s685Qvmk/E2JKGvo7jcu0U1dcNq1DPQRN16nV1fSZtaip4eGqmSfArUNn21tOE6UMOEeKtXos5xq+np1JVbyvXU1Ulnj4zotcLC/Z0GeoPlCY7objda3bEdRNc1Y+pbU0UEkbpGMRjJEYrNVrmtaiZoiexM8yxnsKwnCFPiYUc4w2nq8tZznDJxv7iM3PjZyknlLDS3abtDUsXYxvGPb5JeL9WLX3KRjGPnWNjFcjWo1vg1ET2InjkZO1tKFlT8Dbx4sdXjXn+st69xVuJKVV5aWPUbTDp8xvBhpLI27M7O2mWjbUrSxLVNp1TJYkm1ddG5fHP4mPnsWxnW8O4a5y1l4b6Ws4yXFK+r0aapwlotFospdT5v+4KFRpkxhVw1MMt31qaotzLVJTdlhSFaZv1WJHqarcvFUciI7NfaVlsqzTUuJqpcfOXnjdOc59W7qIRvK8FFRlok4pYWMPfpuf1vXrLNDpzxpQXaS4suzJJ5aGO2yNlpIXRyU7PqMcxWaq5fbln8SnU2NZVKbpuGnGct7zxnvaec/8AB5Su61GUJwl9GPFX+Ho6/XqaGq5qqmaLRvLyYjEH+G/zsX/pznhn+bP1qj/Mdr/ow/P3+X3P8hx2cqMSbpUfrEv8S/8AIJEYB6LhH5QmPMD2KGz2q9NZQUyqtKyppIah1KqrmqxOkY5Wf0XLxKkqk5JJvdpnnx0Z/wC9RGMVHOOfXHNn6iHD2nfGuGaathpbrHUJV1a3Bz6+khqnx1S+CzsdK1yseqe9CMZOEYwjpxd3SulZ6+clL58pTlrxt/Xjdp1cxjMRaVcU4rorrSXW6dqp7pXtudYzs8TPS1LWaiPza1FT6PhkmSfApqKjGMVui219ct/b/wCE+PLMpZ1kkn9S3Gbt/wAofHlvu9xuO2IqqW4NhbVQ1dDBLBL6JqNid6JzNRHNREyciIvh45lbwktdd7cvW+ddDKfFWIroXF9XQ+n1la06dca2m+Xi6pd21tReHNfcIq+miqIalW/VV0T2q1NX3ZImWWSeBGEnCHEW7OenXp15z2Xzpcd78Y9XR9RRxDpexdiujvFJdbw6sp7tPDUVkboYkR74m6seWTU1EangjW5J8Cm4qSSfM3L1tYb69NNT1PEnJc6UfUnlLq11NPJHhXuX7Lq/4W/naAzVwRAAAAAAAAAAB7Jo600WjCmEqO011HWvmp1k+nTsY5rkc9zve5MvrZf0Oj7H4S2thZQtq0JNxzuw97b52uk1i92XVuK8qsJLDxvz0Y6DZPWJw3uV25UfmGa+WNh5E+yPxFh4kuPKj2v3D1icN7lduVH5g+WNh5E+yPxDxJceVHtfuHrE4b3K7cqPzB8sbDyJ9kfiHiS48qPa/cPWJw3uV25UfmD5Y2HkT7I/EPElx5Ue1+4esThvcrtyo/MHyxsPIn2R+IeJLjyo9r9w9YnDe5XblR+YPljYeRPsj8Q8SXHlR7X7h6xOG9yu3Kj8wfLGw8ifZH4h4kuPKj2v3D1icN7lduVH5g+WNh5E+yPxDxJceVHtfuHrE4b3K7cqPzB8sbDyJ9kfiHiS48qPa/cPWJw3uV25UfmD5Y2HkT7I/EPElx5Ue1+4/MvyisPpE9YqC5ukyXVa+ONEVfdmuuuXAjLhjY4fFpzz9S+I9WxLjOsl+33Hg9hkZDfLdJI5rI2VMbnOcuSIiOTNVU4nfRlO0qxistxl+5nWuDFanb7esK1aSjCNam228JJTi223oklq29x0pQaUsPU1DTQvrotaNjWLqzR5eCZKqLrJ/wAG9bI4eWlhs62tK1nXcqcIxeILGYxSePnrTToRnuEf9Gs9r7bvdpW+2bJQrVak0nX1SnNySeINZw9cNrPOT97OG9+Zzo+sy3KPYeaXHs4/Ga3yS3Hpqw+8Pux3s4b35nOj6xyj2Hmlx7OPxjkluPTVh94fdjvZw3vzOdH1jlHsPNLj2cfjHJLcemrD7w+7Hezhvfmc6PrHKPYeaXHs4/GOSW49NWH3h92O9nDe/M50fWOUew80uPZx+Mcktx6asPvD7sd7OG9+Zzo+sco9h5pcezj8Y5Jbj01YfeH3Y72cN78znR9Y5R7DzS49nH4xyS3Hpqw+8Pux3s4b35nOj6xyj2Hmlx7OPxjkluPTVh94fdjvZw3vzOdH1jlHsPNLj2cfjHJLcemrD7w+7Hezhvfmc6PrHKPYeaXHs4/GOSW49NWH3h92O9nDe/M50fWOUew80uPZx+Mcktx6asPvD7sd7OG9+Zzo+sco9h5pcezj8Y5Jbj01YfeH3Y72cN78znR9Y5R7DzS49nH4xyS3Hpqw+8Pux3s4b35nOj6xyj2Hmlx7OPxjkluPTVh94fdjvZw3vzOdH1jlHsPNLj2cfjHJLcemrD7w+7Hezhvfmc6PrHKPYeaXHs4/GOSW49NWH3h92O9nDe/M50fWOUew80uPZx+Mcktx6asPvD7sxeItKNilo2TU9XHLJSypUei9NHnJqoqo1PpL4r7PYatt7hdQ2zKyhb2taLpV6dRuUElxY5zuk9defC6ze+DPA2PBO32vc3e1bSr4azr0oxp1uNJzkotaOMc/Ra0y8tJI5qMKclNjfiCjkcr1SdquXNURjVy/rrA9yfNu0X7xy29QGRt2i/eOW3qAyNu0X7xy29QGRt2i/eOW3qAyNu0X7xy29QGRt2i/eOW3qAyNu0X7xy29QGRt2i/eOW3qAyV6+8009FLDEkqvkREze1ERMlRftX7AMmFB4ACfs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gHx1MiRveyaOXVTNUbrIqJnl70T7UAIQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAT036Gq/DT87QCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAE9N+hqvw0/O0AgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABPTfoar8NPztAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAT036Gq/DT87QCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAE9N+hqvw0/O0AgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABPTfoar8NPztAIAAAf/2Q==) ###### Test alert trigger[​](#test-alert-trigger "Direct link to Test alert trigger") You can manually stop an FE node to test the alert. At this point, the heart-shaped symbol to the right of Frontends Status will change from green to yellow and then to red. **Green**: Indicates that during the last periodic check, the status of each instance of the metric item was normal, and no alert was triggered. The green status does not guarantee that the current node is in normal status. There may be a delay in status change after a node service anomaly, but typically, the delay is not in the order of minutes. ![MA-41](/assets/images/monitor41-d6555dd45cee2a87e315e538a7fff036.png) **Yellow**: Indicates that during the last periodic check, an instance of the metric item was found abnormal, but the abnormal state duration has not yet reached the "Duration" configured above. At this point, Grafana will not send an alert and will continue periodic checks until the abnormal state duration reaches the configured "Duration". During this period, if the status is restored, the symbol will change back to green. ![MA-42](/assets/images/monitor42-44ec0dd64f7e71b1f398710742990b94.jpeg) **Red**: When the abnormal state duration reaches the configured "Duration", the symbol changes to red, and Grafana will send an email alert. The symbol will remain red until the abnormal state is resolved, at which point it will change back to green. ![MA-43](/assets/images/monitor43-859a3b060d8b224da48e1cf53f593e18.jpeg) ###### Manually pause Alerts[​](#manually-pause-alerts "Direct link to Manually pause Alerts") Suppose the anomaly requires an extended period for resolution or the alerts are continuously triggered for some reasons other than anomaly. You can temporarily pause the evaluation of the alert rule to prevent Grafana from persistently sending alert emails. Navigate to the Alert tab corresponding to the metric item on the Dashboard and click the edit icon: ![MA-44](/assets/images/monitor44-f370c012a1066414412bf5024cfba667.png) In the **Alert Evaluation Behavior** section, toggle the **Pause Evaluation** switch to the ON position. ![MA-45](/assets/images/monitor45-2ef24b9bf18cc0429210e63aff8a9554.png) > **NOTE** > > After pausing the evaluation, you will receive an email notifying you that the service is restored. ###### Configure the alert rule for BE[​](#configure-the-alert-rule-for-be "Direct link to Configure the alert rule for BE") You can follow the above process to configure alert rules for BE. Editing the configuration for the metric item Backends Status: 1. In the **Set an alert rule name** section, configure the name as "\[PROD]Backends Status". 2. In the **Set a query and alert condition** section, set PromSQL to `(up{group="be"})`, and use the same settings as those in the FE alert rule for other items. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "BE node in your StarRocks production cluster failed, please check! Stack information for BE failure will be printed in the BE log file **be.out**. You can identify the cause based on the logs". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### 3.4.2 Query alert rule[​](#342-query-alert-rule "Direct link to 3.4.2 Query alert rule") The metric item for query failures is **Query Error** under **Query Statistic**. Configure the alert rule for the metric item "Query Error" as follows: 1. In the **Set an alert rule name** section, configure the name as "\[PROD] Query Error". 2. In the **Set a query and alert condition** section, remove section **B**. Set **Input** in section **A** to **C**. In section **C**, use the default value for PromQL, which is `rate(starrocks_fe_query_err{job="StarRocks_Cluster01"}[1m])`, representing the number of failed queries per minute divided by 60s. This includes both failed queries and queries that exceeded the timeout limit. Then, in section **D**, configure the rule as `A IS ABOVE 0.05`. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "High query failure rate, please check the resource usage or configure query timeout reasonably. If queries are failing due to timeouts, you can adjust the query timeout by setting the system variable `query_timeout`". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### 3.4.3 User operation failure alert rule[​](#343-user-operation-failure-alert-rule "Direct link to 3.4.3 User operation failure alert rule") This item monitors the rate of Schema Change operation failures, corresponding to the metric item **Schema Change** under **BE tasks**. It should be configured to alert when greater than 0. 1. In the **Set an alert rule name** section, configure the name as "\[PROD] Schema Change". 2. In the **Set a query and alert condition** section, remove section **A**. Set **Input** in section **C** to **B**. In section **B**, use the default value for PromQL, which is `irate(starrocks_be_engine_requests_total{job="StarRocks_Cluster01", type="create_rollup", status="failed"}[1m])`, representing the number of failed Schema Change tasks per minute divided by 60s. Then, in section **D**, configure the rule as `C IS ABOVE 0`. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "Failed Schema Change tasks detected, please check promptly. You can increase the memory limit available for Schema Change by adjusting the BE configuration parameter `memory_limitation_per_thread_for_schema_change`, which is set to 2GB by default". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### 3.4.4 StarRocks operation failure alert rule[​](#344-starrocks-operation-failure-alert-rule "Direct link to 3.4.4 StarRocks operation failure alert rule") ###### BE Compaction Score[​](#be-compaction-score "Direct link to BE Compaction Score") This item corresponds to **BE Compaction Score** under **Cluster Overview**, and is used to monitor the compaction pressure on the cluster. 1. In the **Set an alert rule name** section, configure the name as "\[PROD] BE Compaction Score". 2. In the **Set a query and alert condition** section, configure the rule in section C as `B IS ABOVE 0`. You can use default values for other items. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "High compaction pressure. Please check whether there are high-frequency or high concurrency loading tasks and reduce the loading frequency. If the cluster has sufficient CPU, memory, and I/O resources, consider adjusting the cluster compaction strategy". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### Clone[​](#clone "Direct link to Clone") This item corresponds to **Clone** in **BE tasks** and is mainly used to monitor replica balancing or replica repair operations within StarRocks, which usually should not fail. 1. In the **Set an alert rule name** section, configure the name as "\[PROD] Clone". 2. In the **Set a query and alert condition** section, remove section A. Set **Input** in section **C** to **B**. In section **B**, use the default value for PromQL, which is `irate(starrocks_be_engine_requests_total{job="StarRocks_Cluster01", type="clone", status="failed"}[1m])`, representing the number of failed Clone tasks per minute divided by 60s. Then, in section **D**, configure the rule as `C IS ABOVE 0`. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "Detected a failure in the clone task. Please check the cluster BE status, disk status, and network status". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### 3.4.5 Service availability alert rule[​](#345-service-availability-alert-rule "Direct link to 3.4.5 Service availability alert rule") This item monitors the metadata log count in BDB, corresponding to the **Meta Log Count** monitoring item under **Cluster Overview**. 1. In the **Set an alert rule name** section, configure the name as "\[PROD] Meta Log Count". 2. In the **Set a query and alert condition** section, configure the rule in section **C** as `B IS ABOVE 100000`. You can use default values for other items. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "Detected that the metadata count in FE BDB is significantly higher than the expected value, which can indicate a failed Checkpoint operation. Please check whether the Xmx heap memory configuration in the FE configuration file **fe.conf** is reasonable". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### 3.4.6 System overload alert rule[​](#346-system-overload-alert-rule "Direct link to 3.4.6 System overload alert rule") ###### BE CPU Idle[​](#be-cpu-idle "Direct link to BE CPU Idle") This item monitors the CPU idle rate on BE nodes. 1. In the **Set an alert rule name** section, configure the name as "\[PROD] BE CPU Idle". 2. In the **Set a query and alert condition** section, configure the rule in section C as `B IS BELOW 10`. You can use default values for other items. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "Detected that BE CPU load is consistently high. It will impact other tasks in the cluster. Please check whether the cluster is abnormal or if there is a CPU resource bottleneck". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### BE Memory[​](#be-memory "Direct link to BE Memory") This item corresponds to **BE Mem** under **BE**, monitoring the memory usage on BE nodes. 1. In the **Set an alert rule name** section, configure the name as "\[PROD] BE Mem". 2. In the **Set a query and alert condition** section, configure PromSQL as `starrocks_be_process_mem_bytes{job="StarRocks_Cluster01"}/(*1024*1024*1024)`, where `` needs to be replaced with the current BE node's available memory limit, that is, the server's memory size multiplied by the value of the BE configuration item `mem_limit`. Example: `starrocks_be_process_mem_bytes{job="StarRocks_Cluster01"}/(49*1024*1024*1024)`. Then, in section **C**, configure the rule as `B IS ABOVE 0.9`. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "Detected that BE memory usage is consistently high. To prevent query failure, please consider expanding memory size or adding BE nodes". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### Disks Avail Capacity[​](#disks-avail-capacity "Direct link to Disks Avail Capacity") This item corresponds to **Disk Usage** under **BE**, monitoring the remaining space ratio in the directory where the BE storage path is located. 1. In the **Set an alert rule name** section, configure the name as "\[PROD] Disks Avail Capacity". 2. In the **Set a query and alert condition** section, configure the rule in section **C** as `B`` IS BELOW 0.2`. You can use default values for other items. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "Detected that BE disk available space is below 20%, please release disk space or expand the disk". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### FE JVM Heap Stat[​](#fe-jvm-heap-stat "Direct link to FE JVM Heap Stat") This item corresponds to **Cluster FE JVM Heap Stat** under **Overview**, monitoring the proportion of FE's JVM memory usage to FE heap memory limit. 1. In the **Set an alert rule name** section, configure the name as "\[PROD] FE JVM Heap Stat". 2. In the **Set a query and alert condition** section, configure the rule in section **C** as `B IS ABOVE 80`. You can use default values for other items. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "Detected that FE heap memory usage is high, please adjust the heap memory limit in the FE configuration file **fe.conf**". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. #### Appendix[​](#appendix "Direct link to Appendix") ##### Enable Service Detection for Prometheus[​](#enable-service-detection-for-prometheus "Direct link to Enable Service Detection for Prometheus") You can enable Service Detection for Prometheus so that it can automatically detect the services (nodes) after the cluster is scaled in or out. note The following section uses AWS as an example. 1. Grant the EC2 instance that hosts your Prometheus service the following permissions using IAM Policy: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:DescribeInstances", "ec2:DescribeTags" ], "Resource": "*" } ] } ``` For detailed instructions of authentication to AWS resources, see [Authenticate to AWS resources](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md). With these permissions, Prometheus is able to list the instances and their tags in the region. 2. Add the `ec2_sd_configs` and `relabel_configs` sections to **prometheus/prometheus.yml**. Example: ```yaml global: scrape_interval: 15s # Set the global scrape interval to 15s. The default is 1 min. evaluation_interval: 15s # Set the global rule evaluation interval to 15s. The default is 1 min. scrape_configs: - job_name: 'StarRocks_Cluster01' metrics_path: '/metrics' ec2_sd_configs: - region: us-west-2 port: 8030 filters: - name: tag:ClusterName values: ['test-stage-20251021'] - name: tag:ProcessType values: ['FE'] - region: us-west-2 port: 8040 filters: - name: tag:ClusterName values: ['test-stage-20251021'] - name: tag:ProcessType values: ['BE'] relabel_configs: - source_labels: [__meta_ec2_tag_ClusterName] regex: test-stage-20251021 target_label: cluster replacement: test-stage-20251021 - source_labels: [__meta_ec2_tag_ProcessType] regex: FE target_label: group replacement: fe - source_labels: [__meta_ec2_tag_ProcessType] regex: BE target_label: group replacement: be ``` #### Q\&A[​](#qa "Direct link to Q\&A") ##### Q: Why can't the Dashboard detect anomalies?[​](#q-why-cant-the-dashboard-detect-anomalies "Direct link to Q: Why can't the Dashboard detect anomalies?") A: Grafana Dashboard relies on the system time of the server it is hosted on to fetch values for monitoring items. If the Grafana Dashboard page remains unchanged after a cluster anomaly, you can check if the servers' system clock is synchronized and then perform cluster time calibration. ##### Q: How can I implement alert grading?[​](#q-how-can-i-implement-alert-grading "Direct link to Q: How can I implement alert grading?") A: Taking the Query Error item as an example, you can create two alert rules for it with different alert thresholds. For example: * **Risk Level**: Set the failure rate greater than 0.05, indicating a risk. Send the alert to the development team. * **Severity Level**: Set the failure rate greater than 0.20, indicating a severity. At this point, the alert notification will be sent to both the development and operations teams simultaneously. ##### Q: How can I retrieve more detailed metrics, including table-level metrics, materialized view metrics, and connection statistics with user labels?[​](#q-how-can-i-retrieve-more-detailed-metrics-including-table-level-metrics-materialized-view-metrics-and-connection-statistics-with-user-labels "Direct link to Q: How can I retrieve more detailed metrics, including table-level metrics, materialized view metrics, and connection statistics with user labels?") A: By default, the `/metrics` endpoint collects metrics in a minified mode to minimize performance impact. To retrieve detailed metrics, you need to add specific parameters to the request and provide Basic Authentication credentials for a user with ADMIN privileges. **Supported Parameters:** * `with_table_metrics=all`: Collects all table-level metrics. * `with_materialized_view_metrics=all`: Collects all materialized view metrics. * `with_user_connections=all`: Collects connection statistics categorized by user labels. **Authentication Requirement:** These parameters take effect only when the request includes valid Basic Authentication credentials for an ADMIN user. If the request is anonymous or the user lacks ADMIN privileges, these parameters are ignored, and only default metrics are returned. **Example Curl Command:** ```bash curl -u : \ "http://:/metrics?with_table_metrics=all&with_materialized_view_metrics=all&with_user_connections=all" ``` **Prometheus Configuration Example:** To enable detailed metric collection in Prometheus, configure `params` and `basic_auth` in your `prometheus.yml`: ```yaml scrape_configs: - job_name: 'StarRocks_Detailed_Metrics' metrics_path: '/metrics' params: with_table_metrics: ['all'] with_materialized_view_metrics: ['all'] with_user_connections: ['all'] basic_auth: username: '' password: '' static_configs: - targets: [':'] ``` note Collecting all table and materialized view metrics may increase the load on the FE node. Use these parameters with caution in large-scale environments. --- ### Manage Alerts This topic introduces various alert items from different dimensions, including business continuity, cluster availability, and machine load, and provides corresponding resolutions. note In the following examples, all variables are prefixed with `$`. They should be replaced according to your business environment. For example, `$job_name` should be replaced with the corresponding Job Name in the Prometheus configuration, and `$fe_leader` should be replaced with the IP address of the Leader FE. #### Service Suspension Alerts[​](#service-suspension-alerts "Direct link to Service Suspension Alerts") ##### FE Service Suspension[​](#fe-service-suspension "Direct link to FE Service Suspension") **PromSQL** ```plain count(up{group="fe", job="$job_name"}) >= 3 ``` **Alert Description** An alert is triggered when the number of active FE nodes falls below a specified value. You can adjust this value based on the actual number of FE nodes. **Resolution** Try to restart the suspended FE node. ##### BE Service Suspension[​](#be-service-suspension "Direct link to BE Service Suspension") **PromSQL** ```plain node_info{type="be_node_num", job="$job_name",state="dead"} > 1 ``` **Alert Description** An alert is triggered when more than one BE node is suspended. **Resolution** Try to restart the suspended BE node. #### Machine Load Alerts[​](#machine-load-alerts "Direct link to Machine Load Alerts") ##### BE CPU Alert[​](#be-cpu-alert "Direct link to BE CPU Alert") **PromSQL** ```plain (1-(sum(rate(starrocks_be_cpu{mode="idle", job="$job_name",instance=~".*"}[5m])) by (job, instance)) / (sum(rate(starrocks_be_cpu{job="$job_name",host=~".*"}[5m])) by (job, instance))) * 100 > 90 ``` **Alert Description** An alert is triggered when BE CPU Utilization exceeds 90%. **Resolution** Check whether there are large queries or large-scale data loading and forward the details to the support team for further investigation. 1. Use the `top` command to check resource usage by processes. ```bash top -Hp $be_pid ``` 2. Use the `perf` command to collect and analyze performance data. ```bash # Execute the command for 1-2 minutes, and terminate it by pressing CTRL+C. sudo perf top -p $be_pid -g >/tmp/perf.txt ``` note In emergencies, to quickly restore service, you can try to restart the corresponding BE node after preserving the stack. An emergency here refers to a situation where the BE node's CPU utilization remains abnormally high, and no effective means are available to reduce CPU usage. ##### Memory Alert[​](#memory-alert "Direct link to Memory Alert") **PromSQL** ```plain (1-node_memory_MemAvailable_bytes{instance=~".*"}/node_memory_MemTotal_bytes{instance=~".*"})*100 > 90 ``` **Alert Description** An alert is triggered when memory usage exceeds 90%. **Resolution** Refer to the [Get Heap Profile](https://github.com/StarRocks/starrocks/pull/35322) for troubleshooting. note * In emergencies, you can try to restart the corresponding BE service to restore the service. An emergency here refers to a situation where the BE node's memory usage remains abnormally high, and no effective means are available to reduce memory usage. * If other mixed-deployed services are affecting the system, you may consider terminating those services in emergencies. ##### Disk Alerts[​](#disk-alerts "Direct link to Disk Alerts") ###### Disk Load Alert[​](#disk-load-alert "Direct link to Disk Load Alert") **PromSQL** ```sql rate(node_disk_io_time_seconds_total{instance=~".*"}[1m]) * 100 > 90 ``` **Alert Description** An alert is triggered when disk load exceeds 90%. **Resolution** If the cluster triggers a `node_disk_io_time_seconds_total` alert, first check if there are any business changes. If so, consider rolling back the changes to maintain the previous resource balance. If no changes are identified or rollback is not possible, consider whether normal business growth is driving the need for resource expansion. You can use the `iotop` tool to analyze disk I/O usage. `iotop` has a UI similar to `top` and includes information such as `pid`, `user`, and I/O. You can also use the following SQL query to identify the tablets consuming significant I/O and trace them back to specific tasks and tables. ```sql -- "all" indicates all services. 10 indicates the collection lasts 10 seconds. 3 indicates fetching the top 3 results. ADMIN EXECUTE ON $backend_id 'System.print(ExecEnv.io_profile_and_get_topn_stats("all", 10, 3))'; ``` ###### Root Path Capacity Alert[​](#root-path-capacity-alert "Direct link to Root Path Capacity Alert") **PromSQL** ```sql node_filesystem_free_bytes{mountpoint="/"} /1024/1024/1024 < 5 ``` **Alert Description** An alert is triggered when the available space in the root directory is less than 5GB. **Resolution** Common directories that may occupy significant space include **/var**, \*\*/\*\***opt**, and **/tmp**. Use the following command to check for large files and clear unnecessary files. ```bash du -sh / --max-depth=1 ``` ###### Data Disk Capacity Alert[​](#data-disk-capacity-alert "Direct link to Data Disk Capacity Alert") **PromSQL** ```bash (SUM(starrocks_be_disks_total_capacity{job="$job"}) by (host, path) - SUM(starrocks_be_disks_avail_capacity{job="$job"}) by (host, path)) / SUM(starrocks_be_disks_total_capacity{job="$job"}) by (host, path) * 100 > 90 ``` **Alert Description** An alert is triggered when disk capacity utilization exceeds 90%. **Resolution** 1. Check if there have been changes in the loaded data volume. Monitor the `load_bytes` metric in Grafana. If there has been a significant increase in data loading volume, you may need to scale the system resources. 2. Check for any DROP operations. If data loading volume has not changed much, run `SHOW BACKENDS`. If the reported disk usage does not match the actual usage, check the FE Audit Log for recent DROP DATABASE, TABLE, or PARTITION operations. Metadata for these operations remains in FE memory for one day, allowing you to restore data using the RECOVER statement within 24 hours to avoid misoperations. After recovery, the actual disk usage may exceed what is shown in `SHOW BACKENDS`. The retention period of deleted data in memory can be adjusted using the FE dynamic parameter `catalog_trash_expire_second` (default value: 86400). ```bash ADMIN SET FRONTEND CONFIG ("catalog_trash_expire_second"="86400"); ``` To persist this change, add the configuration item to the FE configuration file **fe.conf**. After that, deleted data will be moved to the **trash** directory on BE nodes (`$storage_root_path/trash`). By default, deleted data is kept in the **trash** directory for one day, which may also result in the actual disk usage exceeding what is shown in `SHOW BACKENDS`. The retention time of deleted data in the **trash** directory can be adjusted using the BE dynamic parameter `trash_file_expire_time_sec` (default value: 86400). ```bash curl http://$be_ip:$be_http_port/api/update_config?trash_file_expire_time_sec=86400 ``` ###### FE Metadata Disk Capacity Alert[​](#fe-metadata-disk-capacity-alert "Direct link to FE Metadata Disk Capacity Alert") **PromSQL** ```bash node_filesystem_free_bytes{mountpoint="${meta_path}"} /1024/1024/1024 < 10 ``` **Alert Description** An alert is triggered when the available disk space for FE metadata is less than 10GB. **Resolution** Use the following commands to check for directories occupying large amounts of space and clear unnecessary files. The metadata path is specified by the `meta_dir` configuration in **fe.conf**. ```bash du -sh /${meta_dir} --max-depth=1 ``` If the metadata directory occupies a lot of space, it is usually because the **bdb** directory is large, possibly due to CheckPoint failure. Refer to the [CheckPoint Failure Alert](#checkpoint-failure-alert) for troubleshooting. If this method does not solve the issue, contact the technical support team. #### Cluster Service Exception Alerts[​](#cluster-service-exception-alerts "Direct link to Cluster Service Exception Alerts") ##### Compaction Failure Alerts[​](#compaction-failure-alerts "Direct link to Compaction Failure Alerts") ###### Cumulative Compaction Failure Alert[​](#cumulative-compaction-failure-alert "Direct link to Cumulative Compaction Failure Alert") **PromSQL** ```bash increase(starrocks_be_engine_requests_total{job="$job_name" ,status="failed",type="cumulative_compaction"}[1m]) > 3 increase(starrocks_be_engine_requests_total{job="$job_name" ,status="failed",type="base_compaction"}[1m]) > 3 ``` **Alert Description** An alert is triggered when there are three failures in Cumulative Compaction or Base Compaction within the last minute. **Resolution** Search the log of the corresponding BE node for the following keywords to identify the involved tablet. ```bash grep -E 'compaction' be.INFO | grep failed ``` A log record like the following indicates a Compaction failure. ```plain W0924 17:52:56:537041 123639 comaction_task_cpp:193] compaction task:8482. tablet:8423674 failed. ``` You can check the context of the log to analyze the failure. Typically, the failure may have been caused by a DROP TABLE or PARTITION operation during the Compaction process. The system has an internal retry mechanism for Compaction, and you can also manually set the tablet's status to BAD and trigger a Clone task to repair it. note Before performing the following operation, ensure that the table has at least three complete replicas. ```bash ADMIN SET REPLICA STATUS PROPERTIES("tablet_id" = "$tablet_id", "backend_id" = "$backend_id", "status" = "bad"); ``` ###### High Compaction Pressure Alert[​](#high-compaction-pressure-alert "Direct link to High Compaction Pressure Alert") **PromSQL** ```bash starrocks_fe_max_tablet_compaction_score{job="$job_name",instance="$fe_leader"} > 100 ``` **Alert Description** An alert is triggered when the highest Compaction Score exceeds 100, indicating high Compaction pressure. **Resolution** This alert is typically caused by frequent loading, `INSERT INTO VALUES`, or `DELETE` operations (at a rate of 1 per second). It is recommended to set the interval between loading or DELETE tasks to more than 5 seconds and avoid submitting high concurrency DELETE tasks. ###### Exceeding Version Count Alert[​](#exceeding-version-count-alert "Direct link to Exceeding Version Count Alert") **PromSQL** ```bash starrocks_be_max_tablet_rowset_num{job="$job_name"} > 700 ``` **Alert Description** An alert is triggered when a tablet on a BE node has more than 700 data versions. **Resolution** Use the following command to check the tablet with excessive versions: ```sql SELECT BE_ID,TABLET_ID FROM information_schema.be_tablets WHERE NUM_ROWSET>700; ``` Example for Tablet with ID `2889156`: ```sql SHOW TABLET 2889156; ``` Execute the command returned in the `DetailCmd` field: ```sql SHOW PROC '/dbs/2601148/2889154/partitions/2889153/2889155/2889156'; ``` ![show proc replica](/assets/images/alert_show_proc_3-b5ec100163dd7cf0d4b942da44532d5e.png) Under normal circumstances, as shown, all three replicas should be in `NORMAL` status, and other metrics like `RowCount` and `DataSize` should remain consistent. If only one replica exceeds the version limit of 700, you can trigger a Clone task based on other replicas using the following command: ```sql ADMIN SET REPLICA STATUS PROPERTIES("tablet_id" = "$tablet_id", "backend_id" = "$backend_id", "status" = "bad"); ``` If two or more replicas exceed the version limit, you can temporarily increase the version count limit: ```bash # Replace be_ip with the IP of the BE node which stores the tablet that exceeds the version limit. # The default be_http_port is 8040. # The default value of tablet_max_versions is 1000. curl -XPOST http://$be_ip:$be_http_port/api/update_config?tablet_max_versions=2000 ``` ##### CheckPoint Failure Alert[​](#checkpoint-failure-alert "Direct link to CheckPoint Failure Alert") **PromSQL** ```bash starrocks_fe_meta_log_count{job="$job_name",instance="$fe_master"} > 100000 ``` **Alert Description** An alert is triggered when the FE node's BDB log count exceeds 100,000. By default, the system performs a CheckPoint when the BDB log count exceeds 50,000, and then resets the count to 0. **Resolution** This alert indicates that a CheckPoint was not performed. You need to investigate the FE logs to analyze the CheckPoint process and resolve the issue: In the **fe.log** of the Leader FE node, search for records like `begin to generate new image: image.xxxx`. If found, it means the system has started generating a new image. Continue checking the logs for records like `checkpoint finished save image.xxxx` to confirm successful image creation. If you find `Exception when generate new image file`, the image generation failed. You should carefully handle the metadata based on the specific error. It is recommended to contact the support team for further analysis. ##### Excessive FE Thread Count Alert[​](#excessive-fe-thread-count-alert "Direct link to Excessive FE Thread Count Alert") **PromSQL** ```bash starrocks_fe_thread_pool{job="$job_name", type!="completed_task_count"} > 3000 ``` **Alert Description** An alert is triggered when the number of threads on the FE exceeds 3000. **Resolution** The default thread count limit for FE and BE nodes is 4096. A large number of UNION ALL queries typically lead to an excessive thread count. It is recommended to reduce the concurrency of UNION ALL queries and adjust the system variable `pipeline_dop`. If it is not possible to adjust SQL query granularity, you can globally adjust `pipeline_dop`: ```sql SET GLOBAL pipeline_dop=8; ``` note In emergencies, to restore services quickly, you can increase the FE dynamic parameter `thrift_server_max_worker_threads` (default value: 4096). ```sql ADMIN SET FRONTEND CONFIG ("thrift_server_max_worker_threads"="8192"); ``` ##### High FE JVM Usage Alert[​](#high-fe-jvm-usage-alert "Direct link to High FE JVM Usage Alert") **PromSQL** ```sql sum(jvm_heap_size_bytes{job="$job_name", type="used"}) * 100 / sum(jvm_heap_size_bytes{job="$job_name", type="max"}) > 90 ``` **Alert Description** An alert is triggered when the JVM usage on an FE node exceeds 90%. **Resolution** This alert indicates that JVM usage is too high. You can use the `jmap` command to analyze the situation. Since detailed monitoring information for this metric is still under development, direct insights are limited. Perform the following actions and send the results to the support team for analysis: ```bash # Note that specifying `live` in the command may cause FE to restart. jmap -histo[:live] $fe_pid > jmap.dump ``` note In emergencies, to quickly restore services, you can restart the corresponding FE node or increase the JVM (Xmx) size and then restart the FE service. #### Service Availability Alerts[​](#service-availability-alerts "Direct link to Service Availability Alerts") ##### Loading Exception Alerts[​](#loading-exception-alerts "Direct link to Loading Exception Alerts") ###### Loading Failure Alert[​](#loading-failure-alert "Direct link to Loading Failure Alert") **PromSQL** ```sql rate(starrocks_fe_txn_failed{job="$job_name",instance="$fe_master"}[5m]) * 100 > 5 ``` **Alert Description** An alert is triggered when the number of failed loading transactions exceeds 5% of the total. **Resolution** Check the logs of the Leader FE node to find information about the loading errors. Search for the keyword `status: ABORTED` to identify failed loading tasks. ```plain 2024-04-09 18:34:02.363+08:00 INFO (thrift-server-pool-8845163|12111749) [DatabaseTransactionMgr.abortTransaction():1279] transaction:[TransactionState. txn_id: 7398864, label: 967009-2f20a55e-368d-48cf-833a-762cf1fe07c5, db id: 10139, table id list: 155532, callback id: 967009, coordinator: FE: 192.168.2.1, transaction status: ABORTED, error replicas num: 0, replica ids: , prepare time: 1712658795053, commit time: -1, finish time: 1712658842360, total cost: 47307ms, reason: [E1008]Reached timeout=30000ms @192.168.1.1:8060 attachment: RLTaskTxnCommitAttachment [filteredRows=0, loadedRows=0, unselectedRows=0, receivedBytes=1033110486, taskExecutionTimeMs=0, taskId=TUniqueId(hi:3395895943098091727, lo:-8990743770681178171), jobId=967009, progress=KafkaProgress [partitionIdToOffset=2_1211970882|7_1211893755]]] successfully rollback ``` ###### Routine Load Consumption Delay Alert[​](#routine-load-consumption-delay-alert "Direct link to Routine Load Consumption Delay Alert") **PromSQL** ```sql (sum by (job_name)(starrocks_fe_routine_load_max_lag_of_partition{job="$job_name",instance="$fe_mater"})) > 300000 starrocks_fe_routine_load_jobs{job="$job_name",host="$fe_mater",state="NEED_SCHEDULE"} > 3 starrocks_fe_routine_load_jobs{job="$job_name",host="$fe_mater",state="PAUSED"} > 0 starrocks_fe_routine_load_jobs{job="$job_name",host="$fe_mater",state="UNSTABLE"} > 0 ``` **Alert Description** * An alert is triggered when over 300,000 entries are delayed in consumption. * An alert is triggered when the number of pending Routine Load tasks exceeds 3. * An alert is triggered when there are tasks in the `PAUSED` state. * An alert is triggered when there are tasks in the `UNSTABLE` state. **Resolution** 1. First, check if the Routine Load task status is `RUNNING`. ```sql SHOW ROUTINE LOAD FROM $db; ``` Pay attention to the `State` field in the returned data. 2. If any Routine Load task is in the `PAUSED` state, examine the `ReasonOfStateChanged`, `ErrorLogUrls`, and `TrackingSQL` fields. Typically, executing the SQL query in `TrackingSQL` can reveal the specific error. Example: ![Tracking SQL](/assets/images/alert_routine_load_tracking-ea00801a160616a69c2e799c541fd285.png) 3. If the Routine Load task status is `RUNNING`, you can try to increase the task’s concurrency. The concurrency of individual Routine Load jobs is determined by the minimum value of the following four parameters: * `kafka_partition_num`: Number of partitions in the Kafka Topic. * `desired_concurrent_number`: The set concurrency for the task. * `alive_be_num`: Number of live BE nodes. * `max_routine_load_task_concurrent_num`: FE configuration parameter, with a default value of 5. In most cases, you may need to adjust the task’s concurrency or the number of Kafka Topic partitions (contact Kafka support if necessary). The following example shows how to set concurrency for the task. ```sql ALTER ROUTINE LOAD FOR ${routine_load_jobname} PROPERTIES ( "desired_concurrent_number" = "5" ); ``` ###### Loading Transaction Limit Alert for a Single Database[​](#loading-transaction-limit-alert-for-a-single-database "Direct link to Loading Transaction Limit Alert for a Single Database") **PromSQL** ```sql sum(starrocks_fe_txn_running{job="$job_name"}) by(db) > 900 ``` **Alert Description** An alert is triggered when the number of loading transactions for a single database exceeds 900 (100 in versions prior to v3.1). **Resolution** This alert is typically triggered by a large number of newly added loading tasks. You can temporarily increase the limit on loading transactions for a single database. ```sql ADMIN SET FRONTEND CONFIG ("max_running_txn_num_per_db" = "2000"); ``` ##### Query Exception Alerts[​](#query-exception-alerts "Direct link to Query Exception Alerts") ###### Query Latency Alert[​](#query-latency-alert "Direct link to Query Latency Alert") **PromSQL** ```sql starrocks_fe_query_latency_ms{job="$job_name", quantile="0.95"} > 5000 ``` **Alert Description** An alert is triggered when the P95 query latency exceeds 5 seconds. **Resolution** 1. Investigate whether there are any big queries. Check whether large queries have consumed significant machine resources during the exception, leading to other queries timing out or failing. * Execute `show proc '/current_queries';` to view the `QueryId` of big queries. If you need to quickly restore service, you can use the `KILL` command to terminate the long-running queries. ```sql mysql> SHOW PROC '/current_queries'; +--------------------------------------+--------------+------------+------+-----------+----------------+----------------+------------------+----------+ | QueryId | ConnectionId | Database | User | ScanBytes | ProcessRows | CPUCostSeconds | MemoryUsageBytes | ExecTime | +--------------------------------------+--------------+------------+------+-----------+----------------+----------------+------------------+----------+ | 7c56495f-ae8b-11ed-8ebf-00163e00accc | 4 | tpcds_100g | root | 37.88 MB | 1075769 Rows | 11.13 Seconds | 146.70 MB | 3804 | | 7d543160-ae8b-11ed-8ebf-00163e00accc | 6 | tpcds_100g | root | 13.02 GB | 487873176 Rows | 81.23 Seconds | 6.37 GB | 2090 | +--------------------------------------+--------------+------------+------+-----------+----------------+----------------+------------------+----------+ 2 rows in set (0.01 sec) ``` * You can also restart the BE nodes with high CPU utilization to resolve the issue. 2. Check if the machine resources are sufficient. Verify whether CPU, memory, Disk I/O, and network traffic during the exception are normal. If anomalies are detected, investigate the root cause by examining peak traffic variations and cluster resource usage. If the issue persists, consider restarting the affected node. note In emergencies, you can resolve the issue by: * Reducing business traffic and restarting the affected BE node if a sudden traffic spike caused resource overuse and query failure. * Expanding node capacity if high resource usage is due to normal operations. ###### Query Failure Alert[​](#query-failure-alert "Direct link to Query Failure Alert") **PromSQL** ```plain sum by (job,instance)(starrocks_fe_query_err_rate{job="$job_name"}) * 100 > 10 # This PromSQL is supported from v3.1.15, v3.2.11, and v3.3.3 onwards. increase(starrocks_fe_query_internal_err{job="$job_name"})[1m] >10 ``` **Alert Description** An alert is triggered when the query failure rate exceeds 0.1/second or 10 failed queries occur within one minute. **Resolution** When this alert is triggered, check the logs to identify the queries that failed. ```bash grep 'State=ERR' fe.audit.log ``` If you have the AuditLoader plugin installed, you can locate the corresponding queries using the following query. ```bash SELECT stmt FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE state='ERR'; ``` Note that queries that fail due to syntax errors or timeouts are also recorded in `starrocks_fe_query_err_rate`. For query failures caused by kernel issues, search the `fe.log` for the error and obtain the complete stack trace and [Query Dump](https://docs.starrocks.io/docs/faq/Dump_query.md), and contact the support team for troubleshooting. ###### Query Overload Alert[​](#query-overload-alert "Direct link to Query Overload Alert") **PromSQL** ```bash abs((sum by (exported_job)(rate(starrocks_fe_query_total{process="FE",job="$job_name"}[3m]))-sum by (exported_job)(rate(starrocks_fe_query_total{process="FE",job="$job_name"}[3m] offset 1m)))/sum by (exported_job)(rate(starrocks_fe_query_total{process="FE",job="$job_name"}[3m]))) * 100 > 100 abs((sum(starrocks_fe_connection_total{job="$job_name"})-sum(starrocks_fe_connection_total{job="$job_name"} offset 3m))/sum(starrocks_fe_connection_total{job="$job_name"})) * 100 > 100 ``` **Alert Description** An alert is triggered when the QPS or the number of connections increases by 100% within the last minute. **Resolution** Check whether the high-frequency queries in the `fe.audit.log` are expected. If there are legitimate changes in business behavior (for example, new services going live or increased data volumes), monitor machine load and scale BE nodes as needed. ###### User Connection Limit Exceeded Alert[​](#user-connection-limit-exceeded-alert "Direct link to User Connection Limit Exceeded Alert") **PromSQL** ```bash sum(starrocks_fe_connection_total{job="$job_name"}) by(user) > 90 ``` **Alert Description** An alert is triggered when the number of user connections exceeds 90. (User connection limits are supported from versions v3.1.16, v3.2.12, and v3.3.4 onward.) **Resolution** Use the SQL command `SHOW PROCESSLIST` to check if the number of current connections is as expected. You can terminate unexpected connections using the `KILL` command. Additionally, ensure that frontend services are not holding connections open for too long, and consider adjusting the system variable `wait_timeout` (Unit: Seconds) to accelerate the system's automatic termination of idle connections. ```bash SET wait_timeout = 3600; ``` note In emergencies, you can increase the user connection limit temporarily to restore service: * For v3.1.16, v3.2.12, and v3.3.4 or later: ```bash ALTER USER 'jack' SET PROPERTIES ("max_user_connections" = "1000"); ``` * For v2.5 and earlier: ```bash SET PROPERTY FOR 'jack' 'max_user_connections' = '1000'; ``` ##### Schema Change Exception Alert[​](#schema-change-exception-alert "Direct link to Schema Change Exception Alert") **PromSQL** ```bash increase(starrocks_be_engine_requests_total{job="$job_name",type="schema_change", status="failed"}[1m]) > 1 ``` **Alert Description** An alert is triggered when more than one Schema Change task fails in the last minute. **Resolution** Run the following statement to check if the `Msg` field contains any error messages: ```bash SHOW ALTER COLUMN FROM $db; ``` If no message is found, search for the JobId from the previous step in the Leader FE logs to retrieve the context. * Schema Change Out of Memory If the Schema Change fails due to insufficient memory, search the **be.WARNING** logs for `failed to process the version`, `failed to process the schema change from tablet`, or `Memory of schema change task exceeded limit` to identify log records shown in the following: ```bash fail to execute schema change: Memory of schema change task exceed limit. DirectSchemaChange Used: 2149621304, Limit: 2147483648. You can change the limit by modify BE config [memory_limitation_per_thread_for_schema_change] ``` The memory limit error is typically caused by exceeding the 2GB memory limit for a single Schema Change, controlled by the BE dynamic parameter `memory_limitation_per_thread_for_schema_change`. You can modify this parameter to resolve the issue. ```bash curl -XPOST http://be_host:http_port/api/update_config?memory_limitation_per_thread_for_schema_change=8 ``` * Schema Change Timeout Except for adding columns, which is a lightweight implementation, most Schema Changes involve creating a large number of new tablets, rewriting the original data, and implementing the operation via SWAP. ```plain Create replicas failed. Error: Error replicas:21539953=99583471, 21539953=99583467, 21539953=99599851 ``` You can address this by: * Increasing the timeout for creating tablets (Default: 10 seconds). ```bash ADMIN SET FRONTEND CONFIG ("tablet_create_timeout_second"="60"); ``` * Increasing the number of threads for creating tablets (default: 3). ```bash curl -XPOST http://be_host:http_port/api/update_config?alter_tablet_worker_count=6 ``` * Non-Normal Tablet State 1. If a tablet is in a non-normal state, search the **be.WARNING** logs for `tablet is not normal` and execute `SHOW PROC '/statistic'` to check the cluster-level `UnhealthyTabletNum`. ![show statistic](/assets/images/alert_show_statistic-0b98665c37c4ad9d02dd67870099f871.png) 2. Execute `SHOW PROC '/statistic/$DbId'` to check the unhealthy tablet number in the specified database. ![show statistic db](/assets/images/alert_show_statistic_db-d7769f340e126de5cd9161e4a3920cbc.png) 3. Execute `SHOW TABLET $tablet_id` to view the table information of the corresponding tablet. ![show tablet](/assets/images/alert_show_tablet-24f35ed634afe110d11f2edce74be565.png) 4. Execute the command returned in the `DetailCmd` field to identify the cause of the unhealthy tablets. ![show proc](/assets/images/alert_show_proc-a71de5e2f30f015ef8e3483c0ff69abe.png) Typically, unhealthy as well as inconsistent replicas are usually caused by high-frequency loading, where the progress of writes to different replicas is not synchronized. You can check if the table has a large number of real-time writes and reduce the number of abnormal replicas by reducing the frequency of loading or temporarily suspending the service and retrying the task thereafter. note In emergencies, to restore the service, you can set the non-Normal replicas as Bad to trigger a Clone task. ```bash ADMIN SET REPLICA STATUS PROPERTIES("tablet_id" = "$tablet_id", "backend_id" = "$backend_id", "status" = "bad"); ``` Before performing this operation, ensure the table has at least three complete replicas with only one non-normal replica. ##### Materialized View Refresh Exception Alert[​](#materialized-view-refresh-exception-alert "Direct link to Materialized View Refresh Exception Alert") **PromSQL** ```bash increase(starrocks_fe_mv_refresh_total_failed_jobs[5m]) > 0 ``` **Alert Description** An alert is triggered when more than one materialized view refresh fails in the last five minutes. **Resolution** 1. Check the materialized views that failed to refresh. ```sql SELECT TABLE_NAME,IS_ACTIVE,INACTIVE_REASON,TASK_NAME FROM information_schema.materialized_views WHERE LAST_REFRESH_STATE !=" SUCCESS"; ``` 2. Try manually refreshing the materialized view. ```sql REFRESH MATERIALIZED VIEW $mv_name; ``` 3. If the materialized view is in the `INACTIVE` state, try to manually activate it. ```sql ALTER MATERIALIZED VIEW $mv_name ACTIVE; ``` 4. Investigate the cause of the refresh failure. ```sql SELECT * FROM information_schema.task_runs WHERE task_name ='mv-112517' \G ``` --- ### Metric details #### [📄️ Metrics a through c](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/a-c.md) [Alphabetical a - c](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/a-c.md) #### [📄️ Metrics d through h](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/d-h.md) [Alphabetical d - h](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/d-h.md) #### [📄️ Metrics i through p](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/i-p.md) [Alphabetical i - p](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/i-p.md) #### [📄️ Metrics q through r](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/q-r.md) [Alphabetical q - r](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/q-r.md) #### [📄️ Metrics s](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/s.md) [Alphabetical s](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/s.md) #### [📄️ Metrics t through z](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/t-z.md) [Alphabetical t - z](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/t-z.md) --- ### Metrics a through c note Metrics for materialized views and shared-data clusters are detailed in the corresponding sections: * [Metrics for asynchronous materialized view metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics-materialized_view.md) * [Metrics for Shared-data Dashboard metrics, and Starlet Dashboard metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics-shared-data.md) For more information on how to build a monitoring service for your StarRocks cluster, see [Monitor and Alert](https://docs.starrocks.io/docs/administration/management/monitoring.md). #### `active_scan_context_count`[​](#active_scan_context_count "Direct link to active_scan_context_count") * Unit: Count * Description: Total number of scan tasks created by Flink/Spark SQL. #### `async_delta_writer_queue_count`[​](#async_delta_writer_queue_count "Direct link to async_delta_writer_queue_count") * Unit: Count * Description: Queued task count in the tablet delta writer thread pool. #### `base_compaction_task_byte_per_second`[​](#base_compaction_task_byte_per_second "Direct link to base_compaction_task_byte_per_second") * Unit: Bytes/s * Description: Estimated rate of base compactions. #### `base_compaction_task_cost_time_ms`[​](#base_compaction_task_cost_time_ms "Direct link to base_compaction_task_cost_time_ms") * Unit: ms * Description: Total time spent on base compactions. #### `be_base_compaction_bytes_per_second`[​](#be_base_compaction_bytes_per_second "Direct link to be_base_compaction_bytes_per_second") * Unit: Bytes/s * Type: Average * Description: Base compaction speed of BE. #### `be_base_compaction_failed`[​](#be_base_compaction_failed "Direct link to be_base_compaction_failed") * Unit: Count/s * Type: Average * Description: Base compaction failure of BE. #### `be_base_compaction_requests`[​](#be_base_compaction_requests "Direct link to be_base_compaction_requests") * Unit: Count/s * Type: Average * Description: Base compaction request of BE. #### `be_base_compaction_rowsets_per_second`[​](#be_base_compaction_rowsets_per_second "Direct link to be_base_compaction_rowsets_per_second") * Unit: Count * Type: Average * Description: Base compaction speed of BE rowsets. #### `be_broker_count`[​](#be_broker_count "Direct link to be_broker_count") * Unit: Count * Type: Average * Description: Number of brokers. #### `be_brpc_endpoint_count`[​](#be_brpc_endpoint_count "Direct link to be_brpc_endpoint_count") * Unit: Count * Type: Average * Description: Number of StubCache in bRPC. #### `be_bytes_read_per_second`[​](#be_bytes_read_per_second "Direct link to be_bytes_read_per_second") * Unit: Bytes/s * Type: Average * Description: Read speed of BE. #### `be_bytes_written_per_second`[​](#be_bytes_written_per_second "Direct link to be_bytes_written_per_second") * Unit: Bytes/s * Type: Average * Description: Write speed of BE. #### `be_clone_failed`[​](#be_clone_failed "Direct link to be_clone_failed") * Unit: Count/s * Type: Average * Description: BE clone failure. #### `be_clone_total_requests`[​](#be_clone_total_requests "Direct link to be_clone_total_requests") * Unit: Count/s * Type: Average * Description: Clone request of BE. #### `be_create_rollup_failed`[​](#be_create_rollup_failed "Direct link to be_create_rollup_failed") * Unit: Count/s * Type: Average * Description: Materialized view creation failure of BE. #### `be_create_rollup_requests`[​](#be_create_rollup_requests "Direct link to be_create_rollup_requests") * Unit: Count/s * Type: Average * Description: Materialized view creation request of BE. #### `be_create_tablet_failed`[​](#be_create_tablet_failed "Direct link to be_create_tablet_failed") * Unit: Count/s * Type: Average * Description: Tablet creation failure of BE. #### `be_create_tablet_requests`[​](#be_create_tablet_requests "Direct link to be_create_tablet_requests") * Unit: Count/s * Type: Average * Description: Tablet creation request of BE. #### `be_cumulative_compaction_bytes_per_second`[​](#be_cumulative_compaction_bytes_per_second "Direct link to be_cumulative_compaction_bytes_per_second") * Unit: Bytes/s * Type: Average * Description: Cumulative compaction speed of BE. #### `be_cumulative_compaction_failed`[​](#be_cumulative_compaction_failed "Direct link to be_cumulative_compaction_failed") * Unit: Count/s * Type: Average * Description: Cumulative compaction failure of BE. #### `be_cumulative_compaction_requests`[​](#be_cumulative_compaction_requests "Direct link to be_cumulative_compaction_requests") * Unit: Count/s * Type: Average * Description: Cumulative compaction request of BE. #### `be_cumulative_compaction_rowsets_per_second`[​](#be_cumulative_compaction_rowsets_per_second "Direct link to be_cumulative_compaction_rowsets_per_second") * Unit: Count * Type: Average * Description: Cumulative compaction speed of BE rowsets. #### `be_delete_failed`[​](#be_delete_failed "Direct link to be_delete_failed") * Unit: Count/s * Type: Average * Description: Delete failure of BE. #### `be_delete_requests`[​](#be_delete_requests "Direct link to be_delete_requests") * Unit: Count/s * Type: Average * Description: Delete request of BE. #### `be_finish_task_failed`[​](#be_finish_task_failed "Direct link to be_finish_task_failed") * Unit: Count/s * Type: Average * Description: Task failure of BE. #### `be_finish_task_requests`[​](#be_finish_task_requests "Direct link to be_finish_task_requests") * Unit: Count/s * Type: Average * Description: Task finish request of BE. #### `be_fragment_endpoint_count`[​](#be_fragment_endpoint_count "Direct link to be_fragment_endpoint_count") * Unit: Count * Type: Average * Description: Number of BE DataStream. #### `be_fragment_request_latency_avg`[​](#be_fragment_request_latency_avg "Direct link to be_fragment_request_latency_avg") * Unit: ms * Type: Average * Description: Latency of fragment requests. #### `be_fragment_requests_per_second`[​](#be_fragment_requests_per_second "Direct link to be_fragment_requests_per_second") * Unit: Count/s * Type: Average * Description: Number of fragment requests. #### `be_http_request_latency_avg`[​](#be_http_request_latency_avg "Direct link to be_http_request_latency_avg") * Unit: ms * Type: Average * Description: Latency of HTTP requests. #### `be_http_request_send_bytes_per_second`[​](#be_http_request_send_bytes_per_second "Direct link to be_http_request_send_bytes_per_second") * Unit: Bytes/s * Type: Average * Description: Number of bytes sent for HTTP requests. #### `be_http_requests_per_second`[​](#be_http_requests_per_second "Direct link to be_http_requests_per_second") * Unit: Count/s * Type: Average * Description: Number of HTTP requests. #### `be_publish_failed`[​](#be_publish_failed "Direct link to be_publish_failed") * Unit: Count/s * Type: Average * Description: Version release failure of BE. #### `be_publish_requests`[​](#be_publish_requests "Direct link to be_publish_requests") * Unit: Count/s * Type: Average * Description: Version publish request of BE. #### `be_report_disk_failed`[​](#be_report_disk_failed "Direct link to be_report_disk_failed") * Unit: Count/s * Type: Average * Description: Disk report failure of BE. #### `be_report_disk_requests`[​](#be_report_disk_requests "Direct link to be_report_disk_requests") * Unit: Count/s * Type: Average * Description: Disk report request of BE. #### `be_report_tables_failed`[​](#be_report_tables_failed "Direct link to be_report_tables_failed") * Unit: Count/s * Type: Average * Description: Table report failure of BE. #### `be_report_tablet_failed`[​](#be_report_tablet_failed "Direct link to be_report_tablet_failed") * Unit: Count/s * Type: Average * Description: Tablet report failure of BE. #### `be_report_tablet_requests`[​](#be_report_tablet_requests "Direct link to be_report_tablet_requests") * Unit: Count/s * Type: Average * Description: Tablet report request of BE. #### `be_report_tablets_requests`[​](#be_report_tablets_requests "Direct link to be_report_tablets_requests") * Unit: Count/s * Type: Average * Description: Tablet report request of BE. #### `be_report_task_failed`[​](#be_report_task_failed "Direct link to be_report_task_failed") * Unit: Count/s * Type: Average * Description: Task report failure of BE. #### `be_report_task_requests`[​](#be_report_task_requests "Direct link to be_report_task_requests") * Unit: Count/s * Type: Average * Description: Task report request of BE. #### `be_schema_change_failed`[​](#be_schema_change_failed "Direct link to be_schema_change_failed") * Unit: Count/s * Type: Average * Description: Schema change failure of BE. #### `be_schema_change_requests`[​](#be_schema_change_requests "Direct link to be_schema_change_requests") * Unit: Count/s * Type: Average * Description: Schema change report request of BE. #### `be_storage_migrate_requests`[​](#be_storage_migrate_requests "Direct link to be_storage_migrate_requests") * Unit: Count/s * Type: Average * Description: Migration request of BE. #### `binary_column_pool_bytes`[​](#binary_column_pool_bytes "Direct link to binary_column_pool_bytes") * Unit: Bytes * Description: Memory used by the BINARY column pool. #### `bitmap_index_mem_bytes`[​](#bitmap_index_mem_bytes "Direct link to bitmap_index_mem_bytes") * Unit: Bytes * Description: Memory used by bitmap indexes. #### `block_cache_hit_bytes`[​](#block_cache_hit_bytes "Direct link to block_cache_hit_bytes") * Unit: Bytes * Type: Counter * Description: Cumulative bytes of block cache hits. For now, only the cache hit bytes for external table is being counted. #### `block_cache_miss_bytes`[​](#block_cache_miss_bytes "Direct link to block_cache_miss_bytes") * Unit: Bytes * Type: Counter * Description: Cumulative bytes of block cache misses. For now, only the cache miss bytes for external table is being counted. #### `block_cache_hit_count`[​](#block_cache_hit_count "Direct link to block_cache_hit_count") * Unit: Count * Type: Counter * Description: Cumulative count of block cache hits. #### `block_cache_miss_count`[​](#block_cache_miss_count "Direct link to block_cache_miss_count") * Unit: Count * Type: Counter * Description: Cumulative count of block cache misses. #### `blocks_created_total (Deprecated)`[​](#blocks_created_total-deprecated "Direct link to blocks_created_total-deprecated") #### `blocks_deleted_total (Deprecated)`[​](#blocks_deleted_total-deprecated "Direct link to blocks_deleted_total-deprecated") #### `blocks_open_reading (Deprecated)`[​](#blocks_open_reading-deprecated "Direct link to blocks_open_reading-deprecated") #### `blocks_open_writing (Deprecated)`[​](#blocks_open_writing-deprecated "Direct link to blocks_open_writing-deprecated") #### `bloom_filter_index_mem_bytes`[​](#bloom_filter_index_mem_bytes "Direct link to bloom_filter_index_mem_bytes") * Unit: Bytes * Description: Memory used by Bloomfilter indexes. #### `broker_count`[​](#broker_count "Direct link to broker_count") * Unit: Count * Description: Total number of filesystem brokers (by host address) created. #### `brpc_endpoint_stub_count`[​](#brpc_endpoint_stub_count "Direct link to brpc_endpoint_stub_count") * Unit: Count * Description: Total number of bRPC stubs (by address). #### `builtin_inverted_index_mem_bytes`[​](#builtin_inverted_index_mem_bytes "Direct link to builtin_inverted_index_mem_bytes") * Unit: Bytes * Description: Memory used by builtin inverted indexes. #### `bytes_read_total (Deprecated)`[​](#bytes_read_total-deprecated "Direct link to bytes_read_total-deprecated") #### `bytes_written_total`[​](#bytes_written_total "Direct link to bytes_written_total") * Unit: Bytes * Description: Total bytes written (sectors write \* 512). #### `central_column_pool_bytes (Deprecated)`[​](#central_column_pool_bytes-deprecated "Direct link to central_column_pool_bytes-deprecated") #### `chunk_allocator_mem_bytes`[​](#chunk_allocator_mem_bytes "Direct link to chunk_allocator_mem_bytes") * Unit: Bytes * Description: Memory used by the chunk allocator. #### `chunk_pool_local_core_alloc_count`[​](#chunk_pool_local_core_alloc_count "Direct link to chunk_pool_local_core_alloc_count") * Unit: Count * Description: Memory chunk allocation/cache metrics. #### `chunk_pool_other_core_alloc_count`[​](#chunk_pool_other_core_alloc_count "Direct link to chunk_pool_other_core_alloc_count") * Unit: Count * Description: Memory chunk allocation/cache metrics. #### `chunk_pool_system_alloc_cost_ns`[​](#chunk_pool_system_alloc_cost_ns "Direct link to chunk_pool_system_alloc_cost_ns") * Unit: ns * Description: Memory chunk allocation/cache metrics. #### `chunk_pool_system_alloc_count`[​](#chunk_pool_system_alloc_count "Direct link to chunk_pool_system_alloc_count") * Unit: Count * Description: Memory chunk allocation/cache metrics. #### `chunk_pool_system_free_cost_ns`[​](#chunk_pool_system_free_cost_ns "Direct link to chunk_pool_system_free_cost_ns") * Unit: ns * Description: Memory chunk allocation/cache metrics. #### `chunk_pool_system_free_count`[​](#chunk_pool_system_free_count "Direct link to chunk_pool_system_free_count") * Unit: Count * Description: Memory chunk allocation/cache metrics. #### `clone_mem_bytes`[​](#clone_mem_bytes "Direct link to clone_mem_bytes") * Unit: Bytes * Description: Memory used for replica clone. #### `cluster_snapshot_consecutive_failures`[​](#cluster_snapshot_consecutive_failures "Direct link to cluster_snapshot_consecutive_failures") * Unit: Count * Description: Number of consecutive failed automated cluster snapshot jobs since the last successful one. A persistently increasing value indicates the snapshot storage volume is misconfigured or unreachable. #### `cluster_snapshot_last_finished_time`[​](#cluster_snapshot_last_finished_time "Direct link to cluster_snapshot_last_finished_time") * Unit: Milliseconds * Description: Epoch time (in milliseconds) of the last finished automated cluster snapshot, or 0 if none has finished. #### `column_metadata_mem_bytes`[​](#column_metadata_mem_bytes "Direct link to column_metadata_mem_bytes") * Unit: Bytes * Description: Memory used by column metadata. #### `column_partial_update_apply_duration_us`[​](#column_partial_update_apply_duration_us "Direct link to column_partial_update_apply_duration_us") * Unit: us * Description: Total time spent on partial updates for columns' APPLY tasks (Column mode). #### `column_partial_update_apply_total`[​](#column_partial_update_apply_total "Direct link to column_partial_update_apply_total") * Unit: Count * Description: Total number of APPLY for partial updates by column (Column mode) #### `column_pool_mem_bytes`[​](#column_pool_mem_bytes "Direct link to column_pool_mem_bytes") * Unit: Bytes * Description: Memory used by the column pools. #### `column_zonemap_index_mem_bytes`[​](#column_zonemap_index_mem_bytes "Direct link to column_zonemap_index_mem_bytes") * Unit: Bytes * Description: Memory used by column zonemaps. #### `compaction_bytes_total`[​](#compaction_bytes_total "Direct link to compaction_bytes_total") * Unit: Bytes * Description: Total merged bytes from base compactions and cumulative compactions. #### `compaction_deltas_total`[​](#compaction_deltas_total "Direct link to compaction_deltas_total") * Unit: Count * Description: Total number of merged rowsets from base compactions and cumulative compactions. #### `compaction_mem_bytes`[​](#compaction_mem_bytes "Direct link to compaction_mem_bytes") * Unit: Bytes * Description: Memory used by compactions. #### `consistency_mem_bytes`[​](#consistency_mem_bytes "Direct link to consistency_mem_bytes") * Unit: Bytes * Description: Memory used by replica consistency checks. #### `cpu`[​](#cpu "Direct link to cpu") * Unit: - * Description: CPU usage information returned by `/proc/stat`. #### `cpu_guest`[​](#cpu_guest "Direct link to cpu_guest") * Unit: - * Type: Average * Description: cpu\_guest usage rate. #### `cpu_idle`[​](#cpu_idle "Direct link to cpu_idle") * Unit: - * Type: Average * Description: cpu\_idle usage rate. #### `cpu_iowait`[​](#cpu_iowait "Direct link to cpu_iowait") * Unit: - * Type: Average * Description: cpu\_iowait usage rate. #### `cpu_irq`[​](#cpu_irq "Direct link to cpu_irq") * Unit: - * Type: Average * Description: cpu\_irq usage rate. #### `cpu_nice`[​](#cpu_nice "Direct link to cpu_nice") * Unit: - * Type: Average * Description: cpu\_nice usage rate. #### `cpu_softirq`[​](#cpu_softirq "Direct link to cpu_softirq") * Unit: - * Type: Average * Description: cpu\_softirq usage rate. #### `cpu_steal`[​](#cpu_steal "Direct link to cpu_steal") * Unit: - * Type: Average * Description: cpu\_steal usage rate. #### `cpu_system`[​](#cpu_system "Direct link to cpu_system") * Unit: - * Type: Average * Description: cpu\_system usage rate. #### `cpu_user`[​](#cpu_user "Direct link to cpu_user") * Unit: - * Type: Average * Description: cpu\_user usage rate. #### `cpu_util`[​](#cpu_util "Direct link to cpu_util") * Unit: - * Type: Average * Description: CPU usage rate. #### `cumulative_compaction_task_byte_per_second`[​](#cumulative_compaction_task_byte_per_second "Direct link to cumulative_compaction_task_byte_per_second") * Unit: Bytes/s * Description: Rate of bytes processed during cumulative compactions. #### `cumulative_compaction_task_cost_time_ms`[​](#cumulative_compaction_task_cost_time_ms "Direct link to cumulative_compaction_task_cost_time_ms") * Unit: ms * Description: Total time spent on cumulative compactions. --- ### Metrics d through h note Metrics for materialized views and shared-data clusters are detailed in the corresponding sections: * [Metrics for asynchronous materialized view metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics-materialized_view.md) * [Metrics for Shared-data Dashboard metrics, and Starlet Dashboard metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics-shared-data.md) For more information on how to build a monitoring service for your StarRocks cluster, see [Monitor and Alert](https://docs.starrocks.io/docs/administration/management/monitoring.md). #### `data_stream_receiver_count`[​](#data_stream_receiver_count "Direct link to data_stream_receiver_count") * Unit: Count * Description: Cumulative number of instances serving as Exchange receivers in BE. #### `datacache_disk_quota_bytes`[​](#datacache_disk_quota_bytes "Direct link to datacache_disk_quota_bytes") * Unit: Bytes * Type: Gauge * Description: The configured disk quota for datacache. #### `datacache_disk_used_bytes`[​](#datacache_disk_used_bytes "Direct link to datacache_disk_used_bytes") * Unit: Bytes * Type: Gauge * Description: The current disk usage of datacache. #### `datacache_mem_quota_bytes`[​](#datacache_mem_quota_bytes "Direct link to datacache_mem_quota_bytes") * Unit: Bytes * Type: Gauge * Description: The configured memory quota for datacache. #### `datacache_mem_used_bytes`[​](#datacache_mem_used_bytes "Direct link to datacache_mem_used_bytes") * Unit: Bytes * Type: Gauge * Description: The current memory usage of datacache. #### `datacache_meta_used_bytes`[​](#datacache_meta_used_bytes "Direct link to datacache_meta_used_bytes") * Unit: Bytes * Type: Gauge * Description: The memory usage for datacache metadata. #### `date_column_pool_bytes`[​](#date_column_pool_bytes "Direct link to date_column_pool_bytes") * Unit: Bytes * Description: Memory used by the DATE column pool. #### `datetime_column_pool_bytes`[​](#datetime_column_pool_bytes "Direct link to datetime_column_pool_bytes") * Unit: Bytes * Description: Memory used by the DATETIME column pool. #### `decimal_column_pool_bytes`[​](#decimal_column_pool_bytes "Direct link to decimal_column_pool_bytes") * Unit: Bytes * Description: Memory used by the DECIMAL column pool. #### `default_cache_hit_bytes`[​](#default_cache_hit_bytes "Direct link to default_cache_hit_bytes") * Unit: Bytes * Type: Cumulative * Description: Cumulative bytes read from the Data Cache instance `default_cache` on cache hits. In a shared-data cluster where [`datacache_unified_instance_enable`](https://docs.starrocks.io/docs/administration/configuration/BE_parameters/shared_lake_other.md#datacache_unified_instance_enable) is `true` (default), this instance also caches the data of cloud-native tables, and for cloud-native tables this metric counts the same reads as `fslib_open_cache_hits`. Data Cache reads of external tables on the same node are also counted. If `datacache_unified_instance_enable` is `false`, cloud-native tables use a separate cache instance, whose metrics are exported as `cloud_cache_hit_bytes`, `cloud_cache_miss_bytes`, `cloud_cache_hit_count`, and `cloud_cache_miss_count`. This metric is exported only when the BE configuration item `dump_metrics_with_bvar` is `true` (default). It is exported with the Prometheus type `gauge`, but its value only increases until the BE or CN restarts, so use `rate()` on it. The byte-based cache hit ratio is `sum(rate(default_cache_hit_bytes[5m])) / (sum(rate(default_cache_hit_bytes[5m])) + sum(rate(default_cache_miss_bytes[5m])))`. Unlike the count-based hit ratio calculated from `fslib_open_cache_hits` and `fslib_open_cache_misses`, it is not lowered much by many small misses. #### `default_cache_hit_count`[​](#default_cache_hit_count "Direct link to default_cache_hit_count") * Unit: Count * Type: Cumulative * Description: Cumulative number of cache hits of the Data Cache instance `default_cache`. For cloud-native tables, it counts the same reads as `fslib_open_cache_hits`. For the scope and export conditions, see `default_cache_hit_bytes`. #### `default_cache_miss_bytes`[​](#default_cache_miss_bytes "Direct link to default_cache_miss_bytes") * Unit: Bytes * Type: Cumulative * Description: Cumulative bytes requested from the Data Cache instance `default_cache` on cache misses. It is the size that the read requested, not the size then read from object storage, which is aligned to cache blocks. A miss on a cached file size entry adds 0. For the scope and export conditions, see `default_cache_hit_bytes`. #### `default_cache_miss_count`[​](#default_cache_miss_count "Direct link to default_cache_miss_count") * Unit: Count * Type: Cumulative * Description: Cumulative number of cache misses of the Data Cache instance `default_cache`. For cloud-native tables, it counts the same reads as `fslib_open_cache_misses`. For the scope and export conditions, see `default_cache_hit_bytes`. #### `delta_column_group_get_hit_cache`[​](#delta_column_group_get_hit_cache "Direct link to delta_column_group_get_hit_cache") * Unit: Count * Description: Total number of delta column group cache hits (for Primary Key tables only). #### `delta_column_group_get_non_pk_hit_cache`[​](#delta_column_group_get_non_pk_hit_cache "Direct link to delta_column_group_get_non_pk_hit_cache") * Unit: Count * Description: Total number of hits in the delta column group cache (for non-Primary Key tables). #### `delta_column_group_get_non_pk_total`[​](#delta_column_group_get_non_pk_total "Direct link to delta_column_group_get_non_pk_total") * Unit: Count * Description: Total number of times to get delta column group (for non-Primary Key tables only). #### `delta_column_group_get_total`[​](#delta_column_group_get_total "Direct link to delta_column_group_get_total") * Unit: Count * Description: Total number of times to get delta column groups (for Primary Key tables). #### `disk_bytes_read`[​](#disk_bytes_read "Direct link to disk_bytes_read") * Unit: Bytes * Description: Total bytes read from the disk(sectors read \* 512). #### `disk_bytes_written`[​](#disk_bytes_written "Direct link to disk_bytes_written") * Unit: Bytes * Description: Total bytes written to disk. #### `disk_free`[​](#disk_free "Direct link to disk_free") * Unit: Bytes * Type: Average * Description: Free disk capacity. #### `disk_io_svctm`[​](#disk_io_svctm "Direct link to disk_io_svctm") * Unit: ms * Type: Average * Description: Disk IO service time. #### `disk_io_time_ms`[​](#disk_io_time_ms "Direct link to disk_io_time_ms") * Unit: ms * Description: Time spent on I/Os. #### `disk_io_time_weigthed`[​](#disk_io_time_weigthed "Direct link to disk_io_time_weigthed") * Unit: ms * Description: Weighted time spent on I/Os. #### `disk_io_util`[​](#disk_io_util "Direct link to disk_io_util") * Unit: - * Type: Average * Description: Disk usage. #### `disk_read_time_ms`[​](#disk_read_time_ms "Direct link to disk_read_time_ms") * Unit: ms * Description: Time spent on reading from disk. Unit: ms. #### `disk_reads_completed`[​](#disk_reads_completed "Direct link to disk_reads_completed") * Unit: Count * Description: Number of successfully completed disk reads. #### `disk_sync_total (Deprecated)`[​](#disk_sync_total-deprecated "Direct link to disk_sync_total-deprecated") #### `disk_used`[​](#disk_used "Direct link to disk_used") * Unit: Bytes * Type: Average * Description: Used disk capacity. #### `disk_write_time_ms`[​](#disk_write_time_ms "Direct link to disk_write_time_ms") * Unit: ms * Description: Time spent on disk writing. Unit: ms. #### `disk_writes_completed`[​](#disk_writes_completed "Direct link to disk_writes_completed") * Unit: Count * Description: Total number of successfully completed disk writes. #### `disks_avail_capacity`[​](#disks_avail_capacity "Direct link to disks_avail_capacity") * Description: Available capacity of a specific disk. #### `disks_data_used_capacity`[​](#disks_data_used_capacity "Direct link to disks_data_used_capacity") * Description: Used capacity of each disk (represented by a storage path). #### `disks_state`[​](#disks_state "Direct link to disks_state") * Unit: - * Description: State of each disk. `1` indicates that the disk is in use, and `0` indicates that it is not in use. #### `disks_total_capacity`[​](#disks_total_capacity "Direct link to disks_total_capacity") * Description: Total capacity of the disk. #### `double_column_pool_bytes`[​](#double_column_pool_bytes "Direct link to double_column_pool_bytes") * Unit: Bytes * Description: Memory used by the DOUBLE column pool. #### `encryption_keys_created`[​](#encryption_keys_created "Direct link to encryption_keys_created") * Unit: Count * Type: Cumulative * Description: number of file encryption keys created for file encryption #### `encryption_keys_in_cache`[​](#encryption_keys_in_cache "Direct link to encryption_keys_in_cache") * Unit: Count * Type: Instantaneous * Description: number of encryption keys currently in key cache #### `encryption_keys_unwrapped`[​](#encryption_keys_unwrapped "Direct link to encryption_keys_unwrapped") * Unit: Count * Type: Cumulative * Description: number of encryption meta unwrapped for file decryption #### `engine_requests_total`[​](#engine_requests_total "Direct link to engine_requests_total") * Unit: Count * Description: Total count of all types of requests between BE and FE, including CREATE TABLE, Publish Version, and tablet clone. #### `fd_num_limit`[​](#fd_num_limit "Direct link to fd_num_limit") * Unit: Count * Description: Maximum number of file descriptors. #### `fd_num_used`[​](#fd_num_used "Direct link to fd_num_used") * Unit: Count * Description: Number of file descriptors currently in use. #### `fe_cancelled_broker_load_job`[​](#fe_cancelled_broker_load_job "Direct link to fe_cancelled_broker_load_job") * Unit: Count * Type: Average * Description: Number of cancelled broker jobs. #### `fe_cancelled_delete_load_job`[​](#fe_cancelled_delete_load_job "Direct link to fe_cancelled_delete_load_job") * Unit: Count * Type: Average * Description: Number of cancelled delete jobs. #### `fe_cancelled_hadoop_load_job`[​](#fe_cancelled_hadoop_load_job "Direct link to fe_cancelled_hadoop_load_job") * Unit: Count * Type: Average * Description: Number of cancelled hadoop jobs. #### `fe_cancelled_insert_load_job`[​](#fe_cancelled_insert_load_job "Direct link to fe_cancelled_insert_load_job") * Unit: Count * Type: Average * Description: Number of cancelled insert jobs. #### `fe_checkpoint_push_per_second`[​](#fe_checkpoint_push_per_second "Direct link to fe_checkpoint_push_per_second") * Unit: Count/s * Type: Average * Description: Number of FE checkpoints. #### `fe_committed_broker_load_job`[​](#fe_committed_broker_load_job "Direct link to fe_committed_broker_load_job") * Unit: Count * Type: Average * Description: Number of committed broker jobs. #### `fe_committed_delete_load_job`[​](#fe_committed_delete_load_job "Direct link to fe_committed_delete_load_job") * Unit: Count * Type: Average * Description: Number of committed delete jobs. #### `fe_committed_hadoop_load_job`[​](#fe_committed_hadoop_load_job "Direct link to fe_committed_hadoop_load_job") * Unit: Count * Type: Average * Description: Number of committed hadoop jobs. #### `fe_committed_insert_load_job`[​](#fe_committed_insert_load_job "Direct link to fe_committed_insert_load_job") * Unit: Count * Type: Average * Description: Number of committed insert jobs. #### `fe_connection_total`[​](#fe_connection_total "Direct link to fe_connection_total") * Unit: Count * Type: Cumulative * Description: Total number of FE connections. #### `fe_connections_per_second`[​](#fe_connections_per_second "Direct link to fe_connections_per_second") * Unit: Count/s * Type: Average * Description: New connection rate of FE. #### `fe_edit_log_read`[​](#fe_edit_log_read "Direct link to fe_edit_log_read") * Unit: Count/s * Type: Average * Description: Read speed of FE edit log. #### `fe_edit_log_size_bytes`[​](#fe_edit_log_size_bytes "Direct link to fe_edit_log_size_bytes") * Unit: Bytes/s * Type: Average * Description: Size of FE edit log. #### `fe_edit_log_write`[​](#fe_edit_log_write "Direct link to fe_edit_log_write") * Unit: Bytes/s * Type: Average * Description: Write speed of FE edit log. #### `fe_finished_broker_load_job`[​](#fe_finished_broker_load_job "Direct link to fe_finished_broker_load_job") * Unit: Count * Type: Average * Description: Number of finished broker jobs. #### `fe_finished_delete_load_job`[​](#fe_finished_delete_load_job "Direct link to fe_finished_delete_load_job") * Unit: Count * Type: Average * Description: Number of finished delete jobs. #### `fe_finished_hadoop_load_job`[​](#fe_finished_hadoop_load_job "Direct link to fe_finished_hadoop_load_job") * Unit: Count * Type: Average * Description: Number of completedhadoop jobs. #### `fe_finished_insert_load_job`[​](#fe_finished_insert_load_job "Direct link to fe_finished_insert_load_job") * Unit: Count * Type: Average * Description: Number of completed insert jobs. #### `fe_loading_broker_load_job`[​](#fe_loading_broker_load_job "Direct link to fe_loading_broker_load_job") * Unit: Count * Type: Average * Description: Number of loading broker jobs. #### `fe_loading_delete_load_job`[​](#fe_loading_delete_load_job "Direct link to fe_loading_delete_load_job") * Unit: Count * Type: Average * Description: Number of loading delete jobs. #### `fe_loading_hadoop_load_job`[​](#fe_loading_hadoop_load_job "Direct link to fe_loading_hadoop_load_job") * Unit: Count * Type: Average * Description: Number of loading hadoop jobs. #### `fe_loading_insert_load_job`[​](#fe_loading_insert_load_job "Direct link to fe_loading_insert_load_job") * Unit: Count * Type: Average * Description: Number of loading insert jobs. #### `fe_pending_broker_load_job`[​](#fe_pending_broker_load_job "Direct link to fe_pending_broker_load_job") * Unit: Count * Type: Average * Description: Number of pending broker jobs. #### `fe_pending_delete_load_job`[​](#fe_pending_delete_load_job "Direct link to fe_pending_delete_load_job") * Unit: Count * Type: Average * Description: Number of pending delete jobs. #### `fe_pending_hadoop_load_job`[​](#fe_pending_hadoop_load_job "Direct link to fe_pending_hadoop_load_job") * Unit: Count * Type: Average * Description: Number of pending hadoop jobs. #### `fe_pending_insert_load_job`[​](#fe_pending_insert_load_job "Direct link to fe_pending_insert_load_job") * Unit: Count * Type: Average * Description: Number of pending insert jobs. #### `fe_rollup_running_alter_job`[​](#fe_rollup_running_alter_job "Direct link to fe_rollup_running_alter_job") * Unit: Count * Type: Average * Description: Number of jobs created in rollup. #### `fe_schema_change_running_job`[​](#fe_schema_change_running_job "Direct link to fe_schema_change_running_job") * Unit: Count * Type: Average * Description: Number of jobs in schema change. #### `float_column_pool_bytes`[​](#float_column_pool_bytes "Direct link to float_column_pool_bytes") * Unit: Bytes * Description: Memory used by the FLOAT column pool. #### `fragment_endpoint_count`[​](#fragment_endpoint_count "Direct link to fragment_endpoint_count") * Unit: Count * Description: Cumulative number of instances serving as Exchange senders in BE. #### `fragment_request_duration_us`[​](#fragment_request_duration_us "Direct link to fragment_request_duration_us") * Unit: us * Description: Cumulative execution time of fragment instances (for non-pipeline engine). #### `fragment_requests_total`[​](#fragment_requests_total "Direct link to fragment_requests_total") * Unit: Count * Description: Total fragment instances executing on a BE (for non-pipeline engine). #### `fslib_open_cache_hits`[​](#fslib_open_cache_hits "Direct link to fslib_open_cache_hits") * Unit: Count * Type: Counter * Labels: `fstype` (`cachefs`) * Description: Cumulative number of Data Cache hits when reading the files of cloud-native tables in a shared-data cluster. Each read covers at most one cache block. Because every read counts as one regardless of its size, many small misses, such as metadata reads, can lower the hit ratio calculated from this metric and `fslib_open_cache_misses` even when most bytes are still read from the cache. For the byte-based metrics, see `default_cache_hit_bytes`. #### `fslib_open_cache_misses`[​](#fslib_open_cache_misses "Direct link to fslib_open_cache_misses") * Unit: Count * Type: Counter * Labels: `fstype` (`cachefs`) * Description: Cumulative number of Data Cache misses when reading the files of cloud-native tables in a shared-data cluster. Each read covers at most one cache block. For the byte-based metrics, see `default_cache_miss_bytes`. #### `hive_write_bytes`[​](#hive_write_bytes "Direct link to hive_write_bytes") * Unit: Bytes * Type: Cumulative * Labels: `write_type` (`insert` or `overwrite`) * Description: Total written bytes from Hive write tasks (`INSERT`, `INSERT OVERWRITE`). This represents the total size of data files written to the Hive table. `write_type` distinguishes between the operation types. #### `hive_write_duration_ms_total`[​](#hive_write_duration_ms_total "Direct link to hive_write_duration_ms_total") * Unit: Millisecond * Type: Cumulative * Labels: `write_type` (`insert` or `overwrite`) * Description: Total execution time of Hive write tasks (`INSERT`, `INSERT OVERWRITE`) in milliseconds. The duration of each task is added after it ends. `write_type` distinguishes between the operation types. #### `hive_write_files`[​](#hive_write_files "Direct link to hive_write_files") * Unit: Count * Type: Cumulative * Labels: `write_type` (`insert` or `overwrite`) * Description: Total number of data files written to Hive from write tasks (`INSERT`, `INSERT OVERWRITE`). This represents the count of data files written to the Hive table. `write_type` distinguishes between the operation types. #### `hive_write_rows`[​](#hive_write_rows "Direct link to hive_write_rows") * Unit: Rows * Type: Cumulative * Labels: `write_type` (`insert` or `overwrite`) * Description: Total written rows from Hive write tasks (`INSERT`, `INSERT OVERWRITE`). This represents the number of rows written to the Hive table. `write_type` distinguishes between the operation types. #### `hive_write_total`[​](#hive_write_total "Direct link to hive_write_total") * Unit: Count * Type: Cumulative * Labels: * `status` (`success` or `failed`) * `reason` (`none`, `timeout`, `oom`, `access_denied`, `unknown`) * `write_type` (`insert` or `overwrite`) * Description: Total number of `INSERT` or `INSERT OVERWRITE` tasks that target Hive tables. The metric is incremented by 1 after each task ends, regardless of success or failure. `write_type` distinguishes between the operation types. #### `http_request_send_bytes (Deprecated)`[​](#http_request_send_bytes-deprecated "Direct link to http_request_send_bytes-deprecated") #### `http_requests_total (Deprecated)`[​](#http_requests_total-deprecated "Direct link to http_requests_total-deprecated") --- ### Metrics i through p note Metrics for materialized views and shared-data clusters are detailed in the corresponding sections: * [Metrics for asynchronous materialized view metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics-materialized_view.md) * [Metrics for Shared-data Dashboard metrics, and Starlet Dashboard metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics-shared-data.md) For more information on how to build a monitoring service for your StarRocks cluster, see [Monitor and Alert](https://docs.starrocks.io/docs/administration/management/monitoring.md). #### `iceberg_compaction_duration_ms_total`[​](#iceberg_compaction_duration_ms_total "Direct link to iceberg_compaction_duration_ms_total") * Unit: Millisecond * Type: Cumulative * Labels: `compaction_type` (`manual` or `auto`) * Description: Total time spent running Iceberg compaction tasks. #### `iceberg_compaction_input_files_total`[​](#iceberg_compaction_input_files_total "Direct link to iceberg_compaction_input_files_total") * Unit: Count * Type: Cumulative * Labels: `compaction_type` (`manual` or `auto`) * Description: Total number of data files read by Iceberg compaction tasks. #### `iceberg_compaction_output_files_total`[​](#iceberg_compaction_output_files_total "Direct link to iceberg_compaction_output_files_total") * Unit: Count * Type: Cumulative * Labels: `compaction_type` (`manual` or `auto`) * Description: Total number of data files produced by Iceberg compaction tasks. #### `iceberg_compaction_removed_delete_files_total`[​](#iceberg_compaction_removed_delete_files_total "Direct link to iceberg_compaction_removed_delete_files_total") * Unit: Count * Type: Cumulative * Labels: `compaction_type` (`manual` or `auto`) * Description: Total number of delete files removed by Iceberg manual compaction tasks. #### `iceberg_compaction_total`[​](#iceberg_compaction_total "Direct link to iceberg_compaction_total") * Unit: Count * Type: Cumulative * Labels: `compaction_type` (`manual` or `auto`) * Description: Total number of Iceberg compaction (`rewrite_data_files`) tasks. #### `iceberg_delete_bytes`[​](#iceberg_delete_bytes "Direct link to iceberg_delete_bytes") * Unit: Bytes * Type: Cumulative * Labels: `delete_type` (`position` or `metadata`) * Description: Total deleted bytes from Iceberg `DELETE` tasks. For `metadata` delete, this represents the size of deleted data files. For `position` delete, this represents the size of position delete files created. #### `iceberg_delete_duration_ms_total`[​](#iceberg_delete_duration_ms_total "Direct link to iceberg_delete_duration_ms_total") * Unit: Millisecond * Type: Cumulative * Labels: `delete_type` (`position` or `metadata`) * Description: Total execution time of Iceberg `DELETE` tasks in milliseconds. The duration of each task is added after it ends. `delete_type` distinguishes between two delete methods. #### `iceberg_delete_rows`[​](#iceberg_delete_rows "Direct link to iceberg_delete_rows") * Unit: Rows * Type: Cumulative * Labels: `delete_type` (`position` or `metadata`) * Description: Total deleted rows from Iceberg `DELETE` tasks. For `metadata` delete, this represents the number of rows in deleted data files. For `position` delete, this represents the number of position deletes created. #### `iceberg_delete_total`[​](#iceberg_delete_total "Direct link to iceberg_delete_total") * Unit: Count * Type: Cumulative * Labels: * `status` (`success` or `failed`) * `reason` (`none`, `timeout`, `oom`, `access_denied`, `unknown`) * `delete_type` (`position` or `metadata`) * Description: Total number of `DELETE` tasks that target Iceberg tables. The metric is incremented by 1 after each task ends, regardless of success or failure. `delete_type` distinguishes between two delete methods: `position` (generates position delete files) and `metadata` (metadata-level delete). #### `iceberg_metadata_table_query_total`[​](#iceberg_metadata_table_query_total "Direct link to iceberg_metadata_table_query_total") * Unit: Count * Type: Cumulative * Labels: `metadata_table` (`refs`, `history`, `metadata_log_entries`, `snapshots`, `manifests`, `files`, `partitions`, or `properties`) * Description: Total number of SQL queries that access Iceberg metadata tables. Each query is counted under the `metadata_table` label that identifies the metadata table being accessed. #### `iceberg_time_travel_query_total`[​](#iceberg_time_travel_query_total "Direct link to iceberg_time_travel_query_total") * Unit: Count * Type: Cumulative * Labels: `time_travel_type` (`branch`, `tag`, `snapshot`, or `timestamp`) for the categorized series. * Description: Total number of Iceberg time travel queries. The unlabeled series counts each time travel query once. The labeled series count each distinct time travel type used by the query. `snapshot` means `FOR VERSION AS OF `, `branch` and `tag` mean `FOR VERSION AS OF `, and `timestamp` means `FOR TIMESTAMP AS OF ...`. #### `iceberg_write_bytes`[​](#iceberg_write_bytes "Direct link to iceberg_write_bytes") * Unit: Bytes * Type: Cumulative * Labels: `write_type` (`insert`, `overwrite`, or `ctas`) * Description: Total written bytes from Iceberg write tasks (`INSERT`, `INSERT OVERWRITE`, `CTAS`). This represents the total size of data files written to the Iceberg table. `write_type` distinguishes between the operation types. #### `iceberg_write_duration_ms_total`[​](#iceberg_write_duration_ms_total "Direct link to iceberg_write_duration_ms_total") * Unit: Millisecond * Type: Cumulative * Labels: `write_type` (`insert`, `overwrite`, or `ctas`) * Description: Total execution time of Iceberg write tasks (`INSERT`, `INSERT OVERWRITE`, `CTAS`) in milliseconds. The duration of each task is added after it ends. `write_type` distinguishes between the operation types. #### `iceberg_write_files`[​](#iceberg_write_files "Direct link to iceberg_write_files") * Unit: Count * Type: Cumulative * Labels: `write_type` (`insert`, `overwrite`, or `ctas`) * Description: Total number of data files written to Iceberg from write tasks (`INSERT`, `INSERT OVERWRITE`, `CTAS`). This represents the count of data files written to the Iceberg table. `write_type` distinguishes between the operation types. #### `iceberg_write_rows`[​](#iceberg_write_rows "Direct link to iceberg_write_rows") * Unit: Rows * Type: Cumulative * Labels: `write_type` (`insert`, `overwrite`, or `ctas`) * Description: Total written rows from Iceberg write tasks (`INSERT`, `INSERT OVERWRITE`, `CTAS`). This represents the number of rows written to the Iceberg table. `write_type` distinguishes between the operation types. #### `iceberg_write_total`[​](#iceberg_write_total "Direct link to iceberg_write_total") * Unit: Count * Type: Cumulative * Labels: * `status` (`success` or `failed`) * `reason` (`none`, `timeout`, `oom`, `access_denied`, `unknown`) * `write_type` (`insert`, `overwrite`, or `ctas`) * Description: Total number of `INSERT`, `INSERT OVERWRITE`, or `CTAS` tasks that target Iceberg tables. The metric is incremented by 1 after each task ends, regardless of success or failure. `write_type` distinguishes between the operation types. #### `int128_column_pool_bytes`[​](#int128_column_pool_bytes "Direct link to int128_column_pool_bytes") * Unit: Bytes * Description: Memory used by the INT128 column pool. #### `int16_column_pool_bytes`[​](#int16_column_pool_bytes "Direct link to int16_column_pool_bytes") * Unit: Bytes * Description: Memory used by the INT16 column pool. #### `int32_column_pool_bytes`[​](#int32_column_pool_bytes "Direct link to int32_column_pool_bytes") * Unit: Bytes * Description: Memory used by the INT32 column pool. #### `int64_column_pool_bytes`[​](#int64_column_pool_bytes "Direct link to int64_column_pool_bytes") * Unit: Bytes * Description: Memory used by the INT64 column pool. #### `int8_column_pool_bytes`[​](#int8_column_pool_bytes "Direct link to int8_column_pool_bytes") * Unit: Bytes * Description: Memory used by the INT8 column pool. #### `jemalloc_active_bytes`[​](#jemalloc_active_bytes "Direct link to jemalloc_active_bytes") * Unit: Bytes * Description: Total bytes in active pages allocated by the application. #### `jemalloc_allocated_bytes`[​](#jemalloc_allocated_bytes "Direct link to jemalloc_allocated_bytes") * Unit: Bytes * Description: Total number of bytes allocated by the application. #### `jemalloc_dirty_bytes`[​](#jemalloc_dirty_bytes "Direct link to jemalloc_dirty_bytes") * Unit: Bytes * Description: Total number of bytes in unused dirty pages, which have not yet been `madvise`d back to the operating system and can be reused for new allocations without a page fault. #### `jemalloc_mapped_bytes`[​](#jemalloc_mapped_bytes "Direct link to jemalloc_mapped_bytes") * Unit: Bytes * Description: Total number of bytes in active extents mapped by the allocator. #### `jemalloc_metadata_bytes`[​](#jemalloc_metadata_bytes "Direct link to jemalloc_metadata_bytes") * Unit: Bytes * Description: Total number of bytes dedicated to metadata, comprising base allocations used for bootstrap-sensitive allocator metadata structures and internal allocations. The usage of transparent huge pages is not included in this item. #### `jemalloc_metadata_thp`[​](#jemalloc_metadata_thp "Direct link to jemalloc_metadata_thp") * Unit: Count * Description: Number of Transparent Huge Pages used for metadata. #### `jemalloc_muzzy_bytes`[​](#jemalloc_muzzy_bytes "Direct link to jemalloc_muzzy_bytes") * Unit: Bytes * Description: Total number of bytes in unused muzzy pages, an intermediate decay state between dirty and retained where the pages have been `madvise`d (for example with `MADV_FREE`) but the mapping is still retained. #### `jemalloc_resident_bytes`[​](#jemalloc_resident_bytes "Direct link to jemalloc_resident_bytes") * Unit: Bytes * Description: Maximum number of bytes in physically resident data pages mapped by the allocator, comprising all pages dedicated to allocator metadata, pages backing active allocations, and unused dirty pages. #### `jemalloc_retained_bytes`[​](#jemalloc_retained_bytes "Direct link to jemalloc_retained_bytes") * Unit: Bytes * Description: Total bytes of virtual memory mappings that were retained rather than returned to the operating system via operations like munmap(2). #### `jit_cache_mem_bytes`[​](#jit_cache_mem_bytes "Direct link to jit_cache_mem_bytes") * Unit: Bytes * Description: Memory used by jit compiled function cache. #### `lake_compaction_failed`[​](#lake_compaction_failed "Direct link to lake_compaction_failed") * Unit: Count * Description: Counter of failed lake compaction jobs. #### `lake_compaction_partial_success`[​](#lake_compaction_partial_success "Direct link to lake_compaction_partial_success") * Unit: Count * Description: Counter of partially successful lake compaction jobs. #### `lake_compaction_running`[​](#lake_compaction_running "Direct link to lake_compaction_running") * Unit: Count * Description: Number of currently running lake compaction jobs. #### `lake_compaction_running_tasks`[​](#lake_compaction_running_tasks "Direct link to lake_compaction_running_tasks") * Unit: Count * Description: Number of tablets currently being compacted across all running shared-data (lake) compaction jobs. This is the same unit the scheduler caps with the `lake_compaction_max_tasks` config, and is finer-grained than `lake_compaction_running`, which counts compaction jobs (one per partition) — a single job fans out into one tablet-level task per tablet. Carries an `is_leader` label; follower FEs export the metric with `is_leader="false"` and value 0, so dashboards should filter on `is_leader="true"`. #### `lake_compaction_score_at_trigger`[​](#lake_compaction_score_at_trigger "Direct link to lake_compaction_score_at_trigger") * Unit: Score * Type: Gauge * Description: Compaction score of the most recent partition that triggered a lake compaction job, rounded to the nearest integer. The value is the partition's *max* tablet-level score (`Quantiles.getMax()`), matching the criterion the scheduler uses to pick partitions for compaction. Updated once per partition per trigger; the gauge holds the value of the most recent update. This gauge does not decay: on the leader FE, when no compactions are running it retains the last trigger's value (it is not reset to 0). The value is process-local (an in-memory counter on the leader, not persisted), so after an FE leader failover the newly promoted leader starts at 0 and reports 0 until its first compaction trigger — it does not inherit the previous leader's value. Alert on it together with `lake_compaction_running > 0` rather than reading it in isolation. Carries an `is_leader` label; follower FEs export `is_leader="false"` and return 0, so dashboards should filter on `is_leader="true"`. #### `lake_compaction_success`[​](#lake_compaction_success "Direct link to lake_compaction_success") * Unit: Count * Description: Counter of successful lake compaction jobs. #### `lake_vacuum_del_file_batch_size_minute`[​](#lake_vacuum_del_file_batch_size_minute "Direct link to lake_vacuum_del_file_batch_size_minute") * Unit: Count (files per batch) * Type: Gauge * Description: Recent (60s) mean of files per `DeleteObjects` batch sent by Vacuum on shared-data clusters. #### `lake_vacuum_del_file_retries_minute`[​](#lake_vacuum_del_file_retries_minute "Direct link to lake_vacuum_del_file_retries_minute") * Unit: Count * Type: Gauge * Description: Number of Vacuum delete retries triggered in the last 60s on shared-data clusters. Surfaces transient object-storage throttling (SlowDown / try-again). #### `load_bytes`[​](#load_bytes "Direct link to load_bytes") * Unit: Bytes * Description: Total loaded bytes. #### `load_channel_count`[​](#load_channel_count "Direct link to load_channel_count") * Unit: Count * Description: Total number of loading channels. #### `load_mem_bytes`[​](#load_mem_bytes "Direct link to load_mem_bytes") * Unit: Bytes * Description: Memory cost of data loading. #### `load_rows`[​](#load_rows "Direct link to load_rows") * Unit: Count * Description: Total loaded rows. #### `load_rpc_threadpool_size`[​](#load_rpc_threadpool_size "Direct link to load_rpc_threadpool_size") * Unit: Count * Description: The current size of the RPC thread pool, which is used for handling Routine Load and loading via table functions. The default value is 10, with a maximum value of 1000. This value is dynamically adjusted based on the usage of the thread pool. #### `load_sink_close_active_threads`[​](#load_sink_close_active_threads "Direct link to load_sink_close_active_threads") * Unit: Count * Type: Instantaneous * Description: The number of load sink closes that are running in the load sink close thread pool. In a shared-data cluster, each close writes the combined transaction log of a load to object storage. #### `load_sink_close_execute_time_ns_total`[​](#load_sink_close_execute_time_ns_total "Direct link to load_sink_close_execute_time_ns_total") * Unit: Nanoseconds * Type: Cumulative * Description: The total execution time of completed load sink close tasks, including waits and object-storage writes performed during close. Updated when each task finishes. #### `load_sink_close_executed_tasks_total`[​](#load_sink_close_executed_tasks_total "Direct link to load_sink_close_executed_tasks_total") * Unit: Count * Type: Cumulative * Description: The number of load sink close tasks completed by the thread pool, including closes that return an error. #### `load_sink_close_pending_time_ns_total`[​](#load_sink_close_pending_time_ns_total "Direct link to load_sink_close_pending_time_ns_total") * Unit: Nanoseconds * Type: Cumulative * Description: The total time completed load sink close tasks spent waiting in the thread pool queue. Updated when each task finishes; currently queued tasks are not yet included. #### `load_sink_close_queue_count`[​](#load_sink_close_queue_count "Direct link to load_sink_close_queue_count") * Unit: Count * Type: Instantaneous * Description: The number of load sink closes waiting in the queue of the load sink close thread pool. A value that keeps growing means object-storage writes are slower than loads finish. #### `load_sink_close_running_threads`[​](#load_sink_close_running_threads "Direct link to load_sink_close_running_threads") * Unit: Count * Type: Instantaneous * Description: The number of threads in the load sink close thread pool. #### `load_sink_close_threadpool_size`[​](#load_sink_close_threadpool_size "Direct link to load_sink_close_threadpool_size") * Unit: Count * Type: Instantaneous * Description: The maximum number of threads in the load sink close thread pool, set by `load_sink_close_thread_pool_num_max`. #### `local_column_pool_bytes (Deprecated)`[​](#local_column_pool_bytes-deprecated "Direct link to local_column_pool_bytes-deprecated") #### `max_disk_io_util_percent`[​](#max_disk_io_util_percent "Direct link to max_disk_io_util_percent") * Unit: - * Description: Maximum disk I/O utilization percentage. #### `max_network_receive_bytes_rate`[​](#max_network_receive_bytes_rate "Direct link to max_network_receive_bytes_rate") * Unit: Bytes * Description: Total bytes received over the network (maximum value among all network interfaces). #### `max_network_send_bytes_rate`[​](#max_network_send_bytes_rate "Direct link to max_network_send_bytes_rate") * Unit: Bytes * Description: Total bytes sent over the network (maximum value among all network interfaces). #### `memory_pool_bytes_total`[​](#memory_pool_bytes_total "Direct link to memory_pool_bytes_total") * Unit: Bytes * Description: Memory used by the memory pool. #### `memtable_flush_duration_us`[​](#memtable_flush_duration_us "Direct link to memtable_flush_duration_us") * Unit: us * Description: Total time spent on memtable flush. #### `memtable_flush_queue_count`[​](#memtable_flush_queue_count "Direct link to memtable_flush_queue_count") * Unit: Count * Description: Queued task count in the memtable flush thread pool. #### `memtable_flush_total`[​](#memtable_flush_total "Direct link to memtable_flush_total") * Unit: Count * Description: Total number of memtable flushes. #### `merge_commit_append_pipe`[​](#merge_commit_append_pipe "Direct link to merge_commit_append_pipe") * Unit: microsecond * Type: Summary * Description: Time spent appending data to the stream load pipe during merge commit. #### `merge_commit_fail_total`[​](#merge_commit_fail_total "Direct link to merge_commit_fail_total") * Unit: Count * Type: Cumulative * Description: Merge commit requests that failed. #### `merge_commit_pending`[​](#merge_commit_pending "Direct link to merge_commit_pending") * Unit: microsecond * Type: Summary * Description: Time merge commit tasks spend waiting in the pending queue before execution. #### `merge_commit_pending_bytes`[​](#merge_commit_pending_bytes "Direct link to merge_commit_pending_bytes") * Unit: Bytes * Type: Instantaneous * Description: Total bytes of data held by pending merge commit tasks. #### `merge_commit_pending_total`[​](#merge_commit_pending_total "Direct link to merge_commit_pending_total") * Unit: Count * Type: Instantaneous * Description: Merge commit tasks currently waiting in the execution queue. #### `merge_commit_register_pipe_total`[​](#merge_commit_register_pipe_total "Direct link to merge_commit_register_pipe_total") * Unit: Count * Type: Cumulative * Description: Stream load pipes registered for merge commit operations. #### `merge_commit_request`[​](#merge_commit_request "Direct link to merge_commit_request") * Unit: microsecond * Type: Summary * Description: End-to-end processing latency for merge commit requests. #### `merge_commit_request_bytes`[​](#merge_commit_request_bytes "Direct link to merge_commit_request_bytes") * Unit: Bytes * Type: Cumulative * Description: Total bytes of data received across merge commit requests. #### `merge_commit_request_total`[​](#merge_commit_request_total "Direct link to merge_commit_request_total") * Unit: Count * Type: Cumulative * Description: Total number of merge commit requests received by BE. #### `merge_commit_send_rpc_total`[​](#merge_commit_send_rpc_total "Direct link to merge_commit_send_rpc_total") * Unit: Count * Type: Cumulative * Description: RPC requests sent to FE for starting merge commit operations. #### `merge_commit_success_total`[​](#merge_commit_success_total "Direct link to merge_commit_success_total") * Unit: Count * Type: Cumulative * Description: Merge commit requests that finished successfully. #### `merge_commit_unregister_pipe_total`[​](#merge_commit_unregister_pipe_total "Direct link to merge_commit_unregister_pipe_total") * Unit: Count * Type: Cumulative * Description: Stream load pipes unregistered from merge commit operations. Latency metrics expose percentile series such as `merge_commit_request_latency_99` and `merge_commit_request_latency_90`, reported in microseconds. The end-to-end latency obeys: `merge_commit_request = merge_commit_pending + merge_commit_wait_plan + merge_commit_append_pipe + merge_commit_wait_finish` > **Note**: Before v3.4.11, v3.5.12, and v4.0.4, these latency metrics were reported in nanoseconds. #### `merge_commit_wait_finish`[​](#merge_commit_wait_finish "Direct link to merge_commit_wait_finish") * Unit: microsecond * Type: Summary * Description: Time spent waiting for merge commit load operations to finish. #### `merge_commit_wait_plan`[​](#merge_commit_wait_plan "Direct link to merge_commit_wait_plan") * Unit: microsecond * Type: Summary * Description: Combined latency for the RPC request and waiting for the stream load pipe to become available. #### `meta_replay_lag_second`[​](#meta_replay_lag_second "Direct link to meta_replay_lag_second") * Unit: Seconds * Type: Gauge * Description: How far the metadata replayed by this FE lags behind the Leader's clock. The Leader FE writes a timestamp into the journal every 10 seconds, and this metric is the age of the most recent timestamp that this node has replayed. Unlike `max_journal_replay_lag`, which only the Leader reports, this metric is reported by the lagging node itself, and it keeps growing while a single journal entry is stuck in replay. The Leader FE always reports `0`, because it writes the timestamps instead of replaying them. Once this value exceeds `meta_delay_toleration_second`, the node stops serving reads from its own metadata and forwards its queries to the Leader. Two cases are exempt: reads continue while `ignore_meta_check` is `true`, and a node that has replayed nothing since the previous check keeps whatever read availability it already had, unless it has also lost contact with the Leader, because falling behind a Leader that is writing nothing says nothing about this node. This second case only stops a node from being taken out of service; it never returns one that has already stopped serving. #### `meta_request_duration`[​](#meta_request_duration "Direct link to meta_request_duration") * Unit: us * Description: Total meta read/write duration. #### `meta_request_total`[​](#meta_request_total "Direct link to meta_request_total") * Unit: Count * Description: Total number of meta read/write requests. #### `metadata_mem_bytes (Deprecated)`[​](#metadata_mem_bytes-deprecated "Direct link to metadata_mem_bytes-deprecated") #### `network_receive_bytes`[​](#network_receive_bytes "Direct link to network_receive_bytes") * Unit: Bytes * Description: Total bytes received via network. #### `network_receive_packets`[​](#network_receive_packets "Direct link to network_receive_packets") * Unit: Count * Description: Total number of packets received through the network. #### `network_send_bytes`[​](#network_send_bytes "Direct link to network_send_bytes") * Unit: Bytes * Description: Number of bytes sent over the network. #### `network_send_packets`[​](#network_send_packets "Direct link to network_send_packets") * Unit: Count * Description: Total number of packets sent through the network. #### `ordinal_index_mem_bytes`[​](#ordinal_index_mem_bytes "Direct link to ordinal_index_mem_bytes") * Unit: Bytes * Description: Memory used by ordinal indexes. #### `page_cache_capacity`[​](#page_cache_capacity "Direct link to page_cache_capacity") * Description: Capacity of the storage page cache. #### `page_cache_hit_count`[​](#page_cache_hit_count "Direct link to page_cache_hit_count") * Unit: Count * Description: Total number of hits in the storage page cache. #### `page_cache_insert_count`[​](#page_cache_insert_count "Direct link to page_cache_insert_count") * Unit: Count * Description: Total number of insert operations in the storage page cache. #### `page_cache_insert_evict_count`[​](#page_cache_insert_evict_count "Direct link to page_cache_insert_evict_count") * Unit: Count * Description: Total number of cache entries evicted during insert operations due to capacity constraints. #### `page_cache_lookup_count`[​](#page_cache_lookup_count "Direct link to page_cache_lookup_count") * Unit: Count * Description: Total number of storage page cache lookups. #### `page_cache_release_evict_count`[​](#page_cache_release_evict_count "Direct link to page_cache_release_evict_count") * Unit: Count * Description: Total number of cache entries evicted during release operations when cache usage exceeds capacity. #### `pip_query_ctx_cnt`[​](#pip_query_ctx_cnt "Direct link to pip_query_ctx_cnt") * Unit: Count * Description: Total number of currently running queries in the BE. #### `pipe_driver_execution_time`[​](#pipe_driver_execution_time "Direct link to pipe_driver_execution_time") * Description: Cumulative time spent by PipelineDriver executors on processing PipelineDrivers. #### `pipe_driver_queue_len`[​](#pipe_driver_queue_len "Direct link to pipe_driver_queue_len") * Unit: Count * Description: Current number of ready drivers in the ready queue waiting for scheduling in BE. #### `pipe_driver_schedule_count`[​](#pipe_driver_schedule_count "Direct link to pipe_driver_schedule_count") * Unit: Count * Description: Cumulative number of driver scheduling times for pipeline executors in the BE. #### `pipe_drivers`[​](#pipe_drivers "Direct link to pipe_drivers") * Unit: Count * Description: Current logical driver capacity reserved by BE admission control. Includes reservations held while a fragment instantiates drivers and operators, as well as reservations retained after instantiation until they are released. The value follows logical DOP accounting, including colocate execution groups, and is not a count of instantiated driver objects or running threads. Releasing a reservation, including on preparation failure or fragment completion, decreases the value. #### `pipe_poller_block_queue_len`[​](#pipe_poller_block_queue_len "Direct link to pipe_poller_block_queue_len") * Unit: Count * Description: Current length of the block queue of PipelineDriverPoller in the pipeline engine. #### `pipe_poller_max_driver_check`[​](#pipe_poller_max_driver_check "Direct link to pipe_poller_max_driver_check") * Unit: microsecond * Type: Summary * Description: Time taken by the slowest single driver check in one pass of PipelineDriverPoller. A large value means that checking one blocked driver held up the whole pass. #### `pipe_poller_pass`[​](#pipe_poller_pass "Direct link to pipe_poller_pass") * Unit: microsecond * Type: Summary * Description: Time taken by one pass of PipelineDriverPoller over its blocked drivers. Each executor set has a single poller thread, so a long pass delays every driver that waits on it. The maximum in the recent window is exposed as `pipe_poller_pass_max_latency`. #### `pipe_prepare_pool_queue_len`[​](#pipe_prepare_pool_queue_len "Direct link to pipe_prepare_pool_queue_len") * Unit: Count * Description: Queued task count in the pipeline PREPARE thread pool. This is an instantaneous value. #### `pipe_scan_executor_queuing`[​](#pipe_scan_executor_queuing "Direct link to pipe_scan_executor_queuing") * Unit: Count * Description: Current number of pending asynchronous I/O tasks launched by Scan Operators. #### `pk_index_compaction_queue_count`[​](#pk_index_compaction_queue_count "Direct link to pk_index_compaction_queue_count") * Unit: Count * Description: Queued task count in the Primary Key index compaction thread pool. #### `pk_index_sst_read_error_total`[​](#pk_index_sst_read_error_total "Direct link to pk_index_sst_read_error_total") * Type: Counter * Unit: Count * Description: Total number of SST file read failures in the lake Primary Key persistent index. Incremented when SST multi-get (read) operations fail, or when compaction detects data corruption while reading input SST files. #### `pk_index_sst_write_error_total`[​](#pk_index_sst_write_error_total "Direct link to pk_index_sst_write_error_total") * Type: Counter * Unit: Count * Description: Total number of SST file write failures in the lake Primary Key persistent index. Incremented when SST file build fails. #### `plan_fragment_count`[​](#plan_fragment_count "Direct link to plan_fragment_count") * Unit: Count * Description: Number of currently running query plan fragments. #### `process_fd_num_limit_hard`[​](#process_fd_num_limit_hard "Direct link to process_fd_num_limit_hard") * Unit: Count * Description: Hard limit on the maximum number of file descriptors. #### `process_fd_num_limit_soft`[​](#process_fd_num_limit_soft "Direct link to process_fd_num_limit_soft") * Unit: Count * Description: Soft limit on the maximum number of file descriptors. Please note that this item indicates the soft limit. You can set the hard limit using the `ulimit` command. #### `process_fd_num_used`[​](#process_fd_num_used "Direct link to process_fd_num_used") * Unit: Count * Description: Number of file descriptors currently in use in this BE process. #### `process_mem_bytes`[​](#process_mem_bytes "Direct link to process_mem_bytes") * Unit: Bytes * Description: Memory used by this process. #### `process_thread_num`[​](#process_thread_num "Direct link to process_thread_num") * Unit: Count * Description: Total number of threads in this process. #### `publish_version_queue_count`[​](#publish_version_queue_count "Direct link to publish_version_queue_count") * Unit: Count * Description: Queued task count in the Publish Version thread pool. #### `push_request_duration_us`[​](#push_request_duration_us "Direct link to push_request_duration_us") * Unit: us * Description: Total time spent on Spark Load. #### `push_request_write_bytes`[​](#push_request_write_bytes "Direct link to push_request_write_bytes") * Unit: Bytes * Description: Total bytes written via Spark Load. #### `push_request_write_bytes_per_second`[​](#push_request_write_bytes_per_second "Direct link to push_request_write_bytes_per_second") * Unit: Bytes/s * Description: Data writing rate of Spark Load. #### `push_request_write_rows`[​](#push_request_write_rows "Direct link to push_request_write_rows") * Unit: Count * Description: Total rows written via Spark Load. #### `push_requests_total`[​](#push_requests_total "Direct link to push_requests_total") * Unit: Count * Description: Total number of successful and failed Spark Load requests. --- ### Metrics q through r #### `query_cache_capacity`[​](#query_cache_capacity "Direct link to query_cache_capacity") * Description: Capacity of query cache. #### `query_cache_hit_count`[​](#query_cache_hit_count "Direct link to query_cache_hit_count") * Unit: Count * Description: Number of query cache hits. #### `query_cache_hit_ratio`[​](#query_cache_hit_ratio "Direct link to query_cache_hit_ratio") * Unit: - * Description: Hit ratio of query cache. #### `query_cache_lookup_count`[​](#query_cache_lookup_count "Direct link to query_cache_lookup_count") * Unit: Count * Description: Total number of query cache lookups. #### `query_cache_usage`[​](#query_cache_usage "Direct link to query_cache_usage") * Unit: Bytes * Description: Current query cache usages. #### `query_cache_usage_ratio`[​](#query_cache_usage_ratio "Direct link to query_cache_usage_ratio") * Unit: - * Description: Current query cache usage ratio. #### `query_mem_bytes`[​](#query_mem_bytes "Direct link to query_mem_bytes") * Unit: Bytes * Description: Memory used by queries. #### `query_scan_bytes`[​](#query_scan_bytes "Direct link to query_scan_bytes") * Unit: Bytes * Description: Total number of scanned bytes. #### `query_scan_bytes_per_second`[​](#query_scan_bytes_per_second "Direct link to query_scan_bytes_per_second") * Unit: Bytes/s * Description: Estimated rate of scanned bytes per second. #### `query_scan_rows`[​](#query_scan_rows "Direct link to query_scan_rows") * Unit: Count * Description: Total number of scanned rows. #### `query_spill_trigger_total`[​](#query_spill_trigger_total "Direct link to query_spill_trigger_total") * Unit: Count * Labels: `storage_type` * Description: Number of spillable operator instances that triggered at least one spill, broken down by storage backend (`local`, `remote`). Incremented once per operator instance at the first flush callback. #### `query_spill_bytes_write_total`[​](#query_spill_bytes_write_total "Direct link to query_spill_bytes_write_total") * Unit: Bytes * Labels: `storage_type` * Description: Cumulative payload bytes written by spillable operators to spill storage, broken down by storage backend (`local`, `remote`). #### `query_spill_bytes_read_total`[​](#query_spill_bytes_read_total "Direct link to query_spill_bytes_read_total") * Unit: Bytes * Labels: `storage_type` * Description: Cumulative payload bytes read back from spill storage during restore, broken down by storage backend. #### `query_spill_blocks_write_total`[​](#query_spill_blocks_write_total "Direct link to query_spill_blocks_write_total") * Unit: Count * Labels: `storage_type` * Description: Number of spill blocks allocated for writing, broken down by storage backend. Useful for estimating IO count scale on the write path. #### `query_spill_blocks_read_total`[​](#query_spill_blocks_read_total "Direct link to query_spill_blocks_read_total") * Unit: Count * Labels: `storage_type` * Description: Number of spill blocks opened for reading, broken down by storage backend. Useful for estimating IO count scale on the read path. #### `query_spill_write_io_duration_ns_total`[​](#query_spill_write_io_duration_ns_total "Direct link to query_spill_write_io_duration_ns_total") * Unit: Nanoseconds * Labels: `storage_type` * Description: Cumulative wall-clock time spent in write-side spill IO (block append and flush), broken down by storage backend. Useful for tracking write-side spill performance. #### `query_spill_read_io_duration_ns_total`[​](#query_spill_read_io_duration_ns_total "Direct link to query_spill_read_io_duration_ns_total") * Unit: Nanoseconds * Labels: `storage_type` * Description: Cumulative wall-clock time spent in read-side spill IO (block reads during restore), broken down by storage backend. Useful for tracking read-side spill performance. #### `ranger_audit_init_failed`[​](#ranger_audit_init_failed "Direct link to ranger_audit_init_failed") * Unit: - * Type: Gauge * Description: Whether this FE failed to initialize Ranger audit. `1` means authorization still works but this FE writes no Ranger audit event until it is restarted with a fixed audit configuration; the FE log names the destination that failed. `0` otherwise. A common cause is an audit destination that the FE does not ship: only the `file` and `solr` Ranger audit destinations are available. #### `readable_blocks_total (Deprecated)`[​](#readable_blocks_total-deprecated "Direct link to readable_blocks_total-deprecated") #### `recycle_bin_database_num`[​](#recycle_bin_database_num "Direct link to recycle_bin_database_num") * Unit: Count * Description: Number of databases currently held in the FE catalog recycle bin. #### `recycle_bin_partition_num`[​](#recycle_bin_partition_num "Direct link to recycle_bin_partition_num") * Unit: Count * Description: Number of partitions currently held in the FE catalog recycle bin. #### `recycle_bin_table_num`[​](#recycle_bin_table_num "Direct link to recycle_bin_table_num") * Unit: Count * Description: Number of tables currently held in the FE catalog recycle bin. #### `resource_group_bigquery_count`[​](#resource_group_bigquery_count "Direct link to resource_group_bigquery_count") * Unit: Count * Description: Number of queries in each resource group that have triggered the limit for big queries. This is an instantaneous value. #### `resource_group_concurrency_overflow_count`[​](#resource_group_concurrency_overflow_count "Direct link to resource_group_concurrency_overflow_count") * Unit: Count * Description: Number of queries in each resource group that have triggered the concurrency limit. This is an instantaneous value. #### `resource_group_connector_scan_use_ratio (Deprecated)`[​](#resource_group_connector_scan_use_ratio-deprecated "Direct link to resource_group_connector_scan_use_ratio-deprecated") * Unit: - * Description: Ratio of external table scan thread time slices used by each resource group to the total used by all resource groups. This is an average value over the time interval between two metric retrievals. #### `resource_group_cpu_limit_ratio`[​](#resource_group_cpu_limit_ratio "Direct link to resource_group_cpu_limit_ratio") * Unit: - * Description: Ratio of the CPU core limit for each resource group to the total CPU core limit for all resource groups. This is an instantaneous value. #### `resource_group_cpu_use_ratio (Deprecated)`[​](#resource_group_cpu_use_ratio-deprecated "Direct link to resource_group_cpu_use_ratio-deprecated") * Unit: - * Description: Ratio of pipeline thread time slices used by each resource group to the total used by all resource groups. This is an average value over the time interval between two metric retrievals. #### `resource_group_driver_queue_level_accu_time_ns`[​](#resource_group_driver_queue_level_accu_time_ns "Direct link to resource_group_driver_queue_level_accu_time_ns") * Unit: ns * Description: Execution time accumulated by each level of the pipeline driver queue of each resource group, divided by the factor of the level. The `level` label is the level, from 0 to 7, and a driver moves to a higher level as it runs longer. The next driver is taken from the non-empty level with the smallest value. When a level turns from empty to non-empty, its value is raised to at least the value of the level scheduled last. To keep values within the numeric range, all levels and the scheduling watermark occasionally subtract a common time origin. This gauge can decrease during that rebase; the cumulative ready-wait and schedule-count counters are preserved. #### `resource_group_driver_queue_level_schedule_count`[​](#resource_group_driver_queue_level_schedule_count "Direct link to resource_group_driver_queue_level_schedule_count") * Unit: Count * Description: Cumulative number of drivers taken from each level of the pipeline driver queue of each resource group. This counter includes draining resource-group versions and retains their final totals after removal. Updating, deleting, or recreating a group with the same name preserves the total within the BE/CN process. #### `resource_group_driver_queue_level_wait_time_ns`[​](#resource_group_driver_queue_level_wait_time_ns "Direct link to resource_group_driver_queue_level_wait_time_ns") * Unit: ns * Description: Cumulative time that the drivers taken from each level of the pipeline driver queue of each resource group waited in the ready queue. Like the schedule-count counter, it includes draining versions and preserves totals across group updates, deletion, and recreation with the same name within the BE/CN process. Divide its increase by the increase of `resource_group_driver_queue_level_schedule_count` to get the average wait of the level. #### `resource_group_inuse_cpu_cores`[​](#resource_group_inuse_cpu_cores "Direct link to resource_group_inuse_cpu_cores") * Unit: Count * Description: Estimated number of CPU cores currently in use by each resource group. This is an average value over the time interval between two metric retrievals. #### `resource_group_mem_inuse_bytes`[​](#resource_group_mem_inuse_bytes "Direct link to resource_group_mem_inuse_bytes") * Unit: Bytes * Description: Currently used memory by each resource group, measured in bytes. This is an instantaneous value. #### `resource_group_mem_limit_bytes`[​](#resource_group_mem_limit_bytes "Direct link to resource_group_mem_limit_bytes") * Unit: Bytes * Description: Memory limit for each resource group, measured in bytes. This is an instantaneous value. #### `resource_group_running_queries`[​](#resource_group_running_queries "Direct link to resource_group_running_queries") * Unit: Count * Description: Number of queries currently running in each resource group. This is an instantaneous value. #### `resource_group_scan_use_ratio (Deprecated)`[​](#resource_group_scan_use_ratio-deprecated "Direct link to resource_group_scan_use_ratio-deprecated") * Unit: - * Description: Ratio of internal table scan thread time slices used by each resource group to the total used by all resource groups. This is an average value over the time interval between two metric retrievals. #### `resource_group_total_queries`[​](#resource_group_total_queries "Direct link to resource_group_total_queries") * Unit: Count * Description: Total number of queries executed in each resource group, including those currently running. This is an instantaneous value. #### `result_block_queue_count`[​](#result_block_queue_count "Direct link to result_block_queue_count") * Unit: Count * Description: Number of results in the result block queue. #### `result_buffer_block_count`[​](#result_buffer_block_count "Direct link to result_buffer_block_count") * Unit: Count * Description: Number of blocks in the result buffer. #### `routine_load_task_count`[​](#routine_load_task_count "Direct link to routine_load_task_count") * Unit: Count * Description: Number of currently running Routine Load tasks. #### `rowset_count_generated_and_in_use`[​](#rowset_count_generated_and_in_use "Direct link to rowset_count_generated_and_in_use") * Unit: Count * Description: Number of rowset IDs currently in use. #### `rowset_metadata_mem_bytes`[​](#rowset_metadata_mem_bytes "Direct link to rowset_metadata_mem_bytes") * Unit: Bytes * Description: Total number of bytes for rowset metadata. #### `running_base_compaction_task_num`[​](#running_base_compaction_task_num "Direct link to running_base_compaction_task_num") * Unit: Count * Description: Total number of running base compaction tasks. #### `running_cumulative_compaction_task_num`[​](#running_cumulative_compaction_task_num "Direct link to running_cumulative_compaction_task_num") * Unit: Count * Description: Total number of running cumulative compactions. #### `running_update_compaction_task_num`[​](#running_update_compaction_task_num "Direct link to running_update_compaction_task_num") * Unit: Count * Description: Total number of currently running Primary Key table compaction tasks. --- ### Metrics s #### `schema_change_mem_bytes`[​](#schema_change_mem_bytes "Direct link to schema_change_mem_bytes") * Unit: Bytes * Description: Memory used for Schema Change. #### `segment_flush_queue_count`[​](#segment_flush_queue_count "Direct link to segment_flush_queue_count") * Unit: Count * Description: Number of queued tasks in the segment flush thread pool. #### `segment_metadata_mem_bytes`[​](#segment_metadata_mem_bytes "Direct link to segment_metadata_mem_bytes") * Unit: Bytes * Description: Memory used by segment metadata. #### `segment_read`[​](#segment_read "Direct link to segment_read") * Unit: Count * Description: Total number of segment reads. #### `segment_replicate_queue_count`[​](#segment_replicate_queue_count "Direct link to segment_replicate_queue_count") * Unit: Count * Description: Queued task count in the Segment Replicate thread pool. #### `segment_zonemap_mem_bytes`[​](#segment_zonemap_mem_bytes "Direct link to segment_zonemap_mem_bytes") * Unit: Bytes * Description: Memory used by segment zonemap. #### `short_key_index_mem_bytes`[​](#short_key_index_mem_bytes "Direct link to short_key_index_mem_bytes") * Unit: Bytes * Description: Memory used by the short key index. #### `small_file_cache_count`[​](#small_file_cache_count "Direct link to small_file_cache_count") * Unit: Count * Description: Number of small file caches. #### `spill_disk_bytes_used`[​](#spill_disk_bytes_used "Direct link to spill_disk_bytes_used") * Unit: Bytes * Labels: `storage_type` * Description: Current disk bytes reserved across all spill storage directories. The `storage_type=local` variant aggregates the live reserved bytes across every directory managed by the BE's spill `DirManager`. The `storage_type=remote` variant is reported for completeness and is currently always 0 because remote spill storage is tracked per-query rather than globally. #### `snmp`[​](#snmp "Direct link to snmp") * Unit: - * Description: Metrics returned by `/proc/net/snmp`. #### `sort_key_sampling_data_page_fallback_total`[​](#sort_key_sampling_data_page_fallback_total "Direct link to sort_key_sampling_data_page_fallback_total") * Unit: Count * Description: Cumulative number of times data-page sort-key sampling declined a segment and fell back to that segment's coarse `[min, max]` range: an unexpected schema, a `sort_key_idxes` that is not the segment schema's own, a failed page read, a short read, or a sample outside the segment's declared bounds. Sampling is fail-open, so this never fails a split; a rising value means tablet split and range-split compaction are working from coarser boundaries than intended. #### `sort_key_sampling_data_page_latency`[​](#sort_key_sampling_data_page_latency "Direct link to sort_key_sampling_data_page_latency") * Unit: us * Description: Time spent sampling one segment's sort key from its data pages, published as a bvar latency series (average, percentiles, max, qps and count). Covers every attempt that got past the sample-budget check, including attempts that then fell back, so a tablet that is slow and keeps failing open is still visible here. #### `sort_key_sampling_data_page_segments_total`[​](#sort_key_sampling_data_page_segments_total "Direct link to sort_key_sampling_data_page_segments_total") * Unit: Count * Description: Cumulative number of segments for which sampling committed to the data-page path. Counted once the sampling geometry is settled and before the first read, so a segment whose read then falls back is counted too — the counter exists to show that the free short-key-index path was NOT taken. For the I/O actually paid, read `sort_key_sampling_read_bytes_total` instead. #### `sort_key_sampling_read_bytes_total`[​](#sort_key_sampling_read_bytes_total "Direct link to sort_key_sampling_read_bytes_total") * Unit: Bytes * Description: Cumulative compressed bytes read from segment data pages while sampling sort keys. Charged from the page reader's own accounting, so it counts only transfers that really happened, including those on an attempt that later fell back. This is the metric to watch for the I/O cost of split-point sampling. #### `sort_key_sampling_rowsets_opened_total`[​](#sort_key_sampling_rowsets_opened_total "Direct link to sort_key_sampling_rowsets_opened_total") * Unit: Count * Description: Cumulative number of rowsets whose segment files were opened in order to sample them. The open is gated on at least one of the rowset's segments having a non-zero sample budget, so this rises only where sampling is actually attempted. Read it against `sort_key_sampling_samples_total`: rowsets opened while no samples are published means the sample budget is not reaching the segments being read. #### `sort_key_sampling_samples_total`[​](#sort_key_sampling_samples_total "Direct link to sort_key_sampling_samples_total") * Unit: Count * Description: Cumulative number of sort-key samples published, by either sampling path. An increase here with no increase in `sort_key_sampling_data_page_segments_total` means the samples came from the free short-key-index path. #### `sort_key_sampling_short_key_index_fallback_total`[​](#sort_key_sampling_short_key_index_fallback_total "Direct link to sort_key_sampling_short_key_index_fallback_total") * Unit: Count * Description: Cumulative number of times short-key-index sort-key sampling was entered for a segment and then declined it: geometry that does not cross-check against the segment's own row count, an index entry that fails to decode, or a sample that is out of order or outside the segment's declared bounds. Declining is not the end of the road for the segment: it goes on to data-page sampling whenever a data-page budget was apportioned to it, and only falls back to its coarse `[min, max]` range if that budget is zero or that path yields nothing either. Note what this does NOT count: a segment of fewer than two index blocks, which leaves this path without incrementing anything and gets sampled at row granularity instead; and a schema whose short key index cannot encode the whole sort key (a VARCHAR key, for example), which never enters this path at all, so such tables leave this counter at zero and appear in `sort_key_sampling_data_page_segments_total` instead. #### `sort_key_sampling_short_key_index_latency`[​](#sort_key_sampling_short_key_index_latency "Direct link to sort_key_sampling_short_key_index_latency") * Unit: us * Description: Time spent sampling one segment's sort key from its short key index, published as a bvar latency series (average, percentiles, max, qps and count). This path reads no data pages, so values here should stay far below `sort_key_sampling_data_page_latency`. #### `starrocks_be_clone_task_copy_bytes`[​](#starrocks_be_clone_task_copy_bytes "Direct link to starrocks_be_clone_task_copy_bytes") * Unit: Bytes * Type: Cumulative * Description: The total file size copied by Clone tasks in the BE node, including both INTER\_NODE and INTRA\_NODE types. #### `starrocks_be_clone_task_copy_duration_ms`[​](#starrocks_be_clone_task_copy_duration_ms "Direct link to starrocks_be_clone_task_copy_duration_ms") * Unit: ms * Type: Cumulative * Description: The total time for copy consumed by Clone tasks in the BE node, including both INTER\_NODE and INTRA\_NODE types. #### `starrocks_be_exec_state_report_active_threads`[​](#starrocks_be_exec_state_report_active_threads "Direct link to starrocks_be_exec_state_report_active_threads") * Unit: Count * Type: Instantaneous * Description: The number of tasks being executed in the thread pool that reports the execution status of the Fragment instance. #### `starrocks_be_exec_state_report_queue_count`[​](#starrocks_be_exec_state_report_queue_count "Direct link to starrocks_be_exec_state_report_queue_count") * Unit: Count * Type: Instantaneous * Description: The number of tasks queued in the thread pool that reports the execution status of the Fragment instance, up to a maximum of 1000. #### `starrocks_be_exec_state_report_running_threads`[​](#starrocks_be_exec_state_report_running_threads "Direct link to starrocks_be_exec_state_report_running_threads") * Unit: Count * Type: Instantaneous * Description: The number of threads in the thread pool that reports the execution status of the Fragment instance, with a minimum of 1 and a maximum of 2. #### `starrocks_be_exec_state_report_threadpool_size`[​](#starrocks_be_exec_state_report_threadpool_size "Direct link to starrocks_be_exec_state_report_threadpool_size") * Unit: Count * Type: Instantaneous * Description: The maximum number of threads in the thread pool that reports the execution status of the Fragment instance, defaults to 2. #### `starrocks_be_files_scan_num_bytes_read`[​](#starrocks_be_files_scan_num_bytes_read "Direct link to starrocks_be_files_scan_num_bytes_read") * Unit: Bytes * Description: Total bytes read from external storage. Labels: `file_format`, `scan_type`. #### `starrocks_be_files_scan_num_files_read`[​](#starrocks_be_files_scan_num_files_read "Direct link to starrocks_be_files_scan_num_files_read") * Unit: Count * Description: Number of files read from external storage (CSV, Parquet, ORC, JSON, Avro). Labels: `file_format`, `scan_type`. #### `starrocks_be_files_scan_num_raw_rows_read`[​](#starrocks_be_files_scan_num_raw_rows_read "Direct link to starrocks_be_files_scan_num_raw_rows_read") * Unit: Count * Description: Total raw rows read from external storage before format validation and predicate filtering. Labels: `file_format`, `scan_type`. #### `starrocks_be_files_scan_num_rows_return`[​](#starrocks_be_files_scan_num_rows_return "Direct link to starrocks_be_files_scan_num_rows_return") * Unit: Count * Description: Number of rows returned after predicate filtering. Labels: `file_format`, `scan_type`. #### `starrocks_be_files_scan_num_valid_rows_read`[​](#starrocks_be_files_scan_num_valid_rows_read "Direct link to starrocks_be_files_scan_num_valid_rows_read") * Unit: Count * Description: Number of valid rows read (excluding rows with invalid format). Labels: `file_format`, `scan_type`. #### `starrocks_be_lake_tablet_metadata_get_not_found_total`[​](#starrocks_be_lake_tablet_metadata_get_not_found_total "Direct link to starrocks_be_lake_tablet_metadata_get_not_found_total") * Unit: Count * Type: Cumulative * Description: Shared-data only. Total number of remote storage read attempts for lake tablet metadata that return `NotFound`. A tablet's metadata lives either in its own metadata object or inside a bundled metadata file, never in both, and reads of either location are counted, including the bundle reads issued by vacuum. Each failed fallback read is counted separately. Cache misses and successful reads do not increment this metric. Reads of a source cluster's metadata during replication and of snapshot metadata during restore are not counted. #### `starrocks_be_mem_pool_mem_limit_bytes`[​](#starrocks_be_mem_pool_mem_limit_bytes "Direct link to starrocks_be_mem_pool_mem_limit_bytes") * Unit: Bytes * Type: Instantaneous * Description: Memory limit for each memory pool, measured in bytes. #### `starrocks_be_mem_pool_mem_usage_bytes`[​](#starrocks_be_mem_pool_mem_usage_bytes "Direct link to starrocks_be_mem_pool_mem_usage_bytes") * Unit: Bytes * Type: Instantaneous * Description: Currently used total memory by each memory pool, measured in bytes. #### `starrocks_be_mem_pool_mem_usage_ratio`[​](#starrocks_be_mem_pool_mem_usage_ratio "Direct link to starrocks_be_mem_pool_mem_usage_ratio") * Unit: - * Type: Instantaneous * Description: Ratio of the memory usage of the memory pool to the memory limit of the memory pool. #### `starrocks_be_mem_pool_workgroup_count`[​](#starrocks_be_mem_pool_workgroup_count "Direct link to starrocks_be_mem_pool_workgroup_count") * Unit: Count * Type: Instantaneous * Description: Number of resource groups assigned to each memory pool. #### `starrocks_be_pipe_prepare_pool_queue_len`[​](#starrocks_be_pipe_prepare_pool_queue_len "Direct link to starrocks_be_pipe_prepare_pool_queue_len") * Unit: Count * Type: Instantaneous * Description: Instantaneous value of pipeline prepare thread pool task queue length. #### `starrocks_be_priority_exec_state_report_active_threads`[​](#starrocks_be_priority_exec_state_report_active_threads "Direct link to starrocks_be_priority_exec_state_report_active_threads") * Unit: Count * Type: Instantaneous * Description: The number of tasks being executed in the thread pool that reports the final execution state of the Fragment instance. #### `starrocks_be_priority_exec_state_report_queue_count`[​](#starrocks_be_priority_exec_state_report_queue_count "Direct link to starrocks_be_priority_exec_state_report_queue_count") * Unit: Count * Type: Instantaneous * Description: The number of tasks queued in the thread pool that reports the final execution status of the Fragment instance, up to a maximum of 2147483647. #### `starrocks_be_priority_exec_state_report_running_threads`[​](#starrocks_be_priority_exec_state_report_running_threads "Direct link to starrocks_be_priority_exec_state_report_running_threads") * Unit: Count * Type: Instantaneous * Description: The number of threads in the thread pool that reports the final execution status of the Fragment instance, with a minimum of 1 and a maximum of 2. #### `starrocks_be_priority_exec_state_report_threadpool_size`[​](#starrocks_be_priority_exec_state_report_threadpool_size "Direct link to starrocks_be_priority_exec_state_report_threadpool_size") * Unit: Count * Type: Instantaneous * Description: The maximum number of threads in the thread pool that reports the final execution status of the Fragment instance, defaults to 2. #### `starrocks_be_resource_group_cpu_limit_ratio`[​](#starrocks_be_resource_group_cpu_limit_ratio "Direct link to starrocks_be_resource_group_cpu_limit_ratio") * Unit: - * Type: Instantaneous * Description: Instantaneous value of resource group cpu quota ratio. #### `starrocks_be_resource_group_cpu_use_ratio`[​](#starrocks_be_resource_group_cpu_use_ratio "Direct link to starrocks_be_resource_group_cpu_use_ratio") * Unit: - * Type: Average * Description: The ratio of CPU time used by the resource group to the CPU time of all resource groups. #### `starrocks_be_resource_group_mem_inuse_bytes`[​](#starrocks_be_resource_group_mem_inuse_bytes "Direct link to starrocks_be_resource_group_mem_inuse_bytes") * Unit: Bytes * Type: Instantaneous * Description: Instantaneous value of resource group memory usage. #### `starrocks_be_resource_group_mem_limit_bytes`[​](#starrocks_be_resource_group_mem_limit_bytes "Direct link to starrocks_be_resource_group_mem_limit_bytes") * Unit: Bytes * Type: Instantaneous * Description: Instantaneous value of resource group memory quota. #### `starrocks_be_segment_file_not_found_total`[​](#starrocks_be_segment_file_not_found_total "Direct link to starrocks_be_segment_file_not_found_total") * Unit: Count * Description: Total number of times a segment file was not found (file missing) during segment open. A continuously increasing value may indicate data loss or storage inconsistency. #### `starrocks_be_staros_shard_info_fallback_total`[​](#starrocks_be_staros_shard_info_fallback_total "Direct link to starrocks_be_staros_shard_info_fallback_total") * Unit: Count * Type: Cumulative * Description: Shared-data only. Total number of actual starmgr RPCs (`g_starlet->get_shard_info()`) that the BE's StarOSWorker had to issue because the requested shard info was not in the local cache (i.e. the FE had not pushed the shard to this BE before a query/compaction/lake operation referenced it). Only counted when the starlet readiness check passes and the RPC is actually dispatched; starlet-not-ready timeouts are not included. Should normally be near zero. A sustained or rising rate is a strong signal that FE-side task or node selection is scheduling work on a BE that does not yet have the shard, or that shard push propagation from FE is lagging. Recommended alert: high per-BE rate over a 5-minute window. #### `starrocks_be_staros_shard_info_fallback_failed_total`[​](#starrocks_be_staros_shard_info_fallback_failed_total "Direct link to starrocks_be_staros_shard_info_fallback_failed_total") * Unit: Count * Type: Cumulative * Description: Shared-data only. Subset of `starrocks_be_staros_shard_info_fallback_total` where the starmgr RPC returned a non-OK status. Use the ratio `failed_total / fallback_total` to alert on transient starmgr errors separately from routine successful fallbacks. #### `starrocks_be_staros_shard_count`[​](#starrocks_be_staros_shard_count "Direct link to starrocks_be_staros_shard_count") * Unit: Count * Type: Instantaneous * Description: Shared-data only. Number of shards currently assigned to this BE's StarOSWorker (size of the worker's local shard table). Updated synchronously inside `StarOSWorker::add_shard` and `StarOSWorker::remove_shard` (push-on-mutation), so the value reflects the last shard table mutation rather than being recomputed at scrape time. The gauge is not reset on BE shutdown and will retain its last value until the next mutation. Use it to observe shard distribution balance across BEs and to detect drift from the FE-side placement. #### `starrocks_fe_alter_duration_ms`[​](#starrocks_fe_alter_duration_ms "Direct link to starrocks_fe_alter_duration_ms") * Unit: ms * Type: Summary * Labels: `execution_mode` (`fse`, `legacy_fse`, or `rewrite`), `is_leader` * Description: Time in milliseconds to apply an ALTER TABLE change, per statement. Reported only by the Leader FE (`is_leader="true"`). Includes the 0.75/0.95/0.98/0.99/0.999 quantiles plus `_sum` and `_count`. The `execution_mode` label indicates how the change was applied: * `fse`: the current Fast Schema Evolution (FSE), applied immediately while the statement runs. * `legacy_fse`: the legacy FSE path, which runs in the background and is usually far slower. Occurs only in shared-data clusters with `cloud_native_fast_schema_evolution_v2` disabled (enabled by default, which uses `fse`). * `rewrite`: the change had to physically rewrite the table data, which also runs in the background. #### `starrocks_fe_alter_operation_total`[​](#starrocks_fe_alter_operation_total "Direct link to starrocks_fe_alter_operation_total") * Unit: Count * Type: Cumulative * Labels: `type` (`add_column`, `drop_column`, or `modify_column`), `is_leader` * Description: Number of ALTER TABLE column operations, by type. A single statement can contain several operations — for example, `ADD COLUMN a, DROP COLUMN b` — and each is counted separately under its type. Renames, reorders, and comment-only changes are not counted. Reported only by the Leader FE (`is_leader="true"`). #### `starrocks_fe_clone_task_copy_bytes`[​](#starrocks_fe_clone_task_copy_bytes "Direct link to starrocks_fe_clone_task_copy_bytes") * Unit: Bytes * Type: Cumulative * Description: The total file size copied by Clone tasks in the cluster, including both INTER\_NODE and INTRA\_NODE types. #### `starrocks_fe_clone_task_copy_duration_ms`[​](#starrocks_fe_clone_task_copy_duration_ms "Direct link to starrocks_fe_clone_task_copy_duration_ms") * Unit: ms * Type: Cumulative * Description: The total time for copy consumed by Clone tasks in the cluster, including both INTER\_NODE and INTRA\_NODE types. #### `starrocks_fe_clone_task_success`[​](#starrocks_fe_clone_task_success "Direct link to starrocks_fe_clone_task_success") * Unit: Count * Type: Cumulative * Description: The number of successfully executed Clone tasks in the cluster. #### `starrocks_fe_clone_task_total`[​](#starrocks_fe_clone_task_total "Direct link to starrocks_fe_clone_task_total") * Unit: Count * Type: Cumulative * Description: The total number of Clone tasks in the cluster. #### `starrocks_fe_last_finished_job_timestamp`[​](#starrocks_fe_last_finished_job_timestamp "Direct link to starrocks_fe_last_finished_job_timestamp") * Unit: ms * Type: Instantaneous * Description: Indicates the end time of the last query or loading under the specific warehouse. For a shared-nothing cluster, this item only monitors the default warehouse. #### `starrocks_fe_max_journal_replay_lag`[​](#starrocks_fe_max_journal_replay_lag "Direct link to starrocks_fe_max_journal_replay_lag") * Unit: Count * Type: Instantaneous * Description: The largest number of metadata journals that any alive Follower or Observer FE still has to replay to catch up with the Leader. Reported only by the Leader FE (`is_leader="true"`), which is the only node that knows both its own journal write position and, through heartbeats, every other node's replayed journal ID. FE nodes that are not alive are excluded, because the journal ID they last reported is frozen at their final successful heartbeat; use the `Alive` column of `SHOW FRONTENDS` to detect those instead. It returns `0` when every other node has caught up, when no other FE node is alive, and on a single-FE cluster. A high or continuously growing value means metadata replay on at least one node is falling behind, which causes that node to serve stale metadata and, once `meta_delay_toleration_second` is exceeded, to forward its queries to the Leader. #### `starrocks_fe_memory_usage`[​](#starrocks_fe_memory_usage "Direct link to starrocks_fe_memory_usage") * Unit: Bytes or Count * Type: Instantaneous * Description: Indicates the memory statistics for various modules under the specific warehouse. For a shared-nothing cluster, this item only monitors the default warehouse. #### `starrocks_fe_meta_log_count`[​](#starrocks_fe_meta_log_count "Direct link to starrocks_fe_meta_log_count") * Unit: Count * Type: Instantaneous * Description: The number of Edit Logs without a checkpoint. A value within `100000` is considered reasonable.. #### `starrocks_fe_publish_version_daemon_loop_total`[​](#starrocks_fe_publish_version_daemon_loop_total "Direct link to starrocks_fe_publish_version_daemon_loop_total") * Unit: Count * Type: Cumulative * Description: Total number of `publish-version-daemon` loop runs on this FE node. The following metrics are `summary`-type metrics that provide latency distributions for different phases of a transaction. These metrics are reported exclusively by the Leader FE node. Each metric includes the following outputs: * **Quantiles**: Latency values at different percentile boundaries. These are exposed via the `quantile` label, which can have values of `0.75`, `0.95`, `0.98`, `0.99`, and `0.999`. * **`_sum`**: The total cumulative time spent in this phase, for example, `starrocks_fe_txn_total_latency_ms_sum`. * **`_count`**: The total number of transactions recorded for this phase, for example, `starrocks_fe_txn_total_latency_ms_count`. All transaction metrics share the following labels: * `type`: Categorizes transactions by their load job source type (for example, `all`, `stream_load`, `routine_load`). This allows for monitoring both overall transaction performance and the performance of specific load types. The reported groups can be configured via the FE parameter [`txn_latency_metric_report_groups`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#txn_latency_metric_report_groups). * `is_leader`: Indicates whether the reporting FE node is the Leader. Only the Leader FE (`is_leader="true"`) reports actual metric values. Followers will have `is_leader="false"` and report no data. #### `starrocks_fe_query_resource_group`[​](#starrocks_fe_query_resource_group "Direct link to starrocks_fe_query_resource_group") * Unit: Count * Type: Cumulative * Description: Indicates the total number of queries executed under the specific resource group. #### `starrocks_fe_query_resource_group`[​](#starrocks_fe_query_resource_group-1 "Direct link to starrocks_fe_query_resource_group-1") * Unit: Count * Type: Cumulative * Description: The number of queries for each resource group. #### `starrocks_fe_query_resource_group_err`[​](#starrocks_fe_query_resource_group_err "Direct link to starrocks_fe_query_resource_group_err") * Unit: Count * Type: Cumulative * Description: Indicates the number of failed queries under the specific resource group. #### `starrocks_fe_query_resource_group_err`[​](#starrocks_fe_query_resource_group_err-1 "Direct link to starrocks_fe_query_resource_group_err-1") * Unit: Count * Type: Cumulative * Description: The number of incorrect queries for each resource group. #### `starrocks_fe_query_resource_group_latency`[​](#starrocks_fe_query_resource_group_latency "Direct link to starrocks_fe_query_resource_group_latency") * Unit: ms * Type: Cumulative * Description: Indicates the latency statistics for queries under the specific resource group. #### `starrocks_fe_query_resource_group_latency`[​](#starrocks_fe_query_resource_group_latency-1 "Direct link to starrocks_fe_query_resource_group_latency-1") * Unit: Seconds * Type: Average * Description: the query latency percentile for each resource group. #### `starrocks_fe_routine_load_error_rows`[​](#starrocks_fe_routine_load_error_rows "Direct link to starrocks_fe_routine_load_error_rows") * Unit: Count * Description: The total number of error rows encountered during data loading by all Routine Load jobs. #### `starrocks_fe_routine_load_jobs`[​](#starrocks_fe_routine_load_jobs "Direct link to starrocks_fe_routine_load_jobs") * Unit: Count * Description: The total number of Routine Load jobs in different states. For example: ```plaintext starrocks_fe_routine_load_jobs{state="NEED_SCHEDULE"} 0 starrocks_fe_routine_load_jobs{state="RUNNING"} 1 starrocks_fe_routine_load_jobs{state="PAUSED"} 0 starrocks_fe_routine_load_jobs{state="STOPPED"} 0 starrocks_fe_routine_load_jobs{state="CANCELLED"} 1 starrocks_fe_routine_load_jobs{state="UNSTABLE"} 0 ``` #### `starrocks_fe_routine_load_max_lag_of_partition`[​](#starrocks_fe_routine_load_max_lag_of_partition "Direct link to starrocks_fe_routine_load_max_lag_of_partition") * Unit: - * Description: The maximum Kafka partition offset lag for each Routine Load job. It is collected only when the FE configuration `enable_routine_load_lag_metrics` is set to `true` and the offset lag is greater than or equal to the FE configuration `min_routine_load_lag_for_metrics`. By default, `enable_routine_load_lag_metrics` is `false`, and `min_routine_load_lag_for_metrics` is `10000`. #### `starrocks_fe_routine_load_max_lag_time_of_partition`[​](#starrocks_fe_routine_load_max_lag_time_of_partition "Direct link to starrocks_fe_routine_load_max_lag_time_of_partition") * Unit: Seconds * Description: The maximum Kafka partition offset timestamp lag for each Routine Load job. It is collected only when the FE configuration `enable_routine_load_lag_time_metrics` is set to `true`. By default, `enable_routine_load_lag_time_metrics` is `false`. #### `starrocks_fe_routine_load_paused`[​](#starrocks_fe_routine_load_paused "Direct link to starrocks_fe_routine_load_paused") * Unit: Count * Description: The total number of times Routine Load jobs are paused. #### `starrocks_fe_routine_load_receive_bytes`[​](#starrocks_fe_routine_load_receive_bytes "Direct link to starrocks_fe_routine_load_receive_bytes") * Unit: Byte * Description: The total amount of data loaded by all Routine Load jobs. #### `starrocks_fe_routine_load_rows`[​](#starrocks_fe_routine_load_rows "Direct link to starrocks_fe_routine_load_rows") * Unit: Count * Description: The total number of rows loaded by all Routine Load jobs. #### `starrocks_fe_safe_mode`[​](#starrocks_fe_safe_mode "Direct link to starrocks_fe_safe_mode") * Unit: - * Type: Instantaneous * Description: Indicates whether Safe Mode is enabled. Valid values: `0` (disabled) and `1` (enabled). When Safe Mode is enabled, the cluster no longer accepts any loading requests. #### `starrocks_fe_scheduled_pending_tablet_num`[​](#starrocks_fe_scheduled_pending_tablet_num "Direct link to starrocks_fe_scheduled_pending_tablet_num") * Unit: Count * Type: Instantaneous * Description: The number of Clone tasks in Pending state FE scheduled, including both BALANCE and REPAIR types. #### `starrocks_fe_scheduled_running_tablet_num`[​](#starrocks_fe_scheduled_running_tablet_num "Direct link to starrocks_fe_scheduled_running_tablet_num") * Unit: Count * Type: Instantaneous * Description: The number of Clone tasks in Running state FE scheduled, including both BALANCE and REPAIR types. #### `starrocks_fe_slow_lock_held_time_ms`[​](#starrocks_fe_slow_lock_held_time_ms "Direct link to starrocks_fe_slow_lock_held_time_ms") * Unit: ms * Type: Summary * Description: Histogram tracking the lock held time (in milliseconds) when slow locks are detected. This metric is updated when lock wait time exceeds the `slow_lock_threshold_ms` configuration parameter. It tracks the maximum lock held time among all lock owners when a slow lock event is detected. Each metric includes quantile values (0.75, 0.95, 0.98, 0.99, 0.999), `_sum`, and `_count` outputs. Note: This metric may not accurately reflect the exact lock held time under high contention, because the metric is updated once the wait time exceeds the threshold, but the held time may continue to increase until the owner completes its operation and releases the lock. However, this metric can still be updated even when deadlock occurs. #### `starrocks_fe_slow_lock_wait_time_ms`[​](#starrocks_fe_slow_lock_wait_time_ms "Direct link to starrocks_fe_slow_lock_wait_time_ms") * Unit: ms * Type: Summary * Description: Histogram tracking the lock wait time (in milliseconds) when slow locks are detected. This metric is updated when lock wait time exceeds the `slow_lock_threshold_ms` configuration parameter. It accurately tracks how long threads wait to acquire locks during lock contention scenarios. Each metric includes quantile values (0.75, 0.95, 0.98, 0.99, 0.999), `_sum`, and `_count` outputs. This metric provides precise wait time measurements. Note: This metric cannot be updated when deadlock occurs, hence it cannot be used to detect deadlock situations. #### `starrocks_fe_sql_block_hit_count`[​](#starrocks_fe_sql_block_hit_count "Direct link to starrocks_fe_sql_block_hit_count") * Unit: Count * Description: The number of times blacklisted SQL has been intercepted. #### `starrocks_fe_tablet_pre_split_eligibility_skipped`[​](#starrocks_fe_tablet_pre_split_eligibility_skipped "Direct link to starrocks_fe_tablet_pre_split_eligibility_skipped") * Unit: Count * Type: Cumulative * Labels: `reason`, the skip reason. The values are listed in the tables below. A sampled column is a partition column or a sort-key column of the target table. * Description: Total Sample-Based Tablet Pre-Split invocations that the FE-side eligibility gate declined before any sampler ran, broken down by the specific reason. Operators can use this counter to attribute "pre-split not running" to a single eligibility branch at a glance. In the multi-partition path the same counter also records per-partition skip reasons emitted by the grouper and the per-partition re-resolve. Values recorded for any load: | `reason` | Recorded when | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `disabled_by_config` | The `enable_tablet_pre_split_for_*` FE parameter for this load type is off. | | `disabled_by_session` | The session sets `enable_tablet_pre_split` to `false`. | | `not_range_distribution` | The target table does not use range distribution. | | `table_not_normal` | The target table is not in the NORMAL state, for example during an ALTER. | | `unsupported_sort_key` | The sort key is empty, or a sort-key column is not of a scalar type. | | `has_materialized_view_or_rollup` | The target table has a visible rollup or synchronous materialized view. | | `metadata_not_resolved` | The partition or its base index cannot be found, usually because an ALTER ran during the load. | | `multiple_base_index_tablets` | The partition's base index already has more than one tablet, which is common when a partition is loaded again. | | `partition_not_empty` | The partition already holds data. | | `source_missing_sampled_column` | A sampled column has no source column or literal in the load, for example because the `FILES()` file lacks it or the INSERT column list omits it. | | `unsupported_sampled_projection` | A sampled column comes from an expression the sampler cannot reproduce, such as `from_unixtime(ts)` or NULL, or from a `FILES()` column that type push-down reads at another column's type. | | `unsupported_generated_column` | A sampled column is a generated column the sampler cannot compute. The FE log names the column and the reason. | | `estimate_unavailable` | An INSERT from an external-catalog table cannot be sized from its table statistics, so the tablet count cannot be chosen. | Values recorded per partition, when a load writes several partitions: | `reason` | Recorded when | | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `unsupported_partition_column_type` | A partition column has a type that cannot be sampled, such as STRUCT or ARRAY. | | `invalid_partition_value` | A sampled partition value cannot form a valid partition, such as NULL in a NOT NULL column or a date that cannot be parsed. | | `grouper_empty` | Every sampled row was dropped while grouping rows by partition. | | `no_matching_partition` | On a manually range-partitioned table, a sampled value falls outside every range the load can write. The value is dropped, and no partition is created for it. | | `stale_catalog_state` | A partition disappeared between sampling and pre-split, for example through a concurrent DROP or REPLACE. | | `partition_not_eligible_post_create` | After it was created, the partition no longer qualifies, usually because it holds data or has more than one tablet. | #### `starrocks_fe_tablet_pre_split_sampler_invocations`[​](#starrocks_fe_tablet_pre_split_sampler_invocations "Direct link to starrocks_fe_tablet_pre_split_sampler_invocations") * Unit: Count * Type: Cumulative * Description: Total sampler invocations driven by Sample-Based Tablet Pre-Split. Incremented once per eligible invocation when the production sampler pipeline starts a sample. #### `starrocks_fe_tablet_pre_split_sampler_failed`[​](#starrocks_fe_tablet_pre_split_sampler_failed "Direct link to starrocks_fe_tablet_pre_split_sampler_failed") * Unit: Count * Type: Cumulative * Labels: `reason` — the post-eligibility failure category (lower-cased SkipReason), one of `sample_failed` (sampler executor threw), `timeout_pre_submit` (sample + plan + build phase exceeded `tablet_pre_split_pre_submit_timeout_seconds`), `submit_failed` (`TabletReshardJobMgr` rejected admission), `pre_create_failed` (multi-partition path: `LocalMetastore.addPartitions` threw while pre-creating a target partition — that one partition is dropped from the combined submit and falls back to BE runtime auto-create; sibling partitions in the same load continue). * Description: Total times the sampler attempted but did not produce an admitted reshard job, broken down by reason. Distinct from `tablet_pre_split_eligibility_skipped` (sampler never ran) and from `tablet_pre_split_tier_used` (which records the tier that succeeded). Meta-tier → data-tier fallback alone is not a failure; it is tracked via `tablet_pre_split_tier_used{tier=data_tier}`. #### `starrocks_fe_tablet_pre_split_tier_used`[​](#starrocks_fe_tablet_pre_split_tier_used "Direct link to starrocks_fe_tablet_pre_split_tier_used") * Unit: Count * Type: Cumulative * Labels: `tier` — `meta_tier` (boundaries computed from Parquet/ORC row-group statistics; no row data read) or `data_tier` (boundaries computed from actual row samples collected via a FILES sub-query — covers both direct data-tier invocations and meta-tier → data-tier fallbacks). * Description: Total Sample-Based Tablet Pre-Split invocations by which sampler tier produced the boundaries. #### `starrocks_fe_tablet_pre_split_boundaries_planned`[​](#starrocks_fe_tablet_pre_split_boundaries_planned "Direct link to starrocks_fe_tablet_pre_split_boundaries_planned") * Unit: Count * Type: Histogram * Description: Number of boundary tuples produced by the planner per invocation. Equals `effectiveTabletCount - 1` (a K-tablet split needs K-1 cut points). #### `starrocks_fe_tablet_pre_split_partitions_total`[​](#starrocks_fe_tablet_pre_split_partitions_total "Direct link to starrocks_fe_tablet_pre_split_partitions_total") * Unit: Count * Type: Cumulative * Description: Multi-partition (P2-a) counter. Total predicted target partitions counted by the Sample-Based Tablet Pre-Split coordinator — one increment per `PartitionSamples` entry that survived the grouper. Combined with `tablet_pre_split_partitions_capped` and the `tablet_pre_split_pre_create{result=...}` family this tells operators how many partitions each multi-partition invocation actually acts on. Stays at zero for the single-partition path. #### `starrocks_fe_tablet_pre_split_partitions_capped`[​](#starrocks_fe_tablet_pre_split_partitions_capped "Direct link to starrocks_fe_tablet_pre_split_partitions_capped") * Unit: Count * Type: Cumulative * Description: Multi-partition counter. Number of predicted target partitions the grouper dropped because the per-load count exceeded `tablet_pre_split_max_partitions_per_load`. The grouper keeps the heaviest partitions (by input bytes when the data tier sampled a subset of files whose partition values come from the file paths, otherwise by sample count) and drops the lightest; dropped partitions fall back to BE runtime auto-create with no pre-split. Sustained non-zero values mean the cap is biting — consider raising `tablet_pre_split_max_partitions_per_load` or reducing partition cardinality on the load. #### `starrocks_fe_tablet_pre_split_pre_create`[​](#starrocks_fe_tablet_pre_split_pre_create "Direct link to starrocks_fe_tablet_pre_split_pre_create") * Unit: Count * Type: Cumulative * Labels: `result` — `succeeded` (`LocalMetastore.addPartitions` returned normally — the partition was created or silently deduped), `failed` (`addPartitions` threw, e.g. concurrent ALTER or journal failure; the affected partition falls back to BE runtime auto-create and is also recorded under `tablet_pre_split_sampler_failed{reason=pre_create_failed}`), `already_exists` (the partition was found in the catalog at pre-create time — concurrent loader race; the coordinator reuses the existing partition). * Description: Multi-partition (P2-a) counter. Number of partition pre-create attempts the coordinator issued via `LocalMetastore.addPartitions`, broken down by outcome. Total attempts = sum of all three labels. Stays at zero for the single-partition path. #### `starrocks_fe_tablet_pre_split_pre_submit_wait_ms`[​](#starrocks_fe_tablet_pre_split_pre_submit_wait_ms "Direct link to starrocks_fe_tablet_pre_split_pre_submit_wait_ms") * Unit: ms * Type: Histogram * Description: Wall-clock time spent in the pre-submit phase of Sample-Based Tablet Pre-Split (sample + plan + build reshard job). Capped by `tablet_pre_split_pre_submit_timeout_seconds`. #### `starrocks_fe_tablet_pre_split_post_submit_wait_ms`[​](#starrocks_fe_tablet_pre_split_post_submit_wait_ms "Direct link to starrocks_fe_tablet_pre_split_post_submit_wait_ms") * Unit: ms * Type: Histogram * Description: Wall-clock time the coordinator spent awaiting `FINISHED` on the admitted Sample-Based Tablet Pre-Split reshard job. Fires on all production load kinds — INSERT-from-FILES and INSERT-from-table (both via `InsertPreSplitHook`, called from `StmtExecutor` before `StatementPlanner.plan` opens the load txn) and Broker Load (via `BrokerLoadPreSplitHook`, called from `BrokerLoadJob.createLoadingTask` before `beginTxn` opens `T_load`), all sync-awaiting through the shared `PreSplitFlow` — and on the optional `runPreSplit` synchronous-await wrapper used by tests. In all cases the trigger load itself plans against the post-split layout. #### `starrocks_fe_tablet_pre_split_post_submit_hard_cap`[​](#starrocks_fe_tablet_pre_split_post_submit_hard_cap "Direct link to starrocks_fe_tablet_pre_split_post_submit_hard_cap") * Unit: Count * Type: Cumulative * Description: Total Sample-Based Tablet Pre-Split post-submit hard-cap events. Incremented when the admitted reshard job did not reach `FINISHED` within `tablet_pre_split_post_submit_wait_seconds`. Fires on every production load kind on timeout — INSERT-from-FILES, INSERT-from-table, and Broker Load (all sync-await through the shared `PreSplitFlow`) — as well as the `runPreSplit` synchronous-await wrapper. The load then proceeds without abort against the currently visible tablet layout (still the original layout if the daemon hasn't transitioned, or partially / fully post-split if the daemon raced past the wait); `tablet_pre_split_load_abort` is NOT incremented because the load itself is not aborted. #### `starrocks_fe_tablet_pre_split_load_abort`[​](#starrocks_fe_tablet_pre_split_load_abort "Direct link to starrocks_fe_tablet_pre_split_load_abort") * Unit: Count * Type: Cumulative * Description: Total load transactions aborted because Sample-Based Tablet Pre-Split could not confirm the admitted reshard job reached `FINISHED` in time. Sibling counter of `tablet_pre_split_post_submit_hard_cap`. Production load paths proceed without abort against the currently visible layout on post-submit timeout rather than abort, so this counter stays at zero in production today; it only fires when a caller uses the strict `runPreSplit` wrapper (tests, or a future caller that opts into abort-on-timeout). #### `starrocks_fe_tablet_reshard_job_total`[​](#starrocks_fe_tablet_reshard_job_total "Direct link to starrocks_fe_tablet_reshard_job_total") * Unit: Count * Type: Cumulative * Labels: `type` (`split` or `merge`) * Description: Total tablet reshard jobs admitted by `TabletReshardJobMgr`, counted once the job has been queued and journaled. Covers both automatic reshard (tablet split and merge) and jobs submitted by Sample-Based Tablet Pre-Split. Per-state job counts are also exposed through the generic `job` gauge with the labels `job="tablet_reshard"`, `type`, and `state`. Only the Leader FE increments this counter; it stays at zero on followers and restarts from zero after an FE restart or a leader change. #### `starrocks_fe_tablet_reshard_job_finished`[​](#starrocks_fe_tablet_reshard_job_finished "Direct link to starrocks_fe_tablet_reshard_job_finished") * Unit: Count * Type: Cumulative * Labels: `type` (`split` or `merge`) * Description: Total tablet reshard jobs that reached `FINISHED`. Compare with `tablet_reshard_job_total` to see how many admitted jobs have completed. Only the Leader FE increments this counter. #### `starrocks_fe_tablet_reshard_job_aborted`[​](#starrocks_fe_tablet_reshard_job_aborted "Direct link to starrocks_fe_tablet_reshard_job_aborted") * Unit: Count * Type: Cumulative * Labels: `type` (`split` or `merge`) * Description: Total tablet reshard jobs that ended in `ABORTED`. A job can only abort before its transaction commits; once committed there is no rollback path, so a failure after that point is retried instead of aborted and is never counted here. Use `tablet_reshard_publish_failed` to detect that case. Only the Leader FE increments this counter. #### `starrocks_fe_tablet_reshard_publish_failed`[​](#starrocks_fe_tablet_reshard_publish_failed "Direct link to starrocks_fe_tablet_reshard_publish_failed") * Unit: Count * Type: Cumulative * Description: Total reshard publish attempts that failed and will be retried, one increment per attempt. Because a reshard transaction is already committed by publish time, a failed publish is retried with exponential backoff (1s, doubling, capped at 30s) until it succeeds: the job stays `RUNNING` and never aborts, so `tablet_reshard_job_aborted` stays flat while the job is stuck. This counter is therefore the metric to alert on. At the 30s cap a permanently failing partition keeps it moving about twice a minute, so `increase(starrocks_fe_tablet_reshard_publish_failed[5m]) > 0` catches a stuck job within roughly a minute of the first failure. The reason for each failure is reported in the `ERROR_MESSAGE` column of `information_schema.tablet_reshard_jobs` and in `fe.warn.log`. Only the Leader FE increments this counter. #### `starrocks_fe_tablet_reshard_parallel_tablets`[​](#starrocks_fe_tablet_reshard_parallel_tablets "Direct link to starrocks_fe_tablet_reshard_parallel_tablets") * Unit: Count * Type: Instantaneous * Description: Number of tablets currently being resharded across all unfinished reshard jobs, that is, the live reshard parallelism. Non-Leader FE nodes report `0`. #### `starrocks_fe_tablet_max_compaction_score`[​](#starrocks_fe_tablet_max_compaction_score "Direct link to starrocks_fe_tablet_max_compaction_score") * Unit: Count * Type: Instantaneous * Description: Indicates the highest Compaction Score on each BE node. #### `starrocks_fe_tablet_num`[​](#starrocks_fe_tablet_num "Direct link to starrocks_fe_tablet_num") * Unit: Count * Type: Instantaneous * Description: Indicates the number of tablets on each BE node. #### `starrocks_fe_txn_max_committed_pending_publish_ms`[​](#starrocks_fe_txn_max_committed_pending_publish_ms "Direct link to starrocks_fe_txn_max_committed_pending_publish_ms") * Unit: ms * Type: Instantaneous * Description: The longest time, in milliseconds, that a transaction is currently sitting in the `COMMITTED` status pending publish to `VISIBLE`, that is, the age of the oldest committed-but-not-yet-published transaction. Unlike `starrocks_fe_txn_publish_*` metrics, which are summaries recorded after a transaction finishes, this is a live gauge of the worst-case in-flight wait. The value is reported per database via the `db` label and only by the Leader FE node (`is_leader="true"`). It returns `0` when no committed transaction is pending publish. A high or continuously growing value indicates that version publishing is stuck or lagging behind commits. #### `starrocks_fe_txn_publish_ack_latency_ms`[​](#starrocks_fe_txn_publish_ack_latency_ms "Direct link to starrocks_fe_txn_publish_ack_latency_ms") * Unit: ms * Type: Summary * Description: The final acknowledgment latency, from `ready-to-finish` time to the final `finish` time when the transaction is marked as `VISIBLE`. This metric includes final acknowledgment steps after the transaction is ready to finish. #### `starrocks_fe_txn_publish_can_finish_latency_ms`[​](#starrocks_fe_txn_publish_can_finish_latency_ms "Direct link to starrocks_fe_txn_publish_can_finish_latency_ms") * Unit: ms * Type: Summary * Description: The latency from `publish` task completion to the moment `canTxnFinish()` first returns true, measured from `publish version finish` time to `ready-to-finish` time. #### `starrocks_fe_txn_publish_execute_latency_ms`[​](#starrocks_fe_txn_publish_execute_latency_ms "Direct link to starrocks_fe_txn_publish_execute_latency_ms") * Unit: ms * Type: Summary * Description: The active execution time of the `publish` task, from when the task is picked up to when it finishes. This metric represents the actual time being spent to make the transaction's changes visible. #### `starrocks_fe_txn_publish_latency_ms`[​](#starrocks_fe_txn_publish_latency_ms "Direct link to starrocks_fe_txn_publish_latency_ms") * Unit: ms * Type: Summary * Description: The latency of the `publish` phase, from `commit` time to `finish` time. This is the duration it takes for a committed transaction to become visible to queries. It is the sum of the `schedule`, `execute`, `can_finish`, and `ack` sub-phases. #### `starrocks_fe_txn_publish_schedule_latency_ms`[​](#starrocks_fe_txn_publish_schedule_latency_ms "Direct link to starrocks_fe_txn_publish_schedule_latency_ms") * Unit: ms * Type: Summary * Description: The time a transaction spends waiting to be published after it has been committed, measured from `commit` time to when the publish task is picked up. This metric reflects scheduling delays or queueing time in the `publish` pipeline. #### `starrocks_fe_txn_total_latency_ms`[​](#starrocks_fe_txn_total_latency_ms "Direct link to starrocks_fe_txn_total_latency_ms") * Unit: ms * Type: Summary * Description: The total latency for a transaction to complete, measured from the `prepare` time to the `finish` time. This metric represents the full end-to-end duration of a transaction. #### `starrocks_fe_txn_write_latency_ms`[​](#starrocks_fe_txn_write_latency_ms "Direct link to starrocks_fe_txn_write_latency_ms") * Unit: ms * Type: Summary * Description: The latency of the `write` phase of a transaction, from `prepare` time to `commit` time. This metric isolates the performance of the data writing and preparation stage before the transaction is ready to be published. #### `starrocks_fe_unfinished_backup_job`[​](#starrocks_fe_unfinished_backup_job "Direct link to starrocks_fe_unfinished_backup_job") * Unit: Count * Type: Instantaneous * Description: Indicates the number of running BACKUP tasks under the specific warehouse. For a shared-nothing cluster, this item only monitors the default warehouse. For a shared-data cluster, this value is always `0`. #### `starrocks_fe_unfinished_query`[​](#starrocks_fe_unfinished_query "Direct link to starrocks_fe_unfinished_query") * Unit: Count * Type: Instantaneous * Description: Indicates the number of queries currently running under the specific warehouse. For a shared-nothing cluster, this item only monitors the default warehouse. #### `starrocks_fe_unfinished_restore_job`[​](#starrocks_fe_unfinished_restore_job "Direct link to starrocks_fe_unfinished_restore_job") * Unit: Count * Type: Instantaneous * Description: Indicates the number of running RESTORE tasks under the specific warehouse. For a shared-nothing cluster, this item only monitors the default warehouse. For a shared-data cluster, this value is always `0`. #### `storage_page_cache_mem_bytes`[​](#storage_page_cache_mem_bytes "Direct link to storage_page_cache_mem_bytes") * Unit: Bytes * Description: Memory used by storage page cache. #### `stream_load`[​](#stream_load "Direct link to stream_load") * Unit: - * Description: Total loaded rows and received bytes. #### `stream_load_pipe_count`[​](#stream_load_pipe_count "Direct link to stream_load_pipe_count") * Unit: Count * Description: Number of currently running Stream Load tasks. #### `streaming_load_bytes`[​](#streaming_load_bytes "Direct link to streaming_load_bytes") * Unit: Bytes * Description: Total bytes loaded by Stream Load. #### `streaming_load_current_processing`[​](#streaming_load_current_processing "Direct link to streaming_load_current_processing") * Unit: Count * Description: Number of currently running Stream Load tasks. #### `streaming_load_duration_ms`[​](#streaming_load_duration_ms "Direct link to streaming_load_duration_ms") * Unit: ms * Description: Total time spent on Stream Load. #### `streaming_load_requests_total`[​](#streaming_load_requests_total "Direct link to streaming_load_requests_total") * Unit: Count * Description: Total number of Stream Load requests. ###### SPLIT[​](#split "Direct link to SPLIT") --- ### Metrics t through z #### `tablet_base_max_compaction_score`[​](#tablet_base_max_compaction_score "Direct link to tablet_base_max_compaction_score") * Unit: - * Description: Highest base compaction score of tablets in this BE. #### `tablet_cumulative_max_compaction_score`[​](#tablet_cumulative_max_compaction_score "Direct link to tablet_cumulative_max_compaction_score") * Unit: - * Description: Highest cumulative compaction score of tablets in this BE. #### `tablet_metadata_mem_bytes`[​](#tablet_metadata_mem_bytes "Direct link to tablet_metadata_mem_bytes") * Unit: Bytes * Description: Memory used by tablet metadata. #### `tablet_pre_split_data_tier_file_selection`[​](#tablet_pre_split_data_tier_file_selection "Direct link to tablet_pre_split_data_tier_file_selection") * Unit: Count * Type: Cumulative * Labels: `mode` — `subset` (the sample scanned a subset of the files, sized by `tablet_pre_split_data_tier_scan_byte_limit`), `under_limit` (the input was within the limit, so every file was scanned), `disabled` (the limit is `0`, so every file was scanned), `all_selected` (the input exceeded the limit, but taking whole files, at least one per partition and at least `tablet_pre_split_data_tier_min_scan_files` in all, still selected every file), `partition_from_file_data` (the input exceeded the limit, but a partition column is read from the file data rather than from the path, or path and literal partition columns are mixed, so every file was scanned: a subset could miss whole partitions), `path_not_expressible` (`INSERT INTO ... SELECT FROM FILES()` only: a selected file's path contains a character FILES cannot take as an exact path — `,`, `*`, `?`, `[`, `{`, `\`, a `:` in the path after the host, or leading or trailing whitespace — so the sample scanned every file through the statement's own `path`). * Description: Total data-tier samples of Broker Load and `INSERT INTO ... SELECT FROM FILES()` loads, by how the scanned files were chosen. #### `tablet_pre_split_data_tier_scanned_bytes_percent`[​](#tablet_pre_split_data_tier_scanned_bytes_percent "Direct link to tablet_pre_split_data_tier_scanned_bytes_percent") * Unit: - * Type: Histogram * Description: Share of the input bytes that each data-tier sample of a Broker Load or `INSERT INTO ... SELECT FROM FILES()` load scanned, as a percentage rounded down to an integer, so `100` means every file was scanned. #### `tablet_schema_mem_bytes`[​](#tablet_schema_mem_bytes "Direct link to tablet_schema_mem_bytes") * Unit: Bytes * Description: Memory used by tablet schema. #### `tablet_update_max_compaction_score`[​](#tablet_update_max_compaction_score "Direct link to tablet_update_max_compaction_score") * Unit: - * Description: Highest compaction score of tablets in Primary Key tables in the current BE. #### `threadpool_task_exception_total`[​](#threadpool_task_exception_total "Direct link to threadpool_task_exception_total") * Unit: Count * Description: Cumulative number of task exceptions caught and swallowed by ThreadPool worker threads across the BE process. Increments only when [`enable_threadpool_catch_task_exception`](https://docs.starrocks.io/docs/administration/configuration/BE_parameters/log_server_meta.md#enable_threadpool_catch_task_exception) is `true`. When that item is `false` (default), this metric stays unchanged because there is no enclosing catch clause. Use it to alert on swallowed failures while catch mode is enabled; pool name and exception detail remain in the BE ERROR logs. #### `thrift_client_retries_total`[​](#thrift_client_retries_total "Direct link to thrift_client_retries_total") * Unit: Count * Type: Cumulative * Description: Total number of retries that FE Thrift RPC calls made after their connection broke. Before each retry, the FE waits a random interval with exponential backoff (100 to 200 ms before the first retry, with both bounds doubling for each further retry until the wait reaches 500 to 1000 ms), then reopens the connection and resends the request. Calls that time out are not retried and are not counted. The number of retries per call is bounded by `thrift_rpc_retry_times`. A rising value means the FE keeps losing its Thrift connections to other nodes, for example because the Thrift worker pool of the Leader FE is saturated and closing new connections; read it together with `thrift_server_rejected_connections_total` on that node. #### `thrift_connections_total`[​](#thrift_connections_total "Direct link to thrift_connections_total") * Unit: Count * Description: Total number of thrift connections (including finished connections). #### `thrift_current_connections (Deprecated)`[​](#thrift_current_connections-deprecated "Direct link to thrift_current_connections-deprecated") #### `thrift_opened_clients`[​](#thrift_opened_clients "Direct link to thrift_opened_clients") * Unit: Count * Description: Number of currently opened thrift clients. #### `thrift_server_acceptor_stall_ms`[​](#thrift_server_acceptor_stall_ms "Direct link to thrift_server_acceptor_stall_ms") * Unit: ms * Type: Instantaneous * Description: Milliseconds since the FE Thrift accept loop last returned a connection. A wedged acceptor makes this climb, but so does an FE with no Thrift traffic, so read it together with the connection arrival rate rather than alerting on it alone. `0` also covers the case where no acceptor is running at all, before the Thrift server starts and after it stops, so a threshold alert on this metric cannot tell a dead Thrift service from a healthy one. #### `thrift_server_expired_connections_total`[​](#thrift_server_expired_connections_total "Direct link to thrift_server_expired_connections_total") * Unit: Count * Type: Cumulative * Description: Total number of connections that the FE Thrift server closed without serving because they had waited in the pending queue longer than `thrift_server_queue_timeout_ms`. That check is disabled by default, so this counter stays at 0 until an operator arms the timeout. A rising value means the FE was working through a backlog of connections whose callers had almost certainly given up already; read it together with `thrift_server_queue_wait_ms` to see how far behind the queue had fallen. #### `thrift_server_queue_wait_ms`[​](#thrift_server_queue_wait_ms "Direct link to thrift_server_queue_wait_ms") * Unit: ms * Type: Instantaneous * Description: Quantiles of how long connections waited in the FE Thrift server pending queue before a worker thread picked them up. Sampled for every dequeued connection, including the ones then dropped as expired, so the distribution is not truncated at `thrift_server_queue_timeout_ms`. This is the leading indicator of Thrift saturation: it climbs while the worker pool is still keeping up, well before `thrift_server_rejected_connections_total` starts to move. #### `thrift_server_rejected_connections_total`[​](#thrift_server_rejected_connections_total "Direct link to thrift_server_rejected_connections_total") * Unit: Count * Type: Cumulative * Description: Total number of connections the FE Thrift server closed immediately because its worker pool was saturated. Every rejection is counted, including those whose log warning was suppressed by rate limiting. A rising value means clients are being turned away; read it with the `thread_pool` metrics for `thrift-server-pool`, where the rate of increase indicates how far worker capacity is short. #### `thrift_used_clients`[​](#thrift_used_clients "Direct link to thrift_used_clients") * Unit: Count * Description: Number of thrift clients in use currently. #### `total_column_pool_bytes (Deprecated)`[​](#total_column_pool_bytes-deprecated "Direct link to total_column_pool_bytes-deprecated") #### `transaction_streaming_load_bytes`[​](#transaction_streaming_load_bytes "Direct link to transaction_streaming_load_bytes") * Unit: Bytes * Description: Total loading bytes of transaction load. #### `transaction_streaming_load_current_processing`[​](#transaction_streaming_load_current_processing "Direct link to transaction_streaming_load_current_processing") * Unit: Count * Description: Number of currently running transactional Stream Load tasks. #### `transaction_streaming_load_duration_ms`[​](#transaction_streaming_load_duration_ms "Direct link to transaction_streaming_load_duration_ms") * Unit: ms * Description: Total time spent on Stream Load transaction Interface. #### `transaction_streaming_load_requests_total`[​](#transaction_streaming_load_requests_total "Direct link to transaction_streaming_load_requests_total") * Unit: Count * Description: Total number of transaction load requests. #### `txn_request`[​](#txn_request "Direct link to txn_request") * Unit: - * Description: Transaction requests of BEGIN, COMMIT, ROLLBACK, and EXEC. #### `uint8_column_pool_bytes`[​](#uint8_column_pool_bytes "Direct link to uint8_column_pool_bytes") * Unit: Bytes * Description: Bytes used by the UINT8 column pool. #### `unused_rowsets_count`[​](#unused_rowsets_count "Direct link to unused_rowsets_count") * Unit: Count * Description: Total number of unused rowsets. Please note that these rowsets will be reclaimed later. #### `update_apply_queue_count`[​](#update_apply_queue_count "Direct link to update_apply_queue_count") * Unit: Count * Description: Queued task count in the Primary Key table transaction APPLY thread pool. #### `update_compaction_duration_us`[​](#update_compaction_duration_us "Direct link to update_compaction_duration_us") * Unit: us * Description: Total time spent on Primary Key table compactions. #### `update_compaction_outputs_bytes_total`[​](#update_compaction_outputs_bytes_total "Direct link to update_compaction_outputs_bytes_total") * Unit: Bytes * Description: Total bytes written by Primary Key table compactions. #### `update_compaction_outputs_total`[​](#update_compaction_outputs_total "Direct link to update_compaction_outputs_total") * Unit: Count * Description: Total number of Primary Key table compactions. #### `update_compaction_task_byte_per_second`[​](#update_compaction_task_byte_per_second "Direct link to update_compaction_task_byte_per_second") * Unit: Bytes/s * Description: Estimated rate of Primary Key table compactions. #### `update_compaction_task_cost_time_ns`[​](#update_compaction_task_cost_time_ns "Direct link to update_compaction_task_cost_time_ns") * Unit: ns * Description: Total time spent on the Primary Key table compactions. #### `update_del_vector_bytes_total`[​](#update_del_vector_bytes_total "Direct link to update_del_vector_bytes_total") * Unit: Bytes * Description: Total memory used for caching DELETE vectors in Primary Key tables. #### `update_del_vector_deletes_new`[​](#update_del_vector_deletes_new "Direct link to update_del_vector_deletes_new") * Unit: Count * Description: Total number of newly generated DELETE vectors used in Primary Key tables. #### `update_del_vector_deletes_total (Deprecated)`[​](#update_del_vector_deletes_total-deprecated "Direct link to update_del_vector_deletes_total-deprecated") #### `update_del_vector_dels_num (Deprecated)`[​](#update_del_vector_dels_num-deprecated "Direct link to update_del_vector_dels_num-deprecated") #### `update_del_vector_num`[​](#update_del_vector_num "Direct link to update_del_vector_num") * Unit: Count * Description: Number of the DELETE vector cache items in Primary Key tables. #### `update_mem_bytes`[​](#update_mem_bytes "Direct link to update_mem_bytes") * Unit: Bytes * Description: Memory used by Primary Key table APPLY tasks and Primary Key index. #### `update_primary_index_bytes_total`[​](#update_primary_index_bytes_total "Direct link to update_primary_index_bytes_total") * Unit: Bytes * Description: Total memory cost of the Primary Key index. #### `update_primary_index_num`[​](#update_primary_index_num "Direct link to update_primary_index_num") * Unit: Count * Description: Number of Primary Key indexes cached in memory. #### `update_rowset_commit_apply_duration_us`[​](#update_rowset_commit_apply_duration_us "Direct link to update_rowset_commit_apply_duration_us") * Unit: us * Description: Total time spent on Primary Key table APPLY tasks. #### `update_rowset_commit_apply_total`[​](#update_rowset_commit_apply_total "Direct link to update_rowset_commit_apply_total") * Unit: Count * Description: Total number of COMMIT and APPLY for Primary Key tables. #### `update_rowset_commit_request_failed`[​](#update_rowset_commit_request_failed "Direct link to update_rowset_commit_request_failed") * Unit: Count * Description: Total number of failed rowset COMMIT requests in Primary Key tables. #### `update_rowset_commit_request_total`[​](#update_rowset_commit_request_total "Direct link to update_rowset_commit_request_total") * Unit: Count * Description: Total number of rowset COMMIT requests in Primary Key tables. #### `wait_base_compaction_task_num`[​](#wait_base_compaction_task_num "Direct link to wait_base_compaction_task_num") * Unit: Count * Description: Number of base compaction tasks waiting for execution. #### `wait_cumulative_compaction_task_num`[​](#wait_cumulative_compaction_task_num "Direct link to wait_cumulative_compaction_task_num") * Unit: Count * Description: Number of cumulative compaction tasks waiting for execution. #### `writable_blocks_total (Deprecated)`[​](#writable_blocks_total-deprecated "Direct link to writable_blocks_total-deprecated") --- ### General Monitoring Metrics note Metrics for materialized views and shared-data clusters are detailed in the corresponding sections: * [Metrics for asynchronous materialized view metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics-materialized_view.md) * [Metrics for Shared-data Dashboard metrics, and Starlet Dashboard metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics-shared-data.md) For more information on how to build a monitoring service for your StarRocks cluster, see [Monitor and Alert](https://docs.starrocks.io/docs/administration/management/monitoring.md). Monitoring metrics are listed alphabetically in these files: * [a - c](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/a-c.md) * [d - h](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/d-h.md) * [i - p](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/i-p.md) * [q - r](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/q-r.md) * [s](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/s.md) * [t - z](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/t-z.md) --- ### Monitoring Metrics for Asynchronous Materialized Views From v3.1 onwards, StarRocks supports metrics for asynchronous materialized views. To allow Prometheus to access the materialized view metadata in your cluster, you must add the following configurations in the Prometheus configuration file **prometheus/prometheus.yml**: ```yaml global: .... scrape_configs: - job_name: 'dev' metrics_path: '/metrics' # Add the following configurations. basic_auth: username: 'root' password: '' params: 'with_materialized_view_metrics' : ['all'] .... ``` * `username`: The username used to log into your StarRocks cluster. Unless using the root account, the user must be granted both the `user_admin` and `db_admin` roles. * `password`: The password used to log into your StarRocks cluster. * `'with_materialized_view_metrics'`: The scope of the metrics to collect. Valid values: * `'all'`: All metrics relevant to materialized views are collected. * `'minified'`: Gauge metrics and metrics whose values are `0` will not be collected. #### Metric items[​](#metric-items "Direct link to Metric items") ##### mv\_refresh\_jobs[​](#mv_refresh_jobs "Direct link to mv_refresh_jobs") * Type: Counter * Description: Total number of refresh jobs triggered for the materialized view. A refresh job corresponds to one user-initiated or scheduled refresh; a single job may execute multiple task runs internally. Each job is counted once when it reaches a terminal state. MERGED task runs (sub-tasks merged into a later batch) are not counted. ##### mv\_refresh\_total\_success\_jobs[​](#mv_refresh_total_success_jobs "Direct link to mv_refresh_total_success_jobs") * Type: Counter * Description: Number of refresh jobs that completed successfully. Counted once per job. ##### mv\_refresh\_total\_failed\_jobs[​](#mv_refresh_total_failed_jobs "Direct link to mv_refresh_total_failed_jobs") * Type: Counter * Description: Number of refresh jobs that failed. Counted once per job. ##### mv\_refresh\_total\_empty\_jobs[​](#mv_refresh_total_empty_jobs "Direct link to mv_refresh_total_empty_jobs") * Type: Counter * Description: Number of canceled refresh jobs of the materialized view because the data to refresh is empty. ##### mv\_refresh\_total\_retry\_meta\_count[​](#mv_refresh_total_retry_meta_count "Direct link to mv_refresh_total_retry_meta_count") * Type: Counter * Description: Number of times when the materialized view refresh job checks whether the base table is updated. ##### mv\_query\_total\_count[​](#mv_query_total_count "Direct link to mv_query_total_count") * Type: Counter * Description: Number of times when the materialized view is used in the pre-processing of a query. ##### mv\_query\_total\_hit\_count[​](#mv_query_total_hit_count "Direct link to mv_query_total_hit_count") * Type: Counter * Description: Number of times when the materialized view is considered able to rewrite a query in the query plan. This value may appear higher because the final query plan may skip rewriting due to a high cost. ##### mv\_query\_total\_considered\_count[​](#mv_query_total_considered_count "Direct link to mv_query_total_considered_count") * Type: Counter * Description: Number of times when the materialized view rewrites a query (excluding direct queries against the materialized view). ##### mv\_query\_total\_matched\_count[​](#mv_query_total_matched_count "Direct link to mv_query_total_matched_count") * Type: Counter * Description: Number of times when the materialized view is involved in the final plan of a query (including direct queries against the materialized view). ##### mv\_refresh\_pending\_jobs[​](#mv_refresh_pending_jobs "Direct link to mv_refresh_pending_jobs") * Type: Gauge * Description:| Number of currently pending refresh jobs of the materialized view. ##### mv\_refresh\_running\_jobs[​](#mv_refresh_running_jobs "Direct link to mv_refresh_running_jobs") * Type: Gauge * Description:| Number of currently running refresh jobs of the materialized view. ##### mv\_row\_count[​](#mv_row_count "Direct link to mv_row_count") * Type: Gauge * Description:| Row count of the materialized view. ##### mv\_storage\_size[​](#mv_storage_size "Direct link to mv_storage_size") * Type: Gauge * Description:| Size of the materialized view. Unit: Byte. ##### mv\_inactive\_state[​](#mv_inactive_state "Direct link to mv_inactive_state") * Type: Gauge * Description:| Status of the materialized view. Valid values: `0`(active) and `1`(inactive). ##### mv\_partition\_count[​](#mv_partition_count "Direct link to mv_partition_count") * Type: Gauge * Description:| Number of partitions in the materialized view. The value is `0` if the materialized view is not partitioned. ##### mv\_refresh\_duration[​](#mv_refresh_duration "Direct link to mv_refresh_duration") * Type: Histogram * Description: Wall-clock duration of refresh jobs, in milliseconds. For multi-batch jobs, measured from the first task run start to the final task run completion. ##### mv\_global\_count[​](#mv_global_count "Direct link to mv_global_count") * Type: Gauge * Description: Current number of asynchronous materialized views in the cluster, with labels `refresh_mode` (the materialized view's refresh mode) and `status` (`ACTIVE` or `INACTIVE`). This metric is always emitted, regardless of the per-materialized-view metrics privilege. ##### mv\_global\_query\_rewrite\_queries\_total[​](#mv_global_query_rewrite_queries_total "Direct link to mv_global_query_rewrite_queries_total") * Type: Counter * Description: Number of queries grouped by materialized view rewrite outcome, with label `state`: `HIT` (the query was rewritten to use a materialized view), `NO_HIT` (rewrite was enabled but no materialized view was used), or `DISABLED` (materialized view rewrite was disabled by the session variable or the FE configuration). Counted once per query. ##### mv\_global\_query\_mv\_usage\_total[​](#mv_global_query_mv_usage_total "Direct link to mv_global_query_mv_usage_total") * Type: Counter * Description: Number of times materialized views are used by queries, with labels `usage_type` (`REWRITE` if a query was rewritten to use the materialized view, or `DIRECT` if a query reads the materialized view directly) and `refresh_mode`. ##### mv\_global\_refresh\_jobs\_total[​](#mv_global_refresh_jobs_total "Direct link to mv_global_refresh_jobs_total") * Type: Counter * Description: Total number of materialized view refresh jobs across all materialized views, with label `warehouse_name`. This is the fleet-level aggregate of `mv_refresh_jobs`: counted once per job on its terminal task run, excluding `MERGED` runs. Always emitted, regardless of the per-materialized-view metrics privilege. ##### mv\_global\_refresh\_success\_jobs\_total[​](#mv_global_refresh_success_jobs_total "Direct link to mv_global_refresh_success_jobs_total") * Type: Counter * Description: Total number of successful materialized view refresh jobs across all materialized views, by `warehouse_name`. ##### mv\_global\_refresh\_failed\_jobs\_total[​](#mv_global_refresh_failed_jobs_total "Direct link to mv_global_refresh_failed_jobs_total") * Type: Counter * Description: Total number of failed materialized view refresh jobs across all materialized views, by `warehouse_name`. ##### mv\_global\_refresh\_duration[​](#mv_global_refresh_duration "Direct link to mv_global_refresh_duration") * Type: Histogram * Description: Per-job wall-clock duration of materialized view refresh jobs, in milliseconds, by `warehouse_name`. ##### mv\_global\_refresh\_pending\_jobs[​](#mv_global_refresh_pending_jobs "Direct link to mv_global_refresh_pending_jobs") * Type: Gauge * Description: Current number of pending materialized view refresh jobs across all materialized views, aggregated by `warehouse_name`. ##### mv\_global\_refresh\_running\_jobs[​](#mv_global_refresh_running_jobs "Direct link to mv_global_refresh_running_jobs") * Type: Gauge * Description: Current number of running materialized view refresh jobs across all materialized views, aggregated by `warehouse_name`. --- ### Monitoring Metrics for Shared-data Clusters StarRocks provides two Dashboard templates for shared-data clusters: * [Shared-data Dashboard](#shared-data-dashboard) * [Starlet Dashboard](#starlet-dashboard) #### Shared-data Dashboard[​](#shared-data-dashboard "Direct link to Shared-data Dashboard") Shared-data Dashboard includes the following categories of monitoring metrics: * [Publish Version](#publish-version) * [Metadata](#metadata) * [Metacache](#metacache) * [Vacuum](#vacuum) * [Loading](#loading) ##### Publish Version[​](#publish-version "Direct link to Publish Version") ###### Latency / QPS[​](#latency--qps "Direct link to Latency / QPS") * Description: Quantile latency, average latency, and QPS of Public Version tasks. ###### Queued Tasks[​](#queued-tasks "Direct link to Queued Tasks") * Description: The number of Public Version tasks in the queue. ##### Metadata[​](#metadata "Direct link to Metadata") ###### Get Tablet Metadata[​](#get-tablet-metadata "Direct link to Get Tablet Metadata") * Description: Quantile latency, average latency, and QPS of Get Tablet Metadata tasks. ###### Put Tablet Metadata[​](#put-tablet-metadata "Direct link to Put Tablet Metadata") * Description: Quantile latency, average latency, and QPS of Put Tablet Metadata tasks. ###### Get Txn Log[​](#get-txn-log "Direct link to Get Txn Log") * Description: Quantile latency, average latency, and QPS of Get Txn Log tasks. ###### Put Txn Log[​](#put-txn-log "Direct link to Put Txn Log") * Description: Quantile latency, average latency, and QPS of Put Txn Log tasks. ##### Metacache[​](#metacache "Direct link to Metacache") ###### Metacache Usage[​](#metacache-usage "Direct link to Metacache Usage") * Description: Metacache utilization rate. ###### Delvec Cache Miss Per Minute[​](#delvec-cache-miss-per-minute "Direct link to Delvec Cache Miss Per Minute") * Description: Number of cache misses in Delvec Cache per minute. ###### Metadata Cache Miss Per Minute[​](#metadata-cache-miss-per-minute "Direct link to Metadata Cache Miss Per Minute") * Description: Number of cache misses in Metadata Cache per minute. ###### Txn Log Cache Miss Per Minute[​](#txn-log-cache-miss-per-minute "Direct link to Txn Log Cache Miss Per Minute") * Description: Number of cache misses in Txn Log Cache per minute. ###### Segment Cache Miss Per Minute[​](#segment-cache-miss-per-minute "Direct link to Segment Cache Miss Per Minute") * Description: Number of cache misses in Segment Cache per minute. ##### Vacuum[​](#vacuum "Direct link to Vacuum") ###### Vacuum Deletes[​](#vacuum-deletes "Direct link to Vacuum Deletes") * Description: Quantile latency, average latency, and QPS of Vacuum Deletes tasks. ###### Errors[​](#errors "Direct link to Errors") * Description: Number of failed Vacuum Deletes operations. ##### Loading[​](#loading "Direct link to Loading") ###### Queue Size[​](#queue-size "Direct link to Queue Size") * Description: Queue size of BE Async Delta Writer. #### Starlet Dashboard[​](#starlet-dashboard "Direct link to Starlet Dashboard") Starlet Dashboard includes the following categories of monitoring metrics: * [FSLIB READ IO METRICS](#fslib-read-io-metrics) * [FSLIB WRITE IO METRICS](#fslib-write-io-metrics) * [S3 IO METRICS](#s3-io-metrics) * [FSLIB CACHE METRICS](#fslib-cache-metrics) * [FSLIB FS METRICS](#fslib-fs-metrics) ##### FSLIB READ IO METRICS[​](#fslib-read-io-metrics "Direct link to FSLIB READ IO METRICS") ###### fslib read io\_latency (quantile)[​](#fslib-read-io_latency-quantile "Direct link to fslib read io_latency (quantile)") * Type: Histogram * Description: Quantile latency for S3 reads. ###### fslib read io\_latency (average)[​](#fslib-read-io_latency-average "Direct link to fslib read io_latency (average)") * Type: Counter * Description: Average latency for S3 reads. ###### fslib total read data[​](#fslib-total-read-data "Direct link to fslib total read data") * Type: Counter * Description: Total data size for S3 reads. ###### fslib read iosize (quantile)[​](#fslib-read-iosize-quantile "Direct link to fslib read iosize (quantile)") * Type: Histogram * Description: Quantile I/O size for S3 reads. ###### fslib read iosize (average)[​](#fslib-read-iosize-average "Direct link to fslib read iosize (average)") * Type: Counter * Description: Average I/O size for S3 reads. ###### fslib read throughput[​](#fslib-read-throughput "Direct link to fslib read throughput") * Type: Counter * Description: I/O throughput per second for S3 reads. ###### fslib read iops[​](#fslib-read-iops "Direct link to fslib read iops") * Type: Counter * Description: Number of I/O operations per second for S3 reads. ##### FSLIB WRITE IO METRICS[​](#fslib-write-io-metrics "Direct link to FSLIB WRITE IO METRICS") ###### fslib write io\_latency (quantile)[​](#fslib-write-io_latency-quantile "Direct link to fslib write io_latency (quantile)") * Type: Histogram * Description: Quantile latency for application writes. Please note that this value may appear lower because this metric monitors only data written to the buffer. ###### fslib write io\_latency (average)[​](#fslib-write-io_latency-average "Direct link to fslib write io_latency (average)") * Type: Counter * Description: Average latency for application writes. Please note that this value may appear lower because this metric monitors only data written to the buffer. ###### fslib total write data[​](#fslib-total-write-data "Direct link to fslib total write data") * Type: Counter * Description: Total data size for application writes. ###### fslib write iosize (quantile)[​](#fslib-write-iosize-quantile "Direct link to fslib write iosize (quantile)") * Type: Histogram * Description: Quantile I/O size for application writes. ###### fslib write iosize (average)[​](#fslib-write-iosize-average "Direct link to fslib write iosize (average)") * Type: Counter * Description: Average I/O size for application writes. ###### fslib write throughput[​](#fslib-write-throughput "Direct link to fslib write throughput") * Type: Counter * Description: I/O throughput per second for application writes. ##### S3 IO METRICS[​](#s3-io-metrics "Direct link to S3 IO METRICS") ###### fslib s3 single upload iops[​](#fslib-s3-single-upload-iops "Direct link to fslib s3 single upload iops") * Type: Counter * Description: Number of invocations per second for S3 Put Object. ###### fslib s3 single upload iosize (quantile)[​](#fslib-s3-single-upload-iosize-quantile "Direct link to fslib s3 single upload iosize (quantile)") * Type: Histogram * Description: Quantile I/O size for S3 Put Object. ###### fslib s3 single upload latency (quantile)[​](#fslib-s3-single-upload-latency-quantile "Direct link to fslib s3 single upload latency (quantile)") * Type: Histogram * Description: Quantile latency for S3 Put Object. ###### fslib s3 multi upload iops[​](#fslib-s3-multi-upload-iops "Direct link to fslib s3 multi upload iops") * Type: Counter * Description: Number of invocations per second for S3 Multi Upload Object. ###### fslib s3 multi upload iosize (quantile)[​](#fslib-s3-multi-upload-iosize-quantile "Direct link to fslib s3 multi upload iosize (quantile)") * Type: Histogram * Description: Quantile I/O size for S3 Multi Upload Object. ###### fslib s3 multi upload latency (quantile)[​](#fslib-s3-multi-upload-latency-quantile "Direct link to fslib s3 multi upload latency (quantile)") * Type: Histogram * Description: Quantile latency for S3 Multi Upload Object. ###### fslib s3 complete multi upload latency (quantile)[​](#fslib-s3-complete-multi-upload-latency-quantile "Direct link to fslib s3 complete multi upload latency (quantile)") * Type: Histogram * Description: Quantile latency for S3 Complete Multi Upload Object. ##### FSLIB CACHE METRICS[​](#fslib-cache-metrics "Direct link to FSLIB CACHE METRICS") ###### fslib cache hit ratio[​](#fslib-cache-hit-ratio "Direct link to fslib cache hit ratio") * Type: Counter * Description: Cache hit ratio. ###### fslib cache hits/misses[​](#fslib-cache-hitsmisses "Direct link to fslib cache hits/misses") * Type: Counter * Description: Number of cache hits/misses per second. ##### FSLIB FS METRICS[​](#fslib-fs-metrics "Direct link to FSLIB FS METRICS") ###### fslib alive fs instances count[​](#fslib-alive-fs-instances-count "Direct link to fslib alive fs instances count") * Type: Gauge * Description: Number of file system instances that are alive. ###### fslib open files[​](#fslib-open-files "Direct link to fslib open files") * Type: Counter * Description: Cumulative number of opened files. ###### fslib create files[​](#fslib-create-files "Direct link to fslib create files") * Type: Counter * Description: Average number of files created per second. ###### filesystem meta operations[​](#filesystem-meta-operations "Direct link to filesystem meta operations") * Type: Counter * Description: Average number of directory listing operations per second. ###### fslib async caches[​](#fslib-async-caches "Direct link to fslib async caches") * Type: Counter * Description: Cumulative number of files in asynchronous cache. ###### fslib create files (TOTAL)[​](#fslib-create-files-total "Direct link to fslib create files (TOTAL)") * Type: Counter * Description: Cumulative number of files created. ###### fslib async tasks[​](#fslib-async-tasks "Direct link to fslib async tasks") * Type: Counter * Description: Cumulative number of asynchronous tasks in the queue. --- ### Process Profile (Proc Profile) The **Process Profile** (Proc Profile) feature provides a built-in mechanism to collect and visualize performance profiles for StarRocks Frontend (FE) and Backend (BE) processes. By generating flame graphs for CPU and memory allocation, it helps developers and administrators diagnose performance bottlenecks, high resource utilization, and complex runtime issues directly from the Web UI. #### Overview[​](#overview "Direct link to Overview") Process Profiling is a system-level diagnostic tool that captures the state of the StarRocks processes over a period of time. Unlike Query Profiles, which focus on individual SQL execution, Process Profiles provide a holistic view of what the processes are doing, including background tasks, metadata management, and internal synchronization. #### Page Examples[​](#page-examples "Direct link to Page Examples") The Proc Profile interface is integrated into the StarRocks Web UI under the **proc profiles** tab. ##### Profile List View[​](#profile-list-view "Direct link to Profile List View") The main page displays a list of collected profile files for the selected node. You can switch between FE and different BE nodes using the tabs at the top. ![Profile List Example](/assets/images/proc_profile_1-60bd9228b396079059a710aca842f293.png) ##### Flame Graph[​](#flame-graph "Direct link to Flame Graph") ![Flame Graph](/assets/images/proc_profile_2-33e9ad52afee3e204664f4eede245e48.png) A Flame Graph is a visualization tool used to show the resource consumption distribution of functions or code paths in a program. It represents the call stack using stacked rectangles (usually in a horizontal "flame" shape), where: * **Each box represents a function (or method) call.** * **The width of a box represents the amount of resources consumed by that function (such as CPU time, memory allocation frequency, or lock wait time). The wider the box, the more resources are consumed.** * **Vertical stacking represents the calling relationship, with the bottom layer being the entry function and child functions stacked on top.** Flame graphs help developers and operations personnel quickly identify code hotspots, performance bottlenecks, and call paths with the highest resource consumption, and are commonly used for performance tuning and troubleshooting. In StarRocks Process Profile, flame graphs are used to visually display the data distribution of CPU usage, memory allocation, and (in BE scenarios) lock contention, helping to locate the most time-consuming code segments or call paths. #### Use Scenarios[​](#use-scenarios "Direct link to Use Scenarios") * **CPU Hotspot Analysis**: Identify which code paths or functions are consuming the most CPU cycles. * **Memory Allocation Profiling**: Track where memory is being frequently allocated to find potential memory pressure sources. * **R\&D Troubleshooting**: Assist developers in analyzing complex bugs or performance regressions in production environments without needing external profiling tools. #### Feature Description[​](#feature-description "Direct link to Feature Description") ##### How to Use[​](#how-to-use "Direct link to How to Use") 1. **Access**: Open the StarRocks Web UI (default port 8030) and click on the **proc profiles** tab. 2. **Select Node**: Choose the **FE** tab or a specific **BE** node tab. 3. **Visualize**: Click the **View** link for any entry. * For **FE**, it extracts and displays a pre-generated HTML flame graph. * For **BE**, it may convert a raw `pprof` file into an SVG flame graph on the fly (the first view might take a few seconds as it performs the conversion). 4. **Interact**: Use the flame graph to zoom into specific call stacks, search for function names, or analyze stack depth. ##### Information Provided[​](#information-provided "Direct link to Information Provided") * **Type**: CPU, Memory (Allocation), or Contention (BE only). * **Timestamp**: When the profile collection was completed. * **File Size**: Size of the compressed profile data. * **Flame Graph**: A hierarchical visualization where the width of each box represents the relative resource consumption (CPU time, allocation frequency, or lock wait time). #### Configuration[​](#configuration "Direct link to Configuration") The profiling feature performs function-level sampling to generate flame graphs for visualization. ##### Frontend (FE) Configuration[​](#frontend-fe-configuration "Direct link to Frontend (FE) Configuration") FE profiling is managed by an internal daemon and uses **AsyncProfiler** for data collection. You can configure these in `fe.conf`. | Parameter | Default | Description | | --------------------------------------- | ------------------ | --------------------------------------------------------------- | | `proc_profile_cpu_enable` | `true` | Whether to enable automatic CPU profiling for FE. | | `proc_profile_mem_enable` | `true` | Whether to enable automatic memory allocation profiling for FE. | | `proc_profile_collect_time_s` | `120` | Duration (seconds) for each profile collection. | | `proc_profile_jstack_depth` | `128` | Maximum Java stack depth to collect. | | `proc_profile_file_retained_days` | `1` | How many days to retain profile files. | | `proc_profile_file_retained_size_bytes` | `2147483648` (2GB) | Maximum total size of retained profile files. | ##### Backend (BE) Configuration[​](#backend-be-configuration "Direct link to Backend (BE) Configuration") BE profiles are collected using the built-in **gperftools** and are typically collected via a background script or manual triggers. The collected data is then converted into flame graphs using **pprof**. ###### Configuration in `be.conf`[​](#configuration-in-beconf "Direct link to configuration-in-beconf") | Parameter | Default | Description | | ----------------------------- | ---------------------------------- | ---------------------------------------------------------------------------------------------- | | `brpc_port` | `8060` | Port used by collection scripts to fetch data from BE. | | `sys_log_dir` | `${STARROCKS_HOME}/log` | Base directory for storing collected profiles (stored in the `proc_profile` subdirectory). | | `flamegraph_tool_dir` | `${STARROCKS_HOME}/bin/flamegraph` | Path to conversion tools (**pprof**, `flamegraph.pl`). | | `COLLECT_BE_PROFILE_INTERVAL` | `60` | Collection interval in seconds when running the `collect_be_profile.sh` script in daemon mode. | ###### Manual BE Collection Options[​](#manual-be-collection-options "Direct link to Manual BE Collection Options") The `collect_be_profile.sh` script supports the following command-line options: | Option | Default | Description | | ------------------ | ------------------ | ----------------------------------------------------------- | | `--profiling-type` | `cpu` | Type of profile to collect: `cpu`, `contention`, or `both`. | | `--duration` | `10` | Duration (seconds) for each profile collection. | | `--interval` | `60` | Interval (seconds) between collections in daemon mode. | | `--cleanup-days` | `1` | Number of days to retain profile files. | | `--cleanup-size` | `2147483648` (2GB) | Maximum total size of retained profile files. | | `--daemon` | - | Run the collection script in daemon mode in the background. | ##### Manual BE Collection Example[​](#manual-be-collection-example "Direct link to Manual BE Collection Example") You can use the provided script to trigger or schedule BE collection: ```bash # Collect a 30-second CPU profile ./bin/collect_be_profile.sh --profiling-type cpu --duration 30 # Run in daemon mode to collect profiles every hour ./bin/collect_be_profile.sh --daemon --interval 3600 ``` --- ### Resource management #### [📄️ Resource group](https://docs.starrocks.io/docs/administration/management/resource_management/resource_group.md) [How to use resource groups in StarRocks to isolate resources and run multiple workloads concurrently in a single cluster.](https://docs.starrocks.io/docs/administration/management/resource_management/resource_group.md) #### [📄️ Query queues](https://docs.starrocks.io/docs/administration/management/resource_management/query_queues.md) [From v2.5, StarRocks supports query queues.](https://docs.starrocks.io/docs/administration/management/resource_management/query_queues.md) #### [📄️ Query Management](https://docs.starrocks.io/docs/administration/management/resource_management/Query_management.md) [Configure per-user query properties in StarRocks: connection limits, query timeouts, and resource allocation using ALTER USER SET PROPERTIES.](https://docs.starrocks.io/docs/administration/management/resource_management/Query_management.md) #### [📄️ Memory Management](https://docs.starrocks.io/docs/administration/management/resource_management/Memory_management.md) [Overview of memory classification and memory management strategies in StarRocks.](https://docs.starrocks.io/docs/administration/management/resource_management/Memory_management.md) #### [📄️ Spill to disk](https://docs.starrocks.io/docs/administration/management/resource_management/spill_to_disk.md) [How to configure StarRocks to spill intermediate query results to disk or object storage to prevent out-of-memory errors.](https://docs.starrocks.io/docs/administration/management/resource_management/spill_to_disk.md) #### [📄️ Load Balancing](https://docs.starrocks.io/docs/administration/management/resource_management/Load_balance.md) [When deploying multiple FE nodes, users can deploy a load balancing layer on top of the FEs to achieve high availability.](https://docs.starrocks.io/docs/administration/management/resource_management/Load_balance.md) #### [📄️ Manage replica](https://docs.starrocks.io/docs/administration/management/resource_management/Replica.md) [Manage data replicas in your StarRocks cluster.](https://docs.starrocks.io/docs/administration/management/resource_management/Replica.md) #### [📄️ Blacklist Management](https://docs.starrocks.io/docs/administration/management/resource_management/Blacklist.md) [In some cases, administrators need to disable certain patterns of SQL to avoid SQL from triggering cluster crashes or unexpected high concurrent queries.](https://docs.starrocks.io/docs/administration/management/resource_management/Blacklist.md) #### [📄️ Add labels on BEs](https://docs.starrocks.io/docs/administration/management/resource_management/be_label.md) [Since v3.2.8, StarRocks supports adding labels on BEs. When creating a table or an asynchronous materialized view, you can specify the label of a certain...](https://docs.starrocks.io/docs/administration/management/resource_management/be_label.md) #### [📄️ File manager](https://docs.starrocks.io/docs/administration/management/resource_management/filemanager.md) [With file manager, you can create, view, and delete files, such as the files that are used to access external data sources: public key files, private key...](https://docs.starrocks.io/docs/administration/management/resource_management/filemanager.md) --- ### Add labels on BEs Since v3.2.8, StarRocks supports adding labels on BEs. When creating a table or an asynchronous materialized view, you can specify the label of a certain group of BE nodes. This ensures that data replicas are distributed only on the BE nodes associated with that label. Data replicas will be evenly distributed among nodes with the same label, enhancing data high availability and resource isolation. #### Usage[​](#usage "Direct link to Usage") ##### Add labels on BEs[​](#add-labels-on-bes-1 "Direct link to Add labels on BEs") Suppose that one StarRocks cluster includes six BEs which are distributed evenly across three racks. You can add labels on BEs based on the racks where the BEs are located. ```sql ALTER SYSTEM MODIFY BACKEND "172.xx.xx.46:9050" SET ("labels.location" = "rack:rack1"); ALTER SYSTEM MODIFY BACKEND "172.xx.xx.47:9050" SET ("labels.location" = "rack:rack1"); ALTER SYSTEM MODIFY BACKEND "172.xx.xx.48:9050" SET ("labels.location" = "rack:rack2"); ALTER SYSTEM MODIFY BACKEND "172.xx.xx.49:9050" SET ("labels.location" = "rack:rack2"); ALTER SYSTEM MODIFY BACKEND "172.xx.xx.50:9050" SET ("labels.location" = "rack:rack3"); ALTER SYSTEM MODIFY BACKEND "172.xx.xx.51:9050" SET ("labels.location" = "rack:rack3"); ``` After adding labels, you can execute `SHOW BACKENDS;` and view the labels of BEs in the `Location` field of the returned result. If you need to modify the labels of BEs, you can execute `ALTER SYSTEM MODIFY BACKEND "172.xx.xx.48:9050" SET ("labels.location" = "rack:xxx");`. ##### Use labels to specify table data distribution on BE nodes[​](#use-labels-to-specify-table-data-distribution-on-be-nodes "Direct link to Use labels to specify table data distribution on BE nodes") If you need to specify the locations to which a table's data is distributed, for example, distributing a table's data across two racks, rack1 and rack2, you can add labels to the table. After labels are added, all the replicas of the same tablet in the table are distributed across labels in a Round-Robin approach. Moreover, if multiple replicas of the same tablet exist within the same label, these replicas will be distributed as evenly as possible across different BEs in that label. note * If the total number of BE nodes associated with the labels is fewer than the number of replicas, the system will preferentially ensure there are enough replicas. In this case, replicas may not be distributed as the label specified. * The label to be associated with a table must already exist. Otherwise, an error `Getting analyzing error. Detail message: Cannot find any backend with location: rack:xxx` will occur. ###### At table creation[​](#at-table-creation "Direct link to At table creation") You can use the property `"labels.location"` to distribute the table's data across rack 1 and rack 2 at table creation: ```sql CREATE TABLE example_table ( order_id bigint NOT NULL, dt date NOT NULL, user_id INT NOT NULL, good_id INT NOT NULL, cnt int NOT NULL, revenue int NOT NULL ) PROPERTIES ("labels.location" = "rack:rack1,rack:rack2"); ``` For newly created tables, the default value of the table property `labels.location` is `*`, indicating that replicas are evenly distributed across all labels. If the data distribution of a newly created table does not need to be aware of the geographical locations of servers in the cluster, you can manually set the table property `"labels.location" = ""`. ###### After table creation[​](#after-table-creation "Direct link to After table creation") If you need to modify the data distribution location of the table after table creation, for example, modify the location to rack 1, rack 2, and rack 3, you can execute the following statement: ```sql ALTER TABLE example_table SET ("labels.location" = "rack:rack1,rack:rack2,rack:rack3"); ``` note If you have upgraded StarRocks to version 3.2.8 or later, for historical tables created before the upgrade, data is not distributed based on labels by default. If you need to distribute data of a historical table based on labels, you can execute the following statement to add labels to the historical table: ```sql ALTER TABLE example_table1 SET ("labels.location" = "rack:rack1,rack:rack2"); ``` ##### Use labels to specify materialized view data distribution on BE nodes[​](#use-labels-to-specify-materialized-view-data-distribution-on-be-nodes "Direct link to Use labels to specify materialized view data distribution on BE nodes") If you need to specify the locations to which an asynchronous materialized view's data is distributed, for example, distributing data across two racks, rack1 and rack2, you can add labels to the materialized view. After labels are added, all the replicas of the same tablet in the materialized view are distributed across labels in a Round-Robin approach. Moreover, if multiple replicas of the same tablet exist within the same label, these replicas will be distributed as evenly as possible across different BEs in that label. note * If the total number of BE nodes associated with the labels is fewer than the number of replicas, the system will preferentially ensure there are enough replicas. In this case, replicas may not be distributed as the label specified. * The labels to be associated with the materialized view must already exist. Otherwise, an error `Getting analyzing error. Detail message: Cannot find any backend with location: rack:xxx` will occur. ###### At materialized view creation[​](#at-materialized-view-creation "Direct link to At materialized view creation") If you want to distribute the materialized view's data across rack 1 and rack 2 while creating it, you can execute the following statement: ```sql CREATE MATERIALIZED VIEW mv_example_mv DISTRIBUTED BY RANDOM PROPERTIES ( "labels.location" = "rack:rack1,rack:rack2") as select order_id, dt from example_table; ``` For newly created materialized view, the default value of the property `labels.location` is `*`, indicating that replicas are evenly distributed across all labels. If the data distribution of a newly created materialized view does not need to be aware of the geographical locations of servers in the cluster, you can manually set the property `"labels.location" = ""`. ###### After materialized view creation[​](#after-materialized-view-creation "Direct link to After materialized view creation") If you need to modify the data distribution location of the materialized view after it is created, for example, modify the location to rack 1, rack 2, and rack 3, you can execute the following statement: ```sql ALTER MATERIALIZED VIEW mv_example_mv SET ("labels.location" = "rack:rack1,rack:rack2,rack:rack3"); ``` note If you have upgraded StarRocks to version 3.2.8 or later, for existing materialized views created before the upgrade, data is not distributed based on labels by default. If you need to distribute data of an existing based on labels, you can execute the following statement to add labels to the materialized view: ```sql ALTER TABLE example_mv1 SET ("labels.location" = "rack:rack1,rack:rack2"); ``` --- ### Blacklist Management In some cases, administrators need to disable certain patterns of SQL to avoid SQL from triggering cluster crashes or unexpected high concurrent queries. The blacklist is only for SELECT statements, INSERT statements (from v3.1 onwards), and CTAS statements (from v3.4 onwards). StarRocks allows users to add, view, and delete SQL blacklists. #### Syntax[​](#syntax "Direct link to Syntax") Enable SQL blacklisting via `enable_sql_blacklist`. The default is False (off). ```sql admin set frontend config ("enable_sql_blacklist" = "true"); ``` The admin user who has ADMIN\_PRIV privileges can manage blacklists by executing the following commands: ```sql ADD SQLBLACKLIST ""; DELETE SQLBLACKLIST ; SHOW SQLBLACKLIST; ``` * When `enable_sql_blacklist` is true, every SQL query needs to be filtered by sqlblacklist. If it matches, the user will be informed that the SQL is in the blacklist. Otherwise, the SQL will be executed normally. The message may be as follows when the SQL is blacklisted: `ERROR 1064 (HY000): Access denied; sql 'select count (*) from test_all_type_select_2556' is in blacklist` #### Add blacklist[​](#add-blacklist "Direct link to Add blacklist") ```sql ADD SQLBLACKLIST ""; ``` **sql** is a regular expression for a certain type of SQL. tip Currently, StarRocks supports adding SELECT statements to the SQL Blacklist. Since SQL itself contains the common characters `(`, `)`, `*`, `.` that may be mixed up with the semantics of regular expressions, we need to distinguish those by using escape characters. Given that `(` and `)` are used too often in SQL, there is no need to use escape characters. Other special characters need to use the escape character `\` as a prefix. For example: * Prohibit `count(\*)`: ```sql ADD SQLBLACKLIST "select count(\\*) from .+"; ``` * Prohibit `count(distinct)`: ```sql ADD SQLBLACKLIST "select count(distinct .+) from .+"; ``` * Prohibit order by limit `x`, `y`, `1 <= x <=7`, `5 <=y <=7`: ```sql ADD SQLBLACKLIST "select id_int from test_all_type_select1 order by id_int limit [1-7], [5-7]"; ``` * Prohibit complex SQL: ```sql ADD SQLBLACKLIST "select id_int \\* 4, id_tinyint, id_varchar from test_all_type_nullable except select id_int, id_tinyint, id_varchar from test_basic except select (id_int \\* 9 \\- 8) \\/ 2, id_tinyint, id_varchar from test_all_type_nullable2 except select id_int, id_tinyint, id_varchar from test_basic_nullable"; ``` * Prohibit all INSERT INTO statements: ```sql ADD SQLBLACKLIST "(?i)^insert\\s+into\\s+.*"; ``` * Prohibit all INSERT INTO ... VALUES statements: ```sql ADD SQLBLACKLIST "(?i)^insert\\s+into\\s+.*values\\s*\\("; ``` * Prohibit all INSERT INTO ... VALUES statements except those against the system-defined view `_statistics_.column_statistics`: ```sql ADD SQLBLACKLIST "(?i)^insert\\s+into\\s+(?!column_statistics\\b).*values\\s*\\("; ``` #### View blacklist[​](#view-blacklist "Direct link to View blacklist") ```sql SHOW SQLBLACKLIST; ``` Result format: `Index | Forbidden SQL` For example: ```sql mysql> show sqlblacklist; +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Index | Forbidden SQL | +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 1 | select count\(\*\) from .+ | | 2 | select id_int \* 4, id_tinyint, id_varchar from test_all_type_nullable except select id_int, id_tinyint, id_varchar from test_basic except select \(id_int \* 9 \- 8\) \/ 2, id_tinyint, id_varchar from test_all_type_nullable2 except select id_int, id_tinyint, id_varchar from test_basic_nullable | | 3 | select id_int from test_all_type_select1 order by id_int limit [1-7], [5-7] | | 4 | select count\(distinct .+\) from .+ | +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` The SQL shown in `Forbidden SQL` is escaped for all SQL semantic characters. #### Delete blacklist[​](#delete-blacklist "Direct link to Delete blacklist") ```sql DELETE SQLBLACKLIST ; ``` `` is a list of SQL IDs separated by comma (,). For example, delete the No.3 and No.4 SQLs in the above blacklist: ```sql delete sqlblacklist 3, 4; ``` Then, the remaining sqlblacklist is as follows: ```sql mysql> show sqlblacklist; +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Index | Forbidden SQL | +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 1 | select count\(\*\) from .+ | | 2 | select id_int \* 4, id_tinyint, id_varchar from test_all_type_nullable except select id_int, id_tinyint, id_varchar from test_basic except select \(id_int \* 9 \- 8\) \/ 2, id_tinyint, id_varchar from test_all_type_nullable2 except select id_int, id_tinyint, id_varchar from test_basic_nullable | +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` --- ### File manager With file manager, you can create, view, and delete files, such as the files that are used to access external data sources: public key files, private key files, and certificate files. You can reference or access the created files by using commands. #### Basic concepts[​](#basic-concepts "Direct link to Basic concepts") **File**: refers to the file that is created and saved in StarRocks. After a file is created and stored in StarRocks, StarRocks assigns a unique ID to the file. You can find a file based on the database name, catalog, and file name. In a database, only an admin user can create and delete files, and all users who have permissions to access a database can use the files that belong to the database. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") * Configure the following parameters for each FE. * `small_file_dir`: the path in which the uploaded files are stored. The default path is `small_files/`, which is in the runtime directory of the FE. You need to specify this parameter in the **fe.conf** file and then restart the FE to allow the change to take effect. * `max_small_file_size_bytes`: the maximum size of a single file. The default value of this parameter is 1 MB. If the size of a file exceeds the value of this parameter, the file cannot be created. You can specify this parameter by using the [ADMIN SET CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) statement. * `max_small_file_number`: the maximum number of files that can be created within a cluster. The default value of this parameter is 100. If the number of files that you have created reaches the value of this parameter, you cannot continue creating files. You can specify this parameter by using the ADMIN SET CONFIG statement. > Note: Increasing the values of the two parameters causes an increase in the memory usage of the FE. Therefore, we recommend that you do not increase the values of the two parameters unless necessary. * Configure the following parameters for each BE. `small_file_dir`: the path in which the downloaded files are stored. The default path is `lib/small_files/`, which is in the runtime directory of the BE. You can specify this parameter in the **be.conf** file. #### Create a file[​](#create-a-file "Direct link to Create a file") You can execute the CREATE FILE statement to create a file. For more information, see [CREATE FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/file/CREATE_FILE.md). After a file is created, the file is uploaded and persisted in StarRocks. #### View a file[​](#view-a-file "Direct link to View a file") You can execute the SHOW FILE statement to view the information about a file stored in a database. For more information, see [SHOW FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/file/SHOW_FILE.md). #### Delete a file[​](#delete-a-file "Direct link to Delete a file") You can execute the DROP FILE statement to delete a file. For more information, see [DROP FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/file/DROP_FILE.md). #### How an FE and a BE use a file[​](#how-an-fe-and-a-be-use-a-file "Direct link to How an FE and a BE use a file") * **FE**: The SmallFileMgr class stores the data related to the file in the specified directory of the FE. Then the SmallFileMgr class returns a local file path for the FE to use the file. * **BE**: The BE calls the **/api/get\_small\_file API** (HTTP) to download the file to its specified directory and record the information of the file. When the BE requests to use the file, the BE checks whether the file has been downloaded and then verifies the file. If the file pass the verification, the path of the file is returned. If the file fails the verification, the file is deleted and re-downloaded from the FE. When a BE restarts, it preloads the downloaded file into its memory. --- ### Load Balancing When deploying multiple FE nodes, users can deploy a load balancing layer on top of the FEs to achieve high availability. The following are some high availability options: #### Code approach[​](#code-approach "Direct link to Code approach") One way is to implement code at the application layer to perform retry and load balancing. For example, if a connection is broken, it will automatically retry on other connections. This approach requires users to configure multiple FE node addresses. #### JDBC Connector[​](#jdbc-connector "Direct link to JDBC Connector") JDBC connector supports automatic retry: ```sql jdbc:mysql:loadbalance://[host1][:port],[host2][:port][,[host3][:port]]...[/[database]][?propertyName1=propertyValue1[&propertyName2=propertyValue2]...] ``` #### ProxySQL[​](#proxysql "Direct link to ProxySQL") ProxySQL is a MySQL proxy layer that supports read/write separation, query routing, SQL caching, dynamic load configuration, failover, and SQL filtering. StarRocks FE is responsible for receiving connection and query requests, and it’s horizontally scalable and highly available. However FE requires users to set up a proxy layer on top of it to achieve automatic load balancing. See the following steps for setup: ##### 1. Install relevant dependencies[​](#1-install-relevant-dependencies "Direct link to 1. Install relevant dependencies") ```shell yum install -y gnutls perl-DBD-MySQL perl-DBI perl-devel ``` ##### 2. Download the installation package[​](#2-download-the-installation-package "Direct link to 2. Download the installation package") ```shell wget https://github.com/sysown/proxysql/releases/download/v2.0.14/proxysql-2.0.14-1-centos7.x86_64.rpm ``` ##### 3. Decompress to the current directory[​](#3-decompress-to-the-current-directory "Direct link to 3. Decompress to the current directory") ```shell rpm2cpio proxysql-2.0.14-1-centos7.x86_64.rpm | cpio -ivdm ``` ##### 4. Modify the configuration file[​](#4-modify-the-configuration-file "Direct link to 4. Modify the configuration file") ```shell vim ./etc/proxysql.cnf ``` Direct to a directory that the user has privilege to access (absolute path): ```vim datadir="/var/lib/proxysql" errorlog="/var/lib/proxysql/proxysql.log" ``` ##### 5. Start[​](#5-start "Direct link to 5. Start") ```shell ./usr/bin/proxysql -c ./etc/proxysql.cnf --no-monitor ``` ##### 6. Log in[​](#6-log-in "Direct link to 6. Log in") ```shell mysql -u admin -padmin -h 127.0.0.1 -P6032 ``` ##### 7. Configure the global log[​](#7-configure-the-global-log "Direct link to 7. Configure the global log") ```shell SET mysql-eventslog_filename='proxysql_queries.log'; SET mysql-eventslog_default_log=1; SET mysql-eventslog_format=2; LOAD MYSQL VARIABLES TO RUNTIME; SAVE MYSQL VARIABLES TO DISK; ``` ##### 8. Insert into the leader node[​](#8-insert-into-the-leader-node "Direct link to 8. Insert into the leader node") ```sql insert into mysql_servers(hostgroup_id, hostname, port) values(1, '172.xx.xx.139', 9030); ``` ##### 9. Insert the observer nodes[​](#9-insert-the-observer-nodes "Direct link to 9. Insert the observer nodes") ```sql insert into mysql_servers(hostgroup_id, hostname, port) values(2, '172.xx.xx.139', 9030); insert into mysql_servers(hostgroup_id, hostname, port) values(2, '172.xx.xx.140', 9030); ``` ##### 10. Load the configuration[​](#10-load-the-configuration "Direct link to 10. Load the configuration") ```sql load mysql servers to runtime; save mysql servers to disk; ``` ##### 11. Configure the username and password[​](#11-configure-the-username-and-password "Direct link to 11. Configure the username and password") ```sql insert into mysql_users(username, password, active, default_hostgroup, backend, frontend) values('root', '*94BDCEBE19083CE2A1F959FD02F964C7AF4CFC29', 1, 1, 1, 1); ``` ##### 12. Load the configuration[​](#12-load-the-configuration "Direct link to 12. Load the configuration") ```sql load mysql users to runtime; save mysql users to disk; ``` ##### 13. Write to the proxy rules[​](#13-write-to-the-proxy-rules "Direct link to 13. Write to the proxy rules") ```sql insert into mysql_query_rules(rule_id, active, match_digest, destination_hostgroup, mirror_hostgroup, apply) values(1, 1, '.', 1, 2, 1); ``` ##### 14. Load the configuration[​](#14-load-the-configuration "Direct link to 14. Load the configuration") ```sql load mysql query rules to runtime; save mysql query rules to disk; ``` --- ### Memory Management This section briefly introduces memory classification and StarRocks’ methods of managing memory. #### Memory Classification[​](#memory-classification "Direct link to Memory Classification") Explanation: | Metric | Name | Description | | ------------- | ------------------------------- | ----------------------------------------------------------------------------------------------- | | `process` | Total memory used of BE | | | `query_pool` | Memory used by data querying | Consists of two parts: memory used by the execution layer and memory used by the storage layer. | | `load` | Memory used by data loading | Generally MemTable | | `table_meta` | Metadata memory | S Schema, Tablet metadata, RowSet metadata, Column metadata, ColumnReader, IndexReader | | `compaction` | Multi-version memory compaction | compaction that happens after data import is complete | | `snapshot` | Snapshot memory | Generally used for clone, little memory usage | | `column_pool` | Column pool memory | Request to release column cache for accelerated column | | `page_cache` | BE's own PageCache | The default is off, the user can turn it on by modifying the BE file | #### Memory-related configuration[​](#memory-related-configuration "Direct link to Memory-related configuration") * **BE Configuration** | Name | Default | Description | | ------------------------------------------------ | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | vector\_chunk\_size | 4096 | Number of chunk rows | | mem\_limit | 90% | BE process memory upper limit. You can set it as a percentage ("80%") or a physical limit ("100G"). The default hard limit is 90% of the server's memory size, and the soft limit is 80%. You need to configure this parameter if you want to deploy StarRocks with other memory-intensive services on a same server. | | disable\_storage\_page\_cache | false | The boolean value to control whether to disable PageCache. When PageCache is enabled, StarRocks caches the recently scanned data. PageCache can significantly improve the query performance when similar queries are repeated frequently. `true` indicates to disable PageCache. Use this item together with `storage_page_cache_limit`, you can accelerate query performance in scenarios with sufficient memory resources and much data scan. The default value of this item has been changed from `true` to `false` since StarRocks v2.4. | | write\_buffer\_size | 104857600 | The capacity limit of a single MemTable, exceeding which a disk swipe will be performed. | | load\_process\_max\_memory\_limit\_bytes | 107374182400 | The upper limit of memory resources that can be taken up by all load processes on a BE node. Its value is the smaller one between `mem_limit * load_process_max_memory_limit_percent / 100` and `load_process_max_memory_limit_bytes`. If this threshold is exceeded, a flush and backpressure will be triggered. | | load\_process\_max\_memory\_limit\_percent | 30 | The maximum percentage of memory resources that can be taken up by all load processes on a BE node. Its value is the smaller one between `mem_limit * load_process_max_memory_limit_percent / 100` and `load_process_max_memory_limit_bytes`. If this threshold is exceeded, a flush and backpressure will be triggered. | | default\_load\_mem\_limit | 2147483648 | If the memory limit on the receiving side is reached for a single import instance, a disk swipe will be triggered. This needs to be modified with the Session variable `load_mem_limit` to take effect. This parameter is mutable when the Event-based Compaction Framework is enabled. | | max\_compaction\_concurrency | -1 | The maximum concurrency of compactions (both Base Compaction and Cumulative Compaction). The value -1 indicates that no limit is imposed on the concurrency. | | cumulative\_compaction\_check\_interval\_seconds | 1 | Interval of compaction check | * **Session variables** | Name | Default | Description | | ----------------- | ------- | --------------------------------------------------------------------------------------- | | query\_mem\_limit | 0 | Memory limit of a query on each BE node | | load\_mem\_limit | 0 | Memory limit of a single import task. If the value is 0, `exec_mem_limit` will be taken | #### View memory usage[​](#view-memory-usage "Direct link to View memory usage") * **`mem_tracker`** ```bash //View the overall memory statistics // View fine-grained memory statistics ``` * **`tcmalloc`** ```bash ``` ```plain ------------------------------------------------ MALLOC: 777276768 ( 741.3 MiB) Bytes in use by application MALLOC: + 8851890176 ( 8441.8 MiB) Bytes in page heap freelist MALLOC: + 143722232 ( 137.1 MiB) Bytes in central cache freelist MALLOC: + 21869824 ( 20.9 MiB) Bytes in transfer cache freelist MALLOC: + 832509608 ( 793.9 MiB) Bytes in thread cache freelists MALLOC: + 58195968 ( 55.5 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 10685464576 (10190.5 MiB) Actual memory used (physical + swap) MALLOC: + 25231564800 (24062.7 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 35917029376 (34253.1 MiB) Virtual address space used MALLOC: MALLOC: 112388 Spans in use MALLOC: 335 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. ``` The memory queried by this method is accurate. However, some memory in StarRocks is reserved but not in use. TcMalloc counts the memory that is reserved, not the memory used. Here `Bytes in use by application` refers to the memory currently in use. * **metrics** ```bash curl -XGET http://be_ip:be_http_port/metrics | grep 'mem' curl -XGET http://be_ip:be_http_port/metrics | grep 'column_pool' ``` The value of metrics is updated every 10 seconds. It is possible to monitor some of the memory statistics with older versions. --- ### Query Management #### Number of user connections[​](#number-of-user-connections "Direct link to Number of user connections") `Property` is set for user granularity. To set the maximum number of connections between Client and FE, use the following command. ```sql ALTER USER '' SET PROPERTIES ("key"="value", ...) ``` User properties include the resources assigned to the user. The properties set here are for the user, not `user_identity`. That is, if two users `jack'@'%` and `jack'@'192.%` are created by the `CREATE USER` statement, then the `ALTER USER SET PROPERTIES` statement can work on the user `jack`, not `jack'@'%` or `jack'@'192.%`. Example 1: ```sql -- For the user `jack`, change the maximum number of connections to 1000 ALTER USER 'jack' SET PROPERTIES ("max_user_connections" = "1000"); -- Check the connection limit for the root user SHOW PROPERTY FOR 'root'; ``` #### Query-related session variables[​](#query-related-session-variables "Direct link to Query-related session variables") The session variables can be set by 'key' = 'value', which can limit the concurrency, memory and other query parameters in the current session. For example: * parallel\_fragment\_exec\_instance\_num The parallelism of the query with a default value of 1. It indicates the number of fragment instances on each BE. You can set this to half the number of CPU cores of the BE to improve query performance. * query\_mem\_limit Memory limit of a query on each BE node, can be adjusted when a query reports insufficient memory. * load\_mem\_limit Memory limit for import, can be adjusted when an import job reports insufficient memory. Example 2: ```sql set parallel_fragment_exec_instance_num = 8; set query_mem_limit = 137438953472; ``` #### capacity quota of database storage[​](#capacity-quota-of-database-storage "Direct link to capacity quota of database storage") The capacity quota of database storage is unlimited by default. And you can change quota value by using `alter database`. ```sql ALTER DATABASE db_name SET DATA QUOTA quota; ``` The quota units are: B/K/KB/M/MB/G/GB/T/TB/P/PB Example 3: ```sql ALTER DATABASE example_db SET DATA QUOTA 10T; ``` #### Kill queries[​](#kill-queries "Direct link to Kill queries") To terminate a query on a particular connection with the following command: ```sql kill connection_id; ``` The `connection_id` can be seen by `show processlist;` or `select connection_id();`. ```plain show processlist; +------+------------+---------------------+-----------------+---------------+---------+------+-------+------+ | Id | User | Host | Cluster | Db | Command | Time | State | Info | +------+------------+---------------------+-----------------+---------------+---------+------+-------+------+ | 1 | starrocksmgr | 172.26.34.147:56208 | default_cluster | starrocks_monitor | Sleep | 8 | | | | 129 | root | 172.26.92.139:54818 | default_cluster | | Query | 0 | | | | 114 | test | 172.26.34.147:57974 | default_cluster | ssb_100g | Query | 3 | | | | 3 | starrocksmgr | 172.26.34.147:57268 | default_cluster | starrocks_monitor | Sleep | 8 | | | | 100 | root | 172.26.34.147:58472 | default_cluster | ssb_100 | Sleep | 637 | | | | 117 | starrocksmgr | 172.26.34.147:33790 | default_cluster | starrocks_monitor | Sleep | 8 | | | | 6 | starrocksmgr | 172.26.34.147:57632 | default_cluster | starrocks_monitor | Sleep | 8 | | | | 119 | starrocksmgr | 172.26.34.147:33804 | default_cluster | starrocks_monitor | Sleep | 8 | | | | 111 | root | 172.26.92.139:55472 | default_cluster | | Sleep | 2758 | | | +------+------------+---------------------+-----------------+---------------+---------+------+-------+------+ 9 rows in set (0.00 sec) mysql> select connection_id(); +-----------------+ | CONNECTION_ID() | +-----------------+ | 98 | +-----------------+ mysql> kill 114; Query OK, 0 rows affected (0.02 sec) ``` --- ### Query queues This topic describes how to manage query queues in StarRocks. From v2.5, StarRocks supports query queues. With query queues enabled, StarRocks automatically queues the incoming queries when the concurrency threshold or resource limit is reached, thereby avoiding the overload deteriorating. Pending queries wait in a queue until there is enough compute resources available to begin execution. The Query Queue feature has two versions: * [**Query Queue v1**](#query-queue-v1): Triggers queuing based on query concurrency, BE memory usage, and BE CPU usage. The original query queue configurations and behaviors in this topic belong to v1. From v3.1.4 onwards, v1 supports setting query queues on the resource group level. * [**Query Queue v2**](#query-queue-v2): Supported from v3.3 onwards. v2 estimates the BE resources consumed by each query, represents BE resources as logical slots, and queues and schedules queries based on the number of slots each query needs. #### Query Queue v1[​](#query-queue-v1 "Direct link to Query Queue v1") Query Queue v1 supports setting thresholds on CPU usage, memory usage, and query concurrency to trigger query queues. **Roadmap**: | Version | Global query queue | Resource group-level query queue | Collective concurrency management | Dynamic concurrency adjustment | | ------- | ------------------ | -------------------------------- | --------------------------------- | ------------------------------ | | v2.5 | ✅ | ❌ | ❌ | ❌ | | v3.1.4 | ✅ | ✅ | ✅ | ✅ | ##### Enable Query Queue v1[​](#enable-query-queue-v1 "Direct link to Enable Query Queue v1") Query queues are disabled by default. You can enable global or resource group-level query queues for INSERT loading, SELECT queries, and statistics queries by setting corresponding global session variables. ###### Enable global query queues[​](#enable-global-query-queues "Direct link to Enable global query queues") * Enable query queues for loading tasks: ```sql SET GLOBAL enable_query_queue_load = true; ``` * Enable query queues for SELECT queries: ```sql SET GLOBAL enable_query_queue_select = true; ``` * Enable query queues for statistics queries: ```sql SET GLOBAL enable_query_queue_statistic = true; ``` ###### Enable resource group-level query queues[​](#enable-resource-group-level-query-queues "Direct link to Enable resource group-level query queues") From v3.1.4 onwards, StarRocks supports setting query queues on the resource group level. To enable the resource group-level query queues, you also need to set `enable_group_level_query_queue` in addition to the global session variables mentioned above. ```sql SET GLOBAL enable_group_level_query_queue = true; ``` ##### Specify resource thresholds[​](#specify-resource-thresholds "Direct link to Specify resource thresholds") ###### Specify resource thresholds for global query queues[​](#specify-resource-thresholds-for-global-query-queues "Direct link to Specify resource thresholds for global query queues") You can set the thresholds that trigger query queues via the following global session variables: | **Variable** | **Default** | **Description** | | ---------------------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | query\_queue\_concurrency\_limit | 0 | The upper limit of concurrent queries on a BE. It takes effect only after being set greater than `0`. Setting it to `0` indicates no limit is imposed. | | query\_queue\_mem\_used\_pct\_limit | 0 | The upper limit of memory usage percentage on a BE. It takes effect only after being set greater than `0`. Setting it to `0` indicates no limit is imposed. Range: \[0, 1] | | query\_queue\_cpu\_used\_permille\_limit | 0 | The upper limit of CPU usage permille (CPU usage \* 1000) on a BE. It takes effect only after being set greater than `0`. Setting it to `0` indicates no limit is imposed. Range: \[0, 1000] | note * After Query Queue v2 is enabled, `query_queue_cpu_used_permille_limit` is no longer supported for queue triggering, and `query_queue_mem_used_pct_limit` still applies. See [Memory admission](#memory-admission). * By default, BE reports resource usage to FE at one-second intervals. You can change this interval by setting the BE configuration item `report_resource_usage_interval_ms`. ###### Specify resource thresholds for resource group-level query queues[​](#specify-resource-thresholds-for-resource-group-level-query-queues "Direct link to Specify resource thresholds for resource group-level query queues") From v3.1.4 onwards, you can set individual concurrency limits (`concurrency_limit`) and CPU core limits (`max_cpu_cores`) when creating a resource group. When a query is initiated, if any of the resource consumptions exceed the resource threshold at either the global or resource group level, the query will be placed in queue until all resource consumptions are within the threshold. | **Variable** | **Default** | **Description** | | ------------------ | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | concurrency\_limit | 0 | The concurrency limit for the resource group on a single BE node. It takes effect only when it is set to greater than `0`. | | max\_cpu\_cores | 0 | The CPU core limit for this resource group on a single BE node. It takes effect only when it is set to greater than `0`. Range: \[0, `avg_be_cpu_cores`], where `avg_be_cpu_cores` represents the average number of CPU cores across all BE nodes. | You can use SHOW USAGE RESOURCE GROUPS to view the resource usage information for each resource group on each BE node, as described in [View Resource Group Usage Information](https://docs.starrocks.io/docs/administration/management/resource_management/resource_group.md#view-resource-group-usage-information). ###### Manage query concurrency[​](#manage-query-concurrency "Direct link to Manage query concurrency") When the number of running queries (`num_running_queries`) exceeds the global or resource group's `concurrency_limit`, incoming queries are placed in the queue. The way to obtain `num_running_queries` differs between versions `<` v3.1.4 and `>=` v3.1.4. * In versions `<` v3.1.4, `num_running_queries` is reported by BEs at the interval specified in `report_resource_usage_interval_ms`. Therefore, there might be some delay in the identification of changes in `num_running_queries`. For example, if the `num_running_queries` reported by BEs at the moment does not exceed the global or resource group's `concurrency_limit`, but incoming queries arrive and exceed the `concurrency_limit` before the next report, these incoming queries will be executed without waiting in the queue. * In versions `>=` v3.1.4, all running queries are collectively managed by the Leader FE. Each Follower FE notifies the Leader FE when initiating or finishing a query, allowing the StarRocks to handle scenarios where there is a sudden increase in queries exceeding the `concurrency_limit`. ##### Configure Query Queue v1[​](#configure-query-queue-v1 "Direct link to Configure Query Queue v1") You can set the capacity of a query queue and the maximum timeout of queries in queues via the following global session variables: | **Variable** | **Default** | **Description** | | -------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | query\_queue\_max\_queued\_queries | 1024 | The upper limit of queries in a queue. When this threshold is reached, incoming queries are rejected. It takes effect only after being set greater than `0`. | | query\_queue\_pending\_timeout\_second | 300 | The maximum timeout of a pending query in a queue. When this threshold is reached, the corresponding query is rejected. Unit: second. | ##### Configure dynamic adjustment of query concurrency[​](#configure-dynamic-adjustment-of-query-concurrency "Direct link to Configure dynamic adjustment of query concurrency") Starting from version v3.1.4, for queries managed by the query queue and run by the Pipeline Engine, StarRocks can dynamically adjust the query concurrency `pipeline_dop` for incoming queries based on the current number of running queries `num_running_queries`, the number of fragments `num_fragments`, and the query concurrency `pipeline_dop`. This allows you to dynamically control query concurrency while minimizing scheduling overhead, ensuring optimal BE resource utilization. For more information about fragments and query concurrency `pipeline_dop`, see [Query Management - Adjusting Query Concurrency](https://docs.starrocks.io/docs/administration/management/resource_management/Query_management.md). For each query under a query queue, StarRocks maintains a concept of drivers, which represent the concurrent fragments of a query on a single BE. Its logical value `num_drivers`, which represents the total concurrency of all fragments of that query on a single BE, is equal to `num_fragments * pipeline_dop`. When a new query arrives, StarRocks adjusts the query concurrency `pipeline_dop` based on the following rules: * The more the number of running drivers `num_drivers` exceeds the low water limit of concurrent drivers `query_queue_driver_low_water`, the lower the query concurrency `pipeline_dop` is adjusted to. * StarRocks restrains the number of running drivers `num_drivers` below the high water limit of concurrent drivers for queries `query_queue_driver_high_water`. You can configure the dynamic adjustment of query concurrency `pipeline_dop` using the following global session variables: | **Variable** | **Default** | **Description** | | --------------------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | query\_queue\_driver\_high\_water | -1 | The high water limit of concurrent drivers for a query. It takes effect only when it is set to a non-negative value. When set to `0`, it is equivalent to `avg_be_cpu_cores * 16`, where `avg_be_cpu_cores` represents the average number of CPU cores across all BE nodes. When set to a value greater than `0`, that value is used directly. | | query\_queue\_driver\_low\_water | -1 | The lower limit of concurrent drivers for queries. It takes effect only when it is set to a non-negative value. When set to `0`, it is equivalent to `avg_be_cpu_cores * 8`. When set to a value greater than `0`, that value is used directly. | #### Query Queue v2[​](#query-queue-v2 "Direct link to Query Queue v2") From v3.3 onwards, StarRocks supports Query Queue v2. In Query Queue v2, query queues are no longer triggered based on fixed thresholds for query concurrency, BE memory usage, or BE CPU usage. Instead, it estimates the BE resources required by each query and queues and schedules queries based on logical slots. If there are not enough available slots, the query waits in the queue until enough slots are released. ##### Configure Query Queue v2[​](#configure-query-queue-v2 "Direct link to Configure Query Queue v2") Query Queue v2 is enabled and tuned through FE configuration items. Changes to `enable_query_queue_v2` require restarting FE nodes to take effect. | Configuration item | Default | Description | | -------------------------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enable_query_queue_v2` | `false` (v3.3 to v4.0)
`true` (from v4.1 onwards) | Whether to enable Query Queue v2. When this item is set to `true`, StarRocks uses the v2 slot-based query scheduling mechanism. | | `query_queue_v2_concurrency_level` | `4` | The logical concurrency level used by Query Queue v2 to calculate the total number of cluster slots. A larger value allows the system to admit more queries. This is a relative tuning parameter. | | `query_queue_slots_estimator_strategy` | `PBE` | The slot estimation strategy used for queue-based queries. Valid values: `PBE` (parallelism-based, the default), `MBE` (memory-cost-based), and `CBE` (CPU-cost-based). PBE estimates a query's slots from scan parallelism, capped by the worker count: for OLAP tables it uses the number of scan ranges left after pruning, so only very small queries fall below the worker count; a connector/external scan is treated as a full-parallelism scan (the worker count) rather than a single-slot query. MBE estimates slots from the query's memory cost divided by `query_queue_v2_mem_bytes_per_slot`. CBE estimates slots from the plan CPU cost divided by `query_queue_v2_cpu_costs_per_slot`. MBE and CBE per-query slots are additionally capped by `number_of_workers * max(1, pipeline_dop / 2)`. The legacy values `MAX` and `MIN` are still accepted for forward compatibility and are treated as the default estimator; any other value is rejected by configuration validation. | | `query_queue_v2_schedule_strategy` | `SWRR` | The scheduling policy used by Query Queue V2 to order pending queries. Supported values (case-insensitive) are `SWRR` (Smooth Weighted Round Robin) — the default, suitable for mixed/hybrid workloads that need fair weighted sharing — and `SJF` (Short Job First + Aging) — prioritizes short jobs while using aging to avoid starvation. The value is parsed with case-insensitive enum lookup; an unrecognized value is logged as an error and the default policy is used. This configuration only affects behavior when Query Queue V2 is enabled and interacts with V2 sizing settings such as `query_queue_v2_concurrency_level`. | | `query_queue_v2_mem_bytes_per_slot` | `0` | Per-slot memory target used by the memory-cost-based estimator (MBE). When `query_queue_slots_estimator_strategy` is `MBE`, the total slots are derived from the warehouse memory budget, and a query's slots are estimated from its total memory cost divided by this value, capped by `number_of_workers * max(1, pipeline_dop / 2)`. If it is non-positive, Query Queue V2 uses the average worker memory per core. | | `query_queue_v2_cpu_costs_per_slot` | `1000000000` | Per-slot CPU cost threshold used by the CPU-cost-based estimator (CBE) to estimate how many slots a query needs from its plan CPU cost. The scheduler computes slots as `ceil(plan_cpu_costs / query_queue_v2_cpu_costs_per_slot)` and clamps the result to the range `[1, min(totalSlots, number_of_workers * max(1, pipeline_dop / 2))]`. A non-positive value is normalized to `1`. Increasing this value reduces slots allocated per query (favoring fewer, larger-slot queries); decreasing it increases slots per query. | | `query_queue_mem_used_pct_limit` | `0` | A global session variable. The memory usage limit of a node, as a fraction of the memory limit of the node. While the process memory usage of any alive node reaches this value, new queries wait in the queue. Range: \[0, 1]. `0` means no memory gating. See [Memory admission](#memory-admission). | | query\_queue\_concurrency\_limit | 0 | The upper limit of concurrent queries on a BE. It takes effect only after being set greater than `0`. Setting it to `0` indicates no limit is imposed. | note `query_queue_cpu_used_permille_limit` applies only to Query Queue v1. For how Query Queue v2 applies `query_queue_mem_used_pct_limit`, see [Memory admission](#memory-admission). ##### Resource slots[​](#resource-slots "Direct link to Resource slots") Query Queue v2 represents BE resources as logical slots: * **Total cluster slots**: StarRocks sets a logical total number of slots for the entire cluster. This total is positively correlated with the number of BEs and BE CPU cores, and is also affected by `query_queue_v2_concurrency_level`. * **Slots required by a query**: StarRocks estimates the number of slots required by each query. The estimation is based on factors such as statistics, query complexity, the number of fragments, estimated input and output data volumes of complex operators, and DOP. ##### Queuing logic[​](#queuing-logic "Direct link to Queuing logic") When the number of slots required by a query exceeds the current number of remaining slots, the query waits in the queue. Query Queue v2 preferentially satisfies queries that require fewer slots, allowing small queries to obtain resources first and avoiding head-of-line blocking where a large query at the head of the queue blocks later small queries. The entire queuing logic is completed on FE, including setting the total number of cluster slots, estimating the number of slots required by a query, and deciding which query's slot requirement to satisfy first. Slot scheduling does not depend on the actual resource usage of BEs. The only input taken from actual usage is the optional memory usage limit `query_queue_mem_used_pct_limit`, which makes new queries wait while a node uses too much memory. See [Memory admission](#memory-admission). ##### Choose an estimation strategy[​](#choose-an-estimation-strategy "Direct link to Choose an estimation strategy") ###### PBE[​](#pbe "Direct link to PBE") Parallel-based estimation (PBE) strategy is best for: * Normal reporting queries * Mixed point lookups and large queries * Users who do not want to understand cost model details * DBAs who want stable, simple, and explainable queueing behavior first Expected behaviors with PBE include: * Point lookups or queries that scan little data after pruning use fewer slots * Queries that scan larger ranges use more slots * Small queries are more likely to get execution resources during peak hours The following example sets PBE as the strategy: ```sql ADMIN SET FRONTEND CONFIG ("query_queue_slots_estimator_strategy" = "PBE"); ``` ###### MBE[​](#mbe "Direct link to MBE") Memory cost-based estimation (MBE) strategy is suitable for dealing with memory pressure, such as large joins, large aggregations, or high-cardinality aggregations. The following example sets MBE as the strategy, and allocates 2 GB of memory to each slot: ```sql ADMIN SET FRONTEND CONFIG ("query_queue_slots_estimator_strategy" = "MBE"); ADMIN SET FRONTEND CONFIG ("query_queue_v2_mem_bytes_per_slot" = "2147483648"); ``` MBE divides the query’s total memory cost by this value to get query slots, and divides the warehouse memory budget by this value to get total slots. Tune the MBE strategy in the following directions: **Symptom: Memory still gets saturated easily** * **Adjustment**: Decrease `query_queue_v2_concurrency_level` * **Effect**: Directly lowers the MBE total memory budget **Symptom: Queue is too long but BE memory still has room** * **Adjustment**: Increase `query_queue_v2_concurrency_level` * **Effect**: Directly raises the MBE total memory budget **Symptom: `max_slots` is very small and integer rounding is visible** * **Adjustment**: Decrease `query_queue_v2_mem_bytes_per_slot` * **Effect**: Uses a finer memory slot granularity and reduces coarse rounding error ###### CBE[​](#cbe "Direct link to CBE") CPU cost-based estimation (CBE) strategy is suitable for dealing with memory pressure, such as compute-heavy SQL, complex expressions, or heavy CPU work after scanning. The following example sets CBE as the strategy, and set the CPU cost threshold to `1000000000` for each slot: ```sql ADMIN SET FRONTEND CONFIG ("query_queue_slots_estimator_strategy" = "CBE"); ADMIN SET FRONTEND CONFIG ("query_queue_v2_cpu_costs_per_slot" = "1000000000"); ``` **Symptom: CPU is often saturated** * **Adjustment**: Decrease `query_queue_v2_cpu_costs_per_slot` * **Effect**: The same CPU cost for more slots, making concurrency more conservative **Symptom: Queries queue noticeably but CPU still has room** * **Adjustment**: Increase `query_queue_v2_cpu_costs_per_slot` * **Effect**: The same CPU cost for fewer slots, making concurrency looser ##### Tune concurrency capacity[​](#tune-concurrency-capacity "Direct link to Tune concurrency capacity") If you only want to increase or decrease overall concurrency, do not switch between PBE, MBE, and CBE first. Tune total slot capacity first: ```sql ADMIN SET FRONTEND CONFIG ("query_queue_v2_concurrency_level" = ""); ``` Recommended process: 1. Start with default value `4`. 2. Observe `remain_slots`, `max_slots`, `query_pending_length`, CPU, memory, and query latency. 3. If there is resource headroom but queries queue noticeably, gradually increase `query_queue_v2_concurrency_level`. 4. If resources are often saturated or queries interfere with each other heavily, gradually decrease `query_queue_v2_concurrency_level`. 5. Make small changes each time, such as 10% to 25%, and observe one business peak period before making another change. **Tuning Priority**: Use `query_queue_v2_concurrency_level` to tune overall capacity first. Consider switching to MBE or PBE only after that. Do not change multiple parameters at the same time at the beginning, because it becomes hard to tell which parameter caused the effect. ###### Fallback Concurrency Cap[​](#fallback-concurrency-cap "Direct link to Fallback Concurrency Cap") `query_queue_concurrency_limit` is a fallback concurrency cap and applies to PBE, MBE, and CBE. Query Queue V2 first estimates the slots required by a query with the current estimator and checks whether total slots are available. After that, it checks whether the current number of running queries has reached `query_queue_concurrency_limit`. The default value `0` means unlimited. Set it only when you need an absolute cap on the number of concurrently running queries: ```sql ALTER WAREHOUSE default_warehouse SET ("query_queue_concurrency_limit" = "8"); ``` Use `query_queue_v2_concurrency_level` to tune resource capacity first. Use `query_queue_concurrency_limit` only when you need to explicitly limit the number of queries run at the same time. ##### Memory admission[​](#memory-admission "Direct link to Memory admission") Slots do not measure the memory that queries use. Memory admission controls memory with two settings: * `query_queue_mem_used_pct_limit`: the memory usage limit of a node. While the process memory usage of any alive node reaches this fraction of the node's memory limit, new queries wait in the queue. It is the same global session variable that Query Queue v1 uses. * `query_mem_limit`: the existing session variable that sets the memory limit of a query on each BE node. `query_queue_mem_used_pct_limit` defaults to `0`, which means that no query waits because of memory. Set it with `SET GLOBAL`. ###### Memory usage limit[​](#memory-usage-limit "Direct link to Memory usage limit") `query_queue_mem_used_pct_limit` is a fraction in \[0, 1], such as `0.8` for 80%. It is compared with the memory usage of each node: the process memory that the node uses, divided by the memory limit of the node. The process memory includes not only queries but also loads, compaction, metadata, and the in-memory data cache. A query fails when either its query pool or the BE process runs out of memory, so comparing the process memory keeps a buffer against both limits, and new queries wait before a node runs out of memory. The memory usage limit works as follows: * Each BE and CN node reports its memory usage every second, as set by the BE configuration item `report_resource_usage_interval_ms`. Only the leader FE receives these reports, and the leader FE makes the admission decisions. * While the memory usage of any alive node is at or above the limit, new queries wait in the queue. Running queries are not affected. Once every node is below the limit, the queued queries are admitted again. The decision is binary: there is no gradual throttling between the two states. * The memory usage limit is checked together with the slot pool and `query_queue_concurrency_limit`. A query is admitted only when all three allow it. * When no query that was admitted by the queue is running, one query is still admitted even if a node is over the limit. The memory may be held by loads, caches, or queries that bypass the queue, and keeping queued queries waiting cannot release it. * A node whose latest report is older than the global variable `query_queue_fresh_resource_usage_interval_ms` (`5000` milliseconds by default) does not take part in the decision, so that a node that stops reporting cannot keep new queries waiting. A node that is not alive does not take part either. * The memory usage limit does not read or depend on resource groups. Memory admission is part of the admission decision of Query Queue v2, so it applies only to queries that go through the query queue. The nodes are all BE and CN nodes of the cluster in a shared-nothing cluster, and all CN nodes of the cluster in a shared-data cluster. ###### Choose a limit[​](#choose-a-limit "Direct link to Choose a limit") A large part of the process memory is usually the in-memory data cache, which holds the decompressed pages that queries read from segment files. Its size is limited by the BE configuration item `datacache_mem_size` (`20%` of the process memory limit by default), and it grows as queries read data. BE gives this memory back when the process memory is high: above the BE configuration item `memory_high_level` (`75` percent of the process memory limit by default), BE evicts the cache gradually, and above `memory_urgent_level` (`85` by default), BE shrinks the cache immediately. We recommend a limit between `0.75` and `0.85`: * Within this range, when queries need more memory, the cache gives memory back as the process memory usage rises above 75%, and new queries wait only until the cache has shrunk. * Below `0.75`, BE does not release the cache, so a node with a large cache can keep new queries waiting. The cluster then admits one query at a time, when nothing is running. * A lower limit leaves more room for the queries that are already running to grow. On a node that holds little besides queries, the room left before the query pool limit (90% of the process memory limit by default, as set by the BE configuration item `query_max_memory_limit_percent`) is 0.9 minus the limit. ###### Per-query memory limit on each node[​](#per-query-memory-limit-on-each-node "Direct link to Per-query memory limit on each node") The session variable `query_mem_limit` sets the memory limit of a query on each BE node, in bytes. The default value `0` means no limit. To set it for the current session, run `SET query_mem_limit = ;`. To set it for all future sessions, run `SET GLOBAL query_mem_limit = ;`. For more information, see [query\_mem\_limit](https://docs.starrocks.io/docs/sql-reference/System_variable.md#query_mem_limit). On each node, BE applies the smallest of `query_mem_limit`, the resource group property `big_query_mem_limit`, and the limits of the parent memory trackers. When a query exceeds `query_mem_limit`, the error message contains `Mem usage has exceed the limit of single query, You can change the limit by set session variable query_mem_limit.` ###### Configure memory admission[​](#configure-memory-admission "Direct link to Configure memory admission") The following example sets the memory usage limit: ```sql SET GLOBAL query_queue_mem_used_pct_limit = 0.8; ``` The following example limits each query to 8 GB of memory on each BE node in all future sessions: ```sql SET GLOBAL query_mem_limit = 8589934592; ``` Whenever you set `query_queue_mem_used_pct_limit`, also set `query_mem_limit`. See [Limitations](#limitations) for the reason. ###### Monitor memory admission[​](#monitor-memory-admission "Direct link to Monitor memory admission") To see the memory usage that is compared with the limit, run [SHOW BACKENDS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md) in a shared-nothing cluster or [SHOW COMPUTE NODES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md) in a shared-data cluster. The `MemUsedPct` column is the memory usage of each node, as a percentage of its memory limit. Only the leader FE receives the memory usage that the nodes report, so run the statement on the leader FE. ###### Limitations[​](#limitations "Direct link to Limitations") * Admission reserves no memory. A newly admitted query adds to the usage only after it allocates memory and the next report arrives, so the queries admitted in one batch can together overshoot the limit. The slot pool and `query_queue_concurrency_limit` bound how many queries a batch admits, but only `query_mem_limit` bounds how much memory each of them can use on a node. This is why the FE log records a warning, once for each limit value, when the memory usage limit is set but the global default of `query_mem_limit` is `0`. * One node at or above the limit makes all new queries wait, even if the other nodes have free memory, because queries usually run on all nodes. #### Monitor query queues[​](#monitor-query-queues "Direct link to Monitor query queues") You can view information related to query queues using the following methods. ##### SHOW PROC[​](#show-proc "Direct link to SHOW PROC") You can check the number of running queries, and memory and CPU usages in BE nodes using [SHOW PROC](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md): ```plain mysql> SHOW PROC '/backends'\G *************************** 1. row *************************** ... NumRunningQueries: 0 MemUsedPct: 0.79 % CpuUsedPct: 0.0 % ``` ##### SHOW PROCESSLIST[​](#show-processlist "Direct link to SHOW PROCESSLIST") You can check if a query is in a queue (when `IsPending` is `true`) using [SHOW PROCESSLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROCESSLIST.md): ```plain mysql> SHOW PROCESSLIST; +------+------+---------------------+-------+---------+---------------------+------+-------+-------------------+-----------+ | Id | User | Host | Db | Command | ConnectionStartTime | Time | State | Info | IsPending | +------+------+---------------------+-------+---------+---------------------+------+-------+-------------------+-----------+ | 2 | root | xxx.xx.xxx.xx:xxxxx | | Query | 2022-11-24 18:08:29 | 0 | OK | SHOW PROCESSLIST | false | +------+------+---------------------+-------+---------+---------------------+------+-------+-------------------+-----------+ ``` ##### FE audit log[​](#fe-audit-log "Direct link to FE audit log") You can check the FE audit log file **fe.audit.log**. The field `PendingTimeMs` indicates the time a query spent waiting in a queue, and its unit is milliseconds. ##### Monitoring metrics[​](#monitoring-metrics "Direct link to Monitoring metrics") You can obtain metrics of query queues in StarRocks using the [Monitor and Alert](https://docs.starrocks.io/docs/administration/management/monitoring.md) feature. The following FE metrics are derived from the statistical data of each FE node. | Metric | Unit | Type | Description | | ----------------------------------------------------- | ----- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | starrocks\_fe\_query\_queue\_pending | Count | Instantaneous | The current number of queries in the queue. | | starrocks\_fe\_query\_queue\_total | Count | Instantaneous | The total number of queries historically queued (including those currently running). | | starrocks\_fe\_query\_queue\_timeout | Count | Instantaneous | The total number of queries that have timed out while in the queue. | | starrocks\_fe\_resource\_group\_query\_queue\_total | Count | Instantaneous | The total number of queries historically queued in this resource group (including those currently running). The `name` label indicates the name of the resource group. This metric is supported from v3.1.4 onwards. | | starrocks\_fe\_resource\_group\_query\_queue\_pending | Count | Instantaneous | The number of queries currently in the queue for this resource group. The `name` label indicates the name of the resource group. This metric is supported from v3.1.4 onwards. | | starrocks\_fe\_resource\_group\_query\_queue\_timeout | Count | Instantaneous | The number of queries that have timed out while in the queue for this resource group. The `name` label indicates the name of the resource group. This metric is supported from v3.1.4 onwards. | ##### SHOW RUNNING QUERIES[​](#show-running-queries "Direct link to SHOW RUNNING QUERIES") From v3.1.4 onwards, StarRocks supports the SQL statement `SHOW RUNNING QUERIES`, which is used to display queue information for each query. The meanings of each field are as follows: * `QueryId`: The ID of the query. * `ResourceGroupId`: The ID of the resource group that the query hit. When there is no hit on a user-defined resource group, it will be displayed as "-". * `StartTime`: The start time of the query. * `PendingTimeout`: The time when the PENDING query will time out in the queue. * `QueryTimeout`: The time when the query times out. * `State`: The queue state of the query, where "PENDING" indicates it is in the queue, and "RUNNING" indicates it is currently executing. * `Slots`: The logical resource quantity requested by the query. In Query Queue v1, this value is usually `1`. In Query Queue v2, this value is the estimated number of slots for the query. * `Frontend`: The FE node that initiated the query. * `FeStartTime`: The start time of the FE node that initiated the query. Example: ```plain MySQL [(none)]> SHOW RUNNING QUERIES; +--------------------------------------+-----------------+---------------------+---------------------+---------------------+-----------+-------+---------------------------------+---------------------+ | QueryId | ResourceGroupId | StartTime | PendingTimeout | QueryTimeout | State | Slots | Frontend | FeStartTime | +--------------------------------------+-----------------+---------------------+---------------------+---------------------+-----------+-------+---------------------------------+---------------------+ | a46f68c6-3b49-11ee-8b43-00163e10863a | - | 2023-08-15 16:56:37 | 2023-08-15 17:01:37 | 2023-08-15 17:01:37 | RUNNING | 1 | 127.00.00.01_9010_1692069711535 | 2023-08-15 16:37:03 | | a6935989-3b49-11ee-935a-00163e13bca3 | 12003 | 2023-08-15 16:56:40 | 2023-08-15 17:01:40 | 2023-08-15 17:01:40 | RUNNING | 1 | 127.00.00.02_9010_1692069658426 | 2023-08-15 16:37:03 | | a7b5e137-3b49-11ee-8b43-00163e10863a | 12003 | 2023-08-15 16:56:42 | 2023-08-15 17:01:42 | 2023-08-15 17:01:42 | PENDING | 1 | 127.00.00.03_9010_1692069711535 | 2023-08-15 16:37:03 | +--------------------------------------+-----------------+---------------------+---------------------+---------------------+-----------+-------+---------------------------------+---------------------+ ``` --- ### Manage replica Manage data replicas in your StarRocks cluster. This topic includes two sections - [shared-nothing](#shared-nothing) and [shared-data](#shared-data). #### Shared-nothing[​](#shared-nothing "Direct link to Shared-nothing") ##### Overview[​](#overview "Direct link to Overview") For native tables in shared-nothing clusters, StarRocks adopts a multi-replica strategy to guarantee the high availability of data. When you create a table, you must specify the replica count of the table using the table property `replication_num` (Default value: `3`). When a loading transaction starts, data is simultaneously loaded into the specified number of replicas. The transaction is returned with success only after the data is stored in the majority of replicas. For detailed information, see [Write quorum](#write-quorum). Nonetheless, StarRocks allows you to specify a lower write quorum for a table to achieve better loading performance. StarRocks stores multiple replicas across different BE nodes. For example, if you want to store three replicas for a table, you must deploy at least three BE nodes in your StarRocks cluster. If any of the replicas fail, StarRocks clones a healthy replica, partially or wholly, from another BE node to repair the failed replica. By using the Multi-Version Concurrency Control (MVCC) technique, StarRocks accelerates the repairing of the replica by duplicating the physical copies of these multi-version data. ##### Loading data into a multi-replica table[​](#loading-data-into-a-multi-replica-table "Direct link to Loading data into a multi-replica table") ![Replica-1](/assets/images/replica-1-39f5580148f6b7a0ee79929f5ae45c7a.png) The routine of a loading transaction is as follows: 1. The client submits a loading request to FE. 2. FE chooses a BE node as the Coordinator BE node of this loading transaction, and generates an execution plan for the transaction. 3. The Coordinator BE node reads the data to be loaded from the client. 4. The Coordinator BE node dispatches the data to all the replicas of tablets. > **NOTE** > > A tablet is a logical slice of a table. A table has multiple tablets, and each tablet has replication\_num replicas. The number of tablets in a table is determined by the `bucket_size` property of the table. 5. After the data is loaded and stored in all the tablets, FE makes the loaded data visible. 6. FE returns loading success to the client. Such a routine guarantees service availability even under extreme scenarios. ##### Write quorum[​](#write-quorum "Direct link to Write quorum") Loading data into a multi-replica table can be very time-consuming. If you want to improve the loading performance and you can tolerate relatively lower data availability, you can set a lower write quorum for tables. A write quorum refers to the minimum number of replicas that need to acknowledge a write operation before it is considered successful. You can specify write quorum by adding the property `write_quorum` when you [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md), or add this property to an existing table using [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md). This property is supported from v2.5. `write_quorum` supports the following values: * `MAJORITY`: Default value. When the majority of data replicas return loading success, StarRocks returns loading task success. Otherwise, StarRocks returns loading task failed. * `ONE`: When one of the data replicas returns loading success, StarRocks returns loading task success. Otherwise, StarRocks returns loading task failed. * `ALL`: When all of the data replicas return loading success, StarRocks returns loading task success. Otherwise, StarRocks returns loading task failed. ##### Automatic replica repair[​](#automatic-replica-repair "Direct link to Automatic replica repair") Replicas can fail because certain BE nodes crash or some loading tasks fail. StarRocks automatically repairs these failed replicas. Every `tablet_sched_checker_interval_seconds`, default 20 seconds, Tablet Checker in FE scans all tablet replicas of all tables in your StarRocks cluster, and judges if a replica is healthy by checking the version number of the currently visible data and the health status of the BE node. If the visible version of a replica lags behind those of the other replicas, StarRocks performs an incremental clone to repair the failed replica. If a BE node fails to receive heartbeats or is dropped from the cluster, or the replica is too lagged to be repaired by an incremental clone, StarRocks performs a full clone to repair the lost replica. After detecting tablet replicas that need repair, FE generates a tablet scheduling task, and adds the task to the scheduling task queue. Tablet Scheduler in the FE receives the scheduling task from the queue, creates clone tasks for each failed replica in accordance with the clone type they need, and assigns the tasks to the executor BE nodes. A clone task is essentially copying data from a source BE node (which has a healthy replica), and loading the data into the destination BE node (which has a failed replica). For a replica with a lagged data version, FE assigns an incremental clone task to the BE executor that stores the failed replica, and informs the executor BE node from which peer BE node it can find a healthy replica and clone the new data. If a replica is lost, FE chooses a surviving BE node as the executor BE node, creates an empty replica in the BE node, and assigns a full clone task to the BE node. For each clone task, regardless of its type, the executor BE node duplicates the physical data files from a healthy replica, and then updates its metadata accordingly. After the clone task is completed, the executor BE node reports task success to Tablet Scheduler in FE. After removing the redundant tablet replicas, FE updates its metadata, marking the completion of the replica repair. ![Replica-2](/assets/images/replica-2-146e16e99c04cdec3405f0dcb7fdc262.png) During tablet repair, StarRocks can still execute queries. StarRocks can load data into the table as long as the number of healthy replicas satisfies `write_quorum`. ##### Repair replica manually[​](#repair-replica-manually "Direct link to Repair replica manually") The manual replica repair consists of two steps: 1. Check the replica status. 2. Set the replica priority level. ###### Check replica status[​](#check-replica-status "Direct link to Check replica status") Follow these steps to check the replica status of tablets to identify the unhealthy (failed) tablets. 1. **Check the status of all tablets in the cluster.** ```sql SHOW PROC '/statistic'; ``` Example: ```plain mysql> SHOW PROC '/statistic'; +----------+-----------------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+ | DbId | DbName | TableNum | PartitionNum | IndexNum | TabletNum | ReplicaNum | UnhealthyTabletNum | InconsistentTabletNum | +----------+-----------------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+ | 35153636 | default_cluster:DF_Newrisk | 3 | 3 | 3 | 96 | 288 | 0 | 0 | | 48297972 | default_cluster:PaperData | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | 5909381 | default_cluster:UM_TEST | 7 | 7 | 10 | 320 | 960 | 1 | 0 | | Total | 240 | 10 | 10 | 13 | 416 | 1248 | 1 | 0 | +----------+-----------------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+ ``` * `UnhealthyTabletNum`: indicates the number of unhealthy tablets in the corresponding database. * `InconsistentTabletNum`: indicates the number of tablets whose replicas are inconsistent. If the value of `UnhealthyTabletNum` or `InconsistentTabletNum` is not `0` in a specific database, you can check the unhealthy tablets in the database with its `DbId`. ```sql SHOW PROC '/statistic/' ``` Example: ```plain mysql> SHOW PROC '/statistic/5909381'; +------------------+---------------------+ | UnhealthyTablets | InconsistentTablets | +------------------+---------------------+ | [40467980] | [] | +------------------+---------------------+ ``` The ID of the unhealthy tablet is returned in the field `UnhealthyTablets`. 2. **Check the tablet status in a specific table or partition.** You can use the WHERE clause in [ADMIN SHOW REPLICA STATUS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_REPLICA_STATUS.md) to filter the tablets with a certain `STATUS`. ```sql ADMIN SHOW REPLICA STATUS FROM [PARTITION ([, , ...])] [WHERE STATUS = {'OK'|'DEAD'|'VERSION_ERROR'|'SCHEMA_ERROR'|'MISSING'}] ``` Example: ```plain mysql> ADMIN SHOW REPLICA STATUS FROM tbl PARTITION (p1, p2) WHERE STATUS = "OK"; +----------+-----------+-----------+---------+-------------------+--------------------+------------------+------------+------------+-------+--------+--------+ | TabletId | ReplicaId | BackendId | Version | LastFailedVersion | LastSuccessVersion | CommittedVersion | SchemaHash | VersionNum | IsBad | State | Status | +----------+-----------+-----------+---------+-------------------+--------------------+------------------+------------+------------+-------+--------+--------+ | 29502429 | 29502432 | 10006 | 2 | -1 | 2 | 1 | -1 | 2 | false | NORMAL | OK | | 29502429 | 36885996 | 10002 | 2 | -1 | -1 | 1 | -1 | 2 | false | NORMAL | OK | | 29502429 | 48100551 | 10007 | 2 | -1 | -1 | 1 | -1 | 2 | false | NORMAL | OK | | 29502433 | 29502434 | 10001 | 2 | -1 | 2 | 1 | -1 | 2 | false | NORMAL | OK | | 29502433 | 44900737 | 10004 | 2 | -1 | -1 | 1 | -1 | 2 | false | NORMAL | OK | | 29502433 | 48369135 | 10006 | 2 | -1 | -1 | 1 | -1 | 2 | false | NORMAL | OK | +----------+-----------+-----------+---------+-------------------+--------------------+------------------+------------+------------+-------+--------+--------+ ``` If the field `IsBad` is `true`, this tablet is corrupted. For detailed information provided in the field `Status`, see [ADMIN SHOW REPLICA STATUS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_REPLICA_STATUS.md). You can further explore the details of tablets in the table using [SHOW TABLET](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLET.md). ```sql SHOW TABLET FROM ``` Example: ```plain mysql> SHOW TABLET FROM tbl1; +----------+-----------+-----------+------------+---------+-------------+-------------------+-----------------------+------------------+----------------------+---------------+----------+----------+--------+-------------------------+--------------+----------------------+--------------+----------------------+----------------------+----------------------+ | TabletId | ReplicaId | BackendId | SchemaHash | Version | VersionHash | LstSuccessVersion | LstSuccessVersionHash | LstFailedVersion | LstFailedVersionHash | LstFailedTime | DataSize | RowCount | State | LstConsistencyCheckTime | CheckVersion | CheckVersionHash | VersionCount | PathHash | MetaUrl | CompactionStatus | +----------+-----------+-----------+------------+---------+-------------+-------------------+-----------------------+------------------+----------------------+---------------+----------+----------+--------+-------------------------+--------------+----------------------+--------------+----------------------+----------------------+----------------------+ | 29502429 | 29502432 | 10006 | 1421156361 | 2 | 0 | 2 | 0 | -1 | 0 | N/A | 784 | 0 | NORMAL | N/A | -1 | -1 | 2 | -5822326203532286804 | url | url | | 29502429 | 36885996 | 10002 | 1421156361 | 2 | 0 | -1 | 0 | -1 | 0 | N/A | 784 | 0 | NORMAL | N/A | -1 | -1 | 2 | -1441285706148429853 | url | url | | 29502429 | 48100551 | 10007 | 1421156361 | 2 | 0 | -1 | 0 | -1 | 0 | N/A | 784 | 0 | NORMAL | N/A | -1 | -1 | 2 | -4784691547051455525 | url | url | +----------+-----------+-----------+------------+---------+-------------+-------------------+-----------------------+------------------+----------------------+---------------+----------+----------+--------+-------------------------+--------------+----------------------+--------------+----------------------+----------------------+----------------------+ ``` The returned results show the size, row count, version, and URL of the tablets. The field `State` returned by SHOW TABLET indicates the task state of the tablet, including `CLONE`, `SCHEMA_CHANGE`, and `ROLLUP`. You can also check the replica distribution of a specific table or partition to see if these replicas are distributed evenly using [ADMIN SHOW REPLICA DISTRIBUTION](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_REPLICA_DISTRIBUTION.md). ```sql ADMIN SHOW REPLICA DISTRIBUTION FROM ``` Example: ```plain mysql> ADMIN SHOW REPLICA DISTRIBUTION FROM tbl1; +-----------+------------+-------+---------+ | BackendId | ReplicaNum | Graph | Percent | +-----------+------------+-------+---------+ | 10000 | 7 | | 7.29 % | | 10001 | 9 | | 9.38 % | | 10002 | 7 | | 7.29 % | | 10003 | 7 | | 7.29 % | | 10004 | 9 | | 9.38 % | | 10005 | 11 | > | 11.46 % | | 10006 | 18 | > | 18.75 % | | 10007 | 15 | > | 15.62 % | | 10008 | 13 | > | 13.54 % | +-----------+------------+-------+---------+ ``` The returned results show the number of tablet replicas on each BE node, and their corresponding percentages. 3. **Check the** **replica** **status of a specific tablet.** With the `TabletId` of the unhealthy tablets you obtained in the preceding procedures, you can examine the replica statues of them. ```sql SHOW TABLET ``` Example: ```plain mysql> SHOW TABLET 29502553; +------------------------+-----------+---------------+-----------+----------+----------+-------------+----------+--------+---------------------------------------------------------------------------+ | DbName | TableName | PartitionName | IndexName | DbId | TableId | PartitionId | IndexId | IsSync | DetailCmd | +------------------------+-----------+---------------+-----------+----------+----------+-------------+----------+--------+---------------------------------------------------------------------------+ | default_cluster:test | test | test | test | 29502391 | 29502428 | 29502427 | 29502428 | true | SHOW PROC '/dbs/29502391/29502428/partitions/29502427/29502428/29502553'; | +------------------------+-----------+---------------+-----------+----------+----------+-------------+----------+--------+---------------------------------------------------------------------------+ ``` The returned results show detailed information about the database, table, partition, and index (Rollup) of the tablet. You can copy the SQL statement in the field `DetailCmd` to further examine the replica status of the tablet. Example: ```plain mysql> SHOW PROC '/dbs/29502391/29502428/partitions/29502427/29502428/29502553'; +-----------+-----------+---------+-------------+-------------------+-----------------------+------------------+----------------------+---------------+------------+----------+----------+--------+-------+--------------+----------------------+----------+------------------+ | ReplicaId | BackendId | Version | VersionHash | LstSuccessVersion | LstSuccessVersionHash | LstFailedVersion | LstFailedVersionHash | LstFailedTime | SchemaHash | DataSize | RowCount | State | IsBad | VersionCount | PathHash | MetaUrl | CompactionStatus | +-----------+-----------+---------+-------------+-------------------+-----------------------+------------------+----------------------+---------------+------------+----------+----------+--------+-------+--------------+----------------------+----------+------------------+ | 43734060 | 10004 | 2 | 0 | -1 | 0 | -1 | 0 | N/A | -1 | 784 | 0 | NORMAL | false | 2 | -8566523878520798656 | url | url | | 29502555 | 10002 | 2 | 0 | 2 | 0 | -1 | 0 | N/A | -1 | 784 | 0 | NORMAL | false | 2 | 1885826196444191611 | url | url | | 39279319 | 10007 | 2 | 0 | -1 | 0 | -1 | 0 | N/A | -1 | 784 | 0 | NORMAL | false | 2 | 1656508631294397870 | url | url | +-----------+-----------+---------+-------------+-------------------+-----------------------+------------------+----------------------+---------------+------------+----------+----------+--------+-------+--------------+----------------------+----------+------------------+ ``` The returned results show all the replicas of the tablet. ###### Set replica priority level[​](#set-replica-priority-level "Direct link to Set replica priority level") Tablet Scheduler automatically assigns a different priority level to each different type of clone task. If you want the tablets from a certain table or certain partitions to be repaired at the earliest opportunity, you can manually assign a `VERY_HIGH` priority level to them using [ADMIN REPAIR TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md). ```sql ADMIN REPAIR TABLE [PARTITION ([, , ...])] ``` > **NOTE** > > * Executing this SQL statement only submits a hint to modify the priority level of the tablets to be repaired. It does not guarantee that these tablets can be successfully repaired. > * Tablet Scheduler might still assign different priority levels to these tablets after you execute this SQL statement. > * When the Leader FE node is changed or restarted, the hint this SQL statement submitted expires. You can cancel this operation using [ADMIN CANCEL REPAIR TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_CANCEL_REPAIR.md). ```sql ADMIN CANCEL REPAIR TABLE [PARTITION ([, , ...])] ``` ##### Replica balancing[​](#replica-balancing "Direct link to Replica balancing") StarRocks automatically balances the tablets across BE nodes. To move a tablet from a high-load node to a low-load node, StarRocks first creates a replica of the tablet in the low-load node, and then drops the corresponding replica on the high-load node. If different types of storage mediums are used in the cluster, StarRocks categorizes all the BE nodes in accordance with the storage medium types. StarRocks moves the tablet across the BE nodes of the same storage medium type whenever possible. Replicas of the same tablet are stored on different BE nodes. ###### BE load[​](#be-load "Direct link to BE load") StarRocks shows the load statistics of each BE node in the cluster using `ClusterLoadStatistics` (CLS). Tablet Scheduler triggers the replica balancing based on `ClusterLoadStatistics`. StarRocks evaluates the **disk utilization** and the **replica count** of each BE node and calculates a `loadScore` accordingly. The higher the `loadScore` of a BE node, the higher the load the node has. Tablet Scheduler updates `ClusterLoadStatistics` every one minute. `capacityCoefficient` and `replicaNumCoefficient`are the weighting factors for the disk utilization and the replica count respectively. The sum of `capacityCoefficient` and `replicaNumCoefficient` is one. `capacityCoefficient` is dynamically adjusted according to the actual disk usage. When the overall disk utilization of a BE node is below 50%, the `capacityCoefficient` value is 0.5. When the disk utilization is above 75%, the value is 1. You can configure this limit via the FE configuration item `capacity_used_percent_high_water`. If the utilization is between 50% and 75%, `capacityCoefficient` increases smoothly based on this formula: ```sql capacityCoefficient= 2 * Disk utilization - 0.5 ``` `capacityCoefficient` ensures that when the disk usage is exceedingly high, the `loadScore` of this BE node gets higher, forcing the system to reduce the load on this BE node at the earliest opportunity. ###### Balancing policy[​](#balancing-policy "Direct link to Balancing policy") Each time Tablet Scheduler schedules tablets, it selects a certain number of healthy tablets as the candidate tablets to be balanced through Load Balancer. Next time when scheduling tablets, Tablet Scheduler balances these healthy tablets. ###### View System Balance Status[​](#view-system-balance-status "Direct link to View System Balance Status") You can view the current overall balance status of the system and the details of different balance types. * **View the current overall balance status of the system:** ```sql SHOW PROC '/cluster_balance/balance_stat'; ``` Example: ```plain +---------------+--------------------------------+----------+----------------+----------------+ | StorageMedium | BalanceType | Balanced | PendingTablets | RunningTablets | +---------------+--------------------------------+----------+----------------+----------------+ | HDD | inter-node disk usage | true | 0 | 0 | | HDD | inter-node tablet distribution | true | 0 | 0 | | HDD | intra-node disk usage | true | 0 | 0 | | HDD | intra-node tablet distribution | true | 0 | 0 | | HDD | colocation group | true | 0 | 0 | | HDD | label-aware location | true | 0 | 0 | +---------------+--------------------------------+----------+----------------+----------------+ ``` * `StorageMedium`: Storage medium. * `BalanceType`: Type of balance. * `Balanced`: Whether the balanced state is achieved. * `PendingTablets`: Number of tablets with task status Pending. * `RunningTablets`: Number of tablets with task status Running. * **View the balance of disk utilization by node:** ```sql SHOW PROC '/cluster_balance/cluster_load_stat'; ``` Example: ```plain +---------------+----------------------------------------------------------------------------------------------------------------------+ | StorageMedium | ClusterDiskBalanceStat | +---------------+----------------------------------------------------------------------------------------------------------------------+ | HDD | {"balanced":false,"maxBeId":1,"minBeId":2,"maxUsedPercent":0.9,"minUsedPercent":0.1,"type":"INTER_NODE_DISK_USAGE"} | | SSD | {"balanced":true} | +---------------+----------------------------------------------------------------------------------------------------------------------+ ``` * `StorageMedium`: Storage medium. * `ClusterDiskBalanceStat`: Balance status across nodes based on disk usage. If not balanced, displays the maximum and minimum disk utilization and the corresponding BEs. * **View the balance of disk usage within node:** ```sql SHOW PROC '/cluster_balance/cluster_load_stat/HDD'; ``` Example: ```plain +-------+-----------------+-----------+--------------+--------------+-------------+------------+----------+-----------+-------+-------+---------------------------------------------------------------------------------------------------------------------------------------------+ | BeId | Cluster | Available | UsedCapacity | Capacity | UsedPercent | ReplicaNum | CapCoeff | ReplCoeff | Score | Class | BackendDiskBalanceStat | +-------+-----------------+-----------+--------------+--------------+-------------+------------+----------+-----------+-------+-------+---------------------------------------------------------------------------------------------------------------------------------------------+ | 10004 | default_cluster | true | 651509602 | 243695955810 | 0.267 | 339 | 0.5 | 0.5 | 1.0 | MID | {"maxUsedPercent":0.9,"minUsedPercent":0.1,"beId":1,"maxPath":"/disk1","minPath":"/disk2","type":"INTRA_NODE_DISK_USAGE","balanced":false} | | 10005 | default_cluster | true | 651509602 | 243695955810 | 0.267 | 339 | 0.5 | 0.5 | 1.0 | MID | {"balanced":true} | +-------+-----------------+-----------+--------------+--------------+-------------+------------+----------+-----------+-------+-------+---------------------------------------------------------------------------------------------------------------------------------------------+ ``` * `BeId`: ID of the BE node. * `BackendDiskBalanceStat`: Balance status between disks within the node based on disk utilization. If not balanced, displays the maximum and minimum disk usage and the corresponding disk paths. * **View balanced distribution by tablet:** ```sql SHOW PROC '/dbs/ssb/lineorder/partitions/lineorder'; ``` Example: ```plain +---------+-----------+--------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+ | IndexId | IndexName | State | LastConsistencyCheckTime | TabletBalanceStat | +---------+-----------+--------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+ | 11129 | lineorder | NORMAL | NULL | {"maxTabletNum":23,"minTabletNum":21,"maxBeId":10012,"minBeId":10013,"type":"INTER_NODE_TABLET_DISTRIBUTION","balanced":false} | | 11230 | lineorder | NORMAL | NULL | {"maxTabletNum":23,"minTabletNum":21,"beId":10012,"maxPath":"/disk1","minPath":"/disk2","type":"INTRA_NODE_TABLET_DISTRIBUTION","balanced":false} | | 10432 | lineorder | NORMAL | NULL | {"tabletId":10435,"currentBes":[10002,10004],"expectedBes":[10003,10004],"type":"COLOCATION_GROUP","balanced":false} | | 10436 | lineorder | NORMAL | NULL | {"tabletId":10438,"currentBes":[10005,10006],"expectedLocations":{"rack":["rack1","rack2"]},"type":"LABEL_AWARE_LOCATION","balanced":false} | +---------+-----------+--------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+ ``` * `IndexId`: ID of the Materialized Index within the partition. * `TabletBalanceStat`: Balance status of tablet distribution between nodes or within nodes. If not balanced, displays the details of the imbalance, including Colocation Group, Label-aware Location. * **View partitions with imbalanced tablet distribution:** ```sql SELECT DB_NAME, TABLE_NAME, PARTITION_NAME, TABLET_BALANCED FROM information_schema.partitions_meta WHERE TABLET_BALANCED = 0; ``` Example: ```plain +--------------+---------------+----------------+-----------------+ | DB_NAME | TABLE_NAME | PARTITION_NAME | TABLET_BALANCED | +--------------+---------------+----------------+-----------------+ | ssb | lineorder | lineorder | 0 | +--------------+---------------+----------------+-----------------+ ``` * `TABLET_BALANCED`: whether the tablet distribution is balanced. ###### Check tablet scheduling tasks[​](#check-tablet-scheduling-tasks "Direct link to Check tablet scheduling tasks") You can check tablet scheduling tasks that are pending, running, and finished. * **Check pending tablet scheduling tasks** ```sql SHOW PROC '/cluster_balance/pending_tablets'; ``` Example: ```plain +----------+--------+-----------------+---------+----------+----------+-------+---------+--------+----------+---------+---------------------+---------------------+---------------------+----------+------+-------------+---------------+---------------------+------------+---------------------+--------+---------------------+-------------------------------+ | TabletId | Type | Status | State | OrigPrio | DynmPrio | SrcBe | SrcPath | DestBe | DestPath | Timeout | Create | LstSched | LstVisit | Finished | Rate | FailedSched | FailedRunning | LstAdjPrio | VisibleVer | VisibleVerHash | CmtVer | CmtVerHash | ErrMsg | +----------+--------+-----------------+---------+----------+----------+-------+---------+--------+----------+---------+---------------------+---------------------+---------------------+----------+------+-------------+---------------+---------------------+------------+---------------------+--------+---------------------+-------------------------------+ | 4203036 | REPAIR | REPLICA_MISSING | PENDING | HIGH | LOW | -1 | -1 | -1 | -1 | 0 | 2019-02-21 15:00:20 | 2019-02-24 11:18:41 | 2019-02-24 11:18:41 | N/A | N/A | 2 | 0 | 2019-02-21 15:00:43 | 1 | 0 | 2 | 0 | unable to find source replica | +----------+--------+-----------------+---------+----------+----------+-------+---------+--------+----------+---------+---------------------+---------------------+---------------------+----------+------+-------------+---------------+---------------------+------------+---------------------+--------+---------------------+-------------------------------+ ``` * `TabletId`: the ID of the tablet pending to be scheduled. A scheduled task is only for one tablet. * `Type`: the task type. Valid values: REPAIR and BALANCE. * `Status`: the current status of the tablet, such as REPLICA\_MISSING. * `State`: the state of the scheduling task. Valid values: PENDING, RUNNING, FINISHED, CANCELLED, TIMEOUT, and UNEXPECTED. * `OrigPrio`: the original priority of the task. * `DynmPrio`: the current priority of the task after the dynamic adjustment. * `SrcBe`: the ID of the source BE node. * `SrcPath`: the hash value of the path to the source BE node. * `DestBe`: the ID of the destination BE node. * `DestPath`: the hash value of the path to the destination BE node. * `Timeout`: the timeout of the task when the task is scheduled successfully. Unit: second. * `Create`: the time when the task was created. * `LstSched`: the time when the task was scheduled most recently. * `LstVisit`: the time when the task was visited most recently. To visit the task here means to schedule the task or to report its execution. * `Finished`: the time when the task is finished. * `Rate`: the rate at which the data is cloned. * `FailedSched`: the number of task scheduling failures. * `FailedRunning`: the number of task execution failures. * `LstAdjPrio`: the time when the task priority was adjusted most recently. * `CmtVer`, `CmtVerHash`, `VisibleVer`, and `VisibleVerHash`: the version information used to execute the clone task. * `ErrMsg`: the error message that occurs when the task is scheduled and running. * **Check running tablet scheduling tasks** ```sql SHOW PROC '/cluster_balance/running_tablets'; ``` The returned results are identical to those of the pending tasks. * **Check finished tablet scheduling tasks** ```sql SHOW PROC '/cluster_balance/history_tablets'; ``` The returned results are identical to those of the pending tasks. If the `State` of the task is `FINISHED`, the task is completed successfully. If not, please check the `ErrMsg` field for the cause of the task failure. ##### Resource control[​](#resource-control "Direct link to Resource control") Because StarRocks repairs and balances tablets by cloning tablets from one BE node to another, the I/O load of a BE node can increase dramatically if the node executes such tasks too frequently in a short time. To avoid this situation, StarRocks sets a concurrency limit on clone tasks for each BE node. The minimum unit of resource control is a disk, which is a data storage path (`storage_root_path`) you have specified in the BE configuration file. By default, StarRocks allocates two slots for each disk to process tablet repair tasks. A clone task occupies one slot on the source BE node and one on the destination BE node. If all the slots on a BE node are occupied, StarRocks stops scheduling tasks to the node. You can increase the number of slots on a BE node by increasing the value of the FE dynamic parameter `tablet_sched_slot_num_per_path`. StarRocks allocates two slots specifically for tablet balancing tasks to avoid the situation that a high-load BE node fails to release the disk space by balancing tablets because tablet repair tasks constantly occupy the slots. #### Shared-data[​](#shared-data "Direct link to Shared-data") From v4.1 onwards, StarRocks supports repairing the data replica of cloud-native tables in shared-data clusters. ##### Overview[​](#overview-1 "Direct link to Overview") In a shared-data architecture, data is stored in single-replica mode on remote storage systems such as object storage or HDFS in order to reduce storage costs. Unlike traditional shared-nothing architectures, this design cannot rely on multiple replicas to automatically recover data when files are lost. As a result, if the effective metadata version maintained by FE references metadata or data files that no longer exist in remote storage, data ingestion and query operations will fail with "File not found" errors, potentially rendering the service unavailable. Such file loss may occur under the following circumstances: * **Accidental deletion**: Object storage files are mistakenly removed due to operational errors. * **Consistency issues**: In extreme cases, the storage system experiences delayed consistency or metadata loss. * **Software defects**: System bugs cause files to be cleaned up prematurely. Traditional snapshot-based restore mechanisms are often time-consuming and costly, making them unsuitable for fast recovery in production environments. To address this issue, StarRocks provides a low-cost, second-level recovery mechanism. By scanning historical metadata versions, the system identifies the most recent healthy version in which all required files are present, and rolls back the Tablet metadata to that version. This approach sacrifices a small amount of recent data in exchange for rapid restoration of table availability. ###### Mechanism[​](#mechanism "Direct link to Mechanism") This feature is an extension of the ADMIN REPAIR TABLE statement for cloud-native tables in shared-data clusters. It operates through the following mechanisms: 1. **Automated Detection** The FE coordinates Compute Nodes (CNs) to probe historical Tablet metadata versions in reverse chronological order and in batches. 2. **Deterministic Path Derivation** Metadata file paths are derived deterministically, allowing direct probing without performing expensive and inefficient object storage List Objects operations. 3. **Multi-Strategy Recovery Decisions** Two recovery strategies are supported—Strict Consistency and Maximum Availability—to accommodate different business requirements. 4. **Metadata Reset Capability** When metadata is completely unavailable, the system can create empty Tablets (Empty Tablet Recovery) to prevent a small number of corrupted Tablets from blocking the availability of the entire table. ###### Key Benefits[​](#key-benefits "Direct link to Key Benefits") * **Ultra-fast recovery** Only metadata (KB/MB scale) is modified. No data movement is required, enabling second-level recovery even for PB-scale tables. * **Low operational cost** No additional replicas or expensive object storage API calls are needed. ##### Check Tablet Status[​](#check-tablet-status "Direct link to Check Tablet Status") Before executing a repair, you can use the [ADMIN SHOW TABLET STATUS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_TABLET_STATUS.md) statement to check the file integrity of each Tablet in a cloud-native table or cloud-native materialized view, and identify whether metadata or data files are missing. **Only cloud-native tables and cloud-native materialized views in shared-data clusters support this statement.** ###### Syntax[​](#syntax "Direct link to Syntax") ```sql ADMIN SHOW TABLET STATUS FROM [.] [PARTITION ( [, , ...])] [WHERE STATUS [=|!=] {'NORMAL'|'MISSING_META'|'MISSING_DATA'}] [PROPERTIES ("max_missing_data_files_to_show" = "")] ``` **Tablet status values:** | Status | Description | | ------------- | --------------------------------------- | | NORMAL | Metadata and data files are all intact. | | MISSING\_META | Metadata file is missing. | | MISSING\_DATA | Data file(s) are missing. | **Return columns:** | Column | Description | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | TabletId | Tablet ID. | | PartitionId | Physical partition ID of the partition the Tablet belongs to. | | Version | Current visible version of the Tablet. | | Status | Tablet status. See status values above. | | MissingDataFileCount | Number of missing data files. Populated when `Status` is `MISSING_DATA`. | | MissingDataFiles | List of missing data file paths. Limited by `max_missing_data_files_to_show` (default: 5). Populated when `Status` is `MISSING_DATA`. | ###### Example: Check abnormal tablets in a partition[​](#example-check-abnormal-tablets-in-a-partition "Direct link to Example: Check abnormal tablets in a partition") ```sql ADMIN SHOW TABLET STATUS FROM my_cloud_table PARTITION (p20250101) WHERE STATUS != "NORMAL"; ``` ##### Usage[​](#usage "Direct link to Usage") ###### Syntax[​](#syntax-1 "Direct link to Syntax") Use the ADMIN REPAIR TABLE statement with PROPERTIES to control recovery behavior. ```sql ADMIN REPAIR TABLE [PARTITION (, ...)] PROPERTIES ( 'enforce_consistent_version' = 'true', 'allow_empty_tablet_recovery' = 'false' ); ``` **Properties** | Property | Type | Default | Description | | ------------------------------ | ------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enforce\_consistent\_version | Boolean | True | Whether to enforce all tablets in a partition to roll back to a consistent version. If this item is set to `true`, the system will search for a consistent version in the history that is valid for all tablets to perform the rollback, ensuring data version alignment across the partition. If it is set to `false`, each tablet in the partition is allowed to rollback to its latest available valid version. The versions of different tablets may be inconsistent, but this maximizes data preservation. | | allow\_empty\_tablet\_recovery | Boolean | False | Whether to allow recovery by creating empty tablets. This item takes effect only when `enforce_consistent_version` is `false`. If this item is set to `true`, when metadata is missing for all versions of some tablets but valid metadata exists for at least one tablet, the system attempts to create empty tablets to fill the missing versions. If metadata for all versions of all tablets is lost, recovery is impossible. | | dry\_run | Boolean | False | Whether to return the repair plan without actually executing the repair. If set to `true`, the system evaluates the recoverability of each partition and returns the repair plan without performing any actual metadata rollback. Useful for previewing the repair effect before execution. See [Preview the Repair Plan (Dry Run)](#preview-the-repair-plan-dry-run). | ##### Examples[​](#examples "Direct link to Examples") ###### Example 1: Strict Consistency Recovery (Recommended for Strong Consistency)[​](#example-1-strict-consistency-recovery-recommended-for-strong-consistency "Direct link to Example 1: Strict Consistency Recovery (Recommended for Strong Consistency)") Restore all Tablets in partition `p20250101` to the most recent uniform and complete version. ```sql ADMIN REPAIR TABLE my_cloud_table PARTITION (p20250101); ``` ###### Example 2: Maximum Availability Recovery[​](#example-2-maximum-availability-recovery "Direct link to Example 2: Maximum Availability Recovery") Restore each Tablet in partition p20250101 to its own most recent valid version, allowing version divergence. ```sql ADMIN REPAIR TABLE my_cloud_table PARTITION (p20250101) PROPERTIES ( 'enforce_consistent_version' = 'false' ); ``` ###### Example 3: Allow Empty Tablet Recovery[​](#example-3-allow-empty-tablet-recovery "Direct link to Example 3: Allow Empty Tablet Recovery") Allow Tablets with completely missing metadata to be recovered as empty Tablets. ```sql ADMIN REPAIR TABLE my_cloud_table PARTITION (p20250101) PROPERTIES ( 'enforce_consistent_version' = 'false', 'allow_empty_tablet_recovery' = 'true' ); ``` ##### Preview the Repair Plan (Dry Run)[​](#preview-the-repair-plan-dry-run "Direct link to Preview the Repair Plan (Dry Run)") Before executing the repair, you can use `dry_run` mode to preview the repair plan. This lets you check the recoverability of each partition and the version each tablet would roll back to, without actually modifying any metadata. **`dry_run` mode is only supported for cloud-native tables in shared-data clusters.** ###### Syntax[​](#syntax-2 "Direct link to Syntax") ```sql ADMIN REPAIR TABLE [PARTITION (, ...)] PROPERTIES ( 'dry_run' = 'true' ); ``` ###### Return Columns[​](#return-columns "Direct link to Return Columns") | Column | Description | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | PartitionId | Partition ID. | | VisibleVersion | Current visible version. | | RepairStatus | Repair status: `NORMAL` (all tablets healthy, no repair needed), `RECOVERABLE` (missing files but recoverable), `UNRECOVERABLE` (missing files and unrecoverable), `UNKNOWN` (exception during probing). | | TabletRecoverInfo | JSON array listing the version each tablet will roll back to. Only populated when `RepairStatus` is `RECOVERABLE`. | | ErrorMsg | Error message. Only populated when `RepairStatus` is `UNRECOVERABLE` or `UNKNOWN`. | ###### Example: Preview the repair plan for partition p1[​](#example-preview-the-repair-plan-for-partition-p1 "Direct link to Example: Preview the repair plan for partition p1") ```sql ADMIN REPAIR TABLE my_cloud_table PARTITION (p1) PROPERTIES ("dry_run" = "true"); ``` Example output: ```plain +-------------+----------------+--------------+-----------------------------------------------------------------------+---------+ | PartitionId | VisibleVersion | RepairStatus | TabletRecoverInfo | ErrorMsg| +-------------+----------------+--------------+-----------------------------------------------------------------------+---------+ | 10001 | 100 | RECOVERABLE | [{"tabletId":20001,"version":98},{"tabletId":20002,"version":98}] | | | 10002 | 100 | NORMAL | [] | | +-------------+----------------+--------------+-----------------------------------------------------------------------+---------+ ``` After confirming the repair plan, remove `dry_run` (or set it to `false`) to execute the actual repair. ##### Limitations and Recommendations[​](#limitations-and-recommendations "Direct link to Limitations and Recommendations") * Repairing replicas is supported for shared-data clusters from v3.5 onwards. * Currently, setting `PROPERTIES` is applicable only to shared-data tables. * Take the following points into consideration when performing materialized view recovery: * Asynchronous materialized views require a manual refresh after recovery. * Synchronous materialized views must use Strict Consistency Recovery to ensure consistency between base tables and rollup data. * Recovery is best-effort. A small amount of data loss is expected, depending on the available historical versions and the selected recovery strategy. --- ### Resource group This topic describes the resource group feature of StarRocks. ![resource group](/assets/images/resource_group-48d40767073c72d56e922e11a48e213c.png) With this feature, you could simultaneously run several workloads in a single cluster, including short query, ad-hoc query, ETL jobs, to save extra cost of deploying multiple clusters. From technical perspective, the execution engine would schedule concurrent workloads according to users' specification and isolate the interference among them. The roadmap of Resource Group: * Since v2.2, StarRocks supports limiting resource consumption for queries and implementing isolation and efficient use of resources among tenants in the same cluster. * In StarRocks v2.3, you can further restrict the resource consumption for big queries, and prevent the cluster resources from getting exhausted by oversized query requests, to guarantee the system stability. * StarRocks v2.5 supports limiting computation resource consumption for data loading (INSERT). * From v3.3.5 onwards, StarRocks supports imposing hard limits on CPU resources. | | Internal Table | External Table | Big Query Restriction | INSERT INTO | Broker Load | Routine Load, Stream Load, Schema Change | CPU Hard Limit | | --------------- | -------------- | -------------- | --------------------- | ----------- | ----------- | ---------------------------------------- | -------------- | | 2.2 | √ | × | × | × | × | × | x | | 2.3 | √ | √ | √ | × | × | × | x | | 2.5 | √ | √ | √ | √ | × | × | x | | 3.1 & 3.2 | √ | √ | √ | √ | √ | × | x | | 3.3.5 and later | √ | √ | √ | √ | √ | × | √ | #### Terms[​](#terms "Direct link to Terms") This section describes the terms that you must understand before you use the resource group feature. ##### resource group[​](#resource-group-1 "Direct link to resource group") Each resource group is a set of computing resources from a specific BE. You can divide each BE of your cluster into multiple resource groups. When a query is assigned to a resource group, StarRocks allocates CPU and memory resources to the resource group based on the resource quotas that you specified for the resource group. You can specify CPU and memory resource quotas for a resource group on a BE by using the following parameters: | Parameter | Description | Value Range | Default | | ------------------------------ | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | ------------------ | | cpu\_weight | The CPU scheduling weight of this resource group on a BE node. | (0, `avg_be_cpu_cores`] (takes effect when greater than 0) | 0 | | cpu\_weight\_percent | The CPU scheduling weight percentage of this resource group on a BE node. Supported from v4.1. | \[0, 100] (takes effect when greater than 0) | 0 | | exclusive\_cpu\_cores | CPU hard isolation parameter for this resource group. | (0, `min_be_cpu_cores - 1`] (takes effect when greater than 0) | 0 | | exclusive\_cpu\_percent | CPU hard isolation percentage for this resource group. Supported from v4.1. | \[0, 100] (takes effect when greater than 0) | 0 | | mem\_limit | The percentage of memory available for queries by this resource group on the current BE node. | (0, 1] (required) | - | | mem\_pool | Groups resource groups to share a memory limit. | String | default\_mem\_pool | | spill\_mem\_limit\_threshold | Memory usage threshold that triggers spilling to disk. | (0, 1] | 1.0 | | concurrency\_limit | Maximum number of concurrent queries in this resource group. | Integer (takes effect when greater than 0) | 0 | | big\_query\_cpu\_second\_limit | Maximum CPU time (in seconds) for big query tasks on each BE node. | Integer (takes effect when greater than 0) | 0 | | big\_query\_scan\_rows\_limit | Maximum number of rows big query tasks can scan on each BE node. | Integer (takes effect when greater than 0) | 0 | | big\_query\_mem\_limit | Maximum memory big query tasks can use on each BE node. | Integer (takes effect when greater than 0) | 0 | ###### CPU resource parameters[​](#cpu-resource-parameters "Direct link to CPU resource parameters") ###### `cpu_weight` and `cpu_weight_percent`[​](#cpu_weight-and-cpu_weight_percent "Direct link to cpu_weight-and-cpu_weight_percent") These parameters specify the CPU scheduling weight of a resource group on a single BE node, determining the relative share of CPU time allocated to tasks from this group. Before v3.3.5, they were referred to as `cpu_core_limit`. You can use one of the following parameters to set the CPU scheduling weight: * `cpu_weight`: Directly sets the CPU scheduling weight. Its value range is (0, `avg_be_cpu_cores`], where `avg_be_cpu_cores` is the average number of CPU cores across all BE nodes. The parameter is effective only when it is set to greater than 0. * `cpu_weight_percent`: Supported from v4.1. Sets the CPU scheduling weight as a percentage. Its value range is \[0, 100]. The parameter is effective only when it is set to greater than 0. If `min_be_cpu_cores * cpu_weight_percent / 100 < 1`, the system returns an error, where `min_be_cpu_cores` is the minimum number of CPU cores across all BE nodes. At runtime, each BE converts `cpu_weight_percent` to the actual `cpu_weight` based on the number of CPU cores on that BE (`be_cpu_cores`): `cpu_weight = be_cpu_cores * cpu_weight_percent / 100`. Only one of `cpu_weight`, `cpu_weight_percent`, `exclusive_cpu_cores`, and `exclusive_cpu_percent` can be greater than 0. > **NOTE** > > For example, suppose three resource groups, rg1, rg2, and rg3, have cpu\_weight values of 2, 6, and 8, respectively. On a fully loaded BE node, these groups would receive 12.5%, 37.5%, and 50% of the CPU time. If the node is not fully loaded and rg1 and rg2 are under load while rg3 is idle, rg1 and rg2 would receive 25% and 75% of the CPU time, respectively. ###### `exclusive_cpu_cores` and `exclusive_cpu_percent`[​](#exclusive_cpu_cores-and-exclusive_cpu_percent "Direct link to exclusive_cpu_cores-and-exclusive_cpu_percent") These parameters define CPU hard isolation for a resource group. It has two implications: * **Exclusive**: Reserves a specified number of CPU cores exclusively for this resource group, making them unavailable to other groups, even when idle. * **Quota**: Limits the resource group to only using these reserved CPU cores, preventing it from using available CPU resources from other groups. You can use one of the following parameters to set CPU hard isolation: * `exclusive_cpu_cores`: Directly sets the number of reserved CPU cores. Its value range is (0, `min_be_cpu_cores - 1`]. The parameter is effective only when it is set to greater than 0. * `exclusive_cpu_percent`: Supported from v4.1. Sets the number of reserved CPU cores as a percentage. Its value range is \[0, 100]. The parameter is effective only when it is set to greater than 0. If `min_be_cpu_cores * exclusive_cpu_percent / 100 < 1`, the system returns an error. At runtime, each BE converts `exclusive_cpu_percent` to the actual `exclusive_cpu_cores` based on the number of CPU cores on that BE (`be_cpu_cores`): `exclusive_cpu_cores = be_cpu_cores * exclusive_cpu_percent / 100`. * Resource groups with `exclusive_cpu_cores` or `exclusive_cpu_percent` greater than 0 are called Exclusive Resource Groups, and the CPU cores allocated to them are called Exclusive Cores. Other groups are called Shared Resource Groups and run on Shared Cores. * The total number of `exclusive_cpu_cores` across all Exclusive Resource Groups cannot exceed `min_be_cpu_cores - 1`. If `exclusive_cpu_percent` is used, the system first converts it to CPU cores based on `min_be_cpu_cores * exclusive_cpu_percent / 100` and then calculates the total. The upper limit is set to leave at least one Shared Core available. The relationship between `exclusive_cpu_cores`, `exclusive_cpu_percent`, `cpu_weight`, and `cpu_weight_percent`: Only one of `cpu_weight`, `cpu_weight_percent`, `exclusive_cpu_cores`, and `exclusive_cpu_percent` can be active at a time. Exclusive Resource Groups operate on their own reserved Exclusive Cores without requiring a share of CPU time through `cpu_weight` or `cpu_weight_percent`. You can configure whether Shared Resource Groups can borrow Exclusive Cores from Exclusive Resource Groups using the BE configuration `enable_resource_group_cpu_borrowing`. When set to `true` (default), Shared Resouce Groups can borrow CPU resources when Exclusive Resouce Groups are idle. To modify this configuration dynamically, use the following command: ```sql UPDATE information_schema.be_configs SET VALUE = "false" WHERE NAME = "enable_resource_group_cpu_borrowing"; ``` ###### Memory resource parameters[​](#memory-resource-parameters "Direct link to Memory resource parameters") ###### `mem_limit`[​](#mem_limit "Direct link to mem_limit") Specifies the percentage of memory (query pool) available to the resource group on the current BE node. The value range is (0,1]. ###### `mem_pool`[​](#mem_pool "Direct link to mem_pool") Since v4.0, specifies a shared memory pool identifier. Resource groups with the same mem\_pool identifier draw from a shared memory pool, collectively limited by the `mem_limit`. If not specified, the resource group is assigned to `default_mem_pool`, and its memory usage is limited solely by its own `mem_limit`. All resource groups sharing the same `mem_pool` must be configured with an identical `mem_limit`. To limit two resource groups to consume 50% of memory collectively, it could be defined in the following way: ```sql CREATE RESOURCE GROUP rg1 TO (db='db1') WITH ( 'mem_limit' = '50%', 'mem_pool' = 'shared_pool' ); CREATE RESOURCE GROUP rg2 TO (db='db1') WITH ( 'mem_limit' = '50%', 'mem_pool' = 'shared_pool' ); ``` ###### `spill_mem_limit_threshold`[​](#spill_mem_limit_threshold "Direct link to spill_mem_limit_threshold") Defines the memory usage threshold that triggers spilling to disk. The value range is (0,1], with the default being 1 (inactive). Introduced in v3.1.7. * When automatic spilling is enabled (`spill_mode` set to `auto`), but resource groups are disabled, the system will spill intermediate results to disk when a query’s memory usage exceeds 80% of `query_mem_limit`. * When resource groups are enabled, spilling will occur if: * The total memory usage of all queries in the group exceeds `current BE memory limit * mem_limit * spill_mem_limit_threshold`, or * The memory usage of the current query exceeds 80% of `query_mem_limit`. ###### Query concurrency parameters[​](#query-concurrency-parameters "Direct link to Query concurrency parameters") ###### `concurrency_limit`[​](#concurrency_limit "Direct link to concurrency_limit") Defines the maximum number of concurrent queries in the resource group to prevent system overload. Effective only when greater than 0, with a default value of 0. ###### Big query resource parameters[​](#big-query-resource-parameters "Direct link to Big query resource parameters") You can configure resource limits specifically for large queries using the following parameters: ###### `big_query_cpu_second_limit`[​](#big_query_cpu_second_limit "Direct link to big_query_cpu_second_limit") Specifies the maximum CPU time (in seconds) that large query tasks can use on each BE node, summing the actual CPU time used by parallel tasks. Effective only when greater than 0, with a default value of 0. ###### `big_query_scan_rows_limit`[​](#big_query_scan_rows_limit "Direct link to big_query_scan_rows_limit") Sets a limit on the number of rows large query tasks can scan on each BE node. Effective only when greater than 0, with a default value of 0. ###### `big_query_mem_limit`[​](#big_query_mem_limit "Direct link to big_query_mem_limit") Defines the maximum memory large query tasks can use on each BE node, in bytes. Effective only when greater than 0, with a default value of 0. > **NOTE** > > When a query running in a resource group exceeds the above big query limit, the query will be terminated with an error. You can also view error messages in the `ErrorCode` column of the FE node **fe.audit.log**. ###### Type (Deprecated Since v3.3.5)[​](#type-deprecated-since-v335 "Direct link to Type (Deprecated Since v3.3.5)") Before v3.3.5, StarRocks allowed setting the `type` of a resource group to `short_query`. However, the parameter `type` has been deprecated and replaced by `exclusive_cpu_cores`. For any existing resource groups of this type, the system will automatically convert them to an Exclusive resource group where the `exclusive_cpu_cores` value equals the `cpu_weight` after upgrading to v3.3.5. ###### System-defined resource groups[​](#system-defined-resource-groups "Direct link to System-defined resource groups") There are two system-defined resource groups in each StarRocks instance: `default_wg` and `default_mv_wg`. You can modify the configuration of system-defined resource groups using the ALTER RESOURCE GROUP command, but you cannot define classifiers for them or delete system-defined resource groups. ###### default\_wg[​](#default_wg "Direct link to default_wg") `default_wg` will be assigned to regular queries that are under the management of resource groups but don't match any classifier. The default resource limits of `default_wg` are as follows: * `cpu_weight`: The number of CPU cores of the BE. * `mem_limit`: 100%. * `concurrency_limit`: 0. * `big_query_cpu_second_limit`: 0. * `big_query_scan_rows_limit`: 0. * `big_query_mem_limit`: 0. * `spill_mem_limit_threshold`: 1. ###### default\_mv\_wg[​](#default_mv_wg "Direct link to default_mv_wg") `default_mv_wg` will be assigned to asynchronous materialized view refresh tasks if no resource group is allocated to the corresponding materialized view in the property `resource_group` during materialized view creation. The default resource limits of `default_mv_wg` are as follows: * `cpu_weight`: 1. * `mem_limit`: 80%. * `concurrency_limit`: 0. * `spill_mem_limit_threshold`: 80%. ##### classifier[​](#classifier "Direct link to classifier") Each classifier holds one or more conditions that can be matched to the properties of queries. StarRocks identifies the classifier that best matches each query based on the match conditions and assigns resources for running the query. Classifiers support the following conditions: * `user`: the name of the user. * `role`: the role of the user. * `query_type`: the type of the query. `SELECT` and `INSERT` (from v2.5) are supported. When INSERT INTO or BROKER LOAD tasks hit a resource group with `query_type` as `insert`, the BE node reserves the specified CPU resources for the tasks. * `source_ip`: the CIDR block from which the query is initiated. * `db`: the database which the query accesses. It can be specified by strings separated by commas `,`. * `plan_cpu_cost_range`: The estimated CPU cost range of the query. The format is `(DOUBLE, DOUBLE]`. The default value is NULL, indicating no such restriction. The `PlanCpuCost` column in `fe.audit.log` represents the system's estimate of the CPU cost for the query. This parameter is supported from v3.1.4 onwards. * `plan_mem_cost_range`: The system-estimated memory cost range of a query. The format is `(DOUBLE, DOUBLE]`. The default value is NULL, indicating no such restriction. The `PlanMemCost` column in `fe.audit.log` represents the system's estimate of the memory cost for the query. This parameter is supported from v3.1.4 onwards. A classifier matches a query only when one or all conditions of the classifier match the information about the query. If multiple classifiers match a query, StarRocks calculates the degree of matching between the query and each classifier and identifies the classifier with the highest degree of matching. > **NOTE** > > You can view the resource group to which a query belongs in the `ResourceGroup` column of the FE node **fe.audit.log** or by running `EXPLAIN VERBOSE `, as described in [View the resource group of a query](#view-the-resource-group-of-a-query). StarRocks calculates the degree of matching between a query and a classifier by using the following rules: * If the classifier has the same value of `user` as the query, the degree of matching of the classifier increases by 1. * If the classifier has the same value of `role` as the query, the degree of matching of the classifier increases by 1. * If the classifier has the same value of `query_type` as the query, the degree of matching of the classifier increases by 1 plus the number obtained from the following calculation: 1/Number of `query_type` fields in the classifier. * If the classifier has the same value of `source_ip` as the query, the degree of matching of the classifier increases by 1 plus the number obtained from the following calculation: (32 - `cidr_prefix`)/64. * If the classifier has the same value of `db` as the query, the degree of matching of the classifier increases by 10. * If the query's CPU cost falls within the `plan_cpu_cost_range`, the degree of matching of the classifier increases by 1. * If the query's memory cost falls within the `plan_mem_cost_range`, the degree of matching of the classifier increases by 1. If multiple classifiers match a query, the classifier with a larger number of conditions has a higher degree of matching. ```plain -- Classifier B has more conditions than Classifier A. Therefore, Classifier B has a higher degree of matching than Classifier A. classifier A (user='Alice') classifier B (user='Alice', source_ip = '192.168.1.0/24') ``` If multiple matching classifiers have the same number of conditions, the classifier whose conditions are described more accurately has a higher degree of matching. ```plain -- The CIDR block that is specified in Classifier B is smaller in range than Classifier A. Therefore, Classifier B has a higher degree of matching than Classifier A. classifier A (user='Alice', source_ip = '192.168.1.0/16') classifier B (user='Alice', source_ip = '192.168.1.0/24') -- Classifier C has fewer query types specified in it than Classifier D. Therefore, Classifier C has a higher degree of matching than Classifier D. classifier C (user='Alice', query_type in ('select')) classifier D (user='Alice', query_type in ('insert','select')) ``` If multiple classifiers have the same degree of matching, one of the classifiers will be randomly selected. ```plain -- If a query simultaneously queries both db1 and db2 and the classifiers E and F have the -- highest degree of matching among the hit classifiers, one of E and F will be randomly selected. classifier E (db='db1') classifier F (db='db2') ``` #### Isolate computing resources[​](#isolate-computing-resources "Direct link to Isolate computing resources") You can isolate computing resources among queries by configuring resource groups and classifiers. ##### Enable resource groups[​](#enable-resource-groups "Direct link to Enable resource groups") To use resource group, you must enable Pipeline Engine for your StarRocks cluster: ```sql -- Enable Pipeline Engine in the current session. SET enable_pipeline_engine = true; -- Enable Pipeline Engine globally. SET GLOBAL enable_pipeline_engine = true; ``` > **NOTE** > > From v3.1.0 onwards, Resource Group is enabled by default, and the session variable `enable_resource_group` is deprecated. ##### Create resource groups and classifiers[​](#create-resource-groups-and-classifiers "Direct link to Create resource groups and classifiers") Execute the following statement to create a resource group, associate the resource group with a classifier, and allocate computing resources to the resource group: ```sql CREATE RESOURCE GROUP TO ( user='string', role='string', query_type in ('select'), source_ip='cidr' ) --Create a classifier. If you create more than one classifier, separate the classifiers with commas (`,`). WITH ( "{ cpu_weight | exclusive_cpu_cores }" = "INT", "mem_limit" = "m%", "concurrency_limit" = "INT", "type" = "str" --The type of the resource group. Set the value to normal. ); ``` Example: ```sql CREATE RESOURCE GROUP rg1 TO (user='rg1_user1', role='rg1_role1', query_type in ('select'), source_ip='192.168.x.x/24'), (user='rg1_user2', query_type in ('select'), source_ip='192.168.x.x/24'), (user='rg1_user3', source_ip='192.168.x.x/24'), (user='rg1_user4'), (db='db1') WITH ( 'exclusive_cpu_cores' = '10', 'mem_limit' = '20%', 'big_query_cpu_second_limit' = '100', 'big_query_scan_rows_limit' = '100000', 'big_query_mem_limit' = '1073741824' ); ``` ##### Specify resource group (Optional)[​](#specify-resource-group-optional "Direct link to Specify resource group (Optional)") You can specify resource group for the current session directly, including `default_wg` and `default_mv_wg`. ```sql SET resource_group = 'group_name'; ``` ##### View resource groups and classifiers[​](#view-resource-groups-and-classifiers "Direct link to View resource groups and classifiers") Execute the following statement to query all resource groups and classifiers: ```sql SHOW RESOURCE GROUPS ALL; ``` Execute the following statement to query the resource groups and classifiers of the logged-in user: ```sql SHOW RESOURCE GROUPS; ``` Execute the following statement to query a specified resource group and its classifiers: ```sql SHOW RESOURCE GROUP group_name; ``` Example: ```plain mysql> SHOW RESOURCE GROUPS ALL; +---------------+-------+------------+---------------------+-----------+----------------------------+---------------------------+---------------------+-------------------+---------------------------+----------------------------------------+ | name | id | cpu_weight | exclusive_cpu_cores | mem_limit | big_query_cpu_second_limit | big_query_scan_rows_limit | big_query_mem_limit | concurrency_limit | spill_mem_limit_threshold | classifiers | +---------------+-------+------------+---------------------+-----------+----------------------------+---------------------------+---------------------+-------------------+---------------------------+----------------------------------------+ | default_mv_wg | 3 | 1 | 0 | 80.0% | 0 | 0 | 0 | null | 80% | (id=0, weight=0.0) | | default_wg | 2 | 1 | 0 | 100.0% | 0 | 0 | 0 | null | 100% | (id=0, weight=0.0) | | rge1 | 15015 | 0 | 6 | 90.0% | 0 | 0 | 0 | null | 100% | (id=15016, weight=1.0, user=rg1_user) | | rgs1 | 15017 | 8 | 0 | 90.0% | 0 | 0 | 0 | null | 100% | (id=15018, weight=1.0, user=rgs1_user) | | rgs2 | 15019 | 8 | 0 | 90.0% | 0 | 0 | 0 | null | 100% | (id=15020, weight=1.0, user=rgs2_user) | +---------------+-------+------------+---------------------+-----------+----------------------------+---------------------------+---------------------+-------------------+---------------------------+----------------------------------------+ ``` > **NOTE** > > In the preceding example, `weight` indicates the degree of matching. To query all fields of a resource group, including deprecated fields. By adding the keyword `VERBOSE` to the three commands mentioned above, you can view all fields of the resource group, including deprecated ones, such as `type` and `max_cpu_cores`. ```sql SHOW VERBOSE RESOURCE GROUPS ALL; SHOW VERBOSE RESOURCE GROUPS; SHOW VERBOSE RESOURCE GROUP group_name; ``` ##### Manage resource groups and classifiers[​](#manage-resource-groups-and-classifiers "Direct link to Manage resource groups and classifiers") You can modify the resource quotas for each resource group. You can also add or delete classifiers from resource groups. Execute the following statement to modify the resource quotas for an existing resource group: ```sql ALTER RESOURCE GROUP group_name WITH ( 'cpu_core_limit' = 'INT', 'mem_limit' = 'm%' ); ``` Execute the following statement to delete a resource group: ```sql DROP RESOURCE GROUP group_name; ``` Execute the following statement to add a classifier to a resource group: ```sql ALTER RESOURCE GROUP ADD (user='string', role='string', query_type in ('select'), source_ip='cidr'); ``` Execute the following statement to delete a classifier from a resource group: ```sql ALTER RESOURCE GROUP DROP (CLASSIFIER_ID_1, CLASSIFIER_ID_2, ...); ``` Execute the following statement to delete all classifiers of a resource group: ```sql ALTER RESOURCE GROUP DROP ALL; ``` #### Observe resource groups[​](#observe-resource-groups "Direct link to Observe resource groups") ##### View the resource group of a query[​](#view-the-resource-group-of-a-query "Direct link to View the resource group of a query") For queries that have not yet been executed, you can view the resource group matched by the query from the `RESOURCE GROUP` field returned by `EXPLAIN VERBOSE `. While a query is running, you can check which resource group the query has hit from the `ResourceGroup` field returned by `SHOW PROC '/current_queries'` and `SHOW PROC '/global_current_queries'`. After a query has completed, you can view the resource group that the query matched by checking the `ResourceGroup` field in the **fe.audit.log** file on the FE node. * If the query is not under the management of resource groups, the column value is an empty string `""`. * If the query is under the management of resource groups but doesn't match any classifier, it will be assigned to the default resource group `default_wg`. ##### Monitoring resource groups[​](#monitoring-resource-groups "Direct link to Monitoring resource groups") You can set up [monitoring and alerting](https://docs.starrocks.io/docs/administration/management/monitoring.md) for your resource groups. Resource group-related FE and BE metrics are as follows. All the metrics below have a `name` label indicating their corresponding resource group. ##### FE metrics[​](#fe-metrics "Direct link to FE metrics") The following FE metrics only provide statistics within the current FE node: | Metric | Unit | Type | Description | | ----------------------------------------------------- | ----- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | starrocks\_fe\_query\_resource\_group | Count | Instantaneous | The number of queries historically run in this resource group (including those currently running). | | starrocks\_fe\_query\_resource\_group\_latency | ms | Instantaneous | The query latency percentile for this resource group. The label `type` indicates specific percentiles, including `mean`, `75_quantile`, `95_quantile`, `98_quantile`, `99_quantile`, `999_quantile`. | | starrocks\_fe\_query\_resource\_group\_err | Count | Instantaneous | The number of queries in this resource group that encountered an error. | | starrocks\_fe\_resource\_group\_query\_queue\_total | Count | Instantaneous | The total number of queries historically queued in this resource group (including those currently running). This metric is supported from v3.1.4 onwards. It is valid only when query queues are enabled. | | starrocks\_fe\_resource\_group\_query\_queue\_pending | Count | Instantaneous | The number of queries currently in the queue of this resource group. This metric is supported from v3.1.4 onwards. It is valid only when query queues are enabled. | | starrocks\_fe\_resource\_group\_query\_queue\_timeout | Count | Instantaneous | The number of queries in this resource group that have timed out while in the queue. This metric is supported from v3.1.4 onwards. It is valid only when query queues are enabled. | ##### BE metrics[​](#be-metrics "Direct link to BE metrics") | Metric | Unit | Type | Description | | --------------------------------------------- | ---------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | resource\_group\_running\_queries | Count | Instantaneous | The number of queries currently running in this resource group. | | resource\_group\_total\_queries | Count | Instantaneous | The number of queries historically run in this resource group (including those currently running). | | resource\_group\_bigquery\_count | Count | Instantaneous | The number of queries in this resource group that triggered the big query limit. | | resource\_group\_concurrency\_overflow\_count | Count | Instantaneous | The number of queries in this resource group that triggered the `concurrency_limit` limit. | | resource\_group\_mem\_limit\_bytes | Bytes | Instantaneous | The memory limit for this resource group. | | resource\_group\_mem\_inuse\_bytes | Bytes | Instantaneous | The memory currently in use by this resource group. | | resource\_group\_cpu\_limit\_ratio | Percentage | Instantaneous | The ratio of this resource group's `cpu_core_limit` to the total `cpu_core_limit` across all resource groups. | | resource\_group\_inuse\_cpu\_cores | Count | Average | The estimated number of CPU cores in use by this resource group. This value is an approximate estimate. It represents the average value calculated based on the statistics from two consecutive metric collections. This metric is supported from v3.1.4 onwards. | | resource\_group\_cpu\_use\_ratio | Percentage | Average | **Deprecated** The ratio of the Pipeline thread time slices used by this resource group to the total Pipeline thread time slices used by all resource groups. It represents the average value calculated based on the statistics from two consecutive metric collections. | | resource\_group\_connector\_scan\_use\_ratio | Percentage | Average | **Deprecated** The ratio of the external table Scan thread time slices used by this resource group to the total Pipeline thread time slices used by all resource groups. It represents the average value calculated based on the statistics from two consecutive metric collections. | | resource\_group\_scan\_use\_ratio | Percentage | Average | **Deprecated** The ratio of the internal table Scan thread time slices used by this resource group to the total Pipeline thread time slices used by all resource groups. It represents the average value calculated based on the statistics from two consecutive metric collections. | ##### View resource group usage information[​](#view-resource-group-usage-information "Direct link to View resource group usage information") From v3.1.4 onwards, StarRocks supports the SQL statement [SHOW USAGE RESOURCE GROUPS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group/SHOW_USAGE_RESOURCE_GROUPS.md), which is used to display usage information for each resource group across BEs. The descriptions of each field are as follows: * `Name`: The name of the resource group. * `Id`: The ID of the resource group. * `Backend`: The BE's IP or FQDN. * `BEInUseCpuCores`: The number of CPU cores currently in use by this resource group on this BE. This value is an approximate estimate. * `BEInUseMemBytes`: The number of memory bytes currently in use by this resource group on this BE. * `BERunningQueries`: The number of queries from this resource group that are still running on this BE. Please note: * BEs periodically report this resource usage information to the Leader FE at the interval specified in `report_resource_usage_interval_ms`, which is by default set to 1 second. * The results will only show rows where at least one of `BEInUseCpuCores`/`BEInUseMemBytes`/`BERunningQueries` is a positive number. In other words, the information is displayed only when a resource group is actively using some resources on a BE. Example: ```plain MySQL [(none)]> SHOW USAGE RESOURCE GROUPS; +------------+----+-----------+-----------------+-----------------+------------------+ | Name | Id | Backend | BEInUseCpuCores | BEInUseMemBytes | BERunningQueries | +------------+----+-----------+-----------------+-----------------+------------------+ | default_wg | 0 | 127.0.0.1 | 0.100 | 1 | 5 | +------------+----+-----------+-----------------+-----------------+------------------+ | default_wg | 0 | 127.0.0.2 | 0.200 | 2 | 6 | +------------+----+-----------+-----------------+-----------------+------------------+ | wg1 | 0 | 127.0.0.1 | 0.300 | 3 | 7 | +------------+----+-----------+-----------------+-----------------+------------------+ | wg2 | 0 | 127.0.0.1 | 0.400 | 4 | 8 | +------------+----+-----------+-----------------+-----------------+------------------+ ``` ##### View thread information for Exclusive and Shared resource groups[​](#view-thread-information-for-exclusive-and-shared-resource-groups "Direct link to View thread information for Exclusive and Shared resource groups") Query execution mainly involves three thread pools: `pip_exec`, `pip_scan`, and `pip_con_scan`. * Exclusive resource groups run in their dedicated thread pools and are bound to the Exclusive CPU cores allocated to them. * Shared resource groups run in shared thread pools and are bound to the remaining Shared CPU cores. The threads in these three pools follow the naming convention `{ pip_exec | pip_scan | pip_con_scan }_{ com | }`, where `com` refers to the shared thread pool, and `` refers to the ID of the Exclusive resource group. You can view the CPU information bound to each BE thread through the system-defined view `information_schema.be_threads`. The fields `BE_ID`, `NAME`, and `BOUND_CPUS` represent the BE's ID, the name of the thread, and the number of CPU cores bound to that thread, respectively. ```sql select * from information_schema.be_threads where name like '%pip_exec%'; select * from information_schema.be_threads where name like '%pip_scan%'; select * from information_schema.be_threads where name like '%pip_con_scan%'; ``` Example: ```sql select BE_ID, NAME, FINISHED_TASKS, BOUND_CPUS from information_schema.be_threads where name like '%pip_exec_com%' and be_id = 10223; +-------+--------------+----------------+------------+ | BE_ID | NAME | FINISHED_TASKS | BOUND_CPUS | +-------+--------------+----------------+------------+ | 10223 | pip_exec_com | 2091295 | 10 | | 10223 | pip_exec_com | 2088025 | 10 | | 10223 | pip_exec_com | 1637603 | 6 | | 10223 | pip_exec_com | 1641260 | 6 | | 10223 | pip_exec_com | 1634197 | 6 | | 10223 | pip_exec_com | 1633804 | 6 | | 10223 | pip_exec_com | 1638184 | 6 | | 10223 | pip_exec_com | 1636374 | 6 | | 10223 | pip_exec_com | 2095951 | 10 | | 10223 | pip_exec_com | 2095248 | 10 | | 10223 | pip_exec_com | 2098745 | 10 | | 10223 | pip_exec_com | 2085338 | 10 | | 10223 | pip_exec_com | 2101221 | 10 | | 10223 | pip_exec_com | 2093901 | 10 | | 10223 | pip_exec_com | 2092364 | 10 | | 10223 | pip_exec_com | 2091366 | 10 | +-------+--------------+----------------+------------+ ``` --- ### Spill to disk Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) This topic describes how to spill intermediate computation results of large operators to local disks and object storage. #### Overview[​](#overview "Direct link to Overview") For database systems that rely on in-memory computing for query execution, like StarRocks, they can consume substantial memory resources when processing queries with aggregate, sort, and join operators on a big dataset. When memory limits are reached, these queries are forcibly terminated due to out-of-memory (OOM). However, there are still chances that you want certain memory-intensive tasks to be completed stably and performance is not your top priority, for example, building a materialized view, or performing a lightweight ETL with INSERT INTO SELECT. These tasks can easily exhaust your memory resources and thereby block other queries running in your cluster. Usually, to address this issue, you can only fine-tune these tasks individually, and rely on your resource isolation strategy to control the query concurrency. This could be particularly inconvenient and likely to fail under some extreme scenarios. From StarRocks v3.0.1, StarRocks supports spilling the intermediate results of some memory-intensive operators to disks. With this feature, you can trade a tolerable drop in performance for a significant reduction in memory usage, thereby improving system availability. Currently, StarRocks' spilling feature supports the following operators: * Aggregate operators * Sort operators * Hash join (LEFT JOIN, RIGHT JOIN, FULL JOIN, OUTER JOIN, SEMI JOIN, and INNER JOIN) operators * CTE operators (Supported from v3.3.4 onwards) #### Enable intermediate result spilling[​](#enable-intermediate-result-spilling "Direct link to Enable intermediate result spilling") Follow these steps to enable intermediate result spilling: 1. Specify the local spill directory `spill_local_storage_dir`, which stores the spilled intermediate result on the local disk, in the BE configuration file **be.conf** or the CN configuration file **cn.conf**, and restart the cluster to allow the modification to take effect. ```properties spill_local_storage_dir=/[;/] ``` > **NOTE** > > * You can specify multiple `spill_local_storage_dir` by separating them with semicolons (`;`). > * In a production environment, we strongly recommend you use different disks for data storage and spilling. When intermediate results are spilled to disk, there could be a significant increase in both writing load and disk usage. If the same disk is used, this surge can impact other queries or tasks running in the cluster. 2. Execute the following statement to enable intermediate result spilling: ```sql SET enable_spill = true; ``` 3. Configure the mode of intermediate result spilling using the session variable `spill_mode`: ```sql SET spill_mode = { "auto" | "force" }; ``` > **NOTE** > > Each time a query with spilling completes, StarRocks automatically clears the spilled data the query produces. If BE crashes before clearing the data, StarRocks clears it when the BE is restarted. | **Variable** | **Default** | **Description** | | ------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_spill | false | Whether to enable intermediate result spilling. If it is set to `true`, StarRocks spills the intermediate results to disk to reduce the memory usage when processing aggregate, sort, or join operators in queries. | | spill\_mode | auto | The execution mode of intermediate result spilling. Valid values:- `auto`: Spilling is automatically triggered when the memory usage threshold is reached.
- `force`: StarRocks forcibly executes spilling for all relevant operators, regardless of memory usage.This variable takes effect only when the variable `enable_spill` is set to `true`. | #### \[Preview] Spill intermediate result to object storage[​](#preview-spill-intermediate-result-to-object-storage "Direct link to [Preview] Spill intermediate result to object storage") From v3.3.0 onwards, StarRocks supports spilling intermediate results to object storage. tip Before enabling spilling to object storage, you must create a storage volume to define the object storage you want to use. For detailed instruction on creating a storage volume, see [CREATE STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md). After you have enabled spilling in the previous step, you can further set these system variables to allow the intermediate results to be spilled to object storage: ```sql SET enable_spill_to_remote_storage = true; -- Replace with the name of the storage volume which you want to use. SET spill_storage_volume = ''; ``` After spilling to object storage has been enabled, the intermediate results of queries that triggered spilling will be first stored in the local disks of the BE or CN nodes, and then the object storage if the capacity limit of the local disks is reached. Please note that, if the storage volume you specified for `spill_storage_volume` does not exist, spilling to object storage will not be enabled. #### Limitations[​](#limitations "Direct link to Limitations") * Not all OOM issues can be resolved by spilling. For example, StarRocks cannot release the memory used for expression evaluation. * Usually, queries with spilling involved indicate a tenfold increase in query latency. We recommend you extend the query timeout for these queries by setting the session variable `query_timeout`. * There is a significant performance drop in spilling to object storage compared to spilling to local disks. * `spill_local_storage_dir` of each BE or CN node is shared among all queries running on the node. Currently, StarRocks does not support setting a size limit of spilled data to local disks individually for each query. Therefore, concurrent queries involved spilling may impact one another. --- ### Scale in and out This topic describes how to scale in and out the node of StarRocks. #### Scale FE in and out[​](#scale-fe-in-and-out "Direct link to Scale FE in and out") StarRocks has two types of FE nodes: Follower and Observer. Followers are involved in election voting and writing. Observers are only used to synchronize logs and extend read performance. > * The number of follower FEs (including leader) must be odd, and it is recommended to deploy 3 of them to form a High Availability (HA) mode. > * When the FE is in high availability deployment (1 leader, 2 followers), it is recommended to add Observer FEs for better read performance. ##### Scale FE out[​](#scale-fe-out "Direct link to Scale FE out") After deploying the FE node and starting the service, run the following command to scale FE out. ```sql alter system add follower "fe_host:edit_log_port"; alter system add observer "fe_host:edit_log_port"; ``` ##### Scale FE in[​](#scale-fe-in "Direct link to Scale FE in") FE scale-in is similar to the scale-out. Run the following command to scale FE in. ```sql alter system drop follower "fe_host:edit_log_port"; alter system drop observer "fe_host:edit_log_port"; ``` After the expansion and contraction, you can view the node information by running `show proc '/frontends';`. #### Scale BE in and out[​](#scale-be-in-and-out "Direct link to Scale BE in and out") StarRocks will automatically perform load-balancing after BE's are scaled in or out without affecting the overall performance. When you add a new BE node, the system's Tablet Scheduler will detect the new node and its low load. It will then start moving tablets from high-load BE nodes to the new, low-load BE node to ensure an even distribution of data and load across the entire cluster. The balancing process is based on a loadScore calculated for each BE, which considers both disk utilization and replica count. The system aims to move tablets from nodes with a higher loadScore to nodes with a lower loadScore. You can check the FE configuration parameter `tablet_sched_disable_balance` to ensure that automatic balancing is not disabled (the parameter is false by default, which means that tablet balancing is enabled by default). More details are in the [manage replica docs](https://docs.starrocks.io/docs/administration/management/resource_management/Replica.md). ##### Scale BE out[​](#scale-be-out "Direct link to Scale BE out") Run the following command to scale BE out. ```sql alter system add backend 'be_host:be_heartbeat_service_port'; ``` Run the following command to check the BE status. ```sql show proc '/backends'; ``` ##### Scale BE in[​](#scale-be-in "Direct link to Scale BE in") There are two ways to scale in a BE node – `DROP` and `DECOMMISSION`. `DROP` will delete the BE node immediately, and the lost duplicates will be made up by FE scheduling. `DECOMMISSION` will make sure the duplicates are made up first, and then drop the BE node. `DECOMMISSION` is a bit more friendly and is recommended for BE scale-in. The commands of both methods are similar: * `alter system decommission backend "be_host:be_heartbeat_service_port";` * `alter system drop backend "be_host:be_heartbeat_service_port";` Drop backend is a dangerous operation, so you need to confirm it twice before executing it * `alter system drop backend "be_host:be_heartbeat_service_port";` #### Scale CN in and out[​](#scale-cn-in-and-out "Direct link to Scale CN in and out") ##### Scale CN out[​](#scale-cn-out "Direct link to Scale CN out") Run the following command to scale CN out. ```sql ALTER SYSTEM ADD COMPUTE NODE "cn_host:cn_heartbeat_service_port"; ``` Run the following command to check the CN status. ```sql SHOW PROC '/compute_nodes'; ``` ##### Scale CN in[​](#scale-cn-in "Direct link to Scale CN in") CN scale-in is similar to the scale-out. Run the following command to scale CN in. ```sql ALTER SYSTEM DROP COMPUTE NODE "cn_host:cn_heartbeat_service_port"; ``` You can view the node information by running `SHOW PROC '/compute_nodes';`. --- ### Configure a time zone This topic describes how to configure a time zone and the impacts of time zone settings. #### Configure a session-level time zone or a global time zone[​](#configure-a-session-level-time-zone-or-a-global-time-zone "Direct link to Configure a session-level time zone or a global time zone") You can configure a session-level time zone or a global time zone for your StarRocks cluster using the `time_zone` parameter. * To configure a session-level time zone, execute the command `SET time_zone = 'xxx';`. You can configure different time zones for different sessions. The time zone setting becomes invalid if you disconnect with FEs. * To configure a global time zone, execute the command `SET global time_zone = 'xxx';`. The time zone setting is persisted in FEs and is valid even if you disconnect with FEs. > **Note** > > Before you load data into StarRocks, modify the global time zone of your StarRocks cluster to the same value of the `system_time_zone` parameter. Otherwise, after data loading, data of the DATE type are incorrect. The `system_time_zone` parameter refers to the time zone of the machines that are used to host FEs. When the machines are started, the time zone of the machines is recorded as the value of this parameter. You cannot manually configure this parameter. ##### Time zone format[​](#time-zone-format "Direct link to Time zone format") The value of the `time_zone` parameter is not case-sensitive. The value of the parameter can be in one of the following formats. | **Format** | **Example** | | -------------- | ---------------------------------------------------------------------------------- | | UTC offset | `SET time_zone = '+10:00';` `SET global time_zone = '-6:00';` | | Time zone name | `SET time_zone = 'Asia/Shanghai';` `SET global time_zone = 'America/Los_Angeles';` | For more information about time zone format, see [List of tz database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). > **Note** > > Time zone abbreviations are not supported except for CST. If you set the value of `time_zone` to `CST`, StarRocks converts `CST` into `Asia/Shanghai`. ##### Default time zone[​](#default-time-zone "Direct link to Default time zone") The default value of the `time_zone` parameter is `Asia/Shanghai`. #### View time zone settings[​](#view-time-zone-settings "Direct link to View time zone settings") To view the time zone setting, run the following command. ```plaintext SHOW VARIABLES LIKE '%time_zone%'; ``` #### Impacts of time zone settings[​](#impacts-of-time-zone-settings "Direct link to Impacts of time zone settings") * Time zone settings affect the time values returned by the SHOW LOAD and SHOW BACKENDS statements. However, the settings do not affect the value specified in the `LESS THAN` clause when the partitioning columns specified in CREATE TABLE statement are of the DATE or DATETIME type. The settings also do not affect data of the DATE and DATETIME types. * Time zone settings affect the display and storage of the following functions: * **from\_unixtime**: returns a date and time of your specified time zone based on a specified UTC timestamp. For example, if the global time zone of your StarRocks cluster is `Asia/Shanghai`, `select FROM_UNIXTIME(0);` returns `1970-01-01 08:00:00`. * **unix\_timestamp**: returns a UTC timestamp based on the date and time of your specified time zone. For example, if the global time zone of your StarRocks cluster is `Asia/Shanghai`, `select UNIX_TIMESTAMP('1970-01-01 08:00:00');` returns `0`. * **curtime**: returns the current time of your specified time zone. For example, if the current time of a specified time zone is 16:34:05. `select CURTIME();` returns `16:34:05`. * **now**: returns the current date and time of your specified time zone. For example, if the current date and time of a specified time zone is 2021-02-11 16:34:13, `select NOW();` returns `2021-02-11 16:34:13`. * **convert\_tz**: converts the date and time from one time zone to another. For example, `select CONVERT_TZ('2021-08-01 11:11:11', 'Asia/Shanghai', 'America/Los_Angeles');` returns `2021-07-31 20:11:11`. --- ### Metadata Recovery This topic describes how to recover the metadata in your StarRocks clusters when FE nodes encounter different exceptions. Generally, you may have to resort to metadata recovery when and only when one of the following issues occurs: * [FE fails to restart](#fe-fails-to-restart) * [FE fails to provide services](#fe-fails-to-provide-services) * [Recover metadata on a new FE node with the metadata backup](#recover-metadata-on-a-new-fe-node-with-metadata-backup). Check the issue you encountered, follow the solution provided in the corresponding section, and perform any recommended actions. #### FE fails to restart[​](#fe-fails-to-restart "Direct link to FE fails to restart") FE nodes may fail to restart if the metadata is damaged or incompatible with the cluster after rollback. ##### Metadata incompatibility after rollback[​](#metadata-incompatibility-after-rollback "Direct link to Metadata incompatibility after rollback") When you downgrade your StarRocks cluster, FEs may fail to restart if the metadata is incompatible with that before the downgrade. You can identify this issue if you encounter the following exception while downgrading the cluster: ```plain UNKNOWN Operation Type xxx ``` You can follow these steps to recover the metadata and start FE: 1. Stop all FE nodes. 2. Back up the metadata directories `meta_dir` of all FE nodes. 3. Add the configuration `metadata_ignore_unknown_operation_type = true` to the configuration files **fe.conf** of all FE nodes. 4. Start all FE nodes, and check whether your data and metadata are intact. 5. If both your data and metadata are intact, execute the following statement to create an image file for your metadata: ```sql ALTER SYSTEM CREATE IMAGE; ``` 6. After the new image file is transmitted to the directory **meta/image** of all FE nodes, you can remove the configuration `metadata_ignore_unknown_operation_type= true` from all FE configuration files and restart the FE nodes. ##### Metadata damage[​](#metadata-damage "Direct link to Metadata damage") Both damage to BDBJE metadata and StarRocks metadata will cause failures to restart. ###### BDBJE metadata damage[​](#bdbje-metadata-damage "Direct link to BDBJE metadata damage") ###### VLSN Bug[​](#vlsn-bug "Direct link to VLSN Bug") You can identify the VLSN bug based on the following error message: ```plain recoveryTracker should overlap or follow on disk last VLSN of 6,684,650 recoveryFirst= 6,684,652 UNEXPECTED_STATE_FATAL: Unexpected internal state, unable to continue. Environment is invalid and must be closed. ``` You can follow these steps to fix this issue: 1. Clear the metadata directory `meta_dir` of the FE node that throws this exception. 2. Restart the FE node using the Leader FE node as the helper. ```bash # Replace with the IP address (priority_networks) # of the Leader FE node, and replace (Default: 9010) with # the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : --daemon ``` tip * This bug has been fixed in StarRocks v3.1. You can avoid this issue by upgrading your cluster to v3.1 or later. * This solution does not apply if more than half of the FE nodes have encountered this issue. You must follow the instructions provided in [Measure of the last resort](#7-measure-of-the-last-resort) to fix the issue. ###### RollbackException[​](#rollbackexception "Direct link to RollbackException") You can identify this issue based on the following error message: ```plain must rollback 1 total commits(1 of which were durable) to the earliest point indicated by transaction id=-14752149 time=2022-01-12 14:36:28.21 vlsn=28,069,415 lsn=0x1174/0x16e durable=false in order to rejoin the replication group. All existing ReplicatedEnvironment handles must be closed and reinstantiated. Log files were truncated to file 0x4467, offset 0x269, vlsn 28,069,413 HARD_RECOVERY: Rolled back past transaction commit or abort. Must run recovery by re-opening Environment handles Environment is invalid and must be closed. ``` This issue occurs when the Leader FE node writes BDBJE metadata but fails to synchronize it to Follower FE nodes before it hangs. After being restarted, the original Leader becomes a Follower, corrupting the metadata. To solve this issue, you only need to restart this node again to wipe out the dirty metadata. ###### ReplicaWriteException[​](#replicawriteexception "Direct link to ReplicaWriteException") You can identify this issue based on the keyword `removeReplicaDb` from the FE log **fe.log**. ```plain Caused by: com.sleepycat.je.rep.ReplicaWriteException: (JE 18.3.16) Problem closing transaction 25000090. The current state is:REPLICA. The node transitioned to this state at:Fri Feb 23 01:31:00 UTC 2024 Problem seen replaying entry NameLN_TX/14 vlsn=1,902,818,939 isReplicated="1" txn=-953505106 dbop=REMOVE Originally thrown by HA thread: REPLICA 10.233.132.23_9010_1684154162022(6) at com.sleepycat.je.rep.txn.ReadonlyTxn.disallowReplicaWrite(ReadonlyTxn.java:114) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.dbi.DbTree.checkReplicaWrite(DbTree.java:880) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.dbi.DbTree.doCreateDb(DbTree.java:579) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.dbi.DbTree.createInternalDb(DbTree.java:507) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.cleaner.ExtinctionScanner.openDb(ExtinctionScanner.java:357) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.cleaner.ExtinctionScanner.prepareForDbExtinction(ExtinctionScanner.java:1703) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.dbi.DbTree.doRemoveDb(DbTree.java:1208) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.dbi.DbTree.removeReplicaDb(DbTree.java:1261) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.Replay.applyNameLN(Replay.java:996) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.Replay.replayEntry(Replay.java:722) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.Replica$ReplayThread.run(Replica.java:1225) ~[starrocks-bdb-je-18.3.16.jar:?] ``` This issue occurs when the BDBJE version of the failed FE node (v18.3.\*) mismatches that of the Leader FE node (v7.3.7). You can follow these steps to fix this issue: 1. Drop the failed Follower or Observer node. ```sql -- To drop a Follower node, replace with the IP address (priority_networks) -- of the Follower node, and replace (Default: 9010) with -- the Follower node's edit_log_port. ALTER SYSTEM DROP FOLLOWER ":"; -- To drop an Observer node, replace with the IP address (priority_networks) -- of the Observer node, and replace (Default: 9010) with -- the Observer node's edit_log_port. ALTER SYSTEM DROP OBSERVER ":"; ``` 2. Add the failed node back to the cluster. ```sql -- Add the Follower node: ALTER SYSTEM ADD FOLLOWER ":"; -- Add the Observer node: ALTER SYSTEM ADD OBSERVER ":"; ``` 3. Clear the metadata directory `meta_dir` of the failed node. 4. Restart the failed node using the Leader FE node as the helper. ```bash # Replace with the IP address (priority_networks) # of the Leader FE node, and replace (Default: 9010) with # the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : --daemon ``` 5. After the failed node recovers to a healthy status, you need to upgrade the BDBJE packages in your cluster to **starrocks-bdb-je-18.3.16.jar** (or upgrade your StarRocks cluster to v3.0 or later), following the order of Followers first and then the Leader. ###### InsufficientLogException[​](#insufficientlogexception "Direct link to InsufficientLogException") You can identify this issue based on the following error message: ```plain xxx INSUFFICIENT_LOG: Log files at this node are obsolete. Environment is invalid and must be closed. ``` This issue occurs when the Follower node requires full metadata synchronization. It may occur when one of the following situations happens: * The metadata on the Follower node lags behind that of the Leader node, which has already done a CheckPoint of metadata within itself. The Follower node cannot perform incremental updates on its metadata, thus full metadata synchronization is required. * The original Leader node writes and checkpoints its metadata, but fails to synchronize it to Follower FE nodes before it hangs. After being restarted, it becomes a Follower node. With dirty metadata checkpointed, the Follower node cannot perform incremental deletion of its metadata, thus full metadata synchronization is required. Please note that this exception will be thrown when a new Follower node is added to the cluster. In this case, you do not need to take any action. If this exception is thrown for an existing Follower node or the Leader node, you only need to restart the node. ###### HANDSHAKE\_ERROR: Error during the handshake between two nodes[​](#handshake_error-error-during-the-handshake-between-two-nodes "Direct link to HANDSHAKE_ERROR: Error during the handshake between two nodes") You can identify this issue based on the following error message: ```plain 2023-11-13 21:51:55,271 WARN (replayer|82) [BDBJournalCursor.wrapDatabaseException():97] failed to get DB names for 1 times!Got EnvironmentFailureExce com.sleepycat.je.EnvironmentFailureException: (JE 18.3.16) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 10.26.5.115_9010_1697071897979(1):/data1/meta/bdb A replica with the name: 10.26.5.115_9010_1697071897979(1) is already active with the Feeder:null HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:230) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1835) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1844) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.Environment.checkOpen(Environment.java:2697) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.Environment.getDatabaseNames(Environment.java:2455) ~[starrocks-bdb-je-18.3.16.jar:?] at com.starrocks.journal.bdbje.BDBEnvironment.getDatabaseNamesWithPrefix(BDBEnvironment.java:478) ~[starrocks-fe.jar:?] at com.starrocks.journal.bdbje.BDBJournalCursor.refresh(BDBJournalCursor.java:177) ~[starrocks-fe.jar:?] at com.starrocks.server.GlobalStateMgr$5.runOneCycle(GlobalStateMgr.java:2148) ~[starrocks-fe.jar:?] at com.starrocks.common.util.Daemon.run(Daemon.java:115) ~[starrocks-fe.jar:?] at com.starrocks.server.GlobalStateMgr$5.run(GlobalStateMgr.java:2216) ~[starrocks-fe.jar:?] Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 10.26.5.115_9010_1697071897979(1):/data1/meta/bdb A replica with the name: 10.26.5.115_9010_1697071897979(1) is already active with the Feeder:null HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. Originally thrown by HA thread: UNKNOWN 10.26.5.115_9010_1697071897979(1) Originally thrown by HA thread: UNKNOWN 10.26.5.115_9010_1697071897979(1) at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.negotiateProtocol(ReplicaFeederHandshake.java:198) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.execute(ReplicaFeederHandshake.java:250) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.Replica.initReplicaLoop(Replica.java:709) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoopInternal(Replica.java:485) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoop(Replica.java:412) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1869) ~[starrocks-bdb-je-18.3.16.jar:?] ``` This issue occurs when the original Leader node hangs and becomes alive again while the surviving Follower nodes are trying to elect a new Leader node. The Follower nodes will try to establish a new connection with the original Leader node. However, the Leader node will reject the connection request because the old connection still exists. Once the request is rejected, the Follower node will set the environment as invalid and throw this exception. To solve this issue, you can either increase the JVM heap size or use the G1 GC algorithm. ###### DatabaseNotFoundException[​](#databasenotfoundexception "Direct link to DatabaseNotFoundException") You can identify this issue based on the following error message: ```plain 2024-01-05 12:47:21,087 INFO (main|1) [BDBEnvironment.ensureHelperInLocal():340] skip check local environment because helper node and local node are identical. 2024-01-05 12:47:21,339 ERROR (MASTER 172.17.0.1_9112_1704430041062(-1)|1) [StarRocksFE.start():186] StarRocksFE start failed com.sleepycat.je.DatabaseNotFoundException: (JE 18.3.16) _jeRepGroupDB at com.sleepycat.je.rep.impl.RepImpl.openGroupDb(RepImpl.java:1974) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.RepImpl.getGroupDb(RepImpl.java:1912) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.RepGroupDB.reinitFirstNode(RepGroupDB.java:1439) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.RepNode.reinitSelfElect(RepNode.java:1686) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.RepNode.startup(RepNode.java:874) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.RepNode.joinGroup(RepNode.java:2153) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.RepImpl.joinGroup(RepImpl.java:618) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.ReplicatedEnvironment.joinGroup(ReplicatedEnvironment.java:558) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:619) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:464) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:538) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.util.DbResetRepGroup.reset(DbResetRepGroup.java:262) ~[starrocks-bdb-je-18.3.16.jar:?] at com.starrocks.journal.bdbje.BDBEnvironment.initConfigs(BDBEnvironment.java:188) ~[starrocks-fe.jar:?] at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:174) ~[starrocks-fe.jar:?] at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:153) ~[starrocks-fe.jar:?] at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?] at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1201) ~[starrocks-fe.jar:?] at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1150) ~[starrocks-fe.jar:?] at com.starrocks.StarRocksFE.start(StarRocksFE.java:129) ~[starrocks-fe.jar:?] at com.starrocks.StarRocksFE.main(StarRocksFE.java:83) ~[starrocks-fe.jar:?] ``` This issue occurs when you add the configuration `metadata_failure_recovery = true` in the FE configuration file **fe.conf**. To solve this issue, you need to remove the configuration and restart the node. ###### StarRocks metadata damage[​](#starrocks-metadata-damage "Direct link to StarRocks metadata damage") You can identify the StarRocks metadata damage issue based on one of the following error messages: ```plain failed to load journal type xxx ``` Or ```plain catch exception when replaying ``` warning Before proceeding to recover the metadata by following the solution provided below, you are strongly advised to seek assistance from the technical experts in the StarRocks community, because this solution may lead to **data loss**. You can follow these steps to fix this issue: ###### Ignore Error Journal ID (Preferred)[​](#ignore-error-journal-id-preferred "Direct link to Ignore Error Journal ID (Preferred)") 1. Shut down all FE nodes. 2. Back up the metadata directories of all FE nodes. 3. Locate the erroneous journal ID in the logs. `xxx` in the following log represents the erroneous journal ID. ```plain got interrupt exception or inconsistent exception when replay journal xxx, will exit ``` 4. Add the following configuration to all **fe.conf** files and start the FE nodes. ```plain metadata_journal_skip_bad_journal_ids=xxx ``` 5. If the startup still again, identify the new failed journal ID through Step 3, add it to the **fe.conf**, and then restart the nodes with the previous configurations unchanged. ```plain metadata_journal_skip_bad_journal_ids=xxx,yyy ``` 6. If the system still fails to start after the above steps, or if there are too many failed journal IDs, proceed to Recovery Mode. ###### Recovery Mode[​](#recovery-mode "Direct link to Recovery Mode") 1. Stop all FE nodes. 2. Back up the metadata directories `meta_dir` of all FE nodes. 3. Add the configuration `metadata_enable_recovery_mode = true` to the configuration files **fe.conf** of all FE nodes. Note that data loading is forbidden in this mode. 4. Start all FE nodes, and query tables in the cluster to check whether your data is intact. You must wait until metadata recovery is completed if the following error is returned when you query these tables: ```plain ERROR 1064 (HY000): capture_consistent_versions error: version already been compacted. ``` You can execute the following statement from the Leader FE node to view the progress of metadata recovery: ```sql SHOW PROC '/meta_recovery'; ``` This statement will show the partitions that failed to be recovered. You can follow the advice returned to recover the partitions. If nothing is returned, it indicates the recovery is successful. 5. If both your data and metadata are intact, execute the following statement to create an image file for your metadata: ```sql ALTER SYSTEM CREATE IMAGE; ``` 6. After the new image file is transmitted to the directory **meta/image** of all FE nodes, you can remove the configuration `metadata_enable_recovery_mode = true` from all FE configuration files and restart the FE nodes. #### FE fails to provide services[​](#fe-fails-to-provide-services "Direct link to FE fails to provide services") FE will not provide services when Follower FE nodes fail to perform Leader election. When this issue occurs, you may find the following log record repeating: ```plain wait globalStateMgr to be ready. FE type: INIT. is ready: false ``` A variety of exceptions may cause this issue. You are strongly advised to troubleshoot the issue step by step following the sections below. Applying inappropriate solutions will deteriorate the problem and probably lead to data loss. ##### 1. The majority of Follower nodes are not running[​](#1-the-majority-of-follower-nodes-are-not-running "Direct link to 1. The majority of Follower nodes are not running") If the majority of the Follower nodes are not running, the FE group will not provide services. Here, 'majority' indicates `1 + (Follower node count/2)`. Please note that the Leader FE node itself is a Follower, but Observer nodes are not Followers. * You can identify the role of each FE node from the **fe/meta/image/ROLE** file: ```bash cat fe/meta/image/ROLE #Fri Jan 19 20:03:14 CST 2024 role=FOLLOWER hostType=IP name=172.26.92.154_9312_1705568349984 ``` * You can view the total number of Follower nodes from the BDBJE log: ```bash grep "Current group size" fe/meta/bdb/je.info.0 # The example output indicates there are three Follower nodes in the cluster. 2024-01-24 08:21:44.754 UTC INFO [172.26.92.139_29917_1698226672727] Current group size: 3 ``` To solve this issue, you need to start all Follower nodes in the cluster. If they cannot be restarted, please refer to [The measure of last resort](#10-measure-of-the-last-resort). ##### 2. Node IP is changed[​](#2-node-ip-is-changed "Direct link to 2. Node IP is changed") If the `priority_networks` of the node is not configured, the FE node will randomly select an available IP address once it is restarted. If the IP address recorded in the BDBJE metadata is different from that used to start the node, the FE will not provide services. * You can view the IP address recorded in the BDBJE metadata from the **fe/meta/image/ROLE** file: ```bash cat fe/meta/image/ROLE #Fri Jan 19 20:03:14 CST 2024 role=FOLLOWER hostType=IP name=172.26.92.154_9312_1705568349984 ``` The value `172.26.92.154` before the first underscore is the IP address recorded in the BDBJE metadata. * You can view the IP address used to start the node from the FE log: ```bash grep "IP:" fe/log/fe.log 2024-02-06 14:33:58,211 INFO (main|1) [FrontendOptions.initAddrUseIp():249] Use IP init local addr, IP: /172.17.0.1 2024-02-06 14:34:27,689 INFO (main|1) [FrontendOptions.initAddrUseIp():249] Use IP init local addr, IP: /172.17.0.1 ``` To solve this issue, you need to set the `priority_networks` of the node in the FE configuration file **fe.conf** to the IP address recorded in **fe/meta/image/ROLE**, and restart the node. ##### 3. System clock among nodes is not synchronized[​](#3-system-clock-among-nodes-is-not-synchronized "Direct link to 3. System clock among nodes is not synchronized") You can identify this issue based on the following error message from **fe.out**, **fe.log** or **fe/meta//bdb/je.info.0**: ```plain com.sleepycat.je.EnvironmentFailureException: (JE 7.3.7) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 7.3.7) 172.26.92.139_29917_1631006307557(2180):xxx Clock delta: 11020 ms. between Feeder: 172.26.92.154_29917_1641969377236 and this Replica exceeds max permissible delta: 5000 ms. HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. fetchRoot of 0x1278/0x1fcbb8 state=0 expires=never ``` You must synchronize the system clock among all nodes. ##### 4. Available disk space is insufficient[​](#4-available-disk-space-is-insufficient "Direct link to 4. Available disk space is insufficient") After you upgrade StarRocks to v3.0 or later, or upgrade BDBJE to v18 or later, the node may fail to restart when the available space of the disk that stores `meta_dir` is less than 5 GB. You can view the BDBJE version from the **.jar** package under the directory **fe/lib**. To solve this issue, you can scale up the disk, or allocate a dedicated disk with larger capacity for the FE metadata. ##### 5. `edit_log_port` is changed[​](#5-edit_log_port-is-changed "Direct link to 5-edit_log_port-is-changed") If the `edit_log_port` recorded in the BDBJE metadata is different from that configured in **fe.conf**, the FE will not provide services. You can view the `edit_log_port` recorded in the BDBJE metadata from the **fe/meta/image/ROLE** file: ```bash cat fe/meta/image/ROLE #Fri Jan 19 20:03:14 CST 2024 role=FOLLOWER hostType=IP name=172.26.92.154_9312_1705568349984 ``` The value `9312` before the second underscore is the `edit_log_port` recorded in the BDBJE metadata. To solve this issue, you need to set the `edit_log_port` of the node in the FE configuration file **fe.conf** to the `edit_log_port` recorded in **fe/meta/image/ROLE**, and restart the node. ##### 6. JVM heap size is insufficient[​](#6-jvm-heap-size-is-insufficient "Direct link to 6. JVM heap size is insufficient") You can view the JVM memory usage using the `jstat` command: ```plain jstat -gcutil pid 1000 1000 S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 0.00 100.00 27.78 95.45 97.77 94.45 24 0.226 1 0.065 0.291 0.00 100.00 44.44 95.45 97.77 94.45 24 0.226 1 0.065 0.291 0.00 100.00 55.56 95.45 97.77 94.45 24 0.226 1 0.065 0.291 0.00 100.00 72.22 95.45 97.77 94.45 24 0.226 1 0.065 0.291 0.00 100.00 88.89 95.45 97.77 94.45 24 0.226 1 0.065 0.291 0.00 100.00 5.26 98.88 97.80 94.45 25 0.231 1 0.065 0.297 0.00 100.00 21.05 98.88 97.80 94.45 25 0.231 1 0.065 0.297 0.00 100.00 31.58 98.88 97.80 94.45 25 0.231 1 0.065 0.297 0.00 100.00 47.37 98.88 97.80 94.45 25 0.231 1 0.065 0.297 0.00 100.00 63.16 98.88 97.80 94.45 25 0.231 1 0.065 0.297 0.00 100.00 73.68 98.88 97.80 94.45 25 0.231 1 0.065 0.297 ``` If the percentages shown in field `O` remain high, it indicates that the JVM heap size is insufficient. To solve this issue, you must increase the JVM heap size. ##### 7. Latch timeout. com.sleepycat.je.log.LogbufferPool\_FullLatch[​](#7-latch-timeout-comsleepycatjeloglogbufferpool_fulllatch "Direct link to 7. Latch timeout. com.sleepycat.je.log.LogbufferPool_FullLatch") You can identify this issue based on the following error message: ```plain Environment invalid because of previous exception: xxx Latch timeout. com.sleepycat.je.log.LogbufferPool_FullLatch xxx' at com.sleepycat.je.EnvironmentFailureException.unexpectedState(EnvironmentFailureException.java:459) at com.sleepycat.je.latch.LatchSupport.handleTimeout(LatchSupport.java:211) at com.sleepycat.je.latch.LatchWithStatsImpl.acquireExclusive(LatchWithStatsImpl.java:87) at com.sleepycat.je.log.LogBufferPool.bumpCurrent(LogBufferPool.java:527) at com.sleepycat.je.log.LogManager.flushInternal(LogManager.java:1373) at com.sleepycat.je.log.LogManager.flushNoSync(LogManager.java:1337) at com.sleepycat.je.log.LogFlusher$FlushTask.run(LogFlusher.java:232) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) ``` This issue occurs when there is excessive pressure on the local disk of the FE node. To solve this issue, you can allocate a dedicated disk for the FE metadata, or replace the disk with a high-performance one. ##### 8. InsufficientReplicasException[​](#8-insufficientreplicasexception "Direct link to 8. InsufficientReplicasException") You can identify this issue based on the following error message: ```plain com.sleepycat.je.rep.InsufficientReplicasException: (JE 7.3.7) Commit policy: SIMPLE_MAJORITY required 1 replica. But none were active with this master. ``` This issue occurs when the Leader FE node or Follower FE nodes use excessive memory resources, leading to Full GC. To solve this issue, you can either increase the JVM heap size or use the G1 GC algorithm. ##### 9. UnknownMasterException[​](#9-unknownmasterexception "Direct link to 9. UnknownMasterException") You can identify this issue based on the following error message: ```plain com.sleepycat.je.rep.UnknownMasterException: (JE 18.3.16) Could not determine master from helpers at:[/xxx.xxx.xxx.xxx:9010, /xxx.xxx.xxx.xxx:9010] at com.sleepycat.je.rep.elections.Learner.findMaster(Learner.java:443) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.util.ReplicationGroupAdmin.getMasterSocket(ReplicationGroupAdmin.java:186) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.util.ReplicationGroupAdmin.doMessageExchange(ReplicationGroupAdmin.java:607) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.util.ReplicationGroupAdmin.getGroup(ReplicationGroupAdmin.java:406) ~[starrocks-bdb-je-18.3.16.jar:?] at com.starrocks.ha.BDBHA.getElectableNodes(BDBHA.java:178) ~[starrocks-fe.jar:?] at com.starrocks.common.proc.FrontendsProcNode.getFrontendsInfo(FrontendsProcNode.java:96) ~[starrocks-fe.jar:?] at com.starrocks.common.proc.FrontendsProcNode.fetchResult(FrontendsProcNode.java:80) ~[starrocks-fe.jar:?] at com.starrocks.sql.ast.ShowProcStmt.getMetaData(ShowProcStmt.java:74) ~[starrocks-fe.jar:?] at com.starrocks.qe.ShowExecutor.handleShowProc(ShowExecutor.java:872) ~[starrocks-fe.jar:?] at com.starrocks.qe.ShowExecutor.execute(ShowExecutor.java:286) ~[starrocks-fe.jar:?] at com.starrocks.qe.StmtExecutor.handleShow(StmtExecutor.java:1574) ~[starrocks-fe.jar:?] at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:688) ~[starrocks-fe.jar:?] at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:336) ~[starrocks-fe.jar:?] at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:530) ~[starrocks-fe.jar:?] at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:838) ~[starrocks-fe.jar:?] at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] ``` When executing `SHOW FRONTENDS`, if the Leader FE node cannot be found, there could be several reasons: * If it is observed that more than half of the FE nodes are undergoing Full GC, and the duration is notably long. * Or if the log contains the keyword `java.lang.OutOfMemoryError: Java heap space`. It is because there is insufficient memory. You need to increase the JVM memory allocation. ##### 10. Measure of the last resort[​](#10-measure-of-the-last-resort "Direct link to 10. Measure of the last resort") warning You can try the following solution as the last resort only when none of the preceding solutions works. This solution is designed only for extreme cases, such as: * The majority of Follower nodes cannot be restarted. * Follower nodes cannot perform Leader election due to bugs of BDBJE. * Exceptions other than those mentioned in preceding sections. Follow these steps to recover the metadata: 1. Stop all FE nodes. 2. Back up the metadata directories `meta_dir` of all FE nodes. 3. Run the following commands on **all the servers that host the FE nodes** to identify the node with the latest metadata. ```bash # You need to specify the exact .jar package the node uses in the command # as the package varies according to the StarRocks version. java -jar fe/lib/starrocks-bdb-je-18.3.16.jar DbPrintLog -h meta/bdb/ -vd ``` Example output: ```bash file 0x3b numRepRecords = 24479 firstVLSN = 1,434,126 lastVLSN = 1,458,604 file 0x3c numRepRecords = 22541 firstVLSN = 1,458,605 lastVLSN = 1,481,145 file 0x3d numRepRecords = 25176 firstVLSN = 1,481,146 lastVLSN = 1,506,321 ...... file 0x74 numRepRecords = 26903 firstVLSN = 2,927,458 lastVLSN = 2,954,360 file 0x75 numRepRecords = 26496 firstVLSN = 2,954,361 lastVLSN = 2,980,856 file 0x76 numRepRecords = 18727 firstVLSN = 2,980,857 lastVLSN = 2,999,583 ... 0 files at end First file: 0x3b Last file: 0x76 ``` The node with the largest `lastVLSN` value has the latest metadata. 4. Find the role (Follower or Observer) of the FE node that has the latest metadata from the **fe/meta/image/ROLE** file: ```bash cat fe/meta/image/ROLE #Fri Jan 19 20:03:14 CST 2024 role=FOLLOWER hostType=IP name=172.26.92.154_9312_1705568349984 ``` If multiple nodes have the latest metadata, it is recommended to proceed with a Follower node. If multiple Follower nodes have the latest metadata, you can proceed with any of them. 5. Perform the corresponding operations based on the role of the FE node you chose in the previous step. * Proceed with Follower Node * Proceed with Observer Node If a Follower node has the latest metadata, perform the following operations: 1. Add the following configurations to **fe.conf**: ```properties bdbje_reset_election_group = true ``` 2. Restart the node, and check whether your data and metadata are intact. 3. Check whether the current FE node is the Leader FE node. ```sql SHOW FRONTENDS; ``` * If the field `Alive` is `true`, this FE node is properly started and added to the cluster. * If the field `Role` is `LEADER`, this FE node is the Leader FE node. 4. If the data and metadata are intact, and the role of the node is Leader, you can remove the configuration you added earlier and restart the node. If an Observer node has the latest metadata, perform the following operations: 1. Change the role of the FE node from `OBSERVER` to `FOLLOWER` in the **fe/meta/image/ROLE** file. 2. Add the following configurations to **fe.conf**: ```properties bdbje_reset_election_group = true ``` 3. Restart the node, and check whether your data and metadata are intact. 4. Check whether the current FE node is the Leader FE node. ```sql SHOW FRONTENDS; ``` * If the field `Alive` is `true`, this FE node is properly started and added to the cluster. * If the field `Role` is `LEADER`, this FE node is the Leader FE node. 5. If the data and metadata are intact, and the role of the node is Leader, you can remove the configuration you added earlier. However, **do not restart the node**. 6. Add a new Follower node (on a new server) to the cluster. ```sql ALTER SYSTEM ADD FOLLOWER ":"; ``` 7. Start a new FE node on the new server using the temporary Leader FE node as the helper. ```bash # Replace with the IP address (priority_networks) # of the Leader FE node, and replace (Default: 9010) with # the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : --daemon ``` 8. Once the new FE node is started successfully, check the status and roles of both FE nodes: ```sql SHOW FRONTENDS; ``` * If the field `Alive` is `true`, this FE node is properly started and added to the cluster. * If the field `Role` is `FOLLOWER`, this FE node is a Follower FE node. * If the field `Role` is `LEADER`, this FE node is the Leader FE node. 9. If the new Follower is successfully running in the cluster, you can then stop all nodes. 10. Add the following configurations to the **fe.conf of the new Follower only**: * For StarRocks v2.5, v3.0, v3.1.9 and earlier patch versions, and v3.2.4 and earlier patch versions: ```properties metadata_failure_recovery = true ``` * For StarRocks v3.1.10 and later patch versions, v3.2.5 and later patch versions, and v3.3 and later: ```properties bdbje_reset_election_group = true ``` 11. Restart the new Follower node, and check whether your data and metadata are intact. 12. Check whether the current FE node is the Leader FE node. ```sql SHOW FRONTENDS; ``` * If the field `Alive` is `true`, this FE node is properly started and added to the cluster. * If the field `Role` is `LEADER`, this FE node is the Leader FE node. 13. If the data and metadata are intact, and the role of the node is Leader, you can remove the configuration you added earlier and restart the node. 5) Clear the metadata directories `meta_dir` of the FE nodes that you want to add back to the cluster. 6) Start new Follower nodes using the new Leader FE node as the helper. ```bash # Replace with the IP address (priority_networks) # of the Leader FE node, and replace (Default: 9010) with # the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : --daemon ``` 7) Add the Follower nodes back to the cluster. ```sql ALTER SYSTEM ADD FOLLOWER ":"; ``` After all nodes are added back to the cluster, the metadata is successfully recovered. #### Recover metadata on a new FE node with metadata backup[​](#recover-metadata-on-a-new-fe-node-with-metadata-backup "Direct link to Recover metadata on a new FE node with metadata backup") Follow these steps if you want to start a new FE node with the metadata backup: 1. Copy the backup metadata directory `meta_dir` to the new FE node. 2. In the configuration file of the FE node, set `bdbje_reset_election_group` to `true`. ```properties bdbje_reset_election_group = true ``` 3. Start the FE node. ```bash ./fe/bin/start_fe.sh ``` 4. Check whether the current FE node is the Leader FE node. ```sql SHOW FRONTENDS; ``` If the field `Role` is `LEADER`, this FE node is the Leader FE node. Make sure its IP address is the that of the current FE node. 5. If the data and metadata are intact, and the role of the node is Leader, you must remove the configuration `bdbje_reset_election_group` and restart the node. 6. Now you have successfully start a new Leader FE node with the metadata backup. You can add new Follower nodes using the new Leader FE node as the helper. ```bash # Replace with the IP address (priority_networks) # of the Leader FE node, and replace (Default: 9010) with # the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : --daemon ``` #### Metadata recovery-related configurations[​](#metadata-recovery-related-configurations "Direct link to Metadata recovery-related configurations") tip You must remove the following configuration once the metadata recovery is completed. * [bdbje\_reset\_election\_group](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#bdbje_reset_election_group) * [metadata\_enable\_recovery\_mode](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#metadata_enable_recovery_mode) * [metadata\_ignore\_unknown\_operation\_type](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#metadata_ignore_unknown_operation_type) --- ### SQL Digest This topic introduces the SQL Digest feature of StarRocks. This feature is supported from v3.3.6 onwards. #### Overview[​](#overview "Direct link to Overview") SQL Digest is a fingerprint generated by historical SQL statements with parameters removed. It helps cluster SQL statements with the same structure but different parameters. Common use cases of SQL Digest include: * Finding other SQL statements with the same structure but different parameters in query history * Tracking execution frequency, cumulative time, and other statistics of SQL with the same structure * Identifying the most time-consuming SQL patterns in the system In StarRocks, SQL Digests are mainly recorded through audit logs **fe.audit.log**. For example, execute the following two SQL statements: ```sql SELECT count(*) FROM lineorder WHERE lo_orderdate > '19920101'; SELECT count(*) FROM lineorder WHERE lo_orderdate > '19920202'; ``` Two same Digest will be generated in **fe.audit.log**: ```sql Digest=f58bb71850d112014f773717830e7f77 Digest=f58bb71850d112014f773717830e7f77 ``` #### Usage[​](#usage "Direct link to Usage") ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") To enable this feature, you must set the FE configuration item `enable_sql_digest` to `true`. Execute the following statement to enable it dynamically: ```sql ADMIN SET FRONTEND CONFIG ('enable_sql_digest'='true'); ``` To enable it permanently, you must add `enable_sql_digest = true` to the FE configuration file `fe.conf` and restart FE. After enabling this feature, you can install the [AuditLoader](https://docs.starrocks.io/docs/administration/management/audit_loader.md) plugin to perform statistical analysis on SQL statements. ##### Find similar SQL[​](#find-similar-sql "Direct link to Find similar SQL") ```sql SELECT * FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE digest = '' LIMIT 1; ``` ##### Track daily execution count and time of similar SQL[​](#track-daily-execution-count-and-time-of-similar-sql "Direct link to Track daily execution count and time of similar SQL") ```sql SELECT date_trunc('day', `timestamp`) query_date, count(*), sum(queryTime), sum(scanRows), sum(cpuCostNs), sum(memCostBytes) FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE digest = '' GROUP BY query_date ORDER BY query_date DESC LIMIT 30; ``` ##### Calculate average execution time of similar SQL[​](#calculate-average-execution-time-of-similar-sql "Direct link to Calculate average execution time of similar SQL") ```sql SELECT avg(queryTime), min(queryTime), max(queryTime), stddev(queryTime) FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE digest = ''; ``` ##### Aggregate similar SQL to analyze the most time-consuming pattern[​](#aggregate-similar-sql-to-analyze-the-most-time-consuming-pattern "Direct link to Aggregate similar SQL to analyze the most time-consuming pattern") ```sql WITH top_sql AS ( SELECT digest, sum(queryTime) FROM starrocks_audit_db__.starrocks_audit_tbl__ GROUP BY digest ORDER BY sum(queryTime) DESC LIMIT 10 ) SELECT * FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE digest IN (SELECT digest FROM top_sql); ``` #### Parameter normalization rules[​](#parameter-normalization-rules "Direct link to Parameter normalization rules") * Constant values in SQL will be normalized. For example, similar SQL statements with `WHERE a = 1` and `WHERE a = 2` will have the same Digest. * IN predicates will be normalized. For example, similar SQL statements with `IN (1,2,3)` and `IN (1,2)` will have the same Digest. * `LIMIT N` clauses will be normalized. For example, similar SQL statements with `LIMIT 10` and `LIMIT 30` will have the same Digest. --- ### Deploy and Manage StarRocks with StarGo This topic describes how to deploy and manage StarRocks clusters with StarGo. StarGo is a command line tool for multiple StarRocks clusters management. You can easily deploy, check, upgrade, downgrade, start, and stop multiple clusters through StarGo. #### Install StarGo[​](#install-stargo "Direct link to Install StarGo") Download the following files to your central control node: * **sr-ctl**: The binary file of StarGo. You do not need to install it after downloading it. * **sr-c1.yaml**: The template for the deployment configuration file. * **repo.yaml**: The configuration file for the download path of StarRocks installer. > Note You can access `http://cdn-thirdparty.starrocks.com` to get the corresponding installation index files and installers. ```shell wget https://github.com/wangtianyi2004/starrocks-controller/raw/main/stargo-pkg.tar.gz wget https://github.com/wangtianyi2004/starrocks-controller/blob/main/sr-c1.yaml wget https://github.com/wangtianyi2004/starrocks-controller/blob/main/repo.yaml ``` Grant **sr-ctl** access. ```shell chmod 751 sr-ctl ``` #### Deploy StarRocks cluster[​](#deploy-starrocks-cluster "Direct link to Deploy StarRocks cluster") You can deploy a StarRocks cluster with StarGo. ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * The cluster to be deployed must have at least one central control node and three deployment nodes. All nodes can be deployed on one machine. * You need to deploy StarGo on the central control node. * You need to build mutual SSH authentication between the central control node and three deployment nodes. The following example builds mutual authentication between central control node sr-dev@r0 and three deployment nodes starrocks@r1, starrocks@r2, and starrocks@r3. ```plain ## Build the mutual authentication between sr-dev@r0 and starrocks@r1, 2, 3. [sr-dev@r0 ~]$ ssh-keygen [sr-dev@r0 ~]$ ssh-copy-id starrocks@r1 [sr-dev@r0 ~]$ ssh-copy-id starrocks@r2 [sr-dev@r0 ~]$ ssh-copy-id starrocks@r3 ## Verify the mutual authentication between sr-dev@r0 and starrocks@r1, 2, 3. [sr-dev@r0 ~]$ ssh starrocks@r1 date [sr-dev@r0 ~]$ ssh starrocks@r2 date [sr-dev@r0 ~]$ ssh starrocks@r3 date ``` ##### Create configuration file[​](#create-configuration-file "Direct link to Create configuration file") Create the StarRocks deployment topology file based on the following YAML template. See [Configuration](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md) for detailed information. ```yaml global: user: "starrocks" ## The current OS user. ssh_port: 22 fe_servers: - host: 192.168.XX.XX ssh_port: 22 http_port: 8030 rpc_port: 9020 query_port: 9030 edit_log_port: 9010 deploy_dir: StarRocks/fe meta_dir: StarRocks/fe/meta log_dir: StarRocks/fe/log priority_networks: 192.168.XX.XX/24 # Specify the unique IP for current node when the machine has multiple IP addresses. config: sys_log_level: "INFO" - host: 192.168.XX.XX ssh_port: 22 http_port: 8030 rpc_port: 9020 query_port: 9030 edit_log_port: 9010 deploy_dir: StarRocks/fe meta_dir: StarRocks/fe/meta log_dir: StarRocks/fe/log priority_networks: 192.168.XX.XX/24 # Specify the unique IP for current node when the machine has multiple IP addresses. config: sys_log_level: "INFO" - host: 192.168.XX.XX ssh_port: 22 http_port: 8030 rpc_port: 9020 query_port: 9030 edit_log_port: 9010 deploy_dir: StarRocks/fe meta_dir: StarRocks/fe/meta log_dir: StarRocks/fe/log priority_networks: 192.168.XX.XX/24 # Specify the unique IP for current node when the machine has multiple IP addresses. config: sys_log_level: "INFO" be_servers: - host: 192.168.XX.XX ssh_port: 22 be_port: 9060 be_http_port: 8040 heartbeat_service_port: 9050 brpc_port: 8060 deploy_dir : StarRocks/be storage_dir: StarRocks/be/storage log_dir: StarRocks/be/log priority_networks: 192.168.XX.XX/24 # Specify the unique IP for current node when the machine has multiple IP addresses. config: create_tablet_worker_count: 3 - host: 192.168.XX.XX ssh_port: 22 be_port: 9060 be_http_port: 8040 heartbeat_service_port: 9050 brpc_port: 8060 deploy_dir : StarRocks/be storage_dir: StarRocks/be/storage log_dir: StarRocks/be/log priority_networks: 192.168.XX.XX/24 # Specify the unique IP for current node when the machine has multiple IP addresses. config: create_tablet_worker_count: 3 - host: 192.168.XX.XX ssh_port: 22 be_port: 9060 be_http_port: 8040 heartbeat_service_port: 9050 brpc_port: 8060 deploy_dir : StarRocks/be storage_dir: StarRocks/be/storage log_dir: StarRocks/be/log priority_networks: 192.168.XX.XX/24 # Specify the unique IP for current node when the machine has multiple IP addresses. config: create_tablet_worker_count: 3 ``` ##### Create deployment directory (Optional)[​](#create-deployment-directory-optional "Direct link to Create deployment directory (Optional)") If the paths under which StarRocks to be deployed does not exist, and you have the privilege to create such paths, you do not have create these paths, and StarGo will create them for you based on the configuration file. If the the paths already exist, make sure you have the write access to them. You can also create necessary deployment directories on each node by running the following commands. * Create **meta** directory on FE nodes. ```shell mkdir -p StarRocks/fe/meta ``` * Create **storage** directory on BE nodes. ```shell mkdir -p StarRocks/be/storage ``` > Caution Make sure the above paths are identical with the configuration items `meta_dir` and `storage_dir` in the configuration file. ##### Deploy StarRocks[​](#deploy-starrocks "Direct link to Deploy StarRocks") Deploy StarRocks cluster by running the following command. ```shell ./sr-ctl cluster deploy ``` | Parameter | Description | | -------------- | ------------------------------- | | cluster\_name | Name of the cluster to deploy. | | version | StarRocks version. | | topology\_file | Name of the configuration file. | If the deployment is successful, the cluster will be started automatically. When beStatus and feStatus are true, the cluster is started successfully. Example: ```plain [sr-dev@r0 ~]$ ./sr-ctl cluster deploy sr-c1 v2.0.1 sr-c1.yaml [20220301-234817 OUTPUT] Deploy cluster [clusterName = sr-c1, clusterVersion = v2.0.1, metaFile = sr-c1.yaml] [20220301-234836 OUTPUT] PRE CHECK DEPLOY ENV: PreCheck FE: IP ssh auth meta dir deploy dir http port rpc port query port edit log port -------------------- --------------- ------------------------- ------------------------- --------------- --------------- --------------- --------------- 192.168.xx.xx PASS PASS PASS PASS PASS PASS PASS 192.168.xx.xx PASS PASS PASS PASS PASS PASS PASS 192.168.xx.xx PASS PASS PASS PASS PASS PASS PASS PreCheck BE: IP ssh auth storage dir deploy dir webSer port heartbeat port brpc port be port -------------------- --------------- ------------------------- ------------------------- --------------- --------------- --------------- --------------- 192.168.xx.xx PASS PASS PASS PASS PASS PASS PASS 192.168.xx.xx PASS PASS PASS PASS PASS PASS PASS 192.168.xx.xx PASS PASS PASS PASS PASS PASS PASS [20220301-234836 OUTPUT] PreCheck successfully. RESPECT [20220301-234836 OUTPUT] Create the deploy folder ... [20220301-234838 OUTPUT] Download StarRocks package & jdk ... [20220302-000515 INFO] The file starrocks-2.0.1-quickstart.tar.gz [1227406189] download successfully [20220302-000515 OUTPUT] Download done. [20220302-000515 OUTPUT] Decompress StarRocks pakcage & jdk ... [20220302-000520 INFO] The tar file /home/sr-dev/.starrocks-controller/download/starrocks-2.0.1-quickstart.tar.gz has been decompressed under /home/sr-dev/.starrocks-controller/download [20220302-000547 INFO] The tar file /home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1.tar.gz has been decompressed under /home/sr-dev/.starrocks-controller/download [20220302-000556 INFO] The tar file /home/sr-dev/.starrocks-controller/download/jdk-8u301-linux-x64.tar.gz has been decompressed under /home/sr-dev/.starrocks-controller/download [20220302-000556 OUTPUT] Distribute FE Dir ... [20220302-000603 INFO] Upload dir feSourceDir = [/home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1/fe] to feTargetDir = [StarRocks/fe] on FeHost = [192.168.xx.xx] [20220302-000615 INFO] Upload dir JDKSourceDir = [/home/sr-dev/.starrocks-controller/download/jdk1.8.0_301] to JDKTargetDir = [StarRocks/fe/jdk] on FeHost = [192.168.xx.xx] [20220302-000615 INFO] Modify JAVA_HOME: host = [192.168.xx.xx], filePath = [StarRocks/fe/bin/start_fe.sh] [20220302-000622 INFO] Upload dir feSourceDir = [/home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1/fe] to feTargetDir = [StarRocks/fe] on FeHost = [192.168.xx.xx] [20220302-000634 INFO] Upload dir JDKSourceDir = [/home/sr-dev/.starrocks-controller/download/jdk1.8.0_301] to JDKTargetDir = [StarRocks/fe/jdk] on FeHost = [192.168.xx.xx] [20220302-000634 INFO] Modify JAVA_HOME: host = [192.168.xx.xx], filePath = [StarRocks/fe/bin/start_fe.sh] [20220302-000640 INFO] Upload dir feSourceDir = [/home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1/fe] to feTargetDir = [StarRocks/fe] on FeHost = [192.168.xx.xx] [20220302-000652 INFO] Upload dir JDKSourceDir = [/home/sr-dev/.starrocks-controller/download/jdk1.8.0_301] to JDKTargetDir = [StarRocks/fe/jdk] on FeHost = [192.168.xx.xx] [20220302-000652 INFO] Modify JAVA_HOME: host = [192.168.xx.xx], filePath = [StarRocks/fe/bin/start_fe.sh] [20220302-000652 OUTPUT] Distribute BE Dir ... [20220302-000728 INFO] Upload dir BeSourceDir = [/home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1/be] to BeTargetDir = [StarRocks/be] on BeHost = [192.168.xx.xx] [20220302-000752 INFO] Upload dir BeSourceDir = [/home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1/be] to BeTargetDir = [StarRocks/be] on BeHost = [192.168.xx.xx] [20220302-000815 INFO] Upload dir BeSourceDir = [/home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1/be] to BeTargetDir = [StarRocks/be] on BeHost = [192.168.xx.xx] [20220302-000815 OUTPUT] Modify configuration for FE nodes & BE nodes ... ############################################# START FE CLUSTER ############################################# ############################################# START FE CLUSTER ############################################# [20220302-000816 INFO] Starting leader FE node [host = 192.168.xx.xx, editLogPort = 9010] [20220302-000836 INFO] The FE node start succefully [host = 192.168.xx.xx, queryPort = 9030] [20220302-000836 INFO] Starting follower FE node [host = 192.168.xx.xx, editLogPort = 9010] [20220302-000857 INFO] The FE node start succefully [host = 192.168.xx.xx, queryPort = 9030] [20220302-000857 INFO] Starting follower FE node [host = 192.168.xx.xx, editLogPort = 9010] [20220302-000918 INFO] The FE node start succefully [host = 192.168.xx.xx, queryPort = 9030] [20220302-000918 INFO] List all FE status: feHost = 192.168.xx.xx feQueryPort = 9030 feStatus = true feHost = 192.168.xx.xx feQueryPort = 9030 feStatus = true feHost = 192.168.xx.xx feQueryPort = 9030 feStatus = true ############################################# START BE CLUSTER ############################################# ############################################# START BE CLUSTER ############################################# [20220302-000918 INFO] Starting BE node [BeHost = 192.168.xx.xx HeartbeatServicePort = 9050] [20220302-000939 INFO] The BE node start succefully [host = 192.168.xx.xx, heartbeatServicePort = 9050] [20220302-000939 INFO] Starting BE node [BeHost = 192.168.xx.xx HeartbeatServicePort = 9050] [20220302-001000 INFO] The BE node start succefully [host = 192.168.xx.xx, heartbeatServicePort = 9050] [20220302-001000 INFO] Starting BE node [BeHost = 192.168.xx.xx HeartbeatServicePort = 9050] [20220302-001020 INFO] The BE node start succefully [host = 192.168.xx.xx, heartbeatServicePort = 9050] [20220302-001020 OUTPUT] List all BE status: beHost = 192.168.xx.xx beHeartbeatServicePort = 9050 beStatus = true beHost = 192.168.xx.xx beHeartbeatServicePort = 9050 beStatus = true beHost = 192.168.xx.xx beHeartbeatServicePort = 9050 beStatus = true ``` You can test the cluster by [viewing cluster information](#view-cluster-information). You can also test it by connecting the cluster with MySQL client. ```shell mysql -h 127.0.0.1 -P9030 -uroot ``` #### View cluster information[​](#view-cluster-information "Direct link to View cluster information") You can view the information of the cluster that StarGo manages. ##### View the information of all clusters[​](#view-the-information-of-all-clusters "Direct link to View the information of all clusters") View the information of all clusters by running the following command. ```shell ./sr-ctl cluster list ``` Example: ```shell [sr-dev@r0 ~]$ ./sr-ctl cluster list [20220302-001640 OUTPUT] List all clusters ClusterName User CreateDate MetaPath PrivateKey --------------- ---------- ------------------------- ------------------------------------------------------------ -------------------------------------------------- sr-c1 starrocks 2022-03-02 00:08:15 /home/sr-dev/.starrocks-controller/cluster/sr-c1 /home/sr-dev/.ssh/id_rsa ``` ##### View the information of a specific cluster[​](#view-the-information-of-a-specific-cluster "Direct link to View the information of a specific cluster") View the information of a specific cluster by running the following command. ```shell ./sr-ctl cluster display ``` Example: ```plain [sr-dev@r0 ~]$ ./sr-ctl cluster display sr-c1 [20220302-002310 OUTPUT] Display cluster [clusterName = sr-c1] clusterName = sr-c1 ID ROLE HOST PORT STAT DATADIR DEPLOYDIR -------------------------- ------ -------------------- --------------- ---------- -------------------------------------------------- -------------------------------------------------- 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage ``` #### Start cluster[​](#start-cluster "Direct link to Start cluster") You can start StarRocks clusters via StarGo. ##### Start all nodes in a cluster[​](#start-all-nodes-in-a-cluster "Direct link to Start all nodes in a cluster") Start all nodes in a cluster by running the following command. ```shell ./sr-ctl cluster start ``` Example: ```plain [root@nd1 sr-controller]# ./sr-ctl cluster start sr-c1 [20220303-190404 OUTPUT] Start cluster [clusterName = sr-c1] [20220303-190404 INFO] Starting FE node [FeHost = 192.168.xx.xx, EditLogPort = 9010] [20220303-190435 INFO] Starting FE node [FeHost = 192.168.xx.xx, EditLogPort = 9010] [20220303-190446 INFO] Starting FE node [FeHost = 192.168.xx.xx, EditLogPort = 9010] [20220303-190457 INFO] Starting BE node [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220303-190458 INFO] Starting BE node [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220303-190458 INFO] Starting BE node [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] ``` ##### Start nodes of a specific role[​](#start-nodes-of-a-specific-role "Direct link to Start nodes of a specific role") * Start all FE nodes in a cluster. ```shell ./sr-ctl cluster start --role FE ``` * Start all BE nodes in a cluster. ```shell ./sr-ctl cluster start --role BE ``` Example: ```plain [root@nd1 sr-controller]# ./sr-ctl cluster start sr-c1 --role FE [20220303-191529 OUTPUT] Start cluster [clusterName = sr-c1] [20220303-191529 INFO] Starting FE cluster .... [20220303-191529 INFO] Starting FE node [FeHost = 192.168.xx.xx, EditLogPort = 9010] [20220303-191600 INFO] Starting FE node [FeHost = 192.168.xx.xx, EditLogPort = 9010] [20220303-191610 INFO] Starting FE node [FeHost = 192.168.xx.xx, EditLogPort = 9010] [root@nd1 sr-controller]# ./sr-ctl cluster start sr-c1 --role BE [20220303-194215 OUTPUT] Start cluster [clusterName = sr-c1] [20220303-194215 INFO] Starting BE node [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220303-194216 INFO] Starting BE node [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220303-194217 INFO] Starting BE node [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220303-194217 INFO] Starting BE cluster ... ``` ##### Start a specific node[​](#start-a-specific-node "Direct link to Start a specific node") Start a specific node in the cluster. Currently, only BE nodes are supported. ```shell ./sr-ctl cluster start --node ``` You can check the ID of a specific node by [viewing the information of a specific cluster](#view-the-information-of-a-specific-cluster). Example: ```plain [root@nd1 sr-controller]# ./sr-ctl cluster start sr-c1 --node 192.168.xx.xx:9060 [20220303-194714 OUTPUT] Start cluster [clusterName = sr-c1] [20220303-194714 INFO] Start BE node. [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] ``` #### Stop cluster[​](#stop-cluster "Direct link to Stop cluster") You can stop StarRocks clusters via StarGo. ##### Stop all nodes in a cluster[​](#stop-all-nodes-in-a-cluster "Direct link to Stop all nodes in a cluster") Stop all nodes in a cluster by running the following command. ```shell ./sr-ctl cluster stop ``` Example: ```plain [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster stop sr-c1 [20220302-180140 OUTPUT] Stop cluster [clusterName = sr-c1] [20220302-180140 OUTPUT] Stop cluster sr-c1 [20220302-180140 INFO] Waiting for stoping FE node [FeHost = 192.168.xx.xx] [20220302-180143 OUTPUT] The FE node stop succefully [host = 192.168.xx.xx, queryPort = 9030] [20220302-180143 INFO] Waiting for stoping FE node [FeHost = 192.168.xx.xx] [20220302-180145 OUTPUT] The FE node stop succefully [host = 192.168.xx.xx, queryPort = 9030] [20220302-180145 INFO] Waiting for stoping FE node [FeHost = 192.168.xx.xx] [20220302-180148 OUTPUT] The FE node stop succefully [host = 192.168.xx.xx, queryPort = 9030] [20220302-180148 OUTPUT] Stop cluster sr-c1 [20220302-180148 INFO] Waiting for stoping BE node [BeHost = 192.168.xx.xx] [20220302-180148 INFO] The BE node stop succefully [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220302-180148 INFO] Waiting for stoping BE node [BeHost = 192.168.xx.xx] [20220302-180149 INFO] The BE node stop succefully [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220302-180149 INFO] Waiting for stoping BE node [BeHost = 192.168.xx.xx] [20220302-180149 INFO] The BE node stop succefully [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] ``` ##### Stop nodes of a specific role[​](#stop-nodes-of-a-specific-role "Direct link to Stop nodes of a specific role") * Stop all FE nodes in a cluster. ```shell ./sr-ctl cluster stop --role FE ``` * Stop all BE nodes in a cluster. ```shell ./sr-ctl cluster stop --role BE ``` Example: ```plain [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster stop sr-c1 --role BE [20220302-180624 OUTPUT] Stop cluster [clusterName = sr-c1] [20220302-180624 OUTPUT] Stop cluster sr-c1 [20220302-180624 INFO] Waiting for stoping BE node [BeHost = 192.168.xx.xx] [20220302-180624 INFO] The BE node stop succefully [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220302-180624 INFO] Waiting for stoping BE node [BeHost = 192.168.xx.xx] [20220302-180625 INFO] The BE node stop succefully [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220302-180625 INFO] Waiting for stoping BE node [BeHost = 192.168.xx.xx] [20220302-180625 INFO] The BE node stop succefully [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220302-180625 INFO] Stopping BE cluster ... ########################################################################### [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster stop sr-c1 --role FE [20220302-180849 OUTPUT] Stop cluster [clusterName = sr-c1] [20220302-180849 INFO] Stopping FE cluster .... [20220302-180849 OUTPUT] Stop cluster sr-c1 [20220302-180849 INFO] Waiting for stoping FE node [FeHost = 192.168.xx.xx] [20220302-180851 OUTPUT] The FE node stop succefully [host = 192.168.xx.xx, queryPort = 9030] [20220302-180851 INFO] Waiting for stoping FE node [FeHost = 192.168.xx.xx] [20220302-180854 OUTPUT] The FE node stop succefully [host = 192.168.xx.xx, queryPort = 9030] [20220302-180854 INFO] Waiting for stoping FE node [FeHost = 192.168.xx.xx] [20220302-180856 OUTPUT] The FE node stop succefully [host = 192.168.xx.xx, queryPort = 9030] ``` ##### Stop a specific node[​](#stop-a-specific-node "Direct link to Stop a specific node") Stop a specific node in the cluster. ```shell ./sr-ctl cluster stop --node ``` You can check the ID of a specific node by [viewing the information of a specific cluster](#view-the-information-of-a-specific-cluster). Example: ```plain [root@nd1 sr-controller]# ./sr-ctl cluster display sr-c1 [20220303-185400 OUTPUT] Display cluster [clusterName = sr-c1] clusterName = sr-c1 [20220303-185400 WARN] All FE nodes are down, please start FE node and display the cluster status again. ID ROLE HOST PORT STAT DATADIR DEPLOYDIR -------------------------- ------ -------------------- --------------- ---------- -------------------------------------------------- -------------------------------------------------- 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 DOWN StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 DOWN StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 DOWN StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 DOWN StarRocks/be /dataStarRocks/be/storage 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 DOWN StarRocks/be /dataStarRocks/be/storage 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 DOWN StarRocks/be /dataStarRocks/be/storage [root@nd1 sr-controller]# ./sr-ctl cluster stop sr-c1 --node 192.168.xx.xx:9060 [20220303-185510 OUTPUT] Stop cluster [clusterName = sr-c1] [20220303-185510 INFO] Stopping BE node. [BeHost = 192.168.xx.xx] [20220303-185510 INFO] Waiting for stoping BE node [BeHost = 192.168.xx.xx] ``` #### Scale cluster out[​](#scale-cluster-out "Direct link to Scale cluster out") You can scale a cluster out via StarGo. ##### Create configuration file[​](#create-configuration-file-1 "Direct link to Create configuration file") Create the scale-out task topology file based on the following template. You can specify the file to add FE and/or BE nodes based on your demand. See [Configuration](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md) for detailed information. ```yaml # Add an FE node. fe_servers: - host: 192.168.xx.xx # The IP address of the new FE node. ssh_port: 22 http_port: 8030 rpc_port: 9020 query_port: 9030 edit_log_port: 9010 deploy_dir: StarRocks/fe meta_dir: StarRocks/fe/meta log_dir: StarRocks/fe/log priority_networks: 192.168.xx.xx/24 # Specify the unique IP for current node when the machine has multiple IP addresses. config: sys_log_level: "INFO" sys_log_delete_age: "1d" # Add a BE node. be_servers: - host: 192.168.xx.xx # The IP address of the new BE node. ssh_port: 22 be_port: 9060 be_http_port: 8040 heartbeat_service_port: 9050 brpc_port: 8060 deploy_dir : StarRocks/be storage_dir: StarRocks/be/storage log_dir: StarRocks/be/log config: create_tablet_worker_count: 3 ``` ##### Build SSH mutual authentication[​](#build-ssh-mutual-authentication "Direct link to Build SSH mutual authentication") If you are adding a new node to the cluster, you must build mutual authentication between the new node and the central control node. See [Prerequisites](#prerequisites) for detailed instruction. ##### Create deployment directory (Optional)[​](#create-deployment-directory-optional-1 "Direct link to Create deployment directory (Optional)") If the path under which the new node to be deployed does not exist, and you have the privilege to create such path, you do not have create these path, and StarGo will create them for you based on the configuration file. If the the paths already exist, make sure you have the write access to them. You can also create necessary deployment directories on each node by running the following commands. * Create **meta** directory on FE nodes. ```shell mkdir -p StarRocks/fe/meta ``` * Create **storage** directory on BE nodes. ```shell mkdir -p StarRocks/be/storage ``` > Caution Make sure the above paths are identical with the configuration items `meta_dir` and `storage_dir` in the configuration file. ##### Scale the cluster out[​](#scale-the-cluster-out "Direct link to Scale the cluster out") Scale the cluster out by running the following command. ```shell ./sr-ctl cluster scale-out ``` Example: ```plain # Status of the cluster before scale-out. [root@nd1 sr-controller]# ./sr-ctl cluster display sr-test [20220503-210047 OUTPUT] Display cluster [clusterName = sr-test] clusterName = sr-test clusterVerison = v2.0.1 ID ROLE HOST PORT STAT DATADIR DEPLOYDIR -------------------------- ------ -------------------- --------------- ---------- -------------------------------------------------- -------------------------------------------------- 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP /opt/starrocks-test/fe /opt/starrocks-test/fe/meta 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP /opt/starrocks-test/be /opt/starrocks-test/be/storage # Scale the cluster out. [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster scale-out sr-test sr-out.yaml [20220503-213725 OUTPUT] Scale out cluster. [ClusterName = sr-test] [20220503-213731 OUTPUT] PRE CHECK DEPLOY ENV: PreCheck FE: IP ssh auth meta dir deploy dir http port rpc port query port edit log port -------------------- --------------- ------------------------------ ------------------------------ --------------- --------------- --------------- --------------- 192.168.xx.xx PASS PASS PASS PASS PASS PASS PASS PreCheck BE: IP ssh auth storage dir deploy dir webSer port heartbeat port brpc port be port -------------------- --------------- ------------------------------ ------------------------------ --------------- --------------- --------------- --------------- 192.168.xx.xx PASS PASS PASS PASS PASS PASS PASS [20220503-213731 OUTPUT] PreCheck successfully. RESPECT [20220503-213731 OUTPUT] Create the deploy folder ... [20220503-213732 OUTPUT] Download StarRocks package & jdk ... [20220503-213732 INFO] The package has already exist [fileName = starrocks-2.0.1-quickstart.tar.gz, fileSize = 1227406189, fileModTime = 2022-05-03 17:32:03.478661923 +0800 CST] [20220503-213732 OUTPUT] Download done. [20220503-213732 OUTPUT] Decompress StarRocks pakage & jdk ... [20220503-213741 INFO] The tar file /home/sr-dev/.starrocks-controller/download/starrocks-2.0.1-quickstart.tar.gz has been decompressed under /home/sr-dev/.starrocks-controller/download [20220503-213837 INFO] The tar file /home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1.tar.gz has been decompressed under /home/sr-dev/.starrocks-controller/download [20220503-213837 INFO] The tar file /home/sr-dev/.starrocks-controller/download/jdk-8u301-linux-x64.tar.gz has been decompressed under /home/sr-dev/.starrocks-controller/download [20220503-213837 OUTPUT] Distribute FE Dir ... [20220503-213845 INFO] Upload dir feSourceDir = [/home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1/fe] to feTargetDir = [StarRocks/fe] on FeHost = [192.168.xx.xx] [20220503-213857 INFO] Upload dir JDKSourceDir = [/home/sr-dev/.starrocks-controller/download/jdk1.8.0_301] to JDKTargetDir = [StarRocks/fe/jdk] on FeHost = [192.168.xx.xx] [20220503-213857 INFO] Modify JAVA_HOME: host = [192.168.xx.xx], filePath = [StarRocks/fe/bin/start_fe.sh] [20220503-213857 OUTPUT] Distribute BE Dir ... [20220503-213924 INFO] Upload dir BeSourceDir = [/home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1/be] to BeTargetDir = [StarRocks/be] on BeHost = [192.168.xx.xx] [20220503-213924 OUTPUT] Modify configuration for FE nodes & BE nodes ... ############################################# SCALE OUT FE CLUSTER ############################################# ############################################# SCALE OUT FE CLUSTER ############################################# [20220503-213925 INFO] Starting follower FE node [host = 192.168.xx.xx, editLogPort = 9010] [20220503-213945 INFO] The FE node start succefully [host = 192.168.xx.xx, queryPort = 9030] [20220503-213945 INFO] List all FE status: feHost = 192.168.xx.xx feQueryPort = 9030 feStatus = true ############################################# START BE CLUSTER ############################################# ############################################# START BE CLUSTER ############################################# [20220503-213945 INFO] Starting BE node [BeHost = 192.168.xx.xx HeartbeatServicePort = 9050] [20220503-214016 INFO] The BE node start succefully [host = 192.168.xx.xx, heartbeatServicePort = 9050] [20220503-214016 OUTPUT] List all BE status: beHost = 192.168.xx.xx beHeartbeatServicePort = 9050 beStatus = true # Status of the cluster after scale-out. [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster display sr-test [20220503-214302 OUTPUT] Display cluster [clusterName = sr-test] clusterName = sr-test clusterVerison = v2.0.1 ID ROLE HOST PORT STAT DATADIR DEPLOYDIR -------------------------- ------ -------------------- --------------- ---------- -------------------------------------------------- -------------------------------------------------- 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP /opt/starrocks-test/fe /opt/starrocks-test/fe/meta 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP StarRocks/fe StarRocks/fe/meta 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP /opt/starrocks-test/be /opt/starrocks-test/be/storage 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be StarRocks/be/storage ``` #### Scale cluster in[​](#scale-cluster-in "Direct link to Scale cluster in") Remove a node in the cluster by running the following command. ```shell ./sr-ctl cluster scale-in --node ``` You can check the ID of a specific node by [viewing the information of a specific cluster](#view-the-information-of-a-specific-cluster). Example: ```plain [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster display sr-c1 [20220505-145649 OUTPUT] Display cluster [clusterName = sr-c1] clusterName = sr-c1 clusterVerison = v2.0.1 ID ROLE HOST PORT STAT DATADIR DEPLOYDIR -------------------------- ------ -------------------- --------------- ---------- -------------------------------------------------- -------------------------------------------------- 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster scale-in sr-c1 --node 192.168.88.83:9010 [20220621-010553 OUTPUT] Scale in cluster [clusterName = sr-c1, nodeId = 192.168.88.83:9010] [20220621-010553 INFO] Waiting for stoping FE node [FeHost = 192.168.88.83] [20220621-010606 OUTPUT] Scale in FE node successfully. [clusterName = sr-c1, nodeId = 192.168.88.83:9010] [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster display sr-c1 [20220621-010623 OUTPUT] Display cluster [clusterName = sr-c1] clusterName = sr-c1 clusterVerison = ID ROLE HOST PORT STAT DATADIR DEPLOYDIR -------------------------- ------ -------------------- --------------- ---------- -------------------------------------------------- -------------------------------------------------- 192.168.88.84:9010 FE 192.168.xx.xx 9010/9030 UP StarRocks/fe /dataStarRocks/fe/meta 192.168.88.85:9010 FE 192.168.xx.xx 9010/9030 UP/L StarRocks/fe /dataStarRocks/fe/meta 192.168.88.83:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage 192.168.88.84:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage 192.168.88.85:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage ``` #### Upgrade or downgrade the cluster[​](#upgrade-or-downgrade-the-cluster "Direct link to Upgrade or downgrade the cluster") You can upgrade or downgrade a cluster via StarGo. * Upgrade a cluster. ```shell ./sr-ctl cluster upgrade ``` * Downgrade a cluster. ```shell ./sr-ctl cluster downgrade ``` Example: ```plain [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster list [20220515-195827 OUTPUT] List all clusters ClusterName Version User CreateDate MetaPath PrivateKey --------------- ---------- ---------- ------------------------- ------------------------------------------------------------ -------------------------------------------------- sr-test2 v2.0.1 test222 2022-05-15 19:35:36 /home/sr-dev/.starrocks-controller/cluster/sr-test2 /home/sr-dev/.ssh/id_rsa [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster upgrade sr-test2 v2.1.3 [20220515-200358 OUTPUT] List all clusters ClusterName Version User CreateDate MetaPath PrivateKey --------------- ---------- ---------- ------------------------- ------------------------------------------------------------ -------------------------------------------------- sr-test2 v2.1.3 test222 2022-05-15 20:03:01 /home/sr-dev/.starrocks-controller/cluster/sr-test2 /home/sr-dev/.ssh/id_rsa [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster downgrade sr-test2 v2.0.1 [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster list [20220515-200915 OUTPUT] List all clusters ClusterName Version User CreateDate MetaPath PrivateKey --------------- ---------- ---------- ------------------------- ------------------------------------------------------------ -------------------------------------------------- sr-test2 v2.0.1 test222 2022-05-15 20:08:40 /home/sr-dev/.starrocks-controller/cluster/sr-test2 /home/sr-dev/.ssh/id_rsa ``` #### Relevant commands[​](#relevant-commands "Direct link to Relevant commands") | Command | Description | | --------- | ------------------------------------------- | | deploy | Deploy a cluster. | | start | Start a cluster. | | stop | Stop a cluster. | | scale-in | Scale in a cluster. | | scale-out | Scale out a cluster. | | upgrade | Upgrade a cluster. | | downgrade | Downgrade a cluster | | display | View the information of a specific cluater. | | list | View all clusters. | --- ### User Privileges and Authentication #### [🗃️ Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication.md) [5 items](https://docs.starrocks.io/docs/administration/user_privs/authentication.md) #### [🗃️ Authorization](https://docs.starrocks.io/docs/administration/user_privs/authorization.md) [6 items](https://docs.starrocks.io/docs/administration/user_privs/authorization.md) #### [📄️ Authenticate User Groups](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md) [Enable Group Provider in StarRocks to authenticate, and authorize user groups from external authentication systems.](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md) #### [📄️ SSL Authentication](https://docs.starrocks.io/docs/administration/user_privs/ssl_authentication.md) [From v3.4.1 onwards, StarRocks supports secure connections encrypted by SSL.](https://docs.starrocks.io/docs/administration/user_privs/ssl_authentication.md) --- ### Authentication #### [📄️ Native Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/native_authentication.md) [Create and manage users using the native authentication within StarRocks through SQL commands.](https://docs.starrocks.io/docs/administration/user_privs/authentication/native_authentication.md) #### [📄️ Authenticate with Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md) [Integrate StarRocks with external authentication systems using security integration.](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md) #### [📄️ LDAP Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/ldap_authentication.md) [In addition to native password-based authentication, StarRocks also supports the LDAP authentication.](https://docs.starrocks.io/docs/administration/user_privs/authentication/ldap_authentication.md) #### [📄️ JSON Web Token Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/jwt_authentication.md) [From v3.5.0 onwards, StarRocks supports authenticating client access using JSON Web Tokens.](https://docs.starrocks.io/docs/administration/user_privs/authentication/jwt_authentication.md) #### [📄️ OAuth 2.0 Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/oauth2_authentication.md) [From v3.5.0 onwards, StarRocks supports authenticating client access using OAuth 2.0.](https://docs.starrocks.io/docs/administration/user_privs/authentication/oauth2_authentication.md) --- ### JSON Web Token Authentication This topic describes how to enable JSON Web Token authentication in StarRocks. From v3.5.0 onwards, StarRocks supports authenticating client access using JSON Web Tokens. JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. This information can be verified and trusted because it is digitally signed. JWTs can be signed using a secret (with the HMAC algorithm) or a public/private key pair using RSA or ECDSA. This topic describes how to manually create and authenticate users using JWT in StarRocks. For instructions on how to integrate StarRocks with JWT authentication using security integration, see [Authenticate with Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md). For more information on how to authenticate user groups using JWT, see [Authenticate User Groups](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") If you want to connect to StarRocks from a MySQL client, the MySQL client version must be 9.2 or later. #### Create a user with JWT[​](#create-a-user-with-jwt "Direct link to Create a user with JWT") When creating a user, specify the authentication method as JWT by `IDENTIFIED WITH authentication_jwt [AS '{xxx}']`. `{xxx}` is the JWT properties of the user. In addition to the following method, you can configure the default JWT properties in the FE configuration file. You need to manually modify all **fe.conf** files and restart all FEs for configuration to take effect. After the FE configurations are set, StarRocks will use the default properties specified in your configuration file and you can omit the `AS '{xxx}'` part. Syntax: ```sql CREATE USER IDENTIFIED WITH authentication_jwt [AS '{ "jwks_url": "", "principal_field": "", "required_issuer": "", "required_audience": "" }'] ``` | Property | Corresponding FE Configuration | Description | | ------------------- | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `jwks_url` | `jwt_jwks_url` | The URL to the JSON Web Key Set (JWKS) service or the path to the public key local file under the `fe/conf` directory. | | `principal_field` | `jwt_principal_field` | The string used to identify the field that indicates the subject (`sub`) in the JWT. The default value is `sub`. The value of this field must be identical with the username for logging in to StarRocks. | | `required_issuer` | `jwt_required_issuer` | (Optional) The list of strings used to identify the issuers (`iss`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT issuer. | | `required_audience` | `jwt_required_audience` | (Optional) The list of strings used to identify the audience (`aud`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT audience. | Example: ```sql CREATE USER tom IDENTIFIED WITH authentication_jwt AS '{ "jwks_url": "http://localhost:38080/realms/master/protocol/jwt/certs", "principal_field": "preferred_username", "required_issuer": "http://localhost:38080/realms/master", "required_audience": "starrocks" }'; ``` If you have set the JWT properties in the FE configuration files, you can directly execute the following statement: ```sql CREATE USER tom IDENTIFIED WITH authentication_jwt; ``` #### Connect from MySQL client with JWT[​](#connect-from-mysql-client-with-jwt "Direct link to Connect from MySQL client with JWT") To connect from a MySQL client to StarRocks using JWT, you need to enable the `authentication_openid-connect_client` plugin, and pass the necessary token (using the path to the token file) to authenticate the mapped user. Syntax: ```bash mysql -h -P --authentication-openid-connect-client-id-token-file= -u ``` Example: ```bash mysql -h 127.0.0.1 -P 9030 --authentication-openid-connect-client-id-token-file=/path/to/token/file -u tom ``` --- ### LDAP Authentication In addition to native password-based authentication, StarRocks also supports the LDAP authentication. This topic describes how to manually create and authenticate users using LDAP in StarRocks. For instructions on how to integrate StarRocks with your LDAP service using security integration, see [Authenticate with Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md). For more information on how to authenticate user groups in your LDAP service, see [Authenticate User Groups](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). #### Enable LDAD authentication[​](#enable-ldad-authentication "Direct link to Enable LDAD authentication") To use LDAP authentication, you need to add the LDAP service into the FE node configuration first. ```properties # Add the LDAP service IP address. authentication_ldap_simple_server_host = # Add the LDAP service port, with a default value of 389. authentication_ldap_simple_server_port = # Whether to allow non-encrypted connections to the LDAP server. Default value: `true`. Setting this value to `false` indicates that SSL encryption is required to access LDAP. authentication_ldap_simple_ssl_conn_allow_insecure = # Local path to store the SSL CA certificate of the LDAP server. Supports pem and jks formats. You do not need to set this item if the certificate is issued by a trusted organization. authentication_ldap_simple_ssl_conn_trust_store_path = # The password used to access the locally stored SSL CA certificate of the LDAP server. pem-formatted certificates do not require a password. Only jsk-formatted certificates do. authentication_ldap_simple_ssl_conn_trust_store_pwd = ``` If you wish to authenticate users by means of StarRocks retrieving them directly in the LDAP system (search-and-bind mode), you will need to **add the following additional configuration items**. ```properties # Add the Base DN of the user, specifying the user's retrieval range. authentication_ldap_simple_bind_base_dn = # Add the name of the attribute that carries the login name. Default: uid (OpenLDAP). # On Active Directory use sAMAccountName. authentication_ldap_simple_user_search_attr = # Add the admin DN for retrieving users. authentication_ldap_simple_bind_root_dn = # Add the admin password for retrieving users. authentication_ldap_simple_bind_root_pwd = ``` If you wish to use **direct bind mode** (skip the search step and bind directly with a constructed DN), you can configure a DN pattern instead. This is useful when the user DN structure is predictable. ```properties # The DN pattern for direct bind authentication. # Use ${USER} as a placeholder for the username. # Multiple patterns can be separated by semicolon ';'. authentication_ldap_simple_bind_dn_pattern = ``` For example: `uid=${USER},ou=People,dc=example,dc=com` If you have users across multiple OUs, you can specify multiple patterns separated by semicolons: `uid=${USER},ou=Engineering,dc=example,dc=com;uid=${USER},ou=Marketing,dc=example,dc=com` The system will try each pattern in order and return the first successful bind. note The pattern must produce a valid LDAP Distinguished Name (DN). UPN-style patterns like `${USER}@corp.example.com` are not supported, because the result is not a DN and would break downstream group lookups. If your DN contains `@` in an attribute value (e.g., `uid=${USER}@corp.example.com,ou=People,dc=example,dc=com`), that is valid. #### DN Matching Mechanism[​](#dn-matching-mechanism "Direct link to DN Matching Mechanism") Starting from v3.5.0, StarRocks supports recording and passing user Distinguished Name (DN) information during LDAP authentication to provide more accurate group resolution. ##### How it Works[​](#how-it-works "Direct link to How it Works") 1. **Authentication Phase**: LDAPAuthProvider records both pieces of information after successful user authentication: * Login username (for traditional group matching) * User's complete DN (for DN-based group matching) 2. **Group Resolution Phase**: LDAPGroupProvider determines the matching strategy based on the `ldap_user_search_attr` parameter configuration: * **When `ldap_user_search_attr` is configured**, it uses username as the key for group matching. * **When `ldap_user_search_attr` is not configured**, it uses DN as the key for group matching. ##### Use Cases[​](#use-cases "Direct link to Use Cases") * **Traditional LDAP Environment**: Group members use simple usernames (such as `cn` attribute). Administrators need to configure `ldap_user_search_attr`. * **Microsoft AD Environment**: Group members may lack username attributes. `ldap_user_search_attr` cannot be configured. The system will use DN directly for matching. * **Mixed Environment**: Flexible switching between both matching methods is supported. #### Authentication priority[​](#authentication-priority "Direct link to Authentication priority") When a user logs in with LDAP authentication, StarRocks determines the user's DN using the following priority: 1. **Per-user DN**: If the user was created with an explicit DN (`CREATE USER ... AS 'dn'`), that DN is used directly. 2. **Direct bind via DN pattern**: If `authentication_ldap_simple_bind_dn_pattern` is configured, the system constructs the DN from the pattern and attempts to bind directly. Multiple patterns are tried in order. 3. **Search-and-bind**: If neither of the above applies, the system uses the admin account to search for the user in LDAP, then binds with the found DN. #### Resolve groups from the user entry (memberOf)[​](#resolve-groups-from-the-user-entry-memberof "Direct link to Resolve groups from the user entry (memberOf)") From v4.2 onwards, the groups of an LDAP user can be read from the group membership attribute of the user's own entry, instead of declaring in the configuration which groups to look at. A group newly created in the directory then takes effect on the next login without any configuration change. Two FE configuration items control it, both dynamic: ```properties # Where the groups come from: group_provider (default) | memberof | both authentication_ldap_simple_group_source = group_provider # Name of the attribute on the user entry that carries the group membership authentication_ldap_simple_memberof_attr = memberOf ``` * `group_provider` (default) keeps the behavior of earlier versions: only the group providers are used. * `memberof` uses only the attribute of the user's own entry. Configured group providers are ignored, but their configuration is kept. * `both` returns the union of the two. `memberOf` fits Active Directory and OpenLDAP with the `memberof` overlay installed. Oracle Directory Server and 389 Directory Server use `isMemberOf`. These are cluster-wide defaults. A security integration can override both with properties of the same name, which is the recommended way to configure them per LDAP service — see [Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md) for the full behavior, including how the attribute is read in each authentication mode and what is not covered (nested groups, the Active Directory primary group, and cross-domain groups). note A user created with an explicit DN (`CREATE USER ... IDENTIFIED WITH authentication_ldap_simple AS ''`) does **not** support this feature: that form is the legacy per-user mechanism and is scheduled for deprecation, so its group resolution is unchanged. Create the user without `AS ''`, or use a security integration. #### Case sensitivity[​](#case-sensitivity "Direct link to Case sensitivity") There is no single answer for the whole LDAP configuration - some values are compared by the FE and some by the directory - so the table below says which is which. | What | Compared where | Case-sensitive? | | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Attribute names: `authentication_ldap_simple_user_search_attr`, `authentication_ldap_simple_memberof_attr` | the directory, and the FE when it reads the answer back | **No.** An LDAP attribute description is case-insensitive by definition, so `memberof` and `memberOf` are the same attribute. | | `authentication_ldap_simple_group_source` | the FE | **No.** Surrounding blanks are ignored too. An unrecognized value is refused by `CREATE` / `ALTER SECURITY INTEGRATION`, and falls back to `group_provider` with an ERROR in the log when it arrives through the FE configuration. | | DN values: `authentication_ldap_simple_bind_root_dn`, `..._bind_base_dn`, `..._bind_dn_pattern` | the directory | **Decided by the directory.** The attribute types in a DN (`cn=`, `ou=`, `dc=`) are always case-insensitive; whether their values are depends on the matching rule of each attribute, which for the usual `cn` / `ou` / `dc` is case-insensitive. | | The login name | the FE, before the directory sees it | **No.** It is lower-cased before it is put into the search filter or substituted into the bind DN pattern, which matches how Active Directory treats account names. | | Group names | the FE | **No for matching, yes for what you see.** `GRANT ... TO EXTERNAL GROUP` and `permitted_groups` are matched ignoring case, but the name itself is never rewritten: `current_group()` and the group set handed to Apache Ranger keep exactly the spelling the directory returned. Ranger matches group names case-sensitively, so its policies must use the directory's spelling. | #### Create a user with LDAP[​](#create-a-user-with-ldap "Direct link to Create a user with LDAP") When creating a user, specify the authentication method as LDAP authentication by `IDENTIFIED WITH authentication_ldap_simple AS 'xxx'`. xxx is the DN (Distinguished Name) of the user in LDAP. Example 1: Create a user with an explicit DN. ```sql CREATE USER tom IDENTIFIED WITH authentication_ldap_simple AS 'uid=tom,ou=company,dc=example,dc=com' ``` Example 2: Create a user without specifying the DN. The system will resolve the DN at login time using either the DN pattern (direct bind) or search-and-bind, depending on the configuration. ```sql CREATE USER tom IDENTIFIED WITH authentication_ldap_simple ``` If using **search-and-bind** mode, the following additional FE configuration is needed: * `authentication_ldap_simple_bind_base_dn`: The base DN of the user, specifying the retrieval range of the user. * `authentication_ldap_simple_user_search_attr`: The name of the attribute in the LDAP object that identifies the user, uid by default. * `authentication_ldap_simple_bind_root_dn`: The DN of the administrator account used to retrieve the user information. * `authentication_ldap_simple_bind_root_pwd`: The password of the administrator account used when retrieving the user information. If using **direct bind** mode, configure `authentication_ldap_simple_bind_dn_pattern` instead. This does not require an admin account. #### Authenticate users[​](#authenticate-users "Direct link to Authenticate users") LDAP authentication requires the client to pass on a clear-text password to StarRocks. There are three ways to pass on a clear-text password: ##### Connect from MySQL client with LDAP[​](#connect-from-mysql-client-with-ldap "Direct link to Connect from MySQL client with LDAP") Add `--default-auth mysql_clear_password --enable-cleartext-plugin` when executing: ```sql mysql -utom -P9030 -h127.0.0.1 -p --default-auth mysql_clear_password --enable-cleartext-plugin ``` ##### Connect from JDBC/ODBC client with LDAP[​](#connect-from-jdbcodbc-client-with-ldap "Direct link to Connect from JDBC/ODBC client with LDAP") * **JDBC** Note that when you use JDBC connections, you must enable SSL on the server side. For more information, see [SSL Authentication](https://docs.starrocks.io/docs/administration/user_privs/ssl_authentication.md). JDBC 5: ```java Properties properties = new Properties(); properties.put("authenticationPlugins", "com.mysql.jdbc.authentication.MysqlClearPasswordPlugin"); properties.put("defaultAuthenticationPlugin", "com.mysql.jdbc.authentication.MysqlClearPasswordPlugin"); properties.put("disabledAuthenticationPlugins", "com.mysql.jdbc.authentication.MysqlNativePasswordPlugin"); ``` JDBC 8: ```java Properties properties = new Properties(); properties.put("authenticationPlugins", "com.mysql.cj.protocol.a.authentication.MysqlClearPasswordPlugin"); properties.put("defaultAuthenticationPlugin", "com.mysql.cj.protocol.a.authentication.MysqlClearPasswordPlugin"); properties.put("disabledAuthenticationPlugins", "com.mysql.cj.protocol.a.authentication.MysqlNativePasswordPlugin"); ``` * **ODBC** Add `default\_auth=mysql_clear_password` and `ENABLE_CLEARTEXT\_PLUGIN=1` in the DSN of ODBC: , along with username and password. --- ### Native Authentication Create and manage users using the native authentication within StarRocks through SQL commands. StarRocks native authentication is a password-based authentication method. In addition to that, StarRocks also supports integrating with external authentication systems such as LDAP. For more instructions, see [Authenticate with Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md). note Users with the system-defined role `user_admin` can create users, alter users, and drop users in StarRocks. After a StarRocks cluster is created, the system automatically generates a `root` user with root privileges. Because the `root` user and role have all privileges of the system, we recommend that you create new users and roles for subsequent operations to prevent any risky operations. Keep the password of the `root` user properly. #### Create user[​](#create-user "Direct link to Create user") You can create a user by specifying the user identity, the authentication method, and optionally the default role. To enable the native authentication for the user, you need to explicitly specify the password in plaintext or ciphertext. The following example creates the user `jack`, allows it to connect only from the IP address `172.10.1.10`, enables the native authentication, sets the password to `12345` in plaintext, and assigns the role `example_role` to it as its default role: ```sql CREATE USER jack@'172.10.1.10' IDENTIFIED BY '12345' DEFAULT ROLE 'example_role'; ``` note * StarRocks encrypts users' passwords before storing them. You can get the encrypted password using the password() function. * A system-defined default role `PUBLIC` is assigned to a user if no default role is specified during user creation. The default role of a user is automatically activated when the user connects to StarRocks. For instructions on how to enable all (default and granted) roles for a user after connection, see [Enable all roles](https://docs.starrocks.io/docs/administration/user_privs/authorization/User_privilege.md#enable-all-roles). For more information and advanced instructions on creating a user, see [CREATE USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_USER.md). #### Alter user[​](#alter-user "Direct link to Alter user") You can alter the password, default role, or property for a user. For instructions on how to alter the default role for a user, see [Alter default role](https://docs.starrocks.io/docs/administration/user_privs/authorization/User_privilege.md#alter-the-default-role-of-a-user). ##### Alter the property of a user[​](#alter-the-property-of-a-user "Direct link to Alter the property of a user") You can set the property of a user using [ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md). The following example sets the maximum number of connections for user `jack` to `1000`. User identities that have the same user name share the same property. Therefore, you only need to set the property for `jack` and this setting takes effect for all the user identities with the user name `jack`. ```sql ALTER USER 'jack' SET PROPERTIES ("max_user_connections" = "1000"); ``` ##### Reset password for a user[​](#reset-password-for-a-user "Direct link to Reset password for a user") You can reset the password for a user using [SET PASSWORD](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_PASSWORD.md) or [ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md). > **NOTE** > > * Any user can reset their own passwords without needing any privileges. Both the following examples reset the password of `jack` to `54321`: * Reset the password using SET PASSWORD: ```sql SET PASSWORD FOR jack@'172.10.1.10' = PASSWORD('54321'); ``` * Reset the password using ALTER USER: ```sql ALTER USER jack@'172.10.1.10' IDENTIFIED BY '54321'; ``` ###### Reset lost root password[​](#reset-lost-root-password "Direct link to Reset lost root password") Only the `root` user itself can set its password. If you have lost the password of the `root` user and cannot connect to StarRocks, you can reset it by following these procedures: 1. Add the following configuration item to the configuration files **fe/conf/fe.conf** of **all FE nodes** to disable user authentication: ```yaml enable_auth_check = false ``` 2. Restart **all FE nodes** to allow the configuration to take effect. ```bash ./fe/bin/stop_fe.sh ./fe/bin/start_fe.sh ``` 3. Connect from a MySQL client to StarRocks via the `root` user. You do not need to specify the password when user authentication is disabled. ```bash mysql -h -P -uroot ``` 4. Reset the password for the `root` user. ```sql SET PASSWORD for root = PASSWORD('xxxxxx'); ``` 5. Re-enable user authentication by setting the configuration item `enable_auth_check` to `true` in the configuration files **fe/conf/fe.conf** of **all FE nodes**. ```yaml enable_auth_check = true ``` 6. Restart **all FE nodes** to allow the configuration to take effect. ```bash ./fe/bin/stop_fe.sh ./fe/bin/start_fe.sh ``` 7. Connect from a MySQL client to StarRocks using the `root` user and the new password to verify whether the password is reset successfully. ```bash mysql -h -P -uroot -p ``` #### Drop a user[​](#drop-a-user "Direct link to Drop a user") You can drop a user using [DROP USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/DROP_USER.md). The following example drops the user `jack`: ```sql DROP USER jack@'172.10.1.10'; ``` #### View users[​](#view-users "Direct link to View users") You can view all the users within the StarRocks cluster using SHOW USERS. ```sql SHOW USERS; ``` #### View user property[​](#view-user-property "Direct link to View user property") You can view the property of a user using [SHOW PROPERTY](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_PROPERTY.md). The following example shows the property of the user `jack`: ```sql SHOW PROPERTY FOR 'jack'; ``` Or to view a specific property: ```sql SHOW PROPERTY FOR 'jack' LIKE 'max_user_connections'; ``` --- ### OAuth 2.0 Authentication This topic describes how to enable OAuth 2.0 authentication in StarRocks. From v3.5.0 onwards, StarRocks supports authenticating client access using OAuth 2.0. You can enable OAuth 2.0 authentication over HTTP for the Web UI and the JDBC driver. StarRocks uses the [Authorization Code](https://tools.ietf.org/html/rfc6749#section-1.3.1) flow which exchanges an authorization code for a token. Generally, the flow includes the following steps: 1. The StarRocks coordinator redirects the user’s browser to the Authorization Server. 2. The user authenticates from the Authorization Server. 3. After the request is approved, the browser is redirected back to the StarRocks FE with an authorization code. 4. the StarRocks coordinator exchanges the authorization code for a token. This topic describes how to manually create and authenticate users using OAuth 2.0 in StarRocks. For instructions on how to integrate StarRocks with your OAuth 2.0 service using security integration, see [Authenticate with Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md). For more information on how to authenticate user groups in your OAuth 2.0 service, see [Authenticate User Groups](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") If you want to connect to StarRocks from a MySQL client, the MySQL client version must be 9.2 or later. For more information, see [MySQL official document](https://dev.mysql.com/doc/refman/9.7/en/openid-pluggable-authentication.html). #### Create a user with OAuth 2.0[​](#create-a-user-with-oauth-20 "Direct link to Create a user with OAuth 2.0") When creating a user, specify the authentication method as OAuth 2.0 by `IDENTIFIED WITH authentication_oauth2 [AS '{xxx}']`. `{xxx}` is the OAuth 2.0 properties of the user. In addition to the following method, you can configure the default OAuth 2.0 properties in the FE configuration file. You need to manually modify all **fe.conf** files and restart all FEs for configuration to take effect. After the FE configurations are set, StarRocks will use the default properties specified in your configuration file and you can omit the `AS '{xxx}'` part. Syntax: ```sql CREATE USER IDENTIFIED WITH authentication_oauth2 [AS '{ "auth_server_url": "", "token_server_url": "", "client_id": "", "client_secret": "", "redirect_url": "", "jwks_url": "", "principal_field": "", "required_issuer": "", "required_audience": "" }'] ``` | Property | Corresponding FE Configuration | Description | | ------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `auth_server_url` | `oauth2_auth_server_url` | The authorization URL. The URL to which the users’ browser will be redirected in order to begin the OAuth 2.0 authorization process. | | `token_server_url` | `oauth2_token_server_url` | The URL of the endpoint on the authorization server from which StarRocks obtains the access token. | | `client_id` | `oauth2_client_id` | The public identifier of the StarRocks client. | | `client_secret` | `oauth2_client_secret` | The secret used to authorize StarRocks client with the authorization server. | | `redirect_url` | `oauth2_redirect_url` | The URL to which the users’ browser will be redirected after the OAuth 2.0 authentication succeeds. The authorization code will be sent to this URL. In most cases, it need to be configured as `http://:/api/oauth2`. | | `jwks_url` | `oauth2_jwks_url` | The URL to the JSON Web Key Set (JWKS) service or the path to the local file under the `conf` directory. | | `principal_field` | `oauth2_principal_field` | The string used to identify the field that indicates the subject (`sub`) in the JWT. The default value is `sub`. The value of this field must be identical with the username for logging in to StarRocks. | | `required_issuer` | `oauth2_required_issuer` | (Optional) The list of strings used to identify the issuers (`iss`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT issuer. | | `required_audience` | `oauth2_required_audience` | (Optional) The list of strings used to identify the audience (`aud`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT audience. | Example: ```sql CREATE USER tom IDENTIFIED WITH authentication_oauth2 AS '{ "auth_server_url": "http://localhost:38080/realms/master/protocol/openid-connect/auth", "token_server_url": "http://localhost:38080/realms/master/protocol/openid-connect/token", "client_id": "12345", "client_secret": "LsWyD9vPcM3LHxLZfzJsuoBwWQFBLcoR", "redirect_url": "http://localhost:8030/api/oauth2", "jwks_url": "http://localhost:38080/realms/master/protocol/openid-connect/certs", "principal_field": "preferred_username", "required_issuer": "http://localhost:38080/realms/master", "required_audience": "12345" }'; ``` If you have set the OAuth 2.0 properties in the FE configuration files, you can directly execute the following statement: ```sql CREATE USER tom IDENTIFIED WITH authentication_oauth2; ``` #### Connect from JDBC client with OAuth 2.0[​](#connect-from-jdbc-client-with-oauth-20 "Direct link to Connect from JDBC client with OAuth 2.0") StarRocks supports the MySQL protocol. You can customize a MySQL plugin to automatically launch the browser login method. For the example code of the JDBC OAuth2 plugin, see the official document for [starrocks-jdbc-oauth2-plugin](https://github.com/StarRocks/starrocks/tree/main/contrib/starrocks-jdbc-oauth2-plugin). #### Connect from MySQL client with OAuth 2.0[​](#connect-from-mysql-client-with-oauth-20 "Direct link to Connect from MySQL client with OAuth 2.0") If you cannot access a browser in your environment (such as using terminal or server), you can also access StarRocks via native MySQL client or JDBC driver: * When you first connect to StarRocks, a URL will be returned. * You need to access this URL on a browser and complete the authentication. * After the authentication, you can then interact with StarRocks. --- ### Authenticate with Security Integration Integrate StarRocks with external authentication systems using security integration. By creating a security integration within your StarRocks cluster, you can allow access of your external authentication service to StarRocks. With the security integration, you do not need to manually create users within StarRocks. When a user tries to log in using an external identity, StarRocks will use the corresponding security integration according to the configuration in `authentication_chain` to authenticate the user. After the authentication is successful and the user is allowed to log in, StarRocks creates a virtual user in the session for the user to perform subsequent operations. Please note that if you use the security integration to configure an external authentication method, you must also [integrate StarRocks with Apache Ranger](https://docs.starrocks.io/docs/administration/user_privs/authorization/ranger_plugin.md) to enable external authorization. Currently, integrating Security Integration with the StarRocks native authorization is not supported. You can also enable [Group Provider](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md) for StarRocks to access the group information in you external authentication systems, thus allowing creating, authenticating, and authorizing user groups in StarRocks. Manually creating and managing users with external authentication services are also supported in case of specific corner cases. For more instructions, you can refer to [See also](#see-also). #### Create a security integration[​](#create-a-security-integration "Direct link to Create a security integration") Currently, StarRocks' security integration supports the following authentication systems: * LDAP * JSON Web Token (JWT) * OAuth 2.0 note StarRocks does not offer connectivity checks when you create a security integration. ##### Create a security integration with LDAP[​](#create-a-security-integration-with-ldap "Direct link to Create a security integration with LDAP") ###### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE SECURITY INTEGRATION PROPERTIES ( "type" = "authentication_ldap_simple", "authentication_ldap_simple_server_host" = "", "authentication_ldap_simple_server_port" = "", "authentication_ldap_simple_bind_base_dn" = "", "authentication_ldap_simple_user_search_attr" = "", "authentication_ldap_simple_bind_root_dn" = "", "authentication_ldap_simple_bind_root_pwd" = "", "authentication_ldap_simple_bind_dn_pattern" = "", "authentication_ldap_simple_ssl_conn_allow_insecure" = "{true | false}", "authentication_ldap_simple_ssl_conn_trust_store_path" = "", "authentication_ldap_simple_ssl_conn_trust_store_pwd" = "", "comment" = "" ) ``` ###### Parameters[​](#parameters "Direct link to Parameters") ###### security\_integration\_name[​](#security_integration_name "Direct link to security_integration_name") * Required: Yes * Description: The name of the security integration.
**NOTE**
The security integration name is globally unique. You cannot specify this parameter as `native`. ###### type[​](#type "Direct link to type") * Required: Yes * Description: The type of the security integration. Specify it as `authentication_ldap_simple`. ###### authentication\_ldap\_simple\_server\_host[​](#authentication_ldap_simple_server_host "Direct link to authentication_ldap_simple_server_host") * Required: No * Description: The IP address of your LDAP service. Default: `127.0.0.1`. ###### authentication\_ldap\_simple\_server\_port[​](#authentication_ldap_simple_server_port "Direct link to authentication_ldap_simple_server_port") * Required: No * Description: The port of your LDAP service. Default: `389`. ###### authentication\_ldap\_simple\_bind\_base\_dn[​](#authentication_ldap_simple_bind_base_dn "Direct link to authentication_ldap_simple_bind_base_dn") * Required: No * Description: The base Distinguished Name (DN) of the LDAP user for which the cluster searches. Required when using search-and-bind mode. Not needed when using direct bind mode with `authentication_ldap_simple_bind_dn_pattern`. ###### authentication\_ldap\_simple\_user\_search\_attr[​](#authentication_ldap_simple_user_search_attr "Direct link to authentication_ldap_simple_user_search_attr") * Required: No * Description: The attribute of a user entry that carries the login name. It is interpolated into the search filter of search-and-bind mode, so it must be the attribute whose value the user actually types when logging in. Default: `uid`, the convention on OpenLDAP, where it is usually also the entry's RDN (`uid=alice,ou=People,dc=example,dc=com`). Not used in direct bind mode. On Active Directory, set this to `sAMAccountName` An AD entry's RDN is the display name (`CN=Dana Scully,OU=People,DC=company,DC=com`), while the name a user types to log in is its `sAMAccountName` (`dscully`). AD's schema does contain `uid`, but it holds no value unless an administrator populated it, so the default finds nobody and every login fails. Two settings follow from this choice: * **Use search-and-bind, not direct bind.** `sAMAccountName` is not part of an AD entry's DN, so no `authentication_ldap_simple_bind_dn_pattern` can produce one. Leave that property unset and set `authentication_ldap_simple_bind_base_dn`, `authentication_ldap_simple_bind_root_dn` and `authentication_ldap_simple_bind_root_pwd` instead. * **A Group Provider cannot match members by `sAMAccountName` either.** An AD group lists its members by DN (`member: CN=Dana Scully,OU=People,...`), and that DN carries no `sAMAccountName`, so a group provider configured with `"ldap_user_search_attr" = "sAMAccountName"` matches nobody. Either leave the group provider's own `ldap_user_search_attr` unset, so that it matches by DN, or read the groups from the user's own entry with `authentication_ldap_simple_group_source = memberof`. note **DN Passing Mechanism**: LDAP security integration supports DN passing functionality. * After successful authentication, the system records both the user's login name and complete DN. * When combined with Group Provider, DN information is automatically passed to the Group Provider. * If `ldap_user_search_attr` is not configured for the Group Provider, DN will be used for group matching. * This mechanism is particularly suitable for complex LDAP environments like Microsoft AD. For more details, see the DN matching mechanism in [Authenticate User Groups](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). ###### authentication\_ldap\_simple\_bind\_root\_dn[​](#authentication_ldap_simple_bind_root_dn "Direct link to authentication_ldap_simple_bind_root_dn") * Required: No * Description: The admin DN of your LDAP service. Required when using search-and-bind mode. ###### authentication\_ldap\_simple\_bind\_root\_pwd[​](#authentication_ldap_simple_bind_root_pwd "Direct link to authentication_ldap_simple_bind_root_pwd") * Required: No * Description: The admin password of your LDAP service. Required when using search-and-bind mode. ###### authentication\_ldap\_simple\_bind\_dn\_pattern[​](#authentication_ldap_simple_bind_dn_pattern "Direct link to authentication_ldap_simple_bind_dn_pattern") * Required: No * Description: The DN pattern for direct bind authentication. Use `${USER}` as a placeholder for the username. The pattern must produce a valid LDAP Distinguished Name (DN); UPN-style patterns like `${USER}@domain` are not supported. For example, `uid=${USER},ou=People,dc=example,dc=com`. Multiple patterns can be separated by semicolons, and the system will try each pattern in order until one succeeds. When this parameter is set, the system skips the search step and directly binds with the constructed DN, so `authentication_ldap_simple_bind_base_dn`, `authentication_ldap_simple_user_search_attr`, `authentication_ldap_simple_bind_root_dn`, and `authentication_ldap_simple_bind_root_pwd` are not required. ###### authentication\_ldap\_simple\_group\_source[​](#authentication_ldap_simple_group_source "Direct link to authentication_ldap_simple_group_source") * Required: No * Description: Where the groups of an LDAP-authenticated user come from. Valid values: * `group_provider` (default): only the group providers configured in `group_provider` are used. This is the behavior of earlier versions. * `memberof`: only the group membership attribute of the user's own LDAP entry is used. Group providers configured in `group_provider` are ignored, but the configuration is kept, so switching back only takes one `ALTER SECURITY INTEGRATION`. * `both`: the union of the two sources. With `memberof` or `both`, a group newly created in the directory takes effect on the next login without any configuration change here. The resolved groups take part in role mapping via `GRANT ... TO EXTERNAL GROUP`, in the `permitted_groups` login check, in `current_group()`, and in Apache Ranger authorization, exactly like the groups of a group provider. The cluster-wide default is the FE configuration item of the same name. Supported from v4.2 onwards. ###### authentication\_ldap\_simple\_memberof\_attr[​](#authentication_ldap_simple_memberof_attr "Direct link to authentication_ldap_simple_memberof_attr") * Required: No * Description: The name of the attribute on the user entry that carries its group membership, used when `authentication_ldap_simple_group_source` is `memberof` or `both`. Matched ignoring case, like every LDAP attribute name. Default: `memberOf`, which fits Active Directory and OpenLDAP with the `memberof` overlay installed. Oracle Directory Server and 389 Directory Server use `isMemberOf`. The cluster-wide default is the FE configuration item of the same name. Supported from v4.2 onwards. Reading group membership from the user entry * Only direct membership is resolved. Nested groups, the Active Directory primary group (`Domain Users` by default), and groups from another domain or forest are not included, because they do not appear in the attribute. * The group name is the value of the first RDN of each group DN. For example, `CN=SR Analysts,OU=Groups,DC=company,DC=com` becomes `SR Analysts`. The original case is kept. * The attribute is read on the connection that authentication already opened, so no additional LDAP connection is made. In search-and-bind mode the number of requests does not change at all; in direct bind mode (`authentication_ldap_simple_bind_dn_pattern`) one read request is added, because a bind response cannot carry attributes. * In direct bind mode the user reads its own attribute. If the directory does not allow that and `authentication_ldap_simple_bind_root_dn` and `authentication_ldap_simple_bind_root_pwd` are set, the FE retries once with that account, so there is no need to change the authentication mode. If neither reader can see the attribute, the group set is empty and the login still succeeds. * If the group set is empty and `permitted_groups` is set, the user is rejected, because the intersection is necessarily empty. * The group set is computed at login and stored in the session. A membership change in the directory takes effect on the next login, not in a running session. * The legacy per-user form `CREATE USER ... IDENTIFIED WITH authentication_ldap_simple AS ''` does not support this feature. Use a security integration instead. * `EXECUTE AS` resolves the groups of the **target** identity from that identity's own configuration: a native-password user has no LDAP identity and only gets its group providers, while `EXECUTE AS EXTERNAL USER` uses the first `authentication_ldap_simple` integration of `authentication_chain`. Impersonation never presents the target's password, so the attribute is read with `authentication_ldap_simple_bind_root_dn` and `authentication_ldap_simple_bind_root_pwd`; with no service account configured, only the group providers contribute. ###### authentication\_ldap\_simple\_ssl\_conn\_allow\_insecure[​](#authentication_ldap_simple_ssl_conn_allow_insecure "Direct link to authentication_ldap_simple_ssl_conn_allow_insecure") * Required: No * Description: Whether to allow non-encrypted connections to the LDAP server. Default value: `true`. Setting this value to `false` indicates that SSL encryption is required to access LDAP. ###### authentication\_ldap\_simple\_ssl\_conn\_trust\_store\_path[​](#authentication_ldap_simple_ssl_conn_trust_store_path "Direct link to authentication_ldap_simple_ssl_conn_trust_store_path") * Required: No * Description: Local path to store the SSL CA certificate of the LDAP server. Supports pem and jks formats. You do not need to set this item if the certificate is issued by a trusted organization. ###### ldap\_ssl\_conn\_trust\_store\_pwd[​](#ldap_ssl_conn_trust_store_pwd "Direct link to ldap_ssl_conn_trust_store_pwd") * Required: No * Description: The password used to access the locally stored SSL CA certificate of the LDAP server. pem-formatted certificates do not require a password. Only jsk-formatted certificates do. ###### group\_provider[​](#group_provider "Direct link to group_provider") * Required: No * Description: The name of the group provider(s) to be combined with the security integration. Multiple group providers are separated by commas. Once set, StarRocks will record the user's group information under each specified provider upon login. Supported from v3.5 onwards. For detailed instructions on enabling Group Provider, see [Authenticate User Groups](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). ###### permitted\_groups[​](#permitted_groups "Direct link to permitted_groups") * Required: No * Description: The name of group(s) whose members are allowed to log in to StarRocks. Multiple groups are separated by commas. Make sure that the specified groups can be retrieved by the combined group provider(s). Supported from v3.5 onwards. ###### comment[​](#comment "Direct link to comment") * Required: No * Description: The description of the security integration. ##### Create a security integration with JWT[​](#create-a-security-integration-with-jwt "Direct link to Create a security integration with JWT") ###### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE SECURITY INTEGRATION PROPERTIES ( "type" = "authentication_jwt", "jwks_url" = "", "principal_field" = "", "required_issuer" = "", "required_audience" = "" "comment" = "" ); ``` ###### Parameters[​](#parameters "Direct link to Parameters") ###### security\_integration\_name[​](#security_integration_name "Direct link to security_integration_name") * Required: Yes * Description: The name of the security integration.
**NOTE**
The security integration name is globally unique. You cannot specify this parameter as `native`. ###### type[​](#type "Direct link to type") * Required: Yes * Description: The type of the security integration. Specify it as `jwt`. ###### jwks\_url[​](#jwks_url "Direct link to jwks_url") * Required: Yes * Description: The URL to the JSON Web Key Set (JWKS) service or the path to the local file under the `fe/conf` directory. ###### principal\_field[​](#principal_field "Direct link to principal_field") * Required: Yes * Description: The string used to identify the field that indicates the subject (`sub`) in the JWT. The default value is `sub`. The value of this field must be identical with the username for logging in to StarRocks. ###### required\_issuer[​](#required_issuer "Direct link to required_issuer") * Required: No * Description: The list of strings used to identify the issuers (`iss`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT issuer. ###### required\_audience[​](#required_audience "Direct link to required_audience") * Required: No * Description: The list of strings used to identify the audience (`aud`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT audience. ###### comment[​](#comment "Direct link to comment") * Required: No * Description: The description of the security integration. ##### Create a security integration with OAuth 2.0[​](#create-a-security-integration-with-oauth-20 "Direct link to Create a security integration with OAuth 2.0") ###### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE SECURITY INTEGRATION PROPERTIES ( "type" = "authentication_oauth2", "auth_server_url" = "", "token_server_url" = "", "client_id" = "", "client_secret" = "", "redirect_url" = "", "jwks_url" = "", "principal_field" = "", "required_issuer" = "", "required_audience" = "" "comment" = "" ) ``` ###### Parameters[​](#parameters "Direct link to Parameters") ###### security\_integration\_name[​](#security_integration_name "Direct link to security_integration_name") * Required: Yes * Description: The name of the security integration.
**NOTE**
The security integration name is globally unique. You cannot specify this parameter as `native`. ###### auth\_server\_url[​](#auth_server_url "Direct link to auth_server_url") * Required: Yes * Description: The authorization URL. The URL to which the users’ browser will be redirected in order to begin the OAuth 2.0 authorization process. ###### token\_server\_url[​](#token_server_url "Direct link to token_server_url") * Required: Yes * Description: The URL of the endpoint on the authorization server from which StarRocks obtains the access token. ###### client\_id[​](#client_id "Direct link to client_id") * Required: Yes * Description: The public identifier of the StarRocks client. ###### client\_secret[​](#client_secret "Direct link to client_secret") * Required: Yes * Description: The secret used to authorize StarRocks client with the authorization server. ###### redirect\_url[​](#redirect_url "Direct link to redirect_url") * Required: Yes * Description: The URL to which the users’ browser will be redirected after the OAuth 2.0 authentication succeeds. The authorization code will be sent to this URL. In most cases, it need to be configured as `http://:/api/oauth2`. ###### type[​](#type "Direct link to type") * Required: Yes * Description: The type of the security integration. Specify it as `authentication_oauth2`. ###### jwks\_url[​](#jwks_url "Direct link to jwks_url") * Required: Yes * Description: The URL to the JSON Web Key Set (JWKS) service or the path to the local file under the `fe/conf` directory. ###### principal\_field[​](#principal_field "Direct link to principal_field") * Required: Yes * Description: The string used to identify the field that indicates the subject (`sub`) in the JWT. The default value is `sub`. The value of this field must be identical with the username for logging in to StarRocks. ###### required\_issuer[​](#required_issuer "Direct link to required_issuer") * Required: No * Description: The list of strings used to identify the issuers (`iss`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT issuer. ###### required\_audience[​](#required_audience "Direct link to required_audience") * Required: No * Description: The list of strings used to identify the audience (`aud`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT audience. ###### comment[​](#comment "Direct link to comment") * Required: No * Description: The description of the security integration. #### Configure authentication chain[​](#configure-authentication-chain "Direct link to Configure authentication chain") After the security integration is created, it is added to your StarRocks cluster as a new authentication method. You must enable the security integration by setting the order of the authentication methods via the FE dynamic configuration item `authentication_chain`. ```sql ADMIN SET FRONTEND CONFIG ( "authentication_chain" = "[... ,]" ); ``` note * StarRocks prioritizes native authentication for local users. If a local user with the same username does not exist, authentication is performed in the order configured by `authentication_chain`. If login fails using the native authentication method, the cluster will try the next authentication method in the specified order. * You can specify multiple security integrations in `authentication_chain` except for OAuth 2.0 security integration. You cannot specify multiple OAuth 2.0 security integrations or one with other security integrations. You can check the value of `authentication_chain` using the following statement: ```sql ADMIN SHOW FRONTEND CONFIG LIKE 'authentication_chain'; ``` #### Manage security integrations[​](#manage-security-integrations "Direct link to Manage security integrations") ##### Alter security integration[​](#alter-security-integration "Direct link to Alter security integration") You can alter the configuration of an existing security integration using the following statement: ```sql ALTER SECURITY INTEGRATION SET ( "key"="value"[, ...] ) ``` note You cannot alter the `type` of a security integration. ##### Drop security integration[​](#drop-security-integration "Direct link to Drop security integration") You can drop an existing security integration using the following statement: ```sql DROP SECURITY INTEGRATION ``` ##### View security integration[​](#view-security-integration "Direct link to View security integration") You can view all security integrations in your cluster using the following statement: ```sql SHOW SECURITY INTEGRATIONS; ``` Example: ```plain SHOW SECURITY INTEGRATIONS; +--------+--------+---------+ | Name | Type | Comment | +--------+--------+---------+ | LDAP1 | LDAP | NULL | +--------+--------+---------+ ``` | **Parameter** | **Description** | | ------------- | ------------------------------------------------------------------------------------------------------------------------------ | | Name | The name of the security integration. | | Type | The type of the security integration. | | Comment | The description of the security integration. `NULL` is returned when no description is specified for the security integration. | You can check the details of a security integration using the following statement: ```sql SHOW CREATE SECURITY INTEGRATION ``` Example: ```plain SHOW CREATE SECURITY INTEGRATION LDAP1； +----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Security Integration | Create Security Integration | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | LDAP1 | CREATE SECURITY INTEGRATION LDAP1 PROPERTIES ( "type" = "authentication_ldap_simple", "authentication_ldap_simple_server_host" = "", "authentication_ldap_simple_server_port" = "", "authentication_ldap_simple_bind_base_dn" = "", "authentication_ldap_simple_user_search_attr" = "" "authentication_ldap_simple_bind_root_dn" = "", "authentication_ldap_simple_bind_root_pwd" = "", "authentication_ldap_simple_ssl_conn_allow_insecure" = "{true | false}", "authentication_ldap_simple_ssl_conn_trust_store_path" = "", "authentication_ldap_simple_ssl_conn_trust_store_pwd" = "", "comment" = "" )| +----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` note `ldap_bind_root_pwd` is masked when SHOW CREATE SECURITY INTEGRATION is executed. #### Connect to StarRocks via a security integration[​](#connect-to-starrocks-via-a-security-integration "Direct link to Connect to StarRocks via a security integration") * For instructions on how to connect to StarRocks via LDAP, see [LDAP Authentication - Connect to StarRocks](https://docs.starrocks.io/docs/administration/user_privs/authentication/ldap_authentication.md#connect-from-mysql-client-with-ldap). * For instructions on how to connect to StarRocks via JWT, see [JSON Web Token Authentication - Connect to StarRocks](https://docs.starrocks.io/docs/administration/user_privs/authentication/jwt_authentication.md#connect-from-mysql-client-with-jwt). * For instructions on how to connect to StarRocks via OAuth 2.0, see [OAuth 2.0 Authentication - Connect to StarRocks](https://docs.starrocks.io/docs/administration/user_privs/authentication/oauth2_authentication.md#connect-from-jdbc-client-with-oauth-20). #### See also[​](#see-also "Direct link to See also") * For instructions on how to manually authenticate users via LDAP in StarRocks, see [LDAP Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/ldap_authentication.md). * For instructions on how to manually authenticate users via JSON Web Token in StarRocks, see [JSON Web Token Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/jwt_authentication.md). * For instructions on how to manually authenticate users via OAuth 2.0 in StarRocks, see [OAuth 2.0 Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/oauth2_authentication.md). * For instructions on how to authenticate user groups, see [Authenticate User Groups](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). --- ### Authorization #### [📄️ Overview of privileges](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md) [Overview of StarRocks privilege system, covering role-based access control (RBAC) and identity-based access control (IBAC) models.](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md) #### [📄️ Privilege Items](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md) [Privileges granted to a user or role determine which operations the user or role can perform on certain objects.](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md) #### [📄️ Built-in Roles](https://docs.starrocks.io/docs/administration/user_privs/authorization/built_in_roles.md) [Reference for the five built-in StarRocks roles: db\_admin, cluster\_admin, user\_admin, security\_admin, and public.](https://docs.starrocks.io/docs/administration/user_privs/authorization/built_in_roles.md) #### [📄️ Manage user privileges](https://docs.starrocks.io/docs/administration/user_privs/authorization/User_privilege.md) [How to manage user privileges in StarRocks using role-based access control (RBAC) and identity-based access control (IBAC).](https://docs.starrocks.io/docs/administration/user_privs/authorization/User_privilege.md) #### [📄️ Manage permissions with Apache Ranger](https://docs.starrocks.io/docs/administration/user_privs/authorization/ranger_plugin.md) [How to manage StarRocks access permissions with Apache Ranger for centralized, fine-grained access control.](https://docs.starrocks.io/docs/administration/user_privs/authorization/ranger_plugin.md) #### [📄️ Privilege FAQ](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_faq.md) [FAQ for common StarRocks privilege and authorization questions, including role activation issues.](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_faq.md) --- ### Built-in Roles supported by StarRocks In a StarRocks cluster, there are FIVE built-in roles: * `db_admin` * `cluster_admin` * `user_admin` * `security_admin` * `public` Each of the `admin` roles is granted with different privileges to allow them to perform administrative operations on their specific domain. By default, the `public` role has no privileges and is granted to every user that can access the cluster. For details of the privileges described below, see [Privilege Item](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md). #### `db_admin`[​](#db_admin "Direct link to db_admin") `db_admin` is the built-in database administrator. It has all data-related privileges and some basic privileges on operations and maintenance. * Focused on management of databases and data * Unavailable for user or cluster management * Immutable role Privilege scope: | Privilege Level | Privilege Item | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | SYSTEM | - CREATE RESOURCE
- PLUGIN
- FILE
- BLACKLIST
- OPERATE
- CREATE EXTERNAL CATALOG
- REPOSITORY
- CREATE RESOURCE GROUP
- CREATE GLOBAL FUNCTION
- CREATE STORAGE VOLUME
- SECURITY | | CATALOG | - USAGE
- DROP
- ALTER
- CREATE DATABASE | | DATABASE | - DROP
- ALTER
- CREATE TABLE
- CREATE VIEW
- CREATE MATERIALIZED VIEW
- CREATE FUNCTION
- CREATE PIPE
- CREATE MASKING POLICY
- CREATE ROW ACCESS POLICY | | TABLE | - DROP
- ALTER
- INSERT
- UPDATE
- DELETE
- SELECT
- EXPORT | | VIEW | - DROP
- ALTER
- SELECT | | MATERIALIZED VIEW | - DROP
- ALTER
- SELECT
- REFRESH | | RESOURCE | - USAGE
- DROP
- ALTER | | RESOURCE GROUP | - DROP
- ALTER | | FUNCTION | - USAGE
- DROP | | GLOBAL FUNCTION | - USAGE
- DROP | | STORAGE VOLUME | - USAGE
- DROP
- ALTER | | PIPE | - USAGE
- DROP
- ALTER | #### `cluster_admin`[​](#cluster_admin "Direct link to cluster_admin") `cluster_admin` is the built-in cluster administrator. * Focused on management of cluster infrastructure * Granted with privileges on node management * Immutable role Privilege scope: | Privilege Level | Privilege Item | | --------------- | -------------- | | SYSTEM | NODE | #### `user_admin`[​](#user_admin "Direct link to user_admin") `user_admin` is the built-in user administrator. It can be used to manage users, roles, and authorization. * Focused on management of users and privileges * Able to create, alter, and drop users * Able to grant or revoke privileges or roles * Immutable role Privilege scope: | Privilege Level | Privilege Item | | --------------- | -------------- | | SYSTEM | GRANT | #### `security_admin`[​](#security_admin "Direct link to security_admin") `security_admin` is the built-in security administrator. It can be used to manage security integrations and group providers. * Focused on management of system security * Able to manage security-related configurations and strategies * Immutable role Privilege scope: | Privilege Level | Privilege Item | | --------------- | ------------------------- | | SYSTEM | - SECURITY
- OPERATE | #### `public`[​](#public "Direct link to public") `public` is the built-in role that is granted to every user that can access the cluster. By default, it has no privilege. * Automatically granted and activated to all cluster users * Mutable role. You can grant privileges or roles to this role if you want to grant them to all cluster users. --- ### Privilege FAQ #### Why is the error message "no permission" still reported even after the required role has been assigned to a user?[​](#why-is-the-error-message-no-permission-still-reported-even-after-the-required-role-has-been-assigned-to-a-user "Direct link to Why is the error message \"no permission\" still reported even after the required role has been assigned to a user?") This error may happen if the role is not activated. You can run `select current_role();` to query the roles that have been activated for the user in the current session. If the required role is not activated, run [SET ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_ROLE.md) to activate this role and perform operations using this role. If you want roles to be automatically activated upon login, the `user_admin` role can run [SET DEFAULT ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_DEFAULT_ROLE.md) or [ALTER USER DEFAULT ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md) to set a default role for each user. After the default role is set, it will be automatically activated when the user logs in. If you want all the assigned roles of all users to be automatically activated upon login, you can run the following command. This operation requires the OPERATE permission at the System level. ```sql SET GLOBAL activate_all_roles_on_login = TRUE; ``` However, we recommend that you follow the principle of "least privilege" by setting default roles with limited privileges to prevent potential risks. For example: * Common users can set the `read_only` role that has only the SELECT privilege as the default role, while avoiding setting roles with privileges like ALTER, DROP, and INSERT as default roles. * Administrators can set the `db_admin` role as the default role, while avoiding setting the `node_admin` role, which has the privilege to add and drop nodes, as the default role. This approach helps ensure that users are assigned roles with appropriate permissions, reducing the risk of unintended operations. You can run [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to assign the required privileges or roles to users. #### I have granted a user the privilege on all tables in a database (`GRANT ALL ON ALL TABLES IN DATABASE TO USER ;`), but the user still cannot create tables in the database. Why?[​](#i-have-granted-a-user-the-privilege-on-all-tables-in-a-database-grant-all-on-all-tables-in-database-db_name-to-user-user_identity-but-the-user-still-cannot-create-tables-in-the-database-why "Direct link to i-have-granted-a-user-the-privilege-on-all-tables-in-a-database-grant-all-on-all-tables-in-database-db_name-to-user-user_identity-but-the-user-still-cannot-create-tables-in-the-database-why") Creating tables within a database requires the database-level CREATE TABLE privilege. You need to grant the privilege to the user. ```sql GRANT CREATE TABLE ON DATABASE TO USER ;; ``` #### I have granted a user all the privileges on a database using `GRANT ALL ON DATABASE TO USER ;`, but nothing is returned when the user runs `SHOW TABLES;` in this database. Why?[​](#i-have-granted-a-user-all-the-privileges-on-a-database-using-grant-all-on-database-db_name-to-user-user_identity-but-nothing-is-returned-when-the-user-runs-show-tables-in-this-database-why "Direct link to i-have-granted-a-user-all-the-privileges-on-a-database-using-grant-all-on-database-db_name-to-user-user_identity-but-nothing-is-returned-when-the-user-runs-show-tables-in-this-database-why") `SHOW TABLES;` returns only tables on which the user has any privilege. If the user has no privilege on a table, this table will not be returned. You can grant any privilege on all tables in this database (using SELECT for example) to the user: ```sql GRANT SELECT ON ALL TABLES IN DATABASE TO USER ; ``` The statement above is equivalent to `GRANT select_priv ON db.* TO ;` used in versions earlier than v3.0. #### What privileges are required to access the StarRocks Web Console `http://:`?[​](#what-privileges-are-required-to-access-the-starrocks-web-console-httpfe_ipfe_http_port "Direct link to what-privileges-are-required-to-access-the-starrocks-web-console-httpfe_ipfe_http_port") The user must have the `db_admin` and `user_admin` roles. #### How did the privilege retention mechanism change before and after StarRocks v3.0?[​](#how-did-the-privilege-retention-mechanism-change-before-and-after-starrocks-v30 "Direct link to How did the privilege retention mechanism change before and after StarRocks v3.0?") Before v3.0, after a user is granted privileges on a table, the privileges would still remain even if the table was dropped and recreated. Starting from v3.0, privileges will no longer be retained after a table is dropped and recreated. #### How to query users and granted privileges in StarRocks?[​](#how-to-query-users-and-granted-privileges-in-starrocks "Direct link to How to query users and granted privileges in StarRocks?") You can obtain the full user list via querying the system view `sys.grants_to_users` or executing SHOW USERS, and then query each user individually using `SHOW GRANTS FOR `. #### What is the impact on FE resources when querying system views on privilege metadata of large numbers of users and tables?[​](#what-is-the-impact-on-fe-resources-when-querying-system-views-on-privilege-metadata-of-large-numbers-of-users-and-tables "Direct link to What is the impact on FE resources when querying system views on privilege metadata of large numbers of users and tables?") When the number of users or tables is very large, queries on system views `sys.grants_to_users`, `sys.grants_to_roles`, and `sys.role_edges` may take a long time. These views are computed in real time, consuming a proportion of FE resources. Therefore, it is not recommended to run such operations frequently at large scale. #### Will recreating a catalog cause permission loss? How should privileges be backed up and restored?[​](#will-recreating-a-catalog-cause-permission-loss-how-should-privileges-be-backed-up-and-restored "Direct link to Will recreating a catalog cause permission loss? How should privileges be backed up and restored?") Yes. Recreating a catalog will cause its relative privileges to be lost. You should back up all user privileges first and restore them after the catalog is recreated. #### Is there a tool that supports automatic permission migration?[​](#is-there-a-tool-that-supports-automatic-permission-migration "Direct link to Is there a tool that supports automatic permission migration?") Not at the moment. Users must manually back up and restore permissions using SHOW GRANTS for each user. #### Are there restrictions on using the KILL command? Can it be restricted to only killing a user’s own queries?[​](#are-there-restrictions-on-using-the-kill-command-can-it-be-restricted-to-only-killing-a-users-own-queries "Direct link to Are there restrictions on using the KILL command? Can it be restricted to only killing a user’s own queries?") Yes. The KILL command now requires the OPERATE privilege, and a user can only kill queries that were initiated by themselves. #### Why do the granted privileges change after renaming or deleting a table? Can the system retain old permissions while adding permissions for the renamed table?[​](#why-do-the-granted-privileges-change-after-renaming-or-deleting-a-table-can-the-system-retain-old-permissions-while-adding-permissions-for-the-renamed-table "Direct link to Why do the granted privileges change after renaming or deleting a table? Can the system retain old permissions while adding permissions for the renamed table?") For native tables, privileges are tied to the table ID, not the table name. This ensures data security, as table names can change arbitrarily. If privileges were to follow table names, it could cause data leakage. Similarly, when a table is dropped, its permissions are removed because the object no longer exists. For external tables, historical versions behaved the same as internal tables. However, because external table metadata is not managed by StarRocks, delays or privilege loss can occur. To address this, future versions will use table-name–based permission management for external tables, which aligns with the expected behavior. #### How to back up user privileges?[​](#how-to-back-up-user-privileges "Direct link to How to back up user privileges?") Below is a sample script for backing up the user privilege information in the cluster. ```bash #!/bin/bash # MySQL connection info HOST="" PORT="9030" USER="root" PASSWORD="" OUTPUT_FILE="user_privileges.txt" # Clear output file > $OUTPUT_FILE # Get user list users=$(mysql -h$HOST -P$PORT -u$USER -p$PASSWORD -e "SHOW USERS;" | sed -e '1d' -e '/^+/d') # Loop through users and get privileges for user in $users; do echo "Privileges for $user:" >> $OUTPUT_FILE mysql -h$HOST -P$PORT -u$USER -p$PASSWORD -e "SHOW GRANTS FOR $user;" >> $OUTPUT_FILE echo "" >> $OUTPUT_FILE done echo "All user privileges have been written to $OUTPUT_FILE" ``` #### Why does granting USAGE on a normal function produce the error “Unexpected input 'IN', the most similar input is TO.”? What is the correct way to grant permissions on functions?[​](#why-does-granting-usage-on-a-normal-function-produce-the-error-unexpected-input-in-the-most-similar-input-is-to-what-is-the-correct-way-to-grant-permissions-on-functions "Direct link to why-does-granting-usage-on-a-normal-function-produce-the-error-unexpected-input-in-the-most-similar-input-is-to-what-is-the-correct-way-to-grant-permissions-on-functions") Normal functions cannot be granted using IN ALL DATABASES; they can only be granted within the current database. While global functions are granted on the ALL DATABASES scale. --- ### Privileges supported by StarRocks Privileges granted to a user or role determine which operations the user or role can perform on certain objects. Privileges can be used to implement fine-grained access control to safeguard data security. This topic describes privileges provided by StarRocks on different objects and their meanings. Privileges are granted and revoked by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). The privileges that can be granted on an object are specific to the object type. For example, table privileges are different from database privileges. important The privileges described in this topic are available only from v3.0. The privilege framework and syntax in v3.0 are not backward compatible with those in earlier versions. After an upgrade to v3.0, most of your original privileges are still retained except those for specific operations. For the detailed differences, see [Upgrade notes](#upgrade-notes) at the end of this topic. #### Privilege list[​](#privilege-list "Direct link to Privilege list") This section describes privileges that are available on different objects. ##### SYSTEM[​](#system "Direct link to SYSTEM") | Privilege | Description | | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | NODE | Operates nodes, such as adding, deleting, or decommissioning nodes. To ensure cluster security, this privilege cannot be directly granted to users or roles. The `cluster_admin` role has this privilege. | | GRANT | Creates a user or role, alters a user or role, or grants privileges to a user or role. This privilege cannot be directly granted to users or roles. The `user_admin` role has this privilege. | | CREATE RESOURCE GROUP | Creates a resource group. | | CREATE RESOURCE | Creates resources for Spark Load jobs or external tables. | | CREATE EXTERNAL CATALOG | Creates an external catalog. | | PLUGIN | Installs or uninstalls a plugin. | | REPOSITORY | Creates, deletes, or views repositories. | | BLACKLIST | Creates, deletes, or displays SQL blacklists and BE Blacklist. | | FILE | Creates, deletes, or views files. | | OPERATE | Manages replicas, configuration items, variables, and transactions. | | CREATE GLOBAL FUNCTION | Creates a global UDF. | | CREATE STORAGE VOLUME | Creates a storage volume for a remote storage system. | | SECURITY | Creates or manages security integrations and group providers. | ##### RESOURCE GROUP[​](#resource-group "Direct link to RESOURCE GROUP") | Privilege | Description | | --------- | ------------------------------------------------- | | ALTER | Adds or deletes classifiers for a resource group. | | DROP | Deletes a resource group. | | ALL | Has all the above privileges on a resource group. | ##### RESOURCE[​](#resource "Direct link to RESOURCE") | Privilege | Description | | --------- | ------------------------------------------- | | USAGE | Uses a resource. | | ALTER | Alters a resource. | | DROP | Deletes a resource. | | ALL | Has all the above privileges on a resource. | ##### USER[​](#user "Direct link to USER") | Privilege | Description | | ----------- | ---------------------------------------------- | | IMPERSONATE | Allows user A to perform operations as user B. | ##### GLOBAL FUNCTION (Global UDFs)[​](#global-function-global-udfs "Direct link to GLOBAL FUNCTION (Global UDFs)") | Privilege | Description | | --------- | ------------------------------------------- | | USAGE | Uses a function in a query. | | DROP | Deletes a function. | | ALL | Has all the above privileges on a function. | ##### CATALOG[​](#catalog "Direct link to CATALOG") | Object | Privilege | Description | | -------------------------- | --------------- | ----------------------------------------------------- | | CATALOG (internal catalog) | USAGE | Uses the internal catalog (default\_catalog). | | CATALOG (internal catalog) | CREATE DATABASE | Creates databases in the internal catalog. | | CATALOG (internal catalog) | ALL | Has all the above privileges on the internal catalog. | | CATALOG (external catalog) | USAGE | Uses an external catalog. | | CATALOG (external catalog) | DROP | Deletes an external catalog. | | CATALOG (external catalog) | ALL | Has all the above privileges on the external catalog. | > Notes: StarRocks internal catalog cannot be deleted. ##### DATABASE[​](#database "Direct link to DATABASE") | Privilege | Description | | ------------------------ | --------------------------------------------------------------------------------- | | ALTER | Sets properties for a database, rename a database, or sets quotas for a database. | | DROP | Deletes a database. | | CREATE TABLE | Creates tables in a database. | | CREATE VIEW | Creates a view. | | CREATE FUNCTION | Creates a function. | | CREATE MATERIALIZED VIEW | Creates a materialized view. | | ALL | Has all the above privileges on a database. | ##### TABLE[​](#table "Direct link to TABLE") | Privilege | Description | | --------- | ------------------------------------------------------------------------------------------------ | | ALTER | Modifies a table or refreshes metadata in an external table. | | DROP | Drops a table. | | SELECT | Queries data in a table. | | INSERT | Inserts data into a table. | | UPDATE | Updates data in a table. | | EXPORT | Exports data from a StarRocks table. | | DELETE | Deletes data from a table based on the specified condition or deletes all the data from a table. | | ALL | Has all the above privileges on a table. | ##### VIEW[​](#view "Direct link to VIEW") | Privilege | Description | | --------- | --------------------------------------- | | SELECT | Queries data in a view. | | ALTER | Modifies the definition of a view. | | DROP | Deletes a logical view. | | ALL | Has all the above privileges on a view. | ##### MATERIALIZED VIEW[​](#materialized-view "Direct link to MATERIALIZED VIEW") | Privilege | Description | | --------- | ---------------------------------------------------- | | SELECT | Queries a materialized view to accelerate queries. | | ALTER | Changes a materialized view. | | REFRESH | Refreshes a materialized view. | | DROP | Deletes a materialized view. | | ALL | Has all the above privileges on a materialized view. | ##### FUNCTION (Database-level UDFs)[​](#function-database-level-udfs "Direct link to FUNCTION (Database-level UDFs)") | Privilege | Description | | --------- | ------------------------------------------- | | USAGE | Uses a function. | | DROP | Deletes a function. | | ALL | Has all the above privileges on a function. | ##### STORAGE VOLUME[​](#storage-volume "Direct link to STORAGE VOLUME") | Privilege | Description | | --------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | ALTER | Alters the credential properties, comment, or status (enabled) of a storage volume, and sets a storage volume as the default storage volume. | | DROP | Drops a storage volume. | | USAGE | Describes a storage volume. | | ALL | Has all the above privileges on a storage volume. | #### Upgrade notes[​](#upgrade-notes "Direct link to Upgrade notes") During an upgrade from v2.x to v3.0, some of your operations may be unable to perform due to the introduction of the new privilege system. The following table describes the changes before and after the upgrade. | **Operation** | **Commands involved** | **Before** | **After** | | --------------------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Change table | ALTER TABLE, CANCEL ALTER TABLE | Users who have the `LOAD_PRIV` privilege on a table or the database to which the table belongs can perform the `ALTER TABLE` and `CANCEL ALTER TABLE` operations. | You must have the ALTER privilege on the table to perform these two operations. | | Refresh external table | REFRESH EXTERNAL TABLE | Users who have the `LOAD_PRIV` privilege on an external table can refresh the external table. | You must have the ALTER privilege on the external table to perform this operation. | | Backup and restore | BACKUP, RESTORE | Users who have the `LOAD_PRIV` privilege on a database can back up and restore the database or any table in the database. | The administrator must grant backup and restore privileges to users again after the upgrade. | | Recover after deletion | RECOVER | Users who have the `ALTER_PRIV`, `CREATE_PRIV`, and `DROP_PRIV` privileges on the database and table can recover the database and table. | You must have the CREATE DATABASE privilege on the default\_catalog to recover the database. You must have the CREATE TABLE privilege on the database and the DROP privilege on the table. | | Create and change users | CREATE USER, ALTER USER | Users who have the `GRANT_PRIV` privilege on the database can create and change users. | You must have the `user_admin` role to create and change users. | | Grant and revoke privileges | GRANT, REVOKE | Users who have the `GRANT_PRIV` privilege on an object can grant privileges on the object to other users or roles. | After the upgrade, you can still grant the privileges you already have on that object to other users or roles after the upgrade.
In the new privilege system: - You must have the `user_admin` role to grant privileges to other users or roles.
- If your GRANT statement includes `WITH GRANT OPTION`, you can grant the privileges involved in the statement to other users or roles. | In v2.x, StarRocks does not fully implement role-based access control (RBAC). When you assign a role to a user, StarRocks directly grants all the privileges of the role to the user, instead of the role itself. Therefore, the user does not actually own the role. In v3.0, StarRocks renovates its privilege system. After an upgrade to v3.0, your original roles are retained but there is still no ownership between users and roles. If you want to use the new RBAC system, perform the GRANT operation to assign roles and privileges. --- ### Manage permissions with Apache Ranger [Apache Ranger](https://ranger.apache.org/) provides a centralized security management framework that allows users to customize access policies through a visual web page. This helps determine which roles can access which data and exercise fine-grained data access control for various components and services in the Hadoop ecosystem. Apache Ranger provides the following core modules: * **Ranger Admin**: the core module of Ranger with a built-in web page. Users can create and update security policies on this page or through a REST interface. Plugins of various components of the Hadoop ecosystem poll and pull these policies at a regular basis. * **Agent Plugin**: plugins of components embedded in the Hadoop ecosystem. These plugins pull security policies from Ranger Admin on a regular basis and store the policies in local files. When users access a component, the corresponding plugin assesses the request based on the configured security policy and sends the authentication results to the corresponding component. * **User Sync**: used to pull user and user group information, and synchronize the permission data of users and user groups to Ranger's database. In addition to the native RBAC privilege system, StarRocks v3.1.9 also supports access control through Apache Ranger. Currently, StarRocks supports: * Creates access policies, masking policies, and row-level filter policies through Apache Ranger. * Ranger audit logs. * **Ranger Servers that use Kerberos for authentication are not supported.** * **Ranger policy conditions that are evaluated by JavaScript are not supported**, such as the `expression` condition and the `accessed-after-expiry` condition of tag-based policies. See the note below. * You can register multiple StarRocks Services in the same Apache Ranger service to manage privileges in different StarRocks clusters. note StarRocks FE runs on JDK 17 or later, which does not include a JavaScript engine. When Ranger cannot evaluate a JavaScript condition, it treats the condition as satisfied and logs `failed to evaluate condition '': script engine not found` each time. For example, an allow item with an `expression` condition then grants access without checking the expression. Do not use these conditions in the policies that apply to StarRocks. When it loads policies, the FE also logs `failed to initialize condition 'accessed-after-expiry'`. This comes from the `EXPIRES_ON` policy of the tag service that Ranger Admin links to a service by default. The message has no effect as long as no resource has the `EXPIRES_ON` tag. Access to a resource with this tag is denied even before its expiry date. This topic describes the permission control methods and integration process of StarRocks and Apache Ranger. For information on how to create security policies on Ranger to manage data security, see the [Apache Ranger official website](https://ranger.apache.org/). From v3.5.0 onwards, StarRocks supports Group Provider to collect group information from external authentication systems for user group management. For more information, see [Authenticate User Groups](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). #### Permission control method[​](#permission-control-method "Direct link to Permission control method") StarRocks integrated with Apache Ranger provides the following permission control methods: * Create StarRocks Service in Ranger to implement permission control. When users access StarRocks internal tables, external tables, or other objects, access control is performed according to the access policies configured in StarRocks Service. * When users access an external data source, the external service (such as the Hive Service) on Apache Ranger can be reused for access control. StarRocks can match Ranger services with different External Catalogs and implements access control based on the Ranger service corresponding to the data source. After StarRocks is integrating with Apache Ranger, you can achieve the following access control patterns: * Use Apache Ranger to uniformly manage access to StarRocks internal tables, external tables, and all objects. * Use Apache Ranger to manage access to StarRocks internal tables and objects. For External Catalogs, reuse the policy of the corresponding external service on Ranger for access control. * Use Apache Ranger to manage access to External Catalogs by reusing the Service corresponding to the external data source. Use StarRocks native RBAC privilege system to manage access to StarRocks internal tables and objects. **Authentication process** 1. You can also use LDAP for user authentication, then use Ranger to synchronize LDAP users and configure access rules for them. StarRocks can also complete user login authentication through LDAP. 2. When users initiate a query, StarRocks parses the query statement, passes user information and required privileges to Apache Ranger. Ranger determines whether the user has the required privilege based on the access policy configured in the corresponding Service, and returns the authentication result to StarRocks. If the user has access, StarRocks returns the query data; if not, StarRocks returns an error. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Apache Ranger 2.1.0 or later has been installed. For the instructions on how to install Apache Ranger, see [Ranger quick start](https://ranger.apache.org/quick_start_guide.html). * All StarRocks FE machines have access to Apache Ranger. You can check this by running the following command on each FE machine: ```sql telnet ``` If `Connected to ` is displayed, the connection is successful. #### Integrate StarRocks Service with Ranger[​](#integrate-starrocks-service-with-ranger "Direct link to Integrate StarRocks Service with Ranger") ##### (Optional) Install ranger-starrocks-plugin[​](#optional-install-ranger-starrocks-plugin "Direct link to (Optional) Install ranger-starrocks-plugin") note The main purpose of this step is to use Ranger's resource name autocomplete feature. When authoring policies in Ranger Admin, users need to enter the name of the resources whose access need to be protected. To make it easier for users to enter the resource names, Ranger Admin provides the autocomplete feature, which looks up the available resources in the service that match the input entered so far and automatically completes the resource name. If you do not have the permissions to operate the Ranger cluster or do not need this feature, you can skip this step. Also, please notice that if you didn't install the ranger-starrocks-plugin, then you cannot use `test connection` when creating StarRocks service. However, that doesn't mean that you can not create the service successfully. 1. Create the `starrocks` folder in the Ranger Admin directory `ews/webapp/WEB-INF/classes/ranger-plugins`. ```sql mkdir {path-to-ranger}/ews/webapp/WEB-INF/classes/ranger-plugins/starrocks ``` 2. Download [plugin-starrocks/target/ranger-starrocks-plugin-3.0.0-SNAPSHOT.jar](https://www.starrocks.io/download/community) and [mysql-connector-j.jar](https://dev.mysql.com/downloads/connector/j/), and place them in the `starrocks` folder. 3. Restart Ranger Admin. * Ranger 0.5.x: ```sql ranger-admin restart ``` * Ranger 2.x and above: ```sql ./ews/ranger-admin-services.sh restart ``` ##### Configure StarRocks Service on Ranger Admin[​](#configure-starrocks-service-on-ranger-admin "Direct link to Configure StarRocks Service on Ranger Admin") note This step configures the StarRocks Service on Ranger so that users can perform access control on StarRocks objects through Ranger. 1. Copy [ranger-servicedef-starrocks.json](https://github.com/StarRocks/starrocks/blob/main/conf/ranger/ranger-servicedef-starrocks.json) to any directory of the StarRocks FE machine or Ranger machine. ```sql wget https://raw.githubusercontent.com/StarRocks/starrocks/main/conf/ranger/ranger-servicedef-starrocks.json ``` note If you do not need Ranger's autocomplete feature (which means you did not install the ranger-starrocks-plugin), you must set `implClass` in the .json file to empty: ```json "implClass": "", ``` If you need Ranger's autocomplete feature (which means you have installed the ranger-starrocks-plugin), you must set `implClass` in the .json file to `org.apache.ranger.services.starrocks.RangerServiceStarRocks`: ```json "implClass": "org.apache.ranger.services.starrocks.RangerServiceStarRocks", ``` 2. Add StarRocks Service by running the following command as a Ranger administrator. ```bash curl -u : \ -X POST -H "Accept: application/json" \ -H "Content-Type: application/json" http://:/service/plugins/definitions -d@ranger-servicedef-starrocks.json ``` 3. Access `http://:/login.jsp` to log in to the Apache Ranger page. The STARROCKS service appears on the page. ![home](/assets/images/ranger_home-c446eb652890149c921b41f81e301926.png) 4. Click the plus sign (`+`) after **STARROCKS** to configure StarRocks Service. ![service detail](/assets/images/ranger_service_details-df17ab63545b9b49e05f6d3c56477186.png) ![property](/assets/images/ranger_properties-d836dd82cdc123eafdaf4406cd1105fc.png) * `Service Name`: You must enter a service name. * `Display Name`: The name you want to display for the service under STARROCKS. If it is not specified, `Service Name` will be displayed. * `Username` and `Password`: FE username and password, used to auto-complete object names when creating policies. The two parameters do not affect the connectivity between StarRocks and Ranger. If you want to use auto-completion, configure at least one user with the `db_admin` role activated. * `jdbc.url`: Enter the StarRocks FE IP address and port. The following figure shows a configuration example. ![example](/assets/images/ranger_show_config-1112dfa7140bc9972ed4628a26a8ec09.png) The following figure shows the added service. ![added service](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAc8AAABxCAIAAAA8rb/NAAAeNUlEQVR4nOzdCXxTZb438OdkT5qtSZOugZaW7itQKAUKLes4oMg24yiXUWHuzMdRX2eucx2v+qqvjuOIMzLX915H9M4rgisqssjSUlm0QAuFtpS2dEv3Nc2+Nct5P+1TDjFt06QkRcr/++HD5/TkyXmenLS/PHnOc85hkCSJAAAABBJJkrTb3QYAAJjmcKcW0hYAAAKIGj+AtAUAgEBxHaqFtAUAgIBwOyoGaQsAAFMB0hYAAPxv9HQvSFsAAPCzMWfWQtoCAEDAwXxbAADws/FOGYO0BQCAwIKzGwAAwM88XAsB0hYAAAIIziUDAAA/89CxhaNkAAAQQHDmLgAATB04SgYAAH4z4bXCIW0BAGAqMEavam7ruh0tAQCAO5hr3zZGEU6todaPkbYIoUGTbqpaCAAA08KNsGXyBGM+DiMJAABwy7y4vyOkLQAATAVIWwAACBRq0BbObgAAgCkCaQsAAFMB0hYAAKYCpC0AAAQcQRCQtgAAMBUgbQEAYCqMfS6ZBw31DWUXy7RaretKkUg0b9682bNn+7VtAIA7ld1mN5lMBEEIhGOfWOXTplQDKtugbbwCDAZDKpUyWUyfNmsymew2+3iPMllMLpfrY0sn4Fvatra2Fp0sQggJBAKCIEiSxP9rtdqTJ09yOGyFYoZfmkU6SY1GYzAYWGyWUCDk8vz8sgOFRKoBldlsDpGGcLicCUuaTCaxSCwQCBAx+TrtNrtWqzUajUFBQUKh0NffOQD8zmF37Pton9lsRgilpaUtWrRo0pvq6e45dvwY3pQHLBZr5cqVCoXCm22q+lWFhYUarcZzMalUunLFSnGw2Jf2usMJiZd9S9uzZ88ihB74+QMisch1vU6r++jjj86cOfvggw/eSsuwq1VXL5ResNlufpTJ5fIli5fI5DKD3rB3314Pz01ISMjPz0cIlXxfUllViT/3Hn3kUYJ2M8+KTxZfr78++rl8Pj81NTUzIxNnn5fFMIPeUFRU1N3TTa1hMpnZ87LTM9Ldnj5myYU5C5OTk/EGzSbzB3s+wC3fvn07LuOwOw4eOtjT04Pfv02bNkmlUrvNfu78uerqatftx8XGLVq06I75fALT0eDgIJWParV68hsiUWFh4YRRi2ssKira+tBWBnOCTHM6nMeOH9Pr9RNuU6VSnSg8sWXLFl9a/AOuUetz2ur1eg6H4xa1CCGhSMhgMPR6fYuyxe0h3P/lC/hSqdSbKi6WXbx46aLbyt7e3i+/+nLzps0sFsvz050O53Ct6FrNNbzGbrcrlcqYWTE3mzTOKc0Gg+H8+fMajWbZsmXeFxvq8re0Hj121O3qljabreRcibJFec9P7qF+A8Yreebsme6e7oKCgqGX4HTi9Q6Hg3pRhw4doqJ2w4YNQzuTRIcPH3ZNbayhsaGnt2fL5i3QyQV3Op1OZzAaPBSIjo7u7e01mUwIIavVqtaoZTKZ522qNerRUSsWi9PS0giXDtT1+uvd3d0DAwMGvYEv4E/cVmLiSyX4PG5LEGN/6Q0NDe3o6Dh67Oh4T0xPT8/NzfW8cbvNjqOWIIjVq1bLQ+Vmk/lC6YXW1laSJMsuli0vWL5gwQJcWD2gxn1PFouVlZWFV4ZIQxBCSqXSbr85InO1+qpr2lLiZ8fPmjVr6H0atLYoW5qamxBCtbW12fOyg/hBXhYzGoxUgIaFhSUlJXE53N6+3kuXLpEk2dnZef78+cVLFiOERpfk8XgdHR1XrlwZenevX1coFKPHvp0O5+EjI6nKYDDuv/9+/LnV2dmJV/L5/NWrVvP5fLVaXfxtscFg0Ov19Q31Q51lAO5kVIdjTHPmzJk/f75Oqzvw9QEcuA67p/LYmOO/PC4vJTnF9dsqnUHv7u7GnaFJtn4U39LWrWPsavny5Y0NjTabjUb7wTwHEpGkk2xsaqysrJTJZJ6PpKk1I186OBzOjBkzaHQaj8fLX5Z/+PBhvOsZTAYVrO3t7ThtBXwBtRK7Wn0VL7BYrMHBwY6ODrPZPHrMWy6XR8dE4+WE+IT33n8PZ7RWp3VNW8/FTp85jfdJTEzM6lWr8Rs2Y+aM2NjYTz/9FDcmOTlZIpWMWVKhUEiCJcXfFiOEqqur3fYP6SSPfHOks7MTDzhs3LCRGkXq7evFC+Fh4TL50Oc5l8fNy8u7cP4CHo7wsJ8BuNPhqB36Yi0USiQSnLaT1tnVWVxcPPTl8haOoIwJd0/xH77Pfdvx8Hi8tPS08R5NS0t7/3/er6io8Jy2weJgvGA2m/d8uCc5KXnmzJmhoaGbt2z2viUWs6W9vX1oa8HBSYlJJedKEEJ1dXWZmZkenmUfhpdlIeN+GRldrKtr5OLr8+bNc32rgoODo6KicEu6e7olUsl4JePi4oKCgkhE0ml0t+qOfHOko6MDR+3mTZuFIiH1EDUyU99QPzAwkJiYqFAoZgzzckcB4C8Ws+Xz/Z8bjcbRD7W3t7/zzjvUj0lJSUuXLr3F6qioRST67vvv8F/ZLWKxJxilvHV+S1vP8BjihH1yBpORnp5eWVk59P5ZLOWXy8svlxMEoVAosrOzJxyRwerq6vBCclLyrFmzcNpevXp1dNq2trXio2dmk7mhsQGvDAkJcRvx9FDMNmijXpRU4j4wHR4ejn8PBgYGPJSk0WmRUZGjXwhJktSvEZ1O5/F4ro9GRkRKJVLVgAohpBpQfV/yPQ7l+Pj4uXPm8oJ43uwrAPxCo9GMGbWjtba2TmL7ISEhQbygltaW0VF79erVSWzQTWpq6uJFi0f6QCSaZA93nKFbakhgitKWdJI4CyYsmbswVygQnr9wnupCkiTZOmz58uXeTOmtulqFF+Li4rg8bnBwsFqtNhgMqn6VNOQHMYc367pGLBbfd+99bhv0UMxiseA1DAZj9DvE5YyMXRiNRuq46pglJ2SxWM6cOVOwvIBaQ6PT7rvvvgulF1znJNhsturq6rq6uo0bNgZLgn2uBoAAG+/Aj2epqamJCYnFxcV8AT+gUVtaWmrQGwoKCia8paNPcODeatraBm1VVVWNTY1Go5EYL0WIkWGL/v7+Dz74YCgpaLTo6OjsedljTEolUGpaampqand3t7JF2djYSB1APHny5KyYWXSG+9dtV319fQaDAY/89qv6kWooGfEclOpr1Xl5eW67AH8ADA4O4jUajebatWsZmRleFgsKGhnetdvtpJN0nWeGENIbRlouEon4fL6Hkp0dnXgXuXVyCYKIj4/HvfXr9dfj4uJmzLw5UMBis5YsWbIwZ2Fbe1trS2tjUyNuod1uP1l8ctOmTR52FAB+JJVKIyMjqe6tw+Gg/mzpdLpAcPMEh/jZ8ZPY/lBGE+jmuCqJvvvuO+rwDGUSEekWteXl5Xh6g9HkVVfdS35IW51W99VXX5ktZhyg4xUjSZJGo4lEIqfTaTAYcMXV1dUqlWr9+vWuJRsbGru6u/DgTlh4WFh4WE5OTk93z4GvD+D92NvbGx4R7qFJ16pHJn5ZLJYjR464PlRTU7N40WIa/WY7F+UuSk1LxZ8Z5ZfLL1++jBAqu1jmlrYeitHoNC6Xi/utyhZlTMwPZj40NIwMO0ilUg8luzq7Dh46iI/pPfLII65buHfdveHh4eoBNT4mdqLwxEMPPoQ/oi5dvIT3fM6CnJhhS5ctra6uxnOi+/v7HXaH508mAPyFyWKuW7eO+pGaM47H09auXeufajxG7SSMGbUIITxkMckWjhP4BEFMPm2dDueXX31psVhyF+YmJib6Osa8Z88ePMHClUqlwl8NCIKgzj8JDQsNlYfi2U52x7hn2uEm1dbVjvcoSZJKpXJW7KzRDzFZzHlz5125coUkSbvdbjaZxzw7YMxisbGxuM0lJSUR4RFsDhsXrq2ppT7eIyOHeqzjlbx85TJemPHDM/EIgsAfLStXrtz30T7caS06WYR/d5uVzf39/fhZVIc3JTkFp+3kPucBuCNQ3zJvRVRU1JhRGyCTSVtq7v3BQwctFsviRYtxv88nTY1NJpMJZ5CroXAZTp6qqioGgxEXF0cjaE3NTdQcfs8HypqamnDEhIaGrlm9hlpfU1NTWlaKJ2ONmbZ4ep1EIlGpVLhfPN65WKOLzc+eX11dTZKkXq//5NNP8Czarq6uxsZG/JQ5c+bgA1yjS/L5/BZlCzUonJo69p4UCAW5C3Px4b729vba2trExMSIiAicticKT+Tm5oaHh9tt9isVV0aeIhBMeFINAHek4SEFg8HQ2dV5K5sZGBjQ6XRCkdDPUTv+uLRvf5AkSeIRzLKysu7u7oT4hElEbV9f34nCE3Q6fcWKFW4PKRSKuNg4fNz/8jDXR7PnZXM4ni4+QA2ZpySnuMZlUlISTtvOzk4PZwEKBAIcoyaTycMhJrdiLDZr9arVx08cJ0nSbDa7vW1hoWFz58zFy55Lpqenh4WHjVdpenp6TW0NHoA+ffq0IkqRPS+7qanJYDDY7fYzZ864lS/ILxh/PwFwZ6COq9fU1LieuUAicswJtt58w6aG10wm04GvD8jlcqVS6ak83T/DcT7fl4wgCD6f39zUfOnSJalEml+Q72uVtkHbgQMHEELr1q0b8xI7BQUFOTk5brMXWCxWXl7e3Llzx2kWwlN0qS6w2/gpl8cNCQnBy/X19eO1TS6T44W2tjYPL2F0seiY6Ad/8WBYWJhbm3Nzc9evX+86eDpmSSaTuXz58glOtCPQ6lWr8SJJksdPHGeymBs3bhx9wphUIr133b2eR7cBCCgud+QvjiCI0V9hvccX8CUSCUKou7v7zNkz1L+zZ8+OvqaMQCCgJux7ECwOZrNHBvFMJpPnqOUH8YVCoa/NHj3vAn/nHuPcsOa2rkGTbsyt7P1wr8VqsdvtBEH8ctsvqZFH732x/4u+/r68vLwJzislkdFo1Oq0dDpdKBT6/dJngUIitUZtMVskEskEO+dGSZFY5DaR1lcOu0Ov1xuNRg6XIxTANcDAtKJRawqLCvG3SQ/EIvHKlSvdpniOp62trbCwcMLBXy6Xu2b1mtCwUF/aO4TFE0ZH3exRURnrY9ru3YvnV224f6M81KtzDVydPn26pqYmJSVlyZIlvj4XAHDXMpvNnq5vy2T42mWxDdoGBgY8nG/FZDGlEunkDn6weMIYRTiO1slfA4zNZhsMhsyMTF+j1ulwXrx0saamRhYig6gFAPiEO8yPG2SymJPotPqEIAhqTgHmW9pmZWYVnSyqqKzAs2K9mWOEZ9f29vbiSf5uE2wBAGBaGh2PvqVt3Oy4wcHB8svl+Fqr3uNyuEnJSdnzst3OoQIAgOln5LDYDy+a6Nu4LQAAAM/GO0oG99wFAAB/cuvCUpe4hbQFAICpAGkLAAD+M/5VaSBtAQAggGDcFgAA/G38ObEwbgsAAAHkesQM0hYAAPzBY8cW0hYAAALFbSrY2OeSsXg+X2QMAADuWh4uY0A9NHbaRoZ6deEyAAAAbmnb0TP29SFhJAEAAPzP7YqLvt1z11lcRJZeQPV15MBAwFp4ByAkEhSfQFuQSyzz+dYVAIDpx8s7rnqVtqRK5dz1JllVecutmg6GPmzOn3OcP0ecLqY98TsieOKbcwAAgFcjCc6dr0PUjkZeuex8a+ftbgUA4EfKrc87cdo6C4+TtdcC2aQ7GFlZ4Swuut2tAAD8uLjlLP7Ri5GE8yXUIrFwEf0XW1FUVEAaeKdob3d89CF57nv8E3muBBW436odAHCX8HLQ1qszd521NdQy/YEH7/aoRQhFRQ3tB0pTw+1sDADgDuFF39ZkurmsmBHQ1twxXPbDXT5DA0wPVqtVq9WazWYve2oTIgiCy+WKRCI22/1W/1NZ12BLm/7DLyxVdcjh8EtdiE7npCUItm5kzVT49DzfZoABAKYlq9Xa3d0tk8nCw8PpdLpftulwOPR6fXd3d1hYmGsITmVdgy1tfX/4k99y9kZllivXLFV1sr886zlwR3+WwNkNANzttFqtTCYTi8X+ir/hLiBdLBbLZDKtVnu76tJ/+IWfo5bicAxt3EeQtgDc7cxms0AgCMSWBQKB2Wy+XXVZquoCUdGkN37HpO3FNlVlp/p2twKAaYgkST/2NF3R6fTRc6GmrK5AdWwnu3E/pG15+8D6D7679e14tr+irbi+x9dnmQYdOW8XtWtMPj3rnxcaXz8JU4wBAP7kh7S12Bw1/QZ/NMb/SJKs01msdt8+hfRWe5/RGrBGAQDuOgRB+DYnYdDheKO4Zk9le6/Vvi5a+uo96Vc7NS8V1/Ra7WveO/3XtRkRIt7/OVF9uKGn12q/Z6b0jbWZESLuc99ULImRnWnuPd+u/s97s/7ybe2vcmJf+7bmNzmxs2XC14uvfdHYlyHhbc2csW1+LING2B3Od0rq91a01eksm+Jkz69ImSHhU20wWGz//k2FmMN4eU3GFxWtf/2+vk5nWSAXPFeQtHiW3LW1LQOGR/aXIYQe3V/2eE7cxswZ39Z3//Xs9e+6dVtmy59bmaIQB/XoLC8cr/ysvlfOZmxKCnt+Zdrfz9a9f7m112q/2qs7tn2p//Y2AOAuQhCEV1cTH8/Ra517Ktu/+EUOj0n/08lrfyq69vKatO3zop8urn1tTZpCHPR68bUrPZrPH1jAZTIe/7p836XmpwuSm9Wmz2sq18aFPp+fZLY5v27ur1UZHl8YGyvlr93z/YaEsNIdS1sGDI8drhhKxpy4d0rq3y9v+fvazGAea+ep2l99eYlKPa3Ztu2zCyEc5rNrMzs0xn89WrVvfVZ6RPD+itZ1n5T2/fs9DPrN3rqMz31pRcq6T0p/vzh+rkJ6uX1gw+cXP75/zlsywQdlzfd+8H3pb5f/7UyN1mov/9eleqt92+dlSfLWzRmKHr21z2h5Ki/BT7sdAOCt6upqpVLpuobJZGZnZ7PZ7NbW1sTExEBUyp2bxoqOQghZrzdbqmpvfYNuUUsQhM8jCWwGvddqr+/TCznMN+7NeiovQcbnJIeK5GxGVpREwGH+NCniH/fPSwkTc5l0MYepMo18H88KFb55X9ay2WH4x/+8N+vBuTGVnVoeg/bST9JmywQrEsKfXTL77QtNQ4+WNv/v/KQlsfLUcPGff5rx6NwYu8OJEGrXmn+2rySIQf/vTfOCWAw2Y+ijorpba3M4fp07++zDi0lEtgwYLrcP4H8kSWZFShBCyWGiCBF3X3nLjrTIzMjgIBZjx8JYpXHwvLKfy2QoNaZ2jXmWlH9o2+Jls0NjpIKZwTw5n5MVJbn1nQ4A8N5777134sQJt5VWq9XpdO7fv/+xxx775ptv/F4pUxEueeQBvBzy24cZUrHfq8B869uuiA97Iz9xV0n9I4crMiS8Py5NjJf/4J46PBbjlaLqc52aGBFXY7FFi7l4fX6MzLVYZuTQ6+nUm9JkAtaNA5SJcpHSONhvtPZa7bNCRoYOwoTczVkjJ269f7VjTkjQxW6txmyT8emhQs6nG+a+e6HpT+eb5GzGEzmzkkJFey8p91/rxOU/+8XCCCGXqrRBbTzdodld1UGt6dCan8hLIBHafqAcD328sCIZiXzaJQDcLfR6/alTp5qbm41GY1hY2Pz581NSUvxbxblz59577z3cE6SYTKZXXnklOzt76dKlR48eveeee/xVneTRn7OiFfQQCY3F4mamDnVC2azQl/7NodZaaus1+w54s5HRIwbj8S1tu/Tm1Unh2xfG9egs715o+PlX5cromzFqd5JbPrnwWHbMu1uyWXT6a0XVWssgfkjA+UFFzOGEDQliXx8wOkmSNrxzO3VmOZsh5rIQQr16Cy7ZrjF9XK58cunQl/pHUyNf+2n6un9+99zRync2zddYBhVi3v5ti/QW24naru3fVKaFif5jZep/rEylKjJa7dSyQsh5MSbuybyRbyKVHWqFJKhLa3oyL+GFVakNffoXTlz9tyMVhx7J82mfAHA3KCoqevvtt+12+/r16zUazd69ez/55JOUlJSXXnrJj/NnSZIcHbW7du3icDhvv/32H/7wh/nz5/urLoSQ7uvjEW+9TBcOtZ8hu3F7MEmwjcnU/+1dP1aEQ9m3kYRj17p++Vlpt84sE7AzI4b6pzQC0Qmi12pv15jMVnuv1R4l4tEIWmWH+v9VtJnsTg+hvzQutE5nebekwWJzXO/VvfxtzfasGQwa8Whq5M6z19u1pl695S+nanoMVtz/jRJxmXT639Zmflbfe7i6Xaky5P7P2SvtA3w2c44ieLgxhFsVNNrQmla10TToWJsY8V9lysvtAxab41B1++ZPLtAR8efimlcLq02D9hhpUJJMQB8e9qXTiS69pc9gmeR+BWB6OXbs2MGDB/Pz8zdu3KhUKk+fPo3XL1269JlnnnE7g8uPcNTabLbTp09HRERkZWUF+/Xi/fZ+NXI4Bju6et/6h+6bIsOZcz0v/9Wh1ZF2h2PgVl8UMcx1jW992wfmzCxtVyW9fRIhJGcz3v1JuoDDjJMLMiS8tP8qPvkvuS8sinv0SAU6UjEnJOhXc2e+UtK4Prl7zBwcTk/eoZ/Pf/zQlT+erkMIPZapeGzxUB/22RXJfzxSkfZ/ixFCW5PDn8yLv9H6of+TwkQvL5n9LwevXH9ixXO5sfl7Ri4I+cScGTkzQ9yq4DLpm+JkGz6/+OelCTty454aMBYMl18gF/z3ukwhl/ncypTffHVJ8eZxhFCCkPOP++cihLKjpH8/3xT/9yL1s2t928EATDuFhYWnTp16/fXXd+7caTKZzp07h9fb7faMjAy5XP7ss8++8cYbPB7Pv/WaTKbnn39erVa3tLRERES8+eabcrncv1WMcJLISZJO59A/hxM5nQGpxde05XOYu7cs+JvVZrE7xBwWngAg5bFP/boAF5gTJflVTpzdSYq4TITQ9gVxQg6zID6M2sJchcQ1whbPkl9+cpXaPChgMajpBBIe+x+b579lc5Ak4rFGRnX/vDaTetbjSxIeXzKUy79flvTbxfF6i43PZnKYY5+gsnvLgt03ln+9aPaOhXF6qw2PVyCE4mXCoh35WsugkySDuWwc6PNnSut+v8anPQPAdNXY2JiVlcXlcjMyMnbt2kWtDwkJiYyMFAgENpvNaDT6PW1379791FNPffzxxzab7bXXXnME9MSwW+Pl0O1kzm7gs5khQRzXuVaugtgMHLUIIRGXOVan1l0wlzV6a1wmnYpaD9gMegifM17UjkanEVTUYgSBxFyWhMf2pqkA3G127NjR0NDw6aefrl27ds2akV6IVCp94YUXNBrNc8899+STT8pkMr/Xq1Kp3nnnHZPJ9Oabb1ZVVZWXl/u9ikBzG0yAKy4CADyh0+nPPPPMzp07d+3a9bOf/Wzr1q1arTY8PLysrGzXrl2/+93v/DgHVigUajQasViMENq2bZtOp4uOjhaLxbW1tatWrfJXLZ4EsssFaQsAmACdTn/66adLS0t3797d2dlJkiSLxVqwYMGrr77q317tww8//OKLL+p0OteVBEEsXLjQ77PNMOegjRUZFrLjIcRgIBqNNyedLuDbVX67ABbu23p7XzIiOJhU36i7rRVu34DwfriBCIaTIMD0R6PRcoYFtJaMjIy33noroFW40R08Jly3GtFu9mmdRpP2kPsZFhPyMHRLPeRF3zZ6FlJfwouOj/fRH9qGIiJ8bcq00t7u+HjfzR9jY29nYwAAt0B36KTu0MlA14IDd+K0pa1a47g8krbkue/tN+41CzDinnW3uwkAgNtvwu7txHMSiJyFRHpGANo2HRDzF9Cy5tzuVgAAfhSI8Sc20Wg0r2aA0f7X74mMTL+2ajogMrPov3n8drcCAHBn8GpOAhEsob/4CnnqW+f5ElRfd5ff05uQSlF8Am3+QmJZ/u1uCwDgx8VD99aHGWDEsnw65AsA0w5BEA6HIxC3C3M4HG7pM5V1ITo9gLcm8/0l3DF3gQQABAiXy9Xr9YHYsl6v53K5t6suTloAbwjgYePjdW8hbQG424lEor6+Po1G48drETgcDo1G09fXJxKJblddgq0bJ9ED9QqdPrTx8Y0ZuGNMWWhu64oMlfq7dQCAHy+r1arVas1ms5cXxp4QQRBcLlckErHZ7NtY12BLm/7DLyxVdX4bUqDTOWkJgq0bWTMV4xXp6FHFKMJHvzpIWwAA8CecttQJuxQYSQAAgIBwG0+AtAUAgEBxDVxIWwAACCAqcCFtAQAgsHDgQtoCAEDA+XzPXQAAAJMDaQsAAFNh7OskdPSoprwlAAAwnXl1Y14AAAC3CEYSAABgKkDaAgDAVIC0BQCAqQBpCwAAUwHSFgAApgKkLQAATAVIWwAAmAqQtgAAMBX+fwAAAP//mYeA7H9mpYUAAAAASUVORK5CYII=) 5. Click **Test connection** to test the connectivity, and save it after the connection is successful. If you didn't install ranger-starrocks-plugin, then you can skip test connection and create directly. ##### Enable Ranger on the FE nodes[​](#enable-ranger-on-the-fe-nodes "Direct link to Enable Ranger on the FE nodes") 1. On each FE machine of the StarRocks cluster, create [ranger-starrocks-security.xml](https://github.com/StarRocks/ranger/blob/master/plugin-starrocks/conf/ranger-starrocks-security.xml) in the `fe/conf` folder and copy the content. You must modify the following two parameters and save the modifications: * `ranger.plugin.starrocks.service.name`: Change to the name of the StarRocks Service you created in Step 4. * `ranger.plugin.starrocks.policy.rest the url`: Change to the address of the Ranger Admin. If you need to modify other configurations, refer to official documentation of Apache Ranger. For example, you can modify `ranger.plugin.starrocks.policy.pollIntervalMs` to change the interval for pulling policy changes. ```sql vim ranger-starrocks-security.xml ... ranger.plugin.starrocks.service.name starrocks -- Change it to the StarRocks Service name. Name of the Ranger service containing policies for this StarRocks instance ... ... ranger.plugin.starrocks.policy.rest.url http://localhost:6080 -- Change it to Ranger Admin address. URL to Ranger Admin ... ``` 2. (Optional) If you want to use the Audit Log service of Ranger, you need to create the [ranger-starrocks-audit.xml](https://github.com/StarRocks/starrocks/blob/main/conf/ranger/ranger-starrocks-audit.xml) file in the `fe/conf` folder of each FE machine. Copy the content, **replace `solr_url` in `xasecure.audit.solr.solr_url` with your own `solr_url`**, and save the file. StarRocks supports two destinations for Ranger audit logs, and you can use only one of them at a time: * Solr: Use the `xasecure.audit.solr.*` properties that the template uses. Configuring Solr with `xasecure.audit.destination.solr` is not supported. * Local file: Set `xasecure.audit.destination.file` to `true` and `xasecure.audit.destination.file.dir` to the directory of the audit files. Once any `xasecure.audit.destination.*` property is enabled, Ranger ignores all the `xasecure.audit..*` properties, including the Solr ones. This also applies to `ranger-hive-audit.xml`. 3. Add the configuration `access_control = ranger` to all FE configuration files. ```sql vim fe.conf access_control=ranger ``` 4. Restart all FE machines. ```sql -- Switch to the FE folder. cd.. bin/stop_fe.sh bin/start_fe.sh ``` #### Reuse other services to control access to external tables[​](#reuse-other-services-to-control-access-to-external-tables "Direct link to Reuse other services to control access to external tables") For External Catalog, you can reuse external services (such as Hive Service) for access control. StarRocks supports matching different Ranger external services for different Catalogs. When users access an external table, the system implements access control based on the access policy of the Ranger Service corresponding to the external table. The user permissions are consistent with the Ranger user with the same name. 1. Copy Hive's Ranger configuration files [ranger-hive-security.xml](https://github.com/StarRocks/ranger/blob/master/hive-agent/conf/ranger-hive-security.xml) and [ranger-hive-audit.xml](https://github.com/StarRocks/ranger/blob/master/hive-agent/conf/ranger-hive-audit.xml) to the `fe/conf` file of all FE machines. Make sure Ranger's IP and port are correct. 2. Restart all FE machines. 3. Configure External Catalog. * When you create an External Catalog, add the property `"ranger.plugin.hive.service.name"`. ```sql CREATE EXTERNAL CATALOG hive_catalog_1 PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "ranger.plugin.hive.service.name" = "" ) ``` * You can also add this property to an existing External Catalog. ```sql ALTER CATALOG hive_catalog_1 SET ("ranger.plugin.hive.service.name" = ""); ``` ​ This operation changes the authentication method of an existing Catalog to Ranger-based authentication. #### What to do next[​](#what-to-do-next "Direct link to What to do next") After adding a StarRocks Service, you can click the service to create access control policies for the service and assign different permissions to different users or user groups. When users access StarRocks data, access control will be implemented based on these policies. --- ### Manage user privileges This topic describes how to manage users, roles, and privileges in StarRocks. StarRocks employs both role-based access control (RBAC) and identity-based access control (IBAC) to manage privileges within a StarRocks cluster, allowing cluster administrators to easily restrict privileges within the cluster on different granular levels. Within a StarRocks cluster, privileges can be granted to users or roles. A role is a collection of privileges that can be assigned to users or other roles in the cluster as needed. A user can be granted one or more roles, which determine their permissions on different objects. #### View privilege and role information[​](#view-privilege-and-role-information "Direct link to View privilege and role information") Users with the system-defined role `user_admin` can view all the user and role information within the StarRocks cluster. ##### View privilege information[​](#view-privilege-information "Direct link to View privilege information") You can view the privileges granted to a user or a role using [SHOW GRANTS](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_GRANTS.md). * View the privileges of the current user. ```sql SHOW GRANTS; ``` > **NOTE** > > Any user can view their own privileges without needing any privileges. * View the privileges of a specific user. The following example shows the privileges of the user `jack`: ```sql SHOW GRANTS FOR jack@'172.10.1.10'; ``` * View the privileges of a specific role. The following example shows the privileges of the role `example_role`: ```sql SHOW GRANTS FOR ROLE example_role; ``` ##### View roles[​](#view-roles "Direct link to View roles") You can view all the roles within the StarRocks cluster using [SHOW ROLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_ROLES.md). ```sql SHOW ROLES; ``` #### Manage roles[​](#manage-roles "Direct link to Manage roles") Users with the system-defined role `user_admin` can create, grant, revoke, or drop roles in StarRocks. ##### Create a role[​](#create-a-role "Direct link to Create a role") You can create a role using [CREATE ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_ROLE.md). By default, a user can have a maximum of 64 roles. You can adjust this setting by using the FE dynamic parameter `privilege_max_total_roles_per_user`. A role can have a maximum of 16 inheritance levels. You can adjust this setting by using the FE dynamic parameter `privilege_max_role_depth`. The following example creates the role `example_role`: ```sql CREATE ROLE example_role; ``` ##### Grant a role[​](#grant-a-role "Direct link to Grant a role") You can grant roles to a user or another role using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md). * Grant a role to a user. The following example grants the role `example_role` to the user `jack`: ```sql GRANT example_role TO USER jack@'172.10.1.10'; ``` * Grant a role to another role. The following example grants the role `example_role` to the role `test_role`: ```sql GRANT example_role TO ROLE test_role; ``` * Grant a role to a user group. You can identify user groups from external authentication systems via [Group Provider](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). The following example grants the role `example_role` to the user group `analysts`: ```sql GRANT example_role TO EXTERNAL GROUP analysts; ``` ##### Alter the default role of a user[​](#alter-the-default-role-of-a-user "Direct link to Alter the default role of a user") The default role of a user is automatically activated when the user connects to StarRocks. For instructions on how to enable all (default and granted) roles for a user after connection, see [Enable all roles](#enable-all-roles). You can set the default role of a user using [SET DEFAULT ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_DEFAULT_ROLE.md) or [ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md). Both of the following examples set the default role of `jack` to `db1_admin`. Note that `db1_admin` must have been assigned to `jack`. * Set the default role using SET DEFAULT ROLE: ```sql SET DEFAULT ROLE 'db1_admin' TO jack@'172.10.1.10'; ``` * Set the default role using ALTER USER: ```sql ALTER USER jack@'172.10.1.10' DEFAULT ROLE 'db1_admin'; ``` ##### Revoke a role[​](#revoke-a-role "Direct link to Revoke a role") You can revoke roles from a user or another role using [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). > **NOTE** > > You cannot revoke the system-defined default role `PUBLIC` from a user. * Revoke a role from a user. The following example revokes the role `example_role` from the user `jack`: ```sql REVOKE example_role FROM USER jack@'172.10.1.10'; ``` * Revoke a role from another role. The following example revokes the role `example_role` from the role `test_role`: ```sql REVOKE example_role FROM ROLE test_role; ``` ##### Drop a role[​](#drop-a-role "Direct link to Drop a role") You can drop a role using [DROP ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/DROP_ROLE.md). The following example drops the role `example_role`: ```sql DROP ROLE example_role; ``` > **CAUTION** > > System-defined roles cannot be dropped. ##### Enable all roles[​](#enable-all-roles "Direct link to Enable all roles") The default roles of a user are roles that are automatically activated each time the user connects to the StarRocks cluster. If you want to enable all the roles (default and granted roles) for all StarRocks users when they connect to the StarRocks cluster, you can perform the following operation. This operation requires the system privilege OPERATE. ```sql SET GLOBAL activate_all_roles_on_login = TRUE; ``` You can also use SET ROLE to activate the roles assigned to you. For example, user `jack@'172.10.1.10'` has roles `db_admin` and `user_admin` but they are not default roles of the user and are not automatically activated when the user connects to StarRocks. If `jack@'172.10.1.10'` needs to activate `db_admin` and `user_admin`, he can run `SET ROLE db_admin, user_admin;`. Note that SET ROLE overwrites original roles. If you want to enable all your roles, run SET ROLE ALL. #### Manage privileges[​](#manage-privileges "Direct link to Manage privileges") Users with the system-defined role `user_admin` can grant or revoke privileges in StarRocks. ##### Grant privileges[​](#grant-privileges "Direct link to Grant privileges") You can grant privileges to a user or a role using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md). * Grant a privilege to a user. The following example grants the SELECT privilege on the table `sr_member` to the user `jack`, and allows `jack` to grant this privilege to other users or roles (by specifying WITH GRANT OPTION in the SQL): ```sql GRANT SELECT ON TABLE sr_member TO USER jack@'172.10.1.10' WITH GRANT OPTION; ``` * Grant a privilege to a role. The following example grants the SELECT privilege on the table `sr_member` to the role `example_role`: ```sql GRANT SELECT ON TABLE sr_member TO ROLE example_role; ``` ##### Revoke privileges[​](#revoke-privileges "Direct link to Revoke privileges") You can revoke privileges from a user or a role using [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). * Revoke a privilege from a user. The following example revokes the SELECT privilege on the table `sr_member` from the user `jack`, and disallows `jack` to grant this privilege to other users or roles: ```sql REVOKE SELECT ON TABLE sr_member FROM USER jack@'172.10.1.10'; ``` * Revoke a privilege from a role. The following example revokes the SELECT privilege on the table `sr_member` from the role `example_role`: ```sql REVOKE SELECT ON TABLE sr_member FROM ROLE example_role; ``` #### Best practices[​](#best-practices "Direct link to Best practices") ##### Multi-service access control[​](#multi-service-access-control "Direct link to Multi-service access control") Usually, a company-owned StarRocks cluster is managed by a sole service provider and maintains multiple lines of business (LOBs), each of which uses one or more databases. As shown below, a StarRocks cluster's users include members from the service provider and two LOBs (A and B). Each LOB is operated by two roles - analysts and executives. Analysts generate and analyze business statements, and executives query the statements. ![User Privileges](/assets/images/user_privilege_1-e144a82f732892eea1d207996c34a990.png) LOB A independently manages the database `DB_A`, and LOB B the database `DB_B`. LOB A and LOB B use different tables in `DB_C`. `DB_PUBLIC` can be accessed by all members of both LOBs. ![User Privileges](/assets/images/user_privilege_2-053e5c148bb7de7aa89561c05ac3222b.png) Because different members perform different operations on different databases and tables, we recommend you create roles in accordance with their services and positions, apply only the necessary privileges to each role, and assign these roles to corresponding members. As shown below: ![User Privileges](/assets/images/user_privilege_3-abd736675549c8da604cd515ea0c080f.png) 1. Assign the system-defined roles `db_admin`, `user_admin`, and `cluster_admin` to cluster maintainers, set `db_admin` and `user_admin` as their default roles for daily maintenance, and manually activate the role `cluster_admin` when they need to operate the nodes of the cluster. Example: ```sql GRANT db_admin, user_admin, cluster_admin TO USER user_platform; ALTER USER user_platform DEFAULT ROLE db_admin, user_admin; ``` 2. Create users for each member within the LOBs, and set complex passwords for each user. 3. Create roles for each position within the LOBs, and apply the corresponding privileges to each role. For the director of each LOB, grant their role the maximum collection of the privileges their LOBs need, and the corresponding GRANT privileges (by specifying WITH GRANT OPTION in the statement). Therefore, they can assign these privileges to the members of their LOB. Set the role as their default role if their daily work requires it. Example: ```sql GRANT SELECT, ALTER, INSERT, UPDATE, DELETE ON ALL TABLES IN DATABASE DB_A TO ROLE linea_admin WITH GRANT OPTION; GRANT SELECT, ALTER, INSERT, UPDATE, DELETE ON TABLE TABLE_C1, TABLE_C2, TABLE_C3 TO ROLE linea_admin WITH GRANT OPTION; GRANT linea_admin TO USER user_linea_admin; ALTER USER user_linea_admin DEFAULT ROLE linea_admin; ``` For analysts and executives, assign them the role with the corresponding privileges. Example: ```sql GRANT SELECT ON ALL TABLES IN DATABASE DB_A TO ROLE linea_query; GRANT SELECT ON TABLE TABLE_C1, TABLE_C2, TABLE_C3 TO ROLE linea_query; GRANT linea_query TO USER user_linea_salesa; GRANT linea_query TO USER user_linea_salesb; ALTER USER user_linea_salesa DEFAULT ROLE linea_query; ALTER USER user_linea_salesb DEFAULT ROLE linea_query; ``` 4. For the database `DB_PUBLIC`, which can be accessed by all cluster users, grant the SELECT privilege on `DB_PUBLIC` to the system-defined role `public`. Example: ```sql GRANT SELECT ON ALL TABLES IN DATABASE DB_PUBLIC TO ROLE public; ``` You can assign roles to others to achieve role inheritance in complicated scenarios. For example, if analysts require privileges to write into and query tables in `DB_PUBLIC`, and executives can only query these tables, you can create roles `public_analysis` and `public_sales`, apply relevant privileges to the roles, and assign them to the original roles of analysts and executives respectively. Example: ```sql CREATE ROLE public_analysis; CREATE ROLE public_sales; GRANT SELECT, ALTER, INSERT, UPDATE, DELETE ON ALL TABLES IN DATABASE DB_PUBLIC TO ROLE public_analysis; GRANT SELECT ON ALL TABLES IN DATABASE DB_PUBLIC TO ROLE public_sales; GRANT public_analysis TO ROLE linea_analysis; GRANT public_analysis TO ROLE lineb_analysis; GRANT public_sales TO ROLE linea_query; GRANT public_sales TO ROLE lineb_query; ``` ##### Customize roles based on scenarios[​](#customize-roles-based-on-scenarios "Direct link to Customize roles based on scenarios") We recommend you customize roles to manage privileges and users. The following examples classify a few combinations of privileges for some common scenarios. ###### Grant global read-only privileges on StarRocks tables[​](#grant-global-read-only-privileges-on-starrocks-tables "Direct link to Grant global read-only privileges on StarRocks tables") ```sql -- Create a role. CREATE ROLE read_only; -- Grant the USAGE privilege on all catalogs to the role. GRANT USAGE ON ALL CATALOGS TO ROLE read_only; -- Grant the privilege to query all tables to the role. GRANT SELECT ON ALL TABLES IN ALL DATABASES TO ROLE read_only; -- Grant the privilege to query all views to the role. GRANT SELECT ON ALL VIEWS IN ALL DATABASES TO ROLE read_only; -- Grant the privilege to query all materialized views and the privilege to accelerate queries with them to the role. GRANT SELECT ON ALL MATERIALIZED VIEWS IN ALL DATABASES TO ROLE read_only; ``` And you can further grant the privilege to use UDFs in queries: ```sql -- Grant the USAGE privilege on all database-level UDF to the role. GRANT USAGE ON ALL FUNCTIONS IN ALL DATABASES TO ROLE read_only; -- Grant the USAGE privilege on global UDF to the role. GRANT USAGE ON ALL GLOBAL FUNCTIONS TO ROLE read_only; ``` ###### Grant global write privileges on StarRocks tables[​](#grant-global-write-privileges-on-starrocks-tables "Direct link to Grant global write privileges on StarRocks tables") ```sql -- Create a role. CREATE ROLE write_only; -- Grant the USAGE privilege on all catalogs to the role. GRANT USAGE ON ALL CATALOGS TO ROLE write_only; -- Grant the INSERT and UPDATE privileges on all tables to the role. GRANT INSERT, UPDATE ON ALL TABLES IN ALL DATABASES TO ROLE write_only; -- Grant the REFRESH privilege on all materialized views to the role. GRANT REFRESH ON ALL MATERIALIZED VIEWS IN ALL DATABASES TO ROLE write_only; ``` ###### Grant read-only privileges on a specific external catalog[​](#grant-read-only-privileges-on-a-specific-external-catalog "Direct link to Grant read-only privileges on a specific external catalog") ```sql -- Create a role. CREATE ROLE read_catalog_only; -- Grant the USAGE privilege on the destination catalog to the role. GRANT USAGE ON CATALOG hive_catalog TO ROLE read_catalog_only; -- Switch to the corresponding catalog. SET CATALOG hive_catalog; -- Grant the privileges to query all tables and all views in the external catalog. GRANT SELECT ON ALL TABLES IN ALL DATABASES TO ROLE read_catalog_only; ``` tip For views in external catalogs, you can query only Hive table views (since v3.1). ###### Grant write-only privileges on a specific external catalog[​](#grant-write-only-privileges-on-a-specific-external-catalog "Direct link to Grant write-only privileges on a specific external catalog") You can only write data into Iceberg tables (since v3.1) and Hive tables (since v3.2). ```sql -- Create a role. CREATE ROLE write_catalog_only; -- Grant the USAGE privilege on the destination catalog to the role. GRANT USAGE ON CATALOG iceberg_catalog TO ROLE read_catalog_only; -- Switch to the corresponding catalog. SET CATALOG iceberg_catalog; -- Grant the privilege to write data into Iceberg tables. GRANT INSERT ON ALL TABLES IN ALL DATABASES TO ROLE write_catalog_only; ``` ###### Grant admin privileges on a specific database[​](#grant-admin-privileges-on-a-specific-database "Direct link to Grant admin privileges on a specific database") ```sql -- Create a role. CREATE ROLE db1_admin; -- Grant ALL privileges on the destination database to the role. This role can create tables, views, materialized views, and UDFs in this database. And it also can drop or modify this database. GRANT ALL ON DATABASE db1 TO ROLE db1_admin; -- Switch to the corresponding catalog. SET CATALOG iceberg_catalog; -- Grant all privileges on tables, views, materialized views, and UDFs in this database to the role. GRANT ALL ON ALL TABLES IN DATABASE db1 TO ROLE db1_admin; GRANT ALL ON ALL VIEWS IN DATABASE db1 TO ROLE db1_admin; GRANT ALL ON ALL MATERIALIZED VIEWS IN DATABASE db1 TO ROLE db1_admin; GRANT ALL ON ALL FUNCTIONS IN DATABASE db1 TO ROLE db1_admin; ``` ###### Grant privileges to perform backup and restore operations on global, database, table, and partition levels[​](#grant-privileges-to-perform-backup-and-restore-operations-on-global-database-table-and-partition-levels "Direct link to Grant privileges to perform backup and restore operations on global, database, table, and partition levels") * Grant privileges to perform global backup and restore operations: The privileges to perform global backup and restore operations allow the role to back up and restore any database, table, or partition. It requires the REPOSITORY privilege on the SYSTEM level, the privileges to create databases in the default catalog, to create tables in any database, and to load and export data on any table. ```sql -- Create a role. CREATE ROLE recover; -- Grant the REPOSITORY privilege on the SYSTEM level. GRANT REPOSITORY ON SYSTEM TO ROLE recover; -- Grant the privilege to create databases in the default catalog. GRANT CREATE DATABASE ON CATALOG default_catalog TO ROLE recover; -- Grant the privilege to create tables in any database. GRANT CREATE TABLE ON ALL DATABASES TO ROLE recover; -- Grant the privilege to load and export data on any table. GRANT INSERT, EXPORT ON ALL TABLES IN ALL DATABASES TO ROLE recover; ``` * Grant the privileges to perform database-level backup and restore operations: The privileges to perform database-level backup and restore operations require the REPOSITORY privilege on the SYSTEM level, the privilege to create databases in the default catalog, the privilege to create tables in any database, the privilege to load data into any table, and the privilege export data from any table in the database to be backed up. ```sql -- Create a role. CREATE ROLE recover_db; -- Grant the REPOSITORY privilege on the SYSTEM level. GRANT REPOSITORY ON SYSTEM TO ROLE recover_db; -- Grant the privilege to create databases. GRANT CREATE DATABASE ON CATALOG default_catalog TO ROLE recover_db; -- Grant the privilege to create tables. GRANT CREATE TABLE ON ALL DATABASES TO ROLE recover_db; -- Grant the privilege to load data into any table. GRANT INSERT ON ALL TABLES IN ALL DATABASES TO ROLE recover_db; -- Grant the privilege to export data from any table in the database to be backed up. GRANT EXPORT ON ALL TABLES IN DATABASE TO ROLE recover_db; ``` * Grant the privileges to perform table-level backup and restore operations: The privileges to perform table-level backup and restore operations require the REPOSITORY privilege on the SYSTEM level, the privilege to create tables in corresponding databases, the privilege to load data into any table in the database, and the privilege to export data from the table to be backed up. ```sql -- Create a role. CREATE ROLE recover_tbl; -- Grant the REPOSITORY privilege on the SYSTEM level. GRANT REPOSITORY ON SYSTEM TO ROLE recover_tbl; -- Grant the privilege to create tables in corresponding databases. GRANT CREATE TABLE ON DATABASE TO ROLE recover_tbl; -- Grant the privilege to load data into any table in a database. GRANT INSERT ON ALL TABLES IN DATABASE TO ROLE recover_db; -- Grant the privilege to export data from the table you want to back up. GRANT EXPORT ON TABLE TO ROLE recover_tbl; ``` * Grant the privileges to perform partition-level backup and restore operations: The privileges to perform partition-level backup and restore operations require the REPOSITORY privilege on the SYSTEM level, and the privilege to load and export data on the corresponding table. ```sql -- Create a role. CREATE ROLE recover_par; -- Grant the REPOSITORY privilege on the SYSTEM level. GRANT REPOSITORY ON SYSTEM TO ROLE recover_par; -- Grant the privilege to load and export data on the corresponding table. GRANT INSERT, EXPORT ON TABLE TO ROLE recover_par; ``` --- ### Overview of privileges This topic describes the basic concepts of StarRocks' privilege system. Privileges determine which users can perform which operations on which objects, so that you can more securely manage data and resources in a fine-grained manner. > NOTE: The privileges described in this topic are available only from v3.0. The privilege framework and syntax in v3.0 are not backward compatible with those in earlier versions. After an upgrade to v3.0, most of your original privileges are still retained except those for specific operations. For the detailed differences, see \[Upgrade notes] in [Privileges supported in StarRocks](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md). StarRocks adopts two privilege models: * Role-based access control (RBAC): Privileges are assigned to roles, which are then assigned to users. In this case, privileges are passed to users through roles. * Identity-based access control (IBAC): Privileges are directly assigned to user identities. Therefore, the maximum privilege scope of each user identity is the union of its own privileges and the privileges of the roles assigned to this user identity. **Basic concepts** for understanding StarRocks' privilege system: * **Object**: An entity to which access can be granted. Unless allowed by a grant, access is denied. Examples of objects include CATALOG, DATABASE, TABLE, and VIEW. For more information, see [Privileges supported in StarRocks](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md). * **Privilege**: A defined level of access to an object. Multiple privileges can be used to control the granularity of access granted on an object. Privileges are object-specific. Different objects may have different privileges. Examples of privileges include SELECT, ALTER, and DROP. * **User identity**: the unique identity of a user and also the entity to which privileges can be granted. User identity is represented as `username@'userhost'`, consisting of username and the IP from which the user logs in. Use identity simplifies attribute configuration. User identities that share the same user name share the same attribute. If you configure an attribute for a username, this attribute takes effect on all user identities that share this username. * **Role**: An entity to which privileges can be granted. Roles are an abstract collection of privileges. Roles can in turn assigned to users. Roles can also be assigned to other roles, creating a role hierarchy. To facilitate data management, StarRocks provides system-defined roles. To allow for more flexibility, you can also create custom roles according to business requirements. The following figure shows an example of privilege management under the RBAC and IBAC privilege models. In the models, access to objects is allowed through privileges assigned to roles and users. Roles are in turn assigned to other roles or users. ![privilege management](/assets/images/privilege-manage-1b5789075b4868ca896d1acc014df68b.png) #### Objects and privileges[​](#objects-and-privileges "Direct link to Objects and privileges") Objects have a logical hierarchy, which is related to the concept they represent. For example, Database is contained in Catalog, and Table, View, Materialized View, and Function are contained in Database. The following figure shows the object hierarchy in the StarRocks system. ![privilege objects](/assets/images/privilege-object-c7b962dd11136a98e64759615847e4c9.png) Each object has a set of privilege items that can be granted. These privileges define which operations can be performed on these objects. You can grant and revoke privileges from roles or users through the [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md) commands. #### Users[​](#users "Direct link to Users") ##### User identity[​](#user-identity "Direct link to User identity") In StarRocks, each user is identified by a unique user ID. It consists of the IP address (user host) and username, in the format of `username @'userhost'`. StarRocks identifies users with the same username but from different IP addresses as different user identities. For example, `user1@'172.65.xx.1'` and `172.65.xx.2'` are two user identities. Another representation of user identity is `username @['domain']`, where `domain` is a domain name that can be resolved by DNS as a set of IP addresses. `username @['domain']` is finally represented as a set of `username@'userhost'`. You can use `%` for the `userhost` part for fuzzy match. If `userhost` is not specified, it defaults to `'%'`, which means users of the same name logged in from any host. ##### Grant privileges to users[​](#grant-privileges-to-users "Direct link to Grant privileges to users") Users are entities to which privileges can be granted. Both privileges and roles can be assigned to users. The maximum privilege scope of each user identity is the union of its own privileges and the privileges of the roles assigned to this user identity. StarRocks ensures that each user can only perform authorized operations. We recommend that you **use roles to pass privileges** in most cases. For example, after you create a role, you can grant privileges to the role and then assign the role to users. If you want to grant temporary or special privileges, you can directly grant them to users. This simplifies privilege management and offers flexibility. #### Roles[​](#roles "Direct link to Roles") Roles are the entities to which privileges can be granted and revoked. Roles can be seen as a collection of privileges that can be assigned to users, to allow them to perform required actions. A user can be assigned multiple roles so they can perform different actions using separate sets of privileges. To simplify management, StarRocks recommends that you **manage privileges through roles**. Special and temporary privileges can be directly granted to users. To facilitate management, StarRocks provides several **system-defined roles** with specific privileges, which helps you meet daily management and maintenance requirements. You can also flexibly **customize roles** to meet specific business needs and security needs. Note that the privilege scope of system-defined roles cannot be modified. After a role is activated, users can perform operations that are authorized by the role. You can set **default roles** that are automatically activated when the user logs in. Users can also manually activate a role owned by this used in the current session. ##### System-defined roles[​](#system-defined-roles "Direct link to System-defined roles") StarRocks provides several types of system-defined roles. ![roles](/assets/images/privilege-role-bd6646cc406da9c1e890bb5f108a3764.png) * `root`: has global privileges. By default, the `root` user has the `root` role. After a StarRocks cluster is created, the system automatically generates a root user with root privileges. Because the root user and role have all privileges of the system, we recommend that you create new users and roles for subsequent operations to prevent any risky operations. Keep the password of the root user properly. * `cluster_admin`: has cluster management privileges to perform node-related operations, such as adding or dropping nodes. `cluster_admin` has the privileges to add, drop, and decommission cluster nodes. We recommend that you do not assign `cluster_admin` or any custom roles that contain this role as a default role to any user, to prevent unexpected node changes. - `db_admin`: has database management privileges, including the privileges to perform all operations on catalog, database, table, view, materialized view, function, global function, resource group, and plug-ins. - `user_admin`: has administrative privileges on users and roles, including privileges to create users, roles, and privileges. The above system-defined roles are designed to aggregate complex database privileges to facilitate your daily management. **The privilege scope of the above roles cannot be modified.** In addition, if you need to grant specific privileges to all users, StarRocks also provides a system-defined role `public`. - `public`: This role is owned by any user and activated by default in any session, including adding new users. The `public` role does not have any privileges by default. You can modify the privileges scope of this role. ##### Custom roles[​](#custom-roles "Direct link to Custom roles") You can create custom roles to meet specific business requirements and modify their privilege scope. At the same time, for the convenience of management, you can assign roles to other roles to create privilege hierarchy and inheritance. Then, the privileges associated with a role are inherited by another role. ###### Role hierarchy and privilege inheritance[​](#role-hierarchy-and-privilege-inheritance "Direct link to Role hierarchy and privilege inheritance") The following figure shows an example of privilege inheritance. > Note: The maximum number of inheritance levels for a role is 16 by default. The inheritance relationship cannot be bidirectional. ![role inheritance](/assets/images/privilege-role_inheri-5cf6cbc58151a23afa650d0057df6e54.png) As shown in the figure: * `role_s` is assigned to `role_p`. `role_p` implicitly inherits `priv_1` of `role_s`. * `role_p` is assigned to `role_g`, `role_g` implicitly inherits `priv_2` of `role_p` and `priv_1` of `role_s`. * After a role is assigned to a user, the user also has the privileges of this role. ##### Active roles[​](#active-roles "Direct link to Active roles") Active roles allow users to apply the privileges of the role under the current session. You can use `SELECT CURRENT_ROLE();` to view active roles in the current session. For more information, see [current\_role](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/current_role.md). ###### Default roles[​](#default-roles "Direct link to Default roles") Default roles are automatically activated when the user logs in to the cluster. It can be a role owned by one or more users. The administrator can set default roles using the `DEFAULT ROLE` keyword in [CREATE USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_USER.md) and can change default roles using [ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md). Users can also change their default roles using [SET DEFAULT ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_DEFAULT_ROLE.md). Default roles provide basic privilege protection for users. For example, User A has `role_query` and `role_delete`, which has query and delete privilege respectively. We recommend that you only use `role_query` as the default role to prevent data loss caused by high-risk operations such as `DELETE` or `TRUNCATE`. If you need to perform these operations, you can do it after manually setting active roles. A user who does not have a default role still has the `public` role, which is automatically activated after the user logs in to the cluster. ###### Manually activate roles[​](#manually-activate-roles "Direct link to Manually activate roles") In addition to default roles, users can also manually activate one or more existing roles within a session. You can use [SHOW GRANTS](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_GRANTS.md) to view the privileges and roles that can be activated, and use [SET ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_ROLE.md) to configure active roles that are effective in the current session. Note that the SET ROLE command overwrites each other. For example, after a user logs in, the `default_role` is activated by default. Then the user runs `SET ROLE role_s`. At this time, the user has only the privileges of `role_s` and their own privileges. `default_role` is overwritten. #### References[​](#references "Direct link to References") * [Privileges supported by StarRocks](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md) * [Manage user privileges](https://docs.starrocks.io/docs/administration/user_privs/authorization/User_privilege.md) --- ### Authenticate User Groups Enable Group Provider in StarRocks to authenticate, and authorize user groups from external authentication systems. From v3.5.0 onwards, StarRocks supports Group Provider to collect group information from external authentication systems for user group management. #### Overview[​](#overview "Direct link to Overview") To deepen its integration with external user authentication and authorization systems, such as LDAP and Apache Ranger, StarRocks supports collecting user group information for a better experience on the collective user management. With a group provider, you can fetch the group information from external user systems for different purposes. Group information is independent and can be integrated flexibly into authentication, authorization, or other processes without being tightly coupled to any specific workflow. A Group Provider is essentially a mapping between users and groups. Any process that requires group information can query this mapping as needed. ##### Workflow[​](#workflow "Direct link to Workflow") The following flow chart uses LDAP and Apache Ranger as an example to explain the workflow of Group Provider. ![Group Provider](/assets/images/group_provider-cfcd33985a599de8f9403e8fa8357a19.png) #### Create a group provider[​](#create-a-group-provider "Direct link to Create a group provider") StarRocks supports three types of group providers: * **LDAP group provider**: Search and match users with groups in your LDAP service * **Unix group provider**: Search and match users with groups in your operating system * **File group provider**: Search and match users with groups defined by a file ##### Syntax[​](#syntax "Direct link to Syntax") * LDAP group provider: ```sql CREATE GROUP PROVIDER PROPERTIES ( "type" = "ldap", ldap_info, ldap_search_group_arg, ldap_search_attr, [ldap_cache_attr] ) ldap_info ::= "ldap_conn_url" = "", "ldap_bind_root_dn" = "", "ldap_bind_root_pwd" = "", "ldap_bind_base_dn" = "", ["ldap_conn_timeout" = "",] ["ldap_conn_read_timeout" = ""] ["ldap_ssl_conn_allow_insecure" = ""] ["ldap_ssl_conn_trust_store_path" = ""] ["ldap_ssl_conn_trust_store_pwd" = ""] ldap_search_group_arg ::= { "ldap_group_dn" = "" | "ldap_group_filter" = "" }, "ldap_group_identifier_attr" = "" ldap_search_user_arg ::= "ldap_group_member_attr" = "", "ldap_user_search_attr" = "" ldap_cache_arg ::= ["ldap_cache_refresh_interval" = "",] ["ldap_cache_max_stale_time" = ""] ``` * Unix group provider: ```sql CREATE GROUP PROVIDER PROPERTIES ( "type" = "unix" ) ``` * File group provider: ```sql CREATE GROUP PROVIDER PROPERTIES ( "type" = "file", "group_file_url" = "" ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### `type`[​](#type "Direct link to type") The type of the group provider to create. Valid values: * `ldap`: Creates an LDAP group provider. When this value is set, you need to specify `ldap_info`, `ldap_search_group_arg`, `ldap_search_user_arg`, and optionally `ldap_cache_arg`. * `unix`: Creates a Unix group provider. * `file`: Creates a File group provider. When this value is set, you need to specify `group_file_url`. ###### `group_file_url`[​](#group_file_url "Direct link to group_file_url") The URL or relative path (under `fe/conf`) to the file that defines the user groups. note A group file contains a list of groups and their members. You can define a group in each line where the group name and members are separated by a colon. Multiple users are separated by commas. Example: `group_name:user_1,user_2,user_3`. An `http://` or `https://` URL is fetched with the connect and read timeouts set by the FE configuration items `group_provider_http_connect_timeout_ms` (default 5000) and `group_provider_http_read_timeout_ms` (default 30000). If the server does not answer within them, the statement fails and the group provider keeps its previous configuration. ###### `ldap_info` parameter group[​](#ldap_info-parameter-group "Direct link to ldap_info-parameter-group") The information used to connect to your LDAP service. ###### `ldap_conn_url`[​](#ldap_conn_url "Direct link to ldap_conn_url") The URL of your LDAP server. Format: `ldap://:)`. ###### `ldap_bind_root_dn`[​](#ldap_bind_root_dn "Direct link to ldap_bind_root_dn") The admin Distinguished Name (DN) of your LDAP service. ###### `ldap_bind_root_pwd`[​](#ldap_bind_root_pwd "Direct link to ldap_bind_root_pwd") The admin password of your LDAP service. ###### `ldap_bind_base_dn`[​](#ldap_bind_base_dn "Direct link to ldap_bind_base_dn") The base DN of the LDAP user for which the cluster searches. ###### `ldap_conn_timeout`[​](#ldap_conn_timeout "Direct link to ldap_conn_timeout") Optional. The timeout duration for the connection to your LDAP service. ###### `ldap_conn_read_timeout`[​](#ldap_conn_read_timeout "Direct link to ldap_conn_read_timeout") Optional. The timeout duration for the read operations in the connection to your LDAP service. ###### `ldap_ssl_conn_allow_insecure`[​](#ldap_ssl_conn_allow_insecure "Direct link to ldap_ssl_conn_allow_insecure") Optional. Whether to allow non-encrypted connections to the LDAP server. Default value: `true`. Setting this value to `false` indicates that SSL encryption is required to access LDAP. ###### `ldap_ssl_conn_trust_store_path`[​](#ldap_ssl_conn_trust_store_path "Direct link to ldap_ssl_conn_trust_store_path") Optional. Local path to store the SSL CA certificate of the LDAP server. Supports pem and jks formats. You do not need to set this item if the certificate is issued by a trusted organization. ###### `ldap_ssl_conn_trust_store_pwd`[​](#ldap_ssl_conn_trust_store_pwd "Direct link to ldap_ssl_conn_trust_store_pwd") Optional. The password used to access the locally stored SSL CA certificate of the LDAP server. pem-formatted certificates do not require a password. Only jsk-formatted certificates do. ###### `ldap_search_group_arg` parameter group[​](#ldap_search_group_arg-parameter-group "Direct link to ldap_search_group_arg-parameter-group") The arguments used to control how StarRocks searches for a group. note You can only specify either `ldap_group_dn` or `ldap_group_filter`. Specifying both is not supported. ###### `ldap_group_dn`[​](#ldap_group_dn "Direct link to ldap_group_dn") The DN of the group to be searched for. The group will be queried directly using this DN. Example: `"cn=ldapgroup1,ou=Group,dc=starrocks,dc=com;cn=ldapgroup2,ou=Group,dc=starrocks,dc=com"`. ###### `ldap_group_filter`[​](#ldap_group_filter "Direct link to ldap_group_filter") A customized group filter that can be recognized by the LDAP server. It will be sent directly to your LDAP server for searching for the group. Example: `(&(objectClass=groupOfNames)(cn=testgroup))`. ###### `ldap_group_identifier_attr`[​](#ldap_group_identifier_attr "Direct link to ldap_group_identifier_attr") The attribute used as the identifier for the group name. ###### `ldap_search_user_arg` parameter group[​](#ldap_search_user_arg-parameter-group "Direct link to ldap_search_user_arg-parameter-group") The arguments used to control how StarRocks identifies for a user in a group. ###### `ldap_group_member_attr`[​](#ldap_group_member_attr "Direct link to ldap_group_member_attr") The attribute that represents group members. Valid values: `member` and `memberUid`. ###### `ldap_user_search_attr`[​](#ldap_user_search_attr "Direct link to ldap_user_search_attr") Specifies how to extract the user identifier from the member attribute value. You can explicitly define an attribute (for example, `cn` or `uid`) or use a regular expression. note **DN Matching Mechanism** * **When `ldap_user_search_attr` is configured**, the system extracts the specified value from group member DNs and uses it as usernames, and uses login username as key during group search. * **When `ldap_user_search_attr` is not configured**, the system uses the complete DN directly as user identifier, and uses the DN recorded during authentication as key during group search. This design enables LDAP Group Provider to adapt to different LDAP environments, especially complex environments like Microsoft AD. tip **Interaction with `authentication_ldap_simple_bind_dn_pattern`** When using DN pattern authentication (e.g., `uid=${USER}@abc.com,ou=People,dc=example,dc=com`) where the `${USER}` substitution is embedded within a larger attribute value: * **Recommended**: Do not configure `ldap_user_search_attr`. The system will use the complete DN for group matching, which avoids extraction errors. * **If you must configure it**: Use a regex that extracts only the username portion. For example, if the DN pattern is `uid=${USER}@abc.com,ou=People,dc=example,dc=com`, set `ldap_user_search_attr` to `uid=([^,@]+)@abc.com` to extract only the `${USER}` part. A simple `uid` or `uid=([^,]+)` would incorrectly extract `alice@abc.com` instead of `alice`. ###### `ldap_cache_arg` parameter group[​](#ldap_cache_arg-parameter-group "Direct link to ldap_cache_arg-parameter-group") The argument used to define the cache behavior for the LDAP group information. ###### `ldap_cache_refresh_interval`[​](#ldap_cache_refresh_interval "Direct link to ldap_cache_refresh_interval") Optional. The interval at which StarRocks automatically refreshes the cached LDAP group information. Unit: Seconds. Default: `300`. ###### `ldap_cache_max_stale_time`[​](#ldap_cache_max_stale_time "Direct link to ldap_cache_max_stale_time") Optional. How long the last successfully built cache keeps being served while refreshes keep failing. Unit: Seconds. Default: `3600`. If a refresh cannot reach the LDAP server, StarRocks keeps serving the previous cache until it has been stale for longer than this value, after which the cache is dropped and every user resolves to an empty group set. Set this item to `0` to drop the cache as soon as a refresh fails. note Dropping the cache immediately turns a brief LDAP outage into a cluster-wide login failure when `permitted_groups` is configured, because an empty group set can never intersect the allowed list. Raising this value trades a longer window of stale group information for tolerance of directory downtime. ##### Example[​](#example "Direct link to Example") Suppose an LDAP server contains the following group and member information. ```plain -- Group information # testgroup, Group, starrocks.com dn: cn=testgroup,ou=Group,dc=starrocks,dc=com objectClass: groupOfNames cn: testgroup member: uid=test,ou=people,dc=starrocks,dc=com member: uid=tom,ou=people,dc=starrocks,dc=com -- User information # test, People, starrocks.com dn: cn=test,ou=People,dc=starrocks,dc=com objectClass: inetOrgPerson cn: test uid: test sn: FTE userPassword:: ``` Create a group provider `ldap_group_provider` for members in `testgroup`: ```sql CREATE GROUP PROVIDER ldap_group_provider PROPERTIES( "type"="ldap", "ldap_conn_url"="ldap://xxxx:xxx", "ldap_bind_root_dn"="cn=admin,dc=starrocks,dc=com", "ldap_bind_root_pwd"="123456", "ldap_bind_base_dn"="dc=starrocks,dc=com", "ldap_group_filter"="(&(objectClass=groupOfNames)(cn=testgroup))", "ldap_group_identifier_attr"="cn", "ldap_group_member_attr"="member", "ldap_user_search_attr"="uid=([^,]+)" ) ``` The above example uses `ldap_group_filter` to search for a group with the `groupOfNames` objectClass and a `cn` of `testgroup`. Therefore, `cn` is specified in `ldap_group_identifier_attr` to identify the group. `ldap_group_member_attr` is set to `member` so that the `member` attribute is used in the `groupOfNames` objectClass to identify members. `ldap_user_search_attr` is set to an expression `uid=([^,]+)`, which is used to identify users in the `member` attribute. ##### Microsoft AD Environment Example[​](#microsoft-ad-environment-example "Direct link to Microsoft AD Environment Example") Suppose a Microsoft AD server contains the following group and member information: ```plain -- Group information # ADGroup, Groups, company.com dn: CN=ADGroup,OU=Groups,DC=company,DC=com objectClass: group cn: ADGroup member: CN=John Doe,OU=Users,DC=company,DC=com member: CN=Jane Smith,OU=Users,DC=company,DC=com -- User information # John Doe, Users, company.com dn: CN=John Doe,OU=Users,DC=company,DC=com objectClass: user cn: John Doe sAMAccountName: johndoe ``` Create a Group Provider for Microsoft AD environment: ```sql CREATE GROUP PROVIDER ad_group_provider PROPERTIES( "type"="ldap", "ldap_conn_url"="ldap://ad.company.com:389", "ldap_bind_root_dn"="CN=admin,OU=Users,DC=company,DC=com", "ldap_bind_root_pwd"="password", "ldap_bind_base_dn"="DC=company,DC=com", "ldap_group_filter"="(&(objectClass=group)(cn=ADGroup))", "ldap_group_identifier_attr"="cn", "ldap_group_member_attr"="member" -- Note: Do not configure ldap_user_search_attr, system will use complete DN for matching ) ``` In this example, since `ldap_user_search_attr` is not configured, the system will: 1. During group cache construction, directly use the complete DN (for example, `CN=John Doe,OU=Users,DC=company,DC=com`) as user identifier. 2. During group search, use the DN recorded during authentication as key to search user's groups. This approach is particularly suitable for Microsoft AD environments, as group members in AD may lack simple username attributes. #### Alter a group provider[​](#alter-a-group-provider "Direct link to Alter a group provider") You can update the properties of an existing group provider without dropping and recreating it. The new configuration is validated and warmed up before it takes effect: the old instance keeps serving requests until the new one is ready, so group lookups never return empty during the change. ##### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql ALTER GROUP PROVIDER SET ( "" = "" [, "" = "" ...] ) ``` ##### Notes[​](#notes "Direct link to Notes") * Only the listed properties are changed; properties that are not listed keep their previous values. * The `type` property cannot be changed. Drop and recreate the provider if you need to switch its type. * `SET` adds or overwrites a property, but it cannot remove one. Two properties that are not allowed to coexist therefore cannot be swapped with `ALTER`: an LDAP group provider accepts exactly one of `ldap_group_dn` and `ldap_group_filter`, so moving from one to the other means dropping and recreating the provider. * A property that the provider type does not define and that the provider does not already have is rejected. Property names are matched without regard to case, so `SET ("LDAP_BIND_ROOT_PWD" = ...)` updates `ldap_bind_root_pwd`; a misspelled name fails the statement instead of being stored beside the property you meant to change. * The new configuration is verified synchronously. If it is not usable (for example, the LDAP bind credentials are wrong or the server is unreachable), the statement fails and the provider keeps its previous configuration. * Credentials are never echoed back. `SHOW CREATE GROUP PROVIDER` prints `***` instead of the values of `ldap_bind_root_pwd` and `ldap_ssl_conn_trust_store_pwd`, and the statement recorded in the audit log has them redacted. ##### Example[​](#example-1 "Direct link to Example") ```sql -- Rotate the LDAP bind password ALTER GROUP PROVIDER ldap_group_provider SET ("ldap_bind_root_pwd" = ""); -- Adjust the cache refresh interval ALTER GROUP PROVIDER ldap_group_provider SET ("ldap_cache_refresh_interval" = "10"); -- Update multiple properties at once ALTER GROUP PROVIDER ldap_group_provider SET ( "ldap_conn_url" = "ldaps://new-host:636", "ldap_ssl_conn_trust_store_path" = "/etc/ssl/new-truststore.jks" ); ``` #### View group providers[​](#view-group-providers "Direct link to View group providers") ```sql -- List every group provider with its type. The Comment column is always NULL: no statement sets a comment. SHOW GROUP PROVIDERS; -- Show the statement that would recreate one group provider. Credentials such as -- `ldap_bind_root_pwd` and `ldap_ssl_conn_trust_store_pwd` are printed as `***`, so the output cannot be -- replayed as-is - fill the passwords back in before reusing it. SHOW CREATE GROUP PROVIDER ; ``` Both statements require the `SECURITY` privilege on the SYSTEM level. #### Drop a group provider[​](#drop-a-group-provider "Direct link to Drop a group provider") ```sql DROP GROUP PROVIDER [IF EXISTS] ; ``` Dropping a group provider takes effect immediately: users who log in afterwards no longer carry the groups it resolved, so any role that was granted `TO EXTERNAL GROUP` for those groups stops applying. Security integrations keep the dropped name in their `group_provider` property, and simply resolve no groups through it. To change the properties of a group provider, use [ALTER GROUP PROVIDER](#alter-a-group-provider) instead of dropping and recreating it. #### Combine group provider with a security integration[​](#combine-group-provider-with-a-security-integration "Direct link to Combine group provider with a security integration") After creating the group provider, you can combine it with a security integration to allow users specified by the group provider to log in to StarRocks. For more information on creating a security integration, see [Authenticate with Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md). ##### Syntax[​](#syntax-2 "Direct link to Syntax") ```sql ALTER SECURITY INTEGRATION SET ( "group_provider" = "", "permitted_groups" = "" ) ``` ##### Parameters[​](#parameters-1 "Direct link to Parameters") ###### `group_provider`[​](#group_provider "Direct link to group_provider") The name of the group provider(s) to be combined with the security integration. Multiple group providers are separated by commas. Once set, StarRocks will record the user's group information under each specified provider upon login. When this property is not set, the security integration falls back to the FE configuration item `group_provider`, the cluster-wide default list. Note that versions earlier than v4.2 consulted no group provider at all in that case, so setting the FE configuration item takes effect for such an integration only from v4.2 onwards. ###### `permitted_groups`[​](#permitted_groups "Direct link to permitted_groups") Optional. The name of group(s) whose members are allowed to log in to StarRocks. Multiple groups are separated by commas. Make sure that the specified groups can be retrieved by the combined group provider(s). ##### Example[​](#example-2 "Direct link to Example") ```sql ALTER SECURITY INTEGRATION LDAP SET ( "group_provider"="ldap_group_provider", "permitted_groups"="testgroup" ); ``` #### Grant role to a user group[​](#grant-role-to-a-user-group "Direct link to Grant role to a user group") You can grant roles to a user group via [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md). The following example grants the role `example_role` to the user group `analysts`: ```sql GRANT example_role TO EXTERNAL GROUP analysts; ``` #### Combine group provider with external authorization system (Apache Ranger)[​](#combine-group-provider-with-external-authorization-system-apache-ranger "Direct link to Combine group provider with external authorization system (Apache Ranger)") Once you configure the associated group provider in the security integration, StarRocks will record the user's group information upon login. This group information will then be automatically included in the authorization process with Ranger, eliminating the need for additional configuration. For more instructions on integrating StarRocks with Ranger, see [Manage permissions with Apache Ranger](https://docs.starrocks.io/docs/administration/user_privs/authorization/ranger_plugin.md). --- ### SSL Authentication From v3.4.1 onwards, StarRocks supports secure connections encrypted by SSL. Unlike the traditional cleartext connections to DBMS, SSL connections provide endpoint verification and data encryption to ensure that the data transmitted between clients and StarRocks cannot be read by unauthorized users. #### Enable SSL authentication[​](#enable-ssl-authentication "Direct link to Enable SSL authentication") ##### Configure FE nodes[​](#configure-fe-nodes "Direct link to Configure FE nodes") To enable SSL authentication in StarRocks, configure the following parameters in the FE configuration file **fe.conf**: * `ssl_keystore_location`: Specifies the path to the keystore file that stores the SSL certificate and key. * `ssl_keystore_password`: The password for accessing the keystore file. StarRocks requires this password to read the keystore file. * `ssl_key_password`: The password for accessing the key. StarRocks requires this password to retrieve the key from the keystore. * `ssl_force_secure_transport`: Whether to force SSL authentication. Default value: `FALSE`. If this item is set to `TRUE`, the system will reject connections that are not encrypted with SSL. Example: ```properties ssl_keystore_location = // Path to the keystore file ssl_keystore_password = // Password for the keystore file ssl_key_password = // Password for accessing the key ``` ##### Generate SSL certificate[​](#generate-ssl-certificate "Direct link to Generate SSL certificate") In a production environment, it is recommended to use certificates provided by certificate authorities. In a development environment, you can generate a custom SSL certificate. Use the following command to generate an SSL certificate: ```bash keytool -genkeypair -alias starrocks \ -keypass \ -keyalg RSA -keysize 1024 -validity 365 \ -keystore \ -storepass ``` Parameters: * `-keypass`: The key password, corresponding to `ssl_key_password` in **fe.conf**. * `-storepass`: The keystore file password, corresponding to `ssl_keystore_password` in **fe.conf**. * `-keystore`: The storage path of the keystore file, corresponding to `ssl_keystore_location` in **fe.conf**. ##### Enable SSL on client[​](#enable-ssl-on-client "Direct link to Enable SSL on client") StarRocks is compatible with the MySQL protocol. For MySQL clients, SSL authentication is enabled by default. For JDBC connections, add the following options: ```properties useSSL=true verifyServerCertificate=false ``` #### Disable SSL authentication[​](#disable-ssl-authentication "Direct link to Disable SSL authentication") To disable SSL authentication, follow these steps: * **MySQL client**: Add the option `--ssl-mode=DISABLED`. * **JDBC**: Remove `useSSL=true` and `verifyServerCertificate=false`. #### LDAP authentication[​](#ldap-authentication "Direct link to LDAP authentication") See [Authentication methods](https://docs.starrocks.io/docs/administration/user_privs/authentication/ldap_authentication.md) for detailed instructions on enabling LDAP authentication. For JDBC connections, since StarRocks supports SSL authentication, you do not need to customize `AuthPlugin`. You can use the built-in `MysqlClearPasswordPlugin`. * When using JDBC 5 with LDAP authentication, configure the following settings: ```properties authenticationPlugins: com.mysql.jdbc.authentication.MysqlClearPasswordPlugin defaultAuthenticationPlugin: com.mysql.jdbc.authentication.MysqlClearPasswordPlugin disabledAuthenticationPlugins: com.mysql.jdbc.authentication.MysqlNativePasswordPlugin ``` * When using JDBC 8 with LDAP authentication, configure the following settings: ```properties authenticationPlugins: com.mysql.cj.protocol.a.authentication.MysqlClearPasswordPlugin defaultAuthenticationPlugin: com.mysql.cj.protocol.a.authentication.MysqlClearPasswordPlugin disabledAuthenticationPlugins: com.mysql.cj.protocol.a.authentication.MysqlNativePasswordPlugin ``` #### FAQ[​](#faq "Direct link to FAQ") ###### Q1: When I connect to StarRocks using DBeaver, an error is returned "Unable to load authentication plugin 'mysql\_native\_password'"[​](#q1-when-i-connect-to-starrocks-using-dbeaver-an-error-is-returned-unable-to-load-authentication-plugin-mysql_native_password "Direct link to Q1: When I connect to StarRocks using DBeaver, an error is returned \"Unable to load authentication plugin 'mysql_native_password'\"") A: You need to upgrade JDBC 5 to version 5.1.46 or later. --- ## Benchmarking ### SSB Flat-table Benchmarking Star schema benchmark (SSB) is designed to test basic performance metrics of OLAP database products. SSB uses a star schema test set that is widely applied in academia and industry. For more information, see [Star Schema Benchmark](https://www.cs.umb.edu/~poneil/StarSchemaB.PDF). ClickHouse flattens the star schema into a wide flat table and rewrites the SSB into a single-table benchmark. For more information, see [Star schema benchmark of ClickHouse](https://clickhouse.com/docs/getting-started/example-datasets/star-schema) This test compares the performance of StarRocks, Apache Druid, and ClickHouse against SSB single-table datasets. #### Test conclusions[​](#test-conclusions "Direct link to Test conclusions") The test is performed on an OLAP table in a shared-nothing StarRocks cluster, together with ClickHouse and Apache Druid, against the same dataset. Based on the results from 13 queries performed on the 100 GB SSB-Flat dataset, StarRocks has an overall query performance **1.87x that of ClickHouse and 4.75x that of Apache Druid**. The unit of the results are milliseconds. ![SSB-SR](/assets/images/SSB-SR-a6741eb2b92ecf5dedf472be518c29dd.png) #### Test preparation[​](#test-preparation "Direct link to Test preparation") ##### Hardware[​](#hardware "Direct link to Hardware") StarRocks, Apache Druid, and ClickHouse are deployed on hosts of the same configurations - [AWS m7i.4xlarge](https://aws.amazon.com/ec2/instance-types/m7i/?nc1=h_ls). | | **Spec** | | ------------------------ | ---------- | | Instance Number | 5 | | vCPU | 16 | | Memory (GiB) | 64 | | Network Bandwidth (Gbps) | Up to 12.5 | | EBS Bandwidth (Gbps) | Up to 10 | ##### Software[​](#software "Direct link to Software") | | **StarRocks** | **ClickHouse** | **Apache Druid** | | ----------------- | ----------------- | -------------- | ------------------------------------------------------------ | | **Cluster Size** | One FE, Three BEs | Three nodes | One Master Server, one Query Servers, and three Data Servers | | **Version** | 3.5.0 | 25.3.3.42 | 33.0.0 | | **Release Date** | 2025.6.13 | 2025.4.22 | 2025.4.29 | | **Configuration** | Default | Default | Default | #### Test results[​](#test-results "Direct link to Test results") The following table shows the performance test results on 13 queries. The unit of query latency is milliseconds. All queries are warmed up 1 time and then executed 3 times to take the average value as the result. `ClickHouse vs StarRocks` and `Druid vs StarRocks` in the table header means using the query response time of ClickHouse/Druid to divide the query response time of StarRocks. A larger value indicates better performance of StarRocks. | Query | StarRocks | ClickHouse | Druid | ClickHouse vs StarRocks | Druid vs StarRocks | | ----- | --------- | ---------- | ----- | ----------------------- | ------------------ | | SUM | 992 | 1858 | 4710 | 1.87 | 4.75 | | Q01 | 30 | 49 | 330 | 1.63 | 11.00 | | Q02 | 16 | 31 | 260 | 1.94 | 16.25 | | Q03 | 26 | 29 | 250 | 1.12 | 9.62 | | Q04 | 143 | 197 | 420 | 1.38 | 2.94 | | Q05 | 120 | 179 | 440 | 1.49 | 3.67 | | Q06 | 63 | 158 | 320 | 2.51 | 5.08 | | Q07 | 133 | 249 | 510 | 1.87 | 3.83 | | Q08 | 90 | 197 | 380 | 2.19 | 4.22 | | Q09 | 86 | 150 | 350 | 1.74 | 4.07 | | Q10 | 20 | 33 | 250 | 1.65 | 12.50 | | Q11 | 156 | 340 | 550 | 2.18 | 3.53 | | Q12 | 66 | 133 | 330 | 2.02 | 5.00 | | Q13 | 43 | 113 | 320 | 2.63 | 7.44 | --- ### TPC-DS test SQL ```sql -- query 1 with customer_total_return as (select sr_customer_sk as ctr_customer_sk ,sr_store_sk as ctr_store_sk ,sum(SR_RETURN_AMT) as ctr_total_return from store_returns ,date_dim where sr_returned_date_sk = d_date_sk and d_year =2000 group by sr_customer_sk ,sr_store_sk) select c_customer_id from customer_total_return ctr1 ,store ,customer where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2 from customer_total_return ctr2 where ctr1.ctr_store_sk = ctr2.ctr_store_sk) and s_store_sk = ctr1.ctr_store_sk and s_state = 'TN' and ctr1.ctr_customer_sk = c_customer_sk order by c_customer_id limit 100; -- query 2 with wscs as (select sold_date_sk ,sales_price from (select ws_sold_date_sk sold_date_sk ,ws_ext_sales_price sales_price from web_sales union all select cs_sold_date_sk sold_date_sk ,cs_ext_sales_price sales_price from catalog_sales) t), wswscs as (select d_week_seq, sum(case when (d_day_name='Sunday') then sales_price else null end) sun_sales, sum(case when (d_day_name='Monday') then sales_price else null end) mon_sales, sum(case when (d_day_name='Tuesday') then sales_price else null end) tue_sales, sum(case when (d_day_name='Wednesday') then sales_price else null end) wed_sales, sum(case when (d_day_name='Thursday') then sales_price else null end) thu_sales, sum(case when (d_day_name='Friday') then sales_price else null end) fri_sales, sum(case when (d_day_name='Saturday') then sales_price else null end) sat_sales from wscs ,date_dim where d_date_sk = sold_date_sk group by d_week_seq) select d_week_seq1 ,round(sun_sales1/sun_sales2,2) ,round(mon_sales1/mon_sales2,2) ,round(tue_sales1/tue_sales2,2) ,round(wed_sales1/wed_sales2,2) ,round(thu_sales1/thu_sales2,2) ,round(fri_sales1/fri_sales2,2) ,round(sat_sales1/sat_sales2,2) from (select wswscs.d_week_seq d_week_seq1 ,sun_sales sun_sales1 ,mon_sales mon_sales1 ,tue_sales tue_sales1 ,wed_sales wed_sales1 ,thu_sales thu_sales1 ,fri_sales fri_sales1 ,sat_sales sat_sales1 from wswscs,date_dim where date_dim.d_week_seq = wswscs.d_week_seq and d_year = 2001) y, (select wswscs.d_week_seq d_week_seq2 ,sun_sales sun_sales2 ,mon_sales mon_sales2 ,tue_sales tue_sales2 ,wed_sales wed_sales2 ,thu_sales thu_sales2 ,fri_sales fri_sales2 ,sat_sales sat_sales2 from wswscs ,date_dim where date_dim.d_week_seq = wswscs.d_week_seq and d_year = 2001+1) z where d_week_seq1=d_week_seq2-53 order by d_week_seq1; -- query 3 select dt.d_year ,item.i_brand_id brand_id ,item.i_brand brand ,sum(ss_ext_sales_price) sum_agg from date_dim dt ,store_sales ,item where dt.d_date_sk = store_sales.ss_sold_date_sk and store_sales.ss_item_sk = item.i_item_sk and item.i_manufact_id = 128 and dt.d_moy=11 group by dt.d_year ,item.i_brand ,item.i_brand_id order by dt.d_year ,sum_agg desc ,brand_id limit 100; -- query 4 with year_total as ( select c_customer_id customer_id ,c_first_name customer_first_name ,c_last_name customer_last_name ,c_preferred_cust_flag customer_preferred_cust_flag ,c_birth_country customer_birth_country ,c_login customer_login ,c_email_address customer_email_address ,d_year dyear ,sum(((ss_ext_list_price-ss_ext_wholesale_cost-ss_ext_discount_amt)+ss_ext_sales_price)/2) year_total ,'s' sale_type from customer ,store_sales ,date_dim where c_customer_sk = ss_customer_sk and ss_sold_date_sk = d_date_sk group by c_customer_id ,c_first_name ,c_last_name ,c_preferred_cust_flag ,c_birth_country ,c_login ,c_email_address ,d_year union all select c_customer_id customer_id ,c_first_name customer_first_name ,c_last_name customer_last_name ,c_preferred_cust_flag customer_preferred_cust_flag ,c_birth_country customer_birth_country ,c_login customer_login ,c_email_address customer_email_address ,d_year dyear ,sum((((cs_ext_list_price-cs_ext_wholesale_cost-cs_ext_discount_amt)+cs_ext_sales_price)/2) ) year_total ,'c' sale_type from customer ,catalog_sales ,date_dim where c_customer_sk = cs_bill_customer_sk and cs_sold_date_sk = d_date_sk group by c_customer_id ,c_first_name ,c_last_name ,c_preferred_cust_flag ,c_birth_country ,c_login ,c_email_address ,d_year union all select c_customer_id customer_id ,c_first_name customer_first_name ,c_last_name customer_last_name ,c_preferred_cust_flag customer_preferred_cust_flag ,c_birth_country customer_birth_country ,c_login customer_login ,c_email_address customer_email_address ,d_year dyear ,sum((((ws_ext_list_price-ws_ext_wholesale_cost-ws_ext_discount_amt)+ws_ext_sales_price)/2) ) year_total ,'w' sale_type from customer ,web_sales ,date_dim where c_customer_sk = ws_bill_customer_sk and ws_sold_date_sk = d_date_sk group by c_customer_id ,c_first_name ,c_last_name ,c_preferred_cust_flag ,c_birth_country ,c_login ,c_email_address ,d_year ) select t_s_secyear.customer_id ,t_s_secyear.customer_first_name ,t_s_secyear.customer_last_name ,t_s_secyear.customer_preferred_cust_flag from year_total t_s_firstyear ,year_total t_s_secyear ,year_total t_c_firstyear ,year_total t_c_secyear ,year_total t_w_firstyear ,year_total t_w_secyear where t_s_secyear.customer_id = t_s_firstyear.customer_id and t_s_firstyear.customer_id = t_c_secyear.customer_id and t_s_firstyear.customer_id = t_c_firstyear.customer_id and t_s_firstyear.customer_id = t_w_firstyear.customer_id and t_s_firstyear.customer_id = t_w_secyear.customer_id and t_s_firstyear.sale_type = 's' and t_c_firstyear.sale_type = 'c' and t_w_firstyear.sale_type = 'w' and t_s_secyear.sale_type = 's' and t_c_secyear.sale_type = 'c' and t_w_secyear.sale_type = 'w' and t_s_firstyear.dyear = 2001 and t_s_secyear.dyear = 2001+1 and t_c_firstyear.dyear = 2001 and t_c_secyear.dyear = 2001+1 and t_w_firstyear.dyear = 2001 and t_w_secyear.dyear = 2001+1 and t_s_firstyear.year_total > 0 and t_c_firstyear.year_total > 0 and t_w_firstyear.year_total > 0 and case when t_c_firstyear.year_total > 0 then t_c_secyear.year_total / t_c_firstyear.year_total else null end > case when t_s_firstyear.year_total > 0 then t_s_secyear.year_total / t_s_firstyear.year_total else null end and case when t_c_firstyear.year_total > 0 then t_c_secyear.year_total / t_c_firstyear.year_total else null end > case when t_w_firstyear.year_total > 0 then t_w_secyear.year_total / t_w_firstyear.year_total else null end order by t_s_secyear.customer_id ,t_s_secyear.customer_first_name ,t_s_secyear.customer_last_name ,t_s_secyear.customer_preferred_cust_flag limit 100; -- query 5 with ssr as (select s_store_id, sum(sales_price) as sales, sum(profit) as profit, sum(return_amt) as returns, sum(net_loss) as profit_loss from ( select ss_store_sk as store_sk, ss_sold_date_sk as date_sk, ss_ext_sales_price as sales_price, ss_net_profit as profit, cast(0 as decimal(7,2)) as return_amt, cast(0 as decimal(7,2)) as net_loss from store_sales union all select sr_store_sk as store_sk, sr_returned_date_sk as date_sk, cast(0 as decimal(7,2)) as sales_price, cast(0 as decimal(7,2)) as profit, sr_return_amt as return_amt, sr_net_loss as net_loss from store_returns ) salesreturns, date_dim, store where date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 14) and store_sk = s_store_sk group by s_store_id) , csr as (select cp_catalog_page_id, sum(sales_price) as sales, sum(profit) as profit, sum(return_amt) as returns, sum(net_loss) as profit_loss from ( select cs_catalog_page_sk as page_sk, cs_sold_date_sk as date_sk, cs_ext_sales_price as sales_price, cs_net_profit as profit, cast(0 as decimal(7,2)) as return_amt, cast(0 as decimal(7,2)) as net_loss from catalog_sales union all select cr_catalog_page_sk as page_sk, cr_returned_date_sk as date_sk, cast(0 as decimal(7,2)) as sales_price, cast(0 as decimal(7,2)) as profit, cr_return_amount as return_amt, cr_net_loss as net_loss from catalog_returns ) salesreturns, date_dim, catalog_page where date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 14) and page_sk = cp_catalog_page_sk group by cp_catalog_page_id) , wsr as (select web_site_id, sum(sales_price) as sales, sum(profit) as profit, sum(return_amt) as returns, sum(net_loss) as profit_loss from ( select ws_web_site_sk as wsr_web_site_sk, ws_sold_date_sk as date_sk, ws_ext_sales_price as sales_price, ws_net_profit as profit, cast(0 as decimal(7,2)) as return_amt, cast(0 as decimal(7,2)) as net_loss from web_sales union all select ws_web_site_sk as wsr_web_site_sk, wr_returned_date_sk as date_sk, cast(0 as decimal(7,2)) as sales_price, cast(0 as decimal(7,2)) as profit, wr_return_amt as return_amt, wr_net_loss as net_loss from web_returns left outer join web_sales on ( wr_item_sk = ws_item_sk and wr_order_number = ws_order_number) ) salesreturns, date_dim, web_site where date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 14) and wsr_web_site_sk = web_site_sk group by web_site_id) select channel , id , sum(sales) as sales , sum(returns) as returns , sum(profit) as profit from (select 'store channel' as channel , 'store' || s_store_id as id , sales , returns , (profit - profit_loss) as profit from ssr union all select 'catalog channel' as channel , 'catalog_page' || cp_catalog_page_id as id , sales , returns , (profit - profit_loss) as profit from csr union all select 'web channel' as channel , 'web_site' || web_site_id as id , sales , returns , (profit - profit_loss) as profit from wsr ) x group by rollup (channel, id) order by channel ,id limit 100; -- query 6 select a.ca_state state, count(*) cnt from customer_address a ,customer c ,store_sales s ,date_dim d ,item i where a.ca_address_sk = c.c_current_addr_sk and c.c_customer_sk = s.ss_customer_sk and s.ss_sold_date_sk = d.d_date_sk and s.ss_item_sk = i.i_item_sk and d.d_month_seq = (select distinct (d_month_seq) from date_dim where d_year = 2001 and d_moy = 1 ) and i.i_current_price > 1.2 * (select avg(j.i_current_price) from item j where j.i_category = i.i_category) group by a.ca_state having count(*) >= 10 order by cnt, a.ca_state limit 100; -- query 7 select i_item_id, avg(ss_quantity) agg1, avg(ss_list_price) agg2, avg(ss_coupon_amt) agg3, avg(ss_sales_price) agg4 from store_sales, customer_demographics, date_dim, item, promotion where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and ss_cdemo_sk = cd_demo_sk and ss_promo_sk = p_promo_sk and cd_gender = 'M' and cd_marital_status = 'S' and cd_education_status = 'College' and (p_channel_email = 'N' or p_channel_event = 'N') and d_year = 2000 group by i_item_id order by i_item_id limit 100; -- query 8 select s_store_name ,sum(ss_net_profit) from store_sales ,date_dim ,store, (select ca_zip from ( SELECT substr(ca_zip,1,5) ca_zip FROM customer_address WHERE substr(ca_zip,1,5) IN ( '24128','76232','65084','87816','83926','77556','20548','26231','43848','15126', '91137','61265','98294','25782','17920','18426','98235','40081','84093','28577', '55565','17183','54601','67897','22752','86284','18376','38607','45200','21756', '29741','96765','23932','89360','29839','25989','28898','91068','72550','10390', '18845','47770','82636','41367','76638','86198','81312','37126','39192','88424', '72175','81426','53672','10445','42666','66864','66708','41248','48583','82276', '18842','78890','49448','14089','38122','34425','79077','19849','43285','39861', '66162','77610','13695','99543','83444','83041','12305','57665','68341','25003', '57834','62878','49130','81096','18840','27700','23470','50412','21195','16021', '76107','71954','68309','18119','98359','64544','10336','86379','27068','39736', '98569','28915','24206','56529','57647','54917','42961','91110','63981','14922', '36420','23006','67467','32754','30903','20260','31671','51798','72325','85816', '68621','13955','36446','41766','68806','16725','15146','22744','35850','88086', '51649','18270','52867','39972','96976','63792','11376','94898','13595','10516', '90225','58943','39371','94945','28587','96576','57855','28488','26105','83933', '25858','34322','44438','73171','30122','34102','22685','71256','78451','54364', '13354','45375','40558','56458','28286','45266','47305','69399','83921','26233', '11101','15371','69913','35942','15882','25631','24610','44165','99076','33786', '70738','26653','14328','72305','62496','22152','10144','64147','48425','14663', '21076','18799','30450','63089','81019','68893','24996','51200','51211','45692', '92712','70466','79994','22437','25280','38935','71791','73134','56571','14060', '19505','72425','56575','74351','68786','51650','20004','18383','76614','11634', '18906','15765','41368','73241','76698','78567','97189','28545','76231','75691', '22246','51061','90578','56691','68014','51103','94167','57047','14867','73520', '15734','63435','25733','35474','24676','94627','53535','17879','15559','53268', '59166','11928','59402','33282','45721','43933','68101','33515','36634','71286', '19736','58058','55253','67473','41918','19515','36495','19430','22351','77191', '91393','49156','50298','87501','18652','53179','18767','63193','23968','65164', '68880','21286','72823','58470','67301','13394','31016','70372','67030','40604', '24317','45748','39127','26065','77721','31029','31880','60576','24671','45549', '13376','50016','33123','19769','22927','97789','46081','72151','15723','46136', '51949','68100','96888','64528','14171','79777','28709','11489','25103','32213', '78668','22245','15798','27156','37930','62971','21337','51622','67853','10567', '38415','15455','58263','42029','60279','37125','56240','88190','50308','26859', '64457','89091','82136','62377','36233','63837','58078','17043','30010','60099', '28810','98025','29178','87343','73273','30469','64034','39516','86057','21309', '90257','67875','40162','11356','73650','61810','72013','30431','22461','19512', '13375','55307','30625','83849','68908','26689','96451','38193','46820','88885', '84935','69035','83144','47537','56616','94983','48033','69952','25486','61547', '27385','61860','58048','56910','16807','17871','35258','31387','35458','35576') intersect select ca_zip from (SELECT substr(ca_zip,1,5) ca_zip,count(*) cnt FROM customer_address, customer WHERE ca_address_sk = c_current_addr_sk and c_preferred_cust_flag='Y' group by ca_zip having count(*) > 10)A1)A2) V1 where ss_store_sk = s_store_sk and ss_sold_date_sk = d_date_sk and d_qoy = 2 and d_year = 1998 and (substr(s_zip,1,2) = substr(V1.ca_zip,1,2)) group by s_store_name order by s_store_name limit 100; -- query 9 select case when (select count(*) from store_sales where ss_quantity between 1 and 20) > 74219 then (select avg(ss_ext_discount_amt) from store_sales where ss_quantity between 1 and 20) else (select avg(ss_net_paid) from store_sales where ss_quantity between 1 and 20) end bucket1 , case when (select count(*) from store_sales where ss_quantity between 21 and 40) > 122840 then (select avg(ss_ext_discount_amt) from store_sales where ss_quantity between 21 and 40) else (select avg(ss_net_paid) from store_sales where ss_quantity between 21 and 40) end bucket2, case when (select count(*) from store_sales where ss_quantity between 41 and 60) > 56580 then (select avg(ss_ext_discount_amt) from store_sales where ss_quantity between 41 and 60) else (select avg(ss_net_paid) from store_sales where ss_quantity between 41 and 60) end bucket3, case when (select count(*) from store_sales where ss_quantity between 61 and 80) > 10097 then (select avg(ss_ext_discount_amt) from store_sales where ss_quantity between 61 and 80) else (select avg(ss_net_paid) from store_sales where ss_quantity between 61 and 80) end bucket4, case when (select count(*) from store_sales where ss_quantity between 81 and 100) > 165306 then (select avg(ss_ext_discount_amt) from store_sales where ss_quantity between 81 and 100) else (select avg(ss_net_paid) from store_sales where ss_quantity between 81 and 100) end bucket5 from reason where r_reason_sk = 1 ; -- query 10 select cd_gender, cd_marital_status, cd_education_status, count(*) cnt1, cd_purchase_estimate, count(*) cnt2, cd_credit_rating, count(*) cnt3, cd_dep_count, count(*) cnt4, cd_dep_employed_count, count(*) cnt5, cd_dep_college_count, count(*) cnt6 from customer c,customer_address ca,customer_demographics where c.c_current_addr_sk = ca.ca_address_sk and ca_county in ('Rush County','Toole County','Jefferson County','Dona Ana County','La Porte County') and cd_demo_sk = c.c_current_cdemo_sk and exists (select * from store_sales,date_dim where c.c_customer_sk = ss_customer_sk and ss_sold_date_sk = d_date_sk and d_year = 2002 and d_moy between 1 and 1+3) and exists (select * from ( select ws_bill_customer_sk as customer_sk, d_year,d_moy from web_sales, date_dim where ws_sold_date_sk = d_date_sk and d_year = 2002 and d_moy between 1 and 1+3 union all select cs_ship_customer_sk as customer_sk, d_year, d_moy from catalog_sales, date_dim where cs_sold_date_sk = d_date_sk and d_year = 2002 and d_moy between 1 and 1+3 ) x where c.c_customer_sk = customer_sk) group by cd_gender, cd_marital_status, cd_education_status, cd_purchase_estimate, cd_credit_rating, cd_dep_count, cd_dep_employed_count, cd_dep_college_count order by cd_gender, cd_marital_status, cd_education_status, cd_purchase_estimate, cd_credit_rating, cd_dep_count, cd_dep_employed_count, cd_dep_college_count limit 100; -- query 11 with year_total as ( select c_customer_id customer_id ,c_first_name customer_first_name ,c_last_name customer_last_name ,c_preferred_cust_flag customer_preferred_cust_flag ,c_birth_country customer_birth_country ,c_login customer_login ,c_email_address customer_email_address ,d_year dyear ,sum(ss_ext_list_price-ss_ext_discount_amt) year_total ,'s' sale_type from customer ,store_sales ,date_dim where c_customer_sk = ss_customer_sk and ss_sold_date_sk = d_date_sk group by c_customer_id ,c_first_name ,c_last_name ,c_preferred_cust_flag ,c_birth_country ,c_login ,c_email_address ,d_year union all select c_customer_id customer_id ,c_first_name customer_first_name ,c_last_name customer_last_name ,c_preferred_cust_flag customer_preferred_cust_flag ,c_birth_country customer_birth_country ,c_login customer_login ,c_email_address customer_email_address ,d_year dyear ,sum(ws_ext_list_price-ws_ext_discount_amt) year_total ,'w' sale_type from customer ,web_sales ,date_dim where c_customer_sk = ws_bill_customer_sk and ws_sold_date_sk = d_date_sk group by c_customer_id ,c_first_name ,c_last_name ,c_preferred_cust_flag ,c_birth_country ,c_login ,c_email_address ,d_year ) select t_s_secyear.customer_id ,t_s_secyear.customer_first_name ,t_s_secyear.customer_last_name ,t_s_secyear.customer_preferred_cust_flag from year_total t_s_firstyear ,year_total t_s_secyear ,year_total t_w_firstyear ,year_total t_w_secyear where t_s_secyear.customer_id = t_s_firstyear.customer_id and t_s_firstyear.customer_id = t_w_secyear.customer_id and t_s_firstyear.customer_id = t_w_firstyear.customer_id and t_s_firstyear.sale_type = 's' and t_w_firstyear.sale_type = 'w' and t_s_secyear.sale_type = 's' and t_w_secyear.sale_type = 'w' and t_s_firstyear.dyear = 2001 and t_s_secyear.dyear = 2001+1 and t_w_firstyear.dyear = 2001 and t_w_secyear.dyear = 2001+1 and t_s_firstyear.year_total > 0 and t_w_firstyear.year_total > 0 and case when t_w_firstyear.year_total > 0 then t_w_secyear.year_total / t_w_firstyear.year_total else 0.0 end > case when t_s_firstyear.year_total > 0 then t_s_secyear.year_total / t_s_firstyear.year_total else 0.0 end order by t_s_secyear.customer_id ,t_s_secyear.customer_first_name ,t_s_secyear.customer_last_name ,t_s_secyear.customer_preferred_cust_flag limit 100; -- query 12 select i_item_id ,i_item_desc ,i_category ,i_class ,i_current_price ,sum(ws_ext_sales_price) as itemrevenue ,sum(ws_ext_sales_price)*100/sum(sum(ws_ext_sales_price)) over (partition by i_class) as revenueratio from web_sales ,item ,date_dim where ws_item_sk = i_item_sk and i_category in ('Sports', 'Books', 'Home') and ws_sold_date_sk = d_date_sk and d_date between cast('1999-02-22' as date) and date_add(cast('1999-02-22' as date), 30) group by i_item_id ,i_item_desc ,i_category ,i_class ,i_current_price order by i_category ,i_class ,i_item_id ,i_item_desc ,revenueratio limit 100; -- query 13 select avg(ss_quantity) ,avg(ss_ext_sales_price) ,avg(ss_ext_wholesale_cost) ,sum(ss_ext_wholesale_cost) from store_sales ,store ,customer_demographics ,household_demographics ,customer_address ,date_dim where s_store_sk = ss_store_sk and ss_sold_date_sk = d_date_sk and d_year = 2001 and((ss_hdemo_sk=hd_demo_sk and cd_demo_sk = ss_cdemo_sk and cd_marital_status = 'M' and cd_education_status = 'Advanced Degree' and ss_sales_price between 100.00 and 150.00 and hd_dep_count = 3 )or (ss_hdemo_sk=hd_demo_sk and cd_demo_sk = ss_cdemo_sk and cd_marital_status = 'S' and cd_education_status = 'College' and ss_sales_price between 50.00 and 100.00 and hd_dep_count = 1 ) or (ss_hdemo_sk=hd_demo_sk and cd_demo_sk = ss_cdemo_sk and cd_marital_status = 'W' and cd_education_status = '2 yr Degree' and ss_sales_price between 150.00 and 200.00 and hd_dep_count = 1 )) and((ss_addr_sk = ca_address_sk and ca_country = 'United States' and ca_state in ('TX', 'OH', 'TX') and ss_net_profit between 100 and 200 ) or (ss_addr_sk = ca_address_sk and ca_country = 'United States' and ca_state in ('OR', 'NM', 'KY') and ss_net_profit between 150 and 300 ) or (ss_addr_sk = ca_address_sk and ca_country = 'United States' and ca_state in ('VA', 'TX', 'MS') and ss_net_profit between 50 and 250 )) ; -- query 14 with cross_items as (select i_item_sk ss_item_sk from item, (select iss.i_brand_id brand_id ,iss.i_class_id class_id ,iss.i_category_id category_id from store_sales ,item iss ,date_dim d1 where ss_item_sk = iss.i_item_sk and ss_sold_date_sk = d1.d_date_sk and d1.d_year between 1999 AND 1999 + 2 intersect select ics.i_brand_id ,ics.i_class_id ,ics.i_category_id from catalog_sales ,item ics ,date_dim d2 where cs_item_sk = ics.i_item_sk and cs_sold_date_sk = d2.d_date_sk and d2.d_year between 1999 AND 1999 + 2 intersect select iws.i_brand_id ,iws.i_class_id ,iws.i_category_id from web_sales ,item iws ,date_dim d3 where ws_item_sk = iws.i_item_sk and ws_sold_date_sk = d3.d_date_sk and d3.d_year between 1999 AND 1999 + 2) t where i_brand_id = brand_id and i_class_id = class_id and i_category_id = category_id ), avg_sales as (select avg(quantity*list_price) average_sales from (select ss_quantity quantity ,ss_list_price list_price from store_sales ,date_dim where ss_sold_date_sk = d_date_sk and d_year between 1999 and 1999 + 2 union all select cs_quantity quantity ,cs_list_price list_price from catalog_sales ,date_dim where cs_sold_date_sk = d_date_sk and d_year between 1999 and 1999 + 2 union all select ws_quantity quantity ,ws_list_price list_price from web_sales ,date_dim where ws_sold_date_sk = d_date_sk and d_year between 1999 and 1999 + 2) x) select channel, i_brand_id,i_class_id,i_category_id,sum(sales), sum(number_sales) from( select 'store' channel, i_brand_id,i_class_id ,i_category_id,sum(ss_quantity*ss_list_price) sales , count(*) number_sales from store_sales ,item ,date_dim where ss_item_sk in (select ss_item_sk from cross_items) and ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and d_year = 1999+2 and d_moy = 11 group by i_brand_id,i_class_id,i_category_id having sum(ss_quantity*ss_list_price) > (select average_sales from avg_sales) union all select 'catalog' channel, i_brand_id,i_class_id,i_category_id, sum(cs_quantity*cs_list_price) sales, count(*) number_sales from catalog_sales ,item ,date_dim where cs_item_sk in (select ss_item_sk from cross_items) and cs_item_sk = i_item_sk and cs_sold_date_sk = d_date_sk and d_year = 1999+2 and d_moy = 11 group by i_brand_id,i_class_id,i_category_id having sum(cs_quantity*cs_list_price) > (select average_sales from avg_sales) union all select 'web' channel, i_brand_id,i_class_id,i_category_id, sum(ws_quantity*ws_list_price) sales , count(*) number_sales from web_sales ,item ,date_dim where ws_item_sk in (select ss_item_sk from cross_items) and ws_item_sk = i_item_sk and ws_sold_date_sk = d_date_sk and d_year = 1999+2 and d_moy = 11 group by i_brand_id,i_class_id,i_category_id having sum(ws_quantity*ws_list_price) > (select average_sales from avg_sales) ) y group by rollup (channel, i_brand_id,i_class_id,i_category_id) order by channel,i_brand_id,i_class_id,i_category_id limit 100; with cross_items as (select i_item_sk ss_item_sk from item, (select iss.i_brand_id brand_id ,iss.i_class_id class_id ,iss.i_category_id category_id from store_sales ,item iss ,date_dim d1 where ss_item_sk = iss.i_item_sk and ss_sold_date_sk = d1.d_date_sk and d1.d_year between 1999 AND 1999 + 2 intersect select ics.i_brand_id ,ics.i_class_id ,ics.i_category_id from catalog_sales ,item ics ,date_dim d2 where cs_item_sk = ics.i_item_sk and cs_sold_date_sk = d2.d_date_sk and d2.d_year between 1999 AND 1999 + 2 intersect select iws.i_brand_id ,iws.i_class_id ,iws.i_category_id from web_sales ,item iws ,date_dim d3 where ws_item_sk = iws.i_item_sk and ws_sold_date_sk = d3.d_date_sk and d3.d_year between 1999 AND 1999 + 2) x where i_brand_id = brand_id and i_class_id = class_id and i_category_id = category_id ), avg_sales as (select avg(quantity*list_price) average_sales from (select ss_quantity quantity ,ss_list_price list_price from store_sales ,date_dim where ss_sold_date_sk = d_date_sk and d_year between 1999 and 1999 + 2 union all select cs_quantity quantity ,cs_list_price list_price from catalog_sales ,date_dim where cs_sold_date_sk = d_date_sk and d_year between 1999 and 1999 + 2 union all select ws_quantity quantity ,ws_list_price list_price from web_sales ,date_dim where ws_sold_date_sk = d_date_sk and d_year between 1999 and 1999 + 2) x) select this_year.channel ty_channel ,this_year.i_brand_id ty_brand ,this_year.i_class_id ty_class ,this_year.i_category_id ty_category ,this_year.sales ty_sales ,this_year.number_sales ty_number_sales ,last_year.channel ly_channel ,last_year.i_brand_id ly_brand ,last_year.i_class_id ly_class ,last_year.i_category_id ly_category ,last_year.sales ly_sales ,last_year.number_sales ly_number_sales from (select 'store' channel, i_brand_id,i_class_id,i_category_id ,sum(ss_quantity*ss_list_price) sales, count(*) number_sales from store_sales ,item ,date_dim where ss_item_sk in (select ss_item_sk from cross_items) and ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and d_week_seq = (select d_week_seq from date_dim where d_year = 1999 + 1 and d_moy = 12 and d_dom = 11) group by i_brand_id,i_class_id,i_category_id having sum(ss_quantity*ss_list_price) > (select average_sales from avg_sales)) this_year, (select 'store' channel, i_brand_id,i_class_id ,i_category_id, sum(ss_quantity*ss_list_price) sales, count(*) number_sales from store_sales ,item ,date_dim where ss_item_sk in (select ss_item_sk from cross_items) and ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and d_week_seq = (select d_week_seq from date_dim where d_year = 1999 and d_moy = 12 and d_dom = 11) group by i_brand_id,i_class_id,i_category_id having sum(ss_quantity*ss_list_price) > (select average_sales from avg_sales)) last_year where this_year.i_brand_id= last_year.i_brand_id and this_year.i_class_id = last_year.i_class_id and this_year.i_category_id = last_year.i_category_id order by this_year.channel, this_year.i_brand_id, this_year.i_class_id, this_year.i_category_id limit 100; -- query 15 select ca_zip ,sum(cs_sales_price) from catalog_sales ,customer ,customer_address ,date_dim where cs_bill_customer_sk = c_customer_sk and c_current_addr_sk = ca_address_sk and ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475', '85392', '85460', '80348', '81792') or ca_state in ('CA','WA','GA') or cs_sales_price > 500) and cs_sold_date_sk = d_date_sk and d_qoy = 2 and d_year = 2001 group by ca_zip order by ca_zip limit 100; -- query 16 select count(distinct cs_order_number) as "order count" ,sum(cs_ext_ship_cost) as "total shipping cost" ,sum(cs_net_profit) as "total net profit" from catalog_sales cs1 ,date_dim ,customer_address ,call_center where d_date between '2002-2-01' and date_add(cast('2002-2-01' as date), 60) and cs1.cs_ship_date_sk = d_date_sk and cs1.cs_ship_addr_sk = ca_address_sk and ca_state = 'GA' and cs1.cs_call_center_sk = cc_call_center_sk and cc_county in ('Williamson County', 'Williamson County', 'Williamson County', 'Williamson County', 'Williamson County') and exists (select * from catalog_sales cs2 where cs1.cs_order_number = cs2.cs_order_number and cs1.cs_warehouse_sk <> cs2.cs_warehouse_sk) and not exists(select * from catalog_returns cr1 where cs1.cs_order_number = cr1.cr_order_number) order by count(distinct cs_order_number) limit 100; -- query 17 select i_item_id ,i_item_desc ,s_state ,count(ss_quantity) as store_sales_quantitycount ,avg(ss_quantity) as store_sales_quantityave ,stddev_samp(ss_quantity) as store_sales_quantitystdev ,stddev_samp(ss_quantity)/avg(ss_quantity) as store_sales_quantitycov ,count(sr_return_quantity) as store_returns_quantitycount ,avg(sr_return_quantity) as store_returns_quantityave ,stddev_samp(sr_return_quantity) as store_returns_quantitystdev ,stddev_samp(sr_return_quantity)/avg(sr_return_quantity) as store_returns_quantitycov ,count(cs_quantity) as catalog_sales_quantitycount ,avg(cs_quantity) as catalog_sales_quantityave ,stddev_samp(cs_quantity) as catalog_sales_quantitystdev ,stddev_samp(cs_quantity)/avg(cs_quantity) as catalog_sales_quantitycov from store_sales ,store_returns ,catalog_sales ,date_dim d1 ,date_dim d2 ,date_dim d3 ,store ,item where d1.d_quarter_name = '2001Q1' and d1.d_date_sk = ss_sold_date_sk and i_item_sk = ss_item_sk and s_store_sk = ss_store_sk and ss_customer_sk = sr_customer_sk and ss_item_sk = sr_item_sk and ss_ticket_number = sr_ticket_number and sr_returned_date_sk = d2.d_date_sk and d2.d_quarter_name in ('2001Q1','2001Q2','2001Q3') and sr_customer_sk = cs_bill_customer_sk and sr_item_sk = cs_item_sk and cs_sold_date_sk = d3.d_date_sk and d3.d_quarter_name in ('2001Q1','2001Q2','2001Q3') group by i_item_id ,i_item_desc ,s_state order by i_item_id ,i_item_desc ,s_state limit 100; -- query 18 select i_item_id, ca_country, ca_state, ca_county, avg( cast(cs_quantity as decimal(12,2))) agg1, avg( cast(cs_list_price as decimal(12,2))) agg2, avg( cast(cs_coupon_amt as decimal(12,2))) agg3, avg( cast(cs_sales_price as decimal(12,2))) agg4, avg( cast(cs_net_profit as decimal(12,2))) agg5, avg( cast(c_birth_year as decimal(12,2))) agg6, avg( cast(cd1.cd_dep_count as decimal(12,2))) agg7 from catalog_sales, customer_demographics cd1, customer_demographics cd2, customer, customer_address, date_dim, item where cs_sold_date_sk = d_date_sk and cs_item_sk = i_item_sk and cs_bill_cdemo_sk = cd1.cd_demo_sk and cs_bill_customer_sk = c_customer_sk and cd1.cd_gender = 'F' and cd1.cd_education_status = 'Unknown' and c_current_cdemo_sk = cd2.cd_demo_sk and c_current_addr_sk = ca_address_sk and c_birth_month in (1, 6, 8, 9, 12, 2) and d_year = 1998 and ca_state in ('MS','IN','ND' ,'OK','NM','VA','MS') group by rollup (i_item_id, ca_country, ca_state, ca_county) order by ca_country, ca_state, ca_county, i_item_id limit 100; -- query 19 select i_brand_id brand_id, i_brand brand, i_manufact_id, i_manufact, sum(ss_ext_sales_price) ext_price from date_dim, store_sales, item,customer,customer_address,store where d_date_sk = ss_sold_date_sk and ss_item_sk = i_item_sk and i_manager_id=8 and d_moy=11 and d_year=1998 and ss_customer_sk = c_customer_sk and c_current_addr_sk = ca_address_sk and substr(ca_zip,1,5) <> substr(s_zip,1,5) and ss_store_sk = s_store_sk group by i_brand ,i_brand_id ,i_manufact_id ,i_manufact order by ext_price desc ,i_brand ,i_brand_id ,i_manufact_id ,i_manufact limit 100 ; -- query 20 select i_item_id ,i_item_desc ,i_category ,i_class ,i_current_price ,sum(cs_ext_sales_price) as itemrevenue ,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over (partition by i_class) as revenueratio from catalog_sales ,item ,date_dim where cs_item_sk = i_item_sk and i_category in ('Sports','Books','Home') and cs_sold_date_sk = d_date_sk and d_date between cast('1999-02-22' as date) and date_add(cast('1999-02-22' as date), 30) group by i_item_id ,i_item_desc ,i_category ,i_class ,i_current_price order by i_category ,i_class ,i_item_id ,i_item_desc ,revenueratio limit 100; -- query 21 select * from(select w_warehouse_name ,i_item_id ,sum(case when (cast(d_date as date) < cast ('2000-03-11' as date)) then inv_quantity_on_hand else 0 end) as inv_before ,sum(case when (cast(d_date as date) >= cast ('2000-03-11' as date)) then inv_quantity_on_hand else 0 end) as inv_after from inventory ,warehouse ,item ,date_dim where i_current_price between 0.99 and 1.49 and i_item_sk = inv_item_sk and inv_warehouse_sk = w_warehouse_sk and inv_date_sk = d_date_sk and d_date between date_add(cast ('2000-03-11' as date), -30) and date_add(cast ('2000-03-11' as date), 30) group by w_warehouse_name, i_item_id) x where (case when inv_before > 0 then inv_after / inv_before else null end) between 2.0/3.0 and 3.0/2.0 order by w_warehouse_name ,i_item_id limit 100; -- query 22 select i_product_name ,i_brand ,i_class ,i_category ,avg(inv_quantity_on_hand) qoh from inventory ,date_dim ,item where inv_date_sk=d_date_sk and inv_item_sk=i_item_sk and d_month_seq between 1200 and 1200 + 11 group by rollup(i_product_name ,i_brand ,i_class ,i_category) order by qoh, i_product_name, i_brand, i_class, i_category limit 100; -- query 23 with frequent_ss_items as (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date solddate,count(*) cnt from store_sales ,date_dim ,item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and d_year in (2000,2000+1,2000+2,2000+3) group by substr(i_item_desc,1,30),i_item_sk,d_date having count(*) >4), max_store_sales as (select max(csales) tpcds_cmax from (select c_customer_sk,sum(ss_quantity*ss_sales_price) csales from store_sales ,customer ,date_dim where ss_customer_sk = c_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (2000,2000+1,2000+2,2000+3) group by c_customer_sk) t1), best_ss_customer as (select c_customer_sk,sum(ss_quantity*ss_sales_price) ssales from store_sales ,customer where ss_customer_sk = c_customer_sk group by c_customer_sk having sum(ss_quantity*ss_sales_price) > (50/100.0) * (select * from max_store_sales)) select sum(sales) from (select cs_quantity*cs_list_price sales from catalog_sales ,date_dim where d_year = 2000 and d_moy = 2 and cs_sold_date_sk = d_date_sk and cs_item_sk in (select item_sk from frequent_ss_items) and cs_bill_customer_sk in (select c_customer_sk from best_ss_customer) union all select ws_quantity*ws_list_price sales from web_sales ,date_dim where d_year = 2000 and d_moy = 2 and ws_sold_date_sk = d_date_sk and ws_item_sk in (select item_sk from frequent_ss_items) and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer)) t2 limit 100; with frequent_ss_items as (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date solddate,count(*) cnt from store_sales ,date_dim ,item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and d_year in (2000,2000 + 1,2000 + 2,2000 + 3) group by substr(i_item_desc,1,30),i_item_sk,d_date having count(*) >4), max_store_sales as (select max(csales) tpcds_cmax from (select c_customer_sk,sum(ss_quantity*ss_sales_price) csales from store_sales ,customer ,date_dim where ss_customer_sk = c_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (2000,2000+1,2000+2,2000+3) group by c_customer_sk) t1), best_ss_customer as (select c_customer_sk,sum(ss_quantity*ss_sales_price) ssales from store_sales ,customer where ss_customer_sk = c_customer_sk group by c_customer_sk having sum(ss_quantity*ss_sales_price) > (50/100.0) * (select * from max_store_sales)) select c_last_name,c_first_name,sales from (select c_last_name,c_first_name,sum(cs_quantity*cs_list_price) sales from catalog_sales ,customer ,date_dim where d_year = 2000 and d_moy = 2 and cs_sold_date_sk = d_date_sk and cs_item_sk in (select item_sk from frequent_ss_items) and cs_bill_customer_sk in (select c_customer_sk from best_ss_customer) and cs_bill_customer_sk = c_customer_sk group by c_last_name,c_first_name union all select c_last_name,c_first_name,sum(ws_quantity*ws_list_price) sales from web_sales ,customer ,date_dim where d_year = 2000 and d_moy = 2 and ws_sold_date_sk = d_date_sk and ws_item_sk in (select item_sk from frequent_ss_items) and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer) and ws_bill_customer_sk = c_customer_sk group by c_last_name,c_first_name) t2 order by c_last_name,c_first_name,sales limit 100; -- query 24 with ssales as (select c_last_name ,c_first_name ,s_store_name ,ca_state ,s_state ,i_color ,i_current_price ,i_manager_id ,i_units ,i_size ,sum(ss_net_paid) netpaid from store_sales ,store_returns ,store ,item ,customer ,customer_address where ss_ticket_number = sr_ticket_number and ss_item_sk = sr_item_sk and ss_customer_sk = c_customer_sk and ss_item_sk = i_item_sk and ss_store_sk = s_store_sk and c_current_addr_sk = ca_address_sk and c_birth_country <> upper(ca_country) and s_zip = ca_zip and s_market_id=8 group by c_last_name ,c_first_name ,s_store_name ,ca_state ,s_state ,i_color ,i_current_price ,i_manager_id ,i_units ,i_size) select c_last_name ,c_first_name ,s_store_name ,sum(netpaid) paid from ssales where i_color = 'peach' group by c_last_name ,c_first_name ,s_store_name having sum(netpaid) > (select 0.05*avg(netpaid) from ssales) order by c_last_name ,c_first_name ,s_store_name ; with ssales as (select c_last_name ,c_first_name ,s_store_name ,ca_state ,s_state ,i_color ,i_current_price ,i_manager_id ,i_units ,i_size ,sum(ss_net_paid) netpaid from store_sales ,store_returns ,store ,item ,customer ,customer_address where ss_ticket_number = sr_ticket_number and ss_item_sk = sr_item_sk and ss_customer_sk = c_customer_sk and ss_item_sk = i_item_sk and ss_store_sk = s_store_sk and c_current_addr_sk = ca_address_sk and c_birth_country <> upper(ca_country) and s_zip = ca_zip and s_market_id = 8 group by c_last_name ,c_first_name ,s_store_name ,ca_state ,s_state ,i_color ,i_current_price ,i_manager_id ,i_units ,i_size) select c_last_name ,c_first_name ,s_store_name ,sum(netpaid) paid from ssales where i_color = 'saddle' group by c_last_name ,c_first_name ,s_store_name having sum(netpaid) > (select 0.05*avg(netpaid) from ssales) order by c_last_name ,c_first_name ,s_store_name ; -- query 25 select i_item_id ,i_item_desc ,s_store_id ,s_store_name ,sum(ss_net_profit) as store_sales_profit ,sum(sr_net_loss) as store_returns_loss ,sum(cs_net_profit) as catalog_sales_profit from store_sales ,store_returns ,catalog_sales ,date_dim d1 ,date_dim d2 ,date_dim d3 ,store ,item where d1.d_moy = 4 and d1.d_year = 2001 and d1.d_date_sk = ss_sold_date_sk and i_item_sk = ss_item_sk and s_store_sk = ss_store_sk and ss_customer_sk = sr_customer_sk and ss_item_sk = sr_item_sk and ss_ticket_number = sr_ticket_number and sr_returned_date_sk = d2.d_date_sk and d2.d_moy between 4 and 10 and d2.d_year = 2001 and sr_customer_sk = cs_bill_customer_sk and sr_item_sk = cs_item_sk and cs_sold_date_sk = d3.d_date_sk and d3.d_moy between 4 and 10 and d3.d_year = 2001 group by i_item_id ,i_item_desc ,s_store_id ,s_store_name order by i_item_id ,i_item_desc ,s_store_id ,s_store_name limit 100; -- query 26 select i_item_id, avg(cs_quantity) agg1, avg(cs_list_price) agg2, avg(cs_coupon_amt) agg3, avg(cs_sales_price) agg4 from catalog_sales, customer_demographics, date_dim, item, promotion where cs_sold_date_sk = d_date_sk and cs_item_sk = i_item_sk and cs_bill_cdemo_sk = cd_demo_sk and cs_promo_sk = p_promo_sk and cd_gender = 'M' and cd_marital_status = 'S' and cd_education_status = 'College' and (p_channel_email = 'N' or p_channel_event = 'N') and d_year = 2000 group by i_item_id order by i_item_id limit 100; -- query 27 select i_item_id, s_state, grouping(s_state) g_state, avg(ss_quantity) agg1, avg(ss_list_price) agg2, avg(ss_coupon_amt) agg3, avg(ss_sales_price) agg4 from store_sales, customer_demographics, date_dim, store, item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and ss_store_sk = s_store_sk and ss_cdemo_sk = cd_demo_sk and cd_gender = 'M' and cd_marital_status = 'S' and cd_education_status = 'College' and d_year = 2002 and s_state in ('TN','TN', 'TN', 'TN', 'TN', 'TN') group by rollup (i_item_id, s_state) order by i_item_id ,s_state limit 100; -- query 28 select * from (select avg(ss_list_price) B1_LP ,count(ss_list_price) B1_CNT ,count(distinct ss_list_price) B1_CNTD from store_sales where ss_quantity between 0 and 5 and (ss_list_price between 8 and 8+10 or ss_coupon_amt between 459 and 459+1000 or ss_wholesale_cost between 57 and 57+20)) B1, (select avg(ss_list_price) B2_LP ,count(ss_list_price) B2_CNT ,count(distinct ss_list_price) B2_CNTD from store_sales where ss_quantity between 6 and 10 and (ss_list_price between 90 and 90+10 or ss_coupon_amt between 2323 and 2323+1000 or ss_wholesale_cost between 31 and 31+20)) B2, (select avg(ss_list_price) B3_LP ,count(ss_list_price) B3_CNT ,count(distinct ss_list_price) B3_CNTD from store_sales where ss_quantity between 11 and 15 and (ss_list_price between 142 and 142+10 or ss_coupon_amt between 12214 and 12214+1000 or ss_wholesale_cost between 79 and 79+20)) B3, (select avg(ss_list_price) B4_LP ,count(ss_list_price) B4_CNT ,count(distinct ss_list_price) B4_CNTD from store_sales where ss_quantity between 16 and 20 and (ss_list_price between 135 and 135+10 or ss_coupon_amt between 6071 and 6071+1000 or ss_wholesale_cost between 38 and 38+20)) B4, (select avg(ss_list_price) B5_LP ,count(ss_list_price) B5_CNT ,count(distinct ss_list_price) B5_CNTD from store_sales where ss_quantity between 21 and 25 and (ss_list_price between 122 and 122+10 or ss_coupon_amt between 836 and 836+1000 or ss_wholesale_cost between 17 and 17+20)) B5, (select avg(ss_list_price) B6_LP ,count(ss_list_price) B6_CNT ,count(distinct ss_list_price) B6_CNTD from store_sales where ss_quantity between 26 and 30 and (ss_list_price between 154 and 154+10 or ss_coupon_amt between 7326 and 7326+1000 or ss_wholesale_cost between 7 and 7+20)) B6 limit 100; -- query 29 select i_item_id ,i_item_desc ,s_store_id ,s_store_name ,sum(ss_quantity) as store_sales_quantity ,sum(sr_return_quantity) as store_returns_quantity ,sum(cs_quantity) as catalog_sales_quantity from store_sales ,store_returns ,catalog_sales ,date_dim d1 ,date_dim d2 ,date_dim d3 ,store ,item where d1.d_moy = 9 and d1.d_year = 1999 and d1.d_date_sk = ss_sold_date_sk and i_item_sk = ss_item_sk and s_store_sk = ss_store_sk and ss_customer_sk = sr_customer_sk and ss_item_sk = sr_item_sk and ss_ticket_number = sr_ticket_number and sr_returned_date_sk = d2.d_date_sk and d2.d_moy between 9 and 9 + 3 and d2.d_year = 1999 and sr_customer_sk = cs_bill_customer_sk and sr_item_sk = cs_item_sk and cs_sold_date_sk = d3.d_date_sk and d3.d_year in (1999,1999+1,1999+2) group by i_item_id ,i_item_desc ,s_store_id ,s_store_name order by i_item_id ,i_item_desc ,s_store_id ,s_store_name limit 100; -- query 30 with customer_total_return as (select wr_returning_customer_sk as ctr_customer_sk ,ca_state as ctr_state, sum(wr_return_amt) as ctr_total_return from web_returns ,date_dim ,customer_address where wr_returned_date_sk = d_date_sk and d_year =2002 and wr_returning_addr_sk = ca_address_sk group by wr_returning_customer_sk ,ca_state) select c_customer_id,c_salutation,c_first_name,c_last_name,c_preferred_cust_flag ,c_birth_day,c_birth_month,c_birth_year,c_birth_country,c_login,c_email_address ,c_last_review_date,ctr_total_return from customer_total_return ctr1 ,customer_address ,customer where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2 from customer_total_return ctr2 where ctr1.ctr_state = ctr2.ctr_state) and ca_address_sk = c_current_addr_sk and ca_state = 'GA' and ctr1.ctr_customer_sk = c_customer_sk order by c_customer_id,c_salutation,c_first_name,c_last_name,c_preferred_cust_flag ,c_birth_day,c_birth_month,c_birth_year,c_birth_country,c_login,c_email_address ,c_last_review_date,ctr_total_return limit 100; -- query 31 with ss as (select ca_county,d_qoy, d_year,sum(ss_ext_sales_price) as store_sales from store_sales,date_dim,customer_address where ss_sold_date_sk = d_date_sk and ss_addr_sk=ca_address_sk group by ca_county,d_qoy, d_year), ws as (select ca_county,d_qoy, d_year,sum(ws_ext_sales_price) as web_sales from web_sales,date_dim,customer_address where ws_sold_date_sk = d_date_sk and ws_bill_addr_sk=ca_address_sk group by ca_county,d_qoy, d_year) select ss1.ca_county ,ss1.d_year ,ws2.web_sales/ws1.web_sales web_q1_q2_increase ,ss2.store_sales/ss1.store_sales store_q1_q2_increase ,ws3.web_sales/ws2.web_sales web_q2_q3_increase ,ss3.store_sales/ss2.store_sales store_q2_q3_increase from ss ss1 ,ss ss2 ,ss ss3 ,ws ws1 ,ws ws2 ,ws ws3 where ss1.d_qoy = 1 and ss1.d_year = 2000 and ss1.ca_county = ss2.ca_county and ss2.d_qoy = 2 and ss2.d_year = 2000 and ss2.ca_county = ss3.ca_county and ss3.d_qoy = 3 and ss3.d_year = 2000 and ss1.ca_county = ws1.ca_county and ws1.d_qoy = 1 and ws1.d_year = 2000 and ws1.ca_county = ws2.ca_county and ws2.d_qoy = 2 and ws2.d_year = 2000 and ws1.ca_county = ws3.ca_county and ws3.d_qoy = 3 and ws3.d_year =2000 and case when ws1.web_sales > 0 then ws2.web_sales/ws1.web_sales else null end > case when ss1.store_sales > 0 then ss2.store_sales/ss1.store_sales else null end and case when ws2.web_sales > 0 then ws3.web_sales/ws2.web_sales else null end > case when ss2.store_sales > 0 then ss3.store_sales/ss2.store_sales else null end order by ss1.ca_county; -- query 32 select sum(cs_ext_discount_amt) as "excess discount amount" from catalog_sales ,item ,date_dim where i_manufact_id = 977 and i_item_sk = cs_item_sk and d_date between '2000-01-27' and date_add(cast('2000-01-27' as date), 90) and d_date_sk = cs_sold_date_sk and cs_ext_discount_amt > ( select 1.3 * avg(cs_ext_discount_amt) from catalog_sales ,date_dim where cs_item_sk = i_item_sk and d_date between '2000-01-27' and date_add(cast('2000-01-27' as date), 90) and d_date_sk = cs_sold_date_sk ) limit 100; -- query 33 with ss as ( select i_manufact_id,sum(ss_ext_sales_price) total_sales from store_sales, date_dim, customer_address, item where i_manufact_id in (select i_manufact_id from item where i_category in ('Electronics')) and ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and d_year = 1998 and d_moy = 5 and ss_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_manufact_id), cs as ( select i_manufact_id,sum(cs_ext_sales_price) total_sales from catalog_sales, date_dim, customer_address, item where i_manufact_id in (select i_manufact_id from item where i_category in ('Electronics')) and cs_item_sk = i_item_sk and cs_sold_date_sk = d_date_sk and d_year = 1998 and d_moy = 5 and cs_bill_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_manufact_id), ws as ( select i_manufact_id,sum(ws_ext_sales_price) total_sales from web_sales, date_dim, customer_address, item where i_manufact_id in (select i_manufact_id from item where i_category in ('Electronics')) and ws_item_sk = i_item_sk and ws_sold_date_sk = d_date_sk and d_year = 1998 and d_moy = 5 and ws_bill_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_manufact_id) select i_manufact_id ,sum(total_sales) total_sales from (select * from ss union all select * from cs union all select * from ws) tmp1 group by i_manufact_id order by total_sales limit 100; -- query 34 select c_last_name ,c_first_name ,c_salutation ,c_preferred_cust_flag ,ss_ticket_number ,cnt from (select ss_ticket_number ,ss_customer_sk ,count(*) cnt from store_sales,date_dim,store,household_demographics where store_sales.ss_sold_date_sk = date_dim.d_date_sk and store_sales.ss_store_sk = store.s_store_sk and store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk and (date_dim.d_dom between 1 and 3 or date_dim.d_dom between 25 and 28) and (household_demographics.hd_buy_potential = '>10000' or household_demographics.hd_buy_potential = 'Unknown') and household_demographics.hd_vehicle_count > 0 and (case when household_demographics.hd_vehicle_count > 0 then household_demographics.hd_dep_count/ household_demographics.hd_vehicle_count else null end) > 1.2 and date_dim.d_year in (1999,1999+1,1999+2) and store.s_county in ('Williamson County','Williamson County','Williamson County','Williamson County', 'Williamson County','Williamson County','Williamson County','Williamson County') group by ss_ticket_number,ss_customer_sk) dn,customer where ss_customer_sk = c_customer_sk and cnt between 15 and 20 order by c_last_name,c_first_name,c_salutation,c_preferred_cust_flag desc, ss_ticket_number; -- query 35 select ca_state, cd_gender, cd_marital_status, cd_dep_count, count(*) cnt1, min(cd_dep_count), max(cd_dep_count), avg(cd_dep_count), cd_dep_employed_count, count(*) cnt2, min(cd_dep_employed_count), max(cd_dep_employed_count), avg(cd_dep_employed_count), cd_dep_college_count, count(*) cnt3, min(cd_dep_college_count), max(cd_dep_college_count), avg(cd_dep_college_count) from customer c,customer_address ca,customer_demographics where c.c_current_addr_sk = ca.ca_address_sk and cd_demo_sk = c.c_current_cdemo_sk and exists (select * from store_sales,date_dim where c.c_customer_sk = ss_customer_sk and ss_sold_date_sk = d_date_sk and d_year = 2002 and d_qoy < 4) and exists (select * from (select ws_bill_customer_sk customsk from web_sales,date_dim where ws_sold_date_sk = d_date_sk and d_year = 2002 and d_qoy < 4 union all select cs_ship_customer_sk customsk from catalog_sales,date_dim where cs_sold_date_sk = d_date_sk and d_year = 2002 and d_qoy < 4)x where x.customsk = c.c_customer_sk) group by ca_state, cd_gender, cd_marital_status, cd_dep_count, cd_dep_employed_count, cd_dep_college_count order by ca_state, cd_gender, cd_marital_status, cd_dep_count, cd_dep_employed_count, cd_dep_college_count limit 100; -- query 36 select sum(ss_net_profit)/sum(ss_ext_sales_price) as gross_margin ,i_category ,i_class ,grouping(i_category)+grouping(i_class) as lochierarchy ,rank() over ( partition by grouping(i_category)+grouping(i_class), case when grouping(i_class) = 0 then i_category end order by sum(ss_net_profit)/sum(ss_ext_sales_price) asc) as rank_within_parent from store_sales ,date_dim d1 ,item ,store where d1.d_year = 2001 and d1.d_date_sk = ss_sold_date_sk and i_item_sk = ss_item_sk and s_store_sk = ss_store_sk and s_state in ('TN','TN','TN','TN', 'TN','TN','TN','TN') group by rollup(i_category,i_class) order by lochierarchy desc ,case when lochierarchy = 0 then i_category end ,rank_within_parent limit 100; -- query 37 select i_item_id ,i_item_desc ,i_current_price from item, inventory, date_dim, catalog_sales where i_current_price between 68 and 68 + 30 and inv_item_sk = i_item_sk and d_date_sk=inv_date_sk and d_date between cast('2000-02-01' as date) and date_add(cast('2000-02-01' as date), 60) and i_manufact_id in (677,940,694,808) and inv_quantity_on_hand between 100 and 500 and cs_item_sk = i_item_sk group by i_item_id,i_item_desc,i_current_price order by i_item_id limit 100; -- query 38 select count(*) from ( select distinct c_last_name, c_first_name, d_date from store_sales, date_dim, customer where store_sales.ss_sold_date_sk = date_dim.d_date_sk and store_sales.ss_customer_sk = customer.c_customer_sk and d_month_seq between 1200 and 1200 + 11 intersect select distinct c_last_name, c_first_name, d_date from catalog_sales, date_dim, customer where catalog_sales.cs_sold_date_sk = date_dim.d_date_sk and catalog_sales.cs_bill_customer_sk = customer.c_customer_sk and d_month_seq between 1200 and 1200 + 11 intersect select distinct c_last_name, c_first_name, d_date from web_sales, date_dim, customer where web_sales.ws_sold_date_sk = date_dim.d_date_sk and web_sales.ws_bill_customer_sk = customer.c_customer_sk and d_month_seq between 1200 and 1200 + 11 ) hot_cust limit 100; -- query 39 with inv as (select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy ,stdev,mean, case mean when 0 then null else stdev/mean end cov from(select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy ,stddev_samp(inv_quantity_on_hand) stdev,avg(inv_quantity_on_hand) mean from inventory ,item ,warehouse ,date_dim where inv_item_sk = i_item_sk and inv_warehouse_sk = w_warehouse_sk and inv_date_sk = d_date_sk and d_year =2001 group by w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy) foo where case mean when 0 then 0 else stdev/mean end > 1) select inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean, inv1.cov ,inv2.w_warehouse_sk,inv2.i_item_sk,inv2.d_moy,inv2.mean, inv2.cov from inv inv1,inv inv2 where inv1.i_item_sk = inv2.i_item_sk and inv1.w_warehouse_sk = inv2.w_warehouse_sk and inv1.d_moy=1 and inv2.d_moy=1+1 order by inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean,inv1.cov ,inv2.d_moy,inv2.mean, inv2.cov ; with inv as (select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy ,stdev,mean, case mean when 0 then null else stdev/mean end cov from(select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy ,stddev_samp(inv_quantity_on_hand) stdev,avg(inv_quantity_on_hand) mean from inventory ,item ,warehouse ,date_dim where inv_item_sk = i_item_sk and inv_warehouse_sk = w_warehouse_sk and inv_date_sk = d_date_sk and d_year =2001 group by w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy) foo where case mean when 0 then 0 else stdev/mean end > 1) select inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean, inv1.cov ,inv2.w_warehouse_sk,inv2.i_item_sk,inv2.d_moy,inv2.mean, inv2.cov from inv inv1,inv inv2 where inv1.i_item_sk = inv2.i_item_sk and inv1.w_warehouse_sk = inv2.w_warehouse_sk and inv1.d_moy=1 and inv2.d_moy=1+1 and inv1.cov > 1.5 order by inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean,inv1.cov ,inv2.d_moy,inv2.mean, inv2.cov ; -- query 40 select w_state ,i_item_id ,sum(case when (cast(d_date as date) < cast ('2000-03-11' as date)) then cs_sales_price - coalesce(cr_refunded_cash,0) else 0 end) as sales_before ,sum(case when (cast(d_date as date) >= cast ('2000-03-11' as date)) then cs_sales_price - coalesce(cr_refunded_cash,0) else 0 end) as sales_after from catalog_sales left outer join catalog_returns on (cs_order_number = cr_order_number and cs_item_sk = cr_item_sk) ,warehouse ,item ,date_dim where i_current_price between 0.99 and 1.49 and i_item_sk = cs_item_sk and cs_warehouse_sk = w_warehouse_sk and cs_sold_date_sk = d_date_sk and d_date between date_add(cast ('2000-03-11' as date), -30) and date_add(cast ('2000-03-11' as date), 30) group by w_state,i_item_id order by w_state,i_item_id limit 100; -- query 41 select distinct(i_product_name) from item i1 where i_manufact_id between 738 and 738+40 and (select count(*) as item_cnt from item where (i_manufact = i1.i_manufact and ((i_category = 'Women' and (i_color = 'powder' or i_color = 'khaki') and (i_units = 'Ounce' or i_units = 'Oz') and (i_size = 'medium' or i_size = 'extra large') ) or (i_category = 'Women' and (i_color = 'brown' or i_color = 'honeydew') and (i_units = 'Bunch' or i_units = 'Ton') and (i_size = 'N/A' or i_size = 'small') ) or (i_category = 'Men' and (i_color = 'floral' or i_color = 'deep') and (i_units = 'N/A' or i_units = 'Dozen') and (i_size = 'petite' or i_size = 'large') ) or (i_category = 'Men' and (i_color = 'light' or i_color = 'cornflower') and (i_units = 'Box' or i_units = 'Pound') and (i_size = 'medium' or i_size = 'extra large') ))) or (i_manufact = i1.i_manufact and ((i_category = 'Women' and (i_color = 'midnight' or i_color = 'snow') and (i_units = 'Pallet' or i_units = 'Gross') and (i_size = 'medium' or i_size = 'extra large') ) or (i_category = 'Women' and (i_color = 'cyan' or i_color = 'papaya') and (i_units = 'Cup' or i_units = 'Dram') and (i_size = 'N/A' or i_size = 'small') ) or (i_category = 'Men' and (i_color = 'orange' or i_color = 'frosted') and (i_units = 'Each' or i_units = 'Tbl') and (i_size = 'petite' or i_size = 'large') ) or (i_category = 'Men' and (i_color = 'forest' or i_color = 'ghost') and (i_units = 'Lb' or i_units = 'Bundle') and (i_size = 'medium' or i_size = 'extra large') )))) > 0 order by i_product_name limit 100; -- query 42 select dt.d_year ,item.i_category_id ,item.i_category ,sum(ss_ext_sales_price) from date_dim dt ,store_sales ,item where dt.d_date_sk = store_sales.ss_sold_date_sk and store_sales.ss_item_sk = item.i_item_sk and item.i_manager_id = 1 and dt.d_moy=11 and dt.d_year=2000 group by dt.d_year ,item.i_category_id ,item.i_category order by sum(ss_ext_sales_price) desc,dt.d_year ,item.i_category_id ,item.i_category limit 100 ; -- query 43 select s_store_name, s_store_id, sum(case when (d_day_name='Sunday') then ss_sales_price else null end) sun_sales, sum(case when (d_day_name='Monday') then ss_sales_price else null end) mon_sales, sum(case when (d_day_name='Tuesday') then ss_sales_price else null end) tue_sales, sum(case when (d_day_name='Wednesday') then ss_sales_price else null end) wed_sales, sum(case when (d_day_name='Thursday') then ss_sales_price else null end) thu_sales, sum(case when (d_day_name='Friday') then ss_sales_price else null end) fri_sales, sum(case when (d_day_name='Saturday') then ss_sales_price else null end) sat_sales from date_dim, store_sales, store where d_date_sk = ss_sold_date_sk and s_store_sk = ss_store_sk and s_gmt_offset = -5 and d_year = 2000 group by s_store_name, s_store_id order by s_store_name, s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales limit 100; -- query 44 select asceding.rnk, i1.i_product_name best_performing, i2.i_product_name worst_performing from(select * from (select item_sk,rank() over (order by rank_col asc) rnk from (select ss_item_sk item_sk,avg(ss_net_profit) rank_col from store_sales ss1 where ss_store_sk = 4 group by ss_item_sk having avg(ss_net_profit) > 0.9*(select avg(ss_net_profit) rank_col from store_sales where ss_store_sk = 4 and ss_addr_sk is null group by ss_store_sk))V1)V11 where rnk < 11) asceding, (select * from (select item_sk,rank() over (order by rank_col desc) rnk from (select ss_item_sk item_sk,avg(ss_net_profit) rank_col from store_sales ss1 where ss_store_sk = 4 group by ss_item_sk having avg(ss_net_profit) > 0.9*(select avg(ss_net_profit) rank_col from store_sales where ss_store_sk = 4 and ss_addr_sk is null group by ss_store_sk))V2)V21 where rnk < 11) descending, item i1, item i2 where asceding.rnk = descending.rnk and i1.i_item_sk=asceding.item_sk and i2.i_item_sk=descending.item_sk order by asceding.rnk limit 100; -- query 45 select ca_zip, ca_city, sum(ws_sales_price) from web_sales, customer, customer_address, date_dim, item where ws_bill_customer_sk = c_customer_sk and c_current_addr_sk = ca_address_sk and ws_item_sk = i_item_sk and ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475', '85392', '85460', '80348', '81792') or i_item_id in (select i_item_id from item where i_item_sk in (2, 3, 5, 7, 11, 13, 17, 19, 23, 29) ) ) and ws_sold_date_sk = d_date_sk and d_qoy = 2 and d_year = 2001 group by ca_zip, ca_city order by ca_zip, ca_city limit 100; -- query 46 select c_last_name ,c_first_name ,ca_city ,bought_city ,ss_ticket_number ,amt,profit from (select ss_ticket_number ,ss_customer_sk ,ca_city bought_city ,sum(ss_coupon_amt) amt ,sum(ss_net_profit) profit from store_sales,date_dim,store,household_demographics,customer_address where store_sales.ss_sold_date_sk = date_dim.d_date_sk and store_sales.ss_store_sk = store.s_store_sk and store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk and store_sales.ss_addr_sk = customer_address.ca_address_sk and (household_demographics.hd_dep_count = 4 or household_demographics.hd_vehicle_count= 3) and date_dim.d_dow in (6,0) and date_dim.d_year in (1999,1999+1,1999+2) and store.s_city in ('Fairview','Midway','Fairview','Fairview','Fairview') group by ss_ticket_number,ss_customer_sk,ss_addr_sk,ca_city) dn,customer,customer_address current_addr where ss_customer_sk = c_customer_sk and customer.c_current_addr_sk = current_addr.ca_address_sk and current_addr.ca_city <> bought_city order by c_last_name ,c_first_name ,ca_city ,bought_city ,ss_ticket_number limit 100; -- query 47 with v1 as( select i_category, i_brand, s_store_name, s_company_name, d_year, d_moy, sum(ss_sales_price) sum_sales, avg(sum(ss_sales_price)) over (partition by i_category, i_brand, s_store_name, s_company_name, d_year) avg_monthly_sales, rank() over (partition by i_category, i_brand, s_store_name, s_company_name order by d_year, d_moy) rn from item, store_sales, date_dim, store where ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and ss_store_sk = s_store_sk and ( d_year = 1999 or ( d_year = 1999-1 and d_moy =12) or ( d_year = 1999+1 and d_moy =1) ) group by i_category, i_brand, s_store_name, s_company_name, d_year, d_moy), v2 as( select v1.i_category, v1.i_brand, v1.s_store_name, v1.s_company_name ,v1.d_year, v1.d_moy ,v1.avg_monthly_sales ,v1.sum_sales, v1_lag.sum_sales psum, v1_lead.sum_sales nsum from v1, v1 v1_lag, v1 v1_lead where v1.i_category = v1_lag.i_category and v1.i_category = v1_lead.i_category and v1.i_brand = v1_lag.i_brand and v1.i_brand = v1_lead.i_brand and v1.s_store_name = v1_lag.s_store_name and v1.s_store_name = v1_lead.s_store_name and v1.s_company_name = v1_lag.s_company_name and v1.s_company_name = v1_lead.s_company_name and v1.rn = v1_lag.rn + 1 and v1.rn = v1_lead.rn - 1) select * from v2 where d_year = 1999 and avg_monthly_sales > 0 and case when avg_monthly_sales > 0 then abs(sum_sales - avg_monthly_sales) / avg_monthly_sales else null end > 0.1 order by sum_sales - avg_monthly_sales, s_store_name limit 100; -- query 48 select sum (ss_quantity) from store_sales, store, customer_demographics, customer_address, date_dim where s_store_sk = ss_store_sk and ss_sold_date_sk = d_date_sk and d_year = 2000 and ( ( cd_demo_sk = ss_cdemo_sk and cd_marital_status = 'M' and cd_education_status = '4 yr Degree' and ss_sales_price between 100.00 and 150.00 ) or ( cd_demo_sk = ss_cdemo_sk and cd_marital_status = 'D' and cd_education_status = '2 yr Degree' and ss_sales_price between 50.00 and 100.00 ) or ( cd_demo_sk = ss_cdemo_sk and cd_marital_status = 'S' and cd_education_status = 'College' and ss_sales_price between 150.00 and 200.00 ) ) and ( ( ss_addr_sk = ca_address_sk and ca_country = 'United States' and ca_state in ('CO', 'OH', 'TX') and ss_net_profit between 0 and 2000 ) or (ss_addr_sk = ca_address_sk and ca_country = 'United States' and ca_state in ('OR', 'MN', 'KY') and ss_net_profit between 150 and 3000 ) or (ss_addr_sk = ca_address_sk and ca_country = 'United States' and ca_state in ('VA', 'CA', 'MS') and ss_net_profit between 50 and 25000 ) ) ; -- query 49 select channel, item, return_ratio, return_rank, currency_rank from (select 'web' as channel ,web.item ,web.return_ratio ,web.return_rank ,web.currency_rank from ( select item ,return_ratio ,currency_ratio ,rank() over (order by return_ratio) as return_rank ,rank() over (order by currency_ratio) as currency_rank from ( select ws.ws_item_sk as item ,(cast(sum(coalesce(wr.wr_return_quantity,0)) as decimal(15,4))/ cast(sum(coalesce(ws.ws_quantity,0)) as decimal(15,4) )) as return_ratio ,(cast(sum(coalesce(wr.wr_return_amt,0)) as decimal(15,4))/ cast(sum(coalesce(ws.ws_net_paid,0)) as decimal(15,4) )) as currency_ratio from web_sales ws left outer join web_returns wr on (ws.ws_order_number = wr.wr_order_number and ws.ws_item_sk = wr.wr_item_sk) ,date_dim where wr.wr_return_amt > 10000 and ws.ws_net_profit > 1 and ws.ws_net_paid > 0 and ws.ws_quantity > 0 and ws_sold_date_sk = d_date_sk and d_year = 2001 and d_moy = 12 group by ws.ws_item_sk ) in_web ) web where ( web.return_rank <= 10 or web.currency_rank <= 10 ) union select 'catalog' as channel ,catalog.item ,catalog.return_ratio ,catalog.return_rank ,catalog.currency_rank from ( select item ,return_ratio ,currency_ratio ,rank() over (order by return_ratio) as return_rank ,rank() over (order by currency_ratio) as currency_rank from ( select cs.cs_item_sk as item ,(cast(sum(coalesce(cr.cr_return_quantity,0)) as decimal(15,4))/ cast(sum(coalesce(cs.cs_quantity,0)) as decimal(15,4) )) as return_ratio ,(cast(sum(coalesce(cr.cr_return_amount,0)) as decimal(15,4))/ cast(sum(coalesce(cs.cs_net_paid,0)) as decimal(15,4) )) as currency_ratio from catalog_sales cs left outer join catalog_returns cr on (cs.cs_order_number = cr.cr_order_number and cs.cs_item_sk = cr.cr_item_sk) ,date_dim where cr.cr_return_amount > 10000 and cs.cs_net_profit > 1 and cs.cs_net_paid > 0 and cs.cs_quantity > 0 and cs_sold_date_sk = d_date_sk and d_year = 2001 and d_moy = 12 group by cs.cs_item_sk ) in_cat ) catalog where ( catalog.return_rank <= 10 or catalog.currency_rank <=10 ) union select 'store' as channel ,store.item ,store.return_ratio ,store.return_rank ,store.currency_rank from ( select item ,return_ratio ,currency_ratio ,rank() over (order by return_ratio) as return_rank ,rank() over (order by currency_ratio) as currency_rank from ( select sts.ss_item_sk as item ,(cast(sum(coalesce(sr.sr_return_quantity,0)) as decimal(15,4))/cast(sum(coalesce(sts.ss_quantity,0)) as decimal(15,4) )) as return_ratio ,(cast(sum(coalesce(sr.sr_return_amt,0)) as decimal(15,4))/cast(sum(coalesce(sts.ss_net_paid,0)) as decimal(15,4) )) as currency_ratio from store_sales sts left outer join store_returns sr on (sts.ss_ticket_number = sr.sr_ticket_number and sts.ss_item_sk = sr.sr_item_sk) ,date_dim where sr.sr_return_amt > 10000 and sts.ss_net_profit > 1 and sts.ss_net_paid > 0 and sts.ss_quantity > 0 and ss_sold_date_sk = d_date_sk and d_year = 2001 and d_moy = 12 group by sts.ss_item_sk ) in_store ) store where ( store.return_rank <= 10 or store.currency_rank <= 10 ) ) t1 order by 1,4,5,2 limit 100; -- query 50 select s_store_name ,s_company_id ,s_street_number ,s_street_name ,s_street_type ,s_suite_number ,s_city ,s_county ,s_state ,s_zip ,sum(case when (sr_returned_date_sk - ss_sold_date_sk <= 30 ) then 1 else 0 end) as "30 days" ,sum(case when (sr_returned_date_sk - ss_sold_date_sk > 30) and (sr_returned_date_sk - ss_sold_date_sk <= 60) then 1 else 0 end ) as "31-60 days" ,sum(case when (sr_returned_date_sk - ss_sold_date_sk > 60) and (sr_returned_date_sk - ss_sold_date_sk <= 90) then 1 else 0 end) as "61-90 days" ,sum(case when (sr_returned_date_sk - ss_sold_date_sk > 90) and (sr_returned_date_sk - ss_sold_date_sk <= 120) then 1 else 0 end) as "91-120 days" ,sum(case when (sr_returned_date_sk - ss_sold_date_sk > 120) then 1 else 0 end) as ">120 days" from store_sales ,store_returns ,store ,date_dim d1 ,date_dim d2 where d2.d_year = 2001 and d2.d_moy = 8 and ss_ticket_number = sr_ticket_number and ss_item_sk = sr_item_sk and ss_sold_date_sk = d1.d_date_sk and sr_returned_date_sk = d2.d_date_sk and ss_customer_sk = sr_customer_sk and ss_store_sk = s_store_sk group by s_store_name ,s_company_id ,s_street_number ,s_street_name ,s_street_type ,s_suite_number ,s_city ,s_county ,s_state ,s_zip order by s_store_name ,s_company_id ,s_street_number ,s_street_name ,s_street_type ,s_suite_number ,s_city ,s_county ,s_state ,s_zip limit 100; -- query 51 WITH web_v1 as ( select ws_item_sk item_sk, d_date, sum(sum(ws_sales_price)) over (partition by ws_item_sk order by d_date rows between unbounded preceding and current row) cume_sales from web_sales ,date_dim where ws_sold_date_sk=d_date_sk and d_month_seq between 1200 and 1200+11 and ws_item_sk is not NULL group by ws_item_sk, d_date), store_v1 as ( select ss_item_sk item_sk, d_date, sum(sum(ss_sales_price)) over (partition by ss_item_sk order by d_date rows between unbounded preceding and current row) cume_sales from store_sales ,date_dim where ss_sold_date_sk=d_date_sk and d_month_seq between 1200 and 1200+11 and ss_item_sk is not NULL group by ss_item_sk, d_date) select * from (select item_sk ,d_date ,web_sales ,store_sales ,max(web_sales) over (partition by item_sk order by d_date rows between unbounded preceding and current row) web_cumulative ,max(store_sales) over (partition by item_sk order by d_date rows between unbounded preceding and current row) store_cumulative from (select case when web.item_sk is not null then web.item_sk else store.item_sk end item_sk ,case when web.d_date is not null then web.d_date else store.d_date end d_date ,web.cume_sales web_sales ,store.cume_sales store_sales from web_v1 web full outer join store_v1 store on (web.item_sk = store.item_sk and web.d_date = store.d_date) )x )y where web_cumulative > store_cumulative order by item_sk ,d_date limit 100; -- query 52 select dt.d_year ,item.i_brand_id brand_id ,item.i_brand brand ,sum(ss_ext_sales_price) ext_price from date_dim dt ,store_sales ,item where dt.d_date_sk = store_sales.ss_sold_date_sk and store_sales.ss_item_sk = item.i_item_sk and item.i_manager_id = 1 and dt.d_moy=11 and dt.d_year=2000 group by dt.d_year ,item.i_brand ,item.i_brand_id order by dt.d_year ,ext_price desc ,brand_id limit 100 ; -- query 53 select * from (select i_manufact_id, sum(ss_sales_price) sum_sales, avg(sum(ss_sales_price)) over (partition by i_manufact_id) avg_quarterly_sales from item, store_sales, date_dim, store where ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and ss_store_sk = s_store_sk and d_month_seq in (1200,1200+1,1200+2,1200+3,1200+4,1200+5,1200+6,1200+7,1200+8,1200+9,1200+10,1200+11) and ((i_category in ('Books','Children','Electronics') and i_class in ('personal','portable','reference','self-help') and i_brand in ('scholaramalgamalg #14','scholaramalgamalg #7', 'exportiunivamalg #9','scholaramalgamalg #9')) or(i_category in ('Women','Music','Men') and i_class in ('accessories','classical','fragrances','pants') and i_brand in ('amalgimporto #1','edu packscholar #1','exportiimporto #1', 'importoamalg #1'))) group by i_manufact_id, d_qoy ) tmp1 where case when avg_quarterly_sales > 0 then abs (sum_sales - avg_quarterly_sales)/ avg_quarterly_sales else null end > 0.1 order by avg_quarterly_sales, sum_sales, i_manufact_id limit 100; -- query 54 with my_customers as ( select distinct c_customer_sk , c_current_addr_sk from ( select cs_sold_date_sk sold_date_sk, cs_bill_customer_sk customer_sk, cs_item_sk item_sk from catalog_sales union all select ws_sold_date_sk sold_date_sk, ws_bill_customer_sk customer_sk, ws_item_sk item_sk from web_sales ) cs_or_ws_sales, item, date_dim, customer where sold_date_sk = d_date_sk and item_sk = i_item_sk and i_category = 'Women' and i_class = 'maternity' and c_customer_sk = cs_or_ws_sales.customer_sk and d_moy = 12 and d_year = 1998 ) , my_revenue as ( select c_customer_sk, sum(ss_ext_sales_price) as revenue from my_customers, store_sales, customer_address, store, date_dim where c_current_addr_sk = ca_address_sk and ca_county = s_county and ca_state = s_state and ss_sold_date_sk = d_date_sk and c_customer_sk = ss_customer_sk and d_month_seq between (select distinct d_month_seq+1 from date_dim where d_year = 1998 and d_moy = 12) and (select distinct d_month_seq+3 from date_dim where d_year = 1998 and d_moy = 12) group by c_customer_sk ) , segments as (select cast((revenue/50) as int) as segment from my_revenue ) select segment, count(*) as num_customers, segment*50 as segment_base from segments group by segment order by segment, num_customers limit 100; -- query 55 select i_brand_id brand_id, i_brand brand, sum(ss_ext_sales_price) ext_price from date_dim, store_sales, item where d_date_sk = ss_sold_date_sk and ss_item_sk = i_item_sk and i_manager_id=28 and d_moy=11 and d_year=1999 group by i_brand, i_brand_id order by ext_price desc, i_brand_id limit 100 ; -- query 56 with ss as ( select i_item_id,sum(ss_ext_sales_price) total_sales from store_sales, date_dim, customer_address, item where i_item_id in (select i_item_id from item where i_color in ('slate', 'blanched', 'burnished')) and ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and d_year = 2001 and d_moy = 2 and ss_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_item_id), cs as ( select i_item_id,sum(cs_ext_sales_price) total_sales from catalog_sales, date_dim, customer_address, item where i_item_id in (select i_item_id from item where i_color in ('slate', 'blanched', 'burnished')) and cs_item_sk = i_item_sk and cs_sold_date_sk = d_date_sk and d_year = 2001 and d_moy = 2 and cs_bill_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_item_id), ws as ( select i_item_id,sum(ws_ext_sales_price) total_sales from web_sales, date_dim, customer_address, item where i_item_id in (select i_item_id from item where i_color in ('slate', 'blanched', 'burnished')) and ws_item_sk = i_item_sk and ws_sold_date_sk = d_date_sk and d_year = 2001 and d_moy = 2 and ws_bill_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_item_id) select i_item_id ,sum(total_sales) total_sales from (select * from ss union all select * from cs union all select * from ws) tmp1 group by i_item_id order by total_sales, i_item_id limit 100; -- query 57 with v1 as( select i_category, i_brand, cc_name, d_year, d_moy, sum(cs_sales_price) sum_sales, avg(sum(cs_sales_price)) over (partition by i_category, i_brand, cc_name, d_year) avg_monthly_sales, rank() over (partition by i_category, i_brand, cc_name order by d_year, d_moy) rn from item, catalog_sales, date_dim, call_center where cs_item_sk = i_item_sk and cs_sold_date_sk = d_date_sk and cc_call_center_sk= cs_call_center_sk and ( d_year = 1999 or ( d_year = 1999-1 and d_moy =12) or ( d_year = 1999+1 and d_moy =1) ) group by i_category, i_brand, cc_name , d_year, d_moy), v2 as( select v1.i_category, v1.i_brand, v1.cc_name ,v1.d_year, v1.d_moy ,v1.avg_monthly_sales ,v1.sum_sales, v1_lag.sum_sales psum, v1_lead.sum_sales nsum from v1, v1 v1_lag, v1 v1_lead where v1.i_category = v1_lag.i_category and v1.i_category = v1_lead.i_category and v1.i_brand = v1_lag.i_brand and v1.i_brand = v1_lead.i_brand and v1. cc_name = v1_lag. cc_name and v1. cc_name = v1_lead. cc_name and v1.rn = v1_lag.rn + 1 and v1.rn = v1_lead.rn - 1) select * from v2 where d_year = 1999 and avg_monthly_sales > 0 and case when avg_monthly_sales > 0 then abs(sum_sales - avg_monthly_sales) / avg_monthly_sales else null end > 0.1 order by sum_sales - avg_monthly_sales, cc_name limit 100; -- query 58 with ss_items as (select i_item_id item_id ,sum(ss_ext_sales_price) ss_item_rev from store_sales ,item ,date_dim where ss_item_sk = i_item_sk and d_date in (select d_date from date_dim where d_week_seq = (select d_week_seq from date_dim where d_date = '2000-01-03')) and ss_sold_date_sk = d_date_sk group by i_item_id), cs_items as (select i_item_id item_id ,sum(cs_ext_sales_price) cs_item_rev from catalog_sales ,item ,date_dim where cs_item_sk = i_item_sk and d_date in (select d_date from date_dim where d_week_seq = (select d_week_seq from date_dim where d_date = '2000-01-03')) and cs_sold_date_sk = d_date_sk group by i_item_id), ws_items as (select i_item_id item_id ,sum(ws_ext_sales_price) ws_item_rev from web_sales ,item ,date_dim where ws_item_sk = i_item_sk and d_date in (select d_date from date_dim where d_week_seq =(select d_week_seq from date_dim where d_date = '2000-01-03')) and ws_sold_date_sk = d_date_sk group by i_item_id) select ss_items.item_id ,ss_item_rev ,ss_item_rev/((ss_item_rev+cs_item_rev+ws_item_rev)/3) * 100 ss_dev ,cs_item_rev ,cs_item_rev/((ss_item_rev+cs_item_rev+ws_item_rev)/3) * 100 cs_dev ,ws_item_rev ,ws_item_rev/((ss_item_rev+cs_item_rev+ws_item_rev)/3) * 100 ws_dev ,(ss_item_rev+cs_item_rev+ws_item_rev)/3 average from ss_items,cs_items,ws_items where ss_items.item_id=cs_items.item_id and ss_items.item_id=ws_items.item_id and ss_item_rev between 0.9 * cs_item_rev and 1.1 * cs_item_rev and ss_item_rev between 0.9 * ws_item_rev and 1.1 * ws_item_rev and cs_item_rev between 0.9 * ss_item_rev and 1.1 * ss_item_rev and cs_item_rev between 0.9 * ws_item_rev and 1.1 * ws_item_rev and ws_item_rev between 0.9 * ss_item_rev and 1.1 * ss_item_rev and ws_item_rev between 0.9 * cs_item_rev and 1.1 * cs_item_rev order by item_id ,ss_item_rev limit 100; -- query 59 with wss as (select d_week_seq, ss_store_sk, sum(case when (d_day_name='Sunday') then ss_sales_price else null end) sun_sales, sum(case when (d_day_name='Monday') then ss_sales_price else null end) mon_sales, sum(case when (d_day_name='Tuesday') then ss_sales_price else null end) tue_sales, sum(case when (d_day_name='Wednesday') then ss_sales_price else null end) wed_sales, sum(case when (d_day_name='Thursday') then ss_sales_price else null end) thu_sales, sum(case when (d_day_name='Friday') then ss_sales_price else null end) fri_sales, sum(case when (d_day_name='Saturday') then ss_sales_price else null end) sat_sales from store_sales,date_dim where d_date_sk = ss_sold_date_sk group by d_week_seq,ss_store_sk ) select s_store_name1,s_store_id1,d_week_seq1 ,sun_sales1/sun_sales2,mon_sales1/mon_sales2 ,tue_sales1/tue_sales2,wed_sales1/wed_sales2,thu_sales1/thu_sales2 ,fri_sales1/fri_sales2,sat_sales1/sat_sales2 from (select s_store_name s_store_name1,wss.d_week_seq d_week_seq1 ,s_store_id s_store_id1,sun_sales sun_sales1 ,mon_sales mon_sales1,tue_sales tue_sales1 ,wed_sales wed_sales1,thu_sales thu_sales1 ,fri_sales fri_sales1,sat_sales sat_sales1 from wss,store,date_dim d where d.d_week_seq = wss.d_week_seq and ss_store_sk = s_store_sk and d_month_seq between 1212 and 1212 + 11) y, (select s_store_name s_store_name2,wss.d_week_seq d_week_seq2 ,s_store_id s_store_id2,sun_sales sun_sales2 ,mon_sales mon_sales2,tue_sales tue_sales2 ,wed_sales wed_sales2,thu_sales thu_sales2 ,fri_sales fri_sales2,sat_sales sat_sales2 from wss,store,date_dim d where d.d_week_seq = wss.d_week_seq and ss_store_sk = s_store_sk and d_month_seq between 1212+ 12 and 1212 + 23) x where s_store_id1=s_store_id2 and d_week_seq1=d_week_seq2-52 order by s_store_name1,s_store_id1,d_week_seq1 limit 100; -- query 60 with ss as ( select i_item_id,sum(ss_ext_sales_price) total_sales from store_sales, date_dim, customer_address, item where i_item_id in (select i_item_id from item where i_category in ('Music')) and ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and d_year = 1998 and d_moy = 9 and ss_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_item_id), cs as ( select i_item_id,sum(cs_ext_sales_price) total_sales from catalog_sales, date_dim, customer_address, item where i_item_id in (select i_item_id from item where i_category in ('Music')) and cs_item_sk = i_item_sk and cs_sold_date_sk = d_date_sk and d_year = 1998 and d_moy = 9 and cs_bill_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_item_id), ws as ( select i_item_id,sum(ws_ext_sales_price) total_sales from web_sales, date_dim, customer_address, item where i_item_id in (select i_item_id from item where i_category in ('Music')) and ws_item_sk = i_item_sk and ws_sold_date_sk = d_date_sk and d_year = 1998 and d_moy = 9 and ws_bill_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_item_id) select i_item_id ,sum(total_sales) total_sales from (select * from ss union all select * from cs union all select * from ws) tmp1 group by i_item_id order by i_item_id ,total_sales limit 100; -- query 61 select promotions,total,cast(promotions as decimal(15,4))/cast(total as decimal(15,4))*100 from (select sum(ss_ext_sales_price) promotions from store_sales ,store ,promotion ,date_dim ,customer ,customer_address ,item where ss_sold_date_sk = d_date_sk and ss_store_sk = s_store_sk and ss_promo_sk = p_promo_sk and ss_customer_sk= c_customer_sk and ca_address_sk = c_current_addr_sk and ss_item_sk = i_item_sk and ca_gmt_offset = -5 and i_category = 'Jewelry' and (p_channel_dmail = 'Y' or p_channel_email = 'Y' or p_channel_tv = 'Y') and s_gmt_offset = -5 and d_year = 1998 and d_moy = 11) promotional_sales, (select sum(ss_ext_sales_price) total from store_sales ,store ,date_dim ,customer ,customer_address ,item where ss_sold_date_sk = d_date_sk and ss_store_sk = s_store_sk and ss_customer_sk= c_customer_sk and ca_address_sk = c_current_addr_sk and ss_item_sk = i_item_sk and ca_gmt_offset = -5 and i_category = 'Jewelry' and s_gmt_offset = -5 and d_year = 1998 and d_moy = 11) all_sales order by promotions, total limit 100; -- query 62 select substr(w_warehouse_name,1,20) ,sm_type ,web_name ,sum(case when (ws_ship_date_sk - ws_sold_date_sk <= 30 ) then 1 else 0 end) as "30 days" ,sum(case when (ws_ship_date_sk - ws_sold_date_sk > 30) and (ws_ship_date_sk - ws_sold_date_sk <= 60) then 1 else 0 end ) as "31-60 days" ,sum(case when (ws_ship_date_sk - ws_sold_date_sk > 60) and (ws_ship_date_sk - ws_sold_date_sk <= 90) then 1 else 0 end) as "61-90 days" ,sum(case when (ws_ship_date_sk - ws_sold_date_sk > 90) and (ws_ship_date_sk - ws_sold_date_sk <= 120) then 1 else 0 end) as "91-120 days" ,sum(case when (ws_ship_date_sk - ws_sold_date_sk > 120) then 1 else 0 end) as ">120 days" from web_sales ,warehouse ,ship_mode ,web_site ,date_dim where d_month_seq between 1200 and 1200 + 11 and ws_ship_date_sk = d_date_sk and ws_warehouse_sk = w_warehouse_sk and ws_ship_mode_sk = sm_ship_mode_sk and ws_web_site_sk = web_site_sk group by substr(w_warehouse_name,1,20) ,sm_type ,web_name order by substr(w_warehouse_name,1,20) ,sm_type ,web_name limit 100; -- query 63 select * from (select i_manager_id ,sum(ss_sales_price) sum_sales ,avg(sum(ss_sales_price)) over (partition by i_manager_id) avg_monthly_sales from item ,store_sales ,date_dim ,store where ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and ss_store_sk = s_store_sk and d_month_seq in (1200,1200+1,1200+2,1200+3,1200+4,1200+5,1200+6,1200+7,1200+8,1200+9,1200+10,1200+11) and (( i_category in ('Books','Children','Electronics') and i_class in ('personal','portable','reference','self-help') and i_brand in ('scholaramalgamalg #14','scholaramalgamalg #7', 'exportiunivamalg #9','scholaramalgamalg #9')) or( i_category in ('Women','Music','Men') and i_class in ('accessories','classical','fragrances','pants') and i_brand in ('amalgimporto #1','edu packscholar #1','exportiimporto #1', 'importoamalg #1'))) group by i_manager_id, d_moy) tmp1 where case when avg_monthly_sales > 0 then abs (sum_sales - avg_monthly_sales) / avg_monthly_sales else null end > 0.1 order by i_manager_id ,avg_monthly_sales ,sum_sales limit 100; -- query 64 with cs_ui as (select cs_item_sk ,sum(cs_ext_list_price) as sale,sum(cr_refunded_cash+cr_reversed_charge+cr_store_credit) as refund from catalog_sales ,catalog_returns where cs_item_sk = cr_item_sk and cs_order_number = cr_order_number group by cs_item_sk having sum(cs_ext_list_price)>2*sum(cr_refunded_cash+cr_reversed_charge+cr_store_credit)), cross_sales as (select i_product_name product_name ,i_item_sk item_sk ,s_store_name store_name ,s_zip store_zip ,ad1.ca_street_number b_street_number ,ad1.ca_street_name b_street_name ,ad1.ca_city b_city ,ad1.ca_zip b_zip ,ad2.ca_street_number c_street_number ,ad2.ca_street_name c_street_name ,ad2.ca_city c_city ,ad2.ca_zip c_zip ,d1.d_year as syear ,d2.d_year as fsyear ,d3.d_year s2year ,count(*) cnt ,sum(ss_wholesale_cost) s1 ,sum(ss_list_price) s2 ,sum(ss_coupon_amt) s3 FROM store_sales ,store_returns ,cs_ui ,date_dim d1 ,date_dim d2 ,date_dim d3 ,store ,customer ,customer_demographics cd1 ,customer_demographics cd2 ,promotion ,household_demographics hd1 ,household_demographics hd2 ,customer_address ad1 ,customer_address ad2 ,income_band ib1 ,income_band ib2 ,item WHERE ss_store_sk = s_store_sk AND ss_sold_date_sk = d1.d_date_sk AND ss_customer_sk = c_customer_sk AND ss_cdemo_sk= cd1.cd_demo_sk AND ss_hdemo_sk = hd1.hd_demo_sk AND ss_addr_sk = ad1.ca_address_sk and ss_item_sk = i_item_sk and ss_item_sk = sr_item_sk and ss_ticket_number = sr_ticket_number and ss_item_sk = cs_ui.cs_item_sk and c_current_cdemo_sk = cd2.cd_demo_sk AND c_current_hdemo_sk = hd2.hd_demo_sk AND c_current_addr_sk = ad2.ca_address_sk and c_first_sales_date_sk = d2.d_date_sk and c_first_shipto_date_sk = d3.d_date_sk and ss_promo_sk = p_promo_sk and hd1.hd_income_band_sk = ib1.ib_income_band_sk and hd2.hd_income_band_sk = ib2.ib_income_band_sk and cd1.cd_marital_status <> cd2.cd_marital_status and i_color in ('purple','burlywood','indian','spring','floral','medium') and i_current_price between 64 and 64 + 10 and i_current_price between 64 + 1 and 64 + 15 group by i_product_name ,i_item_sk ,s_store_name ,s_zip ,ad1.ca_street_number ,ad1.ca_street_name ,ad1.ca_city ,ad1.ca_zip ,ad2.ca_street_number ,ad2.ca_street_name ,ad2.ca_city ,ad2.ca_zip ,d1.d_year ,d2.d_year ,d3.d_year ) select cs1.product_name ,cs1.store_name ,cs1.store_zip ,cs1.b_street_number ,cs1.b_street_name ,cs1.b_city ,cs1.b_zip ,cs1.c_street_number ,cs1.c_street_name ,cs1.c_city ,cs1.c_zip ,cs1.syear ,cs1.cnt ,cs1.s1 as s11 ,cs1.s2 as s21 ,cs1.s3 as s31 ,cs2.s1 as s12 ,cs2.s2 as s22 ,cs2.s3 as s32 ,cs2.syear ,cs2.cnt from cross_sales cs1,cross_sales cs2 where cs1.item_sk=cs2.item_sk and cs1.syear = 1999 and cs2.syear = 1999 + 1 and cs2.cnt <= cs1.cnt and cs1.store_name = cs2.store_name and cs1.store_zip = cs2.store_zip order by cs1.product_name ,cs1.store_name ,cs2.cnt ,cs1.s1 ,cs2.s1; -- query 65 select s_store_name, i_item_desc, sc.revenue, i_current_price, i_wholesale_cost, i_brand from store, item, (select ss_store_sk, avg(revenue) as ave from (select ss_store_sk, ss_item_sk, sum(ss_sales_price) as revenue from store_sales, date_dim where ss_sold_date_sk = d_date_sk and d_month_seq between 1176 and 1176+11 group by ss_store_sk, ss_item_sk) sa group by ss_store_sk) sb, (select ss_store_sk, ss_item_sk, sum(ss_sales_price) as revenue from store_sales, date_dim where ss_sold_date_sk = d_date_sk and d_month_seq between 1176 and 1176+11 group by ss_store_sk, ss_item_sk) sc where sb.ss_store_sk = sc.ss_store_sk and sc.revenue <= 0.1 * sb.ave and s_store_sk = sc.ss_store_sk and i_item_sk = sc.ss_item_sk order by s_store_name, i_item_desc limit 100; -- query 66 select w_warehouse_name ,w_warehouse_sq_ft ,w_city ,w_county ,w_state ,w_country ,ship_carriers ,year ,sum(jan_sales) as jan_sales ,sum(feb_sales) as feb_sales ,sum(mar_sales) as mar_sales ,sum(apr_sales) as apr_sales ,sum(may_sales) as may_sales ,sum(jun_sales) as jun_sales ,sum(jul_sales) as jul_sales ,sum(aug_sales) as aug_sales ,sum(sep_sales) as sep_sales ,sum(oct_sales) as oct_sales ,sum(nov_sales) as nov_sales ,sum(dec_sales) as dec_sales ,sum(jan_sales/w_warehouse_sq_ft) as jan_sales_per_sq_foot ,sum(feb_sales/w_warehouse_sq_ft) as feb_sales_per_sq_foot ,sum(mar_sales/w_warehouse_sq_ft) as mar_sales_per_sq_foot ,sum(apr_sales/w_warehouse_sq_ft) as apr_sales_per_sq_foot ,sum(may_sales/w_warehouse_sq_ft) as may_sales_per_sq_foot ,sum(jun_sales/w_warehouse_sq_ft) as jun_sales_per_sq_foot ,sum(jul_sales/w_warehouse_sq_ft) as jul_sales_per_sq_foot ,sum(aug_sales/w_warehouse_sq_ft) as aug_sales_per_sq_foot ,sum(sep_sales/w_warehouse_sq_ft) as sep_sales_per_sq_foot ,sum(oct_sales/w_warehouse_sq_ft) as oct_sales_per_sq_foot ,sum(nov_sales/w_warehouse_sq_ft) as nov_sales_per_sq_foot ,sum(dec_sales/w_warehouse_sq_ft) as dec_sales_per_sq_foot ,sum(jan_net) as jan_net ,sum(feb_net) as feb_net ,sum(mar_net) as mar_net ,sum(apr_net) as apr_net ,sum(may_net) as may_net ,sum(jun_net) as jun_net ,sum(jul_net) as jul_net ,sum(aug_net) as aug_net ,sum(sep_net) as sep_net ,sum(oct_net) as oct_net ,sum(nov_net) as nov_net ,sum(dec_net) as dec_net from ( select w_warehouse_name ,w_warehouse_sq_ft ,w_city ,w_county ,w_state ,w_country ,'DHL' || ',' || 'BARIAN' as ship_carriers ,d_year as year ,sum(case when d_moy = 1 then ws_ext_sales_price* ws_quantity else 0 end) as jan_sales ,sum(case when d_moy = 2 then ws_ext_sales_price* ws_quantity else 0 end) as feb_sales ,sum(case when d_moy = 3 then ws_ext_sales_price* ws_quantity else 0 end) as mar_sales ,sum(case when d_moy = 4 then ws_ext_sales_price* ws_quantity else 0 end) as apr_sales ,sum(case when d_moy = 5 then ws_ext_sales_price* ws_quantity else 0 end) as may_sales ,sum(case when d_moy = 6 then ws_ext_sales_price* ws_quantity else 0 end) as jun_sales ,sum(case when d_moy = 7 then ws_ext_sales_price* ws_quantity else 0 end) as jul_sales ,sum(case when d_moy = 8 then ws_ext_sales_price* ws_quantity else 0 end) as aug_sales ,sum(case when d_moy = 9 then ws_ext_sales_price* ws_quantity else 0 end) as sep_sales ,sum(case when d_moy = 10 then ws_ext_sales_price* ws_quantity else 0 end) as oct_sales ,sum(case when d_moy = 11 then ws_ext_sales_price* ws_quantity else 0 end) as nov_sales ,sum(case when d_moy = 12 then ws_ext_sales_price* ws_quantity else 0 end) as dec_sales ,sum(case when d_moy = 1 then ws_net_paid * ws_quantity else 0 end) as jan_net ,sum(case when d_moy = 2 then ws_net_paid * ws_quantity else 0 end) as feb_net ,sum(case when d_moy = 3 then ws_net_paid * ws_quantity else 0 end) as mar_net ,sum(case when d_moy = 4 then ws_net_paid * ws_quantity else 0 end) as apr_net ,sum(case when d_moy = 5 then ws_net_paid * ws_quantity else 0 end) as may_net ,sum(case when d_moy = 6 then ws_net_paid * ws_quantity else 0 end) as jun_net ,sum(case when d_moy = 7 then ws_net_paid * ws_quantity else 0 end) as jul_net ,sum(case when d_moy = 8 then ws_net_paid * ws_quantity else 0 end) as aug_net ,sum(case when d_moy = 9 then ws_net_paid * ws_quantity else 0 end) as sep_net ,sum(case when d_moy = 10 then ws_net_paid * ws_quantity else 0 end) as oct_net ,sum(case when d_moy = 11 then ws_net_paid * ws_quantity else 0 end) as nov_net ,sum(case when d_moy = 12 then ws_net_paid * ws_quantity else 0 end) as dec_net from web_sales ,warehouse ,date_dim ,time_dim ,ship_mode where ws_warehouse_sk = w_warehouse_sk and ws_sold_date_sk = d_date_sk and ws_sold_time_sk = t_time_sk and ws_ship_mode_sk = sm_ship_mode_sk and d_year = 2001 and t_time between 30838 and 30838+28800 and sm_carrier in ('DHL','BARIAN') group by w_warehouse_name ,w_warehouse_sq_ft ,w_city ,w_county ,w_state ,w_country ,d_year union all select w_warehouse_name ,w_warehouse_sq_ft ,w_city ,w_county ,w_state ,w_country ,'DHL' || ',' || 'BARIAN' as ship_carriers ,d_year as year ,sum(case when d_moy = 1 then cs_sales_price* cs_quantity else 0 end) as jan_sales ,sum(case when d_moy = 2 then cs_sales_price* cs_quantity else 0 end) as feb_sales ,sum(case when d_moy = 3 then cs_sales_price* cs_quantity else 0 end) as mar_sales ,sum(case when d_moy = 4 then cs_sales_price* cs_quantity else 0 end) as apr_sales ,sum(case when d_moy = 5 then cs_sales_price* cs_quantity else 0 end) as may_sales ,sum(case when d_moy = 6 then cs_sales_price* cs_quantity else 0 end) as jun_sales ,sum(case when d_moy = 7 then cs_sales_price* cs_quantity else 0 end) as jul_sales ,sum(case when d_moy = 8 then cs_sales_price* cs_quantity else 0 end) as aug_sales ,sum(case when d_moy = 9 then cs_sales_price* cs_quantity else 0 end) as sep_sales ,sum(case when d_moy = 10 then cs_sales_price* cs_quantity else 0 end) as oct_sales ,sum(case when d_moy = 11 then cs_sales_price* cs_quantity else 0 end) as nov_sales ,sum(case when d_moy = 12 then cs_sales_price* cs_quantity else 0 end) as dec_sales ,sum(case when d_moy = 1 then cs_net_paid_inc_tax * cs_quantity else 0 end) as jan_net ,sum(case when d_moy = 2 then cs_net_paid_inc_tax * cs_quantity else 0 end) as feb_net ,sum(case when d_moy = 3 then cs_net_paid_inc_tax * cs_quantity else 0 end) as mar_net ,sum(case when d_moy = 4 then cs_net_paid_inc_tax * cs_quantity else 0 end) as apr_net ,sum(case when d_moy = 5 then cs_net_paid_inc_tax * cs_quantity else 0 end) as may_net ,sum(case when d_moy = 6 then cs_net_paid_inc_tax * cs_quantity else 0 end) as jun_net ,sum(case when d_moy = 7 then cs_net_paid_inc_tax * cs_quantity else 0 end) as jul_net ,sum(case when d_moy = 8 then cs_net_paid_inc_tax * cs_quantity else 0 end) as aug_net ,sum(case when d_moy = 9 then cs_net_paid_inc_tax * cs_quantity else 0 end) as sep_net ,sum(case when d_moy = 10 then cs_net_paid_inc_tax * cs_quantity else 0 end) as oct_net ,sum(case when d_moy = 11 then cs_net_paid_inc_tax * cs_quantity else 0 end) as nov_net ,sum(case when d_moy = 12 then cs_net_paid_inc_tax * cs_quantity else 0 end) as dec_net from catalog_sales ,warehouse ,date_dim ,time_dim ,ship_mode where cs_warehouse_sk = w_warehouse_sk and cs_sold_date_sk = d_date_sk and cs_sold_time_sk = t_time_sk and cs_ship_mode_sk = sm_ship_mode_sk and d_year = 2001 and t_time between 30838 AND 30838+28800 and sm_carrier in ('DHL','BARIAN') group by w_warehouse_name ,w_warehouse_sq_ft ,w_city ,w_county ,w_state ,w_country ,d_year ) x group by w_warehouse_name ,w_warehouse_sq_ft ,w_city ,w_county ,w_state ,w_country ,ship_carriers ,year order by w_warehouse_name limit 100; -- query 67 select * from (select i_category ,i_class ,i_brand ,i_product_name ,d_year ,d_qoy ,d_moy ,s_store_id ,sumsales ,rank() over (partition by i_category order by sumsales desc) rk from (select i_category ,i_class ,i_brand ,i_product_name ,d_year ,d_qoy ,d_moy ,s_store_id ,sum(coalesce(ss_sales_price*ss_quantity,0)) sumsales from store_sales ,date_dim ,store ,item where ss_sold_date_sk=d_date_sk and ss_item_sk=i_item_sk and ss_store_sk = s_store_sk and d_month_seq between 1200 and 1200+11 group by rollup(i_category, i_class, i_brand, i_product_name, d_year, d_qoy, d_moy,s_store_id))dw1) dw2 where rk <= 100 order by i_category ,i_class ,i_brand ,i_product_name ,d_year ,d_qoy ,d_moy ,s_store_id ,sumsales ,rk limit 100; -- query 68 select c_last_name ,c_first_name ,ca_city ,bought_city ,ss_ticket_number ,extended_price ,extended_tax ,list_price from (select ss_ticket_number ,ss_customer_sk ,ca_city bought_city ,sum(ss_ext_sales_price) extended_price ,sum(ss_ext_list_price) list_price ,sum(ss_ext_tax) extended_tax from store_sales ,date_dim ,store ,household_demographics ,customer_address where store_sales.ss_sold_date_sk = date_dim.d_date_sk and store_sales.ss_store_sk = store.s_store_sk and store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk and store_sales.ss_addr_sk = customer_address.ca_address_sk and date_dim.d_dom between 1 and 2 and (household_demographics.hd_dep_count = 4 or household_demographics.hd_vehicle_count= 3) and date_dim.d_year in (1999,1999+1,1999+2) and store.s_city in ('Fairview','Midway') group by ss_ticket_number ,ss_customer_sk ,ss_addr_sk,ca_city) dn ,customer ,customer_address current_addr where ss_customer_sk = c_customer_sk and customer.c_current_addr_sk = current_addr.ca_address_sk and current_addr.ca_city <> bought_city order by c_last_name ,ss_ticket_number limit 100; -- query 69 select cd_gender, cd_marital_status, cd_education_status, count(*) cnt1, cd_purchase_estimate, count(*) cnt2, cd_credit_rating, count(*) cnt3 from customer c,customer_address ca,customer_demographics where c.c_current_addr_sk = ca.ca_address_sk and ca_state in ('KY', 'GA', 'NM') and cd_demo_sk = c.c_current_cdemo_sk and exists (select * from store_sales,date_dim where c.c_customer_sk = ss_customer_sk and ss_sold_date_sk = d_date_sk and d_year = 2001 and d_moy between 4 and 4+2) and (not exists (select * from web_sales,date_dim where c.c_customer_sk = ws_bill_customer_sk and ws_sold_date_sk = d_date_sk and d_year = 2001 and d_moy between 4 and 4+2) and not exists (select * from catalog_sales,date_dim where c.c_customer_sk = cs_ship_customer_sk and cs_sold_date_sk = d_date_sk and d_year = 2001 and d_moy between 4 and 4+2)) group by cd_gender, cd_marital_status, cd_education_status, cd_purchase_estimate, cd_credit_rating order by cd_gender, cd_marital_status, cd_education_status, cd_purchase_estimate, cd_credit_rating limit 100; -- query 70 select sum(ss_net_profit) as total_sum ,s_state ,s_county ,grouping(s_state)+grouping(s_county) as lochierarchy ,rank() over ( partition by grouping(s_state)+grouping(s_county), case when grouping(s_county) = 0 then s_state end order by sum(ss_net_profit) desc) as rank_within_parent from store_sales ,date_dim d1 ,store where d1.d_month_seq between 1200 and 1200+11 and d1.d_date_sk = ss_sold_date_sk and s_store_sk = ss_store_sk and s_state in ( select s_state from (select s_state as s_state, rank() over ( partition by s_state order by sum(ss_net_profit) desc) as ranking from store_sales, store, date_dim where d_month_seq between 1200 and 1200+11 and d_date_sk = ss_sold_date_sk and s_store_sk = ss_store_sk group by s_state ) tmp1 where ranking <= 5 ) group by rollup(s_state,s_county) order by lochierarchy desc ,case when lochierarchy = 0 then s_state end ,rank_within_parent limit 100; -- query 71 select i_brand_id brand_id, i_brand brand,t_hour,t_minute, sum(ext_price) ext_price from item, (select ws_ext_sales_price as ext_price, ws_sold_date_sk as sold_date_sk, ws_item_sk as sold_item_sk, ws_sold_time_sk as time_sk from web_sales,date_dim where d_date_sk = ws_sold_date_sk and d_moy=11 and d_year=1999 union all select cs_ext_sales_price as ext_price, cs_sold_date_sk as sold_date_sk, cs_item_sk as sold_item_sk, cs_sold_time_sk as time_sk from catalog_sales,date_dim where d_date_sk = cs_sold_date_sk and d_moy=11 and d_year=1999 union all select ss_ext_sales_price as ext_price, ss_sold_date_sk as sold_date_sk, ss_item_sk as sold_item_sk, ss_sold_time_sk as time_sk from store_sales,date_dim where d_date_sk = ss_sold_date_sk and d_moy=11 and d_year=1999 ) tmp,time_dim where sold_item_sk = i_item_sk and i_manager_id=1 and time_sk = t_time_sk and (t_meal_time = 'breakfast' or t_meal_time = 'dinner') group by i_brand, i_brand_id,t_hour,t_minute order by ext_price desc, i_brand_id ; -- query 72 select i_item_desc ,w_warehouse_name ,d1.d_week_seq ,sum(case when p_promo_sk is null then 1 else 0 end) no_promo ,sum(case when p_promo_sk is not null then 1 else 0 end) promo ,count(*) total_cnt from catalog_sales join inventory on (cs_item_sk = inv_item_sk) join warehouse on (w_warehouse_sk=inv_warehouse_sk) join item on (i_item_sk = cs_item_sk) join customer_demographics on (cs_bill_cdemo_sk = cd_demo_sk) join household_demographics on (cs_bill_hdemo_sk = hd_demo_sk) join date_dim d1 on (cs_sold_date_sk = d1.d_date_sk) join date_dim d2 on (inv_date_sk = d2.d_date_sk) join date_dim d3 on (cs_ship_date_sk = d3.d_date_sk) left outer join promotion on (cs_promo_sk=p_promo_sk) left outer join catalog_returns on (cr_item_sk = cs_item_sk and cr_order_number = cs_order_number) where d1.d_week_seq = d2.d_week_seq and inv_quantity_on_hand < cs_quantity and d3.d_date > d1.d_date + 5 and hd_buy_potential = '>10000' and d1.d_year = 1999 and cd_marital_status = 'D' group by i_item_desc,w_warehouse_name,d1.d_week_seq order by total_cnt desc, i_item_desc, w_warehouse_name, d_week_seq limit 100; -- query 73 select c_last_name ,c_first_name ,c_salutation ,c_preferred_cust_flag ,ss_ticket_number ,cnt from (select ss_ticket_number ,ss_customer_sk ,count(*) cnt from store_sales,date_dim,store,household_demographics where store_sales.ss_sold_date_sk = date_dim.d_date_sk and store_sales.ss_store_sk = store.s_store_sk and store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk and date_dim.d_dom between 1 and 2 and (household_demographics.hd_buy_potential = '>10000' or household_demographics.hd_buy_potential = 'Unknown') and household_demographics.hd_vehicle_count > 0 and case when household_demographics.hd_vehicle_count > 0 then household_demographics.hd_dep_count/ household_demographics.hd_vehicle_count else null end > 1 and date_dim.d_year in (1999,1999+1,1999+2) and store.s_county in ('Williamson County', 'Franklin Parish', 'Bronx County', 'Orange County') group by ss_ticket_number,ss_customer_sk) dj,customer where ss_customer_sk = c_customer_sk and cnt between 1 and 5 order by cnt desc, c_last_name asc; -- query 74 with year_total as ( select c_customer_id customer_id ,c_first_name customer_first_name ,c_last_name customer_last_name ,d_year as year ,sum(ss_net_paid) year_total ,'s' sale_type from customer ,store_sales ,date_dim where c_customer_sk = ss_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (2001,2001+1) group by c_customer_id ,c_first_name ,c_last_name ,d_year union all select c_customer_id customer_id ,c_first_name customer_first_name ,c_last_name customer_last_name ,d_year as year ,sum(ws_net_paid) year_total ,'w' sale_type from customer ,web_sales ,date_dim where c_customer_sk = ws_bill_customer_sk and ws_sold_date_sk = d_date_sk and d_year in (2001,2001+1) group by c_customer_id ,c_first_name ,c_last_name ,d_year ) select t_s_secyear.customer_id, t_s_secyear.customer_first_name, t_s_secyear.customer_last_name from year_total t_s_firstyear ,year_total t_s_secyear ,year_total t_w_firstyear ,year_total t_w_secyear where t_s_secyear.customer_id = t_s_firstyear.customer_id and t_s_firstyear.customer_id = t_w_secyear.customer_id and t_s_firstyear.customer_id = t_w_firstyear.customer_id and t_s_firstyear.sale_type = 's' and t_w_firstyear.sale_type = 'w' and t_s_secyear.sale_type = 's' and t_w_secyear.sale_type = 'w' and t_s_firstyear.year = 2001 and t_s_secyear.year = 2001+1 and t_w_firstyear.year = 2001 and t_w_secyear.year = 2001+1 and t_s_firstyear.year_total > 0 and t_w_firstyear.year_total > 0 and case when t_w_firstyear.year_total > 0 then t_w_secyear.year_total / t_w_firstyear.year_total else null end > case when t_s_firstyear.year_total > 0 then t_s_secyear.year_total / t_s_firstyear.year_total else null end order by 1, 1, 1 limit 100; -- query 75 WITH all_sales AS ( SELECT d_year ,i_brand_id ,i_class_id ,i_category_id ,i_manufact_id ,SUM(sales_cnt) AS sales_cnt ,SUM(sales_amt) AS sales_amt FROM (SELECT d_year ,i_brand_id ,i_class_id ,i_category_id ,i_manufact_id ,cs_quantity - COALESCE(cr_return_quantity,0) AS sales_cnt ,cs_ext_sales_price - COALESCE(cr_return_amount,0.0) AS sales_amt FROM catalog_sales JOIN item ON i_item_sk=cs_item_sk JOIN date_dim ON d_date_sk=cs_sold_date_sk LEFT JOIN catalog_returns ON (cs_order_number=cr_order_number AND cs_item_sk=cr_item_sk) WHERE i_category='Books' UNION SELECT d_year ,i_brand_id ,i_class_id ,i_category_id ,i_manufact_id ,ss_quantity - COALESCE(sr_return_quantity,0) AS sales_cnt ,ss_ext_sales_price - COALESCE(sr_return_amt,0.0) AS sales_amt FROM store_sales JOIN item ON i_item_sk=ss_item_sk JOIN date_dim ON d_date_sk=ss_sold_date_sk LEFT JOIN store_returns ON (ss_ticket_number=sr_ticket_number AND ss_item_sk=sr_item_sk) WHERE i_category='Books' UNION SELECT d_year ,i_brand_id ,i_class_id ,i_category_id ,i_manufact_id ,ws_quantity - COALESCE(wr_return_quantity,0) AS sales_cnt ,ws_ext_sales_price - COALESCE(wr_return_amt,0.0) AS sales_amt FROM web_sales JOIN item ON i_item_sk=ws_item_sk JOIN date_dim ON d_date_sk=ws_sold_date_sk LEFT JOIN web_returns ON (ws_order_number=wr_order_number AND ws_item_sk=wr_item_sk) WHERE i_category='Books') sales_detail GROUP BY d_year, i_brand_id, i_class_id, i_category_id, i_manufact_id) SELECT prev_yr.d_year AS prev_year ,curr_yr.d_year AS year ,curr_yr.i_brand_id ,curr_yr.i_class_id ,curr_yr.i_category_id ,curr_yr.i_manufact_id ,prev_yr.sales_cnt AS prev_yr_cnt ,curr_yr.sales_cnt AS curr_yr_cnt ,curr_yr.sales_cnt-prev_yr.sales_cnt AS sales_cnt_diff ,curr_yr.sales_amt-prev_yr.sales_amt AS sales_amt_diff FROM all_sales curr_yr, all_sales prev_yr WHERE curr_yr.i_brand_id=prev_yr.i_brand_id AND curr_yr.i_class_id=prev_yr.i_class_id AND curr_yr.i_category_id=prev_yr.i_category_id AND curr_yr.i_manufact_id=prev_yr.i_manufact_id AND curr_yr.d_year=2002 AND prev_yr.d_year=2002-1 AND CAST(curr_yr.sales_cnt AS DECIMAL(17,2))/CAST(prev_yr.sales_cnt AS DECIMAL(17,2))<0.9 ORDER BY sales_cnt_diff,sales_amt_diff limit 100; -- query 76 select channel, col_name, d_year, d_qoy, i_category, COUNT(*) sales_cnt, SUM(ext_sales_price) sales_amt FROM ( SELECT 'store' as channel, 'ss_store_sk' col_name, d_year, d_qoy, i_category, ss_ext_sales_price ext_sales_price FROM store_sales, item, date_dim WHERE ss_store_sk IS NULL AND ss_sold_date_sk=d_date_sk AND ss_item_sk=i_item_sk UNION ALL SELECT 'web' as channel, 'ws_ship_customer_sk' col_name, d_year, d_qoy, i_category, ws_ext_sales_price ext_sales_price FROM web_sales, item, date_dim WHERE ws_ship_customer_sk IS NULL AND ws_sold_date_sk=d_date_sk AND ws_item_sk=i_item_sk UNION ALL SELECT 'catalog' as channel, 'cs_ship_addr_sk' col_name, d_year, d_qoy, i_category, cs_ext_sales_price ext_sales_price FROM catalog_sales, item, date_dim WHERE cs_ship_addr_sk IS NULL AND cs_sold_date_sk=d_date_sk AND cs_item_sk=i_item_sk) foo GROUP BY channel, col_name, d_year, d_qoy, i_category ORDER BY channel, col_name, d_year, d_qoy, i_category limit 100; -- query 77 with ss as (select s_store_sk, sum(ss_ext_sales_price) as sales, sum(ss_net_profit) as profit from store_sales, date_dim, store where ss_sold_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) and ss_store_sk = s_store_sk group by s_store_sk) , sr as (select s_store_sk, sum(sr_return_amt) as returns, sum(sr_net_loss) as profit_loss from store_returns, date_dim, store where sr_returned_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) and sr_store_sk = s_store_sk group by s_store_sk), cs as (select cs_call_center_sk, sum(cs_ext_sales_price) as sales, sum(cs_net_profit) as profit from catalog_sales, date_dim where cs_sold_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) group by cs_call_center_sk ), cr as (select cr_call_center_sk, sum(cr_return_amount) as returns, sum(cr_net_loss) as profit_loss from catalog_returns, date_dim where cr_returned_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) group by cr_call_center_sk ), ws as ( select wp_web_page_sk, sum(ws_ext_sales_price) as sales, sum(ws_net_profit) as profit from web_sales, date_dim, web_page where ws_sold_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) and ws_web_page_sk = wp_web_page_sk group by wp_web_page_sk), wr as (select wp_web_page_sk, sum(wr_return_amt) as returns, sum(wr_net_loss) as profit_loss from web_returns, date_dim, web_page where wr_returned_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) and wr_web_page_sk = wp_web_page_sk group by wp_web_page_sk) select channel , id , sum(sales) as sales , sum(returns) as returns , sum(profit) as profit from (select 'store channel' as channel , ss.s_store_sk as id , sales , coalesce(returns, 0) as returns , (profit - coalesce(profit_loss,0)) as profit from ss left join sr on ss.s_store_sk = sr.s_store_sk union all select 'catalog channel' as channel , cs_call_center_sk as id , sales , returns , (profit - profit_loss) as profit from cs , cr union all select 'web channel' as channel , ws.wp_web_page_sk as id , sales , coalesce(returns, 0) returns , (profit - coalesce(profit_loss,0)) as profit from ws left join wr on ws.wp_web_page_sk = wr.wp_web_page_sk ) x group by rollup (channel, id) order by channel ,id limit 100; -- query 78 with ws as (select d_year AS ws_sold_year, ws_item_sk, ws_bill_customer_sk ws_customer_sk, sum(ws_quantity) ws_qty, sum(ws_wholesale_cost) ws_wc, sum(ws_sales_price) ws_sp from web_sales left join web_returns on wr_order_number=ws_order_number and ws_item_sk=wr_item_sk join date_dim on ws_sold_date_sk = d_date_sk where wr_order_number is null group by d_year, ws_item_sk, ws_bill_customer_sk ), cs as (select d_year AS cs_sold_year, cs_item_sk, cs_bill_customer_sk cs_customer_sk, sum(cs_quantity) cs_qty, sum(cs_wholesale_cost) cs_wc, sum(cs_sales_price) cs_sp from catalog_sales left join catalog_returns on cr_order_number=cs_order_number and cs_item_sk=cr_item_sk join date_dim on cs_sold_date_sk = d_date_sk where cr_order_number is null group by d_year, cs_item_sk, cs_bill_customer_sk ), ss as (select d_year AS ss_sold_year, ss_item_sk, ss_customer_sk, sum(ss_quantity) ss_qty, sum(ss_wholesale_cost) ss_wc, sum(ss_sales_price) ss_sp from store_sales left join store_returns on sr_ticket_number=ss_ticket_number and ss_item_sk=sr_item_sk join date_dim on ss_sold_date_sk = d_date_sk where sr_ticket_number is null group by d_year, ss_item_sk, ss_customer_sk ) select ss_sold_year, ss_item_sk, ss_customer_sk, round(ss_qty/(coalesce(ws_qty,0)+coalesce(cs_qty,0)),2) ratio, ss_qty store_qty, ss_wc store_wholesale_cost, ss_sp store_sales_price, coalesce(ws_qty,0)+coalesce(cs_qty,0) other_chan_qty, coalesce(ws_wc,0)+coalesce(cs_wc,0) other_chan_wholesale_cost, coalesce(ws_sp,0)+coalesce(cs_sp,0) other_chan_sales_price from ss left join ws on (ws_sold_year=ss_sold_year and ws_item_sk=ss_item_sk and ws_customer_sk=ss_customer_sk) left join cs on (cs_sold_year=ss_sold_year and cs_item_sk=ss_item_sk and cs_customer_sk=ss_customer_sk) where (coalesce(ws_qty,0)>0 or coalesce(cs_qty, 0)>0) and ss_sold_year=2000 order by ss_sold_year, ss_item_sk, ss_customer_sk, ss_qty desc, ss_wc desc, ss_sp desc, other_chan_qty, other_chan_wholesale_cost, other_chan_sales_price, ratio limit 100; -- query 79 select c_last_name,c_first_name,substr(s_city,1,30),ss_ticket_number,amt,profit from (select ss_ticket_number ,ss_customer_sk ,store.s_city ,sum(ss_coupon_amt) amt ,sum(ss_net_profit) profit from store_sales,date_dim,store,household_demographics where store_sales.ss_sold_date_sk = date_dim.d_date_sk and store_sales.ss_store_sk = store.s_store_sk and store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk and (household_demographics.hd_dep_count = 6 or household_demographics.hd_vehicle_count > 2) and date_dim.d_dow = 1 and date_dim.d_year in (1999,1999+1,1999+2) and store.s_number_employees between 200 and 295 group by ss_ticket_number,ss_customer_sk,ss_addr_sk,store.s_city) ms,customer where ss_customer_sk = c_customer_sk order by c_last_name,c_first_name,substr(s_city,1,30), profit limit 100; -- query 80 with ssr as (select s_store_id as store_id, sum(ss_ext_sales_price) as sales, sum(coalesce(sr_return_amt, 0)) as returns, sum(ss_net_profit - coalesce(sr_net_loss, 0)) as profit from store_sales left outer join store_returns on (ss_item_sk = sr_item_sk and ss_ticket_number = sr_ticket_number), date_dim, store, item, promotion where ss_sold_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) and ss_store_sk = s_store_sk and ss_item_sk = i_item_sk and i_current_price > 50 and ss_promo_sk = p_promo_sk and p_channel_tv = 'N' group by s_store_id) , csr as (select cp_catalog_page_id as catalog_page_id, sum(cs_ext_sales_price) as sales, sum(coalesce(cr_return_amount, 0)) as returns, sum(cs_net_profit - coalesce(cr_net_loss, 0)) as profit from catalog_sales left outer join catalog_returns on (cs_item_sk = cr_item_sk and cs_order_number = cr_order_number), date_dim, catalog_page, item, promotion where cs_sold_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) and cs_catalog_page_sk = cp_catalog_page_sk and cs_item_sk = i_item_sk and i_current_price > 50 and cs_promo_sk = p_promo_sk and p_channel_tv = 'N' group by cp_catalog_page_id) , wsr as (select web_site_id, sum(ws_ext_sales_price) as sales, sum(coalesce(wr_return_amt, 0)) as returns, sum(ws_net_profit - coalesce(wr_net_loss, 0)) as profit from web_sales left outer join web_returns on (ws_item_sk = wr_item_sk and ws_order_number = wr_order_number), date_dim, web_site, item, promotion where ws_sold_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) and ws_web_site_sk = web_site_sk and ws_item_sk = i_item_sk and i_current_price > 50 and ws_promo_sk = p_promo_sk and p_channel_tv = 'N' group by web_site_id) select channel , id , sum(sales) as sales , sum(returns) as returns , sum(profit) as profit from (select 'store channel' as channel , 'store' || store_id as id , sales , returns , profit from ssr union all select 'catalog channel' as channel , 'catalog_page' || catalog_page_id as id , sales , returns , profit from csr union all select 'web channel' as channel , 'web_site' || web_site_id as id , sales , returns , profit from wsr ) x group by rollup (channel, id) order by channel ,id limit 100; -- query 81 with customer_total_return as (select cr_returning_customer_sk as ctr_customer_sk ,ca_state as ctr_state, sum(cr_return_amt_inc_tax) as ctr_total_return from catalog_returns ,date_dim ,customer_address where cr_returned_date_sk = d_date_sk and d_year =2000 and cr_returning_addr_sk = ca_address_sk group by cr_returning_customer_sk ,ca_state ) select c_customer_id,c_salutation,c_first_name,c_last_name,ca_street_number,ca_street_name ,ca_street_type,ca_suite_number,ca_city,ca_county,ca_state,ca_zip,ca_country,ca_gmt_offset ,ca_location_type,ctr_total_return from customer_total_return ctr1 ,customer_address ,customer where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2 from customer_total_return ctr2 where ctr1.ctr_state = ctr2.ctr_state) and ca_address_sk = c_current_addr_sk and ca_state = 'GA' and ctr1.ctr_customer_sk = c_customer_sk order by c_customer_id,c_salutation,c_first_name,c_last_name,ca_street_number,ca_street_name ,ca_street_type,ca_suite_number,ca_city,ca_county,ca_state,ca_zip,ca_country,ca_gmt_offset ,ca_location_type,ctr_total_return limit 100; -- query 82 select i_item_id ,i_item_desc ,i_current_price from item, inventory, date_dim, store_sales where i_current_price between 62 and 62+30 and inv_item_sk = i_item_sk and d_date_sk=inv_date_sk and d_date between cast('2000-05-25' as date) and date_add(cast('2000-05-25' as date), 60) and i_manufact_id in (129,270,821,423) and inv_quantity_on_hand between 100 and 500 and ss_item_sk = i_item_sk group by i_item_id,i_item_desc,i_current_price order by i_item_id limit 100; -- query 83 with sr_items as (select i_item_id item_id, sum(sr_return_quantity) sr_item_qty from store_returns, item, date_dim where sr_item_sk = i_item_sk and d_date in (select d_date from date_dim where d_week_seq in (select d_week_seq from date_dim where d_date in ('2000-06-30','2000-09-27','2000-11-17'))) and sr_returned_date_sk = d_date_sk group by i_item_id), cr_items as (select i_item_id item_id, sum(cr_return_quantity) cr_item_qty from catalog_returns, item, date_dim where cr_item_sk = i_item_sk and d_date in (select d_date from date_dim where d_week_seq in (select d_week_seq from date_dim where d_date in ('2000-06-30','2000-09-27','2000-11-17'))) and cr_returned_date_sk = d_date_sk group by i_item_id), wr_items as (select i_item_id item_id, sum(wr_return_quantity) wr_item_qty from web_returns, item, date_dim where wr_item_sk = i_item_sk and d_date in (select d_date from date_dim where d_week_seq in (select d_week_seq from date_dim where d_date in ('2000-06-30','2000-09-27','2000-11-17'))) and wr_returned_date_sk = d_date_sk group by i_item_id) select sr_items.item_id ,sr_item_qty ,sr_item_qty/(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 * 100 sr_dev ,cr_item_qty ,cr_item_qty/(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 * 100 cr_dev ,wr_item_qty ,wr_item_qty/(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 * 100 wr_dev ,(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 average from sr_items ,cr_items ,wr_items where sr_items.item_id=cr_items.item_id and sr_items.item_id=wr_items.item_id order by sr_items.item_id ,sr_item_qty limit 100; -- query 84 select c_customer_id as customer_id , coalesce(c_last_name,'') || ', ' || coalesce(c_first_name,'') as customername from customer ,customer_address ,customer_demographics ,household_demographics ,income_band ,store_returns where ca_city = 'Edgewood' and c_current_addr_sk = ca_address_sk and ib_lower_bound >= 38128 and ib_upper_bound <= 38128 + 50000 and ib_income_band_sk = hd_income_band_sk and cd_demo_sk = c_current_cdemo_sk and hd_demo_sk = c_current_hdemo_sk and sr_cdemo_sk = cd_demo_sk order by c_customer_id limit 100; -- query 85 select substr(r_reason_desc,1,20) ,avg(ws_quantity) ,avg(wr_refunded_cash) ,avg(wr_fee) from web_sales, web_returns, web_page, customer_demographics cd1, customer_demographics cd2, customer_address, date_dim, reason where ws_web_page_sk = wp_web_page_sk and ws_item_sk = wr_item_sk and ws_order_number = wr_order_number and ws_sold_date_sk = d_date_sk and d_year = 2000 and cd1.cd_demo_sk = wr_refunded_cdemo_sk and cd2.cd_demo_sk = wr_returning_cdemo_sk and ca_address_sk = wr_refunded_addr_sk and r_reason_sk = wr_reason_sk and ( ( cd1.cd_marital_status = 'M' and cd1.cd_marital_status = cd2.cd_marital_status and cd1.cd_education_status = 'Advanced Degree' and cd1.cd_education_status = cd2.cd_education_status and ws_sales_price between 100.00 and 150.00 ) or ( cd1.cd_marital_status = 'S' and cd1.cd_marital_status = cd2.cd_marital_status and cd1.cd_education_status = 'College' and cd1.cd_education_status = cd2.cd_education_status and ws_sales_price between 50.00 and 100.00 ) or ( cd1.cd_marital_status = 'W' and cd1.cd_marital_status = cd2.cd_marital_status and cd1.cd_education_status = '2 yr Degree' and cd1.cd_education_status = cd2.cd_education_status and ws_sales_price between 150.00 and 200.00 ) ) and ( ( ca_country = 'United States' and ca_state in ('IN', 'OH', 'NJ') and ws_net_profit between 100 and 200 ) or ( ca_country = 'United States' and ca_state in ('WI', 'CT', 'KY') and ws_net_profit between 150 and 300 ) or ( ca_country = 'United States' and ca_state in ('LA', 'IA', 'AR') and ws_net_profit between 50 and 250 ) ) group by r_reason_desc order by substr(r_reason_desc,1,20) ,avg(ws_quantity) ,avg(wr_refunded_cash) ,avg(wr_fee) limit 100; -- query 86 select sum(ws_net_paid) as total_sum ,i_category ,i_class ,grouping(i_category)+grouping(i_class) as lochierarchy ,rank() over ( partition by grouping(i_category)+grouping(i_class), case when grouping(i_class) = 0 then i_category end order by sum(ws_net_paid) desc) as rank_within_parent from web_sales ,date_dim d1 ,item where d1.d_month_seq between 1200 and 1200+11 and d1.d_date_sk = ws_sold_date_sk and i_item_sk = ws_item_sk group by rollup(i_category,i_class) order by lochierarchy desc, case when lochierarchy = 0 then i_category end, rank_within_parent limit 100; -- query 87 select count(*) from ((select distinct c_last_name, c_first_name, d_date from store_sales, date_dim, customer where store_sales.ss_sold_date_sk = date_dim.d_date_sk and store_sales.ss_customer_sk = customer.c_customer_sk and d_month_seq between 1200 and 1200+11) except (select distinct c_last_name, c_first_name, d_date from catalog_sales, date_dim, customer where catalog_sales.cs_sold_date_sk = date_dim.d_date_sk and catalog_sales.cs_bill_customer_sk = customer.c_customer_sk and d_month_seq between 1200 and 1200+11) except (select distinct c_last_name, c_first_name, d_date from web_sales, date_dim, customer where web_sales.ws_sold_date_sk = date_dim.d_date_sk and web_sales.ws_bill_customer_sk = customer.c_customer_sk and d_month_seq between 1200 and 1200+11) ) cool_cust ; -- query 88 select * from (select count(*) h8_30_to_9 from store_sales, household_demographics , time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 8 and time_dim.t_minute >= 30 and ((household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2) or (household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or (household_demographics.hd_dep_count = 0 and household_demographics.hd_vehicle_count<=0+2)) and store.s_store_name = 'ese') s1, (select count(*) h9_to_9_30 from store_sales, household_demographics , time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 9 and time_dim.t_minute < 30 and ((household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2) or (household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or (household_demographics.hd_dep_count = 0 and household_demographics.hd_vehicle_count<=0+2)) and store.s_store_name = 'ese') s2, (select count(*) h9_30_to_10 from store_sales, household_demographics , time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 9 and time_dim.t_minute >= 30 and ((household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2) or (household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or (household_demographics.hd_dep_count = 0 and household_demographics.hd_vehicle_count<=0+2)) and store.s_store_name = 'ese') s3, (select count(*) h10_to_10_30 from store_sales, household_demographics , time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 10 and time_dim.t_minute < 30 and ((household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2) or (household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or (household_demographics.hd_dep_count = 0 and household_demographics.hd_vehicle_count<=0+2)) and store.s_store_name = 'ese') s4, (select count(*) h10_30_to_11 from store_sales, household_demographics , time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 10 and time_dim.t_minute >= 30 and ((household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2) or (household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or (household_demographics.hd_dep_count = 0 and household_demographics.hd_vehicle_count<=0+2)) and store.s_store_name = 'ese') s5, (select count(*) h11_to_11_30 from store_sales, household_demographics , time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 11 and time_dim.t_minute < 30 and ((household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2) or (household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or (household_demographics.hd_dep_count = 0 and household_demographics.hd_vehicle_count<=0+2)) and store.s_store_name = 'ese') s6, (select count(*) h11_30_to_12 from store_sales, household_demographics , time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 11 and time_dim.t_minute >= 30 and ((household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2) or (household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or (household_demographics.hd_dep_count = 0 and household_demographics.hd_vehicle_count<=0+2)) and store.s_store_name = 'ese') s7, (select count(*) h12_to_12_30 from store_sales, household_demographics , time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 12 and time_dim.t_minute < 30 and ((household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2) or (household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or (household_demographics.hd_dep_count = 0 and household_demographics.hd_vehicle_count<=0+2)) and store.s_store_name = 'ese') s8 ; -- query 89 select * from( select i_category, i_class, i_brand, s_store_name, s_company_name, d_moy, sum(ss_sales_price) sum_sales, avg(sum(ss_sales_price)) over (partition by i_category, i_brand, s_store_name, s_company_name) avg_monthly_sales from item, store_sales, date_dim, store where ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and ss_store_sk = s_store_sk and d_year in (1999) and ((i_category in ('Books','Electronics','Sports') and i_class in ('computers','stereo','football') ) or (i_category in ('Men','Jewelry','Women') and i_class in ('shirts','birdal','dresses') )) group by i_category, i_class, i_brand, s_store_name, s_company_name, d_moy) tmp1 where case when (avg_monthly_sales <> 0) then (abs(sum_sales - avg_monthly_sales) / avg_monthly_sales) else null end > 0.1 order by sum_sales - avg_monthly_sales, s_store_name limit 100; -- query 90 select cast(amc as decimal(15,4))/cast(pmc as decimal(15,4)) am_pm_ratio from ( select count(*) amc from web_sales, household_demographics , time_dim, web_page where ws_sold_time_sk = time_dim.t_time_sk and ws_ship_hdemo_sk = household_demographics.hd_demo_sk and ws_web_page_sk = web_page.wp_web_page_sk and time_dim.t_hour between 8 and 8+1 and household_demographics.hd_dep_count = 6 and web_page.wp_char_count between 5000 and 5200) at, ( select count(*) pmc from web_sales, household_demographics , time_dim, web_page where ws_sold_time_sk = time_dim.t_time_sk and ws_ship_hdemo_sk = household_demographics.hd_demo_sk and ws_web_page_sk = web_page.wp_web_page_sk and time_dim.t_hour between 19 and 19+1 and household_demographics.hd_dep_count = 6 and web_page.wp_char_count between 5000 and 5200) pt order by am_pm_ratio limit 100; -- query 91 select cc_call_center_id Call_Center, cc_name Call_Center_Name, cc_manager Manager, sum(cr_net_loss) Returns_Loss from call_center, catalog_returns, date_dim, customer, customer_address, customer_demographics, household_demographics where cr_call_center_sk = cc_call_center_sk and cr_returned_date_sk = d_date_sk and cr_returning_customer_sk= c_customer_sk and cd_demo_sk = c_current_cdemo_sk and hd_demo_sk = c_current_hdemo_sk and ca_address_sk = c_current_addr_sk and d_year = 1998 and d_moy = 11 and ( (cd_marital_status = 'M' and cd_education_status = 'Unknown') or(cd_marital_status = 'W' and cd_education_status = 'Advanced Degree')) and hd_buy_potential like 'Unknown%' and ca_gmt_offset = -7 group by cc_call_center_id,cc_name,cc_manager,cd_marital_status,cd_education_status order by sum(cr_net_loss) desc; -- query 92 select sum(ws_ext_discount_amt) as "Excess Discount Amount" from web_sales ,item ,date_dim where i_manufact_id = 350 and i_item_sk = ws_item_sk and d_date between '2000-01-27' and date_add(cast('2000-01-27' as date), 90) and d_date_sk = ws_sold_date_sk and ws_ext_discount_amt > ( SELECT 1.3 * avg(ws_ext_discount_amt) FROM web_sales ,date_dim WHERE ws_item_sk = i_item_sk and d_date between '2000-01-27' and date_add(cast('2000-01-27' as date), 90) and d_date_sk = ws_sold_date_sk ) order by sum(ws_ext_discount_amt) limit 100; -- query 93 select ss_customer_sk ,sum(act_sales) sumsales from (select ss_item_sk ,ss_ticket_number ,ss_customer_sk ,case when sr_return_quantity is not null then (ss_quantity-sr_return_quantity)*ss_sales_price else (ss_quantity*ss_sales_price) end act_sales from store_sales left outer join store_returns on (sr_item_sk = ss_item_sk and sr_ticket_number = ss_ticket_number) ,reason where sr_reason_sk = r_reason_sk and r_reason_desc = 'reason 28') t group by ss_customer_sk order by sumsales, ss_customer_sk limit 100; -- query 94 select count(distinct ws_order_number) as "order count" ,sum(ws_ext_ship_cost) as "total shipping cost" ,sum(ws_net_profit) as "total net profit" from web_sales ws1 ,date_dim ,customer_address ,web_site where d_date between '1999-2-01' and date_add(cast('1999-2-01' as date), 60) and ws1.ws_ship_date_sk = d_date_sk and ws1.ws_ship_addr_sk = ca_address_sk and ca_state = 'IL' and ws1.ws_web_site_sk = web_site_sk and web_company_name = 'pri' and exists (select * from web_sales ws2 where ws1.ws_order_number = ws2.ws_order_number and ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk) and not exists(select * from web_returns wr1 where ws1.ws_order_number = wr1.wr_order_number) order by count(distinct ws_order_number) limit 100; -- query 95 with ws_wh as (select ws1.ws_order_number,ws1.ws_warehouse_sk wh1,ws2.ws_warehouse_sk wh2 from web_sales ws1,web_sales ws2 where ws1.ws_order_number = ws2.ws_order_number and ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk) select count(distinct ws_order_number) as "order count" ,sum(ws_ext_ship_cost) as "total shipping cost" ,sum(ws_net_profit) as "total net profit" from web_sales ws1 ,date_dim ,customer_address ,web_site where d_date between '1999-2-01' and date_add(cast('1999-2-01' as date), 60) and ws1.ws_ship_date_sk = d_date_sk and ws1.ws_ship_addr_sk = ca_address_sk and ca_state = 'IL' and ws1.ws_web_site_sk = web_site_sk and web_company_name = 'pri' and ws1.ws_order_number in (select ws_order_number from ws_wh) and ws1.ws_order_number in (select wr_order_number from web_returns,ws_wh where wr_order_number = ws_wh.ws_order_number) order by count(distinct ws_order_number) limit 100; -- query 96 select count(*) from store_sales ,household_demographics ,time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 20 and time_dim.t_minute >= 30 and household_demographics.hd_dep_count = 7 and store.s_store_name = 'ese' order by count(*) limit 100; -- query 97 with ssci as ( select ss_customer_sk customer_sk ,ss_item_sk item_sk from store_sales,date_dim where ss_sold_date_sk = d_date_sk and d_month_seq between 1200 and 1200 + 11 group by ss_customer_sk ,ss_item_sk), csci as( select cs_bill_customer_sk customer_sk ,cs_item_sk item_sk from catalog_sales,date_dim where cs_sold_date_sk = d_date_sk and d_month_seq between 1200 and 1200 + 11 group by cs_bill_customer_sk ,cs_item_sk) select sum(case when ssci.customer_sk is not null and csci.customer_sk is null then 1 else 0 end) store_only ,sum(case when ssci.customer_sk is null and csci.customer_sk is not null then 1 else 0 end) catalog_only ,sum(case when ssci.customer_sk is not null and csci.customer_sk is not null then 1 else 0 end) store_and_catalog from ssci full outer join csci on (ssci.customer_sk=csci.customer_sk and ssci.item_sk = csci.item_sk) limit 100; -- query 98 select i_item_id ,i_item_desc ,i_category ,i_class ,i_current_price ,sum(ss_ext_sales_price) as itemrevenue ,sum(ss_ext_sales_price)*100/sum(sum(ss_ext_sales_price)) over (partition by i_class) as revenueratio from store_sales ,item ,date_dim where ss_item_sk = i_item_sk and i_category in ('Sports', 'Books', 'Home') and ss_sold_date_sk = d_date_sk and d_date between cast('1999-02-22' as date) and date_add(cast('1999-02-22' as date), 30) group by i_item_id ,i_item_desc ,i_category ,i_class ,i_current_price order by i_category ,i_class ,i_item_id ,i_item_desc ,revenueratio; -- query 99 select substr(w_warehouse_name,1,20) ,sm_type ,cc_name ,sum(case when (cs_ship_date_sk - cs_sold_date_sk <= 30 ) then 1 else 0 end) as "30 days" ,sum(case when (cs_ship_date_sk - cs_sold_date_sk > 30) and (cs_ship_date_sk - cs_sold_date_sk <= 60) then 1 else 0 end ) as "31-60 days" ,sum(case when (cs_ship_date_sk - cs_sold_date_sk > 60) and (cs_ship_date_sk - cs_sold_date_sk <= 90) then 1 else 0 end) as "61-90 days" ,sum(case when (cs_ship_date_sk - cs_sold_date_sk > 90) and (cs_ship_date_sk - cs_sold_date_sk <= 120) then 1 else 0 end) as "91-120 days" ,sum(case when (cs_ship_date_sk - cs_sold_date_sk > 120) then 1 else 0 end) as ">120 days" from catalog_sales ,warehouse ,ship_mode ,call_center ,date_dim where d_month_seq between 1200 and 1200 + 11 and cs_ship_date_sk = d_date_sk and cs_warehouse_sk = w_warehouse_sk and cs_ship_mode_sk = sm_ship_mode_sk and cs_call_center_sk = cc_call_center_sk group by substr(w_warehouse_name,1,20) ,sm_type ,cc_name order by substr(w_warehouse_name,1,20) ,sm_type ,cc_name limit 100; ``` --- ### TPC-DS Benchmarking TPC-DS is a decision support benchmark developed by the Transaction Processing Performance Council (TPC). It uses more comprehensive test datasets and complex SQL queries than TPC-H. TPC-DS models several generally applicable aspects of a decision support system, including queries and data maintenance. TPC-DS aims to provide a comprehensive and realistic workload for testing and evaluating the performance of database systems in a retail environment. The TPC-DS benchmark simulates the sales and return data of three sales channels (stores, Internet, and catalog) in a retail enterprise. In addition to creating tables for sales and return data models, it also includes a simple inventory system and a promotion system. This benchmark tests a total of 99 complex SQL queries against 24 tables whose data size ranges from 1 GB to 3 TB. The main performance metric is the response time of each query, which is the duration between the time a query is submitted to the time the result is returned. #### Test Conclusions[​](#test-conclusions "Direct link to Test Conclusions") The test is performed against the TPC-DS 1 TB dataset on StarRocks and Trino. The unit of the following results are milliseconds. ![TPC-DS-SR](/assets/images/TPC-DS-SR-298c467caf034bdc5c5de30658886659.png) For StarRocks, the test queries are performed on its native tables (under both shared-nothing and shared-data architecture) and Iceberg Catalog (under shared-data architecture). For Trino, the test queries are performed on the same Iceberg Catalog. Both tests for StarRocks and Trino on the Iceberg Catalog use AWS Glue as the metastore, and Parquet-formatted data with ZSTD compression. The test results are: * Queries performed on the OLAP table in the StarRocks shared-nothing cluster took 314 seconds. * Queries performed on the OLAP table in the StarRocks shared-data cluster took 326 seconds. * Queries performed on Iceberg Catalog in the StarRocks shared-data cluster took 368 seconds. * Queries performed on Iceberg Catalog in Trino took 2552 seconds. The conclusions are: * **StarRocks delivers comparable performance in both shared-nothing and shared-data architecture** In OLAP table query scenarios, StarRocks in the shared-nothing mode achieves 1.04× the performance of the shared-data mode, indicating minimal performance overhead from storage-compute separation. * **StarRocks significantly outperforms Trino in Lakehouse query scenarios out of the box, with even greater advantages in OLAP table queries** * In Iceberg Catalog query scenarios, StarRocks delivers 6.93× the performance of Trino. * In OLAP table queries, the performance gap widens further—StarRocks achieves 8.13× (under shared-nothing mode) and 7.82× (under shared-data mode) the performance of Trino. #### Test Preparation[​](#test-preparation "Direct link to Test Preparation") ##### Hardware[​](#hardware "Direct link to Hardware") StarRocks and Trino are deployed on hosts of the same configurations - [AWS m6id.4xlarge](https://aws.amazon.com/ec2/instance-types/m6i/?nc1=h_ls). | | **Spec** | | ------------------------ | -------------- | | Instance Number | 5 | | vCPU | 16 | | Memory (GiB) | 64 | | Instance Storage (GB) | 1x950 NVMe SSD | | Network Bandwidth (Gbps) | Up to 12.5 | | EBS Bandwidth (Gbps) | Up to 10 | ##### Software[​](#software "Direct link to Software") | | **StarRocks** | **Trino** | | ----------------- | ------------------------------------------ | -------------------------------------------------------------- | | **Cluster Size** | One FE, four BE | One Coordinator, four Workers | | **Version** | 3.5.0 | 475 | | **Release Date** | 2025.6.13 | 2025.4.23 | | **Configuration** | `set query_mem_limit=66571993088;` (62 GB) | `query.max-memory=224GB`
`query.max-memory-per-node=35GB` | #### Test results[​](#test-results "Direct link to Test results") The following table shows the performance test results on 99 queries. The unit of query latency is milliseconds. All queries are warmed up 1 time and then executed 3 times to take the average value as the result. `Trino vs StarRocks on Iceberg` in the table header means using the query response time of Trino to divide the query response time of StarRocks. A larger value indicates better performance of StarRocks. Before the test on Iceberg Catalog, `ANALYZE TABLE` and `EXPLAIN COST` statements are executed on StarRocks to collect statistics. | Query | StarRocks | | | Trino | Trino vs StarRocks on Iceberg | | ----- | -------------- | ----------- | --------------- | --------------- | ----------------------------- | | | OLAP Table | | Iceberg Catalog | Iceberg Catalog | | | | Shared-nothing | Shared-data | Share-data | | | | SUM | 313582 | 326163 | 368404 | 2552076 | 6.93 | | Q01 | 675 | 636 | 727 | 2276 | 3.13 | | Q02 | 935 | 925 | 1949 | 21289 | 10.92 | | Q03 | 220 | 140 | 399 | 1764 | 4.42 | | Q04 | 17185 | 16206 | 16744 | 200249 | 11.96 | | Q05 | 417 | 543 | 1724 | 5574 | 3.23 | | Q06 | 200 | 180 | 343 | 2909 | 8.48 | | Q07 | 814 | 959 | 1795 | 3638 | 2.03 | | Q08 | 235 | 181 | 319 | 2534 | 7.94 | | Q09 | 7097 | 6503 | 23153 | 36125 | 1.56 | | Q10 | 297 | 343 | 497 | 2471 | 4.97 | | Q11 | 11229 | 10249 | 9777 | 133230 | 13.63 | | Q12 | 128 | 130 | 207 | 1067 | 5.15 | | Q13 | 417 | 447 | 2218 | 18297 | 8.25 | | Q14 | 15112 | 15055 | 15310 | 143172 | 9.35 | | Q15 | 455 | 474 | 525 | 2158 | 4.11 | | Q16 | 476 | 523 | 975 | 14035 | 14.39 | | Q17 | 895 | 948 | 1645 | 5487 | 3.34 | | Q18 | 802 | 973 | 1544 | 4736 | 3.07 | | Q19 | 209 | 268 | 507 | 1870 | 3.69 | | Q20 | 147 | 176 | 253 | 1257 | 4.97 | | Q21 | 83 | 80 | 405 | 945 | 2.33 | | Q22 | 1462 | 1823 | 2384 | 7871 | 3.30 | | Q23 | 82887 | 84426 | 50719 | 488653 | 9.63 | | Q24 | 7649 | 9064 | 11902 | 43436 | 3.65 | | Q25 | 795 | 919 | 1404 | 4347 | 3.10 | | Q26 | 376 | 511 | 868 | 2961 | 3.41 | | Q27 | 458 | 532 | 1397 | 4219 | 3.02 | | Q28 | 6489 | 6004 | 18727 | 24678 | 1.32 | | Q29 | 1296 | 2168 | 2129 | 10640 | 5.00 | | Q30 | 295 | 330 | 529 | 2770 | 5.24 | | Q31 | 2172 | 2162 | 2535 | 10260 | 4.05 | | Q32 | 110 | 225 | 277 | 1035 | 3.74 | | Q33 | 235 | 245 | 734 | 2743 | 3.74 | | Q34 | 402 | 422 | 696 | 2455 | 3.53 | | Q35 | 1353 | 1393 | 1562 | 8023 | 5.14 | | Q36 | 553 | 559 | 1193 | 3342 | 2.80 | | Q37 | 263 | 243 | 217 | 6178 | 28.47 | | Q38 | 5002 | 5207 | 5519 | 38408 | 6.96 | | Q39 | 356 | 367 | 820 | 5696 | 6.95 | | Q40 | 124 | 134 | 575 | 1554 | 2.70 | | Q41 | 43 | 50 | 76 | 271 | 3.57 | | Q42 | 81 | 104 | 197 | 996 | 5.06 | | Q43 | 487 | 733 | 722 | 4108 | 5.69 | | Q44 | 2087 | 2102 | 9364 | 16468 | 1.76 | | Q45 | 431 | 470 | 514 | 2212 | 4.30 | | Q46 | 1468 | 1829 | 1976 | 4780 | 2.42 | | Q47 | 2921 | 3120 | 3130 | 42704 | 13.64 | | Q48 | 467 | 573 | 1441 | 12464 | 8.65 | | Q49 | 496 | 547 | 1715 | 3827 | 2.23 | | Q50 | 3768 | 5135 | 3899 | 21771 | 5.58 | | Q51 | 4826 | 4633 | 3244 | 12247 | 3.78 | | Q52 | 87 | 106 | 219 | 1007 | 4.60 | | Q53 | 446 | 635 | 955 | 1768 | 1.85 | | Q54 | 198 | 266 | 763 | 31864 | 41.76 | | Q55 | 82 | 105 | 221 | 1187 | 5.37 | | Q56 | 175 | 186 | 499 | 2133 | 4.27 | | Q57 | 1848 | 1787 | 1916 | 26042 | 13.59 | | Q58 | 241 | 245 | 503 | 3032 | 6.03 | | Q59 | 3935 | 3908 | 3160 | 20819 | 6.59 | | Q60 | 262 | 292 | 643 | 2749 | 4.28 | | Q61 | 330 | 344 | 913 | 2054 | 2.25 | | Q62 | 500 | 519 | 840 | 5104 | 6.08 | | Q63 | 448 | 629 | 937 | 1719 | 1.83 | | Q64 | 4508 | 8886 | 8602 | 16990 | 1.98 | | Q65 | 5690 | 5617 | 5445 | 20616 | 3.79 | | Q66 | 387 | 390 | 1335 | 3511 | 2.63 | | Q67 | 31286 | 29240 | 21439 | 89430 | 4.17 | | Q68 | 240 | 341 | 683 | 2883 | 4.22 | | Q69 | 260 | 292 | 501 | 2181 | 4.35 | | Q70 | 2904 | 3012 | 3751 | 21499 | 5.73 | | Q71 | 272 | 1111 | 631 | 2578 | 4.09 | | Q72 | 1642 | 1899 | 4969 | 504573 | 101.54 | | Q73 | 186 | 228 | 388 | 1933 | 4.98 | | Q74 | 9433 | 9133 | 8890 | 78071 | 8.78 | | Q75 | 6446 | 7718 | 9082 | 26270 | 2.89 | | Q76 | 2356 | 2331 | 5028 | 11266 | 2.24 | | Q77 | 238 | 241 | 701 | 2785 | 3.97 | | Q78 | 17583 | 16590 | 18614 | 51497 | 2.77 | | Q79 | 2201 | 2474 | 1597 | 5572 | 3.49 | | Q80 | 800 | 855 | 2672 | 4762 | 1.78 | | Q81 | 518 | 533 | 769 | 4040 | 5.25 | | Q82 | 623 | 613 | 854 | 11622 | 13.61 | | Q83 | 158 | 217 | 404 | 2274 | 5.63 | | Q84 | 195 | 434 | 351 | 3086 | 8.79 | | Q85 | 455 | 733 | 1087 | 5424 | 4.99 | | Q86 | 725 | 736 | 819 | 3422 | 4.18 | | Q87 | 4816 | 5004 | 5535 | 41300 | 7.46 | | Q88 | 8712 | 8444 | 19465 | 30738 | 1.58 | | Q89 | 530 | 730 | 915 | 2227 | 2.43 | | Q90 | 587 | 581 | 1150 | 3890 | 3.38 | | Q91 | 95 | 96 | 227 | 2035 | 8.96 | | Q92 | 78 | 140 | 228 | 994 | 4.36 | | Q93 | 3723 | 4705 | 5781 | 22926 | 3.97 | | Q94 | 601 | 963 | 962 | 5884 | 6.12 | | Q95 | 1747 | 3106 | 2575 | 29607 | 11.50 | | Q96 | 1444 | 1428 | 2562 | 4905 | 1.91 | | Q97 | 4795 | 4844 | 5383 | 22726 | 4.22 | | Q98 | 284 | 323 | 365 | 1911 | 5.24 | | Q99 | 1132 | 1184 | 1594 | 10970 | 6.88 | --- ### TPC-H Benchmark TPC-H is a decision support benchmark developed by the Transaction Processing Performance Council (TPC). It consists of a suite of business oriented ad-hoc queries and concurrent data modifications. TPC-H can be used to build models based on real production environments to simulate the data warehouse of a sales system. This test uses eight tables with a data size ranging from 1 GB to 3 TB. A total of 22 queries are tested and the main performance metrics are the response time of each query, which is the duration between the time a query is submitted to the time the result is returned. #### 1. Test conclusion[​](#1-test-conclusion "Direct link to 1. Test conclusion") A comparative test was conducted on the TPC-H 100G scale dataset, with a total of 22 queries. The results are as follows: ![TPCH 100G results](/assets/images/tpch-c2ac4e648428201f0083970d29fb34ad.png) StarRocks tested two methods using local storage and Hive table queries. StarRocks Hive table and Trino query the same data. The data is stored in ORC format and compressed in zlib format. The latency for StarRocks to query data from its native storage is 21s, that for StarRocks to query Hive external tables is 92s, and that for Trino to query Hive external tables was 187s. #### 2. Test Preparation[​](#2-test-preparation "Direct link to 2. Test Preparation") ##### 2.1 Hardware environment[​](#21-hardware-environment "Direct link to 2.1 Hardware environment") | Machine | 4 Cloud hosts | | ----------------- | ----------------------------------------------------- | | CPU | 16core Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz | | Memory | 64 GB | | Network Bandwidth | 5 Gbits/s | | Disk | ESSD cloud disk | ##### 2.2 Software environment[​](#22-software-environment "Direct link to 2.2 Software environment") StarRocks and Trino are deployed on machines with the same configuration. StarRocks has 1 FE and 3 BEs deployed. Trino has 1 Coordinator and 3 Workers deployed. * Kernel version: Linux 3.10.0-1127.13.1.el7.x86\_64 * Operating system version: CentOS Linux release 7.8.2003 * Software version: StarRocks Community Edition 3.0, Trino-419, Hive-2.3.9 #### 3 Test data and results[​](#3-test-data-and-results "Direct link to 3 Test data and results") ##### 3.1 Test data[​](#31-test-data "Direct link to 3.1 Test data") | table | number of rows | | -------- | -------------- | | customer | 15 million | | lineitem | 600 million | | nation | 25 | | orders | 150 million | | part | 20 million | | partsupp | 80 million | | region | 5 | | supplier | 1 million | ##### 3.2 Test results[​](#32-test-results "Direct link to 3.2 Test results") note The unit of query results is ms. Lower is better. All queries are warmed up 1 time and then executed 3 times to take the average value as the result. | Query | StarRocks-native-3.0 | StarRocks-3.0-Hive external | Trino-419 | | ----- | -------------------- | --------------------------- | --------- | | Q1 | 1540 | 5660 | 8811 | | Q2 | 100 | 1593 | 3009 | | Q3 | 700 | 5286 | 7891 | | Q4 | 423 | 2110 | 5760 | | Q5 | 1180 | 4453 | 9181 | | Q6 | 56 | 2806 | 4029 | | Q7 | 903 | 4910 | 7158 | | Q8 | 546 | 4766 | 8014 | | Q9 | 2553 | 8010 | 18460 | | Q10 | 776 | 8190 | 9997 | | Q11 | 206 | 920 | 2088 | | Q12 | 166 | 2916 | 4852 | | Q13 | 1663 | 3420 | 7203 | | Q14 | 146 | 3286 | 4995 | | Q15 | 123 | 4173 | 9688 | | Q16 | 353 | 1126 | 2545 | | Q17 | 296 | 3426 | 18970 | | Q18 | 2713 | 7960 | 21763 | | Q19 | 246 | 4406 | 6586 | | Q20 | 176 | 3280 | 6632 | | Q21 | 1410 | 7933 | 16873 | | Q22 | 350 | 1190 | 2788 | | SUM | 16625 | 91820 | 187293 | #### 4. Test process[​](#4-test-process "Direct link to 4. Test process") ##### 4.1 Query StarRocks Native Table[​](#41-query-starrocks-native-table "Direct link to 4.1 Query StarRocks Native Table") ###### 4.1.1 Generate data[​](#411-generate-data "Direct link to 4.1.1 Generate data") Download the tpch-poc tool package to generate data for the TPC-H standard test set `scale factor=100`. ```bash wget https://starrocks-public.oss-cn-zhangjiakou.aliyuncs.com/tpch-poc-1.0.zip unzip tpch-poc-1.0 cd tpch-poc-1.0 sh bin/gen_data/gen-tpch.sh 100 data_100 ``` ###### 4.1.2 Create table structure[​](#412-create-table-structure "Direct link to 4.1.2 Create table structure") Modify the configuration file `conf/starrocks.conf`, specify the cluster address (host and port), and then perform the table creation operation. ```sql sh bin/create_db_table.sh ddl_100 ``` ###### 4.1.3 Import data[​](#413-import-data "Direct link to 4.1.3 Import data") ```python sh bin/stream_load.sh data_100 ``` ###### 4.1.4 Query data[​](#414-query-data "Direct link to 4.1.4 Query data") ```python sh bin/benchmark.sh ``` ##### 4.2 Query Hive external tables with StarRocks[​](#42-query-hive-external-tables-with-starrocks "Direct link to 4.2 Query Hive external tables with StarRocks") ###### 4.2.1 Create table structure[​](#421-create-table-structure "Direct link to 4.2.1 Create table structure") Create an external table in Hive. The external table storage format is ORC and the compression format is zlib. For detailed table creation statements, see [5.3](#53-hive-external-table-creation-orc-storage-format). This Hive external table is used for both StarRocks and Trino queries. ###### 4.2.2 Import data[​](#422-import-data "Direct link to 4.2.2 Import data") Upload the TPC-H CSV original data generated in step 4.1.1 to the HDFS specified path (this article uses the path `/user/tmp/csv/`), and then create an external table in Hive. For detailed table creation statements, see [5.4](#54-hive-external-table-creation-csv-storage-format). The storage format of this Hive external table is CSV, and the storage path is `/user/tmp/csv/`, the path where the CSV original data is uploaded. Import the data of the external table in CSV format into the external table in ORC format using `INSERT INTO`, so that the data is obtained in the storage format of ORC and the compression format of zlib. The import command is as follows: ```sql use tpch_hive_csv; insert into tpch_hive_orc.customer select * from customer; insert into tpch_hive_orc.lineitem select * from lineitem; insert into tpch_hive_orc.nation select * from nation; insert into tpch_hive_orc.orders select * from orders; insert into tpch_hive_orc.part select * from part; insert into tpch_hive_orc.partsupp select * from partsupp; insert into tpch_hive_orc.region select * from region; insert into tpch_hive_orc.supplier select * from supplier; ``` ###### 4.2.3 Query data[​](#423-query-data "Direct link to 4.2.3 Query data") StarRocks uses the Catalog function to query Hive table data. For detailed operations, see [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md). #### 5. Query SQL and create table statements[​](#5-query-sql-and-create-table-statements "Direct link to 5. Query SQL and create table statements") ##### 5.1 TPC-H query SQL[​](#51-tpc-h-query-sql "Direct link to 5.1 TPC-H query SQL") ```sql --Q1 select l_returnflag, l_linestatus, sum(l_quantity) as sum_qty, sum(l_extendedprice) as sum_base_price, sum(l_extendedprice * (1 - l_discount)) as sum_disc_price, sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge, avg(l_quantity) as avg_qty, avg(l_extendedprice) as avg_price, avg(l_discount) as avg_disc, count(*) as count_order from lineitem where l_shipdate <= date '1998-12-01' - interval '90' day group by l_returnflag, l_linestatus order by l_returnflag, l_linestatus; --Q2 select s_acctbal, s_name, n_name, p_partkey, p_mfgr, s_address, s_phone, s_comment from part, supplier, partsupp, nation, region where p_partkey = ps_partkey and s_suppkey = ps_suppkey and p_size = 15 and p_type like '%BRASS' and s_nationkey = n_nationkey and n_regionkey = r_regionkey and r_name = 'EUROPE' and ps_supplycost = ( select min(ps_supplycost) from partsupp, supplier, nation, region where p_partkey = ps_partkey and s_suppkey = ps_suppkey and s_nationkey = n_nationkey and n_regionkey = r_regionkey and r_name = 'EUROPE' ) order by s_acctbal desc, n_name, s_name, p_partkey limit 100; --Q3 select l_orderkey, sum(l_extendedprice * (1 - l_discount)) as revenue, o_orderdate, o_shippriority from customer, orders, lineitem where c_mktsegment = 'BUILDING' and c_custkey = o_custkey and l_orderkey = o_orderkey and o_orderdate < date '1995-03-15' and l_shipdate > date '1995-03-15' group by l_orderkey, o_orderdate, o_shippriority order by revenue desc, o_orderdate limit 10; --Q4 select o_orderpriority, count(*) as order_count from orders where o_orderdate >= date '1993-07-01' and o_orderdate < date '1993-07-01' + interval '3' month and exists ( select * from lineitem where l_orderkey = o_orderkey and l_commitdate < l_receiptdate ) group by o_orderpriority order by o_orderpriority; --Q5 select n_name, sum(l_extendedprice * (1 - l_discount)) as revenue from customer, orders, lineitem, supplier, nation, region where c_custkey = o_custkey and l_orderkey = o_orderkey and l_suppkey = s_suppkey and c_nationkey = s_nationkey and s_nationkey = n_nationkey and n_regionkey = r_regionkey and r_name = 'ASIA' and o_orderdate >= date '1994-01-01' and o_orderdate < date '1994-01-01' + interval '1' year group by n_name order by revenue desc; --Q6 select sum(l_extendedprice * l_discount) as revenue from lineitem where l_shipdate >= date '1994-01-01' and l_shipdate < date '1994-01-01' + interval '1' year and l_discount between .06 - 0.01 and .06 + 0.01 and l_quantity < 24; --Q7 select supp_nation, cust_nation, l_year, sum(volume) as revenue from ( select n1.n_name as supp_nation, n2.n_name as cust_nation, extract(year from l_shipdate) as l_year, l_extendedprice * (1 - l_discount) as volume from supplier, lineitem, orders, customer, nation n1, nation n2 where s_suppkey = l_suppkey and o_orderkey = l_orderkey and c_custkey = o_custkey and s_nationkey = n1.n_nationkey and c_nationkey = n2.n_nationkey and ( (n1.n_name = 'FRANCE' and n2.n_name = 'GERMANY') or (n1.n_name = 'GERMANY' and n2.n_name = 'FRANCE') ) and l_shipdate between date '1995-01-01' and date '1996-12-31' ) as shipping group by supp_nation, cust_nation, l_year order by supp_nation, cust_nation, l_year; --Q8 select o_year, sum(case when nation = 'BRAZIL' then volume else 0 end) / sum(volume) as mkt_share from ( select extract(year from o_orderdate) as o_year, l_extendedprice * (1 - l_discount) as volume, n2.n_name as nation from part, supplier, lineitem, orders, customer, nation n1, nation n2, region where p_partkey = l_partkey and s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey and c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name = 'AMERICA' and s_nationkey = n2.n_nationkey and o_orderdate between date '1995-01-01' and date '1996-12-31' and p_type = 'ECONOMY ANODIZED STEEL' ) as all_nations group by o_year order by o_year; --Q9 select nation, o_year, sum(amount) as sum_profit from ( select n_name as nation, extract(year from o_orderdate) as o_year, l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity as amount from part, supplier, lineitem, partsupp, orders, nation where s_suppkey = l_suppkey and ps_suppkey = l_suppkey and ps_partkey = l_partkey and p_partkey = l_partkey and o_orderkey = l_orderkey and s_nationkey = n_nationkey and p_name like '%green%' ) as profit group by nation, o_year order by nation, o_year desc; --Q10 select c_custkey, c_name, sum(l_extendedprice * (1 - l_discount)) as revenue, c_acctbal, n_name, c_address, c_phone, c_comment from customer, orders, lineitem, nation where c_custkey = o_custkey and l_orderkey = o_orderkey and o_orderdate >= date '1993-10-01' and o_orderdate < date '1993-10-01' + interval '3' month and l_returnflag = 'R' and c_nationkey = n_nationkey group by c_custkey, c_name, c_acctbal, c_phone, n_name, c_address, c_comment order by revenue desc limit 20; --Q11 select ps_partkey, sum(ps_supplycost * ps_availqty) as value from partsupp, supplier, nation where ps_suppkey = s_suppkey and s_nationkey = n_nationkey and n_name = 'GERMANY' group by ps_partkey having sum(ps_supplycost * ps_availqty) > ( select sum(ps_supplycost * ps_availqty) * 0.000001 from partsupp, supplier, nation where ps_suppkey = s_suppkey and s_nationkey = n_nationkey and n_name = 'GERMANY' ) order by value desc; --Q12 select l_shipmode, sum(case when o_orderpriority = '1-URGENT' or o_orderpriority = '2-HIGH' then 1 else 0 end) as high_line_count, sum(case when o_orderpriority <> '1-URGENT' and o_orderpriority <> '2-HIGH' then 1 else 0 end) as low_line_count from orders, lineitem where o_orderkey = l_orderkey and l_shipmode in ('MAIL', 'SHIP') and l_commitdate < l_receiptdate and l_shipdate < l_commitdate and l_receiptdate >= date '1994-01-01' and l_receiptdate < date '1994-01-01' + interval '1' year group by l_shipmode order by l_shipmode; --Q13 select c_count, count(*) as custdist from ( select c_custkey, count(o_orderkey) as c_count from customer left outer join orders on c_custkey = o_custkey and o_comment not like '%special%requests%' group by c_custkey ) as c_orders group by c_count order by custdist desc, c_count desc; --Q14 select 100.00 * sum(case when p_type like 'PROMO%' then l_extendedprice * (1 - l_discount) else 0 end) / sum(l_extendedprice * (1 - l_discount)) as promo_revenue from lineitem, part where l_partkey = p_partkey and l_shipdate >= date '1995-09-01' and l_shipdate < date '1995-09-01' + interval '1' month; --Q15 select s_suppkey, s_name, s_address, s_phone, total_revenue from supplier, revenue0 where s_suppkey = supplier_no and total_revenue = ( select max(total_revenue) from revenue0 ) order by s_suppkey; --Q16 select p_brand, p_type, p_size, count(distinct ps_suppkey) as supplier_cnt from partsupp, part where p_partkey = ps_partkey and p_brand <> 'Brand#45' and p_type not like 'MEDIUM POLISHED%' and p_size in (49, 14, 23, 45, 19, 3, 36, 9) and ps_suppkey not in ( select s_suppkey from supplier where s_comment like '%Customer%Complaints%' ) group by p_brand, p_type, p_size order by supplier_cnt desc, p_brand, p_type, p_size; --Q17 select sum(l_extendedprice) / 7.0 as avg_yearly from lineitem, part where p_partkey = l_partkey and p_brand = 'Brand#23' and p_container = 'MED BOX' and l_quantity < ( select 0.2 *avg(l_quantity) from lineitem where l_partkey = p_partkey ); --Q18 select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice, sum(l_quantity) from customer, orders, lineitem where o_orderkey in ( select l_orderkey from lineitem group by l_orderkey having sum(l_quantity) > 300 ) and c_custkey = o_custkey and o_orderkey = l_orderkey group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice order by o_totalprice desc, o_orderdate limit 100; --Q19 select sum(l_extendedprice* (1 - l_discount)) as revenue from lineitem, part where ( p_partkey = l_partkey and p_brand = 'Brand#12' and p_container in ('SM CASE', 'SM BOX', 'SM PACK', 'SM PKG') and l_quantity >= 1 and l_quantity <= 1 + 10 and p_size between 1 and 5 and l_shipmode in ('AIR', 'AIR REG') and l_shipinstruct = 'DELIVER IN PERSON' ) or ( p_partkey = l_partkey and p_brand = 'Brand#23' and p_container in ('MED BAG', 'MED BOX', 'MED PKG', 'MED PACK') and l_quantity >= 10 and l_quantity <= 10 + 10 and p_size between 1 and 10 and l_shipmode in ('AIR', 'AIR REG') and l_shipinstruct = 'DELIVER IN PERSON' ) or ( p_partkey = l_partkey and p_brand = 'Brand#34' and p_container in ('LG CASE', 'LG BOX', 'LG PACK', 'LG PKG') and l_quantity >= 20 and l_quantity <= 20 + 10 and p_size between 1 and 15 and l_shipmode in ('AIR', 'AIR REG') and l_shipinstruct = 'DELIVER IN PERSON' ); --Q20 select s_name, s_address from supplier, nation where s_suppkey in ( select ps_suppkey from partsupp where ps_partkey in ( select p_partkey from part where p_name like 'forest%' ) and ps_availqty > ( select 0.5 * sum(l_quantity) from lineitem where l_partkey = ps_partkey and l_suppkey = ps_suppkey and l_shipdate >= date '1994-01-01' and l_shipdate < date '1994-01-01' + interval '1' year ) ) and s_nationkey = n_nationkey and n_name = 'CANADA' order by s_name; --Q21 select s_name, count(*) as numwait from supplier, lineitem l1, orders, nation where s_suppkey = l1.l_suppkey and o_orderkey = l1.l_orderkey and o_orderstatus = 'F' and l1.l_receiptdate > l1.l_commitdate and exists ( select * from lineitem l2 where l2.l_orderkey = l1.l_orderkey and l2.l_suppkey <> l1.l_suppkey ) and not exists ( select * from lineitem l3 where l3.l_orderkey = l1.l_orderkey and l3.l_suppkey <> l1.l_suppkey and l3.l_receiptdate > l3.l_commitdate ) and s_nationkey = n_nationkey and n_name = 'SAUDI ARABIA' group by s_name order by numwait DESC, s_name limit 100; --Q22 select cntrycode, count(*) as numcust, sum(c_acctbal) as totacctbal from ( select substring(c_phone, 1, 2) as cntrycode, c_acctbal from customer where substring(c_phone, 1, 2) in ('13', '31', '23', '29', '30', '18', '17') and c_acctbal > ( select avg(c_acctbal) from customer where c_acctbal > 0.00 and substring(c_phone, 1, 2) in ('13', '31', '23', '29', '30', '18', '17') ) and not exists ( select * from orders where o_custkey = c_custkey ) ) as custsale group by cntrycode order by cntrycode; ``` ##### 5.2 StarRocks Native table creation[​](#52-starrocks-native-table-creation "Direct link to 5.2 StarRocks Native table creation") ```sql #Create table customer drop table if exists customer; CREATE TABLE customer ( c_custkey int NOT NULL, c_name VARCHAR(25) NOT NULL, c_address VARCHAR(40) NOT NULL, c_nationkey int NOT NULL, c_phone VARCHAR(15) NOT NULL, c_acctbal decimal(15, 2) NOT NULL, c_mktsegment VARCHAR(10) NOT NULL, c_comment VARCHAR(117) NOT NULL )ENGINE=OLAP DUPLICATE KEY(`c_custkey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 24 PROPERTIES ( "replication_num" = "1" ); #Create table lineitem drop table if exists lineitem; CREATE TABLE lineitem ( l_shipdate DATE NOT NULL, l_orderkey int NOT NULL, l_linenumber int not null, l_partkey int NOT NULL, l_suppkey int not null, l_quantity decimal(15, 2) NOT NULL, l_extendedprice decimal(15, 2) NOT NULL, l_discount decimal(15, 2) NOT NULL, l_tax decimal(15, 2) NOT NULL, l_returnflag VARCHAR(1) NOT NULL, l_linestatus VARCHAR(1) NOT NULL, l_commitdate DATE NOT NULL, l_receiptdate DATE NOT NULL, l_shipinstruct VARCHAR(25) NOT NULL, l_shipmode VARCHAR(10) NOT NULL, l_comment VARCHAR(44) NOT NULL )ENGINE=OLAP DUPLICATE KEY(`l_shipdate`, `l_orderkey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`l_orderkey`) BUCKETS 96 PROPERTIES ( "replication_num" = "1", "colocate_with" = "tpch2" ); #Create table nation drop table if exists nation; CREATE TABLE `nation` ( `n_nationkey` int(11) NOT NULL, `n_name` varchar(25) NOT NULL, `n_regionkey` int(11) NOT NULL, `n_comment` varchar(152) NULL ) ENGINE=OLAP DUPLICATE KEY(`N_NATIONKEY`) COMMENT "OLAP" DISTRIBUTED BY HASH(`N_NATIONKEY`) BUCKETS 1 PROPERTIES ( "replication_num" = "3" ); #Create table orders drop table if exists orders; CREATE TABLE orders ( o_orderkey int NOT NULL, o_orderdate DATE NOT NULL, o_custkey int NOT NULL, o_orderstatus VARCHAR(1) NOT NULL, o_totalprice decimal(15, 2) NOT NULL, o_orderpriority VARCHAR(15) NOT NULL, o_clerk VARCHAR(15) NOT NULL, o_shippriority int NOT NULL, o_comment VARCHAR(79) NOT NULL )ENGINE=OLAP DUPLICATE KEY(`o_orderkey`, `o_orderdate`) COMMENT "OLAP" DISTRIBUTED BY HASH(`o_orderkey`) BUCKETS 96 PROPERTIES ( "replication_num" = "1", "colocate_with" = "tpch2" ); #Create table part drop table if exists part; CREATE TABLE part ( p_partkey int NOT NULL, p_name VARCHAR(55) NOT NULL, p_mfgr VARCHAR(25) NOT NULL, p_brand VARCHAR(10) NOT NULL, p_type VARCHAR(25) NOT NULL, p_size int NOT NULL, p_container VARCHAR(10) NOT NULL, p_retailprice decimal(15, 2) NOT NULL, p_comment VARCHAR(23) NOT NULL )ENGINE=OLAP DUPLICATE KEY(`p_partkey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`p_partkey`) BUCKETS 24 PROPERTIES ( "replication_num" = "1", "colocate_with" = "tpch2p" ); #Create table partsupp drop table if exists partsupp; CREATE TABLE partsupp ( ps_partkey int NOT NULL, ps_suppkey int NOT NULL, ps_availqty int NOT NULL, ps_supplycost decimal(15, 2) NOT NULL, ps_comment VARCHAR(199) NOT NULL )ENGINE=OLAP DUPLICATE KEY(`ps_partkey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`ps_partkey`) BUCKETS 24 PROPERTIES ( "replication_num" = "1", "colocate_with" = "tpch2p" ); #Create table region drop table if exists region; CREATE TABLE region ( r_regionkey int NOT NULL, r_name VARCHAR(25) NOT NULL, r_comment VARCHAR(152) )ENGINE=OLAP DUPLICATE KEY(`r_regionkey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`r_regionkey`) BUCKETS 1 PROPERTIES ( "replication_num" = "3" ); #Create table supplier drop table if exists supplier; CREATE TABLE supplier ( s_suppkey int NOT NULL, s_name VARCHAR(25) NOT NULL, s_address VARCHAR(40) NOT NULL, s_nationkey int NOT NULL, s_phone VARCHAR(15) NOT NULL, s_acctbal decimal(15, 2) NOT NULL, s_comment VARCHAR(101) NOT NULL )ENGINE=OLAP DUPLICATE KEY(`s_suppkey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`s_suppkey`) BUCKETS 12 PROPERTIES ( "replication_num" = "1" ); drop view if exists revenue0; create view revenue0 (supplier_no, total_revenue) as select l_suppkey, sum(l_extendedprice * (1 - l_discount)) from lineitem where l_shipdate >= date '1996-01-01' and l_shipdate < date '1996-01-01' + interval '3' month group by l_suppkey; ``` ##### 5.3 Hive external table creation (ORC storage format)[​](#53-hive-external-table-creation-orc-storage-format "Direct link to 5.3 Hive external table creation (ORC storage format)") ```sql create database tpch_hive_orc; use tpch_hive_orc; --Create table customer CREATE TABLE `customer`( `c_custkey` int, `c_name` varchar(25), `c_address` varchar(40), `c_nationkey` int, `c_phone` varchar(15), `c_acctbal` decimal(15,2), `c_mktsegment` varchar(10), `c_comment` varchar(117)) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/hive/warehouse/tpch_hive_orc.db/customer'; -- Create table lineitem CREATE TABLE `lineitem`( `l_orderkey` bigint, `l_partkey` int, `l_suppkey` int, `l_linenumber` int, `l_quantity` decimal(15,2), `l_extendedprice` decimal(15,2), `l_discount` decimal(15,2), `l_tax` decimal(15,2), `l_returnflag` varchar(1), `l_linestatus` varchar(1), `l_shipdate` date, `l_commitdate` date, `l_receiptdate` date, `l_shipinstruct` varchar(25), `l_shipmode` varchar(10), `l_comment` varchar(44)) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/hive/warehouse/tpch_hive_orc.db/lineitem'; -- Create table nation CREATE TABLE `nation`( `n_nationkey` int, `n_name` varchar(25), `n_regionkey` int, `n_comment` varchar(152)) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/hive/warehouse/tpch_hive_orc.db/nation'; -- Create table orders CREATE TABLE `orders`( `o_orderkey` bigint, `o_custkey` int, `o_orderstatus` varchar(1), `o_totalprice` decimal(15,2), `o_orderdate` date, `o_orderpriority` varchar(15), `o_clerk` varchar(15), `o_shippriority` int, `o_comment` varchar(79)) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/hive/warehouse/tpch_hive_orc.db/orders'; --Create table part CREATE TABLE `part`( `p_partkey` int, `p_name` varchar(55), `p_mfgr` varchar(25), `p_brand` varchar(10), `p_type` varchar(25), `p_size` int, `p_container` varchar(10), `p_retailprice` decimal(15,2), `p_comment` varchar(23)) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/hive/warehouse/tpch_hive_orc.db/part'; --Create table partsupp CREATE TABLE `partsupp`( `ps_partkey` int, `ps_suppkey` int, `ps_availqty` int, `ps_supplycost` decimal(15,2), `ps_comment` varchar(199)) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/hive/warehouse/tpch_hive_orc.db/partsupp'; --Create table region CREATE TABLE `region`( `r_regionkey` int, `r_name` varchar(25), `r_comment` varchar(152)) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/hive/warehouse/tpch_hive_orc.db/region'; --Create table supplier CREATE TABLE `supplier`( `s_suppkey` int, `s_name` varchar(25), `s_address` varchar(40), `s_nationkey` int, `s_phone` varchar(15), `s_acctbal` decimal(15,2), `s_comment` varchar(101)) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/hive/warehouse/tpch_hive_orc.db/supplier'; ``` ##### 5.4 Hive external table creation (CSV storage format)[​](#54-hive-external-table-creation-csv-storage-format "Direct link to 5.4 Hive external table creation (CSV storage format)") ```sql create database tpch_hive_csv; use tpch_hive_csv; --Create the customer external table. CREATE EXTERNAL TABLE `customer`( `c_custkey` int, `c_name` varchar(25), `c_address` varchar(40), `c_nationkey` int, `c_phone` varchar(15), `c_acctbal` double, `c_mktsegment` varchar(10), `c_comment` varchar(117)) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/tmp/csv/customer_csv'; --Create the lineitem external table. CREATE EXTERNAL TABLE `lineitem`( `l_orderkey` int, `l_partkey` int, `l_suppkey` int, `l_linenumber` int, `l_quantity` double, `l_extendedprice` double, `l_discount` double, `l_tax` double, `l_returnflag` varchar(1), `l_linestatus` varchar(1), `l_shipdate` date, `l_commitdate` date, `l_receiptdate` date, `l_shipinstruct` varchar(25), `l_shipmode` varchar(10), `l_comment` varchar(44)) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/tmp/csv/lineitem_csv'; --Create the nation external table. CREATE EXTERNAL TABLE `nation`( `n_nationkey` int, `n_name` varchar(25), `n_regionkey` int, `n_comment` varchar(152)) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/tmp/csv/nation_csv'; --Create the orders external table. CREATE EXTERNAL TABLE `orders`( `o_orderkey` int, `o_custkey` int, `o_orderstatus` varchar(1), `o_totalprice` double, `o_orderdate` date, `o_orderpriority` varchar(15), `o_clerk` varchar(15), `o_shippriority` int, `o_comment` varchar(79)) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/tmp/csv/orders_csv'; --Create the part external table. CREATE EXTERNAL TABLE `part`( `p_partkey` int, `p_name` varchar(55), `p_mfgr` varchar(25), `p_brand` varchar(10), `p_type` varchar(25), `p_size` int, `p_container` varchar(10), `p_retailprice` double, `p_comment` varchar(23)) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/tmp/csv/part_csv'; --Create the partsupp external table. CREATE EXTERNAL TABLE `partsupp`( `ps_partkey` int, `ps_suppkey` int, `ps_availqty` int, `ps_supplycost` double, `ps_comment` varchar(199)) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/tmp/csv/partsupp_csv'; --Create the region external table. CREATE EXTERNAL TABLE `region`( `r_regionkey` int, `r_name` varchar(25), `r_comment` varchar(152)) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/tmp/csv/region_csv'; --Create the supplier external table. CREATE EXTERNAL TABLE `supplier`( `s_suppkey` int, `s_name` varchar(25), `s_address` varchar(40), `s_nationkey` int, `s_phone` varchar(15), `s_acctbal` double, `s_comment` varchar(101)) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/tmp/csv/supplier_csv'; ``` --- ## Best_practices ### Best Practices #### [📄️ Overview](https://docs.starrocks.io/docs/best_practices/overview.md) [These best practices are written by experienced database engineers.](https://docs.starrocks.io/docs/best_practices/overview.md) #### [📄️ Partitioning](https://docs.starrocks.io/docs/best_practices/partitioning.md) [Fast analytics in StarRocks begin with a table layout that matches your query patterns.](https://docs.starrocks.io/docs/best_practices/partitioning.md) #### [📄️ Table clustering](https://docs.starrocks.io/docs/best_practices/table_clustering.md) [A thoughtful sort‑key is the highest‑leverage physical‑design knob in StarRocks.](https://docs.starrocks.io/docs/best_practices/table_clustering.md) #### [📄️ Bucketing](https://docs.starrocks.io/docs/best_practices/bucketing.md) [A concise field guide to choosing between Hash Bucketing and Random Bucketing in StarRocks, including their mechanics, trade‑offs, and recommended use cases.](https://docs.starrocks.io/docs/best_practices/bucketing.md) #### [📄️ Primary Key tables](https://docs.starrocks.io/docs/best_practices/primarykey_table.md) [The Primary Key table uses a new storage engine designed by StarRocks.](https://docs.starrocks.io/docs/best_practices/primarykey_table.md) #### [📄️ Authentication and Authorization](https://docs.starrocks.io/docs/best_practices/authentication_authorization.md) [For detailed instructions on each operation involved below, see links in See Also.](https://docs.starrocks.io/docs/best_practices/authentication_authorization.md) #### [📄️ Audit Log-based Resource Group Configuration](https://docs.starrocks.io/docs/best_practices/audit_log_resource_group.md) [How to use StarRocks audit logs to analyze query patterns and configure resource groups for workload isolation.](https://docs.starrocks.io/docs/best_practices/audit_log_resource_group.md) #### [🗃️ Query Tuning](https://docs.starrocks.io/docs/best_practices/query_tuning.md) [8 items](https://docs.starrocks.io/docs/best_practices/query_tuning.md) #### [📄️ FE / Coordinator Node memory full issue troubleshooting](https://docs.starrocks.io/docs/best_practices/fe_memory_troubleshooting.md) [Prevent 'FE memory full' issues, quickly recover, and find out the cause as much as possible to prevent recurrence.](https://docs.starrocks.io/docs/best_practices/fe_memory_troubleshooting.md) --- ### Audit Log-based Resource Group Configuration In StarRocks, **Resource Groups** provide an effective mechanism for resource isolation by allocating CPU, memory, and concurrency limits based on classifiers such as user identity and query type. This feature is essential for achieving efficient resource utilization in multi-tenant environments. Traditional resource group configuration often relies on empirical judgment. By analyzing historical query data from the audit log table `starrocks_audit_db__.starrocks_audit_tbl__`, administrators can instead adopt a **data-driven approach** to tuning resource groups. Key metrics such as CPU time, memory consumption, and query concurrency offer objective insights into actual workload characteristics. This approach helps: * Prevent query latency caused by resource contention * Protect the cluster from resource exhaustion * Improve overall stability and predictability This topic provides step-by-step tutorial on how to derive appropriate resource group parameters based on workload patterns observed from audit logs. note This tutorial is based on the analysis using AuditLoader plugin, which allows you to query audit logs using SQL statements directly within your cluster. For detailed instructions to install the plugin, see [AuditLoader](https://docs.starrocks.io/docs/administration/management/audit_loader.md). #### CPU Resource Allocation[​](#cpu-resource-allocation "Direct link to CPU Resource Allocation") ##### Objective[​](#objective "Direct link to Objective") Determine per-user CPU consumption and allocate CPU resources proportionally using `cpu_weight` or `exclusive_cpu_cores`. ##### Analysis[​](#analysis "Direct link to Analysis") The following SQL aggregates total CPU time per user (`cpuCostNs`) over the last 30 days, converts it to seconds, and calculates the percentage of total CPU usage. ```sql SELECT user, SUM(cpuCostNs) / 1e9 AS total_cpu_seconds, -- Query the total CPU time. ( SUM(cpuCostNs) / ( SELECT SUM(cpuCostNs) FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE state IN ('EOF','OK') AND timestamp >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) ) ) * 100 AS cpu_usage_percentage -- Calculate the percentage of total CPU usage per user. FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE state IN ('EOF','OK') -- Include queries that are finished only. AND timestamp >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) -- Query the data of the last 30 days. GROUP BY user ORDER BY total_cpu_seconds DESC LIMIT 20; -- List the top 20 users with the most CPU resource consumption. ``` ##### Best Practices[​](#best-practices "Direct link to Best Practices") Assume a fixed number of CPU cores per BE (for example, 64 cores). If a user accounts for 16% (`cpu_usage_percentage`) of total CPU time, allocating approximately `64 × 16% ≈ 11 cores` is reasonable. You can configure the CPU limits for the resource group as follows: * `exclusive_cpu_cores`: * Its value must not exceed the total number of cores on a single BE. * The sum of `exclusive_cpu_cores` of all resource groups must not exceed the total number of cores on a single BE. * `cpu_weight`: * Applies only to **soft-isolation** resource groups. * Determines relative CPU share among competing queries on remaining cores. * Does **not** map directly to a fixed number of CPU cores. #### Memory Management[​](#memory-management "Direct link to Memory Management") ##### Objective[​](#objective-1 "Direct link to Objective") Identify memory-intensive users and define appropriate memory limits and circuit breakers. ##### Analysis[​](#analysis-1 "Direct link to Analysis") The following SQL computes the maximum memory usage per user (`memCostBytes`) for a single query over the last 30 days. ```sql SELECT user, MAX(memCostBytes) / (1024 * 1024) AS max_mem_mb -- Max memory usage (in MB) per query. FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE state IN ('EOF','OK') -- Include queries that are finished only. AND timestamp >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) -- Query the data of the last 30 days. GROUP BY user ORDER BY max_mem_mb DESC LIMIT 20; -- List the top 20 users with the most memory resource consumption. ``` ##### Best Practices[​](#best-practices-1 "Direct link to Best Practices") `max_mem_mb` represents **total memory usage across all BEs**. You can calculate the approximate per-BE memory usage as: `max_mem_mb / number_of_BEs`. You can configure the memory limits for the resource group as follows: * `big_query_mem_limit`: * Protects the cluster from anomalously large queries. * You can set it to a relatively high threshold to avoid false-positive query termination. * `mem_limit`: * In most cases, set it to a high value (for example, `0.9`). #### Concurrency Control[​](#concurrency-control "Direct link to Concurrency Control") ##### Objective[​](#objective-2 "Direct link to Objective") Identify peak query concurrency per user and define appropriate `concurrency_limit` values. ##### Analysis[​](#analysis-2 "Direct link to Analysis") The following SQL analyzes per-minute query concurrency over the last 30 days and extracts the maximum observed concurrency per user. ```sql WITH UserConcurrency AS ( SELECT user, DATE_FORMAT(timestamp, '%Y-%m-%d %H:%i') AS minute_bucket, COUNT(*) AS query_concurrency FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE state IN ('EOF', 'OK') -- Include queries that are finished only. AND timestamp >= DATE_SUB(NOW(), INTERVAL 30 DAY) -- Query the data of the last 30 days. AND LOWER(stmt) LIKE '%select%' -- Include SELECT statements only. GROUP BY user, minute_bucket HAVING query_concurrency > 1 -- Exclude scenarios where concurrency is less than one query per minute. ) SELECT user, minute_bucket, query_concurrency / 60.0 AS query_concurrency_per_second -- Query the per-second concurrency. FROM ( SELECT user, minute_bucket, query_concurrency, ROW_NUMBER() OVER ( PARTITION BY user ORDER BY query_concurrency DESC ) AS rn FROM UserConcurrency ) ranked WHERE rn = 1 -- Keep the highest record for each user. ORDER BY query_concurrency_per_second DESC LIMIT 50; -- List the top 50 users with the highest concurrency. ``` ##### Best Practices[​](#best-practices-2 "Direct link to Best Practices") The above analysis is performed at **minute granularity**. Actual per-second concurrency may be higher. You can configure the concurrency limits for the resource group as follows: * `concurrency_limit` * Set it to approximately **1.5× the observed peak** to provide headroom. * For users with extreme concurrency spikes, you can further enable **Query Queues** to smooth peak load and protect cluster stability. #### Resource Isolation for Asynchronous Materialized Views[​](#resource-isolation-for-asynchronous-materialized-views "Direct link to Resource Isolation for Asynchronous Materialized Views") ##### Objective[​](#objective-3 "Direct link to Objective") Prevent asynchronous materialized view refresh operations from impacting interactive queries. ##### Analysis[​](#analysis-3 "Direct link to Analysis") The following SQL identifies memory-intensive materialized view refresh operations, typically characterized by `INSERT OVERWRITE` statements. ```sql SELECT user, MAX(memCostBytes) / (1024 * 1024) AS max_mem_mb -- Max memory usage (in MB) per query. FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE state IN ('EOF','OK') -- Include queries that are finished only. AND timestamp >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) -- Query the data of the last 30 days. AND LOWER(stmt) LIKE '%insert overwrite%' -- Include materialized view refresh operations only. GROUP BY user ORDER BY max_mem_mb DESC LIMIT 20; -- List the top 20 users with the most memory resource consumption. ``` ##### Best Practices[​](#best-practices-3 "Direct link to Best Practices") StarRocks provides a system-defined resource group (`default_mv_wg`) for materialized view refresh tasks by default. However, customizing a dedicated resource group for materialized view refresh tasks is strongly recommended to enforce strict isolation and prevent materialized view refresh operations from degrading foreground query performance. For instructions on configuring resource group limits, see [Best Practice for CPU Resource Allocation](#best-practices) and [Best Practice for Memory Management](#best-practices-1). The following example only provides guidance on creating and assign a dedicated resource group to materialized view refresh tasks. 1. Create a dedicated resource group for materialized view refresh: ```sql CREATE RESOURCE GROUP rg_mv TO ( user = 'mv_user', query_type IN ('insert', 'select') ) WITH ( 'cpu_weight' = '32', 'mem_limit' = '0.9', 'concurrency_limit' = '10', 'spill_mem_limit_threshold' = '0.5' ); ``` 2. Assign the resource group to a materialized view. * Assign while creating a materialized view: ```sql CREATE MATERIALIZED VIEW mv_example REFRESH ASYNC PROPERTIES ( 'resource_group' = 'rg_mv' ) AS SELECT * FROM example_table; ``` * Assign to an existing materialized view: ```sql ALTER MATERIALIZED VIEW mv_example SET ("resource_group" = "rg_mv"); ``` #### See Also[​](#see-also "Direct link to See Also") * [Resource Group](https://docs.starrocks.io/docs/administration/management/resource_management/resource_group.md) * [Query Queues](https://docs.starrocks.io/docs/administration/management/resource_management/query_queues.md) --- ### Authentication and Authorization This topic aims to provide a coherent guide for best practices on developing your own authentication and authorization workflow. For detailed instructions on each operation involved below, see links in [See Also](#see-also). #### Real-World Enterprise Scenario[​](#real-world-enterprise-scenario "Direct link to Real-World Enterprise Scenario") Large enterprises often have complex organizational structures and a vast number of employees using diverse platforms and tools. From an IT governance perspective, having a unified identity, authentication, and authorization system brings significant advantages: * **Simplified User Management**: Admins no longer need to manually create or delete users and assign permissions across multiple systems. User lifecycle management (for example, onboarding/offboarding) becomes seamless and audit-friendly. * **Improved Security**: A single sign-on (SSO) mechanism eliminates the need for users to manage multiple credentials, reducing the attack surface. * **Role-Aligned Access Control**: Access permissions are typically tied to a user’s role or department. A well-structured identity system enables easier and more accurate authorization decisions. ##### Example[​](#example "Direct link to Example") Suppose that three new employees join different departments of a SaaS company: one Marketing Specialist and two Solution Architects. * **Organizationally**, they belong to different teams. * **From an identity standpoint**, their email accounts serve as their login credentials across internal platforms. * **By access rights**, each of the three is granted access to different platforms: * The Marketing Specialist can log in to Hubspot backend to view new leads. * Solution Architects can access the service console, and manage services for assigned customers. Although all three use the same identity provider, their access rights are strictly enforced: * The Marketing Specialist only has access to Hubspot. * Solution Architects can access the service console, but they cannot access the service for users they are not assigned to. They also cannot access Hubspot. #### Three Layers of Access Control[​](#three-layers-of-access-control "Direct link to Three Layers of Access Control") This example highlights the three key components in an enterprise identity and access flow: 1. **Identity Authentication** – “I am Peter, a verified employee of the SaaS company.” 2. **Access Authentication** – “As a Solution Architect, I am authorized to log in to the service console.” (Not all verified employees should have access to all services.) 3. **Action Authorization** – “As the customer of the SaaS company, I can view the information of our own service, but not other customer’s.” ##### In Database Context[​](#in-database-context "Direct link to In Database Context") These layers of access control also apply to the database system: 1. **Identity Verification**: Confirm the user is a valid employee with their own password. 2. **Access Authentication**: Verify the user or their group has permission to log in to a specific cluster. 3. **Operation Authorization**: Check if the user can run a query, load data, etc. As you can see, authentication and authorization are tightly coupled in practice. A user's authentication request often implies a broader access control requirement. Therefore, it is essential to understand the full access flow. #### Key Concepts[​](#key-concepts "Direct link to Key Concepts") ##### LDAP[​](#ldap "Direct link to LDAP") Lightweight Directory Access Protocol (LDAP) is a protocol for accessing and maintaining distributed directory information. You can think of it as your organization’s global address book: * Each user has a unique path (Distinguished Name, DN). * LDAP stores basic user information, including passwords. * LDAP also manages group structures and membership. * `ldapsearch` queries can retrieve users or groups. LDAP can be used: * As an **authentication** source (to validate usernames and passwords). * As a **group information** provider for access control. ##### UNIX Groups[​](#unix-groups "Direct link to UNIX Groups") Sometimes users mirror LDAP groups locally (on the host OS) for security or isolation reasons, avoiding direct communication with external LDAP servers. These local UNIX groups can be used for authentication or access control enforcement. ##### OAuth, OIDC, and JWT[​](#oauth-oidc-and-jwt "Direct link to OAuth, OIDC, and JWT") tip **Quick Explanation of Terms** * **ID Token**: Proof of identity (I am me.) * **Access Token**: Proof of permission to access certain resources (I can do certain things.) * **OAuth 2.0**: Authorization framework that provides access tokens. * **OIDC**: Authentication layer on top of OAuth. Provides ID and Access Tokens. * **JWT**: Token format. Used by both OAuth and OIDC. **Practical Use:** * **OAuth-based login**: Redirects to an external login page (for example, Google), then back to the cluster. Requires browser access and redirect URL setup in advance. * **JWT-based login**: The user passes a token directly to the cluster, which requires a public key or endpoint setup in advance. #### Features[​](#features "Direct link to Features") The system supports all three layers of access control: 1. **User Authentication** – “I am who I say I am.” 2. **Login Authorization** – “I am allowed to access this cluster.” (It depends on individual or group membership.) 3. **Operation Authorization** – “I can run this query or load this dataset.” (Authorization can be based on identity or group affiliation.) From v3.5 onward, StarRocks provides a modular, composable model to support various combinations of identity and access management components. *Feature Mapping* ![Authentication and Authorization](/assets/images/auth_feature-120a5a00c391862013cf840199fdc35b.png) From the feature's perspective: 1. **Authentication Provider** – Supported protocols: Native user, LDAP, OIDC, and OAuth 2.0. 2. **Group Provider** – Supported sources: LDAP, Operating System, and File-based Configuration. 3. **Authorization System** – Supported systems: Native RBAC & IBAC, and Apache Ranger. ##### Authentication[​](#authentication "Direct link to Authentication") Comparison of supported authentication modes: | Method | CREATE USER (Native user) | CREATE SECURITY INTEGRATION (Session-based dummy user) | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Description | Manually creates users in the cluster. You can associate them with external authentication systems. The user exists explicitly in the cluster. | Defines an external authentication integration. The cluster does not store any user information. You can optionally combine it with a Group Provider to define allowed users. | | Login Process | Users must be pre-created in the cluster. During login, the user is authenticated via StarRocks or via the configured external authentication system (for example, LDAP). Only pre-created users can log in. | Upon login, StarRocks authenticates the user using external identity systems. If succeeds, it creates a temporary, session-scoped "dummy user" internally. This user is discarded after the session ends. | | Authorization Process | Since users exist in the cluster, permissions can be assigned in advance using either the native authorization system or Apache Ranger. | Although users do not persist, you can predefine role-to-group mappings. When a user logs in, the system assigns roles based on their group, enabling RBAC. Apache Ranger can also be used in parallel. | | Pros & Cons, Use Cases | - **Pros**: Full flexibility—supports both native and external authorization systems.
- **Cons**: Requires manual effort for creating users, which can be cumbersome.
- **Recommended for**: Small user bases or cases where the cluster handles access control. | - **Pros**: Easy to set up; only requires external authentication configuration and allowed group definitions.
- **Recommended for**: Ideal for large user bases with role-group mappings. | These authentication modes can coexist. When a user attempts to log in: 1. The cluster first checks if the user exists as a native user and tries to authenticate accordingly. 2. If the user is not found, the cluster proceeds down the `authentication_chain` as defined in the configuration. This hybrid mode provides both flexibility and control, suitable for different organizational requirements. ###### Option 1: Create Native User with External Authentication System[​](#option-1-create-native-user-with-external-authentication-system "Direct link to Option 1: Create Native User with External Authentication System") For example, you can use the following syntax to create a native user with LDAP: ```sql CREATE USER IDENTIFIED WITH authentication_ldap_simple AS 'uid=tom,ou=company,dc=example,dc=com'; ``` Then, you can `GRANT` privileges or roles to the user, or delegate authorization to external systems like Apache Ranger. ###### Option 2: Use Security Integration with External Authentication System[​](#option-2-use-security-integration-with-external-authentication-system "Direct link to Option 2: Use Security Integration with External Authentication System") You can also create a security integration to allow access of your external authentication service to the cluster. ```sql CREATE SECURITY INTEGRATION PROPERTIES ( "type" = "authentication_ldap_simple", "authentication_ldap_simple_server_host" = "", "authentication_ldap_simple_server_port" = "", "authentication_ldap_simple_bind_base_dn" = "", "authentication_ldap_simple_user_search_attr" = "" "authentication_ldap_simple_bind_root_dn" = "", "authentication_ldap_simple_bind_root_pwd" = "", "authentication_ldap_simple_ssl_conn_allow_insecure" = "{true | false}", "authentication_ldap_simple_ssl_conn_trust_store_path" = "", "authentication_ldap_simple_ssl_conn_trust_store_pwd" = "", "comment" = "" ); ``` After that, you need to configure the FE parameter `authentication_chain` and enable the security integration for your cluster. ```sql ADMIN SET FRONTEND CONFIG ( "authentication_chain" = "[... ,]" ); ``` ##### Group Provider (Optional but Recommended)[​](#group-provider-optional-but-recommended "Direct link to Group Provider (Optional but Recommended)") Group information in the cluster is **decoupled** from both the authentication and authorization systems. It serves as a shared layer that can be independently configured and then used across both login control and access control. ###### How Groups Are Used[​](#how-groups-are-used "Direct link to How Groups Are Used") * **Authentication Stage** When used together with a security integration, group membership can define the scope of who is allowed to log in. Only users who pass authentication and belong to a specified group will be allowed to access the cluster. * **Authorization Stage** Group membership is automatically taken into account during authorization. If privileges are granted to a group, all users within that group will inherit the permissions during access checks. ###### Configuration Notes[​](#configuration-notes "Direct link to Configuration Notes") * When configuring a group provider, you should specify: * Groups used to define **who can log in** (login scope) * Groups used to define **who can access specific resources** (authorization) * **Important**: The user identity (for example, username or ID) returned by the group provider **must match** the identity used during authentication and authorization. Inconsistent identifiers will cause permission or login failures. ###### Example[​](#example-1 "Direct link to Example") The following example is based on LDAP. 1. Create a group provider. ```sql -- LDAP Group Provider CREATE GROUP PROVIDER PROPERTIES ( "type" = "ldap", ldap_info, ldap_search_group_arg, ldap_search_attr, [ldap_cache_attr] ) ldap_info ::= "ldap_conn_url" = "", "ldap_bind_root_dn" = "", "ldap_bind_root_pwd" = "", "ldap_bind_base_dn" = "", ["ldap_conn_timeout" = "",] ["ldap_conn_read_timeout" = ""] ldap_search_group_arg ::= { "ldap_group_dn" = "" | "ldap_group_filter" = "" }, "ldap_group_identifier_attr" = "" ldap_search_user_arg ::= "ldap_group_member_attr" = "", "ldap_user_search_attr" = "" ldap_cache_arg ::= "ldap_cache_refresh_interval" = "" ``` 2. Integrate the group provider with a security integration. ```sql ALTER SECURITY INTEGRATION SET ( "group_provider" = "", "permitted_groups" = "" ) ``` 3. Integrate the group provider with the authorization system. You can use either the native authorization or Apache Ranger. * Native authorization: Roles can be assigned to groups. On login, users are automatically assigned roles based on group membership. ```sql GRANT role TO EXTERNAL GROUP ``` * Apache Ranger: Once a user logs in, StarRocks passes group information to Ranger for policy evaluation. ##### Authorization[​](#authorization "Direct link to Authorization") StarRocks supports **both internal and external authorization mechanisms**, which can be used independently or in combination: * **Internal Authorization** StarRocks provides a built-in **RBAC (Role-Based Access Control)** and **IBAC (Identity-Based Access Control)** system. * **RBAC**: Assigns roles to users or groups, and grants privileges to those roles. * **IBAC**: Grants privileges directly to users. * **External Authorization** StarRocks integrates with **Apache Ranger** to support centralized and unified authorization management. Apache Ranger can be used either as an integral solution itself or together with StarRocks' native authorization system. * **Full Ranger Authorization** Both internal tables and external tables (for example, Hive) are authorized via Ranger. * Internal table permissions use the StarRocks plugin for Ranger. * External table permissions can be managed either via the StarRocks plugin or other external service plugins (for example, Hive plugin). * **Hybrid Authorization** * **Internal tables**: Authorized by StarRocks' native system (RBAC/IBAC). * **External tables**: Authorized via Ranger. External table permissions can still be managed either using the StarRocks plugin or through the appropriate external service (for example, Hive, HDFS). This flexibility allows organizations to gradually migrate to centralized authorization or maintain a hybrid model that fits their current infrastructure and security policies. #### Combo Solutions[​](#combo-solutions "Direct link to Combo Solutions") You can choose the solution based on how you want to finish your authentication and authorization workflow. ##### Solution 1: External authentication + External Authorization[​](#solution-1-external-authentication--external-authorization "Direct link to Solution 1: External authentication + External Authorization") You can fully leverage the external authentication and authorization systems to control login and access permissions for the cluster. The overall process is as follows: 1. Use a **security integration** to establish a connection with the external authentication system. 2. Configure the necessary group information for authentication and authorization within the **group provider**. 3. Define the group(s) allowed to log in to the cluster in the **security integration**. Users who belong to these groups will be granted login access. 4. Create a **StarRocks service** in **Apache Ranger** to manage access control for both internal and external tables. For external tables, you may also reuse existing services for authorization. 5. When a user submits a query, the system will send the user's identity along with their group memberships (as configured in the group provider) to Ranger for authorization. 6. If the authorization check passes, the system will proceed to execute the query. note You must ensure that user IDs and group names remain consistent across all integrated systems throughout this process. ![Authentication and Authorization - Solution-1](/assets/images/auth_solution_1-259aba959f1d16cf4683b8436c0020f9.png) ##### Solution 2: External Authentication (Native User) + Internal Authorization[​](#solution-2-external-authentication-native-user--internal-authorization "Direct link to Solution 2: External Authentication (Native User) + Internal Authorization") If you prefer to use the **built-in authorization system** while still relying on **external authentication**, you can follow this approach: 1. **Manually create users** and specify the external authentication method for each user. 2. After the user is created, use standard `GRANT` statements to assign roles or privileges. 3. Once authenticated, the user will be authorized based on the cluster's native permission system. tip While manually created users can still be integrated with a **group provider** and **Ranger**, this approach is more complex and less automated compared to using **security integration**. Therefore, it is **not a recommended best practice**. ##### Solution 3: External Authentication (External Identity) + Internal Authorization[​](#solution-3-external-authentication-external-identity--internal-authorization "Direct link to Solution 3: External Authentication (External Identity) + Internal Authorization") If you prefer to use **StarRocks' built-in authorization system** while still relying on **external authentication**, you can follow this approach: 1. Use a **security integration** to establish a connection with the external authentication system. 2. Configure the necessary group information for authentication and authorization within the **group provider**. 3. Define the group(s) allowed to log in to the StarRocks cluster in the **security integration**. Users who belong to these groups will be granted login access. 4. **Create the necessary roles** within StarRocks and **grant them to external groups**. 5. When a user attempts to log in, they must both pass authentication and belong to an authorized group. Upon successful login, StarRocks will automatically assign the appropriate roles based on group membership. 6. During query execution, StarRocks will enforce **internal RBAC-based authorization** as usual. 7. Additionally, you can combine **Ranger** with this solution. For example, use **StarRocks' native RBAC** for internal table authorization, and use **Ranger** for external table authorization. When performing authorization via Ranger, StarRocks will still pass the **user ID and corresponding group information** to Ranger for access control. ![Authentication and Authorization - Solution-3](/assets/images/auth_solution_3-69183034840fc0cb794b375326e08043.png) #### See also[​](#see-also "Direct link to See also") * **Authentication** * [Native Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/native_authentication.md) * [Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md) * [LDAP Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/ldap_authentication.md) * [OAuth 2.0 Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/oauth2_authentication.md) * [JSON Web Token Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/jwt_authentication.md) * [**Group Provider**](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md) * **Authorization** * [Native Authorization](https://docs.starrocks.io/docs/administration/user_privs/authorization/User_privilege.md) * [Apache Ranger Plugin](https://docs.starrocks.io/docs/administration/user_privs/authorization/ranger_plugin.md) --- ### Bucketing A concise field guide to choosing between Hash Bucketing and Random Bucketing in StarRocks, including their mechanics, trade‑offs, and recommended use cases. *** #### Quick‑Look Comparison[​](#quicklook-comparison "Direct link to Quick‑Look Comparison") | Aspect | Hash Bucketing | Random Bucketing | | ---------------------------------------- | ------------------------------------ | --------------------------------------------------- | | Example | `DISTRIBUTED BY HASH(id) BUCKETS 16` | `DISTRIBUTED BY RANDOM` | | Key declaration | Required HASH(col1, …) | None – rows assigned round‑robin | | Initial bucket count when omitted | Auto‑chosen at CREATE, then fixed | Auto‑chosen at CREATE; can grow if bucket\_size set | | Tablet split / shrink | Manual ALTER … BUCKETS | Automatic split ⇢ growth only (≥ v3.2) | | Skew resistance | Depends on key cardinality | High – uniform by design | | Bucket pruning | ✅ (filters, joins) | 🚫 (full tablet scan) | | Colocate joins | ✅ | 🚫 | | Local aggregation / bucket-shuffle joins | ✅ | 🚫 | | Supported table types | All | Duplicate Key tables only | *** #### Hash Bucketing[​](#hash-bucketing "Direct link to Hash Bucketing") ##### How it Works[​](#how-it-works "Direct link to How it Works") Rows are assigned to tablets by hashing one or more columns. Tablet count is fixed after creation unless manually altered. ##### Requirements[​](#requirements "Direct link to Requirements") * Must pick a stable, evenly, high‑cardinality key up front. The cardinality should typically be 1000 times more than the number of BE nodes to prevent data skew among hash buckets. * Choose an appropriate bucket size initially, ideally ranging between 1 to 10 GB. ##### Strengths[​](#strengths "Direct link to Strengths") * Query locality – selective filters and joins touch fewer tablets. * Colocate joins – fact/dim tables can share hash keys for high‑speed joins. * Predictable layout – rows with the same key always land together. * Local aggregation & bucket‑shuffle joins – identical hash layout across partitions enables local aggregation and reduces data shuffle costs for large join ##### Weaknesses[​](#weaknesses "Direct link to Weaknesses") * Vulnerable to hot tablets if data distribution skews. * Tablet count is static; scaling requires maintenance DDL. * Insufficient tablets can adversely affect data ingestion, data compaction, and query execution parallelism. * Excessive use of tablets will expand the metadata footprint. ##### Example: Dimension‑Fact Join and Tablet Pruning[​](#example-dimensionfact-join-and-tablet-pruning "Direct link to Example: Dimension‑Fact Join and Tablet Pruning") ```sql -- Fact table partitioned and hash‑bucketed by (customer_id) CREATE TABLE sales ( sale_id bigint, customer_id int, sale_date date, amount decimal(10,2) ) ENGINE = OLAP DISTRIBUTED BY HASH(customer_id) BUCKETS 48 PARTITION BY date_trunc('DAY', sale_date) PROPERTIES ("colocate_with" = "group1"); -- Dimension table hash‑bucketed on the same key and bucket count colocated with the sales table CREATE TABLE customers ( customer_id int, region varchar(32), status tinyint ) ENGINE = OLAP DISTRIBUTED BY HASH(customer_id) BUCKETS 48 PROPERTIES ("colocate_with" = "group1"); -- StarRocks can do tablet pruning SELECT sum(amount) FROM sales WHERE customer_id = 123 -- StarRocks can do local aggregation SELECT customer_id, sum(amount) AS total_amount FROM sales GROUP BY customer_id ORDER BY total_amount DESC LIMIT 100; -- StarRocks can do colocate join SELECT c.region, sum(s.amount) FROM sales s JOIN customers c USING (customer_id) WHERE s.sale_date BETWEEN '2025-01-01' AND '2025-01-31' GROUP BY c.region; ``` ###### What do you gain from this example?[​](#what-do-you-gain-from-this-example "Direct link to What do you gain from this example?") * **Tablet pruning**: The customer\_id predicate `WHERE customer_id = 123` enables bucket pruning, allowing the query to access only a single tablet, which lowers latency & CPU cycles, especially for point-lookups. * **Local aggregation**: when the hash distribution key is a subset of the aggregation key, StarRocks can bypass the shuffle aggregation phase, reducing the overall cost. * **Colocated join**: because both tables share bucket number and key, each BE can join its local pair of tablets without network shuffle. ##### When to Use[​](#when-to-use "Direct link to When to Use") * Stable schemas with well‑known distribution filter/join keys. * Data warehousing workloads that benefit from bucket pruning. * You need some specific optimization like colocate join/bucket shuffle join/local aggregation * You are using Aggregate/Primary Key tables. *** #### Random Bucketing[​](#random-bucketing "Direct link to Random Bucketing") ##### How it Works[​](#how-it-works-1 "Direct link to How it Works") Rows are assigned round‑robin; no key specified. With `PROPERTIES ("bucket_size"="")`, StarRocks dynamically splits tablets as partitions grow (v3.2+). ##### Strengths[​](#strengths-1 "Direct link to Strengths") * **Zero design debt**–no keys, no bucket math. * **Skew‑proof writes**–uniform pressure across disks & BEs. * **Elastic growth**–tablet splits keep ingest fast as data or cluster grows. ##### Weaknesses[​](#weaknesses-1 "Direct link to Weaknesses") * **No bucket pruning**–every query scans all tablets in a partition. * **No colocated joins**–keyless layout prevents locality. * Limited to **Duplicate Key** tables today. ##### When to Use[​](#when-to-use-1 "Direct link to When to Use") * Log/event or multi‑tenant SaaS tables where keys change or skew. * Write‑heavy pipelines where uniform ingest throughput is critical. *** #### Operational Guidelines[​](#operational-guidelines "Direct link to Operational Guidelines") * Pick a bucket size (e.g., 1 GiB) for random bucketing to enable auto‑split. * For hash bucketing, monitor tablet size; re‑shard before tablets exceed 5–10 GiB --- ### FE / Coordinator Node memory full issue troubleshooting Prevent "FE memory full" issues, quickly recover, and find out the cause as much as possible to prevent recurrence. note * This information is applicable to FE nodes and Coordinator nodes (shared-data deployments). For readability, we will use "FE" to refer to both types of frontend nodes. * This article mainly analyzes the situation and solutions of in-heap memory, without covering off-heap memory. If the OOM is caused by off-heap memory issues, consider reducing the JVM XMX configuration or expanding the machine's memory. #### FE Memory Structure and Allocation Recommendations[​](#fe-memory-structure-and-allocation-recommendations "Direct link to FE Memory Structure and Allocation Recommendations") ##### FE Memory Composition[​](#fe-memory-composition "Direct link to FE Memory Composition") The front-end service (FE) runs on the JVM, and its memory consumption is from the Java heap and from off-heap memory overhead. ###### Overview of JVM Memory Modules[​](#overview-of-jvm-memory-modules "Direct link to Overview of JVM Memory Modules") All memory allocated through the JVM during the FE process can be roughly divided into the following modules: | Module | Type | Description | | -------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------- | | **Java Heap** | Heap | Memory of Java object instances, such as Plan Cache, metadata, temporary objects, etc. | | **Class** | Off-heap | Structures such as metadata generated when loading a Class | | **Thread** | Off-heap | The memory occupied by threads is mainly the stack space of each thread (usually about 1MB per thread). | | **Code** | Off-heap | Memory occupied by JVM when compiling bytecode into native machine code | | **GC** | Off-heap | Internal working data structures of the garbage collector | | **Compiler** | Off-heap | The memory used by the HotSpot compiler may be similar to that of Code. | | **Other** | Off-heap | Including direct memory such as Direct ByteBuffer, Netty buffer, etc. | | **Symbol** | Off-heap | SymbolTable structures such as string constant pool, integer constant pool, etc. | | **Native Memory Tracking (NMT)** | Off-heap | NMT's own recorded memory usage occupies memory | | **Arena Chunk** | Off-heap | Temporary memory blocks (allocated using malloc) used internally by the JVM for temporary storage and reuse | | **Logging** | Off-heap | Memory occupied by logging systems (such as log4j) | | **Arguments** | Off-heap | Memory occupied by JVM startup parameter passing | | **Module** | Off-heap | Structures used in the Java Module System | Considering the operating characteristics of JVM and StarRocks, the memory of FE mainly consists of the following parts: 1. **JVM Heap Memory (Heap)** * Stores Java object instances, such as Plan Cache, metadata cache, etc. * The maximum size is set via the parameter `-Xmx` and is significantly affected by the garbage collection mechanism. 2. **Non-heap Memory (Metaspace)** * Metadata such as storage class definitions, constant pool, etc. * Java 8 and later replaced PermGen. 3. **Direct Memory** * Off-heap memory used by Netty, ByteBuffer, log buffer, RPC framework, etc. * Allocated by `sun.misc.Unsafe`, usually not controlled by GC. 4. **Thread Stack Memory** * Each Java thread will request a thread stack when created, with a default size of approximately 1MB (configurable via `-Xss`). * When the number of threads is large (such as in high-concurrency scenarios), it is easy to cause memory backlog. 5. **JNI / Native Memory Call** * The database may consume memory when calling the native layer through logging components (such as RocksDB, Netty), Cache, etc. ##### JVM Parameter Configuration Recommendations[​](#jvm-parameter-configuration-recommendations "Direct link to JVM Parameter Configuration Recommendations") In the JVM, the heap memory is configured via `Xmx`. Generally, it is reasonable for the memory used by the process to be less than 130% of `Xmx`. That is to say, in addition to the memory configured by JVM `Xmx`, a portion of off-heap memory will also be used. For example, if JVM is configured with 21g, the RSS memory usage of the process seen in `top` will be approximately 27g. Based on experience, the recommended relationship between the number of tablets and FE memory is: | Tablet Count | Recommended FE Memory Configuration | | ------------ | ----------------------------------- | | Less than 1M | 16 GB | | 1M - 2M | 32 GB | | 2M - 5M | 64 GB | | 5M - 10M | 128 GB | For independently deployed environments, it is recommended to reserve sufficient memory for the system. In addition to the memory reserved for the system, off-heap memory also needs to be considered. For hybrid deployment environments, it is necessary to reserve sufficient memory as much as possible based on the usage considerations of other services to prevent OOM. In the case of independent deployment of FE (mixed deployment is not recommended): 1. When the machine memory is less than 32G, `-Xmx` (Max Heap Size) should be configured to a maximum of 70% of the machine memory. 2. When the machine memory is greater than 32G, `-Xmx` (Max Heap Size) should be configured to a maximum of 80% of the machine memory. 3. Since the FE leader node needs to perform checkpoints, in general, the memory of the leader node will be about twice that of the follower node. In version 3.4+, the leader will select a follower node to perform the checkpoint, then download the follower's results and distribute them to other followers. The leader's memory usage will be close to that of the followers. See [PR #52103](https://github.com/StarRocks/starrocks/pull/52103). ##### FE JVM Monitoring and Memory Profile Deployment[​](#fe-jvm-monitoring-and-memory-profile-deployment "Direct link to FE JVM Monitoring and Memory Profile Deployment") For FE state, heap monitoring and GC monitoring can effectively monitor the state of the service and are also key clues for troubleshooting issues afterwards. It is recommended to configure monitoring in a timely manner, and configure corresponding alerts if necessary. The specific monitoring indicators are as follows: | Indicator | Description | | -------------------------- | ---------------------------------------------------------------------------------- | | `jvm_heap_size_bytes` | Heap memory usage. | | `jvm_non_heap_size_bytes` | Mainly includes metaspace, code cache, etc., does not include all off-heap memory. | | `jvm_old_gc{type="count"}` | Full GC count. | | `jvm_old_gc{type="time"}` | Full GC time. | Monitoring information for these metrics and others is at [Monitoring and alerting](https://docs.starrocks.io/docs/administration/management/monitoring.md) Memory profile can analyze sudden heap increases. It is necessary to confirm whether it works properly after the service installation is completed. In 3.3.6+, the database will periodically print the memory profile and output it to `log/proc_profile`, in HTML format, compressed via tgz. Control configuration: ```properties proc_profile_collect_interval_s = 600 proc_profile_collect_time_s = 300 proc_profile_cpu_enable = true proc_profile_mem_enable = true ``` For versions prior to 3.3.6, you can periodically print the profile via a script. Create a new script file named `mem_profiler.sh` under the installation directory of each FE and copy the following content into the file. note This script has no impact on the FE process, does not affect the normal use of the service, and will automatically clean up files that have expired for 48 hours. ```bash #!/bin/bash mkdir -p mem_alloc_log cleanup_old_files() { find mem_alloc_log -name "alloc-profile-*.html" -mmin +2880 -exec rm -f {} \; } while true do cleanup_old_files current_time=$(date +'%Y-%m-%d-%H-%M-%S') file_name="mem_alloc_log/alloc-profile-${current_time}.html" ./bin/profiler.sh -e alloc --alloc 2m -d 300 -f "$file_name" `cat bin/fe.pid` done ``` ```bash # Background startup chmod +x mem_profiler.sh nohup ./mem_profiler.sh > mem_profiler.out 2>&1 & # Check if the process exists ps aux | grep mem_profiler.sh # Stop the process pkill -f mem_profiler.sh ``` #### FE Crash[​](#fe-crash "Direct link to FE Crash") The causes of FE crashes are roughly divided into 4 types. ##### 1. The FE process has high memory usage and was killed by the operating system[​](#1-the-fe-process-has-high-memory-usage-and-was-killed-by-the-operating-system "Direct link to 1. The FE process has high memory usage and was killed by the operating system") You can check whether this is the cause by querying the system logs: ```bash # Check dmesg dmesg | grep -iE 'out of memory|oom|kill|killed process' # View message log sudo grep -Ei 'killed process|oom.kill|sending SIG' -C5 /var/log/messages /var/log/syslog 2>/dev/null | tail -n 100 ``` After being OOM killed, the log may print the memory usage of system processes. Fields such as `total_vm` and `rss` displayed by OOM are in units of pages, with the default being 1 page = 4KB: * `total_vm = 8793525` → indicates that it has requested approximately 8793525 × 4KB ≈ 33.5 GB of virtual memory * `rss = 7218120` → indicates that approximately 27.5 GB of physical memory has been actually used, which is 7218120 × 4KB Example OOM log: ```text Jun 10 11:20:24 tp-prod-bigdata-sr-ss-fe-2-b kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jun 10 11:21:58 tp-prod-bigdata-sr-ss-fe-2-b kernel: [22654] 1005 22654 8793525 7218120 14943 0 0 java ``` Check the actual RSS usage. If it is very close to the machine's memory, it is generally due to unreasonable JVM allocation. Refer to the [JVM Parameter Configuration Recommendations](#jvm-parameter-configuration-recommendations) section. ##### 2. The FE leader heap memory is high, triggering a Full GC, leading to a leader switch[​](#2-the-fe-leader-heap-memory-is-high-triggering-a-full-gc-leading-to-a-leader-switch "Direct link to 2. The FE leader heap memory is high, triggering a Full GC, leading to a leader switch") When the old leader exits due to a leader switch, the following information is printed in `fe.out`: ```text transfer FE type from LEADER to UNKNOWN. exit ``` or ```text transfer FE type from LEADER to FOLLOWER. exit ``` Most leader switches are caused by Full GC; in a few cases, they are triggered by high leader CPU usage or the execution of `jstack`. **Analyzing GC logs:** Upload the GC log to and click **Pause GC Duration**. If the duration of a single pause GC exceeds 30 seconds, or if there are frequent pause GCs exceeding 10 seconds, it will trigger a leader switch. Click **Heap before GC** to find the time point when heap memory rises. Use this time range to locate the memory profile file for that period, which will identify which module was requesting memory at that time. ##### 3. An internal bug in the FE program caused a crash[​](#3-an-internal-bug-in-the-fe-program-caused-a-crash "Direct link to 3. An internal bug in the FE program caused a crash") This situation is not within the scope of this discussion. Refer to the [Restore Metadata](https://docs.starrocks.io/docs/administration/Meta_recovery.md) documentation. ##### 4. JVM bug causes FE crash[​](#4-jvm-bug-causes-fe-crash "Direct link to 4. JVM bug causes FE crash") This situation is very rare. After encountering it, the cause can be investigated by checking the crash log, located at `log/hs_err_pid%p.log`, where `%p` is the process ID. #### FE Process Is Normal But Memory Is High[​](#fe-process-is-normal-but-memory-is-high "Direct link to FE Process Is Normal But Memory Is High") During the long-term operation of the FE, the in-heap memory usage may abnormally increase, frequently triggering Full GC, which causes queries or imports to slow down. The problem may manifest as heap **sudden increase** or continuous **slow growth**. ##### Process of On-site Investigation[​](#process-of-on-site-investigation "Direct link to Process of On-site Investigation") Open your monitoring tool: [Monitoring and alerting](https://docs.starrocks.io/docs/administration/management/monitoring.md) **Step 1: Check the following monitoring** * **FE JVM Panel**: JVM Heap Memory Used metric **Step 2: Identify the type of memory increase** | Type | Feature | Subsequent Troubleshooting Path | | ------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------ | | **Sudden increase** | Heap memory spikes sharply within a short period (seconds to minutes) | Prioritize looking at QPS, number of connections, and SQL requests | | **Slowly rises** | Memory steadily increases, and post-GC recovery deteriorates. | Prioritize troubleshooting memory leaks or cache accumulation | ##### Memory Spike Investigation[​](#memory-spike-investigation "Direct link to Memory Spike Investigation") Whether the request pressure increases within a short period — high QPS or highly concurrent metadata operations can lead to a surge in memory. **1. Check the following monitoring:** * **QPS Panel**: Has the number of requests received by FE suddenly increased? * **Connections Panel**: Does the number of concurrent connections of FE spike instantaneously? **2. Confirm abnormal SQL through audit tables or logs:** Query high-frequency SQL statements in the last 5-10 minutes: ```sql SELECT * FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE feip = 'IP of memory abnormal node' AND `timestamp` >= now() - INTERVAL 1000 MINUTE ORDER BY `timestamp` DESC; ``` Pay special attention to: * Statements with deep subqueries, multiple JOINs, and complex aggregations * Unconventional business SQL (such as concurrent SQL for datasets generated by the BI platform) * In version 3.3.13+, you can check the FE audit log for the `queryFeMemory` field, which indicates how much memory a query has applied for in total in the FE. * When the FE process exits due to FE OOM, it will automatically print the query being executed and the corresponding `QueryFEAllocatedMemory`. Search keyword in FE log: ```text QueryFEAllocatedMemory ``` **3. Check through the FE log for a large number of metadata operations (table creation and deletion):** ```bash # Create a table grep "Begin to unprotect create table" fe.log # Drop a table grep "Finished drop table" fe.log ``` ##### Memory Slowly Increasing[​](#memory-slowly-increasing "Direct link to Memory Slowly Increasing") **4. The memory configured for the JVM is too small** Monitor the GC status of the FE process to determine whether Full GC occurs frequently. Check the JVM memory usage through the `jstat` command: ```bash jstat -gcutil $pid 1000 1000 ``` Example output: ```text S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 0.00 100.00 27.78 95.45 97.77 94.45 24 0.226 1 0.065 0.291 0.00 100.00 44.44 95.45 97.77 94.45 24 0.226 1 0.065 0.291 0.00 100.00 55.56 95.45 97.77 94.45 24 0.226 1 0.065 0.291 ``` If the `O` (Old Generation) percentage has been relatively high, it indicates a problem with the JVM memory configuration, and the JVM memory needs to be increased. **5. Observe whether there is a leak through the FE's memory tracker (3.3.7+)** You can locate which module has a memory leak by checking the logs of `MemoryUsageTracker`, which prints the memory consumption of each module once every hour: ```text 2025-06-06 16:37:50.633+08:00 INFO (MemoryUsageTracker|74) [MemoryUsageTracker.trackMemory():161] (6ms) Module Dict - CacheDictManager estimated 0B of memory. Contains ColumnDict with 0 object(s). 2025-06-06 16:37:50.657+08:00 INFO (MemoryUsageTracker|74) [MemoryUsageTracker.trackMemory():161] (21ms) Module LocalMetastore - LocalMetastore estimated 3.8MB of memory. Contains Partition with 17473 object(s). 2025-06-06 16:37:50.667+08:00 INFO (MemoryUsageTracker|74) [MemoryUsageTracker.trackMemory():161] (0ms) Module TabletInvertedIndex - TabletInvertedIndex estimated 37.7MB of memory. Contains TabletMeta with 353459 object(s). 2025-06-06 16:37:50.741+08:00 INFO (MemoryUsageTracker|74) [MemoryUsageTracker.trackMemory():108] total tracked memory: 46.8MB, jvm: Process used: 18.6GB, heap used: 4.4GB, non heap used: 289.1MB, direct buffer used: 395.5MB ``` Compare the memory growth of each module across recent log entries to identify which module is growing. ##### On-site Information Collection[​](#on-site-information-collection "Direct link to On-site Information Collection") **If the process has not been restarted yet, collect `jmap` information immediately.** ###### jmap Memory Troubleshooting Process[​](#jmap-memory-troubleshooting-process "Direct link to jmap Memory Troubleshooting Process") **Step 1: Confirm the process PID of FE** ```bash ps aux | grep FE ``` **Step 2: Use jmap** Precautions * `jmap -dump` will pause the FE process for a short period of time. Use with caution when high online stability is required, and it is recommended to notify in advance. * `jmap -histo` generally has no impact on the process; GC triggering may take dozens of milliseconds. * Frequent use of `-dump` to export heap files is not recommended, as it incurs significant overhead. * `histo` is sufficient for preliminary judgment of large objects and does not necessarily require a dump. **`jmap -histo:live` (Use with caution in production environments)** Captures the current in-heap object distribution by forcing a GC. Using the `live` parameter may resolve high memory usage issues: ```bash jmap -histo:live > jmap_histo_$(date +%s).log ``` It is recommended to take two samples and compare the differences: ```bash jmap -histo:live > histo_1.log sleep 60 jmap -histo:live > histo_2.log ``` **`jmap -histo pid` (lightweight, does not trigger Full GC)** ```bash # Retrieves all objects within the current JVM heap, which may include # objects that have been cleaned up — will affect result analysis. jmap -histo > histo.txt ``` **Step 3: Analyze Top N Object Occupancy** View the top entries of the file: ```bash head -n 30 histo_2.log ``` Field explanation: | Column | Meaning | | ---------- | -------------- | | num | Class number | | #instances | Instance count | | #bytes | Total bytes | | class name | Object name | Compare `histo_1` and `histo_2` to identify which classes have significantly increased instances or memory. Example: Classes starting with `java` are utility classes referenced by business classes. Looking from top to bottom, related classes at the front account for a larger proportion. For example, `com.starrocks.lake.LakeTablet` occupying significant memory indicates excessive tablet usage. **Step 4: Export a complete heap dump (if histo is insufficient)** ```bash # Will trigger Full GC. Use with caution. jmap -dump:live,format=b,file=heap_$(date +%s).hprof ``` This file may be very large (several GB) and needs to be opened and analyzed using **MAT** or **VisualVM**. **Step 5: Configure automatic dump on OOM** Add the following to the JVM configuration in `fe.conf` to automatically generate a dump file when FE runs out of memory: ```bash JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true \ -Xmx8192m \ -XX:+UseG1GC \ -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time \ -XX:ErrorFile=${LOG_DIR}/hs_err_pid%p.log \ -XX:+HeapDumpOnOutOfMemoryError \ -XX:HeapDumpPath=${LOG_DIR}/heap_dump_oom.hprof \ -Djava.security.policy=${STARROCKS_HOME}/conf/udf_security.policy \ -Djava.security.krb5.conf=/etc/krb5.conf \ -Dsun.security.krb5.debug=true \ -Dsun.security.spnego.debug=true \ -Dcom.sun.management.jmxremote \ -Dcom.sun.management.jmxremote.authenticate=false \ -Dcom.sun.management.jmxremote.ssl=false \ -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8113" ``` Note: `HeapDumpPath` specifies a directory path, not a file name. ###### Keep Memory Profile Files for Analysis[​](#keep-memory-profile-files-for-analysis "Direct link to Keep Memory Profile Files for Analysis") Starting from version 3.3.6, memory profiles are output to the `log/proc_profile` directory, in HTML format, compressed using tgz. Find the file corresponding to the time of memory increase, decompress it, and open it in a browser to see the flame graph of memory allocation. The wider part of the stack represents more memory allocation. If no memory profile is printed during the period of memory increase, you can turn off the CPU profile and adjust the printing interval to 5 minutes: ```properties proc_profile_cpu_enable = false proc_profile_collect_interval_s = 300 ``` These are dynamic parameters that can be modified directly without restart: ```sql ADMIN SET FRONTEND CONFIG ("proc_profile_cpu_enable" = "false"); ADMIN SET FRONTEND CONFIG ("proc_profile_collect_interval_s" = "300"); ``` #### Emergency Recovery[​](#emergency-recovery "Direct link to Emergency Recovery") **1. Stop the FE service** **2. Adjust the heap memory configuration to a larger value and restart** Modify the `-Xmx` value in `fe.conf`: ```bash JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:+UseG1GC \ -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time \ -XX:ErrorFile=${LOG_DIR}/hs_err_pid%p.log \ -XX:+HeapDumpOnOutOfMemoryError \ -XX:HeapDumpPath=${LOG_DIR}/heap_dump_oom.hprof \ -Djava.security.policy=${STARROCKS_HOME}/conf/udf_security.policy \ -Dcom.sun.management.jmxremote.authenticate=false \ -Dcom.sun.management.jmxremote.ssl=false \ -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8113" ``` **3. If the machine itself has limited memory**, restarting may still result in OOM issues. Consider expanding the machine's memory, then increasing the JVM configuration before restarting. **4. If a historically corresponding PR is found in the case**, it is recommended to upgrade to the corresponding fixed version. #### Case Collection[​](#case-collection "Direct link to Case Collection") ##### Case 1: Frequent and large-scale metadata operations cause high FE memory usage[​](#case-1-frequent-and-large-scale-metadata-operations-cause-high-fe-memory-usage "Direct link to Case 1: Frequent and large-scale metadata operations cause high FE memory usage") **Background**: A client's business requires regular table creation and cleanup, with sequential operations creating dozens of tables per second and synchronous operations deleting them, resulting in the creation of hundreds of thousands of tables throughout the process. **Problem manifestation**: Heap memory and machine CPU continue to rise, and query response slows down. After checking the FE logs, a large number of table creation and deletion operations were found, which led to a sharp increase in service pressure. **Recovery Method**: Stop the table creation and deletion jobs, restart the service, and resume normal use. **Solution**: For this business scenario, when using `DROP TABLE`, `FORCE` must be added; otherwise, the metadata will continue to be occupied. At the same time, reduce the concurrency frequency of table creation. *** ##### Case 2: Complex concurrent SQL causes a sudden increase in FE memory[​](#case-2-complex-concurrent-sql-causes-a-sudden-increase-in-fe-memory "Direct link to Case 2: Complex concurrent SQL causes a sudden increase in FE memory") **Problem manifestation**: Heap memory suddenly increased, and scheduled import tasks were significantly delayed. QPS and the number of connections were within the normal range of variation. Review of audit logs and FE logs revealed many highly complex queries in concurrent scheduled SQL, which resulted in a large amount of memory being occupied during query plan parsing. **Recovery Method**: Expand the memory of the FE service — stop the machine to expand its memory, increase the JVM XMX configuration, and restart the service. **Solution**: Split complex SQL, observe the memory usage during peak job periods, and ensure it stays below 80%. *** ##### Case 3: MV refresh is too frequent, causing the leader FE memory to grow slowly and remain high[​](#case-3-mv-refresh-is-too-frequent-causing-the-leader-fe-memory-to-grow-slowly-and-remain-high "Direct link to Case 3: MV refresh is too frequent, causing the leader FE memory to grow slowly and remain high") **Background**: The user has MV refresh tasks, and many tasks are configured for minute or second-level refresh. **Problem manifestation**: FE leader memory continues to rise, and queries are laggy. Checking the FE log reveals many slow locks, causing occupied resources to fail to be released. Example slow lock log: ```text 2025-06-06 15:41:43.813+08:00 WARN (AutoStatistic|41) [LockManager.logSlowLockTrace():423] LockManager detects slow lock : {"owners":[{"id":13479085,"name":"starrocks-taskrun-pool-22075","type":"INTENTION_SHARED","heldFor":6935,"queryId":"ab976f93-42a9-11f0-98e6-fa163e3710f8","waitTime":0,"stack":["java.base@11.0.20.1/java.net.SocketInputStream.socketRead0(Native Method)","java.base@11.0.20.1/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)","java.base@11.0.20.1/java.net.SocketInputStream.read(SocketInputStream.java:168)","java.base@11.0.20.1/java.net.SocketInputStream.read(SocketInputStream.java:140)","app//org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161)","app//org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128)","app//org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113)","app//org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)","app//org.postgresql.core.PGStream.receiveChar(PGStream.java:453)","app//org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2120)","app//org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356)","app//org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496)","app//org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413)","app//org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:333)","app//org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:319)","app//org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:295)","app//org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:244)","app//org.postgresql.jdbc.PgDatabaseMetaData.getColumns(PgDatabaseMetaData.java:1584)","app//com.starrocks.connector.jdbc.PostgresSchemaResolver.getColumns(PostgresSchemaResolver.java:49)","app//com.starrocks.connector.jdbc.JDBCMetadata.lambda$getTable$1(JDBCMetadata.java:200)","app//com.starrocks.connector.jdbc.JDBCMetadata$$Lambda$1340/0x000014e17aa20960.apply(Unknown Source)","app//com.starrocks.connector.jdbc.JDBCMetaCache.get(JDBCMetaCache.java:73)","app//com.starrocks.connector.jdbc.JDBCMetadata.getTable(JDBCMetadata.java:197)","app//com.starrocks.connector.CatalogConnectorMetadata.getTable(CatalogConnectorMetadata.java:136)","app//com.starrocks.server.MetadataMgr.lambda$getTable$5(MetadataMgr.java:501)","app//com.starrocks.server.MetadataMgr$$Lambda$513/0x000014e2116f0cb0.apply(Unknown Source)","java.base@11.0.20.1/java.util.Optional.map(Optional.java:265)","app//com.starrocks.server.MetadataMgr.getTable(MetadataMgr.java:501)","app//com.starrocks.sql.analyzer.QueryAnalyzer.resolveTable(QueryAnalyzer.java:1391)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.resolveTableRef(QueryAnalyzer.java:482)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.resolveTableRef(QueryAnalyzer.java:420)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.resolveTableRef(QueryAnalyzer.java:420)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:363)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.SelectRelation.accept(SelectRelation.java:232)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryRelation(QueryAnalyzer.java:303)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:293)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.QueryStatement.accept(QueryStatement.java:70)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSubquery(QueryAnalyzer.java:907)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSubquery(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.SubqueryRelation.accept(SubqueryRelation.java:66)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:370)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.SelectRelation.accept(SelectRelation.java:232)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryRelation(QueryAnalyzer.java:303)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:293)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.QueryStatement.accept(QueryStatement.java:70)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSubquery(QueryAnalyzer.java:907)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSubquery(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.SubqueryRelation.accept(SubqueryRelation.java:66)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitJoin(QueryAnalyzer.java:739)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitJoin(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.JoinRelation.accept(JoinRelation.java:134)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:370)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.SelectRelation.accept(SelectRelation.java:232)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryRelation(QueryAnalyzer.java:303)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:293)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.QueryStatement.accept(QueryStatement.java:70)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer.analyze(QueryAnalyzer.java:121)","app//com.starrocks.sql.analyzer.InsertAnalyzer.analyzeWithDeferredLock(InsertAnalyzer.java:74)","app//com.starrocks.sql.analyzer.InsertAnalyzer.analyze(InsertAnalyzer.java:62)","app//com.starrocks.sql.analyzer.Analyzer$AnalyzerVisitor.visitInsertStatement(Analyzer.java:398)","app//com.starrocks.sql.analyzer.Analyzer$AnalyzerVisitor.visitInsertStatement(Analyzer.java:176)","app//com.starrocks.sql.ast.InsertStmt.accept(InsertStmt.java:304)","app//com.starrocks.sql.ast.AstVisitor.visit(AstVisitor.java:80)","app//com.starrocks.sql.analyzer.Analyzer.analyze(Analyzer.java:173)","app//com.starrocks.sql.StatementPlanner.analyzeStatement(StatementPlanner.java:218)","app//com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:116)","app//com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:95)","app//com.starrocks.load.InsertOverwriteJobRunner.executeInsert(InsertOverwriteJobRunner.java:351)","app//com.starrocks.load.InsertOverwriteJobRunner.doLoad(InsertOverwriteJobRunner.java:171)","app//com.starrocks.load.InsertOverwriteJobRunner.handle(InsertOverwriteJobRunner.java:151)","app//com.starrocks.load.InsertOverwriteJobRunner.transferTo(InsertOverwriteJobRunner.java:212)","app//com.starrocks.load.InsertOverwriteJobRunner.prepare(InsertOverwriteJobRunner.java:256)","app//com.starrocks.load.InsertOverwriteJobRunner.handle(InsertOverwriteJobRunner.java:148)","app//com.starrocks.load.InsertOverwriteJobRunner.run(InsertOverwriteJobRunner.java:136)","app//com.starrocks.load.InsertOverwriteJobMgr.executeJob(InsertOverwriteJobMgr.java:91)","app//com.starrocks.qe.StmtExecutor.handleInsertOverwrite(StmtExecutor.java:2152)","app//com.starrocks.qe.StmtExecutor.handleDMLStmt(StmtExecutor.java:2244)","app//com.starrocks.qe.StmtExecutor.handleDMLStmtWithProfile(StmtExecutor.java:2161)","app//com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.refreshMaterializedView(PartitionBasedMvRefreshProcessor.java:1271)","app//com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.doRefreshMaterializedView(PartitionBasedMvRefreshProcessor.java:464)","app//com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.doRefreshMaterializedViewWithRetry(PartitionBasedMvRefreshProcessor.java:373)","app//com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.doMvRefresh(PartitionBasedMvRefreshProcessor.java:332)","app//com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.processTaskRun(PartitionBasedMvRefreshProcessor.java:200)","app//com.starrocks.scheduler.TaskRun.executeTaskRun(TaskRun.java:285)","app//com.starrocks.scheduler.TaskRunExecutor.lambda$executeTaskRun$0(TaskRunExecutor.java:60)","app//com.starrocks.scheduler.TaskRunExecutor$$Lambda$3311/0x000014db15a06cb0.get(Unknown Source)","java.base@11.0.20.1/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)","java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)","java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)","java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)"]}],"waiter":[{"id":13446845,"name":"thrift-server-pool-367701","type":"WRITE","waitTime":6895},{"id":5268046,"name":"lake-publish-task-8973","type":"INTENTION_EXCLUSIVE","waitTime":6895},{"id":5268447,"name":"lake-publish-task-9181","type":"INTENTION_EXCLUSIVE","waitTime":6874},{"id":5267898,"name":"lake-publish-task-8911","type":"INTENTION_EXCLUSIVE","waitTime":6874},{"id":5267726,"name":"lake-publish-task-8827","type":"INTENTION_EXCLUSIVE","waitTime":6874},{"id":13494208,"name":"thrift-server-pool-369089","type":"INTENTION_EXCLUSIVE","waitTime":6857},{"id":13492395,"name":"starrocks-taskrun-pool-22096","type":"INTENTION_EXCLUSIVE","waitTime":6854},{"id":41,"name":"AutoStatistic","type":"INTENTION_SHARED","waitTime":6854,"queryId":"ad195549-42a9-11f0-98e6-fa163e3710f8"},{"id":486,"name":"nioEventLoopGroup-6-33","type":"INTENTION_SHARED","waitTime":6852},{"id":5267924,"name":"lake-publish-task-8925","type":"INTENTION_SHARED","waitTime":6849},{"id":5271934,"name":"lake-publish-task-9265","type":"INTENTION_SHARED","waitTime":6849},{"id":5344442,"name":"lake-publish-task-9312","type":"INTENTION_EXCLUSIVE","waitTime":6786},{"id":13471238,"name":"thrift-server-pool-368401","type":"INTENTION_SHARED","waitTime":55},{"id":13494164,"name":"thrift-server-pool-369065","type":"INTENTION_SHARED","waitTime":55},{"id":13481141,"name":"thrift-server-pool-368676","type":"INTENTION_SHARED","waitTime":54}]} ``` **Recovery Method**: After leader GC, it switches to the primary and self-recovers. **Solution**: Relax the refresh frequency. Adjust non-real-time strongly dependent scenarios to hourly or even daily levels to avoid a large number of minute-level MVs being triggered simultaneously. Once slow locks disappear, the problem is resolved. *** ##### Case 4: Querying external tables in the Iceberg catalog causes an increase in FE memory[​](#case-4-querying-external-tables-in-the-iceberg-catalog-causes-an-increase-in-fe-memory "Direct link to Case 4: Querying external tables in the Iceberg catalog causes an increase in FE memory") **Background**: User executes a query job on an Iceberg table. **Problem manifestation**: After submitting the SQL, the memory of the FE leader increased significantly, and CPU also continued to rise. Other query and import tasks reported errors. Checking the process status revealed that frequent GC was triggered, causing abnormal service status. **Recovery Method**: Restart the leader node and regularly manage the delete files of the corresponding Iceberg table. *** ##### Case 5: Insert memory leak issue causes abnormal increase in FE memory[​](#case-5-insert-memory-leak-issue-causes-abnormal-increase-in-fe-memory "Direct link to Case 5: Insert memory leak issue causes abnormal increase in FE memory") **Background**: During user operation, it was found that the FE memory abnormally increased, the process restarted, and the leader was not properly switched. The main task of the client cluster was import tasks. **Problem manifestation**: Export `jmap` for the leader node process for troubleshooting: ```bash jmap -histo > jmap.txt # This operation will not trigger Full GC ``` Found in the exported file: ```bash 14: 521358 154321968 com.starrocks.load.loadv2.InsertLoadJob ``` The number of `InsertLoadJob` instances is 520,000. When this number exceeds 10,000, it indicates that a memory leak has occurred. * \[Issue]\( * [Fix PR](https://github.com/StarRocks/starrocks/pull/53809) **Affected versions**: * 3.1.0 - 3.1.16 * 3.2.0 - 3.2.12 * 3.3.0 - 3.3.7 **Fixed versions**: 3.1.17+, 3.2.13+, 3.3.8+ **Recovery Method**: Upgrade the cluster version. *** ##### Case 6: async-profiler causes JVM crash, leading to FE crash[​](#case-6-async-profiler-causes-jvm-crash-leading-to-fe-crash "Direct link to Case 6: async-profiler causes JVM crash, leading to FE crash") **Problem manifestation**: FE process crashed or restarted. **Problem Troubleshooting**: When the process is abnormal and the file `hs_err_pid$pid.log` is generated in the log directory, the FE crash may be caused by a JVM crash. ```text --------------- S U M M A R Y ------------ ... --------------- T H R E A D --------------- Current thread: JavaThread "tablet scheduler" ... Native frames: V [libjvm.so] frame::entry_frame_is_first() const V [libjvm.so] forte_fill_call_trace_given_top(...) V [libjvm.so] AsyncGetCallTrace C [libasyncProfiler.so] Profiler::getJavaTraceAsync(...) C [libasyncProfiler.so] Profiler::recordSample(...) C [libasyncProfiler.so] PerfEvents::signalHandler(...) ``` The crash breakpoint `PerfEvents::signalHandler` is caused by `async-profiler`. The tool triggered a segmentation fault inside the JVM while attempting to obtain the Java call stack. In the enterprise version, this tool is deployed by default to periodically obtain process information. **Processing Method**: Stop `async-profiler` CPU information collection: ```sql ADMIN SET FRONTEND CONFIG ("proc_profile_cpu_enable" = "false"); ``` Also add the following to `fe.conf`: ```properties proc_profile_cpu_enable = false ``` *** ##### Case 7: OOM caused by insufficient JVM heap configuration[​](#case-7-oom-caused-by-insufficient-jvm-heap-configuration "Direct link to Case 7: OOM caused by insufficient JVM heap configuration") Under normal circumstances, with 3 FEs, if the leader node crashes due to insufficient memory, the remaining 2 FEs can elect a new leader and provide services. If the 2 follower nodes cannot communicate properly after the leader crashes, the FE that is about to be promoted to leader may also exit due to insufficient metadata replicas, resulting in 2 FEs crashing. **Problem Troubleshooting**: FE leader log shows: ```text java.lang.OutOfMemoryError: Java heap space 2025-08-09 08:22:30.926-06:00 WARN (thrift-server-pool-9443633|65019425) [TIOStreamTransport.close():153] Error closing output stream. java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) ~[?:?] at java.net.SocketOutputStream.write(SocketOutputStream.java:150) ~[?:?] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?] at java.io.FilterOutputStream.close(FilterOutputStream.java:182) ~[?:?] at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:151) ~[libthrift-0.20.0.jar:0.20.0] at org.apache.thrift.transport.TSocket.close(TSocket.java:238) ~[libthrift-0.20.0.jar:0.20.0] at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:326) ~[starrocks-fe.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] 2025-08-09 08:22:30.927-06:00 ERROR (thrift-server-accept|144) [SRTThreadPoolServer.execute():221] ExecutorService threw error: java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space 2025-08-09 08:22:30.931-06:00 WARN (starrocks-mysql-nio-pool-42252|65019486) [StmtExecutor.execute():595] New planner error: ... com.starrocks.sql.common.StarRocksPlannerException: StarRocks planner use long time 3000 ms in memo phase, This probably because 1. FE Full GC, 2. Hive external table fetch metadata took a long time, 3. The SQL is very complex. You could 1. adjust FE JVM config, 2. try query again, 3. enlarge new_planner_optimize_timeout session variable at com.starrocks.sql.optimizer.task.SeriallyTaskScheduler.executeTasks(SeriallyTaskScheduler.java:50) ~[starrocks-fe.jar:?] at com.starrocks.sql.optimizer.Optimizer.memoOptimize(Optimizer.java:900) ~[starrocks-fe.jar:?] at com.starrocks.sql.optimizer.Optimizer.optimizeByCost(Optimizer.java:272) ~[starrocks-fe.jar:?] at com.starrocks.sql.optimizer.Optimizer.optimize(Optimizer.java:196) ~[starrocks-fe.jar:?] at com.starrocks.sql.StatementPlanner.createQueryPlanWithReTry(StatementPlanner.java:348) ~[starrocks-fe.jar:?] at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:138) ~[starrocks-fe.jar:?] at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:95) ~[starrocks-fe.jar:?] at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:580) ~[starrocks-fe.jar:?] at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:389) ~[starrocks-fe.jar:?] at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:598) ~[starrocks-fe.jar:?] at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:936) ~[starrocks-fe.jar:?] at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] 2025-08-09 08:22:30.932-06:00 INFO (TaskCleaner|111) [TaskManager.dropTasks():392] drop tasks:[] 2025-08-09 08:22:35.431-06:00 INFO (main|1) [StarRocksFE.start():123] StarRocks FE starting, version: 3.3.14-ee-5b29ea9 ``` The FE being promoted to leader fails: ```text at com.starrocks.common.util.Daemon.run(Daemon.java:109) ~[starrocks-fe.jar:?] 2025-08-09 08:16:07.844-06:00 ERROR (stateChangeExecutor|86) [GlobalStateMgr.transferToLeader():1333] failed to init journal after transfer to leader! will exit com.starrocks.journal.JournalException: catch exception after retried 3 times at com.starrocks.journal.bdbje.BDBJEJournal.open(BDBJEJournal.java:222) ~[starrocks-fe.jar:?] at com.starrocks.server.GlobalStateMgr.transferToLeader(GlobalStateMgr.java:1323) ~[starrocks-fe.jar:?] at com.starrocks.server.GlobalStateMgr$1.transferToLeader(GlobalStateMgr.java:795) ~[starrocks-fe.jar:?] at com.starrocks.ha.StateChangeExecutor.runOneCycle(StateChangeExecutor.java:125) ~[starrocks-fe.jar:?] at com.starrocks.common.util.Daemon.run(Daemon.java:109) ~[starrocks-fe.jar:?] Caused by: com.sleepycat.je.rep.InsufficientReplicasException: (JE 18.3.20) Commit policy: SIMPLE_MAJORITY required 1 replica. But none were active with this master. at com.sleepycat.je.rep.impl.node.DurabilityQuorum.ensureReplicasForCommit(DurabilityQuorum.java:116) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.rep.impl.RepImpl.txnBeginHook(RepImpl.java:1171) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.rep.txn.MasterTxn.txnBeginHook(MasterTxn.java:195) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.txn.Txn.initTxn(Txn.java:384) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.txn.Txn.(Txn.java:288) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.txn.Txn.(Txn.java:267) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.rep.txn.MasterTxn.(MasterTxn.java:146) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.rep.txn.MasterTxn$1.create(MasterTxn.java:117) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.rep.txn.MasterTxn.create(MasterTxn.java:435) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.rep.impl.RepImpl.createRepUserTxn(RepImpl.java:1145) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.txn.Txn.createAutoTxn(Txn.java:334) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.txn.LockerFactory.getWritableLocker(LockerFactory.java:79) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.Environment.setupDatabase(Environment.java:816) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.Environment.openDatabase(Environment.java:668) ~[starrocks-bdb-je-18.3.20.jar:?] at com.starrocks.journal.bdbje.BDBEnvironment.openDatabase(BDBEnvironment.java:446) ~[starrocks-fe.jar:?] at com.starrocks.journal.bdbje.BDBJEJournal.open(BDBJEJournal.java:213) ~[starrocks-fe.jar:?] ... 4 more 2025-08-09 08:16:07.845-06:00 INFO (Thread-69|145) [StarRocksFE.lambda$addShutdownHook$1():374] start to execute shutdown hook 2025-08-09 08:16:07.852-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:show frontends;, QueryFEAllocatedMemory: 69224 2025-08-09 08:16:07.853-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:show frontends;, QueryFEAllocatedMemory: 31128 2025-08-09 08:16:07.853-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:SET NAMES utf8, QueryFEAllocatedMemory: 64464 2025-08-09 08:16:07.853-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:SET NAMES utf8, QueryFEAllocatedMemory: 119040 2025-08-09 08:16:07.853-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:show frontends;, QueryFEAllocatedMemory: 83760 2025-08-09 08:16:07.853-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:show frontends;, QueryFEAllocatedMemory: 69080 2025-08-09 08:16:07.853-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:show frontends;, QueryFEAllocatedMemory: 83760 2025-08-09 08:16:07.853-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:show frontends;, QueryFEAllocatedMemory: 84072 2025-08-09 08:16:07.853-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:SET NAMES utf8, QueryFEAllocatedMemory: 10048 ``` **Processing Method**: Increase the FE JVM memory configuration, with a recommended minimum of 16GB in production, and adjust later based on the increment of metadata. For recovery, refer to steps 8 and 9 in the [Metadata Recovery](https://docs.starrocks.io/docs/administration/Meta_recovery.md) documentation. *** ##### Case 8: Slow FE response leads to increased time consumption for import tasks[​](#case-8-slow-fe-response-leads-to-increased-time-consumption-for-import-tasks "Direct link to Case 8: Slow FE response leads to increased time consumption for import tasks") **Problem Phenomenon**: The agent prints `jstack` when communicating with FE and encountering a situation where FE does not respond in a timely manner. The logs of the two follower FEs contain the printed message "notify new FE type". There are a large number of `jstack` collection records in the agent log. `jstack` collection exacerbated the unresponsive state duration of FE. **Processing Method**: Turn off `jstack` collection. Version 3.3+ has already turned it off by default. You can remove the `jstack` command in the cluster configuration. *** ##### Case 9: FE Memory Slowly Increasing (version 3.3.13 - 3.3.18)[​](#case-9-fe-memory-slowly-increasing-version-3313---3318 "Direct link to Case 9: FE Memory Slowly Increasing (version 3.3.13 - 3.3.18)") **Version scope**: This issue occurs between versions 3.3.13 and 3.3.18. **Problem Phenomenon**: If the FE heap memory is found to continuously and slowly increase, it may be caused by this issue. Obtaining `jmap` and checking reveals that `com.starrocks.catalog.Replica` occupies excessive memory. All operations involving large amounts of `DROP`, `SWAP`, and `INSERT OVERWRITE` may trigger this issue. This issue is due to the optimization of the tablet deletion path in version 3.3.13, which introduced a memory leak and was fixed in 3.3.18. * Optimization PR in 3.3.13: \[BugFix] Fix recycle bin missing to delete lake mv's expired partitions after mv refreshed * Fix PR in [3.3.18](https://github.com/StarRocks/starrocks/pull/61582) **Processing Method**: * Temporary workaround: Restart FE * Fundamental solution: Upgrade to a fixed version **Fixed versions**: 3.3.18, 3.4.7, 3.5.4 *** #### Appendix: Tool Usage Commands[​](#appendix-tool-usage-commands "Direct link to Appendix: Tool Usage Commands") ##### jstat — Monitor JVM GC Status[​](#jstat--monitor-jvm-gc-status "Direct link to jstat — Monitor JVM GC Status") ```bash jstat -gcutil 1000 10 ``` Prints the GC utilization of the FE process once every 1 second, continuously printing 10 times. **Common indicators:** | Indicator | Meaning | | --------- | --------------------------------------------- | | S0, S1 | Survivor space utilization (Young Generation) | | E | Eden space utilization (Young Generation) | | O | Old Generation utilization | | YGC, YGCT | Young GC count and time | | FGC, FGCT | Full GC count and time | **Purpose**: Quickly determine whether GC is frequent, whether the old generation is approaching full, and whether there is a Full GC. ##### jmap — View JVM Memory Object Distribution[​](#jmap--view-jvm-memory-object-distribution "Direct link to jmap — View JVM Memory Object Distribution") **Method 1: View objects in the current heap (including garbage objects)** ```bash jmap -histo | head -n 30 ``` Counts the number of instances and occupied size of all object types. Use this to check what the large objects in the heap are, such as `byte[]`, `String`, `HashMap`, etc. **Method 2: Check live objects after forcing GC** ```bash jmap -histo:live | head -n 30 ``` Triggers a Full GC, then only displays objects that still survive after the GC. Use this to troubleshoot issues such as leaked objects or cache not being released. --- ### Best Practices These best practices are written by experienced database engineers. Designing for efficiency does more than improve query speed, it decreases costs by reducing storage, CPU, and object storage (e.g., S3) API costs. #### General table design[​](#general-table-design "Direct link to General table design") Three guides covering: * [Partitioning](https://docs.starrocks.io/docs/best_practices/partitioning.md) * [Clustering](https://docs.starrocks.io/docs/best_practices/table_clustering.md) * [Bucketing](https://docs.starrocks.io/docs/best_practices/bucketing.md) Learn about: * The differences between partitioning and bucketing * When to partition * How to choose an efficient sort key * Choosing between hash and random bucketing #### Primary key tables[​](#primary-key-tables "Direct link to Primary key tables") The [Primary Key](https://docs.starrocks.io/docs/best_practices/primarykey_table.md) table uses a new storage engine designed by StarRocks. Its main advantage lies in supporting real-time data updates while ensuring efficient performance for complex ad-hoc queries. In real-time business analytics, decision-making can benefit from Primary Key tables, which use the newest data to analyze results in real-time, mitigating data latency in data analysis. Learn about: * Choosing the type of primary key index * Choosing the primary key * Monitoring and managing memory use * Tuning #### Query Tuning[​](#query-tuning "Direct link to Query Tuning") [Query tuning](https://docs.starrocks.io/docs/best_practices/query_tuning/query_plan_intro.md) is essential for achieving high performance and reliability in StarRocks. This directory brings together practical guides, reference materials, and actionable recipes to help you analyze, diagnose, and optimize query performance at every stage—from writing SQL to interpreting execution details. Effective query tuning in StarRocks typically follows a top-down process: 1. **Identify the Problem** 2. **Collect and Analyze Execution Information** 3. **Locate the Root Cause** 4. **Apply Tuning Strategies** 5. **Validate and Iterate** Whether you're a DBA, developer, or data engineer, these resources will help you: * Diagnose and resolve slow or resource-intensive queries * Understand optimizer choices and execution details * Apply best practices and advanced tuning strategies Start with the [overview](https://docs.starrocks.io/docs/best_practices/query_tuning/query_plan_intro.md), dive into the references as needed, and use the recipes and tips to solve real-world performance challenges in StarRocks. --- ### Partitioning Fast analytics in StarRocks begin with a table layout that matches your query patterns. This guide distills hands‑on experience into clear rules for **partitioning**, helping you: * **Scan less data** via aggressive partition pruning * **Manage lifecycle tasks** (TTL, GDPR deletes, tiering) with metadata‑only ops; * **Scale smoothly** as tenant counts, data volume, or retention windows grow. * **Controlled write amplification**–New data lands in the “hot” partition; compaction happens in historical partitions Keep this advice close when modeling a new table or refactoring an old one—each section gives purpose‑driven criteria, design heuristics, and operational guard‑rails so you avoid costly re‑partitioning down the road. #### Partitioning vs. Bucketing–different jobs[​](#partitioning-vs-bucketingdifferent-jobs "Direct link to Partitioning vs. Bucketing–different jobs") Understanding the distinction between partitioning and bucketing is fundamental when designing performant StarRocks tables. While both help manage large datasets, they serve different purposes: * **Partitioning** allows StarRocks to skip entire blocks of data at query time using partition pruning, and enables metadata-only lifecycle operations like dropping old or tenant-specific data. * **Bucketing**, on the other hand, helps distribute data across tablets to parallelize query execution and balance load, especially when combined with hash functions. | Aspect | Partitioning | Bucketing (Hash/Random) | | ---------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | **Primary goal** | Coarse‑grain data pruning and lifecycle control(TTL, archiving). | Fine‑grain parallelism and data locality inside each partition. | | **Planner visibility** | Partitions are catalog objects; FE can skip them via predicates. | Only equality predicates support bucket pruning | | **Lifecycle ops** | DROP PARTITION is metadata‑only—ideal for GDPR deletes, monthly roll‑off. | Buckets can’t be dropped; they change only with `ALTER TABLE … MODIFY DISTRIBUTED BY`. | | **Typical count** | 10^2–10^4 per table (days, weeks, tenants). | 10–120 per partition; StarRocks `BUCKETS xxx` tunes this. | | **Skew handling** | Merge or split partitions; consider composite/hybrid scheme. | Raise bucket count, hash on compound key, isolate “whales”, or use random bucketing | | **Red flags** | >100 k partitions can introduce significant memory footprint for FE | >200 k tablets per BE; tablets exceeding 10 GB may encounter compaction issues. | #### When should I partition?[​](#when-should-i-partition "Direct link to When should I partition?") | Table type | Partition? | Typical key | | ------------------------------ | ---------- | -------------------------------- | | Fact / event stream | Yes | `date_trunc('day', event_time)` | | Huge dimension (billions rows) | Sometimes | Time or business key change date | | Small dimension / lookup | No | Rely on hash distribution | #### Choosing the partition key[​](#choosing-the-partition-key "Direct link to Choosing the partition key") 1. **Time‑first default**–If 80 % of queries include a time filter, lead with `date_trunc('day', dt)`. 2. **Tenant isolation**–Add `tenant_id` into the key when you need to manage the data in tenant basis 3. **Retention alignment**–Put the column you plan to purge on into the key. 4. **Composite keys**: `PARTITION BY tenant_id, date_trunc('day', dt)` prunes perfectly but creates `#tenants × #days` partitions. Keep below ≈ 100 k total or FE memory & BE compaction suffer. #### Picking granularity[​](#picking-granularity "Direct link to Picking granularity") The granularity of `PARTITION BY date_trunc('day', dt)` should be adjusted based on the use case. You can use "hour," "day," or "month," etc. See [`date_trunc`](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_trunc.md) | Granularity | Use when | Pros | Cons | | ---------------- | -------------------------------- | ------------------------------------- | ----------------------------------- | | Daily (default) | Most BI & reporting | Few partitions (365/yr); simple TTL | Less precise for "last 3 h" queries | | Hourly | > 2 × tablet per day; IoT bursts | Hot‑spot isolation; 24 partitions/day | 8 700 partitions/yr | | Weekly / Monthly | Historical archive | Tiny metadata; merges easy | Coarser pruning | * **Rule of thumb**: Keep each partition ≤ 100 GB and ≤ 20 k tablets/partition across replicas. * **Mixed granularity**: Starting from version 3.4, StarRocks supports mixed granularity by merging historical partitions into coarser granularity. #### Example recipes[​](#example-recipes "Direct link to Example recipes") ##### Click‑stream fact (single‑tenant)[​](#clickstream-fact-singletenant "Direct link to Click‑stream fact (single‑tenant)") ```sql CREATE TABLE click_stream ( user_id BIGINT, event_time DATETIME, url STRING, ... ) DUPLICATE KEY(user_id, event_time) PARTITION BY date_trunc('day', event_time) DISTRIBUTED BY HASH(user_id) BUCKETS xxx; ``` ##### SaaS metrics (multi‑tenant, pattern A)[​](#saas-metrics-multitenant-pattern-a "Direct link to SaaS metrics (multi‑tenant, pattern A)") Recommended for most SaaS workloads. Prunes on time, keeps tenant rows co‑located. ```sql CREATE TABLE metrics ( tenant_id INT, dt DATETIME, metric_name STRING, v DOUBLE ) PRIMARY KEY(tenant_id, dt, metric_name) PARTITION BY date_trunc('DAY', dt) DISTRIBUTED BY HASH(tenant_id) BUCKETS xxx; ``` ##### Whale tenant composite (pattern B)[​](#whale-tenant-composite-pattern-b "Direct link to Whale tenant composite (pattern B)") When tenant-specific DML/DDL is necessary or large-scale tenants are present, be cautious of potential partition explosion. ```sql CREATE TABLE activity ( tenant_id INT, dt DATETIME, id BIGINT, .... ) DUPLICATE KEY(dt, id) PARTITION BY tenant_id, date_trunc('MONTH', dt) DISTRIBUTED BY HASH(id) BUCKETS xxx; ``` --- ### Primary Key tables The Primary Key table uses a new storage engine designed by StarRocks. Its main advantage lies in supporting real-time data updates while ensuring efficient performance for complex ad-hoc queries. In real-time business analytics, decision-making can benefit from Primary Key tables, which use the newest data to analyze results in real-time, which can mitigate data latency in data analysis. However, the primary key is not a free lunch. If used improperly, it can lead to unnecessary resource waste. Therefore, in this section, we will guide you on how to use the primary key model more efficiently to achieve the desired results. #### Choosing Primary Key Index[​](#choosing-primary-key-index "Direct link to Choosing Primary Key Index") The primary index is the most critical component in a Primary Key table. The primary key index which is used to store the mapping between the primary key values and the locations of the data rows identified by the primary key values. Currently, we support three types of primary key index: 1. Full in-memory primary key index. ```sql PROPERTIES ( "enable_persistent_index" = "false" ); ``` 2. Local disk based persistent primary key index. ```sql PROPERTIES ( "enable_persistent_index" = "true", "persistent_index_type" = "LOCAL" ); ``` 3. Cloud native persistent primary key index. ```sql PROPERTIES ( "enable_persistent_index" = "true", "persistent_index_type" = "CLOUD_NATIVE" ); ``` We DO NOT recommend using in-memory indexing, as it can lead to significant memory resource waste. If you are using a shared-data (elastic) StarRocks cluster, we recommend opting for the cloud-native persistent primary index. Unlike the local disk based persistent primary index, it stores the complete index data on remote object storage, with local disks serving only as a cache. Compared to the local disk-based persistent primary index, its advantages include: 1. No dependency on local disk capacity. 2. No need to rebuild indexes after data shard rebalance. #### Choosing the Primary key[​](#choosing-the-primary-key "Direct link to Choosing the Primary key") The primary key usually does not help accelerate queries. You can specify a column different from the primary key as the sort key using the `ORDER BY` clause to speed up queries. Therefore, when selecting a primary key, you only need to consider the uniqueness during data import and update processes. The larger the primary key, the more memory, I/O, and other resources it consumes. Therefore, it is generally recommended to avoid selecting too many or overly large columns as the primary key. The default maximum size for the primary key is 128 bytes, controlled by the `primary_key_limit_size` parameter in `be.conf`. You can increase `primary_key_limit_size` to select a larger primary key, but be aware that this will result in higher resource consumption. How much storage and memory space will a persistent index occupy? ##### Formula for storage space cost[​](#formula-for-storage-space-cost "Direct link to Formula for storage space cost") `(key size + 8 bytes) * row count * 50%` note 50% is the estimated compression efficiency, the actual compression effect depends on the data itself. ##### Formula for memory cost[​](#formula-for-memory-cost "Direct link to Formula for memory cost") `min(l0_max_mem_usage * tablet cnt, update_memory_limit_percent * BE process memory);` #### Memory usage[​](#memory-usage "Direct link to Memory usage") The memory used by Primary Key table can be monitored by mem\_tracker: ```text //View the overall memory statistics http://be_ip:be_http_port/mem_tracker // View primary key table memory statistics http://be_ip:be_http_port/mem_tracker?type=update // View primary key table memory statistics with more details http://be_ip:be_http_port/mem_tracker?type=update&upper_level=4 ``` `update` item in `mem_tracker` record whole memory used by Primary Key table, such as primary key index, delete vector and so on. You can also monitor this `update` item via metrics monitor service. For example, in Grafana, you can check update item via (item in the red box): ![grafana](/assets/images/primary_key-1-09d6340d0330093eb1406fbb8c887fde.png) tip More about [Monitoring and alerting with Prometheus and Grafana](https://docs.starrocks.io/docs/administration/management/monitoring.md) If you are sensitive to memory usage and want to reduce memory consumption during the import process of a PK table, you can achieve this through the following configuration: ```sh be.conf l0_max_mem_usage = (some value which smaller than 104857600, default is 104857600) // Shared-nothing cluster transaction_apply_worker_count = (some value smaller than cpu core number, default is cpu core number) // Shared-data cluster transaction_publish_version_worker_count = (some value smaller than cpu core number, default is cpu core number) ``` `l0_max_mem_usage` controls the max memory usage of persistent primary key index per tablet. `transaction_apply_worker_count` and `transaction_publish_version_worker_count` both control the max thread number which can be used for handling upsert and delete in primary key table. But you need to remember, reducing `l0_max_mem_usage` may increase I/O pressure, while decreasing `transaction_apply_worker_count` or `transaction_publish_version_worker_count` could slow down data ingestion. #### Tradeoff between compaction resource, data freshness and query latency[​](#tradeoff-between-compaction-resource-data-freshness-and-query-latency "Direct link to Tradeoff between compaction resource, data freshness and query latency") Compared to tables in other models, primary key tables require additional operations for primary key index lookups and delete vector generation during data import, updates, and deletions, which introduces extra resource overhead. Therefore, you need to make trade-offs among these three factors: * Compaction resources limitation. * Data freshness * Query latency ###### Data freshness & Query latency[​](#data-freshness--query-latency "Direct link to Data freshness & Query latency") If you want to get better data freshness and also better query latency, that means you will introduce high frequency writes, and also want to make sure they can be compacted as soon as possible. Then you will need more compaction resource to handle these writes: ```sh // shared-data be.conf compact_threads = 4 // shared-nothing be.conf update_compaction_num_threads_per_disk = 1 update_compaction_per_tablet_min_interval_seconds = 120 ``` You can increase `compact_threads` and `update_compaction_num_threads_per_disk`, or decrease `update_compaction_per_tablet_min_interval_seconds` to introduce more compaction resource to handle high frequency writes. How do you know whether current compaction resource and setting can handle current high frequency writes? You can observe it in the following ways: 1. For shared-data cluster, if compaction cannot keep up with the ingestion rate, it may lead to ingestion slowdown or even write failure errors and ingestion stop. a. Ingestion slowdown. You can use `show proc /transactions/{db_name}/running';` to check current running transactions, and if there is any slowdown message like : ```sh Partition's compaction score is larger than 100.0, delay commit for xxxms. You can try to increase compaction concurrency ``` show up in ErrMsg field, that means ingestion slowdown happens. E.g. ```sql show proc '/transactions/test_pri_load_c/running'; ``` ```plain +---------------+----------------------------------------------+------------------+-------------------+--------------------+---------------------+------------+-------------+------------+----------------------------------------------------------------------------------------------------------------------------+--------------------+------------+-----------+--------+ | TransactionId | Label | Coordinator | TransactionStatus | LoadJobSourceType | PrepareTime | CommitTime | PublishTime | FinishTime | Reason | ErrorReplicasCount | ListenerId | TimeoutMs | ErrMsg | +---------------+----------------------------------------------+------------------+-------------------+--------------------+---------------------+------------+-------------+------------+----------------------------------------------------------------------------------------------------------------------------+--------------------+------------+-----------+--------+ | 1034 | stream_load_d2753fbaa0b343acadd5f13de92d44c1 | FE: 172.26.94.39 | PREPARE | FRONTEND_STREAMING | 2024-10-24 13:05:01 | NULL | NULL | NULL | Partition's compaction score is larger than 100.0, delay commit for 6513ms. You can try to increase compaction concurrency, | 0 | 11054 | 86400000 | | +---------------+----------------------------------------------+------------------+-------------------+--------------------+---------------------+------------+-------------+------------+----------------------------------------------------------------------------------------------------------------------------+--------------------+------------+-----------+--------+ ``` b. Ingestion stoppage. If there is an ingestion error like : ```text Failed to load data into partition xxx, because of too large compaction score, current/limit: xxx/xxx. You can reduce the loading job concurrency, or increase compaction concurrency ``` That means Ingestion stop because of compaction can't catch up current high frequency writes. 2. For shared-nothing cluster, there is no ingestion slowdown strategy, if the compaction can't catch up current high frequency writes. Ingestion will fail and return error message: ```text Failed to load data into tablet xxx, because of too many versions, current/limit: xxx/xxx. You can reduce the loading job concurrency, or increase loading data batch size. If you are loading data with Routine Load, you can increase FE configs routine_load_task_consume_second and max_routine_load_batch_size. ``` ###### Data freshness & Compaction resources limitation[​](#data-freshness--compaction-resources-limitation "Direct link to Data freshness & Compaction resources limitation") If you have limited compaction resources but still need to maintain sufficient data freshness, this means you'll need to sacrifice some query latency. You can make these config changes to achieve that: * Shared-data cluster ```text fe.conf lake_ingest_slowdown_threshold = xxx (default is 100, you can increase it) lake_compaction_score_upper_bound = xxx (default is 2000, you can increase it) ``` The `lake_ingest_slowdown_threshold` parameter controls the threshold for triggering ingestion slowdown. When a partition's compaction score exceeds this threshold, the system will begin to slowdown data ingestion. Similarly, `lake_compaction_score_upper_bound` determines the threshold for triggering ingestion stoppage. * Shared-nothing cluster ```text be.conf tablet_max_versions = xxx (default is 1000, you can increase it) ``` `tablet_max_versions` determines the threshold for triggering ingestion stoppage. By increasing these configurations, the system can accommodate more small data files and reduce compaction frequency, but this will also impact query latency. ###### Query latency & Compaction resources limitation[​](#query-latency--compaction-resources-limitation "Direct link to Query latency & Compaction resources limitation") If you want to achieve good query latency with limited compaction resources, you need to reduce write frequency and create larger data batches for ingestion. For specific implementation, please refer to the sections on different ingestion methods, which detail how to reduce ingestion frequency and increase batch size. --- ### Query Tuning Best Practices Learn how to analyze, tune, and optimize queries in StarRocks. #### [📄️ Introduction to Query Tuning](https://docs.starrocks.io/docs/best_practices/query_tuning/query_plan_intro.md) [Query tuning is essential for achieving high performance and reliability in StarRocks.](https://docs.starrocks.io/docs/best_practices/query_tuning/query_plan_intro.md) #### [📄️ Query plan](https://docs.starrocks.io/docs/best_practices/query_tuning/query_planning.md) [Optimizing query performance is a common challenge in analytics systems.](https://docs.starrocks.io/docs/best_practices/query_tuning/query_planning.md) #### [📄️ Query Profile Overview](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md) [Query Profile records execution information for all working nodes involved in a query, helping you quickly identify bottlenecks affecting query performance.](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md) #### [📄️ Query Tuning Recipes](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_tuning_recipes.md) [Query tuning recipes for StarRocks: symptom-to-fix playbook using query profiles to diagnose slow operators, skew, spill, and join issues.](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_tuning_recipes.md) #### [📄️ Schema Tuning Recipes](https://docs.starrocks.io/docs/best_practices/query_tuning/schema_tuning.md) [Best practices for optimizing StarRocks query performance through schema design and table type selection.](https://docs.starrocks.io/docs/best_practices/query_tuning/schema_tuning.md) #### [📄️ Explain Analyze](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_text_based_analysis.md) [How to obtain and analyze text-based Query Profiles in StarRocks to diagnose slow query execution.](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_text_based_analysis.md) #### [📄️ Query Hint](https://docs.starrocks.io/docs/best_practices/query_tuning/query_hint.md) [Query hints are directives or comments that explicitly suggest the query optimizer on how to execute a query.](https://docs.starrocks.io/docs/best_practices/query_tuning/query_hint.md) #### [📄️ Query Profile Metrics](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_operator_metrics.md) [Reference for all raw metrics emitted by StarRocks Query Profile, organized by operator type.](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_operator_metrics.md) --- ### Query Hint Query hints are directives or comments that explicitly suggest the query optimizer on how to execute a query. Currently, StarRocks supports three types of hints: system variable hint (`SET_VAR`), user-defined variable hint (`SET_USER_VARIABLE`), and Join hint. Hints only take effect within a single query. #### System variable hint[​](#system-variable-hint "Direct link to System variable hint") You can use a `SET_VAR` hint to set one or more [system variables](https://docs.starrocks.io/docs/sql-reference/System_variable.md) in SELECT and SUBMIT TASK statements, and then execute the statements. You can also use a `SET_VAR` hint in the SELECT clause included in other statements, such as CREATE MATERIALIZED VIEW AS SELECT and CREATE VIEW AS SELECT. Note that if the `SET_VAR` hint is used in the SELECT clause of CTE, the `SET_VAR` hint does not take effect even if the statement is executed successfully. Compared with [the general usage of system variables](https://docs.starrocks.io/docs/sql-reference/System_variable.md) which takes effect at the session level, the `SET_VAR` hint takes effect at the statement level and does not impact the entire session. ##### Syntax[​](#syntax "Direct link to Syntax") ```sql [...] SELECT /*+ SET_VAR(key=value [, key = value]) */ ... SUBMIT [/*+ SET_VAR(key=value [, key = value]) */] TASK ... ``` ##### Examples[​](#examples "Direct link to Examples") To specify the aggregation mode for an aggregate query, use the `SET_VAR` hint to set the system variables `streaming_preaggregation_mode` and `new_planner_agg_stage` in the aggregate query. ```sql SELECT /*+ SET_VAR (streaming_preaggregation_mode = 'force_streaming',new_planner_agg_stage = '2') */ SUM(sales_amount) AS total_sales_amount FROM sales_orders; ``` To specify the execution timeout for a SUBMIT TASK statement, use the `SET_VAR` Hint to set the system variable `insert_timeout` in the SUBMIT TASK statement. ```sql SUBMIT /*+ SET_VAR(insert_timeout=3) */ TASK AS CREATE TABLE temp AS SELECT count(*) AS cnt FROM tbl1; ``` To specify the subquery execution timeout for creating a materialized view, use the `SET_VAR` hint to set the system variable `query_timeout` in the SELECT clause. ```sql CREATE MATERIALIZED VIEW mv PARTITION BY dt DISTRIBUTED BY HASH(`key`) BUCKETS 10 REFRESH ASYNC AS SELECT /*+ SET_VAR(query_timeout=500) */ * from dual; ``` Specify system variables in a nested query: ```sql -- To specify hints in the main query WITH t AS (SELECT region, sales_amount FROM sales_orders) SELECT /*+ SET_VAR (streaming_preaggregation_mode = 'force_streaming', new_planner_agg_stage = '2') */ SUM(sales_amount) AS total_sales_amount FROM t; ``` #### User-defined variable hint[​](#user-defined-variable-hint "Direct link to User-defined variable hint") You can use a `SET_USER_VARIABLE` hint to set one or more [user-defined variables](https://docs.starrocks.io/docs/sql-reference/user_defined_variables.md) in the SELECT statements or INSERT statements. If other statements contain a SELECT clause, you can also use the `SET_USER_VARIABLE` hint in that SELECT clause. Other statements can be SELECT statements and INSERT statements, but cannot be CREATE MATERIALIZED VIEW AS SELECT statements and CREATE VIEW AS SELECT statements. Note that if the `SET_USER_VARIABLE` hint is used in the SELECT clause of CTE, the `SET_USER_VARIABLE` hint does not take effect even if the statement is executed successfully. Since v3.2.4, StarRocks supports the user-defined variable hint. Compared with [the general usage of user-defined variables](https://docs.starrocks.io/docs/sql-reference/user_defined_variables.md) which takes effect at the session level, the `SET_USER_VARIABLE` hint takes effect at the statement level and does not impact the entire session. ##### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql [...] SELECT /*+ SET_USER_VARIABLE(@var_name = expr [, @var_name = expr]) */ ... INSERT /*+ SET_USER_VARIABLE(@var_name = expr [, @var_name = expr]) */ ... ``` ##### Examples[​](#examples-1 "Direct link to Examples") The following SELECT statement references scalar subqueries `select max(age) from users` and `select min(name) from users`, so you can use a `SET_USER_VARIABLE` hint to set these two scalar subqueries as user-defined variables and then run the query. ```sql SELECT /*+ SET_USER_VARIABLE (@a = (select max(age) from users), @b = (select min(name) from users)) */ * FROM sales_orders where sales_orders.age = @a and sales_orders.name = @b; ``` #### Join hint[​](#join-hint "Direct link to Join hint") For multi-table Join queries, the optimizer usually selects the optimal Join execution method. In special cases, you can use a Join hint to explicitly suggest the Join execution method to the optimizer or disable Join Reorder. Currently, a Join hint supports suggesting Shuffle Join, Broadcast Join, Bucket Shuffle Join, or Colocate Join as a Join execution method. When a Join hint is used, the optimizer does not perform Join Reorder. So you need to select the smaller table as the right table. Additionally, when suggesting [Colocate Join](https://docs.starrocks.io/docs/using_starrocks/Colocate_join.md) or Bucket Shuffle Join as the Join execution method, make sure that the data distribution of the joined table meets the requirements of these Join execution methods. Otherwise, the suggested Join execution method cannot take effect. ##### Syntax[​](#syntax-2 "Direct link to Syntax") ```sql ... JOIN { [BROADCAST] | [SHUFFLE] | [BUCKET] | [COLOCATE] | [UNREORDER]} ... ``` note Join Hint is case-insensitive. ##### Examples[​](#examples-2 "Direct link to Examples") * Shuffle Join If you need to shuffle the data rows with the same bucketing key values from tables A and B onto the same machine before a Join operation is performed, you can hint the Join execution method as Shuffle Join. ```sql select k1 from t1 join [SHUFFLE] t2 on t1.k1 = t2.k2 group by t2.k2; ``` * Broadcast Join If table A is a large table and table B is a small table, you can hint the Join execution method as Broadcast Join. The data of the table B is fully broadcasted to the machines on which the data of table A resides, and then the Join operation is performed. Compared to Shuffle Join, Broadcast Join saves the cost of shuffling the data of table A. ```sql select k1 from t1 join [BROADCAST] t2 on t1.k1 = t2.k2 group by t2.k2; ``` * Bucket Shuffle Join If the Join equijoin expression in the Join query contains the bucketing key of table A, especially when both tables A and B are large tables, you can hint the Join execution method as Bucket Shuffle Join. The data of table B is shuffled to the machines on which the data of table A resides, according to the data distribution of table A, and then the Join operation is performed. Compared to Broadcast Join, Bucket Shuffle Join significantly reduces data transferring because the data of table B is shuffled only once globally. Tables participating in Bucket Shuffle Join must be either non-partitioned or colocated. ```sql select k1 from t1 join [BUCKET] t2 on t1.k1 = t2.k2 group by t2.k2; ``` * Colocate Join If tables A and B belong to the same Colocation Group which is specified during table creation, the data rows with the same bucketing key values from tables A and B are distributed on the same BE node. When the Join equijoin expression contains the bucketing key of tables A and B in the Join query, you can hint the Join execution method as Colocate Join. Data with the same key values are directly joined locally, reducing the time spent on data transmission between nodes and improving query performance. ```sql select k1 from t1 join [COLOCATE] t2 on t1.k1 = t2.k2 group by t2.k2; ``` ##### View Join execution method[​](#view-join-execution-method "Direct link to View Join execution method") Use the `EXPLAIN` command to view the actual Join execution method. If the returned result shows that the Join execution method matches the Join hint, it means that the Join hint is effective. ```sql EXPLAIN select k1 from t1 join [COLOCATE] t2 on t1.k1 = t2.k2 group by t2.k2; ``` ![8-9](/assets/images/8-9-9999cb1cdef4159128d20893137e56e9.png) --- ### Introduction to Query Tuning Query tuning is essential for achieving high performance and reliability in StarRocks. This directory brings together practical guides, reference materials, and actionable recipes to help you analyze, diagnose, and optimize query performance at every stage—from writing SQL to interpreting execution details. Effective query tuning in StarRocks typically follows a top-down process: 1. **Identify the Problem** * Detect slow queries, high resource usage, or unexpected results. * In StarRocks, leverage built-in monitoring tools, query history, and audit logs to quickly identify problematic queries or unusual patterns. * See: **[Query Tuning Recipes](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_tuning_recipes.md)** for symptom-driven diagnosis and **[Query Profile Overview](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md)** for accessing query history and profiles. 2. **Collect and Analyze Execution Information** * Obtain the query plan using `EXPLAIN` or `EXPLAIN ANALYZE`. * Enable and review the Query Profile to gather detailed execution metrics. * See: **[Query Plan Overview](https://docs.starrocks.io/docs/best_practices/query_tuning/query_planning.md)** for understanding query plans, **[Explain Analyze & Text-Based Profile Analysis](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_text_based_analysis.md)** for step-by-step analysis, and **[Query Profile Overview](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md)** for enabling and interpreting profiles. 3. **Locate the Root Cause** * Pinpoint which stage or operator is consuming the most time or resources. * Check for common issues: suboptimal join order, missing indexes, data distribution problems, or inefficient SQL patterns. * See: **[Query Profile Metrics](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_operator_metrics.md)** for a glossary of metrics and operators, and **[Query Tuning Recipes](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_tuning_recipes.md)** for root cause analysis. 4. **Apply Tuning Strategies** * SQL Rewrite: rewrite or optimize the SQL query (e.g., add filters, avoid SELECT \*). * Schema tuning: add indexes, change table types, partitioning, clustering. * Query plan tuning: use hints or variables to guide the optimizer if necessary. * Execution tuning: tune session variables for specific workloads. * See: **[Schema Tuning Recipes](https://docs.starrocks.io/docs/best_practices/query_tuning/schema_tuning.md)** for schema-level optimizations, **[Query Hint](https://docs.starrocks.io/docs/best_practices/query_tuning/query_hint.md)** for optimizer hints, and **[Query Tuning Recipes](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_tuning_recipes.md)** for plan tuning and execution tuning. 5. **Validate and Iterate** * Rerun the query and compare performance before and after changes. * Review the new query plan and profile to ensure improvements. * Repeat the process as needed for further optimization. Whether you're a DBA, developer, or data engineer, these resources will help you: * Diagnose and resolve slow or resource-intensive queries * Understand optimizer choices and execution details * Apply best practices and advanced tuning strategies Start with the overview, dive into the references as needed, and use the recipes and tips to solve real-world performance challenges in StarRocks. --- ### Query plan Optimizing query performance is a common challenge in analytics systems. Slow queries can impair user experience and overall cluster performance. In StarRocks, understanding and interpreting query plans and query profiles is the foundation for diagnosing and improving slow queries. These tools help you: * Identify bottlenecks and expensive operations * Spot suboptimal join strategies or missing indexes * Understand how data is filtered, aggregated, and moved * Troubleshoot and optimize resource usage A **query plan** is a detailed roadmap generated by the StarRocks FE that describes how your SQL statement will be executed. It breaks down the query into a series of operations—such as scans, joins, aggregations, and sorts—and determines the most efficient way to perform them. StarRocks provides several ways to inspect the query plan: 1. **EXPLAIN statement**:
Use `EXPLAIN` to display the logical or physical execution plan for a query. You can add options to control the output: * `EXPLAIN LOGICAL `: Shows the simplified plan. * `EXPLAIN `: Shows the basic physical plan * `EXPLAIN VERBOSE `: Shows the physical plan with detailed information. * `EXPLAIN COSTS `: Includes estimated costs for each operation, which is used to diagnose the statistics issue 2. **EXPLAIN ANALYZE**:
Use `EXPLAIN ANALYZE ` to execute the query and display the actual execution plan along with real runtime statistics. See the [Explain Analyze](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_text_based_analysis.md) documentation for details. Example: ```sql EXPLAIN ANALYZE SELECT * FROM sales_orders WHERE amount > 1000; ``` 3. **Query Profile**:
After running a query, you can view its detailed execution profile, which includes timing, resource usage, and operator-level statistics. See the [Query Profile](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md) documentation for how to access and interpret this information. * **SQL commands**: `SHOW PROFILELIST` and `ANALYZE PROFILE FOR `: can be used to retrieve the execution profile for a specific query. * **FE HTTP Service**: Access query profiles via the StarRocks FE web UI by navigating to the **Query** or **Profile** section, where you can search for and inspect query execution details. * **Managed Version**: In cloud or managed deployments, use the provided web console or monitoring dashboard to view query plans and profiles, often with enhanced visualization and filtering options. Typically, the query plan is used to diagnose issues related to how a query is planned and optimized, while the query profile helps identify performance problems during query execution. In the following sections, we'll explore the key concepts of query execution and walk through a concrete example of analyzing a query plan. #### Query execution flow[​](#query-execution-flow "Direct link to Query execution flow") The lifecycle of a query in StarRocks consists of three main phases: 1. **Planning**: The query undergoes parsing, analysis, and optimization, culminating in the generation of a query plan. 2. **Scheduling**: The scheduler and coordinator distribute the plan to all participating backend nodes. 3. **Execution**: The plan is executed using the pipeline execution engine. ![SQL Execution Flow](/assets/images/execution_flow-f4e4a147a33bea8ecfc538573e8f758e.png) **Plan structure** The StarRocks plan is hierarchical: * **Fragment**: Top-level slice of work; each fragment spawns multiple **FragmentInstances** that run on different backend nodes. * **Pipeline**: Within an instance, a pipeline strings operators together; several **PipelineDrivers** run the same pipeline concurrently on separate CPU cores. * **Operator**: The atomic step—scan, join, aggregate—that actually processes data. ![profile-3](/assets/images/profile-3-079a0ab5168d571ec15182752d97142d.png) **Pipeline execution engine** The Pipeline Engine executes the query plan in a parallel and efficient manner, handling complex plans and large data volumes for high performance and scalability. ![pipeline\_opeartors](/assets/images/pipeline_operators-930f5c2a794b4961d3e2a9f4daf9cf25.png) **Metric merging strategy** By default, StarRocks merges the FragmentInstance and PipelineDriver layers to reduce profile volume, resulting in a simplified three-layer structure: * Fragment * Pipeline * Operator You can control this merging behavior through the session variable `pipeline_profile_level`. #### Example[​](#example "Direct link to Example") ##### How to read a query plan and profile[​](#how-to-read-a-query-plan-and-profile "Direct link to How to read a query plan and profile") 1. **Understand the structure**: Query plans are split into fragments, each representing a stage of execution. Read from the bottom up: scan nodes first, then joins, aggregations, and finally the result. 2. **Overall analysis**: * Check total runtime, memory usage, and CPU/wall time ratio. * Find slow operators by sorting by operator time. * Ensure filters are pushed down where possible. * Look for data skew (uneven operator times or row counts). * Monitor for high memory or disk spill; adjust join order or use rollup views if needed. * Use materialized views and query hints (`BROADCAST`, `SHUFFLE`, `COLOCATE`) to optimize as needed. 3. **Scan operations**: Look for `OlapScanNode` or similar. Note which tables are scanned, what filters are applied, and if pre-aggregation or materialized views are used. 4. **Join operations**: Identify join types (`HASH JOIN`, `BROADCAST`, `SHUFFLE`, `COLOCATE`, `BUCKET SHUFFLE`). The join method affects performance: * **Broadcast**: Small table sent to all nodes; good for small tables. * **Shuffle**: Rows are partitioned and shuffled; good for large tables. * **Colocate**: Tables partitioned the same way; enables local joins. * **Bucket Shuffle**: Only one table is shuffled to reduce network cost. 5. **Aggregation and sorting**: Look for `AGGREGATE`, `TOP-N`, or `ORDER BY`. These can be expensive with large or high-cardinality data. 6. **Data movement**: `EXCHANGE` nodes show data transfer between fragments or nodes. Too much data movement can hurt performance. 7. **Predicate pushdown**: Filters applied early (at scan) reduce downstream data. Check `PREDICATES` or `PushdownPredicates` to see which filters are pushed down. ##### Example query plan[​](#example-query-plan "Direct link to Example query plan") tip This is query 96 from the TPC-DS benchmark. ```sql explain logical select count(*) from store_sales ,household_demographics ,time_dim , store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 8 and time_dim.t_minute >= 30 and household_demographics.hd_dep_count = 5 and store.s_store_name = 'ese' order by count(*) limit 100; ``` The output is a hierarchical plan showing how StarRocks will execute the query. The plan is structured as a tree of operators, read from bottom to top. The logical plan shows the sequence of operations with cost estimates: ```text - Output => [69:count] - TOP-100(FINAL)[69: count ASC NULLS FIRST] Estimates: {row: 1, cpu: 8.00, memory: 8.00, network: 8.00, cost: 68669801.20} - TOP-100(PARTIAL)[69: count ASC NULLS FIRST] Estimates: {row: 1, cpu: 8.00, memory: 8.00, network: 8.00, cost: 68669769.20} - AGGREGATE(GLOBAL) [] Estimates: {row: 1, cpu: 8.00, memory: 8.00, network: 0.00, cost: 68669737.20} 69:count := count(69:count) - EXCHANGE(GATHER) Estimates: {row: 1, cpu: 8.00, memory: 0.00, network: 8.00, cost: 68669717.20} - AGGREGATE(LOCAL) [] Estimates: {row: 1, cpu: 3141.35, memory: 0.80, network: 0.00, cost: 68669701.20} 69:count := count() - HASH/INNER JOIN [9:ss_store_sk = 40:s_store_sk] => [71:auto_fill_col] Estimates: {row: 3490, cpu: 111184.52, memory: 8.80, network: 0.00, cost: 68668128.93} 71:auto_fill_col := 1 - HASH/INNER JOIN [7:ss_hdemo_sk = 25:hd_demo_sk] => [9:ss_store_sk] Estimates: {row: 19940, cpu: 1841177.20, memory: 2880.00, network: 0.00, cost: 68612474.92} - HASH/INNER JOIN [4:ss_sold_time_sk = 30:t_time_sk] => [7:ss_hdemo_sk, 9:ss_store_sk] Estimates: {row: 199876, cpu: 69221191.15, memory: 7077.97, network: 0.00, cost: 67671726.32} - SCAN [store_sales] => [4:ss_sold_time_sk, 7:ss_hdemo_sk, 9:ss_store_sk] Estimates: {row: 5501341, cpu: 66016092.00, memory: 0.00, network: 0.00, cost: 33008046.00} partitionRatio: 1/1, tabletRatio: 192/192 predicate: 7:ss_hdemo_sk IS NOT NULL - EXCHANGE(BROADCAST) Estimates: {row: 1769, cpu: 7077.97, memory: 7077.97, network: 7077.97, cost: 38928.81} - SCAN [time_dim] => [30:t_time_sk] Estimates: {row: 1769, cpu: 21233.90, memory: 0.00, network: 0.00, cost: 10616.95} partitionRatio: 1/1, tabletRatio: 5/5 predicate: 33:t_hour = 8 AND 34:t_minute >= 30 - EXCHANGE(BROADCAST) Estimates: {row: 720, cpu: 2880.00, memory: 2880.00, network: 2880.00, cost: 14400.00} - SCAN [household_demographics] => [25:hd_demo_sk] Estimates: {row: 720, cpu: 5760.00, memory: 0.00, network: 0.00, cost: 2880.00} partitionRatio: 1/1, tabletRatio: 1/1 predicate: 28:hd_dep_count = 5 - EXCHANGE(BROADCAST) Estimates: {row: 2, cpu: 8.80, memory: 8.80, network: 8.80, cost: 44.15} - SCAN [store] => [40:s_store_sk] Estimates: {row: 2, cpu: 17.90, memory: 0.00, network: 0.00, cost: 8.95} partitionRatio: 1/1, tabletRatio: 1/1 predicate: 45:s_store_name = 'ese' ``` **Reading the plan bottom-up** The query plan should be read from the bottom (leaf nodes) upward to the top (root node), following the data flow: 1. **Scan Operations (Bottom Level)**: The `SCAN` operators at the bottom read data from the base tables: * `SCAN [store_sales]` reads the main fact table with predicate `ss_hdemo_sk IS NOT NULL` * `SCAN [time_dim]` reads the time dimension table with predicates `t_hour = 8 AND t_minute >= 30` * `SCAN [household_demographics]` reads the demographics table with predicate `hd_dep_count = 5` * `SCAN [store]` reads the store table with predicate `s_store_name = 'ese'` Each scan operation shows: * **Estimates**: Row count, CPU, memory, network, and cost estimates * **Partition and tablet ratios**: How many partitions/tablets are scanned (e.g., `partitionRatio: 1/1, tabletRatio: 192/192`) * **Predicates**: Query conditions that are pushed down to the scan level, reducing the amount of data read 2. **Data Exchange (Broadcast)**: The `EXCHANGE(BROADCAST)` operations distribute smaller dimension tables to all nodes processing the larger fact table. This is efficient when dimension tables are small compared to the fact table, as seen with `time_dim`, `household_demographics`, and `store` being broadcast. 3. **Join Operations (Middle Level)**: Data flows upward through `HASH/INNER JOIN` operations: * First, `store_sales` is joined with `time_dim` on `ss_sold_time_sk = t_time_sk` * Then, the result is joined with `household_demographics` on `ss_hdemo_sk = hd_demo_sk` * Finally, the result is joined with `store` on `ss_store_sk = s_store_sk` Each join shows the join condition and estimates for the resulting row count and resource usage. 4. **Aggregation (Upper Level)**: * `AGGREGATE(LOCAL)` performs local aggregation on each node, computing `count()` * `EXCHANGE(GATHER)` collects results from all nodes * `AGGREGATE(GLOBAL)` merges the local results into the final count 5. **Final Operations (Top Level)**: * `TOP-100(PARTIAL)` and `TOP-100(FINAL)` operations handle the `ORDER BY count(*) LIMIT 100` clause, selecting the top 100 results after ordering The logical plan provides cost estimates for each operation, helping you understand where the query spends most of its resources. The actual physical execution plan (from `EXPLAIN` or `EXPLAIN VERBOSE`) includes additional details about how operations are distributed across nodes and executed in parallel. --- ### Query Profile Metrics > Authoritative reference for raw metrics emitted by **StarRocks Query Profile**, grouped by operator.
Use it as a glossary; for troubleshooting guidance jump to **query\_profile\_tuning\_recipes.md**. ##### Summary Metrics[​](#summary-metrics "Direct link to Summary Metrics") Basic information about the query execution: | Metric | Description | | -------------------------- | --------------------------------------------------------------------------------------------------- | | Total | The total time consumed by the query, including Planning, Executing, and Profiling phase durations. | | Query State | Query state, possible states include Finished, Error, and Running. | | Query ID | Unique identifier for the query. | | Start Time | Timestamp when the query started. | | End Time | Timestamp when the query ended. | | Total | Total duration of the query. | | Query Type | Type of the query. | | Query State | Current state of the query. | | StarRocks Version | Version of StarRocks used. | | User | User who executed the query. | | Default Db | Default database used for the query. | | Sql Statement | SQL statement executed. | | Variables | Important variables used for the query. | | NonDefaultSessionVariables | Non-default session variables used for the query. | | Collect Profile Time | Time taken to collect the profile. | | IsProfileAsync | Indicates if the profile collection was asynchronous. | ##### Planner Metrics[​](#planner-metrics "Direct link to Planner Metrics") It provides a comprehensive overview of the planner. Typically, if the total time spent on the planner is less than 10ms, it is not a cause for concern. In certain scenarios, the planner may require more time: 1. Complex queries may necessitate additional time for parsing and optimization to ensure an optimal execution plan. 2. The presence of numerous materialized views can increase the time required for query rewriting. 3. When multiple concurrent queries exhaust system resources and the query queue is utilized, the `Pending` time may be prolonged. 4. Queries involving external tables may incur additional time for communication with the external metadata server. Example: ```text - -- Parser[1] 0 - -- Total[1] 3ms - -- Analyzer[1] 0 - -- Lock[1] 0 - -- AnalyzeDatabase[1] 0 - -- AnalyzeTemporaryTable[1] 0 - -- AnalyzeTable[1] 0 - -- Transformer[1] 0 - -- Optimizer[1] 1ms - -- MVPreprocess[1] 0 - -- MVTextRewrite[1] 0 - -- RuleBaseOptimize[1] 0 - -- CostBaseOptimize[1] 0 - -- PhysicalRewrite[1] 0 - -- DynamicRewrite[1] 0 - -- PlanValidate[1] 0 - -- InputDependenciesChecker[1] 0 - -- TypeChecker[1] 0 - -- CTEUniqueChecker[1] 0 - -- ColumnReuseChecker[1] 0 - -- ExecPlanBuild[1] 0 - -- Pending[1] 0 - -- Prepare[1] 0 - -- Deploy[1] 2ms - -- DeployLockInternalTime[1] 2ms - -- DeploySerializeConcurrencyTime[2] 0 - -- DeployStageByStageTime[6] 0 - -- DeployWaitTime[6] 1ms - -- DeployAsyncSendTime[2] 0 - DeployDataSize: 10916 Reason: ``` ##### Execution Overview Metrics[​](#execution-overview-metrics "Direct link to Execution Overview Metrics") High-level execution statistics: | Metric | Description | Rule of Thumb | | --------------------------- | ------------------------------------- | ----------------------------------------- | | FrontendProfileMergeTime | FE-side profile processing time | < 10ms normal | | QueryAllocatedMemoryUsage | Total allocated memory across nodes | | | QueryDeallocatedMemoryUsage | Total deallocated memory across nodes | | | QueryPeakMemoryUsagePerNode | Maximum peak memory per node | < 80% capacity normal | | QuerySumMemoryUsage | Total peak memory across nodes | | | QueryExecutionWallTime | Wall time of execution | | | QueryCumulativeCpuTime | Total CPU time across nodes | Compare with `walltime * totalCpuCores` | | QueryCumulativeOperatorTime | Total operator execution time | Denominator for operator time percentages | | QueryCumulativeNetworkTime | Total Exchange node network time | | | QueryCumulativeScanTime | Total Scan node IO time | | | QueryPeakScheduleTime | Maximum Pipeline ScheduleTime | < 1s normal for simple queries | | QuerySpillBytes | Data spilled to disk | < 1GB normal | ##### Fragment Metrics[​](#fragment-metrics "Direct link to Fragment Metrics") Fragment-level execution details: | Metric | Description | | ------------------------------ | -------------------------------------- | | InstanceNum | Number of FragmentInstances | | InstanceIds | IDs of all FragmentInstances | | BackendNum | Number of participating BEs | | BackendAddresses | BE addresses | | FragmentInstancePrepareTime | Fragment Prepare phase duration | | InstanceAllocatedMemoryUsage | Total allocated memory for instances | | InstanceDeallocatedMemoryUsage | Total deallocated memory for instances | | InstancePeakMemoryUsage | Peak memory across instances | ##### Pipeline Metrics[​](#pipeline-metrics "Direct link to Pipeline Metrics") Pipeline execution details and relationships: ![profile\_pipeline\_time\_relationship](/assets/images/profile_pipeline_time_relationship-439d46dd1ab122cafdf39053fedc34cc.jpeg) Key relationships: * DriverTotalTime = ActiveTime + PendingTime + ScheduleTime * ActiveTime = ∑ OperatorTotalTime + OverheadTime * PendingTime = InputEmptyTime + OutputFullTime + PreconditionBlockTime + PendingFinishTime * InputEmptyTime = FirstInputEmptyTime + FollowupInputEmptyTime | Metric | Description | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | DegreeOfParallelism | Degree of pipeline execution parallelism. | | TotalDegreeOfParallelism | Sum of degrees of parallelism. Since the same Pipeline may execute on multiple machines, this item aggregates all values. | | DriverPrepareTime | Time taken by the Prepare phase. This metric is not included in DriverTotalTime. | | DriverTotalTime | Total execution time of the Pipeline, excluding the time spent in the Prepare phase. | | ActiveTime | Execution time of the Pipeline, including the execution time of each operator and the overall framework overhead, such as time spent in invoking methods like has\_output, need\_input, etc. | | PendingTime | Time the Pipeline is blocked from being scheduled for various reasons. | | InputEmptyTime | Time the Pipeline is blocked due to an empty input queue. | | FirstInputEmptyTime | Time the Pipeline is first blocked due to an empty input queue. The first blocking time is separately calculated because the first blocking is mainly caused by Pipeline dependencies. | | FollowupInputEmptyTime | Time the Pipeline is subsequently blocked due to an empty input queue. | | OutputFullTime | Time the Pipeline is blocked due to a full output queue. | | PreconditionBlockTime | Time the Pipeline is blocked due to unmet dependencies. | | PendingFinishTime | Time the Pipeline is blocked waiting for asynchronous tasks to finish. | | ScheduleTime | Scheduling time of the Pipeline, from entering the ready queue to being scheduled for execution. | | BlockByInputEmpty | Number of times the pipeline is blocked due to InputEmpty. | | BlockByOutputFull | Number of times the pipeline is blocked due to OutputFull. | | BlockByPrecondition | Number of times the pipeline is blocked due to unmet preconditions. | ##### Operator Metrics[​](#operator-metrics "Direct link to Operator Metrics") | Metric | Description | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | PrepareTime | Time spent on preparation. | | OperatorTotalTime | Total time consumed by the Operator. It satisfies the equation: OperatorTotalTime = PullTotalTime + PushTotalTime + SetFinishingTime + SetFinishedTime + CloseTime. It excludes time spent on preparation. | | PullTotalTime | Total time the Operator spends executing push\_chunk. | | PushTotalTime | Total time the Operator spends executing pull\_chunk. | | SetFinishingTime | Total time the Operator spends executing set\_finishing. | | SetFinishedTime | Total time the Operator spends executing set\_finished. | | PushRowNum | Cumulative number of input rows for the Operator. | | PullRowNum | Cumulative number of output rows for the Operator. | | JoinRuntimeFilterEvaluate | Number of times Join Runtime Filter is evaluated. | | JoinRuntimeFilterHashTime | Time spent computing hash for Join Runtime Filter. | | JoinRuntimeFilterInputRows | Number of input rows for Join Runtime Filter. | | JoinRuntimeFilterOutputRows | Number of output rows for Join Runtime Filter. | | JoinRuntimeFilterTime | Time spent on Join Runtime Filter. | ##### Scan Operator[​](#scan-operator "Direct link to Scan Operator") ###### OLAP Scan Operator[​](#olap-scan-operator "Direct link to OLAP Scan Operator") The OLAP\_SCAN Operator is responsible for reading data from StarRocks native tables. | Metric | Description | | -------------------------- | ------------------------------------------------------------------------------------------- | | Table | Table name. | | Rollup | Materialized view name. If no materialized view is hit, it is equivalent to the table name. | | SharedScan | Whether the enable\_shared\_scan session variable is enabled. | | TabletCount | Number of tablets. | | MorselsCount | Number of morsels, which is the basic IO execution unit. | | PushdownPredicates | Number of pushdown predicates. | | Predicates | Predicate expressions. | | BytesRead | Size of data read. | | CompressedBytesRead | Size of compressed data read from disk. | | UncompressedBytesRead | Size of uncompressed data read from disk. | | RowsRead | Number of rows read (after predicate filtering). | | RawRowsRead | Number of raw rows read (before predicate filtering). | | ReadPagesNum | Number of pages read. | | CachedPagesNum | Number of cached pages. | | ChunkBufferCapacity | Capacity of the Chunk Buffer. | | DefaultChunkBufferCapacity | Default capacity of the Chunk Buffer. | | PeakChunkBufferMemoryUsage | Peak memory usage of the Chunk Buffer. | | PeakChunkBufferSize | Peak size of the Chunk Buffer. | | PrepareChunkSourceTime | Time spent preparing the Chunk Source. | | ScanTime | Cumulative scan time. Scan operations are completed in an asynchronous I/O thread pool. | | IOTaskExecTime | Execution time of IO tasks. | | IOTaskWaitTime | Waiting time from successful submission to scheduled execution of IO tasks. | | SubmitTaskCount | Number of times IO tasks are submitted. | | SubmitTaskTime | Time spent on task submission. | | PeakIOTasks | Peak number of IO tasks. | | PeakScanTaskQueueSize | Peak size of the IO task queue. | ###### Connector Scan Operator[​](#connector-scan-operator "Direct link to Connector Scan Operator") It's similar to OLAP\_SCAN operator but used for scan external tables like Iceberg/Hive/Hudi/Detal. | Metric | Description | | -------------------------- | ---------------------------------------------------------------------------------------------- | | DataSourceType | Data source type, can be HiveDataSource, ESDataSource, and so on. | | Table | Table name. | | TabletCount | Number of tablets. | | MorselsCount | Number of morsels. | | Predicates | Predicate expression. | | PredicatesPartition | Predicate expression applied to partitions. | | SharedScan | Whether the `enable_shared_scan` Session variable is enabled. | | ChunkBufferCapacity | Capacity of the Chunk Buffer. | | DefaultChunkBufferCapacity | Default capacity of the Chunk Buffer. | | PeakChunkBufferMemoryUsage | Peak memory usage of the Chunk Buffer. | | PeakChunkBufferSize | Peak size of the Chunk Buffer. | | PrepareChunkSourceTime | Time taken to prepare the Chunk Source. | | ScanTime | Cumulative time for scanning. Scan operation is completed in the asynchronous I/O thread pool. | | IOTaskExecTime | Execution time of I/O tasks. | | IOTaskWaitTime | Waiting time from successful submission to scheduled execution of IO tasks. | | SubmitTaskCount | Number of times IO tasks are submitted. | | SubmitTaskTime | Time taken to submit tasks. | | PeakIOTasks | Peak number of IO tasks. | | PeakScanTaskQueueSize | Peak size of the IO task queue. | ##### Exchange Operator[​](#exchange-operator "Direct link to Exchange Operator") Exchange Operator is responsible for transmitting data between BE nodes. There can be several kinds of exchange operations: GATHER/BROADCAST/SHUFFLE. Typical scenarios that can make Exchange Operator the bottleneck of a query: 1. Broadcast Join: This is a suitable method for a small table. However, in exceptional cases when the optimizer chooses a suboptimal query plan, it can lead to a significant increase in network bandwidth. 2. Shuffle Aggregation/Join: Shuffling a large table can result in a significant increase in network bandwidth. ###### Exchange Sink Operator[​](#exchange-sink-operator "Direct link to Exchange Sink Operator") | Metric | Description | | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ChannelNum | Number of channels. Generally, the number of channels is equal to the number of receivers. | | DestFragments | List of destination FragmentInstance IDs. | | DestID | Destination node ID. | | PartType | Data distribution mode, including: UNPARTITIONED, RANDOM, HASH\_PARTITIONED, and BUCKET\_SHUFFLE\_HASH\_PARTITIONED. | | SerializeChunkTime | Time taken to serialize chunks. | | SerializedBytes | Size of serialized data. | | ShuffleChunkAppendCounter | Number of Chunk Append operations when PartType is HASH\_PARTITIONED or BUCKET\_SHUFFLE\_HASH\_PARTITIONED. | | ShuffleChunkAppendTime | Time taken for Chunk Append operations when PartType is HASH\_PARTITIONED or BUCKET\_SHUFFLE\_HASH\_PARTITIONED. | | ShuffleHashTime | Time taken to calculate hash when PartType is HASH\_PARTITIONED or BUCKET\_SHUFFLE\_HASH\_PARTITIONED. | | RequestSent | Number of data packets sent. | | RequestUnsent | Number of unsent data packets. This metric is non-zero when there is a short-circuit logic; otherwise, it is zero. | | BytesSent | Size of sent data. | | BytesUnsent | Size of unsent data. This metric is non-zero when there is a short-circuit logic; otherwise, it is zero. | | BytesPassThrough | If the destination node is the current node, data will not be transmitted over the network, which is called passthrough data. This metric indicates the size of such passthrough data. Passthrough is controlled by `enable_exchange_pass_through`. | | PassThroughBufferPeakMemoryUsage | Peak memory usage of the PassThrough Buffer. | | CompressTime | Compression time. | | CompressedInputBytes | Size of the serialized (pre-compression) data that was actually fed to the compressor. Chunks skipped by the adaptive compression strategy are not counted. `CompressedInputBytes / CompressedBytes` gives the compression ratio, and `SerializedBytes - CompressedInputBytes` is the size of data that was not compressed. | | CompressedBytes | Size of compressed data. Only chunks that were actually compressed are counted. | | OverallThroughput | Throughput rate. | | NetworkTime | Time taken for data packet transmission (excluding post-reception processing time). | | NetworkBandwidth | Estimated network bandwidth. | | WaitTime | Waiting time due to a full sender queue. | | OverallTime | Total time for the entire transmission process, i.e., from sending the first data packet to confirming the correct reception of the last data packet. | | RpcAvgTime | Average time for RPC. | | RpcCount | Total number of RPCs. | ###### Exchange Source Operator[​](#exchange-source-operator "Direct link to Exchange Source Operator") | Metric | Description | | ------------------------ | ---------------------------------------------- | | RequestReceived | Size of received data packets. | | BytesReceived | Size of received data. | | DecompressChunkTime | Time taken to decompress chunks. | | DeserializeChunkTime | Time taken to deserialize chunks. | | ClosureBlockCount | Number of blocked Closures. | | ClosureBlockTime | Blocked time for Closures. | | ReceiverProcessTotalTime | Total time taken for receiver-side processing. | | WaitLockTime | Lock waiting time. | ##### Aggregate Operator[​](#aggregate-operator "Direct link to Aggregate Operator") **Metrics List** | Metric | Description | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | `GroupingKeys` | `GROUP BY` columns. | | `AggregateFunctions` | Time taken for aggregate function calculations. | | `AggComputeTime` | Time for AggregateFunctions + Group By. | | `ChunkBufferPeakMem` | Peak memory usage of the Chunk Buffer. | | `ChunkBufferPeakSize` | Peak size of the Chunk Buffer. | | `ExprComputeTime` | Time for expression computation. | | `ExprReleaseTime` | Time for expression release. | | `GetResultsTime` | Time to extract aggregate results. | | `HashTableSize` | Size of the Hash Table. | | `HashTableMemoryUsage` | Memory size of the Hash Table. | | `InputRowCount` | Number of input rows. | | `PassThroughRowCount` | In Auto mode, the number of data rows processed in streaming mode after low aggregation leads to degradation to streaming mode. | | `ResultAggAppendTime` | Time taken to append aggregate result columns. | | `ResultGroupByAppendTime` | Time taken to append Group By columns. | | `ResultIteratorTime` | Time to iterate over the Hash Table. | | `StreamingTime` | Processing time in streaming mode. | ##### Join Operator[​](#join-operator "Direct link to Join Operator") **Metrics List** | Metric | Description | | --------------------------- | -------------------------------------------------------------------- | | `DistributionMode` | Distribution type, including: BROADCAST, PARTITIONED, COLOCATE, etc. | | `JoinPredicates` | Join predicates. | | `JoinType` | Join type. | | `BuildBuckets` | Number of buckets in the Hash Table. | | `BuildKeysPerBucket` | Number of keys per bucket in the Hash Table. | | `BuildConjunctEvaluateTime` | Time taken for conjunct evaluation during build phase. | | `BuildHashTableTime` | Time taken to build the Hash Table. | | `ProbeConjunctEvaluateTime` | Time taken for conjunct evaluation during probe phase. | | `SearchHashTableTimer` | Time taken to search the Hash Table. | | `CopyRightTableChunkTime` | Time taken to copy chunks from the right table. | | `OutputBuildColumnTime` | Time taken to output the column of build side. | | `OutputProbeColumnTime` | Time taken to output the column of probe side. | | `HashTableMemoryUsage` | Memory usage of the Hash Table. | | `RuntimeFilterBuildTime` | Time taken to build runtime filters. | | `RuntimeFilterNum` | Number of runtime filters. | ##### Window Function Operator[​](#window-function-operator "Direct link to Window Function Operator") | Metric | Description | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ProcessMode` | Execution mode, including two parts: the first part includes Materializing and Streaming; the second part includes Cumulative, RemovableCumulative, ByDefinition. | | `ComputeTime` | Time taken for window function calculations. | | `PartitionKeys` | Partition columns. | | `AggregateFunctions` | Aggregate functions. | | `ColumnResizeTime` | Time taken for column resizing. | | `PartitionSearchTime` | Time taken to search partition boundaries. | | `PeerGroupSearchTime` | Time taken to search Peer Group boundaries. Meaningful only when the window type is `RANGE`. | | `PeakBufferedRows` | Peak number of rows in the buffer. | | `RemoveUnusedRowsCount` | Number of times unused buffers are removed. | | `RemoveUnusedTotalRows` | Total number of rows removed from unused buffers. | ##### Sort Operator[​](#sort-operator "Direct link to Sort Operator") | Metric | Description | | ------------------ | ----------------------------------------------------------------------- | | `SortKeys` | Sorting keys. | | `SortType` | Query result sorting method: full sorting or sorting the top N results. | | `MaxBufferedBytes` | Peak size of buffered data. | | `MaxBufferedRows` | Peak number of buffered rows. | | `NumSortedRuns` | Number of sorted runs. | | `BuildingTime` | Time taken to maintain internal data structures during sorting. | | `MergingTime` | Time taken to merge sorted runs during sorting. | | `SortingTime` | Time taken for sorting. | | `OutputTime` | Time taken to build the output sorted sequence. | ##### Merge Operator[​](#merge-operator "Direct link to Merge Operator") | Metric | Description | Level | | ---------------------------------------- | ------------------------------------------------------------------------------------ | --------- | | `Limit` | Limit. | Primary | | `Offset` | Offset. | Primary | | `StreamingBatchSize` | Size of data processed per Merge operation when Merge is performed in Streaming mode | Primary | | `LateMaterializationMaxBufferChunkNum` | Maximum number of chunks in the buffer when late materialization is enabled. | Primary | | `OverallStageCount` | Total execution count of all stages. | Primary | | `OverallStageTime` | Total execution time for each stage. | Primary | | `1-InitStageCount` | Execution count of the Init stage. | Secondary | | `2-PrepareStageCount` | Execution count of the Prepare stage. | Secondary | | `3-ProcessStageCount` | Execution count of the Process stage. | Secondary | | `4-SplitChunkStageCount` | Execution count of the SplitChunk stage. | Secondary | | `5-FetchChunkStageCount` | Execution count of the FetchChunk stage. | Secondary | | `6-PendingStageCount` | Execution count of the Pending stage. | Secondary | | `7-FinishedStageCount` | Execution count of the Finished stage. | Secondary | | `1-InitStageTime` | Execution time for the Init stage. | Secondary | | `2-PrepareStageTime` | Execution time for the Prepare stage. | Secondary | | `3-ProcessStageTime` | Execution time for the Process stage. | Secondary | | `4-SplitChunkStageTime` | Time taken for the Split stage. | Secondary | | `5-FetchChunkStageTime` | Time taken for the Fetch stage. | Secondary | | `6-PendingStageTime` | Time taken for the Pending stage. | Secondary | | `7-FinishedStageTime` | Time taken for the Finished stage. | Secondary | | `LateMaterializationGenerateOrdinalTime` | Time taken for generating ordinal columns during late materialization. | Tertiary | | `SortedRunProviderTime` | Time taken to retrieve data from the provider during the Process stage. | Tertiary | ##### TableFunction Operator[​](#tablefunction-operator "Direct link to TableFunction Operator") | Metric | Description | | ------------------------ | -------------------------------------------- | | `TableFunctionExecTime` | Computation time for the Table Function. | | `TableFunctionExecCount` | Number of executions for the Table Function. | ##### Project Operator[​](#project-operator "Direct link to Project Operator") Project Operator is responsible for performing `SELECT `. If there're some expensive expressions in the query, this operator can take significant time. | Metric | Description | | -------------------------- | -------------------------------------------- | | `ExprComputeTime` | Computation time for expressions. | | `CommonSubExprComputeTime` | Computation time for common sub-expressions. | ##### LocalExchange Operator[​](#localexchange-operator "Direct link to LocalExchange Operator") | Metric | Description | | ------------------------------------ | ------------------------------------------------------------------------------- | | Type | Type of Local Exchange, including: `Passthrough`, `Partition`, and `Broadcast`. | | `ShuffleNum` | Number of shuffles. This metric is only valid when `Type` is `Partition`. | | `LocalExchangePeakMemoryUsage` | Peak memory usage. | | `LocalExchangePeakBufferSize` | Peak size of the buffer. | | `LocalExchangePeakBufferMemoryUsage` | Peak memory usage of the buffer. | | `LocalExchangePeakBufferChunkNum` | Peak number of chunks in the buffer. | | `LocalExchangePeakBufferRowNum` | Peak number of rows in the buffer. | | `LocalExchangePeakBufferBytes` | Peak size of data in the buffer. | | `LocalExchangePeakBufferChunkSize` | Peak size of chunks in the buffer. | | `LocalExchangePeakBufferChunkRowNum` | Peak number of rows per chunk in the buffer. | | `LocalExchangePeakBufferChunkBytes` | Peak size of data per chunk in the buffer. | ##### OlapTableSink Operator[​](#olaptablesink-operator "Direct link to OlapTableSink Operator") OlapTableSink Operator is responsible for performing the `INSERT INTO
` operation. tip * An excessive difference between the Max and Min values of the `PushChunkNum` metric of `OlapTableSink` indicates data skew in the upstream operators, which may lead to a bottleneck in loading performance. * `RpcClientSideTime` equals `RpcServerSideTime` plus network transmission time plus RPC framework processing time. If there is a significant difference between `RpcClientSideTime` and `RpcServerSideTime`, consider enabling compression to reduce transmission time. | Metric | Description | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `IndexNum` | Number of the synchronous materialized views created for the destination table. | | `ReplicatedStorage` | Whether Single Leader Replication is enabled. | | `TxnID` | ID of the loading transaction. | | `RowsRead` | Number of rows read from upstream operators. | | `RowsFiltered` | Number of rows filtered out due to inadequate data quality. | | `RowsReturned` | Number of rows written to the destination table. | | `RpcClientSideTime` | Total RPC time consumption for loading recorded by the client side. | | `RpcServerSideTime` | Total RPC time consumption for loading recorded by the server side. | | `PrepareDataTime` | Total time consumption for the data preparation phase, including data format conversion and data quality check. | | `SendDataTime` | Local time consumption for sending the data, including time for serializing and compressing data, and for submitting tasks to the sender queue. | --- ### Query Profile Overview #### Introduction[​](#introduction "Direct link to Introduction") Query Profile records execution information for all working nodes involved in a query, helping you quickly identify bottlenecks affecting query performance. It is a powerful tool for diagnosing and tuning query performance in StarRocks. > From v3.3.0 onwards, StarRocks supports providing Query Profile for data loading with INSERT INTO FILES() and Broker Load. For details of the metrics involved, see [OlapTableSink Operator](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_operator_metrics.md#olaptablesink-operator). #### How to Enable Query Profile[​](#how-to-enable-query-profile "Direct link to How to Enable Query Profile") ##### Enable Query Profile[​](#enable-query-profile "Direct link to Enable Query Profile") You can enable Query Profile by setting the variable `enable_profile` to `true`: ```sql SET enable_profile = true; SET GLOBAL enable_profile = true; ``` ##### Query Profile for Slow Queries[​](#query-profile-for-slow-queries "Direct link to Query Profile for Slow Queries") It is not recommended to enable Query Profile globally in production for long periods, as it may impose additional system overhead. To capture and analyze only slow queries, set the variable `big_query_profile_threshold` to a time duration greater than `0s`. For example, setting it to `30s` means only queries exceeding 30 seconds will trigger Query Profile. ```sql -- 30 seconds SET global big_query_profile_threshold = '30s'; -- 500 milliseconds SET global big_query_profile_threshold = '500ms'; -- 60 minutes SET global big_query_profile_threshold = '60m'; ``` ##### Runtime Query Profile[​](#runtime-query-profile "Direct link to Runtime Query Profile") For long-running queries, it can be difficult to determine progress or detect issues before completion. The Runtime Query Profile feature (v3.1+) collects and reports Query Profile data at fixed intervals during execution, providing real-time insight into query progress and bottlenecks. When Query Profile is enabled, Runtime Query Profile is automatically activated with a default reporting interval of 10 seconds. Adjust the interval with `runtime_profile_report_interval`: ```sql SET runtime_profile_report_interval = 30; ``` ##### Configurations[​](#configurations "Direct link to Configurations") | Configuration Item | Type | Valid Values | Default | Description | | ------------------------------------ | ----------- | ---------------- | ------- | -------------------------------------------------------------------------------------- | | enable\_profile | Session Var | true/false | false | Enable Query Profile | | pipeline\_profile\_level | Session Var | 1/2 | 1 | 1: merge metrics; 2: retain original structure (disables visualization tools) | | runtime\_profile\_report\_interval | Session Var | Positive integer | 10 | Runtime Query Profile report interval (seconds) | | big\_query\_profile\_threshold | Session Var | String | 0s | Enable Query Profile for queries exceeding this duration (e.g., '30s', '500ms', '60m') | | enable\_statistics\_collect\_profile | FE Dynamic | true/false | false | Enable Query Profile for statistics collection-related queries | #### How to Obtain Query Profile[​](#how-to-obtain-query-profile "Direct link to How to Obtain Query Profile") ##### Via Web UI[​](#via-web-ui "Direct link to Via Web UI") 1. Access `http://:` in your browser. 2. Click **queries** in the top navigation. 3. In the **Finished Queries** list, select the query you want to analyze and click the link in the **Profile** column. ![img](/assets/images/profile-1-e14d11ad744041b7af92ed2ee2b98758.png) You will be redirected to the detailed page of the selected Query Profile. ![img](/assets/images/profile-2-2801dbb3484feef652b8a1287838b0e0.png) ##### Via SQL Function (`get_query_profile`)[​](#via-sql-function-get_query_profile "Direct link to via-sql-function-get_query_profile") Example workflow: * `last_query_id()`: Returns the ID of the most recently executed query in your session. Useful for quickly retrieving the profile of your last query. * `show profilelist;`: Lists recent queries along with their IDs and status. Use this to find the `query_id` needed for profile analysis. * `get_query_profile('')`: Returns the detailed execution profile for the specified query. Use this to analyze how a query was executed and where time or resources were spent. ```sql -- Enable the profiling feature. SET enable_profile = true; -- Run a query that performs a scan and aggregation to generate a meaningful profile. -- (Using a system table ensures this works on any cluster) SELECT count(*) FROM information_schema.columns; -- Get the query_id of the query. SELECT last_query_id(); +--------------------------------------+ | last_query_id() | +--------------------------------------+ | 019b364f-10c4-704c-b79a-af2cc3a77b89 | +--------------------------------------+ -- Get the list of profiles SHOW PROFILELIST; -- Obtain the query profile. SELECT get_query_profile('019b364f-10c4-704c-b79a-af2cc3a77b89')\G ``` ##### In Managed Version[​](#in-managed-version "Direct link to In Managed Version") In StarRocks Managed (Enterprise) environments, you can conveniently access query profiles directly from the query history in the web console. The managed UI provides an intuitive, visual representation of each query's execution profile, making it easy to analyze performance and identify bottlenecks without manual SQL commands. #### Interpret Query Profile[​](#interpret-query-profile "Direct link to Interpret Query Profile") ##### Explain Analyze[​](#explain-analyze "Direct link to Explain Analyze") Most users may find it challenging to analyze the raw text directly. StarRocks provides a [Text-based Query Profile Visualized Analysis](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_text_based_analysis.md) method for a more intuitive understanding. ##### Managed Version[​](#managed-version "Direct link to Managed Version") In the StarRocks Enterprise Edition (EE), the Managed Version provides a built-in visualization tool for query profiles. This tool offers an interactive, graphical interface that makes it much easier to interpret complex query execution details compared to raw text output. **Key features of the visualization tool include:** * **Operator-level breakdown:** View the execution plan as a tree or graph, with each operator's metrics (time, rows, memory) clearly displayed. * **Bottleneck highlighting:** Quickly identify slow or resource-intensive operators through color-coded indicators. * **Drill-down capability:** Click on any operator to see detailed statistics, including input/output rows, CPU time, memory usage, and more. **How to use:** 1. Open the StarRocks Managed web console. 2. Navigate to the **Query** or **Query History** section. 3. Select a query and click the **Profile** or **Visualize** button. 4. Explore the visualized profile to analyze performance and identify optimization opportunities. This visualization tool is exclusive to the Managed/Enterprise Edition and is designed to accelerate troubleshooting and performance tuning for complex workloads. --- ### Explain Analyze This document explains how to obtain and analyze text-based Query Profiles in StarRocks. It will help you understand query performance and find ways to optimize your SQL queries. #### Analyze Profiles of Existing Queries Using ANALYZE PROFILE[​](#analyze-profiles-of-existing-queries-using-analyze-profile "Direct link to Analyze Profiles of Existing Queries Using ANALYZE PROFILE") To analyze the text-based Profile of an existing (historical or running) query in you cluster, you first need to use the [SHOW PROFILELIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/SHOW_PROFILELIST.md) statement to obtain a summary of the query. This command lists all queries that have finished successfully, failed with error, and those are still running (for more than 10 seconds and not yet finished). Through this statement, you can get the corresponding Query ID for subsequent analysis. The syntax is as follows: ```sql SHOW PROFILELIST [LIMIT ]; ``` Examples: ```sql SHOW PROFILELIST; SHOW PROFILELIST LIMIT 5; ``` Output: ```plaintext +--------------------------------------+---------------------+-------+----------+-----------------------------------------------------------------------------------------------------------------------------------+ | QueryId | StartTime | Time | State | Statement | +--------------------------------------+---------------------+-------+----------+-----------------------------------------------------------------------------------------------------------------------------------+ | a40456b2-8428-11ee-8d02-6a32f8c68848 | 2023-11-16 10:34:18 | 21ms | Finished | SELECT ROUTINE_NAME FROM INFORMATION_SCHEMA.ROUTINES\n WHERE ROUTINE_TYPE="FUNCTION" AND ROUTINE_SCHEMA = "None" | | a3fc4060-8428-11ee-8d02-6a32f8c68848 | 2023-11-16 10:34:17 | 39ms | Finished | select TABLE_NAME, COLUMN_NAME from information_schema.columns\n where table_schema = 'Non ... | | a3f7d38d-8428-11ee-8d02-6a32f8c68848 | 2023-11-16 10:34:17 | 15ms | Finished | select connection_id() | | a3efbd3b-8428-11ee-8d02-6a32f8c68848 | 2023-11-16 10:34:17 | 16ms | Finished | select connection_id() | | a26ec286-8428-11ee-8d02-6a32f8c68848 | 2023-11-16 10:34:15 | 269ms | Error | EXPLAIN ANALYZE SELECT c_nation, s_nation, year(lo_orderdate) AS year , SUM(lo_revenue) AS revenue FROM lineorder_flat WHERE ... | +--------------------------------------+---------------------+-------+----------+-----------------------------------------------------------------------------------------------------------------------------------+ ``` Once you have the Query ID, you can proceed with Query Profile analysis using the [ANALYZE PROFILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/ANALYZE_PROFILE.md) statement. The syntax is as follows: ```sql ANALYZE PROFILE FROM '' [, [, ...] ] ``` * `Query_ID`: The ID corresponding to the query, obtained from the `SHOW PROFILELIST` statement. * `Node_ID`: Profile node ID. For nodes whose IDs are specified, StarRocks returns detailed metric information for those node. For nodes whose IDs are not specified, StarRocks only returns the summary information. The Profile includes the following sections: * Summary: Summary information of the Profile. * QueryID * Version information * Query status, including `Finished`, `Error`, and `Running`. * Total query time. * Memory usage * Top 10 nodes with the highest CPU usage. * Top 10 nodes with the highest memory usage. * Session variables who has a different value from the default value. * Fragments: Displays metrics for each node in each Fragment. * Time, memory usage, cost estimate information, and output rows for each node. * Nodes with a time usage percentage exceeding 30% are highlighted in red. * Nodes with a time usage percentage exceeding 15% and less than 30% are highlighted in pink. Example 1: Querying the Query Profile without specifying node ID. ![img](/assets/images/text_based_profile_without_node_id-07ed49a240ec43e9f3a498dcefba5c70.jpeg) Example 2: Querying the Query Profile and specifying node ID as `0`. StarRocks returns all detailed metrics for Node ID `0` and highlights metrics with high usage for easier problem identification. ![img](/assets/images/text_based_profile_with_node_id-ed71ee1e7f25bdbb97752e03e7be34d1.jpeg) In addition, the above methods also support the display and analysis of Runtime Query Profile, that is, Profile generated for running queries. When the Query Profile feature is enabled, you can use this method to obtain the Profile of queries that are currently running for more than 10 seconds. Compared to those of the finished queries, the text-based Query Profile for running queries contains the following information: * Operator status: * ⏳: Operators not started. These operators may not have started execution due to dependency relationships. * 🚀: Running operators. * ✅: Operators that have finished execution. * Overall progress: Calculated based on `number of operators that have finished execution / total number of operators`. Due to the lack of detailed information on data rows, this value may be slightly distorted. * Operator progress: Calculated based on `number of rows processed / total number of rows`. If the total number of rows cannot be calculated, the progress is displayed as `?`. Example: ![img](/assets/images/text_based_runtime_profile-7957caf58242cda61abe63939311a74c.jpeg) #### Simulate a query for Profile Analysis Using EXPLAIN ANALYZE[​](#simulate-a-query-for-profile-analysis-using-explain-analyze "Direct link to Simulate a query for Profile Analysis Using EXPLAIN ANALYZE") StarRocks provides the [EXPLAIN ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/EXPLAIN_ANALYZE.md) statement, allowing you to simulate and analyze the profile of a query directly. The syntax is as follows: ```sql EXPLAIN ANALYZE ``` When executing `EXPLAIN ANALYZE`, StarRocks will, by default, enable the Query Profile feature for the current session. Currently, `EXPLAIN ANALYZE` supports two types of SQL statements: SELECT statements and INSERT INTO statements. You can only simulate and analyze the Query Profile of INSERT INTO statements on internal tables within the default catalog of StarRocks. Please note that when simulating and analyzing the Query Profile of INSERT INTO statements, no actual data will be loaded. By default, the import transaction will be aborted to ensure that no unintended changes are made to the data during the analysis. Example 1: Simulate and analyze a SELECT statement. The query result are discarded. ![img](/assets/images/text_based_explain_analyze_select-54f611db544479a1b4df4aadab264388.jpeg) Example 2: Simulate and analyze an INSERT INTO statement. The loading transaction will be aborted. ![img](/assets/images/text_based_explain_analyze_insert-178eb5df5ee41c8a33f648ece8a9bff8.jpeg) #### Limitations[​](#limitations "Direct link to Limitations") * `EXPLAIN ANALYZE INSERT INTO` statements are only supported for tables in the default catalog. * To achieve better visual effects, the output text contains ANSI characters to provide color, highlighting, and other features. It is recommended to use the MyCLI client. For clients that do not support ANSI features, such as the MySQL client, there may be some slight display disorders. Usually, they will not affect the usage. For example: ![img](/assets/images/text_based_profile_not_aligned-ce88feb8e672741e051b69db71806121.jpeg) --- ### Query Tuning Recipes > A pragmatic playbook: **symptom → root cause → proven fixes**.
Use it when you’ve opened a profile and spotted a red-flag metric but still need to answer “*now what?*”. *** #### 1 · Fast Diagnosis Workflow[​](#1--fast-diagnosis-workflow "Direct link to 1 · Fast Diagnosis Workflow") 1. **Skim the Execution Overview**
If `QueryPeakMemoryUsagePerNode > 80 %` or `QuerySpillBytes > 1 GB`, jump straight to the memory & spill recipes. 2. **Find the slowest Pipeline / Operator**
⟶ In *Query Profile UI* click **Sort by OperatorTotalTime %**.
The hottest operator tells you which recipe block to read next (Scan, Join, Aggregate, …). 3. **Confirm the bottleneck subtype**
Each recipe begins with its *signature* metric pattern. Match those before trying the fixes. *** #### 2 · Recipes by Operator[​](#2--recipes-by-operator "Direct link to 2 · Recipes by Operator") ##### 2.1 OLAP / Connector Scan [\[metrics\]](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_operator_metrics.md#scan-operator)[​](#21-olap--connector-scan--metrics "Direct link to 21-olap--connector-scan--metrics") To facilitate a better understanding of the various metrics within the Scan Operator, the following diagram demonstrates the associations between these metrics and storage structures. ![profile\_scan\_relationship](/assets/images/profile_scan_relationship-416858c52e5b2db2f66c2f3da1f948ac.jpeg) To retrieve data from disk and apply the predicates, the storage engine utilize several techniques: 1. **Data Storage**: Encoded and compressed data is stored on disk in segments, accompanied by various indices. 2. **Index Filtering**: The engine leverages indices such as BitmapIndex, BloomfilterIndex, ZonemapIndex, ShortKeyIndex, and NGramIndex to skip unnecessary data. 3. **Pushdown Predicates**: Simple predicates, like `a > 1`, are pushed down to evaluate on specific columns. 4. **Late Materialization**: Only the required columns and filtered rows are retrieved from disk. 5. **Non-Pushdown Predicates**: Predicates that cannot be pushed down are evaluated. 6. **Projection Expression**: Expressions, such as `SELECT a + 1`, are computed. The Scan Operator utilizes an additional thread pool for executing IO tasks. Therefore, the relationship between time metrics for this node is illustrated below: ![profile\_scan\_time\_relationship](/assets/images/profile_scan_time_relationship-ea4e7d277c9e7a17db3efe120866370a.jpeg) ###### Common performance bottlenecks[​](#common-performance-bottlenecks "Direct link to Common performance bottlenecks") **Cold or slow storage** – When `BytesRead`, `ScanTime`, or `IOTaskExecTime` dominate and disk I/O hovers around 80‑100 %, the scan is hitting cold or under‑provisioned storage. Move hot data to NVMe/SSD and enable the Data Cache. Size it via BE `datacache_*` settings (or legacy `block_cache_*`), and enable scan‑time usage via session `enable_scan_datacache`. **Filter push‑down missing** – If `PushdownPredicates` stays near 0 while `ExprFilterRows` is high, predicates aren’t reaching the storage layer. Rewrite them as simple comparisons (avoid `%LIKE%` and wide `OR` chains) or add zonemap/Bloom indexes or materialized views so they can be pushed down. **Thread‑pool starvation** – A high `IOTaskWaitTime` together with a low `PeakIOTasks` signals saturated I/O concurrency. Enable and size the Data Cache (BE `datacache_*` and session `enable_scan_datacache`), move hot data to faster storage (NVMe/SSD) **Data skew across tablets** – A wide gap between the maximum and minimum `OperatorTotalTime` means some tablets do much more work than others. Re‑bucket on a higher‑cardinality key or increase the bucket count to spread the load. **Rowset/segment fragmentation** – Exploding `RowsetsReadCount`/`SegmentsReadCount` plus a long `SegmentInitTime` indicate many tiny rowsets. Trigger a manual compaction and batch small loads so segments merge up‑front. **Accumulated soft deletes** – A large `DeleteFilterRows` implies heavy soft‑delete usage. Run BE compaction to purge soft deletes. ##### 2.2 Aggregate [\[metrics\]](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_operator_metrics.md#aggregate-operator)[​](#22-aggregate--metrics "Direct link to 22-aggregate--metrics") ![aggregation\_operator](/assets/images/aggregation_operator-a61f7fc9ca608ba1be4a9767eea86b74.png) Aggregate Operator is responsible for executing aggregation functions, `GROUP BY`, and `DISTINCT`. **Multi forms of aggregation algorithm** | Form | When the planner chooses it | Internal data structure | Highlights / caveats | | ---------------------------- | --------------------------------------------- | -------------------------------------------- | -------------------------------------------------------------- | | Hash aggregation | keys fit into memory; cardinality not extreme | Compact hash table with SIMD probing | default path, excellent for modest key counts | | Sorted aggregation | input already ordered on the GROUP BY keys | Simple row comparison + running state | zero hash table cost, often 2-3× faster on probing heavy skews | | Spillable aggregation (3.2+) | hash table outsizes memory limit | Hybrid hash/merge with disk spill partitions | prevents OOM, preserves pipeline parallelism | **Multi-Stage Distributed Aggregation** In StarRocks the aggregation is implemented in distributed manner, which can be multi-stage depends on the query pattern and optimizer decision. ```text ┌─────────┐ ┌──────────┐ ┌────────────┐ ┌────────────┐ │ Stage 0 │ local │ Stage 1 │ shard/ │ Stage 2 │ gather/│ Stage 3 │ final │ Partial │───► │ Update │ hash │ Merge │ shard │ Finalize │ output └─────────┘ └──────────┘ └────────────┘ └────────────┘ ``` | Stages | When Used | What Happens | | --------------------------------------------------- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | One-stage | The `DISTRIBUTED BY` is a subset of `GROUP BY`, the partitions are colocated | Partial aggregates immediately become the final result. | | Two-stage (local + global) | Typical distributed `GROUP BY` | Stage 0 inside each BE collapses duplicates adaptively; Stage 1 shuffles data based on `GROUP BY` then perform global aggregation | | Three-stage (local + shuffle + final) | Heavy `DISTINCT` and high-cardinality `GROUP BY` | Stage 0 as above; Stage 1 shuffles by `GROUP BY`, then aggregate by `GROUP BY` and `DISTINCT`; Stage 2 merges partial state as `GROUP BY` | | Four-stage (local + partial + intermediate + final) | Heavy `DISTINCT` and low-cardinality `GROUP BY` | Introduce an additional stage to shuffle by `GROUP BY` and `DISTINCT` to avoid single-point bottleneck | ###### Common performance bottlenecks[​](#common-performance-bottlenecks-1 "Direct link to Common performance bottlenecks") **High‑cardinality GROUP BY** – When `HashTableSize` or `HashTableMemoryUsage` balloons toward the memory limit, the grouping key is too wide or too distinct. Enable sorted streaming aggregation (`enable_streaming_preaggregation = true`), create a roll‑up materialized view, or cast wide string keys to `INT`. **Shuffle skew** – Large differences in `HashTableSize` or `InputRowCount` across fragments reveal an unbalanced shuffle. Add a salt column to the key or use the `DISTINCT [skew]` hint so rows distribute evenly. **State‑heavy aggregate functions** – If `AggregateFunctions` dominates runtime and the functions include `HLL_`, `BITMAP_`, or `COUNT(DISTINCT)`, enormous state objects are being moved around. Pre‑compute HLL/bitmap sketches during ingestion or switch to approximate variants. **Partial aggregation degraded** – A huge `InputRowCount` with modest `AggComputeTime`, plus massive `BytesSent` in the upstream EXCHANGE, means pre‑aggregation was bypassed. Force it back on with `SET streaming_preaggregation_mode = "force_preaggregation"`. **Expensive key expressions** – When `ExprComputeTime` rivals `AggComputeTime`, the GROUP BY keys are computed row by row. Materialize those expressions in a sub‑query or promote them to generated columns. ##### 2.3 Join [\[metrics\]](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_operator_metrics.md#join-operator)[​](#23-join--metrics "Direct link to 23-join--metrics") ![join\_operator](/assets/images/join_operator-260010cf2d3e1088eee85c6fab0f7c1e.png) Join Operator is responsible for implementing explicit join or implicit joins. During execution the join operator is split into Build (hash-table construction) and Probe phases that run in parallel inside the pipeline engine. Vector chunks (up to 4096 rows) are batch-hashed with SIMD; consumed keys generate runtime filters—Bloom or IN filters—pushed back to upstream scans to cut probe input early. **Join Strategies** StarRocks relies on a vectorized, pipeline-friendly hash-join core that can be wired into four physical strategies the cost-based optimizer weighs at plan time: | Strategy | When the optimizer picks it | What makes it fast | | ------------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | | Colocate Join | Both tables belong to the same colocation group (identical bucket keys, bucket count, and replica layout).  | No network shuffle: each BE joins only its local buckets. | | Bucket-Shuffle Join | One of join tables has the same bucket key with join key | Only need to shuffle one join table, which can reduce the network cost | | Broadcast Join | Build side is very small (row/byte thresholds or explicit hint).  | Small table is replicated to every probe node; avoids shuffling large table. | | Shuffle (Hash) Join | General case, keys don’t align. | Hash-partition each row on the join key so probes are balanced across BEs. | ###### Common performance bottlenecks[​](#common-performance-bottlenecks-2 "Direct link to Common performance bottlenecks") **Oversized build side** – Spikes in `BuildHashTableTime` and `HashTableMemoryUsage` show the build side has outgrown memory. Swap probe/build tables, pre‑filter the build table, or enable hash spilling. **Cache‑unfriendly probe** – When `SearchHashTableTime` dominates, the probe side is not cache‑efficient. Sort the probe rows on the join keys and enable runtime filters. **Shuffle skew** – If a single fragment’s `ProbeRows` dwarfs all others, the data is skewed. Switch to a higher‑cardinality key or append a salt such as `key || mod(id, 16)`. **Accidental broadcast** – Join type **BROADCAST** with huge `BytesSent` means a table you thought was small isn’t. Lower `broadcast_row_limit` or enforce shuffle with the `SHUFFLE` hint. **Missing runtime filters** – A tiny `JoinRuntimeFilterEvaluate` together with full‑table scans suggests runtime filters never propagated. Rewrite the join as pure equality and make sure the column types line up. **Non‑equi fallback** – When the operator type is `CROSS` or `NESTLOOP`, an inequality or function prevents a hash join. Add a true equality predicate or pre‑filter the larger table. ##### 2.4 Exchange (Network) [\[metrics\]](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_operator_metrics.md#exchange-operator)[​](#24-exchange-network--metrics "Direct link to 24-exchange-network--metrics") **Oversized shuffle or broadcast** – If `NetworkTime` exceeds 30 % and `BytesSent` is large, the query is shipping too much data. Re‑evaluate the join strategy and reduce the shuffle/broadcast volume (e.g., enforce shuffle instead of broadcast, or pre‑filter upstream). **Receiver backlog** – High `WaitTime` in the sink with sender queues that stay full indicates the receiver cannot keep up. Increase the receiver thread pool (`brpc_num_threads`) and confirm NIC bandwidth and QoS settings. **Enable exchange compression** – When network bandwidth is the bottleneck, compress exchange payloads. Set `SET transmission_compression_type = 'zstd';` and optionally increase `SET transmission_encode_level = 7;` to enable adaptive column encoding. Expect higher CPU usage in exchange for reduced bytes on the wire. ##### 2.5 Sort / Merge / Window[​](#25-sort--merge--window "Direct link to 2.5 Sort / Merge / Window") For ease of understanding various metrics, Merge can be represented as the following state mechanism: ```plaintext ┌────────── PENDING ◄──────────┐ │ │ │ │ ├──────────────◄───────────────┤ │ │ ▼ │ INIT ──► PREPARE ──► SPLIT_CHUNK ──► FETCH_CHUNK ──► FINISHED ▲ | | one traverse from leaf to root | ▼ PROCESS ``` **Sort spilling** – When `MaxBufferedBytes` rises above roughly 2 GB or `SpillBytes` is non‑zero, the sort phase is spilling to disk. Add a `LIMIT`, pre‑aggregate upstream, or raise `sort_spill_threshold` if the machine has enough memory. **Merge starvation** – A high `PendingStageTime` tells you the merge is waiting for upstream chunks. Optimize the producer operator first or enlarge pipeline buffers. **Wide window partitions** – Huge `PeakBufferedRows` inside a window operator point to very broad partitions or an ORDER BY lacking frame limits. Partition more granularly, add `RANGE BETWEEN` bounds, or materialize intermediate aggregates. *** #### 3 · Memory & Spill Cheatsheet[​](#3--memory--spill-cheatsheet "Direct link to 3 · Memory & Spill Cheatsheet") | Threshold | What to watch | Practical action | | --------------------------------- | --------------------------------------------- | --------------------------------------------------------------------- | | **80 %** of BE memory | `QueryPeakMemoryUsagePerNode` | Lower session `exec_mem_limit` or add BE RAM | | Spill detected (`SpillBytes` > 0) | `QuerySpillBytes`, per-operator `SpillBlocks` | Increase memory limit; upgrade to SR 3.2+ for hybrid hash/merge spill | *** #### 4 · Template for Your Post-mortem[​](#4--template-for-your-post-mortem "Direct link to 4 · Template for Your Post-mortem") ````text 1. Symptom – Slow stage: Aggregate (OperatorTotalTime 68 %) – Red-flag: HashTableMemoryUsage 9 GB (> exec_mem_limit) 2. Root cause – GROUP BY high-cardinality UUID 3. Fix applied – Added sorted streaming agg + roll-up MV 4. Outcome – Query runtime ↓ from 95 s ➜ 8 s; memory peak 0.7 GB``` ```` --- ### Schema Tuning Recipes This document provides practical tips and best practices for optimizing query performance in StarRocks through effective schema design and foundational table choices. By understanding how different table types, keys, and distribution strategies impact query execution, you can significantly improve both speed and resource efficiency. Use these guidelines to make informed decisions when designing schemas, selecting table types, and tuning your StarRocks environment for high-performance analytics. #### Table Type Selection[​](#table-type-selection "Direct link to Table Type Selection") StarRocks supports four table types: Duplicate Key table, Aggregate table, Unique Key table, and Primary Key table. All of them are sorted by KEY. * `AGGREGATE KEY`: When records with the same AGGREGATE KEY is loaded into StarRocks, the old and new records are aggregated. Currently, Aggregate tables supports the following aggregate functions: SUM, MIN, MAX, and REPLACE. Aggregate tables support aggregating data in advance, facilitating business statements and multi-dimensional analyses. * `DUPLICATE KEY`: You only need to specify the sort key for a DUPLICATE KEY table. Records with the same DUPLICATE KEY exist at the same time. It is suitable for analyses that do not involve aggregating data in advance. * `UNIQUE KEY`: When records with the same UNIQUE KEY is loaded into StarRocks, the new record overwrites the old one. A UNIQUE KEY tables is similar to an Aggregate table with REPLACE function. Both are suitable for analyses involving constant updates. * `PRIMARY KEY`: Primary Key tables guarantee the uniqueness of records, and allow you to perform realtime updating. ```sql CREATE TABLE site_visit ( siteid INT, city SMALLINT, username VARCHAR(32), pv BIGINT SUM DEFAULT '0' ) AGGREGATE KEY(siteid, city, username) DISTRIBUTED BY HASH(siteid); CREATE TABLE session_data ( visitorid SMALLINT, sessionid BIGINT, visittime DATETIME, city CHAR(20), province CHAR(20), ip varchar(32), browser CHAR(20), url VARCHAR(1024) ) DUPLICATE KEY(visitorid, sessionid) DISTRIBUTED BY HASH(sessionid, visitorid); CREATE TABLE sales_order ( orderid BIGINT, status TINYINT, username VARCHAR(32), amount BIGINT DEFAULT '0' ) UNIQUE KEY(orderid) DISTRIBUTED BY HASH(orderid); CREATE TABLE sales_order ( orderid BIGINT, status TINYINT, username VARCHAR(32), amount BIGINT DEFAULT '0' ) PRIMARY KEY(orderid) DISTRIBUTED BY HASH(orderid); ``` #### Colocate Table[​](#colocate-table "Direct link to Colocate Table") To speed up queries, tables with the same distribution can use a common bucketing column. In that case, data can be joined locally without being transferred across the cluster during the `join` operation. ```sql CREATE TABLE colocate_table ( visitorid SMALLINT, sessionid BIGINT, visittime DATETIME, city CHAR(20), province CHAR(20), ip varchar(32), browser CHAR(20), url VARCHAR(1024) ) DUPLICATE KEY(visitorid, sessionid) DISTRIBUTED BY HASH(sessionid, visitorid) PROPERTIES( "colocate_with" = "group1" ); ``` For more information about colocate join and replica management, see [Colocate join](https://docs.starrocks.io/docs/using_starrocks/Colocate_join.md) #### Flat table and star schema[​](#flat-table-and-star-schema "Direct link to Flat table and star schema") StarRocks supports star schema, which is more flexible in modelling than flat tables. You can create a view to replace flat tables during modelling and then query data from multiple tables to accelerate queries. Flat tables have the following drawbacks: * Costly dimension updates because a flat table usually contains a massive number of dimensions. Each time a dimension is updated, the entire table must be updated. The situation exacerbates as the update frequency increases. * High maintenance cost because flat tables require additional development workloads, storage space, and data backfilling operations. * High data ingestion cost because a flat table has many fields and an Aggregate table may contain even more key fields. During data loading, more fields need to be sorted, which prolongs data loading. If you have high requirements on query concurrency or low latency, you can still use flat tables. #### Partition and bucket[​](#partition-and-bucket "Direct link to Partition and bucket") StarRocks supports two levels of partitioning: the first level is RANGE partition and the second level is HASH bucket. * RANGE partition: RANGE partition is used to divide data into different intervals (can be understood as dividing the original table into multiple sub-tables). Most users choose to set partitions by time, which has the following advantages: * Easier to distinguish between hot and cold data * Be able to leverage StarRocks tiered storage (SSD + SATA) * Faster to delete data by partition * HASH bucket: Divides data into different buckets according to the hash value. * It is recommended to use a column with a high degree of discrimination for bucketing to avoid data skew. * To facilitate data recovery, it is recommended to keep the size of compressed data in each bucket between 100 MB to 1 GB. We recommend you configure an appropriate number of buckets when you create a table or add a partition. * Random bucketing is not recommended. You must explicitly specify the HASH bucketing column when you create a table. #### Sparse index and bloomfilter index[​](#sparse-index-and-bloomfilter-index "Direct link to Sparse index and bloomfilter index") StarRocks stores data in an ordered manner and builds sparse indexes at a granularity of 1024 rows. StarRocks selects a fixed-length prefix (currently 36 bytes) in the schema as the sparse index. When creating a table, it is recommended to place common filter fields at the beginning of the schema declaration. Fields with the highest differentiation and query frequency must be placed first. A VARCHAR field must placed at the end of a sparse index because the index gets truncated from the VARCHAR field. If the VARCHAR field appears first, the index may be less than 36 bytes. Use the above `site_visit` table as an example. The table has four columns: `siteid, city, username, pv`. The sort key contains three columns `siteid, city, username`, which occupy 4, 2, and 32 bytes respectively. So the prefix index (sparse index) can be the first 30 bytes of `siteid + city + username`. In addition to sparse indexes, StarRocks also provides bloomfilter indexes, which are effective for filtering columns with high discrimination. If you want to place VARCHAR fields before other fields, you can create bloomfilter indexes. #### Inverted Index[​](#inverted-index "Direct link to Inverted Index") StarRocks adopts Bitmap Indexing technology to support inverted indexes that can be applied to all columns of the Duplicate Key table and the key column of the Aggregate table and Unique Key table. Bitmap Index is suitable for columns with a small value range, such as gender, city, and province. As the range expands, the bitmap index expands in parallel. #### Materialized view (rollup)[​](#materialized-view-rollup "Direct link to Materialized view (rollup)") A rollup is essentially a materialized index of the original table (base table). When creating a rollup, only some columns of the base table can be selected as the schema, and the order of the fields in the schema can be different from that of the base table. Below are some use cases of using a rollup: * Data aggregation in the base table is not high, because the base table has fields with high differentiation. In this case, you may consider selecting some columns to create rollups. Use the above `site_visit` table as an example: ```sql site_visit(siteid, city, username, pv) ``` `siteid` may lead to poor data aggregation. If you need to frequently calculate PVs by city, you can create a rollup with only `city` and `pv`. ```sql ALTER TABLE site_visit ADD ROLLUP rollup_city(city, pv); ``` * The prefix index in the base table cannot be hit, because the way the base table is built cannot cover all the query patterns. In this case, you may consider creating a rollup to adjust the column order. Use the above `session_data` table as an example: ```sql session_data(visitorid, sessionid, visittime, city, province, ip, browser, url) ``` If there are cases where you need to analyze visits by `browser` and `province` in addition to `visitorid`, you can create a separate rollup: ```sql ALTER TABLE session_data ADD ROLLUP rollup_browser(browser,province,ip,url) DUPLICATE KEY(browser,province); ``` #### Schema change[​](#schema-change "Direct link to Schema change") There are three ways to change schemas in StarRocks: sorted schema change, direct schema change, and linked schema change. * Sorted schema change: Change the sorting of a column and reorder the data. For example, deleting a column in a sorted schema leads to data reorder. `ALTER TABLE site_visit DROP COLUMN city;` * Direct schema change: Transform the data instead of reordering it, for example, changing the column type or adding a column to a sparse index. `ALTER TABLE site_visit MODIFY COLUMN username varchar(64);` * Linked schema change: Complete changes without transforming data, for example, adding columns. `ALTER TABLE site_visit ADD COLUMN click bigint SUM default '0';` It is recommended to choose an appropriate schema when you create tables to accelerate schema changes. --- ### Table clustering A thoughtful sort‑key is the highest‑leverage physical‑design knob in StarRocks. This guide explains how the sort key works under the hood, the systemic benefits it unlocks, and a concrete playbook for picking an effective key for your own workload. #### Example[​](#example "Direct link to Example") Suppose you run a telemetry system that receives billions of rows per day, each tagged with a `device_id` and `ts` (timestamp). Defining `ORDER BY (device_id, ts)` on your fact table ensures: * Point queries on `device_id` return in milliseconds. * Dashboards filter recent time windows for each device prune most data. * Aggregations like `GROUP BY device_id` benefit from streaming aggregation. * Compression improves due to runs of nearby timestamps per device. This simple two-column sort key `ORDER BY (device_id, ts)` delivers I/O reduction, CPU savings, and more stable query performance across billions of rows. ```sql CREATE TABLE telemetry ( device_id VARCHAR, ts DATETIME, value DOUBLE ) ENGINE=OLAP PRIMARY KEY(device_id, ts) PARTITION BY date_trunc('day', ts) DISTRIBUTED BY HASH(device_id) BUCKETS 16 ORDER BY (device_id, ts); ``` *** #### Benefits in Depth[​](#benefits-in-depth "Direct link to Benefits in Depth") 1. Massive I/O Elimination—Segment & Page Pruning How it works: Each segment and 64 KB page stores min/max values for all columns. If a predicate falls outside that range, StarRocks skips the entire chunk and never touches the disk. Example: ```sql SELECT count(*) FROM events WHERE tenant_id = 42 AND ts BETWEEN '2025-05-01' AND '2025-05-07'; ``` With `ORDER BY (tenant_id, ts)` only the segments whose first key equals 42 are considered, and within them only the pages whose ts window overlaps those seven days. A 100 B‑row table may scan less than 1 B rows, turning minutes into seconds. *** 2. Millisecond Point  Look‑Ups—Sparse Prefix Index How it works: A sparse prefix index stores every ~1 Kth sort‑key value. A binary search lands on the right page, then a single disk read (often already cached) returns the row. Example: ```sql SELECT * FROM orders WHERE order_id = 982347234; ``` With `ORDER BY (order_id)` the probe needs ≈ 50 key comparisons across a 50 B‑row table—sub‑10 ms latency even on cold data cache. *** 3. Faster Sorted Aggregation How it works: When the sort key aligns with the GROUP BY clause, StarRocks performs streaming aggregation as it scans—no sorting or hash-table needed. This sorted-aggregation plan scans rows in sort-key order and emits groups on the fly, exploiting CPU cache locality and skipping intermediate materialisation. Example: ```sql SELECT device_id, COUNT(*) FROM telemetry WHERE ts BETWEEN '2025-01-01' AND '2025-01-31' GROUP BY device_id; ``` If the table is `ORDER BY (device_id, ts)`, the engine groups rows as they stream in—without building a hash table or re-sorting. For high-cardinality keys like device\_id, this can reduce both CPU and memory usage dramatically. Streaming aggregation with sorted input typically improves throughput by 2–3× over hash-aggregation for large group cardinalities. *** 4. Higher  Compression &  Hotter  Caches How it works: Sorted data shows small deltas or long runs, accelerating dictionary, RLE, and frame‑of‑reference encodings. Compact pages stream sequentially through CPU caches. Example: A telemetry table sorted by (device\_id, ts) achieved 1.8 × better compression (LZ4) and 25 % lower CPU/scan than the same data ingested unsorted. *** #### How the Sort Key Works[​](#how-the-sort-key-works "Direct link to How the Sort Key Works") The impact of a sort key starts the moment a row is written and persists through every read‑time optimisation. This section walks through that lifecycle—write path ➜ storage hierarchy ➜ segment internals ➜ read path—to show how each layer compounds the value of ordering. 1. Write Path 1. Ingest: rows land in a MemTable, are sorted by the declared sort key, and then flushed as a new Rowset containing one or more ordered Segments. 2. Compaction: Background cumulative/base jobs merge many small Rowsets into larger ones, reclaiming deletes and lowering segment count without re‑sorting, because every source Rowset already shares the same order. 3. Replication: Each Tablet (the shard that owns the Rowset) is synchronously replicated to peer Back‑End nodes, guaranteeing that sorted order is consistent across replicas. ![write path steps](/assets/images/table_clustering-1-33d2c6e3d94f21ba5f3cd79263a2f3e1.png) 2. Storage Hierarchy | Object | What it is | Why is matters for the sort key | | --------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | Partition | A coarse‑grained logical slice of a table (e.g., date or tenant\_id). | Enables planner‑time partition pruning and isolates lifecycle ops (TTL, bulk load). | | Tablet | A hash/random bucket within a partition, independently replicated across Back‑End nodes. | Unit whose rows are physically ordered by the sort key; all intra‑partition pruning starts here. | | MemTable | In‑memory write buffer (~96 MB) that sorts by the declared key before flushing to disk. | Guarantees that every on‑disk Segment is already ordered—No external sort needed later. | | Rowset | Immutable bundle of one or more Segments produced by a flush, streaming load, or compaction cycle. | Append‑only design lets StarRocks ingest concurrently while readers stay lock‑free. | | Segment | Self‑contained columnar file (~512 MB) inside a Rowset carrying data pages plus pruning indexes. | Segment‑level zone‑maps and prefix indexes rely on the order established at the MemTable stage. | 3. Inside a Segment File ![write path steps](/assets/images/table_clustering-2-63760bef534000a53042e303a88d795d.png) Each Segment is self‑describing. From top to bottom you’ll find: * Column data pages 64 KB blocks encoded (Dictionary, RLE, Delta) and compressed (LZ4 default). * Ordinal index Maps a row ordinal → page offset so the engine can jump directly to page n. * Zone‑map index min, max, and has\_null per page and for the whole Segment—first line of defence for pruning. * Short‑key (prefix) index Sparse binary‑search table of the first 36 bytes of the sort key every ~1 K rows—enables millisecond point/range seeks. * Footer & magic number Offsets to every index and a checksum for integrity; lets StarRocks memory‑map just the tail to discover the rest. Because the pages are already sorted by the key, those indexes are tiny yet brutally effective. 4. Read Path 1. Partition pruning (planner‑time) If the WHERE clause constrains the partition key (e.g. `dt BETWEEN '2025‑05‑01' AND '2025‑05‑07'`), the optimizer opens only matching partition directories. 2. Tablet pruning(planner-time) When the equality filter includes the hash distribution column, StarRocks computes the target tablet IDs and schedules just those Tablets. 3. Prefix‑index seek A sparse short‑key index on the leading sort columns homes in on the exact segment or page. 4. Zone‑map pruning min/max metadata per Segment and 64 KB page discards blocks that miss the predicate window. 5. Vectorized scan & late materialization Surviving column pages stream sequentially through CPU caches; only referenced rows\&columns are materialized, keeping memory tight. Because data is committed in key order on every flush, each read‑time pruning layer compounds on the one before it, delivering sub‑second scans on multi‑billion‑row tables. *** #### How to Choose an Effective Sort Key[​](#how-to-choose-an-effective-sort-key "Direct link to How to Choose an Effective Sort Key") 1. Start with Workload Intelligence Analyse the top‑N query patterns first: * Equality predicates (`=` / `IN`). Columns almost always filtered by equality make ideal leading candidates. * Range predicates. Timestamps and numeric ranges typically follow equality columns in the sort key. * Aggregation keys. If a range column also appears in `GROUP BY` clauses, placing it earlier in the key (after selective filters) can enable sorted aggregation. * Join/group-by keys. Consider placing join or grouping keys early if they are common Measure column cardinality: high‑cardinality columns (millions of distinct values) prune best. 2. Heuristics & Rules of Thumb 1. Order rule: (high‑selectivity equality columns) → (primary range column) → (cluster helpers). 2. Cardinality Ordering: Placing low-cardinality columns before high-cardinality columns can enhance data compression. 3. Width: Keep to 3‑5 columns. Very wide keys slow ingest and overflow the 36‑byte prefix‑index limit. 4. String columns: a long leading string column may occupy most or all of the 36-byte limit in the prefix index, preventing subsequent columns in the sort key from being indexed effectively. This reduces the pruning power of the prefix index and degrades point query performance. 3. Coordinate with Other Design Knobs * Partitioning: Choose a partition key that is coarser than the leading sort column (for example, `PARTITION BY date`, `ORDER BY (tenant_id, ts)`). That way partition pruning removes whole date ranges first, and sort pruning cleans up inside. * Bucketing: Using the same columns for both bucketing and clustering serves different purposes. Bucketing ensures even data distribution across the cluster, while sorting enables efficient I/O elimination. * Table type: Primary-Key tables default to using the primary key as the sort key, but they can also specify additional columns to refine physical order and enhance pruning. Aggregate and duplicate tables should follow the analytic predicate-driven sort key strategies discussed above. *** 4. Reference Templates | Scenario | Partition | Sort Key | Rationale | | ----------------- | ----------------------------- | --------------------- | -------------------------------------------------------------------- | | B2C Orders | date\_trunc('day', order\_ts) | (user\_id, order\_ts) | Most queries filter by user first, then recent time ranges. | | IoT Telemetry | date\_trunc('day', ts) | (device\_id, ts) | Device‑scoped time‑series reads dominate. | | SaaS Multi‑Tenant | tenant\_id | (dt, event\_id) | Tenant isolation via partition; sort clusters by day for dashboards. | | Dimension Lookup | none | (dim\_id) | Small table, pure point look‑ups—single‑column suffices. | *** #### Conclusion[​](#conclusion "Direct link to Conclusion") A well‑designed sort key trades a small, predictable ingest overhead for dramatic improvements in scan latency, storage efficiency, and CPU utilisation. By grounding your choice in workload realities, respecting cardinality, and validating with `EXPLAIN`, you can keep StarRocks humming even as data and user counts grow 10× and beyond. --- ## Data_source ### Data Lakehouse ![DLA](/assets/images/1.1-8-dla-c67d601d709b092317fa11eb64ac1783.png) In addition to efficient analytics of local data, StarRocks can work as the compute engine to analyze data stored in data lakes such as Apache Hudi, Apache Iceberg, and Delta Lake. One of the key features of StarRocks is its external catalog, which acts as the linkage to an externally maintained metastore. This functionality provides users with the capability to query external data sources seamlessly, eliminating the need for data migration. As such, users can analyze data from different systems such as HDFS and Amazon S3, in various file formats such as Parquet, ORC, and CSV, etc. The preceding figure shows a data lake analytics scenario where StarRocks is responsible for data computing and analysis, and the data lake is responsible for data storage, organization, and maintenance. Data lakes allow users to store data in open storage formats and use flexible schemas to produce reports on "single source of truth" for various BI, AI, ad-hoc, and reporting use cases. StarRocks fully leverages the advantages of its vectorization engine and CBO, significantly improving the performance of data lake analytics. #### Key ideas[​](#key-ideas "Direct link to Key ideas") * Open Data Formats: Supports a variety of data types, including JSON, Parquet, and Avro, facilitating the storage and processing of both structured and unstructured data. * Metadata Management: Implements a shared metadata layer, often utilizing formats like the Iceberg table format, to organize and govern data efficiently. * Diverse Query Engines: Incorporates multiple engines, like enhanced versions of Presto and Spark, to cater to various analytics and AI use cases. * Governance and Security: Features robust built-in mechanisms for data security, privacy, and compliance, ensuring data integrity and trustworthiness. #### Advantages of Data Lakehouse architecture[​](#advantages-of-data-lakehouse-architecture "Direct link to Advantages of Data Lakehouse architecture") * Flexibility and Scalability: Seamlessly manages diverse data types and scales with the organization’s needs. * Cost-Effectiveness: Offers an economical alternative for data storage and processing, compared to traditional methods. * Enhanced Data Governance: Improves data control, management, and integrity, ensuring reliable and secure data handling. * AI and Analytics Readiness: Perfectly suited for complex analytical tasks, including machine learning and AI-driven data processing. #### StarRocks approach[​](#starrocks-approach "Direct link to StarRocks approach") The key things to consider are: * Standardizing the integration with catalog, or metadata services * Elastic scalability of compute nodes * Flexible caching mechanisms *** #### Catalogs[​](#catalogs "Direct link to Catalogs") StarRocks has two types of catalogs, internal and external. The internal catalog contains metadata for data stored within StarRocks databases. External catalogs are used to work with data stored externally, including the data managed by Hive, Iceberg, Delta Lake, and Hudi. There are many other external systems, links are in the More Information section at the bottom of the page. #### Compute node (CN) scaling[​](#compute-node-cn-scaling "Direct link to Compute node (CN) scaling") Separation of storage and compute reduces the complexity of scaling. Since the StarRocks compute nodes are only storing local cache, nodes can be added or removed based on load. #### Data cache[​](#data-cache "Direct link to Data cache") Cache on the compute nodes is optional. If your compute nodes are spinning up and down quickly based on quickly changing load patterns or your queries are often only on the most recent data it might not make sense to cache data. *** #### [🗃️ Catalog](https://docs.starrocks.io/docs/data_source/catalog.md) [14 items](https://docs.starrocks.io/docs/data_source/catalog.md) #### [🗃️ Data cache](https://docs.starrocks.io/docs/data_source/data_cache.md) [3 items](https://docs.starrocks.io/docs/data_source/data_cache.md) #### [📄️ External table](https://docs.starrocks.io/docs/data_source/External_table.md) [The External Table feature is no longer recommended except for certain corner usage cases, and might be deprecated in future releases.](https://docs.starrocks.io/docs/data_source/External_table.md) #### [📄️ File external table](https://docs.starrocks.io/docs/data_source/file_external_table.md) [File external table is a special type of external table.](https://docs.starrocks.io/docs/data_source/file_external_table.md) #### [📄️ Data lake FAQ](https://docs.starrocks.io/docs/data_source/datalake_faq.md) [FAQ for common data lake analytics issues in StarRocks, including catalog setup and query problems.](https://docs.starrocks.io/docs/data_source/datalake_faq.md) #### [📄️ Feature Support](https://docs.starrocks.io/docs/data_source/feature-support-data-lake-analytics.md) [From v2.3 onwards, StarRocks supports managing external data sources and analyzing data in data lakes via external catalogs.](https://docs.starrocks.io/docs/data_source/feature-support-data-lake-analytics.md) --- ### Catalog overview #### [📄️ Default catalog](https://docs.starrocks.io/docs/data_source/catalog/default_catalog.md) [StarRocks 2.3 and later provide an internal catalog to manage the internal data of StarRocks.](https://docs.starrocks.io/docs/data_source/catalog/default_catalog.md) #### [📄️ Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md) [A Hive catalog is a kind of external catalog that is supported by StarRocks from v2.4 onwards.](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md) #### [🗃️ Iceberg catalog](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md) [7 items](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md) #### [📄️ Hudi catalog](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md) [A Hudi catalog is a kind of external catalog that enables you to query data from Apache Hudi without ingestion.](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md) #### [📄️ Delta Lake catalog](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md) [Query data directly from Delta Lake](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md) #### [📄️ JDBC catalog](https://docs.starrocks.io/docs/data_source/catalog/jdbc_catalog.md) [StarRocks supports JDBC catalogs from v3.0 onwards.](https://docs.starrocks.io/docs/data_source/catalog/jdbc_catalog.md) #### [📄️ Elasticsearch catalog](https://docs.starrocks.io/docs/data_source/catalog/elasticsearch_catalog.md) [StarRocks supports Elasticsearch catalogs from v3.1 onwards.](https://docs.starrocks.io/docs/data_source/catalog/elasticsearch_catalog.md) #### [📄️ Paimon catalog](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog.md) [StarRocks supports Paimon catalogs from v3.1 onwards.](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog.md) #### [📄️ Unified catalog](https://docs.starrocks.io/docs/data_source/catalog/unified_catalog.md) [A unified catalog in StarRocks handles tables from Hive, Iceberg, Hudi, Delta Lake, and Kudu in a single catalog.](https://docs.starrocks.io/docs/data_source/catalog/unified_catalog.md) #### [📄️ MaxCompute catalog](https://docs.starrocks.io/docs/data_source/catalog/maxcompute_catalog.md) [StarRocks supports Alibaba Cloud MaxCompute (previously known as ODPS) catalogs from v3.3 onwards.](https://docs.starrocks.io/docs/data_source/catalog/maxcompute_catalog.md) #### [📄️ Kudu catalog](https://docs.starrocks.io/docs/data_source/catalog/kudu_catalog.md) [StarRocks supports Kudu catalogs from v3.3 onwards.](https://docs.starrocks.io/docs/data_source/catalog/kudu_catalog.md) #### [📄️ Iceberg Lakehouse tutorial](https://docs.starrocks.io/docs/data_source/catalog/icebergtutorial.md) [Iceberg lakehouse in Docker](https://docs.starrocks.io/docs/data_source/catalog/icebergtutorial.md) #### [📄️ Query external data](https://docs.starrocks.io/docs/data_source/catalog/query_external_data.md) [How to query data from external sources in StarRocks using external catalogs.](https://docs.starrocks.io/docs/data_source/catalog/query_external_data.md) #### [📄️ Overview](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md) [StarRocks supports the catalog feature from v2.3 onwards.](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md) --- ### Overview This topic describes what a catalog is, and how to manage and query internal data and external data by using a catalog. StarRocks supports the catalog feature from v2.3 onwards. Catalogs enable you to manage internal and external data in one system and offer a flexible way for you to easily query and analyze data that is stored in various external systems. #### Basic concepts[​](#basic-concepts "Direct link to Basic concepts") * **Internal data**: refers to the data stored in StarRocks. * **External data**: refers to the data stored in an external data source, such as Apache Hive™, Apache Iceberg, Apache Hudi, Delta Lake, and JDBC. #### Catalog[​](#catalog "Direct link to Catalog") Currently, StarRocks provides two types of catalogs: internal catalog and external catalog. ![figure1](/assets/images/3.8.1-e149861830788363c2a5fc3b7ddd4291.png) * **Internal catalog** manages internal data of StarRocks. For example, if you execute the CREATE DATABASE or CREATE TABLE statements to create a database or a table, the database or table is stored in the internal catalog. Each StarRocks cluster has only one internal catalog named [default\_catalog](https://docs.starrocks.io/docs/data_source/catalog/default_catalog.md). * **External catalog** acts like a link to externally managed metastores, which grants StarRocks direct access to external data sources. You can query external data directly with zero data loading or migration. Currently, StarRocks supports the following types of external catalogs: * [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md): used to query data from Hive. * [Iceberg catalog](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md): used to query data from Iceberg. * [Hudi catalog](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md): used to query data from Hudi. * [Delta Lake catalog](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md): used to query data from Delta Lake. * [JDBC catalog](https://docs.starrocks.io/docs/data_source/catalog/jdbc_catalog.md): used to query data from JDBC-compatible data sources. * [Elasticsearch catalog](https://docs.starrocks.io/docs/data_source/catalog/elasticsearch_catalog.md): used to query data from Elasticsearch. Elasticsearch catalogs are supported from v3.1 onwards. * [Paimon catalog](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog.md): used to query data from Paimon. Paimon catalogs are supported from v3.1 onwards. * [Unified catalog](https://docs.starrocks.io/docs/data_source/catalog/unified_catalog.md): used to query data from from Hive, Iceberg, Hudi, and Delta Lake data sources as a unified data source. Unified catalogs are supported from v3.2 onwards. StarRocks interacts with the following two components of external data sources when you query external data: * **Metastore service**: used by the FEs to access the metadata of external data sources. The FEs generate a query execution plan based on the metadata. * **Data storage system**: used to store external data. Both distributed file systems and object storage systems can be used as data storage systems to store data files in various formats. After the FEs distribute the query execution plan to all BEs or CNs, all BEs or CNs scan the target external data in parallel, perform calculations, and then return the query result. #### Access catalog[​](#access-catalog "Direct link to Access catalog") You can use the [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) statement to switch to a specified catalog in the current session. Then, you can query data by using that catalog. #### Query data[​](#query-data "Direct link to Query data") ##### Query internal data[​](#query-internal-data "Direct link to Query internal data") To query data in StarRocks, see [Default catalog](https://docs.starrocks.io/docs/data_source/catalog/default_catalog.md). ##### Query external data[​](#query-external-data "Direct link to Query external data") To query data from external data sources, see [Query external data](https://docs.starrocks.io/docs/data_source/catalog/query_external_data.md). ##### Cross-catalog query[​](#cross-catalog-query "Direct link to Cross-catalog query") To perform a cross-catalog federated query from your current catalog, specify the data you want to query in the `catalog_name.database_name` or `catalog_name.database_name.table_name` format. * Query `hive_table` in `hive_db` when the current session is `default_catalog.olap_db`. ```sql SELECT * FROM hive_catalog.hive_db.hive_table; ``` * Query `olap_table` in `default_catalog` when the current session is `hive_catalog.hive_db`. ```sql SELECT * FROM default_catalog.olap_db.olap_table; ``` * Perform a JOIN query on `hive_table` in `hive_catalog` and `olap_table` in `default_catalog` when the current session is `hive_catalog.hive_db`. ```sql SELECT * FROM hive_table h JOIN default_catalog.olap_db.olap_table o WHERE h.id = o.id; ``` * Perform a JOIN query on `hive_table` in `hive_catalog` and `olap_table` in `default_catalog` by using a JOIN clause when the current session is another catalog. ```sql SELECT * FROM hive_catalog.hive_db.hive_table h JOIN default_catalog.olap_db.olap_table o WHERE h.id = o.id; ``` --- ### Default catalog This topic describes what the default catalog is, and how to query the internal data of StarRocks by using the default catalog. StarRocks 2.3 and later provide an internal catalog to manage the internal data of StarRocks. Each StarRocks cluster has only one internal catalog named `default_catalog`. Currently, you cannot modify the name of the internal catalog or create a new internal catalog. #### Query internal data[​](#query-internal-data "Direct link to Query internal data") 1. Connect your StarRocks cluster. * If you use the MySQL client to connect the StarRocks cluster, you go to `default_catalog` by default after connecting. * If you use JDBC to connect the StarRocks cluster, you can go directly to the destination database in the default catalog by specifying `default_catalog.db_name` when connecting. 2. (Optional) Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view databases: ```sql SHOW DATABASES; ``` Or ```sql SHOW DATABASES FROM ; ``` 3. (Optional) Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to switch to the destination catalog in the current session: ```sql SET CATALOG ; ``` Then, use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify the active database in the current session: ```sql USE ; ``` Or, you can use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to directly go to the active database in the destination catalog: ```sql USE .; ``` 4. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query internal data: ```sql SELECT * FROM ; ``` If you do not specify the active database in the preceding steps, you can directly specify it in a select query: ```sql SELECT * FROM .; ``` Or ```sql SELECT * FROM default_catalog..; ``` #### Examples[​](#examples "Direct link to Examples") To query data in `olap_db.olap_table`, you can perform one of the following operations: ```sql USE olap_db; SELECT * FROM olap_table limit 1; ``` Or ```sql SELECT * FROM olap_db.olap_table limit 1; ``` Or ```sql SELECT * FROM default_catalog.olap_db.olap_table limit 1; ``` #### References[​](#references "Direct link to References") To query data from external data sources, see [Query external data](https://docs.starrocks.io/docs/data_source/catalog/query_external_data.md). --- ### Delta Lake catalog A Delta Lake catalog is a kind of external catalog that enables you to query data from Delta Lake without ingestion. Also, you can directly transform and load data from Delta Lake by using [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) based on Delta Lake catalogs. StarRocks supports Delta Lake catalogs from v2.5 onwards. To ensure successful SQL workloads on your Delta Lake cluster, your StarRocks cluster must be able to access the storage system and metastore of your Delta Lake cluster. StarRocks supports the following storage systems and metastores: * Distributed file system (HDFS) or object storage like AWS S3, Microsoft Azure Storage, Google GCS, or other S3-compatible storage system (for example, MinIO) * Metastore like Hive metastore or AWS Glue note If you choose AWS S3 as storage, you can use HMS or AWS Glue as metastore. If you choose any other storage system, you can only use HMS as metastore. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The file format of Delta Lake that StarRocks supports is Parquet. Parquet files support the following compression formats: SNAPPY, LZ4, ZSTD, GZIP, and NO\_COMPRESSION. #### Integration preparations[​](#integration-preparations "Direct link to Integration preparations") Before you create a Delta Lake catalog, make sure your StarRocks cluster can integrate with the storage system and metastore of your Delta Lake cluster. ##### AWS IAM[​](#aws-iam "Direct link to AWS IAM") If your Delta Lake cluster uses AWS S3 as storage or AWS Glue as metastore, choose your suitable authentication method and make the required preparations to ensure that your StarRocks cluster can access the related AWS cloud resources. The following authentication methods are recommended: * Instance profile * Assumed role * IAM user Of the above-mentioned three authentication methods, instance profile is the most widely used. For more information, see [Preparation for authentication in AWS IAM](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#preparation-for-iam-user-based-authentication). ##### HDFS[​](#hdfs "Direct link to HDFS") If you choose HDFS as storage, configure your StarRocks cluster as follows: * (Optional) Set the username that is used to access your HDFS cluster and Hive metastore. By default, StarRocks uses the username of the FE and BE or CN processes to access your HDFS cluster and Hive metastore. You can also set the username by adding `export HADOOP_USER_NAME=""` at the beginning of the **fe/conf/hadoop\_env.sh** file of each FE and at the beginning of the **be/conf/hadoop\_env.sh** file of each BE or the **cn/conf/hadoop\_env.sh** file of each CN. After you set the username in these files, restart each FE and each BE or CN to make the parameter settings take effect. You can set only one username for each StarRocks cluster. * When you query Delta Lake data, the FEs and BEs or CNs of your StarRocks cluster use the HDFS client to access your HDFS cluster. In most cases, you do not need to configure your StarRocks cluster to achieve that purpose, and StarRocks starts the HDFS client using the default configurations. You need to configure your StarRocks cluster only in the following situations: * High availability (HA) is enabled for your HDFS cluster: Add the **hdfs-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. * View File System (ViewFs) is enabled for your HDFS cluster: Add the **core-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. note If an error indicating an unknown host is returned when you send a query, you must add the mapping between the host names and IP addresses of your HDFS cluster nodes to the **/etc/hosts** path. ##### Kerberos authentication[​](#kerberos-authentication "Direct link to Kerberos authentication") If Kerberos authentication is enabled for your HDFS cluster or Hive metastore, configure your StarRocks cluster as follows: * Run the `kinit -kt keytab_path principal` command on each FE and each BE or CN to obtain Ticket Granting Ticket (TGT) from Key Distribution Center (KDC). To run this command, you must have the permissions to access your HDFS cluster and Hive metastore. Note that accessing KDC with this command is time-sensitive. Therefore, you need to use cron to run this command periodically. * Add `JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"` to the **$FE\_HOME/conf/fe.conf** file of each FE and to the **$BE\_HOME/conf/be.conf** file of each BE or the **$CN\_HOME/conf/cn.conf** file of each CN. In this example, `/etc/krb5.conf` is the save path of the **krb5.conf** file. You can modify the path based on your needs. #### Create a Delta Lake catalog[​](#create-a-delta-lake-catalog "Direct link to Create a Delta Lake catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ( "type" = "deltalake", MetastoreParams, StorageCredentialParams, MetadataUpdateParams ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### catalog\_name[​](#catalog_name "Direct link to catalog_name") The name of the Delta Lake catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### comment[​](#comment "Direct link to comment") The description of the Delta Lake catalog. This parameter is optional. ###### type[​](#type "Direct link to type") The type of your data source. Set the value to `deltalake`. ###### MetastoreParams[​](#metastoreparams "Direct link to MetastoreParams") A set of parameters about how StarRocks integrates with the metastore of your data source. ###### Hive metastore[​](#hive-metastore "Direct link to Hive metastore") If you choose Hive metastore as the metastore of your data source, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "hive", "hive.metastore.uris" = "" ``` note Before querying Delta Lake data, you must add the mapping between the host names and IP addresses of your Hive metastore nodes to the `/etc/hosts` path. Otherwise, StarRocks may fail to access your Hive metastore when you start a query. The following table describes the parameter you need to configure in `MetastoreParams`. | Parameter | Required | Description | | ------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hive.metastore.type | Yes | The type of metastore that you use for your Delta Lake cluster. Set the value to `hive`. | | hive.metastore.uris | Yes | The URI of your Hive metastore. Format: `thrift://:`.
If high availability (HA) is enabled for your Hive metastore, you can specify multiple metastore URIs and separate them with commas (`,`), for example, `"thrift://:,thrift://:,thrift://:"`. | ###### AWS Glue[​](#aws-glue "Direct link to AWS Glue") If you choose AWS Glue as the metastore of your data source, which is supported only when you choose AWS S3 as storage, take one of the following actions: * To choose the instance profile-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "" ``` * To choose the assumed role-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "", "aws.glue.region" = "" ``` * To choose the IAM user-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "" ``` The following table describes the parameters you need to configure in `MetastoreParams`. | Parameter | Required | Description | | ------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hive.metastore.type | Yes | The type of metastore that you use for your Delta Lake cluster. Set the value to `glue`. | | aws.glue.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. | | aws.glue.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS Glue Data Catalog. If you use the assumed role-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.region | Yes | The region in which your AWS Glue Data Catalog resides. Example: `us-west-1`. | | aws.glue.access\_key | No | The access key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.secret\_key | No | The secret key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS Glue and how to configure an access control policy in the AWS IAM Console, see [Authentication parameters for accessing AWS Glue](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-glue). ###### StorageCredentialParams[​](#storagecredentialparams "Direct link to StorageCredentialParams") A set of parameters about how StarRocks integrates with your storage system. This parameter set is optional. If you use HDFS as storage, you do not need to configure `StorageCredentialParams`. If you use AWS S3, other S3-compatible storage system, Microsoft Azure Storage, or Google GCS as storage, you must configure `StorageCredentialParams`. ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") If you choose AWS S3 as storage for your Delta Lake cluster, take one of the following actions: * To choose the instance profile-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "" ``` * To choose the assumed role-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.region" = "" ``` * To choose the IAM user-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ----------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. | | aws.s3.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS S3 bucket. If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.region | Yes | The region in which your AWS S3 bucket resides. Example: `us-west-1`. | | aws.s3.access\_key | No | The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.secret\_key | No | The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in AWS IAM Console, see [Authentication parameters for accessing AWS S3](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-s3). ###### S3-compatible storage system[​](#s3-compatible-storage-system "Direct link to S3-compatible storage system") Delta Lake catalogs support S3-compatible storage systems from v2.5 onwards. If you choose an S3-compatible storage system, such as MinIO, as storage for your Delta Lake cluster, configure `StorageCredentialParams` as follows to ensure a successful integration: ```sql "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ---------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.enable\_ssl | Yes | Specifies whether to enable SSL connection.
Valid values: `true` and `false`. Default value: `true`. | | aws.s3.enable\_path\_style\_access | Yes | Specifies whether to enable path-style access.
Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`.
Path-style URLs use the following format: `https://s3..amazonaws.com//`. For example, if you create a bucket named `DOC-EXAMPLE-BUCKET1` in the US West (Oregon) Region, and you want to access the `alice.jpg` object in that bucket, you can use the following path-style URL: `https://s3.us-west-2.amazonaws.com/DOC-EXAMPLE-BUCKET1/alice.jpg`. | | aws.s3.endpoint | Yes | The endpoint that is used to connect to your S3-compatible storage system instead of AWS S3. | | aws.s3.access\_key | Yes | The access key of your IAM user. | | aws.s3.secret\_key | Yes | The secret key of your IAM user. | ###### Microsoft Azure Storage[​](#microsoft-azure-storage "Direct link to Microsoft Azure Storage") Delta Lake catalogs support Microsoft Azure Storage from v3.0 onwards. ###### Azure Blob Storage[​](#azure-blob-storage "Direct link to Azure Blob Storage") If you choose Blob Storage as storage for your Delta Lake cluster, take one of the following actions: * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | -------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.shared\_key | Yes | The shared key of your Blob Storage account. | * To choose the SAS Token authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | --------------------------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.container | Yes | The name of the blob container that stores your data. | | azure.blob.sas\_token | Yes | The SAS token that is used to access your Blob Storage account. | ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2 "Direct link to Azure Data Lake Storage Gen2") If you choose Data Lake Storage Gen2 as storage for your Delta Lake cluster, take one of the following actions: * To choose the Managed Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_use\_managed\_identity | Yes | Specifies whether to enable the Managed Identity authentication method. Set the value to `true`. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the managed identity. | * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ---------------------------- | ------------ | -------------------------------------------------------------- | | azure.adls2.storage\_account | Yes | The username of your Data Lake Storage Gen2 storage account. | | azure.adls2.shared\_key | Yes | The shared key of your Data Lake Storage Gen2 storage account. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ``` The following table describes the parameters you need to configure `in StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls2.oauth2\_client\_secret | Yes | The value of the new client (application) secret created. | | azure.adls2.oauth2\_client\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | * To choose the Workload Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_token_file" = "", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_token\_file | Yes | The absolute file path to the OAuth2 token file projected into the pod by the Azure Workload Identity webhook. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client ID (application ID) of the Azure AD application (user-assigned managed identity or app registration) associated with the workload identity. | ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1 "Direct link to Azure Data Lake Storage Gen1") If you choose Data Lake Storage Gen1 as storage for your Delta Lake cluster, take one of the following actions: * To choose the Managed Service Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.use_managed_service_identity" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------- | | azure.adls1.use\_managed\_service\_identity | Yes | Specifies whether to enable the Managed Service Identity authentication method. Set the value to `true`. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls1.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls1.oauth2\_credential | Yes | The value of the new client (application) secret created. | | azure.adls1.oauth2\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | ###### Google GCS[​](#google-gcs "Direct link to Google GCS") Delta Lake catalogs support Google GCS from v3.0 onwards. If you choose Google GCS as storage for your Delta Lake cluster, take one of the following actions: * To choose the VM-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.use_compute_engine_service_account" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value example** | **Description** | | ---------------------------------------------- | ----------------- | ----------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | * To choose the service account-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------- | | gcp.gcs.service\_account\_email | "" | `"user@hello.iam.gserviceaccount.com"` | The email address in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the service account. | * To choose the impersonation-based authentication method, configure `StorageCredentialParams` as follows: * Make a VM instance impersonate a service account: ```sql "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value example** | **Description** | | ---------------------------------------------- | ----------------- | ----------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The service account that you want to impersonate. | * Make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account): ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | gcp.gcs.service\_account\_email | "" | `"user@hello.iam.gserviceaccount.com"` | The email address in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the meta service account. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The data service account that you want to impersonate. | ###### MetadataUpdateParams[​](#metadataupdateparams "Direct link to MetadataUpdateParams") A set of parameters about how StarRocks updates the cached metadata of Delta Lake. This parameter set is optional. From v3.3.3 onwards, Delta Lake Catalog supports [Metadata Local Cache and Retrieval](#appendix-metadata-local-cache-and-retrieval). In most cases, you can ignore `MetadataUpdateParams` and do not need to tune the policy parameters in it, because the default values of these parameters already provide you with an out-of-the-box performance. However, if the frequency of data updates in Delta Lake is high, you can tune these parameters to further optimize the performance of automatic asynchronous updates. note In most cases, if your Delta Lake data is updated at a granularity of 1 hour or less, the data update frequency is considered high. | **Parameter** | **Unit** | **Default** | **Description** | | -------------------------------------------------------- | -------- | -------------- | ----------------------------------------------------------------------------------- | | enable\_deltalake\_table\_cache | - | true | Whether to enable Table Cache in the metadata cache for Delta Lake. | | enable\_deltalake\_json\_meta\_cache | - | true | Whether to enable cache for Delta Log JSON files. | | deltalake\_json\_meta\_cache\_ttl\_sec | Second | 48 \* 60 \* 60 | Time-To-Live (TTL) for the Delta Log JSON file cache. | | deltalake\_json\_meta\_cache\_memory\_usage\_ratio | - | 0.1 | The maximum ratio of JVM Heap size occupied by the Delta Log JSON file cache. | | enable\_deltalake\_checkpoint\_meta\_cache | - | true | Whether to enable cache for Delta Log Checkpoint files. | | deltalake\_checkpoint\_meta\_cache\_ttl\_sec | Second | 48 \* 60 \* 60 | Time-To-Live (TTL) for the Delta Log Checkpoint file cache. | | deltalake\_checkpoint\_meta\_cache\_memory\_usage\_ratio | - | 0.1 | The maximum ratio of JVM Heap size occupied by the Delta Log Checkpoint file cache. | ##### Examples[​](#examples "Direct link to Examples") The following examples create a Delta Lake catalog named `deltalake_catalog_hms` or `deltalake_catalog_glue`, depending on the type of metastore you use, to query data from your Delta Lake cluster. ###### HDFS[​](#hdfs-1 "Direct link to HDFS") If you use HDFS as storage, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` ###### AWS S3[​](#aws-s3-1 "Direct link to AWS S3") ###### If you choose instance profile-based credential[​](#if-you-choose-instance-profile-based-credential "Direct link to If you choose instance profile-based credential") * If you use Hive metastore in your Delta Lake cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Delta Lake cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_glue PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` ###### If you choose assumed role-based credential[​](#if-you-choose-assumed-role-based-credential "Direct link to If you choose assumed role-based credential") * If you use Hive metastore in your Delta Lake cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Delta Lake cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_glue PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "arn:aws:iam::081976408565:role/test_glue_role", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` ###### If you choose IAM user-based credential[​](#if-you-choose-iam-user-based-credential "Direct link to If you choose IAM user-based credential") * If you use Hive metastore in your Delta Lake cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Delta Lake cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_glue PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` ###### S3-compatible storage system[​](#s3-compatible-storage-system-1 "Direct link to S3-compatible storage system") Use MinIO as an example. Run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.enable_ssl" = "true", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ); ``` ###### Microsoft Azure Storage[​](#microsoft-azure-storage-1 "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage-1 "Direct link to Azure Blob Storage") * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` * If you choose the SAS Token authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1-1 "Direct link to Azure Data Lake Storage Gen1") * If you choose the Managed Service Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.use_managed_service_identity" = "true" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2-1 "Direct link to Azure Data Lake Storage Gen2") * If you choose the Managed Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` * If you choose the Workload Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### Google GCS[​](#google-gcs-1 "Direct link to Google GCS") * If you choose the VM-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true" ); ``` * If you choose the service account-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ); ``` * If you choose the impersonation-based authentication method: * If you make a VM instance impersonate a service account, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ); ``` * If you make a service account impersonate another service account, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ); ``` #### View Delta Lake catalogs[​](#view-delta-lake-catalogs "Direct link to View Delta Lake catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of a Delta Lake catalog named `deltalake_catalog_glue`: ```sql SHOW CREATE CATALOG deltalake_catalog_glue; ``` #### Switch to a Delta Lake Catalog and a database in it[​](#switch-to-a-delta-lake-catalog-and-a-database-in-it "Direct link to Switch to a Delta Lake Catalog and a database in it") You can use one of the following methods to switch to a Delta Lake catalog and a database in it: * Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to specify a Delta Lake catalog in the current session, and then use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify an active database: ```sql -- Switch to a specified catalog in the current session: SET CATALOG -- Specify the active database in the current session: USE ``` * Directly use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to switch to a Delta Lake catalog and a database in it: ```sql USE . ``` #### Drop a Delta Lake catalog[​](#drop-a-delta-lake-catalog "Direct link to Drop a Delta Lake catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop an external catalog. The following example drops a Delta Lake catalog named `deltalake_catalog_glue`: ```sql DROP Catalog deltalake_catalog_glue; ``` #### View the schema of a Delta Lake table[​](#view-the-schema-of-a-delta-lake-table "Direct link to View the schema of a Delta Lake table") You can use one of the following syntaxes to view the schema of a Delta Lake table: * View schema ```sql DESC[RIBE] .. ``` * View schema and location from the CREATE statement ```sql SHOW CREATE TABLE .. ``` #### Query a Delta Lake table[​](#query-a-delta-lake-table "Direct link to Query a Delta Lake table") 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your Delta Lake cluster: ```sql SHOW DATABASES FROM ``` 2. [Switch to a Delta Lake Catalog and a database in it](#switch-to-a-delta-lake-catalog-and-a-database-in-it). 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT count(*) FROM LIMIT 10 ``` #### Load data from Delta Lake[​](#load-data-from-delta-lake "Direct link to Load data from Delta Lake") Suppose you have an OLAP table named `olap_tbl`, you can transform and load data like below: ```sql INSERT INTO default_catalog.olap_db.olap_tbl SELECT * FROM deltalake_table ``` #### Configure metadata cache and update strategy[​](#configure-metadata-cache-and-update-strategy "Direct link to Configure metadata cache and update strategy") From v3.3.3 onwards, Delta Lake Catalog supports [Metadata Local Cache and Retrieval](#appendix-metadata-local-cache-and-retrieval). You can configure the Delta Lake metadata cache refresh through the following FE parameters: | **Configuration item** | **Default** | **Description** | | -------------------------------------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_background\_refresh\_connector\_metadata | `true` | Whether to enable the periodic Delta Lake metadata cache refresh. After it is enabled, StarRocks polls the metastore (Hive Metastore or AWS Glue) of your Delta Lake cluster, and refreshes the cached metadata of the frequently accessed Delta Lake catalogs to perceive data changes. `true` indicates to enable the Delta Lake metadata cache refresh, and `false` indicates to disable it. | | background\_refresh\_metadata\_interval\_millis | `600000` | The interval between two consecutive Delta Lake metadata cache refreshes. Unit: millisecond. | | background\_refresh\_metadata\_time\_secs\_since\_last\_access\_secs | `86400` | The expiration time of a Delta Lake metadata cache refresh task. For the Delta Lake catalog that has been accessed, if it has not been accessed for more than the specified time, StarRocks stops refreshing its cached metadata. For the Delta Lake catalog that has not been accessed, StarRocks will not refresh its cached metadata. Unit: second. | #### Appendix: Metadata Local Cache and Retrieval[​](#appendix-metadata-local-cache-and-retrieval "Direct link to Appendix: Metadata Local Cache and Retrieval") Because the repeated decompression and parsing of metadata files can introduce unnecessary delays, StarRocks employs a new metadata cache strategy of caching deserialized memory objects. By storing these deserialized files in memory, the system can bypass the decompression and parsing stages for subsequent queries. This caching mechanism allows direct access to the required metadata, significantly reducing retrieval times. As a result, the system becomes more responsive and better suited to meet high query demands and materialized view rewriting needs. You can configure this behavior through the Catalog property [MetadataUpdateParams](#metadataupdateparams) and [relevant configuration items](#configure-metadata-cache-and-update-strategy). #### Feature support[​](#feature-support "Direct link to Feature support") Currently, Delta Lake catalogs support the following table features: * V2 Checkpoint (From v3.3.0 onwards) * Timestamp without Timezone (From v3.3.1 onwards) * Column mapping (From v3.3.6 onwards) * Deletion Vector (From v3.4.1 onwards) --- ### Elasticsearch catalog StarRocks supports Elasticsearch catalogs from v3.1 onwards. StarRocks and Elasticsearch are both popular analytical systems with distinct strengths. StarRocks excels in large-scale distributed computing and supports querying data from Elasticsearch through external tables. Elasticsearch is known for its full-text search capabilities. The combination of StarRocks and Elasticsearch provides a more comprehensive OLAP solution. With Elasticsearch catalogs, you can directly analyze all indexed data in your Elasticsearch cluster by using SQL statements on StarRocks without the need for data migration. Unlike catalogs for other data sources, an Elasticsearch catalog has only one database named `default_db` in it upon creation. Each Elasticsearch index is automatically mapped to a data table and mounted to the `default_db` database. #### Create an Elasticsearch catalog[​](#create-an-elasticsearch-catalog "Direct link to Create an Elasticsearch catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ("key"="value", ...) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### `catalog_name`[​](#catalog_name "Direct link to catalog_name") The name of the Elasticsearch catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### `comment`[​](#comment "Direct link to comment") The description of the Elasticsearch catalog. This parameter is optional. ###### PROPERTIES[​](#properties "Direct link to PROPERTIES") The properties of the Elasticsearch catalog. The following table describes the properties supported for Elasticsearch catalogs. | Parameter | Required | Default value | Description | | ---------------------- | -------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hosts | Yes | None | The connection address of the Elasticsearch cluster. You can specify one or more addresses. StarRocks can parse the Elasticsearch version and index shard allocation from this address. StarRocks communicates with your Elasticsearch cluster based on the address returned by the `GET /_nodes/http` API operation. Therefore, the value of the `hosts` parameter must be the same as the address returned by the `GET /_nodes/http` API operation. Otherwise, BEs or CNs may not be able to communicate with your Elasticsearch cluster. | | type | Yes | None | The type of the data source. Set this parameter to `es` when you create an Elasticsearch catalog. | | user | No | Empty | The username that is used to log in to the Elasticsearch cluster with HTTP basic authentication enabled. Make sure that you have permissions to access paths such as `/cluster/state/ nodes/http` and have permissions to read the index. | | password | No | Empty | The password that is used to log in to the Elasticsearch cluster. | | es.type | No | \_doc | The type of the index. If you want to query data in Elasticsearch 8 and later versions, you do not need to configure this parameter because the mapping types have been removed in Elasticsearch 8 and later versions. | | es.nodes.wan.only | No | FALSE | Specifies whether StarRocks only uses the addresses specified by `hosts` to access the Elasticsearch cluster and fetch data.- `true`: StarRocks only uses the addresses specified by `hosts` to access the Elasticsearch cluster and fetch data and does not sniff data nodes on which the shards of the Elasticsearch index reside. If StarRocks cannot access the addresses of the data nodes inside the Elasticsearch cluster, you need to set this parameter to `true`.
- `false`: StarRocks uses the addresses specified by `hosts` to sniff data nodes on which the shards of the Elasticsearch cluster indexes reside. After StarRocks generates a query execution plan, BEs or CNs directly access the data nodes inside the Elasticsearch cluster to fetch data from the shards of indexes. If StarRocks can access the addresses of the data nodes inside the Elasticsearch cluster, we recommend that you retain the default value `false`. | | es.net.ssl | No | FALSE | Specifies whether the HTTPS protocol can be used to access the Elasticsearch cluster. Only StarRocks v2.4 and later support configuring this parameter.- `true`: Both the HTTPS and HTTP protocols can be used to access your Elasticsearch cluster.
- `false`: Only the HTTP protocol can be used to access your Elasticsearch cluster. | | enable\_docvalue\_scan | No | TRUE | Specifies whether to obtain the values of the target fields from Elasticsearch columnar storage. In most cases, reading data from columnar storage outperforms reading data from row storage. | | enable\_keyword\_sniff | No | TRUE | Specifies whether to sniff TEXT-type fields in Elasticsearch based on KEYWORD-type fields. If this parameter is set to `false`, StarRocks performs matching after tokenization. | ##### Examples[​](#examples "Direct link to Examples") The following example creates an Elasticsearch catalog named `es_test`: ```sql CREATE EXTERNAL CATALOG es_test COMMENT 'test123' PROPERTIES ( "type" = "es", "es.type" = "_doc", "hosts" = "https://xxx:9200", "es.net.ssl" = "true", "user" = "admin", "password" = "xxx", "es.nodes.wan.only" = "true" ); ``` #### Predicate pushdown[​](#predicate-pushdown "Direct link to Predicate pushdown") StarRocks supports pushing the predicates specified in queries against Elasticsearch tables down to Elasticsearch for execution. This minimizes the distance between the query engine and the storage source and improves query performance. The following table lists the operators that can be pushed down to Elasticsearch. | SQL syntax | Elasticsearch syntax | | -------------- | ---------------------- | | `=` | term query | | `in` | terms query | | `>=, <=, >, <` | range | | `and` | bool.filter | | `or` | bool.should | | `not` | bool.must\_not | | `not in` | bool.must\_not + terms | | `esquery` | ES Query DSL | #### Query examples[​](#query-examples "Direct link to Query examples") The `esquery()` function can be used to push Elasticsearch queries, such as match and geoshape queries, that cannot be expressed in SQL down to Elasticsearch for filtering and processing. In the `esquery()` function, the first parameter that specifies a column name is used to associate with the index, and the second parameter is an Elasticsearch query's Elasticsearch Query DSL-based JSON representation enclosed in curly braces (`{}`). The JSON representation can and must have only one root key, such as `match`, `geo_shape`, or `bool`. * Match query ```sql SELECT * FROM es_table WHERE esquery(k4, '{ "match": { "k4": "StarRocks on elasticsearch" } }'); ``` * Geoshape query ```sql SELECT * FROM es_table WHERE esquery(k4, '{ "geo_shape": { "location": { "shape": { "type": "envelope", "coordinates": [ [ 13, 53 ], [ 14, 52 ] ] }, "relation": "within" } } }'); ``` * Boolean query ```sql SELECT * FROM es_table WHERE esquery(k4, ' { "bool": { "must": [ { "terms": { "k1": [ 11, 12 ] } }, { "terms": { "k2": [ 100 ] } } ] } }'); ``` #### Usage notes[​](#usage-notes "Direct link to Usage notes") * From v5.x onwards, Elasticsearch adopts a different data scanning method. StarRocks only supports querying data from Elasticsearch v5.x and later. * StarRocks only supports querying data from Elasticsearch clusters that have HTTP basic authentication enabled. * Some queries, such as `count()`-involved queries, run much slower on StarRocks than on Elasticsearch, because Elasticsearch can directly read the metadata related to the specified number of documents that meet the query conditions without the need to filter the requested data. --- ### Hive catalog A Hive catalog is a kind of external catalog that is supported by StarRocks from v2.4 onwards. Within Hive catalogs, you can: * Directly query data stored in Hive without the need to manually create tables. * Use [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) or asynchronous materialized views (which are supported from v2.5 onwards) to process data stored in Hive and load the data into StarRocks. * Perform operations on StarRocks to create or drop Hive databases and tables, or sink data from StarRocks tables to Parquet-formatted (supported from v3.2 onwards) and ORC- or Textfile-formatted (supported from v3.3 onwards) Hive tables by using [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). To ensure successful SQL workloads on your Hive cluster, your StarRocks cluster must be able to access the storage system and metastore of your Hive cluster. StarRocks supports the following storage systems and metastores: * Distributed file system (HDFS) or object storage like AWS S3, Microsoft Azure Storage, Google GCS, or other S3-compatible storage system (for example, MinIO) * Metastore like Hive metastore or AWS Glue note If you choose AWS S3 as storage, you can use HMS or AWS Glue as metastore. If you choose any other storage system, you can only use HMS as metastore. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * StarRocks supports queries against Hive tables in Parquet, ORC, Textfile, Avro, RCFile, and SequenceFile file formats: * Parquet files support the following compression formats: SNAPPY, LZ4, ZSTD, GZIP, and NO\_COMPRESSION. From v3.1.5 onwards, Parquet files also support the LZO compression format. * ORC files support the following compression formats: ZLIB, SNAPPY, LZO, LZ4, ZSTD, and NO\_COMPRESSION. * Textfile files support the LZO compression format from v3.1.5 onwards. * The data types of Hive that StarRocks does not support are INTERVAL, BINARY, and UNION. Additionally, StarRocks does not support the MAP and STRUCT data types for Textfile-formatted Hive tables. * StarRocks supports sinking data to Parquet-formatted (supported from v3.2 onwards) and ORC- or Textfile-formatted (supported from v3.3 onwards) Hive tables: * Parquet and ORC files support the following compression formats: NO\_COMPRESSION, SNAPPY, LZ4, ZSTD, and GZIP. * Textfile files support the NO\_COMPRESSION compression format. You can use the table property [`compression_codec`](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md#properties) or the system variable [`connector_sink_compression_codec`](https://docs.starrocks.io/docs/sql-reference/System_variable.md#connector_sink_compression_codec) to specify the compression algorithm used for sinking data to Hive tables. When writing to a Hive table, if the table's properties include a compression codec, StarRocks will preferentially use that algorithm to compress the written data. Otherwise, it will use the compression algorithm set in the system variable `connector_sink_compression_codec`. #### Integration preparations[​](#integration-preparations "Direct link to Integration preparations") Before you create a Hive catalog, make sure your StarRocks cluster can integrate with the storage system and metastore of your Hive cluster. ##### AWS IAM[​](#aws-iam "Direct link to AWS IAM") If your Hive cluster uses AWS S3 as storage or AWS Glue as metastore, choose your suitable authentication method and make the required preparations to ensure that your StarRocks cluster can access the related AWS cloud resources. The following authentication methods are recommended: * Instance profile * Assumed role * IAM user Of the above-mentioned three authentication methods, instance profile is the most widely used. For more information, see [Preparation for authentication in AWS IAM](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#preparations). ##### HDFS[​](#hdfs "Direct link to HDFS") If you choose HDFS as storage, configure your StarRocks cluster as follows: * (Optional) Set the username that is used to access your HDFS cluster and Hive metastore. By default, StarRocks uses the username of the FE and BE or CN processes to access your HDFS cluster and Hive metastore. You can also set the username by adding `export HADOOP_USER_NAME=""` at the beginning of the **fe/conf/hadoop\_env.sh** file of each FE and at the beginning of the **be/conf/hadoop\_env.sh** file of each BE or the **cn/conf/hadoop\_env.sh** file of each CN. After you set the username in these files, restart each FE and each BE or CN to make the parameter settings take effect. You can set only one username for each StarRocks cluster. * When you query Hive data, the FEs and BEs or CNs of your StarRocks cluster use the HDFS client to access your HDFS cluster. In most cases, you do not need to configure your StarRocks cluster to achieve that purpose, and StarRocks starts the HDFS client using the default configurations. You need to configure your StarRocks cluster only in the following situations: * High availability (HA) is enabled for your HDFS cluster: Add the **hdfs-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. * View File System (ViewFs) is enabled for your HDFS cluster: Add the **core-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. note If an error indicating an unknown host is returned when you send a query, you must add the mapping between the host names and IP addresses of your HDFS cluster nodes to the **/etc/hosts** path. ##### Kerberos authentication[​](#kerberos-authentication "Direct link to Kerberos authentication") If Kerberos authentication is enabled for your HDFS cluster or Hive metastore, configure your StarRocks cluster as follows: * Run the `kinit -kt keytab_path principal` command on each FE and each BE or CN to obtain Ticket Granting Ticket (TGT) from Key Distribution Center (KDC). To run this command, you must have the permissions to access your HDFS cluster and Hive metastore. Note that accessing KDC with this command is time-sensitive. Therefore, you need to use cron to run this command periodically. * Add `JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"` to the **$FE\_HOME/conf/fe.conf** file of each FE and to the **$BE\_HOME/conf/be.conf** file of each BE or the **$CN\_HOME/conf/cn.conf** file of each CN. In this example, `/etc/krb5.conf` is the save path of the **krb5.conf** file. You can modify the path based on your needs. #### Create a Hive catalog[​](#create-a-hive-catalog "Direct link to Create a Hive catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ( "type" = "hive", GeneralParams, MetastoreParams, StorageCredentialParams, MetadataUpdateParams ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### catalog\_name[​](#catalog_name "Direct link to catalog_name") The name of the Hive catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### comment[​](#comment "Direct link to comment") The description of the Hive catalog. This parameter is optional. ###### type[​](#type "Direct link to type") The type of your data source. Set the value to `hive`. ###### GeneralParams[​](#generalparams "Direct link to GeneralParams") A set of general parameters. The following table describes the parameters you can configure in `GeneralParams`. | Parameter | Required | Description | | -------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_recursive\_listing | No | Specifies whether StarRocks reads data from a table and its partitions and from the subdirectories within the physical locations of the table and its partitions. Valid values: `true` and `false`. Default value: `true`. The value `true` specifies to recursively list subdirectories, and the value `false` specifies to ignore subdirectories. | ###### MetastoreParams[​](#metastoreparams "Direct link to MetastoreParams") A set of parameters about how StarRocks integrates with the metastore of your data source. ###### Hive metastore[​](#hive-metastore "Direct link to Hive metastore") If you choose Hive metastore as the metastore of your data source, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "hive", "hive.metastore.uris" = "" ``` note Before querying Hive data, you must add the mapping between the host names and IP addresses of your Hive metastore nodes to the `/etc/hosts` path. Otherwise, StarRocks may fail to access your Hive metastore when you start a query. The following table describes the parameter you need to configure in `MetastoreParams`. | Parameter | Required | Description | | ------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hive.metastore.type | Yes | The type of metastore that you use for your Hive cluster. Set the value to `hive`. | | hive.metastore.uris | Yes | The URI of your Hive metastore. Format: `thrift://:`.
If high availability (HA) is enabled for your Hive metastore, you can specify multiple metastore URIs and separate them with commas (`,`), for example, `"thrift://:,thrift://:,thrift://:"`. | ###### AWS Glue[​](#aws-glue "Direct link to AWS Glue") If you choose AWS Glue as the metastore of your data source, which is supported only when you choose AWS S3 as storage, take one of the following actions: * To choose the instance profile-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "" ``` * To choose the assumed role-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "", "aws.glue.region" = "" ``` * To choose the IAM user-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "" ``` The following table describes the parameters you need to configure in `MetastoreParams`. | Parameter | Required | Description | | ------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hive.metastore.type | Yes | The type of metastore that you use for your Hive cluster. Set the value to `glue`. | | aws.glue.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication. Valid values: `true` and `false`. Default value: `false`. | | aws.glue.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS Glue Data Catalog. If you use the assumed role-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.region | Yes | The region in which your AWS Glue Data Catalog resides. Example: `us-west-1`. | | aws.glue.access\_key | No | The access key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.secret\_key | No | The secret key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | | hive.metastore.glue.catalogid | No | The ID of the AWS Glue Data Catalog to use. When not specified, the catalog in the current AWS account is used. You must specify this parameter when you need to access a Glue Data Catalog in a different AWS account (cross-account access). | | aws.glue.resource\_share\_type | No | Controls which databases `SHOW DATABASES` lists, by setting the `ResourceShareType` sent to the AWS Glue `GetDatabases` API. Case-insensitive. Valid values: `FOREIGN` (databases shared with your account from another account through AWS Resource Access Manager), `FEDERATED` (databases that reference an external data source such as a JDBC connection), and `ALL` (local databases plus both of the above). When unset, only local databases are listed. This parameter affects listing only. StarRocks still queries a database using the single account set by `hive.metastore.glue.catalogid`, so a listed `FOREIGN` database remains unqueryable unless you also set `hive.metastore.glue.catalogid` to the owner account or create a [resource link](https://docs.aws.amazon.com/lake-formation/latest/dg/resource-links-about.html) for it in your own Data Catalog; a `FEDERATED` database is not a Hive-compatible database and cannot be queried through this catalog. | For information about how to choose an authentication method for accessing AWS Glue and how to configure an access control policy in the AWS IAM Console, see [Authentication parameters for accessing AWS Glue](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-glue). ###### StorageCredentialParams[​](#storagecredentialparams "Direct link to StorageCredentialParams") A set of parameters about how StarRocks integrates with your storage system. This parameter set is optional. If you use HDFS as storage, you do not need to configure `StorageCredentialParams`. If you use AWS S3, other S3-compatible storage system, Microsoft Azure Storage, or Google GCS as storage, you must configure `StorageCredentialParams`. ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") If you choose AWS S3 as storage for your Hive cluster, take one of the following actions: * To choose the instance profile-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "" ``` * To choose the assumed role-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.region" = "" ``` * To choose the IAM user-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ----------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. | | aws.s3.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS S3 bucket. If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.region | Yes | The region in which your AWS S3 bucket resides. Example: `us-west-1`. | | aws.s3.access\_key | No | The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.secret\_key | No | The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in AWS IAM Console, see [Authentication parameters for accessing AWS S3](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-s3). ###### S3-compatible storage system[​](#s3-compatible-storage-system "Direct link to S3-compatible storage system") Hive catalogs support S3-compatible storage systems from v2.5 onwards. If you choose an S3-compatible storage system, such as MinIO, as storage for your Hive cluster, configure `StorageCredentialParams` as follows to ensure a successful integration: ```sql "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ---------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.enable\_ssl | Yes | Specifies whether to enable SSL connection.
Valid values: `true` and `false`. Default value: `true`. | | aws.s3.enable\_path\_style\_access | Yes | Specifies whether to enable path-style access.
Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`.
Path-style URLs use the following format: `https://s3..amazonaws.com//`. For example, if you create a bucket named `DOC-EXAMPLE-BUCKET1` in the US West (Oregon) Region, and you want to access the `alice.jpg` object in that bucket, you can use the following path-style URL: `https://s3.us-west-2.amazonaws.com/DOC-EXAMPLE-BUCKET1/alice.jpg`. | | aws.s3.endpoint | Yes | The endpoint that is used to connect to your S3-compatible storage system instead of AWS S3. | | aws.s3.access\_key | Yes | The access key of your IAM user. | | aws.s3.secret\_key | Yes | The secret key of your IAM user. | ###### Microsoft Azure Storage[​](#microsoft-azure-storage "Direct link to Microsoft Azure Storage") Hive catalogs support Microsoft Azure Storage from v3.0 onwards. ###### Azure Blob Storage[​](#azure-blob-storage "Direct link to Azure Blob Storage") If you choose Blob Storage as storage for your Hive cluster, take one of the following actions: * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | -------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.shared\_key | Yes | The shared key of your Blob Storage account. | * To choose the SAS Token authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | --------------------------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.container | Yes | The name of the blob container that stores your data. | | azure.blob.sas\_token | Yes | The SAS token that is used to access your Blob Storage account. | ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2 "Direct link to Azure Data Lake Storage Gen2") If you choose Data Lake Storage Gen2 as storage for your Hive cluster, take one of the following actions: * To choose the Managed Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_use\_managed\_identity | Yes | Specifies whether to enable the Managed Identity authentication method. Set the value to `true`. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the managed identity. | * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ---------------------------- | ------------ | -------------------------------------------------------------- | | azure.adls2.storage\_account | Yes | The username of your Data Lake Storage Gen2 storage account. | | azure.adls2.shared\_key | Yes | The shared key of your Data Lake Storage Gen2 storage account. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ``` The following table describes the parameters you need to configure `in StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls2.oauth2\_client\_secret | Yes | The value of the new client (application) secret created. | | azure.adls2.oauth2\_client\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | * To choose the Workload Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_token_file" = "", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_token\_file | Yes | The absolute file path to the OAuth2 token file projected into the pod by the Azure Workload Identity webhook. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client ID (application ID) of the Azure AD application (user-assigned managed identity or app registration) associated with the workload identity. | ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1 "Direct link to Azure Data Lake Storage Gen1") If you choose Data Lake Storage Gen1 as storage for your Hive cluster, take one of the following actions: * To choose the Managed Service Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.use_managed_service_identity" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------- | | azure.adls1.use\_managed\_service\_identity | Yes | Specifies whether to enable the Managed Service Identity authentication method. Set the value to `true`. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls1.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls1.oauth2\_credential | Yes | The value of the new client (application) secret created. | | azure.adls1.oauth2\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | ###### Google GCS[​](#google-gcs "Direct link to Google GCS") Hive catalogs support Google GCS from v3.0 onwards. If you choose Google GCS as storage for your Hive cluster, take one of the following actions: * To choose the VM-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.use_compute_engine_service_account" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | * To choose the service account-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------- | | gcp.gcs.service\_account\_email | "" | "" | The email address in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the service account. | * To choose the impersonation-based authentication method, configure `StorageCredentialParams` as follows: * Make a VM instance impersonate a service account: ```sql "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The service account that you want to impersonate. | * Make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account): ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | gcp.gcs.service\_account\_email | "" | "" | The email address in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the meta service account. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The data service account that you want to impersonate. | ###### MetadataUpdateParams[​](#metadataupdateparams "Direct link to MetadataUpdateParams") A set of parameters about how StarRocks updates the cached metadata of Hive. This parameter set is optional. StarRocks implements the [automatic asynchronous update policy](#appendix-understand-metadata-automatic-asynchronous-update) by default. In most cases, you can ignore `MetadataUpdateParams` and do not need to tune the policy parameters in it, because the default values of these parameters already provide you with an out-of-the-box performance. However, if the frequency of data updates in Hive is high, you can tune these parameters to further optimize the performance of automatic asynchronous updates. note In most cases, if your Hive data is updated at a granularity of 1 hour or less, the data update frequency is considered high. | Parameter | Required | Description | | ------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_metastore\_cache | No | Specifies whether StarRocks caches the metadata of Hive tables. Valid values: `true` and `false`. Default value: `true`. The value `true` enables the cache, and the value `false` disables the cache. | | enable\_remote\_file\_cache | No | Specifies whether StarRocks caches the metadata of the underlying data files of Hive tables or partitions. Valid values: `true` and `false`. Default value: `true`. The value `true` enables the cache, and the value `false` disables the cache. | | metastore\_cache\_refresh\_interval\_sec | No | The time interval at which StarRocks asynchronously updates the metadata of Hive tables or partitions cached in itself. Unit: seconds. Default value: `60`, which is one minute. Since v3.3.0, the default value of this property is changed from `7200` to `60`. | | remote\_file\_cache\_refresh\_interval\_sec | No | The time interval at which StarRocks asynchronously updates the metadata of the underlying data files of Hive tables or partitions cached in itself. Unit: seconds. Default value: `60`. | | metastore\_cache\_ttl\_sec | No | The time interval at which StarRocks automatically discards the metadata of Hive tables or partitions cached in itself. Unit: seconds. Default value: `86400`, which is 24 hours. | | remote\_file\_cache\_ttl\_sec | No | The time interval at which StarRocks automatically discards the metadata of the underlying data files of Hive tables or partitions cached in itself. Unit: seconds. Default value: `129600`, which is 36 hours. | | enable\_cache\_list\_names | No | Specifies whether StarRocks caches Hive partition names. Valid values: `true` and `false`. Default value: `true`. The value `true` enables the cache, and the value `false` disables the cache. | | remote\_file\_cache\_memory\_ratio | No | The maximum memory usage ratio for the remote file cache. Default value: `0.1`, which is 10%. Supported from v3.5.6 onwards. | ##### Examples[​](#examples "Direct link to Examples") The following examples create a Hive catalog named `hive_catalog_hms` or `hive_catalog_glue`, depending on the type of metastore you use, to query data from your Hive cluster. ###### HDFS[​](#hdfs-1 "Direct link to HDFS") If you use HDFS as storage, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` ###### AWS S3[​](#aws-s3-1 "Direct link to AWS S3") ###### Instance profile-based authentication[​](#instance-profile-based-authentication "Direct link to Instance profile-based authentication") * If you use Hive metastore in your Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_glue PROPERTIES ( "type" = "hive", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` ###### Assumed role-based authentication[​](#assumed-role-based-authentication "Direct link to Assumed role-based authentication") * If you use Hive metastore in your Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_glue PROPERTIES ( "type" = "hive", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "arn:aws:iam::081976408565:role/test_glue_role", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` ###### IAM user-based authentication[​](#iam-user-based-authentication "Direct link to IAM user-based authentication") * If you use Hive metastore in your Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_glue PROPERTIES ( "type" = "hive", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` ###### S3-compatible storage system[​](#s3-compatible-storage-system-1 "Direct link to S3-compatible storage system") Use MinIO as an example. Run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.enable_ssl" = "true", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ); ``` ###### Microsoft Azure Storage[​](#microsoft-azure-storage-1 "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage-1 "Direct link to Azure Blob Storage") * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` * If you choose the SAS Token authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1-1 "Direct link to Azure Data Lake Storage Gen1") * If you choose the Managed Service Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.use_managed_service_identity" = "true" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2-1 "Direct link to Azure Data Lake Storage Gen2") * If you choose the Managed Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` * If you choose the Workload Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### Google GCS[​](#google-gcs-1 "Direct link to Google GCS") * If you choose the VM-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true" ); ``` * If you choose the service account-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ); ``` * If you choose the impersonation-based authentication method: * If you make a VM instance impersonate a service account, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ); ``` * If you make a service account impersonate another service account, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ); ``` #### View Hive catalogs[​](#view-hive-catalogs "Direct link to View Hive catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of a Hive catalog named `hive_catalog_glue`: ```sql SHOW CREATE CATALOG hive_catalog_glue; ``` #### Switch to a Hive Catalog and a database in it[​](#switch-to-a-hive-catalog-and-a-database-in-it "Direct link to Switch to a Hive Catalog and a database in it") You can use one of the following methods to switch to a Hive catalog and a database in it: * Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to specify a Hive catalog in the current session, and then use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify an active database: ```sql -- Switch to a specified catalog in the current session: SET CATALOG -- Specify the active database in the current session: USE ``` * Directly use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to switch to a Hive catalog and a database in it: ```sql USE . ``` #### Drop a Hive catalog[​](#drop-a-hive-catalog "Direct link to Drop a Hive catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop an external catalog. The following example drops a Hive catalog named `hive_catalog_glue`: ```sql DROP Catalog hive_catalog_glue; ``` #### View the schema of a Hive table[​](#view-the-schema-of-a-hive-table "Direct link to View the schema of a Hive table") You can use one of the following syntaxes to view the schema of a Hive table: * View schema ```sql DESC[RIBE] .. ``` * View schema and location from the CREATE statement ```sql SHOW CREATE TABLE .. ``` #### Query a Hive table[​](#query-a-hive-table "Direct link to Query a Hive table") 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your Hive cluster: ```sql SHOW DATABASES FROM ``` 2. [Switch to a Hive Catalog and a database in it](#switch-to-a-hive-catalog-and-a-database-in-it). 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT count(*) FROM LIMIT 10 ``` #### Load data from Hive[​](#load-data-from-hive "Direct link to Load data from Hive") Suppose you have an OLAP table named `olap_tbl`, you can transform and load data like below: ```sql INSERT INTO default_catalog.olap_db.olap_tbl SELECT * FROM hive_table ``` #### Grant privileges on Hive tables and views[​](#grant-privileges-on-hive-tables-and-views "Direct link to Grant privileges on Hive tables and views") You can use the [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) statement to grant the privileges on all tables and views within a Hive catalog to a specific role. The command syntax is as follows: ```sql GRANT SELECT ON ALL TABLES IN ALL DATABASES TO ROLE ``` For example, use the following commands to create a role named `hive_role_table`, switch to the Hive catalog `hive_catalog`, and then grant the role `hive_role_table` the privilege to query all tables and views within the Hive catalog `hive_catalog`: ```sql -- Create a role named hive_role_table. CREATE ROLE hive_role_table; -- Switch to the Hive catalog hive_catalog. SET CATALOG hive_catalog; -- Grant the role hive_role_table the privilege to query all tables and views within the Hive catalog hive_catalog. GRANT SELECT ON ALL TABLES IN ALL DATABASES TO ROLE hive_role_table; ``` #### Create a Hive database[​](#create-a-hive-database "Direct link to Create a Hive database") Similar to the internal catalog of StarRocks, if you have the [CREATE DATABASE](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md#catalog) privilege on a Hive catalog, you can use the [CREATE DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/CREATE_DATABASE.md) statement to create a database in that Hive catalog. This feature is supported from v3.2 onwards. note You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). note [Switch to a Hive catalog](#switch-to-a-hive-catalog-and-a-database-in-it), and then use the following statement to create a Hive database in that catalog: ```sql CREATE DATABASE [PROPERTIES ("location" = ":///")] ``` The `location` parameter specifies the file path in which you want to create the database, which can be in either HDFS or cloud storage. * When you use Hive metastore as the metastore of your Hive cluster, the `location` parameter defaults to `/`, which is supported by Hive metastore if you do not specify that parameter at database creation. * When you use AWS Glue as the metastore of your Hive cluster, the `location` parameter does not have a default value, and therefore you must specify that parameter at database creation. The `prefix` varies based on the storage system you use: | **Storage system** | **`Prefix`** **value** | | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | HDFS | `hdfs` | | Google GCS | `gs` | | Azure Blob Storage | - If your storage account allows access over HTTP, the `prefix` is `wasb`.
- If your storage account allows access over HTTPS, the `prefix` is `wasbs`. | | Azure Data Lake Storage Gen1 | `adl` | | Azure Data Lake Storage Gen2 | - If your storage account allows access over HTTP, the`prefix` is `abfs`.
- If your storage account allows access over HTTPS, the `prefix` is `abfss`. | | AWS S3 or other S3-compatible storage (for example, MinIO) | `s3` | #### Drop a Hive database[​](#drop-a-hive-database "Direct link to Drop a Hive database") Similar to the internal databases of StarRocks, if you have the [DROP](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md#database) privilege on a Hive database, you can use the [DROP DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/DROP_DATABASE.md) statement to drop that Hive database. This feature is supported from v3.2 onwards. You can only drop empty databases. note You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). When you drop a Hive database, the database's file path on your HDFS cluster or cloud storage will not be dropped along with the database. [Switch to a Hive catalog](#switch-to-a-hive-catalog-and-a-database-in-it), and then use the following statement to drop a Hive database in that catalog: ```sql DROP DATABASE ``` #### Create a Hive table[​](#create-a-hive-table "Direct link to Create a Hive table") Similar to the internal databases of StarRocks, if you have the [CREATE TABLE](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md#database) privilege on a Hive database, you can use the [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md), [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md), or [CREATE TABLE LIKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_LIKE.md) statement to create a managed table in that Hive database. This feature is supported since v3.2 in which version StarRocks only supports creating Parquet-formatted Hive tables. From v3.3 onwards, StarRocks also supports creating ORC- and Textfile-formatted Hive tables. note * You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). * Hive catalogs support CREATE TABLE LIKE from v3.2.4 onwards. [Switch to a Hive catalog and a database in it](#switch-to-a-hive-catalog-and-a-database-in-it), and then use the following syntax to create a Hive managed table in that database. ##### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql CREATE TABLE [IF NOT EXISTS] [database.]table_name (column_definition1[, column_definition2, ... partition_column_definition1,partition_column_definition2...]) [partition_desc] [PROPERTIES ("key" = "value", ...)] [AS SELECT query] [LIKE [database.]] ``` ##### Parameters[​](#parameters-1 "Direct link to Parameters") ###### column\_definition[​](#column_definition "Direct link to column_definition") The syntax of `column_definition` is as follows: ```sql col_name col_type [COMMENT 'comment'] ``` The following table describes the parameters. | Parameter | Description | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | col\_name | The name of the column. | | col\_type | The data type of the column. The following data types are supported: TINYINT, SMALLINT, INT, BIGINT, FLOAT, DOUBLE, DECIMAL, DATE, DATETIME, CHAR, VARCHAR\[(length)], ARRAY, MAP, and STRUCT. The LARGEINT, HLL, and BITMAP data types are not supported. | > **NOTICE** > > All non-partition columns must use `NULL` as the default value. This means that you must specify `DEFAULT "NULL"` for each of the non-partition columns in the table creation statement. Additionally, partition columns must be defined following non-partition columns and cannot use `NULL` as the default value. ###### partition\_desc[​](#partition_desc "Direct link to partition_desc") The syntax of `partition_desc` is as follows: ```sql PARTITION BY (par_col1[, par_col2...]) ``` Currently StarRocks only supports identity transforms, which means that StarRocks creates a partition for each unique partition value. > **NOTICE** > > Partition columns must be defined following non-partition columns. Partition columns support all data types excluding FLOAT, DOUBLE, DECIMAL, and DATETIME and cannot use `NULL` as the default value. Additionally, the sequence of the partition columns declared in `partition_desc` must be consistent with the sequence of the columns defined in `column_definition`. ###### PROPERTIES[​](#properties "Direct link to PROPERTIES") You can specify the table attributes in the `"key" = "value"` format in `properties`. The following table describes a few key properties. | **Property** | **Description** | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | location | The file path in which you want to create the managed table. When you use HMS as metastore, you do not need to specify the `location` parameter, because StarRocks will create the table in the default file path of the current Hive catalog. When you use AWS Glue as metadata service:- If you have specified the `location` parameter for the database in which you want to create the table, you do not need to specify the `location` parameter for the table. As such, the table defaults to the file path of the database to which it belongs.
- If you have not specified the `location` for the database in which you want to create the table, you must specify the `location` parameter for the table. | | file\_format | The file format of the managed table. Supported file formats are Parquet, ORC, and Textfile. ORC and Textfile formats are supported from v3.3 onwards. Valid values: `parquet`, `orc`, and `textfile`. Default value: `parquet`. | | compression\_codec | The compression algorithm used for the managed table. | ##### Examples[​](#examples-1 "Direct link to Examples") The following DDLs use the default file format Parquet as an example. 1. Create a non-partitioned table named `unpartition_tbl`. The table consists of two columns, `id` and `score`, as shown below: ```sql CREATE TABLE unpartition_tbl ( id int, score double ); ``` 2. Create a partitioned table named `partition_tbl_1`. The table consists of three columns, `action`, `id`, and `dt`, of which `id` and `dt` are defined as partition columns, as shown below: ```sql CREATE TABLE partition_tbl_1 ( action varchar(20), id int, dt date ) PARTITION BY (id,dt); ``` 3. Query an existing table named `partition_tbl_1`, and create a partitioned table named `partition_tbl_2` based on the query result of `partition_tbl_1`. For `partition_tbl_2`, `id` and `dt` are defined as partition columns, as shown below: ```sql CREATE TABLE partition_tbl_2 PARTITION BY (k1, k2) AS SELECT * from partition_tbl_1; ``` #### Sink data to a Hive table[​](#sink-data-to-a-hive-table "Direct link to Sink data to a Hive table") Similar to the internal tables of StarRocks, if you have the [INSERT](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md#table) privilege on a Hive table (which can be a managed table or an external table), you can use the [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) statement to sink the data of a StarRocks table to that Hive table. This feature is supported since v3.2 in which version data can be sunk only to Parquet-formatted Hive tables. From v3.3 onwards, StarRocks also supports sinking data to ORC- and Textfile-formatted Hive tables. Note that sinking data to external tables is disabled by default. To sink data to external tables, you must set the [system variable `ENABLE_WRITE_HIVE_EXTERNAL_TABLE`](https://docs.starrocks.io/docs/sql-reference/System_variable.md) to `true`. note * You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). * You can use the table property [`compression_codec`](#properties) or the system variable [`connector_sink_compression_codec`](https://docs.starrocks.io/docs/sql-reference/System_variable.md#connector_sink_compression_codec) to specify the compression algorithm used for sinking data to Hive tables. StarRocks will prioritize using the compression codec specified in the table property. [Switch to a Hive catalog and a database in it](#switch-to-a-hive-catalog-and-a-database-in-it), and then use the following syntax to sink the data of StarRocks table to a Parquet-formatted Hive table in that database. ##### Syntax[​](#syntax-2 "Direct link to Syntax") ```sql INSERT {INTO | OVERWRITE} [ (column_name [, ...]) ] { VALUES ( { expression | DEFAULT } [, ...] ) [, ...] | query } -- If you want to sink data to specified partitions, use the following syntax: INSERT {INTO | OVERWRITE} PARTITION (par_col1= [, par_col2=...]) { VALUES ( { expression | DEFAULT } [, ...] ) [, ...] | query } ``` > **NOTICE** > > Partition columns do not allow `NULL` values. Therefore, you must make sure that no empty values are loaded into the partition columns of the Hive table. ##### Parameters[​](#parameters-2 "Direct link to Parameters") | Parameter | Description | | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | INTO | To append the data of the StarRocks table to the Hive table. | | OVERWRITE | To overwrite the existing data of the Hive table with the data of the StarRocks table. | | column\_name | The name of the destination column to which you want to load data. You can specify one or more columns. If you specify multiple columns, separate them with commas (`,`). You can only specify columns that actually exist in the Hive table, and the destination columns that you specify must include the partition columns of the Hive table. The destination columns you specify are mapped one on one in sequence to the columns of the StarRocks table, regardless of what the destination column names are. If no destination columns are specified, the data is loaded into all columns of the Hive table. If a non-partition column of the StarRocks table cannot be mapped to any column of the Hive table, StarRocks writes the default value `NULL` to the Hive table column. If the INSERT statement contains a query statement whose returned column types differ from the data types of the destination columns, StarRocks performs an implicit conversion on the mismatched columns. If the conversion fails, a syntax parsing error will be returned. | | expression | Expression that assigns values to the destination column. | | DEFAULT | Assigns a default value to the destination column. | | query | Query statement whose result will be loaded into the Hive table. It can be any SQL statement supported by StarRocks. | | PARTITION | The partitions into which you want to load data. You must specify all partition columns of the Hive table in this property. The partition columns that you specify in this property can be in a different sequence than the partition columns that you have defined in the table creation statement. If you specify this property, you cannot specify the `column_name` property. | ##### Examples[​](#examples-2 "Direct link to Examples") The following DMLs use the default file format Parquet as an example. 1. Insert three data rows into the `partition_tbl_1` table: ```sql INSERT INTO partition_tbl_1 VALUES ("buy", 1, "2023-09-01"), ("sell", 2, "2023-09-02"), ("buy", 3, "2023-09-03"); ``` 2. Insert the result of a SELECT query, which contains simple computations, into the `partition_tbl_1` table: ```sql INSERT INTO partition_tbl_1 (id, action, dt) SELECT 1+1, 'buy', '2023-09-03'; ``` 3. Insert the result of a SELECT query, which reads data from the `partition_tbl_1` table, into the same table: ```sql INSERT INTO partition_tbl_1 SELECT 'buy', 1, date_add(dt, INTERVAL 2 DAY) FROM partition_tbl_1 WHERE id=1; ``` 4. Insert the result of a SELECT query into the partitions that meet two conditions, `dt='2023-09-01'` and `id=1`, of the `partition_tbl_2` table: ```sql INSERT INTO partition_tbl_2 SELECT 'order', 1, '2023-09-01'; ``` Or ```sql INSERT INTO partition_tbl_2 partition(dt='2023-09-01',id=1) SELECT 'order'; ``` 5. Overwrite all `action` column values in the partitions that meet two conditions, `dt='2023-09-01'` and `id=1`, of the `partition_tbl_1` table with `close`: ```sql INSERT OVERWRITE partition_tbl_1 SELECT 'close', 1, '2023-09-01'; ``` Or ```sql INSERT OVERWRITE partition_tbl_1 partition(dt='2023-09-01',id=1) SELECT 'close'; ``` #### Truncate a Hive table[​](#truncate-a-hive-table "Direct link to Truncate a Hive table") You can use the [TRUNCATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/TRUNCATE_TABLE.md) statement to quickly delete all data from Hive managed tables. This operation supports: * Truncating all data in non-partitioned tables * Truncating all partitions in a partitioned table * Truncating specific partitions in a partitioned table ##### Syntax[​](#syntax-3 "Direct link to Syntax") ```sql TRUNCATE TABLE TRUNCATE TABLE PARTITION (partition_name = partition_value [, ...]) ``` ##### Parameters[​](#parameters-3 "Direct link to Parameters") * `table_name`: The name of the Hive table that you want to truncate data from. You need to [Switch to a Hive catalog and a database in it](#switch-to-a-hive-catalog-and-a-database-in-it) before you truncate the table in that database. * `partition_name = partition_value`: The name and value of the partition column(s) to identify which partition(s) to truncate. ##### Examples[​](#examples-3 "Direct link to Examples") Switch to a Hive catalog and a database in it, and then use the following statements to truncate a Hive table in that database. 1. Truncate a non-partitioned table: ```sql TRUNCATE TABLE my_table; ``` 2. Truncate all partitions of a partitioned table: ```sql TRUNCATE TABLE my_partitioned_table; ``` 3. Truncate a single-partition partitioned table: ```sql TRUNCATE TABLE my_partitioned_table PARTITION (dt='2023-09-01'); ``` 4. Truncate specific partitions of a multi-partition partitioned table: ```sql TRUNCATE TABLE my_partitioned_table PARTITION (dt='2023-09-01', id=1); TRUNCATE TABLE my_multi_part_table PARTITION (k2='2020-01-02', k3='b'); ``` #### Drop a Hive table[​](#drop-a-hive-table "Direct link to Drop a Hive table") Similar to the internal tables of StarRocks, if you have the [DROP](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md#table) privilege on a Hive table, you can use the [DROP TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md) statement to drop that Hive table. This feature is supported from v3.1 onwards. Note that currently StarRocks supports dropping only managed tables of Hive. note You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). When you drop a Hive table, you must specify the `FORCE` keyword in the DROP TABLE statement. After the operation is complete, the table's file path is retained, but the table's data on your HDFS cluster or cloud storage is all dropped along with the table. Exercise caution when you perform this operation to drop a Hive table. [Switch to a Hive catalog and a database in it](#switch-to-a-hive-catalog-and-a-database-in-it), and then use the following statement to drop a Hive table in that database. ```sql DROP TABLE FORCE ``` #### Manually or automatically update metadata cache[​](#manually-or-automatically-update-metadata-cache "Direct link to Manually or automatically update metadata cache") ##### Manual update[​](#manual-update "Direct link to Manual update") By default, StarRocks caches the metadata of Hive and automatically updates the metadata in asynchronous mode to deliver better performance. Additionally, after some schema changes or table updates are made on a Hive table, you can also use [REFRESH EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/REFRESH_EXTERNAL_TABLE.md) to manually update its metadata, thereby ensuring that StarRocks can obtain up-to-date metadata at its earliest opportunity and generate appropriate execution plans: ```sql REFRESH EXTERNAL TABLE [PARTITION ('partition_name', ...)] ``` You need to manually update metadata in the following situations: * A data file in an existing partition is changed, for example, by running the `INSERT OVERWRITE ... PARTITION ...` command. * Schema changes are made on a Hive table. * An existing Hive table is deleted by using the DROP statement, and a new Hive table with the same name as the deleted Hive table is created. * You have specified `"enable_cache_list_names" = "true"` in `PROPERTIES` at the creation of your Hive catalog, and you want to query new partitions that you just created on your Hive cluster. note From v2.5.5 onwards, StarRocks provides the periodic Hive metadata cache refresh feature. For more information, see the below "[Periodically refresh metadata cache](#periodically-refresh-metadata-cache)" section of this topic. After you enable this feature, StarRocks refreshes your Hive metadata cache every 10 minutes by default. Therefore, manual updates are not needed in most cases. You need to perform a manual update only when you want to query new partitions immediately after the new partitions are created on your Hive cluster. Note that the REFRESH EXTERNAL TABLE refreshes only the tables and partitions cached in your FEs. #### Periodically refresh metadata cache[​](#periodically-refresh-metadata-cache "Direct link to Periodically refresh metadata cache") From v2.5.5 onwards, StarRocks can periodically refresh the cached metadata of the frequently accessed Hive catalogs to perceive data changes. You can configure the Hive metadata cache refresh through the following [FE parameters](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md): | Configuration item | Default | Description | | -------------------------------------------------------------------- | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_background\_refresh\_connector\_metadata | `true` in v3.0
`false` in v2.5 | Whether to enable the periodic Hive metadata cache refresh. After it is enabled, StarRocks polls the metastore (Hive Metastore or AWS Glue) of your Hive cluster, and refreshes the cached metadata of the frequently accessed Hive catalogs to perceive data changes. `true` indicates to enable the Hive metadata cache refresh, and `false` indicates to disable it. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | | background\_refresh\_metadata\_interval\_millis | `600000` (10 minutes) | The interval between two consecutive Hive metadata cache refreshes. Unit: millisecond. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | | background\_refresh\_metadata\_time\_secs\_since\_last\_access\_secs | `86400` (24 hours) | The expiration time of a Hive metadata cache refresh task. For the Hive catalog that has been accessed, if it has not been accessed for more than the specified time, StarRocks stops refreshing its cached metadata. For the Hive catalog that has not been accessed, StarRocks will not refresh its cached metadata. Unit: second. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | Using the periodic Hive metadata cache refresh feature and the metadata automatic asynchronous update policy together significantly accelerates data access, reduces the read load from external data sources, and improves query performance. #### Appendix: Understand metadata automatic asynchronous update[​](#appendix-understand-metadata-automatic-asynchronous-update "Direct link to Appendix: Understand metadata automatic asynchronous update") Automatic asynchronous update is the default policy that StarRocks uses to update the metadata in Hive catalogs. By default (namely, when the `enable_metastore_cache` and `enable_remote_file_cache` parameters are both set to `true`), if a query hits a partition of a Hive table, StarRocks automatically caches the metadata of the partition and the metadata of the underlying data files of the partition. The cached metadata is updated by using the lazy update policy. For example, there is a Hive table named `table2`, which has four partitions: `p1`, `p2`, `p3`, and `p4`. A query hits `p1`, and StarRocks caches the metadata of `p1` and the metadata of the underlying data files of `p1`. Assume that the default time intervals to update and discard the cached metadata are as follows: * The time interval (specified by the `metastore_cache_refresh_interval_sec` parameter) to asynchronously update the cached metadata of `p1` is 60 seconds. * The time interval (specified by the `remote_file_cache_refresh_interval_sec` parameter) to asynchronously update the cached metadata of the underlying data files of `p1` is 60 seconds. * The time interval (specified by the `metastore_cache_ttl_sec` parameter) to automatically discard the cached metadata of `p1` is 24 hours. * The time interval (specified by the `remote_file_cache_ttl_sec` parameter) to automatically discard the cached metadata of the underlying data files of `p1` is 36 hours. The following figure shows the time intervals on a timeline for easier understanding. ![Timeline for updating and discarding cached metadata](/assets/images/hive_catalog_timeline-231c618c403c423e20192c9d3a34c70d.png) Then StarRocks updates or discards the metadata in compliance with the following rules: * If another query hits `p1` again and the current time from the last update is less than 60 seconds, StarRocks does not update the cached metadata of `p1` or the cached metadata of the underlying data files of `p1`. * If another query hits `p1` again and the current time from the last update is more than 60 seconds, StarRocks updates the cached metadata of `p1` and the cached metadata of the underlying data files of `p1`. * If the table has been accessed within 24 hours, the related cache will be refreshed every 10 minutes in the background. * If `p1` has not been accessed within 24 hours from the last update, StarRocks discards the cached metadata of `p1`. The metadata will be cached at the next query. * If `p1` has not been accessed within 36 hours from the last update, StarRocks discards the cached metadata of the underlying data files of `p1`. The metadata will be cached at the next query. --- ### Hudi catalog A Hudi catalog is a kind of external catalog that enables you to query data from Apache Hudi without ingestion. Also, you can directly transform and load data from Hudi by using [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) based on Hudi catalogs. StarRocks supports Hudi catalogs from v2.4 onwards. To ensure successful SQL workloads on your Hudi cluster, your StarRocks cluster must be able to access the storage system and metastore of your Hudi cluster. StarRocks supports the following storage systems and metastores: * Distributed file system (HDFS) or object storage like AWS S3, Microsoft Azure Storage, Google GCS, or other S3-compatible storage system (for example, MinIO) * Metastore like Hive metastore or AWS Glue > **NOTE** > > If you choose AWS S3 as storage, you can use HMS or AWS Glue as metastore. If you choose any other storage system, you can only use HMS as metastore. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The file format of Hudi that StarRocks supports is Parquet. Parquet files support the following compression formats: SNAPPY, LZ4, ZSTD, GZIP, and NO\_COMPRESSION. * StarRocks provides complete support for Copy On Write (COW) tables and Merge On Read (MOR) tables from Hudi. #### Integration preparations[​](#integration-preparations "Direct link to Integration preparations") Before you create a Hudi catalog, make sure your StarRocks cluster can integrate with the storage system and metastore of your Hudi cluster. ##### AWS IAM[​](#aws-iam "Direct link to AWS IAM") If your Hudi cluster uses AWS S3 as storage or AWS Glue as metastore, choose your suitable authentication method and make the required preparations to ensure that your StarRocks cluster can access the related AWS cloud resources. The following authentication methods are recommended: * Instance profile * Assumed role * IAM user Of the above-mentioned three authentication methods, instance profile is the most widely used. For more information, see [Preparation for authentication in AWS IAM](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#preparations). ##### HDFS[​](#hdfs "Direct link to HDFS") If you choose HDFS as storage, configure your StarRocks cluster as follows: * (Optional) Set the username that is used to access your HDFS cluster and Hive metastore. By default, StarRocks uses the username of the FE and BE or CN processes to access your HDFS cluster and Hive metastore. You can also set the username by adding `export HADOOP_USER_NAME=""` at the beginning of the **fe/conf/hadoop\_env.sh** file of each FE and at the beginning of the **be/conf/hadoop\_env.sh** file of each BE or the **cn/conf/hadoop\_env.sh** file of each CN. After you set the username in these files, restart each FE and each BE or CN to make the parameter settings take effect. You can set only one username for each StarRocks cluster. * When you query Hudi data, the FEs and BEs or CNs of your StarRocks cluster use the HDFS client to access your HDFS cluster. In most cases, you do not need to configure your StarRocks cluster to achieve that purpose, and StarRocks starts the HDFS client using the default configurations. You need to configure your StarRocks cluster only in the following situations: * High availability (HA) is enabled for your HDFS cluster: Add the **hdfs-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. * View File System (ViewFs) is enabled for your HDFS cluster: Add the **core-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. > **NOTE** > > If an error indicating an unknown host is returned when you send a query, you must add the mapping between the host names and IP addresses of your HDFS cluster nodes to the **/etc/hosts** path. ##### Kerberos authentication[​](#kerberos-authentication "Direct link to Kerberos authentication") If Kerberos authentication is enabled for your HDFS cluster or Hive metastore, configure your StarRocks cluster as follows: * Run the `kinit -kt keytab_path principal` command on each FE and each BE or CN to obtain Ticket Granting Ticket (TGT) from Key Distribution Center (KDC). To run this command, you must have the permissions to access your HDFS cluster and Hive metastore. Note that accessing KDC with this command is time-sensitive. Therefore, you need to use cron to run this command periodically. * Add `JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"` to the **$FE\_HOME/conf/fe.conf** file of each FE and to the **$BE\_HOME/conf/be.conf** file of each BE or the **$CN\_HOME/conf/cn.conf** file of each CN. In this example, `/etc/krb5.conf` is the save path of the **krb5.conf** file. You can modify the path based on your needs. #### Create a Hudi catalog[​](#create-a-hudi-catalog "Direct link to Create a Hudi catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ( "type" = "hudi", MetastoreParams, StorageCredentialParams, MetadataUpdateParams ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### catalog\_name[​](#catalog_name "Direct link to catalog_name") The name of the Hudi catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### comment[​](#comment "Direct link to comment") The description of the Hudi catalog. This parameter is optional. ###### type[​](#type "Direct link to type") The type of your data source. Set the value to `hudi`. ###### MetastoreParams[​](#metastoreparams "Direct link to MetastoreParams") A set of parameters about how StarRocks integrates with the metastore of your data source. ###### Hive metastore[​](#hive-metastore "Direct link to Hive metastore") If you choose Hive metastore as the metastore of your data source, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "hive", "hive.metastore.uris" = "" ``` > **NOTE** > > Before querying Hudi data, you must add the mapping between the host names and IP addresses of your Hive metastore nodes to the `/etc/hosts` path. Otherwise, StarRocks may fail to access your Hive metastore when you start a query. The following table describes the parameter you need to configure in `MetastoreParams`. | Parameter | Required | Description | | ------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hive.metastore.type | Yes | The type of metastore that you use for your Hudi cluster. Set the value to `hive`. | | hive.metastore.uris | Yes | The URI of your Hive metastore. Format: `thrift://:`.
If high availability (HA) is enabled for your Hive metastore, you can specify multiple metastore URIs and separate them with commas (`,`), for example, `"thrift://:,thrift://:,thrift://:"`. | ###### AWS Glue[​](#aws-glue "Direct link to AWS Glue") If you choose AWS Glue as the metastore of your data source, which is supported only when you choose AWS S3 as storage, take one of the following actions: * To choose the instance profile-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "" ``` * To choose the assumed role-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "", "aws.glue.region" = "" ``` * To choose the IAM user-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "" ``` The following table describes the parameters you need to configure in `MetastoreParams`. | Parameter | Required | Description | | ------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hive.metastore.type | Yes | The type of metastore that you use for your Hudi cluster. Set the value to `glue`. | | aws.glue.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. | | aws.glue.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS Glue Data Catalog. If you use the assumed role-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.region | Yes | The region in which your AWS Glue Data Catalog resides. Example: `us-west-1`. | | aws.glue.access\_key | No | The access key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.secret\_key | No | The secret key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS Glue and how to configure an access control policy in the AWS IAM Console, see [Authentication parameters for accessing AWS Glue](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-glue). ###### StorageCredentialParams[​](#storagecredentialparams "Direct link to StorageCredentialParams") A set of parameters about how StarRocks integrates with your storage system. This parameter set is optional. If you use HDFS as storage, you do not need to configure `StorageCredentialParams`. If you use AWS S3, other S3-compatible storage system, Microsoft Azure Storage, or Google GCS as storage, you must configure `StorageCredentialParams`. ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") If you choose AWS S3 as storage for your Hudi cluster, take one of the following actions: * To choose the instance profile-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "" ``` * To choose the assumed role-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.region" = "" ``` * To choose the IAM user-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ----------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. | | aws.s3.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS S3 bucket. If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.region | Yes | The region in which your AWS S3 bucket resides. Example: `us-west-1`. | | aws.s3.access\_key | No | The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.secret\_key | No | The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in AWS IAM Console, see [Authentication parameters for accessing AWS S3](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-s3). ###### S3-compatible storage system[​](#s3-compatible-storage-system "Direct link to S3-compatible storage system") Hudi catalogs support S3-compatible storage systems from v2.5 onwards. If you choose an S3-compatible storage system, such as MinIO, as storage for your Hudi cluster, configure `StorageCredentialParams` as follows to ensure a successful integration: ```sql "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ---------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.enable\_ssl | Yes | Specifies whether to enable SSL connection.
Valid values: `true` and `false`. Default value: `true`. | | aws.s3.enable\_path\_style\_access | Yes | Specifies whether to enable path-style access.
Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`.
Path-style URLs use the following format: `https://s3..amazonaws.com//`. For example, if you create a bucket named `DOC-EXAMPLE-BUCKET1` in the US West (Oregon) Region, and you want to access the `alice.jpg` object in that bucket, you can use the following path-style URL: `https://s3.us-west-2.amazonaws.com/DOC-EXAMPLE-BUCKET1/alice.jpg`. | | aws.s3.endpoint | Yes | The endpoint that is used to connect to your S3-compatible storage system instead of AWS S3. | | aws.s3.access\_key | Yes | The access key of your IAM user. | | aws.s3.secret\_key | Yes | The secret key of your IAM user. | ###### Microsoft Azure Storage[​](#microsoft-azure-storage "Direct link to Microsoft Azure Storage") Hudi catalogs support Microsoft Azure Storage from v3.0 onwards. ###### Azure Blob Storage[​](#azure-blob-storage "Direct link to Azure Blob Storage") If you choose Blob Storage as storage for your Hudi cluster, take one of the following actions: * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | -------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.shared\_key | Yes | The shared key of your Blob Storage account. | * To choose the SAS Token authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | --------------------------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.container | Yes | The name of the blob container that stores your data. | | azure.blob.sas\_token | Yes | The SAS token that is used to access your Blob Storage account. | ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2 "Direct link to Azure Data Lake Storage Gen2") If you choose Data Lake Storage Gen2 as storage for your Hudi cluster, take one of the following actions: * To choose the Managed Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_use\_managed\_identity | Yes | Specifies whether to enable the Managed Identity authentication method. Set the value to `true`. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the managed identity. | * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ---------------------------- | ------------ | -------------------------------------------------------------- | | azure.adls2.storage\_account | Yes | The username of your Data Lake Storage Gen2 storage account. | | azure.adls2.shared\_key | Yes | The shared key of your Data Lake Storage Gen2 storage account. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ``` The following table describes the parameters you need to configure `in StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls2.oauth2\_client\_secret | Yes | The value of the new client (application) secret created. | | azure.adls2.oauth2\_client\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | * To choose the Workload Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_token_file" = "", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_token\_file | Yes | The absolute file path to the OAuth2 token file projected into the pod by the Azure Workload Identity webhook. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client ID (application ID) of the Azure AD application (user-assigned managed identity or app registration) associated with the workload identity. | ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1 "Direct link to Azure Data Lake Storage Gen1") If you choose Data Lake Storage Gen1 as storage for your Hudi cluster, take one of the following actions: * To choose the Managed Service Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.use_managed_service_identity" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------- | | azure.adls1.use\_managed\_service\_identity | Yes | Specifies whether to enable the Managed Service Identity authentication method. Set the value to `true`. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls1.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls1.oauth2\_credential | Yes | The value of the new client (application) secret created. | | azure.adls1.oauth2\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | ###### Google GCS[​](#google-gcs "Direct link to Google GCS") Hudi catalogs support Google GCS from v3.0 onwards. If you choose Google GCS as storage for your Hudi cluster, take one of the following actions: * To choose the VM-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.use_compute_engine_service_account" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | * To choose the service account-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------- | | gcp.gcs.service\_account\_email | "" | "" | The email address in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the service account. | * To choose the impersonation-based authentication method, configure `StorageCredentialParams` as follows: * Make a VM instance impersonate a service account: ```sql "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The service account that you want to impersonate. | * Make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account): ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | gcp.gcs.service\_account\_email | "" | "" | The email address in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the meta service account. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The data service account that you want to impersonate. | ###### MetadataUpdateParams[​](#metadataupdateparams "Direct link to MetadataUpdateParams") A set of parameters about how StarRocks updates the cached metadata of Hudi. This parameter set is optional. StarRocks implements the [automatic asynchronous update policy](#appendix-understand-metadata-automatic-asynchronous-update) by default. In most cases, you can ignore `MetadataUpdateParams` and do not need to tune the policy parameters in it, because the default values of these parameters already provide you with an out-of-the-box performance. However, if the frequency of data updates in Hudi is high, you can tune these parameters to further optimize the performance of automatic asynchronous updates. > **NOTE** > > In most cases, if your Hudi data is updated at a granularity of 1 hour or less, the data update frequency is considered high. | Parameter | Required | Description | | ------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_metastore\_cache | No | Specifies whether StarRocks caches the metadata of Hudi tables. Valid values: `true` and `false`. Default value: `true`. The value `true` enables the cache, and the value `false` disables the cache. | | enable\_remote\_file\_cache | No | Specifies whether StarRocks caches the metadata of the underlying data files of Hudi tables or partitions. Valid values: `true` and `false`. Default value: `true`. The value `true` enables the cache, and the value `false` disables the cache. | | metastore\_cache\_refresh\_interval\_sec | No | The time interval at which StarRocks asynchronously updates the metadata of Hudi tables or partitions cached in itself. Unit: seconds. Default value: `7200`, which is 2 hours. | | remote\_file\_cache\_refresh\_interval\_sec | No | The time interval at which StarRocks asynchronously updates the metadata of the underlying data files of Hudi tables or partitions cached in itself. Unit: seconds. Default value: `60`. | | metastore\_cache\_ttl\_sec | No | The time interval at which StarRocks automatically discards the metadata of Hudi tables or partitions cached in itself. Unit: seconds. Default value: `86400`, which is 24 hours. | | remote\_file\_cache\_ttl\_sec | No | The time interval at which StarRocks automatically discards the metadata of the underlying data files of Hudi tables or partitions cached in itself. Unit: seconds. Default value: `129600`, which is 36 hours. | ##### Examples[​](#examples "Direct link to Examples") The following examples create a Hudi catalog named `hudi_catalog_hms` or `hudi_catalog_glue`, depending on the type of metastore you use, to query data from your Hudi cluster. ###### HDFS[​](#hdfs-1 "Direct link to HDFS") If you use HDFS as storage, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` ###### AWS S3[​](#aws-s3-1 "Direct link to AWS S3") ###### If you choose instance profile-based credential[​](#if-you-choose-instance-profile-based-credential "Direct link to If you choose instance profile-based credential") * If you use Hive metastore in your Hudi cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Hudi cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_glue PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` ###### If you choose assumed role-based credential[​](#if-you-choose-assumed-role-based-credential "Direct link to If you choose assumed role-based credential") * If you use Hive metastore in your Hudi cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Hudi cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_glue PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "arn:aws:iam::081976408565:role/test_glue_role", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` ###### If you choose IAM user-based credential[​](#if-you-choose-iam-user-based-credential "Direct link to If you choose IAM user-based credential") * If you use Hive metastore in your Hudi cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Hudi cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_glue PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` ###### S3-compatible storage system[​](#s3-compatible-storage-system-1 "Direct link to S3-compatible storage system") Use MinIO as an example. Run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.enable_ssl" = "true", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ); ``` ###### Microsoft Azure Storage[​](#microsoft-azure-storage-1 "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage-1 "Direct link to Azure Blob Storage") * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` * If you choose the SAS Token authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1-1 "Direct link to Azure Data Lake Storage Gen1") * If you choose the Managed Service Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.use_managed_service_identity" = "true" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2-1 "Direct link to Azure Data Lake Storage Gen2") * If you choose the Managed Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` * If you choose the Workload Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### Google GCS[​](#google-gcs-1 "Direct link to Google GCS") * If you choose the VM-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true" ); ``` * If you choose the service account-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ); ``` * If you choose the impersonation-based authentication method: * If you make a VM instance impersonate a service account, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ); ``` * If you make a service account impersonate another service account, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ); ``` #### View Hudi catalogs[​](#view-hudi-catalogs "Direct link to View Hudi catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of a Hudi catalog named `hudi_catalog_glue`: ```sql SHOW CREATE CATALOG hudi_catalog_glue; ``` #### Switch to a Hudi Catalog and a database in it[​](#switch-to-a-hudi-catalog-and-a-database-in-it "Direct link to Switch to a Hudi Catalog and a database in it") You can use one of the following methods to switch to a Hudi catalog and a database in it: * Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to specify a Hudi catalog in the current session, and then use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify an active database: ```sql -- Switch to a specified catalog in the current session: SET CATALOG -- Specify the active database in the current session: USE ``` * Directly use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to switch to a Hudi catalog and a database in it: ```sql USE . ``` #### Drop a Hudi catalog[​](#drop-a-hudi-catalog "Direct link to Drop a Hudi catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop an external catalog. The following example drops a Hudi catalog named `hudi_catalog_glue`: ```sql DROP Catalog hudi_catalog_glue; ``` #### View the schema of a Hudi table[​](#view-the-schema-of-a-hudi-table "Direct link to View the schema of a Hudi table") You can use one of the following syntaxes to view the schema of a Hudi table: * View schema ```sql DESC[RIBE] .. ``` * View schema and location from the CREATE statement ```sql SHOW CREATE TABLE .. ``` #### Query a Hudi table[​](#query-a-hudi-table "Direct link to Query a Hudi table") 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your Hudi cluster: ```sql SHOW DATABASES FROM ``` 2. [Switch to a Hudi Catalog and a database in it](#switch-to-a-hudi-catalog-and-a-database-in-it). 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT count(*) FROM LIMIT 10 ``` #### Load data from Hudi[​](#load-data-from-hudi "Direct link to Load data from Hudi") Suppose you have an OLAP table named `olap_tbl`, you can transform and load data like below: ```sql INSERT INTO default_catalog.olap_db.olap_tbl SELECT * FROM hudi_table ``` #### Manually or automatically update metadata cache[​](#manually-or-automatically-update-metadata-cache "Direct link to Manually or automatically update metadata cache") ##### Manual update[​](#manual-update "Direct link to Manual update") By default, StarRocks caches the metadata of Hudi and automatically updates the metadata in asynchronous mode to deliver better performance. Additionally, after some schema changes or table updates are made on a Hudi table, you can also use [REFRESH EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/REFRESH_EXTERNAL_TABLE.md) to manually update its metadata, thereby ensuring that StarRocks can obtain up-to-date metadata at its earliest opportunity and generate appropriate execution plans: ```sql REFRESH EXTERNAL TABLE [PARTITION ('partition_name', ...)] ``` #### Appendix: Understand metadata automatic asynchronous update[​](#appendix-understand-metadata-automatic-asynchronous-update "Direct link to Appendix: Understand metadata automatic asynchronous update") Automatic asynchronous update is the default policy that StarRocks uses to update the metadata in Hudi catalogs. By default (namely, when the `enable_metastore_cache` and `enable_remote_file_cache` parameters are both set to `true`), if a query hits a partition of a Hudi table, StarRocks automatically caches the metadata of the partition and the metadata of the underlying data files of the partition. The cached metadata is updated by using the lazy update policy. For example, there is a Hudi table named `table2`, which has four partitions: `p1`, `p2`, `p3`, and `p4`. A query hits `p1`, and StarRocks caches the metadata of `p1` and the metadata of the underlying data files of `p1`. Assume that the default time intervals to update and discard the cached metadata are as follows: * The time interval (specified by the `metastore_cache_refresh_interval_sec` parameter) to asynchronously update the cached metadata of `p1` is 2 hours. * The time interval (specified by the `remote_file_cache_refresh_interval_sec` parameter) to asynchronously update the cached metadata of the underlying data files of `p1` is 60 seconds. * The time interval (specified by the `metastore_cache_ttl_sec` parameter) to automatically discard the cached metadata of `p1` is 24 hours. * The time interval (specified by the `remote_file_cache_ttl_sec` parameter) to automatically discard the cached metadata of the underlying data files of `p1` is 36 hours. The following figure shows the time intervals on a timeline for easier understanding. ![Timeline for updating and discarding cached metadata](/assets/images/catalog_timeline-152d44a61a8d93f0f6ea1442dfe9b91f.png) Then StarRocks updates or discards the metadata in compliance with the following rules: * If another query hits `p1` again and the current time from the last update is less than 60 seconds, StarRocks does not update the cached metadata of `p1` or the cached metadata of the underlying data files of `p1`. * If another query hits `p1` again and the current time from the last update is more than 60 seconds, StarRocks updates the cached metadata of the underlying data files of `p1`. * If another query hits `p1` again and the current time from the last update is more than 2 hours, StarRocks updates the cached metadata of `p1`. * If `p1` has not been accessed within 24 hours from the last update, StarRocks discards the cached metadata of `p1`. The metadata will be cached at the next query. * If `p1` has not been accessed within 36 hours from the last update, StarRocks discards the cached metadata of the underlying data files of `p1`. The metadata will be cached at the next query. --- ### Iceberg catalog tip This example uses the Local Climatological Data(LCD) dataset featured in the [StarRocks Basics](https://docs.starrocks.io/docs/quick_start/shared-nothing.md) Quick Start. You can load the data and try the example yourself. An Iceberg catalog is a type of external catalog that is supported by StarRocks from v2.4 onwards. With Iceberg catalogs, you can: * Directly query data stored in Iceberg without the need to manually create tables. * Use [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) or asynchronous materialized views (which are supported from v2.5 onwards) to process data stored in Iceberg and load the data into StarRocks. * Perform operations on StarRocks to create or drop Iceberg databases and tables, or sink data from StarRocks tables to Parquet-formatted Iceberg tables by using [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) (this feature is supported from v3.1 onwards). To ensure successful SQL workloads on your Iceberg cluster, your StarRocks cluster must be able to access the storage system and metastore of your Iceberg cluster. StarRocks supports the following storage systems and metastores: * Distributed file system (HDFS) or object storage like AWS S3, Microsoft Azure Storage, Google GCS, or other S3-compatible storage system (for example, MinIO) * Metastore like Hive metastore, AWS Glue, or Tabular note * If you choose AWS S3 as storage, you can use HMS or AWS Glue as metastore. If you choose any other storage system, you can only use HMS as metastore. * If you choose Tabular as metastore, you need to use the Iceberg REST catalog. #### Usage notes[​](#usage-notes "Direct link to Usage notes") Take note of the following points when you use StarRocks to query data from Iceberg: | **File format** | **Compression format** | **Iceberg table version** | | --------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Parquet | SNAPPY, LZ4, ZSTD, GZIP, and NO\_COMPRESSION | - v1 tables: supported.
- v2 tables: supported from StarRocks v3.1 onwards in which queries on these v2 tables support position deletes. In v3.1.10, v3.2.5, v3.3 and their later versions, queries on v2 tables also support equality deletes. | | ORC | ZLIB, SNAPPY, LZO, LZ4, ZSTD, and NO\_COMPRESSION | - v1 tables: supported.
- v2 tables: supported from StarRocks v3.0 onwards in which queries on these v2 tables support position deletes. In v3.1.8, v3.2.3, v3.3 and their later versions, queries on v2 tables also support equality deletes. | #### Integration preparation[​](#integration-preparation "Direct link to Integration preparation") Before you create an Iceberg catalog, make sure your StarRocks cluster can integrate with the storage system and metastore of your Iceberg cluster. *** ##### Storage[​](#storage "Direct link to Storage") Select the tab that matches your storage type: * AWS S3 * HDFS If your Iceberg cluster uses AWS S3 as storage or AWS Glue as metastore, choose your suitable authentication method and make the required preparations to ensure that your StarRocks cluster can access the related AWS cloud resources. The following authentication methods are recommended: * Instance profile * Assumed role * IAM user Of the above-mentioned three authentication methods, instance profile is the most widely used. For more information, see [Preparation for authentication in AWS IAM](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#preparations). If you choose HDFS as storage, configure your StarRocks cluster as follows: * (Optional) Set the username that is used to access your HDFS cluster and Hive metastore. By default, StarRocks uses the username of the FE and BE or CN processes to access your HDFS cluster and Hive metastore. You can also set the username by adding `export HADOOP_USER_NAME=""` at the beginning of the **fe/conf/hadoop\_env.sh** file of each FE and at the beginning of the **be/conf/hadoop\_env.sh** file of each BE or the **cn/conf/hadoop\_env.sh** file of each CN. After you set the username in these files, restart each FE and each BE or CN to make the parameter settings take effect. You can set only one username for each StarRocks cluster. * When you query Iceberg data, the FEs and BEs or CNs of your StarRocks cluster use the HDFS client to access your HDFS cluster. In most cases, you do not need to configure your StarRocks cluster to achieve that purpose, and StarRocks starts the HDFS client using the default configurations. You need to configure your StarRocks cluster only in the following situations: * High availability (HA) is enabled for your HDFS cluster: Add the **hdfs-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. * View File System (ViewFs) is enabled for your HDFS cluster: Add the **core-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. tip If an error indicating an unknown host is returned when you send a query, you must add the mapping between the host names and IP addresses of your HDFS cluster nodes to the **/etc/hosts** path. ###### Pass HDFS client configurations through Catalog PROPERTIES[​](#pass-hdfs-client-configurations-through-catalog-properties "Direct link to Pass HDFS client configurations through Catalog PROPERTIES") In addition to placing **hdfs-site.xml** under the **conf** directories of FEs/BEs/CNs, you can declare HDFS client configurations directly in the `PROPERTIES` of `CREATE EXTERNAL CATALOG` (for example, the HA-related keys `dfs.nameservices`, `dfs.ha.namenodes.`, `dfs.namenode.rpc-address..`, `dfs.client.failover.proxy.provider.`, `fs.defaultFS`, etc.). Both FEs and BEs/CNs receive these properties and apply them to the HDFS client. The main advantage of this approach is that **a single StarRocks cluster can access multiple independent HDFS HA clusters at the same time through different Iceberg Catalogs**. Because the **conf** directory of each FE/BE/CN can hold only one **hdfs-site.xml**, multiple HDFS HA clusters cannot coexist there. Passing the configuration through Catalog PROPERTIES lets each Catalog carry its own HA configuration, with no interference between Catalogs. For a complete HA example, see the [Examples - HDFS](#examples) section below. *** ###### Kerberos authentication[​](#kerberos-authentication "Direct link to Kerberos authentication") If Kerberos authentication is enabled for your HDFS cluster or Hive metastore, configure your StarRocks cluster as follows: * Run the `kinit -kt keytab_path principal` command on each FE and each BE or CN to obtain Ticket Granting Ticket (TGT) from Key Distribution Center (KDC). To run this command, you must have the permissions to access your HDFS cluster and Hive metastore. Note that accessing KDC with this command is time-sensitive. Therefore, you need to use cron to run this command periodically. * Add `JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"` to the **$FE\_HOME/conf/fe.conf** file of each FE and to the **$BE\_HOME/conf/be.conf** file of each BE or the **$CN\_HOME/conf/cn.conf** file of each CN. In this example, `/etc/krb5.conf` is the save path of the **krb5.conf** file. You can modify the path based on your needs. *** #### Create an Iceberg catalog[​](#create-an-iceberg-catalog "Direct link to Create an Iceberg catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ( "type" = "iceberg", [SecurityParams], MetastoreParams, StorageCredentialParams, MetadataRelatedParams ) ``` *** ##### Parameters[​](#parameters "Direct link to Parameters") ###### catalog\_name[​](#catalog_name "Direct link to catalog_name") The name of the Iceberg catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### comment[​](#comment "Direct link to comment") The description of the Iceberg catalog. This parameter is optional. ###### type[​](#type "Direct link to type") The type of your data source. Set the value to `iceberg`. ###### SecurityParams[​](#securityparams "Direct link to SecurityParams") Parameter(s) about how StarRocks manages data access to the catalog. For detailed instructions on managing data access for Iceberg Catalogs, see [Security Setup for Iceberg REST Catalog](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_rest_security.md). ###### catalog.access.control[​](#catalogaccesscontrol "Direct link to catalog.access.control") The data access control policy. Valid values: * `native` (Default): The StarRocks built-in data access control system is used. * `allowall`: All data access checks are delegated to the Catalog itself. * `ranger`: Data access checks are delegated to Apache Ranger. ###### MetastoreParams[​](#metastoreparams "Direct link to MetastoreParams") A set of parameters about how StarRocks integrates with the metastore of your data source. Choose the tab that matches your metastore type: * Hive metastore * AWS Glue * REST * JDBC ###### Hive metastore[​](#hive-metastore "Direct link to Hive metastore") If you choose Hive metastore as the metastore of your data source, configure `MetastoreParams` as follows: ```sql "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "" ``` note Before querying Iceberg data, you must add the mapping between the host names and IP addresses of your Hive metastore nodes to the `/etc/hosts` path. Otherwise, StarRocks may fail to access your Hive metastore when you start a query. The following table describes the parameter you need to configure in `MetastoreParams`. * `iceberg.catalog.type` * Required: Yes * Description: The type of metastore that you use for your Iceberg cluster. Set the value to `hive`. * `hive.metastore.uris` * Required: Yes * Description: The URI of your Hive metastore. Format: `thrift://:`.
If high availability (HA) is enabled for your Hive metastore, you can specify multiple metastore URIs and separate them with commas (`,`), for example, `"thrift://:,thrift://:,thrift://:"`. ###### AWS Glue[​](#aws-glue "Direct link to AWS Glue") If you choose AWS Glue as the metastore of your data source, which is supported only when you choose AWS S3 as storage, take one of the following actions: * To choose the instance profile-based authentication method, configure `MetastoreParams` as follows: ```sql "iceberg.catalog.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "" ``` * To choose the assumed role-based authentication method, configure `MetastoreParams` as follows: ```sql "iceberg.catalog.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "", "aws.glue.region" = "" ``` * To choose the IAM user-based authentication method, configure `MetastoreParams` as follows: ```sql "iceberg.catalog.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "" ``` `MetastoreParams` for AWS Glue: * `iceberg.catalog.type` * Required: Yes * Description: The type of metastore that you use for your Iceberg cluster. Set the value to `glue`. * `aws.glue.use_instance_profile` * Required: Yes * Description: Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. * `aws.glue.iam_role_arn` * Required: No * Description: The ARN of the IAM role that has privileges on your AWS Glue Data Catalog. If you use the assumed role-based authentication method to access AWS Glue, you must specify this parameter. * `aws.glue.region` * Required: Yes * Description: The region in which your AWS Glue Data Catalog resides. Example: `us-west-1`. * `aws.glue.access_key` * Required: No * Description: The access key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. * `aws.glue.secret_key` * Required: No * Description: The secret key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. * `aws.glue.catalog_id` * Required: No * Description: The ID of the AWS Glue Data Catalog to use. When not specified, the catalog in the current AWS account is used. You must specify this parameter when you need to access a Glue Data Catalog in a different AWS account (cross-account access). For information about how to choose an authentication method for accessing AWS Glue and how to configure an access control policy in the AWS IAM Console, see [Authentication parameters for accessing AWS Glue](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-glue). ###### REST[​](#rest "Direct link to REST") note For detailed instructions on creating an Iceberg REST catalog for S3 tables, see [Create Iceberg REST Catalog for AWS S3 tables](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_rest_s3.md). If you use REST as metastore, you must specify the metastore type as REST (`"iceberg.catalog.type" = "rest"`). Configure `MetastoreParams` as follows: ```sql "iceberg.catalog.type" = "rest", "iceberg.catalog.uri" = "", "iceberg.catalog.security" = "oauth2", "iceberg.catalog.oauth2.credential" = "", "iceberg.catalog.warehouse" = "" ``` `MetastoreParams` for REST catalog: * `iceberg.catalog.type` * Required: Yes * Description: The type of metastore that you use for your Iceberg cluster. Set the value to `rest`. - * `iceberg.catalog.uri` * Required: Yes * Description: The URI of the REST service endpoint. Example: `https://api.tabular.io/ws`. - * `iceberg.catalog.view-endpoints-supported` * Required: No * Description: Whether to use the view endpoints to support view-related operations when the REST service of earlier versions that does not return endpoints in `CatalogConfig`. This parameter is used for backwards compatibility with REST servers of early versions. Default: `false`. * `iceberg.catalog.security` * Required: No * Description: The type of authorization protocol to use. Default: `NONE`. Valid values: `OAUTH2` and `JWT`. When this item is set to `OAUTH2`, either `token` or `credential` is required. When this item is set to `JWT`, the user is required to log in to the StarRocks cluster using the `JWT` method. You can omit `token` or `credential` and StarRocks will use the logged in user's JWT to access the catalog. * `iceberg.catalog.oauth2.token` * Required: No * Description: The bearer token used for interactions with the server. A `token` or `credential` is required for `OAUTH2` authorization protocol. Example: `AbCdEf123456`. * `iceberg.catalog.oauth2.credential` * Required: No * Description: The credential to exchange for a token in the OAuth2 client credentials flow with the server. A `token` or `credential` is required for `OAUTH2` authorization protocol. Example: `AbCdEf123456`. * `iceberg.catalog.oauth2.scope` * Required: No * Description: Scope to be used when communicating with the REST Catalog. Applicable only when `credential` is used. * `iceberg.catalog.oauth2.server-uri` * Required: No * Description: The endpoint to retrieve access token from OAuth2 Server. * `iceberg.catalog.vended-credentials-enabled` * Required: No * Description: Whether to use credentials provided by REST backend for file system access. Default: `true`. * `iceberg.catalog.warehouse` * Required: No * Description: The warehouse location or identifier of the Iceberg catalog. Example: `s3://my_bucket/warehouse_location` or `sandbox`. - * `iceberg.catalog.rest.nested-namespace-enabled` * Required: No * Description: Whether to support querying objects under nested namespace. Default: `false`. * `iceberg.catalog.rest.view-endpoints-enabled` * Required: No * Description: Whether to enable view endpoints for view-related operations. If set to `false`, view operations like `getView` will be disabled. Default: `true`. The following example creates an Iceberg catalog named `tabular` that uses Tabular as metastore: ```sql CREATE EXTERNAL CATALOG tabular PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "rest", "iceberg.catalog.uri" = "https://api.tabular.io/ws", "iceberg.catalog.oauth2.credential" = "t-5Ii8e3FIbT9m0:aaaa-3bbbbbbbbbbbbbbbbbbb", "iceberg.catalog.warehouse" = "sandbox" ); ``` The following example creates an Iceberg catalog named `smith_polaris` that uses Polaris as metastore: ```sql CREATE EXTERNAL CATALOG smith_polaris PROPERTIES ( "iceberg.catalog.uri" = "http://xxx.xx.xx.xxx:8181/api/catalog", "type" = "iceberg", "iceberg.catalog.type" = "rest", "iceberg.catalog.warehouse" = "starrocks_catalog", "iceberg.catalog.security" = "oauth2", "iceberg.catalog.oauth2.credential" = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "iceberg.catalog.oauth2.scope"='PRINCIPAL_ROLE:ALL' ); # `ns1.ns2.tpch_namespace` is a nested namespace create table smith_polaris.`ns1.ns2.tpch_namespace`.tbl (c1 string); mysql> select * from smith_polaris.`ns1.ns2.tpch_namespace`.tbl; +------+ | c1 | +------+ | 1 | | 2 | | 3 | +------+ 3 rows in set (0.34 sec) ``` The following example creates an Iceberg catalog named `r2` that uses Cloudflare R2 Data Catalog as metastore: ```sql CREATE EXTERNAL CATALOG r2 PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "rest", "iceberg.catalog.uri" = "", "iceberg.catalog.security" = "oauth2", "iceberg.catalog.oauth2.token" = "", "iceberg.catalog.warehouse" = "" ); SET CATALOG r2; CREATE DATABASE testdb; SHOW DATABASES FROM r2; +--------------------+ | Database | +--------------------+ | information_schema | | testdb | +--------------------+ 2 rows in set (0.66 sec) ``` The ``,``, and `` values are obtained from the [Cloudflare Dashboard as detailed here](https://developers.cloudflare.com/r2/data-catalog/get-started/#prerequisites). ###### JDBC[​](#jdbc "Direct link to JDBC") If you choose JDBC as the metastore of your data source, configure `MetastoreParams` as follows: ```sql "iceberg.catalog.type" = "jdbc", "iceberg.catalog.uri" = "", "iceberg.catalog.warehouse" = "" ``` The following table describes the parameter you need to configure in `MetastoreParams`. * `iceberg.catalog.type` * Required: Yes * Description: The type of metastore that you use for your Iceberg cluster. Set the value to `jdbc`. * `iceberg.catalog.uri` * Required: Yes * Description: The URI of your database. Format: `jdbc:[mysql\|postgresql]://:/`. * `iceberg.catalog.warehouse` * Required: Yes * Description: The warehouse location or identifier of the Iceberg catalog. Example: `s3://my_bucket/warehouse_location` . * `iceberg.catalog.jdbc.user` * Required: No * Description: The username for the database. * `iceberg.catalog.jdbc.password` * Required: No * Description: The password for the database. * `iceberg.catalog.jdbc.init-catalog-tables` * Required: No * Description: Whether to create the tables `iceberg_namespace_properties` and `iceberg_tables` for storing metadata in the database specified by `iceberg.catalog.uri`. The default value is `false`. Specify `true` if these two tables have not yet been created in the database specified by `iceberg.catalog.uri`. The following example creates an Iceberg catalog named `iceberg_jdbc` and uses JDBC as metastore: ```sql CREATE EXTERNAL CATALOG iceberg_jdbc PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "jdbc", "iceberg.catalog.warehouse" = "s3://my_bucket/warehouse_location", "iceberg.catalog.uri" = "jdbc:mysql://ip:port/db_name", "iceberg.catalog.jdbc.user" = "username", "iceberg.catalog.jdbc.password" = "password", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ); ``` If using MySQL or other custom JDBC drivers, the corresponding JAR files need to be placed in the `fe/lib` and `be/lib/jni-packages` directories. *** ###### `StorageCredentialParams`[​](#storagecredentialparams "Direct link to storagecredentialparams") A set of parameters about how StarRocks integrates with your storage system. This parameter set is optional. Note the following points: * If you use HDFS as storage, you do not need to configure `StorageCredentialParams` and can skip this section. If you use AWS S3, other S3-compatible storage system, Microsoft Azure Storage, or Google GCS as storage, you must configure `StorageCredentialParams`. * If you use Tabular as metastore, you do not need to configure `StorageCredentialParams` and can skip this section. If you use HMS or AWS Glue as metastore, you must configure `StorageCredentialParams`. Choose the tab that matches your storage type: * AWS S3 * HDFS * MinIO * Microsoft Azure Blob Storage * Google GCS ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") If you choose AWS S3 as storage for your Iceberg cluster, take one of the following actions: * To choose the instance profile-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "" ``` * To choose the assumed role-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.region" = "" ``` * To choose the IAM user-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "" ``` * To choose vended credential (supported from v4.0 onwards) with the REST catalog, configure `StorageCredentialParams` as follows: ```sql "aws.s3.region" = "" ``` `StorageCredentialParams` for AWS S3: ###### aws.s3.use\_instance\_profile[​](#awss3use_instance_profile "Direct link to aws.s3.use_instance_profile") * Required: Yes * Description: Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. ###### aws.s3.iam\_role\_arn[​](#awss3iam_role_arn "Direct link to aws.s3.iam_role_arn") * Required: No * Description: The ARN of the IAM role that has privileges on your AWS S3 bucket. If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter. ###### aws.s3.region[​](#awss3region "Direct link to aws.s3.region") * Required: Yes * Description: The region in which your AWS S3 bucket resides. Example: `us-west-1`. ###### aws.s3.access\_key[​](#awss3access_key "Direct link to aws.s3.access_key") * Required: No * Description: The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. ###### aws.s3.secret\_key[​](#awss3secret_key "Direct link to aws.s3.secret_key") * Required: No * Description: The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in AWS IAM Console, see [Authentication parameters for accessing AWS S3](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-s3). When using HDFS storage skip the storage credentials. ###### S3-compatible storage system[​](#s3-compatible-storage-system "Direct link to S3-compatible storage system") Iceberg catalogs support S3-compatible storage systems from v2.5 onwards. If you choose an S3-compatible storage system, such as MinIO, as storage for your Iceberg cluster, configure `StorageCredentialParams` as follows to ensure a successful integration: ```sql "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` `StorageCredentialParams` for MinIO and other S3 compatible systems: ###### aws.s3.enable\_ssl[​](#awss3enable_ssl "Direct link to aws.s3.enable_ssl") * Required: Yes * Description: Specifies whether to enable SSL connection.
Valid values: `true` and `false`. Default value: `true`. ###### aws.s3.enable\_path\_style\_access[​](#awss3enable_path_style_access "Direct link to aws.s3.enable_path_style_access") * Required: Yes * Description: Specifies whether to enable path-style access.
Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`.
Path-style URLs use the following format: `https://s3..amazonaws.com//`. For example, if you create a bucket named `DOC-EXAMPLE-BUCKET1` in the US West (Oregon) Region, and you want to access the `alice.jpg` object in that bucket, you can use the following path-style URL: `https://s3.us-west-2.amazonaws.com/DOC-EXAMPLE-BUCKET1/alice.jpg`. ###### aws.s3.endpoint[​](#awss3endpoint "Direct link to aws.s3.endpoint") * Required: Yes * Description: The endpoint that is used to connect to your S3-compatible storage system instead of AWS S3. ###### aws.s3.access\_key[​](#awss3access_key-1 "Direct link to aws.s3.access_key") * Required: Yes * Description: The access key of your IAM user. ###### aws.s3.secret\_key[​](#awss3secret_key-1 "Direct link to aws.s3.secret_key") * Required: Yes * Description: The secret key of your IAM user. ###### Microsoft Azure Storage[​](#microsoft-azure-storage "Direct link to Microsoft Azure Storage") Iceberg catalogs support Microsoft Azure Storage from v3.0 onwards. ###### Azure Blob Storage[​](#azure-blob-storage "Direct link to Azure Blob Storage") If you choose Blob Storage as storage for your Iceberg cluster, take one of the following actions: * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ``` * To choose the SAS Token authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ``` * To choose REST catalog with vended credential (supported from v4.0 onwards), you do not need to configure `StorageCredentialParams`. `StorageCredentialParams` for Microsoft Azure: ###### azure.blob.storage\_account[​](#azureblobstorage_account "Direct link to azure.blob.storage_account") * Required: Yes * Description: The username of your Blob Storage account. ###### azure.blob.shared\_key[​](#azureblobshared_key "Direct link to azure.blob.shared_key") * Required: Yes * Description: The shared key of your Blob Storage account. ###### azure.blob.account\_name[​](#azureblobaccount_name "Direct link to azure.blob.account_name") * Required: Yes * Description: The username of your Blob Storage account. ###### azure.blob.container[​](#azureblobcontainer "Direct link to azure.blob.container") * Required: Yes * Description: The name of the blob container that stores your data. ###### azure.blob.sas\_token[​](#azureblobsas_token "Direct link to azure.blob.sas_token") * Required: Yes * Description: The SAS token that is used to access your Blob Storage account. ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1 "Direct link to Azure Data Lake Storage Gen1") If you choose Data Lake Storage Gen1 as storage for your Iceberg cluster, take one of the following actions: * To choose the Managed Service Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.use_managed_service_identity" = "true" ``` Or: * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ``` ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2 "Direct link to Azure Data Lake Storage Gen2") If you choose Data Lake Storage Gen2 as storage for your Iceberg cluster, take one of the following actions: * To choose the Managed Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` Or: * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ``` Or: * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ``` * To choose the Workload Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_token_file" = "", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_token\_file | Yes | The absolute file path to the OAuth2 token file projected into the pod by the Azure Workload Identity webhook. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client ID (application ID) of the Azure AD application (user-assigned managed identity or app registration) associated with the workload identity. | * To choose REST catalog with vended credential (supported from v4.0 onwards), you do not need to configure `StorageCredentialParams`. ###### Google GCS[​](#google-gcs "Direct link to Google GCS") Iceberg catalogs support Google GCS from v3.0 onwards. If you choose Google GCS as storage for your Iceberg cluster, take one of the following actions: * To choose the VM-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.use_compute_engine_service_account" = "true" ``` * To choose the service account-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ``` * To choose the impersonation-based authentication method, configure `StorageCredentialParams` as follows: * Make a VM instance impersonate a service account: ```sql "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ``` * Make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account): ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` * To choose REST catalog with vended credential (supported from v4.0 onwards), you do not need to configure `StorageCredentialParams`. note When vended credentials are used, StarRocks accesses GCS directly with the token vended by the REST catalog. Any `gcp.gcs.impersonation_service_account` configured on the catalog is ignored for that access. `StorageCredentialParams` for Google GCS: ###### gcp.gcs.service\_account\_email[​](#gcpgcsservice_account_email "Direct link to gcp.gcs.service_account_email") * Default value: "" * Example: "" * Description: The email address in the JSON file generated at the creation of the service account. ###### gcp.gcs.service\_account\_private\_key\_id[​](#gcpgcsservice_account_private_key_id "Direct link to gcp.gcs.service_account_private_key_id") * Default value: "" * Example: "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" * Description: The private key ID in the JSON file generated at the creation of the service account. ###### gcp.gcs.service\_account\_private\_key[​](#gcpgcsservice_account_private_key "Direct link to gcp.gcs.service_account_private_key") * Default value: "" * Example: "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" * Description: The private key in the JSON file generated at the creation of the service account. ###### gcp.gcs.impersonation\_service\_account[​](#gcpgcsimpersonation_service_account "Direct link to gcp.gcs.impersonation_service_account") * Default value: "" * Example: "hello" * Description: The service account that you want to impersonate. *** ###### MetadataRelatedParams[​](#metadatarelatedparams "Direct link to MetadataRelatedParams") A set of parameters about cache of the Iceberg metadata in StarRocks. This parameter set is optional. From v3.3.3 onwards, StarRocks supports the [periodic metadata refresh strategy](#appendix-a-periodic-metadata-refresh-strategy). In most cases, you can ignore the parameters below and do not need to tune the policy parameters in it, because the default values of these parameters already provide you with performance out-of-the-box. You can adjust the Iceberg metadata parsing mode using the system variable [`plan_mode`](https://docs.starrocks.io/docs/sql-reference/System_variable.md#plan_mode). | **Parameter** | **Default** | **Description** | | -------------------------------------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_iceberg\_metadata\_cache | true | Whether to cache Iceberg-related metadata, including Table Cache, Partition Name Cache, and the Data File Cache and Delete Data File Cache in Manifest. | | iceberg\_manifest\_cache\_with\_column\_statistics | true | Whether to cache column statistics. When enabled, statistics are cached only for the columns where file-level min/max pruning is effective (partition source columns, sort key columns, and identifier columns), bounding the manifest cache footprint on wide tables. | | refresh\_iceberg\_manifest\_min\_length | 2 \* 1024 \* 1024 | The minimum Manifest file length that triggers a Data File Cache refresh. | | iceberg\_data\_file\_cache\_memory\_usage\_ratio | 0.1 | The maximum memory usage ratio for the data file Manifest cache. Supported from v3.5.6 onwards. | | iceberg\_delete\_file\_cache\_memory\_usage\_ratio | 0.1 | The maximum memory usage ratio for the delete file Manifest cache. Supported from v3.5.6 onwards. | | iceberg\_table\_cache\_refresh\_interval\_sec | 60 | The interval (in seconds) at which the asynchronous refresh of the Iceberg table cache is triggered. Supported from v3.5.7 onwards. | Starting from v3.4, StarRocks can obtain statistics of Iceberg tables by reading Iceberg metadata through setting the following parameters, without actively triggering the collection of Iceberg table statistics. | **Parameter** | **Default** | **Description** | | -------------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_get\_stats\_from\_external\_metadata | false | Whether to obtain statistics from Iceberg metadata. When this item is set to `true`, you can further control which type of statistics to collect through the session variable [`enable_get_stats_from_external_metadata`](https://docs.starrocks.io/docs/sql-reference/System_variable.md#enable_get_stats_from_external_metadata). | ##### Examples[​](#examples "Direct link to Examples") The following examples create an Iceberg catalog named `iceberg_catalog_hms` or `iceberg_catalog_glue`, depending on the type of metastore you use, to query data from your Iceberg cluster. Chose the tab that matches your storage type: * AWS S3 * HDFS * MinIO * Microsoft Azure Blob Storage * Google GCS ###### AWS S3[​](#aws-s3-1 "Direct link to AWS S3") ###### If you choose instance profile-based credential[​](#if-you-choose-instance-profile-based-credential "Direct link to If you choose instance profile-based credential") * If you use Hive metastore in your Iceberg cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Iceberg cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_glue PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` ###### If you choose assumed role-based credential[​](#if-you-choose-assumed-role-based-credential "Direct link to If you choose assumed role-based credential") * If you use Hive metastore in your HIceberg cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Iceberg cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_glue PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "arn:aws:iam::081976408565:role/test_glue_role", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` ###### If you choose IAM user-based credential[​](#if-you-choose-iam-user-based-credential "Direct link to If you choose IAM user-based credential") * If you use Hive metastore in your Iceberg cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Iceberg cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_glue PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` ###### If you choose vended credential[​](#if-you-choose-vended-credential "Direct link to If you choose vended credential") If you choose REST catalog with vended credential, run a command like below: ```sql CREATE EXTERNAL CATALOG polaris_s3 PROPERTIES ( "type" = "iceberg", "iceberg.catalog.uri" = "http://xxx:xxx/api/catalog", "iceberg.catalog.type" = "rest", "iceberg.catalog.rest.nested-namespace-enabled"="true", "iceberg.catalog.security" = "oauth2", "iceberg.catalog.oauth2.credential" = "xxxxx:xxxx", "iceberg.catalog.oauth2.scope"='PRINCIPAL_ROLE:ALL', "iceberg.catalog.warehouse" = "iceberg_catalog", "aws.s3.region" = "us-west-2" ); ``` ###### HDFS[​](#hdfs "Direct link to HDFS") If you use HDFS as storage, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` ###### Access an HA-enabled HDFS cluster[​](#access-an-ha-enabled-hdfs-cluster "Direct link to Access an HA-enabled HDFS cluster") If the target HDFS cluster has HA enabled, you can declare the HA configurations directly in `PROPERTIES`: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_ha PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "hadoop.security.authentication" = "simple", -- HDFS HA configurations "dfs.nameservices" = "my_cluster", "dfs.ha.namenodes.my_cluster" = "nn1,nn2", "dfs.namenode.rpc-address.my_cluster.nn1" = "host1:8020", "dfs.namenode.rpc-address.my_cluster.nn2" = "host2:8020", "dfs.client.failover.proxy.provider.my_cluster" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider", "fs.defaultFS" = "hdfs://my_cluster" ); ``` ###### Access multiple HDFS HA clusters simultaneously[​](#access-multiple-hdfs-ha-clusters-simultaneously "Direct link to Access multiple HDFS HA clusters simultaneously") If you need to query Iceberg tables that live on multiple independent HDFS HA clusters from the same StarRocks cluster, create one Catalog per HDFS cluster and let each Catalog carry its own `dfs.nameservices` and related HA parameters. The Catalogs do not interfere with each other. ```sql -- Catalog A: access HDFS HA cluster cluster_a CREATE EXTERNAL CATALOG iceberg_catalog_a PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://hms-a.example.com:9083", "hadoop.security.authentication" = "simple", "username" = "hdfs", "dfs.nameservices" = "cluster_a", "dfs.ha.namenodes.cluster_a" = "nn1,nn2", "dfs.namenode.rpc-address.cluster_a.nn1" = "host-a-1:8020", "dfs.namenode.rpc-address.cluster_a.nn2" = "host-a-2:8020", "dfs.client.failover.proxy.provider.cluster_a" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider", "fs.defaultFS" = "hdfs://cluster_a" ); -- Catalog B: access another HDFS HA cluster cluster_b CREATE EXTERNAL CATALOG iceberg_catalog_b PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://hms-b.example.com:9083", "hadoop.security.authentication" = "simple", "dfs.nameservices" = "cluster_b", "dfs.ha.namenodes.cluster_b" = "nn1,nn2", "dfs.namenode.rpc-address.cluster_b.nn1" = "host-b-1:8020", "dfs.namenode.rpc-address.cluster_b.nn2" = "host-b-2:8020", "dfs.client.failover.proxy.provider.cluster_b" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider", "fs.defaultFS" = "hdfs://cluster_b" ); ``` ###### S3-compatible storage system[​](#s3-compatible-storage-system-1 "Direct link to S3-compatible storage system") Use MinIO as an example. Run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.enable_ssl" = "true", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ); ``` ###### Microsoft Azure Storage[​](#microsoft-azure-storage-1 "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage-1 "Direct link to Azure Blob Storage") * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` * If you choose the SAS Token authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` * If you choose REST catalog with vended credential, run a command like below: ```sql CREATE EXTERNAL CATALOG polaris_azure PROPERTIES ( "type" = "iceberg", "iceberg.catalog.uri" = "http://xxx:xxx/api/catalog", "iceberg.catalog.type" = "rest", "iceberg.catalog.rest.nested-namespace-enabled"="true", "iceberg.catalog.security" = "oauth2", "iceberg.catalog.oauth2.credential" = "xxxxx:xxxx", "iceberg.catalog.oauth2.scope"='PRINCIPAL_ROLE:ALL', "iceberg.catalog.warehouse" = "iceberg_catalog" ); ``` ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1-1 "Direct link to Azure Data Lake Storage Gen1") * If you choose the Managed Service Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.use_managed_service_identity" = "true" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2-1 "Direct link to Azure Data Lake Storage Gen2") * If you choose the Managed Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` * If you choose the Workload Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` * If you choose REST catalog with vended credential, run a command like below: ```sql CREATE EXTERNAL CATALOG polaris_azure PROPERTIES ( "type" = "iceberg", "iceberg.catalog.uri" = "http://xxx:xxx/api/catalog", "iceberg.catalog.type" = "rest", "iceberg.catalog.rest.nested-namespace-enabled"="true", "iceberg.catalog.security" = "oauth2", "iceberg.catalog.oauth2.credential" = "xxxxx:xxxx", "iceberg.catalog.oauth2.scope"='PRINCIPAL_ROLE:ALL', "iceberg.catalog.warehouse" = "iceberg_catalog" ); ``` ###### Google GCS[​](#google-gcs-1 "Direct link to Google GCS") * If you choose the VM-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true" ); ``` * If you choose the service account-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ); ``` * If you choose the impersonation-based authentication method: * If you make a VM instance impersonate a service account, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ); ``` * If you make a service account impersonate another service account, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ); ``` * If you choose REST catalog with vended credential, run a command like below: ```sql CREATE EXTERNAL CATALOG polaris_gcp PROPERTIES ( "type" = "iceberg", "iceberg.catalog.uri" = "http://xxx:xxx/api/catalog", "iceberg.catalog.type" = "rest", "iceberg.catalog.rest.nested-namespace-enabled"="true", "iceberg.catalog.security" = "oauth2", "iceberg.catalog.oauth2.credential" = "xxxxx:xxxx", "iceberg.catalog.oauth2.scope"='PRINCIPAL_ROLE:ALL', "iceberg.catalog.warehouse" = "iceberg_catalog" ); ``` *** #### Use your catalog[​](#use-your-catalog "Direct link to Use your catalog") ##### View Iceberg catalogs[​](#view-iceberg-catalogs "Direct link to View Iceberg catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of an Iceberg catalog named `iceberg_catalog_glue`: ```sql SHOW CREATE CATALOG iceberg_catalog_glue; ``` *** ##### Switch to an Iceberg Catalog and a database in it[​](#switch-to-an-iceberg-catalog-and-a-database-in-it "Direct link to Switch to an Iceberg Catalog and a database in it") You can use one of the following methods to switch to an Iceberg catalog and a database in it: * Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to specify an Iceberg catalog in the current session, and then use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify an active database: ```sql -- Switch to a specified catalog in the current session: SET CATALOG -- Specify the active database in the current session: USE ``` * Directly use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to switch to an Iceberg catalog and a database in it: ```sql USE . ``` *** ##### Drop an Iceberg catalog[​](#drop-an-iceberg-catalog "Direct link to Drop an Iceberg catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop an external catalog. The following example drops an Iceberg catalog named `iceberg_catalog_glue`: ```sql DROP Catalog iceberg_catalog_glue; ``` *** ##### View the schema of an Iceberg table[​](#view-the-schema-of-an-iceberg-table "Direct link to View the schema of an Iceberg table") You can use one of the following syntaxes to view the schema of an Iceberg table: * View schema ```sql DESC[RIBE] .. ``` * View schema and location from the CREATE statement ```sql SHOW CREATE TABLE .. ``` *** ##### Query an Iceberg table[​](#query-an-iceberg-table "Direct link to Query an Iceberg table") 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your Iceberg cluster: ```sql SHOW DATABASES FROM ``` 2. [Switch to an Iceberg catalog and a database in it](#switch-to-an-iceberg-catalog-and-a-database-in-it). 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT count(*) FROM LIMIT 10 ``` *** ##### Iceberg DDL Operations[​](#iceberg-ddl-operations "Direct link to Iceberg DDL Operations") For DDL operations (CREATE/DROP DATABASE, CREATE/DROP TABLE, and CREATE/ALTER VIEW), see [Iceberg DDL operations](https://docs.starrocks.io/docs/data_source/catalog/iceberg/DDL.md). *** ##### Iceberg DML Operations[​](#iceberg-dml-operations "Direct link to Iceberg DML Operations") For DML operations (INSERT), see [Iceberg DML operations](https://docs.starrocks.io/docs/data_source/catalog/iceberg/DML.md). *** ##### Iceberg Stored Procedures[​](#iceberg-stored-procedures "Direct link to Iceberg Stored Procedures") For Iceberg stored procedures (using snapshots, performing manual Compaction), see [Iceberg Stored Procedures](https://docs.starrocks.io/docs/data_source/catalog/iceberg/procedures.md). *** ##### Configure metadata caching[​](#configure-metadata-caching "Direct link to Configure metadata caching") The metadata files of your Iceberg cluster may be stored in remote storage such as AWS S3 or HDFS. By default, StarRocks caches Iceberg metadata in memory. To accelerate queries, StarRocks adopts a two-level metadata caching mechanism, with which it can cache metadata both in memory and on disk. For each initial query, StarRocks caches their computation results. If any subsequent query that is semantically equivalent to a previous query is issued, StarRocks first attempts to retrieve the requested metadata from its caches, and it retrieves the metadata from the remote storage only when the metadata cannot be hit in its caches. StarRocks uses the Least Recently Used (LRU) algorithm to cache and evict data. The basic rules are as follows: * StarRocks first attempts to retrieve the requested metadata from the memory. If the metadata cannot be hit in the memory, StarRock attempts to retrieve the metadata from the disks. The metadata that StarRocks has retrieved from the disks will be loaded into the memory. If the metadata cannot be hit in the disks either, StarRock retrieves the metadata from the remote storage and caches the retrieved metadata in the memory. * StarRocks writes the metadata evicted out of the memory into the disks, but it directly discards the metadata evicted out of the disks. From v3.3.3 onwards, StarRocks supports the [periodic metadata refresh strategy](#appendix-a-periodic-metadata-refresh-strategy). You can adjust the Iceberg metadata caching plan using the system variable [`plan_mode`](https://docs.starrocks.io/docs/sql-reference/System_variable.md#plan_mode). ###### FE Configurations on Iceberg metadata caching[​](#fe-configurations-on-iceberg-metadata-caching "Direct link to FE Configurations on Iceberg metadata caching") ###### enable\_iceberg\_metadata\_disk\_cache[​](#enable_iceberg_metadata_disk_cache "Direct link to enable_iceberg_metadata_disk_cache") * Unit: N/A * Default value: `false` * Description: Specifies whether to enable the disk cache. ###### iceberg\_metadata\_cache\_disk\_path[​](#iceberg_metadata_cache_disk_path "Direct link to iceberg_metadata_cache_disk_path") * Unit: N/A * Default value: `StarRocksFE.STARROCKS_HOME_DIR + "/caches/iceberg"` * Description: The save path of cached metadata files on disk. ###### iceberg\_metadata\_disk\_cache\_capacity[​](#iceberg_metadata_disk_cache_capacity "Direct link to iceberg_metadata_disk_cache_capacity") * Unit: Bytes * Default value: `2147483648`, equivalent to 2 GB * Description: The maximum size of cached metadata allowed on disk. ###### iceberg\_metadata\_memory\_cache\_capacity[​](#iceberg_metadata_memory_cache_capacity "Direct link to iceberg_metadata_memory_cache_capacity") * Unit: Bytes * Default value: `536870912`, equivalent to 512 MB * Description: The maximum size of cached metadata allowed in memory. ###### iceberg\_metadata\_memory\_cache\_expiration\_seconds[​](#iceberg_metadata_memory_cache_expiration_seconds "Direct link to iceberg_metadata_memory_cache_expiration_seconds") * Unit: Seconds * Default value: `86500` * Description: The amount of time after which a cache entry in memory expires counting from its last access. ###### iceberg\_metadata\_disk\_cache\_expiration\_seconds[​](#iceberg_metadata_disk_cache_expiration_seconds "Direct link to iceberg_metadata_disk_cache_expiration_seconds") * Unit: Seconds * Default value: `604800`, equivalent to one week * Description: The amount of time after which a cache entry on disk expires counting from its last access. ###### iceberg\_metadata\_cache\_max\_entry\_size[​](#iceberg_metadata_cache_max_entry_size "Direct link to iceberg_metadata_cache_max_entry_size") * Unit: Bytes * Default value: `8388608`, equivalent to 8 MB * Description: The maximum size of a file that can be cached. Files whose size exceeds the value of this parameter cannot be cached. If a query requests these files, StarRocks retrieves them from the remote storage. ###### enable\_background\_refresh\_connector\_metadata[​](#enable_background_refresh_connector_metadata "Direct link to enable_background_refresh_connector_metadata") * Unit: - * Default value: true * Description: Whether to enable the periodic Iceberg metadata cache refresh. After it is enabled, StarRocks polls the metastore (Hive Metastore or AWS Glue) of your Iceberg cluster, and refreshes the cached metadata of the frequently accessed Iceberg catalogs to perceive data changes. `true` indicates to enable the Iceberg metadata cache refresh, and `false` indicates to disable it. ###### background\_refresh\_metadata\_interval\_millis[​](#background_refresh_metadata_interval_millis "Direct link to background_refresh_metadata_interval_millis") * Unit: Millisecond * Default value: 600000 * Description: The interval between two consecutive Iceberg metadata cache refreshes. - Unit: millisecond. ###### background\_refresh\_metadata\_time\_secs\_since\_last\_access\_sec[​](#background_refresh_metadata_time_secs_since_last_access_sec "Direct link to background_refresh_metadata_time_secs_since_last_access_sec") * Unit: Second * Default value: 86400 * Description: The expiration time of an Iceberg metadata cache refresh task. For the Iceberg catalog that has been accessed, if it has not been accessed for more than the specified time, StarRocks stops refreshing its cached metadata. For the Iceberg catalog that has not been accessed, StarRocks will not refresh its cached metadata. #### Appendix A: Periodic Metadata Refresh Strategy[​](#appendix-a-periodic-metadata-refresh-strategy "Direct link to Appendix A: Periodic Metadata Refresh Strategy") Iceberg supports [snapshots](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_timetravel.md). With the newest snapshot, you can get the newest result. Therefore, only cached snapshots can influence data freshness. As a result, you only need to pay attention to the refresh strategy of cache that contains snapshot. The following flowchart shows the time intervals on a timeline. ![Timeline for updating and discarding cached metadata](/assets/images/iceberg_catalog_timeline-7d661913987e269182bf4c22db626344.png) #### Appendix B: Metadata File Parsing[​](#appendix-b-metadata-file-parsing "Direct link to Appendix B: Metadata File Parsing") * **Distributed Plan for Large volume of Metadata** To handle large volume of metadata effectively, StarRocks employs a distributed approach using multiple BE and CN nodes. This method leverages the parallel computing capabilities of modern query engines, which can distribute tasks such as reading, decompressing, and filtering manifest files across multiple nodes. By processing these manifest files in parallel, the time required for metadata retrieval is significantly reduced, leading to faster job planning. This is particularly beneficial for large queries involving numerous manifest files, as it eliminates single-point bottlenecks and enhances overall query execution efficiency. * **Local Plan for Small volume of Metadata** For smaller queries, where the repeated decompression and parsing of manifest files can introduce unnecessary delays, a different strategy is employed. StarRocks caches deserialized memory objects, especially Avro files, to address this issue. By storing these deserialized files in memory, the system can bypass the decompression and parsing stages for subsequent queries. This caching mechanism allows direct access to the required metadata, significantly reducing retrieval times. As a result, the system becomes more responsive and better suited to meet high query demands and materialized view rewriting needs. * **Adaptive Metadata Retrieval Strategy** (Default) StarRocks is designed to automatically select the appropriate metadata retrieval method based on various factors, including the number of FE and BE/CN nodes, their CPU core counts, and the number of manifest files required for the current query. This adaptive approach ensures that the system dynamically optimizes metadata retrieval without the need for manual adjustment of metadata-related parameters. By doing so, StarRocks provides a seamless experience, balancing between distributed and local plans to achieve optimal query performance under different conditions. You can adjust the Iceberg metadata caching plan using the system variable [`plan_mode`](https://docs.starrocks.io/docs/sql-reference/System_variable.md#plan_mode). --- ### Iceberg DDL operations StarRocks Iceberg Catalog supports a variety of Data Definition Language (DDL) operations, including creating and managing databases, tables, and views. You must have the appropriate privileges to perform DDL operations. For more information about privileges, see [Privileges](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md). #### CREATE DATABASE[​](#create-database "Direct link to CREATE DATABASE") Creates a database in an Iceberg catalog. This feature is supported from v3.1 onwards. ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE DATABASE [IF NOT EXISTS] [PROPERTIES ("location" = ":////")] ``` ##### Parameters[​](#parameters "Direct link to Parameters") `location`: Specifies the file path where the database will be created. Both HDFS and cloud storage are supported. If not specified, the database is created in the default file path of the Iceberg catalog. The `prefix` varies based on the storage system: * HDFS: `hdfs` * Google GCS: `gs` * Azure Blob Storage (HTTP): `wasb` * Azure Blob Storage (HTTPS): `wasbs` * Azure Data Lake Storage Gen1: `adl` * Azure Data Lake Storage Gen2 (HTTP): `abfs` * Azure Data Lake Storage Gen2 (HTTPS): `abfss` * AWS S3 or S3-compatible storage: `s3` ##### Example[​](#example "Direct link to Example") ```sql CREATE DATABASE iceberg_db PROPERTIES ("location" = "s3://my_bucket/iceberg_db/"); ``` #### DROP DATABASE[​](#drop-database "Direct link to DROP DATABASE") Drops an empty database from an Iceberg catalog. This feature is supported from v3.1 onwards. note Only empty databases can be dropped. When you drop a database, the file path in the remote storage is not deleted. ##### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql DROP DATABASE [IF EXISTS] ``` ##### Example[​](#example-1 "Direct link to Example") ```sql DROP DATABASE iceberg_db; ``` #### CREATE TABLE[​](#create-table "Direct link to CREATE TABLE") Creates a table in an Iceberg database. This feature is supported from v3.1 onwards. ##### Syntax[​](#syntax-2 "Direct link to Syntax") ```sql CREATE TABLE [IF NOT EXISTS] [database.]table_name ( column_definition1[, column_definition2, ...], partition_column_definition1, partition_column_definition2, ... ) [partition_desc] [ORDER BY sort_desc] [PROPERTIES ("key" = "value", ...)] [AS SELECT query] ``` ##### Parameters[​](#parameters-1 "Direct link to Parameters") ###### `column_definition`[​](#column_definition "Direct link to column_definition") ```sql col_name col_type [COMMENT 'comment'] [DEFAULT default_value] ``` note All non-partition columns must use `NULL` as the default value. Partition columns must be defined after non-partition columns and cannot use `NULL` as the default value. ###### Default values[​](#default-values "Direct link to Default values") From v4.1 onwards, StarRocks supports setting default values for columns in Iceberg tables. This feature requires Iceberg format version 3 (`"format-version" = "3"`). **Usage:** * **Write-time filling**: When executing an INSERT statement, if a value is not specified for a column, the system automatically uses the column's default value. * **Schema Evolution filling**: When a new column is added to an existing table, reading old data files (that do not contain the new column) will use the new column's default value. **Syntax:** ```sql col_name col_type DEFAULT default_value ``` **Requirements:** * The table must use Iceberg format version 3 (`"format-version" = "3"`). * Default values for numeric types (INT, BIGINT, FLOAT, DOUBLE), BOOLEAN, STRING, and DATE/TIMESTAMP types must be wrapped in quotes. For example: `DEFAULT "18"`, `DEFAULT "100.0"`, `DEFAULT "true"`. **Examples:** * **Create a table with default values:** ```sql CREATE TABLE user_info ( id INT, name STRING, age INT DEFAULT "18", score DOUBLE DEFAULT "100.0", status STRING DEFAULT 'active', is_active BOOLEAN DEFAULT "true" ) PROPERTIES ("format-version" = "3"); ``` * **Add a column with a default value:** ```sql ALTER TABLE user_info ADD COLUMN bonus DOUBLE DEFAULT "50.5"; ``` * **Modify a column's default value:** ```sql ALTER TABLE user_info MODIFY COLUMN status STRING DEFAULT "inactive"; ``` ###### `partition_desc`[​](#partition_desc "Direct link to partition_desc") ```sql PARTITION BY (partition_expr[, partition_expr...]) ``` Each `partition_expr` can be: * `column_name` (identity transform) * `transform_expr(column_name)` * `transform_expr(column_name, parameter)` StarRocks supports partition transformation expressions defined in the Apache Iceberg specification. note Partition columns support all data types except FLOAT, DOUBLE, DECIMAL, and DATETIME. ###### `ORDER BY`[​](#order-by "Direct link to order-by") Specifies sort keys for the Iceberg table. This feature is supported from v4.0 onwards. ```sql ORDER BY (column_name [ASC | DESC] [NULLS FIRST | NULLS LAST], ...) ``` ###### `PROPERTIES`[​](#properties "Direct link to properties") Key table properties: * `location`: File path for the table. Required when using AWS Glue without database-level location. * `file_format`: File format. Only `parquet` (Default) is supported. * `compression_codec`: Compression algorithm. Options: SNAPPY, GZIP, ZSTD, LZ4 (Default: `zstd`). ##### Examples[​](#examples "Direct link to Examples") * **Create a non-partitioned table:** ```sql CREATE TABLE unpartition_tbl ( id int, score double ); ``` * **Create a partitioned table:** ```sql CREATE TABLE partition_tbl ( action varchar(20), id int, dt date ) PARTITION BY (id, dt); ``` * **Create a table with hidden partitions:** ```sql CREATE TABLE hidden_partition_tbl ( action VARCHAR(20), id INT, dt DATE ) PARTITION BY bucket(id, 10), year(dt); ``` * **CREATE TABLE AS SELECT:** ```sql CREATE TABLE new_tbl PARTITION BY (id, dt) AS SELECT * FROM existing_tbl; ``` #### ALTER TABLE to evolve partition spec[​](#alter-table-to-evolve-partition-spec "Direct link to ALTER TABLE to evolve partition spec") Modifies an Iceberg table's partition spec by adding or dropping partition columns. ##### Syntax[​](#syntax-3 "Direct link to Syntax") ```sql ALTER TABLE [catalog.][database.]table_name ADD PARTITION COLUMN partition_expr [, partition_expr ...]; ALTER TABLE [catalog.][database.]table_name DROP PARTITION COLUMN partition_expr [, partition_expr ...]; ``` Supported `partition_expr` formats: * Column name (identity transform) * Transform expressions: `year()`, `month()`, `day()`, `hour()`, `truncate()`, `bucket()` ##### Examples[​](#examples-1 "Direct link to Examples") * **Add partition columns:** ```sql ALTER TABLE sales_data ADD PARTITION COLUMN month(sale_date), bucket(customer_id, 10); ``` * **Drop partition column:** ```sql ALTER TABLE sales_data DROP PARTITION COLUMN day(sale_date); ``` #### DROP TABLE[​](#drop-table "Direct link to DROP TABLE") Drops an Iceberg table. This feature is supported from v3.1 onwards. When you drop a table, the file path and data in the remote storage are not deleted by default. ##### Syntax[​](#syntax-4 "Direct link to Syntax") ```sql DROP TABLE [IF EXISTS] [FORCE] ``` ##### Parameters[​](#parameters-2 "Direct link to Parameters") * `FORCE`: When specified, the table data in the remote storage is deleted, while the file path is retained. ##### Example[​](#example-2 "Direct link to Example") ```sql DROP TABLE iceberg_db.sales_data; -- Force drop the table with its data DROP TABLE iceberg_db.temp_data FORCE; ``` #### CREATE VIEW[​](#create-view "Direct link to CREATE VIEW") Creates an Iceberg view. This feature is supported from v3.5 onwards. Creating an Iceberg view with PROPERTIES is supported from v4.0.3 onwards. ##### Syntax[​](#syntax-5 "Direct link to Syntax") ```sql CREATE VIEW [IF NOT EXISTS] [..] ( [COMMENT 'column comment'] [, [COMMENT 'column comment'], ...] ) [COMMENT 'view comment'] [PROPERTIES ("key" = "value", ...)] AS ``` ##### Example[​](#example-3 "Direct link to Example") * **Create a regular Iceberg view:** ```sql CREATE VIEW IF NOT EXISTS iceberg_db.sales_summary AS SELECT region, SUM(amount) as total_sales FROM iceberg_db.sales GROUP BY region; ``` * **Create an Iceberg view with properties:** ```sql CREATE VIEW IF NOT EXISTS iceberg_db.sales_summary PROPERTIES ( "key1" = "value1" ) AS SELECT region, SUM(amount) as total_sales FROM iceberg_db.sales GROUP BY region; ``` #### ALTER VIEW to update StarRocks dialect[​](#alter-view-to-update-starrocks-dialect "Direct link to ALTER VIEW to update StarRocks dialect") Adds or modifies StarRocks dialect for an existing Iceberg view. This feature is supported from v3.5 onwards. note You can define only one StarRocks dialect for each Iceberg view. ##### Syntax[​](#syntax-6 "Direct link to Syntax") ```sql ALTER VIEW [..] ( [, ] ) { ADD | MODIFY } DIALECT ``` ##### Examples[​](#examples-2 "Direct link to Examples") * **Add StarRocks dialect:** ```sql ALTER VIEW iceberg_db.spark_view ADD DIALECT SELECT k1, k2 FROM iceberg_db.source_table; ``` * **Modify StarRocks dialect:** ```sql ALTER VIEW iceberg_db.spark_view MODIFY DIALECT SELECT k1, k2, k3 FROM iceberg_db.source_table; ``` --- ### Iceberg DML operations StarRocks Iceberg Catalog supports a variety of Data Manipulation Language (DML) operations, including inserting data into Iceberg tables. You must have the appropriate privileges to perform DML operations. For more information about privileges, see [Privileges](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md). #### INSERT[​](#insert "Direct link to INSERT") Inserts data into an Iceberg table. This feature is supported from v3.1 onwards. Similar to loading data into StarRocks native tables, if you have the [INSERT privilege](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md#table) on an Iceberg table, you can use the [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) statement to sink the data to the Iceberg table. Currently, only Parquet-formatted Iceberg tables are supported. ##### Syntax[​](#syntax "Direct link to Syntax") ```sql INSERT {INTO | OVERWRITE} [ (column_name [, ...]) ] { VALUES ( { expression | DEFAULT } [, ...] ) [, ...] | query } -- If you want to sink data to specified partitions, use the following syntax: INSERT {INTO | OVERWRITE} PARTITION (par_col1= [, par_col2=...]) { VALUES ( { expression | DEFAULT } [, ...] ) [, ...] | query } ``` note `NULL` values are not allowed in partition columns. Therefore, you must make sure that no empty values are loaded into the partition columns of the Iceberg table. ##### Parameters[​](#parameters "Direct link to Parameters") ###### INTO[​](#into "Direct link to INTO") Appends the data to the Iceberg table. ###### OVERWRITE[​](#overwrite "Direct link to OVERWRITE") Overwrites the existing data of the Iceberg table. ###### column\_name[​](#column_name "Direct link to column_name") The name of the destination column to which you want to load data. You can specify one or more columns. Multiple columns are separated with commas (`,`). * You can only specify columns that actually exist in the Iceberg table. * The destination columns must include the partition columns of the Iceberg table. * The destination columns are mapped one on one in sequence to the columns in the SELECT statement (source columns), regardless of what the destination column names are. * If no destination columns are specified, the data is loaded into all columns of the Iceberg table. * If a non-partition source column cannot be mapped to any destination column, StarRocks writes the default value `NULL` to the destination column. * If the data types of the source and destination columns mismatch, StarRocks performs an implicit conversion on the mismatched columns. If the conversion fails, a syntax parsing error will be returned. note You cannot specify the `column_name` property if you have specified the PARTITION clause. ###### expression[​](#expression "Direct link to expression") Expression that assigns values to the destination column. ###### DEFAULT[​](#default "Direct link to DEFAULT") Assigns a default value to the destination column. ###### query[​](#query "Direct link to query") Query statement whose result will be loaded into the Iceberg table. It can be any SQL statement supported by StarRocks. ###### PARTITION[​](#partition "Direct link to PARTITION") The partitions into which you want to load data. You must specify all partition columns of the Iceberg table in this property. The partition columns that you specify in this property can be in a different sequence than the partition columns that you have defined in the table creation statement. note You cannot specify the `column_name` property if you have specified the PARTITION clause. ##### Examples[​](#examples "Direct link to Examples") 1. Insert three data rows into the `partition_tbl_1` table: ```sql INSERT INTO partition_tbl_1 VALUES ("buy", 1, "2023-09-01"), ("sell", 2, "2023-09-02"), ("buy", 3, "2023-09-03"); ``` 2. Insert the result of a SELECT query, which contains simple computations, into the `partition_tbl_1` table: ```sql INSERT INTO partition_tbl_1 (id, action, dt) SELECT 1+1, 'buy', '2023-09-03'; ``` 3. Insert the result of a SELECT query, which reads data from the `partition_tbl_1` table, into the same table: ```sql INSERT INTO partition_tbl_1 SELECT 'buy', 1, date_add(dt, INTERVAL 2 DAY) FROM partition_tbl_1 WHERE id=1; ``` 4. Insert the result of a SELECT query into the partitions that meet two conditions, `dt='2023-09-01'` and `id=1`, of the `partition_tbl_2` table: ```sql INSERT INTO partition_tbl_2 SELECT 'order', 1, '2023-09-01'; ``` Or ```sql INSERT INTO partition_tbl_2 partition(dt='2023-09-01',id=1) SELECT 'order'; ``` 5. Overwrite all `action` column values in the partitions that meet two conditions, `dt='2023-09-01'` and `id=1`, of the `partition_tbl_1` table with `close`: ```sql INSERT OVERWRITE partition_tbl_1 SELECT 'close', 1, '2023-09-01'; ``` Or ```sql INSERT OVERWRITE partition_tbl_1 partition(dt='2023-09-01',id=1) SELECT 'close'; ``` #### DELETE[​](#delete "Direct link to DELETE") You can use the DELETE statement to delete data from Iceberg tables based on specified conditions. This feature is supported from v4.1 and later. ##### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql DELETE FROM WHERE ``` ##### Parameters[​](#parameters-1 "Direct link to Parameters") * `table_name`: The name of the Iceberg table you want to delete data from. You can use: * Fully qualified name: `catalog_name.database_name.table_name` * Database-qualified name (after setting catalog): `database_name.table_name` * Table name only (after setting catalog and database): `table_name` * `condition`: The condition to identify which rows to delete. It can include: * Comparison operators: `=`, `!=`, `>`, `<`, `>=`, `<=`, `<>` * Logical operators: `AND`, `OR`, `NOT` * `IN` and `NOT IN` clauses * `BETWEEN` and `LIKE` operators * `IS NULL` and `IS NOT NULL` * Sub-queries with `IN` or `EXISTS` ##### Examples[​](#examples-1 "Direct link to Examples") ###### Basic DELETE operations[​](#basic-delete-operations "Direct link to Basic DELETE operations") Delete rows matching a simple condition: ```sql DELETE FROM iceberg_catalog.db.table1 WHERE id = 3; ``` ###### DELETE with IN and NOT IN[​](#delete-with-in-and-not-in "Direct link to DELETE with IN and NOT IN") Delete multiple rows using IN clause: ```sql DELETE FROM iceberg_catalog.db.table1 WHERE id IN (18, 20, 22); DELETE FROM iceberg_catalog.db.table1 WHERE id NOT IN (100, 101, 102); ``` ###### DELETE with logical operators[​](#delete-with-logical-operators "Direct link to DELETE with logical operators") Combine multiple conditions: ```sql DELETE FROM iceberg_catalog.db.table1 WHERE age > 30 AND salary < 70000; DELETE FROM iceberg_catalog.db.table1 WHERE status = 'inactive' OR last_login < '2023-01-01'; ``` ###### DELETE with pattern matching[​](#delete-with-pattern-matching "Direct link to DELETE with pattern matching") Use LIKE for pattern-based deletion: ```sql DELETE FROM iceberg_catalog.db.table1 WHERE name LIKE 'A%'; DELETE FROM iceberg_catalog.db.table1 WHERE email LIKE '%@example.com'; ``` ###### DELETE with range conditions[​](#delete-with-range-conditions "Direct link to DELETE with range conditions") Use BETWEEN for range-based deletion: ```sql DELETE FROM iceberg_catalog.db.table1 WHERE age BETWEEN 30 AND 40; DELETE FROM iceberg_catalog.db.table1 WHERE created_date BETWEEN '2023-01-01' AND '2023-12-31'; ``` ###### DELETE with NULL checks[​](#delete-with-null-checks "Direct link to DELETE with NULL checks") Delete rows with or without NULL values: ```sql DELETE FROM iceberg_catalog.db.table1 WHERE name IS NULL; DELETE FROM iceberg_catalog.db.table1 WHERE email IS NULL AND phone IS NULL; DELETE FROM iceberg_catalog.db.table1 WHERE age IS NOT NULL; ``` ###### DELETE with sub-queries[​](#delete-with-sub-queries "Direct link to DELETE with sub-queries") Use sub-queries to identify rows to delete: ```sql -- DELETE with IN sub-query DELETE FROM iceberg_catalog.db.table1 WHERE id IN (SELECT id FROM temp_table WHERE expired = true); -- DELETE with EXISTS sub-query DELETE FROM iceberg_catalog.db.table1 t1 WHERE EXISTS (SELECT user_id FROM inactive_users t2 WHERE t2.user_id = t1.user_id); ``` #### TRUNCATE[​](#truncate "Direct link to TRUNCATE") You can use the TRUNCATE TABLE statement to quickly delete all data from Iceberg tables. ##### Syntax[​](#syntax-2 "Direct link to Syntax") ```sql TRUNCATE TABLE ``` ##### Parameters[​](#parameters-2 "Direct link to Parameters") * `table_name`: The name of the Iceberg table that you want to truncate data from. You can use: * Fully qualified name: `catalog_name.database_name.table_name` * Database-qualified name (after setting catalog): `database_name.table_name` * Table name only (after setting catalog and database): `table_name` ##### Examples[​](#examples-2 "Direct link to Examples") ###### Example 1: Truncate a table using fully qualified name[​](#example-1-truncate-a-table-using-fully-qualified-name "Direct link to Example 1: Truncate a table using fully qualified name") ```sql TRUNCATE TABLE iceberg_catalog.my_db.my_table; ``` ###### Example 2: Truncate a table after setting catalog[​](#example-2-truncate-a-table-after-setting-catalog "Direct link to Example 2: Truncate a table after setting catalog") ```sql SET CATALOG iceberg_catalog; TRUNCATE TABLE my_db.my_table; ``` ###### Example 3: Truncate a table after setting catalog and database[​](#example-3-truncate-a-table-after-setting-catalog-and-database "Direct link to Example 3: Truncate a table after setting catalog and database") ```sql SET CATALOG iceberg_catalog; USE my_db; TRUNCATE TABLE my_table; ``` --- ### Iceberg Metadata Tables This topic describes how to inspect the metadata information of Iceberg tables in StarRocks. #### Overview[​](#overview "Direct link to Overview") From V3.4.1 onwards, StarRocks supports Iceberg metadata tables. These metadata tables contain a variety of information about Iceberg tables, such as table change history, snapshots, and manifests. You can query each metadata table by appending the metadata table name to the original table name. Currently, StarRocks supports the following Iceberg metadata tables: | Metadata table | Description | | ---------------------- | ----------------------------------------------------------------------------------------- | | `history` | Shows a log of metadata changes made to the table. | | `metadata_log_entries` | Shows the metadata log entries for the table. | | `snapshots` | Shows details about the table snapshots. | | `manifests` | Shows an overview of the manifests associated with the snapshots in the table’s log. | | `partitions` | Shows details about the partitions in the table. | | `files` | Shows details about the data files and delete files in the current snapshot of the table. | | `refs` | Shows details about the Iceberg references, including branches and tags. | #### Iceberg v3 Row Lineage Metadata Columns[​](#iceberg-v3-row-lineage-metadata-columns "Direct link to Iceberg v3 Row Lineage Metadata Columns") From v4.1 onwards, for Iceberg v3 tables (format-version = 3), StarRocks supports querying the following Row Lineage metadata columns: | Metadata Column | Description | | ------------------------------- | ------------------------------------------------------------------------------------- | | `_row_id` | Unique row identifier within the table (BIGINT). Format: `firstRowId + row_position`. | | `_last_updated_sequence_number` | The commit sequence number when the row was last updated (BIGINT). | Usage: ```sql SELECT _row_id, _last_updated_sequence_number, * FROM [.][.]table; ``` note * The `_row_id` column requires data files to have `firstRowId` metadata. If a data file is missing `firstRowId`, the query will fail with an error. * For newly inserted data, `_row_id` is computed as `firstRowId + row_position` and `_last_updated_sequence_number` is the file-level `dataSequenceNumber`. * After compaction (e.g., Iceberg OPTIMIZE / rewrite-data-files), if the compactor writes `_row_id` and `_last_updated_sequence_number` as physical columns in the data files (as required by the Iceberg v3 spec), StarRocks reads the per-row values from the physical columns, preserving row lineage across compaction. * These metadata columns are only available for Iceberg v3 tables (format-version = 3). #### `history` table[​](#history-table "Direct link to history-table") Usage: ```sql SELECT * FROM [.][.]table$history; ``` Output: | Field | Description | | --------------------- | ------------------------------------------------------------- | | made\_current\_at | The time when the snapshot became the current snapshot. | | snapshot\_id | The ID of the snapshot. | | parent\_id | The ID of the parent snapshot. | | is\_current\_ancestor | Whether this snapshot is an ancestor of the current snapshot. | #### `metadata_log_entries` table[​](#metadata_log_entries-table "Direct link to metadata_log_entries-table") Usage: ```sql SELECT * FROM [.][.]table$metadata_log_entries; ``` Output: | Field | Description | | ------------------------ | ------------------------------------------------------------ | | timestamp | The time when the metadata was recorded. | | file | The location of the metadata file. | | latest\_snapshot\_id | The ID of the latest snapshot when the metadata was updated. | | latest\_schema\_id | The ID of the latest schema when the metadata was updated. | | latest\_sequence\_number | The data sequence number of the metadata file. | #### `snapshots` table[​](#snapshots-table "Direct link to snapshots-table") Usage: ```sql SELECT * FROM [.][.]table$snapshots; ``` Output: | Field | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | committed\_at | The time when the snapshot was committed. | | snapshot\_id | The ID for the snapshot. | | parent\_id | The ID for the parent snapshot. | | operation | The type of operation performed on the Iceberg table. Valid values:- `append`: New data is appended.
- `replace`: Files are removed and replaced without changing the data in the table.
- `overwrite`: Old data is overwritten by new data.
- `delete`: Data is deleted from the table. | | manifest\_list | The list of Avro manifest files that contain detailed information about snapshot changes. | | summary | A summary of the changes made from the previous snapshot to the current snapshot. | #### `manifests` table[​](#manifests-table "Direct link to manifests-table") Usage: ```sql SELECT * FROM [.][.]table$manifests; ``` Output: | Field | Description | | ---------------------------- | --------------------------------------------------------------------------------------- | | path | The location of the manifest file. | | length | The length of the manifest file. | | partition\_spec\_id | The ID for the partition specification that is used to write the manifest file. | | added\_snapshot\_id | The ID of the snapshot during which this manifest entry has been added. | | added\_data\_files\_count | The number of data files with status `ADDED` in the manifest file. | | added\_rows\_count | The total number of rows in all data files with status `ADDED` in the manifest file. | | existing\_data\_files\_count | The number of data files with status `EXISTING` in the manifest file. | | existing\_rows\_count | The total number of rows in all data files with status `EXISTING` in the manifest file. | | deleted\_data\_files\_count | The number of data files with status `DELETED` in the manifest file. | | deleted\_rows\_count | The total number of rows in all data files with status `DELETED` in the manifest file. | | partition\_summaries | Partition range metadata. | #### `partitions` table[​](#partitions-table "Direct link to partitions-table") Usage: ```sql SELECT * FROM [.][.]table$partitions; ``` Output: | Field | Description | | ---------------------------------- | ------------------------------------------------------------------------- | | partition\_value | The mapping of the partition column names to the partition column values. | | spec\_id | The partition Spec ID of files. | | record\_count | The number of records in the partition. | | file\_count | The number of files mapped in the partition. | | total\_data\_file\_size\_in\_bytes | The size of all the data files in the partition. | | position\_delete\_record\_count | The total row count of Position Delete files in the partition. | | position\_delete\_file\_count | The number of Position Delete files in the partition. | | equality\_delete\_record\_count | The total row count of Equality Delete files in the partition. | | equality\_delete\_file\_count | The number of Position Equality files in the partition. | | last\_updated\_at | The time when the partition was updated most recently. | #### `files` table[​](#files-table "Direct link to files-table") Usage: ```sql SELECT * FROM [.][.]table$files; ``` Output: | Field | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------ | | content | The type of content stored in the file. Valid values: `DATA(0)`, `POSITION_DELETES(1)`, and `EQUALITY_DELETES(2)`. | | file\_path | The location of the data file. | | file\_format | The format of the data file. | | spec\_id | The Spec ID that is used to track the file containing a row. | | record\_count | The number of entries contained in the data file. | | file\_size\_in\_bytes | The size of the data file. | | column\_sizes | The mapping between the Iceberg column ID and its corresponding size in the file. | | value\_counts | The mapping between the Iceberg column ID and its corresponding count of entries in the file. | | null\_value\_counts | The mapping between the Iceberg column ID and its corresponding count of `NULL` values in the file. | | nan\_value\_counts | The mapping between the Iceberg column ID and its corresponding count of non- numerical values in the file. | | lower\_bounds | The mapping between the Iceberg column ID and its corresponding lower bound in the file. | | upper\_bounds | The mapping between the Iceberg column ID and its corresponding upper bound in the file. | | split\_offsets | The list of recommended split locations. | | sort\_id | The ID representing sort order for this file. | | equality\_ids | The set of field IDs used for equality comparison in equality delete files. | | key\_metadata | The metadata about the encryption key that is used to encrypt this file, if applicable. | #### `refs` table[​](#refs-table "Direct link to refs-table") Usage: ```sql SELECT * FROM [.][.]table$refs; ``` Output: | Field | Description | | --------------------------- | --------------------------------------------------------------------------------------------------------- | | name | The name of the reference. | | type | The type of the reference. Valid values: `BRANCH` or `TAG`. | | snapshot\_id | The snapshot ID of the reference. | | max\_reference\_age\_in\_ms | The maximum age of the reference before it could be expired. | | min\_snapshots\_to\_keep | For branch only, the minimum number of snapshots to keep in a branch. | | max\_snapshot\_age\_in\_ms | For branch only, the max snapshot age allowed in a branch. Older snapshots in the branch will be expired. | --- ### Create Iceberg REST Catalog for AWS S3 Tables This article explains how to create Iceberg REST Catalog in StarRocks for access to data in AWS S3 tables through the AWS Glue Iceberg REST endpoint. The AWS Glue Iceberg REST endpoint implements the [Iceberg REST Catalog Open API specification](https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml), which provides a standardized interface for interacting with Iceberg tables. To access S3 tables using this endpoint, you need to configure credentials by combining IAM policies and AWS Lake Formation authorization. The following sections will guide you through the access permission setup, including defining required policies, establishing Lake Formation permissions at database and table levels, and using StarRocks to create an Iceberg REST catalog for accessing S3 tables. #### (Optional) Create a table bucket[​](#optional-create-a-table-bucket "Direct link to (Optional) Create a table bucket") You can skip this step if you already have a table bucket for S3 tables. 1. Sign in to the [Amazon S3 Console](https://console.aws.amazon.com/s3) as a user with administrator privileges. 2. In the upper-right corner of the page, select your AWS region. 3. In the left-side navigation pane, choose **Table buckets** from the navigation panel. 4. Click **Create table bucket** to create a table bucket. 5. After creating the table bucket, select it and click **Create table with Athena**. 6. Create a namespace. 7. After creating the namespace, click **Create table with Athena** again to create a table. > **NOTE** > > You can create a Database and Table using Athena, and then query them using StarRocks. Alternatively, you can just create a table bucket, and then use StarRocks to create the database and table. #### Create IAM Policy[​](#create-iam-policy "Direct link to Create IAM Policy") To access S3 tables via the AWS Glue endpoint, create an IAM Policy with permissions for AWS Glue and Lake Formation operations: 1. Sign in to the [Amazon IAM Console](https://console.aws.amazon.com/iam) as a user with administrator privileges. 2. In the upper-right corner of the page, select your AWS region. 3. In the left-side navigation pane, choose **Policies** from the navigation panel. 4. Choose **Create a policy**, select **JSON** in the policy editor. 5. Add the following policy to grant access to AWS Glue and Lake Formation actions. ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "glue:GetCatalog", "glue:GetDatabase", "glue:GetDatabases", "glue:GetTable", "glue:GetTables", "glue:CreateTable", "glue:UpdateTable" ], "Resource": [ "arn:aws:glue:::catalog", "arn:aws:glue:::catalog/s3tablescatalog", "arn:aws:glue:::catalog/s3tablescatalog/", "arn:aws:glue:::table/s3tablescatalog///*", "arn:aws:glue:::database/s3tablescatalog//" ] }, { "Effect": "Allow", "Action": [ "lakeformation:GetDataAccess" ], "Resource": "*" } ] } ``` After creating the IAM policy, attach it to the target IAM user: 1. Choose **Users** from the navigation panel. 2. Select the user that requires the S3 table access. 3. Click **Add permissions** and select **Attach policies directly**. 4. Attach the newly created policy. #### Manage Permissions via Lake Formation[​](#manage-permissions-via-lake-formation "Direct link to Manage Permissions via Lake Formation") To access S3 tables, StarRocks requires Lake Formation to first set up permissions that allow third-party query engines to access S3 tables. 1. Sign in to the [Lake Formation Console](https://console.aws.amazon.com/lakeformation) as a user with administrator privileges. 2. In the upper-right corner of the page, select your AWS region. 3. In the left-side navigation pane, choose **Application integration settings** from the navigation panel. 4. Choose **Allow external engines to access data in Amazon S3 locations with full table access.** Next, grant the above IAM User access permissions in Lake Formation. 1. In the left-side navigation pane of the [Lake Formation Console](https://console.aws.amazon.com/lakeformation), choose **Data permissions** from the navigation panel. 2. Click **Grant**. 3. In the **Principals** section, choose **IAM users and roles**, and select the authorized IAM user from the **IAM users and roles** drop-down list. 4. In the **LF-Tags or catalog resources** section, choose **Named Data Catalog resources**, and select your table bucket you created in the **Catalogs** drop-down list. 5. In the **Catalog permissions** section, choose **Super** for **Catalog permissions**. 6. Click **Grant**. > **NOTE** > > Here, Super permissions are granted for testing convenience. You need to assign appropriate permissions based on actual requirements in a production environment. #### Create Iceberg REST Catalog[​](#create-iceberg-rest-catalog "Direct link to Create Iceberg REST Catalog") Create an Iceberg REST catalog in StarRocks: ```sql CREATE EXTERNAL CATALOG starrocks_lakehouse_s3tables PROPERTIES( "type"="iceberg", "iceberg.catalog.type" = "rest", "iceberg.catalog.uri" = "https://glue..amazonaws.com/iceberg", "iceberg.catalog.rest.sigv4-enabled" = "true", "iceberg.catalog.rest.signing-name" = "glue", "iceberg.catalog.rest.access-key-id" = "", "iceberg.catalog.rest.secret-access-key" = "", "iceberg.catalog.warehouse" = ":s3tablescatalog/", "aws.s3.region" = "" ); ``` You can then create databases and tables and run queries in it. Example: ```sql -- Switch to the catalog StarRocks> SET CATALOG starrocks_lakehouse_s3tables; -- Create database StarRocks> CREATE DATABASE s3table_db; Query OK, 0 rows affected -- Switch database StarRocks> USE s3table_db; Database changed -- Create table StarRocks> CREATE TABLE taxis ( trip_id BIGINT, trip_distance FLOAT, fare_amount DOUBLE, store_and_fwd_flag STRING, vendor_id BIGINT ) PARTITION BY (vendor_id); Query OK, 0 rows affected -- Insert data StarRocks> INSERT INTO taxis VALUES (1000371, 1.8, 15.32, 'N', 1), (1000372, 2.5, 22.15, 'N', 2), (1000373, 0.9, 9.01, 'N', 2), (1000374, 8.4, 42.13, 'Y', 1); Query OK, 4 rows affected -- Query data StarRocks> SELECT * FROM taxis; +---------+---------------+-------------+--------------------+-----------+ | trip_id | trip_distance | fare_amount | store_and_fwd_flag | vendor_id | +---------+---------------+-------------+--------------------+-----------+ | 1000372 | 2.5 | 22.15 | N | 2 | | 1000373 | 0.9 | 9.01 | N | 2 | | 1000371 | 1.8 | 15.32 | N | 1 | | 1000374 | 8.4 | 42.13 | Y | 1 | +---------+---------------+-------------+--------------------+-----------+ 4 rows in set ``` --- ### Security Setup for Iceberg REST Catalog In scenarios where multiple nodes collaboratively access the same data lake, the core challenge is **how to achieve secure, unified, and auditable permission management**. In traditional models, users need to configure storage credentials and perform local permission control individually for each node, which not only increases maintenance costs but also exposes risks of credential leakage and inconsistent permissions. By integrating Iceberg REST Catalog with StarRocks, secure access and unified permission management can be achieved through a combination of JWT (JSON Web Token) authentication and Vended Credentials (temporary credentials). This setup helps to: * **Reduce credential risks**: No need to store high-privilege account information within StarRocks. Credentials are issued temporarily by the Catalog, avoiding leakage. * **Unified and simplified permissions**: Access control for all databases, tables, and views is centrally managed by the Catalog, ensuring consistency across different nodes and avoiding redundant configuration. * **Compliance and simplified operations**: User actions are traceable, facilitating audits. It also reduces the cost of maintaining permissions and storage credentials within StarRocks. #### Security mechanisms[​](#security-mechanisms "Direct link to Security mechanisms") * **JWT authentication** * When a user logs into StarRocks, the JWT Token obtained can be passed through to the Iceberg REST Session Catalog. * The Catalog will authenticate the user based on the JWT Token and execute queries under the real user’s identity. * The benefit is that StarRocks doesn’t need to store high-privilege accounts, which significantly lowers security risks. * **Vended credentials** * After authentication, the Catalog can generate temporary storage access credentials for the user. * Users do not need to configure credentials for the storage layer in StarRocks. * Each time object storage is accessed, StarRocks will use the temporary credentials issued by the Catalog. * This enhances security and simplifies credential management. #### Usage[​](#usage "Direct link to Usage") ##### Step 1. Set up JWT authentication[​](#step-1-set-up-jwt-authentication "Direct link to Step 1. Set up JWT authentication") In StarRocks, configure a **[JWT-based Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md#create-a-security-integration-with-jwt)** or [create a user with JWT authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/jwt_authentication.md). ##### Step 2. Create Iceberg REST Catalog and configure security settings[​](#step-2-create-iceberg-rest-catalog-and-configure-security-settings "Direct link to Step 2. Create Iceberg REST Catalog and configure security settings") ```sql CREATE EXTERNAL CATALOG iceberg_rest_catalog PROPERTIES ( "iceberg.catalog.type" = "rest", "iceberg.catalog.uri" = "", "iceberg.catalog.security" = "jwt", "iceberg.catalog.warehouse" = "", "iceberg.catalog.vended-credentials-enabled" = "true" ); ``` Properties: * `iceberg.catalog.type`: Set this property to `rest`, indicating to use the REST Catalog. * `iceberg.catalog.uri`: API endpoint for the REST Catalog service. * `iceberg.catalog.security`: Set this property to `jwt` to enable JWT authentication. StarRocks will pass the current user’s authentication information to the Catalog. * `iceberg.catalog.warehouse`: Specifies the Iceberg data warehouse path or identifier. * `iceberg.catalog.vended-credentials-enabled`: Set this property to `true` to enable Vended Credentials and allow temporary credentials issuance. ##### Step 3. Grant permissions[​](#step-3-grant-permissions "Direct link to Step 3. Grant permissions") When a user queries the Catalog via StarRocks, permissions are handled at two levels: 1. **StarRocks Internal Object Permissions** Whether the user has permission to view the Catalog object within StarRocks and switch the session to that Catalog using `SET CATALOG`. 2. **Catalog Internal Permissions** Whether the user has appropriate access permissions when accessing databases, tables, or views within the Catalog. In the ideal workflow, StarRocks is responsible for managing the basic permissions required for a user to access Catalog objects after logging into the StarRocks cluster. Fine-grained data access permissions within the Catalog are completely managed by the Catalog itself. This means: * Users must have the `USAGE` permission to the Catalog object in StarRocks so that they can switch to the catalog using `SET CATALOG` or view the catalog information using `SHOW CATALOGS`. * Data-level permission checks are entirely handled by the Catalog. ###### StarRocks Internal Catalog Object Permissions[​](#starrocks-internal-catalog-object-permissions "Direct link to StarRocks Internal Catalog Object Permissions") To execute `SHOW CATALOGS` or `SET CATALOG` in StarRocks, the user needs `USAGE` permission on the corresponding Catalog object. ###### Option 1: Grant Access to All Users[​](#option-1-grant-access-to-all-users "Direct link to Option 1: Grant Access to All Users") If you allow all users to view and switch to a specified Catalog, you can grant the `USAGE` permission of the Catalog to the `public` role. ```sql GRANT USAGE ON CATALOG TO ROLE public; ``` This will automatically grant all users, upon logging into StarRocks, the `public` role and thus the `USAGE` permission of that Catalog. ###### Option 2: Granular Permission Management[​](#option-2-granular-permission-management "Direct link to Option 2: Granular Permission Management") If you want only specific users or groups to view and switch to a Catalog, you can use **Group Provider** and **Role** for finer-grained management: 1. Synchronize the external user group information to StarRocks using [Group Provider](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). 2. Create the corresponding StarRocks role. ```sql CREATE ROLE ; ``` 3. Bind the role to the external user group: ```sql GRANT TO EXTERNAL GROUP ; ``` 4. Grant the role `USAGE` permission of the Catalog: ```sql GRANT USAGE ON CATALOG TO ROLE ; ``` In this case, members of the external group will automatically inherit the assigned role upon logging into StarRocks and gain the `USAGE` permission of the Catalog. ###### Catalog Internal Data Permissions[​](#catalog-internal-data-permissions "Direct link to Catalog Internal Data Permissions") By configuring the Catalog’s properties, permission checks for all objects within the Catalog are delegated to the Catalog itself. The Catalog will manage authentication and permissions centrally, simplifying permission configuration in StarRocks, avoiding redundant authorizations, and ensuring consistent permission rules across different engines. ```sql ALTER CATALOG iceberg_rest_catalog SET PROPERTIES ( "catalog.access.control" = "allowall" ); ``` Properties: * `catalog.access.control`: Set this property to `allowall`. StarRocks will not perform any further permission checks on objects within the Catalog. Instead, all permission management will be handled by the Catalog itself. --- ### Time Travel with Iceberg Catalog Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) This topic introduces StarRocks' Time Travel feature for Iceberg catalogs. This feature is supported from v3.4.0 onwards. #### Overview[​](#overview "Direct link to Overview") Each Iceberg table maintains a metadata snapshot log, which represents the changes applied to it. Databases can perform Time Travel queries against Iceberg tables by accessing these historical snapshots. Iceberg supports branching and tagging snapshots for sophisticated snapshot lifecycle management, allowing each branch or tag to maintain its own lifecycle based on customized retention policies. For more information on Iceberg's branching and tagging feature, see [Official Documentation](https://iceberg.apache.org/docs/latest/branching/). By integrating Iceberg's snapshot branching and tagging feature, StarRocks supports creating and managing branches and tags in Iceberg catalogs, and Time Travel queries against tables within. #### Manage branches, tags, and snapshots[​](#manage-branches-tags-and-snapshots "Direct link to Manage branches, tags, and snapshots") This section introduces how to manage branches, tags, and snapshots. For instructions on Iceberg stored procedures (using snapshots, performing manual Compaction), see [Iceberg Stored Procedures](https://docs.starrocks.io/docs/data_source/catalog/iceberg/procedures.md). ##### Create a branch[​](#create-a-branch "Direct link to Create a branch") ###### `CREATE BRANCH` Syntax[​](#create-branch-syntax "Direct link to create-branch-syntax") ```sql ALTER TABLE [catalog.][database.]table_name CREATE [OR REPLACE] BRANCH [IF NOT EXISTS] [AS OF VERSION ] [RETAIN { DAYS | HOURS | MINUTES }] [WITH SNAPSHOT RETENTION { minSnapshotsToKeep | maxSnapshotAge | minSnapshotsToKeep maxSnapshotAge }] minSnapshotsToKeep ::= SNAPSHOTS maxSnapshotAge ::= { DAYS | HOURS | MINUTES } ``` ###### Parameters[​](#parameters "Direct link to Parameters") * `branch_name`: Name of the branch to create. * `AS OF VERSION`: ID of the snapshot (version) on which to create the branch. * `RETAIN`: Time to retain the branch. Format: ` `. Supported units: `DAYS`, `HOURS`, and `MINUTES`. Example: `7 DAYS`, `12 HOURS`, or `30 MINUTES`. * `WITH SNAPSHOT RETENTION`: The minimum number of snapshots to keep and/or the maximum time to keep the snapshots. ###### Example[​](#example "Direct link to Example") Create a branch `test-branch` based on version (snapshot ID) `12345` of the table `iceberg.sales.order`, retain the branch for `7` days, and keep at least `2` snapshots on the branch. ```sql ALTER TABLE iceberg.sales.order CREATE BRANCH `test-branch` AS OF VERSION 12345 RETAIN 7 DAYS WITH SNAPSHOT RETENTION 2 SNAPSHOTS; ``` Create a branch `test-branch2` based on version (snapshot ID) `12345` of the table `iceberg.sales.order`, retain the branch for `7` days, and keep the snapshot on the branch for at most `2` days. ```sql ALTER TABLE iceberg.sales.order CREATE BRANCH `test-branch2` AS OF VERSION 12345 RETAIN 7 DAYS WITH SNAPSHOT RETENTION 2 DAYS; ``` Create a branch `test-branch3` based on version (snapshot ID) `12345` of the table `iceberg.sales.order`, retain the branch for `7` days, and keep at least `2` snapshots on the branch, each for at most `2` days. ```sql ALTER TABLE iceberg.sales.order CREATE BRANCH `test-branch3` AS OF VERSION 12345 RETAIN 7 DAYS WITH SNAPSHOT RETENTION 2 SNAPSHOTS 2 DAYS; ``` ##### Load data into a specific branch of a table[​](#load-data-into-a-specific-branch-of-a-table "Direct link to Load data into a specific branch of a table") ###### `VERSION AS OF` Syntax[​](#version-as-of-syntax "Direct link to version-as-of-syntax") ```sql INSERT INTO [catalog.][database.]table_name [FOR] VERSION AS OF ``` ###### Parameters[​](#parameters-1 "Direct link to Parameters") * `branch_name`: Name of the table branch into which the data is loaded. * `query_statement`: Query statement whose result will be loaded into the destination table. It can be any SQL statement supported by StarRocks. ###### Example[​](#example-1 "Direct link to Example") Load the result of a query into the branch `test-branch` of the table `iceberg.sales.order`. ```sql INSERT INTO iceberg.sales.order FOR VERSION AS OF `test-branch` SELECT c1, k1 FROM tbl; ``` ##### Create a tag[​](#create-a-tag "Direct link to Create a tag") ###### `CREATE TAG` Syntax[​](#create-tag-syntax "Direct link to create-tag-syntax") ```sql ALTER TABLE [catalog.][database.]table_name CREATE [OR REPLACE] TAG [IF NOT EXISTS] [AS OF VERSION ] [RETAIN { DAYS | HOURS | MINUTES }] ``` ###### Parameters[​](#parameters-2 "Direct link to Parameters") * `tag_name`: Name of the tag to create. * `AS OF VERSION`: ID of the snapshot (version) on which to create the tag. * `RETAIN`: Time to retain the tag. Format: ` `. Supported units: `DAYS`, `HOURS`, and `MINUTES`. Example: `7 DAYS`, `12 HOURS`, or `30 MINUTES`. ###### Example[​](#example-2 "Direct link to Example") Create a tag `test-tag` based on version (snapshot ID) `12345` of the table `iceberg.sales.order`, and retain the tag for `7` days. ```sql ALTER TABLE iceberg.sales.order CREATE TAG `test-tag` AS OF VERSION 12345 RETAIN 7 DAYS; ``` ##### Drop a branch or a tag[​](#drop-a-branch-or-a-tag "Direct link to Drop a branch or a tag") ###### `DROP BRANCH`, `DROP TAG` Syntax[​](#drop-branch-drop-tag-syntax "Direct link to drop-branch-drop-tag-syntax") ```sql ALTER TABLE [catalog.][database.]table_name DROP { BRANCH | TAG } ``` ###### Example[​](#example-3 "Direct link to Example") ```sql ALTER TABLE iceberg.sales.order DROP BRANCH `test-branch`; ALTER TABLE iceberg.sales.order DROP TAG `test-tag`; ``` #### Query with Time Travel[​](#query-with-time-travel "Direct link to Query with Time Travel") ##### Time Travel to a specific branch or tag[​](#time-travel-to-a-specific-branch-or-tag "Direct link to Time Travel to a specific branch or tag") ###### `VERSION AS OF` Syntax[​](#version-as-of-syntax-1 "Direct link to version-as-of-syntax-1") ```sql [FOR] VERSION AS OF '' ``` ###### Parameter[​](#parameter "Direct link to Parameter") `tag_or_branch`: Name of the branch or tag to which you want to Time Travel. If a branch name is specified, the query will Time Travel to the head snapshot of the branch. If a tag name is specified, the query will Time Travel to the snapshot that the tag referenced. ###### Example[​](#example-4 "Direct link to Example") ```sql -- Time Travel to the head snapshot of a branch. SELECT * FROM iceberg.sales.order VERSION AS OF 'test-branch'; -- Time Travel to the snapshot that the tag referenced. SELECT * FROM iceberg.sales.order VERSION AS OF 'test-tag'; ``` ##### Time Travel to a specific snapshot[​](#time-travel-to-a-specific-snapshot "Direct link to Time Travel to a specific snapshot") ###### `VERSION AS OF` Syntax[​](#version-as-of-syntax-2 "Direct link to version-as-of-syntax-2") ```sql [FOR] VERSION AS OF '' ``` ###### Parameter[​](#parameter-1 "Direct link to Parameter") `snapshot_id`: ID of the snapshot to which you want to Time Travel. ###### Example[​](#example-5 "Direct link to Example") ```sql SELECT * FROM iceberg.sales.order VERSION AS OF 12345; ``` ##### Time Travel to a specific datetime or date[​](#time-travel-to-a-specific-datetime-or-date "Direct link to Time Travel to a specific datetime or date") ###### `TIMESTAMP AS OF` Syntax[​](#timestamp-as-of-syntax "Direct link to timestamp-as-of-syntax") ```sql [FOR] TIMESTAMP AS OF { '' | '' | date_and_time_function } ``` ###### Parameter[​](#parameter-2 "Direct link to Parameter") `date_and_time_function`: Any [date and time functions](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/now.md) supported by StarRocks. ###### Example[​](#example-6 "Direct link to Example") ```sql SELECT * FROM iceberg.sales.order TIMESTAMP AS OF '1986-10-26 01:21:00'; SELECT * FROM iceberg.sales.order TIMESTAMP AS OF '1986-10-26'; SELECT * FROM iceberg.sales.order TIMESTAMP AS OF now(); ``` --- ### Iceberg Procedures StarRocks Iceberg Catalog supports a variety of procedures for managing Iceberg tables, including snapshot management, branch management, data maintenance, metadata management, and table management. You must have the appropriate privileges to execute procedures. For more information about privileges, see [Privileges](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md). #### Snapshot management[​](#snapshot-management "Direct link to Snapshot management") ##### Rollback to snapshot[​](#rollback-to-snapshot "Direct link to Rollback to snapshot") Rolls back the table to a specific snapshot. This operation sets the table's current snapshot to the specified snapshot ID. ###### `rollback_to_snapshot` Syntax[​](#rollback_to_snapshot-syntax "Direct link to rollback_to_snapshot-syntax") ```sql ALTER TABLE [catalog.][database.]table_name EXECUTE rollback_to_snapshot() ``` ###### Parameters[​](#parameters "Direct link to Parameters") `snapshot_id`: ID of the snapshot to which you want to roll back the table. ###### Example[​](#example "Direct link to Example") Roll back the table to snapshot with ID 98765: ```sql ALTER TABLE iceberg.sales.order EXECUTE rollback_to_snapshot(98765); ``` ##### Cherry pick a snapshot[​](#cherry-pick-a-snapshot "Direct link to Cherry pick a snapshot") Cherry picks a specific snapshot and applies it to the current state of the table. This operation creates a new snapshot based on an existing snapshot, while the original snapshot remains unchanged. ###### `cherrypick_snapshot` Syntax[​](#cherrypick_snapshot-syntax "Direct link to cherrypick_snapshot-syntax") ```sql ALTER TABLE [catalog.][database.]table_name EXECUTE cherrypick_snapshot() ``` ###### Parameters[​](#parameters-1 "Direct link to Parameters") `snapshot_id`: ID of the snapshot which you want to cherry pick. ###### Example[​](#example-1 "Direct link to Example") ```sql ALTER TABLE iceberg.sales.order EXECUTE cherrypick_snapshot(54321); ``` #### Branch management[​](#branch-management "Direct link to Branch management") ##### Fast forward a branch to another[​](#fast-forward-a-branch-to-another "Direct link to Fast forward a branch to another") Fast-forwards one branch to another branch's latest snapshot. This operation updates the source branch's snapshot to match the target branch's snapshot. ###### `fast_forward` Syntax[​](#fast_forward-syntax "Direct link to fast_forward-syntax") ```sql ALTER TABLE [catalog.][database.]table_name EXECUTE fast_forward('', '') ``` ###### Parameters[​](#parameters-2 "Direct link to Parameters") * `from_branch`: The branch you want to fast forward. Wrap the branch name in quotes. * `to_branch`: The branch to which you want to fast forward the `from_branch`. Wrap the branch name in quotes. ###### Example[​](#example-2 "Direct link to Example") Fast forward the `main` branch to the branch `test-branch`: ```sql ALTER TABLE iceberg.sales.order EXECUTE fast_forward('main', 'test-branch'); ``` #### Data maintenance[​](#data-maintenance "Direct link to Data maintenance") ##### Rewrite data files[​](#rewrite-data-files "Direct link to Rewrite data files") Rewrites data files to optimize file layout. This procedure merges small files to improve query performance and reduce metadata overhead. ###### `rewrite_data_files` Syntax[​](#rewrite_data_files-syntax "Direct link to rewrite_data_files-syntax") ```sql ALTER TABLE [catalog.][database.]table_name EXECUTE rewrite_data_files ("key"=value [,"key"=value, ...]) [WHERE ] ``` ###### Parameters[​](#parameters-3 "Direct link to Parameters") ###### `rewrite_data_files` properties[​](#rewrite_data_files-properties "Direct link to rewrite_data_files-properties") `"key"=value` pairs that declare the manual compaction behaviors. Note that you need to wrap the key in double quotes. ###### `min_file_size_bytes`[​](#min_file_size_bytes "Direct link to min_file_size_bytes") * Description: The upper limit of a small data file. Data files whose size is less than this value will be merged during the compaction. * Unit: Byte * Type: Int * Default: 268,435,456 (256 MB) ###### `batch_size`[​](#batch_size "Direct link to batch_size") * Description: The maximum size of data that can be processed in each batch. * Unit: Byte * Type: Int * Default: 10,737,418,240 (10 GB) ###### `rewrite_all`[​](#rewrite_all "Direct link to rewrite_all") * Description: Whether to rewrite all data files during the compaction, ignoring the parameters that filter data files with specific requirements. * Unit: - * Type: Boolean * Default: false ###### `batch_parallelism`[​](#batch_parallelism "Direct link to batch_parallelism") * Description: The number of parallel batches to process during the compaction. * Unit: - * Type: Int * Default: 1 ###### `WHERE` clause[​](#where-clause "Direct link to where-clause") * Description: The filter predicate used to specify the partition(s) to be involved in the compaction. ###### Example[​](#example-3 "Direct link to Example") The following example performs manual Compaction on specific partitions in the Iceberg table `t1`. The partitions are represented by the clause `WHERE part_col = 'p1'`. In these partitions, data files that are smaller than 134,217,728 bytes (128 MB) will be merged during the Compaction. ```sql ALTER TABLE t1 EXECUTE rewrite_data_files("min_file_size_bytes"= 134217728) WHERE part_col = 'p1'; ``` #### Metadata management[​](#metadata-management "Direct link to Metadata management") ##### Expire snapshots[​](#expire-snapshots "Direct link to Expire snapshots") Expires snapshots older than a specific timestamp. This operation deletes the data files of the expired snapshots, helping to manage storage usage. ###### `expire_snapshots` Syntax[​](#expire_snapshots-syntax "Direct link to expire_snapshots-syntax") ```sql ALTER TABLE [catalog.][database.]table_name EXECUTE expire_snapshots( [ [older_than =] '' ] [, [retain_last =] ] ) ``` ###### Parameters[​](#parameters-4 "Direct link to Parameters") ###### `older_than`[​](#older_than "Direct link to older_than") * Description: The timestamp before which snapshots will be removed. If not specified, files older than 5 days (from the current time) will be removed by default. Format: 'YYYY-MM-DD HH:MM :SS '. * Type: DATETIME * Required: No ###### `retain_last`[​](#retain_last "Direct link to retain_last") * Description: The maximum number of most recent snapshots to retain. The less recent snapshots will be removed when this threshold is reached. If not specified, only one snapshot will be retained by default. * Type: Integer * Required: No ###### Example[​](#example-4 "Direct link to Example") Expire snapshots before '2023-12-17 00:14:38' and retain two snapshots: ```sql -- With the parameter key specified: ALTER TABLE iceberg.sales.order EXECUTE expire_snapshots(older_than = '2023-12-17 00:14:38', retain_last = 2); -- With the parameter key unspecified: ALTER TABLE iceberg.sales.order EXECUTE expire_snapshots('2023-12-17 00:14:38', 2); ``` ##### Remove orphan files[​](#remove-orphan-files "Direct link to Remove orphan files") Removes orphan files from the table that are not referenced by any valid snapshot and are older than a specified timestamp. This operation helps clean up unused files and reclaim storage space. ###### `remove_orphan_files` Syntax[​](#remove_orphan_files-syntax "Direct link to remove_orphan_files-syntax") ```sql ALTER TABLE [catalog.][database.]table_name EXECUTE remove_orphan_files( [ [older_than =] '' ] [, [location =] '' ] ) ``` ###### Parameters[​](#parameters-5 "Direct link to Parameters") ###### `older_than`[​](#older_than-1 "Direct link to older_than-1") * Description: The timestamp before which orphan files will be removed. If not specified, files older than 7 days (from the current time) will be removed by default. Format: 'YYYY-MM-DD HH:MM :SS '. The value must be at least [`iceberg_remove_orphan_files_min_retention_seconds`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters/shared_lake_other.md) (24 hours by default) before the current time. A more recent value is rejected, because deleting files that young can remove data that a concurrent write has not committed yet. * Type: DATETIME * Required: No ###### `location`[​](#location "Direct link to location") * Description: The directory from which you want to remove orphan files. It must be a sub-directory of the table location. If not specified, the table location will be used by default. * Type: STRING * Required: No ###### Example[​](#example-5 "Direct link to Example") Remove orphan files older than '2024-01-01 00:00:00' from the sub-directory `sub_dir` of the table location: ```sql -- With the parameter key specified: ALTER TABLE iceberg.sales.order EXECUTE remove_orphan_files(older_than = '2024-01-01 00:00:00', location = 's3://iceberg-bucket/iceberg_db/iceberg_table/sub_dir'); -- With the parameter key unspecified: ALTER TABLE iceberg.sales.order EXECUTE remove_orphan_files('2024-01-01 00:00:00', 's3://bucket-test/iceberg_db/iceberg_table/sub_dir'); ``` ##### Rewrite manifests[​](#rewrite-manifests "Direct link to Rewrite manifests") Rewrites data manifest files and merges them by partition to avoid performance degradation caused by excessive small manifests. note This operation rewrites the data manifests of the current snapshot only. ###### `rewrite_manifests` Syntax[​](#rewrite_manifests-syntax "Direct link to rewrite_manifests-syntax") ```sql ALTER TABLE [catalog.][database.]table_name EXECUTE rewrite_manifests() ``` ###### Example[​](#example-6 "Direct link to Example") Rewrite the data manifests of the current snapshot: ```sql ALTER TABLE iceberg.sales.order EXECUTE rewrite_manifests() ``` #### Table management[​](#table-management "Direct link to Table management") ##### Add files[​](#add-files "Direct link to Add files") Adds data files to an Iceberg table from either a source table or a specific location. This procedure supports Parquet and ORC file formats. ###### `add_files` Syntax[​](#add_files-syntax "Direct link to add_files-syntax") ```sql ALTER TABLE [catalog.][database.]table_name EXECUTE add_files( [source_table = '' | location = '', file_format = ''] [, recursive = ] ) ``` ###### Parameters[​](#parameters-6 "Direct link to Parameters") Either `source_table` or `location` must be provided, but not both. ###### `source_table`[​](#source_table "Direct link to source_table") * Description: The source table from which to add files. Format: 'catalog.database.table'. * Type: String * Required: No ###### `location`[​](#location-1 "Direct link to location-1") * Description: The directory path or file path from which to add files. * Type: String * Required: No ###### `file_format`[​](#file_format "Direct link to file_format") * Description: The format of the data files. Supported values: 'parquet', 'orc'. * Type: String * Required: No (required when using `location`) ###### `recursive`[​](#recursive "Direct link to recursive") * Description: Whether to recursively scan subdirectories when adding files from a location. * Type: Boolean * Default: true * Required: No ###### Example[​](#example-7 "Direct link to Example") Add files from a source table: ```sql ALTER TABLE iceberg.sales.order EXECUTE add_files(source_table = 'hive_catalog.sales.source_order'); ``` Add files from a specific location with Parquet format: ```sql ALTER TABLE iceberg.sales.order EXECUTE add_files(location = 's3://bucket/data/order/', file_format = 'parquet', recursive = true); ``` Add files from a single file: ```sql ALTER TABLE iceberg.sales.order EXECUTE add_files(location = 's3://bucket/data/order/data.parquet', file_format = 'parquet'); ``` ##### Register table[​](#register-table "Direct link to Register table") Registers an Iceberg table using a metadata file. This procedure allows you to add an existing Iceberg table to the catalog without migrating data. ###### `register_table` Syntax[​](#register_table-syntax "Direct link to register_table-syntax") ```sql CALL [catalog.]system.register_table( database_name = '', table_name = '', metadata_file = '' ) ``` ###### Parameters[​](#parameters-7 "Direct link to Parameters") ###### `database_name`[​](#database_name "Direct link to database_name") * Description: The name of the database in which to register the table. * Type: String * Required: Yes ###### `table_name`[​](#table_name "Direct link to table_name") * Description: The name of the table to register. * Type: String * Required: Yes ###### `metadata_file`[​](#metadata_file "Direct link to metadata_file") * Description: The path to the Iceberg table metadata file (e.g., metadata.json). * Type: String * Required: Yes ###### Example[​](#example-8 "Direct link to Example") Register a table using a metadata file: ```sql CALL iceberg_catalog.system.register_table( database_name = 'sales', table_name = 'order', metadata_file = 's3://bucket/metadata/sales/order/metadata/00001-xxxxx-xxxxx-xxxxx.metadata.json' ); ``` Or use the current catalog: ```sql CALL system.register_table( database_name = 'sales', table_name = 'order', metadata_file = 's3://bucket/metadata/sales/order/metadata/00001-xxxxx-xxxxx-xxxxx.metadata.json' ); ``` --- ### Iceberg Lakehouse tutorial ### Apache Iceberg Lakehouse This guide will get you up and running with Apache Iceberg™ using StarRocks™, including sample code to highlight some powerful features. ##### Docker-Compose[​](#docker-compose "Direct link to Docker-Compose") The fastest way to get started is to use a docker-compose file that uses the `starrocks/fe-ubuntu` and `starrocks/be-ubuntu` images which contain a local StarRocks cluster with a configured Iceberg catalog. To use this, you'll need to install the Docker CLI. Once you have Docker installed, save the yaml below into a file named docker-compose.yml: ```yml services: starrocks-fe: image: starrocks/fe-ubuntu:4.0-latest hostname: starrocks-fe container_name: starrocks-fe user: root command: | bash /opt/starrocks/fe/bin/start_fe.sh --host_type FQDN ports: - 8030:8030 - 9020:9020 - 9030:9030 networks: iceberg_net: environment: - AWS_ACCESS_KEY_ID=admin - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 healthcheck: test: 'mysql -u root -h starrocks-fe -P 9030 -e "SHOW FRONTENDS\G" |grep "Alive: true"' interval: 10s timeout: 5s retries: 3 starrocks-be: image: starrocks/be-ubuntu:4.0-latest command: - /bin/bash - -c - | ulimit -n 65535; echo "# Enable data cache" >> /opt/starrocks/be/conf/be.conf echo "block_cache_enable = true" >> /opt/starrocks/be/conf/be.conf echo "block_cache_mem_size = 536870912" >> /opt/starrocks/be/conf/be.conf echo "block_cache_disk_size = 1073741824" >> /opt/starrocks/be/conf/be.conf sleep 15s mysql --connect-timeout 2 -h starrocks-fe -P 9030 -u root -e "ALTER SYSTEM ADD BACKEND \"starrocks-be:9050\";" bash /opt/starrocks/be/bin/start_be.sh ports: - 8040:8040 hostname: starrocks-be container_name: starrocks-be user: root depends_on: starrocks-fe: condition: service_healthy healthcheck: test: 'mysql -u root -h starrocks-fe -P 9030 -e "SHOW BACKENDS\G" |grep "Alive: true"' interval: 10s timeout: 5s retries: 3 networks: iceberg_net: environment: - HOST_TYPE=FQDN - AWS_EC2_METADATA_DISABLED=true rest: image: apache/iceberg-rest-fixture container_name: iceberg-rest networks: iceberg_net: aliases: - iceberg-rest.minio ports: - 8181:8181 environment: - AWS_ACCESS_KEY_ID=admin - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 - CATALOG_WAREHOUSE=s3://warehouse/ - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO - CATALOG_S3_ENDPOINT=http://minio:9000 minio: image: minio/minio:RELEASE.2024-10-29T16-01-48Z container_name: minio environment: - MINIO_ROOT_USER=admin - MINIO_ROOT_PASSWORD=password - MINIO_DOMAIN=minio networks: iceberg_net: aliases: - warehouse.minio ports: - 9001:9001 - 9000:9000 command: ["server", "/data", "--console-address", ":9001"] mc: depends_on: - minio image: minio/mc:RELEASE.2024-10-29T15-34-59Z container_name: mc networks: iceberg_net: environment: - AWS_ACCESS_KEY_ID=admin - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 entrypoint: > /bin/sh -c " until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done; /usr/bin/mc rm -r --force minio/warehouse; /usr/bin/mc mb minio/warehouse; /usr/bin/mc policy set public minio/warehouse; tail -f /dev/null " networks: iceberg_net: ``` Next, start up the docker containers with this command: ```plain docker compose up --detach --wait --wait-timeout 400 ``` You can then run any of the following commands to start a StarRocks session. ```bash docker exec -it starrocks-fe \ mysql -P 9030 -h 127.0.0.1 -u root --prompt="StarRocks > " ``` ##### Adding and Using a Catalog[​](#adding-and-using-a-catalog "Direct link to Adding and Using a Catalog") ```sql CREATE EXTERNAL CATALOG 'demo' COMMENT "External catalog to Apache Iceberg on MinIO" PROPERTIES ( "type"="iceberg", "iceberg.catalog.type"="rest", "iceberg.catalog.uri"="http://iceberg-rest:8181", "iceberg.catalog.warehouse"="warehouse", "aws.s3.access_key"="admin", "aws.s3.secret_key"="password", "aws.s3.endpoint"="http://minio:9000", "aws.s3.enable_path_style_access"="true" ); ``` ```sql SHOW CATALOGS\G ``` ```sql *************************** 1. row *************************** Catalog: default_catalog Type: Internal Comment: An internal catalog contains this cluster's self-managed tables. *************************** 2. row *************************** Catalog: demo Type: Iceberg Comment: External catalog to Apache Iceberg on MinIO 2 rows in set (0.00 sec) ``` ```sql SET CATALOG demo; ``` ##### Creating and using a database[​](#creating-and-using-a-database "Direct link to Creating and using a database") ```sql CREATE DATABASE nyc; ``` ```sql USE nyc; ``` ##### Creating a table[​](#creating-a-table "Direct link to Creating a table") ```sql CREATE TABLE demo.nyc.taxis ( trip_id bigint, trip_distance float, fare_amount double, store_and_fwd_flag string, vendor_id bigint ) PARTITION BY (vendor_id); ``` ##### Writing Data to a Table[​](#writing-data-to-a-table "Direct link to Writing Data to a Table") ```sql INSERT INTO demo.nyc.taxis VALUES (1000371, 1.8, 15.32, 'N', 1), (1000372, 2.5, 22.15, 'N', 2), (1000373, 0.9, 9.01, 'N', 2), (1000374, 8.4, 42.13, 'Y', 1); ``` ##### Reading Data from a Table[​](#reading-data-from-a-table "Direct link to Reading Data from a Table") ```sql SELECT * FROM demo.nyc.taxis; ``` ##### Verify that the data is stored in object storage[​](#verify-that-the-data-is-stored-in-object-storage "Direct link to Verify that the data is stored in object storage") When you added and used the external catalog, Starrocks started using MinIO as the object store for the `demo.nyc.taxis` table. If you navigate to and then navigate through the Object Browser menu to `warehouse/nyc/taxis/` you can confirm that StarRocks is using MinIO for the storage. tip The username and password for MinIO are in the docker-compose.yml file. You will be prompted to change the password to something better, just ignore this advice for the tutorial. ![img](/assets/images/MinIO-Iceberg-data-8ade61c31be69444bd02b00acafe263c.png) ##### Next Steps[​](#next-steps "Direct link to Next Steps") ###### Adding Iceberg to StarRocks[​](#adding-iceberg-to-starrocks "Direct link to Adding Iceberg to StarRocks") If you already have a StarRocks 3.2.0, or later, environment, it comes with the Iceberg 1.6.0 included. No additional downloads or jars are needed. ###### Learn More[​](#learn-more "Direct link to Learn More") Now that you're up and running with Iceberg and StarRocks, check out the [StarRocks-Iceberg docs](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md) to learn more! --- ### JDBC catalog Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) StarRocks supports JDBC catalogs from v3.0 onwards. A JDBC catalog is a kind of external catalog that enables you to query data from data sources accessed through JDBC without ingestion. Also, you can directly transform and load data from JDBC data sources by using [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) based on JDBC catalogs. JDBC catalogs support MySQL and PostgreSQL from v3.0 onwards, Oracle and SQLServer since v3.2.9 and v3.3.1, and ClickHouse (Experimental) since v3.3.0. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * The FEs and BEs or CNs in your StarRocks cluster can download the JDBC driver from the download URL specified by the `driver_url` parameter. * `JAVA_HOME` in the **$BE\_HOME/bin/start\_be.sh** file on each BE or CN node is properly configured as a path in the JDK environment instead of a path in the JRE environment. For example, you can configure `export JAVA_HOME = `. You must add this configuration at the beginning of the script and restart the BE or CN for the configuration to take effect. #### Create a JDBC catalog[​](#create-a-jdbc-catalog "Direct link to Create a JDBC catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ("key"="value", ...) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### `catalog_name`[​](#catalog_name "Direct link to catalog_name") The name of the JDBC catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### `comment`[​](#comment "Direct link to comment") The description of the JDBC catalog. This parameter is optional. ###### `PROPERTIES`[​](#properties "Direct link to properties") The properties of the JDBC Catalog. `PROPERTIES` must include the following parameters: | **Parameter** | **Description** | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | type | The type of the resource. Set the value to `jdbc`. | | user | The username that is used to connect to the target database. | | password | The password that is used to connect to the target database. | | jdbc\_uri | The URI that the JDBC driver uses to connect to the target database. For MySQL, the URI is in the `"jdbc:mysql://ip:port"` format. For PostgreSQL, the URI is in the `"jdbc:postgresql://ip:port/db_name"` format. For more information: [PostgreSQL](https://jdbc.postgresql.org/documentation/head/connect.html). | | driver\_url | The download URL of the JDBC driver JAR package. An HTTP URL or file URL is supported, for example, `https://repo1.maven.org/maven2/org/postgresql/postgresql/42.3.3/postgresql-42.3.3.jar` and `file:///home/disk1/postgresql-42.3.3.jar`.
**NOTE**
You can also put the JDBC driver to any same path on the FE and BE or CN nodes and set `driver_url` to that path, which must be in the `file:////to/the/driver` format. | | driver\_class | The class name of the JDBC driver. The JDBC driver class names of common database engines are as follows:- MySQL: `com.mysql.jdbc.Driver` (MySQL v5.x and earlier) and `com.mysql.cj.jdbc.Driver` (MySQL v6.x and later)
- PostgreSQL: `org.postgresql.Driver`
- Oracle: `oracle.jdbc.driver.OracleDriver` | | schema\_resolver | (Optional) Explicitly specifies the schema resolver to use. Valid values: `postgresql`, `mysql`, `oracle`, `sqlserver`, `clickhouse`. Use this parameter when working with non-standard JDBC drivers that cannot be auto-detected by driver class name. If not specified, StarRocks will auto-detect the appropriate resolver based on the `driver_class` parameter. | ###### Optional Oracle properties[​](#optional-oracle-properties "Direct link to Optional Oracle properties") When `driver_class` is set to Oracle, you can configure the following optional properties: | **Parameter** | **Default** | **Description** | | ------------------------------ | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | oracle.number.default-scale | 6 | Set it when Oracle `NUMBER` metadata does not provide explicit precision and scale. Valid range: `0` to `38`. | | oracle.temporal.to-datetime | false | Controls Oracle `DATE`, `TIMESTAMP`, and `TIMESTAMP WITH LOCAL TIME ZONE` mapping. If it is set to `true`, these data types are mapped to StarRocks' `DATETIME` type; otherwise, `DATE` remains `DATE`, and `TIMESTAMP` / `TIMESTAMP WITH LOCAL TIME ZONE` are mapped to `VARCHAR(64)`. | | oracle.timestamptz.to-datetime | false | Controls Oracle `TIMESTAMP WITH TIME ZONE` mapping. If it is set to `true`, it is mapped to StarRocks' `DATETIME` type; otherwise, it is mapped to `VARCHAR(64)`. | ###### Optional row-count cache properties[​](#optional-row-count-cache-properties "Direct link to Optional row-count cache properties") StarRocks caches per-table row counts from JDBC sources to avoid blocking query planning. These properties let you tune the cache behavior per catalog. If not set, the global FE configuration values are used. | **Parameter** | **Default** | **Description** | | ------------------------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | jdbc\_row\_count\_cache\_refresh\_sec | 600 | Background refresh interval (seconds). After this interval, the cached value is returned immediately while a reload runs asynchronously in the background. | | jdbc\_row\_count\_cache\_expire\_sec | 1200 | Hard eviction TTL (seconds). Cache entries not accessed within this window are evicted. Must be greater than `jdbc_row_count_cache_refresh_sec`. | | jdbc\_row\_count\_cache\_max\_size | 10000 | Maximum number of table entries in the row-count cache for this catalog. | > **NOTE** > > The FEs download the JDBC driver JAR package at the time of JDBC catalog creation, and the BEs or CNs download the JDBC driver JAR package at the time of the first query. The amount of time taken for the download varies depending on network conditions. ##### Examples[​](#examples "Direct link to Examples") The following example creates five different JDBC catalogs. ```sql -- PostgresSQL CREATE EXTERNAL CATALOG jdbc0 PROPERTIES ( "type"="jdbc", "user"="postgres", "password"="changeme", "jdbc_uri"="jdbc:postgresql://127.0.0.1:5432/jdbc_test", "driver_url"="https://repo1.maven.org/maven2/org/postgresql/postgresql/42.3.3/postgresql-42.3.3.jar", "driver_class"="org.postgresql.Driver" ); -- MySQL CREATE EXTERNAL CATALOG jdbc1 PROPERTIES ( "type"="jdbc", "user"="root", "password"="changeme", "jdbc_uri"="jdbc:mysql://127.0.0.1:3306", "driver_url"="https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar", "driver_class"="com.mysql.cj.jdbc.Driver" ); -- Oracle CREATE EXTERNAL CATALOG jdbc2 PROPERTIES ( "type"="jdbc", "user"="root", "password"="changeme", "jdbc_uri"="jdbc:oracle:thin:@127.0.0.1:1521:ORCL", "driver_url"="https://repo1.maven.org/maven2/com/oracle/database/jdbc/ojdbc10/19.18.0.0/ojdbc10-19.18.0.0.jar", "driver_class"="oracle.jdbc.driver.OracleDriver" ); -- Oracle (with Oracle-specific optional properties) CREATE EXTERNAL CATALOG jdbc2_ext PROPERTIES ( "type"="jdbc", "user"="root", "password"="changeme", "jdbc_uri"="jdbc:oracle:thin:@127.0.0.1:1521/ORCLPDB1", "driver_url"="https://repo1.maven.org/maven2/com/oracle/database/jdbc/ojdbc10/19.18.0.0/ojdbc10-19.18.0.0.jar", "driver_class"="oracle.jdbc.driver.OracleDriver", "oracle.number.default-scale"="6", "oracle.temporal.to-datetime"="true", "oracle.timestamptz.to-datetime"="true" ); -- SQL Server CREATE EXTERNAL CATALOG jdbc3 PROPERTIES ( "type"="jdbc", "user"="root", "password"="changeme", "jdbc_uri"="jdbc:sqlserver://127.0.0.1:1433;databaseName=MyDatabase;", "driver_url"="https://repo1.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/12.4.2.jre11/mssql-jdbc-12.4.2.jre11.jar", "driver_class"="com.microsoft.sqlserver.jdbc.SQLServerDriver" ); -- ClickHouse CREATE EXTERNAL CATALOG jdbc4 PROPERTIES ( "type"="jdbc", "user"="default", "jdbc_uri"="jdbc:clickhouse://127.0.0.1:8443", "driver_url"="https://repo1.maven.org/maven2/com/clickhouse/clickhouse-jdbc/0.4.6/clickhouse-jdbc-0.4.6.jar", "driver_class"="com.clickhouse.jdbc.ClickHouseDriver" ); ``` #### View JDBC catalogs[​](#view-jdbc-catalogs "Direct link to View JDBC catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of a JDBC catalog named `jdbc0`: ```sql SHOW CREATE CATALOG jdbc0; ``` #### Drop a JDBC catalog[​](#drop-a-jdbc-catalog "Direct link to Drop a JDBC catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop a JDBC catalog. The following example drops a JDBC catalog named `jdbc0`: ```sql DROP Catalog jdbc0; ``` #### Query a table in a JDBC catalog[​](#query-a-table-in-a-jdbc-catalog "Direct link to Query a table in a JDBC catalog") 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your JDBC-compatible cluster: ```sql SHOW DATABASES FROM ; ``` 2. Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to switch to the destination catalog in the current session: ```sql SET CATALOG ; ``` Then, use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify the active database in the current session: ```sql USE ; ``` Or, you can use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to directly specify the active database in the destination catalog: ```sql USE .; ``` 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT * FROM ; ``` #### FAQ[​](#faq "Direct link to FAQ") What do I do if an error suggesting "Malformed database URL, failed to parse the main URL sections" is thrown? If you encounter such an error, the URI that you passed in `jdbc_uri` is invalid. Check the URI that you pass and make sure it is valid. For more information, see the parameter descriptions in the "[PROPERTIES](#properties)" section of this topic. --- ### Kudu catalog Experimental feature [Advice on use of experimental features](https://docs.starrocks.io/docs/maturity/features.md) StarRocks supports Kudu catalogs from v3.3 onwards. A Kudu catalog is a kind of external catalog that enables you to query data from Apache Kudu without ingestion. Also, you can directly transform and load data from Kudu by using [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) based on Kudu catalogs. To ensure successful SQL workloads on your Kudu cluster, your StarRocks cluster needs to integrate with the following important components: * Metastore like your kudu file system or Hive metastore #### Usage notes[​](#usage-notes "Direct link to Usage notes") You can only use Kudu catalogs to query data. You cannot use Kudu catalogs to drop, delete, or insert data into your Kudu cluster. #### Integration preparations[​](#integration-preparations "Direct link to Integration preparations") Before you create a Kudu catalog, make sure your StarRocks cluster can integrate with the storage system and metastore of your Kudu cluster. > **NOTE** > > If an error indicating an unknown host is returned when you send a query, you must add the mapping between the host names and IP addresses of your KUDU cluster nodes to the **/etc/hosts** path. ##### Kerberos authentication[​](#kerberos-authentication "Direct link to Kerberos authentication") If Kerberos authentication is enabled for your KUDU cluster or Hive metastore, configure your StarRocks cluster as follows: * Run the `kinit -kt keytab_path principal` command on each FE and each BE to obtain Ticket Granting Ticket (TGT) from Key Distribution Center (KDC). To run this command, you must have the permissions to access your KUDU cluster and Hive metastore. Note that accessing KDC with this command is time-sensitive. Therefore, you need to use cron to run this command periodically. * Add `JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"` to the **$FE\_HOME/conf/fe.conf** file of each FE and to the **$BE\_HOME/conf/be.conf** file of each BE. In this example, `/etc/krb5.conf` is the save path of the **krb5.conf** file. You can modify the path based on your needs. #### Create a Kudu catalog[​](#create-a-kudu-catalog "Direct link to Create a Kudu catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ( "type" = "kudu", CatalogParams ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### catalog\_name[​](#catalog_name "Direct link to catalog_name") The name of the Kudu catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### comment[​](#comment "Direct link to comment") The description of the Kudu catalog. This parameter is optional. ###### type[​](#type "Direct link to type") The type of your data source. Set the value to `kudu`. ###### CatalogParams[​](#catalogparams "Direct link to CatalogParams") A set of parameters about how StarRocks accesses the metadata of your Kudu cluster. The following table describes the parameter you need to configure in `CatalogParams`. | Parameter | Required | Description | | ----------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | kudu.catalog.type | Yes | The type of metastore that you use for your Kudu cluster. Set this parameter to `kudu` or `hive`. | | kudu.master | No | Specifies the Kudu Master address, which defaults to `localhost:7051`. | | hive.metastore.uris | No | The URI of your Hive metastore. Format: `thrift://:`. If high availability (HA) is enabled for your Hive metastore, you can specify multiple metastore URIs and separate them with commas (`,`), for example, `"thrift://:,thrift://:,thrift://:"`. | | kudu.schema-emulation.enabled | No | option to enable or disable the `schema` emulation. By default, it is turned off (false), which means that all tables belong to the `default` `schema`. | | kudu.schema-emulation.prefix | No | The prefix for `schema` emulation should only be set when `kudu.schema-emulation.enabled` = `true`. The default prefix used is empty string: \`\`. | > **NOTE** > > If you use Hive metastore, you must add the mapping between the host names and IP addresses of your Hive metastore nodes to the `/etc/hosts` path before you query Kudu data. Otherwise, StarRocks may fail to access your Hive metastore when you start a query. ##### Examples[​](#examples "Direct link to Examples") * The following examples create a Kudu catalog named `kudu_catalog` whose metastore type `kudu.catalog.type` is set to `kudu` to query data from your Kudu cluster. ```sql CREATE EXTERNAL CATALOG kudu_catalog PROPERTIES ( "type" = "kudu", "kudu.master" = "localhost:7051", "kudu.catalog.type" = "kudu", "kudu.schema-emulation.enabled" = "true", "kudu.schema-emulation.prefix" = "impala::" ); ``` * The following examples create a Kudu catalog named `kudu_catalog` whose metastore type `kudu.catalog.type` is set to `hive` to query data from your Kudu cluster. ```sql CREATE EXTERNAL CATALOG kudu_catalog PROPERTIES ( "type" = "kudu", "kudu.master" = "localhost:7051", "kudu.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "kudu.schema-emulation.enabled" = "true", "kudu.schema-emulation.prefix" = "impala::" ); ``` #### View Kudu catalogs[​](#view-kudu-catalogs "Direct link to View Kudu catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of a Kudu catalog named `kudu_catalog`: ```sql SHOW CREATE CATALOG kudu_catalog; ``` #### Drop a Kudu catalog[​](#drop-a-kudu-catalog "Direct link to Drop a Kudu catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop an external catalog. The following example drops a Kudu catalog named `kudu_catalog`: ```sql DROP Catalog kudu_catalog; ``` #### View the schema of a Kudu table[​](#view-the-schema-of-a-kudu-table "Direct link to View the schema of a Kudu table") You can use one of the following syntaxes to view the schema of a Kudu table: * View schema ```sql DESC[RIBE] ..; ``` * View schema and location from the CREATE statement ```sql SHOW CREATE TABLE ..; ``` #### Query a Kudu table[​](#query-a-kudu-table "Direct link to Query a Kudu table") 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your Kudu cluster: ```sql SHOW DATABASES FROM ; ``` 2. Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to switch to the destination catalog in the current session: ```sql SET CATALOG ; ``` Then, use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify the active database in the current session: ```sql USE ; ``` Or, you can use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to directly specify the active database in the destination catalog: ```sql USE .; ``` 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT count(*) FROM LIMIT 10; ``` #### Load data from Kudu[​](#load-data-from-kudu "Direct link to Load data from Kudu") Suppose you have an OLAP table named `olap_tbl`, you can transform and load data like below: ```sql INSERT INTO default_catalog.olap_db.olap_tbl SELECT * FROM kudu_table; ``` --- ### MaxCompute catalog Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) StarRocks supports Alibaba Cloud MaxCompute (previously known as ODPS) catalogs from v3.3 onwards. A MaxCompute catalog is a kind of external catalog that enables you to query data from MaxCompute without ingestion. With MaxCompute catalogs, you also can directly transform and load the data from MaxCompute by using [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). #### Usage notes[​](#usage-notes "Direct link to Usage notes") You can use MaxCompute catalogs only to query the data from MaxCompute. You cannot use MaxCompute catalogs to drop, delete, or insert data into your MaxCompute cluster. #### Integration preparations[​](#integration-preparations "Direct link to Integration preparations") Before creating a MaxCompute catalog, make sure that your StarRocks cluster can access your MaxCompute service properly. #### Create a MaxCompute catalog[​](#create-a-maxcompute-catalog "Direct link to Create a MaxCompute catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ( "type" = "odps", CatalogParams, ScanParams, CachingMetaParams ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### catalog\_name[​](#catalog_name "Direct link to catalog_name") The name of the MaxCompute catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### comment[​](#comment "Direct link to comment") The description of the MaxCompute catalog. This parameter is optional. ###### type[​](#type "Direct link to type") The type of your data source. Set the value to `odps`. ###### CatalogParams[​](#catalogparams "Direct link to CatalogParams") A set of parameters about how StarRocks accesses the metadata of the MaxCompute cluster. The following table describes the parameter you need to configure in `CatalogParams`. | Parameter | Required | Description | | -------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | odps.endpoint | Yes | The connection address (namely, endpoint) for the MaxCompute service. You need to configure the endpoint according to the region selected when creating the MaxCompute project as well as the network connection mode. For details about the endpoints used in different regions and network connection modes, see [Endpoint](https://www.alibabacloud.com/help/en/maxcompute/user-guide/endpoints). Note that currently only two network connection modes of Alibaba Cloud are supported to provide the best experience: VPC and classic network. | | odps.project | Yes | The name of the MaxCompute project you want to access. If you have created a standard mode workspace, pay attention to the differences between the project names for the production environment and the development environment (\_dev) when configuring this parameter. You can log in to the [MaxCompute Console](https://account.alibabacloud.com/login/login.htm?spm=5176.12901015-2.0.0.593a525cwmiD7c), and obtain the MaxCompute project name on the **Workspace** > **Project Management** page. | | odps.access.id | Yes | The AccessKey ID of the Alibaba Cloud account or RAM user. You can enter the [AccessKey Management](https://ram.console.aliyun.com/manage/ak) page to obtain the AccessKey ID. | | odps.access.key | Yes | The AccessKey Secret matching the AccessKey ID. You can enter the [AccessKey Management](https://ram.console.aliyun.com/manage/ak) page to obtain the AccessKey Secret. | | odps.tunnel.endpoint | No | The public network access link for the Tunnel service. If you have not configured the Tunnel endpoint, Tunnel will automatically route to the Tunnel endpoint matching the network where the MaxCompute service is located. If you have configured the Tunnel endpoint, it will be used as configured and not automatically routed. | | odps.tunnel.quota | Yes | The name of the quota that is used to access MaxCompute. MaxCompute provides two types of resources for data transmission: exclusive resource group for MaxCompute Tunnel (subscription) and storage API (pay-as-you-go). You can perform the following operations to obtain the quota name based on the resource type.**Exclusive resource group for MaxCompute Tunnel**: Log on to the [MaxCompute console](https://maxcompute.console.aliyun.com/). In the top navigation bar, select a region. In the left-side navigation pane, choose Workspace > Quotas to view the available quotas. For more information, see [Manage quotas for computing resources in the MaxCompute console](https://help.aliyun.com/zh/maxcompute/user-guide/manage-quotas-in-the-maxcompute-console).**Storage API**: Log on to the [MaxCompute console](https://maxcompute.console.aliyun.com/). In the left-side navigation pane, choose Tenants > Tenant Property. On the Tenants page, turn on Storage API Switch. For more information, see [Use storage API (pay-as-you-go)](https://help.aliyun.com/zh/maxcompute/user-guide/overview-1). The default name of the storage API is **"pay-as-you-go"**. | ###### ScanParams[​](#scanparams "Direct link to ScanParams") A set of parameters about how StarRocks accesses the files stored in the MaxCompute cluster. This parameter set is optional. The following table describes the parameter you need to configure in `ScanParams`. | Parameter | Required | Description | | --------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | odps.split.policy | No | The shard policy used when for data scanning.
Valid values: `size` (shard by data size) and `row_offset` (shard by number of rows). Default value: `size`.
| | odps.split.row.count | No | The maximum number of rows per shard when `odps.split.policy` is set to `row_offset`.
Default value: `4 * 1024 * 1024 = 4194304`.
| ###### CachingMetaParams[​](#cachingmetaparams "Direct link to CachingMetaParams") A set of parameters about how StarRocks caches the metadata of Hive. This parameter set is optional. The following table describes the parameter you need to configure in `CachingMetaParams`. | Parameter | Required | Description | | ---------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | odps.cache.table.enable | No | Specifies whether StarRocks caches the metadata of MaxCompute tables. Valid values: `true` and `false`. Default value: `true`. The value `true` enables the cache, and the value `false` disables the cache. | | odps.cache.table.expire | No | The time interval, in seconds, at which StarRocks automatically evicts the cached metadata of MaxCompute tables or partitions. Default value: `86400` (24 hours). | | odps.cache.table.size | No | The number of MaxCompute table metadata entries that StarRocks caches. Default value: `1000`. | | odps.cache.partition.enable | No | Specifies whether StarRocks caches the metadata of all partitions for a MaxCompute table. Valid values: `true` and `false`. Default value: `true`. The value `true` enables the cache, and the value `false` disables the cache. | | odps.cache.partition.expire | No | The time interval, in seconds, at which StarRocks automatically evicts the cached metadata of all partitions for a MaxCompute table. Default value: `86400` (24 hours). | | odps.cache.partition.size | No | The number of MaxCompute tables for which StarRocks caches the metadata of all partitions. Default value: `1000`. | | odps.cache.table-name.enable | No | Specifies whether StarRocks caches the information of the tables from the MaxCompute project. Valid values: `true` and `false`. Default value: `false`. The value `true` enables the cache, and the value `false` disables the cache. | | odps.cache.table-name.expire | No | The time interval, in seconds, at which StarRocks automatically evicts the cached information of the tables from the MaxCompute project. Default value: `86400` (24 hours). | | odps.cache.table-name.size | No | The number of MaxCompute projects that StarRocks caches. Default value: `1000`. | ##### Examples[​](#examples "Direct link to Examples") The following example creates a MaxCompute catalog named `odps_catalog` which uses `odps_project` as the warehouse project. ```sql CREATE EXTERNAL CATALOG odps_catalog PROPERTIES ( "type"="odps", "odps.access.id"="", "odps.access.key"="", "odps.endpoint"="", "odps.project"="odps_project" ); ``` #### View MaxCompute catalogs[​](#view-maxcompute-catalogs "Direct link to View MaxCompute catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of a MaxCompute catalog named `odps_catalog`: ```sql SHOW CREATE CATALOG odps_catalog; ``` #### Drop a MaxCompute catalog[​](#drop-a-maxcompute-catalog "Direct link to Drop a MaxCompute catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop an external catalog. The following example drops a MaxCompute catalog named `odps_catalog`: ```sql DROP CATALOG odps_catalog; ``` #### View the schema of a MaxCompute table[​](#view-the-schema-of-a-maxcompute-table "Direct link to View the schema of a MaxCompute table") You can use one of the following syntaxes to view the schema of a MaxCompute table: * View schema ```sql DESC[RIBE] ..; ``` * View schema and location from the CREATE statement ```sql SHOW CREATE TABLE ..; ``` #### Query a MaxCompute table[​](#query-a-maxcompute-table "Direct link to Query a MaxCompute table") 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your MaxCompute cluster: ```sql SHOW DATABASES FROM ; ``` 2. Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to switch to the destination catalog in the current session: ```sql SET CATALOG ; ``` Then, use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify the active database in the current session: ```sql USE ; ``` Or, you can use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to directly specify the active database in the destination catalog: ```sql USE .; ``` 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT count(*) FROM LIMIT 10; ``` #### Load data from MaxCompute[​](#load-data-from-maxcompute "Direct link to Load data from MaxCompute") Suppose that there is an OLAP table named `olap_tbl` in your StarRock cluster and there is a table named `mc_table` in your MaxCompute cluster. You can transform and load the data from the MaxCompute table `mc_table` into the StarRocks table `olap_tbl` like below: ```sql INSERT INTO default_catalog.olap_db.olap_tbl SELECT * FROM mc_table; ``` #### Data type mapping[​](#data-type-mapping "Direct link to Data type mapping") MaxCompute catalogs map the MaxCompute data types to the StarRocks data types. The following table shows the mapping between the MaxCompute data types to the StarRocks data types. | MaxCompute data type | StarRocks data type | | -------------------- | ------------------- | | BOOLEAN | BOOLEAN | | TINYINT | TINYINT | | SMALLINT | SMALLINT | | INT | INT | | BIGINT | BIGINT | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | DECIMAL(p, s) | DECIMAL(p, s) | | STRING | VARCHAR(1073741824) | | VARCHAR(n) | VARCHAR(n) | | CHAR(n) | CHAR(n) | | JSON | VARCHAR(1073741824) | | BINARY | VARBINARY | | DATE | DATE | | DATETIME | DATETIME | | TIMESTAMP | DATETIME | | ARRAY | ARRAY | | MAP | MAP | | STRUCT | STRUCT | note The TIMESTAMP type will lose precision due to type conversion in StarRocks. #### Collect CBO statistics[​](#collect-cbo-statistics "Direct link to Collect CBO statistics") In the current version, MaxCompute catalogs cannot automatically collect CBO statistics for MaxCompute tables, and consequently the optimizer may not be able to generate the optimal query plans. As such, manually scanning the CBO statistics for MaxCompute tables and importing them into StarRocks can effectively expedite queries. Suppose that there is a MaxCompute table named `mc_table` in your MaxCompute cluster. You can create a manual collection task for collecting CBO statistics by using [ANALYZE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/ANALYZE_TABLE.md): ```sql ANALYZE TABLE mc_table; ``` #### Manually update metadata cache[​](#manually-update-metadata-cache "Direct link to Manually update metadata cache") By default, StarRocks caches the metadata of MaxCompute to improve query performance. Therefore, after making schema changes or other updates to a MaxCompute table, you can use [REFRESH EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/REFRESH_EXTERNAL_TABLE.md) to manually update the metadata of the table, thereby ensuring that StarRocks can obtain the most recent metadata promptly: ```sql REFRESH EXTERNAL TABLE [PARTITION ('partition_name', ...)] ``` --- ### Paimon catalog Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) StarRocks supports Paimon catalogs from v3.1 onwards. A Paimon catalog is a kind of external catalog that enables you to query data from Apache Paimon without ingestion. Also, you can directly transform and load data from Paimon by using [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) based on Paimon catalogs. To ensure successful SQL workloads on your Paimon cluster, your StarRocks cluster must be able to access the storage system and metastore of your Paimon cluster. StarRocks supports the following storage systems and metastores: * Distributed file system (HDFS) or object storage like AWS S3, Microsoft Azure Storage, Google GCS, or or other S3-compatible storage system (for example, MinIO) * Metastore like your file system or Hive metastore #### Usage notes[​](#usage-notes "Direct link to Usage notes") You can only use Paimon catalogs to query data. You cannot use Paimon catalogs to drop, delete, or insert data into your Paimon cluster. #### Paimon to StarRocks data types[​](#paimon-to-starrocks-data-types "Direct link to Paimon to StarRocks data types") | Paimon Type | StarRocks Type | | --------------------- | --------------------------- | | `BINARY` | `VARBINARY` | | `VARBINARY` | `VARBINARY` | | `CHAR` | `CHAR(length)` | | `VARCHAR` | `VARCHAR` | | `BOOLEAN` | `BOOLEAN` | | `DECIMAL` | `DECIMAL(precision, scale)` | | `TINYINT` | `TINYINT` | | `SMALLINT` | `SMALLINT` | | `INT` | `INT` | | `BIGINT` | `BIGINT` | | `FLOAT` | `FLOAT` | | `DOUBLE` | `DOUBLE` | | `DATE` | `DATE` | | `TIME` | `TIME` | | `TIMESTAMP` | `DATETIME` | | `LocalZonedTimestamp` | `DATETIME` | | `ARRAY` | `ARRAY` | | `MAP` | `MAP` | | `ROW/STRUCT` | `STRUCT` | #### Integration preparations[​](#integration-preparations "Direct link to Integration preparations") Before you create a Paimon catalog, make sure your StarRocks cluster can integrate with the storage system and metastore of your Paimon cluster. ##### AWS IAM[​](#aws-iam "Direct link to AWS IAM") If your Paimon cluster uses AWS S3 as storage, choose your suitable authentication method and make the required preparations to ensure that your StarRocks cluster can access the related AWS cloud resources. The following authentication methods are recommended: * Instance profile (recommended) * Assumed role * IAM user Of the above-mentioned three authentication methods, instance profile is the most widely used. For more information, see [Preparation for authentication in AWS IAM](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#preparation-for-iam-user-based-authentication). ##### HDFS[​](#hdfs "Direct link to HDFS") If you choose HDFS as storage, configure your StarRocks cluster as follows: * (Optional) Set the username that is used to access your HDFS cluster and Hive metastore. By default, StarRocks uses the username of the FE and BE or CN processes to access your HDFS cluster and Hive metastore. You can also set the username by adding `export HADOOP_USER_NAME=""` at the beginning of the **fe/conf/hadoop\_env.sh** file of each FE and at the beginning of the **be/conf/hadoop\_env.sh** file of each BE or the **cn/conf/hadoop\_env.sh** file of each CN. After you set the username in these files, restart each FE and each BE or CN to make the parameter settings take effect. You can set only one username for each StarRocks cluster. * When you query Paimon data, the FEs and BEs or CNs of your StarRocks cluster use the HDFS client to access your HDFS cluster. In most cases, you do not need to configure your StarRocks cluster to achieve that purpose, and StarRocks starts the HDFS client using the default configurations. You need to configure your StarRocks cluster only in the following situations: * High availability (HA) is enabled for your HDFS cluster: Add the **hdfs-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. * View File System (ViewFs) is enabled for your HDFS cluster: Add the **core-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. > **NOTE** > > If an error indicating an unknown host is returned when you send a query, you must add the mapping between the host names and IP addresses of your HDFS cluster nodes to the **/etc/hosts** path. ##### Kerberos authentication[​](#kerberos-authentication "Direct link to Kerberos authentication") If Kerberos authentication is enabled for your HDFS cluster or Hive metastore, configure your StarRocks cluster as follows: * Run the `kinit -kt keytab_path principal` command on each FE and each BE or CN to obtain Ticket Granting Ticket (TGT) from Key Distribution Center (KDC). To run this command, you must have the permissions to access your HDFS cluster and Hive metastore. Note that accessing KDC with this command is time-sensitive. Therefore, you need to use cron to run this command periodically. * Add `JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"` to the **$FE\_HOME/conf/fe.conf** file of each FE and to the **$BE\_HOME/conf/be.conf** file of each BE or the **$CN\_HOME/conf/cn.conf** file of each CN. In this example, `/etc/krb5.conf` is the save path of the **krb5.conf** file. You can modify the path based on your needs. #### Create a Paimon catalog[​](#create-a-paimon-catalog "Direct link to Create a Paimon catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ( "type" = "paimon", CatalogParams, StorageCredentialParams, ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### catalog\_name[​](#catalog_name "Direct link to catalog_name") The name of the Paimon catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### comment[​](#comment "Direct link to comment") The description of the Paimon catalog. This parameter is optional. ###### type[​](#type "Direct link to type") The type of your data source. Set the value to `paimon`. ###### CatalogParams[​](#catalogparams "Direct link to CatalogParams") A set of parameters about how StarRocks accesses the metadata of your Paimon cluster. The following table describes the parameter you need to configure in `CatalogParams`. | Parameter | Required | Description | | ------------------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | paimon.catalog.type | Yes | The type of metastore that you use for your Paimon cluster. Set this parameter to `filesystem` or `hive`. | | paimon.catalog.warehouse | Yes | The warehouse storage path of your Paimon data. | | hive.metastore.uris | No | The URI of your Hive metastore. Format: `thrift://:`. If high availability (HA) is enabled for your Hive metastore, you can specify multiple metastore URIs and separate them with commas (`,`), for example, `"thrift://:,thrift://:,thrift://:"`. | > **NOTE** > > If you use Hive metastore, you must add the mapping between the host names and IP addresses of your Hive metastore nodes to the `/etc/hosts` path before you query Paimon data. Otherwise, StarRocks may fail to access your Hive metastore when you start a query. ###### StorageCredentialParams[​](#storagecredentialparams "Direct link to StorageCredentialParams") A set of parameters about how StarRocks integrates with your storage system. This parameter set is optional. If you use HDFS as storage, you do not need to configure `StorageCredentialParams`. If you use AWS S3, other S3-compatible storage system, Microsoft Azure Storage, or Google GCS as storage, you must configure `StorageCredentialParams`. ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") If you choose AWS S3 as storage for your Paimon cluster, take one of the following actions: * To choose the instance profile-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.endpoint" = "" ``` * To choose the assumed role-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.endpoint" = "" ``` * To choose the IAM user-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.endpoint" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ----------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. | | aws.s3.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS S3 bucket. If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.endpoint | Yes | The endpoint that is used to connect to your AWS S3 bucket. For example, `https://s3.us-west-2.amazonaws.com`. | | aws.s3.access\_key | No | The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.secret\_key | No | The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in AWS IAM Console, see [Authentication parameters for accessing AWS S3](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-s3). ###### S3-compatible storage system[​](#s3-compatible-storage-system "Direct link to S3-compatible storage system") If you choose an S3-compatible storage system, such as MinIO, as storage for your Paimon cluster, configure `StorageCredentialParams` as follows to ensure a successful integration: ```sql "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ---------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.enable\_ssl | Yes | Specifies whether to enable SSL connection.
Valid values: `true` and `false`. Default value: `true`. | | aws.s3.enable\_path\_style\_access | Yes | Specifies whether to enable path-style access.
Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`.
Path-style URLs use the following format: `https://s3..amazonaws.com//`. For example, if you create a bucket named `DOC-EXAMPLE-BUCKET1` in the US West (Oregon) Region, and you want to access the `alice.jpg` object in that bucket, you can use the following path-style URL: `https://s3.us-west-2.amazonaws.com/DOC-EXAMPLE-BUCKET1/alice.jpg`. | | aws.s3.endpoint | Yes | The endpoint that is used to connect to your S3-compatible storage system instead of AWS S3. | | aws.s3.access\_key | Yes | The access key of your IAM user. | | aws.s3.secret\_key | Yes | The secret key of your IAM user. | ###### Microsoft Azure Storage[​](#microsoft-azure-storage "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage "Direct link to Azure Blob Storage") If you choose Blob Storage as storage for your Paimon cluster, take one of the following actions: * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | --------------------------- | -------- | -------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.shared\_key | Yes | The shared key of your Blob Storage account. | * To choose the SAS Token authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | --------------------------- | -------- | --------------------------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.container | Yes | The name of the blob container that stores your data. | | azure.blob.sas\_token | Yes | The SAS token that is used to access your Blob Storage account. | ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2 "Direct link to Azure Data Lake Storage Gen2") If you choose Data Lake Storage Gen2 as storage for your Paimon cluster, take one of the following actions: * To choose the Managed Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ------------------------------------------ | -------- | ------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_use\_managed\_identity | Yes | Specifies whether to enable the Managed Identity authentication method. Set the value to `true`. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the managed identity. | * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ---------------------------- | -------- | -------------------------------------------------------------- | | azure.adls2.storage\_account | Yes | The username of your Data Lake Storage Gen2 storage account. | | azure.adls2.shared\_key | Yes | The shared key of your Data Lake Storage Gen2 storage account. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ``` The following table describes the parameters you need to configure `in StorageCredentialParams`. | Parameter | Required | Description | | ------------------------------------ | -------- | -------------------------------------------------------------------------- | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls2.oauth2\_client\_secret | Yes | The value of the new client (application) secret created. | | azure.adls2.oauth2\_client\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | * To choose the Workload Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_token_file" = "", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_token\_file | Yes | The absolute file path to the OAuth2 token file projected into the pod by the Azure Workload Identity webhook. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client ID (application ID) of the Azure AD application (user-assigned managed identity or app registration) associated with the workload identity. | ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1 "Direct link to Azure Data Lake Storage Gen1") If you choose Data Lake Storage Gen1 as storage for your Paimon cluster, take one of the following actions: * To choose the Managed Service Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.use_managed_service_identity" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------- | | azure.adls1.use\_managed\_service\_identity | Yes | Specifies whether to enable the Managed Service Identity authentication method. Set the value to `true`. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ------------------------------ | -------- | -------------------------------------------------------------------------- | | azure.adls1.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls1.oauth2\_credential | Yes | The value of the new client (application) secret created. | | azure.adls1.oauth2\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | ###### Google GCS[​](#google-gcs "Direct link to Google GCS") If you choose Google GCS as storage for your Paimon cluster, take one of the following actions: * To choose the VM-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.use_compute_engine_service_account" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Default value | Value example | Description | | ---------------------------------------------- | ------------- | ------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | FALSE | TRUE | Specifies whether to directly use the service account that is bound to your Compute Engine. | * To choose the service account-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Default value | Value example | Description | | ------------------------------------------ | ------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------- | | gcp.gcs.service\_account\_email | "" | "" | The email address in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the service account. | * To choose the impersonation-based authentication method, configure `StorageCredentialParams` as follows: * Make a VM instance impersonate a service account: ```sql "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Default value | Value example | Description | | ---------------------------------------------- | ------------- | ------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | FALSE | TRUE | Specifies whether to directly use the service account that is bound to your Compute Engine. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The service account that you want to impersonate. | * Make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account): ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Default value | Value example | Description | | ------------------------------------------ | ------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | gcp.gcs.service\_account\_email | "" | "" | The email address in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the meta service account. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The data service account that you want to impersonate. | ##### Examples[​](#examples "Direct link to Examples") The following examples create a Paimon catalog named `paimon_catalog_fs` whose metastore type `paimon.catalog.type` is set to `filesystem` to query data from your Paimon cluster. ###### AWS S3[​](#aws-s3-1 "Direct link to AWS S3") * If you choose the instance profile-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "aws.s3.use_instance_profile" = "true", "aws.s3.endpoint" = "" ); ``` * If you choose the assumed role-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.endpoint" = "" ); ``` * If you choose the IAM user-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.endpoint" = "" ); ``` ###### S3-compatible storage system[​](#s3-compatible-storage-system-1 "Direct link to S3-compatible storage system") Use MinIO as an example. Run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "aws.s3.enable_ssl" = "true", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ); ``` ###### Microsoft Azure Storage[​](#microsoft-azure-storage-1 "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage-1 "Direct link to Azure Blob Storage") * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` * If you choose the SAS Token authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1-1 "Direct link to Azure Data Lake Storage Gen1") * If you choose the Managed Service Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "azure.adls1.use_managed_service_identity" = "true" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2-1 "Direct link to Azure Data Lake Storage Gen2") * If you choose the Managed Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` * If you choose the Workload Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### Google GCS[​](#google-gcs-1 "Direct link to Google GCS") * If you choose the VM-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "gcp.gcs.use_compute_engine_service_account" = "true" ); ``` * If you choose the service account-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ); ``` * If you choose the impersonation-based authentication method: * If you make a VM instance impersonate a service account, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ); ``` * If you make a service account impersonate another service account, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ); ``` #### View Paimon catalogs[​](#view-paimon-catalogs "Direct link to View Paimon catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of a Paimon catalog named `paimon_catalog_fs`: ```sql SHOW CREATE CATALOG paimon_catalog_fs; ``` #### Drop a Paimon catalog[​](#drop-a-paimon-catalog "Direct link to Drop a Paimon catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop an external catalog. The following example drops a Paimon catalog named `paimon_catalog_fs`: ```sql DROP Catalog paimon_catalog_fs; ``` #### View the schema of a Paimon table[​](#view-the-schema-of-a-paimon-table "Direct link to View the schema of a Paimon table") You can use one of the following syntaxes to view the schema of a Paimon table: * View schema ```sql DESC[RIBE] ..; ``` * View schema and location from the CREATE statement ```sql SHOW CREATE TABLE ..; ``` #### Query a Paimon table[​](#query-a-paimon-table "Direct link to Query a Paimon table") 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your Paimon cluster: ```sql SHOW DATABASES FROM ; ``` 2. Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to switch to the destination catalog in the current session: ```sql SET CATALOG ; ``` Then, use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify the active database in the current session: ```sql USE ; ``` Or, you can use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to directly specify the active database in the destination catalog: ```sql USE .; ``` 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT count(*) FROM LIMIT 10; ``` #### Load data from Paimon[​](#load-data-from-paimon "Direct link to Load data from Paimon") Suppose you have an OLAP table named `olap_tbl`, you can transform and load data like below: ```sql INSERT INTO default_catalog.olap_db.olap_tbl SELECT * FROM paimon_table; ``` --- ### Query external data This topic guides you through querying data from external data sources by using external catalogs. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") External catalogs are created based on external data sources. For information about supported types of external catalogs, see [Catalog](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md#catalog). #### Procedure[​](#procedure "Direct link to Procedure") 1. Connect your StarRocks cluster. * If you use the MySQL client to connect the StarRocks cluster, you go to `default_catalog` by default after connecting. * If you use JDBC to connect the StarRocks cluster, you can go directly to the destination database in the default catalog by specifying `default_catalog.db_name` when connecting. 2. (Optional) Execute the following statement to view all the catalogs and find the external catalog you have created. See [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to check the output of this statement. ```sql SHOW CATALOGS; ``` 3. (Optional) Execute the following statement to view all the databases in the external catalog. See [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to check the output of this statement. ```sql SHOW DATABASES FROM catalog_name; ``` 4. (Optional) Execute the following statement to go to the destination database in the external catalog. ```sql USE catalog_name.db_name; ``` 5. Query external data. For more usages of the SELECT statement, see [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). ```sql SELECT * FROM table_name; ``` If you do not specify the external catalog and database in the preceding steps, you can directly specify them in a select query. ```sql SELECT * FROM catalog_name.db_name.table_name; ``` #### Examples[​](#examples "Direct link to Examples") If you already created a Hive catalog named `hive1` and want to use `hive1` to query data from `hive_db.hive_table` of an Apache Hive™ cluster, you can perform one of the following operations: ```sql USE hive1.hive_db; SELECT * FROM hive_table limit 1; ``` Or ```sql SELECT * FROM hive1.hive_db.hive_table limit 1; ``` #### References[​](#references "Direct link to References") To query data from your StarRocks cluster, see [Default catalog](https://docs.starrocks.io/docs/data_source/catalog/default_catalog.md). --- ### Unified catalog Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) A unified catalog is a type of external catalog that is provided by StarRocks from v3.2 onwards to handle tables from Apache Hive™, Apache Iceberg, Apache Hudi, Delta Lake, and Apache Kudu data sources as a unified data source without ingestion. With unified catalogs, you can: * Directly query data stored in Hive, Iceberg, Hudi, Delta Lake, Paimon, and Kudu without the need to manually create tables. * Use [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) or asynchronous materialized views (which are supported from v2.5 onwards) to process data stored in Hive, Iceberg, Hudi, Delta Lake, Paimon, and Kudu and load the data into StarRocks. * Perform operations on StarRocks to create or drop Hive and Iceberg databases and tables. To ensure successful SQL workloads on your unified data source, your StarRocks cluster must be able to access the storage system and metastore of your unified data source. StarRocks supports the following storage systems and metastores: * Distributed file system (HDFS) or object storage like AWS S3, Microsoft Azure Storage, Google GCS, or other S3-compatible storage system (for example, MinIO) * Metastore like Hive metastore or AWS Glue > **NOTE** > > If you choose AWS S3 as storage, you can use HMS or AWS Glue as metastore. If you choose any other storage system, you can only use HMS as metastore. #### Limits[​](#limits "Direct link to Limits") One unified catalog supports integrations with only a single storage system and a single metastore service. Therefore, make sure all the data sources you want to integrate as a unified data source with StarRocks use the same storage system and metastore service. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * See the "Usage notes" section in [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md), [Iceberg catalog](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md), [Hudi catalog](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md), [Delta Lake catalog](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md), [Paimon catalog](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog.md), and [Kudu catalog](https://docs.starrocks.io/docs/data_source/catalog/kudu_catalog.md) to understand the file formats and data types supported. * Format-specific operations are supported only for specific table formats. For example, [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) and [DROP TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md) are supported only for Hive and Iceberg, and [REFRESH EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/REFRESH_EXTERNAL_TABLE.md) is supported only for Hive and Hudi. When you create a table within a unified catalog by using the [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement, use the `ENGINE` parameter to specify the table format (Hive or Iceberg). #### Integration preparations[​](#integration-preparations "Direct link to Integration preparations") Before you create a unified catalog, make sure your StarRocks cluster can integrate with the storage system and metastore of your unified data source. ##### AWS IAM[​](#aws-iam "Direct link to AWS IAM") If you use AWS S3 as storage or AWS Glue as metastore, choose your suitable authentication method and make the required preparations to ensure that your StarRocks cluster can access the related AWS cloud resources. For more information, see [Authenticate to AWS resources - Preparations](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#preparations). ##### HDFS[​](#hdfs "Direct link to HDFS") If you choose HDFS as storage, configure your StarRocks cluster as follows: * (Optional) Set the username that is used to access your HDFS cluster and Hive metastore. By default, StarRocks uses the username of the FE and BE or CN processes to access your HDFS cluster and Hive metastore. You can also set the username by adding `export HADOOP_USER_NAME=""` at the beginning of the **fe/conf/hadoop\_env.sh** file of each FE and at the beginning of the **be/conf/hadoop\_env.sh** file of each BE or the **cn/conf/hadoop\_env.sh** file of each CN. After you set the username in these files, restart each FE and each BE or CN to make the parameter settings take effect. You can set only one username for each StarRocks cluster. * When you query data, the FEs and BEs or CNs of your StarRocks cluster use the HDFS client to access your HDFS cluster. In most cases, you do not need to configure your StarRocks cluster to achieve that purpose, and StarRocks starts the HDFS client using the default configurations. You need to configure your StarRocks cluster only in the following situations: * High availability (HA) is enabled for your HDFS cluster: Add the **hdfs-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. * View File System (ViewFs) is enabled for your HDFS cluster: Add the **core-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. > **NOTE** > > If an error indicating an unknown host is returned when you send a query, you must add the mapping between the host names and IP addresses of your HDFS cluster nodes to the **/etc/hosts** path. ##### Kerberos authentication[​](#kerberos-authentication "Direct link to Kerberos authentication") If Kerberos authentication is enabled for your HDFS cluster or Hive metastore, configure your StarRocks cluster as follows: * Run the `kinit -kt keytab_path principal` command on each FE and each BE or CN to obtain Ticket Granting Ticket (TGT) from Key Distribution Center (KDC). To run this command, you must have the permissions to access your HDFS cluster and Hive metastore. Note that accessing KDC with this command is time-sensitive. Therefore, you need to use cron to run this command periodically. * Add `JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"` to the **$FE\_HOME/conf/fe.conf** file of each FE and to the **$BE\_HOME/conf/be.conf** file of each BE or the **$CN\_HOME/conf/cn.conf** file of each CN. In this example, `/etc/krb5.conf` is the save path of the krb5.conf file. You can modify the path based on your needs. #### Create a unified catalog[​](#create-a-unified-catalog "Direct link to Create a unified catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ( "type" = "unified", MetastoreParams, StorageCredentialParams, MetadataUpdateParams, PaimonCatalogParams, KuduCatalogParams ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### catalog\_name[​](#catalog_name "Direct link to catalog_name") The name of the unified catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### comment[​](#comment "Direct link to comment") The description of the unified catalog. This parameter is optional. ###### type[​](#type "Direct link to type") The type of your data source. Set the value to `unified`. ###### MetastoreParams[​](#metastoreparams "Direct link to MetastoreParams") A set of parameters about how StarRocks integrates with your metastore. ###### Hive metastore[​](#hive-metastore "Direct link to Hive metastore") If you choose Hive metastore as the metastore of your unified data source, configure `MetastoreParams` as follows: ```sql "unified.metastore.type" = "hive", "hive.metastore.uris" = "" ``` > **NOTE** > > Before querying data, you must add the mapping between the host names and IP addresses of your Hive metastore nodes to the **/etc/hosts** path. Otherwise, StarRocks may fail to access your Hive metastore when you start a query. The following table describes the parameters you need to configure in `MetastoreParams`. | Parameter | Required | Description | | ---------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | unified.metastore.type | Yes | The type of metastore that you use for your unified data source. Set the value to `hive`. | | hive.metastore.uris | Yes | The URI of your Hive metastore. Format: `thrift://:`. If high availability (HA) is enabled for your Hive metastore, you can specify multiple metastore URIs and separate them with commas (`,`), for example, `"thrift://:,thrift://:,thrift://:"`. | ###### AWS Glue[​](#aws-glue "Direct link to AWS Glue") If you choose AWS Glue as the metastore of your data source, which is supported only when you choose AWS S3 as storage, take one of the following actions: * To choose the instance profile-based authentication method, configure `MetastoreParams` as follows: ```sql "unified.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "" ``` * To choose the assumed role-based authentication method, configure `MetastoreParams` as follows: ```sql "unified.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "", "aws.glue.region" = "" ``` * To choose the IAM user-based authentication method, configure `MetastoreParams` as follows: ```sql "unified.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "" ``` The following table describes the parameters you need to configure in `MetastoreParams`. | Parameter | Required | Description | | ------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | unified.metastore.type | Yes | The type of metastore that you use for your unified data source. Set the value to `glue`. | | aws.glue.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication. Valid values: `true` and `false`. Default value: `false`. | | aws.glue.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS Glue Data Catalog. If you use the assumed role-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.region | Yes | The region in which your AWS Glue Data Catalog resides. Example: `us-west-1`. | | aws.glue.access\_key | No | The access key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.secret\_key | No | The secret key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS Glue and how to configure an access control policy in the AWS IAM Console, see [Authentication parameters for accessing AWS Glue](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-glue). ###### StorageCredentialParams[​](#storagecredentialparams "Direct link to StorageCredentialParams") A set of parameters about how StarRocks integrates with your storage system. This parameter set is optional. If you use HDFS as storage, you do not need to configure `StorageCredentialParams`. If you use AWS S3, other S3-compatible storage system, Microsoft Azure Storage, or Google GCS as storage, you must configure `StorageCredentialParams`. ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") If you choose AWS S3 as storage, take one of the following actions: * To choose the instance profile-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "" ``` * To choose the assumed role-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.region" = "" ``` * To choose the IAM user-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ----------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. | | aws.s3.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS S3 bucket. If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.region | Yes | The region in which your AWS S3 bucket resides. Example: `us-west-1`. | | aws.s3.access\_key | No | The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.secret\_key | No | The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in AWS IAM Console, see [Authentication parameters for accessing AWS S3](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-s3). ###### S3-compatible storage system[​](#s3-compatible-storage-system "Direct link to S3-compatible storage system") If you choose an S3-compatible storage system, such as MinIO, as storage, configure `StorageCredentialParams` as follows to ensure a successful integration: ```sql "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ---------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.enable\_ssl | Yes | Specifies whether to enable SSL connection.
Valid values: `true` and `false`. Default value: `true`. | | aws.s3.enable\_path\_style\_access | Yes | Specifies whether to enable path-style access.
Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`.
Path-style URLs use the following format: `https://s3..amazonaws.com//`. For example, if you create a bucket named `DOC-EXAMPLE-BUCKET1` in the US West (Oregon) Region, and you want to access the `alice.jpg` object in that bucket, you can use the following path-style URL: `https://s3.us-west-2.amazonaws.com/DOC-EXAMPLE-BUCKET1/alice.jpg`. | | aws.s3.endpoint | Yes | The endpoint that is used to connect to your S3-compatible storage system instead of AWS S3. | | aws.s3.access\_key | Yes | The access key of your IAM user. | | aws.s3.secret\_key | Yes | The secret key of your IAM user. | ###### Microsoft Azure Storage[​](#microsoft-azure-storage "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage "Direct link to Azure Blob Storage") If you choose Blob Storage as storage, take one of the following actions: * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | -------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.shared\_key | Yes | The shared key of your Blob Storage account. | * To choose the SAS Token authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | --------------------------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.container | Yes | The name of the blob container that stores your data. | | azure.blob.sas\_token | Yes | The SAS token that is used to access your Blob Storage account. | ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2 "Direct link to Azure Data Lake Storage Gen2") If you choose Data Lake Storage Gen2 as storage, take one of the following actions: * To choose the Managed Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_use\_managed\_identity | Yes | Specifies whether to enable the Managed Identity authentication method. Set the value to `true`. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the managed identity. | * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ---------------------------- | ------------ | -------------------------------------------------------------- | | azure.adls2.storage\_account | Yes | The username of your Data Lake Storage Gen2 storage account. | | azure.adls2.shared\_key | Yes | The shared key of your Data Lake Storage Gen2 storage account. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ``` The following table describes the parameters you need to configure `in StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls2.oauth2\_client\_secret | Yes | The value of the new client (application) secret created. | | azure.adls2.oauth2\_client\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | * To choose the Workload Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_token_file" = "", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_token\_file | Yes | The absolute file path to the OAuth2 token file projected into the pod by the Azure Workload Identity webhook. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client ID (application ID) of the Azure AD application (user-assigned managed identity or app registration) associated with the workload identity. | ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1 "Direct link to Azure Data Lake Storage Gen1") If you choose Data Lake Storage Gen1 as storage, take one of the following actions: * To choose the Managed Service Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.use_managed_service_identity" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------- | | azure.adls1.use\_managed\_service\_identity | Yes | Specifies whether to enable the Managed Service Identity authentication method. Set the value to `true`. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls1.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls1.oauth2\_credential | Yes | The value of the new client (application) secret created. | | azure.adls1.oauth2\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | ###### Google GCS[​](#google-gcs "Direct link to Google GCS") If you choose Google GCS as storage, take one of the following actions: * To choose the VM-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.use_compute_engine_service_account" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | * To choose the service account-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------- | | gcp.gcs.service\_account\_email | "" | "" | The email address in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the service account. | * To choose the impersonation-based authentication method, configure `StorageCredentialParams` as follows: * Make a VM instance impersonate a service account: ```sql "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The service account that you want to impersonate. | * Make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account): ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | gcp.gcs.service\_account\_email | "" | "" | The email address in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the meta service account. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The data service account that you want to impersonate. | ###### MetadataUpdateParams[​](#metadataupdateparams "Direct link to MetadataUpdateParams") A set of parameters about how StarRocks updates the cached metadata of Hive, Hudi, and Delta Lake. This parameter set is optional. For more information about the policies for updating cached metadata from Hive, Hudi, and Delta Lake, see [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md), [Hudi catalog](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md), and [Delta Lake catalog](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md). In most cases, you can ignore `MetadataUpdateParams` and do not need to tune the policy parameters in it, because the default values of these parameters already provide you with an out-of-the-box performance. However, if the frequency of data updates in Hive, Hudi, or Delta Lake is high, you can tune these parameters to further optimize the performance of automatic asynchronous updates. | Parameter | Required | Description | | ------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_metastore\_cache | No | Specifies whether StarRocks caches the metadata of Hive, Hudi, or Delta Lake tables. Valid values: `true` and `false`. Default value: `true`. The value `true` enables the cache, and the value `false` disables the cache. | | enable\_remote\_file\_cache | No | Specifies whether StarRocks caches the metadata of the underlying data files of Hive, Hudi, or Delta Lake tables or partitions. Valid values: `true` and `false`. Default value: `true`. The value true enables the cache, and the value `false` disables the cache. | | metastore\_cache\_refresh\_interval\_sec | No | The time interval at which StarRocks asynchronously updates the metadata of Hive, Hudi, or Delta Lake tables or partitions cached in itself. Unit: seconds. Default value: `7200`, which is 2 hours. | | remote\_file\_cache\_refresh\_interval\_sec | No | The time interval at which StarRocks asynchronously updates the metadata of the underlying data files of Hive, Hudi, or Delta Lake tables or partitions cached in itself. Unit: seconds. Default value: `60`. | | metastore\_cache\_ttl\_sec | No | The time interval at which StarRocks automatically discards the metadata of Hive, Hudi, or Delta Lake tables or partitions cached in itself. Unit: seconds. Default value: `86400`, which is 24 hours. | | remote\_file\_cache\_ttl\_sec | No | The time interval at which StarRocks automatically discards the metadata of the underlying data files of Hive, Hudi, or Delta Lake tables or partitions cached in itself. Unit: seconds. Default value: `129600`, which is 36 hours. | ###### PaimonCatalogParams[​](#paimoncatalogparams "Direct link to PaimonCatalogParams") A set of parameters about how to connect Paimon Catalog. This parameter set is optional. | Parameter | Required | Description | | ------------------------ | -------- | ----------------------------------------------- | | paimon.catalog.warehouse | No | The warehouse storage path of your Paimon data. | ###### KuduCatalogParams[​](#kuducatalogparams "Direct link to KuduCatalogParams") A set of parameters about how to connect Kudu Catalog. This parameter set is optional. | Parameter | Required | Description | | ----------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | kudu.master | No | Specifies the Kudu Master address, which defaults to `localhost:7051`. | | kudu.schema-emulation.enabled | No | option to enable or disable the `schema` emulation. By default, it is turned off (false), which means that all tables belong to the `default` `schema`. | | kudu.schema-emulation.prefix | No | The prefix for `schema` emulation should only be set when `kudu.schema-emulation.enabled` = `true`. The default prefix used is empty string: \`\`. | ##### Examples[​](#examples "Direct link to Examples") The following examples create a unified catalog named `unified_catalog_hms` or `unified_catalog_glue`, depending on the type of metastore you use, to query data from your unified data source. ###### HDFS[​](#hdfs-1 "Direct link to HDFS") If you use HDFS as storage, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` ###### AWS S3[​](#aws-s3-1 "Direct link to AWS S3") ###### Instance profile-based authentication[​](#instance-profile-based-authentication "Direct link to Instance profile-based authentication") * If you use Hive metastore, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue with Amazon EMR, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_glue PROPERTIES ( "type" = "unified", "unified.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` ###### Assumed role-based authentication[​](#assumed-role-based-authentication "Direct link to Assumed role-based authentication") * If you use Hive metastore, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue with Amazon EMR, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_glue PROPERTIES ( "type" = "unified", "unified.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "arn:aws:iam::081976408565:role/test_glue_role", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` ###### IAM user-based authentication[​](#iam-user-based-authentication "Direct link to IAM user-based authentication") * If you use Hive metastore, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue with Amazon EMR, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_glue PROPERTIES ( "type" = "unified", "unified.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` ###### S3-compatible storage system[​](#s3-compatible-storage-system-1 "Direct link to S3-compatible storage system") Use MinIO as an example. Run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.enable_ssl" = "true", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ); ``` ###### Microsoft Azure Storage[​](#microsoft-azure-storage-1 "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage-1 "Direct link to Azure Blob Storage") * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` * If you choose the SAS Token authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1-1 "Direct link to Azure Data Lake Storage Gen1") * If you choose the Managed Service Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.use_managed_service_identity" = "true" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2-1 "Direct link to Azure Data Lake Storage Gen2") * If you choose the Managed Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` * If you choose the Workload Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### Google GCS[​](#google-gcs-1 "Direct link to Google GCS") * If you choose the VM-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true" ); ``` * If you choose the service account-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ); ``` * If you choose the impersonation-based authentication method: * If you make a VM instance impersonate a service account, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ); ``` * If you make a service account impersonate another service account, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ); ``` #### View unified catalogs[​](#view-unified-catalogs "Direct link to View unified catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of a unified catalog named `unified_catalog_glue`: ```sql SHOW CREATE CATALOG unified_catalog_glue; ``` #### Switch to a Unified Catalog and a database in it[​](#switch-to-a-unified-catalog-and-a-database-in-it "Direct link to Switch to a Unified Catalog and a database in it") You can use one of the following methods to switch to a unified catalog and a database in it: * Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to specify a unified catalog in the current session, and then use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify an active database: ```sql -- Switch to a specified catalog in the current session: SET CATALOG -- Specify the active database in the current session: USE ``` * Directly use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to switch to a unified catalog and a database in it: ```sql USE . ``` #### Drop a unified catalog[​](#drop-a-unified-catalog "Direct link to Drop a unified catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop an external catalog. The following example drops a unified catalog named `unified_catalog_glue`: ```sql DROP CATALOG unified_catalog_glue; ``` #### View the schema of a table from a unified catalog[​](#view-the-schema-of-a-table-from-a-unified-catalog "Direct link to View the schema of a table from a unified catalog") You can use one of the following syntaxes to view the schema of a table from a unified catalog: * View schema ```sql DESC[RIBE] .. ``` * View schema and location from the CREATE statement ```sql SHOW CREATE TABLE .. ``` #### Query data from a unified catalog[​](#query-data-from-a-unified-catalog "Direct link to Query data from a unified catalog") To query data from a unified catalog, follow these steps: 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your unified data source with which the unified catalog is associated: ```sql SHOW DATABASES FROM ``` 2. [Switch to a Hive Catalog and a database in it](#switch-to-a-unified-catalog-and-a-database-in-it). 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT count(*) FROM LIMIT 10 ``` #### Load data from Hive, Iceberg, Hudi, Delta Lake, or Kudu[​](#load-data-from-hive-iceberg-hudi-delta-lake-or-kudu "Direct link to Load data from Hive, Iceberg, Hudi, Delta Lake, or Kudu") You can use [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) to load the data of a Hive, Iceberg, Hudi, Delta Lake, or Kudu table into a StarRocks table created within a unified catalog. The following example loads the data of the Hive table `hive_table` into the StarRocks table `test_tbl` created in the database `test_database` that belongs to the unified catalog `unified_catalog`: ```sql INSERT INTO unified_catalog.test_database.test_table SELECT * FROM hive_table ``` #### Create a database in a unified catalog[​](#create-a-database-in-a-unified-catalog "Direct link to Create a database in a unified catalog") Similar to the internal catalog of StarRocks, if you have the CREATE DATABASE privilege on a unified catalog, you can use the CREATE DATABASE statement to create a database in that catalog. > **NOTE** > > You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). StarRocks supports creating only Hive and Iceberg databases in unified catalogs. [Switch to a unified catalog](#switch-to-a-unified-catalog-and-a-database-in-it), and then use the following statement to create a database in that catalog: ```sql CREATE DATABASE [properties ("location" = ":///")] ``` The `location` parameter specifies the file path in which you want to create the database, which can be in either HDFS or cloud storage. * When you use Hive metastore as the metastore of your data source, the `location` parameter defaults to `/`, which is supported by Hive metastore if you do not specify that parameter at database creation. * When you use AWS Glue as the metastore of your data source, the `location` parameter does not have a default value, and therefore you must specify that parameter at database creation. The `prefix` varies based on the storage system you use: | **Storage system** | **`Prefix`** **value** | | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | HDFS | `hdfs` | | Google GCS | `gs` | | Azure Blob Storage | - If your storage account allows access over HTTP, the `prefix` is `wasb`.
- If your storage account allows access over HTTPS, the `prefix` is `wasbs`. | | Azure Data Lake Storage Gen1 | `adl` | | Azure Data Lake Storage Gen2 | - If your storage account allows access over HTTP, the`prefix` is `abfs`.
- If your storage account allows access over HTTPS, the `prefix` is `abfss`. | | AWS S3 or other S3-compatible storage (for example, MinIO) | `s3` | #### Drop a database from a unified catalog[​](#drop-a-database-from-a-unified-catalog "Direct link to Drop a database from a unified catalog") Similar to the internal databases of StarRocks, if you have the [DROP](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md#database) privilege on a database created within a unified catalog, you can use the [DROP DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/DROP_DATABASE.md) statement to drop that database. You can only drop empty databases. > **NOTE** > > You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). StarRocks supports dropping only Hive and Iceberg databases from unified catalogs. When you drop a database from a unified catalog, the database's file path on your HDFS cluster or cloud storage will not be dropped along with the database. [Switch to a unified catalog](#switch-to-a-unified-catalog-and-a-database-in-it), and then use the following statement to drop a database in that catalog: ```sql DROP DATABASE ``` #### Create a table in a unified catalog[​](#create-a-table-in-a-unified-catalog "Direct link to Create a table in a unified catalog") Similar to the internal databases of StarRocks, if you have the [CREATE TABLE](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md#database) privilege on a database created within a unified catalog, you can use the [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) or \[CREATE TABLE AS SELECT ../../sql-reference/sql-statements/table\_bucket\_part\_index/CREATE\_TABLE\_AS\_SELECT.mdELECT.md) statement to create a table in that database. > **NOTE** > > You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). StarRocks supports creating only Hive and Iceberg tables in unified catalogs. [Switch to a Hive Catalog and a database in it](#switch-to-a-unified-catalog-and-a-database-in-it). Then, use [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) to create a Hive or Iceberg table in that database: ```sql CREATE TABLE (column_definition1[, column_definition2, ...] ENGINE = {|hive|iceberg} [partition_desc] ``` For more information, see [Create a Hive table](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md#create-a-hive-table) and [Create an Iceberg table](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md#create-an-iceberg-table). The following example creates a Hive table named `hive_table`. The table consists of three columns `action`, `id`, and `dt`, of which `id` and `dt`are partition columns. ```sql CREATE TABLE hive_table ( action varchar(65533), id int, dt date ) ENGINE = hive PARTITION BY (id,dt); ``` #### Sink data to a table in a unified catalog[​](#sink-data-to-a-table-in-a-unified-catalog "Direct link to Sink data to a table in a unified catalog") Similar to the internal tables of StarRocks, if you have the [INSERT](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md#table) privilege on a table created within a unified catalog, you can use the [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) statement to sink the data of a StarRocks table to that Unified Catalog table (currently only Parquet-formatted Unified Catalog tables are supported). > **NOTE** > > You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). StarRocks supports sinking data only to Hive and Iceberg tables in unified catalogs. [Switch to a Hive Catalog and a database in it](#switch-to-a-unified-catalog-and-a-database-in-it). Then, use [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) to insert data into a Hive or Iceberg table in that database: ```sql INSERT {INTO | OVERWRITE} [ (column_name [, ...]) ] { VALUES ( { expression | DEFAULT } [, ...] ) [, ...] | query } -- If you want to sink data to specified partitions, use the following syntax: INSERT {INTO | OVERWRITE} PARTITION (par_col1= [, par_col2=...]) { VALUES ( { expression | DEFAULT } [, ...] ) [, ...] | query } ``` For more information, see [Sink data to a Hive table ](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md#sink-data-to-a-hive-table)and [Sink data to an Iceberg table](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md#sink-data-to-an-iceberg-table). The following example inserts three data rows to a Hive table named `hive_table`: ```sql INSERT INTO hive_table VALUES ("buy", 1, "2023-09-01"), ("sell", 2, "2023-09-02"), ("buy", 3, "2023-09-03"); ``` #### Drop a table from a unified catalog[​](#drop-a-table-from-a-unified-catalog "Direct link to Drop a table from a unified catalog") Similar to the internal tables of StarRocks, if you have the [DROP](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md#table) privilege on a table created within a unified catalog, you can use the [DROP TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md) statement to drop that table. > **NOTE** > > You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). StarRocks supports dropping only Hive and Iceberg tables from unified catalogs. [Switch to a Hive Catalog and a database in it](#switch-to-a-unified-catalog-and-a-database-in-it). Then, use [DROP TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md) to drop a Hive or Iceberg table in that database: ```sql DROP TABLE ``` For more information, see [Drop a Hive table](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md#drop-a-hive-table) and [Drop an Iceberg table](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md#drop-an-iceberg-table). The following example drops a Hive table named `hive_table`: ```sql DROP TABLE hive_table FORCE ``` --- ### Data Cache Understand the core principles of Data Cache and how to accelerate data queries with Data Cache. Data Cache is used to cache the data of native tables and external tables. This feature is enabled by default from v3.3.0 onwards. And from v4.0 onwards, the in-memory cache and disk cache have been uniformly integrated into the Data Cache system to facilitate management. Data Cache consists of two components: Page Cache (in-memory cache) and Block Cache (disk cache). #### Principles of Page Cache[​](#principles-of-page-cache "Direct link to Principles of Page Cache") As an in-memory cache, Page Cache is responsible for storing data pages of native and external tables after decompression. The size of these pages is not fixed. Currently, Page Cache supports caching the following types of data: * Data pages and index pages of native tables * Footer information of external table data files * Partial decompressed data pages of external tables Page Cache currently uses the LRU (Least Recently Used) strategy for data eviction. #### Principles of Block Cache[​](#principles-of-block-cache "Direct link to Principles of Block Cache") Block Cache is a disk-based cache whose primary function is to cache data files (from external tables, and also from cloud-native tables in shared-data cluster) to local disks. This reduces remote data access latency and improves query efficiency. The size of each data block is fixed. ##### Background and values[​](#background-and-values "Direct link to Background and values") In data lake analytics and cloud-native table scenarios, StarRocks, acting as an OLAP query engine, needs to scan data files stored in HDFS or object storage (hereinafter referred to as the "remote storage system"). This process faces two major performance bottlenecks: * The more files a query needs to read, the greater the remote I/O overhead. * In ad-hoc query scenarios, frequent access to the same data leads to redundant remote I/O consumption. To address these issues, the Block Cache feature has been introduced in v2.5. It splits raw data from the remote storage system into multiple blocks based on a specific strategy and caches these blocks in the local disks of BE or CN nodes. By avoiding repeated retrieval of remote data, the query performance against hot data is significantly improved. ##### Scenarios[​](#scenarios "Direct link to Scenarios") * Query against data from remote storage systems using external catalogs (except JDBC Catalogs). * Query against cloud-native tables in the shared-data clusters. ##### Core mechanism[​](#core-mechanism "Direct link to Core mechanism") ###### Data splitting and cache unit[​](#data-splitting-and-cache-unit "Direct link to Data splitting and cache unit") When the system caches remote files, it splits the original files into blocks of equal size according to the configured strategy. A block is the minimum cache unit, and its size is customizable. Example: If the block size is configured as 1 MB, when querying a 128 MB Parquet file on Amazon S3, the file will be split into 128 consecutive blocks (that is, `[0, 1 MB)`, `[1 MB, 2 MB)`, ..., `[127 MB, 128 MB)`). Each block is assigned a globally unique cache identifier (cache key), which consists of the following three parts: ```plain hash(filename) + fileModificationTime + blockId ``` | **Component** | **Description** | | -------------------- | ------------------------------------------------------------------------------------------------------------------------- | | filename | The name of the data file. | | fileModificationTime | The last time when the data file was modified. | | blockId | The ID assigned to each block when a data file was split. This ID is unique within a single file but not globally unique. | ###### Cache Hit and Read Process[​](#cache-hit-and-read-process "Direct link to Cache Hit and Read Process") Assuming a query hits the block within the range `[1 MB, 2 MB)`, Block Cache proceeds as follows: 1. The system first checks whether the block exists in the local BE node's Block Cache (by matching the cache key). 2. If found (cache hit), the block is read directly from the local disk. 3. If not found (cache miss), the block is fetched from remote storage and synchronized to the local BE node's Block Cache for reuse in subsequent queries. ##### Cache medium[​](#cache-medium "Direct link to Cache medium") Block Cache uses the local disks of BE or CN nodes as its storage medium, and the cache acceleration effect is directly related to disk performance: * It is recommended to use high-performance local disks (for example, NVMe disks) to minimize cache read/write latency. * If disk performance is not optimal, you can increase the number of disks to achieve load balancing and reduce I/O pressure on individual disks. ##### Cache replacement policies[​](#cache-replacement-policies "Direct link to Cache replacement policies") Block Cache supports two data caching and eviction strategies: LRU and SLRU (Segmented LRU). ###### LRU[​](#lru "Direct link to LRU") The LRU strategy is based on the "Least Recently Used" principle - evicting the blocks that have not been accessed for the longest time. It is simple to implement, and suitable for scenarios with stable access patterns. ###### SLRU[​](#slru "Direct link to SLRU") The SLRU strategy divides the cache space into an eviction segment and a protection segment, both following the LRU rules: 1. Data enters the eviction segment on first access. 2. Data in the eviction segment is promoted to the protection segment when accessed again. 3. Data evicted from the protection segment falls back to the eviction segment, while data evicted from the eviction segment is directly removed from the cache. The SLRU strategy can effectively resist sudden sparse traffic, preventing "temporary data accessed only once" from directly evicting hot data in the protection segment. It offers better stability than LRU. #### Enable and configure Data Cache[​](#enable-and-configure-data-cache "Direct link to Enable and configure Data Cache") Data Cache is enabled by default, controlled by the BE configuration item `datacache_enable` (Default: `true`). Page Cache and Block Cache, as two independent components, are also enabled by default. Setting `datacache_enable` to `false` will disable Data Cache overall, that is, both Page Cache and Block Cache. You can also activate or deactivate Page Cache and Block Cache separately using different BE configuration items. * Page Cache (enabled by default) is controlled by `disable_storage_page_cache` (Default: `false`). * Block Cache (enabled by default) is controlled by `block_cache_enable` (Default: `true`). You can further use the following BE configurations to set the maximum memory and disk usage limits for Data Cache, preventing excessive resource occupation: * `datacache_mem_size`: Sets the maximum memory usage limit for Data Cache (used for storing data in Page Cache). * `datacache_disk_size`: Sets the maximum disk usage limit for Data Cache (used for storing data in Block Cache). #### Populate Block Cache[​](#populate-block-cache "Direct link to Populate Block Cache") ##### Population rules[​](#population-rules "Direct link to Population rules") From v3.3.2 onwards, in order to improve the cache hit rate of Block Cache, the system populates Block Cache according to the following rules: * The cache will not be populated for statements that are not `SELECT`, for example, `ANALYZE TABLE` and `INSERT INTO SELECT`. * Queries that scan all partitions of a table will not populate the cache. However, if the table has only one partition, population is performed by default. * Queries that scan all columns of a table will not populate the cache. However, if the table has only one column, population is performed by default. * The cache will not be populated for tables that are not Hive, Paimon, Delta Lake, Hudi, or Iceberg. ##### View Cache Population Behavior[​](#view-cache-population-behavior "Direct link to View Cache Population Behavior") You can view the population behavior for a specific query with the `EXPLAIN VERBOSE` command. Example: ```sql mysql> EXPLAIN VERBOSE SELECT col1 FROM hudi_table; ... | 0:HudiScanNode | | TABLE: hudi_table | | partitions=3/3 | | cardinality=9084 | | avgRowSize=2.0 | | dataCacheOptions={populate: false} | | cardinality: 9084 | +-----------------------------------------+ ``` `dataCacheOptions={populate: false}` indicates that the cache will not be populated because the query will scan all partitions. You can also fine-tune the population behavior of Block Cache via the session variable [`populate_datacache_mode`](https://docs.starrocks.io/docs/sql-reference/System_variable.md#populate_datacache_mode). ##### Population mode[​](#population-mode "Direct link to Population mode") Block Cache supports two modes, that is, synchronous population and asynchronous population. You can choose between them based on your business requirements for "first query performance" and "cache efficiency". **Synchronous Population** * Core Logic: When remote data is read for the first query, the data is immediately cached locally. Subsequent queries can directly reuse the cache. * Pros: High cache efficiency because data caching is completed with a single query. * Cons: Cache operations are executed synchronously with read operations, which may increase latency for the first query. **Asynchronous Population** * Core Logic: For the first query, data reading is prioritized and completed first. Cache writing is executed asynchronously in the background, without blocking the current query process. * Pros: It will not affect the performance of first query and prevent read operations from being delayed due to caching. * Cons: Lower cache efficiency because a single query may not fully cache all accessed data. It requires multiple queries to gradually improve cache coverage. From v3.3.0, asynchronous cache population is enabled by default. You can change the population mode by setting the session variable [`enable_datacache_async_populate_mode`](https://docs.starrocks.io/docs/sql-reference/System_variable.md). ##### Persistence[​](#persistence "Direct link to Persistence") The cached data in disks can be persistent by default, and these data can be reused after BE or CN restarts. #### Check whether a query hits data cache[​](#check-whether-a-query-hits-data-cache "Direct link to Check whether a query hits data cache") You can check whether a query hits Data Cache by analyzing the following metrics in the query profile: * `DataCacheReadBytes`: the size of data that the system reads directly from the memory and disks. * `DataCacheWriteBytes`: the size of data loaded from the remote storage system to the memory and disks. * `BytesRead`: the total amount of data that is read, including data that the system reads from the remote storage system, and its memory and disks. Example 1: In this example, the system reads a large amount of data (7.65 GB) from the remote storage system and only a few data (518.73 MB) from the disks. This means that few Block Caches were hit. ```plain - Table: lineorder - DataCacheReadBytes: 518.73 MB - __MAX_OF_DataCacheReadBytes: 4.73 MB - __MIN_OF_DataCacheReadBytes: 16.00 KB - DataCacheReadCounter: 684 - __MAX_OF_DataCacheReadCounter: 4 - __MIN_OF_DataCacheReadCounter: 0 - DataCacheReadTimer: 737.357us - DataCacheWriteBytes: 7.65 GB - __MAX_OF_DataCacheWriteBytes: 64.39 MB - __MIN_OF_DataCacheWriteBytes: 0.00 - DataCacheWriteCounter: 7.887K (7887) - __MAX_OF_DataCacheWriteCounter: 65 - __MIN_OF_DataCacheWriteCounter: 0 - DataCacheWriteTimer: 23.467ms - __MAX_OF_DataCacheWriteTimer: 62.280ms - __MIN_OF_DataCacheWriteTimer: 0ns - BufferUnplugCount: 15 - __MAX_OF_BufferUnplugCount: 2 - __MIN_OF_BufferUnplugCount: 0 - BytesRead: 7.65 GB - __MAX_OF_BytesRead: 64.39 MB - __MIN_OF_BytesRead: 0.00 ``` Example 2: In this example, the system reads a large amount of data (46.08 GB) from data cache and no data from the remote storage system, which means it reads data only from Block Cache. ```plain Table: lineitem - DataCacheReadBytes: 46.08 GB - __MAX_OF_DataCacheReadBytes: 194.99 MB - __MIN_OF_DataCacheReadBytes: 81.25 MB - DataCacheReadCounter: 72.237K (72237) - __MAX_OF_DataCacheReadCounter: 299 - __MIN_OF_DataCacheReadCounter: 118 - DataCacheReadTimer: 856.481ms - __MAX_OF_DataCacheReadTimer: 1s547ms - __MIN_OF_DataCacheReadTimer: 261.824ms - DataCacheWriteBytes: 0.00 - DataCacheWriteCounter: 0 - DataCacheWriteTimer: 0ns - BufferUnplugCount: 1.231K (1231) - __MAX_OF_BufferUnplugCount: 81 - __MIN_OF_BufferUnplugCount: 35 - BytesRead: 46.08 GB - __MAX_OF_BytesRead: 194.99 MB - __MIN_OF_BytesRead: 81.25 MB ``` #### I/O Adaptor[​](#io-adaptor "Direct link to I/O Adaptor") To prevent significant tail latency in disk access due to high cache disk I/O load, which can lead to negative optimization of the cache system, Data Cache provides the I/O Adaptor feature. This feature routes some cache requests to remote storage when disk load is high, utilizing both local cache and remote storage to improve I/O throughput. This feature is enabled by default. You can enable I/O Adaptor by setting the following system variable: ```sql SET GLOBAL enable_datacache_io_adaptor=true; ``` #### Dynamic scaling[​](#dynamic-scaling "Direct link to Dynamic scaling") Data Cache supports manual adjustment of cache capacity without restarting the BE process, and also supports automatic adjustment of cache capacity. ##### Manual scaling[​](#manual-scaling "Direct link to Manual scaling") You can modify Data Cache's memory limit or disk capacity by dynamically adjusting BE configuration items. Examples: ```sql -- Adjust the Data Cache memory limit for a specific BE instance. UPDATE be_configs SET VALUE="10G" WHERE NAME="datacache_mem_size" and BE_ID=10005; -- Adjust the Data Cache memory ratio limit for all BE instances. UPDATE be_configs SET VALUE="10%" WHERE NAME="datacache_mem_size"; -- Adjust the Data Cache disk limit for all BE instances. UPDATE be_configs SET VALUE="2T" WHERE NAME="datacache_disk_size"; ``` note * Be cautious when adjusting capacities in this way. Make sure not to omit the WHERE clause to avoid modifying irrelevant configuration items. * Cache capacity adjustments made this way will not be persisted and will be lost after the BE or CN process restarts. Therefore, you can first adjust the parameters dynamically as described above, and then manually modify the BE or CN configuration file to ensure that the changes take effect after the next restart. ##### Automatic Scaling[​](#automatic-scaling "Direct link to Automatic Scaling") StarRocks currently supports automatic scaling of disk capacity. If you do not specify the cache disk path and capacity limit in the BE configuration, automatic scaling is enabled by default. You can also enable automatic scaling by adding the following configuration item to the configuration file and restarting the BE or CN process: ```plain datacache_auto_adjust_enable=true ``` After automatic scaling is enabled: * When the disk usage exceeds the threshold specified by the BE configuration `disk_high_level` (default value is `90`, that is, 90% of disk space), the system will automatically evict cache data to free up disk space. * When the disk usage is consistently below the threshold specified by the BE configuration `disk_low_level` (default value is `60`, that is, 60% of disk space), and the current disk space used by Data Cache is full, the system will automatically expand the cache capacity. * When automatically scaling the cache capacity, the system will aim to adjust the cache capacity to the level specified by the BE configuration `disk_safe_level` (default value is `80`, that is, 80% of disk space). #### Cache Sharing[​](#cache-sharing "Direct link to Cache Sharing") Because Data Cache depends on the BE node's local disk, when the cluster are being scaled, changes in data routing can cause cache misses, which can easily lead to significant performance degradation during the elastic scaling. Cache Sharing is used to support accessing cache data between nodes through network. During cluster scaling, if a local cache miss occurs, the system first attempts to fetch cache data from other nodes within the same cluster. Only if all caches miss will the system re-fetch data from the remote storage. This feature effectively reduces the performance jitter caused by cache invalidation during elastic scaling and ensures stable query performance. ![cache sharing workflow](/assets/images/cache_sharing_workflow-4bc1221ee31928095ec36486bf6b03e1.png) You can enable the Cache Sharing feature by configuring the following two items: * Set the FE configuration item `enable_trace_historical_node` to `true`. * Set the system variable `enable_datacache_sharing` to `true`. In addition, you can check the following metrics in query profile to monitor Cache Sharing: * `DataCacheReadPeerCounter`: The read count from other nodes. * `DataCacheReadPeerBytes`: The bytes read from other nodes. * `DataCacheReadPeerTimer`: The time used for accessing cache data from other nodes. #### Configurations and variables[​](#configurations-and-variables "Direct link to Configurations and variables") You can configure Data Cache using the following system variables and parameters. ##### System variables[​](#system-variables "Direct link to System variables") * [populate\_datacache\_mode](https://docs.starrocks.io/docs/sql-reference/System_variable.md#populate_datacache_mode) * [enable\_datacache\_io\_adaptor](https://docs.starrocks.io/docs/sql-reference/System_variable.md#enable_datacache_io_adaptor) * [enable\_file\_metacache](https://docs.starrocks.io/docs/sql-reference/System_variable.md#enable_file_metacache) * [enable\_datacache\_async\_populate\_mode](https://docs.starrocks.io/docs/sql-reference/System_variable.md) * [enable\_datacache\_sharing](https://docs.starrocks.io/docs/sql-reference/System_variable.md#enable_datacache_sharing) ##### FE configurations[​](#fe-configurations "Direct link to FE configurations") * [enable\_trace\_historical\_node](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#enable_trace_historical_node) ##### BE configurations[​](#be-configurations "Direct link to BE configurations") * [datacache\_enable](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#datacache_enable) * [datacache\_mem\_size](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#datacache_mem_size) * [datacache\_disk\_size](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#datacache_disk_size) * [datacache\_auto\_adjust\_enable](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#datacache_auto_adjust_enable) * [datacache\_disk\_high\_level](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#datacache_disk_high_level) * [datacache\_disk\_safe\_level](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#datacache_disk_safe_level) * [datacache\_disk\_low\_level](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#datacache_disk_low_level) * [datacache\_disk\_adjust\_interval\_seconds](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#datacache_disk_adjust_interval_seconds) * [datacache\_disk\_idle\_seconds\_for\_expansion](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#datacache_disk_idle_seconds_for_expansion) * [datacache\_min\_disk\_quota\_for\_adjustment](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#datacache_min_disk_quota_for_adjustment) * [datacache\_eviction\_policy](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#datacache_eviction_policy) * [datacache\_inline\_item\_count\_limit](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#datacache_inline_item_count_limit) --- ### Block cache warmup Some data lake analytics and shared-data cluster scenarios have high performance requirements for queries, such as BI reports and proof of concept (PoC) performance testing. Loading remote data into local block cache can avoid the need to fetch the same data multiple times, significantly speeding up query execution and minimizing resource usage. StarRocks v3.3 introduces the Block Cache Warmup feature, which is an enhancement to [Block Cache](https://docs.starrocks.io/docs/data_source/data_cache.md#principles-of-block-cache). Block Cache is a process of passively populating the cache, in which data is written to the cache during data querying. Block Cache Warmup, however, is an active process of populating the cache. It proactively fetches the desired data from remote storage in advance. #### Scenarios[​](#scenarios "Direct link to Scenarios") * The disk used for block cache has a storage capacity much larger than the amount of data to warm up. If the disk capacity is less than the data to warm up, the expected warmup effect cannot be achieved. For example, if 100 GB data needs to be warmed up but the disk has only 50 GB of space, then only 50 GB data can be loaded to the cache and the previously loaded 50 GB data will be replaced by the 50 GB data that is loaded later. * Data access on the disk used for block cache is relatively stable. If there is a surge in the access volume, the expected warmup effect cannot be achieved. For example, if 100 GB data needs to be warmed up and the disk has 200 GB of space, then the first condition is met. However, if a large amount of new data (150 GB) is written to the cache during the warmup process, or if an unexpected large cold query needs to load 150 GB data to the cache, it may result in the eviction of the warmed data. #### How it works[​](#how-it-works "Direct link to How it works") StarRocks provides the CACHE SELECT syntax to implement Block Cache Warmup. Before using CACHE SELECT, make sure that the Block Cache feature has been enabled. Syntax of CACHE SELECT: ```sql CACHE SELECT [, ...] FROM [.][.] [WHERE ] [PROPERTIES("verbose"="true")] ``` Parameters: * `column_name`: The columns to fetch. You can use `*` to fetch all columns in the external table. * `catalog_name`: The name of the catalog, default is DEFAULT\_CATALOG. If you have switched to the catalog using SET CATALOG, it can be left unspecified. * `db_name`: The name of the database. If you have switched to that database, it can be left unspecified. * `table_name`: The name of the table from which to fetch data. * `boolean_expression`: The filter condition. * `PROPERTIES`: Currently, only the `verbose` property is supported. It is used to return detailed warmup metrics. CACHE SELECT is a synchronous process and it can warm up only one table at a time. Upon successful execution, it will return warmup-related metrics. ##### Warm up all data in an external table[​](#warm-up-all-data-in-an-external-table "Direct link to Warm up all data in an external table") The following example loads all data from external table `lineitem`: ```plaintext mysql> cache select * from hive_catalog.test_db.lineitem; +-----------------+------------------+----------------------+-------------------+ | READ_CACHE_SIZE | WRITE_CACHE_SIZE | AVG_WRITE_CACHE_TIME | TOTAL_CACHE_USAGE | +-----------------+------------------+----------------------+-------------------+ | 48.2MB | 3.7GB | 59ms | 96.83% | +-----------------+------------------+----------------------+-------------------+ 1 row in set (19.56 sec) ``` Return fields: * `READ_CACHE_SIZE`: The total size of data read from the block cache by all nodes. * `WRITE_CACHE_SIZE`: The total size of data written to the block cache by all nodes. * `AVG_WRITE_CACHE_TIME`: The average time taken by each node to write data to the block cache. * `TOTAL_CACHE_USAGE`: The disk space usage of the block cache of the entire cluster after this warmup task is complete. This metric can be used to assess whether the block cache has sufficient space. ##### Warm up specified columns with filter conditions[​](#warm-up-specified-columns-with-filter-conditions "Direct link to Warm up specified columns with filter conditions") You can specify columns and predicates to achieve fine-grained warmup, which helps reduce the amount to data to warm up, reducing disk I/O and CPU consumption. ```plaintext mysql> cache select l_orderkey from hive_catalog.test_db.lineitem where l_shipdate='1994-10-28'; +-----------------+------------------+----------------------+-------------------+ | READ_CACHE_SIZE | WRITE_CACHE_SIZE | AVG_WRITE_CACHE_TIME | TOTAL_CACHE_USAGE | +-----------------+------------------+----------------------+-------------------+ | 957MB | 713.5MB | 3.6ms | 97.33% | +-----------------+------------------+----------------------+-------------------+ 1 row in set (9.07 sec) ``` The following example prefetches a specific column from a cloud-native table `lineorder` in a shared-data cluster: ```plaintext mysql> cache select lo_orderkey from ssb.lineorder; +-----------------+------------------+----------------------+-------------------+ | READ_CACHE_SIZE | WRITE_CACHE_SIZE | AVG_WRITE_CACHE_TIME | TOTAL_CACHE_USAGE | +-----------------+------------------+----------------------+-------------------+ | 118MB | 558.9MB | 200.6ms | 4.66% | +-----------------+------------------+----------------------+-------------------+ 1 row in set (29.88 sec) ``` ##### Warm up in verbose mode[​](#warm-up-in-verbose-mode "Direct link to Warm up in verbose mode") By default, the metrics returned by `CACHE SELECT` are metrics combined on multiple BEs. You can append `PROPERTIES("verbose"="true")` at the end of CACHE SELECT to obtain detailed metrics of each BE. ```plaintext mysql> cache select * from hive_catalog.test_db.lineitem properties("verbose"="true"); +---------------+-----------------+---------------------+------------------+----------------------+-------------------+ | IP | READ_CACHE_SIZE | AVG_READ_CACHE_TIME | WRITE_CACHE_SIZE | AVG_WRITE_CACHE_TIME | TOTAL_CACHE_USAGE | +---------------+-----------------+---------------------+------------------+----------------------+-------------------+ | 172.26.80.233 | 376MB | 127.8micros | 0B | 0s | 3.85% | | 172.26.80.231 | 272.5MB | 121.8micros | 20.7MB | 146.5micros | 3.91% | | 172.26.80.232 | 355.5MB | 147.7micros | 0B | 0s | 3.91% | +---------------+-----------------+---------------------+------------------+----------------------+-------------------+ 3 rows in set (0.54 sec) ``` In verbose mode, an extra metric will be returned: * `AVG_READ_CACHE_TIME`: the average time for each node to read data when block cache is hit. #### Periodic scheduling of CACHE SELECT tasks[​](#periodic-scheduling-of-cache-select-tasks "Direct link to Periodic scheduling of CACHE SELECT tasks") You can use CACHE SELECT with [SUBMIT TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md) to achieve periodic warmup. For example, the following case warms up the `lineitem` table every 5 minutes: ```plaintext mysql> submit task always_cache schedule every(interval 5 minute) as cache select l_orderkey from hive_catalog.test_db.lineitem where l_shipdate='1994-10-28'; +--------------+-----------+ | TaskName | Status | +--------------+-----------+ | always_cache | SUBMITTED | +--------------+-----------+ 1 row in set (0.03 sec) ``` ##### Manage CACHE SELECT tasks[​](#manage-cache-select-tasks "Direct link to Manage CACHE SELECT tasks") ###### View created tasks[​](#view-created-tasks "Direct link to View created tasks") ```plaintext mysql> select * from default_catalog.information_schema.tasks; +--------------+---------------------+-----------------------------------------------------+---------------+------------------------------+---------------------------------------------------------------------+---------------------+------------+ | TASK_NAME | CREATE_TIME | SCHEDULE | CATALOG | DATABASE | DEFINITION | EXPIRE_TIME | PROPERTIES | +--------------+---------------------+-----------------------------------------------------+---------------+------------------------------+---------------------------------------------------------------------+---------------------+------------+ | always_cache | 2024-04-11 16:01:00 | PERIODICAL START(2024-04-11T16:01) EVERY(5 MINUTES) | emr_hive_test | zz_tpch_sf1000_hive_orc_zlib | cache select l_orderkey from lineitem where l_shipdate='1994-10-28' | NULL | | +--------------+---------------------+-----------------------------------------------------+---------------+------------------------------+---------------------------------------------------------------------+---------------------+------------+ 1 row in set (0.21 sec) ``` ###### View task execution history[​](#view-task-execution-history "Direct link to View task execution history") ```plaintext mysql> select * from default_catalog.information_schema.task_runs; +--------------------------------------+--------------+---------------------+---------------------+---------+---------------+------------------------------+---------------------------------------------------------------------+---------------------+------------+---------------+----------+------------------------------------------------------------------------------------------------------------------------+------------+ | QUERY_ID | TASK_NAME | CREATE_TIME | FINISH_TIME | STATE | CATALOG | DATABASE | DEFINITION | EXPIRE_TIME | ERROR_CODE | ERROR_MESSAGE | PROGRESS | EXTRA_MESSAGE | PROPERTIES | +--------------------------------------+--------------+---------------------+---------------------+---------+---------------+------------------------------+---------------------------------------------------------------------+---------------------+------------+---------------+----------+------------------------------------------------------------------------------------------------------------------------+------------+ | 55b30204-f7da-11ee-b03e-7ea526d0b618 | always_cache | 2024-04-11 16:06:00 | 2024-04-11 16:07:22 | SUCCESS | emr_hive_test | zz_tpch_sf1000_hive_orc_zlib | cache select l_orderkey from lineitem where l_shipdate='1994-10-28' | 2024-04-12 16:06:00 | 0 | NULL | 100% | AlreadyCachedSize: 15.7GB, AvgReadCacheTime: 1ms, WriteCacheSize: 0B, AvgWriteCacheTime: 0s, TotalCacheUsage: 75.94% | | | a2e3dc7e-f7d9-11ee-b03e-7ea526d0b618 | always_cache | 2024-04-11 16:01:00 | 2024-04-11 16:02:39 | SUCCESS | emr_hive_test | zz_tpch_sf1000_hive_orc_zlib | cache select l_orderkey from lineitem where l_shipdate='1994-10-28' | 2024-04-12 16:01:00 | 0 | NULL | 100% | AlreadyCachedSize: 15.7GB, AvgReadCacheTime: 1.2ms, WriteCacheSize: 0B, AvgWriteCacheTime: 0s, TotalCacheUsage: 75.87% | | +--------------------------------------+--------------+---------------------+---------------------+---------+---------------+------------------------------+---------------------------------------------------------------------+---------------------+------------+---------------+----------+------------------------------------------------------------------------------------------------------------------------+------------+ 2 rows in set (0.04 sec) ``` The `EXTRA_MESSAGE` field records metrics of CACHE SELECT. ###### Drop tasks[​](#drop-tasks "Direct link to Drop tasks") ```sql DROP TASK ``` #### Use cases[​](#use-cases "Direct link to Use cases") 1. During PoC performance testing, if you want to assess StarRocks' performance without interference from external storage systems, you can use the CACHE SELECT statement to load the data of the table to test into the block cache in advance. 2. The business team need to view BI reports at 8 a.m. every morning. To ensure a relatively stable query performance, you can schedule a CACHE SELECT task to start running at 7 a.m. each day. ```sql mysql> submit task BI schedule START('2024-02-03 07:00:00') EVERY(interval 1 day) AS cache select * from hive_catalog.test_db.lineitem where l_shipdate='1994-10-28'; +--------------+-----------+ | TaskName | Status | +--------------+-----------+ | BI | SUBMITTED | +--------------+-----------+ 1 row in set (0.03 sec) ``` 3. To minimize system resource consumption during warmup, you can specify session variables in the SUBMIT TASK statement. For example, you can designate a resource group for the CACHE SELECT task, adjust the Degree of Parallelism (DOP), and specify the filter condition in WHERE to reduce the impact of warmup on regular queries. ```sql mysql> submit task cache_select properties("pipeline_dop"="1", "resource_group"="warmup") schedule EVERY(interval 1 day) AS cache select * from hive_catalog.test_db.lineitem where l_shipdate>='1994-10-28'; +--------------+-----------+ | TaskName | Status | +--------------+-----------+ | cache_select | SUBMITTED | +--------------+-----------+ 1 row in set (0.03 sec) ``` #### Limits and usage notes[​](#limits-and-usage-notes "Direct link to Limits and usage notes") * To use CACHE SELECT, you must first enable the Block Cache feature and have the SELECT privilege on the destination table. * CACHE SELECT supports warming up only a single table and does not support operators like ORDER BY, LIMIT, or GROUP BY. * CACHE SELECT can be used in both shared-nothing and shared-data clusters. * CACHE SELECT can warm up remote TEXT, ORC, Parquet files. * The data warmed up by CACHE SELECT may not be retained in cache forever. The cached data may still be evicted based on the SLRU rule of the Block Cache feature. * If you are a data lake user, you can check the remaining capacity of the block cache by using `SHOW BACKENDS\G` or `SHOW COMPUTE NODES\G` to assess whether SLRU eviction may occur. * If you are a shared-data cluster user, you can check the block cache usage by viewing the metrics of the shared-data cluster. * Currently, the implementation of CACHE SELECT uses the INSERT INTO BLACKHOLE() approach, which warms up the table following the normal query process. Therefore, the performance overhead of CACHE SELECT is similar to that of regular queries. Improvements will be made in the future to enhance the performance. #### What to expect in later versions[​](#what-to-expect-in-later-versions "Direct link to What to expect in later versions") In the future, StarRocks will introduce adaptive Block Cache Warmup to ensure a higher cache hit rate. --- ### Data Cache observability In earlier versions, there are no rich metrics or efficient methods to monitor the performance, usage, and health of [Data Cache](https://docs.starrocks.io/docs/data_source/data_cache.md). In v3.3, StarRocks improves the observability of Data Cache by offering efficient monitoring methods and more metrics. Users can now check the overall disk and memory usage of the data cache, and related metrics, enhancing monitoring of cache usage. > **NOTE** > > From v3.4.0 onwards, queries against external catalogs and cloud-native tables (in shared-data clusters) use a unified Data Cache instance. Therefore, unless otherwise specified, the following methods default to displaying the metrics of the Data Cache instance itself, which includes the cache usage of queries against both external catalogs and cloud-native tables. #### SQL commands[​](#sql-commands "Direct link to SQL commands") You can run SQL commands to view the capacity and usage of Data Cache on each BE node. ##### SHOW BACKENDS[​](#show-backends "Direct link to SHOW BACKENDS") The `DataCacheMetrics` field records the used disk and memory space of Data Cache on a specific BE. ```sql mysql> show backends\G *************************** 1. row *************************** BackendId: 10004 IP: XXX.XX.XX.XXX HeartbeatPort: 4450 BePort: 4448 HttpPort: 4449 BrpcPort: 4451 LastStartTime: 2023-12-13 20:09:30 LastHeartbeat: 2023-12-13 20:10:43 Alive: true SystemDecommissioned: false ClusterDecommissioned: false TabletNum: 48 DataUsedCapacity: 0.000 B AvailCapacity: 280.103 GB TotalCapacity: 1.968 TB UsedPct: 86.10 % MaxDiskUsedPct: 86.10 % ErrMsg: Version: datacache-heartbeat-c68caf7 Status: {"lastSuccessReportTabletsTime":"2023-12-13 20:10:38"} DataTotalCapacity: 280.103 GB DataUsedPct: 0.00 % CpuCores: 104 NumRunningQueries: 0 MemUsedPct: 0.00 % CpuUsedPct: 0.0 % DataCacheMetrics: Status: Normal, DiskUsage: 0.00GB/2.00GB, MemUsage: 0.00GB/30.46GB 1 row in set (1.90 sec) ``` ##### information\_schema[​](#information_schema "Direct link to information_schema") The `be_datacache_metrics` view in `information_schema` records the following Data Cache-related information. ```bash mysql> select * from information_schema.be_datacache_metrics; +-------+--------+------------------+-----------------+-----------------+----------------+-----------------+----------------------------------------------------------------------------------------------+ | BE_ID | STATUS | DISK_QUOTA_BYTES | DISK_USED_BYTES | MEM_QUOTA_BYTES | MEM_USED_BYTES | META_USED_BYTES | DIR_SPACES | +-------+--------+------------------+-----------------+-----------------+----------------+-----------------+----------------------------------------------------------------------------------------------+ | 10004 | Normal | 2147483648 | 0 | 32706263420 | 0 | 0 | [{"Path":"/home/disk1/datacache","QuotaBytes":2147483648}] | +-------+--------+------------------+-----------------+-----------------+----------------+-----------------+----------------------------------------------------------------------------------------------+ 1 row in set (5.41 sec) ``` * `BE_ID`: the BE ID * `STATUS`: the BE status * `DISK_QUOTA_BYTES`: the disk cache capacity configured by users, in bytes * `DISK_USED_BYTES`: the disk cache space that has been used, in bytes * `MEM_QUOTA_BYTES`: the memory cache capacity configured by users, in bytes * `MEM_USED_BYTES`: the memory cache space that has been used, in bytes * `META_USED_BYTES`: the space used to cache metadata * `DIR_SPACES`: the cache path and its cache size #### API call[​](#api-call "Direct link to API call") Since v3.3.2, StarRocks provides two APIs to get cache metrics, which reflect the cache state at different levels: * `/api/datacache/app_stat`: Query the Block Cache and Page Cache hit rates. * `/api/datacache/stat`: the underlying execution state of Data Cache. This interface is mainly used for maintenance and bottleneck identification of Data Cache. It does not reflect the actual hit rate of the query. Common users do not need to pay attention to this interface. ##### View cache hit metrics[​](#view-cache-hit-metrics "Direct link to View cache hit metrics") View the cache hit metrics by accessing the following API interface: ```bash http://${BE_HOST}:${BE_HTTP_PORT}/api/datacache/app_stat ``` Return: ```bash { "block_cache_hit_bytes": 1642106883, "block_cache_miss_bytes": 8531219739, "block_cache_hit_rate": 0.16, "block_cache_hit_bytes_last_minute": 899037056, "block_cache_miss_bytes_last_minute": 4163253265, "block_cache_hit_rate_last_minute": 0.18, "page_cache_hit_count": 15048, "page_cache_miss_count": 10032, "page_cache_hit_rate": 0.6, "page_cache_hit_count_last_minute": 10032, "page_cache_miss_count_last_minute": 5016, "page_cache_hit_rate_last_minute": 0.67 } ``` | **Metric** | **Description** | | --------------------------------------- | --------------------------------------------------------------------------------------------------- | | block\_cache\_hit\_bytes | Bytes read from the block cache. | | block\_cache\_miss\_bytes | Bytes read from remote storage (Block Cache misses). | | block\_cache\_hit\_rate | Block Cache hit rate, `(block_cache_hit_bytes / (block_cache_hit_bytes + block_cache_miss_bytes))`. | | block\_cache\_hit\_bytes\_last\_minute | Bytes read from the Block Cache in the last minute. | | block\_cache\_miss\_bytes\_last\_minute | Bytes read from the remote storage in the last minute. | | block\_cache\_hit\_rate\_last\_minute | Block Cache hit rate in the last minute. | | page\_cache\_hit\_count | Number of pages read from the Page Cache. | | page\_cache\_miss\_count | Number of pages missed in the Page Cache. | | block\_cache\_hit\_rate | Page Cache hit rate: `(page_cache_hit_count / (page_cache_hit_count + page_cache_miss_count))`. | | page\_cache\_hit\_count\_last\_minute | Number of pages read from the Page Cache in the last minute. | | page\_cache\_miss\_count\_last\_minute | Number of pages missed in the Page Cache in the last minute. | | page\_cache\_hit\_rate\_last\_minute | Page Cache hit rate in the last minute. | ##### View the underlying execution state of Data Cache[​](#view-the-underlying-execution-state-of-data-cache "Direct link to View the underlying execution state of Data Cache") You can get more detailed metrics on the Data Cache by accessing the following API interfaces. ```bash http://${BE_HOST}:${BE_HTTP_PORT}/api/datacache/stat ``` The results are as follows: ```json { "page_cache_mem_quota_bytes": 10679976935, "page_cache_mem_used_bytes": 10663052377, "page_cache_mem_used_rate": 1.0, "page_cache_hit_count": 276890, "page_cache_miss_count": 153126, "page_cache_hit_rate": 0.64, "page_cache_hit_count_last_minute": 11196, "page_cache_miss_count_last_minute": 9982, "page_cache_hit_rate_last_minute": 0.53, "block_cache_status": "NORMAL", "block_cache_disk_quota_bytes": 214748364800, "block_cache_disk_used_bytes": 11371020288, "block_cache_disk_used_rate": 0.05, "block_cache_disk_spaces": "/disk1/sr/be/storage/datacache:107374182400;/disk2/sr/be/storage/datacache:107374182400", "block_cache_meta_used_bytes": 11756727, "block_cache_hit_count": 57707, "block_cache_miss_count": 2556, "block_cache_hit_rate": 0.96, "block_cache_hit_bytes": 15126253744, "block_cache_miss_bytes": 620687633, "block_cache_hit_count_last_minute": 18108, "block_cache_miss_count_last_minute": 2449, "block_cache_hit_bytes_last_minute": 4745613488, "block_cache_miss_bytes_last_minute": 607536783, "block_cache_read_disk_bytes": 15126253744, "block_cache_write_bytes": 11338218093, "block_cache_write_success_count": 43377, "block_cache_write_fail_count": 36394, "block_cache_remove_bytes": 0, "block_cache_remove_success_count": 0, "block_cache_remove_fail_count": 0, "block_cache_current_reading_count": 0, "block_cache_current_writing_count": 0, "block_cache_current_removing_count": 0 } ``` ##### Metric description[​](#metric-description "Direct link to Metric description") | **Metric** | **Description** | | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | page\_cache\_mem\_quota\_bytes | Current memory limit of Page Cache. | | page\_cache\_mem\_used\_bytes | Current actual memory used by Page Cache. | | page\_cache\_mem\_used\_rate | Current memory usage rate of Page Cache. | | page\_cache\_hit\_count | Number of Page Cache hits. | | page\_cache\_miss\_count | Number of Page Cache misses. | | page\_cache\_hit\_rate | Hit rate of Page Cache. | | page\_cache\_hit\_count\_last\_minute | Number of Page Cache hits in the last minute. | | page\_cache\_miss\_count\_last\_minute | Number of Page Cache misses in the last minute. | | page\_cache\_hit\_rate\_last\_minute | Hit rate of Page Cache in the last minute. | | block\_cache\_status | Status of the Block Cache, including:`NORMAL`: The instance runs normally.`ABNORMAL`: Data cannot be read or written into the cache. The issue must be located using logs.`UPDATING`: The instance is being updated, such as the updating state during online scaling. | | block\_cache\_disk\_quota\_bytes | The disk cache capacity of Block Cache configured by users, in bytes. | | block\_cache\_disk\_used\_bytes | The disk cache space that has been used by Block Cache, in bytes. | | block\_cache\_disk\_used\_rate | The actual disk cache usage rate of Block Cache, in percentages. | | block\_cache\_disk\_spaces | The disk cache information of Block Cache configured by users, including each cache path and the cache size. | | block\_cache\_meta\_used\_bytes | The memory space used to cache Block Cache metadata, in bytes. | | block\_cache\_hit\_count | Number of Block Cache hits. | | block\_cache\_miss\_count | Number of cache misses. | | block\_cache\_hit\_rate | Hit rate of Block Cache. | | block\_cache\_hit\_bytes | Number of bytes that are hit in the Block Cache. | | block\_cache\_miss\_bytes | Number of bytes that are missed in the Block Cache. | | block\_cache\_hit\_count\_last\_minute | Number of Block Cache hits in the last minute. | | block\_cache\_miss\_count\_last\_minute | Number of Block Cache misses in the last minute. | | block\_cache\_hit\_bytes\_last\_minute | Number of types hit by Block Cache in the last minute. | | block\_cache\_miss\_bytes\_last\_minute | Number of types missed by Block Cache in the last minute. | | block\_cache\_buffer\_item\_count | The current number of Buffer instances in the Block Cache. Buffer instances refer to common data caches, such as when reading part of the raw data from a remote file and caching the data directly in memory or on disks. | | block\_cache\_buffer\_item\_bytes | Number of types used to cache the Buffer instance in Block Cache. | | block\_cache\_read\_disk\_bytes | Number of bytes read from Block Cache. | | block\_cache\_write\_bytes | Number of bytes written to Block Cache. | | block\_cache\_write\_success\_count | Number of successful Block Cache writes. | | block\_cache\_write\_fail\_count | Number of failed Block Cache writes. | | block\_cache\_remove\_bytes | Number of bytes removed from Block Cache. | | block\_cache\_remove\_success\_count | Number of successful remove operations from Block Cache. | | block\_cache\_remove\_fail\_count | Number of failed remove operations from Block Cache. | | block\_cache\_current\_reading\_count | Number of read operations currently being executed in Block Cache. | | block\_cache\_current\_writing\_count | Number of write operations currently being executed in Block Cache. | | block\_cache\_current\_removing\_count | Number of remove operations currently being executed in Block Cache. | --- ### Data Cache FAQ This topic describes some frequently asked questions (FAQ) and common issues about Data Cache and provides troubleshooting steps and solutions to these issues. #### Enabling Data Cache[​](#enabling-data-cache "Direct link to Enabling Data Cache") ##### How to confirm whether Data Cache is successfully enabled?[​](#how-to-confirm-whether-data-cache-is-successfully-enabled "Direct link to How to confirm whether Data Cache is successfully enabled?") In most cases, you can check whether Data Cache is successfully enabled by any of the following methods: * Execute `SHOW BACKENDS` (or `SHOW COMPUTE NODES`) from your SQL client, and check the value of `DataCacheMetrics`. You can confirm that Data Cache is enabled if the disk or memory cache quota is greater than 0. ```sql mysql> show backends \G *************************** 1. row *************************** BackendId: 89041 IP: X.X.X.X HeartbeatPort: 9050 BePort: 9060 HttpPort: 8040 BrpcPort: 8060 LastStartTime: 2025-05-29 14:45:37 LastHeartbeat: 2025-05-29 19:20:32 Alive: true SystemDecommissioned: false ClusterDecommissioned: false TabletNum: 10 DataUsedCapacity: 0.000 B AvailCapacity: 1.438 TB TotalCapacity: 1.718 TB UsedPct: 16.27 % MaxDiskUsedPct: 16.27 % ErrMsg: Version: main-c15b412 Status: {"lastSuccessReportTabletsTime":"2025-05-29 19:20:30"} DataTotalCapacity: 1.438 TB DataUsedPct: 0.00 % CpuCores: 8 MemLimit: 50.559GB NumRunningQueries: 0 MemUsedPct: 0.50 % CpuUsedPct: 0.2 % DataCacheMetrics: Status: Normal, DiskUsage: 44MB/1TB, MemUsage: 0B/0B Location: StatusCode: OK 1 row in set (0.00 sec) ``` In the above example, the disk cache quota of Data Cache is 1TB, and 44MB is currently in use; while the memory cache quota is 0B, thus memory cache is not enabled. * You can access the BE Web Console (`http://${BE_HOST}:${BE_HTTP_PORT}/api/datacache/stat`) to check the current Data Cache quota, hit rate and other metrics. You can confirm that Data Cache is enbaled if `disk_quota_bytes` or `mem_quota_bytes` is greater than 0. ![Data Cache FAQ - Web Console](/assets/images/data_cache_be_web_console-c0bf542a74f6593c8b61602da56d645d.png) ##### Why is Data Cache not enabled by default?[​](#why-is-data-cache-not-enabled-by-default "Direct link to Why is Data Cache not enabled by default?") From v3.3 onwards, BE will attempt to enable Data Cache upon startup. However, if there is insufficient space available on the current disk, Data Cache will not be enabled automatically. It may be caused by the following situations: * **Percentage**: The current disk usage is high. * **Remaining space**: The remaining disk space size is relatively low. Therefore, if Data Cache is not enabled by default, you can first check the current disk usage and increase disk capacity if necessary. Alternatively, you can manually enable Data Cache by configuring cache quota based on the current available disk space. ```text # disable Data Cache Automatic Adjustment datacache_auto_adjust_enable = false # Set Data Cache disk quota manually datacache_disk_size = 1T ``` #### Using Data Cache[​](#using-data-cache "Direct link to Using Data Cache") ##### What catalog types does Data Cache support?[​](#what-catalog-types-does-data-cache-support "Direct link to What catalog types does Data Cache support?") Data Cache currently supports external catalog types that use StarRocks Native File Reader (such as Parquet/ORC/CSV Reader), including Hive, Iceberg, Hudi, Delta Lake, and Paimon. Catalogs that access data based on JNI (such as JDBC Catalog) are not supported yet. note Some catalogs may use different data access methods based on certain conditions (such as file type, and data status). For example, for the Paimon catalog, StarRocks may automatically choose whether to use Native File Reader or JNI to access data based on the compaction status of the current data. When JNI is used to access Paimon data, Data Cache acceleration is not supported. ##### How can I know that a query hits the cache?[​](#how-can-i-know-that-a-query-hits-the-cache "Direct link to How can I know that a query hits the cache?") You can check Data Cache-related metrics in the corresponding query profile. The metrics `DataCacheReadBytes` and `DataCacheReadCounter` indicates the local cache hit status. ```text - DataCacheReadBytes: 518.73 MB - __MAX_OF_DataCacheReadBytes: 4.73 MB - __MIN_OF_DataCacheReadBytes: 16.00 KB - DataCacheReadCounter: 684 - __MAX_OF_DataCacheReadCounter: 4 - __MIN_OF_DataCacheReadCounter: 0 - DataCacheReadTimer: 737.357us - DataCacheWriteBytes: 7.65 GB - __MAX_OF_DataCacheWriteBytes: 64.39 MB - __MIN_OF_DataCacheWriteBytes: 0.00 - DataCacheWriteCounter: 7.887K (7887) - __MAX_OF_DataCacheWriteCounter: 65 - __MIN_OF_DataCacheWriteCounter: 0 - DataCacheWriteTimer: 23.467ms - __MAX_OF_DataCacheWriteTimer: 62.280ms - __MIN_OF_DataCacheWriteTimer: 0ns ``` ##### Why does a query miss the cache when Data Cache is enabled?[​](#why-does-a-query-miss-the-cache-when-data-cache-is-enabled "Direct link to Why does a query miss the cache when Data Cache is enabled?") Follow these steps for troubleshooting: 1. Check whether Data Cache support the current catalog type. 2. Confirm whether the query statement meets the cache population conditions. In certain cases, Data Cache will reject cache population for some queries. For details, see [Data Cache Population Rules](https://docs.starrocks.io/docs/data_source/data_cache.md#population-rules). The `EXPLAIN VERBOSE` command can be used to check whether a query triggers cache population. Example: ```sql mysql> EXPLAIN VERBOSE SELECT col1 FROM hudi_table; | 0:HudiScanNode | | TABLE: hudi_table | | partitions=3/3 | | cardinality=9084 | | avgRowSize=2.0 | | dataCacheOptions={populate: false} | | cardinality: 9084 | +-----------------------------------------+ ``` In the above example, the `populate` field of the `dataCacheOptions` section is `false`, indicating the cache will not be populated for the query. To enable Data Cache for such queries, you can modify the default population behavior by setting the system variable `populate_datacache_mode` to `always`. #### Data Cache Hit[​](#data-cache-hit "Direct link to Data Cache Hit") ##### Why sometimes the same query needs to be executed multiple times before fully hitting the cache?[​](#why-sometimes-the-same-query-needs-to-be-executed-multiple-times-before-fully-hitting-the-cache "Direct link to Why sometimes the same query needs to be executed multiple times before fully hitting the cache?") In the current version, Data Cache uses asynchronous population by default to reduce its impact on query performance. When using asynchronous population, the system will try to cache the accessed data in the background without affecting the read performance as much as possible. Therefore, executing a query for a single time can only cache a part of the data. You need to run the query for multiple times to cache all the data required by the query. You can also use the synchronous cache population by setting `enable_datacache_async_populate_mode=false`, or warm up the target data in advance by `CACHE SELECT`. ##### Why is it that all the data in the current query has been cached, but there is still a few data accessed remotely?[​](#why-is-it-that-all-the-data-in-the-current-query-has-been-cached-but-there-is-still-a-few-data-accessed-remotely "Direct link to Why is it that all the data in the current query has been cached, but there is still a few data accessed remotely?") In the current version, I/O adaptation is enabled by default to optimize cache performance when disk I/O load is high, which may result in a small number of requests directly accessing the remote storage in some cases. You can disable the I/O adaptation function by setting `enable_datacache_io-adapter` to `false`. #### Others[​](#others "Direct link to Others") ##### How to clear the cached data?[​](#how-to-clear-the-cached-data "Direct link to How to clear the cached data?") Currently, Data Cache does not provide a direct interface to clear the cached data, but you can choose one of the following methods to clear it: * You can clean up cached data by deleting all data (including block files and meta directories) in the `datacache` directory on the BE/CN nodes, and then restarting the nodes. (Recommended) * If you want to avoid restarting the BE/CN nodes, you can also clean cached data indirectly by scaling down the cache quota during runtime. For example, if you have set the disk cache quota to 2TB earlier, you can first scale it down to 0 (the system will automatically clean up the cached data), and then reset it to 2TB. Example: ```sql UPDATE be_configs SET VALUE="0" WHERE NAME="datacache_disk_size" and BE_ID=10005; UPDATE be_configs SET VALUE="2T" WHERE NAME="datacache_disk_size" and BE_ID=10005; ``` note When cleaning cached data during runtime, be cautious with the `WHERE` condition in the statement to avoid accidentally damaging other irrelevant parameters or nodes. ##### How to improve Data Cache performance?[​](#how-to-improve-data-cache-performance "Direct link to How to improve Data Cache performance?") With Data Cache, StarRocks essentially accesses local memory or disk instead of the remote storage. Therefore, the performance is directly related to the local cache medium. If you find that cache access latency is high due to high disk load, you may consider improving the performance of the local cache medium: * Prioritize using high-performance NVME disks as cache disks. * If high-performance disks are not available, you can also increase the number of disks to share the I/O pressure. * Increase the server memory of the BE/CN nodes (rather than the Data Cache memory quota), using the Page Cache of the operating system to reduce the number of direct disk accesses and the disk I/O pressure. --- ### Data lake FAQ This topic describes some commonly asked questions (FAQ) about data lake and provides solutions to these issues. Some metrics mentioned in this topic can be obtained only from the profiles of the SQL queries. To obtain the profiles of SQL queries, you must specify `set enable_profile=true`. #### Slow HDFS DataNodes[​](#slow-hdfs-datanodes "Direct link to Slow HDFS DataNodes") ##### Issue description[​](#issue-description "Direct link to Issue description") When you access the data files stored in your HDFS cluster, you may find a huge difference between the values of the `__MAX_OF_FSIOTime` and `__MIN_OF_FSIOTime` metrics from the profiles of the SQL queries you run. This indicates that some DataNodes in the HDFS cluster are slow. The following example is a typical profile that indicates a slow HDFS DataNode issue: ```plaintext - InputStream: 0 - AppIOBytesRead: 22.72 GB - __MAX_OF_AppIOBytesRead: 187.99 MB - __MIN_OF_AppIOBytesRead: 64.00 KB - AppIOCounter: 964.862K (964862) - __MAX_OF_AppIOCounter: 7.795K (7795) - __MIN_OF_AppIOCounter: 1 - AppIOTime: 1s372ms - __MAX_OF_AppIOTime: 4s358ms - __MIN_OF_AppIOTime: 1.539ms - FSBytesRead: 15.40 GB - __MAX_OF_FSBytesRead: 127.41 MB - __MIN_OF_FSBytesRead: 64.00 KB - FSIOCounter: 1.637K (1637) - __MAX_OF_FSIOCounter: 12 - __MIN_OF_FSIOCounter: 1 - FSIOTime: 9s357ms - __MAX_OF_FSIOTime: 60s335ms - __MIN_OF_FSIOTime: 1.536ms ``` ##### Solution[​](#solution "Direct link to Solution") You can use one of the following solutions to resolve this issue: * **\[Recommended]** Enable the [data cache](https://docs.starrocks.io/docs/data_source/data_cache.md) feature, which eliminates the impact of slow HDFS DataNodes on queries by automatically caching the data from external storage systems to the BEs or CNs of your StarRocks cluster. * **\[Recommended]** Shorten the timeout duration between the HDFS client and DataNode. This solution is suitable when Data Cache cannot help resolve the slow HDFS DataNode issue. * Enable the [Hedged Read](https://hadoop.apache.org/docs/r2.8.3/hadoop-project-dist/hadoop-common/release/2.4.0/RELEASENOTES.2.4.0.html) feature. With this feature enabled, if a read from a block is slow, StarRocks starts up a new read, which runs in parallel to the original read, to read against a different block replica. Whenever one of the two reads returns, the other read is cancelled. **The Hedged Read feature can help accelerate reads, but it also significantly increases heap memory consumption on Java virtual machines (JVMs). Therefore, if your physical machines provide a small memory capacity, we recommend that you do not enable the Hedged Read feature.** ###### \[Recommended] Data Cache[​](#recommended-data-cache "Direct link to [Recommended] Data Cache") See [Data Cache](https://docs.starrocks.io/docs/data_source/data_cache.md). ###### \[Recommended] Shorten timeout duration between HDFS client and DataNode[​](#recommended-shorten-timeout-duration-between-hdfs-client-and-datanode "Direct link to [Recommended] Shorten timeout duration between HDFS client and DataNode") Configure the `dfs.client.socket-timeout` property in the `hdfs-site.xml` file to shorten the timeout duration between the HDFS client and DataNode. (The default timeout duration is 60s, which is a bit long.) As such, when StarRocks encounters a slow DataNode, the connection request from it can time out within a very short period of time and then be forwarded to another DataNode. The following example sets a 5-second timeout duration: ```xml dfs.client.socket-timeout 5000 ``` ###### Hedged Read[​](#hedged-read "Direct link to Hedged Read") Use the following parameters (supported from v3.0 onwards) in the BE or CN configuration file `be.conf` to enable and configure the Hedged Read feature in your HDFS cluster. | Parameter | Default value | Description | | --------------------------------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hdfs\_client\_enable\_hedged\_read | false | Specifies whether to enable the hedged read feature. | | hdfs\_client\_hedged\_read\_threadpool\_size | 128 | Specifies the size of the Hedged Read thread pool on your HDFS client. The thread pool size limits the number of threads to dedicate to the running of hedged reads in your HDFS client. This parameter is equivalent to the `dfs.client.hedged.read.threadpool.size` parameter in the `hdfs-site.xml` file of your HDFS cluster. | | hdfs\_client\_hedged\_read\_threshold\_millis | 2500 | Specifies the number of milliseconds to wait before starting up a hedged read. For example, you have set this parameter to `30`. In this situation, if a read from a block has not returned within 30 milliseconds, your HDFS client immediately starts up a hedged read against a different block replica. This parameter is equivalent to the `dfs.client.hedged.read.threshold.millis` parameter in the `hdfs-site.xml` file of your HDFS cluster. | If the value of any of the following metrics in your query profiles exceeds `0`, the Hedged Read feature is enabled. | Metric | Description | | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | TotalHedgedReadOps | The number of hedged reads that are started up. | | TotalHedgedReadOpsInCurThread | The number of times that StarRocks has to start up a hedged read in the current thread instead of in a new thread because the Hedged Read thread pool has reached its maximum size specified by the `hdfs_client_hedged_read_threadpool_size` parameter. | | TotalHedgedReadOpsWin | The number of times that a hedged read beats its original read. | #### How do I resolve the error “ERROR 1064 (HY000): Type mismatches on column \[is\_refund], JDBC result type is Integer, please set the type to one of tinyint,smallint,int,bigint” when querying a table in the Hive Catalog?[​](#how-do-i-resolve-the-error-error-1064-hy000-type-mismatches-on-column-is_refund-jdbc-result-type-is-integer-please-set-the-type-to-one-of-tinyintsmallintintbigint-when-querying-a-table-in-the-hive-catalog "Direct link to How do I resolve the error “ERROR 1064 (HY000): Type mismatches on column [is_refund], JDBC result type is Integer, please set the type to one of tinyint,smallint,int,bigint” when querying a table in the Hive Catalog?") This issue is caused by an incorrect JDBC connection configuration. Add the parameter `tinyInt1isBit=false` to your JDBC URI to prevent this issue: ```sql "jdbc_uri" = "jdbc:mysql://xxx:3306?database=yl_spmibill&tinyInt1isBit=false" ``` #### Why can’t I query the latest updated data in the Iceberg Catalog (even after refresh or catalog rebuild), and how should I troubleshoot this?[​](#why-cant-i-query-the-latest-updated-data-in-the-iceberg-catalog-even-after-refresh-or-catalog-rebuild-and-how-should-i-troubleshoot-this "Direct link to Why can’t I query the latest updated data in the Iceberg Catalog (even after refresh or catalog rebuild), and how should I troubleshoot this?") First check whether the issue is caused by Data Cache being enabled. Follow these steps to verify: 1. Compare the scanned data files between StarRocks and Spark: * In StarRocks: `select file_path, spec_id from db.table_name$files;` * In Spark: `select file_path, spec_id from db.table_name.files;` 2. If the results are consistent, continue troubleshooting by disabling Data Cache and querying again to see whether the issue persists. Root cause: To update the Iceberg table data is to overwrite old files, which corrupts Iceberg’s historical data. The correct behavior is to generate new file names when writing updates. StarRocks Data Cache uses the file name, file size, and modification time to determine whether cached data is valid. Since Iceberg does not overwrite files and the modification time is always 0, StarRocks incorrectly treats the files as unchanged and reads from cache, resulting in outdated query results. #### FE constantly crashes after queries against tables in external catalog integrated with JuiceFS. How can I solve this?[​](#fe-constantly-crashes-after-queries-against-tables-in-external-catalog-integrated-with-juicefs-how-can-i-solve-this "Direct link to FE constantly crashes after queries against tables in external catalog integrated with JuiceFS. How can I solve this?") To solve this, restart FE after adding the following configuration items to **fe.conf**: ```properties proc_profile_mem_enable=false proc_profile_cpu_enable=false ``` --- ### External table note The External Table feature is no longer recommended except for certain corner usage cases, and might be deprecated in future releases. To manage and query data from external data sources in general scenarios, [External Catalog](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md) is recommended. * From v3.0 onwards, we recommend that you use catalogs to query data from Hive, Iceberg, and Hudi. See [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md), [Iceberg catalog](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md), and [Hudi catalog](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md). * From v3.1 onwards, we recommend that you use [JDBC catalog](https://docs.starrocks.io/docs/data_source/catalog/jdbc_catalog.md) to query data from MySQL and PostgreSQL and use [Elasticsearch catalog](https://docs.starrocks.io/docs/data_source/catalog/elasticsearch_catalog.md) to query data from Elasticsearch. * From v3.2.9 and v3.3.1, we recommend that you use [JDBC catalog](https://docs.starrocks.io/docs/data_source/catalog/jdbc_catalog.md) to query data from Oracle and SQL Server. * The External Table feature was designed to help with loading data into StarRocks, NOT to perform efficient queries against external systems as a normal operation. A more performant solution would be to load the data into StarRocks. StarRocks supports access to other data sources by using external tables. External tables are created based on data tables that are stored in other data sources. StarRocks only stores the metadata of the data tables. You can use external tables to directly query data in other data sources. Currently, except StarRocks external table, all other external tables are deprecated. **You can only write data from another StarRocks cluster into the current StarRocks cluster. You cannot read data from it. For data sources other than StarRocks, you can only read data from these data sources.** From 2.5 onwards, StarRocks provides the Data Cache feature, which accelerates hot data queriers on external data sources. For more information, see [Data Cache](https://docs.starrocks.io/docs/data_source/data_cache.md). #### StarRocks external table[​](#starrocks-external-table "Direct link to StarRocks external table") From StarRocks 1.19 onwards, StarRocks allows you to use a StarRocks external table to write data from one StarRocks cluster to another. This achieves read-write separation and provides better resource isolation. You can first create a destination table in the destination StarRocks cluster. Then, in the source StarRocks cluster, you can create a StarRocks external table that has the same schema as the destination table and specify the information of the destination cluster and table in the `PROPERTIES` field. Data can be written from a source cluster to a destination cluster by using INSERT INTO statement to write into a StarRocks external table. It can help realize the following goals: * Data synchronization between StarRocks clusters. * Read-write separation. Data is written to the source cluster, and data changes from the source cluster are synchronized to the destination cluster, which provides query services. The following code shows how to create a destination table and an external table. ```sql # Create a destination table in the destination StarRocks cluster. CREATE TABLE t ( k1 DATE, k2 INT, k3 SMALLINT, k4 VARCHAR(2048), k5 DATETIME ) ENGINE=olap DISTRIBUTED BY HASH(k1); # Create an external table in the source StarRocks cluster. CREATE EXTERNAL TABLE external_t ( k1 DATE, k2 INT, k3 SMALLINT, k4 VARCHAR(2048), k5 DATETIME ) ENGINE=olap DISTRIBUTED BY HASH(k1) PROPERTIES ( "host" = "127.0.0.1", "port" = "9020", "user" = "user", "password" = "passwd", "database" = "db_test", "table" = "t" ); # Write data from a source cluster to a destination cluster by writing data into the StarRocks external table. The second statement is recommended for the production environment. insert into external_t values ('2020-10-11', 1, 1, 'hello', '2020-10-11 10:00:00'); insert into external_t select * from other_table; ``` Parameters: * **EXTERNAL:** This keyword indicates that the table to be created is an external table. * **host:** This parameter specifies the IP address of the leader FE node of the destination StarRocks cluster. * **port:** This parameter specifies the RPC port of the FE node of the destination StarRocks cluster. note To ensure that the source cluster to which the StarRocks external tables belong can access the destination StarRocks cluster, you must configure your network and firewall to allow access to the following ports: * The RPC port of the FE node. See `rpc_port` in the FE configuration file **fe/fe.conf**. The default RPC port is `9020`. * The bRPC port of the BE node. See `brpc_port` in the BE configuration file **be/be.conf**. The default bRPC port is `8060`. * **user:** This parameter specifies the username used to access the destination StarRocks cluster. * **password:** This parameter specifies the password used to access the destination StarRocks cluster. * **database:** This parameter specifies the database to which the destination table belongs. * **table:** This parameter specifies the name of the destination table. The following limits apply when you use a StarRocks external table: * You can only run the INSERT INTO and SHOW CREATE TABLE commands on a StarRocks external table. Other data writing methods are not supported. In addition, you cannot query data from a StarRocks external table or perform DDL operations on the external table. * The syntax of creating an external table is the same as creating a normal table, but the column names and other information in the external table must be the same as the destination table. * The external table synchronizes table metadata from the destination table every 10 seconds. If a DDL operation is performed on the destination table, there may be a delay for data synchronization between the two tables. #### (Deprecated) External table for a JDBC-compatible database[​](#deprecated-external-table-for-a-jdbc-compatible-database "Direct link to (Deprecated) External table for a JDBC-compatible database") From v2.3.0, StarRocks provides external tables to query JDBC-compatible databases. This way, you can analyze the data of such databases in a blazing fast manner without the need to import the data into StarRocks. This section describes how to create an external table in StarRocks and query data in JDBC-compatible databases. ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before you use a JDBC external table to query data, make sure that the FEs and BEs have access to the download URL of the JDBC driver. The download URL is specified by the `driver_url` parameter in the statement used for creating the JDBC resource. ##### Create and manage JDBC resources[​](#create-and-manage-jdbc-resources "Direct link to Create and manage JDBC resources") ###### Create a JDBC resource[​](#create-a-jdbc-resource "Direct link to Create a JDBC resource") Before you create an external table to query data from a database, you need to create a JDBC resource in StarRocks to manage the connection information of the database. The database must support the JDBC driver and is referred as the "target database". After creating the resource, you can use it to create external tables. Execute the following statement to create a JDBC resource named `jdbc0`: ```sql CREATE EXTERNAL RESOURCE jdbc0 PROPERTIES ( "type"="jdbc", "user"="postgres", "password"="changeme", "jdbc_uri"="jdbc:postgresql://127.0.0.1:5432/jdbc_test", "driver_url"="https://repo1.maven.org/maven2/org/postgresql/postgresql/42.3.3/postgresql-42.3.3.jar", "driver_class"="org.postgresql.Driver" ); ``` The required parameters in `PROPERTIES` are as follows: * `type`: the type of the resource. Set the value to `jdbc`. * `user`: the username that is used to connect to the target database. * `password`: the password that is used to connect to the target database. * `jdbc_uri`: the URI that the JDBC driver uses to connect to the target database. The URI format must satisfy the database URI syntax. For the URI syntax of some common databases, visit the official websites of [Oracle](https://docs.oracle.com/en/database/oracle/oracle-database/21/jjdbc/data-sources-and-URLs.html#GUID-6D8EFA50-AB0F-4A2B-88A0-45B4A67C361E), [PostgreSQL](https://jdbc.postgresql.org/documentation/head/connect.html), [SQL Server](https://learn.microsoft.com/en-us/sql/connect/jdbc/building-the-connection-url?view=sql-server-ver16). > Note: The URI must include the name of the target database. For example, in the preceding code example, `jdbc_test` is the name of the target database that you want to connect. * `driver_url`: the download URL of the JDBC driver JAR package. An HTTP URL or file URL is supported, for example, `https://repo1.maven.org/maven2/org/postgresql/postgresql/42.3.3/postgresql-42.3.3.jar` or `file:///home/disk1/postgresql-42.3.3.jar`. * `driver_class`: the class name of the JDBC driver. The JDBC driver class names of common databases are as follows: * MySQL: com.mysql.jdbc.Driver (MySQL 5.x and earlier), com.mysql.cj.jdbc.Driver (MySQL 6.x and later) * SQL Server: com.microsoft.sqlserver.jdbc.SQLServerDriver * Oracle: oracle.jdbc.driver.OracleDriver * PostgreSQL: org.postgresql.Driver When the resource is being created, the FE downloads the JDBC driver JAR package by using the URL that is specified in the `driver_url` parameter, generates a checksum, and uses the checksum to verify the JDBC driver downloaded by BEs. > Note: If the download of the JDBC driver JAR package fails, the creation of the resource also fails. When BEs query the JDBC external table for the first time and find that the corresponding JDBC driver JAR package does not exist on their machines, BEs download the JDBC driver JAR package by using the URL that is specified in the `driver_url` parameter, and all JDBC driver JAR packages are saved in the `${STARROCKS_HOME}/lib/jdbc_drivers` directory. ###### View JDBC resources[​](#view-jdbc-resources "Direct link to View JDBC resources") Execute the following statement to view all JDBC resources in StarRocks: ```sql SHOW RESOURCES; ``` > Note: The `ResourceType` column is `jdbc`. ###### Delete a JDBC resource[​](#delete-a-jdbc-resource "Direct link to Delete a JDBC resource") Execute the following statement to delete the JDBC resource named `jdbc0`: ```sql DROP RESOURCE "jdbc0"; ``` > Note: After a JDBC resource is deleted, all JDBC external tables that are created by using that JDBC resource are unavailable. However, the data in the target database is not lost. If you still need to use StarRocks to query data in the target database, you can create the JDBC resource and the JDBC external tables again. ##### Create a database[​](#create-a-database "Direct link to Create a database") Execute the following statement to create and access a database named `jdbc_test` in StarRocks: ```sql CREATE DATABASE jdbc_test; USE jdbc_test; ``` > Note: The database name that you specify in the preceding statement does not need to be same as the name of the target database. ##### Create a JDBC external table[​](#create-a-jdbc-external-table "Direct link to Create a JDBC external table") Execute the following statement to create a JDBC external table named `jdbc_tbl` in the database `jdbc_test`: ```sql create external table jdbc_tbl ( `id` bigint NULL, `data` varchar(200) NULL ) ENGINE=jdbc properties ( "resource" = "jdbc0", "table" = "dest_tbl" ); ``` The required parameters in `properties` are as follows: * `resource`: the name of the JDBC resource used to create the external table. * `table`: the target table name in the database. For supported data types and data type mapping between StarRocks and target databases, see [Data type mapping](#data-type-mapping). > Note: > > * Indexes are not supported. > * You cannot use PARTITION BY or DISTRIBUTED BY to specify data distribution rules. ##### Query a JDBC external table[​](#query-a-jdbc-external-table "Direct link to Query a JDBC external table") Before you query JDBC external tables, you must execute the following statement to enable the Pipeline engine: ```sql set enable_pipeline_engine=true; ``` > Note: If the Pipeline engine is already enabled, you can skip this step. Execute the following statement to query the data in the target database by using JDBC external tables. ```sql select * from JDBC_tbl; ``` StarRocks supports predicate pushdown by pushing down filter conditions to the target table. Executing filter conditions as close as possible to the data source can improve query performance. Currently, StarRocks can push down operators, including the binary comparison operators (`>`, `>=`, `=`, `<`, and `<=`), `IN`, `IS NULL`, and `BETWEEN ... AND ...` . However, StarRocks can not push down functions. ##### Data type mapping[​](#data-type-mapping "Direct link to Data type mapping") Currently, StarRocks can only query data of basic types in the target database, such as NUMBER, STRING, TIME, and DATE. If the ranges of data values in the target database are not supported by StarRocks, the query reports an error. The mapping between the target database and StarRocks varies based on the type of the target database. ###### **MySQL and StarRocks**[​](#mysql-and-starrocks "Direct link to mysql-and-starrocks") | MySQL | StarRocks | | ------------ | --------- | | BOOLEAN | BOOLEAN | | TINYINT | TINYINT | | SMALLINT | SMALLINT | | MEDIUMINTINT | INT | | BIGINT | BIGINT | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | DECIMAL | DECIMAL | | CHAR | CHAR | | VARCHAR | VARCHAR | | DATE | DATE | | DATETIME | DATETIME | ###### **Oracle and StarRocks**[​](#oracle-and-starrocks "Direct link to oracle-and-starrocks") | Oracle | StarRocks | | --------------- | --------- | | CHAR | CHAR | | VARCHARVARCHAR2 | VARCHAR | | DATE | DATE | | SMALLINT | SMALLINT | | INT | INT | | BINARY\_FLOAT | FLOAT | | BINARY\_DOUBLE | DOUBLE | | DATE | DATE | | DATETIME | DATETIME | | NUMBER | DECIMAL | ###### **PostgreSQL and StarRocks**[​](#postgresql-and-starrocks "Direct link to postgresql-and-starrocks") | PostgreSQL | StarRocks | | ------------------- | --------- | | SMALLINTSMALLSERIAL | SMALLINT | | INTEGERSERIAL | INT | | BIGINTBIGSERIAL | BIGINT | | BOOLEAN | BOOLEAN | | REAL | FLOAT | | DOUBLE PRECISION | DOUBLE | | DECIMAL | DECIMAL | | TIMESTAMP | DATETIME | | DATE | DATE | | CHAR | CHAR | | VARCHAR | VARCHAR | | TEXT | VARCHAR | ###### **SQL Server and StarRocks**[​](#sql-server-and-starrocks "Direct link to sql-server-and-starrocks") | SQL Server | StarRocks | | ----------------- | --------- | | BOOLEAN | BOOLEAN | | TINYINT | TINYINT | | SMALLINT | SMALLINT | | INT | INT | | BIGINT | BIGINT | | FLOAT | FLOAT | | REAL | DOUBLE | | DECIMALNUMERIC | DECIMAL | | CHAR | CHAR | | VARCHAR | VARCHAR | | DATE | DATE | | DATETIMEDATETIME2 | DATETIME | ##### Limits[​](#limits "Direct link to Limits") * When you create JDBC external tables, you cannot create indexes on the tables or use PARTITION BY and DISTRIBUTED BY to specify data distribution rules for the tables. * When you query JDBC external tables, StarRocks cannot push down functions to the tables. #### (Deprecated) Elasticsearch external table[​](#deprecated-elasticsearch-external-table "Direct link to (Deprecated) Elasticsearch external table") StarRocks and Elasticsearch are two popular analytics systems. StarRocks is performant in large-scale distributed computing. Elasticsearch is ideal for full-text search. StarRocks combined with Elasticsearch can deliver a more complete OLAP solution. ##### Example of creating an Elasticsearch external table[​](#example-of-creating-an-elasticsearch-external-table "Direct link to Example of creating an Elasticsearch external table") ###### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL TABLE elastic_search_external_table ( k1 DATE, k2 INT, k3 SMALLINT, k4 VARCHAR(2048), k5 DATETIME ) ENGINE=ELASTICSEARCH PROPERTIES ( "hosts" = "http://192.168.0.1:9200,http://192.168.0.2:9200", "user" = "root", "password" = "root", "index" = "tindex", "type" = "_doc", "es.net.ssl" = "true" ); ``` The following table describes the parameters. | **Parameter** | **Required** | **Default value** | **Description** | | ---------------------- | ------------ | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hosts | Yes | None | The connection address of the Elasticsearch cluster. You can specify one or more addresses. StarRocks can parse the Elasticsearch version and index shard allocation from this address. StarRocks communicates with your Elasticsearch cluster based on the address returned by the `GET /_nodes/http` API operation. Therefore, the value of the `host` parameter must be the same as the address returned by the `GET /_nodes/http` API operation. Otherwise, BEs may not be able to communicate with your Elasticsearch cluster. | | index | Yes | None | The name of the Elasticsearch index that is created on the table in StarRocks. The name can be an alias. This parameter supports wildcards (`*`). For example, if you set `index` to `hello*`, StarRocks retrieves all indexes whose names start with `hello`. | | user | No | Empty | The username that is used to log in to the Elasticsearch cluster with basic authentication enabled. Make sure you have access to `/*cluster/state/*nodes/http` and the index. | | password | No | Empty | The password that is used to log in to the Elasticsearch cluster. | | type | No | `_doc` | The type of the index. Default value: `_doc`. If you want to query data in Elasticsearch 8 and later versions, you do not need to configure this parameter because the mapping types have been removed in Elasticsearch 8 and later versions. | | es.nodes.wan.only | No | `false` | Specifies whether StarRocks only uses the addresses specified by `hosts` to access the Elasticsearch cluster and fetch data.- `true`: StarRocks only uses the addresses specified by `hosts` to access the Elasticsearch cluster and fetch data and does not sniff data nodes on which the shards of the Elasticsearch index reside. If StarRocks cannot access the addresses of the data nodes inside the Elasticsearch cluster, you need to set this parameter to `true`.
- `false`: StarRocks uses the addresses specified by `host` to sniff data nodes on which the shards of the Elasticsearch cluster indexes reside. After StarRocks generates a query execution plan, the relevant BEs directly access the data nodes inside the Elasticsearch cluster to fetch data from the shards of indexes. If StarRocks can access the addresses of the data nodes inside the Elasticsearch cluster, we recommend that you retain the default value `false`. | | es.net.ssl | No | `false` | Specifies whether the HTTPS protocol can be used to access your Elasticsearch cluster. Only StarRocks 2.4 and later versions support configuring this parameter.- `true`: Both the HTTPS and HTTP protocols can be used to access your Elasticsearch cluster.
- `false`: Only the HTTP protocol can be used to access your Elasticsearch cluster. | | enable\_docvalue\_scan | No | `true` | Specifies whether to obtain the values of the target fields from Elasticsearch columnar storage. In most cases, reading data from columnar storage outperforms reading data from row storage. | | enable\_keyword\_sniff | No | `true` | Specifies whether to sniff TEXT-type fields in Elasticsearch based on KEYWORD-type fields. If this parameter is set to `false`, StarRocks performs matching after tokenization. | ###### Columnar scan for faster queries[​](#columnar-scan-for-faster-queries "Direct link to Columnar scan for faster queries") If you set `enable_docvalue_scan` to `true`, StarRocks follows these rules when it obtains data from Elasticsearch: * **Try and see**: StarRocks automatically checks if columnar storage is enabled for the target fields. If so, StarRocks obtains all values in the target fields from columnar storage. * **Auto-downgrading**: If any one of the target fields is unavailable in columnar storage, StarRocks parses and obtains all values in the target fields from row storage (`_source`). > **NOTE** > > * Columnar storage is unavailable for TEXT-type fields in Elasticsearch. Therefore, if you query fields containing TEXT-type values, StarRocks obtains the values of the fields from `_source`. > * If you query a large number (greater than or equal to 25) of fields, reading field values from `docvalue` does not show noticeable benefits compared with reading field values from `_source`. ###### Sniff KEYWORD-type fields[​](#sniff-keyword-type-fields "Direct link to Sniff KEYWORD-type fields") If you set `enable_keyword_sniff` to `true`, Elasticsearch allows direct data ingestion without an index because it will automatically create an index after ingestion. For STRING-type fields, Elasticsearch will create a field with both TEXT and KEYWORD types. This is how the Multi-Field feature of Elasticsearch works. The mapping is as follows: ```sql "k4": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } ``` For example, to conduct "=" filtering on `k4`, StarRocks on Elasticsearch will convert the filtering operation into an Elasticsearch TermQuery. The original SQL filter is as follows: ```sql k4 = "StarRocks On Elasticsearch" ``` The converted Elasticsearch query DSL is as follows: ```sql "term" : { "k4": "StarRocks On Elasticsearch" } ``` The first field of `k4` is TEXT, and it will be tokenized by the analyzer configured for `k4` (or by the standard analyzer if no analyzer has been configured for `k4`) after data ingestion. As a result, the first field will be tokenized into three terms: `StarRocks`, `On`, and `Elasticsearch`. The details are as follows: ```sql POST /_analyze { "analyzer": "standard", "text": "StarRocks On Elasticsearch" } ``` The tokenization results are as follows: ```sql { "tokens": [ { "token": "starrocks", "start_offset": 0, "end_offset": 5, "type": "", "position": 0 }, { "token": "on", "start_offset": 6, "end_offset": 8, "type": "", "position": 1 }, { "token": "elasticsearch", "start_offset": 9, "end_offset": 11, "type": "", "position": 2 } ] } ``` Suppose you conduct a query as follows: ```sql "term" : { "k4": "StarRocks On Elasticsearch" } ``` There is no term in the dictionary that matches the term `StarRocks On Elasticsearch`, and therefore no result will be returned. However, if you have set `enable_keyword_sniff` to `true`, StarRocks will convert `k4 = "StarRocks On Elasticsearch"` to `k4.keyword = "StarRocks On Elasticsearch"` to match the SQL semantics. The converted `StarRocks On Elasticsearch` query DSL is as follows: ```sql "term" : { "k4.keyword": "StarRocks On Elasticsearch" } ``` `k4.keyword` is of the KEYWORD type. Therefore, the data is written into Elasticsearch as a complete term, allowing for successful matching. ###### Mapping of column data types[​](#mapping-of-column-data-types "Direct link to Mapping of column data types") When you create an external table, you need to specify the data types of columns in the external table based on the data types of columns in the Elasticsearch table. The following table shows the mapping of column data types. | **Elasticsearch** | **StarRocks** | | ----------------- | --------------------------- | | BOOLEAN | BOOLEAN | | BYTE | TINYINT/SMALLINT/INT/BIGINT | | SHORT | SMALLINT/INT/BIGINT | | INTEGER | INT/BIGINT | | LONG | BIGINT | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | KEYWORD | CHAR/VARCHAR | | TEXT | CHAR/VARCHAR | | DATE | DATE/DATETIME | | NESTED | CHAR/VARCHAR | | OBJECT | CHAR/VARCHAR | | ARRAY | ARRAY | > **Note** > > * StarRocks reads the data of the NESTED type by using JSON-related functions. > * Elasticsearch automatically flattens multi-dimensional arrays into one-dimensional arrays. StarRocks does the same. **The support for querying ARRAY data from Elasticsearch is added from v2.5.** ##### Predicate pushdown[​](#predicate-pushdown "Direct link to Predicate pushdown") StarRocks supports predicate pushdown. Filters can be pushed down to Elasticsearch for execution, which improves query performance. The following table lists the operators that support predicate pushdown. | SQL syntax | ES syntax | | -------------- | ---------------------- | | `=` | term query | | `in` | terms query | | `>=, <=, >, <` | range | | `and` | bool.filter | | `or` | bool.should | | `not` | bool.must\_not | | `not in` | bool.must\_not + terms | | `esquery` | ES Query DSL | ##### Examples[​](#examples "Direct link to Examples") The **esquery function** is used to push down queries **that cannot be expressed in SQL** (such as match and geoshape) to Elasticsearch for filtering. The first parameter in the esquery function is used to associate an index. The second parameter is a JSON expression of basic Query DSL, which is enclosed in brackets . **The JSON expression must have but only one root key**, such as match, geo\_shape, or bool. * match query ```sql select * from es_table where esquery(k4, '{ "match": { "k4": "StarRocks on elasticsearch" } }'); ``` * geo-related query ```sql select * from es_table where esquery(k4, '{ "geo_shape": { "location": { "shape": { "type": "envelope", "coordinates": [ [ 13, 53 ], [ 14, 52 ] ] }, "relation": "within" } } }'); ``` * bool query ```sql select * from es_table where esquery(k4, ' { "bool": { "must": [ { "terms": { "k1": [ 11, 12 ] } }, { "terms": { "k2": [ 100 ] } } ] } }'); ``` ##### Usage notes[​](#usage-notes "Direct link to Usage notes") * Elasticsearch earlier than 5.x scans data in a different way than that later than 5.x. Currently, **only versions later than 5.x** are supported. * Elasticsearch clusters with HTTP basic authentication enabled are supported. * Querying data from StarRocks may not be as fast as directly querying data from Elasticsearch, such as count-related queries. The reason is that Elasticsearch directly reads the metadata of target documents without the need to filter the real data, which accelerates the count query. #### (Deprecated) Hive external table[​](#deprecated-hive-external-table "Direct link to (Deprecated) Hive external table") Before using Hive external tables, make sure JDK 1.8 has been installed on your servers. ##### Create a Hive resource[​](#create-a-hive-resource "Direct link to Create a Hive resource") A Hive resource corresponds to a Hive cluster. You must configure the Hive cluster used by StarRocks, such as the Hive metastore address. You must specify the Hive resource that is used by the Hive external table. * Create a Hive resource named hive0. ```sql CREATE EXTERNAL RESOURCE "hive0" PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` * View the resources created in StarRocks. ```sql SHOW RESOURCES; ``` * Delete the resource named `hive0`. ```sql DROP RESOURCE "hive0"; ``` You can modify `hive.metastore.uris` of a Hive resource in StarRocks 2.3 and later versions. For more information, see [ALTER RESOURCE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/ALTER_RESOURCE.md). ##### Create a database[​](#create-a-database-1 "Direct link to Create a database") ```sql CREATE DATABASE hive_test; USE hive_test; ``` ##### Create a Hive external table[​](#create-a-hive-external-table "Direct link to Create a Hive external table") Syntax ```sql CREATE EXTERNAL TABLE table_name ( col_name col_type [NULL | NOT NULL] [COMMENT "comment"] ) ENGINE=HIVE PROPERTIES ( "key" = "value" ); ``` Example: Create the external table `profile_parquet_p7` under the `rawdata` database in the Hive cluster corresponding to the `hive0` resource. ```sql CREATE EXTERNAL TABLE `profile_wos_p7` ( `id` bigint NULL, `first_id` varchar(200) NULL, `second_id` varchar(200) NULL, `p__device_id_list` varchar(200) NULL, `p__is_deleted` bigint NULL, `p_channel` varchar(200) NULL, `p_platform` varchar(200) NULL, `p_source` varchar(200) NULL, `p__city` varchar(200) NULL, `p__province` varchar(200) NULL, `p__update_time` bigint NULL, `p__first_visit_time` bigint NULL, `p__last_seen_time` bigint NULL ) ENGINE=HIVE PROPERTIES ( "resource" = "hive0", "database" = "rawdata", "table" = "profile_parquet_p7" ); ``` Description: * Columns in the external table * The column names must be the same as column names in the Hive table. * The column order **does not need to be** the same as column order in the Hive table. * You can select only **some of the columns in the Hive table**, but you must select all the **partition key columns**. * Partition key columns of an external table do not need to be specified by using `partition by`. They must be defined in the same description list as other columns. You do not need to specify partition information. StarRocks will automatically synchronize this information from the Hive table. * Set `ENGINE` to HIVE. * PROPERTIES: * **hive.resource**: the Hive resource that is used. * **database**: the Hive database. * **table**: the table in Hive. **view** is not supported. * The following table describes the column data type mapping between Hive and StarRocks. | Column type of Hive | Column type of StarRocks | Description | | ------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | INT/INTEGER | INT | | | BIGINT | BIGINT | | | TIMESTAMP | DATETIME | Precision and time zone information will be lost when you convert TIMESTAMP data into DATETIME data. You need to convert TIMESTAMP data into DATETIME data that does not have the time zone offset based on the time zone in sessionVariable. | | STRING | VARCHAR | | | VARCHAR | VARCHAR | | | CHAR | CHAR | | | DOUBLE | DOUBLE | | | FLOAT | FLOAT | | | DECIMAL | DECIMAL | | | ARRAY | ARRAY | | > Note: > > * Currently, the supported Hive storage formats are Parquet, ORC, and CSV. If the storage format is CSV, quotation marks cannot be used as escape characters. > * The SNAPPY and LZ4 compression formats are supported. > * The maximum length of a Hive string column that can be queried is 1 MB. If a string column exceeds 1 MB, it will be processed as a null column. ##### Use a Hive external table[​](#use-a-hive-external-table "Direct link to Use a Hive external table") Query the total number of rows of `profile_wos_p7`. ```sql select count(*) from profile_wos_p7; ``` ##### Update cached Hive table metadata[​](#update-cached-hive-table-metadata "Direct link to Update cached Hive table metadata") * Hive partition information and the related file information are cached in StarRocks. The cache is refreshed at intervals specified by `hive_meta_cache_refresh_interval_s`. The default value is 7200. * The cached data can also be refreshed manually. 1. If a partition is added or deleted from a table in Hive, you must run the `REFRESH EXTERNAL TABLE hive_t` command to refresh the table metadata cached in StarRocks. `hive_t` is the name of the Hive external table in StarRocks. 2. If data in some Hive partitions is updated, you must refresh the cached data in StarRocks by running the `REFRESH EXTERNAL TABLE hive_t PARTITION ('k1=01/k2=02', 'k1=03/k2=04')` command. `hive_t` is the name of the Hive external table in StarRocks. `'k1=01/k2=02'` and `'k1=03/k2=04'` are the names of Hive partitions whose data is updated. 3. When you run `REFRESH EXTERNAL TABLE hive_t`, StarRocks first checks if the column information of the Hive external table is the same as the column information of the Hive table returned by the Hive Metastore. If the schema of the Hive table changes, such as adding columns or removing columns, StarRocks synchronizes the changes to the Hive external table. After synchronization, the column order of the Hive external table remains the same as the column order of the Hive table, with the partition column being the last column. * When Hive data is stored in the Parquet, ORC, and CSV format, you can synchronize schema changes (such as ADD COLUMN and REPLACE COLUMN) of a Hive table to a Hive external table in StarRocks 2.3 and later versions. ##### Access object storage[​](#access-object-storage "Direct link to Access object storage") * The path of the FE configuration file is `fe/conf`, to which the configuration file can be added if you need to customize the Hadoop cluster. For example: If the HDFS cluster uses a highly available nameservice, you need to put `hdfs-site.xml` under `fe/conf`. If HDFS is configured with ViewFs, you need to put the `core-site.xml` under `fe/conf`. * The path of the BE configuration file is `be/conf`, to which the configuration file can be added if you need to customize the Hadoop cluster. For example, if the HDFS cluster using a highly available nameservice, you need to put `hdfs-site.xml` under `be/conf`. If HDFS is configured with ViewFs, you need to put `core-site.xml` under `be/conf`. * On the machine where BE is located, configure JAVA\_HOME as a JDK environment rather than a JRE environment in the BE **startup script** `bin/start_be.sh`, for example, `export JAVA_HOME = `. You must add this configuration at the beginning of the script and restart the BE for the configuration to take effect. * Configure Kerberos support: 1. To log in with `kinit -kt keytab_path principal` to all FE/BE machines, you need to have access to Hive and HDFS. The kinit command login is only good for a period of time and needs to be put into crontab to be executed regularly. 2. Put `hive-site.xml/core-site.xml/hdfs-site.xml` under `fe/conf`, and put `core-site.xml/hdfs-site.xml` under `be/conf`. 3. Add `-Djava.security.krb5.conf=/etc/krb5.conf` to the value of the `JAVA_OPTS` option in the **$FE\_HOME/conf/fe.conf** file. **/etc/krb5.conf** is the save path of the **krb5.conf** file. You can change the path based on your operating system. 4. Directly add `JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"` to the **$BE\_HOME/conf/be.conf** file. **/etc/krb5.conf** is the save path of the **krb5.conf** file. You can change the path based on your operating system. 5. When you add a Hive resource, you must pass in a domain name to `hive.metastore.uris`. In addition, you must add the mapping between Hive/HDFS domain names and IP addresses in the **/etc/hosts** file. * Configure support for AWS S3: Add the following configuration to `fe/conf/core-site.xml` and `be/conf/core-site.xml`. ```xml fs.s3a.access.key ****** fs.s3a.secret.key ****** fs.s3a.endpoint s3.us-west-2.amazonaws.com fs.s3a.connection.maximum 500 ``` 1. `fs.s3a.access.key`: the AWS access key ID. 2. `fs.s3a.secret.key`: the AWS secret key. 3. `fs.s3a.endpoint`: the AWS S3 endpoint to connect to. 4. `fs.s3a.connection.maximum`: the maximum number of concurrent connections from StarRocks to S3. If an error `Timeout waiting for connection from poll` occurs during a query, you can set this parameter to a larger value. #### (Deprecated) Iceberg external table[​](#deprecated-iceberg-external-table "Direct link to (Deprecated) Iceberg external table") From v2.1.0, StarRocks allows you to query data from Apache Iceberg by using external tables. To query data in Iceberg, you need to create an Iceberg external table in StarRocks. When you create the table, you need to establish mapping between the external table and the Iceberg table you want to query. ##### Before you begin[​](#before-you-begin "Direct link to Before you begin") Make sure that StarRocks has permissions to access the metadata service (such as Hive metastore), file system (such as HDFS), and object storage system (such as Amazon S3 and Alibaba Cloud Object Storage Service) used by Apache Iceberg. ##### Precautions[​](#precautions "Direct link to Precautions") * Iceberg external tables can be used to query only the following types of data: * Iceberg v1 tables (Analytic Data Tables). ORC-formatted Iceberg v2 (Row-level Deletes) tables are supported from v3.0 onwards, and Parquet-formatted Iceberg v2 tables are supported from v3.1 onwards. For the differences between Iceberg v1 tables and Iceberg v2 tables, see [Iceberg Table Spec](https://iceberg.apache.org/spec/). * Tables that are compressed in gzip (default format), Zstd, LZ4, or Snappy format. * Files that are stored in Parquet or ORC format. * Iceberg external tables in StarRocks 2.3 and later versions support synchronizing schema changes of an Iceberg table while Iceberg external tables in versions earlier than StarRocks 2.3 do not. If the schema of an Iceberg table changes, you must delete the corresponding external table and create a new one. ##### Procedure[​](#procedure "Direct link to Procedure") ###### Step 1: Create an Iceberg resource[​](#step-1-create-an-iceberg-resource "Direct link to Step 1: Create an Iceberg resource") Before you create an Iceberg external table, you must create an Iceberg resource in StarRocks. The resource is used to manage the Iceberg access information. Additionally, you also need to specify this resource in the statement that is used to create the external table. You can create a resource based on your business requirements: * If the metadata of an Iceberg table is obtained from a Hive metastore, you can create a resource and set the catalog type to `HIVE`. * If the metadata of an Iceberg table is obtained from other services, you need to create a custom catalog. Then create a resource and set the catalog type to `CUSTOM`. ###### Create a resource whose catalog type is `HIVE`[​](#create-a-resource-whose-catalog-type-is-hive "Direct link to create-a-resource-whose-catalog-type-is-hive") For example, create a resource named `iceberg0` and set the catalog type to `HIVE`. ```sql CREATE EXTERNAL RESOURCE "iceberg0" PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "HIVE", "iceberg.catalog.hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` The following table describes the related parameters. | **Parameter** | **Description** | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | type | The resource type. Set the value to `iceberg`. | | iceberg.catalog.type | The catalog type of the resource. Both Hive catalog and custom catalog are supported. If you specify a Hive catalog, set the value to `HIVE`.If you specify a custom catalog, set the value to `CUSTOM`. | | iceberg.catalog.hive.metastore.uris | The URI of the Hive metastore. The parameter value is in the following format: `thrift://< IP address of Iceberg metadata >:< port number >`. The port number defaults to 9083. Apache Iceberg uses a Hive catalog to access the Hive metastore and then queries the metadata of Iceberg tables. | ###### Create a resource whose catalog type is `CUSTOM`[​](#create-a-resource-whose-catalog-type-is-custom "Direct link to create-a-resource-whose-catalog-type-is-custom") A custom catalog needs to inherit the abstract class BaseMetastoreCatalog, and you need to implement the IcebergCatalog interface. Additionally, the class name of a custom catalog cannot be duplicated with the name of the class that already exists in StarRock. After the catalog is created, package the catalog and its related files, and place them under the **fe/lib** path of each frontend (FE). Then restart each FE. After you complete the preceding operations, you can create a resource whose catalog is a custom catalog. For example, create a resource named `iceberg1` and set the catalog type to `CUSTOM`. ```sql CREATE EXTERNAL RESOURCE "iceberg1" PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "CUSTOM", "iceberg.catalog-impl" = "com.starrocks.IcebergCustomCatalog" ); ``` The following table describes the related parameters. | **Parameter** | **Description** | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | type | The resource type. Set the value to `iceberg`. | | iceberg.catalog.type | The catalog type of the resource. Both Hive catalog and custom catalog are supported. If you specify a Hive catalog, set the value to `HIVE`. If you specify a custom catalog, set the value to `CUSTOM`. | | iceberg.catalog-impl | The fully qualified class name of the custom catalog. FEs search for the catalog based on this name. If the catalog contains custom configuration items, you must add them to the `PROPERTIES` parameter as key-value pairs when you create an Iceberg external table. | You can modify `hive.metastore.uris` and `iceberg.catalog-impl`of a Iceberg resource in StarRocks 2.3 and later versions. For more information, see [ALTER RESOURCE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/ALTER_RESOURCE.md). ###### View Iceberg resources[​](#view-iceberg-resources "Direct link to View Iceberg resources") ```sql SHOW RESOURCES; ``` ###### Drop an Iceberg resource[​](#drop-an-iceberg-resource "Direct link to Drop an Iceberg resource") For example, drop a resource named `iceberg0`. ```sql DROP RESOURCE "iceberg0"; ``` Dropping an Iceberg resource makes all external tables that reference this resource unavailable. However, the corresponding data in Apache Iceberg is not deleted. If you still need to query the data in Apache Iceberg, create a new resource and a new external table. ###### Step 2: (Optional) Create a database[​](#step-2-optional-create-a-database "Direct link to Step 2: (Optional) Create a database") For example, create a database named `iceberg_test` in StarRocks. ```sql CREATE DATABASE iceberg_test; USE iceberg_test; ``` > Note: The name of the database in StarRocks can be different from the name of the database in Apache Iceberg. ###### Step 3: Create an Iceberg external table[​](#step-3-create-an-iceberg-external-table "Direct link to Step 3: Create an Iceberg external table") For example, create an Iceberg external table named `iceberg_tbl` in the database `iceberg_test`. ```sql CREATE EXTERNAL TABLE `iceberg_tbl` ( `id` bigint NULL, `data` varchar(200) NULL ) ENGINE=ICEBERG PROPERTIES ( "resource" = "iceberg0", "database" = "iceberg", "table" = "iceberg_table" ); ``` The following table describes the related parameters. | **Parameter** | **Description** | | ------------- | -------------------------------------------------------------------- | | ENGINE | The engine name. Set the value to `ICEBERG`. | | resource | The name of the Iceberg resource that the external table references. | | database | The name of the database to which the Iceberg table belongs. | | table | The name of the Iceberg table. | > Note: > > * The name of the external table can be different from the name of the Iceberg table. > > * The column names of the external table must be the same as those in the Iceberg table. The column order of the two tables can be different. If you define configuration items in the custom catalog and want configuration items to take effect when you query data, you can add the configuration items to the `PROPERTIES` parameter as key-value pairs when you create an external table. For example, if you define a configuration item `custom-catalog.properties` in the custom catalog, you can run the following command to create an external table. ```sql CREATE EXTERNAL TABLE `iceberg_tbl` ( `id` bigint NULL, `data` varchar(200) NULL ) ENGINE=ICEBERG PROPERTIES ( "resource" = "iceberg0", "database" = "iceberg", "table" = "iceberg_table", "custom-catalog.properties" = "my_property" ); ``` When you create an external table, you need to specify the data types of columns in the external table based on the data types of columns in the Iceberg table. The following table shows the mapping of column data types. | **Iceberg table** | **Iceberg external table** | | ----------------- | -------------------------- | | BOOLEAN | BOOLEAN | | INT | TINYINT / SMALLINT / INT | | LONG | BIGINT | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | DECIMAL(P, S) | DECIMAL | | DATE | DATE / DATETIME | | TIME | BIGINT | | TIMESTAMP | DATETIME | | STRING | STRING / VARCHAR | | UUID | STRING / VARCHAR | | FIXED(L) | CHAR | | BINARY | VARCHAR | | LIST | ARRAY | StarRocks does not support querying Iceberg data whose data type is TIMESTAMPTZ, STRUCT, and MAP. ###### Step 4: Query the data in Apache Iceberg[​](#step-4-query-the-data-in-apache-iceberg "Direct link to Step 4: Query the data in Apache Iceberg") After an external table is created, you can query the data in Apache Iceberg by using the external table. ```sql select count(*) from iceberg_tbl; ``` #### (Deprecated) Hudi external table[​](#deprecated-hudi-external-table "Direct link to (Deprecated) Hudi external table") From v2.2.0, StarRocks allows you to query data from Hudi data lakes by using Hudi external tables, thus facilitating blazing-fast data lake analytics. This topic describes how to create a Hudi external table in your StarRocks cluster and use the Hudi external table to query data from a Hudi data lake. ##### Before you begin[​](#before-you-begin-1 "Direct link to Before you begin") Make sure that your StarRocks cluster is granted access to the Hive metastore, HDFS cluster, or bucket with which you can register Hudi tables. ##### Precautions[​](#precautions-1 "Direct link to Precautions") * Hudi external tables for Hudi are read-only and can be used only for queries. * StarRocks supports querying Copy on Write and Merge On Read tables (MOR tables are supported from v2.5). For the differences between these two types of tables, see [Table & Query Types](https://hudi.apache.org/docs/table_types/). * StarRocks supports the following two query types of Hudi: Snapshot Queries and Read Optimized Queries (Hudi only supports performing Read Optimized Queries on Merge On Read tables). Incremental Queries are not supported. For more information about the query types of Hudi, see [Table & Query Types](https://hudi.apache.org/docs/next/table_types/#query-types). * StarRocks supports the following compression formats for Hudi files: gzip, zstd, LZ4, and Snappy. The default compression format for Hudi files is gzip. * StarRocks cannot synchronize schema changes from Hudi managed tables. For more information, see [Schema Evolution](https://hudi.apache.org/docs/schema_evolution/). If the schema of a Hudi managed table is changed, you must delete the associated Hudi external table from your StarRocks cluster and then re-create that external table. ##### Procedure[​](#procedure-1 "Direct link to Procedure") ###### Step 1: Create and manage Hudi resources[​](#step-1-create-and-manage-hudi-resources "Direct link to Step 1: Create and manage Hudi resources") You must create Hudi resources in your StarRocks cluster. The Hudi resources are used to manage the Hudi databases and external tables that you create in your StarRocks cluster. ###### Create a Hudi resource[​](#create-a-hudi-resource "Direct link to Create a Hudi resource") Execute the following statement to create a Hudi resource named `hudi0`: ```sql CREATE EXTERNAL RESOURCE "hudi0" PROPERTIES ( "type" = "hudi", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` The following table describes the parameters. | Parameter | Description | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | type | The type of the Hudi resource. Set the vaue to hudi. | | hive.metastore.uris | The Thrift URI of the Hive metastore to which the Hudi resource connects. After connecting the Hudi resource to a Hive metastore, you can create and manage Hudi tables by using Hive. The Thrift URI is in the `:` format. The default port number is 9083. | From v2.3 onwards, StarRocks allows changing the `hive.metastore.uris` value of a Hudi resource. For more information, see [ALTER RESOURCE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/ALTER_RESOURCE.md). ###### View Hudi resources[​](#view-hudi-resources "Direct link to View Hudi resources") Execute the following statement to view all Hudi resources that are created in your StarRocks cluster: ```sql SHOW RESOURCES; ``` ###### Delete a Hudi resource[​](#delete-a-hudi-resource "Direct link to Delete a Hudi resource") Execute the following statement to delete the Hudi resource named `hudi0`: ```sql DROP RESOURCE "hudi0"; ``` > Note: > > Deleting a Hudi resource causes unavailability of all Hudi external tables that are created by using that Hudi resource. However, the deletion does not affect your data stored in Hudi. If you still want to query your data from Hudi by using StarRocks, you must re-create Hudi resources, Hudi databases, and Hudi external tables in your StarRocks cluster. ###### Step 2: Create Hudi databases[​](#step-2-create-hudi-databases "Direct link to Step 2: Create Hudi databases") Execute the following statement to create and open a Hudi database named `hudi_test` in your StarRocks cluster: ```sql CREATE DATABASE hudi_test; USE hudi_test; ``` > Note: > > The name that you specify for the Hudi database in your StarRocks cluster does not need to be the same as the associated database in Hudi. ###### Step 3: Create Hudi external tables[​](#step-3-create-hudi-external-tables "Direct link to Step 3: Create Hudi external tables") Execute the following statement to create a Hudi external table named `hudi_tbl` in the `hudi_test` Hudi database: ```sql CREATE EXTERNAL TABLE `hudi_tbl` ( `id` bigint NULL, `data` varchar(200) NULL ) ENGINE=HUDI PROPERTIES ( "resource" = "hudi0", "database" = "hudi", "table" = "hudi_table" ); ``` The following table describes the parameters. | Parameter | Description | | --------- | ------------------------------------------------------------------------------------------------- | | ENGINE | The query engine of the Hudi external table. Set the value to `HUDI`. | | resource | The name of the Hudi resource in your StarRocks cluster. | | database | The name of the Hudi database to which the Hudi external table belongs in your StarRocks cluster. | | table | The Hudi managed table with which the Hudi external table is associated. | > Note: > > * The name that you specify for the Hudi external table does not need to be the same as the associated Hudi managed table. > > * The columns in the Hudi external table must have the same names but can be in a different sequence compared to their counterpart columns in the associated Hudi managed table. > > * You can select some or all columns from the associated Hudi managed table and create only the selected columns in the Hudi external table. The following table lists the mapping between the data types supported by Hudi and the data types supported by StarRocks. | Data types supported by Hudi | Data types supported by StarRocks | | ------------------------------- | --------------------------------- | | BOOLEAN | BOOLEAN | | INT | TINYINT/SMALLINT/INT | | DATE | DATE | | TimeMillis/TimeMicros | TIME | | TimestampMillis/TimestampMicros | DATETIME | | LONG | BIGINT | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | STRING | CHAR/VARCHAR | | ARRAY | ARRAY | | DECIMAL | DECIMAL | > **Note** > > StarRocks does not support querying data of the STRUCT or MAP type, nor does it support querying data of the ARRAY type in Merge On Read tables. ###### Step 4: Query data from a Hudi external table[​](#step-4-query-data-from-a-hudi-external-table "Direct link to Step 4: Query data from a Hudi external table") After you create a Hudi external table associated with a specific Hudi managed table, you do not need to load data into the Hudi external table. To query data from Hudi, execute the following statement: ```sql SELECT COUNT(*) FROM hudi_tbl; ``` #### (Deprecated) MySQL external table[​](#deprecated-mysql-external-table "Direct link to (Deprecated) MySQL external table") In the star schema, data is generally divided into dimension tables and fact tables. Dimension tables have less data but involve UPDATE operations. Currently, StarRocks does not support direct UPDATE operations (update can be implemented by using the Unique Key table). In some scenarios, you can store dimension tables in MySQL for direct data read. To query MySQL data, you must create an external table in StarRocks and map it to the table in your MySQL database. You need to specify the MySQL connection information when creating the table. ```sql CREATE EXTERNAL TABLE mysql_external_table ( k1 DATE, k2 INT, k3 SMALLINT, k4 VARCHAR(2048), k5 DATETIME ) ENGINE=mysql PROPERTIES ( "host" = "127.0.0.1", "port" = "3306", "user" = "mysql_user", "password" = "mysql_passwd", "database" = "mysql_db_test", "table" = "mysql_table_test" ); ``` Parameters: * **host**: the connection address of the MySQL database * **port**: the port number of the MySQL database * **user**: the username to log in to MySQL * **password**: the password to log in to MySQL * **database**: the name of the MySQL database * **table**: the name of the table in the MySQL database --- ### Feature Support: Data Lake Analytics From v2.3 onwards, StarRocks supports managing external data sources and analyzing data in data lakes via external catalogs. This document outlines the feature support for external catalogs and the supported version of the features involved. #### Universal features[​](#universal-features "Direct link to Universal features") This section lists the universal features of the External Catalog feature, including storage systems, file readers, credentials, privileges, and Data Cache. ##### External storage systems[​](#external-storage-systems "Direct link to External storage systems") | Storage System | Supported Version | | ----------------------- | ----------------- | | HDFS | v2.3+ | | AWS S3 | v2.3+ | | Microsoft Azure Storage | v3.0+ | | Google GCS | v3.0+ | | Alibaba Cloud OSS | v3.1+ | | Huawei Cloud OBS | v3.1+ | | Tencent Cloud COS | v3.1+ | | Volcengine TOS | v3.1+ | | Kingsoft Cloud KS3 | v3.1+ | | MinIO | v3.1+ | | Ceph S3 | v3.1+ | In addition to the native support for the storage systems listed above, StarRocks also supports the following types of object storage services: * **HDFS-compatible object storage services such as COS Cloud HDFS, OSS-HDFS, and OBS PFS** * **Description**: You need to specify the object storage URI prefix in the BE configuration item `fallback_to_hadoop_fs_list`, and upload the .jar package provided by the cloud vendor to the directory **/lib/hadoop/hdfs/**. Note that you must create the external catalog using the prefix you specified in `fallback_to_hadoop_fs_list`. * **Supported Version(s)**: v3.1.9+, v3.2.4+ * **S3-compatible object storage services other than those listed above** * **Description**: You need to specify the object storage URI prefix in the BE configuration item `s3_compatible_fs_list`. Note that you must create the external catalog using the prefix you specified in `s3_compatible_fs_list`. * **Supported Version(s)**: v3.1.9+, v3.2.4+ ##### Compression formats[​](#compression-formats "Direct link to Compression formats") This section only lists the compression formats supported by each file format. For the file formats supported by each external catalog, please refer to the section on the corresponding external catalog. | File Format | Compression Formats | | ------------ | ----------------------------------------------------------------------------------------------- | | Parquet | NO\_COMPRESSION, SNAPPY, LZ4, ZSTD, GZIP, LZO (v3.1.5+) | | ORC | NO\_COMPRESSION, ZLIB, SNAPPY, LZO, LZ4, ZSTD | | Text | NO\_COMPRESSION, LZO (v3.1.5+) | | Avro | NO\_COMPRESSION (v3.2.1+), DEFLATE (v3.2.1+), SNAPPY (v3.2.1+), BZIP2 (v3.2.1+) | | RCFile | NO\_COMPRESSION (v3.2.1+), DEFLATE (v3.2.1+), SNAPPY (v3.2.1+), GZIP (v3.2.1+) | | SequenceFile | NO\_COMPRESSION (v3.2.1+), DEFLATE (v3.2.1+), SNAPPY (v3.2.1+), BZIP2 (v3.2.1+), GZIP (v3.2.1+) | note The Avro, RCFile, and SequenceFile file formats are read by Java Native Interface (JNI) instead of the native readers within StarRocks. Therefore, the read performance for these file formats may not be as good as that of Parquet and ORC. ##### Management, credential, and access control[​](#management-credential-and-access-control "Direct link to Management, credential, and access control") | Feature | Description | Supported Version(s) | | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | | Information Schema | Supports Information Schema for external catalogs. | v3.2+ | | Data lake access control | Supports StarRocks' native RBAC model for external catalogs. You can manage the privileges of databases, tables, and views (currently, Hive views and Iceberge views only) in external catalogs just like those in the default catalog of StarRocks. | v3.0+ | | Reuse external services on Apache Ranger | Supports reusing the external service (such as the Hive Service) on Apache Ranger for access control. | v3.1.9+ | | Kerberos authentication | Supports Kerberos authentication for HDFS or Hive Metastore. | v2.3+ | ##### Data Cache[​](#data-cache "Direct link to Data Cache") | Feature | Description | Supported Version(s) | | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------- | | Data Cache (Block Cache) | From v2.5 onwards, StarRocks supported the Data Cache feature (then called Block Cache) implemented using CacheLib, which led to limited optimization potential for its extensibility. Starting from v3.0, StarRocks refactored the cache implementation and added new features to Data Cache, resulting in better performance with each subsequent version. | v2.5+ | | Data rebalancing among local disks | Supports data rebalancing strategy to ensure that data skew is controlled under 10%. | v3.2+ | | Replace Block Cache with Data Cache | **Parameter changes**
BE Configurations:- Replace `block_cache_enable` with `datacache_enable`.
- Replace `block_cache_mem_size` with `datacache_mem_size`.
- Replace `block_cache_disk_size` with `datacache_disk_size`.
- Replace `block_cache_disk_path` with `datacache_disk_path`.
- Replace `block_cache_meta_path` with `datacache_meta_path`.
- Replace `block_cache_block_size` with `datacache_block_size`.Session Variables:- Replace `enable_scan_block_cache` with `enable_scan_datacache`. * Replace `enable_populate_block_cache` with `enable_populate_datacache`.After the cluster is upgraded to a version where Data Cache is available, the Block Cache parameters still take effect. The new parameters will override the old ones once Data Cache is enabled. The mixed usage of both groups of parameters is not allowed. Otherwise, some parameters will not take effect. | v3.2+ | | New metrics for API that monitors Data Cache | Supports an individual API that monitors Data Cache including the cache capacity and hits. You can view Data Cache metrics via the interface `http://${BE_HOST}:${BE_HTTP_PORT}/api/datacache/stat`. | v3.2.3+ | | Memory Tracker for Data Cache | Supports Memory Tracker for Data Cache. You can view the memory-related metrics via the interface `http://${BE_HOST}:${BE_HTTP_PORT}/mem_tracker`. | v3.1.8+ | | Data Cache Warmup | By executing CACHE SELECT, you can proactively populate the cache with the desired data from remote storage in advance to prevent the first query from taking too much time fetching the data. CACHE SELECT will not print data or incur calculations. It only fetches data. | v3.3+ | #### Hive Catalog[​](#hive-catalog "Direct link to Hive Catalog") ##### Metadata[​](#metadata "Direct link to Metadata") While executing queries against Hive data through Hive catalogs, StarRocks will cache the table metadata, to reduce the costs from frequent access to the remote storage. This mechanism ensures query performance while maintaining data freshness through the asynchronous refresh and expiration policy. ##### Cached metadata[​](#cached-metadata "Direct link to Cached metadata") StarRocks will cache the following metadata during queries: * **Table- or partition-level metadata** * Content: * Table information: database, table schema, column names, and partition keys * Partition information: partition list, and partition location * Influence: detecting the table existence (whether a table is deleted and/or recreated) * Catalog properties: * `enable_metastore_cache`: Controls whether to enable the metastore cache. Default value: `true`. * `metastore_cache_refresh_interval_sec`: Controls the time interval at which the cached metadata is considered fresh. Default value: `60`. Unit: Seconds. * Location: Metastore (HMS or Glue) * **Partition name list** * Content: List of partition names used to find and prune partitions. Although the partition name list has been collected as the partition information in the above section, there is a separate configuration to enable or disable this feature under certain circumstances. * Influence: detecting the partition existence (whether there is a new partition or a partition is deleted and/or recreated) * Catalog properties: * `enable_cache_list_names`: Controls whether to enable the partition name list cache. Default value: `true`. * `metastore_cache_refresh_interval_sec`: Controls the time interval at which the cached metadata is considered fresh. Default value: `60`. Unit: Seconds. * Location: Metastore (HMS or Glue) * **File-level metadata** * Content: Paths to files under the partition folder. * Influence: Load data into an existing partition. * Catalog properties: * `enable_remote_file_cache`: Controls whether to enable the metadata cache for files in the remote storage. Default value: `true`. * `remote_file_cache_refresh_interval_sec`: Controls the time interval at which the file metadata is considered fresh. Default value: `60`. Unit: Seconds. * `remote_file_cache_memory_ratio`: Controls the ratio of memory that can be used for the file metadata cache. Default value: `0.1` (10%). * Location: Remote storage (HDFS or S3) ##### Asynchronous update policy[​](#asynchronous-update-policy "Direct link to Asynchronous update policy") The following FE configuration item controls the asynchronous metadata update policy: | Configuration item | Default | Description | | -------------------------------------------------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | enable\_background\_refresh\_connector\_metadata | `true` in v3.0
`false` in v2.5 | Whether to enable the periodic metadata cache refresh. After it is enabled, StarRocks polls the metastore, and refreshes the cached metadata of the frequently accessed external catalogs to perceive data changes. `true` indicates to enable the Hive metadata cache refresh, and `false` indicates to disable it. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | | background\_refresh\_metadata\_interval\_millis | `600000` (10 minutes) | The interval between two consecutive metadata cache refreshes. Unit: millisecond. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | | background\_refresh\_metadata\_time\_secs\_since\_last\_access\_secs | `86400` (24 hours) | The expiration time of a metadata cache refresh task. For the external catalog that has been accessed, if it has not been accessed for more than the specified time, StarRocks stops refreshing its cached metadata. For the external catalog that has not been accessed, StarRocks will not refresh its cached metadata. Unit: second. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | ##### Metadata cache behavior[​](#metadata-cache-behavior "Direct link to Metadata cache behavior") This section uses the default behavior to explain the metadata behavior during metadata updates and queries. By default, when a table is queried, StarRocks caches the metadata of the table, partitions, and files, and keeps it active for the next 24 hours. During the 24 hours, the system will ensure that the cache is refreshed at least every 10 minutes (note that 10 minutes is the estimated time for a metadata refresh round. If there are excessive external tables that are pending metadata refresh, the overall metadata refresh interval may be longer than 10 minutes). If a table has not been accessed for more than 24 hours, StarRocks discards the associated metadata. In other words, any query you make within 24 hours will, at worst, use metadata from 10 minutes ago. ![Metadata Behavior](/assets/images/hive_metadata_behavior-8456001517ecaccbe7d64c7a5d634e02.png) In details: 1. Suppose the first query involves the partition `P1` of table `A`. StarRocks caches table-level metadata, partition name lists, and file path information under `P1`. The cache is synchronously populated while the query is executed. 2. If a second query is submitted within 60 seconds after the cache is populated, and hits the partition `P1` of table `A`, StarRocks uses the metadata cache directly, and at this point StarRocks considers all cached metadata to be fresh (`metastore_cache_refresh_interval_sec` and `remote_file_cache_refresh_interval_sec` control the time window in which StarRocks considers metadata to be fresh). 3. If a third query is submitted after 90 seconds, and hits the partition `P1` of table `A`, StarRocks will still use the metadata cache directly to complete the query. However, since it has been more than 60 seconds since the last metadata refresh, StarRocks will consider the metadata to be expired. So StarRocks will start an asynchronous refresh for the expired metadata. The asynchronous refresh will not affect the result of the current query because the query will still use the outdated metadata. 4. Because the partition `P1` of table `A` has been queried, it is estimated that the metadata will be refreshed every 10 minutes (controlled by `background_refresh_metadata_interval_millis`) for the next 24 hours (controlled by `background_refresh_metadata_time_secs_since_last_access_secs`). The actual interval between rounds of the metadata refresh also depends on the overall pending refresh tasks within the system. 5. If table `A` is not involved in any query within 24 hours, StarRocks will remove its metadata cache after 24 hours. ##### Best practices[​](#best-practices "Direct link to Best practices") Hive Catalog's support for Hive Metastore (HMS) and AWS Glue mostly overlaps except that the automatic incremental update feature for HMS is not recommended. The default configuration is recommended in most cases. The performance of metadata retrieval largely depends on the performance of the user's HMS or HDFS NameNode. Please consider all factors and base your judgment on test results. * **\[Default and Recommended] Best performance with a tolerance of minute-level data inconsistency** * **Configuration**: You can use the default setting. Data updated within 10 minutes (by default) is not visible. Old data will be returned to queries within this duration. * **Advantage**: Best query performance. * **Disadvantage**: Data inconsistency caused by latency. * **Supported Version(s)**: v2.5.5+ (Disabled by default in v2.5 and enabled by default in v3.0+) * **Instant visibility of newly loaded data (files) without manual refresh** * **Configuration**: Disable the cache for the metadata of the underlying data files by setting the catalog property `enable_remote_file_cache` to `false`. * **Advantage**: Visibility of file changes with no delay. * **Disadvantage**: Lower performance when the file metadata cache is disabled. Each query must access the file list. * **Supported Version(s)**: v2.5.5+ * **Instant visibility of partition changes without manual refresh** * **Configuration**: Disable the cache for the Hive partition names by setting the catalog property `enable_cache_list_names` to `false`. * **Advantage**: Visibility of partition changes with no delay * **Disadvantage**: Lower performance when the partition name cache is disabled. Each query must access the partition list. * **Supported Version(s)**: v2.5.5+ tip If you demand real-time updates on the data changes whilst the performance of your HMS is not optimized, you can enable the cache, disable the automatic incremental update, and manually refresh the metadata (using REFRESH EXTERNAL TABLE) via a scheduling system whenever there is a data change upstream. ##### Storage system[​](#storage-system "Direct link to Storage system") | Feature | Description | Supported Version(s) | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- | | Recursive sub-directory listing | Enable recursive sub-directory listing by setting the Catalog property `enable_recursive_listing` to `true`. When recursive listing is enabled, StarRocks will read data from a table and its partitions and from the subdirectories within the physical locations of the table and its partitions. This feature is designed to address the issue of multi-layer nested directories. | v2.5.9+
v3.0.4+ (Disabled by default in v2.5 and v3.0, and enabled by default in v3.1+) | ##### File formats and data types[​](#file-formats-and-data-types "Direct link to File formats and data types") ###### File formats[​](#file-formats "Direct link to File formats") | Feature | Supported File Formats | | ------- | ---------------------------------------------- | | Read | Parquet, ORC, TEXT, Avro, RCFile, SequenceFile | | Sink | Parquet (v3.2+), ORC (v3.3+), TEXT (v3.3+) | ###### Data types[​](#data-types "Direct link to Data types") INTERVAL, BINARY, and UNION types are not supported. TEXT-formatted Hive table does not support MAP and STRUCT types. ###### Table types[​](#table-types "Direct link to Table types") Reading Hive transactional tables is not supported. ##### Hive views[​](#hive-views "Direct link to Hive views") StarRocks supports querying Hive views from v3.1.0 onwards. note While StarRocks executes queries against a Hive view, it will try to parse the definition of the view using the syntax of StarRocks and Trino. An error will be returned if StarRocks cannot parse the definition of the view. There is a possibility that StarRocks failed to parse the Hive views created with functions exclusive to Hive or Spark. ##### Query statistics interfaces[​](#query-statistics-interfaces "Direct link to Query statistics interfaces") | Feature | Supported Version(s) | | ------------------------------------------------------------- | -------------------- | | Supports SHOW CREATE TABLE to view Hive table schema | v3.0+ | | Supports ANALYZE to collect statistics | v3.2+ | | Supports collecting histograms and STRUCT subfield statistics | v3.3+ | ##### Data sinking[​](#data-sinking "Direct link to Data sinking") | Feature | Supported Version(s) | Note | | ---------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CREATE DATABASE | v3.2+ | You can choose to specify the location for a database created in Hive or not. If you do not specify the location for the database, you will need to specify the location for the tables created under the database. Otherwise, an error will be returned. If you have specified the location for the database, tables without the location specified will inherit the location of the database. And if you have specified locations for both the database and the table, the table's location will take effect eventually. | | CREATE TABLE | v3.2+ | For both partitioned and non-partitioned tables. | | CREATE TABLE AS SELECT | v3.2+ | | | INSERT INTO/OVERWRITE | v3.2+ | For both partitioned and non-partitioned tables. | | CREATE TABLE LIKE | v3.2.4+ | | | Sink file size | v3.3+ | You can define the maximum size of each data file to be sunk using the session variable `connector_sink_target_max_file_size`. | #### Iceberg Catalog[​](#iceberg-catalog "Direct link to Iceberg Catalog") ##### Metadata[​](#metadata-1 "Direct link to Metadata") While executing queries against Iceberg data through Iceberg catalogs, StarRocks will cache the table metadata, to reduce the costs from frequent access to the remote storage. This mechanism ensures query performance while maintaining data freshness through the asynchronous refresh and expiration policy. ##### Cached metadata[​](#cached-metadata-1 "Direct link to Cached metadata") StarRocks will cache the following metadata during queries: * **Metadata pointer cache** * Content: JSON file of the metadata pointer * Snapshot ID * Location of the manifest lists * Influence: detecting data changes (If a data changes occurs, the snapshot ID will change.) * Catalog properties: * `enable_iceberg_metadata_cache`: Controls whether to enable the Iceberg metadata cache. Default value: `true`. * `iceberg_table_cache_refresh_interval_sec`: Controls the time interval at which the cached metadata is considered fresh. Default value: `60`. Unit: Seconds. * **Metadata cache** * Content: * Manifest for data file path * Manifest for delete file path * Database * Partition (for materialized view rewrite) * Influence: * Will not affect the data freshness for queries because the manifests for data or delete files cannot be changed. * May affect materialized view rewrite, causing queries to miss the materialized view. The partition metadata will be deleted when the snapshot ID is refreshed. Therefore, the new snapshot ID lacks partition metadata, causing misses of the materialized view rewrite. * Catalog properties: * `enable_cache_list_names`: Controls whether to enable the partition name list cache. Default value: `true`. * `metastore_cache_refresh_interval_sec`: Controls the time interval at which the cached metadata is considered fresh. Default value: `60`. Unit: Seconds. * `iceberg_data_file_cache_memory_usage_ratio`: Controls the ratio of memory that can be used for the data file metadata cache. Default value: `0.1` (10%). * `iceberg_delete_file_cache_memory_usage_ratio`: Controls the ratio of memory that can be used for the delete file metadata cache. Default value: `0.1` (10%). ##### Asynchronous update policy[​](#asynchronous-update-policy-1 "Direct link to Asynchronous update policy") The following FE configuration item controls the asynchronous metadata update policy: | Configuration item | Default | Description | | -------------------------------------------------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | enable\_background\_refresh\_connector\_metadata | `true` in v3.0
`false` in v2.5 | Whether to enable the periodic metadata cache refresh. After it is enabled, StarRocks polls the metastore, and refreshes the cached metadata of the frequently accessed external catalogs to perceive data changes. `true` indicates to enable the Hive metadata cache refresh, and `false` indicates to disable it. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | | background\_refresh\_metadata\_interval\_millis | `600000` (10 minutes) | The interval between two consecutive metadata cache refreshes. Unit: millisecond. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | | background\_refresh\_metadata\_time\_secs\_since\_last\_access\_secs | `86400` (24 hours) | The expiration time of a metadata cache refresh task. For the external catalog that has been accessed, if it has not been accessed for more than the specified time, StarRocks stops refreshing its cached metadata. For the external catalog that has not been accessed, StarRocks will not refresh its cached metadata. Unit: second. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | ##### Metadata cache behavior[​](#metadata-cache-behavior-1 "Direct link to Metadata cache behavior") This section uses the default behavior to explain the metadata behavior during metadata updates and queries. By default, when a table is queried, StarRocks caches the metadata of the table, and keeps it active for the next 24 hours. During the 24 hours, the system will ensure that the cache is refreshed at least every 10 minutes (note that 10 minutes is the estimated time for a metadata refresh round. If there are excessive external tables that are pending metadata refresh, the overall metadata refresh interval may be longer than 10 minutes). If a table has not been accessed for more than 24 hours, StarRocks discards the associated metadata. In other words, any query you make within 24 hours will, at worst, use metadata from 10 minutes ago. ![Metadata Behavior](/assets/images/iceberg_metadata_behavior-7544cc5cb78ab4260b20ac0b21f7350c.png) In details: 1. Suppose the first query involves the table `A`. StarRocks caches its latest snapshot and metadata. The cache is synchronously populated while the query is executed. 2. If a second query is submitted within 60 seconds after the cache is populated, and hits the table `A`, StarRocks uses the metadata cache directly, and at this point StarRocks considers all cached metadata to be fresh (`iceberg_table_cache_refresh_interval_sec` controls the time window in which StarRocks considers metadata to be fresh). 3. If a third query is submitted after 90 seconds, and hits the table `A`, StarRocks will still use the metadata cache directly to complete the query. However, since it has been more than 60 seconds since the last metadata refresh, StarRocks will consider the metadata to be expired. So StarRocks will start an asynchronous refresh for the expired metadata. The asynchronous refresh will not affect the result of the current query because the query will still use the outdated metadata. 4. Because the table `A` has been queried, it is estimated that the metadata will be refreshed every 10 minutes (controlled by `background_refresh_metadata_interval_millis`) for the next 24 hours (controlled by `background_refresh_metadata_time_secs_since_last_access_secs`). The actual interval between rounds of the metadata refresh also depends on the overall pending refresh tasks within the system. 5. If table `A` is not involved in any query within 24 hours, StarRocks will remove its metadata cache after 24 hours. ##### Best practices[​](#best-practices-1 "Direct link to Best practices") Iceberg Catalog supports HMS, Glue, and Tabular as its metastore. The default configuration is recommended in most cases. Please note that the default value of the session variable `enable_iceberg_metadata_cache` has been changed to accommodate different scenarios: * From v3.2.1 to v3.2.3, this parameter is set to `true` by default, regardless of what metastore service is used. * In v3.2.4 and later, if the Iceberg cluster uses AWS Glue as metastore, this parameter still defaults to `true`. However, if the Iceberg cluster uses other metastore services such as Hive metastore, this parameter defaults to `false`. * From v3.3.0 onwards, the default value of this parameter is set to `true` again because StarRocks supports the new Iceberg metadata framework. Iceberg Catalog and Hive Catalog now use the same metadata polling mechanism and FE configuration item `background_refresh_metadata_interval_millis`. | Feature | Supported Version(s) | | ----------------------------------------------------------------------------------------------------- | -------------------- | | Distributed metadata plan (Recommended for scenarios with a large volume of metadata) | v3.3+ | | Manifest Cache (Recommended for scenarios with a small volume of metadata but high demand on latency) | v3.3+ | From v3.3.0 onwards, StarRocks supports the metadata reading and caching policies described above. The system will automatically adjust the choice of policy according to the machines in your cluster. Usually, you do not need to change it. Since metadata caching is enabled, it is possible that metadata freshness may be compromised due to performance considerations. Therefore, you can adjust it according to your specific query requirements: * **\[Default and recommended] Optimal performance with tolerance of minute-level data inconsistencies** * **Setting**: No additional setting is required. By default, data updated within 10 minutes is not visible. During this time, queries will return old data. * **Advantages**: Best query performance. * **Disadvantage**: data inconsistency caused by delays. * **New data files generated by import and partition additions or deletions are immediately visible, and no manual refresh is required** * **Setting**: Set the Catalog property `iceberg_meta_cache_ttl_sec` to `0` to allow StarRocks to fetch a new snapshot for each query. * **Advantages**: File and partition changes are visible without delay. * **Disadvantage**: Lower performance due to the snapshot fetching behavior for each query. ##### File formats[​](#file-formats-1 "Direct link to File formats") | Feature | Supported File Formats | | ------- | ---------------------- | | Read | Parquet, ORC | | Sink | Parquet | * Both Parquet-formatted and ORC-formatted Iceberg V1 tables support position deletes and equality deletes. * ORC-formatted Iceberg V2 tables support position deletes from v3.0.0, and Parquet-formatted ones support position deletes from v3.1.0. * ORC-formatted Iceberg V2 tables support equality deletes from v3.1.8 and v3.2.3, and Parquet-formatted ones support equality deletes from v3.2.5. ##### Iceberg views[​](#iceberg-views "Direct link to Iceberg views") StarRocks supports Iceberg views on REST from v3.3.2 and on Hive from v3.4.1. Currently, only Iceberg views created through StarRocks are supported. Starting with v3.5, adding StarRocks syntax style definitions to existing Iceberg views is supported. ##### Query statistics interfaces[​](#query-statistics-interfaces-1 "Direct link to Query statistics interfaces") | Feature | Supported Version(s) | | ------------------------------------------------------------- | -------------------- | | Supports SHOW CREATE TABLE to view Iceberg table schema | v3.0+ | | Supports ANALYZE to collect statistics | v3.2+ | | Supports collecting histograms and STRUCT subfield statistics | v3.3+ | ##### Data sinking[​](#data-sinking-1 "Direct link to Data sinking") | Feature | Supported Version(s) | Note | | ---------------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CREATE DATABASE | v3.1+ | You can choose to specify the location for a database created in Iceberg or not. If you do not specify the location for the database, you will need to specify the location for the tables created under the database. Otherwise, an error will be returned. If you have specified the location for the database, tables without the location specified will inherit the location of the database. And if you have specified locations for both the database and the table, the table's location will take effect eventually. | | CREATE TABLE | v3.1+ | Supports partitioned and non-partitioned tables. Starting from v4.0, supports creating tables with hidden partitions. | | CREATE TABLE AS SELECT | v3.1+ | | | INSERT INTO/OVERWRITE | v3.1+ | For both partitioned and non-partitioned tables. | ##### Miscellaneous supports[​](#miscellaneous-supports "Direct link to Miscellaneous supports") | Feature | Supported Version(s) | | -------------------------------------------------------------------------------------------- | ---------------------------------- | | Supports reading TIMESTAMP-type partition formats `yyyy-MM-ddTHH:mm` and `yyyy-MM-dd HH:mm`. | v2.5.19+
v3.1.9+
v3.2.3+ | | Supports Iceberg metadata table | v3.4.1+ | | Supports Iceberg TimeTravel | v3.4.0+ | #### Hudi Catalog[​](#hudi-catalog "Direct link to Hudi Catalog") * StarRocks supports querying the Parquet-formatted data in Hudi, and supports SNAPPY, LZ4, ZSTD, GZIP, and NO\_COMPRESSION compression formats for Parquet files. * StarRocks fully supports Hudi's Copy On Write (COW) tables and Merge On Read (MOR) tables. * StarRocks supports SHOW CREATE TABLE to view Hudi table schema from v3.0.0 onwards. * StarRocks v3.5.0 supports Hudi 0.15.0. #### Delta Lake Catalog[​](#delta-lake-catalog "Direct link to Delta Lake Catalog") * StarRocks supports querying the Parquet-formatted data in Delta Lake, and supports SNAPPY, LZ4, ZSTD, GZIP, and NO\_COMPRESSION compression formats for Parquet files. * StarRocks does not support querying the MAP-type and STRUCT-type data in Delta Lake. * StarRocks supports SHOW CREATE TABLE to view Delta Lake table schema from v3.0.0 onwards. * Currently, Delta Lake catalogs support the following table features: * V2 Checkpoint (From v3.3.0 onwards) * Timestamp without Timezone (From v3.3.1 onwards) * Column mapping (From v3.3.6 onwards) * Deletion Vector (From v3.4.1 onwards) #### JDBC Catalog[​](#jdbc-catalog "Direct link to JDBC Catalog") | Catalog type | Supported Version(s) | | ------------ | -------------------- | | MySQL | v3.0+ | | PostgreSQL | v3.0+ | | ClickHouse | v3.3+ | | Oracle | v3.2.9+ | | SQL Server | v3.2.9+ | ##### MySQL[​](#mysql "Direct link to MySQL") | Feature | Supported Version(s) | | -------------- | -------------------- | | Metadata cache | v3.3+ | ###### Data type correspondance[​](#data-type-correspondance "Direct link to Data type correspondance") | MySQL | StarRocks | Supported Version(s) | | ----------------- | ------------------- | -------------------- | | BOOLEAN | BOOLEAN | v2.3+ | | BIT | BOOLEAN | v2.3+ | | SIGNED TINYINT | TINYINT | v2.3+ | | UNSIGNED TINYINT | SMALLINT | v3.0.6+
v3.1.2+ | | SIGNED SMALLINT | SMALLINT | v2.3+ | | UNSIGNED SMALLINT | INT | v3.0.6+
v3.1.2+ | | SIGNED INTEGER | INT | v2.3+ | | UNSIGNED INTEGER | BIGINT | v3.0.6+
v3.1.2+ | | SIGNED BIGINT | BIGINT | v2.3+ | | UNSIGNED BIGINT | LARGEINT | v3.0.6+
v3.1.2+ | | FLOAT | FLOAT | v2.3+ | | REAL | FLOAT | v3.0.1+ | | DOUBLE | DOUBLE | v2.3+ | | DECIMAL | DECIMAL32 | v2.3+ | | CHAR | VARCHAR(columnsize) | v2.3+ | | VARCHAR | VARCHAR | v2.3+ | | TEXT | VARCHAR(columnsize) | v3.0.1+ | | DATE | DATE | v2.3+ | | TIME | TIME | v3.1.9+
v3.2.4+ | | TIMESTAMP | DATETIME | v2.3+ | ##### PostgreSQL[​](#postgresql "Direct link to PostgreSQL") ###### Data type correspondance[​](#data-type-correspondance-1 "Direct link to Data type correspondance") | PGSQL | StarRocks | Supported Version(s) | | --------- | ------------------- | -------------------- | | BIT | BOOLEAN | v2.3+ | | SMALLINT | SMALLINT | v2.3+ | | INTEGER | INT | v2.3+ | | BIGINT | BIGINT | v2.3+ | | REAL | FLOAT | v2.3+ | | DOUBLE | DOUBLE | v2.3+ | | NUMERIC | DECIMAL32 | v2.3+ | | CHAR | VARCHAR(columnsize) | v2.3+ | | VARCHAR | VARCHAR | v2.3+ | | TEXT | VARCHAR(columnsize) | v2.3+ | | DATE | DATE | v2.3+ | | TIMESTAMP | DATETIME | v2.3+ | | UUID | VARBINARY | v3.5.3+ | ##### ClickHouse[​](#clickhouse "Direct link to ClickHouse") Supported from v3.3.0 onwards. ##### Oracle[​](#oracle "Direct link to Oracle") Supported from v3.2.9 onwards. ##### SQL Server[​](#sql-server "Direct link to SQL Server") Supported from v3.2.9 onwards. #### Elasticsearch Catalog[​](#elasticsearch-catalog "Direct link to Elasticsearch Catalog") Elasticsearch Catalog is supported from v3.1.0 onwards. #### Paimon Catalog[​](#paimon-catalog "Direct link to Paimon Catalog") Paimon Catalog is supported from v3.1.0 onwards. #### MaxCompute Catalog[​](#maxcompute-catalog "Direct link to MaxCompute Catalog") MaxCompute Catalog is supported from v3.3.0 onwards. #### Kudu Catalog[​](#kudu-catalog "Direct link to Kudu Catalog") Kudu Catalog is supported from v3.3.0 onwards. --- ### File external table File external table is a special type of external table. It allows you to directly query Parquet and ORC data files in external storage systems without loading data into StarRocks. In addition, file external tables do not rely on a metastore. In the current version, StarRocks supports the following external storage systems: HDFS, Amazon S3, and other S3-compatible storage systems. This feature is supported from StarRocks v2.5. note * From v3.1 onwards, StarRocks supports directly loading data from files on cloud storage using the [INSERT](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-directly-from-files-in-an-external-source-using-files) command and the [FILES](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) function, thereby you do not need to create an external catalog or file external table first. Besides, FILES() can automatically infer the table schema of the files, greatly simplifying the process of data loading. * The File External Table feature was designed to help with loading data into StarRocks, NOT to perform efficient queries against external systems as a normal operation. A more performant solution would be to load the data into StarRocks. #### Limits[​](#limits "Direct link to Limits") * File external tables must be created in databases within the [default\_catalog](https://docs.starrocks.io/docs/data_source/catalog/default_catalog.md). You can run [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query catalogs created in the cluster. * Only Parquet, ORC, Avro, RCFile, and SequenceFile data files are supported. * You can only use file external tables to query data in the target data file. Data write operations such as INSERT, DELETE, and DROP are not supported. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before you create a file external table, you must configure your StarRocks cluster so that StarRocks can access the external storage system where the target data file is stored. The configurations required for a file external table are the same as those required for a Hive catalog, except that you do not need to configure a metastore. See [Hive catalog - Integration preparations](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md#integration-preparations) for more information about configurations. #### Create a database (Optional)[​](#create-a-database-optional "Direct link to Create a database (Optional)") After connecting to your StarRocks cluster, you can create a file external table in an existing database or create a new database to manage file external tables. To query existing databases in the cluster, run [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md). Then you can run `USE ` to switch to the target database. The syntax for creating a database is as follows. ```sql CREATE DATABASE [IF NOT EXISTS] ``` #### Create a file external table[​](#create-a-file-external-table "Direct link to Create a file external table") After accessing the target database, you can create a file external table in this database. ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL TABLE ( [NULL | NOT NULL] [COMMENT ""] ) ENGINE=file COMMENT ["comment"] PROPERTIES ( FileLayoutParams, StorageCredentialParams ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") | Parameter | Required | Description | | ---------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | table\_name | Yes | The name of the file external table. The naming conventions are as follows:- The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter.
- The name cannot exceed 64 characters in length. | | col\_name | Yes | The column name in the file external table. The column names in the file external table must be the same as those in the target data file but are not case-sensitive. The order of columns in the file external table can be different from that in the target data file. | | col\_type | Yes | The column type in the file external table. You need to specify this parameter based on the column type in the target data file. For more information, see [Mapping of column types](#mapping-of-column-types). | | NULL | NOT NULL | No | Whether the column in the file external table is allowed to be NULL. - NULL: NULL is allowed.
- NOT NULL: NULL is not allowed.You must specify this modifier based on the following rules:- If this parameter is not specified for the columns in the target data file, you can choose not to specify it for the columns in the file external table or specify NULL for the columns in the file external table. * If NULL is specified for the columns in the target data file, you can choose not to specify this parameter for the columns in the file external table or specify NULL for the columns in the file external table. * If NOT NULL is specified for the columns in the target data file, you must also specify NOT NULL for the columns in the file external table. | | comment | No | The comment of column in the file external table. | | ENGINE | Yes | The type of engine. Set the value to file. | | comment | No | The description of the file external table. | | PROPERTIES | Yes | - `FileLayoutParams`: specifies the path and format of the target file. This property is required.
- `StorageCredentialParams`: specifies the authentication information required for accessing object storage systems. This property is required only for AWS S3 and other S3-compatible storage systems. | ###### FileLayoutParams[​](#filelayoutparams "Direct link to FileLayoutParams") A set of parameters for accessing the target data file. ```sql "path" = "", "format" = "" "enable_recursive_listing" = "{ true | false }" "enable_wildcards" = "{ true | false }" ``` | Parameter | Required | Description | | -------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | path | Yes | The path of the data file. - If the data file is stored in HDFS, the path format is `hdfs://:/`. The default port number is 8020. If you use the default port, you do not need to specify it.
- If the data file is stored in AWS S3 or other S3-compatible storage system, the path format is `s3:////`. Note the following rules when you enter the path:- If you want to access all files in a path, end this parameter with a slash (`/`), such as `hdfs://x.x.x.x/user/hive/warehouse/array2d_parq/data/`. When you run a query, StarRocks traverses all data files under the path. It does not traverse data files by using recursion. * If you want to access a single file, enter a path that directly points to this file, such as `hdfs://x.x.x.x/user/hive/warehouse/array2d_parq/data`. When you run a query, StarRocks only scans this data file. | | format | Yes | The format of the data file. Valid values: `parquet`, `orc`, `avro`, `rctext` or `rcbinary`, and `sequence`. | | enable\_recursive\_listing | No | Specifies whether to recursively transverse all files under the current path. Default value: `true`. The value `true` specifies to recursively list subdirectories, and the value `false` specifies to ignore subdirectories. | | enable\_wildcards | No | Whether to support using wildcards (`*`) in `path`. Default value: `false`. For example, `2024-07-*` is to match all files with the `2024-07-` prefix. This parameter is supported from v3.1.9. | ###### StorageCredentialParams (Optional)[​](#storagecredentialparams-optional "Direct link to StorageCredentialParams (Optional)") A set of parameters about how StarRocks integrates with the target storage system. This parameter set is **optional**. You need to configure `StorageCredentialParams` only when the target storage system is AWS S3 or other S3-compatible storage. For other storage systems, you can ignore `StorageCredentialParams`. ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") If you need to access a data file stored in AWS S3, configure the following authentication parameters in `StorageCredentialParams`. * If you choose the instance profile-based authentication method, configure `StorageCredentialParams` as follows: ```javascript "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "" ``` * If you choose the assumed role-based authentication method, configure `StorageCredentialParams` as follows: ```javascript "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.region" = "" ``` * If you choose the IAM user-based authentication method, configure `StorageCredentialParams` as follows: ```javascript "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "" ``` | Parameter name | Required | Description | | ----------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method when you access AWS S3. Valid values: `true` and `false`. Default value: `false`. | | aws.s3.iam\_role\_arn | Yes | The ARN of the IAM role that has privileges on your AWS S3 bucket.
If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter. Then, StarRocks will assume this role when it accesses the target data file. | | aws.s3.region | Yes | The region in which your AWS S3 bucket resides. Example: us-west-1. | | aws.s3.access\_key | No | The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.secret\_key | No | The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in the AWS IAM Console, see [Authentication parameters for accessing AWS S3](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-s3). ###### S3-compatible storage[​](#s3-compatible-storage "Direct link to S3-compatible storage") If you need to access an S3-compatible storage system, such as MinIO, configure `StorageCredentialParams` as follows to ensure a successful integration: ```sql "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ---------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.enable\_ssl | Yes | Specifies whether to enable SSL connection.
Valid values: `true` and `false`. Default value: `true`. | | aws.s3.enable\_path\_style\_access | Yes | Specifies whether to enable path-style access.
Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`.
Path-style URLs use the following format: `https://s3..amazonaws.com//`. For example, if you create a bucket named `DOC-EXAMPLE-BUCKET1` in the US West (Oregon) Region, and you want to access the `alice.jpg` object in that bucket, you can use the following path-style URL: `https://s3.us-west-2.amazonaws.com/DOC-EXAMPLE-BUCKET1/alice.jpg`. | | aws.s3.endpoint | Yes | The endpoint used to connect to an S3-compatible storage system instead of AWS S3. | | aws.s3.access\_key | Yes | The access key of your IAM user. | | aws.s3.secret\_key | Yes | The secret key of your IAM user. | ###### Mapping of column types[​](#mapping-of-column-types "Direct link to Mapping of column types") The following table provides the mapping of column types between the target data file and the file external table. | Data file | File external table | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | INT/INTEGER | INT | | BIGINT | BIGINT | | TIMESTAMP | DATETIME.
Note that TIMESTAMP is converted to DATETIME without a time zone based on the time zone setting of the current session and loses some of its precision. | | STRING | STRING | | VARCHAR | VARCHAR | | CHAR | CHAR | | DOUBLE | DOUBLE | | FLOAT | FLOAT | | DECIMAL | DECIMAL | | BOOLEAN | BOOLEAN | | ARRAY | ARRAY | | MAP | MAP | | STRUCT | STRUCT | ##### Examples[​](#examples "Direct link to Examples") ###### HDFS[​](#hdfs "Direct link to HDFS") Create a file external table named `t0` to query Parquet data files stored in an HDFS path. ```sql USE db_example; CREATE EXTERNAL TABLE t0 ( name string, id int ) ENGINE=file PROPERTIES ( "path"="hdfs://x.x.x.x:8020/user/hive/warehouse/person_parq/", "format"="parquet" ); ``` ###### AWS S3[​](#aws-s3-1 "Direct link to AWS S3") Example 1: Create a file external table and use **instance profile** to access **a single Parquet file** in AWS S3. ```sql USE db_example; CREATE EXTERNAL TABLE table_1 ( name string, id int ) ENGINE=file PROPERTIES ( "path" = "s3://bucket-test/folder1/raw_0.parquet", "format" = "parquet", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` Example 2: Create a file external table and use **assumed role** to access **all the ORC files** under the target file path in AWS S3. ```sql USE db_example; CREATE EXTERNAL TABLE table_1 ( name string, id int ) ENGINE=file PROPERTIES ( "path" = "s3://bucket-test/folder1/", "format" = "orc", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::51234343412:role/role_name_in_aws_iam", "aws.s3.region" = "us-west-2" ); ``` Example 3: Create a file external table and use **IAM user** to access **all the ORC files** under the file path in AWS S3. ```sql USE db_example; CREATE EXTERNAL TABLE table_1 ( name string, id int ) ENGINE=file PROPERTIES ( "path" = "s3://bucket-test/folder1/", "format" = "orc", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` #### Query a file external table[​](#query-a-file-external-table "Direct link to Query a file external table") Syntax: ```sql SELECT FROM ``` For example, to query data from the file external table `t0` created in [Examples - HDFS](#examples), run the following command: ```plain SELECT * FROM t0; +--------+------+ | name | id | +--------+------+ | jack | 2 | | lily | 1 | +--------+------+ 2 rows in set (0.08 sec) ``` #### Manage file external tables[​](#manage-file-external-tables "Direct link to Manage file external tables") You can view the schema of the table using [DESC](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md) or drop the table using [DROP TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md). --- ## Deployment ### Deployment Overview This chapter describes how to deploy, upgrade, and downgrade a StarRocks cluster in a production environment. #### Deployment Procedure[​](#deployment-procedure "Direct link to Deployment Procedure") A summary of the deployment procedure is as follows and later topics provide the details. The deployment of StarRocks generally follows the steps outlined here: 1. Confirm the [hardware and software requirements](https://docs.starrocks.io/docs/deployment/preparation/deployment_prerequisites.md) for your StarRocks deployment. Check the prerequisites that your servers must meet before deploying StarRocks, including CPU, memory, storage, network, operating system, and dependencies. 2. [Plan your cluster size](https://docs.starrocks.io/docs/deployment/preparation/plan_cluster.md). Plan the number of FE nodes and BE/CN nodes in your cluster, and the hardware specifications of the servers. 3. [Check environment configurations](https://docs.starrocks.io/docs/deployment/preparation/environment_configurations.md). When your servers are ready, you need to check and modify some environment configurations before deploying StarRocks on them. 4. [Prepare deployment files](https://docs.starrocks.io/docs/deployment/preparation/prepare_deployment_files.md). * If you want to deploy StarRocks on x86 architecture, you can directly download and extract the software package provided on our official website. * If you want to deploy StarRocks on ARM architecture, you need to prepare the deployment files from the StarRocks Docker image. * If you want to deploy StarRocks on Kubernetes, you can skip this step. 5. Deploy StarRocks. * If you want to deploy a shared-data StarRocks cluster, which features a disaggregated storage and compute architecture, see [Deploy and use shared-data StarRocks](https://docs.starrocks.io/docs/deployment/deploy_shared_data_manually.md) for instructions. * If you want to deploy a shared-nothing StarRocks cluster, which uses local storage, you have the following options: * [Deploy StarRocks manually](https://docs.starrocks.io/docs/deployment/deploy_manually.md). * [Deploy StarRocks on Kubernetes with operator](https://docs.starrocks.io/docs/deployment/sr_operator.md). * [Deploy StarRocks on Kubernetes with Helm](https://docs.starrocks.io/docs/deployment/helm.md). 6. Perform necessary [post-deployment setup](https://docs.starrocks.io/docs/deployment/manage_deployment/post_deployment_setup.md) measures. Further setup measures are needed before your StarRocks cluster is put into production. These measures include securing the initial account and setting some performance-related system variables. #### Upgrade and Downgrade[​](#upgrade-and-downgrade "Direct link to Upgrade and Downgrade") If you plan to upgrade an existing StarRocks cluster to a later version rather than install StarRocks for the first time, see [Upgrade StarRocks](https://docs.starrocks.io/docs/deployment/manage_deployment/upgrade.md) for information about upgrade procedures and issues that you should consider before upgrading. For instructions to downgrade your StarRocks cluster, see [Downgrade StarRocks](https://docs.starrocks.io/docs/deployment/manage_deployment/downgrade.md). --- ### Deploy Shared-nothing StarRocks Manually tip The preparations for manual deployment are outlined in the [Deployment prerequisites](https://docs.starrocks.io/docs/deployment/preparation/deployment_prerequisites.md) and [Check environment configurations](https://docs.starrocks.io/docs/deployment/preparation/environment_configurations.md) documents. Please start there if you are planning a production deployment. If you are getting started with StarRocks and would like to follow one of the Quick Starts, please refer to [Quick Starts](https://docs.starrocks.io/docs/quick_start.md). This topic describes how to manually deploy a shared-nothing StarRocks cluster (in which BE is responsible for both storage and computing). For other modes of installation, see [Deployment Overview](https://docs.starrocks.io/docs/deployment.md). To deploy a shared-data StarRocks cluster (decoupled storage and computing), see [Deploy Shared-data StarRocks Manually](https://docs.starrocks.io/docs/deployment/deploy_shared_data_manually.md). #### Step 1: Start the Leader FE node[​](#step-1-start-the-leader-fe-node "Direct link to Step 1: Start the Leader FE node") The following procedures are performed on an FE instance. 1. Create a dedicated directory for metadata storage. We recommend storing metadata in a separate directory from the FE deployment files. Make sure that this directory exists and that you have write access to it. ```yaml # Replace with the metadata directory you want to create. mkdir -p ``` 2. Navigate to the directory that stores the [StarRocks FE deployment files](https://docs.starrocks.io/docs/deployment/preparation/prepare_deployment_files.md) you prepared earlier, and modify the FE configuration file **fe/conf/fe.conf**. a. Specify the metadata directory in the configuration item `meta_dir`. ```yaml # Replace with the metadata directory you have created. meta_dir = ``` b. If any of the FE ports mentioned in the [Environment Configuration Checklist](https://docs.starrocks.io/docs/deployment/preparation/environment_configurations.md#fe-ports) are occupied, you must assign valid alternatives in the FE configuration file. ```yaml http_port = aaaa # Default: 8030 rpc_port = bbbb # Default: 9020 query_port = cccc # Default: 9030 edit_log_port = dddd # Default: 9010 ``` > **CAUTION** > > If you want to deploy multiple FE nodes in a cluster, you must assign the same `http_port` to each FE node. c. If you want to enable IP address access for your cluster, you must add the configuration item `priority_networks` in the configuration file and assign a dedicated IP address (in the CIDR format) to the FE node. You can ignore this configuration item if you want to enable [FQDN access](https://docs.starrocks.io/docs/administration/management/enable_fqdn.md) for your cluster. ```yaml priority_networks = x.x.x.x/x ``` > **NOTE** > > * You can run `ifconfig` in your terminal to view the IP address(es) owned by the instance. > * From v3.3.0, StarRocks supports deployment based on IPv6. d. If you have multiple JDKs installed on the instance, and you want to use a specific JDK that is different from the one specified in the environment variable `JAVA_HOME`, you must specify the path where the chosen JDK is installed by adding the configuration item `JAVA_HOME` in the configuration file. ```yaml # Replace with the path where the chosen JDK is installed. JAVA_HOME = ``` For information about advanced configuration items, see [Parameter Configuration - FE configuration items](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md). 3. Start the FE node. * To enable IP address access for your cluster, run the following command to start the FE node: ```bash ./fe/bin/start_fe.sh --daemon ``` * To enable FQDN access for your cluster, run the following command to start the FE node: ```bash ./fe/bin/start_fe.sh --host_type FQDN --daemon ``` Note that you only need to specify the parameter `--host_type` ONCE when you start the node for the first time. > **CAUTION** > > Before starting the FE node with FQDN access enabled, make sure you have assigned hostnames for all instances in **/etc/hosts**. See [Environment Configuration Checklist - Hostnames](https://docs.starrocks.io/docs/deployment/preparation/environment_configurations.md#hostnames) for more information. 4. Check the FE logs to verify if the FE node is started successfully. ```bash cat fe/log/fe.log | grep thrift ``` A record of log like "2022-08-10 16:12:29,911 INFO (UNKNOWN x.x.x.x\_9010\_1660119137253(-1)|1) \[FeServer.start():52] thrift server started with port 9020." suggests that the FE node is started properly. #### Step 2: Start the BE service[​](#step-2-start-the-be-service "Direct link to Step 2: Start the BE service") note You can only add BE nodes to shared-nothing clusters and CN nodes to shared-data clusters. Otherwise, it may lead to unknown behaviors. The following procedures are performed on the BE instances. 1. Create a dedicated directory for data storage. We recommend storing data in a separate directory from the BE deployment directory. Make sure that this directory exists and you have write access to it. ```yaml # Replace with the data storage directory you want to create. mkdir -p ``` 2. Navigate to the directory that stores the [StarRocks BE deployment files](https://docs.starrocks.io/docs/deployment/preparation/prepare_deployment_files.md) you prepared earlier, and modify the BE configuration file **be/conf/be.conf**. a. Specify the data directory in the configuration item `storage_root_path`. Multiple volumes are separated by semicolon (;). Example: `/data1;/data2`. ```yaml # Replace with the data directory you have created. storage_root_path = ``` b. If any of the BE ports mentioned in the [Environment Configuration Checklist](https://docs.starrocks.io/docs/deployment/preparation/environment_configurations.md#be-ports) are occupied, you must assign valid alternatives in the BE configuration file. ```yaml be_port = vvvv # Default: 9060 be_http_port = xxxx # Default: 8040 heartbeat_service_port = yyyy # Default: 9050 brpc_port = zzzz # Default: 8060 starlet_port = uuuu # Default: 9070 ``` c. If you want to enable IP address access for your cluster, you must add the configuration item `priority_networks` in the configuration file and assign a dedicated IP address (in the CIDR format) to the BE node. You can ignore this configuration item if you want to enable FQDN access for your cluster. ```yaml priority_networks = x.x.x.x/x ``` > **NOTE** > > * You can run `ifconfig` in your terminal to view the IP address(es) owned by the instance. > * From v3.3.0, StarRocks supports deployment based on IPv6. d. If you have multiple JDKs installed on the instance, and you want to use a specific JDK that is different from the one specified in the environment variable `JAVA_HOME`, you must specify the path where the chosen JDK is installed by adding the configuration item `JAVA_HOME` in the configuration file. ```yaml # Replace with the path where the chosen JDK is installed. JAVA_HOME = ``` For information about advanced configuration items, see [Parameter Configuration - BE configuration items](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md). 3. Start the BE node. ```bash ./be/bin/start_be.sh --daemon ``` > **CAUTION** > > * Before starting the BE node with FQDN access enabled, make sure you have assigned hostnames for all instances in **/etc/hosts**. See [Environment Configuration Checklist - Hostnames](https://docs.starrocks.io/docs/deployment/preparation/environment_configurations.md#hostnames) for more information. > * You do not need to specify the parameter `--host_type` when you start BE nodes. 4. Check the BE logs to verify if the BE node is started successfully. ```bash cat be/log/be.INFO | grep heartbeat ``` A record of log like "I0810 16:18:44.487284 3310141 task\_worker\_pool.cpp:1387] Waiting to receive first heartbeat from frontend" suggests that the BE node is started properly. 5. You can start new BE nodes by repeating the above procedures on other BE instances. > **NOTE** > > * A high-availability cluster of BEs is automatically formed when at least three BE nodes are deployed and added to a StarRocks cluster. > > * If you want to deploy just one BE node, you must set `default_replication_num` to `1` in the FE configuration file **fe/conf/fe.conf**. > > ```yaml > default_replication_num = 1 > > ``` #### Step 3: Set up the cluster[​](#step-3-set-up-the-cluster "Direct link to Step 3: Set up the cluster") After all FE and BE nodes are started properly, you can set up the StarRocks cluster. The following procedures are performed on a MySQL client. You must have MySQL client 5.5.0 or later installed. 1. Connect to StarRocks via your MySQL client. You need to log in with the initial account `root`, and the password is empty by default. ```bash # Replace with the IP address (priority_networks) or FQDN # of the Leader FE node, and replace (Default: 9030) # with the query_port you specified in fe.conf. mysql -h -P -uroot ``` 2. Check the status of the Leader FE node by executing the following SQL. ```sql SHOW PROC '/frontends'\G ``` Example: ```plain MySQL [(none)]> SHOW PROC '/frontends'\G *************************** 1. row *************************** Name: x.x.x.x_9010_1686810741121 IP: x.x.x.x EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: LEADER ClusterId: 919351034 Join: true Alive: true ReplayedJournalId: 1220 LastHeartbeat: 2023-06-15 15:39:04 IsHelper: true ErrMsg: StartTime: 2023-06-15 14:32:28 Version: 3.0.0-48f4d81 1 row in set (0.01 sec) ``` * If the field `Alive` is `true`, this FE node is properly started and added to the cluster. * If the field `Role` is `FOLLOWER`, this FE node is eligible to be elected as the Leader FE node. * If the field `Role` is `LEADER`, this FE node is the Leader FE node. 3. Add the BE nodes to the cluster. ```sql -- Replace with the IP address (priority_networks) -- or FQDN of the BE nodes, and replace -- with the heartbeat_service_port (Default: 9050) you specified in be.conf. ALTER SYSTEM ADD BACKEND ":"; ``` > **NOTE** > > You can use the preceding command to add multiple BE nodes at a time. Each `:` pair represents one BE node. 4. Check the status of the BE nodes by executing the following SQL. ```sql SHOW PROC '/backends'\G ``` Example: ```plain MySQL [(none)]> SHOW PROC '/backends'\G *************************** 1. row *************************** BackendId: 10007 IP: 172.26.195.67 HeartbeatPort: 9050 BePort: 9060 HttpPort: 8040 BrpcPort: 8060 LastStartTime: 2023-06-15 15:23:08 LastHeartbeat: 2023-06-15 15:57:30 Alive: true SystemDecommissioned: false ClusterDecommissioned: false TabletNum: 30 DataUsedCapacity: 0.000 AvailCapacity: 341.965 GB TotalCapacity: 1.968 TB UsedPct: 83.04 % MaxDiskUsedPct: 83.04 % ErrMsg: Version: 3.0.0-48f4d81 Status: {"lastSuccessReportTabletsTime":"2023-06-15 15:57:08"} DataTotalCapacity: 341.965 GB DataUsedPct: 0.00 % CpuCores: 16 NumRunningQueries: 0 MemUsedPct: 0.01 % CpuUsedPct: 0.0 % ``` If the field `Alive` is `true`, this BE node is properly started and added to the cluster. #### Step 4: (Optional) Deploy a high-availability FE cluster[​](#step-4-optional-deploy-a-high-availability-fe-cluster "Direct link to Step 4: (Optional) Deploy a high-availability FE cluster") A high-availability FE cluster requires at least THREE Follower FE nodes in the StarRocks cluster. After the Leader FE node is started successfully, you can then start two new FE nodes to deploy a high-availability FE cluster. 1. Connect to StarRocks via your MySQL client. You need to log in with the initial account `root`, and the password is empty by default. ```bash # Replace with the IP address (priority_networks) or FQDN # of the Leader FE node, and replace (Default: 9030) # with the query_port you specified in fe.conf. mysql -h -P -uroot ``` 2. Add the new Follower FE node to the cluster by executing the following SQL. ```sql -- Replace with the IP address (priority_networks) -- or FQDN of the new Follower FE node, and replace -- with the edit_log_port (Default: 9010) you specified in fe.conf. ALTER SYSTEM ADD FOLLOWER ":"; ``` > **NOTE** > > * You can use the preceding command to add a single Follower FE nodes each time. > * If you want to add Observer FE nodes, execute `ALTER SYSTEM ADD OBSERVER ":"=`. For detailed instructions, see [ALTER SYSTEM - FE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md). 3. Launch a terminal on the new FE instance, create a dedicated directory for metadata storage, navigate to the directory that stores the StarRocks FE deployment files, and modify the FE configuration file **fe/conf/fe.conf**. For more instructions, see [Step 1: Start the Leader FE node](#step-1-start-the-leader-fe-node). Basically, you can repeat the procedures in Step 1 **except for the command used to start the FE node**. After configuring the Follower FE node, execute the following SQL to assign a helper node for Follower FE node and start the Follower FE node. > **NOTE** > > When adding new Follower FE node to a cluster, you must assign a helper node (essentially an existing Follower FE node) to the new Follower FE node to synchronize the metadata. * To start a new FE node with IP address access, run the following command to start the FE node: ```bash # Replace with the IP address (priority_networks) # of the Leader FE node, and replace (Default: 9010) with # the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : --daemon ``` Note that you only need to specify the parameter `--helper` ONCE when you start the node for the first time. * To start a new FE node with FQDN access, run the following command to start the FE node: ```bash # Replace with the FQDN of the Leader FE node, # and replace (Default: 9010) with the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : \ --host_type FQDN --daemon ``` Note that you only need to specify the parameters `--helper` and `--host_type` ONCE when you start the node for the first time. 4. Check the FE logs to verify if the FE node is started successfully. ```bash cat fe/log/fe.log | grep thrift ``` A record of log like "2022-08-10 16:12:29,911 INFO (UNKNOWN x.x.x.x\_9010\_1660119137253(-1)|1) \[FeServer.start():52] thrift server started with port 9020." suggests that the FE node is started properly. 5. Repeat the preceding procedure 2, 3, and 4 until you have start all the new Follower FE nodes properly, and then check the status of the FE nodes by executing the following SQL from your MySQL client: ```sql SHOW PROC '/frontends'\G ``` Example: ```plain MySQL [(none)]> SHOW PROC '/frontends'\G *************************** 1. row *************************** Name: x.x.x.x_9010_1686810741121 IP: x.x.x.x EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: LEADER ClusterId: 919351034 Join: true Alive: true ReplayedJournalId: 1220 LastHeartbeat: 2023-06-15 15:39:04 IsHelper: true ErrMsg: StartTime: 2023-06-15 14:32:28 Version: 3.0.0-48f4d81 *************************** 2. row *************************** Name: x.x.x.x_9010_1686814080597 IP: x.x.x.x EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: FOLLOWER ClusterId: 919351034 Join: true Alive: true ReplayedJournalId: 1219 LastHeartbeat: 2023-06-15 15:39:04 IsHelper: true ErrMsg: StartTime: 2023-06-15 15:38:53 Version: 3.0.0-48f4d81 *************************** 3. row *************************** Name: x.x.x.x_9010_1686814090833 IP: x.x.x.x EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: FOLLOWER ClusterId: 919351034 Join: true Alive: true ReplayedJournalId: 1219 LastHeartbeat: 2023-06-15 15:39:04 IsHelper: true ErrMsg: StartTime: 2023-06-15 15:37:52 Version: 3.0.0-48f4d81 3 rows in set (0.02 sec) ``` * If the field `Alive` is `true`, this FE node is properly started and added to the cluster. * If the field `Role` is `FOLLOWER`, this FE node is eligible to be elected as the Leader FE node. * If the field `Role` is `LEADER`, this FE node is the Leader FE node. #### Stop the StarRocks cluster[​](#stop-the-starrocks-cluster "Direct link to Stop the StarRocks cluster") You can stop the StarRocks cluster by running the following commands on the corresponding instances. * Stop an FE node. ```bash ./fe/bin/stop_fe.sh ``` * Stop a BE node. ```bash ./be/bin/stop_be.sh ``` #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") Try the following steps to identify the errors that occur when you start the FE or BE nodes: * If an FE node is not started properly, you can identify the problem by checking its log in **fe/log/fe.warn.log**. ```bash cat fe/log/fe.warn.log ``` Having identified and resolved the problem, you must first terminate the current FE process, delete the existing **meta** directory, create a new metadata storage directory, and then restart the FE node with the correct configuration. * If a BE node is not started properly, you can identify the problem by checking its log in **be/log/be.WARNING**. ```bash cat be/log/be.WARNING ``` Having identified and resolved the problem, you must first terminate the existing BE process, delete the existing **storage** directory, create a new data storage directory, and then restart the BE node with the correct configuration. #### What to do next[​](#what-to-do-next "Direct link to What to do next") Having deployed your StarRocks cluster, you can move on to [Post-deployment Setup](https://docs.starrocks.io/docs/deployment/manage_deployment/post_deployment_setup.md) for instructions on initial management measures. --- ### Deploy Shared-data StarRocks Manually tip The preparations for manual deployment are outlined in the [Deployment prerequisites](https://docs.starrocks.io/docs/deployment/preparation/deployment_prerequisites.md) and [Check environment configurations](https://docs.starrocks.io/docs/deployment/preparation/environment_configurations.md) documents. Please start there if you are planning a production deployment. If you are getting started with StarRocks and would like to follow one of the Quick Starts, please refer to [Quick Starts](https://docs.starrocks.io/docs/quick_start.md). This topic describes how to manually deploy a shared-data StarRocks cluster (in which storage and computing are decoupled). For other modes of installation, see [Deployment Overview](https://docs.starrocks.io/docs/deployment.md). To deploy a shared-nothing StarRocks cluster (in which BE is responsible for both storage and computing), see [Deploy Shared-nothing StarRocks Manually](https://docs.starrocks.io/docs/deployment/deploy_manually.md). #### Overview[​](#overview "Direct link to Overview") The shared-data StarRocks cluster is specifically engineered for the cloud on the premise of separation of storage and computing. It allows data to be stored in remote storage (for example, HDFS, AWS S3, Google GCS, Azure Blob Storage, Azure Data Lake Storage Gen2, and MinIO). You can achieve not only cheaper storage and better resource isolation, but elastic scalability for your cluster. The query performance of the shared-data StarRocks cluster aligns with that of a shared-nothing StarRocks cluster when the local disk cache is hit. The StarRocks shared-data cluster is made up of Frontend Engines (FEs) and Compute Nodes (CNs), which replace the classic Backend Engines (BEs) in shared-nothing clusters. Compared to the classic shared-nothing StarRocks architecture, separation of storage and computing offers a wide range of benefits. By decoupling these components, StarRocks provides: * Inexpensive and seamlessly scalable storage. * Elastic scalable computing. Because data is not stored in Compute Nodes (CNs), scaling can be done without data migration or shuffling across nodes. * Local disk cache for hot data to boost query performance. * Asynchronous data ingestion into remote storage, allowing a significant improvement in loading performance. The architecture of the shared-data cluster is as follows: ![Shared-data Architecture](/assets/images/share_data_arch-2aee703af4e8afcf46e5bf9866cd2f2a.png) #### Step 1: Start the Leader FE node[​](#step-1-start-the-leader-fe-node "Direct link to Step 1: Start the Leader FE node") The following procedures are performed on an FE instance. 1. Create a dedicated directory for metadata storage. We recommend storing metadata in a separate directory from the FE deployment files. Make sure that this directory exists and that you have write access to it. ```yaml # Replace with the metadata directory you want to create. mkdir -p ``` 2. Navigate to the directory that stores the [StarRocks FE deployment files](https://docs.starrocks.io/docs/deployment/preparation/prepare_deployment_files.md) you prepared earlier, and modify the FE configuration file **fe/conf/fe.conf**. a. Set shared-data run mode. ```yaml run_mode = shared_data ``` b. Specify the metadata directory in the configuration item `meta_dir`. ```yaml # Replace with the metadata directory you have created. meta_dir = ``` c. If any of the FE ports mentioned in the [Environment Configuration Checklist](https://docs.starrocks.io/docs/deployment/preparation/environment_configurations.md#fe-ports) are occupied, you must assign valid alternatives in the FE configuration file. ```yaml http_port = aaaa # Default: 8030 rpc_port = bbbb # Default: 9020 query_port = cccc # Default: 9030 edit_log_port = dddd # Default: 9010 cloud_native_meta_port = eeee # Default: 6090 ``` > **CAUTION** > > If you want to deploy multiple FE nodes in a cluster, you must assign the same `http_port` to each FE node. d. If you want to enable IP address access for your cluster, you must add the configuration item `priority_networks` in the configuration file and assign a dedicated IP address (in the CIDR format) to the FE node. You can ignore this configuration item if you want to enable [FQDN access](https://docs.starrocks.io/docs/administration/management/enable_fqdn.md) for your cluster. ```yaml priority_networks = x.x.x.x/x ``` > **NOTE** > > * You can run `ifconfig` in your terminal to view the IP address(es) owned by the instance. > * From v3.3.0, StarRocks supports deployment based on IPv6. e. If you have multiple JDKs installed on the instance, and you want to use a specific JDK that is different from the one specified in the environment variable `JAVA_HOME`, you must specify the path where the chosen JDK is installed by adding the configuration item `JAVA_HOME` in the configuration file. ```yaml # Replace with the path where the chosen JDK is installed. JAVA_HOME = ``` For information about advanced configuration items, see [Parameter Configuration - FE configuration items](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md). 3. Start the FE node. * To enable IP address access for your cluster, run the following command to start the FE node: ```bash ./fe/bin/start_fe.sh --daemon ``` * To enable FQDN access for your cluster, run the following command to start the FE node: ```bash ./fe/bin/start_fe.sh --host_type FQDN --daemon ``` Note that you only need to specify the parameter `--host_type` ONCE when you start the node for the first time. > **CAUTION** > > Before starting the FE node with FQDN access enabled, make sure you have assigned hostnames for all instances in **/etc/hosts**. See [Environment Configuration Checklist - Hostnames](https://docs.starrocks.io/docs/deployment/preparation/environment_configurations.md#hostnames) for more information. 4. Check the FE logs to verify if the FE node is started successfully. ```bash cat fe/log/fe.log | grep thrift ``` A record of log like "2022-08-10 16:12:29,911 INFO (UNKNOWN x.x.x.x\_9010\_1660119137253(-1)|1) \[FeServer.start():52] thrift server started with port 9020." suggests that the FE node is started properly. #### Step 2: Start the CN service[​](#step-2-start-the-cn-service "Direct link to Step 2: Start the CN service") note You can only add BE nodes to shared-nothing clusters and CN nodes to shared-data clusters. Otherwise, it may lead to unknown behaviors. The following procedures are performed on the CN instances. You can deploy CN nodes with the BE deployment files. 1. Create a dedicated directory for data cache. We recommend caching data in a separate directory from the CN deployment directory. Make sure that this directory exists and you have write access to it. ```yaml # Replace with the data cache directory you want to create. mkdir -p ``` 2. Navigate to the directory that stores the [StarRocks BE deployment files](https://docs.starrocks.io/docs/deployment/preparation/prepare_deployment_files.md) you prepared earlier, and modify the CN configuration file **be/conf/cn.conf**. a. Specify the data directory in the configuration item `storage_root_path`. Multiple volumes are separated by semicolon (;). Example: `/data1;/data2`. ```yaml # Replace with the data directory you have created. storage_root_path = ``` Local cache is effective when queries are frequent and the data being queried is recent, but there are cases that you may wish to turn off the local cache completely. * In a Kubernetes environment with CN pods that scale up and down in number on demand, the pods may not have storage volumes attached. * When the data being queried is in a data lake in remote storage and most of it is archive (old) data, if the queries are infrequent, the data cache will have a low hit ratio and the benefit may not be worth having the cache. To turn off the data cache set: ```yaml storage_root_path = ``` > **NOTE** > > The data is cached under the directory **`/starlet_cache`**. b. If any of the CN ports mentioned in the [Environment Configuration Checklist](https://docs.starrocks.io/docs/deployment/preparation/environment_configurations.md) are occupied, you must assign valid alternatives in the CN configuration file. ```yaml be_port = vvvv # Default: 9060 be_http_port = xxxx # Default: 8040 heartbeat_service_port = yyyy # Default: 9050 brpc_port = zzzz # Default: 8060 starlet_port = uuuu # Default: 9070 ``` c. If you want to enable IP address access for your cluster, you must add the configuration item `priority_networks` in the configuration file and assign a dedicated IP address (in the CIDR format) to the CN node. You can ignore this configuration item if you want to enable FQDN access for your cluster. ```yaml priority_networks = x.x.x.x/x ``` > **NOTE** > > * You can run `ifconfig` in your terminal to view the IP address(es) owned by the instance. > * From v3.3.0, StarRocks supports deployment based on IPv6. d. If you have multiple JDKs installed on the instance, and you want to use a specific JDK that is different from the one specified in the environment variable `JAVA_HOME`, you must specify the path where the chosen JDK is installed by adding the configuration item `JAVA_HOME` in the configuration file. ```yaml # Replace with the path where the chosen JDK is installed. JAVA_HOME = ``` For information about advanced configuration items, see [Parameter Configuration - BE configuration items](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md) because most of CN's parameters are inherited from BE. 3. Start the CN node. ```bash ./be/bin/start_cn.sh --daemon ``` > **CAUTION** > > * Before starting the CN node with FQDN access enabled, make sure you have assigned hostnames for all instances in **/etc/hosts**. See [Environment Configuration Checklist - Hostnames](https://docs.starrocks.io/docs/deployment/preparation/environment_configurations.md#hostnames) for more information. > * You do not need to specify the parameter `--host_type` when you start CN nodes. 4. Check the CN logs to verify if the CN node is started successfully. ```bash cat be/log/cn.INFO | grep heartbeat ``` A record of log like "I0313 15:03:45.820030 412450 thrift\_server.cpp:375] heartbeat has started listening port on 9050" suggests that the CN node is started properly. 5. You can start new CN nodes by repeating the above procedures on other instances. #### Step 3: Set up the cluster[​](#step-3-set-up-the-cluster "Direct link to Step 3: Set up the cluster") After all FE and CN nodes are started properly, you can set up the StarRocks cluster. The following procedures are performed on a MySQL client. You must have MySQL client 5.5.0 or later installed. 1. Connect to StarRocks via your MySQL client. You need to log in with the initial account `root`, and the password is empty by default. ```bash # Replace with the IP address (priority_networks) or FQDN # of the Leader FE node, and replace (Default: 9030) # with the query_port you specified in fe.conf. mysql -h -P -uroot ``` 2. Check the status of the Leader FE node by executing the following SQL. ```sql SHOW PROC '/frontends'\G ``` Example: ```plain MySQL [(none)]> SHOW PROC '/frontends'\G *************************** 1. row *************************** Name: x.x.x.x_9010_1686810741121 IP: x.x.x.x EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: LEADER ClusterId: 919351034 Join: true Alive: true ReplayedJournalId: 1220 LastHeartbeat: 2023-06-15 15:39:04 IsHelper: true ErrMsg: StartTime: 2023-06-15 14:32:28 Version: 3.0.0-48f4d81 1 row in set (0.01 sec) ``` * If the field `Alive` is `true`, this FE node is properly started and added to the cluster. * If the field `Role` is `FOLLOWER`, this FE node is eligible to be elected as the Leader FE node. * If the field `Role` is `LEADER`, this FE node is the Leader FE node. 3. Add the CN nodes to the cluster. ```sql -- Replace with the IP address (priority_networks) -- or FQDN of the CN node, and replace -- with the heartbeat_service_port (Default: 9050) you specified in cn.conf. ALTER SYSTEM ADD COMPUTE NODE ":"; ``` > **NOTE** > > You can add multiple CN nodes with one SQL. Each `:` pair represents one CN node. 4. Check the status of the CN nodes by executing the following SQL. ```sql SHOW PROC '/compute_nodes'\G ``` Example: ```plain MySQL [(none)]> SHOW PROC '/compute_nodes'\G *************************** 1. row *************************** ComputeNodeId: 10003 IP: x.x.x.x HeartbeatPort: 9050 BePort: 9060 HttpPort: 8040 BrpcPort: 8060 LastStartTime: 2023-03-13 15:11:13 LastHeartbeat: 2023-03-13 15:11:13 Alive: true SystemDecommissioned: false ClusterDecommissioned: false ErrMsg: Version: 2.5.2-c3772fb 1 row in set (0.00 sec) ``` If the field `Alive` is `true`, this CN node is properly started and added to the cluster. #### Step 4: (Optional) Deploy a high-availability FE cluster[​](#step-4-optional-deploy-a-high-availability-fe-cluster "Direct link to Step 4: (Optional) Deploy a high-availability FE cluster") A high-availability FE cluster requires at least THREE Follower FE nodes in the StarRocks cluster. After the Leader FE node is started successfully, you can then start two new FE nodes to deploy a high-availability FE cluster. 1. Connect to StarRocks via your MySQL client. You need to log in with the initial account `root`, and the password is empty by default. ```bash # Replace with the IP address (priority_networks) or FQDN # of the Leader FE node, and replace (Default: 9030) # with the query_port you specified in fe.conf. mysql -h -P -uroot ``` 2. Add the new Follower FE node to the cluster by executing the following SQL. ```sql -- Replace with the IP address (priority_networks) -- or FQDN of the new Follower FE node, and replace -- with the edit_log_port (Default: 9010) you specified in fe.conf. ALTER SYSTEM ADD FOLLOWER ":"; ``` > **NOTE** > > * You can use the preceding command to add a single Follower FE nodes each time. > * If you want to add Observer FE nodes, execute `ALTER SYSTEM ADD OBSERVER ":"=`. For detailed instructions, see [ALTER SYSTEM - FE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md). 3. Launch a terminal on the new FE instance, create a dedicated directory for metadata storage, navigate to the directory that stores the StarRocks FE deployment files, and modify the FE configuration file **fe/conf/fe.conf**. For more instructions, see [Step 1: Start the Leader FE node](#step-1-start-the-leader-fe-node). Basically, you can repeat the procedures in Step 1 **except for the command used to start the FE node**. After configuring the Follower FE node, execute the following SQL to assign a helper node for Follower FE node and start the Follower FE node. > **NOTE** > > When adding new Follower FE node to a cluster, you must assign a helper node (essentially an existing Follower FE node) to the new Follower FE node to synchronize the metadata. * To start a new FE node with IP address access, run the following command to start the FE node: ```bash # Replace with the IP address (priority_networks) # of the Leader FE node, and replace (Default: 9010) with # the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : --daemon ``` Note that you only need to specify the parameter `--helper` ONCE when you start the node for the first time. * To start a new FE node with FQDN access, run the following command to start the FE node: ```bash # Replace with the FQDN of the Leader FE node, # and replace (Default: 9010) with the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : \ --host_type FQDN --daemon ``` Note that you only need to specify the parameters `--helper` and `--host_type` ONCE when you start the node for the first time. 4. Check the FE logs to verify if the FE node is started successfully. ```bash cat fe/log/fe.log | grep thrift ``` A record of log like "2022-08-10 16:12:29,911 INFO (UNKNOWN x.x.x.x\_9010\_1660119137253(-1)|1) \[FeServer.start():52] thrift server started with port 9020." suggests that the FE node is started properly. 5. Repeat the preceding procedure 2, 3, and 4 until you have start all the new Follower FE nodes properly, and then check the status of the FE nodes by executing the following SQL from your MySQL client: ```sql SHOW PROC '/frontends'\G ``` Example: ```plain MySQL [(none)]> SHOW PROC '/frontends'\G *************************** 1. row *************************** Name: x.x.x.x_9010_1686810741121 IP: x.x.x.x EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: LEADER ClusterId: 919351034 Join: true Alive: true ReplayedJournalId: 1220 LastHeartbeat: 2023-06-15 15:39:04 IsHelper: true ErrMsg: StartTime: 2023-06-15 14:32:28 Version: 3.0.0-48f4d81 *************************** 2. row *************************** Name: x.x.x.x_9010_1686814080597 IP: x.x.x.x EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: FOLLOWER ClusterId: 919351034 Join: true Alive: true ReplayedJournalId: 1219 LastHeartbeat: 2023-06-15 15:39:04 IsHelper: true ErrMsg: StartTime: 2023-06-15 15:38:53 Version: 3.0.0-48f4d81 *************************** 3. row *************************** Name: x.x.x.x_9010_1686814090833 IP: x.x.x.x EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: FOLLOWER ClusterId: 919351034 Join: true Alive: true ReplayedJournalId: 1219 LastHeartbeat: 2023-06-15 15:39:04 IsHelper: true ErrMsg: StartTime: 2023-06-15 15:37:52 Version: 3.0.0-48f4d81 3 rows in set (0.02 sec) ``` * If the field `Alive` is `true`, this FE node is properly started and added to the cluster. * If the field `Role` is `FOLLOWER`, this FE node is eligible to be elected as the Leader FE node. * If the field `Role` is `LEADER`, this FE node is the Leader FE node. #### Step 5: Create and set the default storage volume[​](#step-5-create-and-set-the-default-storage-volume "Direct link to Step 5: Create and set the default storage volume") To give your shared-data StarRocks cluster permission to store data in your remote storage, you must reference a storage volume when you create databases or cloud-native tables. A storage volume consists of the properties and credential information of the remote data storage. If you have deployed a new shared-data StarRocks cluster, you must define a default storage volume before you can create databases and tables in the cluster. Choose your cloud provider and service, and execute the corresponding statements to create and set the default storage volume. * AWS S3 * Google Cloud Storage * Azure Blob Storage * Azure Data Lake Storage Gen2 * MinIO * HDFS The following example creates a storage volume `def_volume` for an AWS S3 bucket `defaultbucket` with the IAM user-based credential (Access Key and Secret Key), enables the [Partitioned Prefix](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md#partitioned-prefix) feature, and sets it as the default storage volume: ```sql CREATE STORAGE VOLUME def_volume TYPE = S3 LOCATIONS = ("s3://defaultbucket") PROPERTIES ( "enabled" = "true", "aws.s3.region" = "us-west-2", "aws.s3.endpoint" = "https://s3.us-west-2.amazonaws.com", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "xxxxxxxxxx", "aws.s3.secret_key" = "yyyyyyyyyy", "aws.s3.enable_partitioned_prefix" = "true" ); SET def_volume AS DEFAULT STORAGE VOLUME; ``` The following example creates a storage volume `def_volume` for a GCS bucket `defaultbucket` with Service Account-based credentials, enables the storage volume, and sets it as the default storage volume: ```sql CREATE STORAGE VOLUME def_volume TYPE = GS LOCATIONS = ("gs://defaultbucket") PROPERTIES ( "enabled" = "true", "gcp.gcs.use_compute_engine_service_account" = "false", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ); SET def_volume AS DEFAULT STORAGE VOLUME; ``` The following example creates a storage volume `def_volume` for an Azure Blob Storage bucket `defaultbucket` with shared key access, enables the storage volume, and sets it as the default storage volume: ```sql CREATE STORAGE VOLUME def_volume TYPE = AZBLOB LOCATIONS = ("azblob://defaultbucket/test/") PROPERTIES ( "enabled" = "true", "azure.blob.endpoint" = "", "azure.blob.shared_key" = "" ); SET def_volume AS DEFAULT STORAGE VOLUME; ``` The following example creates a storage volume `adls2` for an Azure Data Lake Storage Gen2 file system `testfilesystem` with SAS token, and disables the storage volume: ```sql CREATE STORAGE VOLUME adls2 TYPE = ADLS2 LOCATIONS = ("adls2://testfilesystem/starrocks") PROPERTIES ( "enabled" = "false", "azure.adls2.endpoint" = "", "azure.adls2.sas_token" = "" ); ``` The following example creates a storage volume `def_volume` for a MinIO bucket `defaultbucket` with Access Key and Secret Key credentials, enables the [Partitioned Prefix](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md#partitioned-prefix) feature, and sets it as the default storage volume: ```sql CREATE STORAGE VOLUME def_volume TYPE = S3 LOCATIONS = ("s3://defaultbucket") PROPERTIES ( "enabled" = "true", "aws.s3.region" = "us-west-2", "aws.s3.endpoint" = "https://hostname.domainname.com:portnumber", "aws.s3.access_key" = "xxxxxxxxxx", "aws.s3.secret_key" = "yyyyyyyyyy", "aws.s3.enable_partitioned_prefix" = "true" ); SET def_volume AS DEFAULT STORAGE VOLUME; ``` The following example creates a storage volume `def_volume` for an HDFS storage, enables the storage volume, and sets it as the default storage volume: ```sql CREATE STORAGE VOLUME def_volume TYPE = HDFS LOCATIONS = ("hdfs://127.0.0.1:9000/user/starrocks/"); SET def_volume AS DEFAULT STORAGE VOLUME; ``` In addition to the authentication methods used above, StarRocks also supports a variety of credentials to access your remote storage. See [CREATE STORAGE VOLUME - Credential information](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md#credential-information) for detailed instructions. #### Use shared-data StarRocks[​](#use-shared-data-starrocks "Direct link to Use shared-data StarRocks") The usage of shared-data StarRocks clusters is also similar to that of a classic shared-nothing StarRocks cluster, except that the shared-data cluster uses storage volumes and cloud-native tables to store data in remote storage. ##### Create a database and a cloud-native table[​](#create-a-database-and-a-cloud-native-table "Direct link to Create a database and a cloud-native table") After you create a default storage volume, you can then create a database and a cloud-native table using this storage volume. Shared-data StarRocks clusters support all [StarRocks table types](https://docs.starrocks.io/docs/table_design/table_types.md). The following example creates a database `cloud_db` and a table `detail_demo` based on Duplicate Key table type, enables the local disk cache, sets the hot data validity duration to one month, and disables asynchronous data ingestion into remote storage: ```sql CREATE DATABASE cloud_db; USE cloud_db; CREATE TABLE IF NOT EXISTS detail_demo ( recruit_date DATE NOT NULL COMMENT "YYYY-MM-DD", region_num TINYINT COMMENT "range [-128, 127]", num_plate SMALLINT COMMENT "range [-32768, 32767] ", tel INT COMMENT "range [-2147483648, 2147483647]", id BIGINT COMMENT "range [-2^63 + 1 ~ 2^63 - 1]", password LARGEINT COMMENT "range [-2^127 + 1 ~ 2^127 - 1]", name CHAR(20) NOT NULL COMMENT "range char(m),m in (1-255) ", profile VARCHAR(500) NOT NULL COMMENT "upper limit value 65533 bytes", ispass BOOLEAN COMMENT "true/false") DUPLICATE KEY(recruit_date, region_num) DISTRIBUTED BY HASH(recruit_date, region_num) PROPERTIES ( "storage_volume" = "def_volume", "datacache.enable" = "true", "datacache.partition_duration" = "1 MONTH" ); ``` > **NOTE** > > The default storage volume is used when you create a database or a cloud-native table in a shared-data StarRocks cluster if no storage volume is specified. ###### `PROPERTIES`[​](#properties "Direct link to properties") In addition to the regular table `PROPERTIES`, you need to specify the following `PROPERTIES` when creating a table for shared-data StarRocks cluster: ###### `datacache.enable`[​](#datacacheenable "Direct link to datacacheenable") Whether to enable the local disk cache. * `true` (Default) When this property is set to `true`, the data to be loaded is simultaneously written into the remote storage and the local disk (as the cache for query acceleration). * `false` When this property is set to `false`, the data is loaded only into the remote storage. > **NOTE** > > To enable the local disk cache, you must specify the directory of the disk in the CN configuration item `storage_root_path`. ###### `datacache.partition_duration`[​](#datacachepartition_duration "Direct link to datacachepartition_duration") The validity duration of the hot data. When the local disk cache is enabled, all data is loaded into the cache. When the cache is full, StarRocks deletes the less recently used data from the cache. When a query needs to scan the deleted data, StarRocks checks if the data is within the duration of validity starting from the current time. If the data is within the duration, StarRocks loads the data into the cache again. If the data is not within the duration, StarRocks does not load it into the cache. This property is a string value that can be specified with the following units: `YEAR`, `MONTH`, `DAY`, and `HOUR`, for example, `7 DAY` and `12 HOUR`. If it is not specified, all data is cached as the hot data. > **NOTE** > > This property is available only when `datacache.enable` is set to `true`. ##### View table information[​](#view-table-information "Direct link to View table information") You can view the information of tables in a specific database using `SHOW PROC "/dbs/"`. See [SHOW PROC](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md) for more information. Example: ```plain mysql> SHOW PROC "/dbs/xxxxx"; +---------+-------------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+------------------------------+ | TableId | TableName | IndexNum | PartitionColumnName | PartitionNum | State | Type | LastConsistencyCheckTime | ReplicaCount | PartitionType | StoragePath | +---------+-------------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+------------------------------+ | 12003 | detail_demo | 1 | NULL | 1 | NORMAL | CLOUD_NATIVE | NULL | 8 | UNPARTITIONED | s3://xxxxxxxxxxxxxx/1/12003/ | +---------+-------------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+------------------------------+ ``` The `Type` of a table in shared-data StarRocks cluster is `CLOUD_NATIVE`. In the field `StoragePath`, StarRocks returns the remote storage directory where the table is stored. ##### Load data into a shared-data StarRocks cluster[​](#load-data-into-a-shared-data-starrocks-cluster "Direct link to Load data into a shared-data StarRocks cluster") Shared-data StarRocks clusters support all loading methods provided by StarRocks. See [Loading options](https://docs.starrocks.io/docs/loading/loading_introduction.md) for more information. ##### Query in a shared-data StarRocks cluster[​](#query-in-a-shared-data-starrocks-cluster "Direct link to Query in a shared-data StarRocks cluster") Tables in a shared-data StarRocks cluster support all types of queries provided by StarRocks. See StarRocks [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) for more information. > **NOTE** > > Shared-data StarRocks clusters support [synchronous materialized views](https://docs.starrocks.io/docs/using_starrocks/Materialized_view-single_table.md) from v3.4.0. #### Stop the StarRocks cluster[​](#stop-the-starrocks-cluster "Direct link to Stop the StarRocks cluster") You can stop the StarRocks cluster by running the following commands on the corresponding instances. * Stop an FE node. ```bash ./fe/bin/stop_fe.sh ``` * Stop a CN node. ```bash ./be/bin/stop_cn.sh ``` #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") Try the following steps to identify the errors that occur when you start the FE or CN nodes: * If an FE node is not started properly, you can identify the problem by checking its log in **fe/log/fe.warn.log**. ```bash cat fe/log/fe.warn.log ``` Having identified and resolved the problem, you must first terminate the current FE process, delete the existing **meta** directory, create a new metadata storage directory, and then restart the FE node with the correct configuration. * If a CN node is not started properly, you can identify the problem by checking its log in **be/log/cn.WARNING**. ```bash cat be/log/cn.WARNING ``` Having identified and resolved the problem, you must first terminate the existing CN process, and then restart the CN node with the correct configuration. #### What to do next[​](#what-to-do-next "Direct link to What to do next") Having deployed your StarRocks cluster, you can move on to [Post-deployment Setup](https://docs.starrocks.io/docs/deployment/manage_deployment/post_deployment_setup.md) for instructions on initial management measures. --- ### Feature Support: Shared-data Clusters tip Each of the features below lists the version number that they were added in. If you are deploying a new cluster please deploy the latest patch release of version 3.2 or higher. #### Overview[​](#overview "Direct link to Overview") Shared-data StarRocks clusters feature a disaggregated storage and compute architecture. This allows data to be stored in remote storage, leading to lower storage costs, optimized resource isolation, and better service elasticity compared to a shared-nothing cluster. This document outlines the feature support for shared-data clusters, covering deployment methods, storage configurations, caching mechanisms, Compaction, Primary Key table functionalities, and performance test results. #### Deployment[​](#deployment "Direct link to Deployment") Shared-data clusters support deployments on physical/virtual machines and on Kubernetes via Operator. Both deployment solutions have the following limitations: * Mixed deployment of shared-nothing and shared-data mode is not supported. * The transformation from a shared-nothing cluster to a shared-data cluster or vice versa is not supported. * Heterogeneous deployments are not supported, meaning the hardware specifications of all CN nodes within a cluster must be the same. ##### StarRocks Kubernetes Operator[​](#starrocks-kubernetes-operator "Direct link to StarRocks Kubernetes Operator") StarRocks offers the [StarRocks Kubernetes Operator](https://github.com/StarRocks/starrocks-kubernetes-operator/releases) for shared-data deployment on Kubernetes. You can scale shared-data clusters by the following methods: * Manual operations. * Automatic scaling using Kubernetes HPA (Horizontal Pod Autoscaler) strategies. #### Storage[​](#storage "Direct link to Storage") Shared-data clusters support building storage volumes on HDFS and object storage. ##### HDFS[​](#hdfs "Direct link to HDFS") ###### Location[​](#location "Direct link to Location") StarRocks supports the following locations for HDFS storage volume: * HDFS: `hdfs://:/` > **NOTE** > > From v3.2, storage volumes support HDFS clusters with the NameNode HA mode enabled. * WebHDFS (Supported from v3.2): `webhdfs://:/` * ViewFS (Supported from v3.2): `viewfs:///` ###### Authentication[​](#authentication "Direct link to Authentication") StarRocks supports the following authentication methods for HDFS storage volume: * Basic * Username (Supported from v3.2) * Kerberos Ticket Cache (Supported from v3.2) > **NOTE** > > StarRocks does not support automatic ticket refresh. You need to set up crontab tasks to refresh the ticket. Authentication using Kerberos Keytab and Principal ID is not yet supported. ###### Usage notes[​](#usage-notes "Direct link to Usage notes") StarRocks supports storage volumes on HDFS and object storage. However, only one HDFS storage volume is allowed in each StarRocks instance. Creating multiple HDFS storage volumes may cause unknown behaviors of StarRocks. ##### Object storage[​](#object-storage "Direct link to Object storage") ###### Location[​](#location-1 "Direct link to Location") StarRocks supports the following object storage services for storage volumes: * S3-compatible object storage services: `s3://` * AWS S3 * GCS, OSS, OBS, COS, TOS, KS3, MinIO, and Ceph S3 * Azure Blob Storage (Supported from v3.1.1): `azblob://` * Azure Data Lake Storage Gen2 (Supported from v3.4.1): `adls2:///` ###### Authentication[​](#authentication-1 "Direct link to Authentication") StarRocks supports the following authentication methods for different object storage services: * AWS S3 * AWS SDK * IAM user-based Credential * Instance Profile * Assumed Role * GCS, OSS, OBS, COS, TOS, KS3, MinIO, and Ceph S3 * Access Key pair * Azure Blob Storage * Shared Key * Shared Access Signatures (SAS) * Azure Data Lake Storage Gen2 * Shared Key * Shared Access Signatures (SAS) ###### Partitioned Prefix[​](#partitioned-prefix "Direct link to Partitioned Prefix") From v3.2.4, StarRocks supports creating storage volumes with the Partitioned Prefix feature for S3-compatible object storage systems. When this feature is enabled, StarRocks distributes the data into multiple partitions (sub-paths) under the bucket. It can easily multiply StarRocks' read and write performance on data files stored in the bucket. ##### Storage volumes[​](#storage-volumes "Direct link to Storage volumes") * From v3.1.0 onwards, storage volumes can be created using the CREATE STORAGE VOLUME statement, and this method is recommended in later versions. * The internal catalog `default_catalog` in shared-data clusters uses the default storage volume for data persistence. You can assign different storage volumes for databases and tables in `default_catalog` by setting the property `storage_volume`. If not configured, the property `storage_volume` is inherited in the order of catalog, database, and table. * Currently, storage volumes can be used only for storing data in cloud-native tables. Future support will include external storage management, data loading, and backup capabilities. #### Cache[​](#cache "Direct link to Cache") ##### Cache types[​](#cache-types "Direct link to Cache types") ###### File Cache[​](#file-cache "Direct link to File Cache") File Cache was the initial caching mechanism introduced along with the shared-data cluster. It loads the cache at the segment file level. File Cache is not recommended in v3.1.7, v3.2.3, and later versions. ###### Data Cache[​](#data-cache "Direct link to Data Cache") Data Cache is supported from v3.1.7 and v3.2.3 onwards to replace File Cache in earlier versions. Data Cache loads data from remote storage in blocks (on the order of MBs) on demand, without needing to load the entire file. It is recommended in the later versions and enabled by default in v3.2.3 and later. ###### Data Cache Warmup[​](#data-cache-warmup "Direct link to Data Cache Warmup") StarRocks v3.3.0 introduces the Data Cache Warmup feature to accelerate queries in data lakes and shared-data clusters. Data Cache Warmup is an active process of populating the cache. By executing CACHE SELECT, you can proactively fetch the desired data from remote storage in advance. ##### Configurations[​](#configurations "Direct link to Configurations") * Table properties: * `datacache.enable`: Whether to enable the local disk cache. Default: `true`. * `datacache.partition_duration`: The validity duration of the cached data. * BE configurations: * `starlet_use_star_cache`: Whether to enable Data Cache. * `starlet_star_cache_disk_size_percent`: The percentage of disk capacity that Data Cache can use at most in a shared-data cluster. ##### Capabilities[​](#capabilities "Direct link to Capabilities") * Data loading generates a local cache, whose eviction is only managed by the cache capacity control mechanism instead of `partition_duration`. * StarRocks supports setting up regular tasks for Data Cache Warmup. ##### Limitations[​](#limitations "Direct link to Limitations") * StarRocks does not support multiple replicas for cached data. #### Compaction[​](#compaction "Direct link to Compaction") ##### Observability[​](#observability "Direct link to Observability") ###### Partition-level Compaction status[​](#partition-level-compaction-status "Direct link to Partition-level Compaction status") From v3.1.9 onwards, you can view the Compaction status of partitions by querying `information_schema.partitions_meta`. We recommend monitoring the following key metrics: * **AvgCS**: Average Compaction score of all tablets in the partition. * **MaxCS**: Maximum Compaction score among all tablets in the partition. ###### Compaction task status[​](#compaction-task-status "Direct link to Compaction task status") From v3.2.0 onwards, you can view the status and progress of Compaction tasks by querying `information_schema.be_cloud_native_compactions`. We recommend monitoring the following key metrics: * **PROGRESS**: Current Compaction progress (in percentage) of the tablet. * **STATUS**: The status of the compaction task. If any error occurs, detailed error messages will be returned in this field. ##### Cancelling Compaction tasks[​](#cancelling-compaction-tasks "Direct link to Cancelling Compaction tasks") You can cancel specific compaction tasks using the CANCEL COMPACTION statement. Example: ```sql CANCEL COMPACTION WHERE TXN_ID = 123; ``` > **NOTE** > > The CANCEL COMPACTION statement must be executed on the Leader FE node. ##### Manual Compaction[​](#manual-compaction "Direct link to Manual Compaction") From v3.1, StarRocks offers a SQL statement for manual Compaction. You can specify the table or partitions for compaction. For more information, refer to [Manual Compaction](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md#manual-compaction). #### Primary Key tables[​](#primary-key-tables "Direct link to Primary Key tables") The following table lists the major features of Primary Key tables and their support status in shared-data clusters: | **Feature** | **Supported Version(s)** | **Description** | | ----------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------- | | Primary Key tables | v3.1.0 | | | Primary Key index persistence | v3.2.0
v3.1.3
v3.3.2 | | | Partial Update | v3.1.0 | Shared-data clusters support Partial Update in Row mode from v3.1.0 onwards and in Column mode from v3.3.1 onwards. | | Conditional Update | v3.1.0 | Currently, the condition only supports 'Greater'. | | Hybrid row-column storage | ❌ | To be supported in future releases. | #### Query performance[​](#query-performance "Direct link to Query performance") The following test compares the query performance of a shared-data cluster with Data Cache disabled, one with Data Cache enabled, one that queries the dataset in Hive, and a shared-nothing cluster. ##### Hardware Specifications[​](#hardware-specifications "Direct link to Hardware Specifications") The cluster used in the test includes one FE node and five CN/BE nodes. The hardware specifications are as follows: | **VM provider** | Alibaba Cloud ECS | | --------------------- | ------------------- | | **FE node** | 8 Core 32 GB Memory | | **CN/BE node** | 8 Core 64 GB Memory | | **Network bandwidth** | 8 Gbits/s | | **Disk** | ESSD | ##### Software version[​](#software-version "Direct link to Software version") StarRocks v3.3.0 ##### Dataset[​](#dataset "Direct link to Dataset") SSB 1TB dataset note The dataset and queries used in this comparison are from the [Star Schema Benchmark](https://docs.starrocks.io/docs/benchmarking/SSB_Benchmarking.md). ##### Test Results[​](#test-results "Direct link to Test Results") The following table shows the performance test results on thirteen queries and the sum of each cluster. The unit of query latency is milliseconds (ms). | **Query** | **Shared-data Without Data Cache** | **Shared-data With Data Cache** | **Hive Catalog Without Data Cache** | **Shared-nothing** | | --------- | ---------------------------------- | ------------------------------- | ----------------------------------- | ------------------ | | **Q01** | 2742 | 858 | 9652 | 3555 | | **Q02** | 2714 | 704 | 8638 | 3183 | | **Q03** | 1908 | 658 | 8163 | 2980 | | **Q04** | 31135 | 8582 | 34604 | 7997 | | **Q05** | 26597 | 7806 | 29183 | 6794 | | **Q06** | 21643 | 7147 | 24401 | 5602 | | **Q07** | 35271 | 15490 | 38904 | 19530 | | **Q08** | 24818 | 7368 | 27598 | 6984 | | **Q09** | 21056 | 6667 | 23587 | 5687 | | **Q10** | 2823 | 912 | 16663 | 3942 | | **Q11** | 50027 | 18947 | 52997 | 19636 | | **Q12** | 10300 | 4919 | 36146 | 8136 | | **Q13** | 7378 | 3386 | 23153 | 6380 | | **SUM** | 238412 | 83444 | 333689 | 100406 | ##### Conclusion[​](#conclusion "Direct link to Conclusion") * The query performance of the shared-data cluster with Data Cache disabled and Parallel Scan and I/O merge optimization enabled is **1.4 times** that of the cluster that queries Hive data. * The query performance of the shared-data cluster with Data Cache enabled and Parallel Scan and I/O merge optimization enabled is **1.2 times** that of the shared-nothing cluster. #### Other features to be supported[​](#other-features-to-be-supported "Direct link to Other features to be supported") * Full-text inverted index * Hybrid row-column storage * Global dictionary object * Backup and restore --- ### Deploy StarRocks with Helm [Helm](https://helm.sh/) is a package manager for Kubernetes. A [Helm Chart](https://helm.sh/docs/topics/charts/) is a Helm package and contains all of the resource definitions necessary to run an application on a Kubernetes cluster. This topic describes how to use Helm to automatically deploy a StarRocks cluster on a Kubernetes cluster. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") * [Create a Kubernetes cluster](https://docs.starrocks.io/docs/deployment/sr_operator.md#create-kubernetes-cluster). * [Install Helm](https://helm.sh/docs/intro/quickstart/). #### Procedure[​](#procedure "Direct link to Procedure") 1. Add the Helm Chart Repo for StarRocks. The Helm Chart contains the definitions of the StarRocks Operator and the custom resource StarRocksCluster. 1. Add the Helm Chart Repo. ```bash helm repo add starrocks https://starrocks.github.io/starrocks-kubernetes-operator ``` 2. Update the Helm Chart Repo to the latest version. ```bash helm repo update ``` 3. View the Helm Chart Repo that you added. ```bash $ helm search repo starrocks NAME CHART VERSION APP VERSION DESCRIPTION starrocks/kube-starrocks 1.8.0 3.1-latest kube-starrocks includes two subcharts, starrock... starrocks/operator 1.8.0 1.8.0 A Helm chart for StarRocks operator starrocks/starrocks 1.8.0 3.1-latest A Helm chart for StarRocks cluster ``` 2. Use the default **[values.yaml](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/helm-charts/charts/kube-starrocks/values.yaml)** of the Helm Chart to deploy the StarRocks Operator and StarRocks cluster, or create a YAML file to customize your deployment configurations. 1. Deployment with default configurations Run the following command to deploy the StarRocks Operator and the StarRocks cluster which consists of one FE and one BE: > Tip > > The default `values.yaml` is configured to deploy: > > * The operator pod with 1/2CPU and 0.8GB RAM > * One FE with 4GB RAM, 4 cores, and 15Gi disk > * One BE with 4GB RAM, 4 cores, and 1Ti disk > > If you do not have these resources available in your Kubernetes cluster then skip to the **Deployment with custom configurations** section and adjust the resources. ```bash $ helm install starrocks starrocks/kube-starrocks # If the following result is returned, the StarRocks Operator and StarRocks cluster are being deployed. NAME: starrocks LAST DEPLOYED: Tue Aug 15 15:12:00 2023 NAMESPACE: starrocks STATUS: deployed REVISION: 1 TEST SUITE: None ``` 3. Deployment with custom configurations * Create a YAML file, for example, **my-values.yaml**, and customize the configurations for the StarRocks Operator and StarRocks cluster in the YAML file. For the supported parameters and descriptions, see the comments in the default **[values.yaml](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/helm-charts/charts/kube-starrocks/values.yaml)** of the Helm Chart. * Run the following command to deploy the StarRocks Operator and StarRocks cluster with the custom configurations in **my-values.yaml**. ```bash helm install -f my-values.yaml starrocks starrocks/kube-starrocks ``` Deployment takes a while. During this period, you can check the deployment status with: ```bash kubectl --namespace default get starrockscluster -l "cluster=kube-starrocks" ``` If the following result is returned, the deployment has been successfully completed. ```bash NAME PHASE FESTATUS BESTATUS CNSTATUS FEPROXYSTATUS kube-starrocks running running running ``` You can also run `kubectl get pods` to check the deployment status. If all Pods are in the `Running` state and all containers within the Pods are `READY`, the deployment has been successfully completed. ```bash kubectl get pods ``` ```bash NAME READY STATUS RESTARTS AGE kube-starrocks-be-0 1/1 Running 0 2m50s kube-starrocks-fe-0 1/1 Running 0 4m31s kube-starrocks-operator-69c5c64595-pc7fv 1/1 Running 0 4m50s ``` #### Next steps[​](#next-steps "Direct link to Next steps") * Access StarRocks cluster You can access the StarRocks cluster from inside and outside the Kubernetes cluster. For detailed instructions, see [Access StarRocks Cluster](https://docs.starrocks.io/docs/deployment/sr_operator.md#access-starrocks-cluster). * Manage StarRocks operator and StarRocks cluster * If you need to update the configurations of the StarRocks operator and StarRocks cluster, see [Helm Upgrade](https://helm.sh/docs/helm/helm_upgrade/). * If you need to uninstall the StarRocks Operator and StarRocks cluster, run the following command: ```bash helm uninstall starrocks ``` #### More information[​](#more-information "Direct link to More information") * The address of the GitHub repository: [starrocks-kubernetes-operator and kube-starrocks Helm Chart](https://github.com/StarRocks/starrocks-kubernetes-operator). * The docs in the GitHub repository provide more information, for example: * If you need to manage objects like the StarRocks cluster via the Kubernetes API, see [API reference](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/api.md). * If you need to mount persistent volumes to FE and BE pods to store FE metadata and logs, as well as BE data and logs, see [Mount Persistent Volumes by Helm Chart](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/mount_persistent_volume_howto.md#2-mounting-persistent-volumes-by-helm-chart). danger If persistent volumes are not mounted, the StarRocks Operator will use emptyDir to store FE metadata and logs, as well as BE data and logs. When containers restart, data will be lost. * If you need to set the root user password: * Manually set the root user's password after deploying the StarRocks cluster, see [Change root user password HOWTO](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/change_root_password_howto.md). * Automatically set the root user's password when deploying the StarRocks cluster, see [Initialize root user password](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/initialize_root_password_howto.md). * How to resolve the following error that occurs after a CREATE TABLE statement is executed in a StarRocks shared-data cluster. * **Error message** ```plaintext ERROR 1064 (HY000): Table replication num should be less than or equal to the number of available BE nodes. You can change this default by setting the replication_num table properties. Current alive backend is [10001]. , table=orders1, default_replication_num=3 ``` * **Solution** This may be because only one BE exists in that StarRocks shared-data cluster, which supports only one replica. However, the default number of replicas is 3. You can modify the number of replicas to 1 in PROPERTIES, such as, `PROPERTIES( "replication_num" = "1" )`. * The address of Helm Chart maintained by StarRocks on Artifact Hub: [kube-starrocks](https://artifacthub.io/packages/helm/kube-starrocks/kube-starrocks). --- ### Manage your deployment #### [📄️ Post-deployment setup](https://docs.starrocks.io/docs/deployment/manage_deployment/post_deployment_setup.md) [Before getting your new StarRocks cluster into production, you must secure the initial account and set the necessary variables and properties to allow your...](https://docs.starrocks.io/docs/deployment/manage_deployment/post_deployment_setup.md) #### [📄️ Upgrade StarRocks](https://docs.starrocks.io/docs/deployment/manage_deployment/upgrade.md) [How to upgrade a StarRocks cluster to a newer version.](https://docs.starrocks.io/docs/deployment/manage_deployment/upgrade.md) #### [📄️ Downgrade StarRocks](https://docs.starrocks.io/docs/deployment/manage_deployment/downgrade.md) [If an exception occurs after you upgrade a StarRocks cluster, you can downgrade it to the earlier version to quickly recover the cluster.](https://docs.starrocks.io/docs/deployment/manage_deployment/downgrade.md) --- ### Downgrade StarRocks This topic describes how to downgrade your StarRocks cluster. If an exception occurs after you upgrade a StarRocks cluster, you can downgrade it to the earlier version to quickly recover the cluster. #### Overview[​](#overview "Direct link to Overview") Review the information in this section before downgrading. Perform any recommended actions. ##### Downgrade paths[​](#downgrade-paths "Direct link to Downgrade paths") * **For patch version downgrade** You can downgrade your StarRocks cluster across patch versions, for example, from v3.5.11 directly to v3.5.6. * **For minor version downgrade** For compatibility and safety reasons, we strongly recommend you downgrade your StarRocks cluster **consecutively from one minor version to another**. For example, to downgrade a StarRocks v3.5 cluster to v3.2, you need to downgrade it in the following order: v3.5.x --> v3.4.x --> v3.3.x --> v3.2.x. * **For major version downgrade** You can only downgrade your StarRocks v4.1 cluster to v4.0.6 and later versions. warning **Downgrade Notes** * After upgrading StarRocks to v4.1, DO NOT downgrade to any v4.0 version below v4.0.6. Due to internal changes in data layout introduced in v4.1 (related to tablet splitting and distribution mechanisms), clusters upgraded to v4.1 may generate metadata and storage structures that are not fully compatible with earlier versions. As a result, downgrade from v4.1 is only supported to v4.0.6 or later. Downgrading to versions prior to v4.0.6 is not supported. This limitation is due to backward compatibility constraints in how earlier versions interpret tablet layout and distribution metadata. ##### Downgrade procedure[​](#downgrade-procedure "Direct link to Downgrade procedure") StarRocks' downgrade procedure is the reverse order of the [upgrade procedure](https://docs.starrocks.io/docs/deployment/manage_deployment/upgrade.md#upgrade-procedure). Therefore, you need to **downgrade** **FEs** **first and then BEs and CNs**. Downgrading them in the wrong order may lead to incompatibility between FEs and BEs/CNs, and thereby cause the service to crash. For FE nodes, you must first downgrade all Follower FE nodes before downgrading the Leader FE node. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") During preparation, you must perform the compatibility configuration if you are up for a minor or major version downgrade. You also need to perform the downgrade availability test on one of the FEs or BEs before downgrading all nodes in the cluster. ##### Perform compatibility configuration[​](#perform-compatibility-configuration "Direct link to Perform compatibility configuration") If you want to downgrade your StarRocks cluster to an earlier minor or major version, you must perform the compatibility configuration. In addition to the universal compatibility configuration, detailed configurations vary depending on the version of the StarRocks cluster you downgrade from. * **Universal compatibility configuration** Before downgrading your StarRocks cluster, you must disable tablet clone. You can skip this step if you have disabled the balancer. ```sql ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "0"); ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "0"); ADMIN SET FRONTEND CONFIG ("disable_balance"="true"); ADMIN SET FRONTEND CONFIG ("disable_colocate_balance"="true"); ``` After the downgrade, you can enable tablet clone again if the status of all BE nodes becomes `Alive`. ```sql ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "10000"); ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "500"); ADMIN SET FRONTEND CONFIG ("disable_balance"="false"); ADMIN SET FRONTEND CONFIG ("disable_colocate_balance"="false"); ``` #### Downgrade FE[​](#downgrade-fe "Direct link to Downgrade FE") note To downgrade a cluster from v3.3.0 or later to v3.2, follow these steps before downgrading: 1. Ensure that all ALTER TABLE SCHEMA CHANGE transactions initiated in the v3.3 cluster are either completed or canceled before downgrading. 2. Clear all transaction history by executing the following command: ```sql ADMIN SET FRONTEND CONFIG ("history_job_keep_max_second" = "0"); ``` 3. Verify that there are no remaining historical records by running the following command: ```sql SHOW PROC '/jobs//schema_change'; ``` After the compatibility configuration and the availability test, you can downgrade the FE nodes. You must first downgrade the Follower FE nodes and then the Leader FE node. 1. Create a metadata snapshot. a. Run [ALTER SYSTEM CREATE IMAGE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md) to create a metadata snapshot. b. You can check whether the image file has been synchronized by viewing the log file **fe.log** of the Leader FE. A record of log like "push image.\* from subdir \[] to other nodes. totally xx nodes, push successful xx nodes" suggests that the image file has been successfully synchronized. 2. Navigate to the working directory of the FE node and stop the node. ```bash # Replace with the deployment directory of the FE node. cd /fe ./bin/stop_fe.sh ``` 3. Replace the original deployment files under **bin**, **lib**, and **spark-dpp** with the ones of the earlier version. ```bash mv lib lib.bak mv bin bin.bak mv spark-dpp spark-dpp.bak cp -r /tmp/StarRocks-x.x.x/fe/lib . cp -r /tmp/StarRocks-x.x.x/fe/bin . cp -r /tmp/StarRocks-x.x.x/fe/spark-dpp . ``` 4. Start the FE node. ```bash ./bin/start_fe.sh --daemon ``` 5. Check if the FE node is started successfully. ```bash ps aux | grep StarRocksFE ``` 6. Repeat the above Step 2 to Step 5 to downgrade other Follower FE nodes, and finally the Leader FE node. > **CAUTION** > > Suppose you have downgraded your cluster after a failed upgrade and you want to upgrade the cluster again, for example, 3.5->4.0->3.5->4.0. To prevent metadata upgrade failure for some Follower FEs, repeat Step 1 to trigger a new snapshot before upgrading. #### Downgrade BE[​](#downgrade-be "Direct link to Downgrade BE") Having downgraded the FE nodes, you can then downgrade the BE nodes in the cluster. 1. Navigate to the working directory of the BE node and stop the node. ```bash # Replace with the deployment directory of the BE node. cd /be ./bin/stop_be.sh ``` 2. Replace the original deployment files under **bin** and **lib** with the ones of the earlier version. ```bash mv lib lib.bak mv bin bin.bak cp -r /tmp/StarRocks-x.x.x/be/lib . cp -r /tmp/StarRocks-x.x.x/be/bin . ``` 3. Start the BE node. ```bash ./bin/start_be.sh --daemon ``` 4. Check if the BE node is started successfully. ```bash ps aux | grep starrocks_be ``` 5. Repeat the above procedures to downgrade other BE nodes. #### Downgrade CN[​](#downgrade-cn "Direct link to Downgrade CN") 1. Navigate to the working directory of the CN node and stop the node gracefully. ```bash # Replace with the deployment directory of the CN node. cd /be ./bin/stop_cn.sh --graceful ``` 2. Replace the original deployment files under **bin** and **lib** with the ones of the earlier version. ```bash mv lib lib.bak mv bin bin.bak cp -r /tmp/StarRocks-x.x.x/be/lib . cp -r /tmp/StarRocks-x.x.x/be/bin . ``` 3. Start the CN node. ```bash ./bin/start_cn.sh --daemon ``` 4. Check if the CN node is started successfully. ```bash ps aux | grep starrocks_be ``` 5. Repeat the above procedures to downgrade other CN nodes. --- ### Post-deployment setup This topic describes tasks that you should perform after deploying StarRocks. Before getting your new StarRocks cluster into production, you must secure the initial account and set the necessary variables and properties to allow your cluster to run properly. #### Secure initial account[​](#secure-initial-account "Direct link to Secure initial account") Upon the creation of a StarRocks cluster, the initial `root` user of the cluster is generated automatically. The `root` user is granted the `root` privileges, which are the collection of all privileges within the cluster. We recommend you secure this user account and avoid using it in production to prevent misuse. StarRocks automatically assigns an empty password to the `root` user when the cluster is created. Follow these procedures to set a new password for the `root` user: 1. Connect to StarRocks via your MySQL client with the username `root` and an empty password. ```sh # Replace with the IP address (priority_networks) or FQDN # of the FE node you connect to, and replace # with the query_port (Default: 9030) you specified in fe.conf. mysql -h -P -uroot ``` 2. Reset the password of the `root` user by executing the following SQL: ```sql -- Replace with the password you want to assign to the root user. SET PASSWORD = PASSWORD('') ``` note * Keep the password properly after resetting it. If you forgot the password, see [Reset lost root password](https://docs.starrocks.io/docs/administration/user_privs/authentication/native_authentication.md#reset-lost-root-password) for detailed instructions. * After completing the post-deployment setup, you can create new users and roles to manage the privileges within your team. See [Manage user privileges](https://docs.starrocks.io/docs/administration/user_privs/authorization/User_privilege.md) for detailed instructions. #### Set necessary system variables[​](#set-necessary-system-variables "Direct link to Set necessary system variables") To allow your StarRocks cluster to work properly in production, you need to set the following system variables: ##### enable\_profile[​](#enable_profile "Direct link to enable_profile") ###### Description[​](#description "Direct link to Description") The boolean switch that controls whether to send the profile of a query for analysis. The default value is `false`, which means no profile is required. Setting this variable to `true` can affect the concurrency of StarRocks. ###### Recommended value[​](#recommended-value "Direct link to Recommended value") false * Set `enable_profile` to `false` globally: ```sql SET GLOBAL enable_profile = false; ``` ##### enable\_pipeline\_engine[​](#enable_pipeline_engine "Direct link to enable_pipeline_engine") ###### Description[​](#description-1 "Direct link to Description") The boolean switch that controls whether to enable the pipeline execution engine. `true` indicates enabled and `false` indicates the opposite. Default value: `true`. ###### Recommended value[​](#recommended-value-1 "Direct link to Recommended value") true * Set `enable_pipeline_engine` to `true` globally: ```sql SET GLOBAL enable_pipeline_engine = true; ``` ##### parallel\_fragment\_exec\_instance\_num[​](#parallel_fragment_exec_instance_num "Direct link to parallel_fragment_exec_instance_num") ###### Description[​](#description-2 "Direct link to Description") The number of instances used to scan nodes on each BE. The default value is `1`. ###### Recommended value[​](#recommended-value-2 "Direct link to Recommended value") If you have enabled the pipeline engine, you can set this variable to `1`. If you have not enabled the pipeline engine, you should set it to half the number of CPU cores. * Set `parallel_fragment_exec_instance_num` to `1` globally: ```sql SET GLOBAL parallel_fragment_exec_instance_num = 1; ``` For more information about system variables, see [System variables](https://docs.starrocks.io/docs/sql-reference/System_variable.md). #### Set user property[​](#set-user-property "Direct link to Set user property") If you have created new users in your cluster, you need to enlarge their maximum connection number (to `1000`, for example): ```sql -- Replace with the username you want to enlarge the maximum connection number for. ALTER USER '' SET PROPERTIES ("max_user_connections" = "1000"); ``` #### What to do next[​](#what-to-do-next "Direct link to What to do next") After deploying and setting up your StarRocks cluster, you can then proceed to design tables that best work for your scenarios. See [Understand StarRocks table design](https://docs.starrocks.io/docs/table_design.md) for detailed instructions on designing a table. --- ### Upgrade StarRocks This topic describes how to upgrade your StarRocks cluster. #### Important information[​](#important-information "Direct link to Important information") important Before upgrading StarRocks you should: * Read the [release notes](https://docs.starrocks.io/releasenotes/release-3.5/) for the StarRocks version that you are upgrading to, and all versions between the current version and the target version and: * Make notes of any behavior changes within StarRocks * Make notes of any changes with integrations between StarRocks and external systems used for importing, exporting, visualization, etc * Verify the [deployment prerequisites](https://docs.starrocks.io/docs/deployment/preparation/deployment_prerequisites.md) for the target version. For example, StarRocks 3.5.x requires JDK 17 and StarRocks 3.4.x on Ubuntu requires JDK 11. #### Overview[​](#overview "Direct link to Overview") Review the information in this section before upgrading, and perform any recommended actions. ##### StarRocks versions[​](#starrocks-versions "Direct link to StarRocks versions") The version of StarRocks is represented by three numbers in the form **Major.Minor.Patch**, for example, `2.5.4`. The first number represents the major version of StarRocks, the second number represents the minor version, and the third number represents the patch version. > **CAUTION** > > Please note that you cannot upgrade an existing shared-nothing cluster to a shared-data cluster, or vice versa. You must deploy a new shared-data cluster. ##### Upgrade paths[​](#upgrade-paths "Direct link to Upgrade paths") * **For patch version upgrade** You can upgrade your StarRocks cluster across patch versions, for example, from v2.2.6 directly to v2.2.11. * **For minor version upgrade** From StarRocks v2.0 onwards, you can upgrade a StarRocks cluster across minor versions, for example, from v2.2.x directly to v2.5.x. However, for compatibility and safety reasons, we strongly recommend you upgrade your StarRocks cluster **consecutively from one minor version to another**. For example, to upgrade a StarRocks v2.2 cluster to v2.5, you need to upgrade it in the following order: v2.2.x --> v2.3.x --> v2.4.x --> v2.5.x. * **For major version upgrade** To upgrade your StarRocks cluster to v3.0, you must first upgrade it to v2.5. > **CAUTION** > > Suppose you need to perform consecutive minor version upgrades, for example, 2.4->2.5->3.0->3.1->3.2, or you have downgraded your cluster after a failed upgrade and you want to upgrade the cluster again, for example, 2.5->3.0->2.5->3.0. To prevent metadata upgrade failure for some Follower FEs, perform the following steps between two consecutive upgrades or after the downgrade before the second trial of upgrade: > > 1. Run [ALTER SYSTEM CREATE IMAGE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md) to create a new image. > 2. Wait for the new image to be synchronized to all Follower FEs. > > You can check whether the image file has been synchronized by viewing the log file **fe.log** of the Leader FE. A record of log like "push image.\* from subdir \[] to other nodes. totally xx nodes, push successful xx nodes" suggests that the image file has been successfully synchronized. ##### Upgrade procedure[​](#upgrade-procedure "Direct link to Upgrade procedure") StarRocks supports **rolling upgrades**, which allow you to upgrade your cluster without stopping the service. By design, BEs and CNs are backward compatible with the FEs. Therefore, you need to **upgrade BEs and CNs first and then FEs** to allow your cluster to run properly while being upgraded. Upgrading them in an inverted order may lead to incompatibility between FEs and BEs/CNs, and thereby cause the service to crash. For FE nodes, you must first upgrade all Follower FE nodes before upgrading the Leader FE node. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") During preparation, you must perform the compatibility configuration if you are up for a minor or major version upgrade. You also need to perform the upgrade availability test on one of the FEs and BEs before upgrading all nodes in the cluster. ##### Perform compatibility configuration[​](#perform-compatibility-configuration "Direct link to Perform compatibility configuration") If you want to upgrade your StarRocks cluster to a later minor or major version, you must perform the compatibility configuration. In addition to the universal compatibility configuration, detailed configurations vary depending on the version of the StarRocks cluster you upgrade from. * **Universal compatibility configuration** Before upgrading your StarRocks cluster, you must disable tablet clone. You can skip this step if you have disabled the balancer. ```sql ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "0"); ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "0"); ADMIN SET FRONTEND CONFIG ("disable_balance"="true"); ADMIN SET FRONTEND CONFIG ("disable_colocate_balance"="true"); ``` After the upgrade, and the status of all BE nodes is `Alive`, you can re-enable tablet clone. ```sql ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "10000"); ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "500"); ADMIN SET FRONTEND CONFIG ("disable_balance"="false"); ADMIN SET FRONTEND CONFIG ("disable_colocate_balance"="false"); ``` * **If you upgrade from v2.0 to later versions** Before upgrading your StarRocks v2.0 cluster, you must set the following BE configuration and system variable. 1. If you have modified the BE configuration item `vector_chunk_size`, you must set it to `4096` before upgrading. Because it is a static parameter, you must modify it in the BE configuration file **be.conf** and restart the node to allow the modification to take effect. 2. Set the system variable `batch_size` to less than or equal to `4096` globally. ```sql SET GLOBAL batch_size = 4096; ``` #### Upgrade BE[​](#upgrade-be "Direct link to Upgrade BE") Having passed the upgrade availability test, you can first upgrade the BE nodes in the cluster. 1. Navigate to the working directory of the BE node and stop the node. ```bash # Replace with the deployment directory of the BE node. cd /be ./bin/stop_be.sh ``` 2. Replace the original deployment files under **bin** and **lib** with the ones of the new version. ```bash mv lib lib.bak mv bin bin.bak cp -r /tmp/StarRocks-x.x.x/be/lib . cp -r /tmp/StarRocks-x.x.x/be/bin . # If a custom function (UDF) was used in the old version, you need to copy the old version's UDF directory to the new lib directory. cp -r lib.bak/udf lib/ ``` 3. Start the BE node. ```bash ./bin/start_be.sh --daemon ``` 4. Check if the BE node is started successfully. ```bash ps aux | grep starrocks_be ``` 5. Repeat the above procedures to upgrade other BE nodes. #### Upgrade CN[​](#upgrade-cn "Direct link to Upgrade CN") 1. Navigate to the working directory of the CN node and stop the node gracefully. ```bash # Replace with the deployment directory of the CN node. cd /be ./bin/stop_cn.sh --graceful ``` 2. Replace the original deployment files under **bin** and **lib** with the ones of the new version. ```bash mv lib lib.bak mv bin bin.bak cp -r /tmp/StarRocks-x.x.x/be/lib . cp -r /tmp/StarRocks-x.x.x/be/bin . # If a custom function (UDF) was used in the old version, you need to copy the old version's UDF directory to the new lib directory. cp -r lib.bak/udf lib/ ``` 3. Start the CN node. ```bash ./bin/start_cn.sh --daemon ``` 4. Check if the CN node is started successfully. ```bash ps aux | grep starrocks_be ``` 5. Repeat the above procedures to upgrade other CN nodes. #### Upgrade FE[​](#upgrade-fe "Direct link to Upgrade FE") After upgrading all BE and CN nodes, you can then upgrade the FE nodes. You must first upgrade the Follower FE nodes and then the Leader FE node. 1. Navigate to the working directory of the FE node and stop the node. ```bash # Replace with the deployment directory of the FE node. cd /fe ./bin/stop_fe.sh ``` 2. Replace the original deployment files under **bin**, **lib**, and **spark-dpp** with the ones of the new version. ```bash mv lib lib.bak mv bin bin.bak mv spark-dpp spark-dpp.bak cp -r /tmp/StarRocks-x.x.x/fe/lib . cp -r /tmp/StarRocks-x.x.x/fe/bin . cp -r /tmp/StarRocks-x.x.x/fe/spark-dpp . ``` 3. Start the FE node. ```bash ./bin/start_fe.sh --daemon ``` 4. Check if the FE node is started successfully. ```bash ps aux | grep StarRocksFE ``` 5. Repeat the above procedures to upgrade other Follower FE nodes, and finally the Leader FE node. --- ### Preparation #### [📄️ Deployment Prerequisites](https://docs.starrocks.io/docs/deployment/preparation/deployment_prerequisites.md) [Hardware and software requirements for servers before deploying StarRocks.](https://docs.starrocks.io/docs/deployment/preparation/deployment_prerequisites.md) #### [📄️ Plan StarRocks Cluster](https://docs.starrocks.io/docs/deployment/preparation/plan_cluster.md) [How to plan node count, CPU, memory, and storage for a StarRocks cluster in production.](https://docs.starrocks.io/docs/deployment/preparation/plan_cluster.md) #### [📄️ Check environment configurations](https://docs.starrocks.io/docs/deployment/preparation/environment_configurations.md) [Environment and system configuration checklist to complete before deploying StarRocks.](https://docs.starrocks.io/docs/deployment/preparation/environment_configurations.md) #### [📄️ Prepare deployment files](https://docs.starrocks.io/docs/deployment/preparation/prepare_deployment_files.md) [How to prepare StarRocks deployment files for x86 or ARM-based CPU deployments.](https://docs.starrocks.io/docs/deployment/preparation/prepare_deployment_files.md) --- ### Deployment Prerequisites This topic describes the hardware and software requirements that your servers must meet before deploying StarRocks. For recommended hardware specifications of your StarRocks cluster, see [Plan your StarRocks cluster](https://docs.starrocks.io/docs/deployment/preparation/plan_cluster.md). #### Hardware[​](#hardware "Direct link to Hardware") ##### CPU[​](#cpu "Direct link to CPU") StarRocks relies on AVX2 instruction sets to fully unleash its vectorization capability. Therefore, in a production environment, we highly recommend you deploy StarRocks on machines with x86 architecture CPUs. You can run the following command in your terminal to check if the CPUs on your machines support the AVX2 instruction sets: ```bash cat /proc/cpuinfo | grep avx2 ``` ##### Memory[​](#memory "Direct link to Memory") No specific requirement is imposed on memory kits used for StarRocks. See [Plan StarRocks cluster - CPU and Memory](https://docs.starrocks.io/docs/deployment/preparation/plan_cluster.md#cpu-and-memory) for the recommended memory size. ##### Storage[​](#storage "Direct link to Storage") StarRocks supports both HDD and SSD as storage medium. If your applications require real-time data analytics, intensive data scans, or random disk access, we strongly recommend you use SSD storage. If your applications involve [Primary Key tables](https://docs.starrocks.io/docs/table_design/table_types/primary_key_table.md) with the persistent index, you must use SSD storage. ##### Network[​](#network "Direct link to Network") We recommend that you use 10 Gigabit Ethernet networking to ensure stable data transmission across nodes within your StarRocks cluster. #### Operating System[​](#operating-system "Direct link to Operating System") StarRocks supports deployments on Red Hat Enterprise Linux 7.9, CentOS Linux 7.9 or Ubuntu Linux 22.04. #### Software[​](#software "Direct link to Software") You must install the corresponding JDK version on the server to run StarRocks. * For StarRocks v3.3 and v3.4, use JDK 11 or later. * For StarRocks v3.5 and later, use JDK 17 or later. important StarRocks does not support JRE. --- ### Check environment configurations This topic lists all environment and system configuration items that you must check and set before deploying StarRocks. Setting these configuration items properly allows your StarRocks cluster to work with high availability and performance. #### Ports[​](#ports "Direct link to Ports") StarRocks uses specific ports for different services. Check whether these ports are occupied on each instance if you have deployed other services on these instances. ##### FE ports[​](#fe-ports "Direct link to FE ports") On the instances used for the FE deployment, you need to check the following ports: * `8030`: FE HTTP server port (`http_port`) * `9020`: FE Thrift server port (`rpc_port`) * `9030`: FE MySQL server port (`query_port`) * `9010`: FE internal communication port (`edit_log_port`) * `6090`: FE cloud-native metadata server RPC listen port (`cloud_native_meta_port`) Run the following commands on the FE instances to check whether these ports are occupied: ```bash netstat -tunlp | grep 8030 netstat -tunlp | grep 9020 netstat -tunlp | grep 9030 netstat -tunlp | grep 9010 netstat -tunlp | grep 6090 ``` If any of the above ports are occupied, you must find alternatives and specify them later when you deploy FE nodes. For detailed instructions, see [Deploy StarRocks - Start the Leader FE node](https://docs.starrocks.io/docs/deployment/deploy_manually.md#step-1-start-the-leader-fe-node). ##### BE ports[​](#be-ports "Direct link to BE ports") On the instances used for the BE deployment, you need to check the following ports: * `9060`: BE Thrift server port (`be_port`) * `8040`: BE HTTP server port (`be_http_port`) * `9050`: BE heartbeat service port (`heartbeat_service_port`) * `8060`: BE bRPC port (`brpc_port`) * `9070`: An extra agent service port for BE and CN (`starlet_port`) Run the following commands on the BE instances to check whether these ports are occupied: ```bash netstat -tunlp | grep 9060 netstat -tunlp | grep 8040 netstat -tunlp | grep 9050 netstat -tunlp | grep 8060 netstat -tunlp | grep 9070 ``` If any of the above ports are occupied, you must find alternatives and specify them later when you deploy BE nodes. For detailed instructions, see [Deploy StarRocks - Start the BE service](https://docs.starrocks.io/docs/deployment/deploy_manually.md#step-2-start-the-be-service). ##### CN ports[​](#cn-ports "Direct link to CN ports") On the instances used for the CN deployment, you need to check the following ports: * `9060`: CN Thrift server port (`be_port`) * `8040`: CN HTTP server port (`be_http_port`) * `9050`: CN heartbeat service port (`heartbeat_service_port`) * `8060`: CN bRPC port (`brpc_port`) * `9070`: An extra agent service port for BE and CN (`starlet_port`) Run the following commands on the CN instances to check whether these ports are occupied: ```bash netstat -tunlp | grep 9060 netstat -tunlp | grep 8040 netstat -tunlp | grep 9050 netstat -tunlp | grep 8060 netstat -tunlp | grep 9070 ``` If any of the above ports are occupied, you must find alternatives and specify them later when you deploy CN nodes. For detailed instructions, see [Deploy StarRocks - Start the CN service](https://docs.starrocks.io/docs/deployment/deploy_shared_data_manually.md#step-2-start-the-cn-service). #### Hostnames[​](#hostnames "Direct link to Hostnames") If you want to [enable FQDN access](https://docs.starrocks.io/docs/administration/management/enable_fqdn.md) for your StarRocks cluster, you must assign a hostname to each instance. In the file **/etc/hosts** on each instance, you must specify the IP addresses and corresponding hostnames of all the other instances in the cluster. important All IP addresses in the file **/etc/hosts** must be unique. #### JDK configuration[​](#jdk-configuration "Direct link to JDK configuration") StarRocks relies on the environment variable `JAVA_HOME` to locate the Java dependency on the instance. Run the following command to check the environment variable `JAVA_HOME`: ```bash echo $JAVA_HOME ``` Follow these steps to set `JAVA_HOME`: 1. Set `JAVA_HOME` in the file **/etc/profile**: ```bash sudo vi /etc/profile # Replace with the path where JDK is installed. export JAVA_HOME= export PATH=$PATH:$JAVA_HOME/bin ``` 2. Bring the change into effect: ```bash source /etc/profile ``` Run the following command to verify the change: ```bash java -version ``` #### CPU scaling governor[​](#cpu-scaling-governor "Direct link to CPU scaling governor") This configuration item is **optional**. You can skip it if your CPU does not support the scaling governor. The CPU scaling governor controls the CPU power mode. If your CPU supports it, we recommend you set it to `performance` for better CPU performance: ```bash echo 'performance' | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor ``` #### Memory configurations[​](#memory-configurations "Direct link to Memory configurations") ##### Memory Overcommit[​](#memory-overcommit "Direct link to Memory Overcommit") Memory Overcommit allows the operating system to overcommit memory resources to processes. We recommend you enable Memory Overcommit. ```bash # Modify the configuration file. cat >> /etc/sysctl.conf << EOF vm.overcommit_memory=1 EOF # Bring the change into effect. sysctl -p ``` ##### Transparent Huge Pages[​](#transparent-huge-pages "Direct link to Transparent Huge Pages") Transparent Huge Pages are enabled by default. We recommend you disable this feature because it can interfere with the memory allocator, and thereby lead to a drop in performance. ```bash # Change the configuration temporarily. echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag # Change the configuration permanently. cat >> /etc/rc.d/rc.local << EOF if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo madvise > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo madvise > /sys/kernel/mm/transparent_hugepage/defrag fi EOF chmod +x /etc/rc.d/rc.local ``` ##### Swap Space[​](#swap-space "Direct link to Swap Space") We recommend you disable Swap Space. Follow these steps to check and disable Swap Space: 1. Disable Swap Space. ```sql swapoff / swapoff -a ``` 2. Delete the Swap Space information from the configuration file **/etc/fstab**. ```bash / swap swap defaults 0 0 ``` 3. Verify that Swap Space is disabled. ```bash free -m ``` ##### Swappiness[​](#swappiness "Direct link to Swappiness") We recommend you disable swappiness to eliminate its impact on performance. ```bash # Modify the configuration file. cat >> /etc/sysctl.conf << EOF vm.swappiness=0 EOF # Bring the change into effect. sysctl -p ``` #### Storage configurations[​](#storage-configurations "Direct link to Storage configurations") We recommend that you choose your suitable scheduler algorithm in accordance with the storage medium you use. You can run the following command to check the scheduler algorithm that you are using: ```bash cat /sys/block/${disk}/queue/scheduler # For example, run cat /sys/block/vdb/queue/scheduler ``` We recommend you use the mq-deadline scheduler for SATA disks and the kyber scheduler algorithm for SSD and NVMe disks. ##### SATA[​](#sata "Direct link to SATA") The mq-deadline scheduler algorithm suits SATA disks. ```bash # Change the configuration temporarily. echo mq-deadline | sudo tee /sys/block/${disk}/queue/scheduler # Change the configuration permanently. cat >> /etc/rc.d/rc.local << EOF echo mq-deadline | sudo tee /sys/block/${disk}/queue/scheduler EOF chmod +x /etc/rc.d/rc.local ``` ##### SSD and NVMe[​](#ssd-and-nvme "Direct link to SSD and NVMe") * If your NVMe or SSD disks support the kyber scheduler algorithm: ```bash # Change the configuration temporarily. echo kyber | sudo tee /sys/block/${disk}/queue/scheduler # Change the configuration permanently. cat >> /etc/rc.d/rc.local << EOF echo kyber | sudo tee /sys/block/${disk}/queue/scheduler EOF chmod +x /etc/rc.d/rc.local ``` * If your NVMe or SSD disks support the none (or noop) scheduler. ```bash # Change the configuration temporarily. echo none | sudo tee /sys/block/vdb/queue/scheduler # Change the configuration permanently. cat >> /etc/rc.d/rc.local << EOF echo none | sudo tee /sys/block/${disk}/queue/scheduler EOF chmod +x /etc/rc.d/rc.local ``` #### SELinux[​](#selinux "Direct link to SELinux") We recommend you disable SELinux. ```bash # Change the configuration temporarily. setenforce 0 # Change the configuration permanently. sed -i 's/SELINUX=.*/SELINUX=disabled/' /etc/selinux/config sed -i 's/SELINUXTYPE/#SELINUXTYPE/' /etc/selinux/config ``` #### Firewall[​](#firewall "Direct link to Firewall") Open the internal ports for FE nodes, BE nodes, and Broker if your firewall is enabled. ```bash systemctl stop firewalld.service systemctl disable firewalld.service ``` #### LANG variable[​](#lang-variable "Direct link to LANG variable") Run the following command to check and configure the LANG variable manually: ```bash # Modify the configuration file. echo "export LANG=en_US.UTF8" >> /etc/profile # Bring the change into effect. source /etc/profile ``` #### Time zone[​](#time-zone "Direct link to Time zone") Set this item in accordance with your actual time zone. The following example sets the time zone to `/Asia/Shanghai`. ```bash cp -f /usr/share/zoneinfo/Asia/Shanghai /etc/localtime hwclock ``` #### ulimit configurations[​](#ulimit-configurations "Direct link to ulimit configurations") Problems can occur with StarRocks if the values of **max file descriptors** and **max user processes** are abnormally small. We recommend you enlarge these values. ```bash cat >> /etc/security/limits.conf << EOF * soft nproc 65535 * hard nproc 65535 * soft nofile 655350 * hard nofile 655350 * soft stack unlimited * hard stack unlimited * hard memlock unlimited * soft memlock unlimited EOF cat >> /etc/security/limits.d/20-nproc.conf << EOF * soft nproc 65535 root soft nproc 65535 EOF ``` #### File system configuration[​](#file-system-configuration "Direct link to File system configuration") We recommend you use the ext4 or xfs journaling file system. You can run the following command to check the mount type: ```bash df -Th ``` #### Network configuration[​](#network-configuration "Direct link to Network configuration") ##### tcp\_abort\_on\_overflow[​](#tcp_abort_on_overflow "Direct link to tcp_abort_on_overflow") Allow the system to reset new connections if the system is currently overflowed with new connection attempts that the daemon(s) can not handle: ```bash # Modify the configuration file. cat >> /etc/sysctl.conf << EOF net.ipv4.tcp_abort_on_overflow=1 EOF # Bring the change into effect. sysctl -p ``` ##### somaxconn[​](#somaxconn "Direct link to somaxconn") Specify the maximum number of connection requests queued for any listening socket to `1024`: ```bash # Modify the configuration file. cat >> /etc/sysctl.conf << EOF net.core.somaxconn=1024 EOF # Bring the change into effect. sysctl -p ``` #### NTP configuration[​](#ntp-configuration "Direct link to NTP configuration") You must configure time synchronization between nodes within your StarRocks cluster to ensure linear consistency of transactions. You can either use the internet time service provided by pool.ntp.org, or use the NTP service built in an offline environment. For example, you can use the NTP service provided by your cloud service provider. 1. Check if the NTP time server or Chrony service exists. ```bash rpm -qa | grep ntp systemctl status chronyd ``` 2. Install the NTP service if there is not one. ```bash sudo yum install ntp ntpdate && \ sudo systemctl start ntpd.service && \ sudo systemctl enable ntpd.service ``` 3. Check the NTP service. ```bash systemctl list-unit-files | grep ntp ``` 4. Check the connectivity and monitoring status of the NTP service. ```bash netstat -tunlp | grep ntp ``` 5. Check if your application is synchronized with the NTP server. ```bash ntpstat ``` 6. Check the state of all the configured NTP servers in your network. ```bash ntpq -p ``` #### High concurrency configurations[​](#high-concurrency-configurations "Direct link to High concurrency configurations") If your StarRocks cluster has a high load concurrency, we recommend you set the following configurations. ##### max\_map\_count[​](#max_map_count "Direct link to max_map_count") Specify the maximum number of memory map areas a process may have as `262144`: ```bash # Modify the configuration file. cat >> /etc/sysctl.conf << EOF vm.max_map_count = 262144 EOF # Bring the change into effect. sysctl -p ``` ##### Other[​](#other "Direct link to Other") ```bash echo 120000 > /proc/sys/kernel/threads-max echo 200000 > /proc/sys/kernel/pid_max ``` --- ### Plan StarRocks Cluster This topic describes how to plan resources for your StarRocks cluster in production from the perspectives of node count, CPU core count, memory size, and storage size. #### Node count[​](#node-count "Direct link to Node count") StarRocks mainly consists of two types of components: FE nodes and BE/CN nodes. Each node must be deployed separately on a physical or virtual machine. ##### FE node count[​](#fe-node-count "Direct link to FE node count") FE nodes are mainly responsible for metadata management, client connection management, query planning, and query scheduling. In production, we recommend you deploy at least **THREE** Follower FE nodes in your StarRocks cluster to prevent single points of failure (SPOFs). The Leader FE will be automatically elected from these three followers. StarRocks uses the Raft protocol to manage the metadata across FE nodes. StarRocks elects a Leader FE node from all Follower FE nodes. Only the Leader FE node can write metadata. The other Follower FE nodes only update their metadata based on the logs from the Leader FE node. Each time the Leader FE node fails, StarRocks re-elects a new Leader FE node as long as more than half of the Follower FE nodes are alive. If your application generates highly concurrent query requests, you can add Observer FE nodes to your cluster. Observer FE nodes only process the query requests and do not participate in the election for the Leader FE node. ##### BE node count[​](#be-node-count "Direct link to BE node count") BE nodes are responsible for data storage and SQL execution in **shared-nothing** clusters. In production, we recommend you deploy at least **THREE** BE nodes in your StarRocks cluster to ensure high data reliability and service availability. A high-availability cluster of BEs is automatically formed when at least three BE nodes are deployed and added to your StarRocks cluster. The failure of one BE node will not affect the overall availability of the BE services. You can increase the number of BE nodes to enable your StarRocks cluster to process highly concurrent queries. ##### CN node count[​](#cn-node-count "Direct link to CN node count") CN nodes are responsible for data caching and SQL execution in **shared-data** clusters. You can increase the number of CN nodes to elastically scale compute resources in your StarRocks cluster. #### CPU and memory[​](#cpu-and-memory "Direct link to CPU and memory") Usually, the FE service does not consume a lot of CPU and memory resources. We recommend allocating 8 CPU cores and 16 GB RAM to each FE node. Unlike the FE service, the BE/CN service can be significantly CPU- and memory-intensive if your application works with highly concurrent or complex queries on a large dataset. Therefore, we recommend allocating 16 CPU cores and 64 GB RAM to each BE/CN node. #### Storage capacity[​](#storage-capacity "Direct link to Storage capacity") ##### FE storage[​](#fe-storage "Direct link to FE storage") Because FE nodes only maintain StarRocks' metadata in their storage, 100 GB of HDD storage is enough for each FE node in most scenarios. ##### BE storage[​](#be-storage "Direct link to BE storage") ###### Estimate initial storage space for BE[​](#estimate-initial-storage-space-for-be "Direct link to Estimate initial storage space for BE") The total storage space that your StarRocks cluster will need is simultaneously influenced by the size of your raw data, the data replica count, and the compression ratio of the data compression algorithm you use. With the following formula, you can estimate the total storage space you will need for all BE nodes: ```plain Total BE storage space = Raw data size * Replica count/Compression ratio Raw data size = Sum of the space taken up by all fields in a row * Row count ``` In StarRocks, data in a table is first divided into multiple partitions and then into multiple tablets. Tablets are the basic logical units of data management in StarRocks. To ensure high data reliability, you can maintain multiple replicas of each tablet and store them across different BEs. By default, StarRocks maintains three replicas. Currently, StarRocks supports four data compression algorithms, which are listed in order from higher to lower compression ratio: zlib, Zstandard (or zstd), LZ4, and Snappy. They can provide a compression ratio from 3:1 to 5:1. After determining the total storage space, you can simply divide it by the number of BE nodes in your cluster to estimate the average storage space per BE node. ###### Add extra storage as you go[​](#add-extra-storage-as-you-go "Direct link to Add extra storage as you go") If the BE storage space runs out as your raw data grows, you can supplement it by scaling your cluster vertically or horizontally, or simply scaling up your cloud storage. * Add new BE nodes to your StarRocks cluster You can add new BE nodes to your StarRocks cluster so that the data can be re-distributed evenly to more nodes. For detailed instructions, see [Scale your StarRocks cluster - Scale BE out](https://docs.starrocks.io/docs/administration/management/Scale_up_down.md). After new BE nodes are added, StarRocks automatically re-balances the data among all BE nodes. Such auto-balancing is supported on all table types. * Add extra storage volumes to your BE nodes You can also add extra storage volumes to existing BE nodes. For detailed instructions, see [Scale your StarRocks cluster - Scale BE up](https://docs.starrocks.io/docs/administration/management/Scale_up_down.md). After extra storage volumes are added, StarRocks automatically re-balances the data in all tables. * Add cloud storage If your StarRocks cluster is deployed on cloud, you can scale up your cloud storage on demand. For detailed instructions, contact your cloud provider. ##### CN storage[​](#cn-storage "Direct link to CN storage") Raw data in shared-data clusters is stored in remote storage. You can scale the storage space whenever needed. The local disk for a CN node is used to cache hot data for query acceleration. You can estimate the local disk space based on the hot data size of your daily business scenario. --- ### Prepare deployment files This topic describes how to prepare StarRocks deployment files. Currently, the binary distribution packages StarRocks provides on [the StarRocks official website](https://www.starrocks.io/download/community) support deployments only on x86-based CPU. If you want to deploy StarRocks with the ARM-based CPU, you need to prepare the deployment files using the StarRocks Docker image. #### For x86-based CPU[​](#for-x86-based-cpu "Direct link to For x86-based CPU") From v3.1.14, v3.2.10, and v3.3.3, StarRocks binary distribution packages are named in the `StarRocks-{Version}-{OS}-{ARCH}.tar.gz` format, where `Version` is a number (for example, `3.3.3`) that indicates the version information of the binary distribution package, `OS` indicates the operating system (including `centos` and `ubuntu`), and `ARCH` indicates the CPU architecture (currently only `amd64` is available, which is equivalent to x86\_64). Make sure that you have chosen the correct version of the package. note In versions earlier than v3.1.14, v3.2.10, and v3.3.3, StarRocks provides binary distribution packages named in the `StarRocks-{Version}.tar.gz` format. Follow these steps to prepare deployment files for the x86-based platform: 1. Obtain the StarRocks binary distribution package directly from the [Download StarRocks](https://www.starrocks.io/download/community) page or by running the following command in your terminal: ```bash # Replace with the version of StarRocks you want to download, for example, 3.3.3, # and replace with centos or ubuntu. wget https://releases.starrocks.io/starrocks/StarRocks---amd64.tar.gz ``` 2. Extract the files in the package. ```bash # Replace with the version of StarRocks you want to download, for example, 3.3.3, # and replace with centos or ubuntu. tar -xzvf StarRocks---amd64.tar.gz ``` The package includes the following directories and files: | **Directory/File** | **Description** | | ------------------------ | -------------------------------------------- | | **apache\_hdfs\_broker** | The deployment directory of the Broker node. | | **fe** | The FE deployment directory. | | **be** | The BE deployment directory. | | **LICENSE.txt** | The StarRocks license file. | | **NOTICE.txt** | The StarRocks notice file. | 3. Dispatch the directory **fe** to all the FE instances and the directory **be** to all the BE or CN instances for [manual deployment](https://docs.starrocks.io/docs/deployment/deploy_manually.md). #### For ARM-based CPU[​](#for-arm-based-cpu "Direct link to For ARM-based CPU") ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") You must have [Docker Engine](https://docs.docker.com/engine/install/) (17.06.0 or later) installed on your machine. ##### Procedures[​](#procedures "Direct link to Procedures") From v3.1.14, v3.2.10, and v3.3.3, StarRocks provides Docker images in the `starrocks/artifacts-{OS}:{Version}` format, where `OS` indicates the operating system (including `centos7` and `ubuntu`), and `Version` is the version number (for example, `3.3.3`). Docker will automatically identify your CPU architecture and pull the corresponding image. Make sure that you have chosen the correct version of the image. note In versions earlier than v3.1.14, v3.2.10, and v3.3.3, StarRocks provides Docker images in the repositories `starrocks/artifacts-ubuntu` and `starrocks/artifacts-centos7`. 1. Download a StarRocks Docker image from [StarRocks Docker Hub](https://hub.docker.com/u/starrocks?page=1\&search=artifacts). You can choose a specific version based on the tag of the image. ```bash # Replace with centos7 or ubuntu, # and replace with the version of StarRocks you want to download, for example, 3.3.3. # e.g. docker pull starrocks/artifacts-centos7:3.3.3 or docker pull starrocks/artifacts-ubuntu:3.3.3 docker pull starrocks/artifacts-: ``` 2. Copy the StarRocks deployment files from the Docker image to your host machine by running the following command: ```bash # Replace with centos7 or ubuntu, # and replace with the version of StarRocks you want to download, for example, 3.3.3. docker run --rm starrocks/artifacts-: \ tar -cf - -C /release . | tar -xvf - ``` 3. Dispatch the deployment files to the corresponding instances for [manual deployment](https://docs.starrocks.io/docs/deployment/deploy_manually.md). --- ### StarRocks Kubernetes Operator Automate deployment and management of a StarRocks cluster on a Kubernetes cluster with the StarRocks Kubernetes Operator. note The StarRocks k8s operator was designed to be a level 2 operator. See to understand more about the capabilities of a level 2 operator. #### How it works[​](#how-it-works "Direct link to How it works") ![img](/assets/images/starrocks_operator-5c0529d834fcc0959839194ce33ec326.png) #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Create Kubernetes cluster[​](#create-kubernetes-cluster "Direct link to Create Kubernetes cluster") You can use the cloud-managed Kubernetes service, such as an [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/?nc1=h_ls) or [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) cluster, or a self-managed Kubernetes cluster. * Create an Amazon EKS cluster 1. Check that [the following command-line tools are installed in your environment](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html): 1. Install and configure AWS command-line tool AWS CLI. 2. Install EKS cluster command-line tool eksctl. 3. Install the Kubernetes cluster command-line tool kubectl. 2. Use one of the following methods to create an EKS cluster: 1. [Use eksctl to quickly create an EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html). 2. [Manually create an EKS cluster with the AWS console and AWS CLI](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html). * Create a GKE cluster Before you start to create a GKE cluster, make sure that you complete all the [prerequisites](https://docs.cloud.google.com/kubernetes-engine/docs/deploy-app-cluster#before-you-begin). Then follow the instructions provided in [Create a GKE cluster](https://docs.cloud.google.com/kubernetes-engine/docs/deploy-app-cluster#create_cluster) to create a GKE cluster. * Create a self-managed Kubernetes cluster Follow the instructions provided in [Bootstrapping clusters with kubeadm](https://kubernetes.io/docs) to create a self-managed Kubernetes cluster. You can use Minikube and Docker Desktop to create a single-node private Kubernetes cluster with minimum steps. ##### Deploy StarRocks Kubernetes Operator[​](#deploy-starrocks-kubernetes-operator "Direct link to Deploy StarRocks Kubernetes Operator") 1. Add the custom resource StarRocksCluster. ```bash kubectl apply -f https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/deploy/starrocks.com_starrocksclusters.yaml ``` 2. Deploy the StarRocks Operator. You can choose to deploy the StarRocks Operator by using a default configuration file or a custom configuration file. 1. Deploy the StarRocks Operator by using a default configuration file. ```bash kubectl apply -f https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/deploy/operator.yaml ``` The StarRocks Operator is deployed to the namespace `starrocks` and manages all StarRocks clusters under all namespaces. 2. Deploy the StarRocks Operator by using a custom configuration file. * Download the configuration file **operator.yaml**, which is used to deploy the StarRocks Operator. ```bash curl -O https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/deploy/operator.yaml ``` * Modify the configuration file **operator.yaml** to suit your needs. * Deploy the StarRocks Operator. ```bash kubectl apply -f operator.yaml ``` 3. Check the running status of the StarRocks Operator. If the pod is in the `Running` state and all containers inside the pod are `READY`, the StarRocks Operator is running as expected. ```bash $ kubectl -n starrocks get pods NAME READY STATUS RESTARTS AGE starrocks-controller-65bb8679-jkbtg 1/1 Running 0 5m6s ``` > **NOTE** > > If you customize the namespace in which the StarRocks Operator is located, you need to replace`starrocks` with the name of your customized namespace. #### Deploy StarRocks Cluster[​](#deploy-starrocks-cluster "Direct link to Deploy StarRocks Cluster") You can directly use the [sample configuration files](https://github.com/StarRocks/starrocks-kubernetes-operator/tree/main/examples/starrocks) provided by StarRocks to deploy a StarRocks cluster (an object instantiated by using the custom resource StarRocks Cluster). For example, you can use **starrocks-fe-and-be.yaml** to deploy a StarRocks cluster that contains three FE nodes and three BE nodes. ```bash kubectl apply -f https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/examples/starrocks/starrocks-fe-and-be.yaml ``` The following table describes a few important fields in the **starrocks-fe-and-be.yaml** file. | **Field** | **Description** | | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Kind | The resource type of the object. The value must be `StarRocksCluster`. | | Metadata | Metadata, in which the following sub-fields are nested:- `name`: the name of the object. Each object name uniquely identifies an object of the same resource type.
- `namespace`: the namespace to which the object belongs. | | Spec | The expected status of the object. Valid values are `starRocksFeSpec`, `starRocksBeSpec`, and `starRocksCnSpec`. | You can also deploy the StarRocks cluster by using a modified configuration file. For supported fields and detailed descriptions, see [api.md](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/api.md). Deploying the StarRocks cluster takes a while. During this period, you can use the command `kubectl -n starrocks get pods` to check the starting status of the StarRocks cluster. If all the pods are in the `Running` state and all containers inside the pods are `READY`, the StarRocks cluster is running as expected. > **NOTE** > > If you customize the namespace in which the StarRocks cluster is located, you need to replace `starrocks` with the name of your customized namespace. ```bash $ kubectl -n starrocks get pods NAME READY STATUS RESTARTS AGE starrocks-controller-65bb8679-jkbtg 1/1 Running 0 22h starrockscluster-sample-be-0 1/1 Running 0 23h starrockscluster-sample-be-1 1/1 Running 0 23h starrockscluster-sample-be-2 1/1 Running 0 22h starrockscluster-sample-fe-0 1/1 Running 0 21h starrockscluster-sample-fe-1 1/1 Running 0 21h starrockscluster-sample-fe-2 1/1 Running 0 22h ``` tip If some pods cannot start after a long period of time, you can use `kubectl logs -n starrocks ` to view the log information or use `kubectl -n starrocks describe pod ` to view the event information to locate the problem. #### Manage StarRocks Cluster[​](#manage-starrocks-cluster "Direct link to Manage StarRocks Cluster") ##### Access StarRocks Cluster[​](#access-starrocks-cluster "Direct link to Access StarRocks Cluster") The components of the StarRocks cluster can be accessed through their associated Services, such as the FE Service. For detailed descriptions of Services and their access addresses, see [api.md](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/api.md) and [Services](https://kubernetes.io/docs/concepts/services-networking/service/). note * Only the FE Service is deployed by default. If you need to deploy the BE Service and CN Service, you need to configure `starRocksBeSpec` and `starRocksCnSpec` in the StarRocks cluster configuration file. * The name of a Service is `--service` by default, for example, `starrockscluster-sample-fe-service`. You can also specify the Service name in the spec of each component. ###### Access StarRocks Cluster from within Kubernetes Cluster[​](#access-starrocks-cluster-from-within-kubernetes-cluster "Direct link to Access StarRocks Cluster from within Kubernetes Cluster") From within the Kubernetes cluster, the StarRocks cluster can be accessed through the FE Service's ClusterIP. 1. Obtain the internal virtual IP address `CLUSTER-IP` and port `PORT(S)` of the FE Service. ```bash $ kubectl -n starrocks get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE be-domain-search ClusterIP None 9050/TCP 23m fe-domain-search ClusterIP None 9030/TCP 25m starrockscluster-sample-fe-service ClusterIP 10.100.162.xxx 8030/TCP,9020/TCP,9030/TCP,9010/TCP 25m ``` 2. Access the StarRocks cluster by using the MySQL client from within the Kubernetes cluster. ```bash mysql -h 10.100.162.xxx -P 9030 -uroot ``` ###### Access StarRocks Cluster from outside Kubernetes Cluster[​](#access-starrocks-cluster-from-outside-kubernetes-cluster "Direct link to Access StarRocks Cluster from outside Kubernetes Cluster") From outside the Kubernetes cluster, you can access the StarRocks cluster through the FE Service's LoadBalancer or NodePort. This topic uses LoadBalancer as an example: 1. Run the command `kubectl -n starrocks edit src starrockscluster-sample` to update the StarRocks cluster configuration file, and change the Service type of `starRocksFeSpec` to `LoadBalancer`. ```yaml starRocksFeSpec: image: starrocks/fe-ubuntu:3.0-latest replicas: 3 requests: cpu: 4 memory: 16Gi service: type: LoadBalancer # specified as LoadBalancer ``` 2. Obtain the IP address `EXTERNAL-IP` and port `PORT(S)` that the FE Service exposes to the outside. ```bash $ kubectl -n starrocks get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE be-domain-search ClusterIP None 9050/TCP 127m fe-domain-search ClusterIP None 9030/TCP 129m starrockscluster-sample-fe-service LoadBalancer 10.100.162.xxx a7509284bf3784983a596c6eec7fc212-618xxxxxx.us-west-2.elb.amazonaws.com 8030:30629/TCP,9020:32544/TCP,9030:32244/TCP,9010:32024/TCP 129m ClusterIP None 9030/TCP 23h ``` 3. Log in to your machine host and access the StarRocks cluster by using the MySQL client. ```bash mysql -h a7509284bf3784983a596c6eec7fc212-618xxxxxx.us-west-2.elb.amazonaws.com -P9030 -uroot ``` ##### Upgrade StarRocks Cluster[​](#upgrade-starrocks-cluster "Direct link to Upgrade StarRocks Cluster") ###### Upgrade BE nodes[​](#upgrade-be-nodes "Direct link to Upgrade BE nodes") Run the following command to specify a new BE image file, such as `starrocks/be-ubuntu:latest`: ```bash kubectl -n starrocks patch starrockscluster starrockscluster-sample --type='merge' -p '{"spec":{"starRocksBeSpec":{"image":"starrocks/be-ubuntu:latest"}}}' ``` ###### Upgrade FE nodes[​](#upgrade-fe-nodes "Direct link to Upgrade FE nodes") Run the following command to specify a new FE image file, such as `starrocks/fe-ubuntu:latest`: ```bash kubectl -n starrocks patch starrockscluster starrockscluster-sample --type='merge' -p '{"spec":{"starRocksFeSpec":{"image":"starrocks/fe-ubuntu:latest"}}}' ``` The upgrade process lasts for a while. You can run the command `kubectl -n starrocks get pods` to view the upgrade progress. ##### Scale StarRocks cluster[​](#scale-starrocks-cluster "Direct link to Scale StarRocks cluster") ###### Scale out BE cluster[​](#scale-out-be-cluster "Direct link to Scale out BE cluster") Run the following command to scale the BE cluster to 9 nodes: ```bash kubectl -n starrocks patch starrockscluster starrockscluster-sample --type='merge' -p '{"spec":{"starRocksBeSpec":{"replicas":9}}}' ``` ##### Scale in BE cluster[​](#scale-in-be-cluster "Direct link to Scale in BE cluster") When scaling in BE nodes, you need to scale them one at a time, and wait for the tablets on the BEs to be re-distributed before proceeding. If there are tables with single replicas, taking a BE node offline may cause data loss if the tablets fail to be redistributed. Execute the following command to scale in a cluster with 10 BE nodes to 9. ```bash kubectl -n starrocks patch starrockscluster starrockscluster-sample --type='merge' -p '{"spec":{"starRocksBeSpec":{"replicas":9}}}' ``` After scaling in, you must manually drop the nodes whose `alive` status is `false`. The redistribution of tablets will take some time. You can check the progress by executing `SHOW PROC '/statistic';`. ###### Scale out FE cluster[​](#scale-out-fe-cluster "Direct link to Scale out FE cluster") Run the following command to scale out the FE cluster to 4 nodes: ```bash kubectl -n starrocks patch starrockscluster starrockscluster-sample --type='merge' -p '{"spec":{"starRocksFeSpec":{"replicas":4}}}' ``` The scaling process lasts for a while. You can use the command `kubectl -n starrocks get pods` to view the scaling progress. ##### Automatic scaling for CN cluster[​](#automatic-scaling-for-cn-cluster "Direct link to Automatic scaling for CN cluster") Run the command `kubectl -n starrocks edit src starrockscluster-sample` to configure the automatic scaling policy for the CN cluster. You can specify the resource metrics for CNs as the average CPU utilization, average memory usage, elastic scaling threshold, upper elastic scaling limit, and lower elastic scaling limit. The upper elastic scaling limit and lower elastic scaling limit specify the maximum number and minimum number of CNs allowed for elastic scaling. note If the automatic scaling policy for the CN cluster is configured, delete the `replicas` field from the `starRocksCnSpec` in the StarRocks cluster configuration file. Kubernetes also supports using `behavior` to customize scaling behaviors according to business scenarios, helping you achieve rapid or slow scaling or disable scaling. For more information about automatic scaling policies, see [Horizontal Pod Scaling](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/). The following is a [template](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/examples/starrocks/deploy_a_starrocks_cluster_with_cn.yaml) provided by StarRocks to help you configure automatic scaling policies: ```yaml starRocksCnSpec: image: starrocks/cn-ubuntu:latest limits: cpu: 16 memory: 64Gi requests: cpu: 16 memory: 64Gi autoScalingPolicy: # Automatic scaling policy of the CN cluster. maxReplicas: 10 # The maximum number of CNs is set to 10. minReplicas: 1 # The minimum number of CNs is set to 1. # operator creates an HPA resource based on the following field. # see https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/ for more information. hpaPolicy: metrics: # Resource metrics - type: Resource resource: name: memory # The average memory usage of CNs is specified as a resource metric. target: # The elastic scaling threshold is 60%. # When the average memory utilization of CNs exceeds 60%, the number of CNs increases for scale-out. # When the average memory utilization of CNs is below 60%, the number of CNs decreases for scale-in. averageUtilization: 60 type: Utilization - type: Resource resource: name: cpu # The average CPU utilization of CNs is specified as a resource metric. target: # The elastic scaling threshold is 60%. # When the average CPU utilization of CNs exceeds 60%, the number of CNs increases for scale-out. # When the average CPU utilization of CNs is below 60%, the number of CNs decreases for scale-in. averageUtilization: 60 type: Utilization behavior: # The scaling behavior is customized according to business scenarios, helping you achieve rapid or slow scaling or disable scaling. scaleUp: policies: - type: Pods value: 1 periodSeconds: 10 scaleDown: selectPolicy: Disabled ``` The following table describes a few important fields: * The upper and lower elastic scaling limit. ```yaml maxReplicas: 10 # The maximum number of CNs is set to 10. minReplicas: 1 # The minimum number of CNs is set to 1. ``` * The elastic scaling threshold. ```yaml # For example, the average CPU utilization of CNs is specified as a resource metric. # The elastic scaling threshold is 60%. # When the average CPU utilization of CNs exceeds 60%, the number of CNs increases for scale-out. # When the average CPU utilization of CNs is below 60%, the number of CNs decreases for scale-in. - type: Resource resource: name: cpu target: averageUtilization: 60 ``` #### FAQ[​](#faq "Direct link to FAQ") **Issue description:** When a custom resource StarRocksCluster is installed using `kubectl apply -f xxx`, an error is returned `The CustomResourceDefinition 'starrocksclusters.starrocks.com' is invalid: metadata.annotations: Too long: must have at most 262144 bytes`. **Cause analysis:** Whenever `kubectl apply -f xxx` is used to create or update resources, a metadata annotation `kubectl.kubernetes.io/last-applied-configuration` is added. This metadata annotation is in JSON format and records the *last-applied-configuration*. `kubectl apply -f xxx`" is suitable for most cases, but in rare situations , such as when the configuration file for the custom resource is too large, it may cause the size of the metadata annotation to exceed the limit. **Solution:** If you install the custom resource StarRocksCluster for the first time, it is recommended to use `kubectl create -f xxx`. If the custom resource StarRocksCluster is already installed in the environment, and you need to update its configuration, it is recommended to use `kubectl replace -f xxx`. --- ## Developers ### Compile StarRocks with Docker This topic describes how to compile StarRocks using Docker. #### Overview[​](#overview "Direct link to Overview") StarRocks provides development environment images for Ubuntu (22.04 and 24.04), CentOS 7.9, and Rocky Linux 9. With the image, you can launch a Docker container and compile StarRocks in the container. note Starting from v4.2, the development environment changes as follows: * CentOS 7 reached its End-of-Life on June 30, 2024, so its build distribution is discontinued and replaced by Rocky Linux 9. CentOS 7 images remain available only for v4.1 and earlier. * The Ubuntu development environment is upgraded from Ubuntu 22.04 (v4.1 and earlier) to Ubuntu 24.04 (v4.2 and later). The image name `starrocks/dev-env-ubuntu` is unchanged. ##### StarRocks version and DEV ENV image[​](#starrocks-version-and-dev-env-image "Direct link to StarRocks version and DEV ENV image") Different branches of StarRocks correspond to different development environment images provided on [StarRocks Docker Hub](https://hub.docker.com/u/starrocks). * For Ubuntu (22.04 for v4.1 and earlier, 24.04 for v4.2 and later): | **Branch name** | **Image name** | | --------------- | --------------------------------------- | | main | starrocks/dev-env-ubuntu:latest | | branch-4.1 | starrocks/dev-env-ubuntu:4.1-latest | | branch-4.0 | starrocks/dev-env-ubuntu:4.0-latest | | branch-3.5 | starrocks/dev-env-ubuntu:3.5-latest | * For CentOS 7.9 (v4.1 and earlier; discontinued from v4.2): | **Branch name** | **Image name** | | --------------- | ------------------------------------ | | branch-4.1 | starrocks/dev-env-centos7:4.1-latest | | branch-4.0 | starrocks/dev-env-centos7:4.0-latest | | branch-3.5 | starrocks/dev-env-centos7:3.5-latest | * For Rocky Linux 9 (v4.2 and later): | **Branch name** | **Image name** | | --------------- | --------------------------------------- | | main | starrocks/dev-env-rocky9:latest | #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before compiling StarRocks, make sure the following requirements are satisfied: * **Hardware** Your machine must have at least 8 GB RAM. * **Software** * Your machine must be running on Ubuntu 22.04 or 24.04, CentOS 7.9, or Rocky Linux 9. * You must have Docker installed on your machine and version v20.10.10 at least. #### Step 1: Download the image[​](#step-1-download-the-image "Direct link to Step 1: Download the image") Download the development environment image by running the following command: ```bash # Replace with the name of the image that you want to download, # for example, `starrocks/dev-env-ubuntu:latest`. # Make sure you have choose the correct image for your OS. docker pull ``` Docker automatically identifies the CPU architecture of your machine and pulls the corresponding image that suits your machine. The `linux/amd64` images are for the x86-based CPUs, and `linux/arm64` images are for the ARM-based CPUs. #### Step 2: Compile StarRocks in a Docker container[​](#step-2-compile-starrocks-in-a-docker-container "Direct link to Step 2: Compile StarRocks in a Docker container") You can launch the development environment Docker container with or without the local host path mounted. We recommend you launch the container with the local host path mounted, so that you can avoid re-downloading the Java dependency during the next compilation, and you do not need to manually copy the binary files from the container to your local host. * **Launch the container with the local host path mounted**: 1. Clone the StarRocks source code to your local host. ```bash git clone https://github.com/StarRocks/starrocks.git ``` 2. Launch the container. ```bash # Replace with the parent directory of the StarRocks source code directory. # Replace with the name of the branch that corresponds to the image name. # Replace with the name of the image that you downloaded. docker run -it -v /.m2:/root/.m2 \ -v /starrocks:/root/starrocks \ --name -d ``` 3. Launch a bash shell inside the container you have launched. ```bash # Replace with the name of the branch that corresponds to the image name. docker exec -it /bin/bash ``` 4. Compile StarRocks in the container. ```bash cd /root/starrocks && ./build.sh ``` * **Launch the container without the local host path mounted**: 1. Launch the container. ```bash # Replace with the name of the branch that corresponds to the image name. # Replace with the name of the image that you downloaded. docker run -it --name -d ``` 2. Launch a bash shell inside the container. ```bash # Replace with the name of the branch that corresponds to the image name. docker exec -it /bin/bash ``` 3. Clone the StarRocks source code to the container. ```bash git clone https://github.com/StarRocks/starrocks.git ``` 4. Compile StarRocks in the container. ```bash cd starrocks && ./build.sh ``` #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") Q: The StarRocks BE building fails, and the following error message has been returned: ```bash g++: fatal error: Killed signal terminated program cc1plus compilation terminated. ``` What should I do? A: This error message indicates a lack of memory in the Docker container. You need to allocate at least 8 GB of memory resources to the container. --- ### Compile StarRocks on Ubuntu This topic describes how to compile StarRocks on the Ubuntu operating system. StarRocks supports compilation on both x86\_64 and AArch64 architectures. note Build StarRocks v4.1 and earlier on Ubuntu 22.04, and StarRocks v4.2 and later on Ubuntu 24.04. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") ##### Install Dependencies[​](#install-dependencies "Direct link to Install Dependencies") Run the following commands to install necessary dependencies: ```bash sudo apt update ``` ```bash sudo apt install build-essential automake bison byacc ccache flex libiberty-dev libtool maven zip python3 python-is-python3 bzip2 -y ``` ##### Install Compiler[​](#install-compiler "Direct link to Install Compiler") If you are using Ubuntu 22.04 or later, run the following command to install the tools and compilers: ```bash sudo apt install cmake gcc g++ openjdk-17-jdk -y ``` If you are using an Ubuntu version earlier than 22.04, run the following commands to check the versions of tools and compilers: 1. Check GCC/G++ versions: ```bash gcc --version g++ --version ``` GCC/G++ versions must be 10.3 or later. If you are using earlier versions, [click here to install GCC/G++](https://gcc.gnu.org/releases.html). 2. Check JDK version: ```bash java --version ``` OpenJDK version must be 17 or later. If you are using an earlier version, [click here to install OpenJDK](https://openjdk.org/install). 3. Check CMake version: ```bash cmake --version ``` CMake version must be 3.20.1 or later. If you are using an earlier version, [click here to install CMake](https://cmake.org/download). #### Compile StarRocks[​](#compile-starrocks "Direct link to Compile StarRocks") ##### Download Source Code[​](#download-source-code "Direct link to Download Source Code") Run the following command to clone the StarRocks repository and navigate into the directory: ```bash git clone https://github.com/StarRocks/starrocks.git cd starrocks ``` ##### Build StarRocks[​](#build-starrocks "Direct link to Build StarRocks") Run the following command to start the compilation: ```bash ./build.sh ``` The default compilation parallelism is equal to **CPU core count/4**. Assuming you have 32 CPU cores, the default parallelism is 8. If you want to adjust the parallelism, you can specify the number of CPU cores to be used for compilation via `-j` in the command line. The following example uses 24 CPU cores for compilation: ```bash ./build.sh -j 24 ``` #### FAQ[​](#faq "Direct link to FAQ") Q-1: Building `aws_cpp_sdk` fails on Ubuntu 20.04 with the error "Error: undefined reference to pthread\_create". How can I resolve this? A: This error occurs due to a lower version of CMake. Please upgrade CMake to version 3.20.1 or above. Q-2: Building StarRocks fails on Ubuntu 24.04 or GCC 12+ with strict warning errors. How can I resolve this? A: To prevent build failures in third-party libraries, export the following flag before building: ```bash export DISABLE_WARNING_AS_ERROR=1 ./build.sh ``` --- ### How to build StarRocks In general, you can build StarRocks by just executing ```text ./build.sh ``` This command will check if all the thirdpary dependencies are ready at first. If all dependencies are ready, it will build StarRocks `Backend` and `Frontend`. After this command executes succefully, the generated binary will be in `output` directory. #### build FE/BE separately[​](#build-febe-separately "Direct link to build FE/BE separately") You don't need to build both FE and BE each time, you can build them separately. For example, you can only build BE by ```text ./build.sh --be ``` and, only build FE by ```text ./build.sh --fe ``` ### How to run unit test Unit tests of BE and FE are separted. In general, you can run BE test by ```text ./run-be-ut.sh ``` run FE test by ```text ./run-fe-ut.sh ``` #### How to run BE UT in command line[​](#how-to-run-be-ut-in-command-line "Direct link to How to run BE UT in command line") Now, BE UT needs some dependency to run, and `./run-be-ut.sh` helps it. But it is not flexible enough. When you want to run UT in the command-line, you can execute ```text UDF_RUNTIME_DIR=./ STARROCKS_HOME=./ LD_LIBRARY_PATH=/usr/lib/jvm/java-18-openjdk-amd64/lib/server ./be/ut_build_ASAN/test/starrocks_test ``` StarRocks Backend UT is built on top of google-test, so you can pass filter to run some of the UT, For example, you want to test only MapColumn related tests, you can execute ```text UDF_RUNTIME_DIR=./ STARROCKS_HOME=./ LD_LIBRARY_PATH=/usr/lib/jvm/java-18-openjdk-amd64/lib/server ./be/ut_build_ASAN/test/starrocks_test --gtest_filter="*MapColumn*" ``` ### Build options #### build with clang[​](#build-with-clang "Direct link to build with clang") You can build StarRocks by `clang` too ```text CC=clang CXX=clang++ ./build.sh --be ``` Then you can see the following similar message in the build message ```text -- compiler Clang version 14.0.0 ``` #### build with different linker[​](#build-with-different-linker "Direct link to build with different linker") The default linker is slow, developer can specify different linker to speed up linking. For example, you can use `lld`, the LLVM-based linker. You need to install `lld` firstly. ```text sudo apt install lld ``` Then you set the environment variable STARROCKS\_LINKER with the linker you want to use. For example: ```text STARROCKS_LINKER=lld ./build.sh --be ``` #### build different type[​](#build-different-type "Direct link to build different type") You can build StarRocks with different types with different BUILD\_TYPE variable, the default BUILD\_TYPE is `RELEASE`. For example, you can build StarRocks with `ASAN` type by ```text BUILD_TYPE=ASAN ./build.sh --be ``` --- ### Protobuf Guides #### Never use required[​](#never-use-required "Direct link to Never use required") As the project involving, any fields may become optional. But if it is defined as required, it can not be removed. So `required` should not be used. #### Never change the ordinal[​](#never-change-the-ordinal "Direct link to Never change the ordinal") To be back compatible, the ordinal of the field SHOULD NOT be changed. ### Naming #### file name[​](#file-name "Direct link to file name") The names of messages are all lowercase, with underscores between words. Files should end in `.proto`. ```text my_message.proto // Good mymessage.proto // Bad my_message.pb // Bad ``` #### Message Name[​](#message-name "Direct link to Message Name") Message names start with a capital letter and have a capital letter for each new word, with no underscores, and with `PB` as postfix: MyMessagePB ```protobuf message MyMessagePB // Good message MyMessage // Bad message My_Message_PB // Bad message myMessagePB // Bad ``` #### field name[​](#field-name "Direct link to field name") The names of messages are all lowercase, with underscores between words. ```text optional int64 my_field = 3; // Good optional int64 myField = 3; // Bad ``` --- ### StarRocks Restful API Standard #### API Format[​](#api-format "Direct link to API Format") 1. The API format follows the pattern: `/api/{version}/{target-object-access-path}/{action}`. 2. `{version}` is denoted as `v{number}`, such as v1, v2, v3, v4, etc. 3. `{target-object-access-path}` is organized in a hierarchical manner, which will be explained in detail later. 4. `{action}` is optional, API implementors should utilize the HTTP METHOD to convey the operation's meaning as much as possible. Only when the HTTP methods' semantics cannot be fulfilled should the action be used. For example, if there is no HTTP method available to rename an object. #### Definition of Target Object Access Path[​](#definition-of-target-object-access-path "Direct link to Definition of Target Object Access Path") 1. Target objects accessed by the REST API need to be categorized and organized into a hierarchical access path. The access path format is as follows: ```text /primary_categories/primary_object/secondary_categories/secondary_object/.../categories/object ``` Taking catalog, database, table, column as examples: ```text /catalogs: Represents all catalogs. /catalogs/hive: Represents the specific catalog object named "hive" under the catalog category. /catalogs/hive/databases: Represents all databases in the "hive" catalog. /catalogs/hive/databases/tpch_100g: Represents the database named "tpch_100g" in the "hive" catalog. /catalogs/hive/databases/tpch_100g/tables: Represents all tables in the "tpch_100g" database. /catalogs/hive/databases/tpch_100g/tables/lineitem: Represents the tpch_100g.lineitem table. /catalogs/hive/databases/tpch_100g/tables/lineitem/columns: Represents all columns in the tpch_100g.lineitem table. /catalogs/hive/databases/tpch_100g/tables/lineitem/columns/l_orderkey: Represents the specific column l_orderkey in the tpch_100g.lineitem table. ``` 2. Categories are named using snake-case, and the last word is in plural form.all the words are in lowercase, and multiple words are connected by underscores (\_). Specific objects are named using their actual names. The hierarchical relationship of the target objects needs to be clearly defined. #### Selection of HTTP Method[​](#selection-of-http-method "Direct link to Selection of HTTP Method") 1. GET: Use the GET method to show a single object and list all objects of a certain category. The GET method's access to objects is read-only and does not provide a request body. ```text # list all of the tables in database ssb_100g GET /api/v2/catalogs/default/databases/ssb_100g/tables # show the table ssb_100g.lineorder GET /api/v2/catalogs/default/databases/ssb_100g/tables/lineorder ``` 2. POST: Used to create objects. Parameters are passed through the request body. It is not idempotent. If the object already exists, the repeated creation will fail and return an error message. ```text POST /api/v2/catalogs/default/databases/ssb_100g/tables/create -d@create_customer.sql ``` 3. PUT: Used to create objects. Parameters are passed through the request body. It is idempotent. If the object already exists, it will return success. PUT method is the CREATE IF NOT EXISTS version of the POST method. ```text PUT /api/v2/databases/ssb_100g/tables/create -d@create_customer.sql ``` 4. DELETE: Used to delete objects. It does not provide a request body. If the object to be deleted does not exist, it will return success. DELETE method has the DROP IF EXISTS semantics. ```text DELETE /api/v2/catalogs/default/databases/ssb_100g/tables/customer ``` 5. PATCH: Used to update objects. It provides a request body, which only contains the partial information that needs to be modified. ```text PATCH /api/v2/databases/ssb_100g/tables/customer -d '{"unique_key_constraints": ["c_custkey"]}' ``` #### Authentication and Authorization[​](#authentication-and-authorization "Direct link to Authentication and Authorization") 1. Authentication and authorization information is passed in the HTTP Request Header. #### HTTP Status Codes[​](#http-status-codes "Direct link to HTTP Status Codes") 1. HTTP status codes are returned by the REST API to indicate the success or failure of an operation. 2. The status codes (2xx) for success operation as follows: * 200 OK: Indicates that the request has been successfully completed. It is used for viewing/listing/deleting/updating objects and querying the status of pending tasks. * 201 Created: Indicates that the object has been successfully created. It is used for PUT/POST methods. The response body must include the object URI for subsequent viewing/listing/deleting/updating. * 202 Accepted: Indicates that the task submission is successful and the task is in a pending state. The response body must include the task URI for subsequent cancellation, deletion, and polling of task status. 3. The error codes (4xx) indicate client errors. Users need to adjust and modify the HTTP request and retry. * 400 Bad Request: Invalid request parameters. * 401 Unauthorized: Missing authentication information, illegal authentication information, authentication failure. * 403 Forbidden: Authentication succeeded, but the user's operation failed the authorization check. No access permission. * 404 Not Found: API URI encoding error. It does not belong to the registered REST API. * 405 Method Not Allowed: Incorrect HTTP Method used. * 406 Not Acceptable: The response format does not match the media type specified in the Accept header. * 415 Not Acceptable: The media type of the request content does not match the media type specified in the Content-Type header. 4. The error codes (5xx) indicate server errors. Users do not need to modify the request and can retry later. * 500 Internal Server Error: Internal server error, similar to Unknown error. * 503 Service Unavailable: The service is temporarily unavailable. For example, the user's access frequency is too high and has reached the rate limit; or the service is currently unable to provide service due to internal status, such as when creating a table with 3 replicas, but only 2 BEs are available; all Tablet replicas involved in a user's query are unavailable. #### HTTP Response Format[​](#http-response-format "Direct link to HTTP Response Format") 1. When the API returns an HTTP code code of 200/201/202, the HTTP response is not empty. The API returns results in JSON format, including top-level fields "code", "message", and "result". All JSON fields are named using camel-case. 2. In a successful API response, the "code" is "0", the "message" is "OK", and the "result" contains the actual results. ```json { "code":"0", "message": "OK", "result": {....} } ``` 3. In a failed API response, the "code" is not "0", the "message" is a simple error message, and the "result" can contain detailed error information, such as error stack traces. ```json { "code":"1", "message": "Analyze error", "result": {....} } ``` #### Parameter Passing[​](#parameter-passing "Direct link to Parameter Passing") 1. API parameters are passed in the precedence order of path, request body, query parameters, and header. Choose the appropriate method for parameter passing. 2. Path parameters: Required parameters that represent the object's hierarchical relationship are placed in the path parameters. ```text /api/v2/warehouses/{warehouseName}/backends/{backendId} /api/v2/warehouses/ware0/backends/10027 ``` 3. Request body: Parameters are passed using application/json. Parameters can be of required or optional types. 4. Query parameters: Using query parameters and request body parameters at the same time is not allowed. For the same API, choose either one. If the number of parameters excluding header parameters and path parameters is not more than 2, query parameters can be used; otherwise, use the request body to pass parameters. 5. HEADER parameters: Headers should be used to pass HTTP standard parameters such as Content-type and Accept are placed in the header, implementors should not abuse http headers to pass customized parameters. When using headers to pass parameters for user extensions, the header name should be in the format `x-starrocks-{name}`, where the name can contain multiple English words, and each word is in lowercase and concatenated by hyphens (-). --- ### Thrift Guides #### Never use required[​](#never-use-required "Direct link to Never use required") As the project involving, any fields may become optional. But if it is defined as required, it can not be removed. So `required` should not be used. #### Never change the ordinal[​](#never-change-the-ordinal "Direct link to Never change the ordinal") To be back compatible, the ordinal of the field SHOULD NOT be changed. ### Naming #### file name[​](#file-name "Direct link to file name") The names of messages are all lowercase, with underscores between words. Files should end in `.thrift`. ```text my_struct.thrift // Good MyStruct.thrift // Bad my_struct.proto // Bad ``` #### struct name[​](#struct-name "Direct link to struct name") Struct names start with a capital letter `T` and have a capital letter for each new word, with no underscores: TMyStruct ```text struct TMyStruct; // Good struct MyStruct; // Bad struct TMy_Struct; // Bad struct TmyStruct; // Bad ``` #### field name[​](#field-name "Direct link to field name") The names of struct members are all lowercase, with underscores between words. ```text 1: optional i64 my_field; // Good 1: optional i64 myField; // Bad ``` --- ### Use the debuginfo file for debugging #### Change description[​](#change-description "Direct link to Change description") From v2.5 onwards, the debuginfo file of BE is stripped from the StarRocks installation package to reduce the size and space usage of the installation package. You can see two packages at [StarRocks website](https://www.starrocks.io/download/community). ![debuginfo](/assets/images/debug_info-5791bd74c1be19fa504931d9c8491749.png) In this figure, you can click `Get Debug Symbol files` to download the debuginfo package. `StarRocks-2.5.10.tar.gz` is the installation package and you can click **Download** to download this package. This change does not affect your download behavior or use of StarRocks. You can download only the installation package for cluster deployment and upgrade. The debuginfo package is only for developers to debug programs using GDB. #### Precautions[​](#precautions "Direct link to Precautions") GDB 12.1 or later is recommended for debugging. #### How to use the debuginfo file[​](#how-to-use-the-debuginfo-file "Direct link to How to use the debuginfo file") 1. Download and decompress the debuginfo package. ```sql wget https://releases.starrocks.io/starrocks/StarRocks-.debuginfo.tar.gz tar -xzvf StarRocks-.debuginfo.tar.gz ``` > **NOTE** > > Replace `` with the version number of the StarRocks installation package you want to download. 2. Load the debuginfo file when you perform GDB debugging. * **Method 1** ```shell objcopy --add-gnu-debuglink=starrocks_be.debug starrocks_be ``` This operation associates the debug info file with your executable file. * **Method 2** ```shell gdb -s starrocks_be.debug -e starrocks_be -c `core_file` ``` The debuginfo file works well with perf and pstack. You can directly use perf and pstack without additional operations. --- ### Setup development environment with Docker This directory provides Docker-based build tools for StarRocks that use the official `starrocks/dev-env-ubuntu:latest` development environment image. This ensures consistent builds across different host systems by using a standardized Ubuntu environment with all required toolchains and dependencies pre-installed. #### 🚀 Quick Start[​](#-quick-start "Direct link to 🚀 Quick Start") ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Docker installed and running * At least 8GB RAM available for Docker * At least 20GB free disk space ##### Simple Commands[​](#simple-commands "Direct link to Simple Commands") ```bash # Open development shell ./docker-dev.sh shell # Build Frontend only ./docker-dev.sh build-fe # Build Backend only ./docker-dev.sh build-be # Build everything ./docker-dev.sh build-all # Clean build everything ./docker-dev.sh clean-build # Run Frontend tests ./docker-dev.sh test-fe ``` #### 📋 Available Tools[​](#-available-tools "Direct link to 📋 Available Tools") ##### 1. `build-in-docker.sh` - Full-Featured Build Script[​](#1-build-in-dockersh---full-featured-build-script "Direct link to 1-build-in-dockersh---full-featured-build-script") The main build script that automatically passes through all `build.sh` options: ```bash # Basic usage ./build-in-docker.sh # Build all (FE + BE) ./build-in-docker.sh --fe # Build Frontend only ./build-in-docker.sh --be # Build Backend only ./build-in-docker.sh --fe --be --clean # Clean and build both # Advanced options ./build-in-docker.sh --be --with-gcov # Build BE with code coverage ./build-in-docker.sh --fe --disable-java-check-style # Skip checkstyle ./build-in-docker.sh --be -j 8 # Build with 8 parallel jobs # Development ./build-in-docker.sh --shell # Interactive shell ./build-in-docker.sh --test # Build and run tests # Custom image ./build-in-docker.sh --image starrocks/dev-env-ubuntu:latest --fe # Future build.sh options work automatically ./build-in-docker.sh --be --new-future-option ``` ##### 2. `docker-dev.sh` - Simple Wrapper[​](#2-docker-devsh---simple-wrapper "Direct link to 2-docker-devsh---simple-wrapper") Quick commands for common tasks: ```bash ./docker-dev.sh shell # Development shell ./docker-dev.sh build-fe # Build Frontend ./docker-dev.sh build-be # Build Backend ./docker-dev.sh build-all # Build everything ./docker-dev.sh clean-build # Clean and build all ./docker-dev.sh test-fe # Run FE tests ./docker-dev.sh test-be # Run BE tests ./docker-dev.sh test-all # Run all tests # Pass through any build.sh options ./docker-dev.sh build --be --with-gcov ./docker-dev.sh build --fe --new-option ``` ##### 3. `docker-compose.dev.yml` - Docker Compose[​](#3-docker-composedevyml---docker-compose "Direct link to 3-docker-composedevyml---docker-compose") For persistent development environments: ```bash # Start development shell docker-compose -f docker-compose.dev.yml run --rm starrocks-dev # Build Frontend docker-compose -f docker-compose.dev.yml run --rm build-fe # Build Backend docker-compose -f docker-compose.dev.yml run --rm build-be # Run tests docker-compose -f docker-compose.dev.yml run --rm test-fe docker-compose -f docker-compose.dev.yml run --rm test-be # Clean up docker-compose -f docker-compose.dev.yml down -v ``` #### 🔧 Configuration[​](#-configuration "Direct link to 🔧 Configuration") ##### Volume Mounts[​](#volume-mounts "Direct link to Volume Mounts") The Docker scripts automatically mount: * **Source code**: `$(pwd):/workspace` - Your local repository * **Maven cache**: `~/.m2:/tmp/.m2` - Maven dependencies cache for faster builds **Note**: The Maven cache is shared between your host system and the Docker container, so dependencies downloaded during builds are persisted and reused across build sessions. ##### Environment Variables[​](#environment-variables "Direct link to Environment Variables") ```bash # Use different Docker image export STARROCKS_DEV_ENV_IMAGE=starrocks/dev-env-ubuntu:latest # Additional Docker options export DOCKER_BUILD_OPTS="--memory=16g --cpus=8" # Set user ID for file permissions export UID=$(id -u) export GID=$(id -g) ``` ##### Custom Build Options[​](#custom-build-options "Direct link to Custom Build Options") All original `build.sh` options are supported: ```bash # Backend build types BUILD_TYPE=Debug ./build-in-docker.sh --be # Debug build BUILD_TYPE=Release ./build-in-docker.sh --be # Release build (default) BUILD_TYPE=Asan ./build-in-docker.sh --be # AddressSanitizer build # Feature flags ./build-in-docker.sh --be --enable-shared-data # Enable shared data ./build-in-docker.sh --be --with-gcov # Code coverage ./build-in-docker.sh --be --with-bench # Benchmarks ./build-in-docker.sh --be --without-avx2 # Disable AVX2 ``` #### 📁 Output and Artifacts[​](#-output-and-artifacts "Direct link to 📁 Output and Artifacts") Build artifacts are created in the `output/` directory: ```text output/ ├── fe/ # Frontend artifacts ├── be/ # Backend artifacts └── java-extensions/ # Java extensions ``` The Docker container mounts your local repository and Maven cache, so all build outputs are available on your host system and dependencies are cached for faster subsequent builds. **Multi-User Support**: Container names include username and user ID to prevent conflicts on shared development machines (e.g., `starrocks-build-username-1001-1234567890`). **Automatic Extensibility**: All unrecognized options are automatically passed through to `build.sh`, so new build options work without updating the Docker scripts. #### 🐛 Troubleshooting[​](#-troubleshooting "Direct link to 🐛 Troubleshooting") ##### Common Issues[​](#common-issues "Direct link to Common Issues") 1. **Permission Issues** ```bash # Fix file permissions sudo chown -R $(id -u):$(id -g) output/ # Or run with correct user export UID=$(id -u) GID=$(id -g) ./build-in-docker.sh --fe ``` 2. **Out of Memory** ```bash # Increase Docker memory limit or reduce parallel jobs ./build-in-docker.sh --be -j 2 ``` 3. **Docker Image Not Found** ```bash # Pull the image manually docker pull starrocks/dev-env-ubuntu:latest ``` 4. **Build Failures** ```bash # Clean build ./build-in-docker.sh --clean --fe --be # Check logs in interactive shell ./build-in-docker.sh --shell ``` ##### Debug Mode[​](#debug-mode "Direct link to Debug Mode") For debugging build issues: ```bash # Open shell and run commands manually ./build-in-docker.sh --shell # Inside container: ./build.sh --fe --clean ./run-fe-ut.sh ``` #### 🔍 Verification[​](#-verification "Direct link to 🔍 Verification") Test that everything works: ```bash # 1. Test Docker setup docker --version docker info # 2. Test image availability docker pull starrocks/dev-env-ubuntu:latest # 3. Test build scripts ./docker-dev.sh shell # Inside container: exit # 4. Test simple build ./docker-dev.sh build-fe ``` #### 📊 Performance Tips[​](#-performance-tips "Direct link to 📊 Performance Tips") 1. **Use parallel builds**: `./build-in-docker.sh --be -j $(nproc)` 2. **Maven cache**: Automatically mounted from `~/.m2` for faster dependency resolution 3. **Memory allocation**: Increase Docker memory limit for faster builds 4. **SSD storage**: Use SSD for Docker storage driver 5. **Persistent volumes**: Use Docker Compose for additional cache persistence #### 🤝 Integration with IDEs[​](#-integration-with-ides "Direct link to 🤝 Integration with IDEs") ##### VS Code with Dev Containers[​](#vs-code-with-dev-containers "Direct link to VS Code with Dev Containers") Create `.devcontainer/devcontainer.json`: ```json { "name": "StarRocks Dev Environment", "image": "starrocks/dev-env-ubuntu:latest", "workspaceMount": "source=${localWorkspaceFolder},target=/workspace,type=bind", "workspaceFolder": "/workspace", "customizations": { "vscode": { "extensions": [ "ms-vscode.cpptools", "redhat.java" ] } } } ``` ##### IntelliJ IDEA[​](#intellij-idea "Direct link to IntelliJ IDEA") Use the Docker integration to run builds and tests within the container environment. #### 📚 Additional Resources[​](#-additional-resources "Direct link to 📚 Additional Resources") * [Original build.sh](https://github.com/StarRocks/starrocks/blob/main/build.sh) for reference #### 🆘 Support[​](#-support "Direct link to 🆘 Support") If you encounter issues: 1. Check the troubleshooting section above 2. Verify Docker setup and image availability 3. Try clean builds with `--clean` flag 4. Open an issue with build logs and system information --- ### Setup IDE for developing StarRocks ### Setup IDE for developing StarRocks Some people want to become StarRocks contributor, but are troubled by the development environment, so here I write a tutorial about it. What is a perfect development toolchain? * Support one click to compile FE and BE. * Support code jump in Clion and IDEA. * All variables in the IDE can be analyzed normally without red lines. * Clion can enable its analysis function normally. * Support FE and BE debug. #### Prepare[​](#prepare "Direct link to Prepare") I use a MacBook(M1) for local coding and a remote server for compiling & testing StarRocks. (Remote server uses Ubuntu 22, **at least need 16GB RAM**). The overall idea is to write code on the MacBook, then automatically synchronize the code to the server through the IDE, and use the server to compile and develop StarRocks. ##### MacBook Setup[​](#macbook-setup "Direct link to MacBook Setup") ###### Thrift 0.20[​](#thrift-020 "Direct link to Thrift 0.20") ```bash brew install cartman-kai/thrift/thrift@0.20 ``` You can check whether Thrift is installed successfully with the following command: ```bash $ thrift -version Thrift version 0.20.0 ``` ###### Protobuf[​](#protobuf "Direct link to Protobuf") Just use the latest version v3 directly, because the latest version of Protobuf is compatible with the v2 version of the Protobuf protocol in StarRocks. ```bash brew install protobuf ``` ###### Maven[​](#maven "Direct link to Maven") ```bash brew install maven ``` ###### OpenJDK 17[​](#openjdk-17 "Direct link to OpenJDK 17") ```bash brew install openjdk@17 ``` ###### Python3[​](#python3 "Direct link to Python3") MacOS comes with it, no installation is needed. ###### Setup system env[​](#setup-system-env "Direct link to Setup system env") ```bash export JAVA_HOME=xxxxx export PYTHON=/usr/bin/python3 ``` ##### Ubuntu server setup[​](#ubuntu-server-setup "Direct link to Ubuntu server setup") ###### Clone StarRocks code[​](#clone-starrocks-code "Direct link to Clone StarRocks code") `git clone https://github.com/StarRocks/starrocks.git` ###### Install required tools for compilation[​](#install-required-tools-for-compilation "Direct link to Install required tools for compilation") ```bash sudo apt update ``` ```bash sudo apt install gcc g++ maven openjdk-17-jdk python3 python-is-python3 unzip cmake bzip2 ccache byacc ccache flex automake libtool bison libiberty-dev build-essential ninja-build curl ``` Setup `JAVA_HOME` env ```bash export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 ``` ###### Do a compilation of StarRocks[​](#do-a-compilation-of-starrocks "Direct link to Do a compilation of StarRocks") ```bash cd starrocks/ ./build.sh ``` The first time compile needs to compile thirdparty, it will require some time. **You have to use gcc for the first compilation, currently, thirdparty can't compile success in clang.** #### IDE Setup[​](#ide-setup "Direct link to IDE Setup") ##### FE[​](#fe "Direct link to FE") FE development is simple because you can compile it in MacOS directly. Just enter `fe` folder and run the command `mvn install -DskipTests`. Then you can use IDEA to open `fe` folder directly, everything is ok. ###### Local debug[​](#local-debug "Direct link to Local debug") The same as other Java applications. You might face OOM error while compiling StarRocks on Intellij IDEA, you can go to Build, Execution, Deployment -> Compiler -> Increase Heap size to a larger value. ###### Remote debug[​](#remote-debug "Direct link to Remote debug") In Ubuntu server, run with `./start_fe.sh --debug`, then use IDEA remote debug to connect it. The default port is 5005, you can change it in `start_fe.sh` scripts. Debug java parameter: `-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005` is just copied from IDEA. ![IDE](/assets/images/ide-1-98b198df0ef5da5f566e18eba4c5cf5f.png) ##### BE[​](#be "Direct link to BE") It is recommended to run `mvn install -DskipTests` first in `fe` to generate the FE-side Java thrift/proto sources. For BE, run the CMake configure/build flow once. The BE thrift/protobuf headers are materialized by CMake during configure/build into the active build directory (for example `be/build_Release/gensrc/gen_cpp`), so you no longer need `cd gensrc && make` for those files. If you need the shared script outputs first, run `make -C gensrc script`. Use Clion to open `be` folder. Enter `Settings`, add `Toolchains`. Add a remote server first, then setup Build Tool, C and C++ Compiler separately. ![IDE](/assets/images/ide-2-d70ce01d1c2227a2f753da205414c5e4.png) In `Settings` / `Deployment`. Change folder `mappings`. ![IDE](/assets/images/ide-3-7f7db478d4179890a26040f538cc6e53.png) In `Settings` / `Cmake`. Change Toolchain to be the remote toolchain just added. Add the following environment variables: ```bash JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 STARROCKS_GCC_HOME=/usr/ STARROCKS_THIRDPARTY=/root/starrocks/thirdparty ``` Notice: Be careful not to check `Include system environment variables`. ![IDE](/assets/images/ide-4-44ca169b84d329083c3bd624627fd200.png) ![IDE](/assets/images/ide-5-b84ccafdc6ed800da98838a727ab1bcf.png) From here on, all setup is complete. After Clion and the remote server are synchronized for a while, the code jump will work normally. ###### Debug[​](#debug "Direct link to Debug") BE debug is a little difficult, you have to use gdb in your remote server. Of course, you can use gdb server + Clion remote gdb, but I don't recommend it, it's too stuck. We need to change `start_backend.sh` script from: ```bash if [ ${RUN_BE} -eq 1 ]; then echo "start time: "$(date) >> $LOG_DIR/be.out if [ ${RUN_DAEMON} -eq 1 ]; then nohup ${STARROCKS_HOME}/lib/starrocks_be "$@" >> $LOG_DIR/be.out 2>&1 > $LOG_DIR/be.out 2>&1 > $LOG_DIR/be.out if [ ${RUN_DAEMON} -eq 1 ]; then nohup ${STARROCKS_HOME}/lib/starrocks_be "$@" >> $LOG_DIR/be.out 2>&1 If you face the error report when debugging for lakehouse, just add `handle SIGSEGV nostop noprint pass` in `~/.gdbinit`. ###### LLVM[​](#llvm "Direct link to LLVM") Of course, you can use LLVM tools to development be. Ubuntu LLVM installtion refer to: Then use the command: `CC=clang-15 CXX=clang++-15 ./build.sh` to compile be. But the premise is that your thirdparty has been compiled with gcc. #### Last[​](#last "Direct link to Last") Feel free to contribute codes to StarRocks. 🫵 --- ### Setup StarRocks FE development environment on IDEA This tutorial is based on macOS and has been tested on Apple Chip(M1, M2). Even if you are not using macOS, you can also refer to this tutorial. #### Requirements[​](#requirements "Direct link to Requirements") ##### Thrift 0.13[​](#thrift-013 "Direct link to Thrift 0.13") There is no 0.13 version of Thrift in the official brew repository; one of our committers created a version in their repo to install. ```bash brew install alberttwong/thrift/thrift@0.13 ``` After installing Thrift successfully, you can check by executing following command: ```bash $ thrift -version Thrift version 0.13.0 ``` ##### Protobuf[​](#protobuf "Direct link to Protobuf") Just use the latest version v3, because the latest version of Protobuf is compatible with the v2 version of the Protobuf used in StarRocks. ```bash brew install protobuf ``` ##### Maven[​](#maven "Direct link to Maven") ```text brew install maven ``` ##### Openjdk 1.8 or 11[​](#openjdk-18-or-11 "Direct link to Openjdk 1.8 or 11") ```bash brew install openjdk@11 ``` ##### Python3[​](#python3 "Direct link to Python3") MacOS is already installed by default. Everyone's Thrift and Protobuf installation directories may be different, you can use the brew list command to inspect: ```bash brew list thrift@0.13.0 brew list protobuf ``` #### Configure the StarRocks[​](#configure-the-starrocks "Direct link to Configure the StarRocks") ##### Download the StarRocks[​](#download-the-starrocks "Direct link to Download the StarRocks") ```text git clone https://github.com/StarRocks/starrocks.git ``` ##### Setup thirdparty directory[​](#setup-thirdparty-directory "Direct link to Setup thirdparty directory") Create `installed/bin` directory in `thirdparty`. ```bash cd starrocks && mkdir -p thirdparty/installed/bin ``` Then create soft link for Thrift and Protobuf respectively. ```bash ln -s /opt/homebrew/bin/thrift thirdparty/installed/bin/thrift ln -s /opt/homebrew/bin/protoc thirdparty/installed/bin/protoc ``` ##### Setting environment variables[​](#setting-environment-variables "Direct link to Setting environment variables") ```bash export JAVA_HOME="/opt/homebrew/Cellar/openjdk@11/11.0.15" # Caution: The jdk version may be different in you desktop export PYTHON=/usr/bin/python3 export STARROCKS_THIRDPARTY=$(pwd)/thirdparty # Caution: Make sure you are in the starrocks directory ``` #### Generate source code[​](#generate-source-code "Direct link to Generate source code") Many source files in FE need to be generated manually, otherwise IDEA will report an error due to missing files. Execute the following command to automatically generate: ```bash make -C gensrc script ./build.sh --be --configure-only cmake --build be/build_Release --target be_proto_codegen be_thrift_codegen ``` The generated BE thrift/protobuf C++ files will appear under the active build directory, for example `be/build_Release/gensrc/gen_cpp`. #### Compile FE[​](#compile-fe "Direct link to Compile FE") Enter `fe` directory and use Maven to compile: ```bash cd fe mvn install -DskipTests ``` #### Open StarRocks in IDEA[​](#open-starrocks-in-idea "Direct link to Open StarRocks in IDEA") 1. Open the `StarRocks` directory in IDEA. 2. Add Coding style setting To standardize the coding style, you should import the `fe/starrocks_intellij_style.xml` code style file in IDEA. ![image-20220701193938856](/assets/images/IDEA-2-8dd4f3fe0da44fb9d1c6f4f035b0a432.png) #### Run StarRocks FE in MacOS[​](#run-starrocks-fe-in-macos "Direct link to Run StarRocks FE in MacOS") Use IDEA to open the `fe` directory. If you execute the Main function directly in `StarRocksFE.java`, some errors will be reported. You only need to do some simple settings to run it smoothly. **NOTICE:** `StarRocksFE.java` is in the `fe/fe-core/src/main/java/com/starrocks` directory. 1. Copy the conf, bin and webroot directories from the StarRocks directory to `fe` directory: ```bash cp -r conf fe/conf cp -r bin fe/bin cp -r webroot fe/webroot ``` 2. Enter the `fe` directory and create the log and meta folders under the `fe` directory: ```bash cd fe mkdir log mkdir meta ``` 3. Set the environment variable, as shown in the following figure: ![image-20220701193938856](/assets/images/IDEA-1-10bdc0d1013b0b63c1ee394608e3091a.png) ```bash export PID_DIR=/Users/smith/Code/starrocks/fe/bin export STARROCKS_HOME=/Users/smith/Code/starrocks/fe export LOG_DIR=/Users/smith/Code/starrocks/fe/log ``` 4. Modify the priority\_networks in `fe/conf/fe.conf` to `127.0.0.1/24` to prevent FE from using the current computer's LAN IP and cause the port fail to bind. 5. Then you have run StarRocks FE successfully. #### DEBUG StarRocks FE in MacOS[​](#debug-starrocks-fe-in-macos "Direct link to DEBUG StarRocks FE in MacOS") If you started the FE with the debug option, you can then attach the IDEA debugger to the FE process. ```text ./start_fe.sh --debug ``` See . --- ### Contribute to StarRocks Contributing to StarRocks is cordially welcome from everyone. Contributing to StarRocks is not limited to contributing code. Below, we list different approaches to contributing to our community. | Report a bug | You can [file an issue](https://github.com/StarRocks/starrocks/issues/new/choose) to report a bug with StarRocks. You can also click `Feedback` in the upper-right corner of the page you are reading in the Documentation Site to report a bug. | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Contribute code** | You can contribute your code by fixing a bug or implementing a feature. | | **Contribute test case** | You can contribute your test cases. | | **Help review code** | If you are an active contributor or committer of StarRocks, you can help us review the pull requests (PRs). | | **Contribute documentation** | StarRocks community maintains a tremendous amount of documentation both in Chinese and English. You can contribute documentation changes by fixing a documentation bug or proposing a new piece of content. | | **Help StarRocks users** | You can help newcomers who meet difficulties in our community. | | **Spread the word about StarRocks** | You can author an article or give a talk about us to help spread our technology to the world. | > **NOTE** > > To contribute documentation, remember to **sign off** your commit using `git commit -s`. Otherwise, the Developer Certificate of Origin (DCO) check will fail and the PR may be blocked. In addition, prefix `[Doc]` to your PR title and select the **Doc** check box. ![Doc title](/assets/images/doctitle-814494a3b89c3e9762496f5bc4fbcc72.png) #### Community resources[​](#community-resources "Direct link to Community resources") The best place to get a wide variety of help about StarRocks is via StarRocks's [Slack Channel](https://docs.starrocks.io/join/) For contribution-related discussions, please go to the **#contributing-to-starrocks** channel. You can also report issues and problems, or suggest new features, on [GitHub](https://github.com/StarRocks/starrocks/). #### Code of conduct[​](#code-of-conduct "Direct link to Code of conduct") Our community strictly adheres to the [code of conduct](https://github.com/StarRocks/starrocks/blob/main/CODE_OF_CONDUCT.md). #### Community roles[​](#community-roles "Direct link to Community roles") ![Community roles](/assets/images/contri-1-3648053cec929acf5bebd3710cb44476.png) Everyone is encouraged to participate in the StarRocks project. Anyone can make an impact by simply being involved in the discussions about new features, project roadmap, architecture, and even reporting issues you are facing. The roles listed below are a few possible ways to get involved in the community, it also defines what is expected from each role. #### User group[​](#user-group "Direct link to User group") ##### Participants[​](#participants "Direct link to Participants") Participants are actively involved in the community and work to make StarRocks better for everyone. As a participant, you can submit issue reports on GitHub, contribute test cases, translate/modify documentation, help answer user questions in forums or communities, participate in events, share your experience with StarRocks, star StarRocks on GitHub, and more. Expectations and responsibilities: * Follow the community's [code of conduct](https://github.com/StarRocks/starrocks/blob/main/CODE_OF_CONDUCT.md). * Be involved in discussions and community events. * Share feedback with the community so everyone else knows what is/isn’t working. * Suggest improvements. ##### Champions[​](#champions "Direct link to Champions") StarRocks Community Champions are a group of passionate community evangelists and pioneers who are well-versed in StarRocks technology. They enjoy sharing the latest developments and products in the community, and their enthusiasm drives the progress and development of the community. ###### Benefits[​](#benefits "Direct link to Benefits") Product * Early access to new products/features. * Participate in exclusive meetings with the project core team. Community * Listed as a StarRocks champion on the website. * Exclusive prizes and badges. Events * Free access to StarRocks' events. * Expense reimbursement for travel and accommodations for events. * VIP seats and VIP events. ###### Requirements[​](#requirements "Direct link to Requirements") 1. Have published at least 3 StarRocks-related original and quality technical content (including articles, tutorials, videos, etc.). 2. Participate in at least 2 online/offline technical sharing sessions as a speaker. 3. Help answer user questions on Slack and other channels. ###### Other Requirements[​](#other-requirements "Direct link to Other Requirements") 1. StarRocks Champions must abide by the StarRocks Community [Code of Conduct.](https://github.com/StarRocks/starrocks/blob/main/CODE_OF_CONDUCT.md) 2. StarRocks Champions are eligible for one year, and must reapply each year based on their contributions in the past year. #### Developer Group[​](#developer-group "Direct link to Developer Group") ##### Contributor[​](#contributor "Direct link to Contributor") Anyone who contributes to StarRocks can become a Contributor. ###### How to become a Contributor?[​](#how-to-become-a-contributor "Direct link to How to become a Contributor?") * 1 merged PR in any StarRocks' public repos. As a Contributor, we expect you to * Actively participate in StarRocks' project development. * Participate in community events (meetups, hackathons, etc.). * Learn and help others learn StarRocks-related technologies. ###### Privileges[​](#privileges "Direct link to Privileges") * Be listed as a StarRocks contributor. ##### Active Contributor[​](#active-contributor "Direct link to Active Contributor") Active contributors are contributors who have made outstanding contributions and sustained commitment to StarRocks. They actively participate in the community by contributing code, improving docs, and helping others. ###### How to become an Active Contributor?[​](#how-to-become-an-active-contributor "Direct link to How to become an Active Contributor?") * Have 5 merged PRs or fixed major bugs. * Participate in more than 5 code reviews. * Actively participate in community events such as online/offline meetups and community discussions. ###### Responsibilities and privileges[​](#responsibilities-and-privileges "Direct link to Responsibilities and privileges") * Join the community meeting and discussion. * Mentor and guide new contributors. * Be listed as a StarRocks Active Contributor. * Be awarded a StarRocks Active Contributor e-certificate. ##### Committer[​](#committer "Direct link to Committer") Committers are contributors who have earned the ability to modify ("commit") source code, documentation, or other technical artifacts in a project’s repository. ###### How to become a Committer?[​](#how-to-become-a-committer "Direct link to How to become a Committer?") * Have a deep understanding of StarRocks' principles and future plans. * Have the ability to deal with various issues that arise in the project promptly. * Lead at least one major development, write and revise related documents * A contributor may become a Committer by a minimum of two votes (excluding the nominator's vote) from the TSC members. ###### Responsibilities and privileges[​](#responsibilities-and-privileges-1 "Direct link to Responsibilities and privileges") * Mentor and guide other members in the community. * Ensure continued health of the subproject. * Be granted write access to StarRocks repos (to be specified). * Be recognized as a StarRocks Committer. ##### Maintainer[​](#maintainer "Direct link to Maintainer") Maintainers are a subset of Committers with additional responsibilities for driving a project’s release and serving on the TSC (optional). ###### How to become a Maintainer?[​](#how-to-become-a-maintainer "Direct link to How to become a Maintainer?") * In-depth understanding of StarRocks principles and a clear understanding of StarRocks' future plans. * Have the ability to deal with project issues promptly. * Lead project development and iterations, and steer the overall direction of the project. * A Committer may become a Maintainer by a minimum of two votes (excluding the nominator's vote) from the TSC members. ###### Responsibilities and privileges[​](#responsibilities-and-privileges-2 "Direct link to Responsibilities and privileges") * Mentor and guide other memberships in the community. * Ensure continued health of the project, such as code quality and test coverage. * Make and approve technical design decisions. * Define milestones and releases. * Nominate new SIG committers to TSC. * Be recognized as a StarRocks Maintainer. #### Contribution process[​](#contribution-process "Direct link to Contribution process") If you don't know how to get started, this is the process we suggest for contributions. This process is designed to help reduce your learning curve and get your pull requests merged more efficiently! 1. Sign the [Contributor License Agreement (CLA)](https://cla-assistant.io/StarRocks/starrocks). 2. Start a discussion by creating a Github [issue](https://github.com/StarRocks/starrocks/issues), or asking on [Slack channel](https://docs.starrocks.io/join/) unless the change is trivial). Before getting your hands on codes, you should comment in the issue body, and inform the maintainer to assign you the issue that you wish to solve. It is recommended to share your plan on how to solve this problem in the issue body as well. * This step helps you identify possible collaborators and reviewers. * Will the change conflict with another change in progress? If so, work with others to minimize impact. * Is this change major? If so, work with others to break the change into smaller steps. 3. Implement the change. * If the change is major, split it into smaller PRs. * Include tests and documentation as necessary. 4. Create a Github [pull request](https://github.com/StarRocks/starrocks/pulls): In StarRocks community, we follow the fork-and-merge GitHub workflow when contributing code. * Create a fork of StarRocks in your GitHub account. * Clone this forked repository to your computer. * Check out a new branch based on the branch you expect to contribute to. * Commit your code changes to the new branch. * Push the branch with code changes to GitHub. * Create a PR to submit your code changes. It is recommended to submit ONE commit in ONE PR. You can follow the [PR templated](https://github.com/StarRocks/starrocks/blob/main/.github/PULL_REQUEST_TEMPLATE.md) when submitting a PR. * Make sure the pull request passes the tests in CI. 5. Review is required by at least 2 reviewers: * For StarRocks project, we require at least 2 lgtm from reviewers (committers) to merge the pull request. * This normally happens within a few days, but may take longer if the change is major, complex, or if a critical reviewer is unavailable. (feel free to ping the reviewer on the pull request). 6. Maintainers merge the pull request after the final changes are accepted. --- ### Jemalloc Heap Profiling This topic describes how to enable and visualize the Jemalloc Heap Profile for StarRocks. note * Enabling Jemalloc Heap Profiling may have an impact on StarRocks' performance. * This solution is available only for StarRocks v3.1.6 and later. #### Enable Jemalloc Heap Profile[​](#enable-jemalloc-heap-profile "Direct link to Enable Jemalloc Heap Profile") Syntax: ```sql ADMIN EXECUTE ON 'System.print(HeapProf.getInstance().enable_prof())' ``` Enabling requires the BE to have been started with `prof:true` in `jemalloc_conf`, which is the default; otherwise the statement fails. The current setting is also visible in `information_schema.be_configs`: query the row whose `NAME` is `jemalloc_conf` and read the `prof_active` option inside its `VALUE`. `be_id`: The ID of BE/CN node. You can get the ID by running SHOW BACKENDS or SHOW COMPUTE NODES. Example: ```sql mysql> admin execute on 10001 'System.print(HeapProf.getInstance().enable_prof())'; +--------+ | result | +--------+ | OK | +--------+ 1 row in set (0.00 sec) ``` You can check whether Jemalloc Heap Profile is enabled or not by using the following syntax: ```sql ADMIN EXECUTE ON 'System.print(HeapProf.getInstance().has_enable())' ``` Example: ```sql mysql> admin execute on 10001 'System.print(HeapProf.getInstance().has_enable())'; +--------+ | result | +--------+ | true | +--------+ 1 row in set (0.01 sec) ``` You can disable Jemalloc Heap Profile by using the following syntax: ```sql ADMIN EXECUTE ON 'System.print(HeapProf.getInstance().disable_prof())' ``` Example: ```sql mysql> admin execute on 10001 'System.print(HeapProf.getInstance().disable_prof())'; +--------+ | result | +--------+ | OK | +--------+ 1 row in set (0.00 sec) ``` #### Collect Jemalloc Heap Profile[​](#collect-jemalloc-heap-profile "Direct link to Collect Jemalloc Heap Profile") Syntax: ```sql ADMIN EXECUTE ON 'System.print(HeapProf.getInstance().dump_dot_snapshot())' ``` Example: ```sql mysql> admin execute on 10001 'System.print(HeapProf.getInstance().dump_dot_snapshot())'; +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | result | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | digraph "/home/disk/opt/env/default/be/lib/starrocks_be; 1.0 MB" { | | node [width=0.375,height=0.25]; | | Legend [shape=box,fontsize=24,shape=plaintext,label="/home/disk/opt/env/default/be/lib/starrocks_be\lTotal MB: 1.0\lFocusing on: 1.0\lDropped nodes with <= 0.0 abs(MB)\lDropped edges with <= 0.0 MB\l"]; | | N1 [label="brpc\nInputMessenger\nOnNewMessages\n0.0 (0.0%)\rof 1.0 (100.0%)\r",shape=box,fontsize=8.0]; | | N2 [label="brpc\nSocket\nProcessEvent\n0.0 (0.0%)\rof 1.0 (100.0%)\r",shape=box,fontsize=8.0]; | | N3 [label="bthread\nTaskGroup\ntask_runner\n0.0 (0.0%)\rof 1.0 (100.0%)\r",shape=box,fontsize=8.0]; | | N4 [label="bthread_make_fcontext\n0.0 (0.0%)\rof 1.0 (100.0%)\r",shape=box,fontsize=8.0]; | | N5 [label="brpc\nInputMessenger\nCutInputMessage\n0.0 (0.0%)\rof 0.5 (50.1%)\r",shape=box,fontsize=8.0]; | | N6 [label="brpc\npolicy\nParseRpcMessage\n0.5 (50.1%)\r",shape=box,fontsize=43.4]; | | N7 [label="brpc\nProcessInputMessage\n0.0 (0.0%)\rof 0.5 (49.9%)\r",shape=box,fontsize=8.0]; | | N8 [label="brpc\npolicy\nProcessRpcRequest\n0.0 (0.0%)\rof 0.5 (49.9%)\r",shape=box,fontsize=8.0]; | | N9 [label="starrocks\nPInternalServiceImplBase\nexecute_command\n0.0 (0.0%)\rof 0.5 (49.9%)\r",shape=box,fontsize=8.0]; | | N10 [label="starrocks\nStorageEngineRef\nbind\n0.0 (0.0%)\rof 0.5 (49.9%)\r",shape=box,fontsize=8.0]; | | N11 [label="starrocks\nexecute_command\n0.0 (0.0%)\rof 0.5 (49.9%)\r",shape=box,fontsize=8.0]; | | N12 [label="starrocks\nexecute_script\n0.0 (0.0%)\rof 0.5 (49.9%)\r",shape=box,fontsize=8.0]; | | N13 [label="std\nmake_unique\n0.5 (49.9%)\r",shape=box,fontsize=43.3]; | | N2 -> N1 [label=1.0, weight=16398, style="setlinewidth(2.000000)"]; | | N3 -> N2 [label=1.0, weight=16398, style="setlinewidth(2.000000)"]; | | N4 -> N3 [label=1.0, weight=16398, style="setlinewidth(2.000000)"]; | | N1 -> N5 [label=0.5, weight=10102, style="setlinewidth(2.000000)"]; | | N5 -> N6 [label=0.5, weight=10102, style="setlinewidth(2.000000)"]; | | N9 -> N11 [label=0.5, weight=10086, style="setlinewidth(2.000000)"]; | | N12 -> N10 [label=0.5, weight=10086, style="setlinewidth(2.000000)"]; | | N11 -> N12 [label=0.5, weight=10086, style="setlinewidth(2.000000)"]; | | N1 -> N7 [label=0.5, weight=10086, style="setlinewidth(2.000000)"]; | | N7 -> N8 [label=0.5, weight=10086, style="setlinewidth(2.000000)"]; | | N10 -> N13 [label=0.5, weight=10086, style="setlinewidth(2.000000)"]; | | N8 -> N9 [label=0.5, weight=10086, style="setlinewidth(2.000000)"]; | | } | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 29 rows in set (30.22 sec) ``` #### Visualize Jemalloc Heap Profile[​](#visualize-jemalloc-heap-profile "Direct link to Visualize Jemalloc Heap Profile") Copy the profile text you collected in the last step, and paste it to [GraphvizOnline](https://dreampuf.github.io/GraphvizOnline/). You can then download the visualized Heap Profile. Example: ![Example - Visualized Heap Profile](/assets/images/visualized_heap_profile-886621b6e11bd0a830b24f807c7409ea.png) --- ### Query Trace Profile This topic introduces how to obtain and analyze query trace profiles. A query trace profile records the debug information for a specified query statement, including time costs, variables & values, and logs. Such information is categorized into several modules, allowing you to debug and identify the performance bottlenecks from different aspects. This feature is supported from v3.2.0 onwards. #### Syntax[​](#syntax "Direct link to Syntax") You can use the following syntax to obtain the trace profile of a query: ```sql TRACE { TIMES | VALUES | LOGS | ALL } [ ] ``` * `TIMES`: Traces the time costs of events in each stage of the specified query. * `VALUES`: Traces the variables and their values of the specified query. * `LOGS`: Traces the log records of the specified query. * `ALL`: Lists all the `TIMES`, `VALUES`, and `LOGS` information in chronological order. * ``: The module you want to trace information from. Valid values: * `BASE`: The base module. * `MV`: The materialized view module. * `OPTIMIZER`: The optimizer module. * `SCHEDULE`: The schedule module. * `EXTERNAL`: The external table-related module. If no module is specified, `BASE` is used. * ``: The query statement whose query trace profile you want to obtain. #### Use cases[​](#use-cases "Direct link to Use cases") ##### Trace the time costs of a query[​](#trace-the-time-costs-of-a-query "Direct link to Trace the time costs of a query") The following example traces the time costs of a query's optimizer module. ```plain MySQL > TRACE TIMES OPTIMIZER SELECT * FROM t1 JOIN t2 ON t1.v1 = t2.v1; +---------------------------------------------------------------------+ | Explain String | +---------------------------------------------------------------------+ | 2ms|-- Total[1] 15ms | | 2ms| -- Analyzer[1] 1ms | | 4ms| -- Transformer[1] 1ms | | 6ms| -- Optimizer[1] 11ms | | 6ms| -- preprocessMvs[1] 0 | | 6ms| -- RuleBaseOptimize[1] 3ms | | 6ms| -- RewriteTreeTask[41] 2ms | | 7ms| -- PushDownJoinOnClauseRule[1] 0 | | 7ms| -- PushDownPredicateProjectRule[2] 0 | | 7ms| -- PushDownPredicateScanRule[2] 0 | | 8ms| -- MergeTwoProjectRule[3] 0 | | 8ms| -- PushDownJoinOnExpressionToChildProject[1] 0 | | 8ms| -- PruneProjectColumnsRule[6] 0 | | 8ms| -- PruneJoinColumnsRule[2] 0 | | 8ms| -- PruneScanColumnRule[4] 0 | | 9ms| -- PruneSubfieldRule[2] 0 | | 9ms| -- PruneProjectRule[6] 0 | | 9ms| -- PartitionPruneRule[2] 0 | | 9ms| -- DistributionPruneRule[2] 0 | | 9ms| -- MergeProjectWithChildRule[3] 0 | | 10ms| -- CostBaseOptimize[1] 6ms | | 10ms| -- OptimizeGroupTask[6] 0 | | 10ms| -- OptimizeExpressionTask[9] 0 | | 10ms| -- ExploreGroupTask[4] 0 | | 10ms| -- DeriveStatsTask[9] 3ms | | 13ms| -- ApplyRuleTask[16] 0 | | 13ms| -- OnlyScanRule[2] 0 | | 14ms| -- HashJoinImplementationRule[2] 0 | | 14ms| -- EnforceAndCostTask[12] 1ms | | 14ms| -- OlapScanImplementationRule[2] 0 | | 15ms| -- OnlyJoinRule[2] 0 | | 15ms| -- JoinCommutativityRule[1] 0 | | 16ms| -- PhysicalRewrite[1] 0 | | 17ms| -- PlanValidate[1] 0 | | 17ms| -- InputDependenciesChecker[1] 0 | | 17ms| -- TypeChecker[1] 0 | | 17ms| -- CTEUniqueChecker[1] 0 | | 17ms| -- ExecPlanBuild[1] 0 | | Tracer Cost: 273us | +---------------------------------------------------------------------+ 39 rows in set (0.029 sec) ``` In the **Explain String** returned by the TRACE TIMES statement, each row (except the last row) corresponds to an event in the specified module (stage) of the query. The last row `Tracer Cost` records the time cost of the tracing process. Take `| 4ms| -- Transformer[1] 1ms` as an example: * The left column records the time point, in the lifecycle of the query, when the event was first executed. * In the right column, following the consecutive hyphens is the name of the event, for example, `Transformer`. * Following the event name, the number in the brackets (`[1]`) indicates the number of times the event was executed. * The last part of this column is the overall time cost of the event, for example, `1ms`. * The records of events are indented based on the depth of the method stack. That is to say, in this example, the first execution of `Transformer` always happens within `Total`. ##### Trace the variables of a query[​](#trace-the-variables-of-a-query "Direct link to Trace the variables of a query") The following example traces the variables & values of a query's MV module. ```plain MySQL > TRACE VALUES MV SELECT t1.v2, sum(t1.v3) FROM t1 JOIN t0 ON t1.v1 = t0.v1 GROUP BY t1.v2; +----------------------------+ | Explain String | +----------------------------+ | 32ms| mv2: Rewrite Succeed | | Tracer Cost: 66us | +----------------------------+ 2 rows in set (0.045 sec) ``` The structure of the **Explain String** returned by the TRACE VALUES statement is similar to that of the TRACE TIMES statement, except that the right column records the variables and settings of the event in the specified module. The above example records that the materialized view is successfully used to rewrite the query. ##### Trace the logs of a query[​](#trace-the-logs-of-a-query "Direct link to Trace the logs of a query") The following example traces the logs of a query's MV module. ```plain MySQL > TRACE LOGS MV SELECT v2, sum(v3) FROM t1 GROUP BY v2; +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 3ms| [MV TRACE] [PREPARE cac571e8-47f9-11ee-abfb-2e95bcb5f199][mv2] [SYNC=false] Prepare MV mv2 success | | 3ms| [MV TRACE] [PREPARE cac571e8-47f9-11ee-abfb-2e95bcb5f199][GLOBAL] [SYNC=false] RelatedMVs: [mv2], CandidateMVs: [mv2] | | 4ms| [MV TRACE] [PREPARE cac571e8-47f9-11ee-abfb-2e95bcb5f199][GLOBAL] [SYNC=true] There are no related mvs for the query plan | | 35ms| [MV TRACE] [REWRITE cac571e8-47f9-11ee-abfb-2e95bcb5f199 TF_MV_AGGREGATE_SCAN_RULE mv2] Rewrite ViewDelta failed: cannot compensate query by using PK/FK constraints | | 35ms| [MV TRACE] [REWRITE cac571e8-47f9-11ee-abfb-2e95bcb5f199 TF_MV_ONLY_SCAN_RULE mv2] MV is not applicable: mv expression is not valid | | 43ms| Query cannot be rewritten, please check the trace logs or `set enable_mv_optimizer_trace_log=on` to find more infos. | | Tracer Cost: 400us | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 7 rows in set (0.056 sec) ``` Alternatively, you can print these logs in the FE log file **fe.log** by setting the variable `trace_log_mode` as follows: ```sql SET trace_log_mode='file'; ``` The default value of `trace_log_mode` is `command`, indicating that the logs are returned as the **Explain String** as shown above. If you set its value to `file`, the logs are printed in the FE log file **fe.log** with the class name being `FileLogTracer`. After you set `trace_log_mode` to `file`, no logs will be returned when you execute the TRACE LOGS statement. Example: ```plain MySQL > TRACE LOGS OPTIMIZER SELECT v1 FROM t1 ; +---------------------+ | Explain String | +---------------------+ | Tracer Cost: 3422us | +---------------------+ 1 row in set (0.023 sec) ``` The log will be printed in **fe.log**. ![img](/assets/images/query_trace_profile-bcdf3ff971f4c20f7f9d591c97b2c5c8.png) --- ### Trace ##### Background[​](#background "Direct link to Background")  A Distributed Trace, more commonly known as a Trace, records the paths taken by requests (made by an application or end-user) as they propagate through multi-service architectures, like microservice and serverless applications. Without tracing, it is challenging to pinpoint the cause of performance problems in a distributed system. It improves the visibility of our application or system’s health and lets us debug behavior that is difficult to reproduce locally. Tracing is essential for distributed systems, which commonly have nondeterministic problems or are too complicated to reproduce locally.  Tracing makes debugging and understanding distributed systems less daunting by breaking down what happens within a request as it flows through a distributed system. A Trace is made of one or more Spans. The first Span represents the Root Span. Each Root Span represents a request from start to finish. The Spans underneath the parent provide a more in-depth context of what occurs during a request (or what steps make up a request). Many Observability back-ends visualize Traces as waterfall diagrams that may look something like this picture. ![trace\_pic1](/assets/images/trace_pic1-1d5e52ca0f5dc47f1b716e1106fdc267.png)  Waterfall diagrams show the parent-child relationship between a Root Span and its child Spans. When a Span encapsulates another Span, this also represents a nested relationship.  Recently SR added a tracing framework. It leverages opentelemetry and jaeger to trace distributed events in the system. * Opentelemetry is an instrumentation/tracing SDK. Developers can use it to instrument code and emit tracing data to an observability backend. It supports many languages. We use java and CPP SDK in SR. * Currently, Jaeger is used as the observability backend. ##### Basic Usage[​](#basic-usage "Direct link to Basic Usage") Steps to enable tracing in SR: 1. Install [Jaeger](https://www.jaegertracing.io/docs/1.76/getting-started/) The guide above uses docker. For simplicity, you can also just download [binary package](https://github.com/jaegertracing/jaeger/releases) and run locally. ```text decster@decster-MS-7C94:~/soft/jaeger-1.31.0-linux-amd64$ ll total 215836 drwxr-xr-x 2 decster decster 4096 02-05 05:01:30 ./ drwxrwxr-x 28 decster decster 4096 05-18 18:24:07 ../ -rwxr-xr-x 1 decster decster 19323884 02-05 05:01:31 example-hotrod* -rwxr-xr-x 1 decster decster 23430444 02-05 05:01:29 jaeger-agent* -rwxr-xr-x 1 decster decster 51694774 02-05 05:01:29 jaeger-all-in-one* -rwxr-xr-x 1 decster decster 41273869 02-05 05:01:30 jaeger-collector* -rwxr-xr-x 1 decster decster 37576660 02-05 05:01:30 jaeger-ingester* -rwxr-xr-x 1 decster decster 47698843 02-05 05:01:30 jaeger-query* decster@decster-MS-7C94:~/soft/jaeger-1.31.0-linux-amd64$ ./jaeger-all-in-one ``` 2. Config FE\&FE to enable tracing. Currently, opentelemetry java & cpp sdk use different protocols, java uses grpc proto, while cpp uses thrift\&UDP, so the endpoint ports are different. ```text fe.conf # Enable jaeger tracing by setting jaeger_grpc_endpoint # jaeger_grpc_endpoint = http://localhost:14250 be.conf # Enable jaeger tracing by setting jaeger_endpoint # jaeger_endpoint = localhost:6831 ``` 3. Open jaeger web UI, usually in `http://localhost:16686/search` 4. Do some data ingestion (streamload/insert into) and search TXN traces on web UI ![trace\_pic2.png](/assets/images/trace_pic2-7667cb8c15ecdfce6a0eb93a365a96e5.png)(trace\_pic2.png) ![trace\_pic3.png](/assets/images/trace_pic3-31201861601577398a87e72701f9f3fa.png)(trace\_pic3.png) ##### Adding traces[​](#adding-traces "Direct link to Adding traces") * To add trace, first get familiar with basic concepts like tracer, span, trace propagation read the [observability primer](https://opentelemetry.io/docs/concepts/observability-primer/) * Read utility class and it's usages in SR: TraceManager.java(java) `common/tracer.h/cpp (cpp)`, it's current usage(like write txn(load/insert/update/delete) trace, and its propagation to BE). * Add your own trace --- ### Version Release Guide Version naming is detailed in the [versioning](https://docs.starrocks.io/docs/maturity/release_numbering.md) documentation. Read that page first to understand **major**, **minor**, and **patch** versioning. #### Release Plan[​](#release-plan "Direct link to Release Plan") * Release a minor version approximately every four months. * Maintain the three latest minor versions (**minor** is the second number in the dot separated version, for example, in 3.4.2 **4** is the minor version). With a minor version released every four months, a minor version should be expected to be supported for at most one year. * Release a patch version within 2-3 weeks for the minor version in maintenance. #### Pull Request Type[​](#pull-request-type "Direct link to Pull Request Type") Every pull request in StarRocks should be titled with a type, including **feature**, **enhancement**, and **bugfix**. ##### Feature[​](#feature "Direct link to Feature") * Definition: A feature is a new capability or functionality that did not previously exist in the database. It adds new behavior or significantly extends the existing functionality. * Examples: * Adding a new type of data structure (e.g., a new table type or index type). * Implementing a new query language feature (e.g., a new SQL function or operator). * Introducing a new API endpoint or interface for interacting with the database. ##### Enhancement[​](#enhancement "Direct link to Enhancement") * Definition: An enhancement is an improvement to an existing feature or functionality. It does not introduce entirely new behavior but makes existing features better, faster, or more efficient. * Examples: * Optimizing the performance of a query execution plan. * Improving the user interface of the database management tool. * Enhancing the security features by adding more granular access controls. ##### Bugfix[​](#bugfix "Direct link to Bugfix") * Definition: A bugfix is a correction of an error or flaw in the existing code. It addresses issues that prevent the database from functioning correctly or as intended. * Examples: * Fixing a crash that occurs under certain query conditions. * Correcting an incorrect result returned by a query. * Resolving a memory leak or resource management issue. #### Cherry-pick Rule[​](#cherry-pick-rule "Direct link to Cherry-pick Rule") We define some status for a minor version to assist cherry-pick management. You can find the version status in the `.github/.status` file. For example, at the time this document was published, StarRocks version 3.4 is in the `feature-freeze` state and version 3.3 is `bugfix-only`. To verify this: ```bash git switch branch-3.3 cat .github/.status ``` ```bash bugfix-only ``` 1. `open`: All types of pull requests can be merged, including feature, enhancement, and bugfix. 2. `feature-freeze`: Only enhancement and bugfix pull requests can be merged. 3. `bugfix-only`: Only bugfix pull requests can be merged. 4. `code-freeze`: No pull requests can be merged except critical CVE fixes. The minor version status will change with some baseline triggers, as shown below, and it can also be changed in advance if necessary. 1. When the minor version branch is created, it becomes `open` and stays `open` until it is released. 2. When the minor version is released, it becomes `feature-freeze`. 3. When the next minor version is released, the previous minor version becomes `bugfix-only`. 4. The minor version stays as `bugfix-only` until three more minor versions are released, and then becomes `code-freeze`. ##### Example[​](#example "Direct link to Example") * branch-5.1 is created, this branch is in the `open` state until it passes through release candidate and is publicly released. * Once version 5.1 is publicly released it enters the `feature-freeze` state. * Once version 5.2 is publicly released 5.1 switches to `bugfix-only`. * When versions 5.1, 5.2, 5.3, and 5.4 are all released: * 5.4 is in the `feature-freeze` state * 5.3 is in the `bugfix-only` state * 5.2 is also in the `bugfix-only` state * 5.1 is in the `code-freeze` state #### JDK Support Policy[​](#jdk-support-policy "Direct link to JDK Support Policy") StarRocks FE, BE, and CN nodes depend on a JDK. The JDK versions that StarRocks requires and recommends follow the Java LTS calendar published by [Eclipse Adoptium](https://adoptium.net/support), not StarRocks version numbers. For the JDK versions of each StarRocks version, see [JDK version by StarRocks version](https://docs.starrocks.io/docs/deployment/preparation/environment_configurations.md#jdk-version-by-starrocks-version). Each StarRocks minor version defines two JDK versions: * **Minimum JDK**: the oldest Java LTS release that Adoptium still builds. The FE does not start on an older JDK. The BE and CN log an error on an older JDK, and Java-dependent features such as Java UDFs and JNI-based connectors are not supported on it. * **Recommended JDK**: the next Java LTS release. It is the JDK that the official container images ship, and it becomes the minimum JDK at the next change. Starting on a JDK that is older than the recommended JDK prints a deprecation warning that names the recommended JDK. Java LTS releases newer than the recommended JDK are not yet covered by this policy. They usually work but are not tested. Non-LTS Java releases are not supported. Currently, the minimum JDK is JDK 17 and the recommended JDK is JDK 21. ##### When the minimum JDK changes[​](#when-the-minimum-jdk-changes "Direct link to When the minimum JDK changes") * The minimum JDK changes when the current minimum reaches its end of availability on Adoptium. At that point, the previous recommended JDK becomes the minimum, and the next Java LTS release becomes the recommended JDK. * The recommended JDK is published one full cycle before it becomes the minimum, so you get about two years of notice through this policy, the JDK matrix, and the startup warning. * The change takes effect in the first minor version released after the date, and only if the previous minor version already printed the startup warning for the new minimum. Otherwise, the change moves to the next minor version. * A minor version keeps its minimum and recommended JDK for its whole life. A patch version never raises the minimum JDK, so upgrading to a patch version never requires a JDK upgrade. Support for a newer JDK can be added to an existing minor version in a patch version. ##### Schedule[​](#schedule "Direct link to Schedule") | Effective from | Minimum JDK | Recommended JDK | Reason | | -------------- | ----------- | --------------- | ----------------------------------- | | Oct 2027 | JDK 21 | JDK 25 | Java 17 reaches end of availability | | Dec 2029 | JDK 25 | JDK 29 | Java 21 reaches end of availability | | Sep 2031 | JDK 29 | JDK 33 | Java 25 reaches end of availability | The dates are the end-of-availability dates published by Adoptium, which Adoptium states as "at least". They are fixed: if Adoptium later extends the availability of a Java version, the schedule does not change. Java 29 and Java 33 assume that Adoptium continues to designate an LTS release every two years. ##### Components that run in another JVM[​](#components-that-run-in-another-jvm "Direct link to Components that run in another JVM") The Spark Load DPP library, the Hive Bitmap UDF library, and the Broker run inside another system's JVM. They target Java 8 and follow the JDK requirements of that system, not this policy. --- ## Faq ### Data Export #### Alibaba cloud OSS backup and restore[​](#alibaba-cloud-oss-backup-and-restore "Direct link to Alibaba cloud OSS backup and restore") StarRocks supports backing up data to alicloud OSS / AWS S3 (or object storage compatible with S3 protocol). Suppose there are two StarRocks clusters, namely DB1 cluster and DB2 cluster. We need to back up the data in DB1 to alicloud OSS and then restore it to DB2 when necessary. The general process of backup and recovery is as follows: ##### Create a cloud repository[​](#create-a-cloud-repository "Direct link to Create a cloud repository") Execute SQL in DB1 and DB2 respectively: ```sql CREATE REPOSITORY `repository name` WITH BROKER `broker_name` ON LOCATION "oss://bucket name/path" PROPERTIES ( "fs.oss.accessKeyId" = "xxx", "fs.oss.accessKeySecret" = "yyy", "fs.oss.endpoint" = "oss-cn-beijing.aliyuncs.com" ); ``` a. Both DB1 and DB2 need to be created, and the created REPOSITORY name should be the same. View the repository: ```sql SHOW REPOSITORIES; ``` b. broker\_ name needs to fill in the broker name in a cluster. View BrokerName: ```sql SHOW BROKER; ``` c. The path after fs.oss.endpoint does not need to have a bucket name. ##### Backup data table[​](#backup-data-table "Direct link to Backup data table") BACKUP the tables to be backed up to the cloud repository in DB1. Execute SQL in DB1: ```sql BACKUP SNAPSHOT [db_name].{snapshot_name} TO `repository_name` ON ( `table_name` [PARTITION (`p1`, ...)], ... ) PROPERTIES ("key"="value", ...); ``` ```plain PROPERTIES currently supports the following properties: "type" = "full": indicates that this is a full update (default). "timeout" = "3600": task timeout. The default is one day. The unit is seconds. ``` StarRocks does not support full database backup at present. We need to specify the tables or partitions to be backed up ON (...), and these tables or partitions will be backed up in parallel. View the backup tasks in progress (note that only one backup task can be performed at the same time): ```sql SHOW BACKUP FROM db_name; ``` After the backup is completed, you can check whether the backup data in the OSS already exists (unnecessary backups need to be deleted in the OSS): ```sql SHOW SNAPSHOT ON OSS repository name; ``` ##### Data restore[​](#data-restore "Direct link to Data restore") For data restore in DB2, there is no need to create a table structure to be restored in DB2. It will be created automatically during the Restore operation. Perform restore SQL: ```sql RESTORE SNAPSHOT [db_name].{snapshot_name} FROMrepository_name`` ON ( 'table_name' [PARTITION ('p1', ...)] [AS 'tbl_alias'], ... ) PROPERTIES ("key"="value", ...); ``` View the restore progress: ```sql SHOW RESTORE; ``` --- ### Broker Load #### 1. Does Broker Load support re-running load jobs that have been run successfully and are in the FINISHED state?[​](#1-does-broker-load-support-re-running-load-jobs-that-have-been-run-successfully-and-are-in-the-finished-state "Direct link to 1. Does Broker Load support re-running load jobs that have been run successfully and are in the FINISHED state?") Broker Load does not support re-running load jobs that have been run successfully and are in the FINISHED state. Also, to prevent data loss and duplication, Broker Load does not allow reusing the labels of successfully run load jobs. You can use [SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md) to view the history of load jobs and find the load job that you want to re-run. Then, you can copy the information of that load job and use the job information, except the label, to create another load job. #### 2. When I load data from HDFS by using Broker Load, what do I do if the date and time values loaded into the destination StarRocks table are 8 hours later than the date and time values from the source data file?[​](#2-when-i-load-data-from-hdfs-by-using-broker-load-what-do-i-do-if-the-date-and-time-values-loaded-into-the-destination-starrocks-table-are-8-hours-later-than-the-date-and-time-values-from-the-source-data-file "Direct link to 2. When I load data from HDFS by using Broker Load, what do I do if the date and time values loaded into the destination StarRocks table are 8 hours later than the date and time values from the source data file?") Both the destination StarRocks table and the Broker Load job are compiled at creation to use a China Standard Time (CST) time zone (specified by using the `timezone` parameter). However, the server is set to run based on a Coordinated Universal Time (UTC) time zone. As a result, 8 extra hours are added to the date and time values from the source data file during data loading. To prevent this issue, do not specify the `timezone` parameter when you create the destination StarRocks table. #### 3. When I load ORC-formatted data by using Broker Load, what do I do if the `ErrorMsg: type:ETL_RUN_FAIL; msg:Cannot cast '' from VARCHAR to ARRAY` error occurs?[​](#3-when-i-load-orc-formatted-data-by-using-broker-load-what-do-i-do-if-the-errormsg-typeetl_run_fail-msgcannot-cast-slot-6-from-varchar-to-arrayvarchar30-error-occurs "Direct link to 3-when-i-load-orc-formatted-data-by-using-broker-load-what-do-i-do-if-the-errormsg-typeetl_run_fail-msgcannot-cast-slot-6-from-varchar-to-arrayvarchar30-error-occurs") The source data file has different column names than the destination StarRocks table. In this situation, you must use the `SET` clause in the load statement to specify the column mapping between the file and the table. When executing the `SET` clause, StarRocks needs to perform a type inference, but it fails in invoking the [cast](https://docs.starrocks.io/docs/sql-reference/sql-functions/cast.md) function to transform the source data to the destination data types. To resolve this issue, make sure that the source data file has the same column names as the destination StarRocks table. As such, the `SET` clause is not needed and therefore StarRocks does not need to invoke the cast function to perform data type conversions. Then the Broker Load job can be run successfully. #### 4. The Broker Load job does not report errors, but why am I unable to query the loaded data?[​](#4-the-broker-load-job-does-not-report-errors-but-why-am-i-unable-to-query-the-loaded-data "Direct link to 4. The Broker Load job does not report errors, but why am I unable to query the loaded data?") Broker Load is an asynchronous loading method. The load job may still fail even if the load statement does not return errors. After you run a Broker Load job, you can use [SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md) to view the result and `errmsg` of the load job. Then, you can modify the job configuration and retry. #### 5. What do I do if the "failed to send batch" or "TabletWriter add batch with unknown id" error occurs?[​](#5-what-do-i-do-if-the-failed-to-send-batch-or-tabletwriter-add-batch-with-unknown-id-error-occurs "Direct link to 5. What do I do if the \"failed to send batch\" or \"TabletWriter add batch with unknown id\" error occurs?") The amount of time taken to write the data exceeds the upper limit, causing a timeout error. To resolve this issue, modify the settings of the [session variable](https://docs.starrocks.io/docs/sql-reference/System_variable.md) `query_timeout` and the [BE configuration item](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#configure-be-static-parameters) `streaming_load_rpc_max_alive_time_sec` based on your business requirements. #### 6. What do I do if the "LOAD-RUN-FAIL; msg:OrcScannerAdapter::init\_include\_columns. col name = xxx not found" error occurs?[​](#6-what-do-i-do-if-the-load-run-fail-msgorcscanneradapterinit_include_columns-col-name--xxx-not-found-error-occurs "Direct link to 6. What do I do if the \"LOAD-RUN-FAIL; msg:OrcScannerAdapter::init_include_columns. col name = xxx not found\" error occurs?") If you are loading Parquet- or ORC-formatted data, check whether the column names held in the first row of the source data file are the same as the column names of the destination StarRocks table. ```sql (tmp_c1,tmp_c2) SET ( id=tmp_c2, name=tmp_c1 ) ``` The preceding example maps the `tmp_c1` and `tmp_c2` columns of the source data file onto the `name` and `id` columns of the destination StarRocks table, respectively. If you do not specify the `SET` clause, the column names specified in the `column_list` parameter are used to declare the column mapping. For more information, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). > **NOTICE** > > If the source data file is an ORC-formatted file generated by Apache Hive™ and the first row of the file holds `(_col0, _col1, _col2, ...)`, the "Invalid Column Name" error may occur. If this error occurs, you need to use the `SET` clause to specify the column mapping. #### 7. How do I handle errors such as the error that causes the Broker Load job to run for an excessively long period of time?[​](#7-how-do-i-handle-errors-such-as-the-error-that-causes-the-broker-load-job-to-run-for-an-excessively-long-period-of-time "Direct link to 7. How do I handle errors such as the error that causes the Broker Load job to run for an excessively long period of time?") View the FE log file **fe.log** and search for the ID of the load job based on the job label. Then, view the BE log file **be.INFO** and retrieve the log records of the load job based on the job ID to locate the root cause of the error. #### 8. How do I configure an Apache HDFS cluster that runs in HA mode?[​](#8-how-do-i-configure-an-apache-hdfs-cluster-that-runs-in-ha-mode "Direct link to 8. How do I configure an Apache HDFS cluster that runs in HA mode?") If an HDFS cluster runs in high availability (HA) mode, configure it as follows: * `dfs.nameservices`: the name of the HDFS cluster, for example, `"dfs.nameservices" = "my_ha"`. * `dfs.ha.namenodes.xxx`: the name of the NameNode in the HDFS cluster. If you specify multiple NameNode names, separate them with commas (`,`). `xxx` is the HDFS cluster name that you have specified in `dfs.nameservices`, for example, `"dfs.ha.namenodes.my_ha" = "my_nn"`. * `dfs.namenode.rpc-address.xxx.nn`: the RPC address of the NameNode in the HDFS cluster. `nn` is the NameNode name that you have specified in `dfs.ha.namenodes.xxx`, for example, `"dfs.namenode.rpc-address.my_ha.my_nn" = "host:port"`. * `dfs.client.failover.proxy.provider`: the provider of the NameNode to which the client will connect. Default value: `org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider`. Example: ```sql ( "dfs.nameservices" = "my-ha", "dfs.ha.namenodes.my-ha" = "my-namenode1, my-namenode2", "dfs.namenode.rpc-address.my-ha.my-namenode1" = "nn1-host:rpc_port", "dfs.namenode.rpc-address.my-ha.my-namenode2" = "nn2-host:rpc_port", "dfs.client.failover.proxy.provider" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider" ) ``` The HA mode can be used with simple authentication or Kerberos authentication. For example, to use simple authentication to access an HDFS cluster that runs in HA mode, you need to specify the following configurations: ```sql ( "username"="user", "password"="passwd", "dfs.nameservices" = "my-ha", "dfs.ha.namenodes.my-ha" = "my_namenode1, my_namenode2", "dfs.namenode.rpc-address.my-ha.my-namenode1" = "nn1-host:rpc_port", "dfs.namenode.rpc-address.my-ha.my-namenode2" = "nn2-host:rpc_port", "dfs.client.failover.proxy.provider" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider" ) ``` You can add the configurations of the HDFS cluster to the **hdfs-site.xml** file. This way, you only need to specify the file path and authentication information when you use brokers to load data from the HDFS cluster. #### 9. How do I configure Hadoop ViewFS Federation?[​](#9-how-do-i-configure-hadoop-viewfs-federation "Direct link to 9. How do I configure Hadoop ViewFS Federation?") Copy the ViewFs-related configuration files `core-site.xml` and `hdfs-site.xml` to the **broker/conf** directory. If you have a custom file system, you also need to copy the file system-related **.jar** files to the **broker/lib** directory. #### 10. When I access an HDFS cluster that requires Kerberos authentication, what do I do if the "Can't get Kerberos realm" error occurs?[​](#10-when-i-access-an-hdfs-cluster-that-requires-kerberos-authentication-what-do-i-do-if-the-cant-get-kerberos-realm-error-occurs "Direct link to 10. When I access an HDFS cluster that requires Kerberos authentication, what do I do if the \"Can't get Kerberos realm\" error occurs?") Check that the **/etc/krb5.conf** file is configured on all hosts on which brokers are deployed. If the error persists, add `-Djava.security.krb5.conf:/etc/krb5.conf` to the end of the `JAVA_OPTS` variable in the broker startup script. #### 11. Will Broker Load fail if the source Hive table is empty?[​](#11-will-broker-load-fail-if-the-source-hive-table-is-empty "Direct link to 11. Will Broker Load fail if the source Hive table is empty?") By default, an empty transaction returns error "all partitions have no load data". You can set the FE configuration `empty_load_as_error` to `false` to allow empty transactions to return success. --- ### DataX #### Why datax writer does not have writemode parameter? Why must the tables be created in the update mode?[​](#why-datax-writer-does-not-have-writemode-parameter-why-must-the-tables-be-created-in-the-update-mode "Direct link to Why datax writer does not have writemode parameter? Why must the tables be created in the update mode?") Currently, only insert is supported. #### How to process keywords when I synchronize data by using DataX Writer?[​](#how-to-process-keywords-when-i-synchronize-data-by-using-datax-writer "Direct link to How to process keywords when I synchronize data by using DataX Writer?") Enclose a keyword in back quotes (\`\`). If you want to download the newest version of DataX, see [DataX Writer.](https://docs.starrocks.io/docs/integrations/loading_tools/DataX-starrocks-writer.md) --- ### Flink Connector #### flink-connector-jdbc\_2.11sink is 8 hours late in StarRocks[​](#flink-connector-jdbc_211sink-is-8-hours-late-in-starrocks "Direct link to flink-connector-jdbc_2.11sink is 8 hours late in StarRocks") **Issue description:** The time generated by localtimestap function is normal in Flink. But it became 8 hours late when sunk to StarRocks. Flink server and StarRocks server are located in the same timezone, namely Asia/Shanghai UTC/GMT+08:00. Flink version is 1.12. Driver: flink-connector-jdbc\_2.11. Can I ask how to resolve this issue? **Solution:** Please try configure the time parameter 'server-time-zone' = 'Asia/Shanghai' in Flink sink table. You may also add \&serverTimezone=Asia/Shanghai in jdbc url. An example is shown below: ```sql CREATE TABLE sk ( sid int, local_dtm TIMESTAMP, curr_dtm TIMESTAMP ) WITH ( 'connector' = 'jdbc', 'url' = 'jdbc:mysql://192.168.110.66:9030/sys_device?characterEncoding=utf-8&serverTimezone=Asia/Shanghai', 'table-name' = 'sink', 'driver' = 'com.mysql.jdbc.Driver', 'username' = 'sr', 'password' = 'sr123', 'server-time-zone' = 'Asia/Shanghai' ); ``` #### In flink import, only the kafka clusters deployed in StarRocks clusters can be imported[​](#in-flink-import-only-the-kafka-clusters-deployed-in-starrocks-clusters-can-be-imported "Direct link to In flink import, only the kafka clusters deployed in StarRocks clusters can be imported") **Issue description:** ```sql failed to query wartermark offset, err: Local: Bad message format ``` **Solution:** Kafka communication needs the hostname. Users need to configure the host name resolution /etc/hosts in StarRocks cluster nodes. #### Can StarRocks export 'create table statements' in batches?[​](#can-starrocks-export-create-table-statements-in-batches "Direct link to Can StarRocks export 'create table statements' in batches?") **Solution:** You can use StarRocks Tools to export the statements. #### Memory requested by BE is not released back to to the operation system[​](#memory-requested-by-be-is-not-released-back-to-to-the-operation-system "Direct link to Memory requested by BE is not released back to to the operation system") This is a normal phenomenon, as large blocks of memory allocated to the database from the operating system are reserved during allocation and deferred during release in order to reuse the memory and make memory allocation more convenient. It is recommended that users validate the test environment by monitoring memory usage over a longer period of time to see if the memory can be released. #### Flink connector does not work after being downloaded[​](#flink-connector-does-not-work-after-being-downloaded "Direct link to Flink connector does not work after being downloaded") **Issue description:** This package needs to be obtained through Aliyun mirror address. **Solution:** Please make sure that the mirror part of `/etc/maven/settings.xml` is all configured to be obtained through Aliyun mirror address. If it is, change it to the following: aliyunmaven centralaliyun public repohttps: //maven.aliyun.com/repository/public #### The meaning of parameter sink.buffer-flush.interval-ms in Flink-connector-StarRocks[​](#the-meaning-of-parameter-sinkbuffer-flushinterval-ms-in-flink-connector-starrocks "Direct link to The meaning of parameter sink.buffer-flush.interval-ms in Flink-connector-StarRocks") **Issue description:** ```plain +----------------------+--------------------------------------------------------------+ | Option | Required | Default | Type | Description | +-------------------------------------------------------------------------------------+ | sink.buffer-flush. | NO | 300000 | String | the flushing time interval, | | interval-ms | | | | range: [1000ms, 3600000ms] | +----------------------+--------------------------------------------------------------+ ``` If this parameter is set as 15s with checkpoint interval being equal to 5 mins, does this value still take effect? **Solution:** Whichever of the three thresholds is reached first, that one will take effect first. This is not affected by the checkpoint interval value which only works for exactly once. Interval-ms is used by at\_least\_once. #### Why does Partial Updates with Flink Connector fail with “NULL value in non-nullable column”?[​](#why-does-partial-updates-with-flink-connector-fail-with-null-value-in-non-nullable-column "Direct link to Why does Partial Updates with Flink Connector fail with “NULL value in non-nullable column”?") Configure the following properties: ```sql sink.properties.partial_update=true sink.properties.columns= ``` #### How to handle the Flink import error with JSON data “The size of this batch exceed the max size \[104857600]”?[​](#how-to-handle-the-flink-import-error-with-json-data-the-size-of-this-batch-exceed-the-max-size-104857600 "Direct link to How to handle the Flink import error with JSON data “The size of this batch exceed the max size [104857600]”?") Reduce batch frequency, or set `sink.properties.ignore_json_size` to `true` (which may cause higher memory usage). #### How to handle if a bigint unsigned field in Flink CDC turned into a string and changed its values?[​](#how-to-handle-if-a-bigint-unsigned-field-in-flink-cdc-turned-into-a-string-and-changed-its-values "Direct link to How to handle if a bigint unsigned field in Flink CDC turned into a string and changed its values?") Add the following configuration: ```sql 'debezium.bigint.unsigned.handling.mode' = 'precise' ``` #### Why do I get “None of hosts in load\_url could be connected” when using Flink connector to import data?[​](#why-do-i-get-none-of-hosts-in-load_url-could-be-connected-when-using-flink-connector-to-import-data "Direct link to Why do I get “None of hosts in load_url could be connected” when using Flink connector to import data?") The `load_url` is unreachable or experiencing timeout. Increase the value of the property `sink.connect.timeout-ms` (Range: \[100, 60000]). --- ### Insert Into #### When performing data insert, each insert in SQL takes up 50 to 100ms. Is there any way to increase efficiency?[​](#when-performing-data-insert-each-insert-in-sql-takes-up-50-to-100ms-is-there-any-way-to-increase-efficiency "Direct link to When performing data insert, each insert in SQL takes up 50 to 100ms. Is there any way to increase efficiency?") It is not recommended to insert data piece by piece to OLAP. It is usually inserted in batches. Both methods take up the same amount of time. #### 'Insert into select' task reports error: index channel has intolerable failure[​](#insert-into-select-task-reports-error-index-channel-has-intolerable-failure "Direct link to 'Insert into select' task reports error: index channel has intolerable failure") You can solve this problem by changing the timeout duration for the Stream Load RPC. Change the following item in **be.conf** and restart the machines to allow the change to take effect: `streaming_load_rpc_max_alive_time_sec`: The RPC timeout for Stream Load. Unit: Seconds. Default: `1200`. Or you can set the INSERT timeout using the following variable: `insert_timeout`: The timeout duration for INSERT statements. Its unit is seconds, and the default value is `14400`. #### The error "execute timeout" occurs when I run the INSERT INTO SELECT command to load a large volume of data[​](#the-error-execute-timeout-occurs-when-i-run-the-insert-into-select-command-to-load-a-large-volume-of-data "Direct link to The error \"execute timeout\" occurs when I run the INSERT INTO SELECT command to load a large volume of data") By default, the INSERT timeout duration is 14400s. You can set the variable `insert_timeout` to extend this duration. The unit is second. #### Why does INSERT INTO SELECT return “Reach limit of connections”?[​](#why-does-insert-into-select-return-reach-limit-of-connections "Direct link to Why does INSERT INTO SELECT return “Reach limit of connections”?") It is because the user connection limit is reached. Increase the value of the user property `max_user_connections`. --- ### Data loading common questions #### 1. What do I do if the "close index channel failed" or "too many tablet versions" error occurs?[​](#1-what-do-i-do-if-the-close-index-channel-failed-or-too-many-tablet-versions-error-occurs "Direct link to 1. What do I do if the \"close index channel failed\" or \"too many tablet versions\" error occurs?") You were running load jobs too frequently, and the data was not compacted in a timely manner. As a result, the number of data versions generated during loading exceeds the maximum number (which defaults to 1000) of data versions that are allowed. Use one of the following methods to resolve this issue: * Increase the amount of data loaded in each individual job, thereby reducing loading frequency. * Modify the some configuration items in the BE configuration file **be.conf** of each BE to accelerate compactions: * For Duplicate Key tables, Aggregate tables, and Unique Key tables, you can appropriately increase the values of `cumulative_compaction_num_threads_per_disk`, `base_compaction_num_threads_per_disk`, and `cumulative_compaction_check_interval_seconds`. Example: ```plain cumulative_compaction_num_threads_per_disk = 4 base_compaction_num_threads_per_disk = 2 cumulative_compaction_check_interval_seconds = 2 ``` * For Primary Key tables, you can appropriately increase the value of `update_compaction_num_threads_per_disk` and decrease the value of `update_compaction_per_tablet_min_interval_seconds`. After you modify the settings of the preceding configuration items, you must observe the memory and I/O to ensure that they are normal. #### 2. What do I do if the "Label Already Exists" error occurs?[​](#2-what-do-i-do-if-the-label-already-exists-error-occurs "Direct link to 2. What do I do if the \"Label Already Exists\" error occurs?") This error occurs because the load job has the same label as another load job, which has been successfully run or is being run, within the same StarRocks database. Stream Load jobs are submitted according to HTTP. In general, request retry logic is embedded in HTTP clients of all programmatic languages. When the StarRocks cluster receives a load job request from an HTTP client, it immediately starts to process the request, but it does not return the job result to the HTTP client in a timely manner. As a result, the HTTP client sends the same load job request again. However, the StarRocks cluster is already processing the first request and therefore returns the `Label Already Exists` error for the second request. Do as follows to check that load jobs submitted by using different loading methods do not have the same label and are not repeatedly submitted: * View the FE log and check whether the label of the failed load job is recorded twice. If the label is recorded twice, the client has submitted the load job request twice. > **NOTE** > > The StarRocks cluster does not distinguish between the labels of load jobs based on loading methods. Therefore, load jobs submitted by using different loading methods may have the same label. * Run SHOW LOAD WHERE LABEL = "xxx" to check for load jobs that have the same label and are in the **FINISHED** state. > **NOTE** > > `xxx` is the label that you want to check. Before you submit a load job, we recommend that you calculate the approximate amount of time required to load the data and then adjust the client-side request timeout period accordingly. This way, you can prevent the client from submitting the load job request multiple times. #### 3. What do I do if the "ETL\_QUALITY\_UNSATISFIED; msg:quality not good enough to cancel" error occurs?[​](#3-what-do-i-do-if-the-etl_quality_unsatisfied-msg-not-good-enough-to-cancel-error-occurs "Direct link to 3-what-do-i-do-if-the-etl_quality_unsatisfied-msg-not-good-enough-to-cancel-error-occurs") Execute [SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md), and use the error URL in the returned execution result to view the error details. Common data quality errors are as follows: * "convert csv string to INT failed." Strings from a source column failed to be transformed into the data type of the matching destination column. For example, `abc` failed to be transformed into a numeric value. * "the length of input is too long than schema." Values from a source column are in lengths that are not supported by the matching destination column. For example, the source column values of CHAR data type exceed the destination column's maximum length specified at table creation, or the source column values of INT data type exceed 4 bytes. * "actual column number is less than schema column number." After a source row is parsed based on the specified column separator, the number of columns obtained is smaller than the number of columns in the destination table. A possible reason is that the column separator specified in the load command or statement differs from the column separator that is actually used in that row. * "actual column number is more than schema column number." After a source row is parsed based on the specified column separator, the number of columns obtained is greater than the number of columns in the destination table. A possible reason is that the column separator specified in the load command or statement differs from the column separator that is actually used in that row. * "the frac part length longer than schema scale." The decimal parts of values from a DECIMAL-type source column exceed the specified length. * "the int part length longer than schema precision." The integer parts of values from a DECIMAL-type source column exceed the specified length. * "there is no corresponding partition for this key." The value in the partition column for a source row is not within the partition range. #### 4. What do I do if RPC times out?[​](#4-what-do-i-do-if-rpc-times-out "Direct link to 4. What do I do if RPC times out?") Check the setting of the `write_buffer_size` configuration item in the BE configuration file **be.conf** of each BE. This configuration item is used to control the maximum size per memory block on the BE. The default maximum size is 100 MB. If the maximum size is exceedingly large, Remote Procedure Call (RPC) may time out. To resolve this issue, adjust the settings of the `write_buffer_size` and `tablet_writer_rpc_timeout_sec` configuration items in the BE configuration file. For more information, see [BE configurations](https://docs.starrocks.io/docs/loading/loading_introduction/loading_considerations.md#be-configurations). #### 5. What do I do if the "Value count does not match column count" error occurs?[​](#5-what-do-i-do-if-the-value-count-does-not-match-column-count-error-occurs "Direct link to 5. What do I do if the \"Value count does not match column count\" error occurs?") After my load job failed, I used the error URL returned in the job result to retrieve the error details and found the "Value count does not match column count" error, which indicates a mismatch between the number of columns in the source data file and the number of columns in the destination StarRocks table: ```java Error: Value count does not match column count. Expect 3, but got 1. Row: 2023-01-01T18:29:00Z,cpu0,80.99 Error: Value count does not match column count. Expect 3, but got 1. Row: 2023-01-01T18:29:10Z,cpu1,75.23 Error: Value count does not match column count. Expect 3, but got 1. Row: 2023-01-01T18:29:20Z,cpu2,59.44 ``` The reason for this issue is as follows: The column separator specified in the load command or statement differs from the column separator that is actually used in the source data file. In the preceding example, the CSV-formatted data file consists of three columns, which are separated with commas (`,`). However, `\t` is specified as the column separator in the load command or statement. As a result, the three columns from the source data file are incorrectly parsed into one column. Specify commas (`,`) as the column separator in the load command or statement. Then, submit the load job again. #### 6. What do I do if the "current running txns on db XXX is 100, larger than limit 100" error occurs?[​](#6-what-do-i-do-if-the-current-running-txns-on-db-xxx-is-100-larger-than-limit-100-error-occurs "Direct link to 6. What do I do if the \"current running txns on db XXX is 100, larger than limit 100\" error occurs?") Increase the value of the FE configuration `max_running_txn_num_per_db`. #### 7. Why do I get a curl ERRORURL saying `be/storage/error_log` does not exist during data import?[​](#7-why-do-i-get-a-curl-errorurl-saying-bestorageerror_log-does-not-exist-during-data-import "Direct link to 7-why-do-i-get-a-curl-errorurl-saying-bestorageerror_log-does-not-exist-during-data-import") BE error logs are kept for 48 hours by default and are cleaned up afterward. You can adjust the retention time using `load_error_log_reserve_hours`. #### 8. How do I troubleshoot the error “Tablet is in error state … prepare\_segment\_writer meet invalid rssid” during import?[​](#8-how-do-i-troubleshoot-the-error-tablet-is-in-error-state--prepare_segment_writer-meet-invalid-rssid-during-import "Direct link to 8. How do I troubleshoot the error “Tablet is in error state … prepare_segment_writer meet invalid rssid” during import?") This issue is usually caused by version lag. Compare tablet versions at the partition level to check whether publish is stuck. Use the following SQL to compare versions: ```sql SELECT * FROM information_schema.be_tablets; SELECT * FROM information_schema.partitions_meta; ``` If only a few tablets are inconsistent, mark the lagging replicas as bad so they can be cloned from healthy ones. If it's caused by an ongoing large table update or schema change, locate the affected partition based on the error and consider deleting and reloading it. If the issue persists, try restarting FE and the problematic BE; if still ineffective, restart all BEs. #### 9. Why does DELETE fail with “failed to execute delete, transaction id xxx, timeout(ms) 30000”?[​](#9-why-does-delete-fail-with-failed-to-execute-delete-transaction-id-xxx-timeoutms-30000 "Direct link to 9. Why does DELETE fail with “failed to execute delete, transaction id xxx, timeout(ms) 30000”?") Increase the value of the FE configuration `load_straggler_wait_second` to 600 (Default: 300). #### 10. How to handle the error “StarRocks planner use long time 3000 ms …”?[​](#10-how-to-handle-the-error-starrocks-planner-use-long-time-3000-ms- "Direct link to 10. How to handle the error “StarRocks planner use long time 3000 ms …”?") The SQL may be too complex. Increase the value of the session variable `new_planner_optimize_timeout`. #### 11. How to fix the error “Primary-key index exceeds the limit.”?[​](#11-how-to-fix-the-error-primary-key-index-exceeds-the-limit "Direct link to 11. How to fix the error “Primary-key index exceeds the limit.”?") It is because that the Primary Key index exceeded memory limits. You can enable persistent index by setting the table property `enable_persistent_index` to `true`. #### 12. How to resolve “current running txns on db XXX is 100, larger than limit 100”?[​](#12-how-to-resolve-current-running-txns-on-db-xxx-is-100-larger-than-limit-100 "Direct link to 12. How to resolve “current running txns on db XXX is 100, larger than limit 100”?") Increase the value of the FE configuration `max_running_txn_num_per_db`. --- ### Routine Load #### How can I improve loading performance?[​](#how-can-i-improve-loading-performance "Direct link to How can I improve loading performance?") **Method 1: Increase the actual load task parallelism** by splitting a load job into as many parallel load tasks as possible. > **NOTICE** > > This method may consume more CPU resources and cause too many tablet versions. The actual load task parallelism is determined by the following formula composed of several parameters, with an upper limit of the number of BE nodes alive or the number of partitions to be consumed. ```plaintext min(alive_be_number, partition_number, desired_concurrent_number, max_routine_load_task_concurrent_num) ``` Parameter description: * `alive_be_number`: the number of BE nodes alive. * `partition_number`: the number of partitions to be consumed. * `desired_concurrent_number`: the desired load task parallelism for a Routine Load job. The default value is `3`. You can set a higher value for this parameter to increase the actual load task parallelism. * If you have not created a Routine Load job, you need to set this parameter when using [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md) to create a Routine Load job. * If you have already created a Routine Load job, you need to use [ALTER ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/ALTER_ROUTINE_LOAD.md) to modify this parameter. * `max_routine_load_task_concurrent_num`: the default maximum task parallelism for a Routine Load job. The default value is `5`. This parameter is a an FE dynamic parameter. For more information and the configuration method, see [Parameter configuration](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#loading-and-unloading). Therefore, when the number of partitions to be consumed and the number of BE nodes alive are greater than the other two parameters, you can increase the values of `desired_concurrent_number` and `max_routine_load_task_concurrent_num` parameters to increase the actual load task parallelism. For example, the number of partitions to be consumed is `7`, the number of live BE nodes is `5`, and `max_routine_load_task_concurrent_num` is the default value `5`. At this time, if you need to increase the load task parallelism to the upper limit, you need to set `desired_concurrent_number` to `5` (the default value is `3`). Then, the actual task parallelism `min(5,7,5,5)` is computed to be `5`. For more parameter descriptions, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). **Method 2: Increase the amount of data consumed by a Routine Load task from one or more partitions.** > **NOTICE** > > This method may cause delay in data loading. The upper limit of the number of messages that a Routine Load task can consume is determined by either the parameter `max_routine_load_batch_size` which means the maximum number of messages that a load task can consume or the parameter `routine_load_task_consume_second` which means the maximum duration of message consumption. Once an load task consumes enough data that meets either requirement, the consumption is complete. These two parameters are FE dynamic parameters. For more information and the configuration method, see [Parameter configuration](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#loading-and-unloading). You can analyze which parameter determines the upper limit of the amount of data consumed by a load task by viewing the log in **be/log/be.INFO**. By increasing that parameter, you can increase the amount of data consumed by a load task. ```plaintext I0325 20:27:50.410579 15259 data_consumer_group.cpp:131] consumer group done: 41448fb1a0ca59ad-30e34dabfa7e47a0. consume time(ms)=3261, received rows=179190, received bytes=9855450, eos: 1, left_time: -261, left_bytes: 514432550, blocking get time(us): 3065086, blocking put time(us): 24855 ``` Normally, the field `left_bytes` in the log is greater than or equal to `0`, indicating that the amount of data consumed by a load task has not exceeded `max_routine_load_batch_size` within `routine_load_task_consume_second`. This means that a batch of scheduled load tasks can consume all data from Kafka without delay in consumption. In this scenario, you can set a larger value for `routine_load_task_consume_second` to increase the amount of data consumed by a load task from one or more partitions. If the field `left_bytes` is less than `0`, it means that the amount of data consumed by a load task has reached `max_routine_load_batch_size` within `routine_load_task_consume_second`. Every time data from Kafka fills the batch of scheduled load tasks. Therefore, it is highly likely that there is remaining data that has not been consumed in Kafka, causing delay in consumption. In this case, you can set a larger value for `max_routine_load_batch_size`. #### What do I do if the result of SHOW ROUTINE LOAD shows that the load job is in the `PAUSED` state?[​](#what-do-i-do-if-the-result-of-show-routine-load-shows-that-the-load-job-is-in-the-paused-state "Direct link to what-do-i-do-if-the-result-of-show-routine-load-shows-that-the-load-job-is-in-the-paused-state") * Check the field `ReasonOfStateChanged` and it reports the error message `Broker: Offset out of range`. **Cause analysis:** The consumer offset of the load job does not exist in the Kafka partition. **Solution:** You can execute [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) and check the latest consumer offset of the load job in the parameter `Progress`. Then, you can verify if the corresponding message exists in the Kafka partition. If it does not exist, it may be because * The consumer offset specified when the load job is created is an offset in the future. * The message at the specified consumer offset in the Kafka partition has been removed before being consumed by the load job. It is recommended to set a reasonable Kafka log cleaning policy and parameters, such as `log.retention.hours and log.retention.bytes`, based on the loading speed. * Check the field `ReasonOfStateChanged` and it doesn't report the error message `Broker: Offset out of range`. **Cause analysis:** The number of error rows in the load task exceeds the threshold `max_error_number`. **Solution:** You can troubleshoot and fix the issue by using error messages in the fields `ReasonOfStateChanged` and `ErrorLogUrls`. * If it is caused by incorrect data format in the data source, you need to check the data format and fix the issue. After successfully fixing the issue, you can use [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) to resume the paused load job. * If it is because that StarRocks cannot parse the data format in the data source, you need to adjust the threshold `max_error_number`. You can first execute [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) to view the value of `max_error_number`, and then use [ALTER ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/ALTER_ROUTINE_LOAD.md) to increase the threshold. After modifying the threshold, you can use [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) to resume the paused load job. #### What do I do if the result of SHOW ROUTINE LOAD shows that the load job is in the `CANCELLED` state?[​](#what-do-i-do-if-the-result-of-show-routine-load-shows-that-the-load-job-is-in-the-cancelled-state "Direct link to what-do-i-do-if-the-result-of-show-routine-load-shows-that-the-load-job-is-in-the-cancelled-state") **Cause analysis:** The load job encountered an exception during loading, such as the table deleted. **Solution:** When troubleshooting and fixing the issue, you can refer to the error messages in the fields `ReasonOfStateChanged` and `ErrorLogUrls`. However, after fixing the issue, you cannot resume the cancelled load job. #### Can Routine Load guarantee consistency semantics when consuming from Kafka and writing to StarRocks?[​](#can-routine-load-guarantee-consistency-semantics-when-consuming-from-kafka-and-writing-to-starrocks "Direct link to Can Routine Load guarantee consistency semantics when consuming from Kafka and writing to StarRocks?") Routine Load guarantees exactly-once semantics. Each load task is a individual transaction. If an error occurs during the execution of the transaction, the transaction is aborted, and the FE does not update the consumption progress of the relevant partitions of the load tasks. When the FE schedules the load tasks from the task queue next time, the load tasks send the consumption request from the last saved consumption position of the partitions, thus ensuring exactly-once semantics. #### What do I do if Routine Load returns an SSL Authentication error?[​](#what-do-i-do-if-routine-load-returns-an-ssl-authentication-error "Direct link to What do I do if Routine Load returns an SSL Authentication error?") **Error Message:** `routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 273ms in state SSL_HANDSHAKE)` **Cause analysis:** The domain in the certificate is different with that of the Kafka Broker. See [more details](https://github.com/confluentinc/librdkafka/issues/4349). **Solution:** Add the property `"property.ssl.endpoint.identification.algorithm"= "none"` to the Routine Load job. #### Why does Routine Load report “JSON data is an array.strip\_outer\_array must be set true”?[​](#why-does-routine-load-report-json-data-is-an-arraystrip_outer_array-must-be-set-true "Direct link to Why does Routine Load report “JSON data is an array.strip_outer_array must be set true”?") Your input data is a JSON array `([{},{}])`. Set the property `strip_outer_array` to `true` to expand it. #### Why do I get “There are more than 100 routine load jobs running” when creating a Routine Load job?[​](#why-do-i-get-there-are-more-than-100-routine-load-jobs-running-when-creating-a-routine-load-job "Direct link to Why do I get “There are more than 100 routine load jobs running” when creating a Routine Load job?") Increase the value of the FE configuration `max_routine_load_job_num`. #### Why does Routine Load job creation fails with “failed to get partition meta” even after configuring SASL?[​](#why-does-routine-load-job-creation-fails-with-failed-to-get-partition-meta-even-after-configuring-sasl "Direct link to Why does Routine Load job creation fails with “failed to get partition meta” even after configuring SASL?") The actual cause can be incorrect SASL configuration. #### How should I handle Routine Load error “Create replicas failed …”?[​](#how-should-i-handle-routine-load-error-create-replicas-failed- "Direct link to How should I handle Routine Load error “Create replicas failed …”?") Adjust the following FE configurations: ```sql admin set frontend config ("tablet_create_timeout_second"="5"); admin set frontend config ("max_create_table_timeout_second"="600"); ``` You set also set them in the configuration file to persist the modification. #### Why does Routine Load report “Bad message format” when consuming Kafka?[​](#why-does-routine-load-report-bad-message-format-when-consuming-kafka "Direct link to Why does Routine Load report “Bad message format” when consuming Kafka?") Kafka uses hostname for communication. Add hostname resolution for Kafka nodes in `/etc/hosts` on all server that host the StarRocks nodes. #### What causes Routine Load to fail with error "failed to send task: failed to submit task. error code: TOO MANY TASKS"?[​](#what-causes-routine-load-to-fail-with-error-failed-to-send-task-failed-to-submit-task-error-code-too-many-tasks "Direct link to What causes Routine Load to fail with error \"failed to send task: failed to submit task. error code: TOO MANY TASKS\"?") It is because the total Routine Load concurrency exceeds cluster capability (which equals to `routine_load_thread_pool_size × number of active BEs`). Solutions: * Reduce loading QPS (recommended cluster QPS < 10). Calculate the cluster QPS based on `cluster routine_load_task_num / routine_load_task_consume_second`. * Increase per-task batch size (> 1 GB) by adjusting `max_routine_load_batch_size` and `routine_load_task_timeout_second`. * Ensure `routine_load_thread_pool_size` is less than half of BE CPU cores. A job’s concurrency is determined by the minimum of the following values: * `kafka_partition_num` * `desired_concurrent_number` * `alive_be_num` * `max_routine_load_task_concurrent_num` You may start adjusting the concurrency by reducing `max_routine_load_task_concurrent_num`. --- ### Stream Load #### 1. Does Stream Load support identifying column names held in the first few rows of a CSV-formatted file, or skipping the first few rows during data reading?[​](#1-does-stream-load-support-identifying-column-names-held-in-the-first-few-rows-of-a-csv-formatted-file-or-skipping-the-first-few-rows-during-data-reading "Direct link to 1. Does Stream Load support identifying column names held in the first few rows of a CSV-formatted file, or skipping the first few rows during data reading?") Stream Load does not support identifying column names held in the first few rows of a CSV-formatted file. Stream Load considers the first few rows to be normal data like the other rows. In v2.5 and earlier, Stream Load does not support skipping the first few rows of a CSV file during data reading. If the first few rows of the CSV file you want to load hold column names, take one of the following actions: * Modify the settings of the tool that you use to export the data. Then, re-export the data as a CSV file that does not hold column names in the first few rows. * Use commands such as `sed -i '1d' filename` to delete the first few rows of the CSV file. * In the load command or statement, use `-H "where: != ''"` to filter out the first few rows of the CSV file. `` is any of the column names held in the first few rows. Note that StarRocks first transforms and then filters the source data. Therefore, if the column names in the first few rows fail to be transformed into their matching destination data types, `NULL` values are returned for them. This means the destination StarRocks table cannot contain columns that are set to `NOT NULL`. * In the load command or statement, add `-H "max_filter_ratio:0.01"` to set a maximum error tolerance that is 1% or lower but can tolerate a few error rows, thereby allowing StarRocks to ignore the data transformation failures in the first few rows. In this case, the Stream Load job can still succeed even if `ErrorURL` is returned to indicate error rows. Do not set `max_filter_ratio` to a large value. If you set `max_filter_ratio` to a large value, some important data quality issues may be missed. From v3.0 onwards, Stream Load supports the `skip_header` parameter, which specifies whether to skip the first few rows of a CSV file. For more information,see [CSV parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#csv-parameters). #### 2. The data to be loaded into the partition column is not of standard DATE or INT type. For example, the data is in a format like 202106.00. How do I transform the data if I load it by using Stream Load?[​](#2-the-data-to-be-loaded-into-the-partition-column-is-not-of-standard-date-or-int-type-for-example-the-data-is-in-a-format-like-20210600-how-do-i-transform-the-data-if-i-load-it-by-using-stream-load "Direct link to 2. The data to be loaded into the partition column is not of standard DATE or INT type. For example, the data is in a format like 202106.00. How do I transform the data if I load it by using Stream Load?") StarRocks supports transforming data at loading. For more information, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md). Suppose that you want to load a CSV-formatted file named `TEST` and the file consists of four columns, `NO`, `DATE`, `VERSION`, and `PRICE`, among which the data from the `DATE` column is in a non-standard format such as 202106.00. If you want to use `DATE` as the partition column in StarRocks, you need to first create a StarRocks table, for example, one that consists of the following four columns: `NO`, `VERSION`, `PRICE`, and `DATE`. Then, you need to specify the data type of the `DATE` column of the StarRocks table as DATE, DATETIME, or INT. Finally, when you create a Stream Load job, you need to specify the following setting in the load command or statement to transform data from the source `DATE` column's data type to the destination column's data type: ```plain -H "columns: NO,DATE_1, VERSION, PRICE, DATE=LEFT(DATE_1,6)" ``` In the preceding example, `DATE_1` can be considered to be a temporarily named column mapping the destination `DATE` column, and the final results loaded into the destination `DATE` column are computed by the `left()` function. Note that you must first list the temporary names of the source columns and then use functions to transform data. The functions supported are scalar functions, including non-aggregate functions and window functions. #### 3. What do I do if my Stream Load job reports the "body exceed max size: 10737418240, limit: 10737418240" error?[​](#3-what-do-i-do-if-my-stream-load-job-reports-the-body-exceed-max-size-10737418240-limit-10737418240-error "Direct link to 3. What do I do if my Stream Load job reports the \"body exceed max size: 10737418240, limit: 10737418240\" error?") The size of the source data file exceeds 10 GB, which is the maximum file size supported by Stream Load. Take one of the following actions: * Use `seq -w 0 n` to split the source data file into smaller files. * Use `curl -XPOST http://be_host:http_port/api/update_config?streaming_load_max_mb=` to adjust the value of the [BE configuration item](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#configure-be-dynamic-parameters) `streaming_load_max_mb` to increase the maximum file size. #### 4. How can I load real NULL values instead of writing “null” into a string column via Stream Load?[​](#4-how-can-i-load-real-null-values-instead-of-writing-null-into-a-string-column-via-stream-load "Direct link to 4. How can I load real NULL values instead of writing “null” into a string column via Stream Load?") Use the replace function: ```bash -H "columns: pk, temp, pd_type=replace(temp,'NULL',NULL)" ``` #### 5. The field name “role” causes a Stream Load error. How should columns be named?[​](#5-the-field-name-role-causes-a-stream-load-error-how-should-columns-be-named "Direct link to 5. The field name “role” causes a Stream Load error. How should columns be named?") `role` is a reserved keyword. Use backticks to wrap the reserved keyword when you use it in names. Example: ```bash -H $'columns:k1,`role`' ``` --- ### Synchronize data from MySQL in real time #### What do I do if a Flink job reports an error?[​](#what-do-i-do-if-a-flink-job-reports-an-error "Direct link to What do I do if a Flink job reports an error?") A Flink job reports the error `Could not execute SQL statement. Reason:org.apache.flink.table.api.ValidationException: One or more required options are missing.` A possible reason is that the required configuration information is missing in multiple sets of rules, such as `[table-rule.1]` and `[table-rule.2]`, in the SMT configuration file **config\_prod.conf**. You can check whether each set of rules, such as `[table-rule.1]` and `[table-rule.2]` is configured with the required database, table, and Flink connector information. #### How can I make Flink automatically restart failed tasks?[​](#how-can-i-make-flink-automatically-restart-failed-tasks "Direct link to How can I make Flink automatically restart failed tasks?") Flink automatically restarts failed tasks through the [checkpointing mechanism](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpointing/) and [restart strategy](https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/state/task_failure_recovery/). For example, if you need to enable the checkpointing mechanism and use the default restart strategy, which is the fixed delay restart strategy, you can configure the following information in the configuration file **flink-conf.yaml**: ```bash execution.checkpointing.interval: 300000 state.backend: filesystem state.checkpoints.dir: file:///tmp/flink-checkpoints-directory ``` Parameter description: > **NOTE** > > For more detailed parameter descriptions in Flink documentation, see [Checkpointing](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpointing/). * `execution.checkpointing.interval`: the base time interval of checkpointing. Unit: millisecond. To enable the checkpointing mechanism, you need to set this parameter to a value greater than `0`. * `state.backend`: specifies the state backend to determine how the state is represented internally, and how and where it is persisted upon checkpointing. Common values are `filesystem` or `rocksdb`. After the checkpointing mechanism is enabled, the state is persisted upon checkpoints to prevent data loss and ensure data consistency after recovery. For more information on state, see [State Backends](https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/state_backends/). * `state.checkpoints.dir`: the directory to which checkpoints are written to. #### How can I manually stop a Flink job and later restore it to the state before stopping?[​](#how-can-i-manually-stop-a-flink-job-and-later-restore-it-to-the-state-before-stopping "Direct link to How can I manually stop a Flink job and later restore it to the state before stopping?") You can manually trigger a [savepoint](https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/savepoints/) when stopping a Flink job (a savepoint is a consistent image of the execution state of a streaming Flink job, and is created based on the checkpointing mechanism). Later, you can restore the Flink job from the specified savepoint. 1. Stop the Flink job with a savepoint. The following command automatically triggers a savepoint for the Flink job `jobId` and stops the Flink job. Additionally, you can specify a target file system directory to store the savepoint. ```bash bin/flink stop --type [native/canonical] --savepointPath [:targetDirectory] :jobId ``` Parameter description: * `jobId`: You can view the Flink job ID from the Flink WebUI or by running `flink list -running` on the command line. * `targetDirectory`: You can specify `state.savepoints.dir` as the default directory for storing savepoints in the Flink configuration file **flink-conf.yml**. When a savepoint is triggered, the savepoint is stored in this default directory and you do not need to specify a directory . ```bash state.savepoints.dir: [file:// or hdfs://]/home/user/savepoints_dir ``` 2. Resubmit the Flink job with the preceding savepoint specified. ```bash ./flink run -c com.starrocks.connector.flink.tools.ExecuteSQL -s savepoints_dir/savepoints-xxxxxxxx flink-connector-starrocks-xxxx.jar -f flink-create.all.sql ``` --- ### Datalake FAQ ### Data lake FAQ This topic describes some commonly asked questions (FAQ) about data lake and provides solutions to these issues. Some metrics mentioned in this topic can be obtained only from the profiles of the SQL queries. To obtain the profiles of SQL queries, you must specify `set enable_profile=true`. #### Slow HDFS DataNodes[​](#slow-hdfs-datanodes "Direct link to Slow HDFS DataNodes") ##### Issue description[​](#issue-description "Direct link to Issue description") When you access the data files stored in your HDFS cluster, you may find a huge difference between the values of the `__MAX_OF_FSIOTime` and `__MIN_OF_FSIOTime` metrics from the profiles of the SQL queries you run. This indicates that some DataNodes in the HDFS cluster are slow. The following example is a typical profile that indicates a slow HDFS DataNode issue: ```plaintext - InputStream: 0 - AppIOBytesRead: 22.72 GB - __MAX_OF_AppIOBytesRead: 187.99 MB - __MIN_OF_AppIOBytesRead: 64.00 KB - AppIOCounter: 964.862K (964862) - __MAX_OF_AppIOCounter: 7.795K (7795) - __MIN_OF_AppIOCounter: 1 - AppIOTime: 1s372ms - __MAX_OF_AppIOTime: 4s358ms - __MIN_OF_AppIOTime: 1.539ms - FSBytesRead: 15.40 GB - __MAX_OF_FSBytesRead: 127.41 MB - __MIN_OF_FSBytesRead: 64.00 KB - FSIOCounter: 1.637K (1637) - __MAX_OF_FSIOCounter: 12 - __MIN_OF_FSIOCounter: 1 - FSIOTime: 9s357ms - __MAX_OF_FSIOTime: 60s335ms - __MIN_OF_FSIOTime: 1.536ms ``` ##### Solution[​](#solution "Direct link to Solution") You can use one of the following solutions to resolve this issue: * **\[Recommended]** Enable the [data cache](https://docs.starrocks.io/docs/data_source/data_cache.md) feature, which eliminates the impact of slow HDFS DataNodes on queries by automatically caching the data from external storage systems to the BEs or CNs of your StarRocks cluster. * **\[Recommended]** Shorten the timeout duration between the HDFS client and DataNode. This solution is suitable when Data Cache cannot help resolve the slow HDFS DataNode issue. * Enable the [Hedged Read](https://hadoop.apache.org/docs/r2.8.3/hadoop-project-dist/hadoop-common/release/2.4.0/RELEASENOTES.2.4.0.html) feature. With this feature enabled, if a read from a block is slow, StarRocks starts up a new read, which runs in parallel to the original read, to read against a different block replica. Whenever one of the two reads returns, the other read is cancelled. **The Hedged Read feature can help accelerate reads, but it also significantly increases heap memory consumption on Java virtual machines (JVMs). Therefore, if your physical machines provide a small memory capacity, we recommend that you do not enable the Hedged Read feature.** ###### \[Recommended] Data Cache[​](#recommended-data-cache "Direct link to [Recommended] Data Cache") See [Data Cache](https://docs.starrocks.io/docs/data_source/data_cache.md). ###### \[Recommended] Shorten timeout duration between HDFS client and DataNode[​](#recommended-shorten-timeout-duration-between-hdfs-client-and-datanode "Direct link to [Recommended] Shorten timeout duration between HDFS client and DataNode") Configure the `dfs.client.socket-timeout` property in the `hdfs-site.xml` file to shorten the timeout duration between the HDFS client and DataNode. (The default timeout duration is 60s, which is a bit long.) As such, when StarRocks encounters a slow DataNode, the connection request from it can time out within a very short period of time and then be forwarded to another DataNode. The following example sets a 5-second timeout duration: ```xml dfs.client.socket-timeout 5000 ``` ###### Hedged Read[​](#hedged-read "Direct link to Hedged Read") Use the following parameters (supported from v3.0 onwards) in the BE or CN configuration file `be.conf` to enable and configure the Hedged Read feature in your HDFS cluster. | Parameter | Default value | Description | | --------------------------------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hdfs\_client\_enable\_hedged\_read | false | Specifies whether to enable the hedged read feature. | | hdfs\_client\_hedged\_read\_threadpool\_size | 128 | Specifies the size of the Hedged Read thread pool on your HDFS client. The thread pool size limits the number of threads to dedicate to the running of hedged reads in your HDFS client. This parameter is equivalent to the `dfs.client.hedged.read.threadpool.size` parameter in the `hdfs-site.xml` file of your HDFS cluster. | | hdfs\_client\_hedged\_read\_threshold\_millis | 2500 | Specifies the number of milliseconds to wait before starting up a hedged read. For example, you have set this parameter to `30`. In this situation, if a read from a block has not returned within 30 milliseconds, your HDFS client immediately starts up a hedged read against a different block replica. This parameter is equivalent to the `dfs.client.hedged.read.threshold.millis` parameter in the `hdfs-site.xml` file of your HDFS cluster. | If the value of any of the following metrics in your query profiles exceeds `0`, the Hedged Read feature is enabled. | Metric | Description | | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | TotalHedgedReadOps | The number of hedged reads that are started up. | | TotalHedgedReadOpsInCurThread | The number of times that StarRocks has to start up a hedged read in the current thread instead of in a new thread because the Hedged Read thread pool has reached its maximum size specified by the `hdfs_client_hedged_read_threadpool_size` parameter. | | TotalHedgedReadOpsWin | The number of times that a hedged read beats its original read. | #### How do I resolve the error “ERROR 1064 (HY000): Type mismatches on column \[is\_refund], JDBC result type is Integer, please set the type to one of tinyint,smallint,int,bigint” when querying a table in the Hive Catalog?[​](#how-do-i-resolve-the-error-error-1064-hy000-type-mismatches-on-column-is_refund-jdbc-result-type-is-integer-please-set-the-type-to-one-of-tinyintsmallintintbigint-when-querying-a-table-in-the-hive-catalog "Direct link to How do I resolve the error “ERROR 1064 (HY000): Type mismatches on column [is_refund], JDBC result type is Integer, please set the type to one of tinyint,smallint,int,bigint” when querying a table in the Hive Catalog?") This issue is caused by an incorrect JDBC connection configuration. Add the parameter `tinyInt1isBit=false` to your JDBC URI to prevent this issue: ```sql "jdbc_uri" = "jdbc:mysql://xxx:3306?database=yl_spmibill&tinyInt1isBit=false" ``` #### Why can’t I query the latest updated data in the Iceberg Catalog (even after refresh or catalog rebuild), and how should I troubleshoot this?[​](#why-cant-i-query-the-latest-updated-data-in-the-iceberg-catalog-even-after-refresh-or-catalog-rebuild-and-how-should-i-troubleshoot-this "Direct link to Why can’t I query the latest updated data in the Iceberg Catalog (even after refresh or catalog rebuild), and how should I troubleshoot this?") First check whether the issue is caused by Data Cache being enabled. Follow these steps to verify: 1. Compare the scanned data files between StarRocks and Spark: * In StarRocks: `select file_path, spec_id from db.table_name$files;` * In Spark: `select file_path, spec_id from db.table_name.files;` 2. If the results are consistent, continue troubleshooting by disabling Data Cache and querying again to see whether the issue persists. Root cause: To update the Iceberg table data is to overwrite old files, which corrupts Iceberg’s historical data. The correct behavior is to generate new file names when writing updates. StarRocks Data Cache uses the file name, file size, and modification time to determine whether cached data is valid. Since Iceberg does not overwrite files and the modification time is always 0, StarRocks incorrectly treats the files as unchanged and reads from cache, resulting in outdated query results. #### FE constantly crashes after queries against tables in external catalog integrated with JuiceFS. How can I solve this?[​](#fe-constantly-crashes-after-queries-against-tables-in-external-catalog-integrated-with-juicefs-how-can-i-solve-this "Direct link to FE constantly crashes after queries against tables in external catalog integrated with JuiceFS. How can I solve this?") To solve this, restart FE after adding the following configuration items to **fe.conf**: ```properties proc_profile_mem_enable=false proc_profile_cpu_enable=false ``` --- ### Deployment This topic provides answers to some frequently asked questions about deployment. #### How do I bind a fixed IP address with the `priority_networks` parameter in the `fe.conf` file?[​](#how-do-i-bind-a-fixed-ip-address-with-the-priority_networks-parameter-in-the-feconf-file "Direct link to how-do-i-bind-a-fixed-ip-address-with-the-priority_networks-parameter-in-the-feconf-file") ##### Problem description[​](#problem-description "Direct link to Problem description") For example, if you have two IP addresses: 192.168.108.23 and 192.168.108.43. You might provide IP addresses as follows: * If you specify the addresses as 192.168.108.23/24, StarRocks will recognize them as 192.168.108.43. * If you specify the addresses as 192.168.108.23/32, StarRocks will recognize them as 127.0.0.1. ##### Solution[​](#solution "Direct link to Solution") There are the following two ways to solve this problem: * Do not add "32" at the end of an IP address or change "32" to "28". * You can also upgrade to StarRocks 2.1 or later. #### Why does the error "StarRocks BE http service did not start correctly, exiting" occur when I start a backend (BE) after installation?[​](#why-does-the-error-starrocks-be-http-service-did-not-start-correctly-exiting-occur-when-i-start-a-backend-be-after-installation "Direct link to Why does the error \"StarRocks BE http service did not start correctly, exiting\" occur when I start a backend (BE) after installation?") When installing a BE, the system reports a startup error: StarRocks Be http service did not start correctly, exiting. This error occurs because the web services port of the BE is occupied. Try to modify the ports in the `be.conf` file and restart the BE. #### What do I do when the error occurs: ERROR 1064 (HY000): Could not initialize class com.starrocks.rpc.BackendServiceProxy?[​](#what-do-i-do-when-the-error-occurs-error-1064-hy000-could-not-initialize-class-comstarrocksrpcbackendserviceproxy "Direct link to What do I do when the error occurs: ERROR 1064 (HY000): Could not initialize class com.starrocks.rpc.BackendServiceProxy?") This error occurs when you run programs in Java Runtime Environment (JRE). To solve this problem, replace JRE with Java Development Kit (JDK). We recommend that you use Oracle's JDK 17 or later. #### Can FE and BE configuration items be modified and then take effect without restarting the cluster?[​](#can-fe-and-be-configuration-items-be-modified-and-then-take-effect-without-restarting-the-cluster "Direct link to Can FE and BE configuration items be modified and then take effect without restarting the cluster?") Yes. Perform the following steps to complete the modifications for an FE and a BE configuration item: * FE: You can complete the modification for an FE in one of the following ways: * SQL ```plaintext ADMIN SET FRONTEND CONFIG ("key" = "value"); ``` Example: ```plaintext ADMIN SET FRONTEND CONFIG ("enable_statistic_collect" = "false"); ``` * Shell ```plaintext curl --location-trusted -u username:password \ http://: ``` Example: ```plaintext curl --location-trusted -u : \ http://192.168.110.101:8030/api/_set_config?enable_statistic_collect=true ``` * BE: You can complete the modification for a BE in the following way: ```plaintext curl -XPOST -u username:password \ http://:/api/update_config?key=value ``` > Note: Make sure that the user has permission to log in remotely. If not, you can grant the permission to the user in the following way: ```plaintext CREATE USER 'test'@'%' IDENTIFIED BY '123456'; GRANT SELECT ON . TO 'test'@'%'; ``` #### Why does the error "Fe type:unknown ,is ready :false." occur when I start an FE during the cluster restart?[​](#why-does-the-error-fe-type-is-ready--occur-when-i-start-an-fe-during-the-cluster-restart "Direct link to why-does-the-error-fe-type-is-ready--occur-when-i-start-an-fe-during-the-cluster-restart") Check if the leader FE is running. If not, restart the FE nodes in your cluster one by one. #### Why does the error "failed to get service info err." occur when I deploy the cluster?[​](#why-does-the-error-failed-to-get-service-info-err-occur-when-i-deploy-the-cluster "Direct link to Why does the error \"failed to get service info err.\" occur when I deploy the cluster?") Check if OpenSSH Daemon (sshd) is enabled. If not, run the `/etc/init.d/sshd`` status` command to enable it. #### Why does the error "Fail to get master client from `cache. ``host= port=0 code=THRIFT_RPC_ERROR`" occur when I start a BE?[​](#why-does-the-error-fail-to-get-master-client-from-cache-host-port0-codethrift_rpc_error-occur-when-i-start-a-be "Direct link to why-does-the-error-fail-to-get-master-client-from-cache-host-port0-codethrift_rpc_error-occur-when-i-start-a-be") Run the `netstat -anp |grep port` command to check whether the ports in the `be.conf` file are occupied. If so, replace the occupied port with a free port and then restart the BE. #### Why does the FE node log on the diagnostics page of StarRocks Manager display "Search log failed." for a newly deployed FE node that is running properly?[​](#why-does-the-fe-node-log-on-the-diagnostics-page-of-starrocks-manager-display-search-log-failed-for-a-newly-deployed-fe-node-that-is-running-properly "Direct link to Why does the FE node log on the diagnostics page of StarRocks Manager display \"Search log failed.\" for a newly deployed FE node that is running properly?") By default, StarRocks Manager obtains the path configuration of the newly deployed FE within 30 seconds. This error occurs when the FE starts slowly or does not respond within 30 seconds due to other reasons. Check the log of Manager Web via the path: `/starrocks-manager-xxx/center/log/webcenter/log/web/``drms.INFO`(you can customize the path). Then find that whether the message "Failed to update FE configurations" display in the log. If so, restart the corresponding FE to obtain the new path configuration. #### Why does the error "exceeds max permissable delta:5000ms." occur when I start an FE?[​](#why-does-the-error-exceeds-max-permissable-delta5000ms-occur-when-i-start-an-fe "Direct link to Why does the error \"exceeds max permissable delta:5000ms.\" occur when I start an FE?") This error occurs when the time difference between two machines is more than 5s. To solve this problem, align the time of these two machines. #### How do I set the `storage_root_path` parameter if there are multiple disks in a BE for data storage?[​](#how-do-i-set-the-storage_root_path-parameter-if-there-are-multiple-disks-in-a-be-for-data-storage "Direct link to how-do-i-set-the-storage_root_path-parameter-if-there-are-multiple-disks-in-a-be-for-data-storage") Configure the `storage_root_path` parameter in the `be.conf` file and separate values of this parameter with `;`. For example: `storage_root_path=/the/path/to/storage1;/the/path/to/storage2;/the/path/to/storage3;` #### Why does the error "invalid cluster id: 209721925." occur after an FE is added to my cluster?[​](#why-does-the-error-invalid-cluster-id-209721925-occur-after-an-fe-is-added-to-my-cluster "Direct link to Why does the error \"invalid cluster id: 209721925.\" occur after an FE is added to my cluster?") If you do not add the `--helper` option for this FE when starting your cluster for the first time, the metadata between two machines is inconsistent, thus this error occurs. To solve this problem, you need to clear all metadata under the meta directory and then add an FE with the `--helper` option. #### Why Alive is `false` when an FE is running and prints log `transfer: follower`?[​](#why-alive-is-false-when-an-fe-is-running-and-prints-log-transfer-follower "Direct link to why-alive-is-false-when-an-fe-is-running-and-prints-log-transfer-follower") This issue occurs when more than half of memory of Java Virtual Machine (JVM) is used and no checkpoint is marked. In general, a checkpoint will be marked after the system accumulates 50,000 pieces of log. We recommend that you modify the JVM's parameters of each FE and restarting these FEs when they are not heavily loaded. #### Query error: “could not initialize class com.starrocks.rpc.BackendServiceProxy”. How do I resolve this?[​](#query-error-could-not-initialize-class-comstarrocksrpcbackendserviceproxy-how-do-i-resolve-this "Direct link to Query error: “could not initialize class com.starrocks.rpc.BackendServiceProxy”. How do I resolve this?") * Verify that the environment variable `$JAVA_HOME` points to the correct JDK path. * Ensure all nodes use the same JDK version. All nodes must use the identical JDK version. #### What are the MySQL version requirements for installing StarRocks?[​](#what-are-the-mysql-version-requirements-for-installing-starrocks "Direct link to What are the MySQL version requirements for installing StarRocks?") MySQL 5.7 or later is recommended to connect to StarRocks. #### If FE and BE are deployed on the same machine, how can I separate them?[​](#if-fe-and-be-are-deployed-on-the-same-machine-how-can-i-separate-them "Direct link to If FE and BE are deployed on the same machine, how can I separate them?") It is recommended to scale out a BE node first. After the cluster finishes balancing, you can then scale in the original BE node. #### What should I do when FE fails to start with the error “Replica exceeds max permissible delta:5000ms”?[​](#what-should-i-do-when-fe-fails-to-start-with-the-error-replica-exceeds-max-permissible-delta5000ms "Direct link to What should I do when FE fails to start with the error “Replica exceeds max permissible delta:5000ms”?") The clocks between FE nodes are not synchronized. The time difference between FE nodes must be less than 5 seconds. #### If I have five physical machines in a production environment, what is the recommended StarRocks deployment?[​](#if-i-have-five-physical-machines-in-a-production-environment-what-is-the-recommended-starrocks-deployment "Direct link to If I have five physical machines in a production environment, what is the recommended StarRocks deployment?") A recommended deployment is 3 FE nodes and 5 BE nodes. #### Should I use the root user to install StarRocks?[​](#should-i-use-the-root-user-to-install-starrocks "Direct link to Should I use the root user to install StarRocks?") It is not recommended to use the root user because it has excessive privileges. Create a dedicated user for installing StarRocks. #### Can I install the MySQL client on any machine?[​](#can-i-install-the-mysql-client-on-any-machine "Direct link to Can I install the MySQL client on any machine?") Yes. It is simply a client tool and does not need to run on the same machine as StarRocks. Make sure the MySQL client can access the cluster. #### Is there a limit to the number of tablets on a BE node? For example, for a server with 64 GB RAM and 16 cores, what is the reasonable range of tablet count?[​](#is-there-a-limit-to-the-number-of-tablets-on-a-be-node-for-example-for-a-server-with-64-gb-ram-and-16-cores-what-is-the-reasonable-range-of-tablet-count "Direct link to Is there a limit to the number of tablets on a BE node? For example, for a server with 64 GB RAM and 16 cores, what is the reasonable range of tablet count?") There is no strict limit for the tablet number. However, for the tablet size, it is recommended to keep each tablet around 1 GB. Proper partitioning and bucketing strategies will help improve query performance. Tablet size planning is important. #### When starting BE, I see the error “error while loading shared libraries: libjvm.so: cannot open shared object file: No such file or directory”. And after manually creating the directory `lib/starrocks_be`, I get a permission denied error. What should I do?[​](#when-starting-be-i-see-the-error-error-while-loading-shared-libraries-libjvmso-cannot-open-shared-object-file-no-such-file-or-directory-and-after-manually-creating-the-directory-libstarrocks_be-i-get-a-permission-denied-error-what-should-i-do "Direct link to when-starting-be-i-see-the-error-error-while-loading-shared-libraries-libjvmso-cannot-open-shared-object-file-no-such-file-or-directory-and-after-manually-creating-the-directory-libstarrocks_be-i-get-a-permission-denied-error-what-should-i-do") There is an issue with the JDK installation. Please reinstall and properly configure your JDK environment. #### Does StarRocks support running on AMD AVX2? Will mixing Intel and AMD servers cause problems?[​](#does-starrocks-support-running-on-amd-avx2-will-mixing-intel-and-amd-servers-cause-problems "Direct link to Does StarRocks support running on AMD AVX2? Will mixing Intel and AMD servers cause problems?") StarRocks can run on AMD. Mixing Intel and AMD servers is not recommended because hardware heterogeneity may cause issues. It is suggested to fully test StarRocks on AMD before migrating. --- ### query_dump interface This topic describes how to use the query\_dump interface to obtain the details of an SQL query and its related information. If you encounter any of the following issues when executing SQL queries with StarRocks, you can use query\_dump to obtain the SQL details and send the information to StarRocks technical support for troubleshooting: * `Unknown Error` is returned when you execute an SQL query or EXPLAIN. * An error message or exception is returned when you execute an SQL query. * Executing an SQL query is not as efficient as expected, or the execution plan can be optimized (for example, partitions can be pruned or Join order can be adjusted). #### Function overview[​](#function-overview "Direct link to Function overview") The query\_dump interface returns the information that FE relies on when executing the SQL, including: * Query statement * Table creation statement * Session variables * Number of BEs * Statistics information (Min, Max values in a column) * Exception information (abnormal stack) * Explain costs info To ensure data privacy, we desensitize the meta information such as database names, table names, and column names. We also use the desensitized metadata to rewrite the query statements. Meta information desensitization is enabled by default. If an exception occurs during the desensitization process, the original information is used. If desensitization needs to be bypassed, you can add "mock=false" in the HTTP URI. #### Syntax[​](#syntax "Direct link to Syntax") HTTP Post ```shell fe_host:fe_http_port/api/query_dump?db=${database}&mock=${value} post_data=${Query} ``` ```shell wget --user=${username} --password=${password} --post-file ${query_file} "http://${fe_host}:${fe_http_port}/api/query_dump?db=${database}&mock={value}" -O ${dump_file} ``` Parameter description: * query\_file: the file containing the query * dump\_file: the output file * db: the database where the SQL query is executed. The `db` parameter is optional if the query includes `use db`. Otherwise, it must be specified. * mock: whether to enable or disable desensitization #### Examples[​](#examples "Direct link to Examples") ##### Disable desensitization[​](#disable-desensitization "Direct link to Disable desensitization") Command: ```shell wget --user=root --password=123 --post-file query_file "http://127.0.0.1:8030/api/query_dump?db=tpch&mock=false" -O dump_file ``` Return data: Data is returned in JSON format. ```json { "statement": "select\n l_returnflag,\n l_linestatus,\n sum(l_quantity) as sum_qty,\n sum(l_extendedprice) as sum_base_price,\n sum(l_extendedprice * (1 - l_discount)) as sum_disc_price,\n sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge,\n avg(l_quantity) as avg_qty,\n avg(l_extendedprice) as avg_price,\n avg(l_discount) as avg_disc,\n count(*) as count_order\nfrom\n lineitem\nwhere\n l_shipdate <= date '1998-12-01'\ngroup by\n l_returnflag,\n l_linestatus\norder by\n l_returnflag,\n l_linestatus ;\n\n", "table_meta": { "tpch.lineitem": "CREATE TABLE `lineitem` (\n `L_ORDERKEY` int(11) NOT NULL COMMENT \"\",\n `L_PARTKEY` int(11) NOT NULL COMMENT \"\",\n `L_SUPPKEY` int(11) NOT NULL COMMENT \"\",\n `L_LINENUMBER` int(11) NOT NULL COMMENT \"\",\n `L_QUANTITY` double NOT NULL COMMENT \"\",\n `L_EXTENDEDPRICE` double NOT NULL COMMENT \"\",\n `L_DISCOUNT` double NOT NULL COMMENT \"\",\n `L_TAX` double NOT NULL COMMENT \"\",\n `L_RETURNFLAG` char(1) NOT NULL COMMENT \"\",\n `L_LINESTATUS` char(1) NOT NULL COMMENT \"\",\n `L_SHIPDATE` date NOT NULL COMMENT \"\",\n `L_COMMITDATE` date NOT NULL COMMENT \"\",\n `L_RECEIPTDATE` date NOT NULL COMMENT \"\",\n `L_SHIPINSTRUCT` char(25) NOT NULL COMMENT \"\",\n `L_SHIPMODE` char(10) NOT NULL COMMENT \"\",\n `L_COMMENT` varchar(44) NOT NULL COMMENT \"\",\n `PAD` char(1) NOT NULL COMMENT \"\"\n) ENGINE=OLAP \nDUPLICATE KEY(`L_ORDERKEY`)\nCOMMENT \"OLAP\"\nDISTRIBUTED BY HASH(`L_ORDERKEY`) BUCKETS 20 \nPROPERTIES (\n\"replication_num\" = \"1\",\n\"in_memory\" = \"false\",\n\"enable_persistent_index\" = \"true\",\n\"replicated_storage\" = \"true\",\n\"compression\" = \"LZ4\"\n);" }, "table_row_count": { "tpch.lineitem": { "lineitem": 3 } }, "column_statistics": { "tpch.lineitem": { "L_TAX": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE", "L_SHIPDATE": "[1.6094304E9, 1.6094304E9, 0.0, 4.0, 1.0] ESTIMATE", "L_EXTENDEDPRICE": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE", "L_DISCOUNT": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE", "L_RETURNFLAG": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE", "L_LINESTATUS": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE", "L_QUANTITY": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE" } }, "explain_info": "PLAN FRAGMENT 0(F02)\n Output Exprs:9: L_RETURNFLAG | 10: L_LINESTATUS | 20: sum | 21: sum | 22: sum | 23: sum | 24: avg | 25: avg | 26: avg | 27: count\n Input Partition: UNPARTITIONED\n RESULT SINK\n\n 6:MERGING-EXCHANGE\n distribution type: GATHER\n cardinality: 1\n column statistics: \n * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n\nPLAN FRAGMENT 1(F01)\n\n Input Partition: HASH_PARTITIONED: 9: L_RETURNFLAG, 10: L_LINESTATUS\n OutPut Partition: UNPARTITIONED\n OutPut Exchange Id: 06\n\n 5:SORT\n | order by: [9, VARCHAR, false] ASC, [10, VARCHAR, false] ASC\n | offset: 0\n | cardinality: 1\n | column statistics: \n | * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 4:AGGREGATE (merge finalize)\n | aggregate: sum[([20: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([21: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([22: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([23: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([24: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([25: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([26: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], count[([27: count, BIGINT, false]); args: ; result: BIGINT; args nullable: true; result nullable: false]\n | group by: [9: L_RETURNFLAG, VARCHAR, false], [10: L_LINESTATUS, VARCHAR, false]\n | cardinality: 1\n | column statistics: \n | * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 3:EXCHANGE\n distribution type: SHUFFLE\n partition exprs: [9: L_RETURNFLAG, VARCHAR, false], [10: L_LINESTATUS, VARCHAR, false]\n cardinality: 1\n\nPLAN FRAGMENT 2(F00)\n\n Input Partition: RANDOM\n OutPut Partition: HASH_PARTITIONED: 9: L_RETURNFLAG, 10: L_LINESTATUS\n OutPut Exchange Id: 03\n\n 2:AGGREGATE (update serialize)\n | STREAMING\n | aggregate: sum[([5: L_QUANTITY, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([6: L_EXTENDEDPRICE, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([18: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([19: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], avg[([5: L_QUANTITY, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([6: L_EXTENDEDPRICE, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([7: L_DISCOUNT, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], count[(*); args: ; result: BIGINT; args nullable: false; result nullable: false]\n | group by: [9: L_RETURNFLAG, VARCHAR, false], [10: L_LINESTATUS, VARCHAR, false]\n | cardinality: 1\n | column statistics: \n | * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 1:Project\n | output columns:\n | 5 <-> [5: L_QUANTITY, DOUBLE, false]\n | 6 <-> [6: L_EXTENDEDPRICE, DOUBLE, false]\n | 7 <-> [7: L_DISCOUNT, DOUBLE, false]\n | 9 <-> [9: L_RETURNFLAG, CHAR, false]\n | 10 <-> [10: L_LINESTATUS, CHAR, false]\n | 18 <-> [29: multiply, DOUBLE, false]\n | 19 <-> [29: multiply, DOUBLE, false] * 1.0 + [8: L_TAX, DOUBLE, false]\n | common expressions:\n | 28 <-> 1.0 - [7: L_DISCOUNT, DOUBLE, false]\n | 29 <-> [6: L_EXTENDEDPRICE, DOUBLE, false] * [28: subtract, DOUBLE, false]\n | cardinality: 1\n | column statistics: \n | * L_QUANTITY-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * L_EXTENDEDPRICE-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * L_DISCOUNT-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 0:OlapScanNode\n table: lineitem, rollup: lineitem\n preAggregation: on\n Predicates: [11: L_SHIPDATE, DATE, false] <= '1998-12-01'\n partitionsRatio=1/1, tabletsRatio=20/20\n tabletList=45030,45032,45034,45036,45038,45040,45042,45044,45046,45048 ...\n actualRows=3, avgRowSize=54.0\n cardinality: 1\n column statistics: \n * L_QUANTITY-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * L_EXTENDEDPRICE-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * L_DISCOUNT-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * L_TAX-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * L_SHIPDATE-->[NaN, NaN, 0.0, 4.0, 1.0] ESTIMATE\n * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n", "session_variables": "{\"partial_update_mode\":\"auto\",\"cbo_cte_reuse\":true,\"character_set_connection\":\"utf8\",\"cbo_use_correlated_join_estimate\":true,\"enable_insert_strict\":true,\"enable_connector_adaptive_io_tasks\":true,\"tx_isolation\":\"REPEATABLE-READ\",\"enable_hive_metadata_cache_with_insert\":false,\"cbo_cte_reuse_rate_v2\":1.15,\"character_set_results\":\"utf8\",\"enable_count_star_optimization\":true,\"query_excluding_mv_names\":\"\",\"enable_rewrite_simple_agg_to_meta_scan\":false,\"enable_adaptive_sink_dop\":true,\"consistent_hash_virtual_number\":32,\"enable_profile\":false,\"load_mem_limit\":0,\"sql_safe_updates\":0,\"runtime_filter_early_return_selectivity\":0.05,\"enable_local_shuffle_agg\":true,\"disable_function_fold_constants\":false,\"select_ratio_threshold\":0.15,\"query_delivery_timeout\":300,\"collation_database\":\"utf8_general_ci\",\"spill_mem_table_size\":104857600,\"cbo_use_lock_db\":false,\"new_planner_agg_stage\":0,\"use_compute_nodes\":-1,\"collation_connection\":\"utf8_general_ci\",\"resource_group\":\"\",\"profile_limit_fold\":true,\"spill_operator_max_bytes\":1048576000,\"cbo_max_reorder_node_use_dp\":10,\"enable_hive_column_stats\":true,\"enable_groupby_use_output_alias\":false,\"forward_to_leader\":false,\"count_distinct_column_buckets\":1024,\"query_cache_agg_cardinality_limit\":5000000,\"cboPushDownAggregateMode_v1\":-1,\"window_partition_mode\":1,\"enable_tablet_internal_parallel_v2\":true,\"interpolate_passthrough\":true,\"enable_incremental_mv\":false,\"SQL_AUTO_IS_NULL\":false,\"event_scheduler\":\"OFF\",\"max_pipeline_dop\":64,\"broadcast_right_table_scale_factor\":10,\"materialized_view_rewrite_mode\":\"DEFAULT\",\"enable_simplify_case_when\":true,\"runtime_join_filter_push_down_limit\":1024000,\"big_query_log_cpu_second_threshold\":480,\"div_precision_increment\":4,\"runtime_adaptive_dop_max_block_rows_per_driver_seq\":16384,\"log_rejected_record_num\":0,\"cbo_push_down_distinct_below_window\":true,\"sql_mode_v2\":32,\"prefer_cte_rewrite\":false,\"hdfs_backend_selector_scan_range_shuffle\":false,\"pipeline_profile_level\":1,\"parallel_fragment_exec_instance_num\":1,\"max_scan_key_num\":-1,\"net_read_timeout\":60,\"streaming_preaggregation_mode\":\"auto\",\"hive_partition_stats_sample_size\":3000,\"enable_mv_planner\":false,\"enable_collect_table_level_scan_stats\":true,\"profile_timeout\":2,\"cbo_push_down_aggregate\":\"global\",\"spill_encode_level\":7,\"enable_query_dump\":false,\"global_runtime_filter_build_max_size\":67108864,\"enable_rewrite_sum_by_associative_rule\":true,\"query_cache_hot_partition_num\":3,\"enable_prune_complex_types\":true,\"query_cache_type\":0,\"max_parallel_scan_instance_num\":-1,\"query_cache_entry_max_rows\":409600,\"enable_mv_optimizer_trace_log\":false,\"connector_io_tasks_per_scan_operator\":16,\"enable_materialized_view_union_rewrite\":true,\"sql_quote_show_create\":true,\"scan_or_to_union_threshold\":50000000,\"enable_exchange_pass_through\":true,\"runtime_profile_report_interval\":10,\"query_cache_entry_max_bytes\":4194304,\"enable_exchange_perf\":false,\"workgroup_id\":0,\"enable_rewrite_groupingsets_to_union_all\":false,\"transmission_compression_type\":\"NO_COMPRESSION\",\"interactive_timeout\":3600,\"use_page_cache\":true,\"big_query_log_scan_bytes_threshold\":10737418240,\"collation_server\":\"utf8_general_ci\",\"tablet_internal_parallel_mode\":\"auto\",\"enable_pipeline\":true,\"spill_mode\":\"auto\",\"enable_query_debug_trace\":false,\"enable_show_all_variables\":false,\"full_sort_max_buffered_bytes\":16777216,\"wait_timeout\":28800,\"transmission_encode_level\":7,\"query_including_mv_names\":\"\",\"transaction_isolation\":\"REPEATABLE-READ\",\"enable_global_runtime_filter\":true,\"enable_load_profile\":false,\"enable_plan_validation\":true,\"load_transmission_compression_type\":\"NO_COMPRESSION\",\"cbo_enable_low_cardinality_optimize\":true,\"scan_use_query_mem_ratio\":0.3,\"new_planner_optimize_timeout\":3000,\"enable_outer_join_reorder\":true,\"force_schedule_local\":false,\"hudi_mor_force_jni_reader\":false,\"cbo_enable_greedy_join_reorder\":true,\"range_pruner_max_predicate\":100,\"enable_rbo_table_prune\":false,\"spillable_operator_mask\":-1,\"rpc_http_min_size\":2147482624,\"cbo_debug_alive_backend_number\":0,\"global_runtime_filter_probe_min_size\":102400,\"scan_or_to_union_limit\":4,\"enable_cbo_table_prune\":false,\"enable_parallel_merge\":true,\"nested_mv_rewrite_max_level\":3,\"net_write_timeout\":60,\"cbo_prune_shuffle_column_rate\":0.1,\"spill_revocable_max_bytes\":0,\"hash_join_push_down_right_table\":true,\"pipeline_sink_dop\":0,\"broadcast_row_limit\":15000000,\"enable_populate_block_cache\":true,\"exec_mem_limit\":2147483648,\"enable_sort_aggregate\":false,\"query_cache_force_populate\":false,\"runtime_filter_on_exchange_node\":false,\"disable_join_reorder\":false,\"enable_rule_based_materialized_view_rewrite\":true,\"connector_scan_use_query_mem_ratio\":0.3,\"net_buffer_length\":16384,\"cbo_prune_subfield\":true,\"full_sort_max_buffered_rows\":1024000,\"query_timeout\":300,\"connector_io_tasks_slow_io_latency_ms\":50,\"cbo_max_reorder_node\":50,\"enable_distinct_column_bucketization\":false,\"enable_big_query_log\":true,\"analyze_mv\":\"sample\",\"runtime_filter_scan_wait_time\":20,\"enable_sync_materialized_view_rewrite\":true,\"prefer_compute_node\":false,\"enable_strict_type\":false,\"group_concat_max_len\":65535,\"parse_tokens_limit\":3500000,\"chunk_size\":4096,\"global_runtime_filter_probe_min_selectivity\":0.5,\"query_mem_limit\":0,\"enable_filter_unused_columns_in_scan_stage\":true,\"enable_scan_block_cache\":false,\"enable_materialized_view_single_table_view_delta_rewrite\":false,\"auto_increment_increment\":1,\"sql_dialect\":\"StarRocks\",\"big_query_log_scan_rows_threshold\":1000000000,\"character_set_client\":\"utf8\",\"autocommit\":true,\"enable_column_expr_predicate\":true,\"enable_runtime_adaptive_dop\":false,\"cbo_cte_max_limit\":10,\"storage_engine\":\"olap\",\"enable_optimizer_trace_log\":false,\"spill_operator_min_bytes\":52428800,\"cbo_enable_dp_join_reorder\":true,\"tx_visible_wait_timeout\":10,\"enable_materialized_view_view_delta_rewrite\":true,\"cbo_max_reorder_node_use_exhaustive\":4,\"enable_sql_digest\":false,\"spill_mem_table_num\":2,\"enable_spill\":false,\"pipeline_dop\":0,\"single_node_exec_plan\":false,\"full_sort_late_materialization_v2\":true,\"join_implementation_mode_v2\":\"auto\",\"sql_select_limit\":9223372036854775807,\"enable_materialized_view_rewrite\":true,\"statistic_collect_parallel\":1,\"hdfs_backend_selector_hash_algorithm\":\"consistent\",\"disable_colocate_join\":false,\"max_pushdown_conditions_per_column\":-1,\"default_table_compression\":\"lz4_frame\",\"runtime_adaptive_dop_max_output_amplification_factor\":0,\"innodb_read_only\":true,\"spill_mem_limit_threshold\":0.8,\"cbo_reorder_threshold_use_exhaustive\":6,\"enable_predicate_reorder\":false,\"enable_query_cache\":false,\"max_allowed_packet\":33554432,\"time_zone\":\"Asia/Shanghai\",\"enable_multicolumn_global_runtime_filter\":false,\"character_set_server\":\"utf8\",\"cbo_use_nth_exec_plan\":0,\"io_tasks_per_scan_operator\":4,\"parallel_exchange_instance_num\":-1,\"enable_shared_scan\":false,\"allow_default_partition\":false}", "be_number": 1, "be_core_stat": { "numOfHardwareCoresPerBe": "{\"10004\":104}", "cachedAvgNumOfHardwareCores": 104 }, "exception": [], "version": "main_querydump", "commit_version": "0c4d8c8d3e" } ``` ##### Enable desensitization (default)[​](#enable-desensitization-default "Direct link to Enable desensitization (default)") Command: ```shell wget --user=root --password=123 --post-file query_file "http://127.0.0.1:8030/api/query_dump?db=tpch -O dump_file ``` Return data: The desensitized data is returned in JSON format. ```json { "statement": "SELECT tbl_mock_001.mock_012, tbl_mock_001.mock_007, sum(tbl_mock_001.mock_010) AS mock_019, sum(tbl_mock_001.mock_005) AS mock_020, sum(tbl_mock_001.mock_005 * (1 - tbl_mock_001.mock_004)) AS mock_021, sum((tbl_mock_001.mock_005 * (1 - tbl_mock_001.mock_004)) * (1 + tbl_mock_001.mock_017)) AS mock_022, avg(tbl_mock_001.mock_010) AS mock_023, avg(tbl_mock_001.mock_005) AS mock_024, avg(tbl_mock_001.mock_004) AS mock_025, count(*) AS mock_026\nFROM db_mock_000.tbl_mock_001\nWHERE tbl_mock_001.mock_013 <= '1998-12-01'\nGROUP BY tbl_mock_001.mock_012, tbl_mock_001.mock_007 ORDER BY tbl_mock_001.mock_012 ASC, tbl_mock_001.mock_007 ASC ", "table_meta": { "db_mock_000.tbl_mock_001": "CREATE TABLE db_mock_000.tbl_mock_001 (\nmock_008 int(11) NOT NULL ,\nmock_009 int(11) NOT NULL ,\nmock_016 int(11) NOT NULL ,\nmock_006 int(11) NOT NULL ,\nmock_010 double NOT NULL ,\nmock_005 double NOT NULL ,\nmock_004 double NOT NULL ,\nmock_017 double NOT NULL ,\nmock_012 char(1) NOT NULL ,\nmock_007 char(1) NOT NULL ,\nmock_013 date NOT NULL ,\nmock_003 date NOT NULL ,\nmock_011 date NOT NULL ,\nmock_014 char(25) NOT NULL ,\nmock_015 char(10) NOT NULL ,\nmock_002 varchar(44) NOT NULL ,\nmock_018 char(1) NOT NULL \n) ENGINE= OLAP \nDUPLICATE KEY(mock_008)\nDISTRIBUTED BY HASH(mock_008) BUCKETS 20 \nPROPERTIES (\n\"replication_num\" = \"1\"\n);" }, "table_row_count": { "db_mock_000.tbl_mock_001": { "tbl_mock_001": 3 } }, "column_statistics": { "db_mock_000.tbl_mock_001": { "mock_017": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE", "mock_013": "[1.6094304E9, 1.6094304E9, 0.0, 4.0, 1.0] ESTIMATE", "mock_005": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE", "mock_004": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE", "mock_012": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE", "mock_007": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE", "mock_010": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE" } }, "explain_info": "PLAN FRAGMENT 0(F02)\n Output Exprs:9: mock_012 | 10: mock_007 | 20: sum | 21: sum | 22: sum | 23: sum | 24: avg | 25: avg | 26: avg | 27: count\n Input Partition: UNPARTITIONED\n RESULT SINK\n\n 6:MERGING-EXCHANGE\n distribution type: GATHER\n cardinality: 1\n column statistics: \n * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n\nPLAN FRAGMENT 1(F01)\n\n Input Partition: HASH_PARTITIONED: 9: mock_012, 10: mock_007\n OutPut Partition: UNPARTITIONED\n OutPut Exchange id: 06\n\n 5:SORT\n | order by: [9, VARCHAR, false] ASC, [10, VARCHAR, false] ASC\n | offset: 0\n | cardinality: 1\n | column statistics: \n | * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 4:AGGREGATE (merge finalize)\n | aggregate: sum[([20: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([21: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([22: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([23: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([24: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([25: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([26: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], count[([27: count, BIGINT, false]); args: ; result: BIGINT; args nullable: true; result nullable: false]\n | group by: [9: mock_012, VARCHAR, false], [10: mock_007, VARCHAR, false]\n | cardinality: 1\n | column statistics: \n | * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 3:EXCHANGE\n distribution type: SHUFFLE\n partition exprs: [9: mock_012, VARCHAR, false], [10: mock_007, VARCHAR, false]\n cardinality: 1\n\nPLAN FRAGMENT 2(F00)\n\n Input Partition: RANDOM\n OutPut Partition: HASH_PARTITIONED: 9: mock_012, 10: mock_007\n OutPut Exchange id: 03\n\n 2:AGGREGATE (update serialize)\n | STREAMING\n | aggregate: sum[([5: mock_010, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([6: mock_005, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([18: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([19: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], avg[([5: mock_010, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([6: mock_005, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([7: mock_004, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], count[(*); args: ; result: BIGINT; args nullable: false; result nullable: false]\n | group by: [9: mock_012, VARCHAR, false], [10: mock_007, VARCHAR, false]\n | cardinality: 1\n | column statistics: \n | * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 1:Project\n | output columns:\n | 5 <-> [5: mock_010, DOUBLE, false]\n | 6 <-> [6: mock_005, DOUBLE, false]\n | 7 <-> [7: mock_004, DOUBLE, false]\n | 9 <-> [9: mock_012, CHAR, false]\n | 10 <-> [10: mock_007, CHAR, false]\n | 18 <-> [29: multiply, DOUBLE, false]\n | 19 <-> [29: multiply, DOUBLE, false] * 1.0 + [8: mock_017, DOUBLE, false]\n | common expressions:\n | 28 <-> 1.0 - [7: mock_004, DOUBLE, false]\n | 29 <-> [6: mock_005, DOUBLE, false] * [28: subtract, DOUBLE, false]\n | cardinality: 1\n | column statistics: \n | * mock_010-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * mock_005-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * mock_004-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 0:OlapScanNode\n table: mock_001, rollup: mock_001\n preAggregation: on\n Predicates: [11: mock_013, DATE, false] <= '1998-12-01'\n partitionsRatio=1/1, tabletsRatio=20/20\n tabletList=45030,45032,45034,45036,45038,45040,45042,45044,45046,45048 ...\n actualRows=3, avgRowSize=54.0\n cardinality: 1\n column statistics: \n * mock_010-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * mock_005-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * mock_004-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * mock_017-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * mock_013-->[NaN, NaN, 0.0, 4.0, 1.0] ESTIMATE\n * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n", "session_variables": "{\"partial_update_mode\":\"auto\",\"cbo_cte_reuse\":true,\"character_set_connection\":\"utf8\",\"cbo_use_correlated_join_estimate\":true,\"enable_insert_strict\":true,\"enable_connector_adaptive_io_tasks\":true,\"tx_isolation\":\"REPEATABLE-READ\",\"enable_hive_metadata_cache_with_insert\":false,\"cbo_cte_reuse_rate_v2\":1.15,\"character_set_results\":\"utf8\",\"enable_count_star_optimization\":true,\"query_excluding_mv_names\":\"\",\"enable_rewrite_simple_agg_to_meta_scan\":false,\"enable_adaptive_sink_dop\":true,\"consistent_hash_virtual_number\":32,\"enable_profile\":false,\"load_mem_limit\":0,\"sql_safe_updates\":0,\"runtime_filter_early_return_selectivity\":0.05,\"enable_local_shuffle_agg\":true,\"disable_function_fold_constants\":false,\"select_ratio_threshold\":0.15,\"query_delivery_timeout\":300,\"collation_database\":\"utf8_general_ci\",\"spill_mem_table_size\":104857600,\"cbo_use_lock_db\":false,\"new_planner_agg_stage\":0,\"use_compute_nodes\":-1,\"collation_connection\":\"utf8_general_ci\",\"resource_group\":\"\",\"profile_limit_fold\":true,\"spill_operator_max_bytes\":1048576000,\"cbo_max_reorder_node_use_dp\":10,\"enable_hive_column_stats\":true,\"enable_groupby_use_output_alias\":false,\"forward_to_leader\":false,\"count_distinct_column_buckets\":1024,\"query_cache_agg_cardinality_limit\":5000000,\"cboPushDownAggregateMode_v1\":-1,\"window_partition_mode\":1,\"enable_tablet_internal_parallel_v2\":true,\"interpolate_passthrough\":true,\"enable_incremental_mv\":false,\"SQL_AUTO_IS_NULL\":false,\"event_scheduler\":\"OFF\",\"max_pipeline_dop\":64,\"broadcast_right_table_scale_factor\":10,\"materialized_view_rewrite_mode\":\"DEFAULT\",\"enable_simplify_case_when\":true,\"runtime_join_filter_push_down_limit\":1024000,\"big_query_log_cpu_second_threshold\":480,\"div_precision_increment\":4,\"runtime_adaptive_dop_max_block_rows_per_driver_seq\":16384,\"log_rejected_record_num\":0,\"cbo_push_down_distinct_below_window\":true,\"sql_mode_v2\":32,\"prefer_cte_rewrite\":false,\"hdfs_backend_selector_scan_range_shuffle\":false,\"pipeline_profile_level\":1,\"parallel_fragment_exec_instance_num\":1,\"max_scan_key_num\":-1,\"net_read_timeout\":60,\"streaming_preaggregation_mode\":\"auto\",\"hive_partition_stats_sample_size\":3000,\"enable_mv_planner\":false,\"enable_collect_table_level_scan_stats\":true,\"profile_timeout\":2,\"cbo_push_down_aggregate\":\"global\",\"spill_encode_level\":7,\"enable_query_dump\":false,\"global_runtime_filter_build_max_size\":67108864,\"enable_rewrite_sum_by_associative_rule\":true,\"query_cache_hot_partition_num\":3,\"enable_prune_complex_types\":true,\"query_cache_type\":0,\"max_parallel_scan_instance_num\":-1,\"query_cache_entry_max_rows\":409600,\"enable_mv_optimizer_trace_log\":false,\"connector_io_tasks_per_scan_operator\":16,\"enable_materialized_view_union_rewrite\":true,\"sql_quote_show_create\":true,\"scan_or_to_union_threshold\":50000000,\"enable_exchange_pass_through\":true,\"runtime_profile_report_interval\":10,\"query_cache_entry_max_bytes\":4194304,\"enable_exchange_perf\":false,\"workgroup_id\":0,\"enable_rewrite_groupingsets_to_union_all\":false,\"transmission_compression_type\":\"NO_COMPRESSION\",\"interactive_timeout\":3600,\"use_page_cache\":true,\"big_query_log_scan_bytes_threshold\":10737418240,\"collation_server\":\"utf8_general_ci\",\"tablet_internal_parallel_mode\":\"auto\",\"enable_pipeline\":true,\"spill_mode\":\"auto\",\"enable_query_debug_trace\":false,\"enable_show_all_variables\":false,\"full_sort_max_buffered_bytes\":16777216,\"wait_timeout\":28800,\"transmission_encode_level\":7,\"query_including_mv_names\":\"\",\"transaction_isolation\":\"REPEATABLE-READ\",\"enable_global_runtime_filter\":true,\"enable_load_profile\":false,\"enable_plan_validation\":true,\"load_transmission_compression_type\":\"NO_COMPRESSION\",\"cbo_enable_low_cardinality_optimize\":true,\"scan_use_query_mem_ratio\":0.3,\"new_planner_optimize_timeout\":3000,\"enable_outer_join_reorder\":true,\"force_schedule_local\":false,\"hudi_mor_force_jni_reader\":false,\"cbo_enable_greedy_join_reorder\":true,\"range_pruner_max_predicate\":100,\"enable_rbo_table_prune\":false,\"spillable_operator_mask\":-1,\"rpc_http_min_size\":2147482624,\"cbo_debug_alive_backend_number\":0,\"global_runtime_filter_probe_min_size\":102400,\"scan_or_to_union_limit\":4,\"enable_cbo_table_prune\":false,\"enable_parallel_merge\":true,\"nested_mv_rewrite_max_level\":3,\"net_write_timeout\":60,\"cbo_prune_shuffle_column_rate\":0.1,\"spill_revocable_max_bytes\":0,\"hash_join_push_down_right_table\":true,\"pipeline_sink_dop\":0,\"broadcast_row_limit\":15000000,\"enable_populate_block_cache\":true,\"exec_mem_limit\":2147483648,\"enable_sort_aggregate\":false,\"query_cache_force_populate\":false,\"runtime_filter_on_exchange_node\":false,\"disable_join_reorder\":false,\"enable_rule_based_materialized_view_rewrite\":true,\"connector_scan_use_query_mem_ratio\":0.3,\"net_buffer_length\":16384,\"cbo_prune_subfield\":true,\"full_sort_max_buffered_rows\":1024000,\"query_timeout\":300,\"connector_io_tasks_slow_io_latency_ms\":50,\"cbo_max_reorder_node\":50,\"enable_distinct_column_bucketization\":false,\"enable_big_query_log\":true,\"analyze_mv\":\"sample\",\"runtime_filter_scan_wait_time\":20,\"enable_sync_materialized_view_rewrite\":true,\"prefer_compute_node\":false,\"enable_strict_type\":false,\"group_concat_max_len\":65535,\"parse_tokens_limit\":3500000,\"chunk_size\":4096,\"global_runtime_filter_probe_min_selectivity\":0.5,\"query_mem_limit\":0,\"enable_filter_unused_columns_in_scan_stage\":true,\"enable_scan_block_cache\":false,\"enable_materialized_view_single_table_view_delta_rewrite\":false,\"auto_increment_increment\":1,\"sql_dialect\":\"StarRocks\",\"big_query_log_scan_rows_threshold\":1000000000,\"character_set_client\":\"utf8\",\"autocommit\":true,\"enable_column_expr_predicate\":true,\"enable_runtime_adaptive_dop\":false,\"cbo_cte_max_limit\":10,\"storage_engine\":\"olap\",\"enable_optimizer_trace_log\":false,\"spill_operator_min_bytes\":52428800,\"cbo_enable_dp_join_reorder\":true,\"tx_visible_wait_timeout\":10,\"enable_materialized_view_view_delta_rewrite\":true,\"cbo_max_reorder_node_use_exhaustive\":4,\"enable_sql_digest\":false,\"spill_mem_table_num\":2,\"enable_spill\":false,\"pipeline_dop\":0,\"single_node_exec_plan\":false,\"full_sort_late_materialization_v2\":true,\"join_implementation_mode_v2\":\"auto\",\"sql_select_limit\":9223372036854775807,\"enable_materialized_view_rewrite\":true,\"statistic_collect_parallel\":1,\"hdfs_backend_selector_hash_algorithm\":\"consistent\",\"disable_colocate_join\":false,\"max_pushdown_conditions_per_column\":-1,\"default_table_compression\":\"lz4_frame\",\"runtime_adaptive_dop_max_output_amplification_factor\":0,\"innodb_read_only\":true,\"spill_mem_limit_threshold\":0.8,\"cbo_reorder_threshold_use_exhaustive\":6,\"enable_predicate_reorder\":false,\"enable_query_cache\":false,\"max_allowed_packet\":33554432,\"time_zone\":\"Asia/Shanghai\",\"enable_multicolumn_global_runtime_filter\":false,\"character_set_server\":\"utf8\",\"cbo_use_nth_exec_plan\":0,\"io_tasks_per_scan_operator\":4,\"parallel_exchange_instance_num\":-1,\"enable_shared_scan\":false,\"allow_default_partition\":false}", "be_number": 1, "be_core_stat": { "numOfHardwareCoresPerBe": "{\"10004\":104}", "cachedAvgNumOfHardwareCores": 104 }, "exception": [], "version": "main_querydump", "commit_version": "0c4d8c8d3e" } ``` --- ### Troubleshooting FE Heap Memory This topic describes how to troubleshoot FE heap memory issues in StarRocks. #### Sudden surge of heap memory allocation[​](#sudden-surge-of-heap-memory-allocation "Direct link to Sudden surge of heap memory allocation") You can identify issues that cause sudden heap memory allocation surge by checking memory profiles (or memory allocate profiles) in StarRocks. From v3.3.6 onwards, StarRocks supports printing memory profiles in `.tgz`-formatted compressed HTML files under directory **fe/log/proc\_profile**. ![FE Memory FAQ - 1](/assets/images/fe_mem_faq_1-cc79083e05ac1d8a54b98b4853069c38.png) You can find the corresponding file based on the time when the memory issue occurred, decompress it, and open it in your browser. In the file, you will see the flame graph of the stack memory allocation. The wider a frame, the more memory resources the stack had been allocated to. For example, in the flame graph below, the width of frame `BDBEnvironment.getDatabaseNamesWithPrefix` exceeds 50% of the graph, indicating that more than half of the memory resources were allocated to this function. ![FE Memory FAQ - 2](/assets/images/fe_mem_faq_2-e49cdbb24bed89137b9c2b628ca00b3a.png) If no memory profile was printed during the surge, you can manually disable the CPU profile, and set the profile printing interval to 5 minutes by setting the following items in the FE configuration file **fe.conf**, and restarting FE. ```properties proc_profile_cpu_enable = false proc_profile_collect_interval_s = 300 ``` ##### Workaround for versions earlier than v3.3.6[​](#workaround-for-versions-earlier-than-v336 "Direct link to Workaround for versions earlier than v3.3.6") For versions earlier than v3.3.6, you can print memory profiles regularly via a script. Run the following Shell script under the **fe** directory: ```bash #!/bin/bash mkdir -p mem_alloc_log while true do current_time=$(date +'%Y-%m-%d-%H-%M-%S') file_name="mem_alloc_log/alloc-profile-${current_time}.html" ./bin/profiler.sh -e alloc --alloc 2m -d 300 -f "$file_name" `cat bin/fe.pid` done ``` #### Slow increase of heap memory usage[​](#slow-increase-of-heap-memory-usage "Direct link to Slow increase of heap memory usage") From v3.3.7 onwards, StarRocks supports printing Memory Usage Tracker logs for tracking memory leak issues. Memory Usage Tracker will regularly record the memory usage of each module. Example: ```plain 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Agent - AgentTaskTracker estimated 0B of memory. Contains AgentTask with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Backup - BackupHandler estimated 0B of memory. Contains BackupOrRestoreJob with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Compaction - CompactionMgr estimated 0B of memory. Contains PartitionStats with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Coordinator - QeProcessorImpl estimated 0B of memory. Contains QueryCoordinator with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Delete - DeleteMgrEPack estimated 0B of memory. Contains DeleteInfo with 0 object(s). DeleteJob with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Dict - CacheDictManager estimated 0B of memory. Contains ColumnDict with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Export - ExportMgr estimated 0B of memory. Contains ExportJob with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Load - InsertOverwriteJobMgr estimated 0B of memory. Contains insertOverwriteJobs with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Load - LoadMgrEPack estimated 245.7KB of memory. Contains LoadJob with 1165 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Load - RoutineLoadMgrEPack estimated 0B of memory. Contains RoutineLoad with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Load - StreamLoadMgrEPack estimated 0B of memory. Contains StreamLoad with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module LocalMetastore - LocalMetastore estimated 7KB of memory. Contains Partition with 45 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module MV - MVTimelinessMgr estimated 0B of memory. Contains mvTimelinessMap with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Profile - ProfileManager estimated 96B of memory. Contains QueryProfile with 4 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Query - QueryTracker estimated 0B of memory. Contains QueryDetail with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Report - ReportHandler estimated 0B of memory. Contains PendingTask with 0 object(s). ReportQueue with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Statistics - CachedStatisticStorage estimated 0B of memory. Contains TableStats with 0 object(s). ColumnStats with 0 object(s). PartitionStats with 0 object(s). HistogramStats with 0 object(s). ConnectorTableStats with 0 object(s). ConnectorHistogramStats with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module TabletInvertedIndex - TabletInvertedIndex estimated 49.8KB of memory. Contains TabletMeta with 208 object(s). TabletCount with 208 object(s). ReplicateCount with 216 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Task - TaskManager estimated 0B of memory. Contains Task with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Task - TaskRunManager estimated 0B of memory. Contains PendingTaskRun with 0 object(s). RunningTaskRun with 0 object(s). HistoryTaskRun with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Transaction - GlobalTransactionMgr estimated 311.4KB of memory. Contains Txn with 1329 object(s). TxnCallbackCount with 0 object(s). 2025-02-05 19:35:23.288+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():111] total tracked memory: 614.2KB, jvm: Process used: 672MB, heap used: 299.9MB, non heap used: 170MB, direct buffer used: 4.2MB ``` --- ### Operation and Maintenance This topic provides answers to some questions related to operation and maintenance. #### Can the `trash` directory be cleaned up?[​](#can-the-trash-directory-be-cleaned-up "Direct link to can-the-trash-directory-be-cleaned-up") You can configure the FE parameter `catalog_trash_expire_second` to specify how long files stay in the FE `trash` directory (Default: 24 hours). The BE parameter `trash_file_expire_time_sec` controls BE trash cleanup intervals (Default: 24 hours). After a DROP TABLE or DROP DATABASE, the data first enters the FE trash and is kept for one day, during which it can be recovered with RECOVER. After that, it moves to the BE trash, which also keeps it for 24 hours. #### Do tablets have a primary–secondary relationship? If some replicas are missing, how does it impact queries?[​](#do-tablets-have-a-primarysecondary-relationship-if-some-replicas-are-missing-how-does-it-impact-queries "Direct link to Do tablets have a primary–secondary relationship? If some replicas are missing, how does it impact queries?") If the table property `replicated_storage` is set to `true`, writes use a primary–secondary mechanism: data is written to the primary replica first and then synchronized to others. Multiple replicas generally do not have a major impact on query performance. #### Can I measure CPU and memory usage for a task through the monitoring interface?[​](#can-i-measure-cpu-and-memory-usage-for-a-task-through-the-monitoring-interface "Direct link to Can I measure CPU and memory usage for a task through the monitoring interface?") You can check the `cpucostns` and `memcostbytes` fields in `fe.audit.log`. #### Creating a materialized view on a Unique Key table returns an error "The aggregation type of column\[now\_time] must be same as the aggregate type of base column in aggregate table". Does Unique Key table support materialized views?[​](#creating-a-materialized-view-on-a-unique-key-table-returns-an-error-the-aggregation-type-of-columnnow_time-must-be-same-as-the-aggregate-type-of-base-column-in-aggregate-table-does-unique-key-table-support-materialized-views "Direct link to Creating a materialized view on a Unique Key table returns an error \"The aggregation type of column[now_time] must be same as the aggregate type of base column in aggregate table\". Does Unique Key table support materialized views?") The error indicates that the aggregation type of the materialized view must match that of the base table. For Unique Key tables, you can only materialized views to adjust their sort key order. For example, if base table `tableA` has columns `k1`, `k2`, `k3`, with `k1` and `k2` as sort keys. While your queries contain the clause `WHERE k3=x` and need to be accelerated prefix index, you may create a materialized view that uses `k3` as the first column: ```sql CREATE MATERIALIZED VIEW k3_as_key AS SELECT k3, k2, k1 FROM tableA; ``` #### How can I get import volume metrics with precise timestamps? Is `query_latency` the metric for average response time?[​](#how-can-i-get-import-volume-metrics-with-precise-timestamps-is-query_latency-the-metric-for-average-response-time "Direct link to how-can-i-get-import-volume-metrics-with-precise-timestamps-is-query_latency-the-metric-for-average-response-time") Table-level import metrics can be obtained from: `http://user:password@fe_host:http_port/metrics?type=json&with_table_metrics=all`. Cluster-level data can be retrieved from: `http://fe_host:http_port/api/show_data` (increment must be calculated manually). `query_latency` provides percentile query response time. #### What is the difference between SHOW PROC '/backends' and SHOW BACKENDS?[​](#what-is-the-difference-between-show-proc-backends-and-show-backends "Direct link to What is the difference between SHOW PROC '/backends' and SHOW BACKENDS?") `SHOW PROC '/backends'` retrieve metadata from the current FE and may lag. While `SHOW BACKENDS` retrieve metadata from the Leader FE and is authoritative. #### Does StarRocks have a timeout mechanism? Why do some client connections persist for a long time?[​](#does-starrocks-have-a-timeout-mechanism-why-do-some-client-connections-persist-for-a-long-time "Direct link to Does StarRocks have a timeout mechanism? Why do some client connections persist for a long time?") Yes. You can configure the system variable `wait_timeout` (Default: 8 hours) to adjust the connection timeout. Example: ```sql SET GLOBAL wait_timeout = 3600; ``` #### Can GRANT authorize multiple tables in one statement?[​](#can-grant-authorize-multiple-tables-in-one-statement "Direct link to Can GRANT authorize multiple tables in one statement?") No. Statements like `GRANT on db1.tb1, db1.tb2` are not supported. #### Can I revoke privileges on a specific table when the ALL TABLES privilege is granted?[​](#can-i-revoke-privileges-on-a-specific-table-when-the-all-tables-privilege-is-granted "Direct link to Can I revoke privileges on a specific table when the ALL TABLES privilege is granted?") Subset revocation is not supported. It is recommended to grant privileges at the database or table level. #### Does StarRocks support table-level and row-level privileges?[​](#does-starrocks-support-table-level-and-row-level-privileges "Direct link to Does StarRocks support table-level and row-level privileges?") Table-level access control is supported. Row- and column-level access controls are not supported in the Open-source Edition. #### If a Primary Key table is not partitioned, does hot–cold data separation still work?[​](#if-a-primary-key-table-is-not-partitioned-does-hotcold-data-separation-still-work "Direct link to If a Primary Key table is not partitioned, does hot–cold data separation still work?") No. Hot–cold separation is based on partitions. #### Can I change the data storage path if the data was mistakenly placed in the root directory?[​](#can-i-change-the-data-storage-path-if-the-data-was-mistakenly-placed-in-the-root-directory "Direct link to Can I change the data storage path if the data was mistakenly placed in the root directory?") Yes. Update `storage_root_path` in `be.conf` to add a new disk, and use semicolons to separate paths. #### How do I check the StarRocks version of FE?[​](#how-do-i-check-the-starrocks-version-of-fe "Direct link to How do I check the StarRocks version of FE?") Run `SHOW FRONTENDS;` and check the `Version` field. #### Does StarRocks support DELETE with nested subqueries?[​](#does-starrocks-support-delete-with-nested-subqueries "Direct link to Does StarRocks support DELETE with nested subqueries?") From v2.3 onwards, Primary Key tables supports full DELETE WHERE syntax. See [Reference - DELETE](https://docs.starrocks.io/zh/docs/sql-reference/sql-statements/table_bucket_part_index/DELETE/) for details. #### For dynamic partitions, if I don't want old partitions automatically cleaned, can I simply omit dynamic\_partition.start?[​](#for-dynamic-partitions-if-i-dont-want-old-partitions-automatically-cleaned-can-i-simply-omit-dynamic_partitionstart "Direct link to For dynamic partitions, if I don't want old partitions automatically cleaned, can I simply omit dynamic_partition.start?") No. Set it to a very large value. #### If a BE machine has faulty memory and needs maintenance, what should be done?[​](#if-a-be-machine-has-faulty-memory-and-needs-maintenance-what-should-be-done "Direct link to If a BE machine has faulty memory and needs maintenance, what should be done?") [Decommission](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md#be) the BE. After repair, add it back to the cluster. #### Can I enforce all future partitions to use SSD by default?[​](#can-i-enforce-all-future-partitions-to-use-ssd-by-default "Direct link to Can I enforce all future partitions to use SSD by default?") No. Default is HDD. Manual configuration is required. #### After adding new BEs, tablets automatically rebalanced. Can I decommission old BEs immediately?[​](#after-adding-new-bes-tablets-automatically-rebalanced-can-i-decommission-old-bes-immediately "Direct link to After adding new BEs, tablets automatically rebalanced. Can I decommission old BEs immediately?") Yes. You do not need to wait until the rebalancing completed. Up to two nodes can be decommissioned at once. #### Will adding new BEs and removing old ones affect performance?[​](#will-adding-new-bes-and-removing-old-ones-affect-performance "Direct link to Will adding new BEs and removing old ones affect performance?") Rebalancing happens automatically and should not affect normal operations. It is recommended to remove nodes one at a time. #### How to replace six BE nodes with six new ones?[​](#how-to-replace-six-be-nodes-with-six-new-ones "Direct link to How to replace six BE nodes with six new ones?") Add six new BE nodes, and then decommission the old ones one by one. #### Why doesn't unhealthy tablet count decrease after decommissioning a node?[​](#why-doesnt-unhealthy-tablet-count-decrease-after-decommissioning-a-node "Direct link to Why doesn't unhealthy tablet count decrease after decommissioning a node?") Check for single-replica tables. Other repairs may be blocked if they continuously retry. #### What does this BE log mean? "tcmalloc: large alloc xxxxxxxx bytes"[​](#what-does-this-be-log-mean-tcmalloc-large-alloc-xxxxxxxx-bytes "Direct link to What does this BE log mean? \"tcmalloc: large alloc xxxxxxxx bytes\"") A large memory allocation request occurred, often caused by large queries. Check the corresponding `query_id` in `be.INFO` to locate the SQL. #### Will tablet migration after adding nodes cause disk I/O fluctuations?[​](#will-tablet-migration-after-adding-nodes-cause-disk-io-fluctuations "Direct link to Will tablet migration after adding nodes cause disk I/O fluctuations?") Yes, temporary I/O fluctuation is expected during balancing. #### What are the recommended data migration methods for deployments on the cloud?[​](#what-are-the-recommended-data-migration-methods-for-deployments-on-the-cloud "Direct link to What are the recommended data migration methods for deployments on the cloud?") * To migrate a single table, you can: * Create a StarRocks external table, and then load the data from it using INSERT INTO SELECT. * Read data from the source BE using a Spark-connector-based programme, and load the data into the target BE using encapsulated STREAM LOAD. * To migrate multiple tables, you can use Backup and Restore. First, back up the data from the source cluster to the remote storage. Then, restore the data from the remote storage to the target cluster. * If your cluster uses HDFS as the remote storage, you can first use `distcp` to migrate the data files, and then use Broker Load to load the data into the target cluster. #### How can I resolve the error "failed to create task: disk ... exceed limit usage"?[​](#how-can-i-resolve-the-error-failed-to-create-task-disk--exceed-limit-usage "Direct link to How can I resolve the error \"failed to create task: disk ... exceed limit usage\"?") The disk is full. Scale up the storage or clean trash. #### FE logs show "tablet migrate failed". How can I solve this?[​](#fe-logs-show-tablet-migrate-failed-how-can-i-solve-this "Direct link to FE logs show \"tablet migrate failed\". How can I solve this?") It is likely because the `storage_root_path` is HDD and the table property `storage_medium` is set to `SSD`. You can set the table property to `HDD`: ```sql ALTER TABLE db.table MODIFY PARTITION (*) SET("storage_medium"="HDD"); ``` #### Can I migrate FE to another machine by copying metadata only?[​](#can-i-migrate-fe-to-another-machine-by-copying-metadata-only "Direct link to Can I migrate FE to another machine by copying metadata only?") No. Use the recommended method: add a new node, and then remove the old one. #### To disks on one BE have uneven usage (500 GB 99%, 2 TB 20%). Why is balancing not happening?[​](#to-disks-on-one-be-have-uneven-usage-500-gb-99-2-tb-20-why-is-balancing-not-happening "Direct link to To disks on one BE have uneven usage (500 GB 99%, 2 TB 20%). Why is balancing not happening?") Balancing assumes equal-sized disks. Use disks of the same size to ensure even distribution. #### If `max_backend_down_time_second` is set to `3600`, does it mean that I must recover the failed BE within one hour?[​](#if-max_backend_down_time_second-is-set-to-3600-does-it-mean-that-i-must-recover-the-failed-be-within-one-hour "Direct link to if-max_backend_down_time_second-is-set-to-3600-does-it-mean-that-i-must-recover-the-failed-be-within-one-hour") If the downtime of the BE exceeds this configuration, FE will replenish replicas on other BE. When the failed BE is added back to the cluster, large rebalancing costs may occur. #### Is there an IDE tool to export table structures and views from the production environment?[​](#is-there-an-ide-tool-to-export-table-structures-and-views-from-the-production-environment "Direct link to Is there an IDE tool to export table structures and views from the production environment?") Yes. See [olapdb-tool](https://github.com/Astralidea/olapdb-tool). #### Can multiple system variables be set in a single SQL (for example, timeout and parallelism)?[​](#can-multiple-system-variables-be-set-in-a-single-sql-for-example-timeout-and-parallelism "Direct link to Can multiple system variables be set in a single SQL (for example, timeout and parallelism)?") Yes. Example: ```sql SELECT /*+ SET_VAR(query_timeout=1, is_report_success=true, parallel_fragment_exec_instance_num=2) */ COUNT(1) FROM table; ``` #### Do VARCHAR columns in sort keys follow the 36-byte prefix limit?[​](#do-varchar-columns-in-sort-keys-follow-the-36-byte-prefix-limit "Direct link to Do VARCHAR columns in sort keys follow the 36-byte prefix limit?") VARCHAR columns are truncated based on actual length. Only the first column gets a short-key index. Place VARCHAR sort keys in the third position if possible. #### BE fails to start with "while lock file" error. How can I deal with it?[​](#be-fails-to-start-with-while-lock-file-error-how-can-i-deal-with-it "Direct link to BE fails to start with \"while lock file\" error. How can I deal with it?") BE process is still running. Kill the daemon process and restart. #### Do clients need to explicitly connect to observer FEs for read-only queries?[​](#do-clients-need-to-explicitly-connect-to-observer-fes-for-read-only-queries "Direct link to Do clients need to explicitly connect to observer FEs for read-only queries?") No. Write requests automatically route to the leader FE; observers serve read-only queries. #### FE exits with `LOG_FILE_NOT_FOUND` caused by too many open files. How can I solve this?[​](#fe-exits-with-log_file_not_found-caused-by-too-many-open-files-how-can-i-solve-this "Direct link to fe-exits-with-log_file_not_found-caused-by-too-many-open-files-how-can-i-solve-this") Check OS file descriptor limits by running `cat /proc/$pid/limits`, and run `lsof -n -p $fe_pid>/tmp/fe_fd.txt` to inspect the file descriptor in use. #### Are there limits on partition and bucket numbers?[​](#are-there-limits-on-partition-and-bucket-numbers "Direct link to Are there limits on partition and bucket numbers?") * For partitions, the default limit `4096` (configurable via the FE configuration `max_partitions_in_one_batch`). * For buckets, there is no limit. The recommended size for each bucket is 1 GB. #### How to manually switch FE leader?[​](#how-to-manually-switch-fe-leader "Direct link to How to manually switch FE leader?") Stop the current Leader, and a new leader will be elected automatically. #### How to replace three FE nodes (1 Leader and 2 Follower) with new machines?[​](#how-to-replace-three-fe-nodes-1-leader-and-2-follower-with-new-machines "Direct link to How to replace three FE nodes (1 Leader and 2 Follower) with new machines?") 1. Add 2 new Followers to the cluster. 2. Remove 1 old Follower. 3. Add one last new Follower. 4. Remove the remaining old Follower. 5. Remove the old Leader. #### Dynamic partition creation not working as expected. Why?[​](#dynamic-partition-creation-not-working-as-expected-why "Direct link to Dynamic partition creation not working as expected. Why?") Dynamic partition check runs every 10 minutes. This behavior is controlled by the FE configuration `dynamic_partition_check_interval_seconds`. #### FE logs frequently show "connect processor exception because，java.io.IOException: Connection reset by peer". Why?[​](#fe-logs-frequently-show-connect-processor-exception-becausejavaioioexception-connection-reset-by-peer-why "Direct link to FE logs frequently show \"connect processor exception because，java.io.IOException: Connection reset by peer\". Why?") It is likely because client-side disconnects, connection pool drops, or network issues. Check OS backlog metrics and network stability. Check if the following metrics have changed: ```bash netstat -s | grep -i LISTEN netstat -s | grep TCPBacklogDrop cat /proc/sys/net/core/somaxconn ``` #### After I execute TRUNCATE statements, when will the storage space to be released?[​](#after-i-execute-truncate-statements-when-will-the-storage-space-to-be-released "Direct link to After I execute TRUNCATE statements, when will the storage space to be released?") Immediately. #### How to check whether bucket distribution is balanced?[​](#how-to-check-whether-bucket-distribution-is-balanced "Direct link to How to check whether bucket distribution is balanced?") Run the following command: ```sql SHOW TABLET FROM db.table PARTITION (); ``` #### How to solve the error "Fail to get master client from cache"?[​](#how-to-solve-the-error-fail-to-get-master-client-from-cache "Direct link to How to solve the error \"Fail to get master client from cache\"?") It is an FE–BE communication failure. Check IP and port connectivity. #### How to migrate StarRocks when IP changes?[​](#how-to-migrate-starrocks-when-ip-changes "Direct link to How to migrate StarRocks when IP changes?") FQDN-mode deployment is recommended. #### Can a non-partitioned table be converted to a partitioned table?[​](#can-a-non-partitioned-table-be-converted-to-a-partitioned-table "Direct link to Can a non-partitioned table be converted to a partitioned table?") No. You can create a new partitioned table and use INSERT INTO SELECT to migrate the data. #### Can I query historical SQL execution? Is there an audit log?[​](#can-i-query-historical-sql-execution-is-there-an-audit-log "Direct link to Can I query historical SQL execution? Is there an audit log?") Yes. See `fe.audit.log`. #### How to convert a non-partitioned table to SSD storage?[​](#how-to-convert-a-non-partitioned-table-to-ssd-storage "Direct link to How to convert a non-partitioned table to SSD storage?") Run the following SQL: ```sql ALTER TABLE db.tbl MODIFY PARTITION (*) SET ("storage_medium"="SSD"); ``` #### After executing ALTER TABLE ADD COLUMN, queries against `information_schema.COLUMNS` show delay. Is this normal?[​](#after-executing-alter-table-add-column-queries-against-information_schemacolumns-show-delay-is-this-normal "Direct link to after-executing-alter-table-add-column-queries-against-information_schemacolumns-show-delay-is-this-normal") Yes. ALTER operations are asynchronous. Check progress with `SHOW ALTER COLUMN`. #### If retention for a dynamic partition table is changed from 366 to 732 days, can historical partitions be auto-created?[​](#if-retention-for-a-dynamic-partition-table-is-changed-from-366-to-732-days-can-historical-partitions-be-auto-created "Direct link to If retention for a dynamic partition table is changed from 366 to 732 days, can historical partitions be auto-created?") Follow these steps: 1. Disable dynamic partitions. ```sql ALTER TABLE db.tbl SET ("dynamic_partition.enable" = "false"); ``` 2. Manually add partitions. ```sql ALTER TABLE db.tbl ADD PARTITIONS START ("2019-01-01") END ("2019-12-31") EVERY (interval 1 day); ``` 3. Re-enable dynamic partitions. ```sql ALTER TABLE db.tbl SET ("dynamic_partition.enable" = "true"); ``` #### Can Routine Load tasks be monitored and alerted when switching from Running to Paused?[​](#can-routine-load-tasks-be-monitored-and-alerted-when-switching-from-running-to-paused "Direct link to Can Routine Load tasks be monitored and alerted when switching from Running to Paused?") Yes. StarRocks supports monitoring metrics for Routine Load tasks and can be connected to alert systems. #### How to diagnose unhealthy replicas?[​](#how-to-diagnose-unhealthy-replicas "Direct link to How to diagnose unhealthy replicas?") Run the following statements to identify UnhealthyTablets: ```sql SHOW PROC '/statistic' SHOW PROC '/statistic/' ``` Then, analyze UnhealthyTablets with `SHOW TABLET tablet_id`. If the result shows that two replicas have consistent data but one replica has inconsistent data—meaning two out of three replicas completed the write successfully—this is considered a successful write. You can then check whether the tablets in UnhealthyTablets are fixed. If they are fixed, it indicates an issue. If the status is changing, you can adjust the loading frequency for the corresponding table. #### Error: "SyntaxErrorException: Reach limit of connections". How to troubleshoot?[​](#error-syntaxerrorexception-reach-limit-of-connections-how-to-troubleshoot "Direct link to Error: \"SyntaxErrorException: Reach limit of connections\". How to troubleshoot?") Increase per-user limits by running the following command: ```sql ALTER USER 'jack' SET PROPERTIES ('max_user_connections'='1000'); ``` Also check load balancers and idle connection accumulation (`wait_timeout`). #### How does XFS and ext4 affect QPS?[​](#how-does-xfs-and-ext4-affect-qps "Direct link to How does XFS and ext4 affect QPS?") StarRocks typically performs better with XFS. #### How long before a BE is considered down and tablet migration begins?[​](#how-long-before-a-be-is-considered-down-and-tablet-migration-begins "Direct link to How long before a BE is considered down and tablet migration begins?") 1. When Heartbeat (default: every 5 seconds) failed 3 times, the BE is marked as not alive. 2. After that, there is an intended delay before executing Clone, which is 60 seconds. 3. Then, the replica clone starts. If BE recovers later, its replicas are deleted. #### Tablet scheduling on new nodes is slow (only 100 at a time). How to adjust?[​](#tablet-scheduling-on-new-nodes-is-slow-only-100-at-a-time-how-to-adjust "Direct link to Tablet scheduling on new nodes is slow (only 100 at a time). How to adjust?") Tune the following FE configurations: ```sql ADMIN SET FRONTEND CONFIG ("schedule_slot_num_per_path"="8"); ADMIN SET FRONTEND CONFIG ("max_scheduling_tablets"="1000"); ADMIN SET FRONTEND CONFIG ("max_balancing_tablets"="1000"); ``` #### Is BACKUP operation serial? It seems only one HDFS directory is changing[​](#is-backup-operation-serial-it-seems-only-one-hdfs-directory-is-changing "Direct link to Is BACKUP operation serial? It seems only one HDFS directory is changing") BACKUP operations are parallel, but upload to HDFS uses a single worker, which is controlled by the BE configuration `upload_worker_count`. Adjust it with caution because it might affect the disk I/O and network I/O. #### How to solve the FE OOM error "OutOfMemoryError: GC overhead limit exceeded"?[​](#how-to-solve-the-fe-oom-error-outofmemoryerror-gc-overhead-limit-exceeded "Direct link to How to solve the FE OOM error \"OutOfMemoryError: GC overhead limit exceeded\"?") Increase FE JVM memory. #### How to solve the error "Cannot truncate a file by broker"?[​](#how-to-solve-the-error-cannot-truncate-a-file-by-broker "Direct link to How to solve the error \"Cannot truncate a file by broker\"?") 1. Check broker logs for error messages. 2. Check BE warning logs for error "remote file checksum is invalid. remote:\*\*\*\* local: \*\*\*\*\*". 3. Search the remote number the broker `apache_hdfs_broker.log` for error "receive a check path request, request detail", and identify the duplicate files. 4. Remove or rename problematic remote files and retry. #### How to verify that disk removal has completed?[​](#how-to-verify-that-disk-removal-has-completed "Direct link to How to verify that disk removal has completed?") Run `SHOW PROC '/statistic'` and ensure `UnhealthyTablet` count is zero. #### How to modify replica count for historical partitions?[​](#how-to-modify-replica-count-for-historical-partitions "Direct link to How to modify replica count for historical partitions?") Run the following command: ```sql ALTER TABLE db.tbl MODIFY PARTITION (*) SET("replication_num"="3"); ``` #### For a three-replica table, if the disk for a BE node is damaged, will replicas automatically recover to maintain three copies?[​](#for-a-three-replica-table-if-the-disk-for-a-be-node-is-damaged-will-replicas-automatically-recover-to-maintain-three-copies "Direct link to For a three-replica table, if the disk for a BE node is damaged, will replicas automatically recover to maintain three copies?") Yes, if enough BE nodes are available. #### How to troubleshoot if the error "reach limit connections" continues even after increasing user limits?[​](#how-to-troubleshoot-if-the-error-reach-limit-connections-continues-even-after-increasing-user-limits "Direct link to How to troubleshoot if the error \"reach limit connections\" continues even after increasing user limits?") Check load balancers (ProxySQL, F5), idle connection accumulation, and reduce `wait_timeout` to 2–4 hours. #### If FE metadata is lost, is all cluster metadata lost?[​](#if-fe-metadata-is-lost-is-all-cluster-metadata-lost "Direct link to If FE metadata is lost, is all cluster metadata lost?") Metadata resides in FE. With only one FE, it's unrecoverable. With multiple FEs, you can re-add the failed node and metadata will be replicated. #### How to solve the loading error "INTERNAL\_ERROR, FE leader shows NullPointerException"?[​](#how-to-solve-the-loading-error-internal_error-fe-leader-shows-nullpointerexception "Direct link to How to solve the loading error \"INTERNAL_ERROR, FE leader shows NullPointerException\"?") Add JVM option `-XX:-OmitStackTraceInFastThrow` and restart FE to get full stack trace. #### How to solve the error "The partition column could not be aggregated column" when setting the partition column for a Primary Key table?[​](#how-to-solve-the-error-the-partition-column-could-not-be-aggregated-column-when-setting-the-partition-column-for-a-primary-key-table "Direct link to How to solve the error \"The partition column could not be aggregated column\" when setting the partition column for a Primary Key table?") The partition columns must be key columns. --- ### Other FAQ This topic provides answers to some general questions. #### Do VARCHAR (32) and STRING occupy the same storage space?[​](#do-varchar-32-and-string-occupy-the-same-storage-space "Direct link to Do VARCHAR (32) and STRING occupy the same storage space?") Both are variable-length data types. When you store data of the same length, VARCHAR (32) and STRING occupy the same storage space. #### Do VARCHAR (32) and STRING perform the same for the data query?[​](#do-varchar-32-and-string-perform-the-same-for-the-data-query "Direct link to Do VARCHAR (32) and STRING perform the same for the data query?") Yes. #### Why do TXT files imported from Oracle still appear garbled after I set the character set to UTF-8?[​](#why-do-txt-files-imported-from-oracle-still-appear-garbled-after-i-set-the-character-set-to-utf-8 "Direct link to Why do TXT files imported from Oracle still appear garbled after I set the character set to UTF-8?") To solve this problem, perform the following steps: 1. For example, there is a file named **original**, whose text is garbled. The character set of this file is ISO-8859-1. Run the following code to obtain the character set of the file. ```plaintext file --mime-encoding origin.txt origin.txt: iso-8859-1 ``` 2. Run the `iconv` command to convert the character set of this file into UTF-8. ```plaintext iconv -f iso-8859-1 -t utf-8 origin.txt > origin_utf-8.txt ``` 3. After the conversion, the text of this file still appears garbled. You can then regrade the character set of this file as GBK and convert the character set into UTF-8 again. ```plaintext iconv -f gbk -t utf-8 origin.txt > origin_utf-8.txt ``` #### Is the length of STRING defined by MySQL the same as that defined by StarRocks?[​](#is-the-length-of-string-defined-by-mysql-the-same-as-that-defined-by-starrocks "Direct link to Is the length of STRING defined by MySQL the same as that defined by StarRocks?") For VARCHAR(n), StarRocks defines "n" by bytes and MySQL defines "n" by characters. According to UTF-8, one Chinese character is equal to three bytes. When StarRocks and MySQL define "n" as the same number, MySQL saves three times as many characters as StarRocks. #### Can the data type of partitioned fields of a table be FLOAT, DOUBLE, or DECIMAL?[​](#can-the-data-type-of-partitioned-fields-of-a-table-be-float-double-or-decimal "Direct link to Can the data type of partitioned fields of a table be FLOAT, DOUBLE, or DECIMAL?") No, only DATE, DATETIME, and INT are supported. #### How to check the storage space that is occupied by the data in a table?[​](#how-to-check-the-storage-space-that-is-occupied-by-the-data-in-a-table "Direct link to How to check the storage space that is occupied by the data in a table?") Execute the SHOW DATA statement to see the corresponding storage space. You can also see the data volume, the number of copies, and the number of rows. **Note**: There is a time delay in data statistics. #### How to request a quota increase for the StarRocks database?[​](#how-to-request-a-quota-increase-for-the-starrocks-database "Direct link to How to request a quota increase for the StarRocks database?") To request a quota increase, run the following code: ```plaintext ALTER DATABASE example_db SET DATA QUOTA 10T; ``` #### Does StarRocks support updating particular fields in a table by executing the UPSERT statement?[​](#does-starrocks-support-updating-particular-fields-in-a-table-by-executing-the-upsert-statement "Direct link to Does StarRocks support updating particular fields in a table by executing the UPSERT statement?") StarRocks 2.2 and later support updating specific fields in a table by using the Primary Key table. StarRocks 1.9 and later support updating all fields in a table by using the Primary Key table. For more information, see [Primary Key table](https://docs.starrocks.io/docs/table_design/table_types/primary_key_table.md) in StarRocks 2.2. #### How to swap the data between two tables or two partitions?[​](#how-to-swap-the-data-between-two-tables-or-two-partitions "Direct link to How to swap the data between two tables or two partitions?") Execute the SWAP WITH statement to swap the data between two tables or two partitions. The SWAP WITH statement is more secure than the INSERT OVERWRITE statement. Before you swap the data, check the data first and then see whether the data after the swapping is consistent with the data before the swapping. * Swap two tables: For example, there is a table named table 1. If you want to replace table 1 with another one, perform the following steps: 1. Create a new table named table 2. ```sql create table2 like table1; ``` 2. Use Stream Load, Broker Load, or Insert Into to load data from table 1 into table 2. 3. Replace table 1 with table 2. ```sql ALTER TABLE table1 SWAP WITH table2; ``` By doing so, the data is loaded accurately into table 1. * Swap two partitions: For example, there is a table named table 1. If you want to replace the partition data in table 1, perform the following steps: 1. Create a temporary partition. ```sql ALTER TABLE table1 ADD TEMPORARY PARTITION tp1 VALUES LESS THAN("2020-02-01"); ``` 2. Load the partition data from table 1 into the temporary partition. 3. Replace the partition of table 1 with the temporary partition. ```sql ALTER TABLE table1 REPLACE PARTITION (p1) WITH TEMPORARY PARTITION (tp1); ``` #### This error "error to open replicated environment, will exit" occurs when I restart a frontend (FE)[​](#this-error-error-to-open-replicated-environment-will-exit-occurs-when-i-restart-a-frontend-fe "Direct link to This error \"error to open replicated environment, will exit\" occurs when I restart a frontend (FE)") This error occurs due to BDBJE's bug. To solve this problem, update the BDBJE version to 1.17 or later. #### This error "Broker list path exception" occurs when I query data from a new Apache Hive table[​](#this-error-broker-list-path-exception-occurs-when-i-query-data-from-a-new-apache-hive-table "Direct link to This error \"Broker list path exception\" occurs when I query data from a new Apache Hive table") ##### Problem description[​](#problem-description "Direct link to Problem description") ```plaintext msg:Broker list path exception path=hdfs://172.31.3.136:9000/user/hive/warehouse/zltest.db/student_info/*, broker=TNetworkAddress(hostname:172.31.4.233, port:8000) ``` ##### Solution[​](#solution "Direct link to Solution") Contact the StarRocks technical support and check whether the address and port of the namenode are correct and whether you have permission to access the address and port of the namenode. #### This error "get hive partition metadata failed" occurs when I query data from a new Apache Hive table[​](#this-error-get-hive-partition-metadata-failed-occurs-when-i-query-data-from-a-new-apache-hive-table "Direct link to This error \"get hive partition metadata failed\" occurs when I query data from a new Apache Hive table") ##### Problem description[​](#problem-description-1 "Direct link to Problem description") ```plaintext msg:get hive partition meta data failed: java.net.UnknownHostException: emr-header-1.cluster-242 ``` ##### Solution[​](#solution-1 "Direct link to Solution") Ensure that the network is connected and upload the **host** file to each backend (BE) in your StarRocks cluster. #### This error "do\_open failed. reason = Invalid ORC postscript length" occurs when I access ORC external table in Apache Hive[​](#this-error-do_open-failed-reason--invalid-orc-postscript-length-occurs-when-i-access-orc-external-table-in-apache-hive "Direct link to This error \"do_open failed. reason = Invalid ORC postscript length\" occurs when I access ORC external table in Apache Hive") ##### Problem description[​](#problem-description-2 "Direct link to Problem description") The metadata of the Apache Hive is cached in the FEs. But there is a two-hours time lag for StarRocks to update the metadata. Before StarRocks finishes the update, If you insert new data or update data in the Apache Hive table, the data in HDFS scanned by the BEs and the data obtained by the FEs are different. Therefore, this error occurs. ```plaintext MySQL [bdp_dim]> select * from dim_page_func_s limit 1; ERROR 1064 (HY000): HdfsOrcScanner::do_open failed. reason = Invalid ORC postscript length ``` ##### Solution[​](#solution-2 "Direct link to Solution") To solve this problem, perform one of the following operations: * Upgrade your current version to StarRocks 2.2 or later. * Manually refresh your Apache Hive table. For more information, see [Metadata caching strategy](https://docs.starrocks.io/docs/data_source/External_table.md). #### This error "caching\_sha2\_password cannot be loaded" occurs when I connect external tables of MySQL[​](#this-error-caching_sha2_password-cannot-be-loaded-occurs-when-i-connect-external-tables-of-mysql "Direct link to This error \"caching_sha2_password cannot be loaded\" occurs when I connect external tables of MySQL") ##### Problem description[​](#problem-description-3 "Direct link to Problem description") The default authentication plugin of MySQL 8.0 is caching\_sha2\_password. The default authentication plugin of MySQL 5.7 is mysql\_native\_password. This error occurs because you use the wrong authentication plugin. ##### Solution[​](#solution-3 "Direct link to Solution") To solve this problem, perform one of the following operations: * Connect to the StarRocks. ```sql ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'yourpassword'; ``` * Modify the `my.cnf` file. ```plaintext vim my.cnf [mysqld] default_authentication_plugin=mysql_native_password ``` #### How to release disk space immediately after deleting a table?[​](#how-to-release-disk-space-immediately-after-deleting-a-table "Direct link to How to release disk space immediately after deleting a table?") If you execute the DROP TABLE statement to delete a table, StarRocks takes a while to release the allocated disk space. To release the allocated disk space immediately, execute the DROP TABLE FORCE statement to delete a table. When you execute the DROP TABLE FORCE statement, the StarRocks deletes the table directly without checking whether there are unfinished events in it. We recommend that you execute the DROP TABLE FORCE statement with caution. Because once the table is deleted, you cannot restore it. #### How to view the current version of StarRocks?[​](#how-to-view-the-current-version-of-starrocks "Direct link to How to view the current version of StarRocks?") Run the `select current_version();` command or the CLI command `./bin/show_fe_version.sh` to view the current version. #### How to set the memory size of an FE?[​](#how-to-set-the-memory-size-of-an-fe "Direct link to How to set the memory size of an FE?") The metadata is stored in the memory used by the FE. You can set the memory size of the FE according to the number of tablets as shown in the table below. For example, if the number of tablets is below 1 million, you should allocate a minimum of 16 GB memory to the FE. You can configure the values of the parameters `-Xms` and `-Xmx` in the **JAVA\_OPTS** configuration item in the **fe.conf** file, and the values of the parameters `-Xms` and `-Xmx` should be consistent. Note that the configuration should be same across all FEs because any of the FEs can be elected as a Leader. | Number of tablets | Memory size of each FE | | ----------------- | ---------------------- | | below 1 million | 16 GB | | 1 ～ 2 million | 32 GB | | 2 ～ 5 million | 64 GB | | 5 ～ 10 million | 128 GB | #### How does StarRocks calculate its query time?[​](#how-does-starrocks-calculate-its-query-time "Direct link to How does StarRocks calculate its query time?") StarRocks supports querying data by using multiple threads. Query time refers to the time used by multiple threads to query data. #### Does StarRocks support setting the path when I export data locally?[​](#does-starrocks-support-setting-the-path-when-i-export-data-locally "Direct link to Does StarRocks support setting the path when I export data locally?") No. #### What are the concurrency upper limits of StarRocks?[​](#what-are-the-concurrency-upper-limits-of-starrocks "Direct link to What are the concurrency upper limits of StarRocks?") You can test the concurrency limitations based on the actual business scenarios or simulated business scenarios. According to the feedback of some users, maximum of 20,000 QPS or 30,000 QPS can be achieved. #### Why is the first-time SSB test performance of StarRocks slower than that done the second time?[​](#why-is-the-first-time-ssb-test-performance-of-starrocks-slower-than-that-done-the-second-time "Direct link to Why is the first-time SSB test performance of StarRocks slower than that done the second time?") The speed to read disks for the first query relates to the performance of disks. After the first query, the page cache is generated for the subsequent queries, so the query is faster than before. #### How many BEs need to be configured at least for a cluster?[​](#how-many-bes-need-to-be-configured-at-least-for-a-cluster "Direct link to How many BEs need to be configured at least for a cluster?") StarRocks supports single node deployment, so you need to configure at least one BE. BEs need to be run with AVX2, so we recommend that you deploy BEs on machines with 8-core and 16GB or higher configurations. #### How to set data permissions when I use Apache Superset to visualize the data in StarRocks?[​](#how-to-set-data-permissions-when-i-use-apache-superset-to-visualize-the-data-in-starrocks "Direct link to How to set data permissions when I use Apache Superset to visualize the data in StarRocks?") You can create a new user account and then set the data permission by granting permissions on the table query to the user. #### Why does the profile fail to display after I set `enable_profile` to `true`?[​](#why-does-the-profile-fail-to-display-after-i-set-enable_profile-to-true "Direct link to why-does-the-profile-fail-to-display-after-i-set-enable_profile-to-true") The report is only submitted to the leader FE for access. #### How to check field annotations in the tables of StarRocks?[​](#how-to-check-field-annotations-in-the-tables-of-starrocks "Direct link to How to check field annotations in the tables of StarRocks?") Run the `show create table xxx` command. #### When I create a table, how to specify the default value for the NOW() function?[​](#when-i-create-a-table-how-to-specify-the-default-value-for-the-now-function "Direct link to When I create a table, how to specify the default value for the NOW() function?") Only StarRocks 2.1 or later version supports specifying the default value for a function. For versions earlier than StarRocks 2.1, you can only specify a constant for a function. #### How can I release the storage space of BE nodes?[​](#how-can-i-release-the-storage-space-of-be-nodes "Direct link to How can I release the storage space of BE nodes?") You can remove the directory `trash` using `rm -rf` command. If you have already restored your data from snapshot, you can remove the directory `snapshot`. #### Can add extra disks to BE nodes?[​](#can-add-extra-disks-to-be-nodes "Direct link to Can add extra disks to BE nodes?") Yes. You can add the disks to the directory specified by the BE configuration item `storage_root_path`. #### How can I prevent expression partition conflicts caused by concurrent execution of loading tasks and partition creation tasks?[​](#how-can-i-prevent-expression-partition-conflicts-caused-by-concurrent-execution-of-loading-tasks-and-partition-creation-tasks "Direct link to How can I prevent expression partition conflicts caused by concurrent execution of loading tasks and partition creation tasks?") Currently, for tables with the expression partitioning strategy, partitions created during loading tasks conflict with those created during ALTER TABLE tasks. Since loading tasks take priority, any conflicting ALTER tasks will fail. To prevent this issue, consider the following workarounds: * If you use coarse time-based partitions (or example, partitioning by day or month), you can prevent ALTER operations from crossing time boundaries, reducing the risk of partition creation failures. * If you use fine-grained time-based partitions (or example, partitioning by hour), you can manually create partitions for a future time range to ensure that ALTER operations are not disrupted by new partition created by loading tasks. You can use the [EXPLAIN ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/EXPLAIN_ANALYZE.md) feature to trigger partition creation by executing an INSERT statement without committing the transaction. This allows you to create the necessary partitions without affecting actual data. The following example demonstrates how to create partitions for the next 8 hours: ```sql CREATE TABLE t( event_time DATETIME ) PARTITION BY date_trunc('hour', event_time); EXPLAIN ANALYZE INSERT INTO t (event_time) SELECT DATE_ADD(NOW(), INTERVAL d hour) FROM table(generate_series(0, 8)) AS g(d); SHOW PARTITIONS FROM t; ``` --- ### Troubleshooting Resource Isolation This topic provides answers to some frequently asked questions about resource isolation. #### Resource Group[​](#resource-group "Direct link to Resource Group") ##### What resources must be configured in a resource group?[​](#what-resources-must-be-configured-in-a-resource-group "Direct link to What resources must be configured in a resource group?") The CPU resource limit must be configured. You must set either `cpu_weight` or `exclusive_cpu_core`, and the values must be greater than 0. ##### Does StarRocks support hard resource limits?[​](#does-starrocks-support-hard-resource-limits "Direct link to Does StarRocks support hard resource limits?") Yes. StarRocks supports hard limits for memory. From v3.3.5 onwards, StarRocks supports hard limits for CPU via `exclusive_cpu_cores`. ##### How is CPU allocated among resource groups?[​](#how-is-cpu-allocated-among-resource-groups "Direct link to How is CPU allocated among resource groups?") When multiple resource groups run queries simultaneously, CPU usage is proportional to each group’s `cpu_core_limit`. If the normal group exceeds the `BE vCPU cores - short_query.cpu_core_limit` within a scheduling cycle, it will not be scheduled further in that cycle. ##### How are resources calculated for the `short_query` resource group?[​](#how-are-resources-calculated-for-the-short_query-resource-group "Direct link to how-are-resources-calculated-for-the-short_query-resource-group") When the `short_query` resource group has running queries, the CPU limit of all normal groups becomes `BE vCPU cores − short_query.cpu_core_limit`. If the `short_query` resource group is idle, its resources can be used by normal groups. ##### How are queries without a matched resource group handled?[​](#how-are-queries-without-a-matched-resource-group-handled "Direct link to How are queries without a matched resource group handled?") They use the default resource group `default_wg`, which has the following resource limits and properties: * `cpu_core_limit` = vCPU cores * `mem_limit` = 100% * `type` = `normal` ##### If the resource group `rg3` has no queries and all resources are allocated to resource groups `rg1` and `rg2`, will those resources be reallocated when `rg3` receives a large query?[​](#if-the-resource-group-rg3-has-no-queries-and-all-resources-are-allocated-to-resource-groups-rg1-and-rg2-will-those-resources-be-reallocated-when-rg3-receives-a-large-query "Direct link to if-the-resource-group-rg3-has-no-queries-and-all-resources-are-allocated-to-resource-groups-rg1-and-rg2-will-those-resources-be-reallocated-when-rg3-receives-a-large-query") Yes. Reclamation happens gradually and stabilizes within tens of milliseconds to seconds. ##### What is the role of classifiers? What if no classifier matches, or two classifiers overlap?[​](#what-is-the-role-of-classifiers-what-if-no-classifier-matches-or-two-classifiers-overlap "Direct link to What is the role of classifiers? What if no classifier matches, or two classifiers overlap?") If no classifier matches, the query falls back to the default resource group `default_wg`. Classifiers have weights; if a query matches multiple classifiers, the one with the highest weight is selected. ##### Why do resource groups require `mem_limit` if BE already uses 90% memory by default?[​](#why-do-resource-groups-require-mem_limit-if-be-already-uses-90-memory-by-default "Direct link to why-do-resource-groups-require-mem_limit-if-be-already-uses-90-memory-by-default") `mem_limit` restricts memory at the resource-group level. It applies only to queries match that resource group. ##### If the resource group's `query_type` is set to `insert`, does INSERT INTO SELECT only limit INSERT or also SELECT?[​](#if-the-resource-groups-query_type-is-set-to-insert-does-insert-into-select-only-limit-insert-or-also-select "Direct link to if-the-resource-groups-query_type-is-set-to-insert-does-insert-into-select-only-limit-insert-or-also-select") Only the SELECT part is limited by the resource group. The INSERT operation is not limited. ##### If the resource groups `mem_limit` is set to `20%`, is the usable memory calculated as `BE_memory * 90% * 20%`? What if total `mem_limit` exceeds 100% when there are multiple resource groups?[​](#if-the-resource-groups-mem_limit-is-set-to-20-is-the-usable-memory-calculated-as-be_memory--90--20-what-if-total-mem_limit-exceeds-100-when-there-are-multiple-resource-groups "Direct link to if-the-resource-groups-mem_limit-is-set-to-20-is-the-usable-memory-calculated-as-be_memory--90--20-what-if-total-mem_limit-exceeds-100-when-there-are-multiple-resource-groups") Total `mem_limit` across groups can exceed 100%. But if a query exceeds its resource group limit, it will fail. ##### How can I verify whether a resource group is applied to a query?[​](#how-can-i-verify-whether-a-resource-group-is-applied-to-a-query "Direct link to How can I verify whether a resource group is applied to a query?") Check `fe.audit.log` or run `EXPLAIN VERBOSE ` to view the matched resource group. ##### Are resource groups defined per cluster or per BE node?[​](#are-resource-groups-defined-per-cluster-or-per-be-node "Direct link to Are resource groups defined per cluster or per BE node?") Resources are divided per BE node, and the resource group configuration applies to all BE nodes in the cluster. ##### How to inspect resource group usage or monitoring metrics?[​](#how-to-inspect-resource-group-usage-or-monitoring-metrics "Direct link to How to inspect resource group usage or monitoring metrics?") Use FE/BE metrics endpoints to view specific resource-group-related metrics. * For FE, collect the following metrics from `fe_host:8030/metrics?type=json`: * `starrocks_fe_query_resource_group`: The number of queries historically run in this resource group (including those currently running). * `starrocks_fe_query_resource_group_latency`: The query latency percentile for this resource group. The label type indicates specific percentiles, including `mean`, `75_quantile`, `95_quantile`, `98_quantile`, `99_quantile`, `999_quantile`. * `starrocks_fe_query_resource_group_err`: The number of queries in this resource group that encountered an error. * For BE, collect the following metrics from `be_host:8040/metrics?type=json`: * `starrocks_be_resource_group_cpu_limit_ratio`: The ratio of this resource group's `cpu_core_limit` to the total `cpu_core_limit` across all resource groups. * `starrocks_be_resource_group_mem_limit_bytes`: The memory limit for this resource group. ##### What is the difference between `short_query` and normal resource groups? Can I create multiple `short_query` resource groups?[​](#what-is-the-difference-between-short_query-and-normal-resource-groups-can-i-create-multiple-short_query-resource-groups "Direct link to what-is-the-difference-between-short_query-and-normal-resource-groups-can-i-create-multiple-short_query-resource-groups") Only one `short_query` resource group is allowed. When there are queries running in `short_query` resource group, it uses actual BE cores, while normal groups share remaining resources proportionally. ##### Does StarRocks provide query priority or large-query-based prioritization?[​](#does-starrocks-provide-query-priority-or-large-query-based-prioritization "Direct link to Does StarRocks provide query priority or large-query-based prioritization?") No priority system exists. A query becomes a “large query” when exceeding any of the configured resource thresholds. ##### Does a resource group belong to a specific BE node or the BE running the query?[​](#does-a-resource-group-belong-to-a-specific-be-node-or-the-be-running-the-query "Direct link to Does a resource group belong to a specific BE node or the BE running the query?") Resource groups apply uniformly across all BEs in the cluster. ##### How should I configure `concurrency_limit`?[​](#how-should-i-configure-concurrency_limit "Direct link to how-should-i-configure-concurrency_limit") It depends on query complexity, cluster size, and workload patterns. ##### How does classifier-based matching work? Is it tied to user and/or database?[​](#how-does-classifier-based-matching-work-is-it-tied-to-user-andor-database "Direct link to How does classifier-based matching work? Is it tied to user and/or database?") Matching depends on classifier attributes such as IP, user, db, role, or `query_type`. ##### How is resource group specified through session variables?[​](#how-is-resource-group-specified-through-session-variables "Direct link to How is resource group specified through session variables?") You can set it as a variable: ```sql SET resource_group = ''; ``` Or specify it in queries via hint: ```sql SELECT /*+ SET_VAR(SET resource_group = '') */ * FROM tbl; ``` ##### Does concurrency control take effect globally, per user, or per BE?[​](#does-concurrency-control-take-effect-globally-per-user-or-per-be "Direct link to Does concurrency control take effect globally, per user, or per BE?") `concurrency_limit` restricts concurrency per resource group, while `pipeline_dop` controls parallelism of a single pipeline instance. ##### Does memory limit take effect globally, per user, or per BE?[​](#does-memory-limit-take-effect-globally-per-user-or-per-be "Direct link to Does memory limit take effect globally, per user, or per BE?") `mem_limit` applies to the resource group per BE. Per-instance memory is controlled by `exec_mem_limit`. ##### Are concurrency and memory limits only effective when resource groups are enabled?[​](#are-concurrency-and-memory-limits-only-effective-when-resource-groups-are-enabled "Direct link to Are concurrency and memory limits only effective when resource groups are enabled?") Concurrency is controlled only by resource groups. Query parallelism is controlled by session variables such as `pipeline_dop`. ##### Will CTAS tasks match resource groups if the `query_type` is set to `INSERT`?[​](#will-ctas-tasks-match-resource-groups-if-the-query_type-is-set-to-insert "Direct link to will-ctas-tasks-match-resource-groups-if-the-query_type-is-set-to-insert") Yes. Resource group will restrict the resources for the SELECT part of CTAS tasks. The big query limits will also take effect if the SELECT part exceeds the threshold. ##### Why can't the queries in the `short_query` resource group consume all CPU?[​](#why-cant-the-queries-in-the-short_query-resource-group-consume-all-cpu "Direct link to why-cant-the-queries-in-the-short_query-resource-group-consume-all-cpu") The `short_query` resource group must leave at least 1 CPU core for normal groups. ##### Without query queue and resource groups, are concurrent queries limited?[​](#without-query-queue-and-resource-groups-are-concurrent-queries-limited "Direct link to Without query queue and resource groups, are concurrent queries limited?") No. Overload results in query timeouts or memory limit errors. ##### With resource groups enabled but query queue disabled, is concurrency limited by resource groups?[​](#with-resource-groups-enabled-but-query-queue-disabled-is-concurrency-limited-by-resource-groups "Direct link to With resource groups enabled but query queue disabled, is concurrency limited by resource groups?") Yes. New queries exceeding the resource group `concurrency_limit` will fail. ##### When is a query recognized as a "big query"?[​](#when-is-a-query-recognized-as-a-big-query "Direct link to When is a query recognized as a \"big query\"?") A query is considered as a big query when it exceeds any of: * `big_query_cpu_second_limit` * `big_query_scan_rows_limit` * `big_query_mem_limit` ##### Can `default_wg` resource limits be changed?[​](#can-default_wg-resource-limits-be-changed "Direct link to can-default_wg-resource-limits-be-changed") No. As a workaround, create a general resource group that can match all queries. Example: ```sql CREATE RESOURCE GROUP general_group TO ( query_type IN ('select', 'insert') ) WITH ( 'cpu_core_limit' = '6', 'mem_limit' = '0.0000000000001' ); ``` ##### Why does the “query\_resource\_group” metric not show newly created groups?[​](#why-does-the-query_resource_group-metric-not-show-newly-created-groups "Direct link to Why does the “query_resource_group” metric not show newly created groups?") The metric is lazy-initialized; it appears only after a query hits that group. ##### If many queries run on a BE, and one hits CPU limit, do all fail?[​](#if-many-queries-run-on-a-be-and-one-hits-cpu-limit-do-all-fail "Direct link to If many queries run on a BE, and one hits CPU limit, do all fail?") Only the query reaching the limit fails. ##### If BE nodes have different memory/CPU sizes, does `mem_limit` or `cpu_core_limit` affect results?[​](#if-be-nodes-have-different-memorycpu-sizes-does-mem_limit-or-cpu_core_limit-affect-results "Direct link to if-be-nodes-have-different-memorycpu-sizes-does-mem_limit-or-cpu_core_limit-affect-results") Memory limit is hard and may cause failures on BEs with less memory size first. CPU is soft and does not cause errors. ##### Are `big_query_` parameters applied per node or globally?[​](#are-big_query_-parameters-applied-per-node-or-globally "Direct link to are-big_query_-parameters-applied-per-node-or-globally") They apply per BE node. ##### How to configure resource groups for Broker Load?[​](#how-to-configure-resource-groups-for-broker-load "Direct link to How to configure resource groups for Broker Load?") Example classifier: `query_type="insert", user="alice"`. ##### `exec_mem_limit`-related questions[​](#exec_mem_limit-related-questions "Direct link to exec_mem_limit-related-questions") Q: How many instances will be generated for a query? A: It is unpredictable because different queries can generate different number of fragments. Q: How to check the number of instance? A: You cannot check the number of instance. Q: If a query consumes 128 GB memory in total and generates 60 instances, while `query_mem_limit=0` and `exec_mem_limit=2G`. Will the query fail? A: The query fails as long as any instance consumes more than 2 GB (`exec_mem_limit`) memory. ##### How to disable global query queue for one resource group only?[​](#how-to-disable-global-query-queue-for-one-resource-group-only "Direct link to How to disable global query queue for one resource group only?") 1. Enable resource group-level query queue: ```sql SET GLOBAL enable_show_all_variables = true; SET enable_group_level_query_queue = true; ``` 2. Disable Query Queue for the current session or at user level: ```sql -- Disable Query Queue for the current session SET enable_query_queue = false; -- Disable Query Queue at user level ALTER USER 'xxx' SET PROPERTIES ("session.enable_query_queue" = "false"); ``` ##### Can queries that doesn't match any resource group be forced to fail?[​](#can-queries-that-doesnt-match-any-resource-group-be-forced-to-fail "Direct link to Can queries that doesn't match any resource group be forced to fail?") No. They will always fall back to the default resource group `default_wg`. #### Query Queue[​](#query-queue "Direct link to Query Queue") ##### How is the query queue memory trigger calculated?[​](#how-is-the-query-queue-memory-trigger-calculated "Direct link to How is the query queue memory trigger calculated?") Query Queue trigger = BE available memory size \* `query_queue_mem_used_pct_limit`. ##### Which takes priority: resource-group concurrency or query-queue concurrency?[​](#which-takes-priority-resource-group-concurrency-or-query-queue-concurrency "Direct link to Which takes priority: resource-group concurrency or query-queue concurrency?") If `enable_group_level_query_queue` is set to: * `false`: global or group limit may trigger first. * `true`: both apply; the smaller limit triggers queueing. ##### If queue size or timeout is reached, will queries fail immediately?[​](#if-queue-size-or-timeout-is-reached-will-queries-fail-immediately "Direct link to If queue size or timeout is reached, will queries fail immediately?") * When `query_queue_max_queued_queries` is reached, the query immediately fails. * When `query_queue_concurrency_limit` is reached, the query waits in queue. ##### What is the difference between resource group limits and query queue limits?[​](#what-is-the-difference-between-resource-group-limits-and-query-queue-limits "Direct link to What is the difference between resource group limits and query queue limits?") Resource groups restrict resource usage per group per BE node. Query queue uses BE-level limits for all queries. `concurrency_limit` and `max_cpu_cores` both apply when resource group-level query queue is enabled. ##### What is the difference between `pipeline_dop`, `exec_mem_limit`, and resource group concurrency limits?[​](#what-is-the-difference-between-pipeline_dop-exec_mem_limit-and-resource-group-concurrency-limits "Direct link to what-is-the-difference-between-pipeline_dop-exec_mem_limit-and-resource-group-concurrency-limits") `pipeline_dop` controls in-query parallelism. Resource groups/query queue control cluster-wide concurrent queries. `query_mem_limit` controls per-query-per-BE memory. --- ### Shared-data This topic provides answers to some frequently asked questions about shared-data clusters. #### Why does table creation fail?[​](#why-does-table-creation-fail "Direct link to Why does table creation fail?") Check the BE log (`be.INFO`) to identify the exact cause. Common causes include: * Misconfigured object storage settings (for example, `aws_s3_path`, `endpoint`, `authentication`). * Object storage service instability or exceptions. * For `CREATE STORAGE VOLUME` or `ALTER STORAGE VOLUME`, StarRocks validates storage accessibility in shared-data mode when the FE configuration `enable_storage_volume_access_check` is enabled (enabled by default). If this check is disabled, validation is skipped. If validation fails, fix credentials, endpoint, or network access first, and then retry. Other errors: Error message: "Error 1064 (HY000): Unexpected exception: Failed to create shards. INVALID\_ARGUMENT: shard info cannot be empty" Cause: This often caused when automatic bucket inference is used while no CN or BE nodes are alive. This issues is fixed in v3.2. #### Why does table creation take too long?[​](#why-does-table-creation-take-too-long "Direct link to Why does table creation take too long?") Excessive bucket numbers (especially in partitioned tables) cause StarRocks to create many tablets. The system needs to write a tablet metadata file for each tablet in the object storage, whose high latency can drastically increase total creation time. You may consider: * Reducing the number of buckets. * Increasing the tablet creation thread pool size via the BE configuration `create_tablet_worker_count`. * Checking and troubleshooting high write latency in object storage. #### Why is data in object storage not cleaned up after dropping a table?[​](#why-is-data-in-object-storage-not-cleaned-up-after-dropping-a-table "Direct link to Why is data in object storage not cleaned up after dropping a table?") StarRocks supports two DROP TABLE modes: * `DROP TABLE xxx`: moves table metadata to FE recycle bin (data is not deleted). * `DROP TABLE xxx FORCE`: immediately deletes table metadata and data. If cleanup fails, check: * Whether `DROP TABLE xxx FORCE` was used. * Whether recycle bin retention parameters are set too high. Parameters include: * FE configuration `catalog_trash_expire_second` * BE configuration `trash_file_expire_time_sec` * FE logs for deletion errors (for example, RPC timeout). Increase RPC timeout if needed. #### How can I find the storage path of table data in object storage?[​](#how-can-i-find-the-storage-path-of-table-data-in-object-storage "Direct link to How can I find the storage path of table data in object storage?") Run the following command to get the storage path. ```sql SHOW PROC '/dbs/'; ``` Example: ```sql mysql> SHOW PROC '/dbs/load_benchmark'; +---------+-------------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+--------------------------------------------------------------------------------------------------------------+ | TableId | TableName | IndexNum | PartitionColumnName | PartitionNum | State | Type | LastConsistencyCheckTime | ReplicaCount | PartitionType | StoragePath | +---------+-------------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+--------------------------------------------------------------------------------------------------------------+ | 17152 | store_sales | 1 | NULL | 1 | NORMAL | CLOUD_NATIVE | NULL | 64 | UNPARTITIONED | s3://starrocks-common/xxxxxxxxx-xxxx_load_benchmark-1699408425544/5ce4ee2c-98ba-470c-afb3-8d0bf4795e48/17152 | +---------+-------------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+--------------------------------------------------------------------------------------------------------------+ 1 row in set (0.18 sec) ``` In versions earlier than v3.1.4, table data is scattered under a single directory. From v3.1.4 onwards, data is organized by partition. The same command displays the table root path, which now contains subdirectories named after Partition IDs, and each partition directory holds sub-directories `data/` (segment data files) and `meta/` (tablet metadata files). #### Why are queries slow in shared-data clusters?[​](#why-are-queries-slow-in-shared-data-clusters "Direct link to Why are queries slow in shared-data clusters?") Common causes include: * Cache miss. * Insufficient compaction, causing too many small segment files and thus excessive I/O. * Bad parallelism (for example, too few tablets). * Improper `datacache.partition_duration` settings, causing caching failures. You need to analyze the Query Profile first to identify the root cause. ##### Cache miss[​](#cache-miss "Direct link to Cache miss") In shared-data clusters, data is stored remotely, so Data Cache is crucial. If queries become unexpectedly slow, check Query Profile metrics such as `CompressedBytesReadRemote` and `IOTimeRemote`. Cache misses may be caused by: * Data Cache disabled during table creation. * Insufficient local cache space. * Tablet migration due to elastic scaling. * Improper `datacache.partition_duration` settings preventing caching. ##### Insufficient compaction[​](#insufficient-compaction "Direct link to Insufficient compaction") Without adequate compaction, many historical data versions remain, increasing the number of segment files accessed during queries. This increases I/O and slows down queries. You can diagnose insufficient compaction by: * Checking Compaction Score for relevant partitions. Compaction Score should remain below ~10. Excessively high Compaction Scores often indicate compaction failures. * Reviewing Query Profile metrics such as `PhySegmentsCount`. If Segment counts are high, compaction may be lagging or stuck. ##### Improper tablet settings[​](#improper-tablet-settings "Direct link to Improper tablet settings") Tablets distribute data across Compute Nodes. Poor bucketing or skewed bucket keys can cause queries to run on only a subset of nodes. Recommendations: * Choose bucket columns that ensure balanced distribution. * Set reasonable bucket numbers (formula: `total data size / (1–5 GB)`). ##### Improper `datacache.partition_duration` settings[​](#improper-datacachepartition_duration-settings "Direct link to improper-datacachepartition_duration-settings") If this value is set to too small, data from “cold” partitions may not be cached, causing repeated remote reads. In Query Profile, if `CompressedBytesReadRemote` or `IOCountRemote` is non-zero, this may be the reason. Tune `datacache.partition_duration` accordingly. #### Why do all queries under a warehouse time out with errors like “Timeout was reached” or “Deadline Exceeded”?[​](#why-do-all-queries-under-a-warehouse-time-out-with-errors-like-timeout-was-reached-or-deadline-exceeded "Direct link to Why do all queries under a warehouse time out with errors like “Timeout was reached” or “Deadline Exceeded”?") Check whether the Compute Nodes under the warehouse can access object storage endpoint. #### How to retrieve tablet metadata in shared-data clusters?[​](#how-to-retrieve-tablet-metadata-in-shared-data-clusters "Direct link to How to retrieve tablet metadata in shared-data clusters?") 1. Obtain visible version by running: ```sql SHOW PARTITIONS FROM ` ``` 2. Execute the following statement to retrieve tablet metadata: ```sql admin execute on 'System.print(StorageEngine.get_lake_tablet_metadata_json(, ))' ``` #### Why is loading slow during high-frequency ingestion?[​](#why-is-loading-slow-during-high-frequency-ingestion "Direct link to Why is loading slow during high-frequency ingestion?") StarRocks serializes transaction commit, so high ingestion rates may hit limits. Monitor the following aspects: * **Loading queue**: If loading queue is full, you can increase I/O worker threads. * **Publish Version latency**: High publish times cause ingestion delays. #### How does compaction work in shared-data clusters, and why does it get stuck?[​](#how-does-compaction-work-in-shared-data-clusters-and-why-does-it-get-stuck "Direct link to How does compaction work in shared-data clusters, and why does it get stuck?") Key behaviors include: * Compaction is scheduled by FE and executed by CN. * Each compaction produces a new version and goes through the process of Write, Commit, and Publish. * FE does not track runtime compaction tasks; stale tasks on BE may block new ones. To clean up stuck compaction tasks, follow these steps: 1. Check version information of the partition, and compare `CompactVersion` and `VisibleVersion`. ```sql SHOW PARTITIONS FROM ``` 2. Check compaction task status. ```sql SHOW PROC '/compactions' ``` ```sql SELECT * FROM information_schema.be_cloud_native_compactions WHERE TXN_ID = ``` 3. Cancel expired tasks. a. Disable compaction and migration. ```sql ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0"); ADMIN SET FRONTEND CONFIG ('tablet_sched_disable_balance' = 'true'); ADMIN SHOW FRONTEND CONFIG LIKE 'lake_compaction_max_tasks'; ADMIN SHOW FRONTEND CONFIG LIKE 'tablet_sched_disable_balance'; ``` b. Restart all BE nodes c. Verify all compactions have failed. ```sql SHOW PROC '/compactions' ``` d. Re-enable compaction & migration. ```sql ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1"); ADMIN SET FRONTEND CONFIG ('tablet_sched_disable_balance' = 'false'); ADMIN SHOW FRONTEND CONFIG LIKE 'lake_compaction_max_tasks'; ADMIN SHOW FRONTEND CONFIG LIKE 'tablet_sched_disable_balance'; ``` #### Why is compaction slow in Kubernetes shared-data clusters?[​](#why-is-compaction-slow-in-kubernetes-shared-data-clusters "Direct link to Why is compaction slow in Kubernetes shared-data clusters?") If ingestion happens in one warehouse but compaction runs in another (for example, `default_warehouse`), compaction must pull data across warehouses with no cache, slowing it down. Solution: * Set BE configuration `lake_enable_vertical_compaction_fill_data_cache` to `true`. * Perform writes in the same warehouse as compaction. #### Why does storage usage appear inflated in shared-data clusters?[​](#why-does-storage-usage-appear-inflated-in-shared-data-clusters "Direct link to Why does storage usage appear inflated in shared-data clusters?") Storage usage on object storage includes all historical versions, while `SHOW DATA` outputs only reflect the latest version. However, if the difference is excessively large, it could be caused by the following issues: * Compaction may be lagging. * Vacuum tasks may be backlogged (check queue size). You may consider tuning compaction or vacuum thread pools if necessary. #### Why do big queries fail with “meta does not exist?[​](#why-do-big-queries-fail-with-meta-does-not-exist "Direct link to Why do big queries fail with “meta does not exist?") The query may be using a data version already compacted and vacuumed. To resolve this, you can increase file retention by modifying the FE configuration `lake_autovacuum_grace_period_minutes`, and then retry the query. #### What causes excessive small files in object storage?[​](#what-causes-excessive-small-files-in-object-storage "Direct link to What causes excessive small files in object storage?") Excessive number of small files may cause performance degradation. Common causes include: * Too many buckets caused by improper bucket settings. * High ingestion frequency causing very small individual loads. Compaction will eventually merge small files, but tuning bucket count and batching ingestion helps prevent performance degradation. --- ### SQL query This topic provides answers to some frequently asked questions about SQL. #### Does StarRocks support caching query results?[​](#does-starrocks-support-caching-query-results "Direct link to Does StarRocks support caching query results?") StarRocks does not directly cache final query results. From v2.5 onwards, StarRocks uses the Query Cache feature to save the intermediate results of first-stage aggregation in the cache. New queries that are semantically equivalent to previous queries can reuse the cached computation results to accelerate computations. Query cache uses BE memory. For more information, see [Query cache](https://docs.starrocks.io/docs/using_starrocks/caching/query_cache.md). #### When a `Null` is included in the calculation, the calculation results of functions are false except for the ISNULL() function[​](#when-a-null-is-included-in-the-calculation-the-calculation-results-of-functions-are-false-except-for-the-isnull-function "Direct link to when-a-null-is-included-in-the-calculation-the-calculation-results-of-functions-are-false-except-for-the-isnull-function") In standard SQL, every calculation that includes an operand with a `NULL` value returns a `NULL`. #### Does StarRocks support the DECODE function?[​](#does-starrocks-support-the-decode-function "Direct link to Does StarRocks support the DECODE function?") StarRocks does not support the DECODE function of the Oracle database. StarRocks is compatible with MySQL, so you can use the CASE WHEN statement. #### Can the latest data be queried immediately after data is loaded into the Primary Key table of StarRocks?[​](#can-the-latest-data-be-queried-immediately-after-data-is-loaded-into-the-primary-key-table-of-starrocks "Direct link to Can the latest data be queried immediately after data is loaded into the Primary Key table of StarRocks?") Yes. StarRocks merges data in a way that references Google Mesa. In StarRocks, a BE triggers the data merge and it has two kinds of compaction to merge data. If the data merge is not completed, it is finished during your query. Therefore, you can read the latest data after data loading. #### Do the utf8mb4 characters stored in StarRocks get truncated or appear garbled?[​](#do-the-utf8mb4-characters-stored-in-starrocks-get-truncated-or-appear-garbled "Direct link to Do the utf8mb4 characters stored in StarRocks get truncated or appear garbled?") No. #### This error "table's state is not normal" occurs when I run the `alter table` command[​](#this-error-tables-state-is-not-normal-occurs-when-i-run-the-alter-table-command "Direct link to this-error-tables-state-is-not-normal-occurs-when-i-run-the-alter-table-command") This error occurs because the previous alteration has not been completed. You can run the following code to check the status of the previous alteration: ```sql show tablet from lineitem where State="ALTER"; ``` The time spent on the alteration operation relates to the data volume. In general, the alteration can be completed in minutes. We recommend that you stop loading data into StarRocks while you are altering tables because data loading lowers the speed at which alteration completes. #### This error "get hive partition meta data failed: java.net.UnknownHostException:hadooptest" occurs when I query the external tables of Apache Hive[​](#this-error-get-hive-partition-meta-data-failed-javanetunknownhostexception-occurs-when-i-query-the-external-tables-of-apache-hive "Direct link to this-error-get-hive-partition-meta-data-failed-javanetunknownhostexception-occurs-when-i-query-the-external-tables-of-apache-hive") This error occurs when the metadata of Apache Hive partitions cannot be obtained. To solve this problem, copy **core-sit.xml** and **hdfs-site.xml** to the **fe.conf** file and the **be.conf** file. #### This error "planner use long time 3000 remaining task num 1" occurs when I query data[​](#this-error-planner-use-long-time-3000-remaining-task-num-1-occurs-when-i-query-data "Direct link to This error \"planner use long time 3000 remaining task num 1\" occurs when I query data") This error occurs usually due to a full garbage collection (full GC), which can be checked by using backend monitoring and the **fe.gc** log. To solve this problem, perform one of the following operations: * Allows SQL's client to access multiple frontends (FEs) simultaneously to spread the load. * Change the heap size of Java Virtual Machine (JVM) from 8 GB to 16 GB in the **fe.conf** file to increase memory and reduce the impact of full GC. #### When cardinality of column A is small, the query results of `select B from tbl order by A limit 10` vary each time[​](#when-cardinality-of-column-a-is-small-the-query-results-of-select-b-from-tbl-order-by-a-limit-10-vary-each-time "Direct link to when-cardinality-of-column-a-is-small-the-query-results-of-select-b-from-tbl-order-by-a-limit-10-vary-each-time") SQL can only guarantee that column A is ordered, and it cannot guarantee that the order of column B is the same for each query. MySQL can guarantee the order of column A and column B because it is a standalone database. StarRocks is a distributed database, of which data stored in the underlying table is in a sharding pattern. The data of column A is distributed across multiple machines, so the order of column B returned by multiple machines may be different for each query, resulting in inconsistent order of B each time. To solve this problem, change `select B from tbl order by A limit 10` to `select B from tbl order by A,B limit 10`. #### Why is there a large gap in column efficiency between SELECT \* and SELECT?[​](#why-is-there-a-large-gap-in-column-efficiency-between-select--and-select "Direct link to Why is there a large gap in column efficiency between SELECT * and SELECT?") To solve this problem, check the profile and see MERGE details: * Check whether the aggregation on the storage layer takes up too much time. * Check whether there are too many indicator columns. If so, aggregate hundreds of columns of millions of rows. ```plaintext MERGE: - aggr: 26s270ms - sort: 15s551ms ``` #### Does DELETE support nested functions?[​](#does-delete-support-nested-functions "Direct link to Does DELETE support nested functions?") Nested functions are not supported, such as `to_days(now())` in `DELETE from test_new WHERE to_days(now())-to_days(publish_time) >7;`. #### How to improve the usage efficiency of a database when there are hundreds of tables in it?[​](#how-to-improve-the-usage-efficiency-of-a-database-when-there-are-hundreds-of-tables-in-it "Direct link to How to improve the usage efficiency of a database when there are hundreds of tables in it?") To improve efficiency, add the `-A` parameter when you connect to MySQL's client server: `mysql -uroot -h127.0.0.1 -P8867 -A`. MySQL's client server does not pre-read database information. #### How to reduce the disk space occupied by the BE log and the FE log?[​](#how-to-reduce-the-disk-space-occupied-by-the-be-log-and-the-fe-log "Direct link to How to reduce the disk space occupied by the BE log and the FE log?") Adjust the log level and corresponding parameters. For more information, see [Parameter Configuration](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md). #### This error "table \*\*\* is colocate table, cannot change replicationNum" occurs when I modify the replication number[​](#this-error-table--is-colocate-table-cannot-change-replicationnum-occurs-when-i-modify-the-replication-number "Direct link to This error \"table *** is colocate table, cannot change replicationNum\" occurs when I modify the replication number") When you create colocated tables, you need to set the `group` property. Therefore, you cannot modify the replication number for a single table. You can perform the following steps to modify the replication number for all tables in a group: 1. Set `group_with` to `empty` for all tables in a group. 2. Set a proper `replication_num` for all tables in a group. 3. Set `group_with` back to its original value. #### Does setting VARCHAR to the maximum value affect storage?[​](#does-setting-varchar-to-the-maximum-value-affect-storage "Direct link to Does setting VARCHAR to the maximum value affect storage?") VARCHAR is a variable-length data type, which has a specified length that can be changed based on the actual data length. Specifying a different varchar length when you create a table has little impact on the query performance on the same data. #### This error "create partititon timeout" occurs when I truncate a table[​](#this-error-create-partititon-timeout-occurs-when-i-truncate-a-table "Direct link to This error \"create partititon timeout\" occurs when I truncate a table") To truncate a table, you need to create the corresponding partitions and then swap them. If there are a larger number of partitions that need to be created, this error occurs. In addition, if there are many data load tasks, the lock will be held for a long time during the compaction process. Therefore, the lock cannot be acquired when you create tables. If there are too many data load tasks, set `tablet_map_shard_size` to `512` in the **be.conf** file to reduce the lock contention. #### This error "Failed to specify server's Kerberos principal name" occurs when I access external tables of Apache Hive[​](#this-error-failed-to-specify-servers-kerberos-principal-name-occurs-when-i-access-external-tables-of-apache-hive "Direct link to This error \"Failed to specify server's Kerberos principal name\" occurs when I access external tables of Apache Hive") Add the following information to **hdfs-site.xml** in the **fe.conf** file and the **be.conf** file: ```html dfs.namenode.kerberos.principal.pattern * ``` #### Is "2021-10" a date format in StarRocks?[​](#is-2021-10-a-date-format-in-starrocks "Direct link to Is \"2021-10\" a date format in StarRocks?") No. #### Can "2021-10" be used as a partition field?[​](#can-2021-10-be-used-as-a-partition-field "Direct link to Can \"2021-10\" be used as a partition field?") No, use functions to change "2021-10" to "2021-10-01" and then use "2021-10-01" as a partition field. #### Where can I query the size of a StarRocks database or table?[​](#where-can-i-query-the-size-of-a-starrocks-database-or-table "Direct link to Where can I query the size of a StarRocks database or table?") You can use the [SHOW DATA](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATA.md) command. `SHOW DATA;` displays the data size and replicas of all tables in the current database. `SHOW DATA FROM .;` displays the data size, number of replicas, and number of rows in a specified table of a specified database. #### In StarRocks on ES, when creating an Elasticsearch external table, if the relevant string length is too long, exceeding 256, and Elasticsearch uses dynamic mapping, using a select statement will result in the inability to query that column[​](#in-starrocks-on-es-when-creating-an-elasticsearch-external-table-if-the-relevant-string-length-is-too-long-exceeding-256-and-elasticsearch-uses-dynamic-mapping-using-a-select-statement-will-result-in-the-inability-to-query-that-column "Direct link to In StarRocks on ES, when creating an Elasticsearch external table, if the relevant string length is too long, exceeding 256, and Elasticsearch uses dynamic mapping, using a select statement will result in the inability to query that column") In dynamic mapping, Elasticsearch's data type is ```json "k4": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } ``` StarRocks uses the keyword data type to convert the query statement. Since the keyword length of the column exceeds 256, the column cannot be queried. Solution: Remove the field mapping ```json "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } ``` to use the text type instead. #### How to quickly count the size of StarRocks databases and tables and the disk resources they occupy?[​](#how-to-quickly-count-the-size-of-starrocks-databases-and-tables-and-the-disk-resources-they-occupy "Direct link to How to quickly count the size of StarRocks databases and tables and the disk resources they occupy?") You can use the [SHOW DATA](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATA.md) command to view the storage size of databases and tables. `SHOW DATA;` displays the data volume and replica count of all tables in the current database. `SHOW DATA FROM .;` displays the data volume, replica count, and row count of a specific table in a specified database. #### Why does using a function on a partition key slow down queries?[​](#why-does-using-a-function-on-a-partition-key-slow-down-queries "Direct link to Why does using a function on a partition key slow down queries?") Using a function on a partition key can lead to inaccurate partition pruning, thereby reducing query performance. #### Why doesn't the DELETE statement support nested functions?[​](#why-doesnt-the-delete-statement-support-nested-functions "Direct link to Why doesn't the DELETE statement support nested functions?") ```sql mysql > DELETE FROM starrocks.ods_sale_branch WHERE create_time >= concat(substr(202201,1,4),'01') and create_time <= concat(substr(202301,1,4),'12'); SQL Error [1064][42000]: Right expr of binary predicate should be value ``` BINARY predicates must be of the `column op literal` type and cannot be expressions. There are currently no plans to support expressions as comparison values. #### How to name columns with reserved keywords?[​](#how-to-name-columns-with-reserved-keywords "Direct link to How to name columns with reserved keywords?") Reserved keywords (e.g., `rank`) need to be escaped, such as using `` `rank` ``. #### How to stop an executing SQL?[​](#how-to-stop-an-executing-sql "Direct link to How to stop an executing SQL?") You can use `show processlist;` to view executing SQL and use `kill ;` to terminate the corresponding SQL. You can also view and manage through `SHOW PROC '/current_queries';`. #### How to clean up idle connections?[​](#how-to-clean-up-idle-connections "Direct link to How to clean up idle connections?") You can control the timeout for idle connections through the session variable `wait_timeout` (unit: seconds). MySQL automatically cleans up idle connections after about 8 hours by default. #### Are multiple SQL segments in UNION ALL executed in parallel?[​](#are-multiple-sql-segments-in-union-all-executed-in-parallel "Direct link to Are multiple SQL segments in UNION ALL executed in parallel?") Yes, they are executed in parallel. #### What should be done if a SQL causes BE to crash?[​](#what-should-be-done-if-a-sql-causes-be-to-crash "Direct link to What should be done if a SQL causes BE to crash?") 1. Based on the `be.out` error stack, find the `query_id` that caused the crash. 2. Find the corresponding SQL in `fe.audit.log` using the `query_id`. Please collect and send the following information to the support team: * `be.out` log * Run `pstack $be_pid > pstack.log` to execute SQL. * Core Dump file Steps to collect Core files: 1. Get the corresponding BE process: ```bash ps aux| grep be ``` 2. Set the Core file size limit to unlimited. ```bash prlimit -p $bePID --core=unlimited:unlimited ``` Verify if the size limit is unlimited. ```bash cat /proc/$bePID/limits ``` If it is not `0`, the system will generate a Core file in the root directory of the BE deployment when the process crashes. #### How to use Hints to control table join optimizer behavior?[​](#how-to-use-hints-to-control-table-join-optimizer-behavior "Direct link to How to use Hints to control table join optimizer behavior?") Supports `broadcast` and `shuffle` Hints. For example: * `select * from a join [broadcast] b on a.id = b.id;` * `select * from a join [shuffle] b on a.id = b.id;` #### How to increase SQL query concurrency?[​](#how-to-increase-sql-query-concurrency "Direct link to How to increase SQL query concurrency?") By adjusting the session variable `pipeline_dop`. #### How to check the execution progress of DDL?[​](#how-to-check-the-execution-progress-of-ddl "Direct link to How to check the execution progress of DDL?") * View all column modification tasks in the default database: ```sql SHOW ALTER TABLE COLUMN; ``` * View the most recent column modification task for a specific table: ```sql SHOW ALTER TABLE COLUMN WHERE TableName="table1" ORDER BY CreateTime DESC LIMIT 1; ``` #### Why does comparing floating-point numbers sometimes result in inconsistent query results?[​](#why-does-comparing-floating-point-numbers-sometimes-result-in-inconsistent-query-results "Direct link to Why does comparing floating-point numbers sometimes result in inconsistent query results?") Directly using floating-point numbers `=` for comparison can lead to instability due to errors. It is recommended to use range checks. #### Why does floating-point calculation result in errors?[​](#why-does-floating-point-calculation-result-in-errors "Direct link to Why does floating-point calculation result in errors?") FLOAT/DOUBLE types have precision errors in `avg`, `sum`, and other calculations, leading to potentially inconsistent query results. For high precision, use the DECIMAL type, but note that performance will decrease by 2-3 times. #### Why does ORDER BY in a subquery not take effect?[​](#why-does-order-by-in-a-subquery-not-take-effect "Direct link to Why does ORDER BY in a subquery not take effect?") In distributed execution, if ORDER BY is not specified in the outer layer of the subquery, global ordering cannot be guaranteed. This is expected behavior. #### Why is the result of row\_number() inconsistent across multiple executions?[​](#why-is-the-result-of-row_number-inconsistent-across-multiple-executions "Direct link to Why is the result of row_number() inconsistent across multiple executions?") If the ORDER BY field has duplicates (e.g., multiple rows with the same `createTime`), SQL standards do not guarantee stable sorting. It is recommended to include a unique field (e.g., `employee_id`) in the ORDER BY to ensure stability. #### What information is needed for SQL optimization or troubleshooting?[​](#what-information-is-needed-for-sql-optimization-or-troubleshooting "Direct link to What information is needed for SQL optimization or troubleshooting?") * `EXPLAIN COSTS ` (includes statistics) * `EXPLAIN VERBOSE ` (includes data types, nullable, optimization strategies) * Query Profile (viewable through the FE Web interface at `http://:` and navigating to the Queries Tab) * Query Dump (obtained via HTTP API) ```bash wget --user=${username} --password=${password} --post-file ${query_file} http://${fe_host}:${fe_http_port}/api/query_dump?db=${database} -O ${dump_file} ``` Query Dump includes the following information: * Query statement * Table schema referenced in the query * Session variables * Number of BEs * Statistics (Min, Max values) * Exception information (exception stack) #### How to check data skew?[​](#how-to-check-data-skew "Direct link to How to check data skew?") Use `ADMIN SHOW REPLICA DISTRIBUTION FROM
` to view the distribution of tablets. #### How to troubleshoot memory-related errors?[​](#how-to-troubleshoot-memory-related-errors "Direct link to How to troubleshoot memory-related errors?") There are three common scenarios: * **Single query memory limit exceeded:** * Error: `Mem usage has exceed the limit of single query, You can change the limit by set session variable exec_mem_limit.` * Solution: Adjust `exec_mem_limit` * **Query pool memory limit exceeded:** * Error: `Mem usage has exceed the limit of query pool` * Solution: Optimize the SQL. * **BE total memory limit exceeded:** * Error: `Mem usage has exceed the limit of BE` * Solution: Analyze memory usage. Memory analysis methods: ```bash curl -XGET -s http://BE_IP:BE_HTTP_PORT/metrics | grep "^starrocks_be_.*_mem_bytes\|^starrocks_be_tcmalloc_bytes_in_use" curl -XGET -s http://BE_IP:BE_HTTP_PORT/mem_tracker ``` *** #### What to do when encountering the error `StarRocks planner use long time xxx ms in logical phase`?[​](#what-to-do-when-encountering-the-error-starrocks-planner-use-long-time-xxx-ms-in-logical-phase "Direct link to what-to-do-when-encountering-the-error-starrocks-planner-use-long-time-xxx-ms-in-logical-phase") 1. Analyze `fe.gc.log` to check for Full GC occurrences. 2. If the SQL execution plan is complex, increase `new_planner_optimize_timeout` (unit: ms): ```sql set global new_planner_optimize_timeout = 6000; ``` #### How to troubleshoot Unknown Error?[​](#how-to-troubleshoot-unknown-error "Direct link to How to troubleshoot Unknown Error?") Try adjusting the following parameters one by one and then re-execute the SQL: ```sql set disable_join_reorder = true; set enable_global_runtime_filter = false; set enable_query_cache = false; set cbo_enable_low_cardinality_optimize = false; ``` Then collect EXPLAIN COSTS, EXPLAIN VERBOSE, PROFILE, and Query Dump, and provide them to the support team. #### What time zone does `select now()` return?[​](#what-time-zone-does-select-now-return "Direct link to what-time-zone-does-select-now-return") It returns the time zone specified by the `time_zone` system variable. FE/BE logs use the machine's local time zone. #### Why does SQL slow down under high concurrency even when resources are normal?[​](#why-does-sql-slow-down-under-high-concurrency-even-when-resources-are-normal "Direct link to Why does SQL slow down under high concurrency even when resources are normal?") The reason is high network or RPC latency. You can adjust the BE parameter `brpc_connection_type` to `pooled` and then restart BE. #### How to disable statistics collection?[​](#how-to-disable-statistics-collection "Direct link to How to disable statistics collection?") * Disable automatic collection: ```sql enable_statistic_collect = false; ``` * Disable import-triggered collection: ```sql enable_statistic_collect_on_first_load = false; ``` * For versions upgraded to v3.3 and above, manually set: ```sql set global analyze_mv = ""; ``` --- ### FAQ: Hadoop 3.4.3 Wildfly Native SSL Library Issue in StarRocks #### Background[​](#background "Direct link to Background") StarRocks upgraded its bundled Hadoop dependency from 3.4.2 to 3.4.3 ([PR #69503](https://github.com/StarRocks/starrocks/pull/69503)). Hadoop 3.4.3 includes [HADOOP-19719](https://issues.apache.org/jira/browse/HADOOP-19719), which upgrades the Wildfly OpenSSL bindings from `2.1.4.Final` to `2.2.5.Final` to add OpenSSL 3.0 compatibility. However, the new native libraries (`libwfssl.so`) were built against **GLIBC 2.34+**, which introduces compatibility issues on older Linux distributions. **Affected StarRocks versions:** 4.1.0+, 4.0.7+, 3.5.14+ ##### Typical Symptoms[​](#typical-symptoms "Direct link to Typical Symptoms") When this issue occurs, CN/BE nodes crash with a **SIGSEGV** (segmentation fault) during SSL context initialization. The FE reports errors like: ```text SQL Error [1064] [42000]: Access storage error. Error message: failed to get file schema: A error occurred: errorCode=2001 errorMessage:Channel inactive error! ``` From [GitHub Issue #70478](https://github.com/StarRocks/starrocks/issues/70478) — querying Parquet files on Azure Data Lake via `FILES()` function: ```text *** SIGSEGV (@0x0) received by PID (TID 0x...) *** @ 0x7b9b67093453 SSL_CTX_new_ex @ 0x7b9b6a09d95a Java_org_wildfly_openssl_SSLImpl_makeSSLContext0 @ 0x7b9a68544be1 (unknown) ``` The Wildfly JNI call `Java_org_wildfly_openssl_SSLImpl_makeSSLContext0` loads `libwfssl.so`, which is compiled against the OpenSSL 3.x ABI. Inside the `starrocks_be` process, however, OpenSSL symbol lookups from `libwfssl.so` are satisfied by the **OpenSSL 1.x symbols that are statically linked into the `starrocks_be` binary** (from StarRocks' thirdparty) before they can ever reach the system's OpenSSL 3.x shared libraries on disk. A 3.x call site (`SSL_CTX_new_ex`) dispatching into a 1.x implementation is what actually crashes the JVM. This is why merely installing OpenSSL 3.x or its development headers on the host does NOT fix the crash -- the BE's in-process OpenSSL 1.x always wins symbol resolution. *** #### Q1: What is the Wildfly OpenSSL native library?[​](#q1-what-is-the-wildfly-openssl-native-library "Direct link to Q1: What is the Wildfly OpenSSL native library?") Hadoop uses the [Wildfly OpenSSL](https://github.com/wildfly-security/wildfly-openssl) library to bind native OpenSSL to the Java JSSE (Java Secure Socket Extension) APIs. This allows Hadoop's S3A, Azure Blob (ABFS), and Azure Data Lake (ADL) connectors to use native OpenSSL for TLS/SSL connections instead of the JVM's built-in SSL implementation. When it works, this provides a performance benefit for high-throughput encrypted I/O. *** #### Q2: What went wrong in Hadoop 3.4.3?[​](#q2-what-went-wrong-in-hadoop-343 "Direct link to Q2: What went wrong in Hadoop 3.4.3?") Hadoop 3.4.3 upgraded Wildfly OpenSSL to `2.2.5.Final` to support OpenSSL 3.0 ([HADOOP-19719](https://issues.apache.org/jira/browse/HADOOP-19719)). The new native library (`libwfssl.so`) was built against **GLIBC 2.34+** and targets the **OpenSSL 3.x ABI**. Because StarRocks' `starrocks_be` binary still statically links OpenSSL 1.x from its thirdparty, this produces two distinct failure modes: | Platform | GLIBC Version | System OpenSSL | Failure Mode | | ------------------------- | ------------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **RHEL 8 / CentOS 8** | 2.28 | 1.1.1 | **Load-time failure**: `UnsatisfiedLinkError: GLIBC_2.34 not found` -- `libwfssl.so` cannot load at all. | | **Ubuntu 22.04** | 2.35 | 3.0.2 | **Runtime ABI collision**: `libwfssl.so` loads, but its OpenSSL symbol lookups resolve to the static OpenSSL 1.x embedded in `starrocks_be`, not the system's OpenSSL 3.x. Crashes in `SSL_CTX_new_ex`. Installing `libssl-dev` does NOT fix this. | | **RHEL 9 / Ubuntu 24.04** | 2.34+ | 3.0+ | Same runtime ABI collision as Ubuntu 22.04: the BE's in-process OpenSSL 1.x shadows the system OpenSSL 3.x. Installing `openssl-devel` does NOT fix this either. | On affected systems, the Wildfly native library crash can cause JVM instability (including `hs_err` crash logs on CN/BE nodes), failed SSL/TLS connections to cloud storage (AWS S3, Azure Blob, Azure Data Lake), and silent fallback to unencrypted or broken connections. *** #### Q3: How do I fix this?[​](#q3-how-do-i-fix-this "Direct link to Q3: How do I fix this?") There are multiple solutions depending on your environment and requirements. ##### Solution 1: Disable Wildfly Native SSL[​](#solution-1-disable-wildfly-native-ssl "Direct link to Solution 1: Disable Wildfly Native SSL") Force Hadoop to use the JVM's built-in SSL implementation instead of the native Wildfly/OpenSSL path. This is the **safest and most portable fix**. Add the following to your `core-site.xml` on **all CN/BE nodes**: ```xml fs.s3a.ssl.channel.mode default_jsse adl.ssl.channel.mode Default_JSSE fs.azure.ssl.channel.mode Default_JSSE ``` **Where to place `core-site.xml`:** * For StarRocks BE/CN: `$STARROCKS_HOME/conf/core-site.xml` * For Broker processes: `$BROKER_HOME/conf/core-site.xml` **Restart all affected services** after making the change. ##### Solution 2: Remove the `wildfly-openssl` JAR from the BE Package[​](#solution-2-remove-the-wildfly-openssl-jar-from-the-be-package "Direct link to solution-2-remove-the-wildfly-openssl-jar-from-the-be-package") The root cause is that `libwfssl.so` (shipped inside `wildfly-openssl-*.Final.jar`) is compiled for the OpenSSL 3.x ABI, while the `starrocks_be` binary still statically links OpenSSL 1.x from the StarRocks thirdparty. Inside the BE process, OpenSSL symbol lookups from `libwfssl.so` are resolved against the 1.x symbols baked into `starrocks_be` before they can reach any system OpenSSL 3.x shared library, and the 3.x-call-into-1.x-implementation dispatch crashes in `SSL_CTX_new_ex`. Installing `libssl-dev` on Ubuntu or `openssl-devel` on RHEL cannot change this: even if those packages upgrade the host's OpenSSL to 3.x, the BE's in-process OpenSSL 1.x still wins symbol resolution every time. Treat any older guidance that suggests `apt install libssl-dev` or `yum install openssl-devel` as ineffective for this crash. A reliable workaround is to delete the offending JAR so Hadoop cannot load the native library at all and falls back to JSSE automatically: ```bash rm -f $STARROCKS_HOME/lib/hadoop/common/wildfly-openssl-2.2.5.Final.jar ``` Run this on every CN/BE node (and on every Broker node if a Broker is deployed) and restart the service. Starting from StarRocks builds that include the fix for [issue #71898](https://github.com/StarRocks/starrocks/issues/71898), the dependency is excluded at the Maven level in `java-extensions/pom.xml` (for `hadoop-common`, `hadoop-aws`, `hadoop-azure`, and `hadoop-azure-datalake`), so the JAR is never produced by the `java-extensions/hadoop-lib` assembly. `build.sh` additionally `rm -rf`s `${STARROCKS_OUTPUT}/be/lib/hadoop/common/wildfly-openssl-2.2.5.Final.jar` from the BE output as a belt-and-braces safety net. No manual cleanup is required after upgrading. *** #### Q4: Which cloud storage connectors are affected?[​](#q4-which-cloud-storage-connectors-are-affected "Direct link to Q4: Which cloud storage connectors are affected?") | Connector | Configuration Property | Recommended Value | | ----------------------------- | --------------------------- | ----------------- | | **AWS S3** (S3A) | `fs.s3a.ssl.channel.mode` | `default_jsse` | | **Azure Data Lake** (ADL) | `adl.ssl.channel.mode` | `Default_JSSE` | | **Azure Blob Storage** (ABFS) | `fs.azure.ssl.channel.mode` | `Default_JSSE` | note The Azure Data Lake Store SDK also supports `AdlStoreOptions.setSSLChannelMode()` programmatically, but for StarRocks the `core-site.xml` approach is the standard method. *** #### Q5: Is there a performance impact from disabling native SSL?[​](#q5-is-there-a-performance-impact-from-disabling-native-ssl "Direct link to Q5: Is there a performance impact from disabling native SSL?") Yes, but it is minor for most workloads. The native OpenSSL path (via Wildfly) can provide better throughput for large-scale encrypted I/O operations. However, the JVM's built-in JSSE implementation is well-optimized in modern JDKs (Java 11+) and is sufficient for most StarRocks data lake query patterns. If you are on a platform that supports the new Wildfly native library (RHEL 9 / Ubuntu 24.04 with GLIBC 2.34+ and OpenSSL 3.0), you can keep the `default` or `openssl` mode for maximum performance. *** #### Q6: How do I verify the fix is working?[​](#q6-how-do-i-verify-the-fix-is-working "Direct link to Q6: How do I verify the fix is working?") After applying the configuration change and restarting services: 1. **Re-run the failing workload** (e.g., query against external S3/Azure tables). 2. **Check BE/CN logs** for the absence of: * `hs_err_pid*.log` files (JVM crash logs) * `UnsatisfiedLinkError` mentioning `GLIBC_2.34` or `libwfssl.so` * `org.wildfly.openssl` error stack traces 3. **Confirm successful SSL connections** in the log: * With `default_jsse`: You should see standard JSSE handshake logs (no Wildfly references). * With `default`: You may see `Failed to load OpenSSL. Falling back to the JSSE` (this is expected and safe). *** #### Q7: Does this affect RHEL 8 / CentOS 8 specifically?[​](#q7-does-this-affect-rhel-8--centos-8-specifically "Direct link to Q7: Does this affect RHEL 8 / CentOS 8 specifically?") Yes. RHEL 8 ships with **GLIBC 2.28**, which is significantly older than the 2.34 required by the new Wildfly native library. On RHEL 8: * The native library **will not load at all** (`UnsatisfiedLinkError`). * The Hadoop 3.4.3 release notes explicitly state: *"as the native libraries were built with GLIBC 2.34+, these do not work on RHEL8. For deployment on those systems, stick with the JVM ssl support."* **Recommendation for RHEL 8:** Always use `default_jsse` mode. Plan migration to RHEL 9 for long-term support. *** #### Q8: What about FIPS-compliant environments?[​](#q8-what-about-fips-compliant-environments "Direct link to Q8: What about FIPS-compliant environments?") Linux distributions with a FIPS-compliant SSL library may not be compatible with the Wildfly native OpenSSL bindings. If you are running in a FIPS-compliant environment, **always use `default_jsse`** (JVM SSL) unless you have verified that the native library is compatible with your specific FIPS OpenSSL implementation. *** #### Q9: What has StarRocks done to mitigate this?[​](#q9-what-has-starrocks-done-to-mitigate-this "Direct link to Q9: What has StarRocks done to mitigate this?") 1. **Exclude `wildfly-openssl` at the Java-extensions build layer** ([issue #71898](https://github.com/StarRocks/starrocks/issues/71898)): recent StarRocks builds add `org.wildfly.openssl:wildfly-openssl` to the `hadoop-common`, `hadoop-aws`, `hadoop-azure`, and `hadoop-azure-datalake` dependencies in `java-extensions/pom.xml`, so `java-extensions/hadoop-lib` no longer ships the JAR to the BE at all. For extra safety, `build.sh` also `rm -rf`s any straggling `wildfly-openssl-2.2.5.Final.jar` from `${STARROCKS_OUTPUT}/be/lib/hadoop/common/`. The exclusion is intentionally temporary: it will be restored once StarRocks' thirdparty OpenSSL is upgraded to 3.x, at which point the in-process ABI will match what `libwfssl.so` expects. 2. **Documentation**: this FAQ and related guidance for configuring `core-site.xml`. > Note: an earlier attempt to install `libssl-dev` inside the Ubuntu runtime Docker image ([PR #70688](https://github.com/StarRocks/starrocks/pull/70688)) has been superseded. The crash is caused by `libwfssl.so`'s OpenSSL 3.x call sites dispatching into the **static OpenSSL 1.x embedded in the `starrocks_be` binary**, not by a missing host-side OpenSSL package -- so installing `libssl-dev` / `openssl-devel` cannot change the outcome. *** #### Quick Reference: Minimal Fix[​](#quick-reference-minimal-fix "Direct link to Quick Reference: Minimal Fix") Pick **either** of the following on **every CN/BE node** (and every Broker node, if deployed), then restart the service. Do NOT rely on installing `libssl-dev` or `openssl-devel` -- those packages do not fix this crash. **Option A -- delete the `wildfly-openssl` JAR:** ```bash rm -f $STARROCKS_HOME/lib/hadoop/common/wildfly-openssl-2.2.5.Final.jar ``` **Option B -- force JSSE via `$STARROCKS_HOME/conf/core-site.xml`:** ```xml fs.s3a.ssl.channel.mode default_jsse adl.ssl.channel.mode Default_JSSE fs.azure.ssl.channel.mode Default_JSSE ``` *** #### References[​](#references "Direct link to References") * [HADOOP-19719: Upgrade to wildfly version with support for openssl 3](https://issues.apache.org/jira/browse/HADOOP-19719) * [HADOOP-19262: Upgrade wildfly-openssl for JDK 17+](https://issues.apache.org/jira/browse/HADOOP-19262) * [FLINK-38284: Flink downstream tracking issue](https://issues.apache.org/jira/browse/FLINK-38284) * [Hadoop S3A Troubleshooting Guide](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/troubleshooting_s3a.html) * [Hadoop S3A Performance Guide (SSL Channel Mode)](https://apache.github.io/hadoop/hadoop-aws/tools/hadoop-aws/performance.html) * [Wildfly OpenSSL GitHub Repository](https://github.com/wildfly-security/wildfly-openssl) * [StarRocks Issue #70478: SIGSEGV on Azure ADLS2 query (4.0.7)](https://github.com/StarRocks/starrocks/issues/70478) * [StarRocks PR #69503: Upgrade Hadoop 3.4.2 to 3.4.3](https://github.com/StarRocks/starrocks/pull/69503) * [StarRocks Issue #71898: Drop `wildfly-openssl` from the BE package](https://github.com/StarRocks/starrocks/issues/71898) * [StarRocks PR #70688: Install libssl-dev for Ubuntu runtime (superseded, does not fix the crash)](https://github.com/StarRocks/starrocks/pull/70688) --- ### table_design_faq #### Troubleshooting Table Design[​](#troubleshooting-table-design "Direct link to Troubleshooting Table Design") This topic provides answers to some frequently asked questions about table design. #### What is the maximum length of the VARCHAR type and how does the length affects query performance?[​](#what-is-the-maximum-length-of-the-varchar-type-and-how-does-the-length-affects-query-performance "Direct link to What is the maximum length of the VARCHAR type and how does the length affects query performance?") The maximum length of the VARCHAR type is 65533, which requires 1 MB storage size. It is recommended to set the VARCHAR length to a minimum value that is necessary. It is because, despite that the VARCHAR type data size is based on the actual length, in query scenarios where memory pre-allocation is required, memory resources are allocated based on the pre-defined length of the VARCHAR types instead of the actual length. For example, for an `address` field, 100 bytes is enough, which means that VARCHAR(100) is recommended over STRING, because the STRING type is equivalent to VARCHAR(65533). --- ## Flink_connector ### Releases of StarRocks Connector for Flink #### Notifications[​](#notifications "Direct link to Notifications") **User guide:** * [Load data into StarRocks using Flink connector](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks/) * [Read data from StarRocks using Flink connector](https://docs.starrocks.io/docs/unloading/Flink_connector/) **Source codes:** [starrocks-connector-for-apache-flink](https://github.com/StarRocks/starrocks-connector-for-apache-flink) **Naming format of the JAR file:** * Flink 1.15 and later: `flink-connector-starrocks-${connector_version}_flink-${flink_version}.jar` * Prior to Flink 1.15: `flink-connector-starrocks-${connector_version}_flink-${flink_version}_${scala_version}.jar` **Methods to obtain the JAR file:** * Directly download the the Flink connector JAR file from the [Maven Central Repository](https://repo1.maven.org/maven2/com/starrocks). * Add the Flink connector as a dependency in your Maven project's `pom.xml` file and download it. For specific instructions, see [user guide](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks/#obtain-flink-connector). * Compile the source codes into Flink connector JAR file. For specific instructions, see [user guide](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks/#obtain-flink-connector). **Version requirements:** | Connector | Flink | StarRocks | Java | Scala | | --------- | ----------------------------- | ------------- | ---- | --------- | | 1.2.16 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.15 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.14 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.12 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.11 | 1.15,1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | > **NOTICE** > > In general, the latest version of the Flink connector only maintains compatibility with the three most recent versions of Flink. #### Release notes[​](#release-notes "Direct link to Release notes") ##### 1.2[​](#12 "Direct link to 1.2") ###### 1.2.16[​](#1216 "Direct link to 1.2.16") Release date: September 24, 2026 ###### Features[​](#features "Direct link to Features") * Derive the `json` columns header from the Flink schema. [#510](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/510) ###### Improvements[​](#improvements "Direct link to Improvements") * Multi-table transaction: batch chunk switches to cut redundant tiny loads. [#506](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/506) * Skip the host probe when only one load URL is configured. [#502](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/502) ###### BugFix[​](#bugfix "Direct link to BugFix") * Multi-table transaction: never park the writer on bytes that cannot be flushed before `txnEnd`. [#512](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/) * Multi-table transaction: recover a lost commit response instead of hanging/failing. [#501](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/501) * Multi-table transaction: reconcile `flushQ` region labels to the live shared label. [#500](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/500) * Fix a shared-state race in the query plan visitor. [#499](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/499) ###### 1.2.15[​](#1215 "Direct link to 1.2.15") Release date: June 18, 2026 ###### Features[​](#features-1 "Direct link to Features") * Added multi-table transaction Stream Load support. [#487](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/487) ###### Improvements[​](#improvements-1 "Direct link to Improvements") * Merge Commit supports logging data quality error messages. [#484](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/484) ###### BugFix[​](#bugfix-1 "Direct link to BugFix") * Fix multi-table transaction concurrency: serialize per-table loads and align cross-table commits. [#491](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/491) * Fallback to FE cancel API when rollback fails for PREPARE-state lingering transactions. [#488](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/488) * Do not quote CURRENT\_TIMESTAMP in the DEFAULT clause when building column statements. [#486](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/486) ###### 1.2.14[​](#1214 "Direct link to 1.2.14") Release date: February 11, 2026 ###### Features[​](#features-2 "Direct link to Features") * Supports Merge Commit. [#474](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/474) ###### Improvements[​](#improvements-2 "Direct link to Improvements") * Supports setting `sink.buffer-flush.interval-ms` to lower than 1 second. [#475](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/475) * Supports configurable transaction Publish timeout. [#480](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/480) ###### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") The following issues have been fixed: * Fix CVE-2023-2976 by upgrading guava version to `32.0.1-jre`. [#467](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/467) ###### 1.2.12[​](#1212 "Direct link to 1.2.12") Release date: September 19, 2025 ###### Improvements[​](#improvements-3 "Direct link to Improvements") * Supports specifying a warehouse for the source. [#423](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/423) * Added security policy. [#434](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/434) * Sanitized sensitive data in the error log. [#446](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/446) * Supports configuring `prepared_timeout` for the Stream Load transaction interface. [#453](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/453) ###### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") The following issues have been fixed: * The source reader was not closed if the open failed. [#441](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/441) * False success caused by any exception in StreamLoadManagerV2.flush. [#451](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/451) ###### 1.2.11[​](#1211 "Direct link to 1.2.11") Release data: June 3, 2025 **Features** * Supports LZ4 compression for the CSV format. [#408](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/408) * Adds support for Flink 1.20. [#409](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/409) **Improvements** * Adds an option to disable wrapping JSON into JSON arrays. [#344](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/344) * Updated FastJSON to resolve CVE-2022-25845. [#394](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/394) * Removed data row metrics from warn logs to avoid exposing payload in logs. [#420](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/420) **Bug Fixes** * Wrong pushdown results caused by the shadow clone of StarRocksDynamicTableSource (After the fix, a deep copy of StarRocksDynamicTableSource will be used). [#421](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/421) ###### 1.2.10[​](#1210 "Direct link to 1.2.10") **Features** * Supports reading JSON columns. [#334](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/334) * Supports reading ARRAY, STRUCT, and MAP columns. [#347](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/347) * Supports LZ4 compression when sinking data with the JSON format. [#354](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/354) * Supports Flink 1.19. [#379](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/379) **Improvements** * Supports configuring socket timeout. [#319](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/319) * The Stream Load transaction interface supports asynchronous `prepare` and `commit` operations. [#328](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/328) * Supports mapping a subset of columns in a StarRocks table to a Flink source table. [#352](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/352) * Supports setting a specific warehouse when using the Stream Load transaction interface. [#361](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/361) **Bug Fixes** Fixed the following issues: * `StarRocksSourceBeReader` in `StarRocksDynamicLookupFunction` is not closed after data reading completes. [#351](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/351) * An exception was thrown when loading an empty JSON string into a JSON column. [#380](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/380) ###### 1.2.9[​](#129 "Direct link to 1.2.9") This release includes some features and bug fixes. The notable change is that the Flink connector is integrated with Flink CDC 3.0 to easily build a streaming ELT pipeline from CDC sources (such as MySQL and Kafka) to StarRocks. You can see [Synchronize data with Flink CDC 3.0 (with schema change supported)](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks/#synchronize-data-with-flink-cdc-30-with-schema-change-supported) for details. **Features** * Implement catalog to support Flink CDC 3.0. [#295](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/295) * Implement new sink API in [FLP-191](https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction) to support Flink CDC 3.0. [#301](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/301) * Support Flink 1.18. [#305](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/305) **Bug Fixes** * Fix misleading thread name and log. [#290](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/290) * Fix wrong stream-load-sdk configurations used for writing to multiple tables. [#298](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/298) ###### 1.2.8[​](#128 "Direct link to 1.2.8") This release includes some improvements and bug fixes. The notable changes are as follows: * Support Flink 1.16 and 1.17. * Recommend to set `sink.label-prefix` when the sink is configured to guarantee the exactly-once semantics. For the specific instructions, see [Exactly Once](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks/#exactly-once). **Improvements** * Support to configure whether to use Stream Load transaction interface to guarantee at-least-once. [#228](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/228) * Add retry metrics for sink V1. [#229](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/229) * No need to getLabelState when EXISTING\_JOB\_STATUS is FINISHED. [#231](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/231) * Remove useless stack trace log for sink V1. [#232](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/232) * \[Refactor] Move StarRocksSinkManagerV2 to stream-load-sdk. [#233](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/233) * Automatically detect partial updates according to a Flink table's schema instead of the `sink.properties.columns` parameter explicitly specified by users. [#235](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/235) * \[Refactor] Move probeTransactionStreamLoad to stream-load-sdk. [#240](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/240) * Add git-commit-id-plugin for stream-load-sdk. [#242](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/242) * Use info log for DefaultStreamLoader#close. [#243](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/243) * Support to generate stream-load-sdk JAR file without dependencies. [#245](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/245) * Replace fastjson with jackson in stream-load-sdk. [#247](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/247) * Support to process update\_before record. [#250](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/250) * Add the Apache license into files. [#251](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/251) * Support to get the exception in stream-load-sdk. [#252](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/252) * Enable `strip_outer_array` and `ignore_json_size` by default. [#259](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/259) * Try to cleanup lingering transactions when a Flink job restores and the sink semantics is exactly-once. [#271](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/271) * Return the first exception after the retrying fails. [#279](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/279) **Bug Fixes** * Fix typos in StarRocksStreamLoadVisitor. [#230](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/230) * Fix the fastjson classloader leak. [#260](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/260) **Tests** * Add the test framework for loading from Kafka to StarRocks. [#249](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/249) **Doc** * Refactor the docs. [#262](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/262) * Improve the doc for the sink. [#268](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/268) [#275](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/275) * Add examples of DataStream API for the sink. [#253](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/253) --- ## Integrations ### Apache Airflow Enables orchestration and scheduling of data workflows with StarRocks using DAGs (Directed Acyclic Graphs) and SQL operators. Use Airflow for data loading and transformation using the `SQLExecuteQueryOperator` and `MySQLHook` without any implementation or complex configuration. [Apache Airflow GitHub repo](https://github.com/apache/airflow). #### Supported features[​](#supported-features "Direct link to Supported features") * SQL Execution through MySQL protocol * Connection management * Transaction support * Parameterized queries * Task dependencies * Retry logic #### Installation[​](#installation "Direct link to Installation") ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Apache Airflow 2.0+ or 3.0+ * Python 3.8+ * Access to a StarRocks cluster (see the [quickstart guide](https://docs.starrocks.io/docs/quick_start/)) ##### Install[​](#install "Direct link to Install") The MySQL provider package is required to use StarRocks as StarRocks uses MySQL protocol. ```sh pip install apache-airflow-providers-mysql ``` Verify the installation by checking the installed providers: ```sh airflow providers list ``` This should list `apache-airflow-providers-mysql` in the output. #### Configuration[​](#configuration "Direct link to Configuration") ##### Create a StarRocks Connection[​](#create-a-starrocks-connection "Direct link to Create a StarRocks Connection") Create a StarRocks connection in the Airflow UI or via environment variable. The name of the connection will be used by the DAGs later. ###### Via Airflow UI[​](#via-airflow-ui "Direct link to Via Airflow UI") 1. Navigate to Admin > Connections 2. Click the + button to add a new connection 3. Configure the connection: * Connection Id: `starrocks_default` * Connection Type: MySQL * Host: `your-starrocks-host.com` * Schema: `your_database` * Login: `your_username` * Password: `your_password` * Port: `9030` ###### Via Airflow CLI[​](#via-airflow-cli "Direct link to Via Airflow CLI") ```sh airflow connections add 'starrocks_default' \ --conn-type 'mysql' \ --conn-host 'your-starrocks-host.com' \ --conn-schema 'your_database' \ --conn-login 'your_username' \ --conn-password 'your_password' \ --conn-port 9030 ``` #### Usage Examples[​](#usage-examples "Direct link to Usage Examples") These examples demonstrate common patterns for integrating StarRocks with Airflow. Each example builds on core concepts while showcasing different approaches to data loading, transformation, and workflow orchestration. **What You'll Learn:** * **Data Loading**: Efficiently load data from CSV files and cloud storage into StarRocks * **Data Transformation**: Execute SQL queries and process results with Python * **Advanced Patterns**: Implement incremental loading, async operations, and query optimization * **Production Best Practices**: Handle errors gracefully and build resilient pipelines All examples use the crash data tables described in the [quickstart guide](https://docs.starrocks.io/docs/quick_start/shared-nothing.md). ##### Data Loading[​](#data-loading "Direct link to Data Loading") ###### Stream Data Loading[​](#stream-data-loading "Direct link to Stream Data Loading") Load large CSV files efficiently using StarRocks Stream Load API. Stream Load is the recommended approach for: * High-throughput data loading (supports parallel loads) * Loading data with column transformations and filtering Stream Load provides better performance than INSERT INTO VALUES statements for large datasets and includes built-in features like error tolerance. Note that this does require the CSV file is accessible on the Airflow worker's filesystem. ```py from airflow.sdk import dag, task from airflow.hooks.base import BaseHook from datetime import datetime import requests from requests.auth import HTTPBasicAuth from urllib.parse import urlparse class PreserveAuthSession(requests.Session): """ Custom session that preserves Authorization header across redirects. StarRocks FE may redirect Stream Load requests to BE nodes. """ def rebuild_auth(self, prepared_request, response): old = urlparse(response.request.url) new = urlparse(prepared_request.url) # Only preserve auth when redirecting to same hostname if old.hostname == new.hostname: prepared_request.headers["Authorization"] = response.request.headers.get("Authorization") @dag( dag_id="starrocks_stream_load_example", schedule=None, start_date=datetime(2024, 1, 1), catchup=False, tags=["starrocks", "stream_load", "example"], ) def starrocks_stream_load_example(): @task def load_csv_to_starrocks(): # Configuration DATABASE = "quickstart" TABLE = "crashdata" CSV_PATH = "/path/to/crashdata.csv" conn = BaseHook.get_connection("starrocks_default") url = f"http://{conn.host}:{conn.port}/api/{DATABASE}/{TABLE}/_stream_load" # Generate unique label from airflow.sdk import get_current_context context = get_current_context() execution_date = context['logical_date'].strftime('%Y%m%d_%H%M%S') label = f"{TABLE}_load_{execution_date}" headers = { "label": label, "column_separator": ",", "skip_header": "1", "max_filter_ratio": "0.1", # Allow up to 10% error rate "Expect": "100-continue", "columns": """ tmp_CRASH_DATE, tmp_CRASH_TIME, CRASH_DATE=str_to_date(concat_ws(' ', tmp_CRASH_DATE, tmp_CRASH_TIME), '%m/%d/%Y %H:%i'), BOROUGH, ZIP_CODE, LATITUDE, LONGITUDE, LOCATION, ON_STREET_NAME, CROSS_STREET_NAME, OFF_STREET_NAME, NUMBER_OF_PERSONS_INJURED, NUMBER_OF_PERSONS_KILLED, NUMBER_OF_PEDESTRIANS_INJURED, NUMBER_OF_PEDESTRIANS_KILLED, NUMBER_OF_CYCLIST_INJURED, NUMBER_OF_CYCLIST_KILLED, NUMBER_OF_MOTORIST_INJURED, NUMBER_OF_MOTORIST_KILLED, CONTRIBUTING_FACTOR_VEHICLE_1, CONTRIBUTING_FACTOR_VEHICLE_2, CONTRIBUTING_FACTOR_VEHICLE_3, CONTRIBUTING_FACTOR_VEHICLE_4, CONTRIBUTING_FACTOR_VEHICLE_5, COLLISION_ID, VEHICLE_TYPE_CODE_1, VEHICLE_TYPE_CODE_2, VEHICLE_TYPE_CODE_3, VEHICLE_TYPE_CODE_4, VEHICLE_TYPE_CODE_5 """.replace("\n", "").replace(" ", ""), } session = PreserveAuthSession() with open(CSV_PATH, "rb") as f: response = session.put( url, headers=headers, data=f, auth=HTTPBasicAuth(conn.login, conn.password or ""), timeout=3600, ) result = response.json() print(f"\nStream Load Response:") print(f" Status: {result.get('Status')}") print(f" Loaded Rows: {result.get('NumberLoadedRows', 0):,}") if result.get("Status") == "Success": return result else: error_msg = result.get("Message", "Unknown error") raise Exception(f"Stream Load failed: {error_msg}") load_csv_to_starrocks() starrocks_stream_load_example() ``` ###### Insert From Files[​](#insert-from-files "Direct link to Insert From Files") Use StarRocks' [FILES()](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files/) table function to load data directly from files. This approach is ideal for: * Loading data from S3, HDFS, Google Cloud Storage * One-step data ingestion with transformations applied during load * Ad-hoc data loads from various file sources `FILES()` supports multiple file formats and storage systems, making it a flexible alternative to Stream Load for certain use cases. The data is read and inserted in a single SQL statement. ```py from airflow.sdk import dag, task from airflow.providers.mysql.hooks.mysql import MySqlHook from datetime import datetime FILE_PATH = "path_to_file_here" @dag( dag_id='crashdata_dynamic_files_load', schedule='@daily', start_date=datetime(2024, 1, 1), catchup=False, tags=['starrocks', 'files', 'dynamic'], ) def crashdata_files(): @task def load_file(file_path: str): hook = MySqlHook(mysql_conn_id='starrocks_default') sql = f""" INSERT INTO crashdata ( CRASH_DATE, BOROUGH, ZIP_CODE, LATITUDE, LONGITUDE, LOCATION, ON_STREET_NAME, CROSS_STREET_NAME, OFF_STREET_NAME, CONTRIBUTING_FACTOR_VEHICLE_1, CONTRIBUTING_FACTOR_VEHICLE_2, COLLISION_ID, VEHICLE_TYPE_CODE_1, VEHICLE_TYPE_CODE_2 ) SELECT STR_TO_DATE(CONCAT_WS(' ', `CRASH DATE`, `CRASH TIME`), '%m/%d/%Y %H:%i'), BOROUGH, `ZIP CODE`, CAST(LATITUDE as INT), CAST(LONGITUDE as INT), LOCATION, `ON STREET NAME`, `CROSS STREET NAME`, `OFF STREET NAME`, `CONTRIBUTING FACTOR VEHICLE 1`, `CONTRIBUTING FACTOR VEHICLE 2`, CAST(`COLLISION_ID` as INT), `VEHICLE TYPE CODE 1`, `VEHICLE TYPE CODE 2` FROM FILES( "path" = "s3://{file_path}", "format" = "parquet", "aws.s3.access_key" = "XXXXXXXXXX", "aws.s3.secret_key" = "YYYYYYYYYY", "aws.s3.region" = "us-west-2" ) """ result = hook.run(sql) return file_path load_file(FILE_PATH) crashdata_files() ``` ##### Data Transformation[​](#data-transformation "Direct link to Data Transformation") Execute SQL queries against StarRocks for table creation and data insertion. This is useful for: * Setting up database schema * Loading small datasets * Running ad-hoc queries ```py from airflow.sdk import dag, chain from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator from datetime import datetime @dag( dag_id='crashdata_basic_setup', schedule='@daily', start_date=datetime(2024, 1, 1), catchup=False, tags=['starrocks', 'crashdata'], ) def crashdata_basic_pipeline(): """Create crashdata table and insert sample NYC crash data.""" create_table = SQLExecuteQueryOperator( task_id='create_crashdata_table', conn_id='starrocks_default', sql=""" CREATE TABLE IF NOT EXISTS crashdata ( CRASH_DATE DATETIME, BOROUGH STRING, ZIP_CODE STRING, LATITUDE INT, LONGITUDE INT, LOCATION STRING, ON_STREET_NAME STRING, CROSS_STREET_NAME STRING, OFF_STREET_NAME STRING, CONTRIBUTING_FACTOR_VEHICLE_1 STRING, CONTRIBUTING_FACTOR_VEHICLE_2 STRING, COLLISION_ID INT, VEHICLE_TYPE_CODE_1 STRING, VEHICLE_TYPE_CODE_2 STRING ) DUPLICATE KEY(CRASH_DATE) DISTRIBUTED BY HASH(COLLISION_ID) BUCKETS 10 PROPERTIES ( "replication_num" = "1" ) """, ) insert_data = SQLExecuteQueryOperator( task_id='insert_sample_data', conn_id='starrocks_default', sql=""" INSERT INTO crashdata VALUES ('2024-01-15 08:30:00', 'MANHATTAN', '10001', 40748817, -73985428, '(40.748817, -73.985428)', '5 AVENUE', 'WEST 34 STREET', NULL, 'Driver Inattention/Distraction', 'Unspecified', 4567890, 'Sedan', 'Taxi'), ('2024-01-15 14:20:00', 'BROOKLYN', '11201', 40693139, -73987664, '(40.693139, -73.987664)', 'FLATBUSH AVENUE', 'ATLANTIC AVENUE', NULL, 'Failure to Yield Right-of-Way', 'Unspecified', 4567891, 'SUV', 'Sedan'), ('2024-01-15 18:45:00', 'QUEENS', '11354', 40767689, -73827426, '(40.767689, -73.827426)', 'NORTHERN BOULEVARD', 'MAIN STREET', NULL, 'Following Too Closely', 'Driver Inattention/Distraction', 4567892, 'Sedan', 'Sedan'), ('2024-01-16 09:15:00', 'BRONX', '10451', 40820679, -73925300, '(40.820679, -73.925300)', 'GRAND CONCOURSE', 'EAST 161 STREET', NULL, 'Unsafe Speed', 'Unspecified', 4567893, 'Truck', 'Sedan') """, ) create_table >> insert_data crashdata_basic_pipeline() ``` ###### More complex operations with MySqlHook[​](#more-complex-operations-with-mysqlhook "Direct link to More complex operations with MySqlHook") Use MySqlHook for advanced data analysis and processing within Python tasks. This approach is useful for: * Running analytical queries and processing results in Python * Combining StarRocks queries with Python libraries (pandas, numpy, etc.) * Implementing complex business logic that requires both SQL and Python * Creating data quality checks and validation workflows MySqlHook provides full programmatic access to query results, enabling sophisticated data transformations and analysis within your DAG. ```py from airflow.sdk import dag, task from airflow.providers.mysql.hooks.mysql import MySqlHook from datetime import datetime @dag( dag_id='crashdata_python_analysis', schedule='@daily', start_date=datetime(2024, 1, 1), catchup=False, tags=['starrocks', 'python', 'analytics'], ) def crashdata_python_pipeline(): @task def analyze_crash_hotspots(): """Identify crash hotspots by borough and street.""" hook = MySqlHook(mysql_conn_id='starrocks_default') # Query to find high-frequency crash locations sql = """ SELECT BOROUGH, ON_STREET_NAME, COUNT(*) as crash_count, COUNT(DISTINCT DATE(CRASH_DATE)) as days_with_crashes FROM crashdata WHERE ON_STREET_NAME IS NOT NULL GROUP BY BOROUGH, ON_STREET_NAME HAVING crash_count >= 3 ORDER BY crash_count DESC LIMIT 10 """ results = hook.get_records(sql) print("Top 10 Crash Hotspots:") for row in results: borough, street, count, days = row print(f"{borough:15} | {street:40} | {count:3} crashes over {days} days") return len(results) @task def calculate_contributing_factors(): """Calculate percentage distribution of contributing factors.""" hook = MySqlHook(mysql_conn_id='starrocks_default') sql = """ SELECT CONTRIBUTING_FACTOR_VEHICLE_1 as factor, COUNT(*) as count, ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER (), 2) as percentage FROM crashdata WHERE CONTRIBUTING_FACTOR_VEHICLE_1 != 'Unspecified' GROUP BY CONTRIBUTING_FACTOR_VEHICLE_1 ORDER BY count DESC """ results = hook.get_records(sql) print("\nContributing Factors Analysis:") for factor, count, percentage in results: print(f"{factor:50} | {count:4} ({percentage}%)") return results # Define task execution order hotspots = analyze_crash_hotspots() factors = calculate_contributing_factors() hotspots >> factors crashdata_python_pipeline() ``` ##### Advanced Patterns[​](#advanced-patterns "Direct link to Advanced Patterns") ###### Incremental Data Loading[​](#incremental-data-loading "Direct link to Incremental Data Loading") Load data incrementally to avoid reprocessing existing records. Incremental loading is essential for: * Efficiently updating tables with new data only * Reducing processing time and resource usage * Managing large datasets that grow over time * Maintaining data freshness without full reloads This pattern uses staging tables and timestamp-based filtering to ensure only new records are loaded, making it ideal for scheduled batch updates. ```py from airflow.sdk import dag, chain from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator from datetime import datetime @dag( dag_id='crashdata_incremental_load', schedule='@hourly', start_date=datetime(2024, 1, 1), catchup=False, tags=['starrocks', 'incremental'], ) def crashdata_incremental_pipeline(): """Incrementally load new crash reports from staging.""" create_staging = SQLExecuteQueryOperator( task_id='create_staging_table', conn_id='starrocks_default', sql=""" CREATE TABLE IF NOT EXISTS crashdata_staging ( CRASH_DATE DATETIME, BOROUGH STRING, ZIP_CODE STRING, LATITUDE INT, LONGITUDE INT, LOCATION STRING, ON_STREET_NAME STRING, CROSS_STREET_NAME STRING, OFF_STREET_NAME STRING, CONTRIBUTING_FACTOR_VEHICLE_1 STRING, CONTRIBUTING_FACTOR_VEHICLE_2 STRING, COLLISION_ID INT, VEHICLE_TYPE_CODE_1 STRING, VEHICLE_TYPE_CODE_2 STRING, loaded_at DATETIME ) DUPLICATE KEY(CRASH_DATE) DISTRIBUTED BY HASH(COLLISION_ID) BUCKETS 10 PROPERTIES ("replication_num" = "1") """, ) incremental_load = SQLExecuteQueryOperator( task_id='load_new_crashes', conn_id='starrocks_default', sql=""" INSERT INTO crashdata SELECT CRASH_DATE, BOROUGH, ZIP_CODE, LATITUDE, LONGITUDE, LOCATION, ON_STREET_NAME, CROSS_STREET_NAME, OFF_STREET_NAME, CONTRIBUTING_FACTOR_VEHICLE_1, CONTRIBUTING_FACTOR_VEHICLE_2, COLLISION_ID, VEHICLE_TYPE_CODE_1, VEHICLE_TYPE_CODE_2 FROM crashdata_staging WHERE loaded_at >= '{{ data_interval_start }}' AND loaded_at < '{{ data_interval_end }}' AND COLLISION_ID NOT IN (SELECT COLLISION_ID FROM crashdata) """, ) create_staging >> incremental_load crashdata_incremental_pipeline() ``` ###### Asynchronous large-scale jobs with `SUBMIT TASK`[​](#asynchronous-large-scale-jobs-with-submit-task "Direct link to asynchronous-large-scale-jobs-with-submit-task") Use `SUBMIT TASK` for long-running queries that shouldn't block the Airflow task. This pattern is beneficial for: * Complex analytical queries that take minutes or hours * Large-scale data transformations (table copies, aggregations) * Resource-intensive operations that might timeout in synchronous mode * Parallel execution of multiple heavy queries * Separating job submission from completion monitoring `SUBMIT TASK` allows Airflow to monitor long-running StarRocks operations without holding database connections open, improving resource efficiency and reliability. ```py from airflow.sdk import dag, chain, task from airflow.providers.mysql.hooks.mysql import MySqlHook from datetime import datetime import time @dag( dag_id='crashdata_submit_task', schedule='@daily', start_date=datetime(2024, 1, 1), catchup=False, tags=['starrocks', 'submit-task'], ) def crashdata_submit_task_pipeline(): """ Example of using SUBMIT TASK for long-running queries. Requires StarRocks 3.4+ for SUBMIT TASK support. """ @task def submit_long_running_query(): """Submit a long-running query as an async task.""" hook = MySqlHook(mysql_conn_id='starrocks_default') submit_sql = """ SUBMIT TASK backup_crashdata AS CREATE TABLE crash_data_backup AS SELECT * FROM crashdata """ conn = hook.get_conn() cursor = conn.cursor() cursor.execute(submit_sql) # Get the task name from result result = cursor.fetchone() task_name = result[0] if result else None cursor.close() conn.close() if task_name: print(f"Task submitted successfully: {task_name}") return task_name else: raise Exception("Failed to submit task") @task def monitor_task_completion(task_name: str): """Monitor the submitted task until completion.""" hook = MySqlHook(mysql_conn_id='starrocks_default') max_wait_time = 600 # 10 minutes poll_interval = 10 # Check every 10 seconds elapsed_time = 0 while elapsed_time < max_wait_time: conn = hook.get_conn() cursor = conn.cursor() # Check task status in information_schema check_sql = f""" SELECT STATE, ERROR_MESSAGE FROM information_schema.task_runs WHERE TASK_NAME = '{task_name}' """ cursor.execute(check_sql) result = cursor.fetchone() if result: state, error_msg = result print(f"[{elapsed_time}s] Task status: {state}") if state == 'SUCCESS': cursor.close() conn.close() return {'status': 'SUCCESS', 'task_name': task_name} elif state == 'FAILED': cursor.close() conn.close() raise Exception(f"Task failed: {error_msg}") cursor.close() conn.close() time.sleep(poll_interval) elapsed_time += poll_interval raise Exception(f"Task did not complete within {max_wait_time} seconds") @task def process_results(): """Process or verify the completed task results.""" print("Task completed successfully - results are now available") return "Processing complete" # Define task flow task_name = submit_long_running_query() monitor_result = monitor_task_completion(task_name) result = process_results() chain(task_name, monitor_result, result) crashdata_submit_task_pipeline() ``` Note that the task name is unique in StarRocks, so future runs may need a qualifier (such as uuid). ###### Materialized Views[​](#materialized-views "Direct link to Materialized Views") Create and manage materialized views for accelerated query performance. Materialized views are ideal for: * Pre-computing complex aggregations for dashboards * Accelerating frequently run analytical queries * Maintaining summary tables that update automatically * Reducing compute costs by avoiding repeated calculations * Serving real-time analytics from pre-aggregated data Materialized views in StarRocks refresh automatically or on-demand, keeping aggregated data fresh while dramatically improving query performance. ```py from airflow.sdk import dag, task from airflow.providers.mysql.hooks.mysql import MySqlHook from datetime import datetime, timedelta @dag( dag_id="starrocks_materialized_view_example", schedule="0 2 * * *", # Run daily at 2 AM start_date=datetime(2024, 1, 1), catchup=False, tags=["starrocks", "materialized_view", "example"], doc_md=__doc__, ) def starrocks_materialized_view_example(): @task def create_materialized_view(): hook = MySqlHook(mysql_conn_id="starrocks_conn") drop_sql = """ DROP MATERIALIZED VIEW IF EXISTS quickstart.mv_daily_crash_stats """ create_sql = """ CREATE MATERIALIZED VIEW IF NOT EXISTS quickstart.mv_daily_crash_stats DISTRIBUTED BY HASH(`crash_date`) REFRESH ASYNC AS SELECT DATE(CRASH_DATE) as crash_date, BOROUGH, COUNT(*) as total_crashes, COUNT(DISTINCT COLLISION_ID) as unique_collisions FROM quickstart.crashdata WHERE CRASH_DATE IS NOT NULL GROUP BY DATE(CRASH_DATE), BOROUGH """ hook.run(drop_sql) hook.run(create_sql) return "mv_daily_crash_stats" @task def refresh_materialized_view(mv_name: str): hook = MySqlHook(mysql_conn_id="starrocks_conn") refresh_sql = f"REFRESH MATERIALIZED VIEW quickstart.{mv_name}" hook.run(refresh_sql) return mv_name @task def check_materialized_view_status(mv_name: str): hook = MySqlHook(mysql_conn_id="starrocks_conn") # Get task name for the MV task_query = f""" SELECT TASK_NAME FROM information_schema.tasks WHERE `DATABASE` = 'quickstart' AND DEFINITION LIKE '%{mv_name}%' ORDER BY CREATE_TIME DESC LIMIT 1 """ task_name = hook.get_first(task_query)[0] # Get latest task run state state_query = f""" SELECT STATE FROM information_schema.task_runs WHERE TASK_NAME = '{task_name}' ORDER BY CREATE_TIME DESC LIMIT 1 """ state = hook.get_first(state_query)[0] print(f"MV: {mv_name} | Task: {task_name} | State: {state}") if state not in ('SUCCESS', 'RUNNING'): raise Exception(f"Materialized view refresh in unexpected state: {state}") return {'task_name': task_name, 'state': state} create = create_materialized_view() refresh = refresh_materialized_view(create) status = check_materialized_view_status(refresh) status starrocks_materialized_view_example() ``` ###### Error Handling[​](#error-handling "Direct link to Error Handling") Implement robust error handling for production reliability. Proper error handling is critical for: * Automatically recovering from transient failures (network issues, timeouts) * Preventing data pipeline disruptions from temporary problems * Providing visibility into failure patterns Airflow's built-in retry mechanisms handle most transient errors. ```py from airflow.sdk import dag from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator from datetime import datetime, timedelta @dag( dag_id='starrocks_with_retries', schedule='@daily', start_date=datetime(2024, 1, 1), catchup=False, default_args={ 'retries': 3, 'retry_delay': timedelta(minutes=5), 'retry_exponential_backoff': True, 'max_retry_delay': timedelta(hours=1), }, ) def starrocks_resilient_pipeline(): critical_query = SQLExecuteQueryOperator( task_id='critical_query', conn_id='starrocks_default', sql='SELECT * FROM important_table', ) starrocks_resilient_pipeline() ``` ##### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") * Verify that port 9030 is accessible from within the Airflow instance * Test the connection (if enabled) from the Airflow UI * If using localhost, use 127.0.0.1 instead --- ### Batch Loading Data #### [📄️ StarRocks Spark Connector](https://docs.starrocks.io/docs/integrations/batch/Spark-connector-starrocks.md) [Load using Spark DataFrames or Spark SQL](https://docs.starrocks.io/docs/integrations/batch/Spark-connector-starrocks.md) #### [📄️ Apache Spark Load](https://docs.starrocks.io/docs/integrations/batch/SparkLoad.md) [Load data in bulk using Spark Load](https://docs.starrocks.io/docs/integrations/batch/SparkLoad.md) --- ### StarRocks Spark Connector ### Load data using Spark connector (recommended) StarRocks provides a self-developed connector named StarRocks Connector for Apache Spark™ (Spark connector for short) to help you load data into a StarRocks table by using Spark. The basic principle is to accumulate the data and then load it all at a time into StarRocks through [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). The Spark connector is implemented based on Spark DataSource V2. A DataSource can be created by using Spark DataFrames or Spark SQL. And both batch and structured streaming modes are supported. > **NOTICE** > > Only users with the SELECT and INSERT privileges on a StarRocks table can load data into this table. You can follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant these privileges to a user. #### Version requirements[​](#version-requirements "Direct link to Version requirements") | Spark connector | Spark | StarRocks | Java | Scala | | --------------- | ------------------ | ------------- | ---- | ----- | | 1.1.4 | 4.0, 4.1 | 2.5 and later | 17 | 2.13 | | 1.1.4 | 3.3, 3.4, 3.5 | 2.5 and later | 8 | 2.12 | | 1.1.3 | 3.2, 3.3, 3.4, 3.5 | 2.5 and later | 8 | 2.12 | | 1.1.2 | 3.2, 3.3, 3.4, 3.5 | 2.5 and later | 8 | 2.12 | | 1.1.1 | 3.2, 3.3, or 3.4 | 2.5 and later | 8 | 2.12 | | 1.1.0 | 3.2, 3.3, or 3.4 | 2.5 and later | 8 | 2.12 | > **NOTICE** > > * Please see [Upgrade Spark connector](#upgrade-spark-connector) for behavior changes among different versions of the Spark connector. > * The Spark connector does not provide MySQL JDBC driver since version 1.1.1, and you need import the driver to the spark classpath manually. You can find the driver on [MySQL site](https://dev.mysql.com/downloads/connector/j/) or [Maven Central](https://repo1.maven.org/maven2/mysql/mysql-connector-java/). #### Obtain Spark connector[​](#obtain-spark-connector "Direct link to Obtain Spark connector") You can obtain the Spark connector JAR file in the following ways: * Directly download the compiled Spark Connector JAR file. * Add the Spark connector as a dependency in your Maven project and then download the JAR file. * Compile the source code of the Spark Connector into a JAR file by yourself. The naming format of the Spark connector JAR file is `starrocks-spark-connector-${spark_version}_${scala_version}-${connector_version}.jar`. For example, if you install Spark 3.2 and Scala 2.12 in your environment and you want to use Spark connector 1.1.0, you can use `starrocks-spark-connector-3.2_2.12-1.1.0.jar`. > **NOTICE** > > In general, the latest version of the Spark connector only maintains compatibility with the three most recent versions of Spark. ##### Download the compiled Jar file[​](#download-the-compiled-jar-file "Direct link to Download the compiled Jar file") Directly download the corresponding version of the Spark connector JAR from the [Maven Central Repository](https://repo1.maven.org/maven2/com/starrocks). ##### Maven Dependency[​](#maven-dependency "Direct link to Maven Dependency") 1. In your Maven project's `pom.xml` file, add the Spark connector as a dependency according to the following format. Replace `spark_version`, `scala_version`, and `connector_version` with the respective versions. ```xml com.starrocks starrocks-spark-connector-${spark_version}_${scala_version} ${connector_version} ``` 2. For example, if the version of Spark in your environment is 3.2, the version of Scala is 2.12, and you choose Spark connector 1.1.0, you need to add the following dependency: ```xml com.starrocks starrocks-spark-connector-3.2_2.12 1.1.0 ``` ##### Compile by yourself[​](#compile-by-yourself "Direct link to Compile by yourself") 1. Download the [Spark connector package](https://github.com/StarRocks/starrocks-connector-for-apache-spark). 2. Execute the following command to compile the source code of Spark connector into a JAR file. Note that `spark_version` is replaced with the corresponding Spark version. ```bash sh build.sh ``` For example, if the Spark version in your environment is 3.2, you need to execute the following command: ```bash sh build.sh 3.2 ``` 3. Go to the `target/` directory to find the Spark connector JAR file, such as `starrocks-spark-connector-3.2_2.12-1.1.0-SNAPSHOT.jar` , generated upon compilation. > **NOTE** > > The name of Spark connector which is not formally released contains the `SNAPSHOT` suffix. #### Parameters[​](#parameters "Direct link to Parameters") ##### `starrocks.fe.http.url`[​](#starrocksfehttpurl "Direct link to starrocksfehttpurl") * **Required**: YES * **Default value**: None * **Description**: The HTTP URL of the FE in your StarRocks cluster. You can specify multiple URLs, which must be separated by a comma (,). Format: `:,:`. Since version 1.1.1, you can also add `http://` prefix to the URL, such as `http://:,http://:`. ##### `starrocks.fe.jdbc.url`[​](#starrocksfejdbcurl "Direct link to starrocksfejdbcurl") * **Required**: YES * **Default value**: None * **Description**: The address that is used to connect to the MySQL server of the FE. Format: `jdbc:mysql://:`. ##### `starrocks.table.identifier`[​](#starrockstableidentifier "Direct link to starrockstableidentifier") * **Required**: YES * **Default value**: None * **Description**: The name of the StarRocks table. Format: `.`. ##### `starrocks.user`[​](#starrocksuser "Direct link to starrocksuser") * **Required**: YES * **Default value**: None * **Description**: The username of your StarRocks cluster account. The user needs the [SELECT and INSERT privileges](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) on the StarRocks table. ##### `starrocks.password`[​](#starrockspassword "Direct link to starrockspassword") * **Required**: YES * **Default value**: None * **Description**: The password of your StarRocks cluster account. ##### `starrocks.write.label.prefix`[​](#starrockswritelabelprefix "Direct link to starrockswritelabelprefix") * **Required**: NO * **Default value**: `spark-` * **Description**: The label prefix used by Stream Load. ##### `starrocks.write.enable.transaction-stream-load`[​](#starrockswriteenabletransaction-stream-load "Direct link to starrockswriteenabletransaction-stream-load") * **Required**: NO * **Default value**: `TRUE` * **Description**: Whether to use [Stream Load transaction interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) to load data. It requires StarRocks v2.5 or later. This feature can load more data in a transaction with less memory usage, and improve performance. note Since 1.1.1, this parameter takes effect only when the value of `starrocks.write.max.retries` is non-positive because Stream Load transaction interface does not support retry. ##### `starrocks.write.buffer.size`[​](#starrockswritebuffersize "Direct link to starrockswritebuffersize") * **Required**: NO * **Default value**: `104857600` * **Description**: The maximum size of data that can be accumulated in memory before being sent to StarRocks at a time. Setting this parameter to a larger value can improve loading performance but may increase loading latency. ##### `starrocks.write.buffer.rows`[​](#starrockswritebufferrows "Direct link to starrockswritebufferrows") * **Required**: NO * **Default value**: Integer.MAX\_VALUE * **Description**: Supported since version 1.1.1. The maximum number of rows that can be accumulated in memory before being sent to StarRocks at a time. ##### `starrocks.write.flush.interval.ms`[​](#starrockswriteflushintervalms "Direct link to starrockswriteflushintervalms") * **Required**: NO * **Default value**: `300000` * **Description**: The interval at which data is sent to StarRocks. This parameter is used to control the loading latency. ##### `starrocks.write.max.retries`[​](#starrockswritemaxretries "Direct link to starrockswritemaxretries") * **Required**: NO * **Default value**: `3` * **Description**: Supported since version 1.1.1. The number of times that the connector retries to perform the Stream Load for the same batch of data if the load fails. note Because Stream Load transaction interface does not support retry. If this parameter is positive, the connector always use Stream Load interface and ignore the value of `starrocks.write.enable.transaction-stream-load`. ##### `starrocks.write.retry.interval.ms`[​](#starrockswriteretryintervalms "Direct link to starrockswriteretryintervalms") * **Required**: NO * **Default value**: `10000` * **Description**: Supported since version 1.1.1. The interval to retry the Stream Load for the same batch of data if the load fails. ##### `starrocks.write.use_bitmap_hash64`[​](#starrockswriteuse_bitmap_hash64 "Direct link to starrockswriteuse_bitmap_hash64") * **Required**: NO * **Default value**: `false` * **Description**: Supported since version 1.1.3. Whether to use the 64-bit hash function to generate bitmap. The default is to use 32-bit hash function. ##### `starrocks.columns`[​](#starrockscolumns "Direct link to starrockscolumns") * **Required**: NO * **Default value**: None * **Description**: The StarRocks table column into which you want to load data. You can specify multiple columns, which must be separated by commas (,), for example, `"col0,col1,col2"`. ##### `starrocks.column.types`[​](#starrockscolumntypes "Direct link to starrockscolumntypes") * **Required**: NO * **Default value**: None * **Description**: Supported since version 1.1.1. Customize the column data types for Spark instead of using the defaults inferred from the StarRocks table and the [default mapping](#data-type-mapping-between-spark-and-starrocks). The parameter value is a schema in DDL format same as the output of Spark [StructType#toDDL](https://github.com/apache/spark/blob/master/sql/api/src/main/scala/org/apache/spark/sql/types/StructType.scala#L449) , such as `col0 INT, col1 STRING, col2 BIGINT`. Note that you only need to specify columns that need customization. One use case is to load data into columns of [BITMAP](#load-data-into-columns-of-bitmap-type) or [HLL](#load-data-into-columns-of-hll-type) type. ##### `starrocks.write.properties.*`[​](#starrockswriteproperties "Direct link to starrockswriteproperties") * **Required**: NO * **Default value**: None * **Description**: The parameters that are used to control Stream Load behavior. For example, the parameter `starrocks.write.properties.format` specifies the format of the data to be loaded, such as CSV or JSON. For a list of supported parameters and their descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ##### `starrocks.write.properties.format`[​](#starrockswritepropertiesformat "Direct link to starrockswritepropertiesformat") * **Required**: NO * **Default value**: `CSV` * **Description**: The file format based on which the Spark connector transforms each batch of data before the data is sent to StarRocks. Valid values: CSV and JSON. ##### `starrocks.write.properties.row_delimiter`[​](#starrockswritepropertiesrow_delimiter "Direct link to starrockswritepropertiesrow_delimiter") * **Required**: NO * **Default value**: `\n` * **Description**: The row delimiter for CSV-formatted data. ##### `starrocks.write.properties.column_separator`[​](#starrockswritepropertiescolumn_separator "Direct link to starrockswritepropertiescolumn_separator") * **Required**: NO * **Default value**: `\t` * **Description**: The column separator for CSV-formatted data. ##### `starrocks.write.properties.partial_update`[​](#starrockswritepropertiespartial_update "Direct link to starrockswritepropertiespartial_update") * **Required**: NO * **Default value**: `FALSE` * **Description**: Whether to use partial updates. Valid values: `TRUE` and `FALSE`. Default value: `FALSE`, indicating to disable this feature. ##### `starrocks.write.properties.partial_update_mode`[​](#starrockswritepropertiespartial_update_mode "Direct link to starrockswritepropertiespartial_update_mode") * **Required**: NO * **Default value**: `row` * **Description**: Specifies the mode for partial updates. Valid values: `row` and `column`. * The value `row` (default) means partial updates in row mode, which is more suitable for real-time updates with many columns and small batches. * The value `column` means partial updates in column mode, which is more suitable for batch updates with few columns and many rows. In such scenarios, enabling the column mode offers faster update speeds. For example, in a table with 100 columns, if only 10 columns (10% of the total) are updated for all rows, the update speed of the column mode is 10 times faster. ##### `starrocks.write.num.partitions`[​](#starrockswritenumpartitions "Direct link to starrockswritenumpartitions") * **Required**: NO * **Default value**: None * **Description**: The number of partitions into which Spark can write data in parallel. When the data volume is small, you can reduce the number of partitions to lower the loading concurrency and frequency. The default value for this parameter is determined by Spark. However, this method may cause Spark Shuffle cost. ##### `starrocks.write.partition.columns`[​](#starrockswritepartitioncolumns "Direct link to starrockswritepartitioncolumns") * **Required**: NO * **Default value**: None * **Description**: The partitioning columns in Spark. The parameter takes effect only when `starrocks.write.num.partitions` is specified. If this parameter is not specified, all columns being written are used for partitioning. ##### `starrocks.timezone`[​](#starrockstimezone "Direct link to starrockstimezone") * **Required**: NO * **Default value**: Default timezone of JVM * **Description**: Supported since 1.1.1. The timezone used to convert Spark `TimestampType` to StarRocks `DATETIME`. The default is the timezone of JVM returned by `ZoneId#systemDefault()`. The format can be a timezone name such as `Asia/Shanghai`, or a zone offset such as `+08:00`. ##### `starrocks.write.socket.timeout.ms`[​](#starrockswritesockettimeoutms "Direct link to starrockswritesockettimeoutms") * **Required**: NO * **Default value**: `-1` * **Description**: Supported since 1.1.3. The time duration for which the HTTP client waits for data. Unit: ms. The default value `-1` means there is no timeout. ##### `starrocks.write.properties.compression`[​](#starrockswritepropertiescompression "Direct link to starrockswritepropertiescompression") * **Required**: NO * **Default value**: None * **Description**: Supported since 1.1.3. The compression algorithm used for Stream Load. Valid values: `lz4_frame`. Compression for JSON format requires the database v3.2.7 or later. Compression for CSV format has no requirement for the database version. #### Data type mapping between Spark and StarRocks[​](#data-type-mapping-between-spark-and-starrocks "Direct link to Data type mapping between Spark and StarRocks") * The default data type mapping is as follows: | Spark data type | StarRocks data type | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | BooleanType | BOOLEAN | | ByteType | TINYINT | | ShortType | SMALLINT | | IntegerType | INT | | LongType | BIGINT | | StringType | LARGEINT | | FloatType | FLOAT | | DoubleType | DOUBLE | | DecimalType | DECIMAL | | StringType | CHAR | | StringType | VARCHAR | | StringType | STRING | | StringType | JSON | | DateType | DATE | | TimestampType | DATETIME | | ArrayType | ARRAY
**NOTE:**
**Supported since version 1.1.1**. For detailed steps, see [Load data into columns of ARRAY type](#load-data-into-columns-of-array-type). | | MapType | MAP
**NOTE:**
**Supported since version 1.1.3**. For detailed steps, see [Load nested columns](#load-nested-columns-struct-array-and-map). | | StructType | STRUCT
**NOTE:**
**Supported since version 1.1.3**. For detailed steps, see [Load nested columns](#load-nested-columns-struct-array-and-map). | * You can also customize the data type mapping. For example, a StarRocks table contains BITMAP and HLL columns, but Spark does not support the two data types. You need to customize the corresponding data types in Spark. For detailed steps, see load data into [BITMAP](#load-data-into-columns-of-bitmap-type) and [HLL](#load-data-into-columns-of-hll-type) columns. **BITMAP and HLL are supported since version 1.1.1**. #### Upgrade Spark connector[​](#upgrade-spark-connector "Direct link to Upgrade Spark connector") ##### Upgrade from version 1.1.0 to 1.1.1[​](#upgrade-from-version-110-to-111 "Direct link to Upgrade from version 1.1.0 to 1.1.1") * Since 1.1.1, the Spark connector does not provide `mysql-connector-java` which is the official JDBC driver for MySQL, because of the limitations of the GPL license used by `mysql-connector-java`. However, the Spark connector still needs the MySQL JDBC driver to connect to StarRocks for the table metadata, so you need to add the driver to the Spark classpath manually. You can find the driver on [MySQL site](https://dev.mysql.com/downloads/connector/j/) or [Maven Central](https://repo1.maven.org/maven2/mysql/mysql-connector-java/). * Since 1.1.1, the connector uses Stream Load interface by default rather than Stream Load transaction interface in version 1.1.0. If you still want to use Stream Load transaction interface, you can set the option `starrocks.write.max.retries` to `0`. Please see the description of `starrocks.write.enable.transaction-stream-load` and `starrocks.write.max.retries` for details. #### Examples[​](#examples "Direct link to Examples") The following examples show how to use the Spark connector to load data into a StarRocks table with Spark DataFrames or Spark SQL. The Spark DataFrames supports both Batch and Structured Streaming modes. For more examples, see [Spark Connector Examples](https://github.com/StarRocks/starrocks-connector-for-apache-spark/tree/main/src/test/java/com/starrocks/connector/spark/examples). ##### Preparations[​](#preparations "Direct link to Preparations") ###### Create a StarRocks table[​](#create-a-starrocks-table "Direct link to Create a StarRocks table") Create a database `test` and create a Primary Key table `score_board`. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Network configuration[​](#network-configuration "Direct link to Network configuration") Ensure that the machine where Spark is located can access the FE nodes of the StarRocks cluster via the `http_port` (default: `8030`) and `query_port` (default: `9030`), and the BE nodes via the `be_http_port` (default: `8040`). ###### Set up your Spark environment[​](#set-up-your-spark-environment "Direct link to Set up your Spark environment") Note that the following examples are run in Spark 3.2.4 and use `spark-shell`, `pyspark` and `spark-sql`. Before running the examples, make sure to place the Spark connector JAR file in the `$SPARK_HOME/jars` directory. ##### Load data with Spark DataFrames[​](#load-data-with-spark-dataframes "Direct link to Load data with Spark DataFrames") The following two examples explain how to load data with Spark DataFrames Batch or Structured Streaming mode. ###### Batch[​](#batch "Direct link to Batch") Construct data in memory and load data into the StarRocks table. 1. You can write the spark application using Scala or Python. For Scala, run the following code snippet in `spark-shell`: ```scala // 1. Create a DataFrame from a sequence. val data = Seq((1, "starrocks", 100), (2, "spark", 100)) val df = data.toDF("id", "name", "score") // 2. Write to StarRocks by configuring the format as "starrocks" and the following options. // You need to modify the options according your own environment. df.write.format("starrocks") .option("starrocks.fe.http.url", "127.0.0.1:8030") .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") .option("starrocks.table.identifier", "test.score_board") .option("starrocks.user", "root") .option("starrocks.password", "") .mode("append") .save() ``` For Python, run the following code snippet in `pyspark`: ```python from pyspark.sql import SparkSession spark = SparkSession \ .builder \ .appName("StarRocks Example") \ .getOrCreate() # 1. Create a DataFrame from a sequence. data = [(1, "starrocks", 100), (2, "spark", 100)] df = spark.sparkContext.parallelize(data) \ .toDF(["id", "name", "score"]) # 2. Write to StarRocks by configuring the format as "starrocks" and the following options. # You need to modify the options according your own environment. df.write.format("starrocks") \ .option("starrocks.fe.http.url", "127.0.0.1:8030") \ .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") \ .option("starrocks.table.identifier", "test.score_board") \ .option("starrocks.user", "root") \ .option("starrocks.password", "") \ .mode("append") \ .save() ``` 2. Query data in the StarRocks table. ```sql MySQL [test]> SELECT * FROM `score_board`; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | spark | 100 | +------+-----------+-------+ 2 rows in set (0.00 sec) ``` ###### Structured Streaming[​](#structured-streaming "Direct link to Structured Streaming") Construct a streaming read of data from a CSV file and load data into the StarRocks table. 1. In the directory `csv-data`, create a CSV file `test.csv` with the following data: ```csv 3,starrocks,100 4,spark,100 ``` 2. You can write the Spark application using Scala or Python. For Scala, run the following code snippet in `spark-shell`: ```scala import org.apache.spark.sql.types.StructType // 1. Create a DataFrame from CSV. val schema = (new StructType() .add("id", "integer") .add("name", "string") .add("score", "integer") ) val df = (spark.readStream .option("sep", ",") .schema(schema) .format("csv") // Replace it with your path to the directory "csv-data". .load("/path/to/csv-data") ) // 2. Write to StarRocks by configuring the format as "starrocks" and the following options. // You need to modify the options according your own environment. val query = (df.writeStream.format("starrocks") .option("starrocks.fe.http.url", "127.0.0.1:8030") .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") .option("starrocks.table.identifier", "test.score_board") .option("starrocks.user", "root") .option("starrocks.password", "") // replace it with your checkpoint directory .option("checkpointLocation", "/path/to/checkpoint") .outputMode("append") .start() ) ``` For Python, run the following code snippet in `pyspark`: ```python from pyspark.sql import SparkSession from pyspark.sql.types import IntegerType, StringType, StructType, StructField spark = SparkSession \ .builder \ .appName("StarRocks SS Example") \ .getOrCreate() # 1. Create a DataFrame from CSV. schema = StructType([ StructField("id", IntegerType()), StructField("name", StringType()), StructField("score", IntegerType()) ]) df = ( spark.readStream .option("sep", ",") .schema(schema) .format("csv") # Replace it with your path to the directory "csv-data". .load("/path/to/csv-data") ) # 2. Write to StarRocks by configuring the format as "starrocks" and the following options. # You need to modify the options according your own environment. query = ( df.writeStream.format("starrocks") .option("starrocks.fe.http.url", "127.0.0.1:8030") .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") .option("starrocks.table.identifier", "test.score_board") .option("starrocks.user", "root") .option("starrocks.password", "") # replace it with your checkpoint directory .option("checkpointLocation", "/path/to/checkpoint") .outputMode("append") .start() ) ``` 3. Query data in the StarRocks table. ```sql MySQL [test]> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 4 | spark | 100 | | 3 | starrocks | 100 | +------+-----------+-------+ 2 rows in set (0.67 sec) ``` ##### Load data with Spark SQL[​](#load-data-with-spark-sql "Direct link to Load data with Spark SQL") The following example explains how to load data with Spark SQL by using the `INSERT INTO` statement in the [Spark SQL CLI](https://spark.apache.org/docs/latest/sql-distributed-sql-engine-spark-sql-cli.html). 1. Execute the following SQL statement in the `spark-sql`: ```sql -- 1. Create a table by configuring the data source as `starrocks` and the following options. -- You need to modify the options according your own environment. CREATE TABLE `score_board` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.score_board", "starrocks.user"="root", "starrocks.password"="" ); -- 2. Insert two rows into the table. INSERT INTO `score_board` VALUES (5, "starrocks", 100), (6, "spark", 100); ``` 2. Query data in the StarRocks table. ```sql MySQL [test]> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 6 | spark | 100 | | 5 | starrocks | 100 | +------+-----------+-------+ 2 rows in set (0.00 sec) ``` #### Best Practices[​](#best-practices "Direct link to Best Practices") ##### Load data to Primary Key table[​](#load-data-to-primary-key-table "Direct link to Load data to Primary Key table") This section will show how to load data to StarRocks Primary Key table to achieve partial updates, and conditional updates. You can see [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md) for the detailed introduction of these features. These examples use Spark SQL. ###### Preparations[​](#preparations-1 "Direct link to Preparations") Create a database `test` and create a Primary Key table `score_board` in StarRocks. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Partial updates[​](#partial-updates "Direct link to Partial updates") This example will show how to only update data in the column `name` through loading: 1. Insert initial data to StarRocks table in MySQL client. ```sql mysql> INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'spark', 100); mysql> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | spark | 100 | +------+-----------+-------+ 2 rows in set (0.02 sec) ``` 2. Create a Spark table `score_board` in Spark SQL client. * Set the option `starrocks.write.properties.partial_update` to `true` which tells the connector to do partial update. * Set the option `starrocks.columns` to `"id,name"` to tell the connector which columns to write. ```sql CREATE TABLE `score_board` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.score_board", "starrocks.user"="root", "starrocks.password"="", "starrocks.write.properties.partial_update"="true", "starrocks.columns"="id,name" ); ``` 3. Insert data into the table in Spark SQL client, and only update the column `name`. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks-update'), (2, 'spark-update'); ``` 4. Query the StarRocks table in MySQL client. You can see that only values for `name` change, and the values for `score` does not change. ```sql mysql> select * from score_board; +------+------------------+-------+ | id | name | score | +------+------------------+-------+ | 1 | starrocks-update | 100 | | 2 | spark-update | 100 | +------+------------------+-------+ 2 rows in set (0.02 sec) ``` ###### Conditional updates[​](#conditional-updates "Direct link to Conditional updates") This example will show how to do conditional updates according to the values of column `score`. The update for an `id` takes effect only when the new value for `score` is has a greater or equal to the old value. 1. Insert initial data to StarRocks table in MySQL client. ```sql mysql> INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'spark', 100); mysql> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | spark | 100 | +------+-----------+-------+ 2 rows in set (0.02 sec) ``` 2. Create a Spark table `score_board` in the following ways. * Set the option `starrocks.write.properties.merge_condition` to `score` which tells the connector to use the column `score` as the condition. * Make sure that the Spark connector use Stream Load interface to load data, rather than Stream Load transaction interface, because the latter does not support this feature. ```sql CREATE TABLE `score_board` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.score_board", "starrocks.user"="root", "starrocks.password"="", "starrocks.write.properties.merge_condition"="score" ); ``` 3. Insert data to the table in Spark SQL client, and update the row whose `id` is 1 with a smaller score value, and the row whose `id` is 2 with a larger score value. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks-update', 99), (2, 'spark-update', 101); ``` 4. Query the StarRocks table in MySQL client. You can see that only the row whose `id` is 2 changes, and the row whose `id` is 1 does not change. ```sql mysql> select * from score_board; +------+--------------+-------+ | id | name | score | +------+--------------+-------+ | 1 | starrocks | 100 | | 2 | spark-update | 101 | +------+--------------+-------+ 2 rows in set (0.03 sec) ``` ##### Load data into columns of BITMAP type[​](#load-data-into-columns-of-bitmap-type "Direct link to Load data into columns of BITMAP type") [`BITMAP`](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/BITMAP.md) is often used to accelerate count distinct, such as counting UV, see [Use Bitmap for exact Count Distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_bitmap.md). Here we take the counting of UV as an example to show how to load data into columns of the `BITMAP` type. **`BITMAP` is supported since version 1.1.1**. 1. Create a StarRocks Aggregate table. In the database `test`, create an Aggregate table `page_uv` where the column `visit_users` is defined as the `BITMAP` type and configured with the aggregate function `BITMAP_UNION`. ```sql CREATE TABLE `test`.`page_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` BITMAP BITMAP_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`); ``` 2. Create a Spark table. The schema of the Spark table is inferred from the StarRocks table, and the Spark does not support the `BITMAP` type. So you need to customize the corresponding column data type in Spark, for example as `BIGINT`, by configuring the option `"starrocks.column.types"="visit_users BIGINT"`. When using Stream Load to ingest data, the connector uses the [`to_bitmap`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/to_bitmap.md) function to convert the data of `BIGINT` type into `BITMAP` type. Run the following DDL in `spark-sql`: ```sql CREATE TABLE `page_uv` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.page_uv", "starrocks.user"="root", "starrocks.password"="", "starrocks.column.types"="visit_users BIGINT" ); ``` 3. Load data into StarRocks table. Run the following DML in `spark-sql`: ```sql INSERT INTO `page_uv` VALUES (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 13), (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 23), (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 33), (1, CAST('2020-06-23 02:30:30' AS TIMESTAMP), 13), (2, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 23); ``` 4. Calculate page UVs from the StarRocks table. ```sql MySQL [test]> SELECT `page_id`, COUNT(DISTINCT `visit_users`) FROM `page_uv` GROUP BY `page_id`; +---------+-----------------------------+ | page_id | count(DISTINCT visit_users) | +---------+-----------------------------+ | 2 | 1 | | 1 | 3 | +---------+-----------------------------+ 2 rows in set (0.01 sec) ``` note The connector uses [`to_bitmap`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/to_bitmap.md) function to convert data of the `TINYINT`, `SMALLINT`, `INTEGER`, and `BIGINT` types in Spark to the `BITMAP` type in StarRocks, and uses [`bitmap_hash`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_hash.md) or [`bitmap_hash64`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_hash64.md) function for other Spark data types. ##### Load data into columns of HLL type[​](#load-data-into-columns-of-hll-type "Direct link to Load data into columns of HLL type") [`HLL`](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/HLL.md) can be used for approximate count distinct, see [Use HLL for approximate count distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_HLL.md). Here we take the counting of UV as an example to show how to load data into columns of the `HLL` type. **`HLL` is supported since version 1.1.1**. 1. Create a StarRocks Aggregate table. In the database `test`, create an Aggregate table `hll_uv` where the column `visit_users` is defined as the `HLL` type and configured with the aggregate function `HLL_UNION`. ```sql CREATE TABLE `hll_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` HLL HLL_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`); ``` 2. Create a Spark table. The schema of the Spark table is inferred from the StarRocks table, and the Spark does not support the `HLL` type. So you need to customize the corresponding column data type in Spark, for example as `BIGINT`, by configuring the option `"starrocks.column.types"="visit_users BIGINT"`. When using Stream Load to ingest data, the connector uses the [`hll_hash`](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_hash.md) function to convert the data of `BIGINT` type into `HLL` type. Run the following DDL in `spark-sql`: ```sql CREATE TABLE `hll_uv` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.hll_uv", "starrocks.user"="root", "starrocks.password"="", "starrocks.column.types"="visit_users BIGINT" ); ``` 3. Load data into StarRocks table. Run the following DML in `spark-sql`: ```sql INSERT INTO `hll_uv` VALUES (3, CAST('2023-07-24 12:00:00' AS TIMESTAMP), 78), (4, CAST('2023-07-24 13:20:10' AS TIMESTAMP), 2), (3, CAST('2023-07-24 12:30:00' AS TIMESTAMP), 674); ``` 4. Calculate page UVs from the StarRocks table. ```sql MySQL [test]> SELECT `page_id`, COUNT(DISTINCT `visit_users`) FROM `hll_uv` GROUP BY `page_id`; +---------+-----------------------------+ | page_id | count(DISTINCT visit_users) | +---------+-----------------------------+ | 4 | 1 | | 3 | 2 | +---------+-----------------------------+ 2 rows in set (0.01 sec) ``` ##### Load data into columns of ARRAY type[​](#load-data-into-columns-of-array-type "Direct link to Load data into columns of ARRAY type") The following example explains how to load data into columns of the [`ARRAY`](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/Array.md) type. 1. Create a StarRocks table. In the database `test`, create a Primary Key table `array_tbl` that includes one `INT` column and two `ARRAY` columns. ```sql CREATE TABLE `array_tbl` ( `id` INT NOT NULL, `a0` ARRAY, `a1` ARRAY> ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`) ; ``` 2. Write data to StarRocks. Because some versions of StarRocks does not provide the metadata of `ARRAY` column, the connector can not infer the corresponding Spark data type for this column. However, you can explicitly specify the corresponding Spark data type of the column in the option `starrocks.column.types`. In this example, you can configure the option as `a0 ARRAY,a1 ARRAY>`. Run the following codes in `spark-shell`: ```scala val data = Seq( | (1, Seq("hello", "starrocks"), Seq(Seq(1, 2), Seq(3, 4))), | (2, Seq("hello", "spark"), Seq(Seq(5, 6, 7), Seq(8, 9, 10))) | ) val df = data.toDF("id", "a0", "a1") df.write .format("starrocks") .option("starrocks.fe.http.url", "127.0.0.1:8030") .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") .option("starrocks.table.identifier", "test.array_tbl") .option("starrocks.user", "root") .option("starrocks.password", "") .option("starrocks.column.types", "a0 ARRAY,a1 ARRAY>") .mode("append") .save() ``` 3. Query data in the StarRocks table. ```sql MySQL [test]> SELECT * FROM `array_tbl`; +------+-----------------------+--------------------+ | id | a0 | a1 | +------+-----------------------+--------------------+ | 1 | ["hello","starrocks"] | [[1,2],[3,4]] | | 2 | ["hello","spark"] | [[5,6,7],[8,9,10]] | +------+-----------------------+--------------------+ 2 rows in set (0.01 sec) ``` #### Load nested columns (STRUCT, ARRAY, and MAP)[​](#load-nested-columns-struct-array-and-map "Direct link to Load nested columns (STRUCT, ARRAY, and MAP)") Loading nested columns is supported since version 1.1.3. The Spark connector supports writing StarRocks columns of type `STRUCT`, `ARRAY`, and `MAP`. You must declare the StarRocks column types using the `starrocks.column.types` option so the connector can serialize the data correctly. ##### Nested data type mapping[​](#nested-data-type-mapping "Direct link to Nested data type mapping") Nested values are serialized as JSON-compatible strings before being sent to StarRocks via Stream Load: * `STRUCT` columns are serialized as a JSON object: `{"field1": value1, "field2": value2}`. * `ARRAY` columns are serialized as a JSON array: `[value1, value2, ...]`. * `MAP` columns are serialized as a JSON object: `{"key1": value1, "key2": value2}`. ##### Example[​](#example "Direct link to Example") Given the following StarRocks table: ```sql CREATE TABLE nested_tbl ( id INT, info STRUCT, tags ARRAY, attributes MAP ) ENGINE=OLAP DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 4; ``` Write data from Spark: ```scala import org.apache.spark.sql.types._ import org.apache.spark.sql.Row val schema = StructType(Seq( StructField("id", IntegerType), StructField("info", StructType(Seq( StructField("type", StringType), StructField("phone", LongType) ))), StructField("tags", ArrayType(StringType)), StructField("attributes", MapType(StringType, StringType)) )) val data = Seq( Row(1, Row("admin", 123456789L), Seq("spark", "starrocks"), Map("env" -> "prod")) ) val df = spark.createDataFrame(spark.sparkContext.parallelize(data), schema) val columnTypes = "info STRUCT, " + "tags ARRAY, " + "attributes MAP" df.write.format("starrocks") .option("starrocks.fenodes", "127.0.0.1:8030") .option("starrocks.table.identifier", "test.nested_tbl") .option("starrocks.user", "root") .option("starrocks.password", "") .option("starrocks.column.types", columnTypes) .mode("append") .save() ``` --- ### Apache Spark Load ### Load data in bulk using Spark Load This load uses external Apache Spark™ resources to pre-process imported data, which improves import performance and saves compute resources. It is mainly used for **initial migration** and **large data import** into StarRocks (data volume up to TB level). Spark load is an **asynchronous** import method that requires users to create Spark-type import jobs via the MySQL protocol and view the import results using `SHOW LOAD`. > **NOTICE** > > * Only users with the INSERT privilege on a StarRocks table can load data into this table. You can follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the required privilege. > * Spark Load can not be used to load data into a Primary Key table. #### Terminology explanation[​](#terminology-explanation "Direct link to Terminology explanation") * **Spark ETL**: Mainly responsible for ETL of data in the import process, including global dictionary construction (BITMAP type), partitioning, sorting, aggregation, etc. * **Broker**: Broker is an independent stateless process. It encapsulates the file system interface and provides StarRocks with the ability to read files from remote storage systems. * **Global Dictionary**: Saves the data structure that maps data from the original value to the encoded value. The original value can be any data type, while the encoded value is an integer. The global dictionary is mainly used in scenarios where exact count distinct is precomputed. #### Fundamentals[​](#fundamentals "Direct link to Fundamentals") The user submits a Spark type import job through the MySQL client;the FE records the metadata and returns the submission result. The execution of the spark load task is divided into the following main phases. 1. The user submits the spark load job to the FE. 2. The FE schedules the submission of the ETL task to the Apache Spark™ cluster for execution. 3. The Apache Spark™ cluster executes the ETL task that includes global dictionary construction (BITMAP type), partitioning, sorting, aggregation, etc. 4. After the ETL task is completed, the FE gets the data path of each preprocessed slice and schedules the relevant BE to execute the Push task. 5. The BE reads data through Broker process from HDFS and converts it into StarRocks storage format. > If you choose not to use Broker process, the BE reads data from HDFS directly. 6. The FE schedules the effective version and completes the import job. The following diagram illustrates the main flow of spark load. ![Spark load](/assets/images/4.3.2-1-7104f83d68f711e7dd819b5f11391eb7.png) *** #### Global Dictionary[​](#global-dictionary "Direct link to Global Dictionary") ##### Applicable Scenarios[​](#applicable-scenarios "Direct link to Applicable Scenarios") Currently, the BITMAP column in StarRocks is implemented using the Roaringbitmap, which only has integer to be the input data type. So if you want to implement precomputation for the BITMAP column in the import process, then you need to convert the input data type to integer. In the existing import process of StarRocks, the data structure of the global dictionary is implemented based on the Hive table, which saves the mapping from the original value to the encoded value. ##### Build Process[​](#build-process "Direct link to Build Process") 1. Read the data from the upstream data source and generate a temporary Hive table, named `hive-table`. 2. Extract the values of the de-emphasized fields of `hive-table` to generate a new Hive table named `distinct-value-table`. 3. Create a new global dictionary table named `dict-table` with one column for the original values and one column for the encoded values. 4. Left join between `distinct-value-table` and `dict-table`, and then use the window function to encode this set. Finally both the original value and the encoded value of the de-duplicated column are written back to `dict-table`. 5. Join between `dict-table` and `hive-table` to finish the job of replacing the original value in `hive-table` with the integer encoded value. 6. `hive-table` will be read by the next time data pre-processing, and then imported into StarRocks after calculation. #### Data Pre-processing[​](#data-pre-processing "Direct link to Data Pre-processing") The basic process of data pre-processing is as follows: 1. Read data from the upstream data source (HDFS file or Hive table). 2. Complete field mapping and calculation for the read data, then generate `bucket-id` based on the partition information. 3. Generate RollupTree based on the Rollup metadata of StarRocks table. 4. Iterate through the RollupTree and perform hierarchical aggregation operations. The Rollup of the next hierarchy can be calculated from the Rollup of the previous hierarchy. 5. Each time the aggregation calculation is completed, the data is bucketed according to `bucket-id` and then written to HDFS. 6. The subsequent Broker process will pull the files from HDFS and import them into the StarRocks BE node. #### Basic Operations[​](#basic-operations "Direct link to Basic Operations") ##### Configuring ETL Clusters[​](#configuring-etl-clusters "Direct link to Configuring ETL Clusters") Apache Spark™ is used as an external computational resource in StarRocks for ETL work. There may be other external resources added to StarRocks, such as Spark/GPU for query, HDFS/S3 for external storage, MapReduce for ETL, etc. Therefore, we introduce `Resource Management` to manage these external resources used by StarRocks. Before submitting a Apache Spark™ import job, configure the Apache Spark™ cluster for performing ETL tasks. The syntax for operation is as follows: ```sql -- create Apache Spark™ resource CREATE EXTERNAL RESOURCE resource_name PROPERTIES ( type = spark, spark_conf_key = spark_conf_value, working_dir = path, broker = broker_name, broker.property_key = property_value ); -- drop Apache Spark™ resource DROP RESOURCE resource_name; -- show resources SHOW RESOURCES SHOW PROC "/resources"; -- privileges GRANT USAGE_PRIV ON RESOURCE resource_name TO user_identityGRANT USAGE_PRIV ON RESOURCE resource_name TO ROLE role_name; REVOKE USAGE_PRIV ON RESOURCE resource_name FROM user_identityREVOKE USAGE_PRIV ON RESOURCE resource_name FROM ROLE role_name; ``` * Create resource **For example**: ```sql -- yarn cluster mode CREATE EXTERNAL RESOURCE "spark0" PROPERTIES ( "type" = "spark", "spark.master" = "yarn", "spark.submit.deployMode" = "cluster", "spark.jars" = "xxx.jar,yyy.jar", "spark.files" = "/tmp/aaa,/tmp/bbb", "spark.executor.memory" = "1g", "spark.yarn.queue" = "queue0", "spark.hadoop.yarn.resourcemanager.address" = "127.0.0.1:9999", "spark.hadoop.fs.defaultFS" = "hdfs://127.0.0.1:10000", "working_dir" = "hdfs://127.0.0.1:10000/tmp/starrocks", "broker" = "broker0", "broker.username" = "user0", "broker.password" = "password0" ); -- yarn HA cluster mode CREATE EXTERNAL RESOURCE "spark1" PROPERTIES ( "type" = "spark", "spark.master" = "yarn", "spark.submit.deployMode" = "cluster", "spark.hadoop.yarn.resourcemanager.ha.enabled" = "true", "spark.hadoop.yarn.resourcemanager.ha.rm-ids" = "rm1,rm2", "spark.hadoop.yarn.resourcemanager.hostname.rm1" = "host1", "spark.hadoop.yarn.resourcemanager.hostname.rm2" = "host2", "spark.hadoop.fs.defaultFS" = "hdfs://127.0.0.1:10000", "working_dir" = "hdfs://127.0.0.1:10000/tmp/starrocks", "broker" = "broker1" ); ``` `resource-name` is the name of the Apache Spark™ resource configured in StarRocks. `PROPERTIES` includes parameters relating to the Apache Spark™ resource, as follows: > **Note** > > For detailed description of Apache Spark™ resource PROPERTIES, please see [CREATE RESOURCE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/CREATE_RESOURCE.md) * Spark related parameters: * `type`: Resource type, required, currently only supports `spark`. * `spark.master`: Required, currently only supports `yarn`. * `spark.submit.deployMode`: The deployment mode of the Apache Spark™ program, required, currently supports both `cluster` and `client`. * `spark.hadoop.fs.defaultFS`: Required if master is yarn. * Parameters related to yarn resource manager, required. * one ResourceManager on a single node `spark.hadoop.yarn.resourcemanager.address`: Address of the single point resource manager. * ResourceManager HA > You can choose to specify ResourceManager's hostname or address. * `spark.hadoop.yarn.resourcemanager.ha.enabled`: Enable the resource manager HA, set to `true`. * `spark.hadoop.yarn.resourcemanager.ha.rm-ids`: list of resource manager logical ids. * `spark.hadoop.yarn.resourcemanager.hostname.rm-id`: For each rm-id, specify the hostname corresponding to the resource manager. * `spark.hadoop.yarn.resourcemanager.address.rm-id`: For each rm-id, specify `host:port` for the client to submit jobs to. * `*working_dir`: The directory used by ETL. Required if Apache Spark™ is used as an ETL resource. For example: `hdfs://host:port/tmp/starrocks`. * Broker related parameters: * `broker`: Broker name. Required if Apache Spark™ is used as an ETL resource. You need to use the `ALTER SYSTEM ADD BROKER` command to complete the configuration in advance. * `broker.property_key`: Information (e.g.authentication information) to be specified when Broker process reads the intermediate file generated by the ETL. **Precaution**: The above is a description of parameters for loading through Broker process. If you intend to load data without Broker process, the following should be noted. * You do not need to specify `broker`. * If you need to configure user authentication, and HA for NameNode nodes, you need to configure the parameters in the hdfs-site.xml file in the HDFS cluster, see [broker\_properties](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#hdfs) for descriptions of parameters. and you need to move the **hdfs-site.xml** file under **$FE\_HOME/conf** for each FE and **$BE\_HOME/conf** for each BE. > Note > > If the HDFS file can only be accessed by a specific user, you still need to specify the HDFS username in `broker.name` and the user password in `broker.password`. * View resources Regular accounts can only view resources to which they have `USAGE-PRIV` access. The root and admin accounts can view all resources. * Resource Permissions Resource permissions are managed through `GRANT REVOKE`, which currently only supports `USAGE-PRIV` permissions. You can give `USAGE-PRIV` permissions to a user or a role. ```sql -- Grant access to spark0 resources to user0 GRANT USAGE_PRIV ON RESOURCE "spark0" TO "user0"@"%"; -- Grant access to spark0 resources to role0 GRANT USAGE_PRIV ON RESOURCE "spark0" TO ROLE "role0"; -- Grant access to all resources to user0 GRANT USAGE_PRIV ON RESOURCE* TO "user0"@"%"; -- Grant access to all resources to role0 GRANT USAGE_PRIV ON RESOURCE* TO ROLE "role0"; -- Revoke the use privileges of spark0 resources from user user0 REVOKE USAGE_PRIV ON RESOURCE "spark0" FROM "user0"@"%"; ``` ##### Configuring Spark Client[​](#configuring-spark-client "Direct link to Configuring Spark Client") Configure the Spark client for FE so that the latter can submit Spark tasks by executing the `spark-submit` command. It is recommended to use the official version of Spark2 2.4.5 or above [spark download address](https://archive.apache.org/dist/spark/). After downloading, please use the following steps to complete the configuration. * Configure `SPARK-HOME` Place the Spark client in a directory on the same machine as the FE, and configure `spark_home_default_dir` in the FE configuration file to this directory, which by default is the `lib/spark2x` path in the FE root directory, and cannot be empty. * **Configure SPARK dependency package** To configure the dependency package, zip and archive all jar files in the jars folder under the Spark client, and configure the `spark_resource_path` item in the FE configuration to this zip file. If this configuration is empty, the FE will try to find the `lib/spark2x/jars/spark-2x.zip` file in the FE root directory. If the FE fails to find it, it will report an error. When the spark load job is submitted, the archived dependency files will be uploaded to the remote repository. The default repository path is under the `working_dir/{cluster_id}` directory named with `--spark-repository--{resource-name}`, which means that a resource in the cluster corresponds to a remote repository. The directory structure is referenced as follows: ```bash ---spark-repository--spark0/ |---archive-1.0.0/ | |\---lib-990325d2c0d1d5e45bf675e54e44fb16-spark-dpp-1.0.0\-jar-with-dependencies.jar | |\---lib-7670c29daf535efe3c9b923f778f61fc-spark-2x.zip |---archive-1.1.0/ | |\---lib-64d5696f99c379af2bee28c1c84271d5-spark-dpp-1.1.0\-jar-with-dependencies.jar | |\---lib-1bbb74bb6b264a270bc7fca3e964160f-spark-2x.zip |---archive-1.2.0/ | |-... ``` In addition to the spark dependencies (named `spark-2x.zip` by default), the FE also uploads the DPP dependencies to the remote repository. If all the dependencies submitted by the spark load already exist in the remote repository, then there is no need to upload the dependencies again, saving the time of repeatedly uploading a large number of files each time. ##### Configuring YARN Client[​](#configuring-yarn-client "Direct link to Configuring YARN Client") Configure the yarn client for FE so that the FE can execute yarn commands to get the status of the running application or kill it.It is recommended to use the official version of Hadoop2 2.5.2 or above ([hadoop download address](https://archive.apache.org/dist/hadoop/common/)). After downloading, please use the following steps to complete the configuration: * **Configure the YARN executable path** Place the downloaded yarn client in a directory on the same machine as the FE, and configure the `yarn_client_path` item in the FE configuration file to the binary executable file of yarn, which by default is the `lib/yarn-client/hadoop/bin/yarn` path in the FE root directory. * **Configure the path to the configuration file needed to generate YARN (optional)** When the FE goes through the yarn client to get the status of the application, or to kill the application, by default StarRocks generates the configuration file required to execute the yarn command in the `lib/yarn-config` path of the FE root directory This path can be modified by configuring the `yarn_config_dir` entry in the FE configuration file, which currently includes `core-site.xml` and `yarn-site.xml`. ##### Create Import Job[​](#create-import-job "Direct link to Create Import Job") **Syntax:** ```sql LOAD LABEL load_label (data_desc, ...) WITH RESOURCE resource_name [resource_properties] [PROPERTIES (key1=value1, ... )] * load_label: db_name.label_name * data_desc: DATA INFILE ('file_path', ...) [NEGATIVE] INTO TABLE tbl_name [PARTITION (p1, p2)] [COLUMNS TERMINATED BY separator ] [(col1, ...)] [COLUMNS FROM PATH AS (col2, ...)] [SET (k1=f1(xx), k2=f2(xx))] [WHERE predicate] DATA FROM TABLE hive_external_tbl [NEGATIVE] INTO TABLE tbl_name [PARTITION (p1, p2)] [SET (k1=f1(xx), k2=f2(xx))] [WHERE predicate] * resource_properties: (key2=value2, ...) ``` **Example 1**: The case where the upstream data source is HDFS ```sql LOAD LABEL db1.label1 ( DATA INFILE("hdfs://abc.com:8888/user/starrocks/test/ml/file1") INTO TABLE tbl1 COLUMNS TERMINATED BY "," (tmp_c1,tmp_c2) SET ( id=tmp_c2, name=tmp_c1 ), DATA INFILE("hdfs://abc.com:8888/user/starrocks/test/ml/file2") INTO TABLE tbl2 COLUMNS TERMINATED BY "," (col1, col2) where col1 > 1 ) WITH RESOURCE 'spark0' ( "spark.executor.memory" = "2g", "spark.shuffle.compress" = "true" ) PROPERTIES ( "timeout" = "3600" ); ``` **Example 2**: The case where the upstream data source is Hive. * Step 1: Create a new hive resource ```sql CREATE EXTERNAL RESOURCE hive0 PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:8080" ); ``` * Step 2: Create a new hive external table ```sql CREATE EXTERNAL TABLE hive_t1 ( k1 INT, K2 SMALLINT, k3 varchar(50), uuid varchar(100) ) ENGINE=hive PROPERTIES ( "resource" = "hive0", "database" = "tmp", "table" = "t1" ); ``` * Step 3: Submit the load command, requiring that the columns in the imported StarRocks table exist in the hive external table. ```sql LOAD LABEL db1.label1 ( DATA FROM TABLE hive_t1 INTO TABLE tbl1 SET ( uuid=bitmap_dict(uuid) ) ) WITH RESOURCE 'spark0' ( "spark.executor.memory" = "2g", "spark.shuffle.compress" = "true" ) PROPERTIES ( "timeout" = "3600" ); ``` Introduction to the parameters in the Spark load: * **Label** Label of the import job. Each import job has a Label that is unique within the database, following the same rules as broker load. * **Data description class parameters** Currently, supported data sources are CSV and Hive table. Other rules are the same as broker load. * **Import Job Parameters** Import job parameters refer to the parameters belonging to the `opt_properties` section of the import statement. These parameters are applicable to the entire import job. The rules are the same as broker load. * **Spark Resource Parameters** Spark resources need to be configured into StarRocks in advance and users need to be given USAGE-PRIV permissions before they can apply the resources to Spark load. Spark resource parameters can be set when the user has a temporary need, such as adding resources for a job and modifying Spark configs. The setting only takes effect on this job and does not affect the existing configurations in the StarRocks cluster. ```sql WITH RESOURCE 'spark0' ( "spark.driver.memory" = "1g", "spark.executor.memory" = "3g" ) ``` * **Import when the data source is Hive** Currently, to use a Hive table in the import process, you need to create an external table of the `Hive` type and then specify its name when submitting the import command. * **Import process to build a global dictionary** In the load command, you can specify the required fields for building the global dictionary in the following format: `StarRocks field name=bitmap_dict(hive table field name)` Note that currently **the global dictionary is only supported when the upstream data source is a Hive table**. * **Load binary type data** Since v2.5.17, Spark Load supports the bitmap\_from\_binary function, which can convert binary data into bitmap data. If the column type of the Hive table or HDFS file is binary and the corresponding column in the StarRocks table is a bitmap-type aggregate column, you can specify the fields in the load command in the following format, `StarRocks field name=bitmap_from_binary(Hive table field name)`. This eliminates the need for building a global dictionary. #### Viewing Import Jobs[​](#viewing-import-jobs "Direct link to Viewing Import Jobs") The Spark load import is asynchronous, as is the broker load. The user must record the label of the import job and use it in the `SHOW LOAD` command to view the import results. The command to view the import is common to all import methods. The example is as follows. Refer to Broker Load for a detailed explanation of returned parameters.The differences are as follows. ```sql mysql> show load order by createtime desc limit 1\G *************************** 1. row *************************** JobId: 76391 Label: label1 State: FINISHED Progress: ETL:100%; LOAD:100% Type: SPARK EtlInfo: unselected.rows=4; dpp.abnorm.ALL=15; dpp.norm.ALL=28133376 TaskInfo: cluster:cluster0; timeout(s):10800; max_filter_ratio:5.0E-5 ErrorMsg: N/A CreateTime: 2019-07-27 11:46:42 EtlStartTime: 2019-07-27 11:46:44 EtlFinishTime: 2019-07-27 11:49:44 LoadStartTime: 2019-07-27 11:49:44 LoadFinishTime: 2019-07-27 11:50:16 URL: http://1.1.1.1:8089/proxy/application_1586619723848_0035/ JobDetails: {"ScannedRows":28133395,"TaskNumber":1,"FileNumber":1,"FileSize":200000} ``` * **State** The current stage of the imported job. PENDING: The job is committed. ETL: Spark ETL is committed. LOADING: The FE schedule an BE to execute push operation. FINISHED: The push is completed and the version is effective. There are two final stages of the import job – `CANCELLED` and `FINISHED`, both indicating the load job is completed. `CANCELLED` indicates import failure and `FINISHED` indicates import success. * **Progress** Description of the import job progress. There are two types of progress –ETL and LOAD, which correspond to the two phases of the import process, ETL and LOADING. * The range of progress for LOAD is 0~100%. `LOAD progress = the number of currently completed tablets of all replications imports / the total number of tablets of this import job * 100%`. * If all tables have been imported, the LOAD progress is 99%, and changes to 100% when the import enters the final validation phase. * The import progress is not linear. If there is no change in progress for a period of time, it does not mean that the import is not executing. * **Type** The type of the import job. SPARK for spark load. * **CreateTime/EtlStartTime/EtlFinishTime/LoadStartTime/LoadFinishTime** These values represent the time when the import was created, when the ETL phase started, when the ETL phase completed, when the LOADING phase started, and when the entire import job was completed. * **JobDetails** Displays the detailed running status of the job, including the number of imported files, total size (in bytes), number of subtasks, number of raw rows being processed, etc. For example: ```json {"ScannedRows":139264,"TaskNumber":1,"FileNumber":1,"FileSize":940754064} ``` * **URL** You can copy the input to your browser to access the web interface of the corresponding application. ##### View Apache Spark™ Launcher commit logs[​](#view-apache-spark-launcher-commit-logs "Direct link to View Apache Spark™ Launcher commit logs") Sometimes users need to view the detailed logs generated during a Apache Spark™ job commit. By default, the logs are saved in the path `log/spark_launcher_log` in the FE root directory named as `spark-launcher-{load-job-id}-{label}.log`. The logs are saved in this directory for a period of time and will be erased when the import information in FE metadata is cleaned up. The default retention time is 3 days. ##### Cancel Import[​](#cancel-import "Direct link to Cancel Import") When the Spark load job status is not `CANCELLED` or `FINISHED`, it can be cancelled manually by the user by specifying the Label of the import job. *** #### Related System Configurations[​](#related-system-configurations "Direct link to Related System Configurations") **FE Configuration:** The following configuration is the system-level configuration of Spark load, which applies to all Spark load import jobs. The configuration values can be adjusted mainly by modifying `fe.conf`. * enable-spark-load: Enable Spark load and resource creation with a default value of false. * spark-load-default-timeout-second: The default timeout for the job is 259200 seconds (3 days). * spark-home-default-dir: The Spark client path (`fe/lib/spark2x`). * spark-resource-path: The path to the packaged S park dependency file (empty by default). * spark-launcher-log-dir: The directory where the commit log of the Spark client is stored (`fe/log/spark-launcher-log`). * yarn-client-path: The path to the yarn binary executable (`fe/lib/yarn-client/hadoop/bin/yarn`). * yarn-config-dir: Yarn's configuration file path (`fe/lib/yarn-config`). *** #### Best Practices[​](#best-practices "Direct link to Best Practices") The most suitable scenario for using Spark load is when the raw data is in the file system (HDFS) and the data volume is in the tens of GB to TB level. Use Stream Load or Broker Load for smaller data volumes. For the full spark load import example, refer to the demo on github: #### FAQs[​](#faqs "Direct link to FAQs") * `Error: When running with master 'yarn' either HADOOP-CONF-DIR or YARN-CONF-DIR must be set in the environment.` Using Spark Load without configuring the `HADOOP-CONF-DIR` environment variable in `spark-env.sh` of the Spark client. * `Error: Cannot run program "xxx/bin/spark-submit": error=2, No such file or directory` The `spark_home_default_dir` configuration item does not specify the Spark client root directory when using Spark Load. * `Error: File xxx/jars/spark-2x.zip does not exist.` The `spark-resource-path` configuration item does not point to the packed zip file when using Spark load. * `Error: yarn client does not exist in path: xxx/yarn-client/hadoop/bin/yarn` The yarn-client-path configuration item does not specify the yarn executable when using Spark load. * `ERROR: Cannot execute hadoop-yarn/bin/... /libexec/yarn-config.sh` When using Hadoop with CDH, you need to configure the `HADOOP_LIBEXEC_DIR` environment variable. Since `hadoop-yarn` and hadoop directories are different, the default `libexec` directory will look for `hadoop-yarn/bin/... /libexec`, while `libexec` is in the hadoop directory. The \`\`\`yarn application status\`\` command to get the Spark task status reported an error causing the import job to fail. --- ### Business Intelligence > Tip: If a BI tool relies on the string max length in query metadata and that length is smaller than actual values (causing empty returns), set the global variable `max_unknown_string_meta_length` (1~1,048,576, default 64) to a higher value so string columns return that length in metadata. #### [📄️ Apache Superset](https://docs.starrocks.io/docs/integrations/BI_integrations/Superset.md) [Apache Superset supports querying and visualizing both internal data and external data in StarRocks.](https://docs.starrocks.io/docs/integrations/BI_integrations/Superset.md) #### [📄️ FineBI](https://docs.starrocks.io/docs/integrations/BI_integrations/FineBI.md) [How to connect FineBI to StarRocks for visualized data analysis.](https://docs.starrocks.io/docs/integrations/BI_integrations/FineBI.md) #### [📄️ Hex](https://docs.starrocks.io/docs/integrations/BI_integrations/Hex.md) [Hex supports querying and visualizing both internal data and external data in StarRocks.](https://docs.starrocks.io/docs/integrations/BI_integrations/Hex.md) #### [📄️ Querybook](https://docs.starrocks.io/docs/integrations/BI_integrations/Querybook.md) [Querybook supports querying and visualizing both internal data and external data in StarRocks.](https://docs.starrocks.io/docs/integrations/BI_integrations/Querybook.md) #### [📄️ QuickBI](https://docs.starrocks.io/docs/integrations/BI_integrations/QuickBI.md) [How to connect QuickBI to StarRocks for visualized data analysis.](https://docs.starrocks.io/docs/integrations/BI_integrations/QuickBI.md) #### [📄️ Rill](https://docs.starrocks.io/docs/integrations/BI_integrations/Rill.md) [Rill supports connecting to StarRocks as an OLAP connector to power Rill dashboards with external tables.](https://docs.starrocks.io/docs/integrations/BI_integrations/Rill.md) #### [📄️ Tableau](https://docs.starrocks.io/docs/integrations/BI_integrations/Tableau_Desktop.md) [How to connect StarRocks to Tableau Desktop and Tableau Server using the StarRocks Tableau JDBC Connector.](https://docs.starrocks.io/docs/integrations/BI_integrations/Tableau_Desktop.md) #### [📄️ Metabase](https://docs.starrocks.io/docs/integrations/BI_integrations/Metabase.md) [Metabase supports querying and visualizing both internal data and external data in StarRocks.](https://docs.starrocks.io/docs/integrations/BI_integrations/Metabase.md) --- ### FineBI This topic describes how to connect FineBI to StarRocks to perform visualized analysis of StarRocks data on FineBI. For details about how to connect to StarRocks on FineBI, see [StarRocks data connection](https://help.fanruan.com/finebi-en/doc-view-5857.html). --- ### Hex Hex supports querying and visualizing both internal data and external data in StarRocks. Add a data connection in Hex. Note that you must select MySQL as the connection type. ![Hex](/assets/images/BI_hex_1-b380296e12a2537f7d48d3f36fc4e094.png) The parameters that you need to configure are described as follows: * **Name**: the name of the data connection. * **Host & port**: the FE host IP address and FE query port of your StarRocks cluster. An example query port is `9030`. * **Database**: the data source that you want to access in your StarRocks cluster. The value of this parameter is in the `.` format. * `catalog_name`: the name of the target catalog in your StarRocks cluster. Both internal and external catalogs are supported. * `database_name`: the name of the target database in your StarRocks cluster. Both internal and external databases are supported. * **Type**: the authentication method that you want to use. Select **Password**. * **User**: the username that is used to log in to your StarRocks cluster, for example, `admin`. * **Password**: the password that is used to log in to your StarRocks cluster. --- ### Metabase Metabase supports querying and visualizing both internal data and external data in StarRocks. Start Metabase and do as follows: 1. In the upper-right corner of the Metabase homepage, click the **Settings** icon and choose **Admin settings**. ![Metabase - Admin settings](/assets/images/Metabase_1-2f1158158723a03d42120e9de4768f3f.png) 2. Choose **Databases** in the top menu bar. 3. On the **Databases** page, click **Add database**. ![Metabase - Add database](/assets/images/Metabase_2-2e0e966419eaa3c520d7b02f9de4eeae.png) 4. On the page that appears, configure the database parameters and click **Save**. * **Database type**: Select **MySQL**. * **Host** and **Port**: Enter the host and port information appropriate for your use case. * **Database name**: Enter a database name in the `.` format. In StarRocks versions earlier than v3.2, you can integrate only the internal catalog of your StarRocks cluster with Metabase. From StarRocks v3.2 onwards, you can integrate both the internal catalog and external catalogs of your StarRocks cluster with Metabase. * **Username** and **Password**: Enter the username and password of your StarRocks cluster user. * **Additional JDBC connection string options**: You must add the property `tinyInt1isBit=false` in this field. Otherwise, there may be an error. The other parameters do not involve StarRocks. Configure them based on your business needs. ![Metabase - Configure database](/assets/images/Metabase_3-c7994bd1edc973e10c1ccce5b63a0883.png) NOTE: Please avoid using DECIMAL data types as Metabase does not understand this StarRocks specific column data type --- ### Querybook Querybook supports querying and visualizing both internal data and external data in StarRocks. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Make sure that you have finished the following preparations: 1. Clone and download the Querybook repository. ```sql git clone git@github.com:pinterest/querybook.git cd querybook ``` 2. Create a file named `local.txt` under the `requirements` folder in the project's root directory. ```sql touch requirements/local.txt ``` 3. Add the required packages. ```sql echo -e "starrocks\nmysqlclient" > requirements/local.txt ``` 4. Start the container. ```sql make ``` #### Integration[​](#integration "Direct link to Integration") Visit the following URL and add a new query engine: ```plain https://localhost:10001/admin/query_engine/ ``` ![Querybook](/assets/images/BI_querybook_1-4805d379448a5c6df3b524aad977f499.png) Take note of the following points: * For **Language**, select **Starrocks**. * For **Executor**, select **sqlalchemy**. * For **Connection\_string**, enter a URI in the StarRocks SQLAlchemy URI format as below: ```sql starrocks://:@:/. ``` The parameters in the URI are described as follows: * `User`: the username that is used to log in to your StarRocks cluster, for example, `admin`. * `Password`: the password that is used to log in to your StarRocks cluster. * `Host`: the FE host IP address of your StarRocks cluster. * `Port`: the FE query port of your StarRocks cluster, for example, `9030`. * `Catalog`: the target catalog in your StarRocks cluster. Both internal and external catalogs are supported. * `Database`: the target database in your StarRocks cluster. Both internal and external databases are supported. --- ### QuickBI This topic describes how to connect QuickBI to StarRocks to perform visualized analysis of StarRocks data on QuickBI. For details about how to connect to StarRocks on QuickBI, see [Add a user-created StarRocks data source](https://www.alibabacloud.com/help/en/quick-bi/user-guide/add-a-user-created-starrocks-data-source). --- ### Rill Rill supports connecting to StarRocks as an OLAP connector to power Rill dashboards with external tables. Rill can query and visualize both internal data and external data in StarRocks. #### Connection[​](#connection "Direct link to Connection") Rill connects to StarRocks using the MySQL protocol. You can configure the connection using either connection parameters or a DSN connection string. ##### Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters") When adding a data source in Rill, select **StarRocks** and configure the following parameters: * **Host**: The FE host IP address or hostname of your StarRocks cluster * **Port**: The MySQL protocol port of the StarRocks FE (default: `9030`) * **Username**: The username for authentication (default: `root`) * **Password**: The password for authentication * **Catalog**: The StarRocks catalog name (default: `default_catalog`). Supports both internal and external catalogs (e.g., Iceberg, Hive) * **Database**: The StarRocks database name * **SSL**: Enable SSL/TLS encryption (default: `false`) ##### Connection String (DSN)[​](#connection-string-dsn "Direct link to Connection String (DSN)") Alternatively, you can use a MySQL-format DSN connection string: ```text user:password@tcp(host:9030)/database?parseTime=true ``` For external catalogs, specify the catalog and database as separate properties. #### External Catalogs[​](#external-catalogs "Direct link to External Catalogs") Rill supports querying data from external catalogs in StarRocks, including Hive, Iceberg, Delta Lake, and other external data sources. Set the `catalog` property to your external catalog name (e.g., `iceberg_catalog`) and the `database` property to the database within that catalog. #### More Information[​](#more-information "Direct link to More Information") For detailed configuration options, examples, troubleshooting, and the most up-to-date information, see the [Rill Data StarRocks connector documentation](https://docs.rilldata.com/developers/build/connectors/olap/starrocks). --- ### Apache Superset Apache Superset supports querying and visualizing both internal data and external data in StarRocks. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Make sure that you have finished the following installations: 1. Install the Python client for StarRocks on your Apache Superset server. ```sql pip install starrocks ``` 2. Install the latest version of Apache Superset. For more information, see [Installing Superset](https://superset.apache.org/docs/intro). #### Integration[​](#integration "Direct link to Integration") Create a database in Apache Superset: ![Apache Superset - 1](/assets/images/BI_superset_1-eff12bff07257eed12c79806812c0da6.png) ![Apache Superset - 2](/assets/images/BI_superset_2-0f8ec523d0d808a9e157ab1a0e91c69a.png) Take note of the following points: * For **SUPPORTED DATABASES**, select **StarRocks**, which will be used as the data source. * For **SQLALCHEMY** **URI**, enter a URI in the StarRocks SQLAlchemy URI format as below: ```sql starrocks://:@:/. ``` The parameters in the URI are described as follows: * `User`: the username that is used to log in to your StarRocks cluster, for example, `admin`. * `Password`: the password that is used to log in to your StarRocks cluster. * `Host`: the FE host IP address of your StarRocks cluster. * `Port`: the FE query port of your StarRocks cluster, for example, `9030`. * `Catalog`: the target catalog in your StarRocks cluster. Both internal and external catalogs are supported. * `Database`: the target database in your StarRocks cluster. Both internal and external databases are supported. --- ### Tableau This topic describes how to connect StarRocks to Tableau Desktop and Tableau Server with StarRocks Tableau JDBC Connector. #### Overview[​](#overview "Direct link to Overview") The StarRocks Tableau JDBC Connector is a custom extension for Tableau Desktop and Tableau Server. It simplifies the process of connecting Tableau to StarRocks and enhances support for standard Tableau functionality, outperforming the default Generic ODBC/JDBC connection. ##### Key Features[​](#key-features "Direct link to Key Features") * LDAP Support: Enables LDAP login with password prompts for secure authentication. * High Compatibility: Achieves 99.99% compatibility in TDVT (Tableau Design Verification Tool) testing, with only one minor failure case. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before proceeding, make sure the following requirements are met: * Tableau Version: Tableau 2020.4 and later * StarRocks Version: v3.2 and later #### Install Connector for Tableau Desktop[​](#install-connector-for-tableau-desktop "Direct link to Install Connector for Tableau Desktop") 1. Download the [MySQL JDBC Driver 8.0.33](https://downloads.mysql.com/archives/c-j/). 2. Store the driver file in the following directory (create the directory if it does not exist): * macOS: `~/Library/Tableau/Drivers` * Windows: `C:\Program Files\Tableau\Drivers` 3. Download the [StarRocks Tableau JDBC Connector file](https://exchange.tableau.com/products/1079). 4. Store the connector file in the following directory: * macOS: `~/Documents/My Tableau Repository/Connectors` * Windows: `C:\Users\[Windows User]\Documents\My Tableau Repository\Connectors` 5. Launch Tableau Desktop. 6. Navigate to **Connect** -> **To a Server** -> **StarRocks JDBC by CelerData**. #### Install Connector for Tableau Server[​](#install-connector-for-tableau-server "Direct link to Install Connector for Tableau Server") 1. Download the [MySQL JDBC Driver 8.0.33](https://downloads.mysql.com/archives/c-j/). 2. Store the driver file in the following directory (create the directory if it does not exist): * Linux: `/opt/tableau/tableau_driver/jdbc` * Windows: `C:\Program Files\Tableau\Drivers` info On Linux, you must permit the "Tableau" user to access the directory. Follow these steps: 1. Create the directory and copy the driver file to the directory: ```bash sudo mkdir -p /opt/tableau/tableau_driver/jdbc # Replace with the absolute path of the driver file. sudo cp /.jar /opt/tableau/tableau_driver/jdbc ``` 2. Grant permission to the the "Tableau" user. ```bash # Replace with the name of the driver file. sudo chmod 755 /opt/tableau/tableau_driver/jdbc/.jar ``` 3. Download the [StarRocks Tableau JDBC Connector file](https://exchange.tableau.com/products/1079). 4. Store the connector file in the following directory of each node: * Linux: `/opt/tableau/connectors` * Windows: `C:\Program Files\Tableau\Connectors` 5. Restart Tableau Server. ```bash tsm restart ``` info You must restart Tableau Server to apply the changes whenever you add, remove, or update a connector. #### Usage notes[​](#usage-notes "Direct link to Usage notes") If LDAP login support is required, you can tick the **Enable LDAP** switch in the **Advanced** tab during configuration. --- ### CSP authentication #### [📄️ AWS](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md) [How to configure StarRocks to authenticate with AWS for accessing S3 and other AWS resources.](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md) #### [📄️ Microsoft Azure Storage](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_azure_storage.md) [How to configure StarRocks to authenticate with Azure Blob Storage and Azure Data Lake Storage.](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_azure_storage.md) #### [📄️ Google Cloud Storage (GCS)](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_gcs.md) [How to configure StarRocks to authenticate with Google Cloud Storage (GCS).](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_gcs.md) --- ### Authenticate to AWS resources StarRocks supports using three authentication methods to integrate with AWS resources: instance profile-based authentication, assumed role-based authentication, and IAM user-based authentication. This topic describes how to configure AWS credentials by using these authentication methods. #### Authentication methods[​](#authentication-methods "Direct link to Authentication methods") ##### Instance profile-based authentication[​](#instance-profile-based-authentication "Direct link to Instance profile-based authentication") The instance profile-based authentication method allows your StarRocks cluster to inherit the privileges specified in the instance profile of the EC2 instance on which the cluster runs. In theory, any cluster user who can log in to the cluster can perform permitted actions on your AWS resources according to the AWS IAM policies you have configured. The typical scenario for this use case is that you do not need any AWS resource access control between multiple cluster users in the cluster. This authentication method means no isolation is required within the same cluster. However, this authentication method still can be seen as a cluster-level safe access control solution, because whoever can log in to the cluster is controlled by the cluster administrator. ##### Assumed role-based authentication[​](#assumed-role-based-authentication "Direct link to Assumed role-based authentication") Unlike instance profile-based authentication, the assumed role-based authentication method supports assuming an AWS IAM role to gain access to your AWS resources. For more information, see [Assuming a role](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-sharing-logs.html). ##### IAM user-based authentication[​](#iam-user-based-authentication "Direct link to IAM user-based authentication") The IAM user-based authentication method supports using IAM user credentials to gain access to your AWS resources. For more information, see [IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html). #### Preparations[​](#preparations "Direct link to Preparations") First, find the IAM role associated with the EC2 instance on which your StarRocks cluster runs (that role is referred to as the EC2 instance role hereinafter in this topic), and obtain the role's ARN. You will need the EC2 instance role for instance profile-based authentication and need the EC2 instance role and its ARN for assumed role-based authentication. Next step, create an IAM policy based on the type of AWS resource you want to access and the specific operation scenario within StarRocks. A policy in AWS IAM declares a set of permissions on a specific AWS resource. After creating a policy, you need to attach it to an IAM role or user. As such, the IAM role or user is assigned the permissions declared in the policy to access the specified AWS resource. > **NOTICE** > > To make these preparations, you must have permission to sign in to the [AWS IAM console](https://us-east-1.console.aws.amazon.com/iamv2/home#/home) and edit IAM users and roles. For the IAM policy you will need to access a specific AWS resource, see the following sections: * [Batch load data from AWS S3](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md#batch-load-data-from-aws-s3) * [Read/write AWS S3](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md#readwrite-aws-s3) * [Integrate with AWS Glue](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md#integrate-with-aws-glue) ##### Preparation for instance profile-based authentication[​](#preparation-for-instance-profile-based-authentication "Direct link to Preparation for instance profile-based authentication") Attach the [IAM policies](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md) for accessing the required AWS resources to the EC2 instance role. ##### Preparation for assumed role-based authentication[​](#preparation-for-assumed-role-based-authentication "Direct link to Preparation for assumed role-based authentication") ###### Create IAM roles and attach policies to them[​](#create-iam-roles-and-attach-policies-to-them "Direct link to Create IAM roles and attach policies to them") Create one or more IAM roles, depending on the AWS resources you want to access. See [Creating IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html). Then, attach the [IAM policies](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md) for accessing the required AWS resources to the IAM roles you create. For example, you want your StarRocks cluster to access AWS S3 and AWS Glue. In this situation, you can choose to create one IAM role (for example, `s3_assumed_role`), and attach the policy for accessing AWS S3 and the policy for accessing AWS Glue both to that role. Alternatively, you can choose to create two different IAM roles (for example, `s3_assumed_role` and `glue_assumed_role`), and attach these policies to the two different roles respectively (that is, attach the policy for accessing AWS S3 to `s3_assumed_role` and the policy for accessing AWS Glue to `glue_assumed_role`). The IAM roles you create will be assumed by the EC2 instance role of the StarRocks cluster to access the specified AWS resources. This section assumes that you have created only one assumed role, `s3_assumed_role`, and have added the policy for accessing AWS S3 and the policy for accessing AWS Glue both to that role. ###### Configure a trust relationship[​](#configure-a-trust-relationship "Direct link to Configure a trust relationship") Configure your assumed role as follows: 1. Sign in to the [AWS IAM console](https://us-east-1.console.aws.amazon.com/iamv2/home#/home). 2. In the left-side navigation pane, choose **Access management** > **Roles**. 3. Find the assumed role (`s3_assumed_role`) and click its name. 4. On the role's details page, click the **Trust relationships** tab, and on the **Trust relationships** tab click **Edit trust policy**. 5. On the **Edit trust policy** page, delete the existing JSON policy document, and paste the following IAM policy, in which you must replace `` with the EC2 instance role's ARN you have obtained above. Then, click **Update policy**. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "sts:AssumeRole" } ] } ``` If you have created different assumed roles for accessing different AWS resources, you need to repeat the preceding steps to configure your other assumed roles. For example, you have created `s3_assumed_role` and `glue_assumed_role` for accessing AWS S3 and AWS Glue respectively. In this situation, you need to repeat the preceding steps to configure `glue_assumed_role`. Configure your EC2 instance role as follows: 1. Sign in to the [AWS IAM console](https://us-east-1.console.aws.amazon.com/iamv2/home#/home). 2. In the left-side navigation pane, choose **Access management** > **Roles**. 3. Find the EC2 instance role and click its name. 4. In the **Permissions policies** section of the role's details page, click **Add permissions** and choose **Create inline policy**. 5. In the **Specify permissions** step, click the **JSON** tab, delete the existing JSON policy document, and paste the following IAM policy, in which you must replace `` with the ARN of the assumed role `s3_assumed_role`. Then, click **Review policy**. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["sts:AssumeRole"], "Resource": [ "" ] } ] } ``` If you have created different assumed roles for accessing different AWS resources, you need to fill the ARNs of all these assumed roles in the **Resource** element of the preceding IAM policy and separate them with a comma (,). For example, you have created`s3_assumed_role` and `glue_assumed_role` for accessing AWS S3 and AWS Glue respectively. In this situation, you need to fill the ARN of `s3_assumed_role` and the ARN of `glue_assumed_role` in the **Resource** element by using the following format: `"",""`. 6. In the **Review Policy** step, enter a policy name and click **Create policy**. ##### Preparation for IAM user-based authentication[​](#preparation-for-iam-user-based-authentication "Direct link to Preparation for IAM user-based authentication") Create an IAM user. See [Creating an IAM user in your AWS account](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html). Then, attach the [IAM policies](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md) for accessing the required AWS resources to the IAM user you create. #### Comparison between authentication methods[​](#comparison-between-authentication-methods "Direct link to Comparison between authentication methods") The following figure provides a high-level explanation of the differences in mechanism between instance profile-based authentication, assumed role-based authentication, and IAM user-based authentication in StarRocks. ![Comparison between authentication methods](/assets/images/authenticate_s3_credential_methods-1f1020387bebbf0fdadc38a85205940d.png) #### Build connections with AWS resources[​](#build-connections-with-aws-resources "Direct link to Build connections with AWS resources") ##### Authentication parameters for accessing AWS S3[​](#authentication-parameters-for-accessing-aws-s3 "Direct link to Authentication parameters for accessing AWS S3") In various scenarios in which StarRocks needs to integrate with AWS S3, for example, when you create external catalogs or file external tables or when you ingest, back up, or restore data from AWS S3, configure the authentication parameters for accessing AWS S3 as follows: * For instance profile-based authentication, set `aws.s3.use_instance_profile` to `true`. * For assumed role-based authentication, set `aws.s3.use_instance_profile` to `true` and configure `aws.s3.iam_role_arn` as the assumed role's ARN that you use to access AWS S3 (for example, the ARN of the assumed role `s3_assumed_role` you have created above). * For IAM user-based authentication, set `aws.s3.use_instance_profile` to `false` and configure `aws.s3.access_key` and `aws.s3.secret_key` as the access key and secret key of your AWS IAM user. The following table describes the parameters. | Parameter | Required | Description | | ----------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. | | aws.s3.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS S3 bucket. If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.access\_key | No | The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.secret\_key | No | The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | ##### Authentication parameters for accessing AWS Glue[​](#authentication-parameters-for-accessing-aws-glue "Direct link to Authentication parameters for accessing AWS Glue") In various scenarios in which StarRocks needs to integrate with AWS Glue, for example, when you create external catalogs, configure the authentication parameters for accessing AWS Glue as follows: * For instance profile-based authentication, set `aws.glue.use_instance_profile` to `true`. * For assumed role-based authentication, set `aws.glue.use_instance_profile` to `true` and configure `aws.glue.iam_role_arn` as the assumed role's ARN that you use to access AWS Glue (for example, the ARN of the assumed role `glue_assumed_role` you have created above). * For IAM user-based authentication, set `aws.glue.use_instance_profile` to `false` and configure `aws.glue.access_key` and `aws.glue.secret_key` as the access key and secret key of your AWS IAM user. The following table describes the parameters. | Parameter | Required | Description | | ------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.glue.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication. Valid values: `true` and `false`. Default value: `false`. | | aws.glue.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS Glue Data Catalog. If you use the assumed role-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.access\_key | No | The access key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.secret\_key | No | The secret key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | #### Integration examples[​](#integration-examples "Direct link to Integration examples") ##### External catalog[​](#external-catalog "Direct link to External catalog") Creating an external catalog in your StarRocks cluster means building integration with the target data lake system, which is composed of two key components: * File storage like AWS S3 to store table files * Metastore like Hive metastore or AWS Glue to store the metadata and locations of table files StarRocks supports the following types of catalogs: * [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md) * [Iceberg catalog](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md) * [Hudi catalog](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md) * [Delta Lake catalog](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md) The following examples create a Hive catalog named `hive_catalog_hms` or `hive_catalog_glue`, depending on the type of metastore you use, to query data from your Hive cluster. For detailed syntax and parameters, see [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md). ###### Instance profile-based authentication[​](#instance-profile-based-authentication-1 "Direct link to Instance profile-based authentication") * If you use Hive metastore in your Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` * If you use AWS Glue in your Amazon EMR Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_glue PROPERTIES ( "type" = "hive", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "us-west-2" ); ``` ###### Assumed role-based authentication[​](#assumed-role-based-authentication-1 "Direct link to Assumed role-based authentication") * If you use Hive metastore in your Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/s3_assumed_role", "aws.s3.region" = "us-west-2", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` * If you use AWS Glue in your Amazon EMR Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_glue PROPERTIES ( "type" = "hive", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/s3_assumed_role", "aws.s3.region" = "us-west-2", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "arn:aws:iam::081976408565:role/glue_assumed_role", "aws.glue.region" = "us-west-2" ); ``` ###### IAM user-based authentication[​](#iam-user-based-authentication-1 "Direct link to IAM user-based authentication") * If you use Hive metastore in your Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` * If you use AWS Glue in your Amazon EMR Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_glue PROPERTIES ( "type" = "hive", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "us-west-2" ); ``` ##### File external table[​](#file-external-table "Direct link to File external table") File external tables must be created in your internal catalog named `default_catalog`. The following examples create a file external table named `file_table` on an existing database named `test_s3_db`. For detailed syntax and parameters, see [File external table](https://docs.starrocks.io/docs/data_source/file_external_table.md). ###### Instance profile-based authentication[​](#instance-profile-based-authentication-2 "Direct link to Instance profile-based authentication") Run a command like below: ```sql CREATE EXTERNAL TABLE test_s3_db.file_table ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "s3://starrocks-test/", "format" = "ORC", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` ###### Assumed role-based authentication[​](#assumed-role-based-authentication-2 "Direct link to Assumed role-based authentication") Run a command like below: ```sql CREATE EXTERNAL TABLE test_s3_db.file_table ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "s3://starrocks-test/", "format" = "ORC", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/s3_assumed_role", "aws.s3.region" = "us-west-2" ); ``` ###### IAM user-based authentication[​](#iam-user-based-authentication-2 "Direct link to IAM user-based authentication") Run a command like below: ```sql CREATE EXTERNAL TABLE test_s3_db.file_table ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "s3://starrocks-test/", "format" = "ORC", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` ##### Ingestion[​](#ingestion "Direct link to Ingestion") You can use LOAD LABEL to load data from AWS S3. The following examples load the data from all Parquet data files stored in the `s3a://test-bucket/test_brokerload_ingestion` path into the `test_ingestion_2` table in an existing database named `test_s3_db`. For detailed syntax and parameters, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Instance profile-based authentication[​](#instance-profile-based-authentication-3 "Direct link to Instance profile-based authentication") Run a command like below: ```sql LOAD LABEL test_s3_db.test_credential_instanceprofile_7 ( DATA INFILE("s3a://test-bucket/test_brokerload_ingestion/*") INTO TABLE test_ingestion_2 FORMAT AS "parquet" ) WITH BROKER ( "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-1" ) PROPERTIES ( "timeout" = "1200" ); ``` ###### Assumed role-based authentication[​](#assumed-role-based-authentication-3 "Direct link to Assumed role-based authentication") Run a command like below: ```sql LOAD LABEL test_s3_db.test_credential_instanceprofile_7 ( DATA INFILE("s3a://test-bucket/test_brokerload_ingestion/*") INTO TABLE test_ingestion_2 FORMAT AS "parquet" ) WITH BROKER ( "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/s3_assumed_role", "aws.s3.region" = "us-west-1" ) PROPERTIES ( "timeout" = "1200" ); ``` ###### IAM user-based authentication[​](#iam-user-based-authentication-3 "Direct link to IAM user-based authentication") Run a command like below: ```sql LOAD LABEL test_s3_db.test_credential_instanceprofile_7 ( DATA INFILE("s3a://test-bucket/test_brokerload_ingestion/*") INTO TABLE test_ingestion_2 FORMAT AS "parquet" ) WITH BROKER ( "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-1" ) PROPERTIES ( "timeout" = "1200" ); ``` --- ### Authenticate to Microsoft Azure Storage From v3.0 onwards, StarRocks can integrate with Microsoft Azure Storage (Azure Blob Storage or Azure Data Lake Storage) in the following scenarios: * Query [Hive](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md), [Iceberg](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md), [Hudi](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md), and [Delta Lake](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md) tables in Azure Storage. * Batch load data from Azure Storage. * Back up data from and restore data to Azure Storage. * Query Parquet and ORC files in Azure Storage. The following table lists the authentication methods that StarRocks supports for each type of Azure Storage. | **Azure Storage type** | **Authentication method** | | ------------------------------------------------- | --------------------------------------------------------------------------------------------- | | Azure Blob Storage | - Shared Key (access key of storage account)
- SAS Token | | Azure Data Lake Storage Gen2 (ADLS Gen2) | - Managed Identity
- Shared Key (access key of storage account)
- Service Principal | | Azure Data Lake Storage Gen1 (ADLS Gen1), retired | - Managed Service Identity
- Service Principal | From v3.5.10 onwards, ADLS Gen2 also supports [Workload Identity](#workload-identity). The Managed Identity method uses the cluster managed identity. In this topic, the cluster managed identity is the user-assigned managed identity attached to the virtual machines (VMs) on which your StarRocks cluster is deployed. For an external catalog, use the credentials of the storage account that stores the data of the external data source. In this topic, Hive catalog, file external table, and Broker Load are used as examples. The Broker Load examples load the data file `file1.parquet` into the table `table1` in the database `mydatabase`. `file1.parquet` and `table1` have the same schema. For information about the parameters in the examples, see [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md), [File external table](https://docs.starrocks.io/docs/data_source/file_external_table.md), and [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). #### Blob Storage[​](#blob-storage "Direct link to Blob Storage") StarRocks supports using one of the following authentication methods to access Blob Storage: * Shared Key * SAS Token > **NOTE** > > When you load data or directly query files from Blob Storage, you must use the wasb or wasbs protocol to access your data: > > * If your storage account allows access over HTTP, use the wasb protocol and write the file path as `wasb://@.blob.core.windows.net//`. > * If your storage account allows access over HTTPS, use the wasbs protocol and write the file path as `wasbs://@.blob.core.windows.net//`. ##### Shared Key (access key of storage account)[​](#shared-key-access-key-of-storage-account "Direct link to Shared Key (access key of storage account)") Access keys authenticate the requests that your StarRocks cluster sends to your storage account. For more information about access keys, see [Manage account access keys - Azure Storage](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal). ###### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | --------------------------- | --------------------------------------------------------- | | azure.blob.storage\_account | The name of your Blob storage account. | | azure.blob.shared\_key | The shared key (access key) of your Blob storage account. | Follow these steps to obtain the shared key of your Blob storage account: 1. Sign in to the [Azure portal](https://portal.azure.com/#home) and go to the **[Storage accounts](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts)** page. 2. Find your storage account and click its name. 3. In the left-side navigation pane, choose **Access keys**. 4. Click **Show** next to the key you want to use, and then click the copy icon to copy the key. ###### Examples[​](#examples "Direct link to Examples") ###### External catalog[​](#external-catalog "Direct link to External catalog") Configure `azure.blob.storage_account` and `azure.blob.shared_key` as follows in the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement: ```sql CREATE EXTERNAL CATALOG hive_catalog_azure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` ###### File external table[​](#file-external-table "Direct link to File external table") Configure `azure.blob.storage_account`, `azure.blob.shared_key`, and the file path (`path`) as follows in the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement: ```sql CREATE EXTERNAL TABLE external_table_azure ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "wasb[s]://@.blob.core.windows.net//", "format" = "ORC", "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` ###### Broker Load[​](#broker-load "Direct link to Broker Load") Configure `azure.blob.storage_account`, `azure.blob.shared_key`, and the file path (`DATA INFILE`) as follows in the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement: ```sql LOAD LABEL mydatabase.label_blob_key ( DATA INFILE("wasb[s]://@.blob.core.windows.net//file1.parquet") INTO TABLE table1 FORMAT AS "parquet" ) WITH BROKER ( "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` ##### SAS Token[​](#sas-token "Direct link to SAS Token") A shared access signature (SAS) is a token that grants restricted access to your storage account. For more information about SAS tokens, see [Delegate access by using a shared access signature - Azure Storage](https://learn.microsoft.com/en-us/rest/api/storageservices/delegate-access-with-shared-access-signature). ###### Parameters[​](#parameters-1 "Direct link to Parameters") | **Parameter** | **Description** | | --------------------------- | ---------------------------------------------------------------------------------- | | azure.blob.storage\_account | The name of your Blob storage account. | | azure.blob.container | The name of the Blob container that stores your data in your Blob storage account. | | azure.blob.sas\_token | The SAS token that is used to access your Blob storage account. | Follow these steps to obtain a SAS token for your Blob storage account: 1. Sign in to the [Azure portal](https://portal.azure.com/#home) and go to the **[Storage accounts](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts)** page. 2. Find your storage account and click its name. 3. In the left-side navigation pane, choose **Shared access signature**. 4. Check the settings in the **Allowed services**, **Allowed resource types**, and **Allowed permissions** sections. Then, select a key from the **Signing key** drop-down list and click **Generate SAS and connection string**. 5. Click the copy icon to copy the generated SAS token. ###### Examples[​](#examples-1 "Direct link to Examples") ###### External catalog[​](#external-catalog-1 "Direct link to External catalog") Configure `azure.blob.storage_account`, `azure.blob.container`, and `azure.blob.sas_token` as follows in the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement: ```sql CREATE EXTERNAL CATALOG hive_catalog_azure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` ###### File external table[​](#file-external-table-1 "Direct link to File external table") Configure `azure.blob.storage_account`, `azure.blob.container`, `azure.blob.sas_token`, and the file path (`path`) as follows in the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement: ```sql CREATE EXTERNAL TABLE external_table_azure ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "wasb[s]://@.blob.core.windows.net//", "format" = "ORC", "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` ###### Broker Load[​](#broker-load-1 "Direct link to Broker Load") Configure `azure.blob.storage_account`, `azure.blob.container`, `azure.blob.sas_token`, and the file path (`DATA INFILE`) as follows in the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement: ```sql LOAD LABEL mydatabase.label_blob_sas ( DATA INFILE("wasb[s]://@.blob.core.windows.net//file1.parquet") INTO TABLE table1 FORMAT AS "parquet" ) WITH BROKER ( "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` #### Data Lake Storage Gen2[​](#data-lake-storage-gen2 "Direct link to Data Lake Storage Gen2") StarRocks supports using one of the following authentication methods to access Data Lake Storage Gen2: * Managed Identity * Shared Key * Service Principal * Workload Identity (from v3.5.10 onwards) > **NOTE** > > When you load data or query files from Data Lake Storage Gen2, you must use the abfs or abfss protocol to access your data: > > * If your storage account allows access over HTTP, use the abfs protocol and write the file path as `abfs://@.dfs.core.windows.net//`. > * If your storage account allows access over HTTPS, use the abfss protocol and write the file path as `abfss://@.dfs.core.windows.net//`. ##### Managed Identity[​](#managed-identity "Direct link to Managed Identity") A user-assigned managed identity lets Azure resources, such as VMs, authenticate to other Azure services without storing credentials. It is a standalone Azure resource. One VM can use more than one user-assigned managed identity, and one user-assigned managed identity can be shared by more than one VM. With this method, StarRocks uses the cluster managed identity to access your storage account. ###### Preparation[​](#preparation "Direct link to Preparation") Before you use this authentication method, make sure that the cluster managed identity exists and has the required permissions on your storage account. First, make sure that a user-assigned managed identity is attached to the VMs on which your StarRocks cluster is deployed, and note its name. If none is attached, create one and add it to these VMs. For more information, see [Configure managed identities on Azure virtual machines](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-to-configure-managed-identities). Then, check that the cluster managed identity has the required permissions on your storage account: 1. Sign in to the [Azure portal](https://portal.azure.com/#home) and go to the **[Storage accounts](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts)** page. 2. Find your storage account and click its name. 3. In the left-side navigation pane, choose **Access control (IAM)**. 4. On the **Role assignments** tab, check the existing role assignments. 5. If the cluster managed identity does not have the required permissions on the storage account, click **Add** and choose **Add role assignment**. Then, add a role assignment: a. On the **Job function roles** tab, select a role and click **Next**. For example, **Storage Blob Data Reader** grants read access, and **Storage Blob Data Contributor** grants read and write access. b. On the **Members** tab, select **Managed identity** for **Assign access to**. c. For **Members**, click **Select members**. In the pane that appears, select your subscription, select **User-assigned managed identity** from the **Managed identity** drop-down list, select the cluster managed identity, and then click **Select**. d. Click **Review + assign**. Review the information, and then click **Review + assign** again to add the role assignment. ###### Parameters[​](#parameters-2 "Direct link to Parameters") | **Parameter** | **Description** | | ------------------------------------------ | ------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_use\_managed\_identity | Specifies whether to enable the Managed Identity authentication method. Set the value to `true`. | | azure.adls2.oauth2\_tenant\_id | The ID of the tenant of your ADLS Gen2 storage account. | | azure.adls2.oauth2\_client\_id | The client ID of the cluster managed identity. | Follow these steps to obtain the tenant ID: 1. Sign in to the [Azure portal](https://portal.azure.com/#home) and go to the **[Tenant properties](https://portal.azure.com/#view/Microsoft_AAD_IAM/TenantProperties.ReactView)** page. 2. Find **Tenant ID** and click the copy icon to copy it. Follow these steps to obtain the client ID of the cluster managed identity: 1. Sign in to the [Azure portal](https://portal.azure.com/#home) and go to the **[Managed Identities](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.ManagedIdentity%2FuserAssignedIdentities)** page. 2. Find the cluster managed identity and click its name. 3. On the **Overview** page, find **Client ID** and click the copy icon to copy it. ###### Examples[​](#examples-2 "Direct link to Examples") ###### External catalog[​](#external-catalog-2 "Direct link to External catalog") Configure `azure.adls2.oauth2_use_managed_identity`, `azure.adls2.oauth2_tenant_id`, and `azure.adls2.oauth2_client_id` as follows in the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement: ```sql CREATE EXTERNAL CATALOG hive_catalog_azure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### File external table[​](#file-external-table-2 "Direct link to File external table") Configure `azure.adls2.oauth2_use_managed_identity`, `azure.adls2.oauth2_tenant_id`, `azure.adls2.oauth2_client_id`, and the file path (`path`) as follows in the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement: ```sql CREATE EXTERNAL TABLE external_table_azure ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "abfs[s]://@.dfs.core.windows.net//", "format" = "ORC", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### Broker Load[​](#broker-load-2 "Direct link to Broker Load") Configure `azure.adls2.oauth2_use_managed_identity`, `azure.adls2.oauth2_tenant_id`, `azure.adls2.oauth2_client_id`, and the file path (`DATA INFILE`) as follows in the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement: ```sql LOAD LABEL mydatabase.label_gen2_identity ( DATA INFILE("abfs[s]://@.dfs.core.windows.net//file1.parquet") INTO TABLE table1 FORMAT AS "parquet" ) WITH BROKER ( "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ##### Shared Key (access key of storage account)[​](#shared-key-access-key-of-storage-account-1 "Direct link to Shared Key (access key of storage account)") Using Shared Key to access ADLS Gen2 works in the same way as using Shared Key to access Blob Storage. For information about how to obtain the shared key, see [Blob Storage > Shared Key](#shared-key-access-key-of-storage-account). ###### Parameters[​](#parameters-3 "Direct link to Parameters") | **Parameter** | **Description** | | ---------------------------- | -------------------------------------------------------------- | | azure.adls2.storage\_account | The name of your ADLS Gen2 storage account. | | azure.adls2.shared\_key | The shared key (access key) of your ADLS Gen2 storage account. | ###### Examples[​](#examples-3 "Direct link to Examples") ###### External catalog[​](#external-catalog-3 "Direct link to External catalog") Configure `azure.adls2.storage_account` and `azure.adls2.shared_key` as follows in the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement: ```sql CREATE EXTERNAL CATALOG hive_catalog_azure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` ###### File external table[​](#file-external-table-3 "Direct link to File external table") Configure `azure.adls2.storage_account`, `azure.adls2.shared_key`, and the file path (`path`) as follows in the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement: ```sql CREATE EXTERNAL TABLE external_table_azure ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "abfs[s]://@.dfs.core.windows.net//", "format" = "ORC", "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` ###### Broker Load[​](#broker-load-3 "Direct link to Broker Load") Configure `azure.adls2.storage_account`, `azure.adls2.shared_key`, and the file path (`DATA INFILE`) as follows in the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement: ```sql LOAD LABEL mydatabase.label_gen2_key ( DATA INFILE("abfs[s]://@.dfs.core.windows.net//file1.parquet") INTO TABLE table1 FORMAT AS "parquet" ) WITH BROKER ( "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` ##### Service Principal[​](#service-principal "Direct link to Service Principal") ###### Preparation[​](#preparation-1 "Direct link to Preparation") Before you use this authentication method, make sure that you have a service principal, and that it has the required permissions on your storage account. For information about how to create a service principal and a client secret, see [Register a Microsoft Entra app and create a service principal](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal). Follow these steps to check the permissions of the service principal: 1. Sign in to the [Azure portal](https://portal.azure.com/#home) and go to the **[Storage accounts](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts)** page. 2. Find your ADLS Gen2 storage account and click its name. 3. In the left-side navigation pane, choose **Access control (IAM)**. 4. On the **Role assignments** tab, check the existing role assignments. 5. If the service principal does not have the required permissions on the storage account, click **Add** and choose **Add role assignment**. Then, add a role assignment: a. On the **Job function roles** tab, select a role and click **Next**. For example, **Storage Blob Data Reader** grants read access, and **Storage Blob Data Contributor** grants read and write access. b. On the **Members** tab, select **User, group, or service principal** for **Assign access to**. c. For **Members**, click **Select members**. In the pane that appears, search for and select the application that you registered, and then click **Select**. d. Click **Review + assign**. Review the information, and then click **Review + assign** again to add the role assignment. ###### Parameters[​](#parameters-4 "Direct link to Parameters") | **Parameter** | **Description** | | ------------------------------------ | ----------------------------------------------------------- | | azure.adls2.oauth2\_client\_id | The application (client) ID of the service principal. | | azure.adls2.oauth2\_client\_secret | The value of the client secret of the service principal. | | azure.adls2.oauth2\_client\_endpoint | The OAuth 2.0 token endpoint (v1) of the service principal. | > **NOTE** > > Azure shows the value of a client secret only once, right after you create it. Save the value before you leave the page. Follow these steps to obtain the application (client) ID and the OAuth 2.0 token endpoint (v1) of the service principal: 1. Sign in to the [Azure portal](https://portal.azure.com/#home) and go to the **[App registrations](https://portal.azure.com/#view/Microsoft_AAD_RegisteredApps/ApplicationsListBlade)** page. 2. Find your registered application and click its name. 3. On the **Overview** page, find **Application (client) ID** and click the copy icon to copy it. 4. Click **Endpoints** at the top of the **Overview** page. 5. In the **Endpoints** pane, find **OAuth 2.0 token endpoint (v1)** and click the copy icon to copy it. ###### Examples[​](#examples-4 "Direct link to Examples") ###### External catalog[​](#external-catalog-4 "Direct link to External catalog") Configure `azure.adls2.oauth2_client_id`, `azure.adls2.oauth2_client_secret`, and `azure.adls2.oauth2_client_endpoint` as follows in the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement: ```sql CREATE EXTERNAL CATALOG hive_catalog_azure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` ###### File external table[​](#file-external-table-4 "Direct link to File external table") Configure `azure.adls2.oauth2_client_id`, `azure.adls2.oauth2_client_secret`, `azure.adls2.oauth2_client_endpoint`, and the file path (`path`) as follows in the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement: ```sql CREATE EXTERNAL TABLE external_table_azure ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "abfs[s]://@.dfs.core.windows.net//", "format" = "ORC", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` ###### Broker Load[​](#broker-load-4 "Direct link to Broker Load") Configure `azure.adls2.oauth2_client_id`, `azure.adls2.oauth2_client_secret`, `azure.adls2.oauth2_client_endpoint`, and the file path (`DATA INFILE`) as follows in the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement: ```sql LOAD LABEL mydatabase.label_gen2_principal ( DATA INFILE("abfs[s]://@.dfs.core.windows.net//file1.parquet") INTO TABLE table1 FORMAT AS "parquet" ) WITH BROKER ( "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` ##### Workload Identity[​](#workload-identity "Direct link to Workload Identity") From v3.5.10 onwards, StarRocks supports Azure Workload Identity as an authentication method when accessing Azure Data Lake Storage Gen2. This authentication method is designed for workloads running inside Azure (for example, AKS pods) where the compute identity is federated with an Azure AD application via a projected token file, avoiding the need to store long-lived credentials. To use Workload Identity authentication, the following Azure-side setup is required before proceeding: * Enable the Azure Workload Identity webhook on your AKS cluster (or equivalent). * Create a federated identity credential linking your Kubernetes service account to an Azure AD application or user-assigned managed identity. * Annotate the Kubernetes pod/service account so that the webhook injects the token projection. The token file will appear at the path configured in `azure.adls2.oauth2_token_file` (commonly `/var/run/secrets/azure/tokens/azure-identity-token`). * Grant the Azure AD identity the necessary RBAC role on the storage account (for example, Storage Blob Data Reader or Storage Blob Data Contributor). ###### External catalog[​](#external-catalog-5 "Direct link to External catalog") Configure `azure.adls2.oauth2_token_file`, `azure.adls2.oauth2_tenant_id`, and `azure.adls2.oauth2_client_id` as follows in the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement: ```sql CREATE EXTERNAL CATALOG hive_catalog_azure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### File external table[​](#file-external-table-5 "Direct link to File external table") Configure `azure.adls2.oauth2_token_file`, `azure.adls2.oauth2_tenant_id`, `azure.adls2.oauth2_client_id`, and the file path (`path`) as follows in the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement: ```sql CREATE EXTERNAL TABLE external_table_azure ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "abfs[s]://@.dfs.core.windows.net//", "format" = "ORC", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### Broker Load[​](#broker-load-5 "Direct link to Broker Load") Configure `azure.adls2.oauth2_token_file`, `azure.adls2.oauth2_tenant_id`, `azure.adls2.oauth2_client_id`, and the file path (`DATA INFILE`) as follows in the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement: ```sql LOAD LABEL mydatabase.label_gen2_workload ( DATA INFILE("abfs[s]://@.dfs.core.windows.net//file1.parquet") INTO TABLE table1 FORMAT AS "parquet" ) WITH BROKER ( "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` #### Data Lake Storage Gen1[​](#data-lake-storage-gen1 "Direct link to Data Lake Storage Gen1") > **NOTE** > > Microsoft retired Azure Data Lake Storage Gen1 on February 29, 2024. The Gen1 sections of this topic are kept for reference. StarRocks supports using one of the following authentication methods to access Data Lake Storage Gen1: * Managed Service Identity * Service Principal > **NOTE** > > When you load data or query files from Data Lake Storage Gen1, you must use the adl protocol to access your data and write the file path as `adl://.azuredatalakestore.net//`. ##### Managed Service Identity[​](#managed-service-identity "Direct link to Managed Service Identity") ###### Parameters[​](#parameters-5 "Direct link to Parameters") | **Parameter** | **Description** | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | azure.adls1.use\_managed\_service\_identity | Specifies whether to enable the Managed Service Identity authentication method. Set the value to `true`. | ###### Examples[​](#examples-5 "Direct link to Examples") ###### External catalog[​](#external-catalog-6 "Direct link to External catalog") Configure `azure.adls1.use_managed_service_identity` as follows in the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement: ```sql CREATE EXTERNAL CATALOG hive_catalog_azure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.use_managed_service_identity" = "true" ); ``` ###### File external table[​](#file-external-table-6 "Direct link to File external table") Configure `azure.adls1.use_managed_service_identity` and the file path (`path`) as follows in the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement: ```sql CREATE EXTERNAL TABLE external_table_azure ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "adl://.azuredatalakestore.net//", "format" = "ORC", "azure.adls1.use_managed_service_identity" = "true" ); ``` ###### Broker Load[​](#broker-load-6 "Direct link to Broker Load") Configure `azure.adls1.use_managed_service_identity` and the file path (`DATA INFILE`) as follows in the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement: ```sql LOAD LABEL mydatabase.label_gen1_identity ( DATA INFILE("adl://.azuredatalakestore.net//file1.parquet") INTO TABLE table1 FORMAT AS "parquet" ) WITH BROKER ( "azure.adls1.use_managed_service_identity" = "true" ); ``` ##### Service Principal[​](#service-principal-1 "Direct link to Service Principal") For information about how to create a service principal and obtain its application (client) ID, client secret, and OAuth 2.0 token endpoint (v1), see [Data Lake Storage Gen2 > Service Principal](#service-principal). ###### Parameters[​](#parameters-6 "Direct link to Parameters") | **Parameter** | **Description** | | ------------------------------ | ----------------------------------------------------------- | | azure.adls1.oauth2\_client\_id | The application (client) ID of the service principal. | | azure.adls1.oauth2\_credential | The value of the client secret of the service principal. | | azure.adls1.oauth2\_endpoint | The OAuth 2.0 token endpoint (v1) of the service principal. | ###### Examples[​](#examples-6 "Direct link to Examples") ###### External catalog[​](#external-catalog-7 "Direct link to External catalog") Configure `azure.adls1.oauth2_client_id`, `azure.adls1.oauth2_credential`, and `azure.adls1.oauth2_endpoint` as follows in the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement: ```sql CREATE EXTERNAL CATALOG hive_catalog_azure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` ###### File external table[​](#file-external-table-7 "Direct link to File external table") Configure `azure.adls1.oauth2_client_id`, `azure.adls1.oauth2_credential`, `azure.adls1.oauth2_endpoint`, and the file path (`path`) as follows in the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement: ```sql CREATE EXTERNAL TABLE external_table_azure ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "adl://.azuredatalakestore.net//", "format" = "ORC", "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` ###### Broker Load[​](#broker-load-7 "Direct link to Broker Load") Configure `azure.adls1.oauth2_client_id`, `azure.adls1.oauth2_credential`, `azure.adls1.oauth2_endpoint`, and the file path (`DATA INFILE`) as follows in the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement: ```sql LOAD LABEL mydatabase.label_gen1_principal ( DATA INFILE("adl://.azuredatalakestore.net//file1.parquet") INTO TABLE table1 FORMAT AS "parquet" ) WITH BROKER ( "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` --- ### Authenticate to Google Cloud Storage #### Authentication methods[​](#authentication-methods "Direct link to Authentication methods") From v3.0 onwards, StarRocks supports using one of the following authentication methods to access Google Cloud Storage (GCS): * VM-based authentication Use the credential attached to Google Cloud Compute Engine to authenticate GCS. * Service account-based authentication Use a service account to authenticate GCS. * Impersonation-based authentication Make a service account or virtual machine (VM) instance impersonate another service account. #### Scenarios[​](#scenarios "Direct link to Scenarios") StarRocks can authenticate to GCS in the following scenarios: * Batch load data from GCS. * Back up data from and restore data to GCS. * Query Parquet and ORC files in GCS. * Query [Hive](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md), [Iceberg](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md), [Hudi](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md), and [Delta Lake](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md) tables in GCS. In this topic, [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md), [file external table](https://docs.starrocks.io/docs/data_source/file_external_table.md), and [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) are used as examples to show how StarRocks integrates with GCS in different scenarios. For information about `StorageCredentialParams` in the examples, see the "[Parameters](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_gcs.md#parameters)" section of this topic. > **NOTE** > > StarRocks supports loading data or directly querying files from GCS only according to the gs protocol. Therefore, when you load data or query files from GCS, you must include `gs` as a prefix in the file path. ##### External catalog[​](#external-catalog "Direct link to External catalog") Use the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement to create a Hive catalog named `hive_catalog_gcs` as follows, in order to query files from GCS: ```sql CREATE EXTERNAL CATALOG hive_catalog_gcs PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", StorageCredentialParams ); ``` ##### File external table[​](#file-external-table "Direct link to File external table") Use the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement to create a file external table named `external_table_gcs` as follows, in order to query a data file named `test_file_external_tbl` from GCS without any metastore: ```sql CREATE EXTERNAL TABLE external_table_gcs ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "gs:////test-gcs/test_file_external_tbl", "format" = "ORC", StorageCredentialParams ); ``` ##### Broker load[​](#broker-load "Direct link to Broker load") Use the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement to create a Broker Load job whose label is `test_db.label000`, in order to batch load data from GCS into the StarRocks table `target_table`: ```sql LOAD LABEL test_db.label000 ( DATA INFILE("gs://bucket_gcs/test_brokerload_ingestion/*") INTO TABLE target_table FORMAT AS "parquet" ) WITH BROKER ( StorageCredentialParams ); ``` #### Parameters[​](#parameters "Direct link to Parameters") `StorageCredentialParams` represents a parameter set that describes how to authenticate to GCS with different authentication methods. ##### VM-based authentication[​](#vm-based-authentication "Direct link to VM-based authentication") If your StarRocks cluster is deployed on a VM instance hosted on Google Cloud Platform (GCP) and you want to use that VM instance to authenticate GCS, configure `StorageCredentialParams` as follows: ```plain "gcp.gcs.use_compute_engine_service_account" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | ##### Service account-based authentication[​](#service-account-based-authentication "Direct link to Service account-based authentication") If you directly use a service account to authenticate GCS, configure `StorageCredentialParams` as follows: ```plain "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------- | | gcp.gcs.service\_account\_email | "" | "`user@hello.iam.gserviceaccount.com`" | The email address in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the service account. | ##### Impersonation-based authentication[​](#impersonation-based-authentication "Direct link to Impersonation-based authentication") ###### Make a VM instance impersonate a service account[​](#make-a-vm-instance-impersonate-a-service-account "Direct link to Make a VM instance impersonate a service account") If your StarRocks cluster is deployed on a VM instance hosted on GCP and you want to make that VM instance impersonate a service account, so as to make StarRocks inherit the privileges from the service account to access GCS, configure `StorageCredentialParams` as follows: ```plain "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The service account that you want to impersonate. | ###### Make a service account impersonate another service account[​](#make-a-service-account-impersonate-another-service-account "Direct link to Make a service account impersonate another service account") If you want to make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account) and make StarRocks inherit the privileges from the data service account to access GCS, configure `StorageCredentialParams` as follows: ```plain "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | gcp.gcs.service\_account\_email | "" | "`user@hello.iam.gserviceaccount.com`" | The email address in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the meta service account. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The data service account that you want to impersonate. | --- ### Data Lakehouse ![DLA](/assets/images/1.1-8-dla-c67d601d709b092317fa11eb64ac1783.png) In addition to efficient analytics of local data, StarRocks can work as the compute engine to analyze data stored in data lakes such as Apache Hudi, Apache Iceberg, and Delta Lake. One of the key features of StarRocks is its external catalog, which acts as the linkage to an externally maintained metastore. This functionality provides users with the capability to query external data sources seamlessly, eliminating the need for data migration. As such, users can analyze data from different systems such as HDFS and Amazon S3, in various file formats such as Parquet, ORC, and CSV, etc. The preceding figure shows a data lake analytics scenario where StarRocks is responsible for data computing and analysis, and the data lake is responsible for data storage, organization, and maintenance. Data lakes allow users to store data in open storage formats and use flexible schemas to produce reports on "single source of truth" for various BI, AI, ad-hoc, and reporting use cases. StarRocks fully leverages the advantages of its vectorization engine and CBO, significantly improving the performance of data lake analytics. #### Key ideas[​](#key-ideas "Direct link to Key ideas") * Open Data Formats: Supports a variety of data types, including JSON, Parquet, and Avro, facilitating the storage and processing of both structured and unstructured data. * Metadata Management: Implements a shared metadata layer, often utilizing formats like the Iceberg table format, to organize and govern data efficiently. * Governance and Security: Features robust built-in mechanisms for data security, privacy, and compliance, ensuring data integrity and trustworthiness. #### Advantages of Data Lakehouse architecture[​](#advantages-of-data-lakehouse-architecture "Direct link to Advantages of Data Lakehouse architecture") * Flexibility and Scalability: Seamlessly manages diverse data types and scales with the organization’s needs. * Cost-Effectiveness: Offers an economical alternative for data storage and processing, compared to traditional methods. * Enhanced Data Governance: Improves data control, management, and integrity, ensuring reliable and secure data handling. * AI and Analytics Readiness: Perfectly suited for complex analytical tasks, including machine learning and AI-driven data processing. #### StarRocks approach[​](#starrocks-approach "Direct link to StarRocks approach") The key things to consider are: * Standardizing the integration with catalog, or metadata services * Elastic scalability of compute nodes * Flexible caching mechanisms *** #### Catalogs[​](#catalogs "Direct link to Catalogs") StarRocks has two types of catalogs, internal and external. The internal catalog contains metadata for data stored within StarRocks databases. External catalogs are used to work with data stored externally, including the data managed by Hive, Iceberg, Delta Lake, and Hudi. There are many other external systems, links are in the More Information section at the bottom of the page. #### Compute node (CN) scaling[​](#compute-node-cn-scaling "Direct link to Compute node (CN) scaling") Separation of storage and compute reduces the complexity of scaling. Since the StarRocks compute nodes are only storing local cache, nodes can be added or removed based on load. #### Data cache[​](#data-cache "Direct link to Data cache") Cache on the compute nodes is optional. If your compute nodes are spinning up and down quickly based on quickly changing load patterns or your queries are often only on the most recent data it might not make sense to cache data. More information is in the [Catalog docs](https://docs.starrocks.io/docs/data_source/catalog.md). --- ### Monitor with Datadog This topic describes how to integrate your StarRocks cluster with [Datadog](https://www.datadoghq.com/), a monitoring and security platform. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before getting started, you must have the following software installed on your instances: * [Datadog Agent](https://docs.datadoghq.com/getting_started/agent/) * Python > **NOTE** > > When you install Datadog Agent for the first time, Python is also installed as a dependency. We recommend you use this Python in the following steps. #### Prepare StarRocks source code[​](#prepare-starrocks-source-code "Direct link to Prepare StarRocks source code") Since Datadog does not provide the integration kit for StarRocks yet, you need to integrate them using the source code. 1. Launch a terminal, navigate to a local directory to which you have both read and write access, and run the following command to create a dedicated directory for StarRocks source code. ```shell mkdir -p starrocks ``` 2. Download the StarRocks source code package using the following command or on [GitHub](https://github.com/StarRocks/starrocks/tags) to the directory you created. ```shell cd starrocks # Replace with the actual version of StarRocks, for example, "2.5.2". wget https://github.com/StarRocks/starrocks/archive/refs/tags/.tar.gz ``` 3. Extract the files in the package. ```shell # Replace with the actual version of StarRocks, for example, "2.5.2". tar -xzvf .tar.gz --strip-components 1 ``` #### Install and configure FE integration kit[​](#install-and-configure-fe-integration-kit "Direct link to Install and configure FE integration kit") 1. Install Datadog integration kit for FE using source code. ```shell /opt/datadog-agent/embedded/bin/pip install contrib/datadog-connector/starrocks_fe ``` 2. Create the FE integration configuration file **/etc/datadog-agent/conf.d/starrocks\_fe.d/conf.yaml**. ```shell sudo mkdir -p /etc/datadog-agent/conf.d/starrocks_fe.d sudo cp contrib/datadog-connector/starrocks_fe/datadog_checks/starrocks_fe/data/conf.yaml.example /etc/datadog-agent/conf.d/starrocks_fe.d/conf.yaml ``` 3. Modify the FE integration configuration file **/etc/datadog-agent/conf.d/starrocks\_fe.d/conf.yaml**. Examples of some important configuration items: | **Config** | **Example** | **Description** | | --------------- | ------------------------------- | ------------------------------------------------------------------------------------------ | | fe\_metric\_url | `http://localhost:8030/metrics` | The URL used to access the StarRocks FE metrics. | | metrics | `- starrocks_fe_*` | Metrics to be monitored on FE. You can use wildcards `*` to match the configuration items. | #### Install and configure BE integration kit[​](#install-and-configure-be-integration-kit "Direct link to Install and configure BE integration kit") 1. Install Datadog integration kit for BE using source code. ```shell /opt/datadog-agent/embedded/bin/pip install contrib/datadog-connector/starrocks_be ``` 2. Create the BE integration configuration file **/etc/datadog-agent/conf.d/starrocks\_be.d/conf.yaml**. ```shell sudo mkdir -p /etc/datadog-agent/conf.d/starrocks_be.d sudo cp contrib/datadog-connector/starrocks_be/datadog_checks/starrocks_be/data/conf.yaml.example /etc/datadog-agent/conf.d/starrocks_be.d/conf.yaml ``` 3. Modify the BE integration configuration file **/etc/datadog-agent/conf.d/starrocks\_be.d/conf.yaml**. Examples of some important configuration items: | **Config** | **Example** | **Description** | | --------------- | ------------------------------- | ------------------------------------------------------------------------------------------ | | be\_metric\_url | `http://localhost:8040/metrics` | The URL used to access the StarRocks BE metrics. | | metrics | `- starrocks_be_*` | Metrics to be monitored on BE. You can use wildcards `*` to match the configuration items. | #### Restart Datadog Agent[​](#restart-datadog-agent "Direct link to Restart Datadog Agent") Restart Datadog Agent to allow the configuration to take effect. ```shell sudo systemctl stop datadog-agent sudo systemctl start datadog-agent ``` #### Verify integration[​](#verify-integration "Direct link to Verify integration") For instructions to verify the integration, see [Datadog Application](https://docs.datadoghq.com/getting_started/application/). #### Uninstall integration kits[​](#uninstall-integration-kits "Direct link to Uninstall integration kits") You can uninstall the integration kits when you no longer need them. * To uninstall FE integration kit, run the following command: ```shell /opt/datadog-agent/embedded/bin/pip uninstall datadog-starrocks-fe ``` * To uninstall BE integration kit, run the following command: ```shell /opt/datadog-agent/embedded/bin/pip uninstall datadog-starrocks-be ``` --- ### dbt `dbt-starrocks` enables the use of `dbt` to transform data in StarRocks using dbt's modeling patterns and best practices. `dbt-starrocks` [GitHub repo](https://github.com/StarRocks/dbt-starrocks). Experimental feature [Advice on use of experimental features](https://docs.starrocks.io/docs/maturity/features.md) #### Supported features[​](#supported-features "Direct link to Supported features") | StarRocks >= 3.1 | StarRocks >= 3.4 | Feature | | ---------------- | ---------------- | ------------------------------------ | | ✅ | ✅ | Table materialization | | ✅ | ✅ | View materialization | | ✅ | ✅ | Materialized View materialization | | ✅ | ✅ | Incremental materialization | | ✅ | ✅ | Primary Key Model | | ✅ | ✅ | Sources | | ✅ | ✅ | Data tests (generic and singular) | | ✅ | ✅ | Unit tests | | ✅ | ✅ | Storing test failures | | ✅ | ✅ | Source freshness (`loaded_at_field`) | | ❌ | ❌ | Metadata-based source freshness | | ✅ | ✅ | Docs generate | | ❌ | ❌ | `persist_docs` | | ✅ | ✅ | Expression Partition | | ❌ | ❌ | Kafka | | ❌ | ✅ | Dynamic Overwrite | | `*` | ✅ | Submit task | | ✅ | ✅ | Microbatch (Insert Overwrite) | | ❌ | ✅ | Microbatch (Dynamic Overwrite) | `*` Verify the specific `submit task` support for your version, see [SUBMIT TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md) #### Installation[​](#installation "Direct link to Installation") Install the StarRocks DBT adapter using pip: ```sh pip install dbt-starrocks ``` #### Verify Installation[​](#verify-installation "Direct link to Verify Installation") Verify the installation by checking the version: ```sh dbt --version ``` This should list `starrocks` under plugins. #### Configuration[​](#configuration "Direct link to Configuration") ##### Profiles[​](#profiles "Direct link to Profiles") Create or update `profiles.yml` with StarRocks-specific settings. ```yaml starrocks_project: target: dev outputs: dev: type: starrocks host: your-starrocks-host.com port: 9030 schema: your_database username: your_username password: your_password catalog: test_catalog ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### `type`[​](#type "Direct link to type") **Description**: The specific adapter to use, this must be set to `starrocks`
**Required?**: Required
**Example**: `starrocks` ###### `host`[​](#host "Direct link to host") **Description**: The hostname to connect to
**Required?**: Required
**Example**: `192.168.100.28` ###### `port`[​](#port "Direct link to port") **Description**: The port to use
**Required?**: Required
**Example**: `9030` ###### `catalog`[​](#catalog "Direct link to catalog") **Description**: Specify the catalog to build models into
**Required?**: Optional
**Example**: `default_catalog` ###### `schema`[​](#schema "Direct link to schema") **Description**: Specify the schema (database in StarRocks) to build models into
**Required?**: Required
**Example**: `analytics` ###### `username`[​](#username "Direct link to username") **Description**: The username to use to connect to the server
**Required?**: Required
**Example**: `dbt_admin` ###### `password`[​](#password "Direct link to password") **Description**: The password to use for authenticating to the server
**Required?**: Required
**Example**: `correct-horse-battery-staple` ###### `version`[​](#version "Direct link to version") **Description**: Let Plugin try to go to a compatible starrocks version
**Required?**: Optional
**Example**: `3.1.0` ###### `use_pure`[​](#use_pure "Direct link to use_pure") **Description**: set to "true" to use C extensions
**Required?**: Optional
**Example**: `true` ###### `is_async`[​](#is_async "Direct link to is_async") **Description**: "true" to submit suitable tasks as etl tasks.
**Required?**: Optional
**Example**: `true` ###### `async_query_timeout`[​](#async_query_timeout "Direct link to async_query_timeout") **Description**: Sets the `query_timeout` value when submitting a task to StarRocks
**Required?**: Optional
**Example**: `300` ##### Sources[​](#sources "Direct link to Sources") Create or update `sources.yml` ```yml sources: - name: your_source database: your_sr_catalog schema: your_sr_database tables: - name: your_table ``` If the catalog is not specified in the schema, it will default to the catalog defined in the profile. Using the profile from earlier, if catalog is not defined, the model will assume the source is located at `test_catalog.your_sr_database`. #### Materializations[​](#materializations "Direct link to Materializations") ##### Table[​](#table "Direct link to Table") Basic Table Configuration ```sql {{ config( materialized='table', engine='OLAP', keys=['id', 'name', 'created_date'], table_type='PRIMARY', distributed_by=['id'], buckets=3, partition_by=['created_date'], properties=[ {"replication_num": "1"} ] ) }} SELECT id, name, email, created_date, last_modified_date FROM {{ source('your_source', 'users') }} ``` #### Configuration Options[​](#configuration-options "Direct link to Configuration Options") * engine: Storage engine (default: `OLAP`) * keys: Columns that define the sort key * table\_type: Table model type * `PRIMARY`: Primary key model (supports upserts and deletes) * `DUPLICATE`: Duplicate key model (allows duplicate rows) * `UNIQUE`: Unique key model (enforces uniqueness) * `distributed_by`: Columns for hash distribution * `buckets`: Number of buckets for data distribution (leave empty for auto bucketing) * `partition_by`: Columns for table partitioning * `partition_by_init`: Initial partition definitions * `properties`: Additional StarRocks table properties #### Tables in External Catalogs[​](#tables-in-external-catalogs "Direct link to Tables in External Catalogs") ##### Read from External into StarRocks[​](#read-from-external-into-starrocks "Direct link to Read from External into StarRocks") This example creates a materialized table in StarRocks containing aggregated data from an external Hive catalog. tip Configure the external catalog if it does not already exist: ```sql CREATE EXTERNAL CATALOG `hive_external` PROPERTIES ( "hive.metastore.uris" = "thrift://127.0.0.1:8087", "type"="hive" ); ``` ```sql {{ config( materialized='table', keys=['product_id', 'order_date'], distributed_by=['product_id'], partition_by=['order_date'] ) }} -- Aggregate data from Hive external catalog into StarRocks table SELECT h.product_id, h.order_date, COUNT(*) as order_count, SUM(h.amount) as total_amount, MAX(h.last_updated) as last_updated FROM {{ source('hive_external', 'orders') }} h GROUP BY h.product_id, h.order_date ``` ##### Write to External[​](#write-to-external "Direct link to Write to External") ```sql {{ config( materialized='table', on_table_exists = 'replace', partition_by=['order_date'], properties={}, catalog='external_catalog', database='test_db' ) }} SELECT * FROM {{ source('iceberg_external', 'orders') }} ``` The configuration for materialization to external catalogs supports fewer options. `on_table_exist`s, `partition_by`, and `properties` are supported. If `catalog` and `database` are not set, the defaults from the profile will be used. ##### Incremental[​](#incremental "Direct link to Incremental") Incremental materializations are supported in StarRocks as well: ```sql {{ config( materialized='incremental', unique_key='id', table_type='PRIMARY', keys=['id'], distributed_by=['id'], incremental_strategy='default' ) }} SELECT id, user_id, event_name, event_timestamp, properties FROM {{ source('raw', 'events') }} {% if is_incremental() %} WHERE event_timestamp > (SELECT MAX(event_timestamp) FROM {{ this }}) {% endif %} ``` ###### Incremental Strategies[​](#incremental-strategies "Direct link to Incremental Strategies") `dbt-starrocks` supports multiple incremental strategies: 1. `append` (default): Simply appends new records without deduplication 2. `insert_overwrite`: Overwrites table partitions with insertion 3. `dynamic_overwrite`: Overwrites, creates, and writes table partitions For more information about which overwrite strategy to use, see the [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) documentation. note Currently, incremental merge is not supported. #### Testing[​](#testing "Direct link to Testing") `dbt-starrocks` supports every dbt test type. No adapter-specific configuration is required — the tests compile to standard SQL that StarRocks executes directly. ##### Data tests[​](#data-tests "Direct link to Data tests") The four built-in generic tests (`not_null`, `unique`, `accepted_values`, and `relationships`) are supported, as are custom generic tests defined in `macros/` and singular tests defined as `.sql` files under `test-paths`. ```yml models: - name: stg_customers columns: - name: id data_tests: [not_null, unique] - name: region data_tests: - accepted_values: values: ['us', 'eu'] - relationships: to: ref('dim_customers') field: region ``` Test severity settings (`severity`, `error_if`, `warn_if`, `fail_calc`, and `limit`) are handled by dbt before any SQL reaches StarRocks and behave as they do on other adapters. ##### Unit tests[​](#unit-tests "Direct link to Unit tests") Unit tests are supported. dbt builds the fixture rows into the compiled query, so no data is written to StarRocks: ```yml unit_tests: - name: test_dim_customers_counts model: dim_customers given: - input: ref('stg_customers') rows: - {id: 1, name: 'a', region: 'us'} - {id: 2, name: 'b', region: 'us'} expect: rows: - {region: 'us', n: 2} ``` Run them with: ```sh dbt test --select test_type:unit ``` ##### Storing test failures[​](#storing-test-failures "Direct link to Storing test failures") `dbt test --store-failures`, the `store_failures` config, and `store_failures_as` (`table` or `view`) are all supported. Failing rows are written to a separate schema named `_dbt_test__audit`, which the adapter creates if it does not exist: ```sql SELECT * FROM `analytics_dbt_test__audit`.`not_null_stg_customers_name`; ``` Failure tables are created with `CREATE TABLE AS`, so they accept the same configuration options as table models. Use this to control how the failure table is stored: ```yml - name: name data_tests: - not_null: config: store_failures: true alias: nn_name_failures distributed_by: ['id'] buckets: 3 ``` Without a `distributed_by` value, the failure table is created with random distribution. tip `store_failures_as: view` stores the failing rows as a view instead of a table, which avoids creating and reloading a table on every test run. ##### Source freshness[​](#source-freshness "Direct link to Source freshness") `dbt source freshness` requires each source table to declare a `loaded_at_field`: ```yml sources: - name: raw tables: - name: events loaded_at_field: updated_at freshness: warn_after: {count: 1, period: hour} error_after: {count: 24, period: hour} ``` warning Metadata-based freshness is not supported. Every source that needs a freshness check must expose a timestamp column. #### Generating documentation[​](#generating-documentation "Direct link to Generating documentation") `dbt docs generate` builds the project catalog by querying `information_schema.tables` and `information_schema.columns`. Models, seeds, views, materialized views, and sources are all included, each with its columns, ordinal positions, and data types. `dbt docs generate --static` and `dbt docs serve` work as well. The descriptions you write in your `.yml` files are read from the dbt manifest, so they appear in the documentation site as usual. However, the following metadata is not collected from StarRocks: * Table and column comments stored in the database. * Table owner. * Table statistics, such as row counts and sizes. Two further details affect how relations are described: * Column types are reported without precision. A `VARCHAR(64)` column appears as `varchar`, and a `DECIMAL(18,4)` column appears as `decimal`. Use [SHOW CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_CREATE_TABLE.md) when the exact type matters. * Materialized views are documented as views. This affects the documentation site only, not `dbt run`. warning `persist_docs` is not supported. Do not set `persist_docs` in `dbt_project.yml`, where it would apply to every model in the project. #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") * Before using external catalogs in dbt, you must create them in StarRocks.[Catalog overview](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md). * External sources should be accessed using the `{{ source('external_source_name', 'table_name' }}` macro. * `dbt seed` was not tested for external catalogs and is not currently supported. * In order for `dbt` to create models in external databases that do not currently exist, the location of the models must be set through properties. * External models need to define the location they are stored at. This location will be defined if the destination database exists and sets the location property. Otherwise, the location needs to be set. * We will currently only support creating external models in databases that already exist. --- ### SQL IDE tools #### [📄️ DataGrip](https://docs.starrocks.io/docs/integrations/IDE_integrations/DataGrip.md) [DataGrip supports querying both internal data and external data in StarRocks.](https://docs.starrocks.io/docs/integrations/IDE_integrations/DataGrip.md) #### [📄️ Dataphin](https://docs.starrocks.io/docs/integrations/IDE_integrations/Dataphin.md) [Dataphin is a cloud-based output of the internal practices of Alibaba Group's OneData data governance methodology.](https://docs.starrocks.io/docs/integrations/IDE_integrations/Dataphin.md) #### [📄️ DBeaver](https://docs.starrocks.io/docs/integrations/IDE_integrations/DBeaver.md) [DBeaver is a SQL client software application and a database administration tool, which offers a helpful assistant that walks you through the process of...](https://docs.starrocks.io/docs/integrations/IDE_integrations/DBeaver.md) #### [📄️ LibreDB Studio](https://docs.starrocks.io/docs/integrations/IDE_integrations/LibreDB_Studio.md) [LibreDB Studio is an open source, browser-based SQL IDE. Connect to StarRocks over its MySQL wire protocol to browse schemas and run queries.](https://docs.starrocks.io/docs/integrations/IDE_integrations/LibreDB_Studio.md) #### [📄️ Jupyter](https://docs.starrocks.io/docs/integrations/IDE_integrations/jupysql.md) [Jupyter notebooks with StarRocks](https://docs.starrocks.io/docs/integrations/IDE_integrations/jupysql.md) #### [📄️ Marimo](https://docs.starrocks.io/docs/integrations/IDE_integrations/marimo.md) [How to integrate StarRocks with Marimo, a reactive Python notebook for reproducible and interactive data analysis.](https://docs.starrocks.io/docs/integrations/IDE_integrations/marimo.md) --- ### DataGrip DataGrip supports querying both internal data and external data in StarRocks. tip [DataGrip docs](https://www.jetbrains.com/help/datagrip/getting-started.html) You can connect to StarRocks from DataGrip using either the native StarRocks JDBC driver (recommended) or the MySQL driver. #### Connect using the StarRocks JDBC driver (recommended)[​](#connect-using-the-starrocks-jdbc-driver-recommended "Direct link to Connect using the StarRocks JDBC driver (recommended)") The StarRocks JDBC driver provides accurate metadata discovery, which enables schema browsing, auto-complete, and table introspection in DataGrip. ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Download the StarRocks JDBC driver JAR. See [StarRocks JDBC Driver](https://docs.starrocks.io/docs/integrations/JDBC_driver.md) for download instructions. ##### Steps[​](#steps "Direct link to Steps") 1. In DataGrip, go to **File** > **Data Sources** (or click the **Database** icon in the toolbar). 2. Click **+** and select **Driver and Data Source**. 3. In the **Drivers** tab, click **+** to create a new driver. * Set the **Name** to `StarRocks`. * Under **Driver Files**, click **+** and add the StarRocks JDBC driver JAR you downloaded. * Set the **Class** to `com.starrocks.cj.jdbc.Driver`. * Set the **URL template** to one of: ```text jdbc:starrocks://{host}:{port} jdbc:starrocks://{host}:{port}/{database} ``` * Click **OK** to save the driver. ![DataGrip - StarRocks driver configuration](/assets/images/IDE_datagrip_starrocks_driver-aaa1e7f2239f055d5b94bf51e1bd6c33.png) 4. Back in the **Data Sources** tab, click **+** and select the **StarRocks** driver you just created. 5. Configure the connection settings: * **Host**: the FE host IP address of your StarRocks cluster. * **Port**: the FE query port of your StarRocks cluster, for example, `9030`. * **Database**: the database to connect to, in the format `[{catalog_name}.]{database_name}`. Both internal and external catalogs are supported. If the catalog is omitted, `default_catalog` is used. * `catalog_name`: the name of the target catalog in your StarRocks cluster. * `database_name`: the name of the target database in your StarRocks cluster. * **User**: the username to log in to your StarRocks cluster, for example, `admin`. * **Password**: the password to log in to your StarRocks cluster. 6. Click **Test Connection** to verify the settings, then click **OK**. note After connecting, DataGrip only loads the database list. Tables are not fetched until you right-click the database in the sidebar and select **Refresh**. This is expected behavior. ##### Browse catalogs and databases in the sidebar[​](#browse-catalogs-and-databases-in-the-sidebar "Direct link to Browse catalogs and databases in the sidebar") After connecting, there are two ways to control what appears in the left sidebar: **Option 1 — Use the URL to scope directly to one catalog/database (simplest)** Edit the **URL** field to include `catalog.database` as shown in the tip above. DataGrip will display that catalog and database in the sidebar immediately, with no further configuration. **Option 2 — Select schemas in data source properties (multiple catalogs/databases)** 1. Double-click the data source (or right-click > **Properties**). 2. Go to the **Schemas** tab. 3. Check the catalogs and databases you want visible in the sidebar. 4. Click **OK**. DataGrip will refresh and display your selection. #### Connect using the MySQL driver[​](#connect-using-the-mysql-driver "Direct link to Connect using the MySQL driver") The MySQL driver is a fallback for the StarRocks JDBC driver. Create a data source in DataGrip. Note that you must select MySQL as the data source. ![DataGrip - 1](/assets/images/BI_datagrip_1-c1fed7e5af5d1cc0a678f2ccba06cac1.png) ![DataGrip - 2](/assets/images/BI_datagrip_2-b4dc9f2ad1041a911ff1baa18c2bdbdd.png) The parameters that you need to configure are described as follows: * **Host**: the FE host IP address of your StarRocks cluster. * **Port**: the FE query port of your StarRocks cluster, for example, `9030`. * **Authentication**: the authentication method that you want to use. Select **Username & Password**. * **User**: the username that is used to log in to your StarRocks cluster, for example, `admin`. * **Password**: the password that is used to log in to your StarRocks cluster. * **Database**: the data source that you want to access in your StarRocks cluster. The value of this parameter is in the `.` format. * `catalog_name`: the name of the target catalog in your StarRocks cluster. Both internal and external catalogs are supported. * `database_name`: the name of the target database in your StarRocks cluster. Both internal and external databases are supported. --- ### Dataphin Dataphin is a cloud-based output of the internal practices of Alibaba Group's OneData data governance methodology. It provides one-stop solution of data integration, construction, management and utilization throughout the entire lifecycle of big data, aming to helping enterprises significantly improve the level of data governance and build an enterprise-level data middle platform that is of high and reliable quality, convenience consumption, and safe and economical production. Dataphin provides a variety of computing platform support and scalable open capabilities to meet the platform technical architecture and specific requirements of enterprises in various industries. There are several ways to integrate Dataphin with StarRocks: * As the source or destination data source for data integration. Data can be read from StarRocks and push into other data sources, or data can be pull from other data sources and written into StarRocks. * As a source table (unbounded scan), dimension table (bounded scan), or result table (streaming sink and batch sink) of flink SQL and datastram development. * As a data warehouse or a datamart. StarRocks can be registered as a compute source, which can be used for SQL scripts development, scheduling, data quality detection, security identification, and other data research and governance tasks. #### Data integration[​](#data-integration "Direct link to Data integration") You can create StarRocks data sources and use StarRocks data sources as source databases or destination databases in offline integration tasks. The procedure is as follows: ##### Create a StarRocks data source[​](#create-a-starrocks-data-source "Direct link to Create a StarRocks data source") ###### Basic information[​](#basic-information "Direct link to Basic information") ![Create a StarRocks data source - 1](/assets/images/create_sr_datasource_1-7bb418ec67fade24f7247048233bae66.png) * **Name**: Required. Enter a data source name. It can only contain Chinese characters, letters, numbers, underscores (\_), and hyphens (-). It cannot exceed 64 characters in length. * **Data source code**: Optional. After you configure the data source code, you can use the `data source code.table` or `data source code.schema.table` format to reference the Flink SQL in the data source. If you want to automatically access the data source in the corresponding environment, use `${data source code}.table` or `${data source code}.schema.table` format access. > **NOTE** > > Currently, only MySQL, Hologres, and MaxCompute data sources are supported. * **Support scenerios**: The scenerios that data source can be applied in. * **Description**: Optional. You can enter a brief description of the data source. A maximum of 128 characters are allowed. * **Environment**: If the business data source distinguishes between production data source and development data source, choose **Prod and Dev**. If the business data source does not distinguish between production and development data sources, choose **Prod**. * **Tags**: You can select tags to label data sources. ###### Configuration information[​](#configuration-information "Direct link to Configuration information") ![Create a StarRocks data source - 2](/assets/images/create_sr_datasource_2-bfe94f501c2d905494c63d768f62c9c1.png) * **JDBC URL**: Required. The format is `jdbc:mysql://:/`. `host` is the IP address of the FE (Front End) host in the StarRocks cluster, `port` is the query port of FE, and `dbname` is the database name. * **Load URL**: Required. The format is `fe_ip:http_port;fe_ip:http_port`. `fe_ip` is the host of the FE (Front End), and `http_port` is the port of the FE. * **Username**: Required. The username of the database. * **Password**: Required. The password of the database. ###### Advanced settings[​](#advanced-settings "Direct link to Advanced settings") ![Create a StarRocks data source - 3](/assets/images/create_sr_datasource_3-873e78fb773c2c2c7a639482954d8250.png) * **connectTimeout**: the connectTimeout (in ms) of the database. The default value is 900000 milliseconds (15 minutes). * **socketTimeout**: the socketTimeout (in ms) of the database. The default value is 1800000 milliseconds (30 minutes). ##### Read data from StarRocks data sources and write data to other data sources[​](#read-data-from-starrocks-data-sources-and-write-data-to-other-data-sources "Direct link to Read data from StarRocks data sources and write data to other data sources") ###### Drag the StarRocks input component to the offline integration task canvas[​](#drag-the-starrocks-input-component-to-the-offline-integration-task-canvas "Direct link to Drag the StarRocks input component to the offline integration task canvas") ![Read data from StarRocks - 1](/assets/images/read_from_sr_datasource_1-7b39a62322439c4213c7ba1e35b7a2e9.png) ###### StarRocks input component configuration[​](#starrocks-input-component-configuration "Direct link to StarRocks input component configuration") ![Read data from StarRocks - 2](/assets/images/read_from_sr_datasource_2-4c7776a751f027d3666ab4533202a508.png) * **Step name**: Enter an appropriate name based on the scenario and location of the current component. * **Data source**: Select the StarRocks data source or project created on Dataphin. Read permission of the data source is required. If there is no satisfied data source, you can add a data source or apply for relevant permissions. * **Source table**: Select a single table or multiple tables with the same table structure as the input. * **Table**: Select the table in the StarRocks data source from the drop-down list. * **Split key**: Used with concurrency configuration. You can use a column in the source data table as the split key. It is recommended to use a primary key or an indexed column as the split key. * **Batch number**: The number of data records extracted in a batch. * **Input Filtering**: Optional. In the following two cases, you need to fill in the filter information: * If you want to filter a certain part of data. * If you need to incrementally append data on a daily basis or obtain full data, you need to fill in the date whose value is set as the system time of Dataphin console. For example, a transaction table in the StarRocks and the transaction creation date of it is set as `${bizdate}`. * **Output fields**: List the related fields based on the input table information. You can rename, remove, add, and move the fields again. In general, fields are renamed to increase the readability of downstream data or facilitate mapping of fields during output. Fields can be removed during the input stage because relevant fields are not needed in application scenarios. The order of fields is changed to ensure that you can effectively merge data or map output data by mapping fields with different names in the same line when multiple input data are merged or output at the downstream side. ###### Select and configure an output component as the destination data source[​](#select-and-configure-an-output-component-as-the-destination-data-source "Direct link to Select and configure an output component as the destination data source") ![Read data from StarRocks - 3](/assets/images/read_from_sr_datasource_3-6b37b3ae1c5dfcea15015c59d61f1f6f.png) ##### Read data from other data sources and write data to StarRocks data sources[​](#read-data-from-other-data-sources-and-write-data-to-starrocks-data-sources "Direct link to Read data from other data sources and write data to StarRocks data sources") ###### Configure the input component in the offline integration task, and select and configure the StarRocks output component as the destination data source[​](#configure-the-input-component-in-the-offline-integration-task-and-select-and-configure-the-starrocks-output-component-as-the-destination-data-source "Direct link to Configure the input component in the offline integration task, and select and configure the StarRocks output component as the destination data source") ![Write data to StarRocks - 1](/assets/images/write_to_sr_datasource_1-8e19eef8d08e3d0e50e62d022e91a3f2.png) ###### Configure StarRocks output component[​](#configure-starrocks-output-component "Direct link to Configure StarRocks output component") ![Write data to StarRocks - 2](/assets/images/write_to_sr_datasource_2-2553b62350851e43640343c494c78947.png) * **Step Name**: enter an appropriate name based on the scenario and location of the current component. * **Data Source**: Select the Dataphin data source or project created in the StarRocks. The data source that the configuration personnel have the synchronous write permission. If the data source is not satisfied, you can add a data source or apply for relevant permissions. * **Table**: Select the table in the StarRocks data source from the drop-down list. * **Generate Target Table by One Click**: If you have not created a target table in StarRocks data source, you can automatically obtain the name, type, and remarks of the fields read from the upstream, and generate a table creation statement. Click to generate a target table with one click. * **CSV import column delimiter**: Use StreamLoad CSV to import. You can configure the CSV import column delimiter. Default value `\t`. Do not specify the default value here. If the data itself contains `\t`, you must use other characters as delimiters. * **CSV import row delimiter**: Use StreamLoad CSV to import. You can configure the CSV import row delimiter. Default value: `\n`. Do not specify the default value here. If the data itself contains `\n`, you must use other characters as delimiters. * **Parse Solution**: Optional. It is some special processing before or after the data is written. The preparation statement is executed before the data is written to the StarRocks data source, and the completion Statement is executed after the data is written. * **Field Mapping**: You can manually select fields for mapping, or use name-based or position-based mapping to process multiple fields at a time based on the fields from upstream input and those in the destination table. #### Real-time Development[​](#real-time-development "Direct link to Real-time Development") ##### Brief introduction[​](#brief-introduction "Direct link to Brief introduction") StarRocks is a fast and scalable real-time analysis Database. It is commonly used in real-time computing to read and write data to meet the needs of real-time data analysis and query. It is widely used in enterprise real-time computing scenarios. It can be used in real-time business monitoring and analysis, real-time user behavior analysis, real-time advertising bidding system, real-time risk control, anti-fraud, real-time monitoring and early warning and other application scenarios. By analyzing and querying data in real time, enterprises can quickly understand business conditions, optimize decisions, provide better services and protect their interests. ##### StarRocks Connector[​](#starrocks-connector "Direct link to StarRocks Connector") The StarRocks connector supports the following information: | **Category** | **Facts and figures** | | ------------------------------------------------------ | ------------------------------------------- | | Supported types | Source Table, dimension table, result table | | Running mode | Stream mode and batch mode | | Data format | JSON and CSV | | Special Metrics | None | | API type | Datastream and SQL | | Support updating or deleting data in the result table? | Yes | ##### How to use it?[​](#how-to-use-it "Direct link to How to use it?") Dataphin supports StarRocks data sources as read and write targets for realtime compute. You can create StarRocks meta tables and use them for realtime compute tasks: ###### Create StarRocks meta table[​](#create-starrocks-meta-table "Direct link to Create StarRocks meta table") 1. Go to **Dataphin** > **R & D** > **Develop** > **Tables**. 2. Click **Create** to select a real-time compute table. ![Create StarRocks meta table - 1](/assets/images/create_sr_metatable_1-4d12085e290b9653a921ef8c71f4a824.png) * **Table type**: Select **Metatable**. * **Metatable**: Input the name of the meta table. The name immutable. * **Datasource**: Select a StarRocks data source. * **Directory**: Select the directory where you want to create a table. * **Description**: Optional. ![Create StarRocks meta table - 2](/assets/images/create_sr_metatable_2-3a638b34be7589e62d3250478a17c032.png) 3. After creating a meta table, you can edit the meta table, including modifying data sources, source tables, meta table fields, and configuring meta table parameters. ![Edit StarRocks meta table](/assets/images/edit_sr_metatable_1-f1098e912569b1f58fd020bfe32a06b0.png) 4. Submit the meta table. ###### Create Flink SQL task to write data from Kafka to StarRocks in real time[​](#create-flink-sql-task-to-write-data-from-kafka-to-starrocks-in-real-time "Direct link to Create Flink SQL task to write data from Kafka to StarRocks in real time") 1. Go to **Dataphin** > **R & D** > **Develop** > **Computing Tasks**. 2. Click **Create Flink SQL task**. ![Create Flink SQL task - Step 2](/assets/images/create_flink_task_step2-679d17bc320ed44c45b106bc05acbb7c.png) 3. Edit Flink SQL code and precompile It. Kafka meta table is used as an input table and StarRocks meta table as an output table. ![Create Flink SQL task - Step 3 - 1](/assets/images/create_flink_task_step3-1-0524f78767a6e4bf233f19e5020f7a34.png) ​ ![Create Flink SQL task - Step 3 - 2](/assets/images/create_flink_task_step3-2-2484f52fca424ae0f8981fa8fec01587.png) 4. After the precompilation is successful, you can debug and submit the code. 5. Testing in the development environment can be performed by printing logs and writing test tables. The test tables can be set in Meta Tables > Properties > debugging test configurations. ![Create Flink SQL task - Step 5 - 1](/assets/images/create_flink_task_step5-1-5a658adaaaa0bd9341dc172f4cf90254.png) ![Create Flink SQL task - Step 5 - 2](/assets/images/create_flink_task_step5-2-7db3c3caf5a4af70894defecb9b28013.png) 6. After a task in the development environment runs normally, you can publish the task and the meta table used to the production environment. ![Create Flink SQL task - Step 6](/assets/images/create_flink_task_step6-97dfc5403fe84086b4eb54248a51b694.png) 7. Start a task in the production environment to write data from Kafka to StarRocks in real time. You can view the status and logs of each metric in the running analysis to learn about the task running status, or configure monitoring alerts for the task. ![Create Flink SQL task - Step 7 - 1](/assets/images/create_flink_task_step7-1-8637be96949330a35d93af698be5f865.png) ![Create Flink SQL task - Step 7 - 2](/assets/images/create_flink_task_step7-2-9bd69161cb2c4e153bdfc9e1f28fd254.png) #### Data warehouse or data mart[​](#data-warehouse-or-data-mart "Direct link to Data warehouse or data mart") ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * The StarRocks version is 3.0.6 or later. * Dataphin is installed and the Dataphin version is 3.12 or later. * Statistics collection must be enabled. After the StarRocks is installed, collection is enabled by default. For more information, see [Gather statistics for CBO](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md). * StarRocks internal catalog (default catalog) is supported, and external catalog is not supported. ##### Connection configuration[​](#connection-configuration "Direct link to Connection configuration") ###### Metadata warehouse settings[​](#metadata-warehouse-settings "Direct link to Metadata warehouse settings") Dataphin can present and display information based on metadata, including table usage information and metadata changes. You can use StarRocks to process and calculate metadata. Therefore, you need to initialize the metadata Computing Engine (metadata warehouse) before using it. The procedure is as follows: 1. Use an administrator account to log on to Dataphin metadata warehouse tenant 2. Go to Administration > System > Metadata Warehouse Configuration a. Click Start b. Select StarRocks c. Configure the parameters. After passing the test connection, click next. d. Complete meta warehouse initialization ![Metadata warehouse settings](/assets/images/metadata_warehouse_settings_1-bd97bd9aecba5212b07bb2e9432bdf59.png) Parameters are described as follows: * **JDBC URL**: JDBC connection string, which is divided into two parts: * Part I: The format is `jdbc:mysql://:/`. `Host` is the IP address of the FE host in the StarRocks cluster. `Port` is the query Port of FE. Default value: `9030`. * Part Two: format is `database? key1 = value1 & key2 = value2`, where `database` is the name of the StarRocks database used for metadata calculation, which is required. The parameter after '?' is optional. * **Load URL**: The format is `fe_ip:http_port;fe_ip:http_port`. `fe_ip` is host of FE (Front End), and `http_port` is th port of FE. * **Username**: The username used to connect to the StarRocks. The user need to have read and write permissions on the database specified in the JDBC URL, and must have access permissions on the following databases and tables: * All table in Information Schema * *statistics*.column\_statistics * *statistics*.table\_statistic\_v1 * **Password**: the password of the link of StarRocks. * **Meta Project**: The name of the project used for metadata processing in Dataphin. It is only used within the Dataphin system. We recommend that you use `dataphin_meta` as the project name. ###### Create StarRocks project and start data development[​](#create-starrocks-project-and-start-data-development "Direct link to Create StarRocks project and start data development") To start data development, follow these steps: 1. Computing settings. 2. Create StarRocks computing source. 3. Create a project. 4. Create StarRocks SQL task. ###### Computing settings[​](#computing-settings "Direct link to Computing settings") The computing settings set the computing engine type and cluster address of the tenant. The detailed steps are as follows: 1. Log on to the Dataphin as a System Administrator or Super Administrator. 2. Go to **Administration** > **System** > **Computation Configuration**. 3. Select **StarRocks** and click **Next**. 4. Enter the JDBC URL and verify it. The format of the JDBC URL is `jdbc:mysql://:/`. `Host` is the IP address of the FE Host in the StarRocks cluster. `Port` is the query Port of FE. Default value: `9030`. ###### StarRocks Computing source[​](#starrocks-computing-source "Direct link to StarRocks Computing source") Computing source is a concept of Dataphin. Its main purpose is to bind and register Dataphin project space with StarRocks storage computing space (database). You must create a computing source for each project. The detailed steps are as follows: 1. Log on to the Dataphin as a system administrator or Super Administrator. 2. Go to **Planning** > **Engine**. 3. Click **Add Computing Engine** in the upper-right corner to create a computing source. The detailed configuration information is as follows: 1. **Essential information** ![Create compute engine - 1](/assets/images/create_compute_engine_1-94a980eb79c9379f42d84c51235c1620.png) * **Computing Engine Type**: Select **StarRocks**. * **Computing Engine Name**: We recommend that you use the same name as the project to be created. For development projects, add the suffix `_dev`. * **Description**: Optional. Enter the description of the computing source. 2. **Configuration information** ![Create compute engine - 2](/assets/images/create_compute_engine_2-47f0deea152a1809e302e97315803c43.png) * **JDBC URL**: The format is `jdbc:mysql://:/`. `Host` is the IP address of the FE Host in the StarRocks cluster. `Port` is the query Port of FE. Default value: `9030`. * **Load URL**: The format is `fe_ip:http_port;fe_ip:http_port`. `fe_ip` is host of FE (Front End), and `http_port` is th port of FE. * **Username**: The username used to connect to the StarRocks. * **Password**: The password of StarRocks. * **Task Resource Group**: you can specify different StarRocks resource groups for tasks with different priorities. When you select do not specify Resource Group, the StarRocks engine determines the resource group to be executed. When you select specify Resource Group, tasks with different priorities are assigned to the specified resource group by the Dataphin.If a resource group is specified in the code of an SQL task or in the materialized configuration of a logical table, the configuration of the Resource Group of the compute source task is ignored when the task is executed. ![Create compute engine - 3](/assets/images/create_compute_engine_3-df002c75045eb66b55afd092cff0da66.png) ###### Dataphin project[​](#dataphin-project "Direct link to Dataphin project") After you create a computing source, you can bind it to a Dataphin project. Dataphin project manages project members, StarRocks storage and computing space, and manages and maintains computing tasks. To create a Dataphin project, follow these steps: 1. Log on to the Dataphin as a System Administrator or Super Administrator. 2. Go to **Planning** > **Project Management**. 3. Click **Create project** in the upper-right corner to create a project. 4. Enter the basic information and select the StarRocks engine created in the previous step from the offline engine. 5. Click **Create**. ###### StarRocks SQL[​](#starrocks-sql "Direct link to StarRocks SQL") After you create a project, you can create a StarRocks SQL task to perform DDL or DML operations on the StarRocks. The detailed steps are as follows: 1. Go to **R & D** > **Develop**. 2. Click '+' in the upper-right corner to create StarRocks SQL task. ![Configure Dataphin project - 1](/assets/images/configure_dataphin_project_1-06d5728dff9a48b8fb5829e0e4f613c2.png) 3. Enter the name and scheduling type to create an SQL task. 4. Enter SQL in the editor to start DDL and DML operations on StarRock. ![Configure Dataphin project - 2](/assets/images/configure_dataphin_project_2-f6f531faebb06c4c0416385b1b8e2968.png) --- ### DBeaver DBeaver is a SQL client software application and a database administration tool, which offers a helpful assistant that walks you through the process of connecting to a database. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Make sure that you have installed DBeaver. You can download the DBeaver Community edition at [https://dbeaver.io](https://dbeaver.io/) or the DBeaver PRO edition at [https://dbeaver.com](https://dbeaver.com/). #### Integration[​](#integration "Direct link to Integration") Follow these steps to connect to a database: 1. Launch DBeaver. 2. Click the plus sign (**+**) icon in the upper-left corner of the DBeaver window or choose **Database** > **New Database Connection** in the menu bar to access the assistant. ![DBeaver - Access the assistant](/assets/images/IDE_dbeaver_1-0cdfa5e0fbf065c8520b49fb33113c84.png) ![DBeaver - Access the assistant](/assets/images/IDE_dbeaver_2-0d9ee4b2c44a326772aa84f8e2ee2a36.png) 3. Select the StarRocks driver. In the **Select your database** step, you are presented with a list of available drivers. Search for **StarRocks** in the search bar, or click **Analytical** in the left-side pane to locate it. Then, double-click the **StarRocks** icon. note If your version of DBeaver does not include the StarRocks driver, you can use the **MySQL** driver as a fallback. ![DBeaver - Select your database](/assets/images/IDE_dbeaver_3-0c417b72eca77a125da29f31e1a1a826.png) 4. Configure the connection to the database. In the **Connection Settings** step, go to the **Main** tab and configure the following essential connection settings: * **Host**: the FE host IP address of your StarRocks cluster. * **Port**: the FE query port of your StarRocks cluster, for example, `9030`. * **Database/Schema**: the target database in your StarRocks cluster. * **Username**: the username that is used to log in to your StarRocks cluster, for example, `admin`. * **Password**: the password that is used to log in to your StarRocks cluster. note Starting with DBeaver 26.0.5, multi-catalog browsing is supported when using the StarRocks driver, allowing you to explore all catalogs in your cluster without specifying a database. ![DBeaver - Connection Settings - Main tab](/assets/images/IDE_dbeaver_4-334f8e0c979e76a1a61724373b39d598.png) You can also view and edit the properties of the StarRocks driver on the **Driver properties** tab if necessary. To edit a specific property, click the row in the **Value** column for that property. ![DBeaver - Connection Settings - Driver properties tab](/assets/images/IDE_dbeaver_5-9d57615f98709513e124a2b075dd867c.png) 5. Test the connection to the database. Click **Test Connection** to verify the accuracy of the connection settings. A dialog box displaying the StarRocks driver's information appears. Click **OK** in the dialog box to confirm the information. After you have successfully configured the connection settings, click **Finish** to complete the process. ![DBeaver - Test Connection](/assets/images/IDE_dbeaver_6-9b86f85c2e17114132b9571cca316ee8.png) 6. Connect to the database. After the connection is established, you can view it in the left-side database connection tree and DBeaver can effectively connect to the database. ![DBeaver - Connect database](/assets/images/IDE_dbeaver_7-e1a369e03c409939f48ea4ce85944ca0.png) --- ### Jupyter This guide describes how to integrate your StarRocks cluster with [Jupyter](https://jupyter.org/), the latest web-based interactive development environment for notebooks, code, and data. All of this is made possible via [JupySQL](https://jupysql.readthedocs.io/) which allows you to run SQL and plot large datasets in Jupyter via a %sql, %%sql, and %sqlplot magics. You can use JupySQL on top of Jupyter to run queries on top of StarRocks. Once the data is loaded into the cluster, you can query and visualize it via SQL plotting. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before getting started, you must have the following software installed locally: * [JupySQL](https://jupysql.readthedocs.io/en/latest/quick-start.html): `pip install jupysql` * Jupyterlab: `pip install jupyterlab` * [SKlearn Evaluation](https://github.com/ploomber/sklearn-evaluation): `pip install sklearn-evaluation` * Python * pymysql: `pip install pymysql` > **NOTE** > > Once you have the above requirements fulfilled, you can open Jupyter lab simply by calling `jupyterlab` - this will open the notebook interface. If Jupyter lab is already running in a notebook, you can simply run the cell bellow to get the dependencies. ```python # Install required packages. %pip install --quiet jupysql sklearn-evaluation pymysql ``` > **NOTE** > > You may need to restart the kernel to use updated packages. ```python import pandas as pd from sklearn_evaluation import plot # Import JupySQL Jupyter extension to create SQL cells. %load_ext sql %config SqlMagic.autocommit=False ``` **You will need to make sure your StarRocks instance is up and reachable for the next stages.** > **NOTE** > > You will need to adjust the connection string according to the instance type you are trying to connect to (url, user, and password). The example below uses a local instance. #### Connecting to StarRocks via JupySQL[​](#connecting-to-starrocks-via-jupysql "Direct link to Connecting to StarRocks via JupySQL") In this example, a docker instance is used, and that is reflecting the data in the connection string. The `root` user is used to connect to the local StarRocks instance, create a database, and check that data can actually be read from and written into the table. ```python %sql mysql+pymysql://root:@localhost:9030 ``` Create and use that JupySQL database: ```python %sql CREATE DATABASE jupysql; ``` ```python %sql USE jupysql; ``` Create a table: ```python %%sql CREATE TABLE tbl(c1 int, c2 int) distributed by hash(c1) properties ("replication_num" = "1"); INSERT INTO tbl VALUES (1, 10), (2, 20), (3, 30); SELECT * FROM tbl; ``` #### Saving and loading queries[​](#saving-and-loading-queries "Direct link to Saving and loading queries") Now after you create a database, you can write some sample data into it and query it. JupySQL allows you to break queries into multiple cells, simplifying the process of building large queries. You can write complex queries, save them, and execute them when needed, in a similar manner to CTEs in SQL. ```python # This is pending for the next JupySQL release. %%sql --save initialize-table --no-execute CREATE TABLE tbl(c1 int, c2 int) distributed by hash(c1) properties ("replication_num" = "1"); INSERT INTO tbl VALUES (1, 1), (2, 2), (3, 3); SELECT * FROM tbl; ``` > **NOTE** > > `--save` stores the query, not the data. Note that we are using `--with;`, this will retrieve previously saved queries, and prepend them (using CTEs). Then, we save the query in `track_fav`. #### Plotting directly on StarRocks[​](#plotting-directly-on-starrocks "Direct link to Plotting directly on StarRocks") JupySQL comes with a few plots by default, allowing you to visualize the data directly in SQL. You can use a bar plot to visualize the data in your newly created table: ```python top_artist = %sql SELECT * FROM tbl top_artist.bar() ``` Now you have a new bar plot without any extra code. You can run SQL directly from your notebook via JupySQL (by ploomber). This adds lots of possibilities around StarRocks for data scientists and engineers. In case that you got stuck or need any support, please reach out to us via Slack. --- ### LibreDB Studio [LibreDB Studio](https://libredb.org) is an open source (MIT licensed) SQL IDE that runs in a browser rather than installing as a desktop application. It ships as a Docker image, a Helm chart, or an npm package, and connects to StarRocks over the MySQL wire protocol using its MySQL driver. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * A running StarRocks cluster reachable from wherever LibreDB Studio runs, and its FE query port (MySQL protocol, default `9030`). * LibreDB Studio itself. Run it with Docker: ```sh docker run -p 3000:3000 ghcr.io/libredb/libredb-studio:latest ``` On first run it prints a generated admin email and password to the container logs (`docker logs `); sign in with those. A Helm chart and an npm package (`npx @libredb/studio`) are also available; see the [LibreDB Studio repository](https://github.com/libredb/libredb-studio) for details. If StarRocks also runs in Docker, put both containers on the same Docker network and connect using the StarRocks container's name rather than `127.0.0.1` or `localhost`. On Docker's default network a container cannot reach the host's published ports by loopback address, and the two containers cannot resolve each other by name unless they share a network: ```sh docker network create starrocks-net docker run --name starrocks --network starrocks-net -p 9030:9030 -p 8030:8030 \ starrocks/allin1-ubuntu:latest docker run --network starrocks-net -p 3000:3000 ghcr.io/libredb/libredb-studio:latest ``` With this setup, use `starrocks` (the container name) as the host in the connection settings below, not an IP address. #### Integration[​](#integration "Direct link to Integration") 1. Open LibreDB Studio in your browser and sign in with the admin credentials from the container logs. An admin account opens to the Admin Dashboard first; click **Editor** in its top bar to reach the SQL editor workspace, whose left sidebar lists connections. 2. Click the **+** button at the top of the sidebar to add a new connection. 3. Select **MySQL** as the connection type. StarRocks speaks the MySQL wire protocol, so there is no separate StarRocks driver to pick. 4. Fill in the connection settings: * **Host**: your FE hostname or IP address (the StarRocks container's name if both run on the same Docker network, as above) * **Port**: the FE query port, `9030` by default (not MySQL's `3306`) * **User** / **Password**: your StarRocks credentials * **Database**: the target database name 5. Click **Test Connection** to verify, then **Establish Connection** to save it. Test Connection reports a connection with no health data (see below for why); the first click of Establish Connection asks you to confirm that with a second click, rather than failing. The connection appears in the sidebar. The SQL editor, the table browser, and the table and storage statistics all work against StarRocks, including correct row counts and sizes once StarRocks' own background statistics collector has caught up with a freshly loaded table. A few surfaces that a MySQL-protocol tool might expect are unavailable: * **Active sessions / connection health**: StarRocks has no `information_schema.PROCESSLIST`, so the active-sessions view and the connection health check do not answer. The connection itself still works; only that one reading is unavailable. * **Slow-query history**: StarRocks has no `performance_schema` database, so this is not available. * **Index information**: StarRocks exposes no secondary-index catalog, so no index information is shown. * **Graphical query plan**: `EXPLAIN FORMAT='json'` is not accepted, so the graphical query-plan view does not render. A plain `EXPLAIN` still runs from the editor. ![LibreDB Studio - Query result against StarRocks](/assets/images/IDE_libredb_studio_1-4ea59b55ebf3e376966c078fc4c129e3.png) --- ### Marimo Integrate your StarRocks cluster with [Marimo](https://marimo.io/), a reactive Python notebook built for reproducibility and interactivity. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") First, start by installing Marimo and setting up a notebook according to the [Marimo quickstart documentation](https://github.com/marimo-team/marimo#quickstart). You will also need the following packages: ```bash pip install starrocks sqlalchemy pandas ``` #### Connecting to StarRocks[​](#connecting-to-starrocks "Direct link to Connecting to StarRocks") Use [SQLAlchemy](https://www.sqlalchemy.org/) to create a connection engine. The connection string format is: ```text starrocks://username:password@host:port/database ``` ```python import marimo as mo import sqlalchemy as sa engine = sa.create_engine("starrocks://username:password@:9030") ``` Replace `` with your StarRocks FE host. #### Using Marimo UI for credentials[​](#using-marimo-ui-for-credentials "Direct link to Using Marimo UI for credentials") To avoid hardcoding credentials, use Marimo's interactive UI elements to collect them at runtime. **Cell 1** — render input fields: ```python user = mo.ui.text(label="Username") pw = mo.ui.text(label="Password", kind="password") mo.hstack([user, pw]) ``` **Cell 2** — create the engine using the entered values: ```python engine = sa.create_engine( f"starrocks://{user.value}:{pw.value}@:9030" ) ``` #### Querying StarRocks[​](#querying-starrocks "Direct link to Querying StarRocks") With the engine established, use pandas to run queries: ```python import pandas as pd df = pd.read_sql("SELECT * FROM my_database.my_table LIMIT 100", engine) mo.ui.table(df) ``` ![Marimo notebook connected to StarRocks](/assets/images/marimo_starrocks-5e392b2109c487ad9d31bf31b0d8a1e2.png) note Multi-catalog support requires Marimo version 0.22.5 or later. --- ### StarRocks JDBC Driver StarRocks provides a native JDBC driver that enables direct connectivity from any JDBC-compatible client, IDE, or application. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Java 8 or later * A running StarRocks cluster #### Download[​](#download "Direct link to Download") The StarRocks JDBC driver is available on [Maven Central](https://central.sonatype.com/artifact/com.starrocks/starrocks-connector-j). You can download the JAR directly from Maven Central, or add it as a dependency in your project using the instructions below. ##### Download via Maven CLI[​](#download-via-maven-cli "Direct link to Download via Maven CLI") If you have Maven installed, you can download the JAR without creating a project: ```bash mvn dependency:get -Dartifact=com.starrocks:starrocks-connector-j:1.1.1 ``` The JAR will be saved to your local Maven repository at: ```text ~/.m2/repository/com/starrocks/starrocks-connector-j/1.1.1/starrocks-connector-j-1.1.1.jar ``` #### Use the JAR in your project[​](#use-the-jar-in-your-project "Direct link to Use the JAR in your project") ##### Maven[​](#maven "Direct link to Maven") Add the following dependency to your `pom.xml`: ```xml com.starrocks starrocks-connector-j 1.1.1 ``` ##### Gradle[​](#gradle "Direct link to Gradle") Add the following dependency to your `build.gradle`: ```groovy implementation 'com.starrocks:starrocks-connector-j:1.1.1' ``` ##### Plain Java[​](#plain-java "Direct link to Plain Java") Download the JAR from [Maven Central](https://central.sonatype.com/artifact/com.starrocks/starrocks-connector-j) and add it to the classpath when compiling and running: ```bash javac -cp starrocks-connector-j-.jar MyApp.java java -cp .:starrocks-connector-j-.jar MyApp ``` #### Connection URL format[​](#connection-url-format "Direct link to Connection URL format") ```text jdbc:starrocks://:/. ``` | Parameter | Description | | --------------- | --------------------------------------------------------------------------------------------------------- | | `fe_host` | The FE host IP address of your StarRocks cluster. | | `fe_query_port` | The FE query port, default `9030`. | | `catalog` | The catalog to connect to. Use `default_catalog` for internal tables, or the name of an external catalog. | | `database` | The database within the catalog. | **Example:** ```text jdbc:starrocks://192.168.1.1:9030/default_catalog.my_database ``` #### Connection properties[​](#connection-properties "Direct link to Connection properties") | Property | Description | | ---------- | ---------------------------------------------------------- | | `user` | The username to log in to StarRocks, for example, `admin`. | | `password` | The password to log in to StarRocks. | #### Metadata discovery[​](#metadata-discovery "Direct link to Metadata discovery") The StarRocks JDBC driver supports standard JDBC metadata APIs (`DatabaseMetaData`), which allow tools to introspect catalogs, schemas, tables, and columns. This enables IDE features such as schema browsing, auto-complete, and table introspection to work out of the box. #### Example: connect from Java[​](#example-connect-from-java "Direct link to Example: connect from Java") ```java import java.sql.Connection; import java.sql.DriverManager; import java.sql.ResultSet; import java.sql.Statement; public class StarRocksExample { public static void main(String[] args) throws Exception { String url = "jdbc:starrocks://192.168.1.1:9030/default_catalog.my_database"; Connection conn = DriverManager.getConnection(url, "admin", "password"); try (Statement stmt = conn.createStatement(); ResultSet rs = stmt.executeQuery("SELECT * FROM my_table LIMIT 10")) { while (rs.next()) { System.out.println(rs.getString(1)); } } conn.close(); } } ``` --- ### Loading tools #### [📄️ StarRocks Migration Tool (SMT)](https://docs.starrocks.io/docs/integrations/loading_tools/SMT.md) [StarRocks Migration Tool (SMT) is a data migration tool provided by StarRocks to load data from source databases through Flink into StarRocks.](https://docs.starrocks.io/docs/integrations/loading_tools/SMT.md) #### [📄️ DataX writer](https://docs.starrocks.io/docs/integrations/loading_tools/DataX-starrocks-writer.md) [The StarRocksWriter plugin allows writing data to StarRocks' destination table.](https://docs.starrocks.io/docs/integrations/loading_tools/DataX-starrocks-writer.md) #### [📄️ BladePipe](https://docs.starrocks.io/docs/integrations/loading_tools/CloudCanal.md) [How to use BladePipe to replicate data in real time from databases and message queues into StarRocks.](https://docs.starrocks.io/docs/integrations/loading_tools/CloudCanal.md) --- ### BladePipe #### Introduction[​](#introduction "Direct link to Introduction") BladePipe is a **real-time end-to-end data replication tool**, moving data between **30+** databases, message queues, search engines, caching systems, real-time data warehouses, data lakes and more, with **ultra-low latency**. It features efficiency, stability and scalability, compatibility with diverse database engines, one-stop management, enhanced security, and complex data transformation. BladePipe helps to break down data silos, increasing the value of data. ![image.png](/assets/images/3.11-1-7e9c0c4be79fb00b9fa73f997d79dbf1.png) #### Features[​](#features "Direct link to Features") BladePipe presents a visualized management interface, allowing you to easily create DataJobs to achieve **schema migration, data migration, synchronization, verification and correction**, and more. In addition, it supports more refined and customized configurations by parameter settings. Now BladePipe supports data movement from the following source DataSources to StarRocks: | Source DataSource | Schema Migration | Data Migration | Data Sync | Verification & Correction | | --------------------------- | ---------------- | -------------- | --------- | ------------------------- | | MySQL/MariaDB/AuroraMySQL | Yes | Yes | Yes | Yes | | Oracle | Yes | Yes | Yes | Yes | | PostgreSQL/AuroraPostgreSQL | Yes | Yes | Yes | Yes | | SQL Server | Yes | Yes | Yes | Yes | | Kafka | No | No | Yes | No | | AutoMQ | No | No | Yes | No | | TiDB | Yes | Yes | Yes | Yes | | Hana | Yes | Yes | Yes | Yes | | PolarDB for MySQL | Yes | Yes | Yes | Yes | | Db2 | Yes | Yes | Yes | Yes | info For more information on supported functions and parameter settings, refer to [BladePipe Connections](https://www.bladepipe.com/docs/dataMigrationAndSync/connection/mysql2/?target=StarRocks). #### Installation[​](#installation "Direct link to Installation") [BladePipe](https://www.bladepipe.com/docs/quick/quick_start/) #### Example[​](#example "Direct link to Example") Taking a MySQL instance as an example, the following section describes how to migrate data from MySQL to StarRocks. ##### Add DataSources[​](#add-datasources "Direct link to Add DataSources") 1. Log in to the [BladePipe Cloud](https://cloud.bladepipe.com/). Click **DataSource** > **Add DataSource**. 2. Select **StarRocks** as the **Type**, and fill in the setup form. * **Client Address**：The port StarRocks provided to MySQL Client. BladePipe queries the metadata in databases via it. * **Account**: The user name of the StarRocks database. The INSERT privilege is required to write data to StarRocks. Follow the instruction provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the database user the INSERT privilege. * **Http Address**：The port used to receive the request from BladePipe to write data to StarRocks. ![image.png](/assets/images/3.11-2-33c064f79fe30ea39142c3766da5f0e0.png) 3. Click **Test Connection**. After the connection is successful, click **Add DataSource** to add the DataSource. 4. Add a MySQL DataSource following the above steps. ##### Create DataJob[​](#create-datajob "Direct link to Create DataJob") 1. Click **DataJob** > [**Create DataJob**](https://www.bladepipe.com/docs/operation/job_manage/create_job/create_full_incre_task/). 2. Select the source and target DataSources, and click **Test Connection** to ensure the connection to the source and target DataSources are both successful. ![image.png](/assets/images/3.11-3-a4b87453ed4bcc144b6f21b1aa4ed831.png) 3. Select **Incremental** for DataJob Type, together with the **Full Data** option. ![image.png](/assets/images/3.11-4-2c50b329100f9ee87f65566814faedd4.png) 4. Select the tables to be replicated. **Note that the target StarRocks tables automatically created after Schema Migration have primary keys, so source tables without primary keys are not supported currently**. ![image.png](/assets/images/3.11-5-12f941b5b9ada9b6abc103dea10c5a92.png) 5. Select the columns to be replicated. ![image.png](/assets/images/3.11-6-781944480bf1b87d1f4250b6591efe15.png) 6. Confirm the DataJob creation. ![image.png](/assets/images/3.11-7-dfe068ebe9474e54bbeed00ba926f6a6.png) 7. The DataJob runs automatically. BladePipe will automatically run the following DataTasks: * **Schema Migration**: The schemas of the source tables will be migrated to the target instance. * **Full Data**: All existing data of the source tables will be fully migrated to the target instance. * **Incremental**: Ongoing data changes will be continuously synchronized to the target instance (with latency less than a minute). ![image.png](/assets/images/3.11-8-9c11d378ec4ac2112b33039d3430c931.png) --- ### DataX writer #### Introduction[​](#introduction "Direct link to Introduction") The StarRocksWriter plugin allows writing data to StarRocks' destination table. Specifically,StarRocksWriter imports data to StarRocks in CSV or JSON format via [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md), and internally caches and bulk imports the data read by `reader` to StarRocks for better write performance. The overall data flow is `source -> Reader -> DataX channel -> Writer -> StarRocks`. [Download the plugin](https://github.com/StarRocks/DataX/releases) Please go to `https://github.com/alibaba/DataX` to download the full package of DataX, and put the starrockswriter plugin into the `datax/plugin/writer/` directory. Use the following command to test: `python datax.py --jvm="-Xms6G -Xmx6G" --loglevel=debug job.json` #### Function Description[​](#function-description "Direct link to Function Description") ##### Sample Configuration[​](#sample-configuration "Direct link to Sample Configuration") Here is a configuration file for reading data from MySQL and loading it to StarRocks. ```json { "job": { "setting": { "speed": { "channel": 1 }, "errorLimit": { "record": 0, "percentage": 0 } }, "content": [ { "reader": { "name": "mysqlreader", "parameter": { "username": "xxxx", "password": "xxxx", "column": [ "k1", "k2", "v1", "v2" ], "connection": [ { "table": [ "table1", "table2" ], "jdbcUrl": [ "jdbc:mysql://127.0.0.1:3306/datax_test1" ] }, { "table": [ "table3", "table4" ], "jdbcUrl": [ "jdbc:mysql://127.0.0.1:3306/datax_test2" ] } ] } }, "writer": { "name": "starrockswriter", "parameter": { "username": "xxxx", "password": "xxxx", "database": "xxxx", "table": "xxxx", "column": ["k1", "k2", "v1", "v2"], "preSql": [], "postSql": [], "jdbcUrl": "jdbc:mysql://172.28.17.100:9030/", "loadUrl": ["172.28.17.100:8030", "172.28.17.100:8030"], "loadProps": {} } } } ] } } ``` #### Starrockswriter Parameter Description[​](#starrockswriter-parameter-description "Direct link to Starrockswriter Parameter Description") * **username** * Description: The user name of the StarRocks database * Required: Yes * Default value: none * **password** * Description: The password for the StarRocks database * Required: Yes * Default: None * **database** * Description: The name of the database for the StarRocks table. * Required: Yes * Default: None * **table** * Description: The name of the table for the StarRocks table. * Required: Yes * Default: None * **loadUrl** * Description: The address of the StarRocks FE for stream load, can be multiple FE addresses, in the form of `fe_ip:fe_http_port`. * Required: yes * Default value: none * **column** * Description: The fields of the destination table **that need to be written to the data**, with the columns separated by commas. Example: "column": \["id", "name", "age"]. > **column configuration item must be specified and cannot be left blank.** Note: We strongly discourage you from leaving it empty, because your job may run incorrectly or fail when you change the number of columns, type, etc. of the destination table. The configuration items must be in the same order as the querySQL or column in the reader. * Required: Yes * Default value: No * **preSql** * Description: The standard statement will be executed before writing data to the destination table. * Required: No * Default: No * **jdbcUrl** * Description: JDBC connection information of the destination database for executing `preSql` and `postSql`. * Required: No * Default: No * **loadProps** * Description: Request parameters for StreamLoad, refer to the StreamLoad introduction page for details. * Required: No * Default value: No #### Type conversion[​](#type-conversion "Direct link to Type conversion") By default, incoming data is converted to strings, with `t` as column separator and `n` as row separator, to form `csv` files for StreamLoad import . To change the column separator, configure `loadProps` properly. ```json "loadProps": { "column_separator": "\\x01", "row_delimiter": "\\x02" } ``` To change the import format to `json`, configure `loadProps` properly. ```json "loadProps": { "format": "json", "strip_outer_array": true } ``` > The `json` format is for the writer to import data to StarRocks in JSON format. #### About time zone[​](#about-time-zone "Direct link to About time zone") If the source tp library is in another time zone, when executing datax.py, add the following parameter after the command line ```json "-Duser.timezone=xx" ``` e.g. If DataX imports Postgrest data and the source library is in UTC time, add the parameter "-Duser.timezone=GMT+0" to startup. --- ### StarRocks Migration Tool (SMT) StarRocks Migration Tool (SMT) is a command-line tool that generates the SQL you need to load data from a source database into StarRocks through Flink. SMT mainly can: * Generate statements to create tables in StarRocks based on information of the source database and the target StarRocks cluster. * Generate SQL statements that can be executed in Flink's SQL client to submit Flink jobs for synchronizing data, which simplifies full or incremental data synchronization in the pipeline. Currently, SMT supports the following source databases: | Source database | Generate statement to create table in StarRocks | Full data synchronization | Incremental data synchronization | | --------------- | ----------------------------------------------- | ------------------------- | -------------------------------- | | MySQL | Supported | Supported | Supported | | PostgreSQL | Supported | Supported | Supported | | Oracle | Supported | Supported | Supported | | Hive | Supported | Supported | Not supported | | ClickHouse | Supported | Supported | Not supported | | SQL Server | Supported | Supported | Supported | | TiDB | Supported | Supported | Supported | Download link: #### What SMT does and does not do[​](#what-smt-does-and-does-not-do "Direct link to What SMT does and does not do") SMT generates SQL files and then exits. It reads the table definitions in your source database and writes two kinds of files to the `result` directory: * `starrocks-create.*.sql`: CREATE TABLE statements, which you run in StarRocks. * `flink-create.*.sql`: Flink SQL that defines a source table and a sink table and submits an `INSERT INTO ... SELECT` job, which you run in the Flink SQL client. SMT does not start, run, or connect to Flink, and it does not move any data. You still install and run a Flink cluster with the Flink CDC connector for your source database and the [Flink connector for StarRocks](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks.md), and you run the generated files yourself. ##### Synchronize data without SMT[​](#synchronize-data-without-smt "Direct link to Synchronize data without SMT") SMT is optional. Instead, you can: * Use a [Flink CDC pipeline](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/data-pipeline/). A pipeline is defined in a single YAML file, creates the StarRocks tables itself, and runs the synchronization. See [Realtime synchronization from PostgreSQL](https://docs.starrocks.io/docs/loading/Flink_cdc_postgres.md), or for MySQL, [Streaming ELT from MySQL to StarRocks](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.6/docs/get-started/quickstart-for-1.20/mysql-to-starrocks/) in the Flink CDC documentation. * Create the StarRocks tables yourself, and write the Flink SQL with the [Flink connector for StarRocks](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks.md) as the sink. This works for any source that Flink can read. #### Steps to use SMT[​](#steps-to-use-smt "Direct link to Steps to use SMT") The generally involved steps are as follows: 1. Configure the **conf/config\_prod.conf** file. 2. Execute **starrocks-migration-tool**. 3. After execution, the SQL scripts are generated in the **result** directory by default. You can then use the SQL scripts in the **result** directory for metadata or data synchronization. #### Configurations of SMT[​](#configurations-of-smt "Direct link to Configurations of SMT") * `[db]`: information to connect the data source. Configure the information to connect the data source corresponding to the database type specified in the `type` patameter. * `[other]`: additional configurations. It is recommended to specify the actual number of BE nodes in the `be_num` patameter. * `flink.starrocks.sink.*`: configurations of flink-connector-starrocks. For detailed configurations and description, see [configuration description](https://github.com/StarRocks/flink-connector-starrocks#sink-options). * `[table-rule.1]`: rule to match tables in the data source. The CREATE TABLE statement is generated based on the regular expressions configured in the rule to match the names of databases and tables in data source. Multiple rules can be configured, and each rule generates a corresponding result file, for example: * `[table-rule.1]` -> `result/starrocks-create.1.sql` * `[table-rule.2]` -> `result/starrocks-create.2.sql` Each rule needs to contain the configurations of database, table, and flink-connector-starrocks. ```bash [table-rule.1] # pattern to match databases for setting properties database = ^ database1.*$ # pattern to match tables for setting properties table = ^.*$ schema = ^.*$ ############################################ ### flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`, they are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.strip_outer_array=true [table-rule.2] # pattern to match databases for setting properties database = ^database2.*$ # pattern to match tables for setting properties table = ^.*$ schema = ^.*$ ############################################ ### flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`, they are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.strip_outer_array=true ``` * A separate rule can be configured for a large table that is split into shards in databases. For example, suppose that the two databases `edu_db_1` and `edu_db_2` contain the tables `course_1` and `course_2` respectively, and these two tables have the same structure. You can use the following rule to load data from these two tables into one StarRocks table for analysis. ```bash [table-rule.3] # pattern to match databases for setting properties database = ^edu_db_[0-9]*$ # pattern to match tables for setting properties table = ^course_[0-9]*$ schema = ^.*$ ``` This rule will automatically form a many-to-one loading relationship. The default name of the table that will be generated in StarRocks is `course__auto_shard`, and you can also modify the table name in the related SOL script, such as `result/starrocks-create.3.sql`. #### Synchronize MySQL to StarRocks[​](#synchronize-mysql-to-starrocks "Direct link to Synchronize MySQL to StarRocks") ##### Introduction[​](#introduction "Direct link to Introduction") Flink CDC connector and SMT can synchronize data from MySQL within subsecond. ![img](/assets/images/load_tools-8e88eab0e5c9d2a228fcd668c63bc3e1.png) As shown in the image, SMT can automatically generate CREATE TABLE statements of the Flink's source and sink tables based on the cluster information and table structure of MySQL and StarRocks. Flink CDC connector reads the MySQL Binlog and Flink-connector-starrocks writes data to StarRocks . ##### Steps[​](#steps "Direct link to Steps") | Dependency | Package name | Download link | | ------------------------- | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | Flink | flink-x.x.x-bin-scala\_2.12.tgz | [Flink](https://flink.apache.org/downloads/) | | Flink CDC connector | flink-sql-connector-mysql-cdc-x.x.x.jar | [Flink CDC connector](https://github.com/apache/flink-cdc/releases) | | Flink-connector-starrocks | flink-connector-starrocks-x.x.x\_flink-x.x.jar | [Flink connector for StarRocks](https://github.com/StarRocks/starrocks-connector-for-apache-flink/releases) | | SMT | smt.tar.gz | [SMT](https://cdn-thirdparty.starrocks.com/smt.tar.gz?r=2) | 1. Download [Flink](https://flink.apache.org/downloads/). Flink 1.11 or later is supported. 2. Download [Flink CDC connector](https://github.com/ververica/flink-cdc-connectors/releases). Make sure that you download the `flink-sql-connector-mysql-cdc-xxx.jar` corresponding to the version of Flink. 3. Download [Flink-connector-starrocks](https://github.com/StarRocks/starrocks-connector-for-apache-flink/releases). 4. Copy **flink-sql-connector-mysql-cdc-xxx.jar** and **flink-connector-starrocks-xxx.jar** to **flink-xxx/lib/**. 5. Download [smt.tar.gz](https://cdn-thirdparty.starrocks.com/smt.tar.gz?r=2). 6. Extract and modify the configuration file of SMT. ```bash [db] host = 192.168.1.1 port = 3306 user = root password = type = mysql [other] # number of backends in StarRocks be_num = 3 # `decimal_v3` is supported since StarRocks-1.18.1 use_decimal_v3 = false # directory to save the converted DDL SQL output_dir = ./result [table-rule.1] # pattern to match databases for setting properties database = ^db$ # pattern to match tables for setting properties table = ^table$ schema = ^.*$ ############################################ ### flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`, they are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.strip_outer_array=true ``` 7. Execute **starrocks-migrate-tool**. All SQL scripts are generated in the result directory. ```bash $./starrocks-migrate-tool $ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql starrocks-external-create.all.sql ``` 8. Use a SQL script whose prefix is **starrocks-create** to generate the table in StarRocks. ```bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ``` 9. Use a SQL script whose prefix is **flink-create** to generate the Flink source and sink tables, and start a Flink job to synchronize data. ```bash bin/sql-client.sh embedded < flink-create.all.sql ``` After the above command is successfully executed, the Flink job to synchronize data keeps running. 10. Observe the status of the Flink job. ```bash bin/flink list ``` If the job execution encounters an error, you can view detailed error information in the Flink logs. Also, you can modify the Flink configurations in the file **conf/flink-conf.yaml**, such as memory and slot. ##### Notes[​](#notes "Direct link to Notes") * How to enable MySQL binlog? 1. Modify /etc/my.cnf: ```plaintext # Enable binlog log-bin=/var/lib/mysql/mysql-bin #log_bin=ON ## Base name of binlog files #log_bin_basename=/var/lib/mysql/mysql-bin ## Index file for binlog files, managing all binlog files #log_bin_index=/var/lib/mysql/mysql-bin.index # Configure server id server-id=1 binlog_format = row ``` 2. Restart mysqld. You can check whether MySQL binlog is enabled by executing `SHOW VARIABLES LIKE 'log_bin';`. #### Synchronize PostgreSQL to StarRocks[​](#synchronize-postgresql-to-starrocks "Direct link to Synchronize PostgreSQL to StarRocks") tip With Flink CDC 3.5 or later, a Flink CDC pipeline can synchronize PostgreSQL to StarRocks without SMT, and creates the StarRocks tables itself. For a step-by-step guide, see [Realtime synchronization from PostgreSQL](https://docs.starrocks.io/docs/loading/Flink_cdc_postgres.md). ##### Introduction[​](#introduction-1 "Direct link to Introduction") Flink CDC connector and SMT can synchronize data from PostgreSQL within subsecond. SMT can automatically generate CREATE TABLE statements of the Flink's source and sink tables based on the cluster information and table structure of PostgreSQL and StarRocks. Flink CDC connector reads the WAL of PostgreSQL and Flink-connector-starrocks writes data to StarRocks . ##### Steps[​](#steps-1 "Direct link to Steps") 1. Download [Flink](https://flink.apache.org/downloads/). The version of Flink is supported to be 1.11 or later. 2. Download [Flink CDC connector](https://github.com/ververica/flink-cdc-connectors/releases). Make sure that you download the flink-sql-connector-postgres-cdc-xxx.jar corresponding to the version of Flink. 3. Download [Flink StarRocks connector](https://github.com/StarRocks/flink-connector-starrocks). 4. Copy **flink-sql-connector-postgres-cdc-xxx.jar** and **flink-connector-starrocks-xxx.jar** to **flink-xxx/lib/**. 5. Download [smt.tar.gz](https://cdn-thirdparty.starrocks.com/smt.tar.gz?r=2). 6. Extract and modify the configuration file of SMT. ```bash [db] host = 192.168.1.1 port = 5432 user = xxx password = xxx type = pgsql [other] # number of backends in StarRocks be_num = 3 # `decimal_v3` is supported since StarRocks-1.18.1 use_decimal_v3 = false # directory to save the converted DDL SQL output_dir = ./result [table-rule.1] # pattern to match databases for setting properties database = ^db$ # pattern to match tables for setting properties table = ^table$ # pattern to match schemas for setting properties schema = ^.*$ ############################################ ### flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`, they are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.strip_outer_array=true ``` 7. Execute **starrocks-migrate-tool**. All SQL scripts are generated in the result directory. ```bash $./starrocks-migrate-tool $ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql ``` 8. Use a SQL script whose prefix is **starrocks-create** to generate the table in StarRocks. ```bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ``` 9. Use a SQL script whose prefix is **flink-create** to generate the Flink source and sink tables, and start a Flink job to synchronize data. ```bash bin/sql-client.sh embedded < flink-create.all.sql ``` After the above command is successfully executed, the Flink job to synchronize data keeps running. 10. Observe the status of the Flink job. ```bash bin/flink list ``` If the job execution encounters an error, you can view detailed error information in the Flink logs. Also, you can modify the Flink configurations in the file **conf/flink-conf.yaml**, such as memory and slot. ##### Notes[​](#notes-1 "Direct link to Notes") * For PostgreSQL `v9.*`, a special flink-cdc configuration as shown below is required (It is recommended to use PostgreSQL `v10.*` or later. Otherwise, you need to install WAL decoding plugins): ```bash ############################################ ############################################ ### flink-cdc plugin configuration for `postgresql` ############################################ ### for `9.*` decoderbufs, wal2json, wal2json_rds, wal2json_streaming, wal2json_rds_streaming ### refer to https://ververica.github.io/flink-cdc-connectors/master/content/connectors/postgres-cdc.html ### and https://debezium.io/documentation/reference/postgres-plugins.html ### flink.cdc.decoding.plugin.name = decoderbufs ``` * How to enable PostgreSQL WAL? ```bash # Allow the Flink hosts to connect. Replace 10.0.0.0/24 with their address range. echo "host all all 10.0.0.0/24 scram-sha-256" >> pg_hba.conf echo "host replication all 10.0.0.0/24 scram-sha-256" >> pg_hba.conf # Enable wal logical replication echo "wal_level = logical" >> postgresql.conf echo "max_wal_senders = 2" >> postgresql.conf echo "max_replication_slots = 8" >> postgresql.conf ``` Specify replica identity FULL for the tables that need to be synchronized. ```sql ALTER TABLE schema_name.table_name REPLICA IDENTITY FULL ``` After making these changes, restart PostgreSQL . #### Synchronize Oracle to StarRocks[​](#synchronize-oracle-to-starrocks "Direct link to Synchronize Oracle to StarRocks") ##### Introduction[​](#introduction-2 "Direct link to Introduction") Flink CDC connector and SMT can synchronize data from Oracle within subsecond. SMT can automatically generate CREATE TABLE statements of the Flink's source and sink tables based on the cluster information and table structure of Oracle and StarRocks. Flink CDC connector reads the logminer of Oracle and Flink-connector-starrocks writes data to StarRocks. ##### Steps[​](#steps-2 "Direct link to Steps") 1. Download [Flink](https://flink.apache.org/downloads/). The version of Flink is supported to be 1.11 or later. 2. Download [Flink CDC connector](https://github.com/ververica/flink-cdc-connectors/releases). Make sure that you download the flink-sql-connector-oracle-cdc-xxx.jar corresponding to the Flink version. 3. Download [Flink StarRocks connector](https://github.com/StarRocks/flink-connector-starrocks). 4. Copy `flink-sql-connector-oracle-cdc-xxx.jar` and `flink-connector-starrocks-xxx.jar` to `flink-xxx/lib/`. 5. Download [smt.tar.gz](https://cdn-thirdparty.starrocks.com/smt.tar.gz?r=2). 6. Extract and modify the configuration file of SMT. ```bash [db] host = 192.168.1.1 port = 1521 user = xxx password = xxx type = oracle [other] # number of backends in StarRocks be_num = 3 # `decimal_v3` is supported since StarRocks-1.18.1 use_decimal_v3 = false # directory to save the converted DDL SQL output_dir = ./result [table-rule.1] # pattern to match databases for setting properties database = ^db$ # pattern to match tables for setting properties table = ^table$ # pattern to match schemas for setting properties schema = ^.*$ ############################################ ### flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`, they are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.strip_outer_array=true ``` 7. Execute **starrocks-migrate-tool**. All SQL scripts are generated in the result directory. ```bash $./starrocks-migrate-tool $ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql ``` 8. Use a SQL script whose prefix is starrocks-create to generate the table in StarRocks. ```bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ``` 9. Use a SQL script whose prefix is flink-create to generate the Flink source and sink tables, and start a Flink job to synchronize data. ```bash bin/sql-client.sh embedded < flink-create.all.sql ``` After the above command is successfully executed, the Flink job to synchronize data keeps running. 10. Observe the status of the Flink job. ```bash bin/flink list ``` If the job execution encounters an error, you can view detailed error information in the Flink logs. Also, you can modify the Flink configurations in the file **conf/flink-conf.yaml**, such as memory and slot. ##### Notes[​](#notes-2 "Direct link to Notes") * Synchronize Oracle using logminer: ```sql # Enable logging alter system set db_recovery_file_dest = '/home/oracle/data' scope=spfile; alter system set db_recovery_file_dest_size = 10G; shutdown immediate; startup mount; alter database archivelog; alter database open; ALTER TABLE schema_name.table_name ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; ALTER DATABASE ADD SUPPLEMENTAL LOG DATA; # Authorize user creation and grant permissions GRANT CREATE SESSION TO flinkuser; GRANT SET CONTAINER TO flinkuser; GRANT SELECT ON V_$DATABASE TO flinkuser; GRANT FLASHBACK ANY TABLE TO flinkuser; GRANT SELECT ANY TABLE TO flinkuser; GRANT SELECT_CATALOG_ROLE TO flinkuser; GRANT EXECUTE_CATALOG_ROLE TO flinkuser; GRANT SELECT ANY TRANSACTION TO flinkuser; GRANT LOGMINING TO flinkuser; GRANT CREATE TABLE TO flinkuser; GRANT LOCK ANY TABLE TO flinkuser; GRANT ALTER ANY TABLE TO flinkuser; GRANT CREATE SEQUENCE TO flinkuser; GRANT EXECUTE ON DBMS_LOGMNR TO flinkuser; GRANT EXECUTE ON DBMS_LOGMNR_D TO flinkuser; GRANT SELECT ON V_$LOG TO flinkuser; GRANT SELECT ON V_$LOG_HISTORY TO flinkuser; GRANT SELECT ON V_$LOGMNR_LOGS TO flinkuser; GRANT SELECT ON V_$LOGMNR_CONTENTS TO flinkuser; GRANT SELECT ON V_$LOGMNR_PARAMETERS TO flinkuser; GRANT SELECT ON V_$LOGFILE TO flinkuser; GRANT SELECT ON V_$ARCHIVED_LOG TO flinkuser; GRANT SELECT ON V_$ARCHIVE_DEST_STATUS TO flinkuser; ``` * The database configurations in the \[table-rule.1] does not support regular expression, so the complete database names need to be specified. * Because Oracle12c supports CDB mode, SMT internally automatically determines whether CDB is enabled and modifies the flink-cdc configuration correspondingly. However, users need to pay attention to whether the c## prefix needs to be added to the configuration of `[db].user` to avoid issues of insufficient permission. #### Synchronize Hive to StarRocks[​](#synchronize-hive-to-starrocks "Direct link to Synchronize Hive to StarRocks") ##### Introduction[​](#introduction-3 "Direct link to Introduction") This guide explains how to use SMT to synchronize Hive data to StarRocks. During synchronization, a Duplicate table in StarRocks is created and a Flink job keeps running to synchronize data. ##### Steps[​](#steps-3 "Direct link to Steps") ###### Preparations[​](#preparations "Direct link to Preparations") ```sql [db] # hiveserver2 service ip host = 127.0.0.1 # hiveserver2 service port port = 10000 user = hive/emr-header-1.cluster-49148 password = type = hive # only takes effect with `type = hive`. # Available values: kerberos, none, nosasl, kerberos_http, none_http, zk, ldap authentication = kerberos ``` The supported authentication methods are as follows: * nosasl, zk: do not need to specify the `user` and `password`. * none, none\_http, ldap: specify the `user` and `password`. * kerberos, kerberos\_http: perform the following steps: * Execute `kadmin.local` on the Hive cluster and check `list_principals` to find the corresponding principal name. For example when the principal name is `hive/emr-header-1.cluster-49148@EMR.49148.COM`, the user needs to be set as `hive/emr-header-1.cluster-49148`, and the password is left empty. * Execute `kinit -kt /path/to/keytab principal` on the machine where SMT is executed and execute `klist` to see if the correct token is generated. ###### Data synchronization[​](#data-synchronization "Direct link to Data synchronization") 1. Execute **starrocks-migrate-tool**. 2. Use a SQL script whose prefix is **starrocks-create** to generate the table in StarRocks. ```bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ``` 3. In **flink/conf/**, create and edit the file **sql-client-defaults.yaml**: ```yaml execution: planner: blink type: batch current-catalog: hive-starrocks catalogs: - name: hive-starrocks type: hive hive-conf-dir: /path/to/apache-hive-xxxx-bin/conf ``` 4. Download the [dependency package](https://nightlies.apache.org/flink/flink-docs-release-1.13/zh/docs/connectors/table/hive/overview/) (flink-sql-connector-hive-xxxx) from the Hive page of the corresponding version of Flink and place it in the `flink/lib` directory. 5. Start the Flink cluster and execute `flink/bin/sql-client.sh embedded < result/flink-create.all.sql` to start data synchronization. #### Synchronize SQL Server to StarRocks[​](#synchronize-sql-server-to-starrocks "Direct link to Synchronize SQL Server to StarRocks") ##### Introduction[​](#introduction-4 "Direct link to Introduction") Flink CDC connector and SMT can synchronize data from SQL Server within subsecond. SMT can automatically generate CREATE TABLE statements of the Flink's source and sink tables based on the cluster information and table structure of SQL Server and StarRocks. Flink CDC connector captures and records row-level changes that occur in the SQL Server database server. The principle is to use the CDC feature provided by SQL Server itself. The CDC capability of SQL Server itself can archive specified changes in the database to specified change tables. The SQL Server CDC connector first reads historical data from the table using JDBC, and then fetches incremental changes from the change tables, thereby achieving full incremental synchronization. Then, Flink-connector-starrocks writes data to StarRocks. ##### Steps[​](#steps-4 "Direct link to Steps") 1. Download [Flink](https://flink.apache.org/downloads/). The version of Flink is supported to be 1.11 or later. 2. Download [Flink CDC connector](https://github.com/ververica/flink-cdc-connectors/releases). Make sure that you download the **flink-sql-connector-sqlserver-cdc-xxx.jar** corresponding to the Flink version. 3. Download [Flink StarRocks connector](https://github.com/StarRocks/flink-connector-starrocks). 4. Copy **flink-sql-connector-sqlserver-cdc-xxx.jar**, **flink-connector-starrocks-xxx.jar** to **flink-xxx/lib/**. 5. Download [smt.tar.gz](https://cdn-thirdparty.starrocks.com/smt.tar.gz?r=2). 6. Extract and modify the configuration file of SMT. ```bash [db] host = 127.0.0.1 port = 1433 user = xxx password = xxx # currently available types: `mysql`, `pgsql`, `oracle`, `hive`, `clickhouse` type = sqlserver [other] # number of backends in StarRocks be_num = 3 # `decimal_v3` is supported since StarRocks-1.18.1 use_decimal_v3 = false # directory to save the converted DDL SQL output_dir = ./result [table-rule.1] # pattern to match databases for setting properties database = ^db$ # pattern to match tables for setting properties table = ^table$ schema = ^.*$ ############################################ ### flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`, they are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.strip_outer_array=true ``` 7. Execute **starrocks-migrate-tool**. All SQL scripts are generated in the `result` directory. ​ \`\`\`Bash $./starrocks-migrate-tool $ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql starrocks-external-create.all.sql ````text 8. Use a SQL script whose prefix is `starrocks-create` to generate the table in StarRocks. ```Bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ```` 9. Use a SQL script whose prefix is `flink-create` to generate the Flink source and sink tables, and start a Flink job to synchronize data. ```bash bin/sql-client.sh embedded < flink-create.all.sql ``` After the above command is successfully executed, the Flink job to synchronize data keeps running. 10. Observe the status of the Flink job. ```bash bin/flink list ``` If the job execution encounters an error, you can view detailed error information in the Flink logs. Also, you can modify the Flink configurations in the file **conf/flink-conf.yaml**, such as memory and slot. ##### Notes[​](#notes-3 "Direct link to Notes") 1. Make sure that the Server Agent Service is enabled. Check if the Server Agent Service is running normally. ```sql EXEC master.dbo.xp_servicecontrol N'QUERYSTATE', N'SQLSERVERAGENT' GO ``` Enable the Server Agent Service. ```bash /opt/mssql/bin/mssql-conf set sqlagent.enabled true ``` 2. Make sure that CDC for the corresponding database is enabled. ​ Check if CDC for the corresponding database is enabled. ```sql select is_cdc_enabled, name from sys.databases where name = 'XXX_databases' GO ``` ​ Enable CDC. ​ :::note ​ When executing this command, make sure that the user `serverRole` is `sysadmin`. ​ ::: ```sql USE XXX_databases GO EXEC sys.sp_cdc_enable_db GO ``` 3. Make sure that CDC for the corresponding table is enabled. ```sql EXEC sys.sp_cdc_enable_table @source_schema = 'XXX_schema', @source_name = 'XXX_table', @role_name = NULL, @supports_net_changes = 0; GO ``` #### Synchronize TiDB to StarRocks[​](#synchronize-tidb-to-starrocks "Direct link to Synchronize TiDB to StarRocks") ##### Introduction[​](#introduction-5 "Direct link to Introduction") Flink CDC connector and SMT can synchronize data from TiDB within subsecond. SMT can automatically generate DDL statements for the Flink's source tables and sink tables based on the cluster information and table structure of TiDB and StarRocks. The Flink CDC connector captures data by directly reading full and incremental data from the underlying TiKV storage. The full data is obtained from ranges partitioned based on keys, and the incremental data is obtained by using the CDC Client provided by TiDB. Subsequently, the data is written to StarRocks through Flink-connector-starrocks. ##### Steps[​](#steps-5 "Direct link to Steps") 1. Download [Flink](https://flink.apache.org/downloads/). The version of Flink is supported to be 1.11 or later. 2. Download [Flink CDC connector](https://github.com/ververica/flink-cdc-connectors/releases). Make sure that you download the **flink-sql-connector-tidb-cdc-xxx.jar** corresponding to the Flink version. 3. Download [Flink StarRocks connector](https://github.com/StarRocks/flink-connector-starrocks). 4. Copy **flink-sql-connector-tidb-cdc-xxx.jar**, **flink-connector-starrocks-xxx.jar** to **flink-xxx/lib/**. 5. Download [smt.tar.gz](https://cdn-thirdparty.starrocks.com/smt.tar.gz?r=2). 6. Extract and modify the configuration file of SMT. ```bash [db] host = 127.0.0.1 port = 4000 user = root password = # currently available types: `mysql`, `pgsql`, `oracle`, `hive`, `clickhouse`, `sqlserver`, `tidb` type = tidb # # only takes effect on `type == hive`. # # Available values: kerberos, none, nosasl, kerberos_http, none_http, zk, ldap # authentication = kerberos [other] # number of backends in StarRocks be_num = 3 # `decimal_v3` is supported since StarRocks-1.18.1 use_decimal_v3 = false # directory to save the converted DDL SQL output_dir = ./result [table-rule.1] # pattern to match databases for setting properties database = ^db$ # pattern to match tables for setting properties table = ^table$ schema = ^.*$ ############################################ ### flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`, they are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.strip_outer_array=true ############################################ ### flink-cdc configuration for `tidb` ############################################ # # Only takes effect on TiDB before v4.0.0. # # TiKV cluster's PD address. # flink.cdc.pd-addresses = 127.0.0.1:2379 ``` 7. Execute *starrocks-migrate-tool*\*. All SQL scripts are generated in the `result` directory. ```bash $./starrocks-migrate-tool $ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql starrocks-external-create.all.sql ``` 8. Use a SQL script whose prefix is `starrocks-create` to generate the table in StarRocks. ```bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ``` 9. Use a SQL script whose prefix is `flink-create` to generate the Flink source and sink tables, and start a Flink job to synchronize data. ```bash bin/sql-client.sh embedded < flink-create.all.sql ``` ​ After the above command is successfully executed, the Flink job to synchronize data keeps running. 10. Observe the status of the Flink job. ```bash bin/flink list ``` If the job execution encounters an error, you can view detailed error information in the Flink logs. Also, you can modify the Flink configurations in the file **conf/flink-conf.yaml**, such as memory and slot. ##### Notes[​](#notes-4 "Direct link to Notes") For TiDB whose version is before v4.0.0, additional configuration of `flink.cdc.pd-addresses` is required. ```plain ############################################ ### flink-cdc configuration for `tidb` ############################################ # # Only takes effect on TiDB before v4.0.0. # # TiKV cluster's PD address. # flink.cdc.pd-addresses = 127.0.0.1:2379 ``` --- ### Schema Management and Migration with SQLAlchemy and Alembic This guide introduces how to manage StarRocks schemas using the Python ecosystem — including SQLAlchemy, Alembic, and sqlacodegen — through the **`starrocks` SQLAlchemy** dialect. It is designed to help you understand **why schema migration is useful** and **how to use it effectively with StarRocks**. #### Overview[​](#overview "Direct link to Overview") Many users manage StarRocks tables, views, and materialized views using SQL DDL directly. However, as projects grow, manually maintaining `ALTER TABLE` statements becomes error-prone and hard to track. The **StarRocks SQLAlchemy dialect (`starrocks`)** provides: * A full SQLAlchemy model layer for StarRocks **tables**, **views**, and **materialized views** * **Declarative** definitions for table schema and table properties (including views and materialized views) * Integration with **Alembic** to allow schema changes to be **detected** and **generated** automatically * Compatibility with tools like **sqlacodegen** for reverse-generating models This allows Python users to maintain StarRocks schemas in a **declarative**, **version-controlled**, and **automated** way. #### Key benefits[​](#key-benefits "Direct link to Key benefits") Although schema migration is traditionally associated with OLTP databases, it is also valuable in data warehousing systems such as StarRocks. Teams use [Alembic](https://alembic.sqlalchemy.org/) together with the StarRocks dialect because of the benefits listed below. ##### Declarative schema definition[​](#declarative-schema-definition "Direct link to Declarative schema definition") Once you define schema in Python [ORM](https://docs.sqlalchemy.org/en/20/orm/quickstart.html#orm-quickstart) models or [SQLAlchemy](https://docs.sqlalchemy.org) core style, writing `ALTER TABLE` statements manually is no longer required. ##### Automatic diffing and autogeneration[​](#automatic-diffing-and-autogeneration "Direct link to Automatic diffing and autogeneration") Alembic compares **current StarRocks schema** with **your SQLAlchemy models**, and generates migration scripts automatically (`CREATE`/`DROP`/`ALTER`). ##### Reviewable, version-controlled migrations[​](#reviewable-version-controlled-migrations "Direct link to Reviewable, version-controlled migrations") Each schema change becomes a migration file (Python), so users can track changes and roll back if needed. ##### Consistent workflow across environments[​](#consistent-workflow-across-environments "Direct link to Consistent workflow across environments") Schema changes can be applied to development, staging, and production with the same process. #### Installation and Connection[​](#installation-and-connection "Direct link to Installation and Connection") ##### Prerequisites\*\*[​](#prerequisites "Direct link to Prerequisites**") * StarRocks Python client: 1.3.2 or later * `SQLAlchemy`: 1.4 or later (SQLAlchemy 2.0 is recommended and is required to use `sqlacodegen`) * `Alembic`: 1.16 or later ##### Installing StarRocks Python client[​](#installing-starrocks-python-client "Direct link to Installing StarRocks Python client") Run the following command to install the StarRocks Python client. ```bash pip install starrocks ``` ##### Connecting to StarRocks[​](#connecting-to-starrocks "Direct link to Connecting to StarRocks") Connect to your StarRocks cluster using the following URL. ```bash starrocks://:@:/[.] ``` * `user`: Username used to connect to the cluster. * `password`: User password. * `FE_host`: FE IP address. * `query_port`: FE `query_port` (Default: 9030). * `catalog`: The name of the catalog where your database locates. * `database`: The name of the database you want to connect. After installation, you can quickly validate connectivity using the following code example: ```python from sqlalchemy import create_engine, text # you need to create `mydatabase` first engine = create_engine("starrocks://root@localhost:9030/mydatabase") with engine.connect() as conn: conn.execute(text("SELECT 1")).fetchall() print("Connection successful!") ``` #### Defining StarRocks Models (Declarative ORM)[​](#defining-starrocks-models-declarative-orm "Direct link to Defining StarRocks Models (Declarative ORM)") The StarRocks dialect supports: * Tables * Views * Materialized Views It also supports StarRocks-specific table attributes such as: * `ENGINE` (OLAP) * Key models (`DUPLICATE KEY`, `PRIMARY KEY`, `UNIQUE KEY`, `AGGREGATE KEY`) * `PARTITION BY` variants (RANGE / LIST / Expression partitioning) * `DISTRIBUTED BY` variants (HASH / RANDOM) * `ORDER BY` * Table properties (e.g., `replication_num`, `storage_medium`) important * StarRocks dialect options are passed as keyword arguments prefixed with `starrocks_`. * The `starrocks_` **prefix must be lowercase**. The suffix is accepted in either case (for example, `PRIMARY_KEY` and `primary_key`). * If you specify a table key (e.g. `starrocks_primary_key="id"`), the involved columns **must** also be marked with `primary_key=True` in `Column(...)`, so that SQLAlchemy metadata and Alembic autogenerate can behave correctly. Examples below reflect the real public API and parameter names. ##### Table Example[​](#table-example "Direct link to Table Example") StarRocks table options can be specified in both ORM (via `__table_args__`) and Core (via `Table(..., starrocks_...=...)`) styles. ###### ORM (Declarative) style[​](#orm-declarative-style "Direct link to ORM (Declarative) style") ```python from sqlalchemy import create_engine from sqlalchemy.orm import Mapped, declarative_base, mapped_column from starrocks import INTEGER, STRING # with the same engine as the quick test engine = create_engine("starrocks://root@localhost:9030/mydatabase") Base = declarative_base() class MyTable(Base): __tablename__ = 'my_orm_table' id: Mapped[int] = mapped_column(INTEGER, primary_key=True) name: Mapped[str] = mapped_column(STRING) __table_args__ = { 'comment': 'table comment', 'starrocks_primary_key': 'id', 'starrocks_distributed_by': 'HASH(id) BUCKETS 10', 'starrocks_properties': {'replication_num': '1'} } # Create the table in the database Base.metadata.create_all(engine) ``` ###### Core style[​](#core-style "Direct link to Core style") ```python from sqlalchemy import Column, MetaData, Table, create_engine from starrocks import INTEGER, VARCHAR # with the same engine as the quick test engine = create_engine("starrocks://root@localhost:9030/mydatabase") metadata = MetaData() my_core_table = Table( 'my_core_table', metadata, Column('id', INTEGER, primary_key=True), Column('name', VARCHAR(50)), # StarRocks-specific arguments starrocks_primary_key='id', starrocks_distributed_by='HASH(id) BUCKETS 10', starrocks_properties={"replication_num": "1"} ) # Create the table in the database metadata.create_all(engine) ``` note For a comprehensive reference of table attributes and data types, see [Reference \[4\]](#references). ##### View Example[​](#view-example "Direct link to View Example") Below is the recommended view definition style, using `columns` as a list of dicts (`name`/`comment`). This example is based on an existing table `my_core_table`. ```python from starrocks.schema import View # Reuse the metadata from the Core table example above metadata = my_core_table.metadata user_view = View( "user_view", metadata, definition="SELECT id, name FROM my_core_table WHERE name IS NOT NULL", columns=[ {"name": "id", "comment": "ID"}, {"name": "name", "comment": "Name"}, ], comment="Active users", ) ``` note For more View options and limitations, see [Reference \[5\]](#references). ##### Materialized View Example[​](#materialized-view-example "Direct link to Materialized View Example") Materialized views are defined similarly. The `starrocks_refresh` property is a syntax string that indicates the refresh strategy. ```python from starrocks.schema import MaterializedView # Reuse the metadata from the Core table example above metadata = my_core_table.metadata # Create a simple Materialized View (asynchronous refresh) user_stats_ = MaterializedView( 'user_stats_', metadata, definition='SELECT id, COUNT(*) AS cnt FROM my_core_table GROUP BY id', starrocks_refresh='ASYNC' ) ``` note For more options and ALTER limitations, see [Reference \[6\]](#references). #### Alembic Integration[​](#alembic-integration "Direct link to Alembic Integration") The StarRocks SQLAlchemy dialect provides full support for: * Create / Drop table * Create / Drop view * Create / Drop materialized view * Detecting supported changes on StarRocks-specific attributes (for example, table properties and distribution) This enables Alembic’s **autogenerate** to work properly. ##### Initializing Alembic[​](#initializing-alembic "Direct link to Initializing Alembic") 1. Initialize Alembic: ```bash alembic init migrations ``` 2. Configure your database URL in `alembic.ini`: ```ini # alembic.ini sqlalchemy.url = starrocks://:@:/[.] ``` 3. Enable StarRocks dialect logging (optional): You can enable the `starrocks` logger in `alembic.ini` to observe the detected changes of a table via logs. For details, see [Reference \[2\]](#references). Edit `env.py` (configure both offline and online paths): ```python from alembic import context from starrocks.alembic import render_column_type, include_object_for_view_ from starrocks.alembic.starrocks import StarRocksImpl # noqa: F401 (ensure impl registered) from myapp.models import Base # adjust to your project target_metadata = Base.metadata def run_migrations_offline() -> None: url = context.config.get_main_option("sqlalchemy.url") context.configure( url=url, target_metadata=target_metadata, render_item=render_column_type, include_object=include_object_for_view_ ) with context.begin_transaction(): context.run_migrations() def run_migrations_online() -> None: # ... create engine and connect as in alembic default env.py ... with connectable.connect() as connection: context.configure( connection=connection, target_metadata=target_metadata, render_item=render_column_type, include_object=include_object_for_view_ ) with context.begin_transaction(): context.run_migrations() ``` ##### Generating migrations automatically[​](#generating-migrations-automatically "Direct link to Generating migrations automatically") ```bash alembic revision --autogenerate -m "initial schema" ``` Alembic will compare SQLAlchemy models with the actual StarRocks schema, and output the correct DDL. ##### Applying migrations[​](#applying-migrations "Direct link to Applying migrations") ```bash alembic upgrade head ``` Downgrade is also supported (where reversible). important StarRocks DDL is not transactional across multiple statements. If an upgrade fails midway, you may need to inspect what has already been applied and **perform manual remediation** (for example, write a compensating migration or run manual DDL) before re-running. #### Supported Schema Change Operations[​](#supported-schema-change-operations "Direct link to Supported Schema Change Operations") The dialect supports Alembic autogenerate for: * **Tables**: create / drop, and diffing of StarRocks-specific attributes declared via `starrocks_*` (within StarRocks ALTER support) * **Views**: create / drop / alter (mainly definition-related changes; some attributes are immutable) * **Materialized Views**: create / drop / alter (limited to mutable clauses such as refresh strategies and properties) Some StarRocks DDL changes are not reversible or not alterable. You can only make these changes by dropping and recreate the table/view/materialized view. If you specify these changes in the dialect, autogenerate will **warn or raise**. #### End-to-End Example (Recommended Reading for Beginners)[​](#end-to-end-example-recommended-reading-for-beginners "Direct link to End-to-End Example (Recommended Reading for Beginners)") This section shows a runnable end-to-end workflow, including where to pause and review generated files. ##### Step 1. Create a project directory and initialize Alembic[​](#step-1-create-a-project-directory-and-initialize-alembic "Direct link to Step 1. Create a project directory and initialize Alembic") ```bash mkdir my_sr_alembic_project cd my_sr_alembic_project alembic init alembic ``` ##### Step 2. Configure `alembic.ini`[​](#step-2-configure-alembicini "Direct link to step-2-configure-alembicini") Edit the URL in `alembic.ini`: ```ini sqlalchemy.url = starrocks://root@localhost:9030/mydatabase ``` ##### Step 3. Define your models[​](#step-3-define-your-models "Direct link to Step 3. Define your models") Create a package for your models: ```bash mkdir -p myapp touch myapp/__init__.py ``` Create `myapp/models.py` and put your table/view/materialized view definitions in the package: note When using Alembic migrations, do not call `metadata.create_all(engine)` in your models module. ```python from sqlalchemy import Column, Table from sqlalchemy.orm import Mapped, declarative_base, mapped_column from starrocks import INTEGER, STRING, VARCHAR from starrocks.schema import MaterializedView, View Base = declarative_base() # --- ORM table --- class MyOrmTable(Base): __tablename__ = "my_orm_table" id: Mapped[int] = mapped_column(INTEGER, primary_key=True) name: Mapped[str] = mapped_column(STRING) __table_args__ = { "comment": "table comment", "starrocks_primary_key": "id", "starrocks_distributed_by": "HASH(id) BUCKETS 10", "starrocks_properties": {"replication_num": "1"}, } # --- Core table on the same metadata (important for Alembic target_metadata) --- my_core_table = Table( "my_core_table", Base.metadata, Column("id", INTEGER, primary_key=True), Column("name", VARCHAR(50)), comment="core table comment", starrocks_primary_key="id", starrocks_distributed_by="HASH(id) BUCKETS 10", starrocks_properties={"replication_num": "1"}, ) # --- View --- user_view = View( "user_view", Base.metadata, definition="SELECT id, name FROM my_core_table WHERE name IS NOT NULL", columns=[ {"name": "id", "comment": "ID"}, {"name": "name", "comment": "Name"}, ], comment="Active users", ) # --- Materialized View --- user_stats_mv = MaterializedView( "user_stats_mv", Base.metadata, definition="SELECT id, COUNT(*) AS cnt FROM my_core_table GROUP BY id", starrocks_refresh="ASYNC", ) ``` ##### Step 4. Configure `env.py` for autogenerate[​](#step-4-configure-envpy-for-autogenerate "Direct link to step-4-configure-envpy-for-autogenerate") Edit `alembic/env.py`: 1. Import `myapp.models` to set the `target_metadata`. 2. Import `render_column_type`, and `include_object_for_view_mv` to set them in both `run_migrations_offline()` and `run_migrations_online()` to properly handle views and MVs, and to properly render StarRocks column types. note You need to add or modify these lines in `env.py`, rather than replace the generated `env.py` file. ```python from alembic import context from starrocks.alembic import render_column_type, include_object_for_view_mv from starrocks.alembic.starrocks import StarRocksImpl # noqa: F401 from myapp.models import Base target_metadata = Base.metadata # Optional: set version table replication for single-BE dev clusters version_table_kwargs = {"starrocks_properties": {"replication_num": "1"}} # In both run_migrations_offline() and run_migrations_online(), ensure: def run_migrations_offline() -> None: url = context.config.get_main_option("sqlalchemy.url") context.configure( url=url, target_metadata=target_metadata, literal_binds=True, render_item=render_column_type, include_object=include_object_for_view_mv, version_table_kwargs=version_table_kwargs, ) def run_migrations_online() -> None: # ... create engine and connect as in alembic default env.py ... with connectable.connect() as connection: context.configure( connection=connection, target_metadata=target_metadata, render_item=render_column_type, include_object=include_object_for_view_mv, version_table_kwargs=version_table_kwargs, ) ``` ##### Step 5. Autogenerate the first revision[​](#step-5-autogenerate-the-first-revision "Direct link to Step 5. Autogenerate the first revision") ```bash alembic revision --autogenerate -m "create initial schema" ``` Pause and review: 1. Check the generated migration file under `alembic/versions/`. 2. Ensure it contains the expected operations (for example, `create_table`, `create_view`, `create_materialized_view`). 3. Make sure it does not contain unexpected drops or alters. ##### Step 6. Preview SQL and apply[​](#step-6-preview-sql-and-apply "Direct link to Step 6. Preview SQL and apply") Preview SQL: ```bash alembic upgrade head --sql ``` Pause and review: 1. Confirm the DDL is in the order you expect. 2. Identify any potentially heavy operations and consider splitting migrations if needed. Apply: ```bash alembic upgrade head ``` important StarRocks DDL is not transactional across multiple statements. If an upgrade fails midway, you may need to inspect what has already been applied and perform manual remediation before re-running. ##### Step 7. Make a change and autogenerate again[​](#step-7-make-a-change-and-autogenerate-again "Direct link to Step 7. Make a change and autogenerate again") Update `myapp/models.py` to: * **Modify an existing table** (`my_core_table`): add a column, or update the table comment, and change one table property. * **Add a new table** (`my_new_table`). note Adding a column can be a time-consuming schema change. StarRocks allows only **one running schema change job per table** at a time. In practice, it is recommended to keep “add/drop/modify columns” changes separate from other heavy changes (for example, additional add/drop columns or mass property changes), and split them into multiple Alembic revisions if needed. ```python from sqlalchemy import Column, Table from starrocks import INTEGER, VARCHAR # Modify an existing table (add a column) # (Update the existing my_core_table definition in-place.) my_core_table = Table( "my_core_table", Base.metadata, Column("id", INTEGER, primary_key=True), Column("name", VARCHAR(50)), Column("age", INTEGER), # added column only starrocks_primary_key='id', starrocks_distributed_by='HASH(id) BUCKETS 10', starrocks_properties={"replication_num": "1"}, ) my_new_table = Table( "my_new_table", Base.metadata, Column("id", INTEGER, primary_key=True), Column("name", VARCHAR(50)), starrocks_primary_key="id", starrocks_distributed_by="HASH(id) BUCKETS 10", starrocks_properties={"replication_num": "1"}, ) ``` ```bash alembic revision --autogenerate -m "add a new table, change a old table" ``` Pause and review: Make sure the new migration contains: * a `create_table(...)` for `my_new_table`, and * expected operations for the `my_core_table` changes (for example, add column / set comment / set properties). Preview SQL and apply: ```bash alembic upgrade head --sql alembic upgrade head ``` #### Using sqlacodegen[​](#using-sqlacodegen "Direct link to Using sqlacodegen") [`sqlacodegen`](https://github.com/agronholm/sqlacodegen) can reverse-generate SQLAlchemy models directly from StarRocks: ```bash sqlacodegen --options include_dialect_options,keep_dialect_types \ --generator tables \ starrocks://:@:/[catalog.] > models.py ``` Supported objects: * Tables * Views * Materialized views * Partitioning, distribution, and order-by clauses, and properties This is useful when onboarding an existing StarRocks schema into Alembic. You can directly use above command to generate the Python script for tables/views/materialized views defined in the **End-to-End Example** section. note * It is recommended to add `--generator tables` when generating Core-style models (ORM generators may reorder columns according to `NOT NULL` / `NULL` attribute). * Key columns may be generated as `NOT NULL`. If you want them nullable, adjust the generated model manually. #### Limitations and Best Practices[​](#limitations-and-best-practices "Direct link to Limitations and Best Practices") * Some StarRocks DDL operations require dropping and recreating the table; autogenerate will warn or raise rather than silently producing unavailable SQL. * Keys model changes (for example, changing DUPLICATE KEY to PRIMARY KEY) are not supported via `ALTER TABLE`; use an explicit plan (usually dropping and recreating with backfill). * StarRocks does not provide transactional DDL across multiple statements; review generated migrations and apply them operationally. If a migration fails midway, you may need to handle rollback **manually**. * For distribution, if you omit the `BUCKETS` clause, StarRocks may auto-assign bucket count; the dialect is designed to avoid noisy diffs in that case. #### Summary[​](#summary "Direct link to Summary") With the StarRocks SQLAlchemy dialect and Alembic integration, you can: * ✔ Use declarative models to define StarRocks schemas * ✔ Automatically detect and generate schema migration scripts * ✔ Use version control for schema evolution * ✔ Manage views and materialized views declaratively * ✔ Reverse-engineer existing schemas using sqlacodegen This brings StarRocks schema management into the modern Python data engineering ecosystem and significantly simplifies cross-environment schema consistency. #### References[​](#references "Direct link to References") \[1]: [starrocks-python-client README](https://github.com/StarRocks/starrocks/blob/main/contrib/starrocks-python-client/README.md) \[2]: [Alembic Integration](https://github.com/StarRocks/starrocks/blob/main/contrib/starrocks-python-client/docs/usage_guide/alembic.md) \[3]: [SQLAlchemy details](https://github.com/StarRocks/starrocks/blob/main/contrib/starrocks-python-client/docs/usage_guide/sqlalchemy.md) \[4]: [Table Support](https://github.com/StarRocks/starrocks/blob/main/contrib/starrocks-python-client/docs/usage_guide/tables.md) \[5]: [View Support](https://github.com/StarRocks/starrocks/blob/main/contrib/starrocks-python-client/docs/usage_guide/views.md) \[6]: [Materialized View Support](https://github.com/StarRocks/starrocks/blob/main/contrib/starrocks-python-client/docs/usage_guide/materialized_views.md) --- ### Apache Flink ### Continuously load data from Apache Flink® StarRocks provides a self-developed connector named StarRocks Connector for Apache Flink® (Flink connector for short) to help you load data into a StarRocks table by using Flink. The basic principle is to accumulate the data and then load it all at a time into StarRocks through [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). The Flink connector supports DataStream API, Table API & SQL, and Python API. It has a higher and more stable performance than [flink-connector-jdbc](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/jdbc/) provided by Apache Flink®. > **NOTICE** > > Loading data into StarRocks tables with Flink connector needs SELECT and INSERT privileges on the target StarRocks table. If you do not have these privileges, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant these privileges to the user that you use to connect to your StarRocks cluster. #### Version requirements[​](#version-requirements "Direct link to Version requirements") | Connector | Flink | StarRocks | Java | Scala | | --------- | ----------------------------- | ------------- | ---- | --------- | | 1.2.16 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.15 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.14 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.12 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.11 | 1.15,1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | #### Obtain Flink connector[​](#obtain-flink-connector "Direct link to Obtain Flink connector") You can obtain the Flink connector JAR file in the following ways: * Directly download the compiled Flink connector JAR file. * Add the Flink connector as a dependency in your Maven project and then download the JAR file. * Compile the source code of the Flink connector into a JAR file by yourself. The naming format of the Flink connector JAR file is as follows: * Since Flink 1.15, it's `flink-connector-starrocks-${connector_version}_flink-${flink_version}.jar`. For example, if you install Flink 1.15 and you want to use Flink connector 1.2.7, you can use `flink-connector-starrocks-1.2.7_flink-1.15.jar`. * Prior to Flink 1.15, it's `flink-connector-starrocks-${connector_version}_flink-${flink_version}_${scala_version}.jar`. For example, if you install Flink 1.14 and Scala 2.12 in your environment, and you want to use Flink connector 1.2.7, you can use `flink-connector-starrocks-1.2.7_flink-1.14_2.12.jar`. > **NOTICE** > > In general, the latest version of the Flink connector only maintains compatibility with the three most recent versions of Flink. ##### Download the compiled Jar file[​](#download-the-compiled-jar-file "Direct link to Download the compiled Jar file") Directly download the corresponding version of the Flink connector Jar file from the [Maven Central Repository](https://repo1.maven.org/maven2/com/starrocks). ##### Maven Dependency[​](#maven-dependency "Direct link to Maven Dependency") In your Maven project's `pom.xml` file, add the Flink connector as a dependency according to the following format. Replace `flink_version`, `scala_version`, and `connector_version` with the respective versions. * In Flink 1.15 and later ```xml com.starrocks flink-connector-starrocks ${connector_version}_flink-${flink_version} ``` * In versions earlier than Flink 1.15 ```xml com.starrocks flink-connector-starrocks ${connector_version}_flink-${flink_version}_${scala_version} ``` ##### Compile by yourself[​](#compile-by-yourself "Direct link to Compile by yourself") 1. Download the [Flink connector source code](https://github.com/StarRocks/starrocks-connector-for-apache-flink). 2. Execute the following command to compile the source code of Flink connector into a JAR file. Note that `flink_version` is replaced with the corresponding Flink version. ```bash sh build.sh ``` For example, if the Flink version in your environment is 1.16, you need to execute the following command: ```bash sh build.sh 1.16 ``` 3. Go to the `target/` directory to find the Flink connector JAR file, such as `flink-connector-starrocks-1.2.7_flink-1.16-SNAPSHOT.jar`, generated upon compilation. > **NOTE** > > The name of Flink connector which is not formally released contains the `SNAPSHOT` suffix. #### Connect to StarRocks[​](#connect-to-starrocks "Direct link to Connect to StarRocks") Set the StarRocks connection options to addresses of the FE: * `jdbc-url`: `jdbc:mysql://:`. The query port defaults to `9030`. * `load-url`: `:`. The HTTP port defaults to `8030`. For ``, use an address that stays the same when an FE restarts or is replaced, such as a DNS name or a load balancer in front of the FEs, not the IP address of an FE. If the address stops leading to an FE, the Flink job can no longer load data. The Flink TaskManagers must be able to reach these FE ports, and also the HTTP port (default `8040`) of every BE or CN, because the FE redirects each load request to a BE or CN. #### Options[​](#options "Direct link to Options") ##### General Options[​](#general-options "Direct link to General Options") ###### connector[​](#connector "Direct link to connector") * **Required**: Yes * **Default value**: NONE * **Description**: The connector that you want to use. The value must be "starrocks". ###### jdbc-url[​](#jdbc-url "Direct link to jdbc-url") * **Required**: Yes * **Default value**: NONE * **Description**: The address that is used to connect to the MySQL server of the FE. You can specify multiple addresses, which must be separated by a comma (,). Format: `jdbc:mysql://:,:,:`. ###### load-url[​](#load-url "Direct link to load-url") * **Required**: Yes * **Default value**: NONE * **Description**: The address that is used to connect to the HTTP server of the FE. You can specify multiple addresses, which must be separated by a semicolon (;). Format: `:;:`. ###### database-name[​](#database-name "Direct link to database-name") * **Required**: Yes * **Default value**: NONE * **Description**: The name of the StarRocks database into which you want to load data. ###### table-name[​](#table-name "Direct link to table-name") * **Required**: Yes * **Default value**: NONE * **Description**: The name of the table that you want to use to load data into StarRocks. ###### username[​](#username "Direct link to username") * **Required**: Yes * **Default value**: NONE * **Description**: The username of the account that you want to use to load data into StarRocks. The account needs [SELECT and INSERT privileges](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) on the target StarRocks table. ###### password[​](#password "Direct link to password") * **Required**: Yes * **Default value**: NONE * **Description**: The password of the preceding account. ###### sink.version[​](#sinkversion "Direct link to sink.version") * **Required**: No * **Default value**: AUTO * **Description**: The interface used to load data. This parameter is supported from Flink connector version 1.2.4 onwards. Valid Values: * `V1`: Use [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) interface to load data. Connectors before 1.2.4 only support this mode. * `V2`: Use [Stream Load transaction](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) interface to load data. It requires StarRocks to be at least version 2.4. Recommends `V2` because it optimizes the memory usage and provides a more stable exactly-once implementation. * `AUTO`: If the version of StarRocks supports transaction Stream Load, will choose `V2` automatically, otherwise choose `V1` ###### sink.label-prefix[​](#sinklabel-prefix "Direct link to sink.label-prefix") * **Required**: No * **Default value**: NONE * **Description**: The label prefix used by Stream Load. Recommend to configure it if you are using exactly-once with connector 1.2.8 and later. See [exactly-once usage notes](#exactly-once). ###### sink.semantic[​](#sinksemantic "Direct link to sink.semantic") * **Required**: No * **Default value**: at-least-once * **Description**: The semantic guaranteed by sink. Valid values: **at-least-once** and **exactly-once**. ###### sink.buffer-flush.max-bytes[​](#sinkbuffer-flushmax-bytes "Direct link to sink.buffer-flush.max-bytes") * **Required**: No * **Default value**: 94371840(90M) * **Description**: The maximum size of data that can be accumulated in memory before being sent to StarRocks at a time. The maximum value ranges from 64 MB to 10 GB. Setting this parameter to a larger value can improve loading performance but may increase loading latency. This parameter only takes effect when `sink.semantic` is set to `at-least-once`. If `sink.semantic` is set to `exactly-once`, the data in memory is flushed when a Flink checkpoint is triggered. In this circumstance, this parameter does not take effect. ###### sink.buffer-flush.max-rows[​](#sinkbuffer-flushmax-rows "Direct link to sink.buffer-flush.max-rows") * **Required**: No * **Default value**: 500000 * **Description**: The maximum number of rows that can be accumulated in memory before being sent to StarRocks at a time. This parameter is available only when `sink.version` is `V1` and `sink.semantic` is `at-least-once`. Valid values: 64000 to 5000000. ###### sink.buffer-flush.interval-ms[​](#sinkbuffer-flushinterval-ms "Direct link to sink.buffer-flush.interval-ms") * **Required**: No * **Default value**: 300000 * **Description**: The interval at which data is flushed. This parameter is available only when `sink.semantic` is `at-least-once`. Unit: ms. Valid value range: * For versions earlier than v1.2.14: \[1000, 3600000] * For v1.2.14 and later: (0, 3600000]. ###### sink.max-retries[​](#sinkmax-retries "Direct link to sink.max-retries") * **Required**: No * **Default value**: 3 * **Description**: The number of times that the system retries to perform the Stream Load job. This parameter is available only when you set `sink.version` to `V1`. Valid values: 0 to 10. ###### sink.connect.timeout-ms[​](#sinkconnecttimeout-ms "Direct link to sink.connect.timeout-ms") * **Required**: No * **Default value**: 30000 * **Description**: The timeout for establishing HTTP connection. Valid values: 100 to 60000. Unit: ms. Before Flink connector v1.2.9, the default value is `1000`. ###### sink.socket.timeout-ms[​](#sinksockettimeout-ms "Direct link to sink.socket.timeout-ms") * **Required**: No * **Default value**: -1 * **Description**: Supported since 1.2.10. The time duration for which the HTTP client waits for data. Unit: ms. The default value `-1` means there is no timeout. ###### sink.sanitize-error-log[​](#sinksanitize-error-log "Direct link to sink.sanitize-error-log") * **Required**: No * **Default value**: false * **Description**: Supported since 1.2.12. Whether to sanitize sensitive data in the error log for production security. When this item is set to `true`, sensitive row data and column values in Stream Load error logs are redacted in both the connector and SDK logs. The value defaults to `false` for backward compatibility. ###### sink.wait-for-continue.timeout-ms[​](#sinkwait-for-continuetimeout-ms "Direct link to sink.wait-for-continue.timeout-ms") * **Required**: No * **Default value**: 10000 * **Description**: Supported since 1.2.7. The timeout for waiting response of HTTP 100-continue from the FE. Valid values: `3000` to `60000`. Unit: ms ###### sink.ignore.update-before[​](#sinkignoreupdate-before "Direct link to sink.ignore.update-before") * **Required**: No * **Default value**: true * **Description**: Supported since version 1.2.8. Whether to ignore `UPDATE_BEFORE` records from Flink when loading data to Primary Key tables. If this parameter is set to false, the record is treated as a delete operation to StarRocks table. ###### sink.json.columns-from-flink-schema[​](#sinkjsoncolumns-from-flink-schema "Direct link to sink.json.columns-from-flink-schema") * **Required**: No * **Default value**: false * **Description**: Supported since 1.2.16. Whether the Stream Load `columns` header is derived from the Flink table schema when the format is `json`. By default no header is sent for `json`, so the server declares every table column and a column the payload does not carry is stored as NULL even when it has a DEFAULT. When `true`, the Flink table's columns are sent as the header, so a column the Flink schema does not declare is left off and the server applies its DEFAULT. Requires a Flink schema and `sink.properties.format=json`, and cannot be combined with `sink.properties.columns` or `sink.properties.jsonpaths`. ###### sink.parallelism[​](#sinkparallelism "Direct link to sink.parallelism") * **Required**: No * **Default value**: NONE * **Description**: The parallelism of loading. Only available for Flink SQL. If this parameter is not specified, Flink planner decides the parallelism. **In the scenario of multi-parallelism, users need to guarantee data is written in the correct order.** ###### sink.properties.\*[​](#sinkproperties "Direct link to sink.properties.*") * **Required**: No * **Default value**: NONE * **Description**: The parameters that are used to control Stream Load behavior. For example, the parameter `sink.properties.format` specifies the format used for Stream Load, such as CSV or JSON. For a list of supported parameters and their descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ###### sink.properties.format[​](#sinkpropertiesformat "Direct link to sink.properties.format") * **Required**: No * **Default value**: csv * **Description**: The format used for Stream Load. The Flink connector will transform each batch of data to the format before sending them to StarRocks. Valid values: `csv` and `json`. ###### sink.properties.column\_separator[​](#sinkpropertiescolumn_separator "Direct link to sink.properties.column_separator") * **Required**: No * **Default value**: \t * **Description**: The column separator for CSV-formatted data. ###### sink.properties.row\_delimiter[​](#sinkpropertiesrow_delimiter "Direct link to sink.properties.row_delimiter") * **Required**: No * **Default value**: \n * **Description**: The row delimiter for CSV-formatted data. ###### sink.properties.max\_filter\_ratio[​](#sinkpropertiesmax_filter_ratio "Direct link to sink.properties.max_filter_ratio") * **Required**: No * **Default value**: 0 * **Description**: The maximum error tolerance of the Stream Load. It's the maximum percentage of data records that can be filtered out due to inadequate data quality. Valid values: `0` to `1`. Default value: `0`. See [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) for details. ###### sink.properties.partial\_update[​](#sinkpropertiespartial_update "Direct link to sink.properties.partial_update") * **Required**: NO * **Default value**: `FALSE` * **Description**: Whether to use partial updates. Valid values: `TRUE` and `FALSE`. Default value: `FALSE`, indicating to disable this feature. ###### sink.properties.partial\_update\_mode[​](#sinkpropertiespartial_update_mode "Direct link to sink.properties.partial_update_mode") * **Required**: NO * **Default value**: `row` * **Description**: Specifies the mode for partial updates. Valid values: `row` and `column`. * The value `row` (default) means partial updates in row mode, which is more suitable for real-time updates with many columns and small batches. * The value `column` means partial updates in column mode, which is more suitable for batch updates with few columns and many rows. In such scenarios, enabling the column mode offers faster update speeds. For example, in a table with 100 columns, if only 10 columns (10% of the total) are updated for all rows, the update speed of the column mode is 10 times faster. ###### sink.properties.strict\_mode[​](#sinkpropertiesstrict_mode "Direct link to sink.properties.strict_mode") * **Required**: No * **Default value**: false * **Description**: Specifies whether to enable the strict mode for Stream Load. It affects the loading behavior when there are unqualified rows, such as inconsistent column values. Valid values: `true` and `false`. Default value: `false`. See [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) for details. ###### sink.properties.compression[​](#sinkpropertiescompression "Direct link to sink.properties.compression") * **Required**: No * **Default value**: NONE * **Description**: The compression algorithm used for Stream Load. Valid values: `lz4_frame`. Compression for the JSON format requires Flink connector 1.2.10+ and StarRocks v3.2.7+. Compression for the CSV format only requires Flink connector 1.2.11+. ###### sink.properties.prepared\_timeout[​](#sinkpropertiesprepared_timeout "Direct link to sink.properties.prepared_timeout") * **Required**: No * **Default value**: NONE * **Description**: Supported since 1.2.12 and only effective when `sink.version` is set to `V2`. Requires StarRocks 3.5.4 or later. Sets the timeout in seconds for the Transaction Stream Load phase from `PREPARED` to `COMMITTED`. Typically, only needed for exactly-once; at-least-once usually does not require setting this (the connector defaults to 300s). If not set in exactly-once, StarRocks FE configuration `prepared_transaction_default_timeout_second` (default 86400s) applies. See [StarRocks Transaction timeout management](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md#transaction-timeout-management). ###### sink.publish-timeout.ms[​](#sinkpublish-timeoutms "Direct link to sink.publish-timeout.ms") * **Required**: No * **Default value**: -1 * **Description**: Supported since 1.2.14 and only effective when `sink.version` is set to `V2`. Timeout in milliseconds for the Publish phase. If a transaction stays in COMMITTED status longer than this timeout, the system will consider it as successful. The default value `-1` means using StarRocks server-side default behavior. When Merge Commit is enabled, the default timeout is 10000 ms. ##### Merge Commit options[​](#merge-commit-options "Direct link to Merge Commit options") Supported from v1.2.14 onwards. Merge Commit allows the system to merge data from multiple subtasks into a single Stream Load transaction for better performance. You can enable this feature by setting `sink.properties.enable_merge_commit` to `true`. For more details about the merge commit feature in StarRocks, see [Merge Commit parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#merge-commit-parameters). The following Stream Load properties are used to control the Merge Commit behavior: ###### sink.properties.enable\_merge\_commit[​](#sinkpropertiesenable_merge_commit "Direct link to sink.properties.enable_merge_commit") * **Required**: No * **Default value**: false * **Description**: Whether to enable Merge Commit. ###### sink.properties.merge\_commit\_interval\_ms[​](#sinkpropertiesmerge_commit_interval_ms "Direct link to sink.properties.merge_commit_interval_ms") * **Required**: Yes (when Merge Commit is enabled) * **Default value**: NONE * **Description**: The Merge Commit time window in milliseconds. The system merges loading requests received within this window into a single transaction. A larger value improves merging efficiency but increases latency. This property must be set when `enable_merge_commit` is set to `true`. ###### sink.properties.merge\_commit\_parallel[​](#sinkpropertiesmerge_commit_parallel "Direct link to sink.properties.merge_commit_parallel") * **Required**: No * **Default value**: 3 * **Description**: The degree of parallelism for the loading plan created for each Merge Commit-enabled transaction. It is different from `sink.parallelism` which controls the parallelism of the Flink sink operator. ###### sink.properties.merge\_commit\_async[​](#sinkpropertiesmerge_commit_async "Direct link to sink.properties.merge_commit_async") * **Required**: No * **Default value**: true * **Description**: The server's return mode for Merge Commit. The default value is `true` (asynchronous), overriding the system default behavior (synchronous) for better throughput. In the asynchronous mode, the server returns immediately after receiving the data. The connector leverages Flink's checkpoint mechanism to ensure no data loss under the asynchronous mode, providing at-least-once guarantee. In most cases, you do not need to change this value. ###### sink.merge-commit.max-concurrent-requests[​](#sinkmerge-commitmax-concurrent-requests "Direct link to sink.merge-commit.max-concurrent-requests") * **Required**: No * **Default value**: Integer.MAX\_VALUE * **Description**: The maximum number of concurrent Stream Load requests. Set this property to `0` to ensure in-order (serial) loading, which is useful for Primary Key tables. A negative value is treated as `Integer.MAX_VALUE` (unlimited concurrency). ###### sink.merge-commit.chunk.size[​](#sinkmerge-commitchunksize "Direct link to sink.merge-commit.chunk.size") * **Required**: No * **Default value**: 20971520 * **Description**: The maximum size of data (in bytes) accumulated in a chunk before it is flushed and sent to StarRocks via a Stream Load request. A larger value improves throughput but increases memory usage and latency; a smaller value reduces memory usage and latency but may lower throughput. When `max-concurrent-requests` is set to `0` (in-order mode), the default value of this property is changed to 500 MB because only one request runs at a time so a larger batch maximizes throughput. #### Data type mapping between Flink and StarRocks[​](#data-type-mapping-between-flink-and-starrocks "Direct link to Data type mapping between Flink and StarRocks") | Flink data type | StarRocks data type | | ------------------------------------- | ------------------- | | BOOLEAN | BOOLEAN | | TINYINT | TINYINT | | SMALLINT | SMALLINT | | INTEGER | INTEGER | | BIGINT | BIGINT | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | DECIMAL | DECIMAL | | BINARY | INT | | CHAR | STRING | | VARCHAR | STRING | | STRING | STRING | | DATE | DATE | | TIMESTAMP\_WITHOUT\_TIME\_ZONE(N) | DATETIME | | TIMESTAMP\_WITH\_LOCAL\_TIME\_ZONE(N) | DATETIME | | ARRAY`<`T`>` | ARRAY`<`T`>` | | MAP`<`KT,VT`>` | JSON STRING | | ROW`<`arg T...`>` | JSON STRING | #### Usage notes[​](#usage-notes "Direct link to Usage notes") ##### Exactly Once[​](#exactly-once "Direct link to Exactly Once") * If you want sink to guarantee exactly-once semantics, we recommend you to upgrade StarRocks to 2.5 or later, and Flink connector to 1.2.4 or later * Since Flink connector 1.2.4, the exactly-once is redesigned based on [Stream Load transaction interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) provided by StarRocks since 2.4. Compared to the previous implementation based on non-transactional Stream Load non-transactional interface, the new implementation reduces memory usage and checkpoint overhead, thereby enhancing real-time performance and stability of loading. * If the version of StarRocks is earlier than 2.4 or the version of Flink connector is earlier than 1.2.4, the sink will automatically choose the implementation based on Stream Load non-transactional interface. * Configurations to guarantee exactly-once * The value of `sink.semantic` needs to be `exactly-once`. * If the version of Flink connector is 1.2.8 and later, it is recommended to specify the value of `sink.label-prefix`. Note that the label prefix must be unique among all types of loading in StarRocks, such as Flink jobs, Routine Load, and Broker Load. * If the label prefix is specified, the Flink connector will use the label prefix to clean up lingering transactions that may be generated in some Flink failure scenarios, such as the Flink job fails when a checkpoint is still in progress. These lingering transactions are generally in `PREPARED` status if you use `SHOW PROC '/transactions//running';` to view them in StarRocks. When the Flink job restores from checkpoint, the Flink connector will find these lingering transactions according to the label prefix and some information in checkpoint, and abort them. The Flink connector can not abort them when the Flink job exits because of the two-phase-commit mechanism to implement the exactly-once. When the Flink job exits, the Flink connector has not received the notification from Flink checkpoint coordinator whether the transactions should be included in a successful checkpoint, and it may lead to data loss if these transactions are aborted anyway. You can have an overview about how to achieve end-to-end exactly-once in Flink in this [blogpost](https://flink.apache.org/2018/02/28/an-overview-of-end-to-end-exactly-once-processing-in-apache-flink-with-apache-kafka-too/). * If the label prefix is not specified, lingering transactions will be cleaned up by StarRocks only after they time out. However the number of running transactions can reach the limitation of StarRocks `max_running_txn_num_per_db` if Flink jobs fail frequently before transactions time out. You can set a smaller timeout for `PREPARED` transactions to make them expired faster when the label prefix is not specified. See the following about how to set the prepared timeout. * If you are certain that the Flink job will eventually recover from checkpoint or savepoint after a long downtime because of stop or continuous failover, please adjust the following StarRocks configurations accordingly, to avoid data loss. * Adjust `PREPARED` transaction timeout. See the following about how to set the timeout. The timeout needs to be larger than the downtime of the Flink job. Otherwise, the lingering transactions that are included in a successful checkpoint may be aborted because of timeout before you restart the Flink job, which leads to data loss. Note that when you set a larger value to this configuration, it is better to specify the value of `sink.label-prefix` so that the lingering transactions can be cleaned according to the label prefix and some information in checkpoint, instead of due to timeout (which may cause data loss). * `label_keep_max_second` and `label_keep_max_num`: StarRocks FE configurations, default values are `259200` and `1000` respectively. For details, see [FE configurations](https://docs.starrocks.io/docs/loading/loading_introduction/loading_considerations.md#fe-configurations). The value of `label_keep_max_second` needs to be larger than the downtime of the Flink job. Otherwise, the Flink connector can not check the state of transactions in StarRocks by using the transaction labels saved in the Flink's savepoint or checkpoint and figure out whether these transactions are committed or not, which may eventually lead to data loss. * How to set the timeout for PREPARED transactions * For Connector 1.2.12+ and StarRocks 3.5.4+, you can set the timeout by configuring the connector parameter `sink.properties.prepared_timeout`. By default, the value is not set, and it falls back to the StarRocks FE's global configuration `prepared_transaction_default_timeout_second` (default value is `86400`). * For other versions of Connector or StarRocks, you can set the timeout by configuring the StarRocks FE's global configuration `prepared_transaction_default_timeout_second` (default value is `86400`). ##### Flush Policy[​](#flush-policy "Direct link to Flush Policy") The Flink connector will buffer the data in memory, and flush them in batch to StarRocks via Stream Load. How the flush is triggered is different between at-least-once and exactly-once. For at-least-once, the flush will be triggered when any of the following conditions are met: * the bytes of buffered rows reaches the limit `sink.buffer-flush.max-bytes` * the number of buffered rows reaches the limit `sink.buffer-flush.max-rows`. (Only valid for sink version V1) * the elapsed time since the last flush reaches the limit `sink.buffer-flush.interval-ms` * a checkpoint is triggered For exactly-once, the flush only happens when a checkpoint is triggered. ##### Merge Commit[​](#merge-commit "Direct link to Merge Commit") Merge Commit helps scale throughput without proportionally increasing StarRocks transaction overhead. Without Merge Commit, each Flink sink subtask maintains its own Stream Load transaction, so increasing `sink.parallelism` leads to more concurrent transactions and higher I/O and Compaction costs on StarRocks. Conversely, keeping parallelism low limits the pipeline's overall capacity. With Merge Commit is enabled, data from multiple sink subtasks is merged into a single transaction within each Merge window. This allows you to increase `sink.parallelism` for higher throughput without increasing the number of transactions. For configuration examples, see [Load data with merge commit](#load-data-with-merge-commit). Below are some important notes when using Merge Commit: * **Single parallelism has no benefit** If the Flink sink parallelism is 1, enabling Merge Commit provides no benefit since there is only one subtask sending data. It may even introduce additional latency due to the Merge Commit time window on the server side. * **Only at-least-once semantic** Merge Commit only guarantees at-least-once semantic. It does not support exactly-once semantic. Do not set `sink.semantic` to `exactly-once` when Merge Commit is enabled. * **Ordering for Primary Key tables** By default, `sink.merge-commit.max-concurrent-requests` is `Integer.MAX_VALUE`, which means a single sink subtask may send multiple Stream Load requests concurrently. This can cause out-of-order loading, which may be problematic for Primary Key tables. To ensure in-order loading, set `sink.merge-commit.max-concurrent-requests` to `0`, which, however, will reduce throughput. Alternatively, you can use Conditional Update to prevent newer data from being overwritten by older data. For configuration examples, see [In-order loading for Primary Key tables](#in-order-loading-for-primary-key-tables). * **End-to-end loading latency** The total loading latency consists of two parts: * **Connector batching latency**: Controlled by `sink.buffer-flush.interval-ms` and `sink.merge-commit.chunk.size`. Data is flushed from the connector when either the chunk size limit is reached or the flush interval elapses, whichever comes first. The maximum connector-side latency is `sink.buffer-flush.interval-ms`. A smaller `sink.buffer-flush.interval-ms` reduces connector-side latency but sends data in smaller batches. * **StarRocks merge window**: Controlled by `sink.properties.merge_commit_interval_ms`. The system waits for this duration to merge requests from multiple subtasks into a single transaction. A larger value improves merging efficiency (more requests will be merged into one transaction) but increases server-side latency. * As a general guideline, set `sink.buffer-flush.interval-ms` to be smaller than or equal to `sink.properties.merge_commit_interval_ms`, so that each subtask can flush at least once within each Merge window. For example, if `merge_commit_interval_ms` is `10000` (10s), you could set `sink.buffer-flush.interval-ms` to `5000` (5 seconds) or less. * **Tuning `sink.parallelism` and `sink.properties.merge_commit_parallel`** These two parameters control parallelism at different layers and should be tuned independently: * `sink.parallelism` controls the number of Flink sink subtasks. Each subtask buffers and sends data to StarRocks. Increase this value when Flink sink operators are CPU- or memory-bound — you can monitor Flink's per-operator CPU and memory usage to determine whether more subtasks are needed. * `sink.properties.merge_commit_parallel` controls the degree of parallelism for the loading plan that StarRocks creates for each Merge Commit transaction. Increase this value when StarRocks becomes the bottleneck. You can monitor the StarRocks metrics [merge\_commit\_pending\_total](https://docs.starrocks.io/docs/administration/management/monitoring/metrics.md#merge_commit_pending_total) (number of pending Merge Commit tasks) and [merge\_commit\_pending\_bytes](https://docs.starrocks.io/docs/administration/management/monitoring/metrics.md#merge_commit_pending_bytes) (bytes held by pending tasks) to determine whether more parallelism is needed on the StarRocks side — sustained high values indicate that the loading plan cannot keep up with incoming data. * **Relationship between `sink.merge-commit.chunk.size` and `sink.buffer-flush.max-bytes`**: * `sink.merge-commit.chunk.size` controls the maximum data size per individual Stream Load request (per chunk). When data in a chunk reaches this size, it is flushed immediately. * `sink.buffer-flush.max-bytes` controls the total memory limit for all cached data across all tables. When the total cached data exceeds this limit, the connector will evict chunks early to free memory. * Therefore, `sink.buffer-flush.max-bytes` should be set larger than `sink.merge-commit.chunk.size` to allow at least one full chunk to be accumulated. In general, `sink.buffer-flush.max-bytes` should be several times larger than `sink.merge-commit.chunk.size`, especially when there are multiple tables or high concurrency. ##### Let server-side DEFAULT values apply with the JSON format[​](#let-server-side-default-values-apply-with-the-json-format "Direct link to Let server-side DEFAULT values apply with the JSON format") With the `json` format, the connector does not send a `columns` header by default. The server therefore includes all table columns in the load. If a column is missing from the JSON payload, the server treats it as an explicit `NULL`, which overrides any `DEFAULT` value defined for that column. For example, if a table contains `ingest_ts DATETIME DEFAULT CURRENT_TIMESTAMP`, the column is loaded as `NULL` unless every row provides a value for `ingest_ts`. To allow server-side `DEFAULT` values to take effect, set `sink.json.columns-from-flink-schema=true`. The connector then uses the Flink table schema to generate the `columns` header, similar to how it handles `csv` when the Flink and StarRocks schemas differ. A StarRocks column that is not defined in the Flink table schema is omitted from the `columns` header, allowing the server to apply its `DEFAULT` value. Therefore, the Flink table definition determines which columns are included in the load: * **Column omitted from the Flink schema:** The connector does not send the column, so StarRocks applies its `DEFAULT` value. * **Column defined in the Flink schema:** The connector sends the value from the row, including `NULL` if the value is null. This option requires a Flink table schema and therefore applies to Flink SQL and sinks created with a schema. For `sink.version=V1`, you must also set `sink.properties.strip_outer_array=true`. The V1 sink sends each batch as a JSON array and does not set this property automatically, unlike V2. The raw `String` DataStream sink does not have a schema and therefore rejects `sink.json.columns-from-flink-schema=true` at startup. For this sink, set `sink.properties.columns` explicitly instead. ##### Monitoring load metrics[​](#monitoring-load-metrics "Direct link to Monitoring load metrics") The Flink connector provides the following metrics to monitor loading. | Metric | Type | Description | | ------------------------ | ------- | ------------------------------------------------------------------ | | totalFlushBytes | counter | successfully flushed bytes. | | totalFlushRows | counter | number of rows successfully flushed. | | totalFlushSucceededTimes | counter | number of times that the data is successfully flushed. | | totalFlushFailedTimes | counter | number of times that the data fails to be flushed. | | totalFilteredRows | counter | number of rows filtered, which is also included in totalFlushRows. | #### Examples[​](#examples "Direct link to Examples") The following examples show how to use the Flink connector to load data into a StarRocks table with Flink SQL or Flink DataStream. ##### Preparations[​](#preparations "Direct link to Preparations") ###### Create a StarRocks table[​](#create-a-starrocks-table "Direct link to Create a StarRocks table") Create a database `test` and create a Primary Key table `score_board`. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Set up Flink environment[​](#set-up-flink-environment "Direct link to Set up Flink environment") * Download Flink binary [Flink 1.15.2](https://archive.apache.org/dist/flink/flink-1.15.2/flink-1.15.2-bin-scala_2.12.tgz), and unzip it to directory `flink-1.15.2`. * Download [Flink connector 1.2.7](https://repo1.maven.org/maven2/com/starrocks/flink-connector-starrocks/1.2.7_flink-1.15/flink-connector-starrocks-1.2.7_flink-1.15.jar), and put it into the directory `flink-1.15.2/lib`. * Run the following commands to start a Flink cluster: ```shell cd flink-1.15.2 ./bin/start-cluster.sh ``` ###### Network configuration[​](#network-configuration "Direct link to Network configuration") Ensure that the machine where Flink is located can access the FE nodes of the StarRocks cluster via the [`http_port`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#http_port) (default: `8030`) and [`query_port`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#query_port) (default: `9030`), and the BE nodes via the [`be_http_port`](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#be_http_port) (default: `8040`). ##### Run with Flink SQL[​](#run-with-flink-sql "Direct link to Run with Flink SQL") * Run the following command to start a Flink SQL client. ```shell ./bin/sql-client.sh ``` * Create a Flink table `score_board`, and insert values into the table via Flink SQL Client. Note you must define the primary key in the Flink DDL if you want to load data into a Primary Key table of StarRocks. It's optional for other types of StarRocks tables. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '' ); INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 100); ``` ##### Run with Flink DataStream[​](#run-with-flink-datastream "Direct link to Run with Flink DataStream") There are several ways to implement a Flink DataStream job according to the type of the input records, such as a CSV Java `String`, a JSON Java `String` or a custom Java object. * The input records are CSV-format `String`. See [LoadCsvRecords](https://github.com/StarRocks/starrocks-connector-for-apache-flink/tree/cd8086cfedc64d5181785bdf5e89a847dc294c1d/examples/src/main/java/com/starrocks/connector/flink/examples/datastream) for a complete example. ```java /** * Generate CSV-format records. Each record has three values separated by "\t". * These values will be loaded to the columns `id`, `name`, and `score` in the StarRocks table. */ String[] records = new String[]{ "1\tstarrocks-csv\t100", "2\tflink-csv\t100" }; DataStream source = env.fromElements(records); /** * Configure the connector with the required properties. * You also need to add properties "sink.properties.format" and "sink.properties.column_separator" * to tell the connector the input records are CSV-format, and the column separator is "\t". * You can also use other column separators in the CSV-format records, * but remember to modify the "sink.properties.column_separator" correspondingly. */ StarRocksSinkOptions options = StarRocksSinkOptions.builder() .withProperty("jdbc-url", jdbcUrl) .withProperty("load-url", loadUrl) .withProperty("database-name", "test") .withProperty("table-name", "score_board") .withProperty("username", "root") .withProperty("password", "") .withProperty("sink.properties.format", "csv") .withProperty("sink.properties.column_separator", "\t") .build(); // Create the sink with the options. SinkFunction starRockSink = StarRocksSink.sink(options); source.addSink(starRockSink); ``` * The input records are JSON-format `String`. See [LoadJsonRecords](https://github.com/StarRocks/starrocks-connector-for-apache-flink/tree/cd8086cfedc64d5181785bdf5e89a847dc294c1d/examples/src/main/java/com/starrocks/connector/flink/examples/datastream) for a complete example. ```java /** * Generate JSON-format records. * Each record has three key-value pairs corresponding to the columns `id`, `name`, and `score` in the StarRocks table. */ String[] records = new String[]{ "{\"id\":1, \"name\":\"starrocks-json\", \"score\":100}", "{\"id\":2, \"name\":\"flink-json\", \"score\":100}", }; DataStream source = env.fromElements(records); /** * Configure the connector with the required properties. * You also need to add properties "sink.properties.format" and "sink.properties.strip_outer_array" * to tell the connector the input records are JSON-format and to strip the outermost array structure. */ StarRocksSinkOptions options = StarRocksSinkOptions.builder() .withProperty("jdbc-url", jdbcUrl) .withProperty("load-url", loadUrl) .withProperty("database-name", "test") .withProperty("table-name", "score_board") .withProperty("username", "root") .withProperty("password", "") .withProperty("sink.properties.format", "json") .withProperty("sink.properties.strip_outer_array", "true") .build(); // Create the sink with the options. SinkFunction starRockSink = StarRocksSink.sink(options); source.addSink(starRockSink); ``` * The input records are custom Java objects. See [LoadCustomJavaRecords](https://github.com/StarRocks/starrocks-connector-for-apache-flink/tree/cd8086cfedc64d5181785bdf5e89a847dc294c1d/examples/src/main/java/com/starrocks/connector/flink/examples/datastream) for a complete example. * In this example, the input record is a simple POJO `RowData`. ```java public static class RowData { public int id; public String name; public int score; public RowData() {} public RowData(int id, String name, int score) { this.id = id; this.name = name; this.score = score; } } ``` * The main program is as follows: ```java // Generate records which use RowData as the container. RowData[] records = new RowData[]{ new RowData(1, "starrocks-rowdata", 100), new RowData(2, "flink-rowdata", 100), }; DataStream source = env.fromElements(records); // Configure the connector with the required properties. StarRocksSinkOptions options = StarRocksSinkOptions.builder() .withProperty("jdbc-url", jdbcUrl) .withProperty("load-url", loadUrl) .withProperty("database-name", "test") .withProperty("table-name", "score_board") .withProperty("username", "root") .withProperty("password", "") .build(); /** * The Flink connector will use a Java object array (Object[]) to represent a row to be loaded into the StarRocks table, * and each element is the value for a column. * You need to define the schema of the Object[] which matches that of the StarRocks table. */ TableSchema schema = TableSchema.builder() .field("id", DataTypes.INT().notNull()) .field("name", DataTypes.STRING()) .field("score", DataTypes.INT()) // When the StarRocks table is a Primary Key table, you must specify notNull(), for example, DataTypes.INT().notNull(), for the primary key `id`. .primaryKey("id") .build(); // Transform the RowData to the Object[] according to the schema. RowDataTransformer transformer = new RowDataTransformer(); // Create the sink with the schema, options, and transformer. SinkFunction starRockSink = StarRocksSink.sink(schema, options, transformer); source.addSink(starRockSink); ``` * The `RowDataTransformer` in the main program is defined as follows: ```java private static class RowDataTransformer implements StarRocksSinkRowBuilder { /** * Set each element of the object array according to the input RowData. * The schema of the array matches that of the StarRocks table. */ @Override public void accept(Object[] internalRow, RowData rowData) { internalRow[0] = rowData.id; internalRow[1] = rowData.name; internalRow[2] = rowData.score; // When the StarRocks table is a Primary Key table, you need to set the last element to indicate whether the data loading is an UPSERT or DELETE operation. internalRow[internalRow.length - 1] = StarRocksSinkOP.UPSERT.ordinal(); } } ``` ##### Synchronize data with Flink CDC 3.0 (with schema change supported)[​](#synchronize-data-with-flink-cdc-30-with-schema-change-supported "Direct link to Synchronize data with Flink CDC 3.0 (with schema change supported)") [Flink CDC 3.0](https://nightlies.apache.org/flink/flink-cdc-docs-stable) framework can be used to easily build a streaming ELT pipeline from CDC sources (such as MySQL and Kafka) to StarRocks. The pipeline can synchronize whole database, merged sharding tables, and schema changes from sources to StarRocks. Since v1.2.9, the Flink connector for StarRocks is integrated into this framework as [StarRocks Pipeline Connector](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.1/docs/connectors/pipeline-connectors/starrocks/). The StarRocks Pipeline Connector supports: * Automatic creation of databases and tables * Schema change synchronization * Full and incremental data synchronization For quick start, see [Streaming ELT from MySQL to StarRocks using Flink CDC 3.0 with StarRocks Pipeline Connector](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.4/docs/get-started/quickstart/mysql-to-starrocks/). It is advised to use StarRocks v3.2.1 and later versions to enable [fast\_schema\_evolution](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md#set-fast-schema-evolution). It will improve the speed of adding or dropping columns and reduce resource usage. #### Best practices[​](#best-practices "Direct link to Best practices") ##### Load data to a Primary Key table[​](#load-data-to-a-primary-key-table "Direct link to Load data to a Primary Key table") This section will show how to load data to a StarRocks Primary Key table to achieve partial updates and conditional updates. You can see [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md) for the introduction of those features. These examples use Flink SQL. ###### Preparations[​](#preparations-1 "Direct link to Preparations") Create a database `test` and create a Primary Key table `score_board` in StarRocks. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Partial update[​](#partial-update "Direct link to Partial update") This example will show how to load data only to columns `id` and `name`. 1. Insert two data rows into the StarRocks table `score_board` in MySQL client. ```sql mysql> INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 100); mysql> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | flink | 100 | +------+-----------+-------+ 2 rows in set (0.02 sec) ``` 2. Create a Flink table `score_board` in Flink SQL client. * Define the DDL which only includes the columns `id` and `name`. * Set the option `sink.properties.partial_update` to `true` which tells the Flink connector to perform partial updates. * If the Flink connector version `<=` 1.2.7, you also need to set the option `sink.properties.columns` to `id,name,__op` to tells the Flink connector which columns need to be updated. Note that you need to append the field `__op` at the end. The field `__op` indicates that the data loading is an UPSERT or DELETE operation, and its values are set by the connector automatically. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.partial_update' = 'true', -- only for Flink connector version <= 1.2.7 'sink.properties.columns' = 'id,name,__op' ); ``` 3. Insert two data rows into the Flink table. The primary keys of the data rows are as same as these of rows in the StarRocks table. but the values in the column `name` are modified. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks-update'), (2, 'flink-update'); ``` 4. Query the StarRocks table in MySQL client. ```sql mysql> select * from score_board; +------+------------------+-------+ | id | name | score | +------+------------------+-------+ | 1 | starrocks-update | 100 | | 2 | flink-update | 100 | +------+------------------+-------+ 2 rows in set (0.02 sec) ``` You can see that only values for `name` change, and the values for `score` do not change. ###### Conditional update[​](#conditional-update "Direct link to Conditional update") This example will show how to do conditional update according to the value of column `score`. The update for an `id` takes effect only when the new value for `score` is has a greater or equal to the old value. 1. Insert two data rows into the StarRocks table in MySQL client. ```sql mysql> INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 100); mysql> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | flink | 100 | +------+-----------+-------+ 2 rows in set (0.02 sec) ``` 2. Create a Flink table `score_board` in the following ways: * Define the DDL including all of columns. * Set the option `sink.properties.merge_condition` to `score` to tell the connector to use the column `score` as the condition. * Set the option `sink.version` to `V1` or `V2`. Both support conditional update. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.merge_condition' = 'score', 'sink.version' = 'V1' ); ``` 3. Insert two data rows into the Flink table. The primary keys of the data rows are as same as these of rows in the StarRocks table. The first data row has a smaller value in the column `score`, and the second data row has a larger value in the column `score`. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks-update', 99), (2, 'flink-update', 101); ``` 4. Query the StarRocks table in MySQL client. ```sql mysql> select * from score_board; +------+--------------+-------+ | id | name | score | +------+--------------+-------+ | 1 | starrocks | 100 | | 2 | flink-update | 101 | +------+--------------+-------+ 2 rows in set (0.03 sec) ``` You can see that only the values of the second data row change, and the values of the first data row do not change. ##### Load data with Merge Commit[​](#load-data-with-merge-commit "Direct link to Load data with Merge Commit") This section shows how to use Merge Commit to improve loading throughput when you have multiple Flink sink subtasks writing to the same StarRocks table. These examples use Flink SQL and StarRocks v3.4.0 or later. ###### Preparations[​](#preparations-2 "Direct link to Preparations") Create a database `test` and create a Primary Key table `score_board` in StarRocks. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Basic configuration[​](#basic-configuration "Direct link to Basic configuration") This Flink SQL enables merge commit with a 10-second merge window. Data from all sink subtasks is merged into a single transaction within each window. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.enable_merge_commit' = 'true', 'sink.properties.merge_commit_interval_ms' = '10000', 'sink.buffer-flush.interval-ms' = '5000' ); ``` Insert data into the Flink table. The data will be loaded into StarRocks via merge commit. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 95), (3, 'spark', 90); ``` ###### In-order loading for Primary Key tables[​](#in-order-loading-for-primary-key-tables "Direct link to In-order loading for Primary Key tables") By default, a single sink subtask may send multiple Stream Load requests concurrently, which can cause out-of-order loading. For Primary Key tables where data ordering matters, there are two approaches to handle this issue. **Method 1: Use `sink.merge-commit.max-concurrent-requests`** Set `sink.merge-commit.max-concurrent-requests` to `0` to ensure each subtask sends requests one at a time. This guarantees in-order loading but may reduce throughput. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.enable_merge_commit' = 'true', 'sink.properties.merge_commit_interval_ms' = '10000', 'sink.buffer-flush.interval-ms' = '5000', 'sink.merge-commit.max-concurrent-requests' = '0' ); INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 95), (3, 'spark', 90); ``` **Method 2: Use Conditional Update** If you want to keep concurrent requests for higher throughput but still prevent older data from overwriting newer data, you can use [Conditional Update](#conditional-update). Set `sink.properties.merge_condition` to a column (for example, a version or timestamp column) so that an update only takes effect when the incoming value is greater than or equal to the existing value. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.enable_merge_commit' = 'true', 'sink.properties.merge_commit_interval_ms' = '10000', 'sink.buffer-flush.interval-ms' = '5000', 'sink.properties.merge_condition' = 'score' ); INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 95), (3, 'spark', 90); ``` With this configuration, concurrent requests are allowed (default `sink.merge-commit.max-concurrent-requests` is `Integer.MAX_VALUE`), but an update to a row only takes effect when the new `score` is greater than or equal to the existing `score`. This prevents newer data from being overwritten by older data even under out-of-order loading. ##### Load data into columns of BITMAP type[​](#load-data-into-columns-of-bitmap-type "Direct link to Load data into columns of BITMAP type") [`BITMAP`](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/BITMAP.md) is often used to accelerate count distinct, such as counting UV, see [Use Bitmap for exact Count Distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_bitmap.md). Here we take the counting of UV as an example to show how to load data into columns of the `BITMAP` type. 1. Create a StarRocks Aggregate table in MySQL client. In the database `test`, create an Aggregate table `page_uv` where the column `visit_users` is defined as the `BITMAP` type and configured with the aggregate function `BITMAP_UNION`. ```sql CREATE TABLE `test`.`page_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` BITMAP BITMAP_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`); ``` 2. Create a Flink table in Flink SQL client. The column `visit_user_id` in the Flink table is of `BIGINT` type, and we want to load this column to the column `visit_users` of `BITMAP` type in the StarRocks table. So when defining the DDL of the Flink table, note that: * Because Flink does not support `BITMAP`, you need to define a column `visit_user_id` as `BIGINT` type to represent the column `visit_users` of `BITMAP` type in the StarRocks table. * You need to set the option `sink.properties.columns` to `page_id,visit_date,user_id,visit_users=to_bitmap(visit_user_id)`, which tells the connector the column mapping between the Flink table and StarRocks table. Also you need to use [`to_bitmap`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/to_bitmap.md) function to tell the connector to convert the data of `BIGINT` type into `BITMAP` type. ```sql CREATE TABLE `page_uv` ( `page_id` INT, `visit_date` TIMESTAMP, `visit_user_id` BIGINT ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'page_uv', 'username' = 'root', 'password' = '', 'sink.properties.columns' = 'page_id,visit_date,visit_user_id,visit_users=to_bitmap(visit_user_id)' ); ``` 3. Load data into Flink table in Flink SQL client. ```sql INSERT INTO `page_uv` VALUES (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 13), (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 23), (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 33), (1, CAST('2020-06-23 02:30:30' AS TIMESTAMP), 13), (2, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 23); ``` 4. Calculate page UVs from the StarRocks table in MySQL client. ```sql MySQL [test]> SELECT `page_id`, COUNT(DISTINCT `visit_users`) FROM `page_uv` GROUP BY `page_id`; +---------+-----------------------------+ | page_id | count(DISTINCT visit_users) | +---------+-----------------------------+ | 2 | 1 | | 1 | 3 | +---------+-----------------------------+ 2 rows in set (0.05 sec) ``` ##### Load data into columns of HLL type[​](#load-data-into-columns-of-hll-type "Direct link to Load data into columns of HLL type") [`HLL`](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/HLL.md) can be used for approximate count distinct, see [Use HLL for approximate count distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_HLL.md). Here we take the counting of UV as an example to show how to load data into columns of the `HLL` type. 1. Create a StarRocks Aggregate table In the database `test`, create an Aggregate table `hll_uv` where the column `visit_users` is defined as the `HLL` type and configured with the aggregate function `HLL_UNION`. ```sql CREATE TABLE `hll_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` HLL HLL_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`); ``` 2. Create a Flink table in Flink SQL client. The column `visit_user_id` in the Flink table is of `BIGINT` type, and we want to load this column to the column `visit_users` of `HLL` type in the StarRocks table. So when defining the DDL of the Flink table, note that: * Because Flink does not support `BITMAP`, you need to define a column `visit_user_id` as `BIGINT` type to represent the column `visit_users` of `HLL` type in the StarRocks table. * You need to set the option `sink.properties.columns` to `page_id,visit_date,user_id,visit_users=hll_hash(visit_user_id)` which tells the connector the column mapping between Flink table and StarRocks table. Also you need to use [`hll_hash`](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_hash.md) function to tell the connector to convert the data of `BIGINT` type into `HLL` type. ```sql CREATE TABLE `hll_uv` ( `page_id` INT, `visit_date` TIMESTAMP, `visit_user_id` BIGINT ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'hll_uv', 'username' = 'root', 'password' = '', 'sink.properties.columns' = 'page_id,visit_date,visit_user_id,visit_users=hll_hash(visit_user_id)' ); ``` 3. Load data into Flink table in Flink SQL client. ```sql INSERT INTO `hll_uv` VALUES (3, CAST('2023-07-24 12:00:00' AS TIMESTAMP), 78), (4, CAST('2023-07-24 13:20:10' AS TIMESTAMP), 2), (3, CAST('2023-07-24 12:30:00' AS TIMESTAMP), 674); ``` 4. Calculate page UVs from the StarRocks table in MySQL client. ```sql mysql> SELECT `page_id`, COUNT(DISTINCT `visit_users`) FROM `hll_uv` GROUP BY `page_id`; **+---------+-----------------------------+ | page_id | count(DISTINCT visit_users) | +---------+-----------------------------+ | 3 | 2 | | 4 | 1 | +---------+-----------------------------+ 2 rows in set (0.04 sec) ``` --- ### Apache Kafka #### [📄️ Kafka connector](https://docs.starrocks.io/docs/integrations/streaming/kafka/Kafka-connector-starrocks.md) [Continuous streaming from Kafka](https://docs.starrocks.io/docs/integrations/streaming/kafka/Kafka-connector-starrocks.md) #### [📄️ Routine Load](https://docs.starrocks.io/docs/integrations/streaming/kafka/RoutineLoad.md) [Load CSV, JSON, or Avro format from a Kafka topic](https://docs.starrocks.io/docs/integrations/streaming/kafka/RoutineLoad.md) #### [📄️ AutoMQ Kafka](https://docs.starrocks.io/docs/integrations/streaming/kafka/automq-routine-load.md) [How to load data into StarRocks from AutoMQ Kafka using Routine Load.](https://docs.starrocks.io/docs/integrations/streaming/kafka/automq-routine-load.md) --- ### AutoMQ Kafka [AutoMQ for Kafka](https://docs.automq.com/automq/what-is-automq/overview) is a cloud-native version of Kafka redesigned for cloud environments. AutoMQ Kafka is [open source](https://github.com/AutoMQ/automq-for-kafka) and fully compatible with the Kafka protocol, fully leveraging cloud benefits. Compared to self-managed Apache Kafka, AutoMQ Kafka, with its cloud-native architecture, offers features like capacity auto scaling, self-balancing of network traffic, move partition in seconds. These features contribute to a significantly lower Total Cost of Ownership (TCO) for users. This article will guide you through importing data into AutoMQ Kafka using StarRocks Routine Load. For an understanding of the basic principles of Routine Load, refer to the section on Routine Load Fundamentals. #### Prepare Environment[​](#prepare-environment "Direct link to Prepare Environment") ##### Prepare StarRocks and test data[​](#prepare-starrocks-and-test-data "Direct link to Prepare StarRocks and test data") Ensure you have a running StarRocks cluster. Creating a database and a Primary Key table for testing: ```sql create database automq_db; create table users ( id bigint NOT NULL, name string NOT NULL, timestamp string NULL, status string NULL ) PRIMARY KEY (id) DISTRIBUTED BY HASH(id) PROPERTIES ( "enable_persistent_index" = "true" ); ``` note If a StarRocks cluster in a staging environment contains only one BE, the number of replicas can be set to `1` in the `PROPERTIES` clause, such as `PROPERTIES( "replication_num" = "1" )`. The default number of replicas is 3, which is also the number recommended for production StarRocks clusters. If you want to use the default number, you do not need to configure the `replication_num` parameter. #### Prepare AutoMQ Kafka and test data[​](#prepare-automq-kafka-and-test-data "Direct link to Prepare AutoMQ Kafka and test data") To prepare your AutoMQ Kafka environment and test data, follow the AutoMQ [Quick Start](https://docs.automq.com/automq/what-is-automq/overview) guide to deploy your AutoMQ Kafka cluster. Ensure that StarRocks can directly connect to your AutoMQ Kafka server. To quickly create a topic named `example_topic` in AutoMQ Kafka and write a test JSON data into it, follow these steps: ##### Create a topic[​](#create-a-topic "Direct link to Create a topic") Use Kafka’s command-line tools to create a topic. Ensure you have access to the Kafka environment and the Kafka service is running. Here is the command to create a topic: ```shell ./kafka-topics.sh --create --topic example_topic --bootstrap-server 10.0.96.4:9092 --partitions 1 --replication-factor 1 ``` > Note: Replace `topic` and `bootstrap-server` with your Kafka server address. To check the result of the topic creation, use this command: ```shell ./kafka-topics.sh --describe example_topic --bootstrap-server 10.0.96.4:9092 ``` ##### Generate test data[​](#generate-test-data "Direct link to Generate test data") Generate a simple JSON format test data ```json { "id": 1, "name": "testuser", "timestamp": "2023-11-10T12:00:00", "status": "active" } ``` ##### Write Test Data[​](#write-test-data "Direct link to Write Test Data") Use Kafka’s command-line tools or programming methods to write test data into example\_topic. Here is an example using command-line tools: ```shell echo '{"id": 1, "name": "testuser", "timestamp": "2023-11-10T12:00:00", "status": "active"}' | sh kafka-console-producer.sh --broker-list 10.0.96.4:9092 --topic example_topic ``` > Note: Replace `topic` and `bootstrap-server` with your Kafka server address. To view the recently written topic data, use the following command: ```shell sh kafka-console-consumer.sh --bootstrap-server 10.0.96.4:9092 --topic example_topic --from-beginning ``` #### Creating a Routine Load Task[​](#creating-a-routine-load-task "Direct link to Creating a Routine Load Task") In the StarRocks command line, create a Routine Load task to continuously import data from the AutoMQ Kafka topic: ```sql CREATE ROUTINE LOAD automq_example_load ON users COLUMNS(id, name, timestamp, status) PROPERTIES ( "desired_concurrent_number" = "5", "format" = "json", "jsonpaths" = "[\"$.id\",\"$.name\",\"$.timestamp\",\"$.status\"]" ) FROM KAFKA ( "kafka_broker_list" = "10.0.96.4:9092", "kafka_topic" = "example_topic", "kafka_partitions" = "0", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` > Note: Replace `kafka_broker_list` with your Kafka server address. ##### Explanation of Parameters[​](#explanation-of-parameters "Direct link to Explanation of Parameters") ###### Data Format[​](#data-format "Direct link to Data Format") Specify the data format as JSON in the "format" = "json" of the PROPERTIES clause. ###### Data Extraction and Transformation[​](#data-extraction-and-transformation "Direct link to Data Extraction and Transformation") To specify the mapping and transformation relationship between the source data and the target table, configure the COLUMNS and jsonpaths parameters. The column names in COLUMNS correspond to the column names of the target table, and their order corresponds to the column order in the source data. The jsonpaths parameter is used to extract the required field data from JSON data, similar to newly generated CSV data. Then the COLUMNS parameter temporarily names the fields in jsonpaths in order. For more explanations on data transformation, please see [Data Transformation during Import](https://docs.starrocks.io/docs/loading/Etl_in_loading.md). > Note: If each JSON object per line has key names and quantities (order is not required) that correspond to the columns of the target table, there is no need to configure COLUMNS. #### Verifying Data Import[​](#verifying-data-import "Direct link to Verifying Data Import") First, we check the Routine Load import job and confirm the Routine Load import task status is in RUNNING status. ```sql show routine load\G ``` Then, querying the corresponding table in the StarRocks database, we can observe that the data has been successfully imported. ```sql StarRocks > select * from users; +------+--------------+---------------------+--------+ | id | name | timestamp | status | +------+--------------+---------------------+--------+ | 1 | testuser | 2023-11-10T12:00:00 | active | | 2 | testuser | 2023-11-10T12:00:00 | active | +------+--------------+---------------------+--------+ 2 rows in set (0.01 sec) ``` --- ### Kafka connector for StarRocks ### Load data using Kafka connector StarRocks provides a self-developed connector named Apache Kafka® connector (StarRocks Connector for Apache Kafka®, Kafka connector for short), as a sink connector, that continuously consumes messages from Kafka and loads them into StarRocks. The Kafka connector guarantees at-least-once semantics. The Kafka connector can seamlessly integrate with Kafka Connect, which allows StarRocks better integrated with the Kafka ecosystem. It is a wise choice if you want to load real-time data into StarRocks. Compared with Routine Load, it is recommended to use the Kafka connector in the following scenarios: * Compared with Routine Load which only supports loading data in CSV, JSON, and Avro formats, Kafka connector can load data in more formats, such as Protobuf. As long as data can be converted into JSON and CSV formats using Kafka Connect's converters, data can be loaded into StarRocks via the Kafka connector. * Customize data transformation, such as Debezium-formatted CDC data. * Load data from multiple Kafka topics. * Load data from Confluent Cloud. * Need finer control over load batch sizes, parallelism, and other parameters to achieve a balance between load speed and resource utilization. #### Preparations[​](#preparations "Direct link to Preparations") ##### Version requirements[​](#version-requirements "Direct link to Version requirements") | Connector | Kafka | StarRocks | Java | | --------- | --------- | ------------- | ---- | | 1.0.6 | 3.4+/4.0+ | 2.5 and later | 8 | | 1.0.5 | 3.4 | 2.5 and later | 8 | | 1.0.4 | 3.4 | 2.5 and later | 8 | | 1.0.3 | 3.4 | 2.5 and later | 8 | ##### Set up Kafka environment[​](#set-up-kafka-environment "Direct link to Set up Kafka environment") Both self-managed Apache Kafka clusters and Confluent Cloud are supported. * For a self-managed Apache Kafka cluster, you can refer to [Apache Kafka quickstart](https://kafka.apache.org/quickstart) to quickly deploy a Kafka cluster. Kafka Connect is already integrated into Kafka. * For Confluent Cloud, make sure that you have a Confluent account and have created a cluster. ##### Download Kafka connector[​](#download-kafka-connector "Direct link to Download Kafka connector") Submit the Kafka connector into Kafka Connect: * Self-managed Kafka cluster: Download [starrocks-connector-for-kafka-x.y.z-with-dependencies.jar](https://github.com/StarRocks/starrocks-connector-for-kafka/releases). * Confluent Cloud: Currently, the Kafka connector is not uploaded to Confluent Hub. You need to download [starrocks-connector-for-kafka-x.y.z-with-dependencies.jar](https://github.com/StarRocks/starrocks-connector-for-kafka/releases), package it into a ZIP file and upload the ZIP file to Confluent Cloud. ##### Network configuration[​](#network-configuration "Direct link to Network configuration") Ensure that the machine where Kafka is located can access the FE nodes of the StarRocks cluster via the [`http_port`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#http_port) (default: `8030`) and [`query_port`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#query_port) (default: `9030`), and the BE nodes via the [`be_http_port`](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#be_http_port) (default: `8040`). #### Usage[​](#usage "Direct link to Usage") This section uses a self-managed Kafka cluster as an example to explain how to configure the Kafka connector and the Kafka Connect, and then run the Kafka Connect to load data into StarRocks. ##### Prepare a dataset[​](#prepare-a-dataset "Direct link to Prepare a dataset") Suppose that JSON-format data exists in the topic `test` in a Kafka cluster. ```json {"id":1,"city":"New York"} {"id":2,"city":"Los Angeles"} {"id":3,"city":"Chicago"} ``` ##### Create a table[​](#create-a-table "Direct link to Create a table") Create the table `test_tbl` in the database `example_db` in the StarRocks cluster according to the keys of the JSON-format data. ```sql CREATE DATABASE example_db; USE example_db; CREATE TABLE test_tbl (id INT, city STRING); ``` ##### Configure Kafka connector and Kafka Connect, and then run Kafka Connect to load data[​](#configure-kafka-connector-and-kafka-connect-and-then-run-kafka-connect-to-load-data "Direct link to Configure Kafka connector and Kafka Connect, and then run Kafka Connect to load data") ###### Run Kafka Connect in standalone mode[​](#run-kafka-connect-in-standalone-mode "Direct link to Run Kafka Connect in standalone mode") 1. Configure the Kafka connector. In the **config** directory under the Kafka installation directory, create the configuration file **connect-StarRocks-sink.properties** for the Kafka connector, and configure the following parameters. For more parameters and descriptions, see [Parameters](#Parameters). info * In this example, the Kafka connector provided by StarRocks is a sink connector that can continuously consume data from Kafka and load data into StarRocks. * If the source data is CDC data, such as data in Debezium format, and the StarRocks table is a Primary Key table, you also need to [configure `transform`](#load-debezium-formatted-cdc-data) in the configuration file **connect-StarRocks-sink.properties** for the Kafka connector provided by StarRocks, to synchronize the source data changes to the Primary Key table. ```yaml name=starrocks-kafka-connector connector.class=com.starrocks.connector.kafka.StarRocksSinkConnector topics=test key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=false # The HTTP URL of the FE in your StarRocks cluster. The default port is 8030. starrocks.http.url=192.168.xxx.xxx:8030 # If the Kafka topic name is different from the StarRocks table name, you need to configure the mapping relationship between them. starrocks.topic2table.map=test:test_tbl # Enter the StarRocks username. starrocks.username=user1 # Enter the StarRocks password. starrocks.password=123456 starrocks.database.name=example_db sink.properties.strip_outer_array=true ``` 2. Configure and run the Kafka Connect. 1. Configure the Kafka Connect. In the configuration file **config/connect-standalone.properties** in the **config** directory, configure the following parameters. For more parameters and descriptions, see [Running Kafka Connect](https://kafka.apache.org/documentation.html#connect_running). ```yaml # The addresses of Kafka brokers. Multiple addresses of Kafka brokers need to be separated by commas (,). # Note that this example uses PLAINTEXT as the security protocol to access the Kafka cluster. If you are using other security protocol to access the Kafka cluster, you need to configure the relevant information in this file. bootstrap.servers=:9092 offset.storage.file.filename=/tmp/connect.offsets offset.flush.interval.ms=10000 key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=false # The absolute path of starrocks-connector-for-kafka-x.y.z-with-dependencies.jar. plugin.path=/home/kafka-connect/starrocks-kafka-connector ``` 2. Run the Kafka Connect. ```bash CLASSPATH=/home/kafka-connect/starrocks-kafka-connector/* bin/connect-standalone.sh config/connect-standalone.properties config/connect-starrocks-sink.properties ``` ###### Run Kafka Connect in distributed mode[​](#run-kafka-connect-in-distributed-mode "Direct link to Run Kafka Connect in distributed mode") 1. Configure and run the Kafka Connect. 1. Configure the Kafka Connect. In the configuration file `config/connect-distributed.properties` in the **config** directory, configure the following parameters. For more parameters and descriptions, refer to [Running Kafka Connect](https://kafka.apache.org/documentation.html#connect_running). ```yaml # The addresses of Kafka brokers. Multiple addresses of Kafka brokers need to be separated by commas (,). # Note that this example uses PLAINTEXT as the security protocol to access the Kafka cluster. If you are using other security protocol to access the Kafka cluster, you need to configure the relevant information in this file. bootstrap.servers=:9092 offset.storage.file.filename=/tmp/connect.offsets offset.flush.interval.ms=10000 key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=false # The absolute path of starrocks-connector-for-kafka-x.y.z-with-dependencies.jar. plugin.path=/home/kafka-connect/starrocks-kafka-connector ``` 2. Run the Kafka Connect. ```bash CLASSPATH=/home/kafka-connect/starrocks-kafka-connector/* bin/connect-distributed.sh config/connect-distributed.properties ``` 2. Configure and create the Kafka connector. Note that in distributed mode, you need to configure and create the Kafka connector through the REST API. For parameters and descriptions, see [Parameters](#Parameters). info * In this example, the Kafka connector provided by StarRocks is a sink connector that can continuously consume data from Kafka and load data into StarRocks. * If the source data is CDC data, such as data in Debezium format, and the StarRocks table is a Primary Key table, you also need to [configure `transform`](#load-debezium-formatted-cdc-data) in the configuration file **connect-StarRocks-sink.properties** for the Kafka connector provided by StarRocks, to synchronize the source data changes to the Primary Key table. ```shell curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ "name":"starrocks-kafka-connector", "config":{ "connector.class":"com.starrocks.connector.kafka.StarRocksSinkConnector", "topics":"test", "key.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter":"org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable":"true", "value.converter.schemas.enable":"false", "starrocks.http.url":"192.168.xxx.xxx:8030", "starrocks.topic2table.map":"test:test_tbl", "starrocks.username":"user1", "starrocks.password":"123456", "starrocks.database.name":"example_db", "sink.properties.strip_outer_array":"true" } }' ``` ###### Query StarRocks table[​](#query-starrocks-table "Direct link to Query StarRocks table") Query the target StarRocks table `test_tbl`. ```mysql MySQL [example_db]> select * from test_tbl; +------+-------------+ | id | city | +------+-------------+ | 1 | New York | | 2 | Los Angeles | | 3 | Chicago | +------+-------------+ 3 rows in set (0.01 sec) ``` The data is successfully loaded when the above result is returned. #### Parameters[​](#parameters "Direct link to Parameters") ##### name[​](#name "Direct link to name") **Required**: YES
**Default value**:
**Description**: Name for this Kafka connector. It must be globally unique among all Kafka connectors within this Kafka Connect cluster. For example, starrocks-kafka-connector. ##### connector.class[​](#connectorclass "Direct link to connector.class") **Required**: YES
**Default value**:
**Description**: Class used by this Kafka connector's sink. Set the value to `com.starrocks.connector.kafka.StarRocksSinkConnector`. ##### topics[​](#topics "Direct link to topics") **Required**:
**Default value**:
**Description**: One or more topics to subscribe to, where each topic corresponds to a StarRocks table. By default, StarRocks assumes that the topic name matches the name of the StarRocks table. So StarRocks determines the target StarRocks table by using the topic name. Please choose either to fill in `topics` or `topics.regex` (below), but not both. However, if the StarRocks table name is not the same as the topic name, then use the optional `starrocks.topic2table.map` parameter (below) to specify the mapping from topic name to table name. ##### topics.regex[​](#topicsregex "Direct link to topics.regex") **Required**:
**Default value**: **Description**: Regular expression to match the one or more topics to subscribe to. For more description, see `topics`. Please choose either to fill in `topics.regex` or `topics` (above), but not both.
##### starrocks.topic2table.map[​](#starrockstopic2tablemap "Direct link to starrocks.topic2table.map") **Required**: NO
**Default value**:
**Description**: The mapping of the StarRocks table name and the topic name when the topic name is different from the StarRocks table name. The format is `:,:,...`. ##### starrocks.http.url[​](#starrockshttpurl "Direct link to starrocks.http.url") **Required**: YES
**Default value**:
**Description**: The HTTP URL of the FE in your StarRocks cluster. The format is `:,:,...`. Multiple addresses are separated by commas (,). For example, `192.168.xxx.xxx:8030,192.168.xxx.xxx:8030`. ##### starrocks.database.name[​](#starrocksdatabasename "Direct link to starrocks.database.name") **Required**: YES
**Default value**:
**Description**: The name of StarRocks database. ##### starrocks.username[​](#starrocksusername "Direct link to starrocks.username") **Required**: YES
**Default value**:
**Description**: The username of your StarRocks cluster account. The user needs the [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) privilege on the StarRocks table. ##### starrocks.password[​](#starrockspassword "Direct link to starrocks.password") **Required**: YES
**Default value**:
**Description**: The password of your StarRocks cluster account. ##### key.converter[​](#keyconverter "Direct link to key.converter") **Required**: NO
**Default value**: Key converter used by Kafka Connect cluster
**Description**: This parameter specifies the key converter for the sink connector (Kafka-connector-starrocks), which is used to deserialize the keys of Kafka data. The default key converter is the one used by Kafka Connect cluster. ##### value.converter[​](#valueconverter "Direct link to value.converter") **Required**: NO
**Default value**: Value converter used by Kafka Connect cluster
**Description**: This parameter specifies the value converter for the sink connector (Kafka-connector-starrocks), which is used to deserialize the values of Kafka data. The default value converter is the one used by Kafka Connect cluster. ##### key.converter.schema.registry.url[​](#keyconverterschemaregistryurl "Direct link to key.converter.schema.registry.url") **Required**: NO
**Default value**:
**Description**: Schema registry URL for the key converter. ##### value.converter.schema.registry.url[​](#valueconverterschemaregistryurl "Direct link to value.converter.schema.registry.url") **Required**: NO
**Default value**:
**Description**: Schema registry URL for the value converter. ##### tasks.max[​](#tasksmax "Direct link to tasks.max") **Required**: NO
**Default value**: 1
**Description**: The upper limit for the number of task threads that the Kafka connector can create, which is usually the same as the number of CPU cores on the worker nodes in the Kafka Connect cluster. You can tune this parameter to control load performance. ##### bufferflush.maxbytes[​](#bufferflushmaxbytes "Direct link to bufferflush.maxbytes") **Required**: NO
**Default value**: 94371840(90M)
**Description**: The maximum size of data that can be accumulated in memory before being sent to StarRocks at a time. The maximum value ranges from 64 MB to 10 GB. Keep in mind that the Stream Load SDK buffer may create multiple Stream Load jobs to buffer data. Therefore, the threshold mentioned here refers to the total data size. ##### bufferflush.intervalms[​](#bufferflushintervalms "Direct link to bufferflush.intervalms") **Required**: NO
**Default value**: 1000
**Description**: Interval for sending a batch of data which controls the load latency. Range: \[1000, 3600000]. ##### connect.timeoutms[​](#connecttimeoutms "Direct link to connect.timeoutms") **Required**: NO
**Default value**: 1000
**Description**: Timeout for connecting to the HTTP URL. Range: \[100, 60000]. ##### sink.properties.\*[​](#sinkproperties "Direct link to sink.properties.*") **Required**:
**Default value**:
**Description**: Stream Load parameters o control load behavior. For example, the parameter `sink.properties.format` specifies the format used for Stream Load, such as CSV or JSON. For a list of supported parameters and their descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ##### sink.properties.format[​](#sinkpropertiesformat "Direct link to sink.properties.format") **Required**: NO
**Default value**: json
**Description**: The format used for Stream Load. The Kafka connector will transform each batch of data to the format before sending them to StarRocks. Valid values: `csv` and `json`. For more information, see [CSV parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#csv-parameters) and [JSON parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#json-parameters). ##### sink.properties.partial\_update[​](#sinkpropertiespartial_update "Direct link to sink.properties.partial_update") **Required**: NO
**Default value**: `FALSE`
**Description**: Whether to use partial updates. Valid values: `TRUE` and `FALSE`. Default value: `FALSE`, indicating to disable this feature. ##### sink.properties.partial\_update\_mode[​](#sinkpropertiespartial_update_mode "Direct link to sink.properties.partial_update_mode") **Required**: NO
**Default value**: `row`
**Description**: Specifies the mode for partial updates. Valid values: `row` and `column`. * The value `row` (default) means partial updates in row mode, which is more suitable for real-time updates with many columns and small batches. * The value `column` means partial updates in column mode, which is more suitable for batch updates with few columns and many rows. In such scenarios, enabling the column mode offers faster update speeds. For example, in a table with 100 columns, if only 10 columns (10% of the total) are updated for all rows, the update speed of the column mode is 10 times faster. #### Usage Notes[​](#usage-notes "Direct link to Usage Notes") ##### Flush Policy[​](#flush-policy "Direct link to Flush Policy") The Kafka connector will buffer the data in memory, and flush them in batch to StarRocks via Stream Load. The flush will be triggered when any of the following conditions are met: * The bytes of buffered rows reaches the limit `bufferflush.maxbytes`. * The elapsed time since the last flush reaches the limit `bufferflush.intervalms`. * The interval at which the connector tries committing offsets for tasks is reached. The interval is controlled by the Kafka Connect configuration [`offset.flush.interval.ms`](https://docs.confluent.io/platform/current/connect/references/allconfigs.html), and the default values is `60000`. For lower data latency, adjust these configurations in the Kafka connector settings. However, more frequent flushes will increase CPU and I/O usage. ##### Limits[​](#limits "Direct link to Limits") * It is not supported to flatten a single message from a Kafka topic into multiple data rows and load into StarRocks. * The sink of the Kafka connector provided by StarRocks guarantees at-least-once semantics. #### Best practices[​](#best-practices "Direct link to Best practices") ##### Load Debezium-formatted CDC data[​](#load-debezium-formatted-cdc-data "Direct link to Load Debezium-formatted CDC data") Debezium is a popular Change Data Capture (CDC) tool that supports monitoring data changes in various database systems and streaming these changes to Kafka. The following example demonstrates how to configure and use the Kafka connector to write PostgreSQL changes to a **Primary Key table** in StarRocks. ###### Step 1: Install and start Kafka[​](#step-1-install-and-start-kafka "Direct link to Step 1: Install and start Kafka") > **NOTE** > > You can skip this step if you have your own Kafka environment. 1. [Download](https://dlcdn.apache.org/kafka/) the latest Kafka release from the official site and extract the package. ```bash tar -xzf kafka_2.13-3.7.0.tgz cd kafka_2.13-3.7.0 ``` 2. Start the Kafka environment. Generate a Kafka cluster UUID. ```bash KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)" ``` Format the log directories. ```bash bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties ``` Start the Kafka server. ```bash bin/kafka-server-start.sh config/kraft/server.properties ``` ###### Step 2: Configure PostgreSQL[​](#step-2-configure-postgresql "Direct link to Step 2: Configure PostgreSQL") 1. Make sure the PostgreSQL user is granted `REPLICATION` privileges. 2. Adjust PostgreSQL configuration. Set `wal_level` to `logical` in **postgresql.conf**. ```properties wal_level = logical ``` Restart the PostgreSQL server to apply changes. ```bash pg_ctl restart ``` 3. Prepare the dataset. Create a table and insert test data. ```sql CREATE TABLE customers ( id int primary key , first_name varchar(65533) NULL, last_name varchar(65533) NULL , email varchar(65533) NULL ); INSERT INTO customers VALUES (1,'a','a','a@a.com'); ``` 4. Verify the CDC log messages in Kafka. ```json { "schema": { "type": "struct", "fields": [ { "type": "struct", "fields": [ { "type": "int32", "optional": false, "field": "id" }, { "type": "string", "optional": true, "field": "first_name" }, { "type": "string", "optional": true, "field": "last_name" }, { "type": "string", "optional": true, "field": "email" } ], "optional": true, "name": "test.public.customers.Value", "field": "before" }, { "type": "struct", "fields": [ { "type": "int32", "optional": false, "field": "id" }, { "type": "string", "optional": true, "field": "first_name" }, { "type": "string", "optional": true, "field": "last_name" }, { "type": "string", "optional": true, "field": "email" } ], "optional": true, "name": "test.public.customers.Value", "field": "after" }, { "type": "struct", "fields": [ { "type": "string", "optional": false, "field": "version" }, { "type": "string", "optional": false, "field": "connector" }, { "type": "string", "optional": false, "field": "name" }, { "type": "int64", "optional": false, "field": "ts_ms" }, { "type": "string", "optional": true, "name": "io.debezium.data.Enum", "version": 1, "parameters": { "allowed": "true,last,false,incremental" }, "default": "false", "field": "snapshot" }, { "type": "string", "optional": false, "field": "db" }, { "type": "string", "optional": true, "field": "sequence" }, { "type": "string", "optional": false, "field": "schema" }, { "type": "string", "optional": false, "field": "table" }, { "type": "int64", "optional": true, "field": "txId" }, { "type": "int64", "optional": true, "field": "lsn" }, { "type": "int64", "optional": true, "field": "xmin" } ], "optional": false, "name": "io.debezium.connector.postgresql.Source", "field": "source" }, { "type": "string", "optional": false, "field": "op" }, { "type": "int64", "optional": true, "field": "ts_ms" }, { "type": "struct", "fields": [ { "type": "string", "optional": false, "field": "id" }, { "type": "int64", "optional": false, "field": "total_order" }, { "type": "int64", "optional": false, "field": "data_collection_order" } ], "optional": true, "name": "event.block", "version": 1, "field": "transaction" } ], "optional": false, "name": "test.public.customers.Envelope", "version": 1 }, "payload": { "before": null, "after": { "id": 1, "first_name": "a", "last_name": "a", "email": "a@a.com" }, "source": { "version": "2.5.3.Final", "connector": "postgresql", "name": "test", "ts_ms": 1714283798721, "snapshot": "false", "db": "postgres", "sequence": "[\"22910216\",\"22910504\"]", "schema": "public", "table": "customers", "txId": 756, "lsn": 22910504, "xmin": null }, "op": "c", "ts_ms": 1714283798790, "transaction": null } } ``` ###### Step 3: Configure StarRocks[​](#step-3-configure-starrocks "Direct link to Step 3: Configure StarRocks") Create a Primary Key table in StarRocks with the same schema as the source table in PostgreSQL. ```sql CREATE TABLE `customers` ( `id` int(11) COMMENT "", `first_name` varchar(65533) NULL COMMENT "", `last_name` varchar(65533) NULL COMMENT "", `email` varchar(65533) NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY hash(id) buckets 1 PROPERTIES ( "bucket_size" = "4294967296", "in_memory" = "false", "enable_persistent_index" = "true", "replicated_storage" = "true", "fast_schema_evolution" = "true" ); ``` ###### Step 4: Install connector[​](#step-4-install-connector "Direct link to Step 4: Install connector") 1. Download the connectors and extract the packages in the **plugins** directory. ```bash mkdir plugins tar -zxvf debezium-debezium-connector-postgresql-2.5.3.zip -C plugins mv starrocks-connector-for-kafka-x.y.z-with-dependencies.jar plugins ``` This directory is the value of the configuration item `plugin.path` in **config/connect-standalone.properties**. ```properties plugin.path=/path/to/kafka_2.13-3.7.0/plugins ``` 2. Configure the PostgreSQL source connector in **pg-source.properties**. ```json { "name": "inventory-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "plugin.name": "pgoutput", "database.hostname": "localhost", "database.port": "5432", "database.user": "postgres", "database.password": "", "database.dbname" : "postgres", "topic.prefix": "test" } } ``` 3. Configure the StarRocks sink connector in **sr-sink.properties**. ```json { "name": "starrocks-kafka-connector", "config": { "connector.class": "com.starrocks.connector.kafka.StarRocksSinkConnector", "tasks.max": "1", "topics": "test.public.customers", "starrocks.http.url": "172.26.195.69:28030", "starrocks.database.name": "test", "starrocks.username": "root", "starrocks.password": "StarRocks@123", "sink.properties.strip_outer_array": "true", "connect.timeoutms": "3000", "starrocks.topic2table.map": "test.public.customers:customers", "transforms": "addfield,unwrap", "transforms.addfield.type": "com.starrocks.connector.kafka.transforms.AddOpFieldForDebeziumRecord", "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", "transforms.unwrap.drop.tombstones": "true", "transforms.unwrap.delete.handling.mode": "rewrite" } } ``` > **NOTE** > > * If the StarRocks table is not a Primary Key table, you do not need to specify the `addfield` transform. > * The unwrap transform is provided by Debezium and is used to unwrap Debezium's complex data structure based on the operation type. For more information, see [New Record State Extraction](https://debezium.io/documentation/reference/stable/transformations/event-flattening.html). 4. Configure Kafka Connect. Configure the following configuration items in the Kafka Connect configuration file **config/connect-standalone.properties**. ```properties # The addresses of Kafka brokers. Multiple addresses of Kafka brokers need to be separated by commas (,). # Note that this example uses PLAINTEXT as the security protocol to access the Kafka cluster. # If you use other security protocol to access the Kafka cluster, configure the relevant information in this part. bootstrap.servers=:9092 offset.storage.file.filename=/tmp/connect.offsets key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=false # The absolute path of starrocks-connector-for-kafka-x.y.z-with-dependencies.jar. plugin.path=/home/kafka-connect/starrocks-kafka-connector # Parameters that control the flush policy. For more information, see the Usage Note section. offset.flush.interval.ms=10000 bufferflush.maxbytes = xxx bufferflush.intervalms = xxx ``` For descriptions of more parameters, see [Running Kafka Connect](https://kafka.apache.org/documentation.html#connect_running). ###### Step 5: Start Kafka Connect in Standalone Mode[​](#step-5-start-kafka-connect-in-standalone-mode "Direct link to Step 5: Start Kafka Connect in Standalone Mode") Run Kafka Connect in standalone mode to initiate the connectors. ```bash bin/connect-standalone.sh config/connect-standalone.properties config/pg-source.properties config/sr-sink.properties ``` ###### Step 6: Verify data ingestion[​](#step-6-verify-data-ingestion "Direct link to Step 6: Verify data ingestion") Test the following operations and ensure the data is correctly ingested into StarRocks. ###### INSERT[​](#insert "Direct link to INSERT") * In PostgreSQL: ```plain postgres=# insert into customers values (2,'b','b','b@b.com'); INSERT 0 1 postgres=# select * from customers; id | first_name | last_name | email ----+------------+-----------+--------- 1 | a | a | a@a.com 2 | b | b | b@b.com (2 rows) ``` * In StarRocks: ```plain MySQL [test]> select * from customers; +------+------------+-----------+---------+ | id | first_name | last_name | email | +------+------------+-----------+---------+ | 1 | a | a | a@a.com | | 2 | b | b | b@b.com | +------+------------+-----------+---------+ 2 rows in set (0.01 sec) ``` ###### UPDATE[​](#update "Direct link to UPDATE") * In PostgreSQL: ```plain postgres=# update customers set email='c@c.com'; UPDATE 2 postgres=# select * from customers; id | first_name | last_name | email ----+------------+-----------+--------- 1 | a | a | c@c.com 2 | b | b | c@c.com (2 rows) ``` * In StarRocks: ```plain MySQL [test]> select * from customers; +------+------------+-----------+---------+ | id | first_name | last_name | email | +------+------------+-----------+---------+ | 1 | a | a | c@c.com | | 2 | b | b | c@c.com | +------+------------+-----------+---------+ 2 rows in set (0.00 sec) ``` ###### DELETE[​](#delete "Direct link to DELETE") * In PostgreSQL: ```plain postgres=# delete from customers where id=1; DELETE 1 postgres=# select * from customers; id | first_name | last_name | email ----+------------+-----------+--------- 2 | b | b | c@c.com (1 row) ``` * In StarRocks: ```plain MySQL [test]> select * from customers; +------+------------+-----------+---------+ | id | first_name | last_name | email | +------+------------+-----------+---------+ | 2 | b | b | c@c.com | +------+------------+-----------+---------+ 1 row in set (0.00 sec) ``` --- ### Routine Load ### Load data using Routine Load tip Try Routine Load out in this [Quick Start](https://docs.starrocks.io/docs/quick_start/routine-load.md) This topic introduces how to create a Routine Load job to stream Kafka messages (events) into StarRocks, and familiarizes you with some basic concepts about Routine Load. To continuously load messages of a stream into StarRocks, you can store the message stream in a Kafka topic, and create a Routine Load job to consume the messages. The Routine Load job persists in StarRocks, generates a series of load tasks to consume the messages in all or part of the partitions in the topic, and loads the messages into StarRocks. A Routine Load job supports exactly-once delivery semantics to guarantee the data loaded into StarRocks is neither lost nor duplicated. Routine Load supports data transformation at data loading and supports data changes made by UPSERT and DELETE operations during data loading. For more information, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md) and [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md). You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. #### Supported data formats[​](#supported-data-formats "Direct link to Supported data formats") Routine Load now supports consuming CSV, JSON, and Avro (supported since v3.0.1) formatted data from a Kafka cluster. > **NOTE** > > For CSV data, take note of the following points: > > * You can use a UTF-8 string, such as a comma (,), tab, or pipe (|), whose length does not exceed 50 bytes as a text delimiter. > * Null values are denoted by using `\N`. For example, a data file consists of three columns, and a record from that data file holds data in the first and third columns but no data in the second column. In this situation, you need to use `\N` in the second column to denote a null value. This means the record must be compiled as `a,\N,b` instead of `a,,b`. `a,,b` denotes that the second column of the record holds an empty string. #### Basic concepts[​](#basic-concepts "Direct link to Basic concepts") ![routine load](/assets/images/4.5.2-1-a6d6d07699a1f32c703d4b80921e2e44.png) ##### Terminology[​](#terminology "Direct link to Terminology") * **Load job** A Routine Load job is a long-running job. As long as its status is RUNNING, a load job continuously generates one or multiple concurrent load tasks which consume the messages in a topic of a Kafka cluster and load the data into StarRocks. * **Load task** A load job is split into multiple load tasks by certain rules. A load task is the basic unit of data loading. As an individual event, a load task implements the load mechanism based on [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md). Multiple load tasks concurrently consume the messages from different partitions of a topic, and load the data into StarRocks. ##### Workflow[​](#workflow "Direct link to Workflow") 1. **Create a Routine Load job.** To load data from Kafka, you need to create a Routine Load job by running the [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md) statement. The FE parses the statement, and creates the job according to the properties you have specified. 2. **The FE splits the job into multiple load tasks.** The FE split the job into multiple load tasks based on certain rules. Each load task is an individual transaction. The splitting rules are as follows: * The FE calculates the actual concurrent number of the load tasks according to the desired concurrent number `desired_concurrent_number`, the partition number in the Kafka topic, and the number of the BE nodes that are alive. * The FE splits the job into load tasks based on the actual concurrent number calculated, and arranges the tasks in the task queue. Each Kafka topic consists of multiple partitions. The relation between the topic partition and the load task is as follows: * A partition is uniquely assigned to a load task, and all messages from the partition are consumed by the load task. * A load task can consume messages from one or more partitions. * All partitions are distributed evenly among load tasks. 3. **Multiple load tasks run concurrently to consume the messages from multiple Kafka topic partitions, and load the data into StarRocks** 1. **The FE schedules and submits load tasks**: the FE schedules the load tasks in the queue on a timely basis, and assigns them to selected Coordinator BE nodes. The interval between load tasks is defined by the configuration item `max_batch_interval`. The FE distributes the load tasks evenly to all BE nodes. See [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md#examples) for more information about `max_batch_interval`. 2. The Coordinator BE starts the load task, consumes messages in partitions, parses and filters the data. A load task lasts until the pre-defined amount of messages are consumed or the pre-defined time limit is reached. The message batch size and time limit are defined in the FE configurations `max_routine_load_batch_size` and `routine_load_task_consume_second`. For detailed information, see [FE Configuration](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md). The Coordinator BE then distributes the messages to the Executor BEs. The Executor BEs write the messages to disks. > **NOTE** > > StarRocks supports access to Kafka via security protocols including SASL\_SSL, SAS\_PLAINTEXT, SSL, and PLAINTEXT. This topic uses connecting to Kafka via PLAINTEXT as an example. If you need to connect to Kafka via other security protocols, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). 4. **The FE generates new load tasks to load data continuously.** After the Executor BEs has written the data to disks, the Coordinator BE reports the result of the load task to the FE. Based on the result, the FE then generates new load tasks to load the data continuously. Or the FE retries the failed tasks to make sure the data loaded into StarRocks is neither lost nor duplicated. #### Create a Routine Load job[​](#create-a-routine-load-job "Direct link to Create a Routine Load job") The following three examples describe how to consume CSV-format, JSON-format and Avro-format data in Kafka, and load the data into StarRocks by creating a Routine Load job. For detailed syntax and parameter descriptions, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### Load CSV-format data[​](#load-csv-format-data "Direct link to Load CSV-format data") This section describes how to create a Routine Load job to consume CSV-format data in a Kafka cluster, and load the data into StarRocks. ###### Prepare a dataset[​](#prepare-a-dataset "Direct link to Prepare a dataset") Suppose there is a CSV-format dataset in the topic `ordertest1` in a Kafka cluster. Every message in the dataset includes six fields: order ID, payment date, customer name, nationality, gender, and price. ```plain 2020050802,2020-05-08,Johann Georg Faust,Deutschland,male,895 2020050802,2020-05-08,Julien Sorel,France,male,893 2020050803,2020-05-08,Dorian Grey,UK,male,1262 2020050901,2020-05-09,Anna Karenina",Russia,female,175 2020051001,2020-05-10,Tess Durbeyfield,US,female,986 2020051101,2020-05-11,Edogawa Conan,japan,male,8924 ``` ###### Create a table[​](#create-a-table "Direct link to Create a table") According to the fields of CSV-format data, create the table `example_tbl1` in the database `example_db`. The following example creates a table with 5 fields excluding the field of customer gender in the CSV-format data. ```sql CREATE TABLE example_db.example_tbl1 ( `order_id` bigint NOT NULL COMMENT "Order ID", `pay_dt` date NOT NULL COMMENT "Payment date", `customer_name` varchar(26) NULL COMMENT "Customer name", `nationality` varchar(26) NULL COMMENT "Nationality", `price`double NULL COMMENT "Price" ) ENGINE=OLAP DUPLICATE KEY (order_id,pay_dt) DISTRIBUTED BY HASH(`order_id`); ``` > **NOTICE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). ###### Submit a Routine Load job[​](#submit-a-routine-load-job "Direct link to Submit a Routine Load job") Execute the following statement to submit a Routine Load job named `example_tbl1_ordertest1` to consume the messages in the topic `ordertest1` and load the data into the table `example_tbl1`. The load task consumes the messages from the initial offset in the specified partitions of the topic. ```sql CREATE ROUTINE LOAD example_db.example_tbl1_ordertest1 ON example_tbl1 COLUMNS TERMINATED BY ",", COLUMNS (order_id, pay_dt, customer_name, nationality, temp_gender, price) PROPERTIES ( "desired_concurrent_number" = "5" ) FROM KAFKA ( "kafka_broker_list" = ":,:", "kafka_topic" = "ordertest1", "kafka_partitions" = "0,1,2,3,4", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` After submitting the load job, you can execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job. * **load job name** There could be multiple load job on a table. Therefore, we recommend you name a load job with the corresponding Kafka topic and the time when the load job is submitted. It helps you distinguish the load job on each table. * **Column separator** The property `COLUMN TERMINATED BY` defines the column separator of the CSV-format data. The default is `\t`. * **Kafka topic partition and offset** You can specify the properties `kafka_partitions` and `kafka_offsets` to specify the partitions and offsets to consume the messages. For example, if you want the load job to consume messages from the Kafka partitions `"0,1,2,3,4"` of the topic `ordertest1` all with the initial offsets, you can specify the properties as follows: If you want the load job to consume messages from the Kafka partitions `"0,1,2,3,4"`and you need to specify a separate starting offset for each partition, you can configure as follows: ```sql "kafka_partitions" ="0,1,2,3,4", "kafka_offsets" = "OFFSET_BEGINNING, OFFSET_END, 1000, 2000, 3000" ``` You can also set the default offsets of all partitions with the property `property.kafka_default_offsets`. ```sql "kafka_partitions" ="0,1,2,3,4", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ``` For detailed information, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). * **Data mapping and transformation** To specify the mapping and transformation relationship between the CSV-format data, and the StarRocks table, you need to use the `COLUMNS` parameter. **Data mapping:** * StarRocks extracts the columns in the CSV-format data and maps them **in sequence** onto the fields declared in the `COLUMNS` parameter. * StarRocks extracts the fields declared in the `COLUMNS` parameter and maps them **by name** onto the columns of StarRocks table. **Data transformation:** And because the example excludes the column of customer gender from the CSV-format data, the field `temp_gender` in `COLUMNS` parameter is used as a placeholder for this field. The other fields are mapped to columns of the StarRocks table `example_tbl1` directly. For more information about data transformation, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md). > **NOTE** > > You do not need to specify the `COLUMNS` parameter if the names, number, and order of the columns in the CSV-format data completely correspond to those of the StarRocks table. * **Task concurrency** When there are many Kafka topic partitions and enough BE nodes, you can accelerate the loading by increasing the task concurrency. To increase the actual load task concurrency, you can increase the desired load task concurrency `desired_concurrent_number` when you create a routine load job. You can also set the dynamic configuration item of FE `max_routine_load_task_concurrent_num` ( default maximum load task currency ) to a larger value. For more information about `max_routine_load_task_concurrent_num`, please see [FE configuration items](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md). The actual task concurrency is defined by the minimum value among the number of BE nodes that are alive, the number of the pre-specified Kafka topic partitions, and the values of `desired_concurrent_number` and `max_routine_load_task_concurrent_num`. In the example, the number of BE nodes that are alive is `5`, the number of the pre-specified Kafka topic partitions is `5`, and the value of `max_routine_load_task_concurrent_num` is `5`. To increase the actual load task concurrency, you can increase the `desired_concurrent_number` from the default value `3` to `5`. For more about the properties, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### Load JSON-format data[​](#load-json-format-data "Direct link to Load JSON-format data") This section describes how to create a Routine Load job to consume JSON-format data in a Kafka cluster, and load the data into StarRocks. ###### Prepare a dataset[​](#prepare-a-dataset-1 "Direct link to Prepare a dataset") Suppose there is a JSON-format dataset in the topic `ordertest2` in a Kafka cluster. The dataset includes six keys: commodity ID, customer name, nationality, payment time, and price. Besides, you want to transform the payment time column into the DATE type, and load it into the `pay_dt` column in the StarRocks table. ```json {"commodity_id": "1", "customer_name": "Mark Twain", "country": "US","pay_time": 1589191487,"price": 875} {"commodity_id": "2", "customer_name": "Oscar Wilde", "country": "UK","pay_time": 1589191487,"price": 895} {"commodity_id": "3", "customer_name": "Antoine de Saint-Exupéry","country": "France","pay_time": 1589191487,"price": 895} ``` > **CAUTION** Each JSON object in a row must be in one Kafka message, otherwise a JSON parsing error is returned. ###### Create a table[​](#create-a-table-1 "Direct link to Create a table") According to the keys of the JSON-format data, create the table `example_tbl2` in the database `example_db`. ```sql CREATE TABLE `example_tbl2` ( `commodity_id` varchar(26) NULL COMMENT "Commodity ID", `customer_name` varchar(26) NULL COMMENT "Customer name", `country` varchar(26) NULL COMMENT "Country", `pay_time` bigint(20) NULL COMMENT "Payment time", `pay_dt` date NULL COMMENT "Payment date", `price`double SUM NULL COMMENT "Price" ) ENGINE=OLAP AGGREGATE KEY(`commodity_id`,`customer_name`,`country`,`pay_time`,`pay_dt`) DISTRIBUTED BY HASH(`commodity_id`); ``` > **NOTICE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). ###### Submit a Routine Load job[​](#submit-a-routine-load-job-1 "Direct link to Submit a Routine Load job") Execute the following statement to submit a Routine Load job named `example_tbl2_ordertest2` to consume the messages in the topic `ordertest2` and load the data into the table `example_tbl2`. The load task consumes the messages from the initial offset in the specified partitions of the topic. ```sql CREATE ROUTINE LOAD example_db.example_tbl2_ordertest2 ON example_tbl2 COLUMNS(commodity_id, customer_name, country, pay_time, price, pay_dt=from_unixtime(pay_time, '%Y%m%d')) PROPERTIES ( "desired_concurrent_number" = "5", "format" = "json", "jsonpaths" = "[\"$.commodity_id\",\"$.customer_name\",\"$.country\",\"$.pay_time\",\"$.price\"]" ) FROM KAFKA ( "kafka_broker_list" =":,:", "kafka_topic" = "ordertest2", "kafka_partitions" ="0,1,2,3,4", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` After submitting the load job, you can execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job. * **Data format** You need to specify `"format" = "json"` in the clause `PROPERTIES` to define that the data format is JSON. * **Data mapping and transformation** To specify the mapping and transformation relationship between the JSON-format data, and the StarRocks table, you need to specify the parameter `COLUMNS` and property`jsonpaths`. The order of fields specified in the `COLUMNS` parameter must match that of the JSON-format data, and the name of fields must match that of the StarRocks table. The property `jsonpaths` is used to extract the required fields from the JSON data. These fields are then named by the property `COLUMNS`. Because the example needs to transform the payment time field to the DATE data type, and load the data into the `pay_dt` column in the StarRocks table, you need to use the from\_unixtime function. The other fields are mapped to fields of the table `example_tbl2` directly. **Data mapping:** * StarRocks extracts the `name` and `code` keys of JSON-format data and maps them onto the keys declared in the `jsonpaths` property. * StarRocks extracts the keys declared in the `jsonpaths` property and maps them **in sequence** onto the fields declared in the `COLUMNS` parameter. * StarRocks extracts the fields declared in the `COLUMNS` parameter and maps them **by name** onto the columns of StarRocks table. **Data transformation**: * Because the example needs to transform the key `pay_time` to the DATE data type, and load the data into the `pay_dt` column in the StarRocks table, you need to use the from\_unixtime function in `COLUMNS` parameter. The other fields are mapped to fields of the table `example_tbl2` directly. * And because the example excludes the column of customer gender from the JSON-format data, the field `temp_gender` in `COLUMNS` parameter is used as a placeholder for this field. The other fields are mapped to columns of the StarRocks table `example_tbl1` directly. For more information about data transformation, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md). > **NOTE** > > You do not need to specify the `COLUMNS` parameter if the names and number of the keys in the JSON object completely match those of fields in the StarRocks table. ##### Load Avro-format data[​](#load-avro-format-data "Direct link to Load Avro-format data") Since v3.0.1, StarRocks supports loading Avro data by using Routine Load. ###### Prepare a dataset[​](#prepare-a-dataset-2 "Direct link to Prepare a dataset") ###### Avro schema[​](#avro-schema "Direct link to Avro schema") 1. Create the following Avro schema file `avro_schema.avsc`: ```json { "type": "record", "name": "sensor_log", "fields" : [ {"name": "id", "type": "long"}, {"name": "name", "type": "string"}, {"name": "checked", "type" : "boolean"}, {"name": "data", "type": "double"}, {"name": "sensor_type", "type": {"type": "enum", "name": "sensor_type_enum", "symbols" : ["TEMPERATURE", "HUMIDITY", "AIR-PRESSURE"]}} ] } ``` 2. Register the Avro schema in the [Schema Registry](https://docs.confluent.io/cloud/current/get-started/schema-registry.html#create-a-schema). ###### Avro data[​](#avro-data "Direct link to Avro data") Prepare the Avro data and send it to the Kafka topic `topic_0`. ###### Create a table[​](#create-a-table-2 "Direct link to Create a table") According to the fields of Avro data, create a table `sensor_log` in the target database `example_db` in the StarRocks cluster. The column names of the table must match the field names in the Avro data. For the data type mapping between the table columns and the Avro data fields, see \[Data types mapping]\(#Data types mapping). ```sql CREATE TABLE example_db.sensor_log ( `id` bigint NOT NULL COMMENT "sensor id", `name` varchar(26) NOT NULL COMMENT "sensor name", `checked` boolean NOT NULL COMMENT "checked", `data` double NULL COMMENT "sensor data", `sensor_type` varchar(26) NOT NULL COMMENT "sensor type" ) ENGINE=OLAP DUPLICATE KEY (id) DISTRIBUTED BY HASH(`id`); ``` > **NOTICE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). ###### Submit a Routine Load job[​](#submit-a-routine-load-job-2 "Direct link to Submit a Routine Load job") Execute the following statement to submit a Routine Load job named `sensor_log_load_job` to consume the Avro messages in the Kafka topic `topic_0` and load the data into the table `sensor_log` in the database `sensor`. The load job consumes the messages from the initial offset in the specified partitions of the topic. ```sql CREATE ROUTINE LOAD example_db.sensor_log_load_job ON sensor_log PROPERTIES ( "format" = "avro" ) FROM KAFKA ( "kafka_broker_list" = ":,:,...", "confluent.schema.registry.url" = "http://172.xx.xxx.xxx:8081", "kafka_topic" = "topic_0", "kafka_partitions" = "0,1,2,3,4,5", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` * Data Format You need to specify `"format = "avro"` in the clause `PROPERTIES` to define that the data format is Avro. * Schema Registry You need to configure `confluent.schema.registry.url` to specify the URL of the Schema Registry where the Avro schema is registered. StarRocks retrieves the Avro schema by using this URL. The format is as follows: ```plaintext confluent.schema.registry.url = http[s]://[:@][:] ``` * Data mapping and transformation To specify the mapping and transformation relationship between the Avro-format data and the StarRocks table, you need to specify the parameter `COLUMNS` and property `jsonpaths`. The order of fields specified in the `COLUMNS` parameter must match that of the fields in the property `jsonpaths`, and the names of fields must match these of the StarRocks table. The property `jsonpaths` is used to extract the required fields from the Avro data. These fields are then named by the property `COLUMNS`. For more information about data transformation, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md). > NOTE > > You do not need to specify the `COLUMNS` parameter if the names and number of the fields in the Avro record completely match those of columns in the StarRocks table. After submitting the load job, you can execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job. ###### Data types mapping[​](#data-types-mapping "Direct link to Data types mapping") The data type mapping between the Avro data fields you want to load and the StarRocks table columns is as follows: ###### Primitive types[​](#primitive-types "Direct link to Primitive types") | Avro | StarRocks | | ------- | --------- | | nul | NULL | | boolean | BOOLEAN | | int | INT | | long | BIGINT | | float | FLOAT | | double | DOUBLE | | bytes | STRING | | string | STRING | ###### Complex types[​](#complex-types "Direct link to Complex types") | Avro | StarRocks | | -------------- | -------------------------------------------------------------------------- | | record | STRUCT, or load the entire RECORD or its subfields into StarRocks as JSON. | | enums | STRING | | arrays | ARRAY | | maps | MAP or JSON | | union(T, null) | NULLABLE(T) | | fixed | STRING | ###### Limits[​](#limits "Direct link to Limits") * Currently, StarRocks does not support schema evolution. * Each Kafka message must only contain a single Avro data record. ##### Access source message metadata[​](#access-source-message-metadata "Direct link to Access source message metadata") When loading JSON- or Avro-format data, you can populate destination columns from a message's Kafka/Pulsar metadata — topic, partition, offset, timestamp, key, and headers — instead of from the message payload. You declare the metadata in an `INCLUDE METADATA (...)` clause that binds each metadata key to an alias; the alias is an ordinary source column you reference from `COLUMNS`. This is useful for auditing (which topic/partition/offset a row came from), event-time processing (using the message timestamp), and routing on a header value. ###### Syntax[​](#syntax "Direct link to Syntax") ```sql INCLUDE METADATA ( [AS ] [, [AS ] ...] ) ``` `INCLUDE METADATA` is a load property; place it among the other load properties (such as `COLUMNS` and `WHERE`), in any order, before the `PROPERTIES` and `FROM` clauses. `AS ` is optional. If it is omitted, the alias defaults to ``. The alias must be unique within the clause, and it must not collide with a payload field, a destination-table column, or a reserved column name. ###### Metadata keys[​](#metadata-keys "Direct link to Metadata keys") The supported keys depend on the data source. | Source | Key | Type | Description | | ------ | ----------------- | ---------------------- | ------------------------------------------------------------------------------------------------- | | KAFKA | `TOPIC` | VARCHAR | Topic name. | | KAFKA | `PARTITION` | INT | Partition number. | | KAFKA | `OFFSET` | BIGINT | Message offset within the partition. | | KAFKA | `TIMESTAMP_MS` | BIGINT | Record timestamp in milliseconds since epoch. `NULL` when the broker reports no timestamp. | | KAFKA | `KEY` | VARCHAR | Message key as raw bytes. `NULL` when the message has no key. | | KAFKA | `HEADERS` | MAP\ | All headers as a map. On duplicate keys, the last value wins. | | PULSAR | `TOPIC` | VARCHAR | The logical topic the job consumes (a partitioned topic's `-partition-N` suffix is not included). | | PULSAR | `PARTITION` | INT | Partition index, parsed from the per-message topic name. `NULL` for a non-partitioned topic. | | PULSAR | `KEY` | VARCHAR | Partition key. `NULL` when the message has no key. | | PULSAR | `MESSAGE_ID` | VARCHAR | Message ID. | | PULSAR | `PUBLISH_TIME_MS` | BIGINT | Publish time in milliseconds since epoch. | | PULSAR | `EVENT_TIME_MS` | BIGINT | Event time in milliseconds since epoch. `NULL` when the producer did not set it. | | PULSAR | `PROPERTIES` | MAP\ | All properties as a map. On duplicate keys, the last value wins. | To read a single header/property value, use `element_at(, '')` over the `HEADERS`/`PROPERTIES` map: the last value wins on a duplicate key, and the result is `NULL` when the key is absent. Header/property values are raw bytes placed into VARCHAR as-is (no UTF-8 validation). ###### Usage notes[​](#usage-notes "Direct link to Usage notes") * `INCLUDE METADATA` is available for `format = json` (Kafka and Pulsar) and `format = avro` (Kafka only; Pulsar Routine Load does not support Avro). It is not supported for CSV, where one message can expand into many rows and per-message metadata would be ambiguous. * A metadata alias is an ordinary source column: reference it anywhere in the `COLUMNS` expressions. If a payload field has the same name as a metadata key, specify a different alias with `AS ` to avoid ambiguity. * `OFFSET` is Kafka-only and `MESSAGE_ID` is Pulsar-only; using a key unsupported by the source raises an error that lists the keys supported for that source. * `HEADERS`/`PROPERTIES` is a `MAP\`. The source headers/properties are an ordered list that may repeat a key; duplicates collapse into the map with the last value winning (an `element_at(map, 'name')` lookup is likewise last-wins, and returns `NULL` when the key is absent). Values are raw bytes stored in VARCHAR as-is — there is no UTF-8 validation or decoding. ###### Example[​](#example "Direct link to Example") Load the order payload field `order_id` together with the source topic, partition, offset, the message timestamp converted to a `DATETIME`, and a `trace-id` header: ```sql CREATE TABLE example_db.orders_with_meta ( order_id BIGINT, src_topic VARCHAR(256), src_partition INT, src_offset BIGINT, msg_time DATETIME, trace_id VARCHAR(128) ) ENGINE = OLAP DUPLICATE KEY(order_id) DISTRIBUTED BY HASH(order_id); CREATE ROUTINE LOAD example_db.orders_with_meta_job ON orders_with_meta INCLUDE METADATA ( TOPIC AS m_topic, PARTITION AS m_partition, OFFSET AS m_offset, TIMESTAMP_MS AS m_timestamp, HEADERS AS m_headers ), COLUMNS ( order_id, src_topic = m_topic, src_partition = m_partition, src_offset = m_offset, msg_time = from_unixtime(m_timestamp / 1000), trace_id = element_at(m_headers, 'trace-id') ) PROPERTIES ( "format" = "json", "jsonpaths" = "[\"$.order_id\"]" ) FROM KAFKA ( "kafka_broker_list" = ":,...", "kafka_topic" = "topic_orders", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` A metadata alias may be used inside an expression (as with `from_unixtime(m_timestamp / 1000)` above); only payload columns are listed in `jsonpaths`. #### Check a load job and task[​](#check-a-load-job-and-task "Direct link to Check a load job and task") ##### Check a load job[​](#check-a-load-job "Direct link to Check a load job") Execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job `example_tbl2_ordertest2`. StarRocks returns the execution state `State`, the statistical information (including the total rows consumed and the total rows loaded) `Statistics`, and the progress of the load job `progress`. If the state of the load job is automatically changed to **PAUSED**, it is possibly because the number of error rows has exceeded the threshold. For detailed instructions on setting this threshold, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). You can check the files `ReasonOfStateChanged` and `ErrorLogUrls` to identify and troubleshoot the problem. Having fixed the problem, you can then execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume the **PAUSED** load job. If the state of the load job is **CANCELLED**, it is possibly because the load job encounters an exception (such as the table has been dropped). You can check the files `ReasonOfStateChanged` and `ErrorLogUrls` to identify and troubleshoot the problem. However, you cannot resume a **CANCELLED** load job. ```sql MySQL [example_db]> SHOW ROUTINE LOAD FOR example_tbl2_ordertest2 \G *************************** 1. row *************************** Id: 63013 Name: example_tbl2_ordertest2 CreateTime: 2022-08-10 17:09:00 PauseTime: NULL EndTime: NULL DbName: default_cluster:example_db TableName: example_tbl2 State: RUNNING DataSourceType: KAFKA CurrentTaskNum: 3 JobProperties: {"partitions":"*","partial_update":"false","columnToColumnExpr":"commodity_id,customer_name,country,pay_time,pay_dt=from_unixtime(`pay_time`, '%Y%m%d'),price","maxBatchIntervalS":"20","whereExpr":"*","dataFormat":"json","timezone":"Asia/Shanghai","format":"json","json_root":"","strict_mode":"false","jsonpaths":"[\"$.commodity_id\",\"$.customer_name\",\"$.country\",\"$.pay_time\",\"$.price\"]","desireTaskConcurrentNum":"3","maxErrorNum":"0","strip_outer_array":"false","currentTaskConcurrentNum":"3","maxBatchRows":"200000"} DataSourceProperties: {"topic":"ordertest2","currentKafkaPartitions":"0,1,2,3,4","brokerList":":,:"} CustomProperties: {"kafka_default_offsets":"OFFSET_BEGINNING"} Statistic: {"receivedBytes":230,"errorRows":0,"committedTaskNum":1,"loadedRows":2,"loadRowsRate":0,"abortedTaskNum":0,"totalRows":2,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":522} Progress: {"0":"1","1":"OFFSET_ZERO","2":"OFFSET_ZERO","3":"OFFSET_ZERO","4":"OFFSET_ZERO"} ReasonOfStateChanged: ErrorLogUrls: OtherMsg: ``` > **CAUTION** > > You cannot check a load job that has stopped or has not yet started. ##### Check a load task[​](#check-a-load-task "Direct link to Check a load task") Execute the [SHOW ROUTINE LOAD TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD_TASK.md) statement to check the load tasks of the load job `example_tbl2_ordertest2`, such as how many tasks are currently running, the Kafka topic partitions that are consumed and the consumption progress `DataSourceProperties`, and the corresponding Coordinator BE node `BeId`. ```sql MySQL [example_db]> SHOW ROUTINE LOAD TASK WHERE JobName = "example_tbl2_ordertest2" \G *************************** 1. row *************************** TaskId: 18c3a823-d73e-4a64-b9cb-b9eced026753 TxnId: -1 TxnStatus: UNKNOWN JobId: 63013 CreateTime: 2022-08-10 17:09:05 LastScheduledTime: 2022-08-10 17:47:27 ExecuteStartTime: NULL Timeout: 60 BeId: -1 DataSourceProperties: {"1":0,"4":0} Message: there is no new data in kafka, wait for 20 seconds to schedule again *************************** 2. row *************************** TaskId: f76c97ac-26aa-4b41-8194-a8ba2063eb00 TxnId: -1 TxnStatus: UNKNOWN JobId: 63013 CreateTime: 2022-08-10 17:09:05 LastScheduledTime: 2022-08-10 17:47:26 ExecuteStartTime: NULL Timeout: 60 BeId: -1 DataSourceProperties: {"2":0} Message: there is no new data in kafka, wait for 20 seconds to schedule again *************************** 3. row *************************** TaskId: 1a327a34-99f4-4f8d-8014-3cd38db99ec6 TxnId: -1 TxnStatus: UNKNOWN JobId: 63013 CreateTime: 2022-08-10 17:09:26 LastScheduledTime: 2022-08-10 17:47:27 ExecuteStartTime: NULL Timeout: 60 BeId: -1 DataSourceProperties: {"0":2,"3":0} Message: there is no new data in kafka, wait for 20 seconds to schedule again ``` #### Pause a load job[​](#pause-a-load-job "Direct link to Pause a load job") You can execute the [PAUSE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/PAUSE_ROUTINE_LOAD.md) statement to pause a load job. The state of the load job will be **PAUSED** after the statement is executed. However, it has not stopped. You can execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume it. You can also check its status with the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. The following example pauses the load job `example_tbl2_ordertest2`: ```sql PAUSE ROUTINE LOAD FOR example_tbl2_ordertest2; ``` #### Resume a load job[​](#resume-a-load-job "Direct link to Resume a load job") You can execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume a paused load job. The state of the load job will be **NEED\_SCHEDULE** temporarily (because the load job is being re-scheduled), and then become **RUNNING**. You can check its status with the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. The following example resumes the paused load job `example_tbl2_ordertest2`: ```sql RESUME ROUTINE LOAD FOR example_tbl2_ordertest2; ``` #### Alter a load job[​](#alter-a-load-job "Direct link to Alter a load job") Before altering a load job, you must pause it with the [PAUSE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/PAUSE_ROUTINE_LOAD.md) statement. Then you can execute the [ALTER ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/ALTER_ROUTINE_LOAD.md). After altering it, you can execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume it, and check its status with the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. Suppose the number of the BE nodes that are alive increases to `6` and the Kafka topic partitions to be consumed is `"0,1,2,3,4,5,6,7"`. If you want to increase the actual load task concurrency, you can execute the following statement to increase the number of desired task concurrency `desired_concurrent_number` to `6` (greater than or equal to the number of BE nodes that are alive), and specify the Kafka topic partitions and initial offsets. > **NOTE** > > Because the actual task concurrency is determined by the minimum value of multiple parameters, you must make sure that the value of the FE dynamic parameter `max_routine_load_task_concurrent_num` is greater than or equal to `6`. ```sql ALTER ROUTINE LOAD FOR example_tbl2_ordertest2 PROPERTIES ( "desired_concurrent_number" = "6" ) FROM kafka ( "kafka_partitions" = "0,1,2,3,4,5,6,7", "kafka_offsets" = "OFFSET_BEGINNING,OFFSET_BEGINNING,OFFSET_BEGINNING,OFFSET_BEGINNING,OFFSET_END,OFFSET_END,OFFSET_END,OFFSET_END" ); ``` #### Stop a load job[​](#stop-a-load-job "Direct link to Stop a load job") You can execute the [STOP ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/STOP_ROUTINE_LOAD.md) statement to stop a load job. The state of the load job will be **STOPPED** after the statement is executed, and you cannot resume a stopped load job. You cannot check the status of a stopped load job with the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. The following example stops the load job `example_tbl2_ordertest2`: ```sql STOP ROUTINE LOAD FOR example_tbl2_ordertest2; ``` --- ### Stream with StarRocks Pipe #### Advantages of Pipe[​](#advantages-of-pipe "Direct link to Advantages of Pipe") Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading.During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that object storage like AWS S3 uses ETag as file digest, and HDFS uses LastModifiedTime as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Data flow[​](#data-flow "Direct link to Data flow") ![Pipe data flow](/assets/images/pipe_data_flow-2a4dc0b44a06c987d9afc0ecf632f5d9.png) #### [📄️ HDFS](https://docs.starrocks.io/docs/integrations/streaming/pipe/hdfs.md) [Pipe example using loading from HDFS](https://docs.starrocks.io/docs/integrations/streaming/pipe/hdfs.md) #### [📄️ S3](https://docs.starrocks.io/docs/integrations/streaming/pipe/s3.md) [Pipe example loading from S3](https://docs.starrocks.io/docs/integrations/streaming/pipe/s3.md) --- ### HDFS Load ### Load data from HDFS StarRocks provides the following options for loading data from HDFS: * Synchronous loading using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md)+[`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) * Asynchronous loading using [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) * Continuous asynchronous loading using [Pipe](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md) Each of these options has its own advantages, which are detailed in the following sections. In most cases, we recommend that you use the INSERT+`FILES()` method, which is much easier to use. However, the INSERT+`FILES()` method currently supports only the Parquet, ORC, and CSV file formats. Therefore, if you need to load data of other file formats such as JSON, or perform data changes such as DELETE during data loading, you can resort to Broker Load. If you need to load a large number of data files with a significant data volume in total (for example, more than 100 GB or even 1 TB), we recommend that you use the Pipe method. Pipe can split the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job and minimizes the need for retries due to data errors. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Make source data ready[​](#make-source-data-ready "Direct link to Make source data ready") Make sure the source data you want to load into StarRocks is properly stored in your HDFS cluster. This topic assumes that you want to load `/user/amber/user_behavior_ten_million_rows.parquet` from HDFS into StarRocks. ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ##### Gather authentication details[​](#gather-authentication-details "Direct link to Gather authentication details") You can use the simple authentication method to establish connections with your HDFS cluster. To use simple authentication, you need to gather the username and password of the account that you can use to access the NameNode of the HDFS cluster. #### Use INSERT+FILES()[​](#use-insertfiles "Direct link to Use INSERT+FILES()") This method is available from v3.1 onwards and currently supports only the Parquet, ORC, and CSV (from v3.3.0 onwards) file formats. ##### Advantages of INSERT+FILES()[​](#advantages-of-insertfiles "Direct link to Advantages of INSERT+FILES()") [`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) can read the file stored in cloud storage based on the path-related properties you specify, infer the table schema of the data in the file, and then return the data from the file as data rows. With `FILES()`, you can: * Query the data directly from HDFS using [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). * Create and load a table using [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) (CTAS). * Load the data into an existing table using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). ##### Typical examples[​](#typical-examples "Direct link to Typical examples") ###### Querying directly from HDFS using SELECT[​](#querying-directly-from-hdfs-using-select "Direct link to Querying directly from HDFS using SELECT") Querying directly from HDFS using SELECT+`FILES()` can give a good preview of the content of a dataset before you create a table. For example: * Get a preview of the dataset without storing the data. * Query for the min and max values and decide what data types to use. * Check for `NULL` values. The following example queries the data file `/user/amber/user_behavior_ten_million_rows.parquet` stored in the HDFS cluster: ```sql SELECT * FROM FILES ( "path" = "hdfs://:/user/amber/user_behavior_ten_million_rows.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ) LIMIT 3; ``` The system returns the following query result: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 543711 | 829192 | 2355072 | pv | 2017-11-27 08:22:37 | | 543711 | 2056618 | 3645362 | pv | 2017-11-27 10:16:46 | | 543711 | 1165492 | 3645362 | pv | 2017-11-27 10:17:00 | +--------+---------+------------+--------------+---------------------+ ``` > **NOTE** > > Notice that the column names as returned above are provided by the Parquet file. ###### Creating and loading a table using CTAS[​](#creating-and-loading-a-table-using-ctas "Direct link to Creating and loading a table using CTAS") This is a continuation of the previous example. The previous query is wrapped in CREATE TABLE AS SELECT (CTAS) to automate the table creation using schema inference. This means StarRocks will infer the table schema, create the table you want, and then load the data into the table. The column names and types are not required to create a table when using the `FILES()` table function with Parquet files as the Parquet format includes the column names. > **NOTE** > > The syntax of CREATE TABLE when using schema inference does not allow setting the number of replicas, so set it before creating the table. The example below is for a system with three replicas: > > ```sql > ADMIN SET FRONTEND CONFIG ('default_replication_num' = "3"); > > ``` Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Use CTAS to create a table and load the data of the data file `/user/amber/user_behavior_ten_million_rows.parquet` into the table: ```sql CREATE TABLE user_behavior_inferred AS SELECT * FROM FILES ( "path" = "hdfs://:/user/amber/user_behavior_ten_million_rows.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ); ``` After creating the table, you can view its schema by using [DESCRIBE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md): ```sql DESCRIBE user_behavior_inferred; ``` The system returns the following query result: ```plain +--------------+-----------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+-----------+------+-------+---------+-------+ | UserID | bigint | YES | true | NULL | | | ItemID | bigint | YES | true | NULL | | | CategoryID | bigint | YES | true | NULL | | | BehaviorType | varbinary | YES | false | NULL | | | Timestamp | varbinary | YES | false | NULL | | +--------------+-----------+------+-------+---------+-------+ ``` Query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_inferred LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+--------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+--------+------------+--------------+---------------------+ | 84 | 56257 | 1879194 | pv | 2017-11-26 05:56:23 | | 84 | 108021 | 2982027 | pv | 2017-12-02 05:43:00 | | 84 | 390657 | 1879194 | pv | 2017-11-28 11:20:30 | +--------+--------+------------+--------------+---------------------+ ``` ###### Loading into an existing table using INSERT[​](#loading-into-an-existing-table-using-insert "Direct link to Loading into an existing table using INSERT") You may want to customize the table that you are inserting into, for example, the: * column data type, nullable setting, or default values * key types and columns * data partitioning and bucketing > **NOTE** > > Creating the most efficient table structure requires knowledge of how the data will be used and the content of the columns. This topic does not cover table design. For information about table design, see [Table types](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md). In this example, we are creating a table based on knowledge of how the table will be queried and the data in the Parquet file. The knowledge of the data in the Parquet file can be gained by querying the file directly in HDFS. * Since a query of the dataset in HDFS indicates that the `Timestamp` column contains data that matches a VARBINARY data type, the column type is specified in the following DDL. * By querying the data in HDFS, you can find that there are no `NULL` values in the dataset, so the DDL does not set any columns as nullable. * Based on knowledge of the expected query types, the sort key and bucketing column are set to the column `UserID`. Your use case might be different for this data, so you might decide to use `ItemID` in addition to or instead of `UserID` for the sort key. Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from HDFS): ```sql CREATE TABLE user_behavior_declared ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` Display the schema so that you can compare it with the inferred schema produced by the `FILES()` table function: ```sql DESCRIBE user_behavior_declared; ``` ```plaintext +--------------+----------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+----------------+------+-------+---------+-------+ | UserID | int | NO | true | NULL | | | ItemID | int | NO | false | NULL | | | CategoryID | int | NO | false | NULL | | | BehaviorType | varchar(65533) | NO | false | NULL | | | Timestamp | varbinary | NO | false | NULL | | +--------------+----------------+------+-------+---------+-------+ 5 rows in set (0.00 sec) ``` tip Compare the schema you just created with the schema inferred earlier using the `FILES()` table function. Look at: * data types * nullable * key fields To better control the schema of the destination table and for better query performance, we recommend that you specify the table schema by hand in production environments. After creating the table, you can load it with INSERT INTO SELECT FROM FILES(): ```sql INSERT INTO user_behavior_declared SELECT * FROM FILES ( "path" = "hdfs://:/user/amber/user_behavior_ten_million_rows.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ); ``` After the load is complete, you can query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_declared LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 107 | 1568743 | 4476428 | pv | 2017-11-25 14:29:53 | | 107 | 470767 | 1020087 | pv | 2017-11-25 14:32:31 | | 107 | 358238 | 1817004 | pv | 2017-11-25 14:43:23 | +--------+---------+------------+--------------+---------------------+ ``` ###### Check load progress[​](#check-load-progress "Direct link to Check load progress") You can query the progress of INSERT jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. Example: ```sql SELECT * FROM information_schema.loads ORDER BY JOB_ID DESC; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'insert_0d86c3f9-851f-11ee-9c3e-00163e044958' \G *************************** 1. row *************************** JOB_ID: 10214 LABEL: insert_0d86c3f9-851f-11ee-9c3e-00163e044958 DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: INSERT PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):300; max_filter_ratio:0.0 CREATE_TIME: 2023-11-17 15:58:14 ETL_START_TIME: 2023-11-17 15:58:14 ETL_FINISH_TIME: 2023-11-17 15:58:14 LOAD_START_TIME: 2023-11-17 15:58:14 LOAD_FINISH_TIME: 2023-11-17 15:58:18 JOB_DETAILS: {"All backends":{"0d86c3f9-851f-11ee-9c3e-00163e044958":[10120]},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":311710786,"InternalTableLoadRows":10000000,"ScanBytes":581574034,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"0d86c3f9-851f-11ee-9c3e-00163e044958":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` > **NOTE** > > INSERT is a synchronous command. If an INSERT job is still running, you need to open another session to check its execution status. #### Use Broker Load[​](#use-broker-load "Direct link to Use Broker Load") An asynchronous Broker Load process handles making the connection to HDFS, pulling the data, and storing the data in StarRocks. This method supports the following file formats: * Parquet * ORC * CSV * JSON (supported from v3.2.3 onwards) ##### Advantages of Broker Load[​](#advantages-of-broker-load "Direct link to Advantages of Broker Load") * Broker Load runs in the background and clients do not need to stay connected for the job to continue. * Broker Load is preferred for long-running jobs, with the default timeout spanning 4 hours. * In addition to Parquet and ORC file format, Broker Load supports CSV file format and JSON file format (JSON file format is supported from v3.2.3 onwards). ##### Data flow[​](#data-flow "Direct link to Data flow") ![Workflow of Broker Load](/assets/images/broker_load_how-to-work_en-bb36de70866e6366b2b21808f0f77be8.png) 1. The user creates a load job. 2. The frontend (FE) creates a query plan and distributes the plan to the backend nodes (BEs) or compute nodes (CNs). 3. The BEs or CNs pull the data from the source and load the data into StarRocks. ##### Typical example[​](#typical-example "Direct link to Typical example") Create a table, start a load process that pulls the data file `/user/amber/user_behavior_ten_million_rows.parquet` from HDFS, and verify the progress and success of the data loading. ###### Create a database and a table[​](#create-a-database-and-a-table "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table has the same schema as the Parquet file that you want to load from HDFS): ```sql CREATE TABLE user_behavior ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Broker Load[​](#start-a-broker-load "Direct link to Start a Broker Load") Run the following command to start a Broker Load job that loads data from the data file `/user/amber/user_behavior_ten_million_rows.parquet` to the `user_behavior` table: ```sql LOAD LABEL user_behavior ( DATA INFILE("hdfs://:/user/amber/user_behavior_ten_million_rows.parquet") INTO TABLE user_behavior FORMAT AS "parquet" ) WITH BROKER ( "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ) PROPERTIES ( "timeout" = "72000" ); ``` This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. * `BROKER`: The connection details for the source. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Check load progress[​](#check-load-progress-1 "Direct link to Check load progress") You can query the progress of Broker Load jobs from the `information_schema.loads` view. This feature is supported from v3.1 onwards. ```sql SELECT * FROM information_schema.loads; ``` For information about the fields provided in the `loads` view, see [Information Schema](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md)). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'user_behavior'; ``` In the output below there are two entries for the load job `user_behavior`: * The first record shows a state of `CANCELLED`. Scroll to `ERROR_MSG`, and you can see that the job has failed due to `listPath failed`. * The second record shows a state of `FINISHED`, which means that the job has succeeded. ```plaintext JOB_ID|LABEL |DATABASE_NAME|STATE |PROGRESS |TYPE |PRIORITY|SCAN_ROWS|FILTERED_ROWS|UNSELECTED_ROWS|SINK_ROWS|ETL_INFO|TASK_INFO |CREATE_TIME |ETL_START_TIME |ETL_FINISH_TIME |LOAD_START_TIME |LOAD_FINISH_TIME |JOB_DETAILS |ERROR_MSG |TRACKING_URL|TRACKING_SQL|REJECTED_RECORD_PATH| ------+-------------------------------------------+-------------+---------+-------------------+------+--------+---------+-------------+---------------+---------+--------+----------------------------------------------------+-------------------+-------------------+-------------------+-------------------+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+------------+------------+--------------------+ 10121|user_behavior |mydatabase |CANCELLED|ETL:N/A; LOAD:N/A |BROKER|NORMAL | 0| 0| 0| 0| |resource:N/A; timeout(s):72000; max_filter_ratio:0.0|2023-08-10 14:59:30| | | |2023-08-10 14:59:34|{"All backends":{},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":0,"InternalTableLoadRows":0,"ScanBytes":0,"ScanRows":0,"TaskNumber":0,"Unfinished backends":{}} |type:ETL_RUN_FAIL; msg:listPath failed| | | | 10106|user_behavior |mydatabase |FINISHED |ETL:100%; LOAD:100%|BROKER|NORMAL | 86953525| 0| 0| 86953525| |resource:N/A; timeout(s):72000; max_filter_ratio:0.0|2023-08-10 14:50:15|2023-08-10 14:50:19|2023-08-10 14:50:19|2023-08-10 14:50:19|2023-08-10 14:55:10|{"All backends":{"a5fe5e1d-d7d0-4826-ba99-c7348f9a5f2f":[10004]},"FileNumber":1,"FileSize":1225637388,"InternalTableLoadBytes":2710603082,"InternalTableLoadRows":86953525,"ScanBytes":1225637388,"ScanRows":86953525,"TaskNumber":1,"Unfinished backends":{"a5| | | | | ``` After you confirm that the load job has finished, you can check a subset of the destination table to see if the data has been successfully loaded. Example: ```sql SELECT * from user_behavior LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 142 | 2869980 | 2939262 | pv | 2017-11-25 03:43:22 | | 142 | 2522236 | 1669167 | pv | 2017-11-25 15:14:12 | | 142 | 3031639 | 3607361 | pv | 2017-11-25 15:19:25 | +--------+---------+------------+--------------+---------------------+ ``` #### Use Pipe[​](#use-pipe "Direct link to Use Pipe") Starting from v3.2, StarRocks provides the Pipe loading method, which currently supports only the Parquet and ORC file formats. ##### Advantages of Pipe[​](#advantages-of-pipe "Direct link to Advantages of Pipe") Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading.During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that HDFS uses LastModifiedTime as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Data flow[​](#data-flow "Direct link to Data flow") ![Pipe data flow](/assets/images/pipe_data_flow-2a4dc0b44a06c987d9afc0ecf632f5d9.png) Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading. During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that HDFS uses `LastModifiedTime` as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Data flow[​](#data-flow-1 "Direct link to Data flow") ![Pipe data flow](/assets/images/pipe_data_flow-2a4dc0b44a06c987d9afc0ecf632f5d9.png) ##### Differences between Pipe and INSERT+FILES()[​](#differences-between-pipe-and-insertfiles "Direct link to Differences between Pipe and INSERT+FILES()") A Pipe job is split into one or more transactions based on the size and number of rows in each data file. Users can query the intermediate results during the loading process. In contrast, an INSERT+`FILES()` job is processed as a single transaction, and users are unable to view the data during the loading process. ##### File loading sequence[​](#file-loading-sequence "Direct link to File loading sequence") For each Pipe job, StarRocks maintains a file queue, from which it fetches and loads data files as micro-batches. Pipe does not ensure that the data files are loaded in the same order as they are uploaded. Therefore, newer data may be loaded prior to older data. ##### Typical example[​](#typical-example-1 "Direct link to Typical example") ###### Create a database and a table[​](#create-a-database-and-a-table-1 "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from HDFS): ```sql CREATE TABLE user_behavior_replica ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Pipe job[​](#start-a-pipe-job "Direct link to Start a Pipe job") Run the following command to start a Pipe job that loads data from the data file `/user/amber/user_behavior_ten_million_rows.parquet` to the `user_behavior_replica` table: ```sql CREATE PIPE user_behavior_replica PROPERTIES ( "AUTO_INGEST" = "TRUE" ) AS INSERT INTO user_behavior_replica SELECT * FROM FILES ( "path" = "hdfs://:/user/amber/user_behavior_ten_million_rows.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ); ``` This job has four main sections: * `pipe_name`: The name of the pipe. The pipe name must be unique within the database to which the pipe belongs. * `INSERT_SQL`: The INSERT INTO SELECT FROM FILES statement that is used to load data from the specified source data file to the destination table. * `PROPERTIES`: A set of optional parameters that specify how to execute the pipe. These include `AUTO_INGEST`, `POLL_INTERVAL`, `BATCH_SIZE`, and `BATCH_FILES`. Specify these properties in the `"key" = "value"` format. For detailed syntax and parameter descriptions, see [CREATE PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md). ###### Check load progress[​](#check-load-progress-2 "Direct link to Check load progress") * Query the progress of Pipe jobs by using [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md). ```sql SHOW PIPES; ``` If you have submitted multiple load jobs, you can filter on the `NAME` associated with the job. Example: ```sql SHOW PIPES WHERE NAME = 'user_behavior_replica' \G *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10252 PIPE_NAME: user_behavior_replica STATE: RUNNING TABLE_NAME: mydatabase.user_behavior_replica LOAD_STATUS: {"loadedFiles":1,"loadedBytes":132251298,"loadingFiles":0,"lastLoadedTime":"2023-11-17 16:13:22"} LAST_ERROR: NULL CREATED_TIME: 2023-11-17 16:13:15 1 row in set (0.00 sec) ``` * Query the progress of Pipe jobs from the [`pipes`](https://docs.starrocks.io/docs/sql-reference/information_schema/pipes.md) view in the StarRocks Information Schema. ```sql SELECT * FROM information_schema.pipes; ``` If you have submitted multiple load jobs, you can filter on the `PIPE_NAME` associated with the job. Example: ```sql SELECT * FROM information_schema.pipes WHERE pipe_name = 'user_behavior_replica' \G *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10252 PIPE_NAME: user_behavior_replica STATE: RUNNING TABLE_NAME: mydatabase.user_behavior_replica LOAD_STATUS: {"loadedFiles":1,"loadedBytes":132251298,"loadingFiles":0,"lastLoadedTime":"2023-11-17 16:13:22"} LAST_ERROR: CREATED_TIME: 2023-11-17 16:13:15 1 row in set (0.00 sec) ``` ###### Check file status[​](#check-file-status "Direct link to Check file status") You can query the load status of the files loaded from the [`pipe_files`](https://docs.starrocks.io/docs/sql-reference/information_schema/pipe_files.md) view in the StarRocks Information Schema. ```sql SELECT * FROM information_schema.pipe_files; ``` If you have submitted multiple load jobs, you can filter on the `PIPE_NAME` associated with the job. Example: ```sql SELECT * FROM information_schema.pipe_files WHERE pipe_name = 'user_behavior_replica' \G *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10252 PIPE_NAME: user_behavior_replica FILE_NAME: hdfs://172.26.195.67:9000/user/amber/user_behavior_ten_million_rows.parquet FILE_VERSION: 1700035418838 FILE_SIZE: 132251298 LAST_MODIFIED: 2023-11-15 08:03:38 LOAD_STATE: FINISHED STAGED_TIME: 2023-11-17 16:13:16 START_LOAD_TIME: 2023-11-17 16:13:17 FINISH_LOAD_TIME: 2023-11-17 16:13:22 ERROR_MSG: 1 row in set (0.02 sec) ``` ###### Manage Pipes[​](#manage-pipes "Direct link to Manage Pipes") You can alter, suspend or resume, drop, or query the pipes you have created and retry to load specific data files. For more information, see [ALTER PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/ALTER_PIPE.md), [SUSPEND or RESUME PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SUSPEND_or_RESUME_PIPE.md), [DROP PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/DROP_PIPE.md), [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md), and [RETRY FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/RETRY_FILE.md). --- ### S3 Load ### Load data from AWS S3 StarRocks provides the following options for loading data from AWS S3: * Synchronous loading using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md)+[`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) * Asynchronous loading using [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) * Continuous asynchronous loading using [Pipe](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md) Each of these options has its own advantages, which are detailed in the following sections. In most cases, we recommend that you use the INSERT+`FILES()` method, which is much easier to use. However, the INSERT+`FILES()` method currently supports only the Parquet, ORC, and CSV file formats. Therefore, if you need to load data of other file formats such as JSON, or perform data changes such as DELETE during data loading, you can resort to Broker Load. If you need to load a large number of data files with a significant data volume in total (for example, more than 100 GB or even 1 TB), we recommend that you use the Pipe method. Pipe can split the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job and minimizes the need for retries due to data errors. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Make source data ready[​](#make-source-data-ready "Direct link to Make source data ready") Make sure the source data you want to load into StarRocks is properly stored in an S3 bucket. You may also consider where the data and the database are located, because data transfer costs are much lower when your bucket and your StarRocks cluster are located in the same region. In this topic, we provide you with a sample dataset in an S3 bucket, `s3://starrocks-examples/user-behavior-10-million-rows.parquet`. You can access that dataset with any valid credentials as the object is readable by any AWS authenticated user. ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ##### Gather authentication details[​](#gather-authentication-details "Direct link to Gather authentication details") The examples in this topic use IAM user-based authentication. To ensure that you have permission to read data from AWS S3, we recommend that you read [Preparation for IAM user-based authentication](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md) and follow the instructions to create an IAM user with proper [IAM policies](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md) configured. In a nutshell, if you practice IAM user-based authentication, you need to gather information about the following AWS resources: * The S3 bucket that stores your data. * The S3 object key (object name) if accessing a specific object in the bucket. Note that the object key can include a prefix if your S3 objects are stored in sub-folders. * The AWS region to which the S3 bucket belongs. * The access key and secret key used as access credentials. For information about all the authentication methods available, see [Authenticate to AWS resources](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md). #### Use INSERT+FILES()[​](#use-insertfiles "Direct link to Use INSERT+FILES()") This method is available from v3.1 onwards and currently supports only the Parquet, ORC, and CSV (from v3.3.0 onwards) file formats. ##### Advantages of INSERT+FILES()[​](#advantages-of-insertfiles "Direct link to Advantages of INSERT+FILES()") [`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) can read the file stored in cloud storage based on the path-related properties you specify, infer the table schema of the data in the file, and then return the data from the file as data rows. With `FILES()`, you can: * Query the data directly from S3 using [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). * Create and load a table using [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) (CTAS). * Load the data into an existing table using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). ##### Typical examples[​](#typical-examples "Direct link to Typical examples") ###### Querying directly from S3 using SELECT[​](#querying-directly-from-s3-using-select "Direct link to Querying directly from S3 using SELECT") Querying directly from S3 using SELECT+`FILES()` can give a good preview of the content of a dataset before you create a table. For example: * Get a preview of the dataset without storing the data. * Query for the min and max values and decide what data types to use. * Check for `NULL` values. The following example queries the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows.parquet`: ```sql SELECT * FROM FILES ( "path" = "s3://starrocks-examples/user-behavior-10-million-rows.parquet", "format" = "parquet", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ) LIMIT 3; ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. The system returns the following query result: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 1 | 2576651 | 149192 | pv | 2017-11-25 01:21:25 | | 1 | 3830808 | 4181361 | pv | 2017-11-25 07:04:53 | | 1 | 4365585 | 2520377 | pv | 2017-11-25 07:49:06 | +--------+---------+------------+--------------+---------------------+ ``` > **NOTE** > > Notice that the column names as returned above are provided by the Parquet file. ###### Creating and loading a table using CTAS[​](#creating-and-loading-a-table-using-ctas "Direct link to Creating and loading a table using CTAS") This is a continuation of the previous example. The previous query is wrapped in CREATE TABLE AS SELECT (CTAS) to automate the table creation using schema inference. This means StarRocks will infer the table schema, create the table you want, and then load the data into the table. The column names and types are not required to create a table when using the `FILES()` table function with Parquet files as the Parquet format includes the column names. > **NOTE** > > The syntax of CREATE TABLE when using schema inference does not allow setting the number of replicas, so set it before creating the table. The example below is for a system with one replica: > > ```sql > ADMIN SET FRONTEND CONFIG ('default_replication_num' = "1"); > > ``` Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Use CTAS to create a table and load the data of the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows.parquet` into the table: ```sql CREATE TABLE user_behavior_inferred AS SELECT * FROM FILES ( "path" = "s3://starrocks-examples/user-behavior-10-million-rows.parquet", "format" = "parquet", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ); ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. After creating the table, you can view its schema by using [DESCRIBE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md): ```sql DESCRIBE user_behavior_inferred; ``` The system returns the following query result: ```plain +--------------+------------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+------------------+------+-------+---------+-------+ | UserID | bigint | YES | true | NULL | | | ItemID | bigint | YES | true | NULL | | | CategoryID | bigint | YES | true | NULL | | | BehaviorType | varchar(1048576) | YES | false | NULL | | | Timestamp | varchar(1048576) | YES | false | NULL | | +--------------+------------------+------+-------+---------+-------+ ``` Query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_inferred LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 225586 | 3694958 | 1040727 | pv | 2017-12-01 00:58:40 | | 225586 | 3726324 | 965809 | pv | 2017-12-01 02:16:02 | | 225586 | 3732495 | 1488813 | pv | 2017-12-01 00:59:46 | +--------+---------+------------+--------------+---------------------+ ``` ###### Loading into an existing table using INSERT[​](#loading-into-an-existing-table-using-insert "Direct link to Loading into an existing table using INSERT") You may want to customize the table that you are inserting into, for example, the: * column data type, nullable setting, or default values * key types and columns * data partitioning and bucketing > **NOTE** > > Creating the most efficient table structure requires knowledge of how the data will be used and the content of the columns. This topic does not cover table design. For information about table design, see [Table types](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md). In this example, we are creating a table based on knowledge of how the table will be queried and the data in the Parquet file. The knowledge of the data in the Parquet file can be gained by querying the file directly in S3. * Since a query of the dataset in S3 indicates that the `Timestamp` column contains data that matches a VARCHAR data type, and StarRocks can cast from VARCHAR to DATETIME, the data type is changed to DATETIME in the following DDL. * By querying the data in S3, you can find that there are no `NULL` values in the dataset, so the DDL could also set all columns as non-nullable. * Based on knowledge of the expected query types, the sort key and bucketing column are set to the column `UserID`. Your use case might be different for this data, so you might decide to use `ItemID` in addition to, or instead of, `UserID` for the sort key. Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand: ```sql CREATE TABLE user_behavior_declared ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp datetime ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` Display the schema so that you can compare it with the inferred schema produced by the `FILES()` table function: ```sql DESCRIBE user_behavior_declared; ``` ```plaintext +--------------+----------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+----------------+------+-------+---------+-------+ | UserID | int | YES | true | NULL | | | ItemID | int | YES | false | NULL | | | CategoryID | int | YES | false | NULL | | | BehaviorType | varchar(65533) | YES | false | NULL | | | Timestamp | datetime | YES | false | NULL | | +--------------+----------------+------+-------+---------+-------+ ``` tip Compare the schema you just created with the schema inferred earlier using the `FILES()` table function. Look at: * data types * nullable * key fields To better control the schema of the destination table and for better query performance, we recommend that you specify the table schema by hand in production environments. After creating the table, you can load it with INSERT INTO SELECT FROM FILES(): ```sql INSERT INTO user_behavior_declared SELECT * FROM FILES ( "path" = "s3://starrocks-examples/user-behavior-10-million-rows.parquet", "format" = "parquet", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ); ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. After the load is complete, you can query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_declared LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 393529 | 3715112 | 883960 | pv | 2017-12-02 02:45:44 | | 393529 | 2650583 | 883960 | pv | 2017-12-02 02:45:59 | | 393529 | 3715112 | 883960 | pv | 2017-12-02 03:00:56 | +--------+---------+------------+--------------+---------------------+ ``` ###### Check load progress[​](#check-load-progress "Direct link to Check load progress") You can query the progress of INSERT jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. Example: ```sql SELECT * FROM information_schema.loads ORDER BY JOB_ID DESC; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'insert_e3b882f5-7eb3-11ee-ae77-00163e267b60' \G *************************** 1. row *************************** JOB_ID: 10243 LABEL: insert_e3b882f5-7eb3-11ee-ae77-00163e267b60 DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: INSERT PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):300; max_filter_ratio:0.0 CREATE_TIME: 2023-11-09 11:56:01 ETL_START_TIME: 2023-11-09 11:56:01 ETL_FINISH_TIME: 2023-11-09 11:56:01 LOAD_START_TIME: 2023-11-09 11:56:01 LOAD_FINISH_TIME: 2023-11-09 11:56:44 JOB_DETAILS: {"All backends":{"e3b882f5-7eb3-11ee-ae77-00163e267b60":[10142]},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":311710786,"InternalTableLoadRows":10000000,"ScanBytes":581574034,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"e3b882f5-7eb3-11ee-ae77-00163e267b60":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` > **NOTE** > > INSERT is a synchronous command. If an INSERT job is still running, you need to open another session to check its execution status. #### Use Broker Load[​](#use-broker-load "Direct link to Use Broker Load") An asynchronous Broker Load process handles making the connection to S3, pulling the data, and storing the data in StarRocks. This method supports the following file formats: * Parquet * ORC * CSV * JSON (supported from v3.2.3 onwards) ##### Advantages of Broker Load[​](#advantages-of-broker-load "Direct link to Advantages of Broker Load") * Broker Load runs in the background and clients do not need to stay connected for the job to continue. * Broker Load is preferred for long-running jobs, with the default timeout spanning 4 hours. * In addition to Parquet and ORC file format, Broker Load supports CSV file format and JSON file format (JSON file format is supported from v3.2.3 onwards). ##### Data flow[​](#data-flow "Direct link to Data flow") ![Workflow of Broker Load](/assets/images/broker_load_how-to-work_en-bb36de70866e6366b2b21808f0f77be8.png) 1. The user creates a load job. 2. The frontend (FE) creates a query plan and distributes the plan to the backend nodes (BEs) or compute nodes (CNs). 3. The BEs or CNs pull the data from the source and load the data into StarRocks. ##### Typical example[​](#typical-example "Direct link to Typical example") Create a table, start a load process that pulls the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows.parquet` from S3, and verify the progress and success of the data loading. ###### Create a database and a table[​](#create-a-database-and-a-table "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table has the same schema as the Parquet file that you want to load from AWS S3): ```sql CREATE TABLE user_behavior ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp datetime ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Broker Load[​](#start-a-broker-load "Direct link to Start a Broker Load") Run the following command to start a Broker Load job that loads data from the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows.parquet` to the `user_behavior` table: ```sql LOAD LABEL user_behavior ( DATA INFILE("s3://starrocks-examples/user-behavior-10-million-rows.parquet") INTO TABLE user_behavior FORMAT AS "parquet" ) WITH BROKER ( "aws.s3.enable_ssl" = "true", "aws.s3.use_instance_profile" = "false", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ) PROPERTIES ( "timeout" = "72000" ); ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. * `BROKER`: The connection details for the source. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Check load progress[​](#check-load-progress-1 "Direct link to Check load progress") You can query the progress of the Broker Load job from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'user_behavior'; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). This record shows a state of `LOADING`, and the progress is 39%. If you see something similar, then run the command again until you see a state of `FINISHED`. ```plaintext JOB_ID: 10466 LABEL: user_behavior DATABASE_NAME: mydatabase STATE: LOADING PROGRESS: ETL:100%; LOAD:39% TYPE: BROKER PRIORITY: NORMAL SCAN_ROWS: 4620288 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 4620288 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):72000; max_filter_ratio:0.0 CREATE_TIME: 2024-02-28 22:11:36 ETL_START_TIME: 2024-02-28 22:11:41 ETL_FINISH_TIME: 2024-02-28 22:11:41 LOAD_START_TIME: 2024-02-28 22:11:41 LOAD_FINISH_TIME: NULL JOB_DETAILS: {"All backends":{"2fb97223-b14c-404b-9be1-83aa9b3a7715":[10004]},"FileNumber":1,"FileSize":136901706,"InternalTableLoadBytes":144032784,"InternalTableLoadRows":4620288,"ScanBytes":143969616,"ScanRows":4620288,"TaskNumber":1,"Unfinished backends":{"2fb97223-b14c-404b-9be1-83aa9b3a7715":[10004]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` After you confirm that the load job has finished, you can check a subset of the destination table to see if the data has been successfully loaded. Example: ```sql SELECT * from user_behavior LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 34 | 856384 | 1029459 | pv | 2017-11-27 14:43:27 | | 34 | 5079705 | 1029459 | pv | 2017-11-27 14:44:13 | | 34 | 4451615 | 1029459 | pv | 2017-11-27 14:45:52 | +--------+---------+------------+--------------+---------------------+ ``` #### Use Pipe[​](#use-pipe "Direct link to Use Pipe") Starting from v3.2, StarRocks provides the Pipe loading method, which currently supports only the Parquet and ORC file formats. ##### Advantages of Pipe[​](#advantages-of-pipe "Direct link to Advantages of Pipe") Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading.During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that object storage like AWS S3 uses ETag as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Data flow[​](#data-flow "Direct link to Data flow") ![Pipe data flow](/assets/images/pipe_data_flow-2a4dc0b44a06c987d9afc0ecf632f5d9.png) Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading.During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that object storage like AWS S3 uses `ETag` as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Differences between Pipe and INSERT+FILES()[​](#differences-between-pipe-and-insertfiles "Direct link to Differences between Pipe and INSERT+FILES()") A Pipe job is split into one or more transactions based on the size and number of rows in each data file. Users can query the intermediate results during the loading process. In contrast, an INSERT+`FILES()` job is processed as a single transaction, and users are unable to view the data during the loading process. ##### File loading sequence[​](#file-loading-sequence "Direct link to File loading sequence") For each Pipe job, StarRocks maintains a file queue, from which it fetches and loads data files as micro-batches. Pipe does not ensure that the data files are loaded in the same order as they are uploaded. Therefore, newer data may be loaded prior to older data. ##### Typical example[​](#typical-example-1 "Direct link to Typical example") ###### Create a database and a table[​](#create-a-database-and-a-table-1 "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from AWS S3): ```sql CREATE TABLE user_behavior_from_pipe ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp datetime ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Pipe job[​](#start-a-pipe-job "Direct link to Start a Pipe job") Run the following command to start a Pipe job that loads data from the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows/` to the `user_behavior_from_pipe` table. This pipe job uses both micro batches, and continuous loading (described above) pipe-specific features. The other examples in this guide load a single Parquet file with 10 million rows. For the pipe example, the same dataset is split into 57 separate files, and these are all stored in one S3 folder. Note in the `CREATE PIPE` command below the `path` is the URI for an S3 folder and rather than providing a filename the URI ends in `/*`. By setting `AUTO_INGEST` and specifying a folder rather than an individual file the pipe job will poll the S3 folder for new files and ingest them as they are added to the folder. ```sql CREATE PIPE user_behavior_pipe PROPERTIES ( "AUTO_INGEST" = "TRUE" ) AS INSERT INTO user_behavior_from_pipe SELECT * FROM FILES ( "path" = "s3://starrocks-examples/user-behavior-10-million-rows/*", "format" = "parquet", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ); ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. This job has four main sections: * `pipe_name`: The name of the pipe. The pipe name must be unique within the database to which the pipe belongs. * `INSERT_SQL`: The INSERT INTO SELECT FROM FILES statement that is used to load data from the specified source data file to the destination table. * `PROPERTIES`: A set of optional parameters that specify how to execute the pipe. These include `AUTO_INGEST`, `POLL_INTERVAL`, `BATCH_SIZE`, and `BATCH_FILES`. Specify these properties in the `"key" = "value"` format. For detailed syntax and parameter descriptions, see [CREATE PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md). ###### Check load progress[​](#check-load-progress-2 "Direct link to Check load progress") * Query the progress of the Pipe job by using [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md) in the current database to which the Pipe job belongs. ```sql SHOW PIPES WHERE NAME = 'user_behavior_pipe' \G ``` The following result is returned: tip In the output shown below the pipe is in the `RUNNING` state. A pipe will stay in the `RUNNING` state until you manually stop it. The output also shows the number of files loaded (57) and the last time that a file was loaded. ```sql *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10476 PIPE_NAME: user_behavior_pipe STATE: RUNNING TABLE_NAME: mydatabase.user_behavior_from_pipe LOAD_STATUS: {"loadedFiles":57,"loadedBytes":295345637,"loadingFiles":0,"lastLoadedTime":"2024-02-28 22:14:19"} LAST_ERROR: NULL CREATED_TIME: 2024-02-28 22:13:41 1 row in set (0.02 sec) ``` * Query the progress of the Pipe job from the [`pipes`](https://docs.starrocks.io/docs/sql-reference/information_schema/pipes.md) view in the StarRocks Information Schema. ```sql SELECT * FROM information_schema.pipes WHERE pipe_name = 'user_behavior_replica' \G ``` The following result is returned: tip Some of the queries in this guide end in `\G` instead of a semicolon (`;`). This causes the MySQL client to output the results in vertical format. If you are using DBeaver or another client you may need to use a semicolon (`;`) rather than `\G`. ```sql *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10217 PIPE_NAME: user_behavior_replica STATE: RUNNING TABLE_NAME: mydatabase.user_behavior_replica LOAD_STATUS: {"loadedFiles":1,"loadedBytes":132251298,"loadingFiles":0,"lastLoadedTime":"2023-11-09 15:35:42"} LAST_ERROR: CREATED_TIME: 9891-01-15 07:51:45 1 row in set (0.01 sec) ``` ###### Check file status[​](#check-file-status "Direct link to Check file status") You can query the load status of the files loaded from the [`pipe_files`](https://docs.starrocks.io/docs/sql-reference/information_schema/pipe_files.md) view in the StarRocks Information Schema. ```sql SELECT * FROM information_schema.pipe_files WHERE pipe_name = 'user_behavior_replica' \G ``` The following result is returned: ```sql *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10217 PIPE_NAME: user_behavior_replica FILE_NAME: s3://starrocks-examples/user-behavior-10-million-rows.parquet FILE_VERSION: e29daa86b1120fea58ad0d047e671787-8 FILE_SIZE: 132251298 LAST_MODIFIED: 2023-11-06 13:25:17 LOAD_STATE: FINISHED STAGED_TIME: 2023-11-09 15:35:02 START_LOAD_TIME: 2023-11-09 15:35:03 FINISH_LOAD_TIME: 2023-11-09 15:35:42 ERROR_MSG: 1 row in set (0.03 sec) ``` ###### Manage Pipe jobs[​](#manage-pipe-jobs "Direct link to Manage Pipe jobs") You can alter, suspend or resume, drop, or query the pipes you have created and retry to load specific data files. For more information, see [ALTER PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/ALTER_PIPE.md), [SUSPEND or RESUME PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SUSPEND_or_RESUME_PIPE.md), [DROP PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/DROP_PIPE.md), [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md), and [RETRY FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/RETRY_FILE.md). --- ### Sink data from RisingWave to StarRocks RisingWave is a distributed SQL streaming database that enables simple, efficient, and reliable processing of streaming data. To quickly get started with RisingWave, see [Get started](https://docs.risingwave.com/get-started/quickstart/). RisingWave provides the data sinking feature to enable users to directly sink data to StarRocks without requiring any other third-party components. This feature can work with all StarRocks table types: Duplicate Key, Primary Key, Aggregate, and Unique Key tables. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * You have a running RisingWave cluster of v1.7 or later. * You can access the target StarRocks table and the StarRocks version is v2.5 or later. * To sink data into a StarRocks table, you must have the SELECT and INSERT privileges on the target table. To grant the privileges, see [GRANT](https://docs.starrocks.io/zh/docs/sql-reference/sql-statements/account-management/GRANT/). tip RisingWave only supports at-least-once semantics for StarRocks Sink, which means that in case of failures, duplicate data may be written. You are recommended to use [StarRocks Primary Key tables](https://docs.starrocks.io/zh/docs/table_design/table_types/primary_key_table/), which can deduplicate data and achieve end-to-end idempotent writes. #### Parameters[​](#parameters "Direct link to Parameters") The following table describes the parameters you need to configure when you sink data from RisingWave to StarRocks. All parameters are required unless otherwise specified. | Parameters | Description | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | connector | Set it to `starrocks`. | | starrocks.host | The IP address of the StarRocks FE node. | | starrocks.query\_port | The query port the FE node. | | starrocks.http\_port | The HTTP port of the FE node. | | starrocks.user | The username used to access the StarRocks cluster. | | starrocks.password | The password associated with the username. | | starrocks.database | The StarRocks database where the target table is located. | | starrocks.table | The StarRocks table to which you want to sink data. | | starrocks.partial\_update | (Optional) Whether to enable the StarRocks partial update feature. Enabling this feature can increase the Sink performance when only a few columns need to be updated. | | type | The data operation type during sink.- `append-only`: Performs only INSERT operations.
- `upsert`: Performs Upsert operations. If this setting is used, the StarRocks target table must be a Primary Key table. | | force\_append\_only | (Optional) When `type` is set to `append-only` but there are also Upsert and Delete operations in the sink process, this setting can force the Sink task to generate append-only data and discard Upsert and Delete data. | | primary\_key | (Optional) The primary key of the StarRocks table. Required if `type` is `upsert`. | #### Data type mapping[​](#data-type-mapping "Direct link to Data type mapping") The following table provides the data type mapping between RisingWave and StarRocks. | RisingWave | StarRocks | | ----------------------------------------------------------------------------- | ------------- | | BOOLEAN | BOOLEAN | | SMALLINT | SMALLINT | | INTEGER | INT | | BIGINT | BIGINT | | REAL | FLOAT | | DOUBLE | DOUBLE | | DECIMAL | DECIMAL | | DATE | DATE | | VARCHAR | VARCHAR | | TIME
(Cast to VARCHAR before sinking to StarRocks) | Not supported | | TIMESTAMP | DATETIME | | TIMESTAMP WITH TIME ZONE
(Cast to TIMESTAMP before sinking to StarRocks) | Not supported | | INTERVAL
(Cast to VARCHAR before sinking to StarRocks) | Not supported | | STRUCT | JSON | | ARRAY | ARRAY | | BYTEA
(Cast to VARCHAR before sinking to StarRocks) | Not supported | | JSONB | JSON | | SERIAL | BIGINT | #### Examples[​](#examples "Direct link to Examples") 1. Create a database `demo` in StarRocks and create a Primary Key table `score_board` in this database. ```sql CREATE DATABASE demo; USE demo; CREATE TABLE demo.score_board( id int(11) NOT NULL COMMENT "", name varchar(65533) NULL DEFAULT "" COMMENT "", score int(11) NOT NULL DEFAULT "0" COMMENT "" ) PRIMARY KEY(id) DISTRIBUTED BY HASH(id); ``` 2. Sink data from RisingWave to StarRocks. ```sql -- Create a table in RisingWave. CREATE TABLE score_board ( id INT PRIMARY KEY, name VARCHAR, score INT ); -- Insert data into the table. INSERT INTO score_board VALUES (1, 'starrocks', 100), (2, 'risingwave', 100); -- Sink data from this table to the StarRocks table. CREATE SINK score_board_sink FROM score_board WITH ( connector = 'starrocks', type = 'upsert', starrocks.host = 'starrocks-fe', starrocks.mysqlport = '9030', starrocks.httpport = '8030', starrocks.user = 'users', starrocks.password = '123456', starrocks.database = 'demo', starrocks.table = 'score_board', primary_key = 'id' ); ``` --- ### Superset Support [Apache Superset](https://superset.apache.org) is a modern data exploration and visualization platform. It uses [SQLAlchemy](https://github.com/StarRocks/starrocks/tree/main/contrib/starrocks-python-client/starrocks) to query data. Although Mysql Dialect can be used, it does not support `largeint`. So we developed [StarRocks Dialect](https://github.com/StarRocks/starrocks/tree/main/contrib/starrocks-python-client/starrocks/). #### Environment[​](#environment "Direct link to Environment") * Python 3.x * mysqlclient (pip install mysqlclient) * [Apache Superset](https://superset.apache.org) Notice: If `mysqlclient` is not installed, an exception will be thrown: ```plain No module named 'MySQLdb' ``` #### Installation[​](#installation "Direct link to Installation") Since `dialect` does not contribute to `SQLAlchemy`, it needs to be installed from the source code. If you install `superset` with Docker, install `sqlalchemy-starrocks` with `root`. Install from [Source Code](https://github.com/StarRocks/starrocks/tree/main/contrib/starrocks-python-client/starrocks) ```shell pip install . ``` Uninstall ```shell pip uninstall sqlalchemy-starrocks ``` #### Usage[​](#usage "Direct link to Usage") To connect to StarRocks with SQLAlchemy, the following URL pattern can be used: ```shell starrocks://:@:/[?charset=utf8] ``` #### Basic Example[​](#basic-example "Direct link to Basic Example") ##### Sqlalchemy Example[​](#sqlalchemy-example "Direct link to Sqlalchemy Example") It is recommended to use python 3.x to connect to the StarRocks database, eg: ```python from sqlalchemy import create_engine import pandas as pd conn = create_engine('starrocks://root:@x.x.x.x:9030/superset_db?charset=utf8') sql = """select * from xxx""" df = pd.read_sql(sql, conn) ``` ##### Superset Example[​](#superset-example "Direct link to Superset Example") In superset, use `Other` database, and set url as: ```shell starrocks://root:@x.x.x.x:9030/superset_db?charset=utf8 ``` --- ## Introduction ### Architecture StarRocks has a simple architecture. The entire system consists of only two types of components; frontends and backends. The frontend nodes are called **FE**s. There are two types of backend nodes, **BE**s, and **CN**s (Compute Nodes). BEs are deployed when local storage for data is used, and CNs are deployed when data is stored on object storage or HDFS. StarRocks does not rely on any external components, simplifying deployment and maintenance. Nodes can be horizontally scaled without service downtime. In addition, StarRocks has a replica mechanism for metadata and service data, which increases data reliability and efficiently prevents single points of failure (SPOFs). StarRocks is compatible with MySQL protocols and supports standard SQL. Users can easily connect to StarRocks from MySQL clients to gain instant and valuable insights. #### Architecture choices[​](#architecture-choices "Direct link to Architecture choices") StarRocks supports shared-nothing (Each BE has a portion of the data on its local storage) and shared-data (all data on object storage or HDFS and each CN has only cache on local storage). You can decide where the data is stored based on your needs. ![Architecture choices](/assets/images/architecture_choices-ddd6ad78779f16a6691ec53a1f86ef06.png) ##### Shared-nothing[​](#shared-nothing "Direct link to Shared-nothing") Local storage provides improved query latency for real-time queries. As a typical massively parallel processing (MPP) database StarRocks supports the shared-nothing architecture. In this architecture, BEs are responsible for both data storage and computation. Direct access to local data on the BE mode allows for local computation, avoiding data transfer and data copying, and providing ultra-fast query and analytics performance. This architecture supports multi-replica data storage, enhancing the cluster's ability to handle high concurrency queries and ensuring data reliability. It is well-suited for scenarios that pursue optimal query performance. ![shared-data-arch](/assets/images/shared-nothing-f5cbe29195b0d7e9c66069d356208cd2.png) ###### Nodes[​](#nodes "Direct link to Nodes") In the shared-nothing architecture, StarRocks consists of two types of nodes: FEs and BEs. * FEs are responsible for metadata management and constructing execution plans. * BEs execute query plans and store data. BEs utilize local storage to accelerate queries and the multi-replica mechanism to ensure high data availability. ###### FE[​](#fe "Direct link to FE") FEs are responsible for metadata management, client connection management, query planning, and query scheduling. Each FE uses BDB JE (Berkeley DB Java Edition) to store and maintain a complete copy of the metadata in its memory, ensuring consistent services across all FEs. FEs can work as the leader, followers, and observers. If the leader node crashes, with followers electing a leader based on the Raft protocol. | **FE Role** | **Metadata management** | **Leader election** | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Leader | The leader FE reads and writes metadata. Follower and observer FEs can only read metadata. They route metadata write requests to the leader FE. The leader FE updates the metadata and then uses the Raft protocol to synchronize the metadata changes to the follower and observer FEs. Data writes are considered successful only after the metadata changes are synchronized to more than half of the follower FEs. | The leader FE, technically speaking, is also a follower node and is elected from follower FEs. To perform leader election, more than half of the follower FEs in the cluster must be active. When the leader FE fails, follower FEs will start another round of leader election. | | Follower | Followers can only read metadata. They synchronize and replay logs from the leader FE to update metadata. | Followers participate in leader election, which requires more than half of the followers in the cluster be active. | | Observer | Observers synchronize and replay logs from the leader FE to update metadata. | Observers are mainly used to increase the query concurrency of the cluster. Observers do not participate in leader election and therefore, will not add leader selection pressure to the cluster. | ###### BE[​](#be "Direct link to BE") BEs are responsible for data storage and SQL execution. * Data storage: BEs have equivalent data storage capabilities. FEs distribute data to BEs based on predefined rules. BEs transform the ingested data, write the data into the required format, and generate indexes for the data. * SQL execution: FEs parse each SQL query into a logical execution plan according to the semantics of the query, and then transform the logical plan into physical execution plans that can be executed on BEs. BEs that store the destination data execute the query. This eliminates the need for data transmission and copy, achieving high query performance. ##### Shared-data[​](#shared-data "Direct link to Shared-data") Object storage and HDFS provide cost, reliability, and scalability benefits. In addition to the scalability of storage, CN nodes can be added and removed without the need to rebalance data since storage and compute are separate. In the shared-data architecture, BEs are replaced with "compute nodes (CNs)" which are responsible only for data compute tasks and caching hot data. Data is stored in low-cost and reliable remote storage systems such as Amazon S3, Google Cloud Storage, Azure Blob Storage, MinIO, etc. When the cache is hit, query performance is comparable to that of the shared-nothing architecture. CN nodes can be added or removed on demand within seconds. This architecture reduces storage cost, ensures better resource isolation, and high elasticity and scalability. The shared-data architecture maintains as simple an architecture as its shared-nothing counterpart. It consists of only two types of nodes: FE and CN. The only difference is users have to provision backend object storage. ![shared-data-arch](/assets/images/shared-data-1fac1b5ab7d46bf34f67c93ecc8e6c28.png) ###### Nodes[​](#nodes-1 "Direct link to Nodes") FEs in the shared-data architecture provide the same functions as in the shared-nothing architecture. BEs are replaced with CNs (Compute Nodes), and the storage function is offloaded to object storage or HDFS. CNs are stateless compute nodes that perform all the functions of BEs, except for the storage of data. ###### Storage[​](#storage "Direct link to Storage") StarRocks shared-data clusters support two storage solutions: object storage (for example, AWS S3, Google GCS, Azure Blob Storage, or MinIO) and HDFS. In a shared-data cluster, the data file format remains consistent with that of a shared-nothing cluster (featuring coupled storage and compute). Data is organized into segment files, and various indexing technologies are reused in cloud-native tables, which are tables used specifically in shared-data clusters. ###### Cache[​](#cache "Direct link to Cache") StarRocks shared-data clusters decouple data storage and computation, allowing each to scale independently, thereby reducing costs and enhancing elasticity. However, this architecture can affect query performance. To mitigate the impact, StarRocks establishes a multi-tiered data access system encompassing memory, local disk, and remote storage to better meet various business needs. Queries against hot data scan the cache directly and then the local disk, while cold data needs to be loaded from the object storage into the local cache to accelerate subsequent queries. By keeping hot data close to compute units, StarRocks achieves truly high-performance computation and cost-effective storage. Moreover, access to cold data has been optimized with data prefetch strategies, effectively eliminating performance limits for queries. Caching can be enabled when creating tables. If caching is enabled, data will be written to both the local disk and backend object storage. During queries, the CN nodes first read data from the local disk. If the data is not found, it will be retrieved from the backend object storage and simultaneously cached on the local disk. #### Feature Differences between Shared-nothing and Shared-data Clusters[​](#feature-differences-between-shared-nothing-and-shared-data-clusters "Direct link to Feature Differences between Shared-nothing and Shared-data Clusters") This section lists the differences of features supported in shared-nothing and shared-data clusters. ##### Real-time Analytics and Primary Key Tables[​](#real-time-analytics-and-primary-key-tables "Direct link to Real-time Analytics and Primary Key Tables") | Feature | Shared-nothing | Shared-data | | ---------------------------------------------------- | -------------- | --------------- | | Conditional Update | v2.5+ | v3.1+ | | Partial Update Column mode | v3.1 | To be supported | | Partial Update Row mode | v2.3+ | v3.1+ | | Partial update with Conditional Update | v3.1+ | v3.4.1+ | | Primary Key Index Persistence | v2.3+ | v3.2+ | | Decoupled ORDER BY columns from Primary Keys | v3.0+ | v3.1+ | | Rich UPDATE and DELETE syntax for Primary Key tables | v3.0+ | v3.1+ | ##### Storage Engine[​](#storage-engine "Direct link to Storage Engine") | Feature | Shared-nothing | Shared-data | | ----------------------------- | -------------- | --------------- | | Fast Schema Evolution | v3.2+ | v4.0+ | | File Bundling | Not applicable | v4.0+ | | Inverted Index | v3.3+ | v4.1+ | | Manual Compaction | v3.1+ | v3.3+ | | Row Store | v3.2+ | To be supported | | Adding/dropping STRUCT fields | v3.3.2+ | v3.3.5+ | | Vector index | v3.4+ | To be supported | ##### Data Distribution[​](#data-distribution "Direct link to Data Distribution") | Feature | Shared-nothing | Shared-data | | --------------------------------------------------- | -------------- | -------------- | | Expression Partitioning | v3.0+ | v3.1.1+ | | List Partitioning | v3.1+ | v3.1.1+ | | Random Bucketing | v3.1+ | v3.2+ | | Random Bucketing Optimization (with Sub-partition) | v3.2+ | v3.2+ | | Unified Syntax for Multi-level Expression Partition | v3.4+ | v3.4+ | | Optimize Table: Change Bucketing | v3.2+ | v3.3+ | | Optimize Table: Change Bucketing Online | v3.3.3+ | Not applicable | | Range-based Distribution | v4.1+ | v4.1+ | ##### Query Performance[​](#query-performance "Direct link to Query Performance") | Feature | Shared-nothing | Shared-data | | ---------------------------------------- | -------------- | ----------- | | Cross-node Data Cache Sharing | Not applicable | v3.5.1+ | | Generated Column | v3.1+ | v3.5+ | | Query Cache | v2.5+ | v3.4+ | | Rollup and Synchronous Materialized view | v1.x+ | v3.3+ | ##### Disaster Recovery[​](#disaster-recovery "Direct link to Disaster Recovery") | Feature | Shared-nothing | Shared-data | | --------------------------------- | -------------- | -------------- | | Backup & Restore | v1.x+ | Not applicable | | BE/CN Blacklist | v3.3+ | v4.0+ | | Cross-cluster Data Migration Tool | v3.3+ | Not applicable | | Cluster Snapshot | Not applicable | v3.4.2+ | ##### Security and Authentication[​](#security-and-authentication "Direct link to Security and Authentication") | Feature | Shared-nothing | Shared-data | | ------------------------ | -------------- | ----------- | | Kerberos Support on HDFS | Not applicable | v3.2+ | #### Learn by doing[​](#learn-by-doing "Direct link to Learn by doing") Try the [Quick Starts](https://docs.starrocks.io/docs/quick_start.md) to get an overview of using StarRocks with realistic scenarios. --- ### Database Features StarRocks offers a rich set of features to deliver a blazing-fast, real-time analytics experience on data at scale. #### MPP framework[​](#mpp-framework "Direct link to MPP framework") StarRocks adopts the massively parallel processing (MPP) framework. One query request is split into multiple physical computing units that can be executed in parallel on multiple machines. Each machine has dedicated CPU and memory resources. The MPP framework fully uses the resources of all CPU cores and machines. The performance of a single query can continuously increase as the cluster is scaled out. ![MPP](/assets/images/1.1-3-mpp-243454cd702a6fac41bbc619a80e0c90.png) In the preceding figure, StarRocks parses an SQL statement into multiple logical execution units (query fragments) according to the semantics of the statement. Each fragment is then implemented by one or multiple physical execution units (fragment instances) based on the computing complexity. A physical execution unit is the smallest scheduling unit in StarRocks. They will be scheduled to backends (BEs) for execution. One logical execution unit can contain one or more operators, such as the Scan, Project, and Agg operators, as shown on the right side of the figure. Each physical execution unit processes only part of the data and the result will be merged to generate the final data. **Parallel execution of logical execution units fully utilizes the resources of all CPU cores and physical machines and accelerates the query speed.** ![MPP](/assets/images/1.1-4-mpp-6e666188e05796fccef4a7f92e97d6c7.png) Unlike the Scatter-Gather framework used by many other data analytics systems, the MPP framework can utilize more resources to process query requests. In the Scatter-Gather framework, only the Gather node can perform the final merge operation. In the MPP framework, data is shuffled to multiple nodes for merge operations. For complex queries, such as Group By on high-cardinality fields and large table joins, StarRocks' MPP framework has noticeable performance advantages over the Scatter-Gather framework. #### Fully vectorized execution engine[​](#fully-vectorized-execution-engine "Direct link to Fully vectorized execution engine") The fully vectorized execution engine makes more efficient use of CPU processing power because this engine organizes and processes data in a columnar manner. Specifically, StarRocks stores data, organizes data in memory, and computes SQL operators all in a columnar manner. Columnar organization makes full use of CPU cache. Columnar computing reduces the number of virtual function calls and branch judgments, resulting in more sufficient CPU instruction flows. The vectorized execution engine also makes full use of SIMD instructions. This engine can complete more data operations with fewer instructions. Tests against standard datasets show that this engine enhances the overall performance of operators by 3 to 10 times. In addition to operator vectorization, StarRocks has implemented other optimizations for the query engine. For example, StarRocks uses the Operation on Encoded Data technology to directly execute operators on encoded strings, without the need for decoding. This noticeably reduces SQL complexity and increases the query speed by more than 2 times. #### Separation of storage and compute[​](#separation-of-storage-and-compute "Direct link to Separation of storage and compute") The [storage-compute separation architecture](https://docs.starrocks.io/docs/introduction/Architecture.md) was introduced from 3.0. In this architecture, computing and storage are decoupled to achieve resource isolation, elastic scaling of compute nodes, and high-performance queries. Storage-compute separation equips StarRocks with better flexibility, higher performance and data availability, and lower cost. ![shared-data](/assets/images/share_data_arch-2aee703af4e8afcf46e5bf9866cd2f2a.png) In storage-compute separation mode, computing and storage are decoupled and can be scaled independently, which eliminates the cost that long exists in the storage-compute coupled mode where storage has to be scaled anytime users want to add computing nodes. In addition, computing can dynamically scale within seconds, improving resource utilization, especially when there are noticeable traffic peaks and valleys. The storage layer leverages the nearly unlimited capacity and high reliability of object storage to achieve massive data storage and data persistence. StarRocks can work with various object storage systems such as AWS S3, Google Cloud Storage, Azure Blob Storage, HDFS, and other S3-compatible storage like MinIO. Users can choose to deploy StarRocks in public clouds, private clouds, or on-premises data centers. StarRocks supports Kubernetes-based deployments and provides an Operator for automated deployment of storage-compute decoupled clusters. StarRocks in storage-compute separation mode provides the same functionalities as the storage-compute coupled mode. The data write and hot data query performance are also the same. Users can perform data updates, data lake analytics, and materialized view acceleration as they do in storage-compute coupled mode. #### Cost-based optimizer[​](#cost-based-optimizer "Direct link to Cost-based optimizer") ![CBO](/assets/images/1.1-5-cbo-3baa9959e4ca41775a33d67400e8c480.png) Performance of multi-table join queries is difficult to optimize. Execution engines alone cannot deliver superior performance because the complexity of execution plans may vary by several orders of magnitude in multi-table join query scenarios. The more the associated tables, the more the execution plans, which makes it NP-hard to choose an optimal plan. Only a query optimizer excellent enough can choose a relatively optimal query plan for efficient multi-table analytics. StarRocks designs a brand-new [CBO](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md) from scratch. This CBO adopts the cascades-like framework and is deeply customized for the vectorized execution engine with a number of optimizations and innovations. These optimizations include the reuse of common table expressions (CTEs), rewriting of subqueries, Lateral Join, Join Reorder, strategy selection for distributed Join execution, and low-cardinality optimization. The CBO supports a total of 99 TPC-DS SQL statements. The CBO enables StarRocks to deliver better multi-table join query performance than competitors, especially in complex multi-table join queries. #### Real-time, updatable columnar storage engine[​](#real-time-updatable-columnar-storage-engine "Direct link to Real-time, updatable columnar storage engine") StarRocks is a columnar storage engine that allows data of the same type to be stored continuously. In columnar storage, data can be encoded in a more efficient way, increasing the compression ratio and lowering the storage cost. Columnar storage also reduces the total data read I/Os, improving query performance. In addition, in most OLAP scenarios, only specific columns are queried. Columnar storage enables users to query only part of the columns, significantly reducing disk I/Os. StarRocks can load data within seconds for near-real-time analytics. StarRocks' storage engine guarantees the atomicity, consistency, isolation, and durability (ACID) of each data ingestion operation. For a data loading transaction, the entire transaction either succeeds or fails. Concurrent transactions do not affect each other, providing transaction-level isolation. ![Realtime](/assets/images/1.1-6-realtime-81890333b24620f95ea35397d23af561.png) StarRocks' storage engine uses the Delete-and-insert pattern, which allows for efficient Partial Update and Upsert operations. The storage engine can quickly filter data using primary key indexes, eliminating the need for Sort and Merge operations at data reading. The engine can also make full use of secondary indexes. It delivers fast and predictable query performance even on huge volume of data updates. #### Intelligent materialized view[​](#intelligent-materialized-view "Direct link to Intelligent materialized view") StarRocks uses intelligent [materialized views](https://docs.starrocks.io/docs/using_starrocks/async_mv/Materialized_view.md) to accelerate queries and data warehouse layering. Different from materialized views of other similar products that requires manual data synchronization with the base table, StarRocks' materialized views automatically update data according to the data changes in the base table without requiring additional maintenance operations. In addition, the selection of materialized views is also automatic. If StarRocks identifies a suitable materialized view (MV) to improve query performance, it will automatically rewrite the query to utilize the MV. This intelligent process significantly enhances query efficiency without requiring manual intervention. StarRocks' MV can replace the traditional ETL data modeling process: Instead of transforming data in the upstream applications, you now have the option to transform data with MV within StarRocks, simplifying the data processing pipeline. For example, in the figure, raw data on data lake can be used to create a normalized table based on an external MV. A denormalized table can be created from normalized tables through asynchronous materialized views. Another MV can be created from normalized tables to support high concurrency queries and better query performance. ![MV](/assets/images/1.1-7-mv-7fb0ae6f680ae698965d29b4a7a6911d.png) #### Data lake analytics[​](#data-lake-analytics "Direct link to Data lake analytics") ![DLA](/assets/images/1.1-8-dla-c67d601d709b092317fa11eb64ac1783.png) In addition to efficient analytics of local data, StarRocks can work as the compute engine to analyze data stored in [data lakes](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md) such as Apache Hive, Apache Iceberg, Apache Hudi, and Delta Lake. One of the key features of StarRocks is its external catalog, which acts as a linkage to an externally maintained metastore. This functionality provides users with the capability to query external data sources seamlessly, eliminating the need for data migration. As such, users can analyze data from different systems such as HDFS and Amazon S3, in various file formats such as Parquet, ORC, and CSV, etc. The preceding figure shows a data lake analytics scenario where StarRocks is responsible for data computing and analysis, and the data lake is responsible for data storage, organization, and maintenance. Data lakes allow users to store data in open storage formats and use flexible schemas to produce reports on "single source of truth" for various BI, AI, ad-hoc, and reporting use cases. StarRocks fully leverages the advantages of its vectorization engine and CBO, significantly improving the performance of data lake analytics. --- ### Community Chat (via Slack) The StarRocks project uses Slack for community chat. You can find our community at [starrocks.slack.com](https://docs.starrocks.io/join/). --- ## Kafka_connector ### Releases of StarRocks Connector for Kafka #### Notifications[​](#notifications "Direct link to Notifications") **User guide:** [Load data using Kafka connector](https://docs.starrocks.io/docs/loading/Kafka-connector-starrocks/) **Source codes:** [starrocks-connector-for-kafka](https://github.com/StarRocks/starrocks-connector-for-kafka) **Naming format of the compressed file:** `starrocks-kafka-connector-${connector_version}.tar.gz` **Download link of the compressed file:** [starrocks-kafka-connector](https://github.com/StarRocks/starrocks-connector-for-kafka/releases) **Version requirements:** | Kafka Connector | StarRocks | Java | | --------------- | ------------- | ---- | | 1.0.3 | 2.1 and later | 8 | | 1.0.2 | 2.1 and later | 8 | | 1.0.1 | 2.1 and later | 8 | | 1.0.0 | 2.1 and later | 8 | #### Release notes[​](#release-notes "Direct link to Release notes") ##### 1.0[​](#10 "Direct link to 1.0") ###### 1.0.3[​](#103 "Direct link to 1.0.3") Release date: December 19, 2023 **Features** Added Apache License as the open-source software license. [#9](https://github.com/StarRocks/starrocks-connector-for-kafka/pull/9) ###### 1.0.2[​](#102 "Direct link to 1.0.2") Release date: December 14, 2023 **Features** Supports the source data to be of DECIMAL type. [#7](https://github.com/StarRocks/starrocks-connector-for-kafka/pull/7) ###### 1.0.1[​](#101 "Direct link to 1.0.1") Release date: November 28, 2023 **Features** * Supports loading Debezium data into Primary Key tables. [#4](https://github.com/StarRocks/starrocks-connector-for-kafka/pull/4) * Supports parsing JSON data without schema registry. [#6](https://github.com/StarRocks/starrocks-connector-for-kafka/pull/6) ###### 1.0.0[​](#100 "Direct link to 1.0.0") Release date: June 25, 2023 **Features** * Supports loading CSV, JSON, Avro, and Protobuf data. * Supports loading data from a self-managed Apache Kafka cluster or Confluent cloud. --- ## Loading ### Load data from HDFS or cloud storage StarRocks provides the loading method MySQL-based Broker Load to help you load a large amount of data from HDFS or cloud storage into StarRocks. Broker Load runs in asynchronous loading mode. After you submit a load job, StarRocks asynchronously runs the job. You need to use the [SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md) statement or the `curl` command to check the result of the job. Broker Load supports single-table loads and multi-table loads. You can load one or multiple data files into one or multiple destination tables by running one Broker Load job. Broker Load ensures the transactional atomicity of each load job that is run to load multiple data files. Atomicity means that the loading of multiple data files in one load job must all succeed or fail. It never happens that the loading of some data files succeeds while the loading of the other files fails. Broker Load supports data transformation at data loading and supports data changes made by UPSERT and DELETE operations during data loading. For more information, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md) and [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md). You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. #### Background information[​](#background-information "Direct link to Background information") In v2.4 and earlier, StarRocks depends on brokers to set up connections between your StarRocks cluster and your external storage system when it runs a Broker Load job. Therefore, you need to input `WITH BROKER ""` to specify the broker you want to use in the load statement. This is called "broker-based loading." A broker is an independent, stateless service that is integrated with a file-system interface. With brokers, StarRocks can access and read data files that are stored in your external storage system, and can use its own computing resources to pre-process and load the data of these data files. From v2.5 onwards, StarRocks no longer depends on brokers to set up connections between your StarRocks cluster and your external storage system when it runs a Broker Load job. Therefore, you no longer need to specify a broker in the load statement, but you still need to retain the `WITH BROKER` keyword. This is called "broker-free loading." When your data is stored in HDFS, you may encounter situations where broker-free loading does not work. This can happen when your data is stored across multiple HDFS clusters or when you have configured multiple Kerberos users. In these situations, you can resort to using broker-based loading instead. To do this successfully, make sure that at least one independent broker group is deployed. For information about how to specify authentication configuration and HA configuration in these situations, see [HDFS](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#hdfs). #### Supported data file formats[​](#supported-data-file-formats "Direct link to Supported data file formats") Broker Load supports the following data file formats: * CSV * Parquet * ORC > **NOTE** > > For CSV data, take note of the following points: > > * You can use a UTF-8 string, such as a comma (,), tab, or pipe (|), whose length does not exceed 50 bytes as a text delimiter. > * Null values are denoted by using `\N`. For example, a data file consists of three columns, and a record from that data file holds data in the first and third columns but no data in the second column. In this situation, you need to use `\N` in the second column to denote a null value. This means the record must be compiled as `a,\N,b` instead of `a,,b`. `a,,b` denotes that the second column of the record holds an empty string. #### Supported storage systems[​](#supported-storage-systems "Direct link to Supported storage systems") Broker Load supports the following storage systems: * HDFS * AWS S3 * Google GCS * Other S3-compatible storage system such as MinIO * Microsoft Azure Storage #### How it works[​](#how-it-works "Direct link to How it works") After you submit a load job to an FE, the FE generates a query plan, splits the query plan into portions based on the number of available BEs and the size of the data file you want to load, and then assigns each portion of the query plan to an available BE. During the load, each involved BE pulls the data of the data file from your HDFS or cloud storage system, pre-processes the data, and then loads the data into your StarRocks cluster. After all BEs finish their portions of the query plan, the FE determines whether the load job is successful. The following figure shows the workflow of a Broker Load job. ![Workflow of Broker Load](/assets/images/broker_load_how-to-work_en-bb36de70866e6366b2b21808f0f77be8.png) #### Basic operations[​](#basic-operations "Direct link to Basic operations") ##### Create a multi-table load job[​](#create-a-multi-table-load-job "Direct link to Create a multi-table load job") This topic uses CSV as an example to describe how to load multiple data files into multiple tables. For information about how to load data in other file formats and about the syntax and parameter descriptions for Broker Load, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). Note that in StarRocks some literals are used as reserved keywords by the SQL language. Do not directly use these keywords in SQL statements. If you want to use such a keyword in an SQL statement, enclose it in a pair of backticks (\`). See [Keywords](https://docs.starrocks.io/docs/sql-reference/sql-statements/keywords.md). ###### Data examples[​](#data-examples "Direct link to Data examples") 1. Create CSV files in your local file system. a. Create a CSV file named `file1.csv`. The file consists of three columns, which represent user ID, user name, and user score in sequence. ```plain 1,Lily,23 2,Rose,23 3,Alice,24 4,Julia,25 ``` b. Create a CSV file named `file2.csv`. The file consists of two columns, which represent city ID and city name in sequence. ```plain 200,'Beijing' ``` 2. Create StarRocks tables in your StarRocks database `test_db`. > **NOTE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). a. Create a Primary Key table named `table1`. The table consists of three columns: `id`, `name`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `table1` ( `id` int(11) NOT NULL COMMENT "user ID", `name` varchar(65533) NULL DEFAULT "" COMMENT "user name", `score` int(11) NOT NULL DEFAULT "0" COMMENT "user score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` b. Create a Primary Key table named `table2`. The table consists of two columns: `id` and `city`, of which `id` is the primary key. ```sql CREATE TABLE `table2` ( `id` int(11) NOT NULL COMMENT "city ID", `city` varchar(65533) NULL DEFAULT "" COMMENT "city name" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` 3. Upload `file1.csv` and `file2.csv` to the `/user/starrocks/` path of your HDFS cluster, to the `input` folder of your AWS S3 bucket `bucket_s3`, to the `input` folder of your Google GCS bucket `bucket_gcs`, to the `input` folder of your MinIO bucket `bucket_minio`, and to the specified paths of your Azure Storage. ###### Load data from HDFS[​](#load-data-from-hdfs "Direct link to Load data from HDFS") Execute the following statement to load `file1.csv` and `file2.csv` from the `/user/starrocks` path of your HDFS cluster into `table1` and `table2`, respectively: ```sql LOAD LABEL test_db.label1 ( DATA INFILE("hdfs://:/user/starrocks/file1.csv") INTO TABLE table1 COLUMNS TERMINATED BY "," (id, name, score) , DATA INFILE("hdfs://:/user/starrocks/file2.csv") INTO TABLE table2 COLUMNS TERMINATED BY "," (id, city) ) WITH BROKER ( StorageCredentialParams ) PROPERTIES ( "timeout" = "3600" ); ``` In the preceding example, `StorageCredentialParams` represents a group of authentication parameters which vary depending on the authentication method you choose. For more information, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#hdfs). ###### Load data from AWS S3[​](#load-data-from-aws-s3 "Direct link to Load data from AWS S3") Execute the following statement to load `file1.csv` and `file2.csv` from the `input` folder of your AWS S3 bucket `bucket_s3` into `table1` and `table2`, respectively: ```sql LOAD LABEL test_db.label2 ( DATA INFILE("s3a://bucket_s3/input/file1.csv") INTO TABLE table1 COLUMNS TERMINATED BY "," (id, name, score) , DATA INFILE("s3a://bucket_s3/input/file2.csv") INTO TABLE table2 COLUMNS TERMINATED BY "," (id, city) ) WITH BROKER ( StorageCredentialParams ); ``` > **NOTE** > > Broker Load supports accessing AWS S3 only according to the S3A protocol. Therefore, when you load data from AWS S3, you must replace `s3://` in the S3 URI you pass as the file path with `s3a://`. In the preceding example, `StorageCredentialParams` represents a group of authentication parameters which vary depending on the authentication method you choose. For more information, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#aws-s3). From v3.1 onwards, StarRocks supports directly loading the data of Parquet-formatted or ORC-formatted files from AWS S3 by using the INSERT command and the TABLE keyword, saving you from the trouble of creating an external table first. For more information, see [Load data using INSERT > Insert data directly from files in an external source using TABLE keyword](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-directly-from-files-in-an-external-source-using-files). ###### Load data from Google GCS[​](#load-data-from-google-gcs "Direct link to Load data from Google GCS") Execute the following statement to load `file1.csv` and `file2.csv` from the `input` folder of your Google GCS bucket `bucket_gcs` into `table1` and `table2`, respectively: ```sql LOAD LABEL test_db.label3 ( DATA INFILE("gs://bucket_gcs/input/file1.csv") INTO TABLE table1 COLUMNS TERMINATED BY "," (id, name, score) , DATA INFILE("gs://bucket_gcs/input/file2.csv") INTO TABLE table2 COLUMNS TERMINATED BY "," (id, city) ) WITH BROKER ( StorageCredentialParams ); ``` > **NOTE** > > Broker Load supports accessing Google GCS only according to the gs protocol. Therefore, when you load data from Google GCS, you must include `gs://` as the prefix in the GCS URI that you pass as the file path. In the preceding example, `StorageCredentialParams` represents a group of authentication parameters which vary depending on the authentication method you choose. For more information, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#google-gcs). ###### Load data from other S3-compatible storage system[​](#load-data-from-other-s3-compatible-storage-system "Direct link to Load data from other S3-compatible storage system") Use MinIO as an example. You can execute the following statement to load `file1.csv` and `file2.csv` from the `input` folder of your MinIO bucket `bucket_minio` into `table1` and `table2`, respectively: ```sql LOAD LABEL test_db.label7 ( DATA INFILE("s3://bucket_minio/input/file1.csv") INTO TABLE table1 COLUMNS TERMINATED BY "," (id, name, score) , DATA INFILE("s3://bucket_minio/input/file2.csv") INTO TABLE table2 COLUMNS TERMINATED BY "," (id, city) ) WITH BROKER ( StorageCredentialParams ); ``` In the preceding example, `StorageCredentialParams` represents a group of authentication parameters which vary depending on the authentication method you choose. For more information, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#other-s3-compatible-storage-system). ###### Load data from Microsoft Azure Storage[​](#load-data-from-microsoft-azure-storage "Direct link to Load data from Microsoft Azure Storage") Execute the following statement to load `file1.csv` and `file2.csv` from the specified paths of your Azure Storage: ```sql LOAD LABEL test_db.label8 ( DATA INFILE("wasb[s]://@.blob.core.windows.net//file1.csv") INTO TABLE table1 COLUMNS TERMINATED BY "," (id, name, score) , DATA INFILE("wasb[s]://@.blob.core.windows.net//file2.csv") INTO TABLE table2 COLUMNS TERMINATED BY "," (id, city) ) WITH BROKER ( StorageCredentialParams ); ``` > **NOTICE** > > When you load data from Azure Storage, you need to determine which prefix to use based on the access protocol and specific storage service that you use. The preceding example uses Blob Storage as an example. > > * When you load data from Blob Storage, you must include `wasb://` or `wasbs://` as a prefix in the file path based on the protocol that is used to access your storage account: > > > > * If your Blob Storage allows access only through HTTP, use `wasb://` as the prefix, for example, `wasb://@.blob.core.windows.net///*`. > * If your Blob Storage allows access only through HTTPS, use `wasbs://` as the prefix, for example, `wasbs://@.blob.core.windows.net///*` > > * When you load data from Data Lake Storage Gen1, you must include `adl://` as a prefix in the file path, for example, `adl://.azuredatalakestore.net//`. > > * When you load data from Data Lake Storage Gen2, you must include `abfs://` or `abfss://` as a prefix in the file path based on the protocol that is used to access your storage account: > > > > * If your Data Lake Storage Gen2 allows access only via HTTP, use `abfs://` as the prefix, for example, `abfs://@.dfs.core.windows.net/`. > * If your Data Lake Storage Gen2 allows access only via HTTPS, use `abfss://` as the prefix, for example, `abfss://@.dfs.core.windows.net/`. In the preceding example, `StorageCredentialParams` represents a group of authentication parameters which vary depending on the authentication method you choose. For more information, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#microsoft-azure-storage). ###### Query data[​](#query-data "Direct link to Query data") After the load of data from your HDFS cluster, AWS S3 bucket, or Google GCS bucket is complete, you can use the SELECT statement to query the data of the StarRocks tables to verify that the load is successful. 1. Execute the following statement to query the data of `table1`: ```sql MySQL [test_db]> SELECT * FROM table1; +------+-------+-------+ | id | name | score | +------+-------+-------+ | 1 | Lily | 23 | | 2 | Rose | 23 | | 3 | Alice | 24 | | 4 | Julia | 25 | +------+-------+-------+ 4 rows in set (0.00 sec) ``` 2. Execute the following statement to query the data of `table2`: ```sql MySQL [test_db]> SELECT * FROM table2; +------+--------+ | id | city | +------+--------+ | 200 | Beijing| +------+--------+ 4 rows in set (0.01 sec) ``` ##### Create a single-table load job[​](#create-a-single-table-load-job "Direct link to Create a single-table load job") You can also load a single data file or all data files from a specified path into a single destination table. Suppose your AWS S3 bucket `bucket_s3` contains a folder named `input`. The `input` folder contains multiple data files, one of which is named `file1.csv`. These data files consist of the same number of columns as `table1` and the columns from each of these data files can be mapped one on one in sequence to the columns from `table1`. To load `file1.csv` into `table1`, execute the following statement: ```sql LOAD LABEL test_db.label_7 ( DATA INFILE("s3a://bucket_s3/input/file1.csv") INTO TABLE table1 COLUMNS TERMINATED BY "," FORMAT AS "CSV" ) WITH BROKER ( StorageCredentialParams ); ``` To load all data files from the `input` folder into `table1`, execute the following statement: ```sql LOAD LABEL test_db.label_8 ( DATA INFILE("s3a://bucket_s3/input/*") INTO TABLE table1 COLUMNS TERMINATED BY "," FORMAT AS "CSV" ) WITH BROKER ( StorageCredentialParams ); ``` In the preceding examples, `StorageCredentialParams` represents a group of authentication parameters which vary depending on the authentication method you choose. For more information, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#aws-s3). ##### View a load job[​](#view-a-load-job "Direct link to View a load job") Broker Load allows you to view a lob job by using the SHOW LOAD statement or the `curl` command. ###### Use SHOW LOAD[​](#use-show-load "Direct link to Use SHOW LOAD") For more information, see [SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md). ###### Use curl[​](#use-curl "Direct link to Use curl") The syntax is as follows: ```bash curl --location-trusted -u : \ 'http://:/api//_load_info?label=' ``` > **NOTE** > > If you use an account for which no password is set, you need to input only `:`. For example, you can run the following command to view the information about a load job, whose label is `label1`, in the `test_db` database: ```bash curl --location-trusted -u : \ 'http://:/api/test_db/_load_info?label=label1' ``` The `curl` command returns the information about the most recently executed load job with the specified label as a JSON object `jobInfo`: ```json {"jobInfo":{"dbName":"default_cluster:test_db","tblNames":["table1_simple"],"label":"label1","state":"FINISHED","failMsg":"","trackingUrl":""},"status":"OK","msg":"Success"}% ``` The following table describes the parameters in `jobInfo`. | **Parameter** | **Description** | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | dbName | The name of the database into which data is loaded | | tblNames | The name of the table into which data is loaded. | | label | The label of the load job. | | state | The status of the load job. Valid values:- `PENDING`: The load job is in queue waiting to be scheduled.
- `QUEUEING`: The load job is in the queue waiting to be scheduled.
- `LOADING`: The load job is running.
- `PREPARED`: The transaction has been committed.
- `FINISHED`: The load job succeeded.
- `CANCELLED`: The load job failed.For more information, see the "Asynchronous loading" section in [Loading concepts](https://docs.starrocks.io/docs/loading/loading_introduction/loading_concepts.md). | | failMsg | The reason why the load job failed. If the `state` value for the load job is `PENDING`, `LOADING`, or `FINISHED`, `NULL` is returned for the `failMsg` parameter. If the `state` value for the load job is `CANCELLED`, the value returned for the `failMsg` parameter consists of two parts: `type` and `msg`.- The `type` part can be any of the following values:
- The `msg` part provides the detailed cause of the load failure. | | trackingUrl | The URL that is used to access the unqualified data detected in the load job. You can use the `curl` or `wget` command to access the URL and obtain the unqualified data. If no unqualified data is detected, `NULL` is returned for the `trackingUrl` parameter. | | status | The status of the HTTP request for the load job. Valid values: `OK` and `Fail`. | | msg | The error information of the HTTP request for the load job. | ##### Cancel a load job[​](#cancel-a-load-job "Direct link to Cancel a load job") When a load job is not in the **CANCELLED** or **FINISHED** stage, you can use the [CANCEL LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/CANCEL_LOAD.md) statement to cancel the job. For example, you can execute the following statement to cancel a load job, whose label is `label1`, in the database `test_db`: ```sql CANCEL LOAD FROM test_db WHERE LABEL = "label"; ``` #### Job splitting and concurrent running[​](#job-splitting-and-concurrent-running "Direct link to Job splitting and concurrent running") A Broker Load job can be split into one or more tasks that concurrently run. The tasks within a load job are run within a single transaction. They must all succeed or fail. StarRocks splits each load job based on how you declare `data_desc` in the `LOAD` statement: * If you declare multiple `data_desc` parameters, each of which specifies a distinct table, a task is generated to load the data of each table. * If you declare multiple `data_desc` parameters, each of which specifies a distinct partition for the same table, a task is generated to load the data of each partition. Additionally, each task can be further split into one or more instances, which are evenly distributed to and concurrently run on the BEs of your StarRocks cluster. StarRocks splits each task based on the following [FE configurations](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md): * `min_bytes_per_broker_scanner`: the minimum amount of data processed by each instance. The default amount is 64 MB. * `load_parallel_instance_num`: the number of concurrent instances allowed in each load job on an individual BE. The default number is 1. You can use the following formula to calculate the number of instances in an individual task: **Number of instances in an individual task = min(Amount of data to be loaded by an individual task/`min_bytes_per_broker_scanner`,`load_parallel_instance_num` x Number of BEs)** In most cases, only one `data_desc` is declared for each load job, each load job is split into only one task, and the task is split into the same number of instances as the number of BEs. #### Related configuration items[​](#related-configuration-items "Direct link to Related configuration items") The [FE configuration item](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md) `max_broker_load_job_concurrency` specifies the maximum number of Broker Load jobs that can be concurrently run within your StarRocks cluster. In StarRocks v2.4 and earlier, if the total number of Broker Load jobs that are submitted within a specific period of time exceeds the maximum number, excessive jobs are queued and scheduled based on their submission time. Since StarRocks v2.5, if the total number of Broker Load jobs that are submitted within a specific period of time exceeds the maximum number, excessive jobs are queued and scheduled based on their priorities. You can specify a priority for a job by using the `priority` parameter at job creation. See [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#opt_properties). You can also use [ALTER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ALTER_LOAD.md) to modify the priority of an existing job that is in the **QUEUEING** or **LOADING** state. --- ### Transform data at loading StarRocks supports data transformation at loading. This feature supports [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md), [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md), and [Routine Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md) but does not support [Spark Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SPARK_LOAD.md). You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. This topic uses CSV data as an example to describe how to extract and transform data at loading. The data file formats that are supported vary depending on the loading method of your choice. > **NOTE** > > For CSV data, you can use a UTF-8 string, such as a comma (,), tab, or pipe (|), whose length does not exceed 50 bytes as a text delimiter. #### Scenarios[​](#scenarios "Direct link to Scenarios") When you load a data file into a StarRocks table, the data of the data file may not be completely mapped onto the data of the StarRocks table. In this situation, you do not need to extract or transform the data before you load it into the StarRocks table. StarRocks can help you extract and transform the data during loading: * Skip columns that do not need to be loaded. You can skip the columns that do not need to be loaded. Additionally, if the columns of the data file are in a different order than the columns of the StarRocks table, you can create a column mapping between the data file and the StarRocks table. * Filter out rows you do not want to load. You can specify filter conditions based on which StarRocks filters out the rows that you do not want to load. * Generate new columns from original columns. Generated columns are special columns that are computed from the original columns of the data file. You can map the generated columns onto the columns of the StarRocks table. * Extract partition field values from a file path. If the data file is generated from Apache Hive™, you can extract partition field values from the file path. #### Data examples[​](#data-examples "Direct link to Data examples") 1. Create data files in your local file system. a. Create a data file named `file1.csv`. The file consists of four columns, which represent user ID, user gender, event date, and event type in sequence. ```plain 354,female,2020-05-20,1 465,male,2020-05-21,2 576,female,2020-05-22,1 687,male,2020-05-23,2 ``` b. Create a data file named `file2.csv`. The file consists of only one column, which represents date. ```plain 2020-05-20 2020-05-21 2020-05-22 2020-05-23 ``` 2. Create tables in your StarRocks database `test_db`. > **NOTE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). a. Create a table named `table1`, which consists of three columns: `event_date`, `event_type`, and `user_id`. ```sql MySQL [test_db]> CREATE TABLE table1 ( `event_date` DATE COMMENT "event date", `event_type` TINYINT COMMENT "event type", `user_id` BIGINT COMMENT "user ID" ) DISTRIBUTED BY HASH(user_id); ``` b. Create a table named `table2`, which consists of four columns: `date`, `year`, `month`, and `day`. ```sql MySQL [test_db]> CREATE TABLE table2 ( `date` DATE COMMENT "date", `year` INT COMMENT "year", `month` TINYINT COMMENT "month", `day` TINYINT COMMENT "day" ) DISTRIBUTED BY HASH(date); ``` 3. Upload `file1.csv` and `file2.csv` to the `/user/starrocks/data/input/` path of your HDFS cluster, publish the data of `file1.csv` to `topic1` of your Kafka cluster, and publish the data of `file2.csv` to `topic2` of your Kafka cluster. #### Skip columns that do not need to be loaded[​](#skip-columns-that-do-not-need-to-be-loaded "Direct link to Skip columns that do not need to be loaded") The data file that you want to load into a StarRocks table may contain some columns that cannot be mapped to any columns of the StarRocks table. In this situation, StarRocks supports loading only the columns that can be mapped from the data file onto the columns of the StarRocks table. This feature supports loading data from the following data sources: * Local file system * HDFS and cloud storage > **NOTE** > > This section uses HDFS as an example. * Kafka In most cases, the columns of a CSV file are not named. For some CSV files, the first row is composed of column names, but StarRocks processes the content of the first row as common data rather than column names. Therefore, when you load a CSV file, you must temporarily name the columns of the CSV file **in sequence** in the job creation statement or command. These temporarily named columns are mapped **by name** onto the columns of the StarRocks table. Take note of the following points about the columns of the data file: * The data of the columns that can be mapped onto and are temporarily named by using the names of the columns in the StarRocks table is directly loaded. * The columns that cannot be mapped onto the columns of the StarRocks table are ignored, the data of these columns are not loaded. * If some columns can be mapped onto the columns of the StarRocks table but are not temporarily named in the job creation statement or command, the load job reports errors. This section uses `file1.csv` and `table1` as an example. The four columns of `file1.csv` are temporarily named as `user_id`, `user_gender`, `event_date`, and `event_type` in sequence. Among the temporarily named columns of `file1.csv`, `user_id`, `event_date`, and `event_type` can be mapped onto specific columns of `table1`, whereas `user_gender` cannot be mapped onto any column of `table1`. Therefore, `user_id`, `event_date`, and `event_type` are loaded into `table1`, but `user_gender` is not. ##### Load data[​](#load-data "Direct link to Load data") ###### Load data from a local file system[​](#load-data-from-a-local-file-system "Direct link to Load data from a local file system") If `file1.csv` is stored in your local file system, run the following command to create a [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) job: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "column_separator:," \ -H "columns: user_id, user_gender, event_date, event_type" \ -T file1.csv -XPUT \ http://:/api/test_db/table1/_stream_load ``` > **NOTE** > > If you choose Stream Load, you must use the `columns` parameter to temporarily name the columns of the data file to create a column mapping between the data file and the StarRocks table. For detailed syntax and parameter descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ###### Load data from an HDFS cluster[​](#load-data-from-an-hdfs-cluster "Direct link to Load data from an HDFS cluster") If `file1.csv` is stored in your HDFS cluster, execute the following statement to create a [Broker Load](https://docs.starrocks.io/docs/loading/hdfs_load.md) job: ```sql LOAD LABEL test_db.label1 ( DATA INFILE("hdfs://:/user/starrocks/data/input/file1.csv") INTO TABLE `table1` FORMAT AS "csv" COLUMNS TERMINATED BY "," (user_id, user_gender, event_date, event_type) ) WITH BROKER; ``` > **NOTE** > > If you choose Broker Load, you must use the `column_list` parameter to temporarily name the columns of the data file to create a column mapping between the data file and the StarRocks table. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Load data from a Kafka cluster[​](#load-data-from-a-kafka-cluster "Direct link to Load data from a Kafka cluster") If the data of `file1.csv` is published to `topic1` of your Kafka cluster, execute the following statement to create a [Routine Load](https://docs.starrocks.io/docs/loading/kafka/RoutineLoad.md) job: ```sql CREATE ROUTINE LOAD test_db.table101 ON table1 COLUMNS TERMINATED BY ",", COLUMNS(user_id, user_gender, event_date, event_type) FROM KAFKA ( "kafka_broker_list" = ":", "kafka_topic" = "topic1", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` > **NOTE** > > If you choose Routine Load, you must use the `COLUMNS` parameter to temporarily name the columns of the data file to create a column mapping between the data file and the StarRocks table. For detailed syntax and parameter descriptions, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### Query data[​](#query-data "Direct link to Query data") After the load of data from your local file system, HDFS cluster, or Kafka cluster is complete, query the data of `table1` to verify that the load is successful: ```sql MySQL [test_db]> SELECT * FROM table1; +------------+------------+---------+ | event_date | event_type | user_id | +------------+------------+---------+ | 2020-05-22 | 1 | 576 | | 2020-05-20 | 1 | 354 | | 2020-05-21 | 2 | 465 | | 2020-05-23 | 2 | 687 | +------------+------------+---------+ 4 rows in set (0.01 sec) ``` #### Filter out rows that you do not want to load[​](#filter-out-rows-that-you-do-not-want-to-load "Direct link to Filter out rows that you do not want to load") When you load a data file into a StarRocks table, you may not want to load specific rows of the data file. In this situation, you can use the WHERE clause to specify the rows that you want to load. StarRocks filters out all rows that do not meet the filter conditions specified in the WHERE clause. This feature supports loading data from the following data sources: * Local file system * HDFS and cloud storage > **NOTE** > > This section uses HDFS as an example. * Kafka This section uses `file1.csv` and `table1` as an example. If you want to load only the rows whose event type is `1` from `file1.csv` into `table1`, you can use the WHERE clause to specify a filter condition `event_type = 1`. ##### Load data[​](#load-data-1 "Direct link to Load data") ###### Load data from a local file system[​](#load-data-from-a-local-file-system-1 "Direct link to Load data from a local file system") If `file1.csv` is stored in your local file system, run the following command to create a [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) job: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "column_separator:," \ -H "columns: user_id, user_gender, event_date, event_type" \ -H "where: event_type=1" \ -T file1.csv -XPUT \ http://:/api/test_db/table1/_stream_load ``` For detailed syntax and parameter descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ###### Load data from an HDFS cluster[​](#load-data-from-an-hdfs-cluster-1 "Direct link to Load data from an HDFS cluster") If `file1.csv` is stored in your HDFS cluster, execute the following statement to create a [Broker Load](https://docs.starrocks.io/docs/loading/hdfs_load.md) job: ```sql LOAD LABEL test_db.label2 ( DATA INFILE("hdfs://:/user/starrocks/data/input/file1.csv") INTO TABLE `table1` FORMAT AS "csv" COLUMNS TERMINATED BY "," (user_id, user_gender, event_date, event_type) WHERE event_type = 1 ) WITH BROKER; ``` For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Load data from a Kafka cluster[​](#load-data-from-a-kafka-cluster-1 "Direct link to Load data from a Kafka cluster") If the data of `file1.csv` is published to `topic1` of your Kafka cluster, execute the following statement to create a [Routine Load](https://docs.starrocks.io/docs/loading/kafka/RoutineLoad.md) job: ```sql CREATE ROUTINE LOAD test_db.table102 ON table1 COLUMNS TERMINATED BY ",", COLUMNS (user_id, user_gender, event_date, event_type), WHERE event_type = 1 FROM KAFKA ( "kafka_broker_list" = ":", "kafka_topic" = "topic1", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` For detailed syntax and parameter descriptions, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### Query data[​](#query-data-1 "Direct link to Query data") After the load of data from your local file system, HDFS cluster, or Kafka cluster is complete, query the data of `table1` to verify that the load is successful: ```sql MySQL [test_db]> SELECT * FROM table1; +------------+------------+---------+ | event_date | event_type | user_id | +------------+------------+---------+ | 2020-05-20 | 1 | 354 | | 2020-05-22 | 1 | 576 | +------------+------------+---------+ 2 rows in set (0.01 sec) ``` #### Generate new columns from original columns[​](#generate-new-columns-from-original-columns "Direct link to Generate new columns from original columns") When you load a data file into a StarRocks table, some data of the data file may require conversions before the data can be loaded into the StarRocks table. In this situation, you can use functions or expressions in the job creation command or statement to implement data conversions. This feature supports loading data from the following data sources: * Local file system * HDFS and cloud storage > **NOTE** > > This section uses HDFS as an example. * Kafka This section uses `file2.csv` and `table2` as an example. `file2.csv` consists of only one column that represents date. You can use the [year](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/year.md), [month](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/month.md), and [day](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/day.md) functions to extract the year, month, and day in each date from `file2.csv` and load the extracted data into the `year`, `month`, and `day` columns of `table2`. ##### Load data[​](#load-data-2 "Direct link to Load data") ###### Load data from a local file system[​](#load-data-from-a-local-file-system-2 "Direct link to Load data from a local file system") If `file2.csv` is stored in your local file system, run the following command to create a [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) job: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "column_separator:," \ -H "columns:date,year=year(date),month=month(date),day=day(date)" \ -T file2.csv -XPUT \ http://:/api/test_db/table2/_stream_load ``` > **NOTE** > > * In the `columns` parameter, you must first temporarily name **all columns** of the data file, and then temporarily name the new columns that you want to generate from the original columns of the data file. As shown in the preceding example, the only column of `file2.csv` is temporarily named as `date`, and then the `year=year(date)`, `month=month(date)`, and `day=day(date)` functions are invoked to generate three new columns, which are temporarily named as `year`, `month`, and `day`. > > * Stream Load does not support `column_name = function(column_name)` but supports `column_name = function(column_name)`. For detailed syntax and parameter descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ###### Load data from an HDFS cluster[​](#load-data-from-an-hdfs-cluster-2 "Direct link to Load data from an HDFS cluster") If `file2.csv` is stored in your HDFS cluster, execute the following statement to create a [Broker Load](https://docs.starrocks.io/docs/loading/hdfs_load.md) job: ```sql LOAD LABEL test_db.label3 ( DATA INFILE("hdfs://:/user/starrocks/data/input/file2.csv") INTO TABLE `table2` FORMAT AS "csv" COLUMNS TERMINATED BY "," (date) SET(year=year(date), month=month(date), day=day(date)) ) WITH BROKER; ``` > **NOTE** > > You must first use the `column_list` parameter to temporarily name **all columns** of the data file, and then use the SET clause to temporarily name the new columns that you want to generate from the original columns of the data file. As shown in the preceding example, the only column of `file2.csv` is temporarily named as `date` in the `column_list` parameter, and then the `year=year(date)`, `month=month(date)`, and `day=day(date)` functions are invoked in the SET clause to generate three new columns, which are temporarily named as `year`, `month`, and `day`. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Load data from a Kafka cluster[​](#load-data-from-a-kafka-cluster-2 "Direct link to Load data from a Kafka cluster") If the data of `file2.csv` is published to `topic2` of your Kafka cluster, execute the following statement to create a [Routine Load](https://docs.starrocks.io/docs/loading/kafka/RoutineLoad.md) job: ```sql CREATE ROUTINE LOAD test_db.table201 ON table2 COLUMNS TERMINATED BY ",", COLUMNS(date,year=year(date),month=month(date),day=day(date)) FROM KAFKA ( "kafka_broker_list" = ":", "kafka_topic" = "topic2", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` > **NOTE** > > In the `COLUMNS` parameter, you must first temporarily name **all columns** of the data file, and then temporarily name the new columns that you want to generate from the original columns of the data file. As shown in the preceding example, the only column of `file2.csv` is temporarily named as `date`, and then the `year=year(date)`, `month=month(date)`, and `day=day(date)` functions are invoked to generate three new columns, which are temporarily named as `year`, `month`, and `day`. For detailed syntax and parameter descriptions, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### Query data[​](#query-data-2 "Direct link to Query data") After the load of data from your local file system, HDFS cluster, or Kafka cluster is complete, query the data of `table2` to verify that the load is successful: ```sql MySQL [test_db]> SELECT * FROM table2; +------------+------+-------+------+ | date | year | month | day | +------------+------+-------+------+ | 2020-05-20 | 2020 | 5 | 20 | | 2020-05-21 | 2020 | 5 | 21 | | 2020-05-22 | 2020 | 5 | 22 | | 2020-05-23 | 2020 | 5 | 23 | +------------+------+-------+------+ 4 rows in set (0.01 sec) ``` #### Extract partition field values from a file path[​](#extract-partition-field-values-from-a-file-path "Direct link to Extract partition field values from a file path") If the file path that you specify contains partition fields, you can use the `COLUMNS FROM PATH AS` parameter to specify the partition fields you want to extract from the file paths. The partition fields in file paths are equivalent to the columns in data files. The `COLUMNS FROM PATH AS` parameter is supported only when you load data from an HDFS cluster. For example, you want to load the following four data files generated from Hive: ```plain /user/starrocks/data/input/date=2020-05-20/data 1,354 /user/starrocks/data/input/date=2020-05-21/data 2,465 /user/starrocks/data/input/date=2020-05-22/data 1,576 /user/starrocks/data/input/date=2020-05-23/data 2,687 ``` The four data files are stored in the `/user/starrocks/data/input/` path of your HDFS cluster. Each of these data files is partitioned by partition field `date` and consists of two columns, which represent event type and user ID in sequence. ##### Load data from an HDFS cluster[​](#load-data-from-an-hdfs-cluster-3 "Direct link to Load data from an HDFS cluster") Execute the following statement to create a [Broker Load](https://docs.starrocks.io/docs/loading/hdfs_load.md) job, which enables you to extract the `date` partition field values from the `/user/starrocks/data/input/` file path and use a wildcard (\*) to specify that you want to load all data files in the file path to `table1`: ```sql LOAD LABEL test_db.label4 ( DATA INFILE("hdfs://:/user/starrocks/data/input/date=*/*") INTO TABLE `table1` FORMAT AS "csv" COLUMNS TERMINATED BY "," (event_type, user_id) COLUMNS FROM PATH AS (date) SET(event_date = date) ) WITH BROKER; ``` > **NOTE** > > In the preceding example, the `date` partition field in the specified file path is equivalent to the `event_date` column of `table1`. Therefore, you need to use the SET clause to map the `date` partition field onto the `event_date` column. If the partition field in the specified file path has the same name as a column of the StarRocks table, you do not need to use the SET clause to create a mapping. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ##### Query data[​](#query-data-3 "Direct link to Query data") After the load of data from your HDFS cluster is complete, query the data of `table1` to verify that the load is successful: ```sql MySQL [test_db]> SELECT * FROM table1; +------------+------------+---------+ | event_date | event_type | user_id | +------------+------------+---------+ | 2020-05-22 | 1 | 576 | | 2020-05-20 | 1 | 354 | | 2020-05-21 | 2 | 465 | | 2020-05-23 | 2 | 687 | +------------+------------+---------+ 4 rows in set (0.01 sec) ``` --- ### Realtime synchronization from MySQL StarRocks supports multiple methods to synchronize data from MySQL to StarRocks in real time, delivering low latency real-time analytics of massive data. This topic describes how to synchronize data from MySQL to StarRocks in real-time (within seconds) through Apache Flink®. You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. #### How it works[​](#how-it-works "Direct link to How it works") tip Flink CDC is used in the synchronization from MySQL to Flink. This topic uses Flink CDC whose version is less than 3.0, so SMT is used to synchronize table schemas. However, if Flink CDC 3.0 is used, it is not necessary to use SMT to synchronize table schemas to StarRocks. Flink CDC 3.0 can even synchronize the schemas of the entire MySQL database, the sharded databases and tables, and also supports schema changes synchronization. For detailed usage, see [Streaming ELT from MySQL to StarRocks](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.4/docs/get-started/quickstart/mysql-to-starrocks/). The following figure illustrates the entire synchronization process. ![img](/assets/images/4.9.2-ef6bd371aaac43f2d8b55e26499a3ee7.png) Real-time synchronization from MySQL through Flink to StarRocks is implemented in two stages: synchronizing database & table schema and synchronizing data. First, the SMT converts MySQL database & table schema into table creation statements for StarRocks. Then, the Flink cluster runs Flink jobs to synchronize full and incremental MySQL data to StarRocks. info The synchronization process guarantees exactly-once semantics. **Synchronization process**: 1. Synchronize database & table schema. The SMT reads the schema of the MySQL database & table to be synchronized and generates SQL files for creating a destination database & table in StarRocks. This operation is based on the MySQL and StarRocks information in SMT's configuration file. 2. Synchronize data. a. The Flink SQL client executes the data loading statement `INSERT INTO SELECT` to submit one or more Flink jobs to the Flink cluster. b. The Flink cluster runs the Flink jobs to obtain data. The Flink CDC connector first reads full historical data from the source database, then seamlessly switches to incremental reading, and sends the data to flink-connector-starrocks. c. flink-connector-starrocks accumulates data in mini-batches, and synchronizes each batch of data to StarRocks. info Only data manipulation language (DML) operations in MySQL can be synchronized to StarRocks. Data definition language (DDL) operations cannot be synchronized. #### Scenarios[​](#scenarios "Direct link to Scenarios") Real-time synchronization from MySQL has a broad range of use cases where data is constantly changed. Take a real-world use case "real-time ranking of commodity sales" as an example. Flink calculates the real-time ranking of commodity sales based on the original order table in MySQL and synchronizes the ranking to StarRocks' Primary Key table in real time. Users can connect a visualization tool to StarRocks to view the ranking in real time to gain on-demand operational insights. #### Preparations[​](#preparations "Direct link to Preparations") ##### Download and install synchronization tools[​](#download-and-install-synchronization-tools "Direct link to Download and install synchronization tools") To synchronize data from MySQL, you need to install the following tools: SMT, Flink, Flink CDC connector, and flink-connector-starrocks. 1. Download and install Flink, and start the Flink cluster. You can also perform this step by following the instructions in [Flink official documentation](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/try-flink/local_installation/). a. Install Java 8 or Java 11 in your operating system before you run Flink. You can run the following command to check the installed Java version. ```bash # View the Java version. java -version # Java 8 is installed if the following output is returned. java version "1.8.0_301" Java(TM) SE Runtime Environment (build 1.8.0_301-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.301-b09, mixed mode) ``` b. Download the [Flink installation package](https://flink.apache.org/downloads/) and decompress it. We recommend that you use Flink 1.14 or later. The minimum allowed version is Flink 1.11. This topic uses Flink 1.14.5. ```bash # Download Flink. wget https://archive.apache.org/dist/flink/flink-1.14.5/flink-1.14.5-bin-scala_2.11.tgz # Decompress Flink. tar -xzf flink-1.14.5-bin-scala_2.11.tgz # Go to the Flink directory. cd flink-1.14.5 ``` c. Start the Flink cluster. ```bash # Start the Flink cluster. ./bin/start-cluster.sh # The Flink cluster is started if the following output is returned. Starting cluster. Starting standalonesession daemon on host. Starting taskexecutor daemon on host. ``` 2. Download [Flink CDC connector](https://github.com/ververica/flink-cdc-connectors/releases). This topic uses MySQL as the data source and therefore, `flink-sql-connector-mysql-cdc-x.x.x.jar` is downloaded. The connector version must match the [Flink](https://github.com/ververica/flink-cdc-connectors/releases) version. This topic uses Flink 1.14.5 and you can download `flink-sql-connector-mysql-cdc-2.2.0.jar`. ```bash wget https://repo1.maven.org/maven2/com/ververica/flink-sql-connector-mysql-cdc/2.1.1/flink-sql-connector-mysql-cdc-2.2.0.jar ``` 3. Download [flink-connector-starrocks](https://search.maven.org/artifact/com.starrocks/flink-connector-starrocks). The version must match the Flink version. > The flink-connector-starrocks package `x.x.x_flink-y.yy _ z.zz.jar` contains three version numbers: > > * `x.x.x` is the version number of flink-connector-starrocks. > * `y.yy` is the supported Flink version. > * `z.zz` is the Scala version supported by Flink. If the Flink version is 1.14.x or earlier, you must download a package that has the Scala version. > > This topic uses Flink 1.14.5 and Scala 2.11. Therefore, you can download the following package: `1.2.3_flink-14_2.11.jar`. 4. Move the JAR packages of Flink CDC connector (`flink-sql-connector-mysql-cdc-2.2.0.jar`) and flink-connector-starrocks (`1.2.3_flink-1.14_2.11.jar`) to the `lib` directory of Flink. > **Note** > > If a Flink cluster is already running in your system, you must stop the Flink cluster and restart it to load and validate the JAR packages. > > ```bash > ./bin/stop-cluster.sh > ./bin/start-cluster.sh > > ``` 5. Download and decompress the [SMT package](https://www.starrocks.io/download/community) and place it in the `flink-1.14.5` directory. StarRocks provides SMT packages for Linux x86 and macos ARM64. You can choose one based on your operating system and CPU. ```bash # for Linux x86 wget https://releases.starrocks.io/resources/smt.tar.gz # for macOS ARM64 wget https://releases.starrocks.io/resources/smt_darwin_arm64.tar.gz ``` ##### Enable MySQL binary log[​](#enable-mysql-binary-log "Direct link to Enable MySQL binary log") To synchronize data from MySQL in real time, the system needs to read data from MySQL binary log (binlog), parse the data, and then synchronize the data to StarRocks. Make sure that MySQL binary log is enabled. 1. Edit the MySQL configuration file `my.cnf` (default path: `/etc/my.cnf`) to enable MySQL binary log. ```bash # Enable MySQL Binlog. log_bin = ON # Configure the save path for the Binlog. log_bin =/var/lib/mysql/mysql-bin # Configure server_id. # If server_id is not configured for MySQL 5.7.3 or later, the MySQL service cannot be used. server_id = 1 # Set the Binlog format to ROW. binlog_format = ROW # The base name of the Binlog file. An identifier is appended to identify each Binlog file. log_bin_basename =/var/lib/mysql/mysql-bin # The index file of Binlog files, which manages the directory of all Binlog files. log_bin_index =/var/lib/mysql/mysql-bin.index ``` 2. Run one of the following commands to restart MySQL for the modified configuration file to take effect. ```bash # Use service to restart MySQL. service mysqld restart # Use mysqld script to restart MySQL. /etc/init.d/mysqld restart ``` 3. Connect to MySQL and check whether MySQL binary log is enabled. ```plain -- Connect to MySQL. mysql -h xxx.xx.xxx.xx -P 3306 -u root -pxxxxxx -- Check whether MySQL binary log is enabled. mysql> SHOW VARIABLES LIKE 'log_bin'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | log_bin | ON | +---------------+-------+ 1 row in set (0.00 sec) ``` ##### Connect to StarRocks[​](#connect-to-starrocks "Direct link to Connect to StarRocks") The SMT configuration file sets these addresses in `flink.starrocks.jdbc-url` and `flink.starrocks.load-url`. Set the StarRocks connection options to addresses of the FE: * `jdbc-url`: `jdbc:mysql://:`. The query port defaults to `9030`. * `load-url`: `:`. The HTTP port defaults to `8030`. For ``, use an address that stays the same when an FE restarts or is replaced, such as a DNS name or a load balancer in front of the FEs, not the IP address of an FE. If the address stops leading to an FE, the Flink job can no longer load data. The Flink TaskManagers must be able to reach these FE ports, and also the HTTP port (default `8040`) of every BE or CN, because the FE redirects each load request to a BE or CN. #### Synchronize database & table schema[​](#synchronize-database--table-schema "Direct link to Synchronize database & table schema") 1. Edit the SMT configuration file. Go to the SMT `conf` directory and edit the configuration file `config_prod.conf`, such as MySQL connection information, the matching rules of the database & table to be synchronized, and configuration information of flink-connector-starrocks. ```bash [db] type = mysql host = xxx.xx.xxx.xx port = 3306 user = user1 password = xxxxxx [other] # Number of BEs in StarRocks be_num = 3 # `decimal_v3` is supported since StarRocks-1.18.1. use_decimal_v3 = true # File to save the converted DDL SQL output_dir = ./result [table-rule.1] # Pattern to match databases for setting properties database = ^demo.*$ # Pattern to match tables for setting properties table = ^.*$ ############################################ ### Flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`. They are auto-generated. ############################################ flink.starrocks.jdbc-url=jdbc:mysql://: flink.starrocks.load-url= : flink.starrocks.username=user2 flink.starrocks.password=xxxxxx flink.starrocks.sink.properties.format=csv flink.starrocks.sink.properties.column_separator=\x01 flink.starrocks.sink.properties.row_delimiter=\x02 flink.starrocks.sink.buffer-flush.interval-ms=15000 ``` * `[db]`: information used to access the source database. * `type`: type of the source database. In this topic, the source database is `mysql`. * `host`: IP address of the MySQL server. * `port`: port number of the MySQL database, defaults to `3306` * `user`: username for accessing the MySQL database * `password`: password of the username * `[table-rule]`: database & table matching rules and the corresponding flink-connector-starrocks configuration. * `Database`, `table`: the names of the database & table in MySQL. Regular expressions are supported. * `flink.starrocks.*`: configuration information of flink-connector-starrocks. For more configurations and information, see [flink-connector-starrocks](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks.md). > If you need to use different flink-connector-starrocks configurations for different tables. For example, if some tables are frequently updated and you need to accelerate data loading, see [Use different flink-connector-starrocks configurations for different tables](#use-different-flink-connector-starrocks-configurations-for-different-tables). If you need to load multiple tables obtained from MySQL sharding into the same StarRocks table, see [Synchronize multiple tables after MySQL sharding to one table in StarRocks](#synchronize-multiple-tables-after-mysql-sharding-to-one-table-in-starrocks). * `[other]`: other information * `be_num`: The number of BEs in your StarRocks cluster (This parameter will be used for setting a reasonable number of tablets in subsequent StarRocks table creation). * `use_decimal_v3`: Whether to enable [Decimal V3](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/DECIMAL.md). After Decimal V3 is enabled, MySQL decimal data will be converted into Decimal V3 data when data is synchronized to StarRocks. * `output_dir`: The path to save the SQL files to be generated. The SQL files will be used to create a database & table in StarRocks and submit a Flink job to the Flink cluster. The default path is `./result` and we recommend that you retain the default settings. 2. Run the SMT to read the database & table schema in MySQL and generate SQL files in the `./result` directory based on the configuration file. The `starrocks-create.all.sql` file is used to create a database & table in StarRocks and the `flink-create.all.sql` file is used to submit a Flink job to the Flink cluster. ```bash # Run the SMT. ./starrocks-migrate-tool # Go to the result directory and check the files in this directory. cd result ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql ``` 3. Run the following command to connect to StarRocks and execute the `starrocks-create.all.sql` file to create a database and table in StarRocks. We recommend that you use the default table creation statement in the SQL file to create a table of the [Primary Key table](https://docs.starrocks.io/docs/table_design/table_types/primary_key_table.md). > **Note** > > You can also modify the table creation statement based on your business needs and create a table that does not use the Primary Key table. However, the DELETE operation in the source MySQL database cannot be synchronized to the non- Primary Key table. Exercise caution when you create such a table. ```bash mysql -h -P -u user2 -pxxxxxx < starrocks-create.all.sql ``` If the data needs to be processed by Flink before it is written to the destination StarRocks table, the table schema will be different between the source and destination tables. In this case, you must modify the table creation statement. In this example, the destination table requires only the `product_id` and `product_name` columns and real-time ranking of commodity sales. You can use the following table creation statement. ```bash CREATE DATABASE IF NOT EXISTS `demo`; CREATE TABLE IF NOT EXISTS `demo`.`orders` ( `product_id` INT(11) NOT NULL COMMENT "", `product_name` STRING NOT NULL COMMENT "", `sales_cnt` BIGINT NOT NULL COMMENT "" ) ENGINE=olap PRIMARY KEY(`product_id`) DISTRIBUTED BY HASH(`product_id`) PROPERTIES ( "replication_num" = "3" ); ``` > **NOTICE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). #### Synchronize data[​](#synchronize-data "Direct link to Synchronize data") Run the Flink cluster and submit a Flink job to continuously synchronize full and incremental data from MySQL to StarRocks. 1. Go to the Flink directory and run the following command to run the `flink-create.all.sql` file on your Flink SQL client. ```bash ./bin/sql-client.sh -f flink-create.all.sql ``` This SQL file defines dynamic tables `source table` and `sink table`, query statement `INSERT INTO SELECT`, and specifies the connector, source database, and destination database. After this file is executed, a Flink job is submitted to the Flink cluster to start data synchronization. > **Note** > > * Make sure that the Flink cluster has been started. You can start the Flink cluster by running `flink/bin/start-cluster.sh`. > * If your Flink version is earlier than 1.13, you may not be able to directly run the SQL file `flink-create.all.sql`. You need to execute SQL statements one by one in this file in the command line interface (CLI) of the SQL client. You also need to escape the `\` character. > > ```bash > 'sink.properties.column_separator' = '\\x01' > 'sink.properties.row_delimiter' = '\\x02' > > ``` **Process data during synchronization**: If you need to process data during synchronization, such as performing GROUP BY or JOIN on the data, you can modify the `flink-create.all.sql` file. The following example calculates real-time ranking of commodity sales by executing COUNT (\*) and GROUP BY. ```bash $ ./bin/sql-client.sh -f flink-create.all.sql No default environment is specified. Searching for '/home/disk1/flink-1.13.6/conf/sql-client-defaults.yaml'...not found. [INFO] Executing SQL from file. Flink SQL> CREATE DATABASE IF NOT EXISTS `default_catalog`.`demo`; [INFO] Execute statement succeed. -- Create a dynamic table `source table` based on the order table in MySQL. Flink SQL> CREATE TABLE IF NOT EXISTS `default_catalog`.`demo`.`orders_src` (`order_id` BIGINT NOT NULL, `product_id` INT NULL, `order_date` TIMESTAMP NOT NULL, `customer_name` STRING NOT NULL, `product_name` STRING NOT NULL, `price` DECIMAL(10, 5) NULL, PRIMARY KEY(`order_id`) NOT ENFORCED ) with ('connector' = 'mysql-cdc', 'hostname' = 'xxx.xx.xxx.xxx', 'port' = '3306', 'username' = 'root', 'password' = '', 'database-name' = 'demo', 'table-name' = 'orders' ); [INFO] Execute statement succeed. -- Create a dynamic table `sink table`. Flink SQL> CREATE TABLE IF NOT EXISTS `default_catalog`.`demo`.`orders_sink` (`product_id` INT NOT NULL, `product_name` STRING NOT NULL, `sales_cnt` BIGINT NOT NULL, PRIMARY KEY(`product_id`) NOT ENFORCED ) with ('sink.max-retries' = '10', 'jdbc-url' = 'jdbc:mysql://:', 'password' = '', 'sink.properties.strip_outer_array' = 'true', 'sink.properties.format' = 'json', 'load-url' = ':', 'username' = 'root', 'sink.buffer-flush.interval-ms' = '15000', 'connector' = 'starrocks', 'database-name' = 'demo', 'table-name' = 'orders' ); [INFO] Execute statement succeed. -- Implement real-time ranking of commodity sales, where `sink table` is dynamically updated to reflect data changes in `source table`. Flink SQL> INSERT INTO `default_catalog`.`demo`.`orders_sink` select product_id,product_name, count(*) as cnt from `default_catalog`.`demo`.`orders_src` group by product_id,product_name; [INFO] Submitting SQL update statement to the cluster... [INFO] SQL update statement has been successfully submitted to the cluster: Job ID: 5ae005c4b3425d8bb13fe660260a35da ``` If you only need to synchronize only a portion of the data, such as data whose payment time is later than December 21, 2021, you can use the `WHERE` clause in `INSERT INTO SELECT` to set a filter condition, such as `WHERE pay_dt > '2021-12-21'`. Data that does not meet this condition will not be synchronized to StarRocks. If the following result is returned, the Flink job has been submitted for full and incremental synchronization. ```sql [INFO] Submitting SQL update statement to the cluster... [INFO] SQL update statement has been successfully submitted to the cluster: Job ID: 5ae005c4b3425d8bb13fe660260a35da ``` 2. You can use the [Flink WebUI](https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/try-flink/flink-operations-playground/#flink-webui) or run the `bin/flink list -running` command on your Flink SQL client to view Flink jobs that are running in the Flink cluster and the job IDs. * Flink WebUI ![img](/assets/images/4.9.3-9e843337839f2cf335718301cd414239.png) * `bin/flink list -running` ```bash $ bin/flink list -running Waiting for response... ------------------ Running/Restarting Jobs ------------------- 13.10.2022 15:03:54 : 040a846f8b58e82eb99c8663424294d5 : insert-into_default_catalog.lily.example_tbl1_sink (RUNNING) -------------------------------------------------------------- ``` > **Note** > > If the job is abnormal, you can perform troubleshooting by using Flink WebUI or by viewing the log file in the `/log` directory of Flink 1.14.5. #### FAQ[​](#faq "Direct link to FAQ") ##### Use different flink-connector-starrocks configurations for different tables[​](#use-different-flink-connector-starrocks-configurations-for-different-tables "Direct link to Use different flink-connector-starrocks configurations for different tables") If some tables in the data source are frequently updated and you want to accelerate the loading speed of flink-connector-starrocks, you must set a separate flink-connector-starrocks configuration for each table in the SMT configuration file `config_prod.conf`. ```bash [table-rule.1] # Pattern to match databases for setting properties database = ^order.*$ # Pattern to match tables for setting properties table = ^.*$ ############################################ ### Flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`. They are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://: flink.starrocks.load-url= : flink.starrocks.username=user2 flink.starrocks.password=xxxxxx flink.starrocks.sink.properties.format=csv flink.starrocks.sink.properties.column_separator=\x01 flink.starrocks.sink.properties.row_delimiter=\x02 flink.starrocks.sink.buffer-flush.interval-ms=15000 [table-rule.2] # Pattern to match databases for setting properties database = ^order2.*$ # Pattern to match tables for setting properties table = ^.*$ ############################################ ### Flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`. They are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://: flink.starrocks.load-url= : flink.starrocks.username=user2 flink.starrocks.password=xxxxxx flink.starrocks.sink.properties.format=csv flink.starrocks.sink.properties.column_separator=\x01 flink.starrocks.sink.properties.row_delimiter=\x02 flink.starrocks.sink.buffer-flush.interval-ms=10000 ``` ##### Synchronize multiple tables after MySQL sharding to one table in StarRocks[​](#synchronize-multiple-tables-after-mysql-sharding-to-one-table-in-starrocks "Direct link to Synchronize multiple tables after MySQL sharding to one table in StarRocks") After sharding is performed, data in one MySQL table may be split into multiple tables or even distributed to multiple databases. All the tables have the same schema. In this case, you can set `[table-rule]` to synchronize these tables to one StarRocks table. For example, MySQL has two databases `edu_db_1` and `edu_db_2`, each of which has two tables `course_1 and course_2`, and the schema of all tables is the same. You can use the following `[table-rule]` configuration to synchronize all the tables to one StarRocks table. > **Note** > > The name of the StarRocks table defaults to `course__auto_shard`. If you need to use a different name, you can modify it in the SQL files `starrocks-create.all.sql` and `flink-create.all.sql` ```bash [table-rule.1] # Pattern to match databases for setting properties database = ^edu_db_[0-9]*$ # Pattern to match tables for setting properties table = ^course_[0-9]*$ ############################################ ### Flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`. They are auto-generated ############################################ flink.starrocks.jdbc-url = jdbc: mysql://xxx.xxx.x.x:xxxx flink.starrocks.load-url = xxx.xxx.x.x:xxxx flink.starrocks.username = user2 flink.starrocks.password = xxxxxx flink.starrocks.sink.properties.format=csv flink.starrocks.sink.properties.column_separator =\x01 flink.starrocks.sink.properties.row_delimiter =\x02 flink.starrocks.sink.buffer-flush.interval-ms = 5000 ``` ##### Import data in JSON format[​](#import-data-in-json-format "Direct link to Import data in JSON format") Data in the preceding example is imported in CSV format. If you are unable to choose a suitable delimiter, you need to replace the following parameters of `flink.starrocks.*` in `[table-rule]`. ```plain flink.starrocks.sink.properties.format=csv flink.starrocks.sink.properties.column_separator =\x01 flink.starrocks.sink.properties.row_delimiter =\x02 ``` Data is imported in JSON format after the following parameters are passed in. ```plain flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.strip_outer_array=true ``` > **Note** > > This method slightly slows down the loading speed. ##### Execute multiple INSERT INTO statements as one Flink job[​](#execute-multiple-insert-into-statements-as-one-flink-job "Direct link to Execute multiple INSERT INTO statements as one Flink job") You can use the [STATEMENT SET](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sqlclient/#execute-a-set-of-sql-statements) syntax in the `flink-create.all.sql` file to execute multiple INSERT INTO statements as one Flink job, which prevents multiple statements from taking up too many Flink job resources and improves the efficiency of executing multiple queries. > **Note** > > Flink supports the STATEMENT SET syntax from 1.13 onwards. 1. Open the `result/flink-create.all.sql` file. 2. Modify the SQL statements in the file. Move all the INSERT INTO statements to the end of the file. Place `EXECUTE STATEMENT SET BEGIN` before the first INSERT INTO statement and place `END;` after the last INSERT INTO statement. > **Note** > > The positions of CREATE DATABASE and CREATE TABLE remain unchanged. ```sql CREATE DATABASE IF NOT EXISTS db; CREATE TABLE IF NOT EXISTS db.a1; CREATE TABLE IF NOT EXISTS db.b1; CREATE TABLE IF NOT EXISTS db.a2; CREATE TABLE IF NOT EXISTS db.b2; EXECUTE STATEMENT SET BEGIN-- one or more INSERT INTO statements INSERT INTO db.a1 SELECT * FROM db.b1; INSERT INTO db.a2 SELECT * FROM db.b2; END; ``` --- ### Realtime synchronization from PostgreSQL This topic shows how to stream data from PostgreSQL into StarRocks with a [Flink CDC](https://nightlies.apache.org/flink/flink-cdc-docs-stable/) pipeline. The pipeline copies the existing rows of each table, then keeps applying every INSERT, UPDATE, and DELETE from PostgreSQL within seconds. It creates the StarRocks tables for you, so there is no separate schema migration step. You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. #### How it works[​](#how-it-works "Direct link to How it works") A Flink CDC pipeline is a YAML file with a **source** (PostgreSQL), a **sink** (StarRocks), and optional **route** and **transform** rules. You submit the file to a Flink cluster, and it runs as a Flink job: 1. The PostgreSQL source takes a snapshot of each matching table, then reads changes from the PostgreSQL write-ahead log (WAL) through a logical replication slot. 2. The StarRocks sink creates a [Primary Key table](https://docs.starrocks.io/docs/table_design/table_types/primary_key_table.md) for each source table that does not already exist in StarRocks, then loads the rows with [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md). Because every table has a primary key, updates overwrite the existing row and deletes remove it. Delivery is at-least-once: after a failure, some changes can be written again, and the Primary Key table makes the repeat harmless. note The pipeline does not synchronize schema changes from PostgreSQL. See [Change a table's schema](#change-a-tables-schema). #### Before you begin[​](#before-you-begin "Direct link to Before you begin") You need: * PostgreSQL 10 or later. This topic uses the built-in `pgoutput` logical decoding plugin, so no server extension is required. * A StarRocks cluster. * A Flink 1.20 cluster, and Java 11 or 17 to run it. * Network access from every Flink TaskManager to PostgreSQL, on its port (default `5432`), and to StarRocks, as described in [Connect to StarRocks](#connect-to-starrocks). * A primary key on every PostgreSQL table you synchronize. StarRocks creates the destination table with the same primary key. #### Step 1: Prepare PostgreSQL[​](#step-1-prepare-postgresql "Direct link to Step 1: Prepare PostgreSQL") Run these steps as a PostgreSQL superuser, or as the owner of the tables. The examples synchronize the tables in the `public` schema of a database named `shop`. 1. Enable logical replication. Set `wal_level` to `logical` in `postgresql.conf`: ```properties wal_level = logical # Each running pipeline uses one replication slot and one WAL sender. max_replication_slots = 4 max_wal_senders = 4 ``` Changing `wal_level` requires a restart of PostgreSQL. For a managed service such as Amazon RDS or Cloud SQL, set the equivalent parameter (for example, `rds.logical_replication = 1`) in the instance's parameter group instead. Check the setting after the restart: ```sql SHOW wal_level; ``` ```plaintext wal_level ----------- logical ``` 2. Create a user for Flink CDC. It needs the `REPLICATION` attribute and read access to the tables: ```sql CREATE ROLE flink_cdc WITH LOGIN REPLICATION PASSWORD ''; GRANT CONNECT ON DATABASE shop TO flink_cdc; GRANT USAGE ON SCHEMA public TO flink_cdc; GRANT SELECT ON ALL TABLES IN SCHEMA public TO flink_cdc; ``` If `pg_hba.conf` restricts replication connections, allow this user to connect from the Flink TaskManager hosts. For example: ```plaintext # TYPE DATABASE USER ADDRESS METHOD host shop flink_cdc 10.0.0.0/24 scram-sha-256 host replication flink_cdc 10.0.0.0/24 scram-sha-256 ``` 3. Create a publication for the tables to synchronize. It must list every table that the pipeline reads (the `tables` option in [Step 4](#step-4-define-the-pipeline)): ```sql CREATE PUBLICATION flink_pub FOR TABLE public.orders, public.customers; ``` tip Create the publication yourself. If none exists, the connector tries to run `CREATE PUBLICATION ... FOR ALL TABLES`, which only a superuser can do. For any other user that fails, and Flink retries until the error it reports is about `max_wal_senders` rather than the missing publication. 4. Set the replica identity of each table to `FULL`, so that the WAL records the complete previous row for every UPDATE and DELETE: ```sql ALTER TABLE public.orders REPLICA IDENTITY FULL; ALTER TABLE public.customers REPLICA IDENTITY FULL; ``` Do this before you start the pipeline. With the default replica identity, the first UPDATE or DELETE fails the Flink job with a `NullPointerException`, and the job keeps failing on that change after every restart. #### Step 2: Prepare StarRocks[​](#step-2-prepare-starrocks "Direct link to Step 2: Prepare StarRocks") Create the destination database and a user for the pipeline: ```sql CREATE DATABASE shop; CREATE USER flink_sink IDENTIFIED BY ''; GRANT CREATE TABLE ON DATABASE shop TO USER flink_sink; GRANT SELECT, INSERT, UPDATE, DELETE, ALTER ON ALL TABLES IN DATABASE shop TO USER flink_sink; ``` #### Step 3: Install Flink CDC[​](#step-3-install-flink-cdc "Direct link to Step 3: Install Flink CDC") 1. Download and start Flink 1.20. See [First steps](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/try-flink/local_installation/) in the Flink documentation. 2. Turn on checkpointing by adding this line to the Flink configuration file, `conf/config.yaml`, and then restart the cluster: ```yaml execution.checkpointing.interval: 10s ``` The PostgreSQL source reports its position in the WAL back to PostgreSQL as checkpoints complete. Without checkpoints, the replication slot keeps every WAL segment since the pipeline started, and the PostgreSQL disk fills up. 3. Download the Flink CDC release that matches your Flink version from the [Flink CDC downloads](https://flink.apache.org/downloads/#apache-flink-cdc) page, and extract it. Starting with Flink CDC 3.6, each release is built for one Flink version, so for Flink 1.20 download `flink-cdc-3.6.0-1.20-bin.tar.gz`, not the `-2.2` package. ```bash tar -xzf flink-cdc-3.6.0-1.20-bin.tar.gz cd flink-cdc-3.6.0-1.20 ``` 4. Download the PostgreSQL and StarRocks pipeline connectors from Maven Central into the `lib` directory of Flink CDC. Use the same version as the Flink CDC release: ```bash cd lib wget https://repo1.maven.org/maven2/org/apache/flink/flink-cdc-pipeline-connector-postgres/3.6.0-1.20/flink-cdc-pipeline-connector-postgres-3.6.0-1.20.jar wget https://repo1.maven.org/maven2/org/apache/flink/flink-cdc-pipeline-connector-starrocks/3.6.0-1.20/flink-cdc-pipeline-connector-starrocks-3.6.0-1.20.jar cd .. ``` #### Step 4: Define the pipeline[​](#step-4-define-the-pipeline "Direct link to Step 4: Define the pipeline") Create a file named `postgres-to-starrocks.yaml`: ```yaml source: type: postgres name: PostgreSQL source hostname: port: 5432 username: flink_cdc password: # Comma-separated ..
entries; the table part can be a # regular expression. List the same tables as the publication. tables: shop.public.orders, shop.public.customers # Replication slot to create and use. Lowercase letters, digits, and underscores only. slot.name: flink_starrocks decoding.plugin.name: pgoutput debezium.publication.name: flink_pub sink: type: starrocks name: StarRocks sink # For the StarRocks addresses, see "Connect to StarRocks" below. jdbc-url: jdbc:mysql://:9030 load-url: :8030 username: flink_sink password: # The default is 300000 (5 minutes). A lower value makes changes visible sooner. sink.buffer-flush.interval-ms: 5000 route: # Write each table in the public schema to the StarRocks database shop. - source-table: public.\.* sink-table: shop.<> replace-symbol: <> pipeline: name: PostgreSQL to StarRocks parallelism: 1 ``` Pay attention to these settings: * `tables` must match the publication. The snapshot copies every table that `tables` selects, but PostgreSQL streams changes only for the tables in the publication. A table outside the publication is copied once and then never updated, and the job reports no error. * `tables` names tables as `..
`. Inside the pipeline, however, a PostgreSQL table is identified only by `.
`, so `route` and `transform` rules match `public.orders`, not `shop.public.orders`. * Without the `route` block, the sink writes each table to a StarRocks database named after the PostgreSQL **schema**, so the tables in `public` land in a database called `public`. * `sink.buffer-flush.interval-ms` defaults to 5 minutes. With the default, a new pipeline looks idle for several minutes after it starts. * To set properties on the tables that the sink creates, add `table.create.properties.` to the `sink` section, for example `table.create.properties.replication_num: 1`. For every source and sink option, see the Flink CDC [Postgres connector](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/pipeline-connectors/postgres/) and [StarRocks connector](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/pipeline-connectors/starrocks/) references. ##### Connect to StarRocks[​](#connect-to-starrocks "Direct link to Connect to StarRocks") Set the StarRocks connection options to addresses of the FE: * `jdbc-url`: `jdbc:mysql://:`. The query port defaults to `9030`. * `load-url`: `:`. The HTTP port defaults to `8030`. For ``, use an address that stays the same when an FE restarts or is replaced, such as a DNS name or a load balancer in front of the FEs, not the IP address of an FE. If the address stops leading to an FE, the Flink job can no longer load data. The Flink TaskManagers must be able to reach these FE ports, and also the HTTP port (default `8040`) of every BE or CN, because the FE redirects each load request to a BE or CN. ##### Columns with a function as the default value[​](#columns-with-a-function-as-the-default-value "Direct link to Columns with a function as the default value") The sink copies each column's default value from PostgreSQL into the StarRocks CREATE TABLE statement. A default that is a function call, such as `DEFAULT now()`, is not valid in StarRocks, and the job fails with an error like this one: ```plaintext Invalid default value for 'created_at': date literal [now()] is invalid. ``` Creating the StarRocks table yourself beforehand does not avoid the error, because the sink still sends its CREATE TABLE statement. Instead, add a `transform` rule that recomputes the column. A computed column has no default value: ```yaml transform: - source-table: public.orders projection: order_id, customer, amount, status, CAST(created_at AS TIMESTAMP(6)) AS created_at ``` List every column of the table in `projection`. A column that is not listed is not synchronized. #### Step 5: Start the pipeline[​](#step-5-start-the-pipeline "Direct link to Step 5: Start the pipeline") Set `FLINK_HOME` to your Flink installation, and submit the pipeline from the Flink CDC directory: ```bash export FLINK_HOME=/path/to/flink-1.20 bin/flink-cdc.sh postgres-to-starrocks.yaml ``` ```plaintext Pipeline has been submitted to cluster. Job ID: cda97bac3b504442d8106a2d70c6074c Job Description: PostgreSQL to StarRocks ``` The job appears in the Flink web UI (by default `http://:8081`). Its status should stay `RUNNING`. If it changes to `RESTARTING`, open the job's **Exceptions** tab, and see [Troubleshooting](#troubleshooting). #### Step 6: Verify the synchronization[​](#step-6-verify-the-synchronization "Direct link to Step 6: Verify the synchronization") 1. After the snapshot finishes, the existing rows are in StarRocks: ```sql SELECT * FROM shop.orders ORDER BY order_id; ``` 2. Change some data in PostgreSQL: ```sql INSERT INTO public.orders (order_id, customer, amount, status) VALUES (5, 'erin', 42.00, 'new'); UPDATE public.orders SET status = 'returned' WHERE order_id = 3; DELETE FROM public.orders WHERE order_id = 4; ``` 3. Within a few seconds (the flush interval plus the load time), run the query in StarRocks again. It returns the same rows as the same query in PostgreSQL. #### Manage the pipeline[​](#manage-the-pipeline "Direct link to Manage the pipeline") ##### Monitor the replication slot[​](#monitor-the-replication-slot "Direct link to Monitor the replication slot") A replication slot keeps WAL on the PostgreSQL server until the pipeline confirms that it has read it. While the pipeline is stopped, or if it falls behind, WAL builds up. Check how much WAL the slot is holding: ```sql SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS retained_wal FROM pg_replication_slots; ``` If you stop a pipeline for good, drop its slot so that PostgreSQL can release the WAL: ```sql SELECT pg_drop_replication_slot('flink_starrocks'); ``` ##### Change a table's schema[​](#change-a-tables-schema "Direct link to Change a table's schema") The pipeline does not pick up schema changes from PostgreSQL. If you add a column in PostgreSQL, the job keeps running, but it does not write the new column, even after you add the column to the StarRocks table. To synchronize a table after a schema change: 1. Make the same change to the StarRocks table, for example with [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md). 2. [Take a new snapshot](#take-a-new-snapshot). ##### Take a new snapshot[​](#take-a-new-snapshot "Direct link to Take a new snapshot") To copy the tables again from the current state of PostgreSQL and then resume streaming: 1. Cancel the Flink job. 2. Drop the replication slot, as shown in [Monitor the replication slot](#monitor-the-replication-slot). 3. Truncate every StarRocks table that the pipeline writes to: ```sql TRUNCATE TABLE shop.orders; TRUNCATE TABLE shop.customers; ``` 4. Submit the pipeline again. The pipeline takes a new snapshot of every table and then continues streaming changes. Do not skip the truncate. Changes made in PostgreSQL between cancelling the job and the new snapshot are not replayed. The snapshot overwrites the rows that still exist, but a row deleted in PostgreSQL during that time would stay in StarRocks. The truncated tables are empty, or only partly loaded, until the snapshot finishes. #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") When the job is `RESTARTING`, the **Exceptions** tab of the Flink web UI shows the cause. Look for the innermost `Caused by` line. | Error | Cause and solution | | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `number of requested standby connections exceeds max_wal_senders` | Often a symptom of an earlier failure, because every retry opens a replication connection. Check the PostgreSQL server log for the first error. A common one is `permission denied for database` on `CREATE PUBLICATION`, which means the publication does not exist. See [Step 1](#step-1-prepare-postgresql). | | `NullPointerException` in `DebeziumSchemaDataTypeInference` or `extractBeforeDataRecord` | A table does not have `REPLICA IDENTITY FULL`. Set it, and then [take a new snapshot](#take-a-new-snapshot). | | `Invalid default value for ''` | A column's PostgreSQL default is a function. See [Columns with a function as the default value](#columns-with-a-function-as-the-default-value). | | `Connect to :8040 ... Connection refused` | The TaskManager cannot reach the BE or CN that the FE redirected the load to. Make sure every BE and CN HTTP port is reachable from the TaskManagers at the address the BE or CN is registered with. | | Nothing arrives in StarRocks, but the job is `RUNNING` | Wait for the flush interval. The default `sink.buffer-flush.interval-ms` is 5 minutes. | | Rows arrive in a database named `public` | The pipeline has no `route` rule. See [Step 4](#step-4-define-the-pipeline). | #### See also[​](#see-also "Direct link to See also") * [Realtime synchronization from MySQL](https://docs.starrocks.io/docs/loading/Flink_cdc_load.md) * [Continuously load data from Apache Flink®](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks.md) * [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md) --- ### Load data from Apache Flink® with Multi-table Transaction StarRocks Flink Connector supports Multi-table Transaction to load data from Flink into multiple tables atomically. #### Use Cases[​](#use-cases "Direct link to Use Cases") When a single Flink job writes to multiple tables within the same StarRocks database in one processing cycle, enabling multi-table transaction guarantees: * **Cross-table atomic commit**: Data written to different table within the same commit cycle becomes visible atomically — all or nothing. * **Source transaction integrity**: A complete upstream transaction (for example, from Kafka) is never split across two StarRocks transactions. * **Sub-second data freshness**: Data continuously flows into StarRocks via `/api/transaction/load`, and is committed at the interval configured by `sink.buffer-flush.interval-ms`. Typical scenarios: * Synchronous writes to a general table and a detail table (for example, `orders` and `order_items`) * Event routing to different partition tables (for example, `events_202601`, `events_202602`) * A single job maintaining multiple interrelated downstream result tables Prerequisites To enable Multi-table Transaction, you must running your cluster on StarRocks v4.0 and later (with the Multi-table Transaction Stream Load support), and StarRocks Flink Connector on v1.2.9 and later. #### Core Capabilities[​](#core-capabilities "Direct link to Core Capabilities") | Capability | Description | | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Cross-table atomic commit | All tables within the same flush cycle share one StarRocks transaction label. The Prepare and Commit operations are unified. | | Source transaction integrity | Commit timing is controlled by the `transactionEnd` flag. Commit only occurs at complete source transaction boundaries. | | Sub-second data visibility | Data is periodically flushed to StarRocks (`/api/transaction/load`). It is committed when the `transactionEnd` and the timer conditions are met | | N:1 transaction mapping | Multiple source transactions can accumulate in a single StarRocks transaction. They do not have to be mapped 1:1. | | Within-partition ordering | `keyBy(sourcePartition)` ensures transactions from the same partition are processed in order within the same sink subtask. | #### Configurations[​](#configurations "Direct link to Configurations") ##### Multi-table Transaction Configurations[​](#multi-table-transaction-configurations "Direct link to Multi-table Transaction Configurations") ###### `sink.transaction.multi-table.enabled`[​](#sinktransactionmulti-tableenabled "Direct link to sinktransactionmulti-tableenabled") * Type: Boolean * Default: `false` * Description: Whether to enable the Multi-table Atomic Transaction mode. ###### `sink.transaction.multi-table.buffer-size`[​](#sinktransactionmulti-tablebuffer-size "Direct link to sinktransactionmulti-tablebuffer-size") * Type: Long * Default: `134217728` (128 MB) * Unit: Bytes * Description: Global buffer size in bytes for the Multi-table Transaction mode. When the total buffered data across all tables reaches this threshold, a flush is triggered. ###### `sink.transaction.multi-table.mini-switch-interval-ms`[​](#sinktransactionmulti-tablemini-switch-interval-ms "Direct link to sinktransactionmulti-tablemini-switch-interval-ms") * Type: Long * Default: `-1` (auto) * Unit: ms * Description: Minimum interval (ms) between per-partition chunk switches. Within the interval, source transactions are batched into a single stream-load, bounding HTTP request count. `-1` auto-derives `min(1000, max(500, sink.buffer-flush.interval-ms/4))`; a positive value overrides it. ###### `sink.transaction.multi-table.min-switch-bytes`[​](#sinktransactionmulti-tablemin-switch-bytes "Direct link to sinktransactionmulti-tablemin-switch-bytes") * Type: Long * Default: `1048576` (1 MB) * Unit: Bytes * Description: An interval-elapsed chunk switch only fires once a region's active chunk has accumulated at least this many bytes, so low-volume partitions batch more source transactions into one stream-load instead of emitting many tiny requests. Half-full headroom, buffer-size memory pressure, or a full `sink.buffer-flush.interval-ms` elapsing since the partition last switched all force a switch regardless — so even with a large threshold a continuous low-volume stream is still switched at roughly the flush interval and never withheld indefinitely. Set `<=0` to disable the size gate and restore the prior per-interval switching behavior. ###### `sink.transaction.multi-table.max-txn-bytes`[​](#sinktransactionmulti-tablemax-txn-bytes "Direct link to sinktransactionmulti-tablemax-txn-bytes") * Type: Long * Default: `0` (unlimited) * Unit: Bytes * Description: Hard limit in bytes on the in-progress source-transaction data one sink subtask may buffer while waiting for txnEnd. The writer never blocks on bytes that cannot be flushed before txnEnd (see Limitation 7), so a single source transaction may grow past `2 × buffer-size`; this option bounds that growth and fails the job with a clear error when exceeded. `0` means no limit: the TaskManager heap is the only bound. ##### Load-related Configurations[​](#load-related-configurations "Direct link to Load-related Configurations") ###### `sink.version`[​](#sinkversion "Direct link to sinkversion") * Recommended Value: `V2` * Description: Required. `V1` does not support the transaction Stream Load interface. ###### `sink.semantic`[​](#sinksemantic "Direct link to sinksemantic") * Recommended Value: `at-least-once` * Description: Multi-table mode currently supports `at-least-once` only. ###### `database-name`[​](#database-name "Direct link to database-name") * Recommended Value: `*` * Description: Wildcard to enable dynamic multi-table routing. ###### `table-name`[​](#table-name "Direct link to table-name") * Recommended Value: `*` * Description: Wildcard to enable dynamic multi-table routing. ###### `sink.buffer-flush.interval-ms`[​](#sinkbuffer-flushinterval-ms "Direct link to sinkbuffer-flushinterval-ms") * Recommended Value: `1000` * Description: Controls the commit cycle. You can set it to `1000` to achieve the freshness of approximately one second. ###### `sink.properties.format`[​](#sinkpropertiesformat "Direct link to sinkpropertiesformat") * Recommended Value: `json` * Description: The data format. ###### `sink.properties.strip_outer_array`[​](#sinkpropertiesstrip_outer_array "Direct link to sinkpropertiesstrip_outer_array") * Recommended Value: `true` * Description: Whether to strip the outermost array structure. #### Interfaces[​](#interfaces "Direct link to Interfaces") ##### `StarRocksRowData`[​](#starrocksrowdata "Direct link to starrocksrowdata") ```java public interface StarRocksRowData { String getUniqueKey(); // Region routing key (nullable; auto-derived from database.table) String getDatabase(); // Target database String getTable(); // Target table String getRow(); // Row data in JSON format /** * Indicates this is the last row of a source transaction batch. * Used by multi-table transaction mode to determine safe commit points: * the connector only commits when the most recent write had this flag set, * ensuring no partial source transaction is committed. */ default boolean isTransactionEnd() { return false; } /** * Returns the source partition ID for this row. * Used by multi-table transaction mode to track per-partition transaction * boundaries. Returns -1 when partition tracking is not applicable. */ default int getSourcePartition() { return -1; } } ``` ##### `DefaultStarRocksRowData`[​](#defaultstarrocksrowdata "Direct link to defaultstarrocksrowdata") ```java public class DefaultStarRocksRowData implements StarRocksRowData { // Basic fields private String uniqueKey; private String database; private String table; private String row; // Multi-table transaction fields private boolean transactionEnd; // Source transaction end marker private int sourcePartition = -1; // Source partition ID (for keyBy ordering) // Constructors public DefaultStarRocksRowData(); public DefaultStarRocksRowData(String database, String table); public DefaultStarRocksRowData(String uniqueKey, String database, String table, String row); // Setters public void setUniqueKey(String uniqueKey); public void setDatabase(String database); public void setTable(String table); public void setRow(String row); public void setTransactionEnd(boolean transactionEnd); public void setSourcePartition(int sourcePartition); // Getters (inherited from StarRocksRowData) public String getUniqueKey(); public String getDatabase(); public String getTable(); public String getRow(); public boolean isTransactionEnd(); public int getSourcePartition(); } ``` ##### User-Implemented Component[​](#user-implemented-component "Direct link to User-Implemented Component") Users need to implement a `KeyedProcessFunction` (referred to as `TransactionAssembler` in this document) that: 1. Keys by source partition and buffers data rows within a transaction 2. Emits all rows only when the source transaction is closed (for example, upon receiving `TXN_END`) 3. Sets `transactionEnd=true` on the last row 4. Sets `sourcePartition` on every row No custom `SinkFunction` is needed — the standard connector API (`SinkFunctionFactory.createSinkFunction()`) handles everything. #### Complete Example[​](#complete-example "Direct link to Complete Example") ##### StarRocks Table DDL[​](#starrocks-table-ddl "Direct link to StarRocks Table DDL") ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`orders` ( `order_id` BIGINT NOT NULL, `customer_id` BIGINT NOT NULL, `total_amount` DECIMAL(10,2) DEFAULT "0", `order_status` VARCHAR(32) DEFAULT "" ) ENGINE=OLAP PRIMARY KEY(`order_id`) DISTRIBUTED BY HASH(`order_id`) PROPERTIES("replication_num" = "1"); CREATE TABLE `test`.`order_items` ( `item_id` BIGINT NOT NULL, `order_id` BIGINT NOT NULL, `product_name` VARCHAR(128) DEFAULT "", `quantity` INT DEFAULT "0", `price` DECIMAL(10,2) DEFAULT "0" ) ENGINE=OLAP PRIMARY KEY(`item_id`) DISTRIBUTED BY HASH(`item_id`) PROPERTIES("replication_num" = "1"); ``` ##### Flink Job Code[​](#flink-job-code "Direct link to Flink Job Code") ```java import com.starrocks.connector.flink.table.data.DefaultStarRocksRowData; import com.starrocks.connector.flink.table.sink.SinkFunctionFactory; import com.starrocks.connector.flink.table.sink.StarRocksSinkOptions; import com.starrocks.data.load.stream.properties.StreamLoadTableProperties; import org.apache.flink.api.common.state.ListState; import org.apache.flink.api.common.state.ListStateDescriptor; import org.apache.flink.api.common.typeinfo.Types; import org.apache.flink.api.java.utils.MultipleParameterTool; import org.apache.flink.configuration.Configuration; import org.apache.flink.streaming.api.datastream.DataStream; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.streaming.api.functions.KeyedProcessFunction; import org.apache.flink.streaming.api.functions.sink.SinkFunction; import org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction; import org.apache.flink.util.Collector; import java.io.Serializable; import java.util.ArrayList; import java.util.List; public class WriteMultipleTablesWithTransaction { // ===================================================== // 1. Source Event Model (define per your business logic) // ===================================================== enum EventType { TXN_BEGIN, DATA, TXN_END } static class TxnEvent implements Serializable { private static final long serialVersionUID = 1L; int partition; String txnId; EventType type; String database; String table; String json; TxnEvent() {} TxnEvent(int partition, String txnId, EventType type, String database, String table, String json) { this.partition = partition; this.txnId = txnId; this.type = type; this.database = database; this.table = table; this.json = json; } static TxnEvent begin(int partition, String txnId) { return new TxnEvent(partition, txnId, EventType.TXN_BEGIN, null, null, null); } static TxnEvent data(int partition, String txnId, String db, String table, String json) { return new TxnEvent(partition, txnId, EventType.DATA, db, table, json); } static TxnEvent end(int partition, String txnId) { return new TxnEvent(partition, txnId, EventType.TXN_END, null, null, null); } } // ===================================================== // 2. TransactionAssembler — Core User Component // ===================================================== /** * Buffers DATA events per partition; on TXN_END emits the complete * transaction's rows as individual DefaultStarRocksRowData records. * * Only emits when a source transaction is fully closed (TXN_END received). * All rows are emitted synchronously within one processElement() call, * so they enter the downstream sink buffer without intervening checkpoint barriers. * * Multiple source transactions accumulate in the sink's buffer between * flush cycles — the connector handles grouping them into StarRocks transactions. */ static class TransactionAssembler extends KeyedProcessFunction { private transient ListState pendingRows; @Override public void open(Configuration parameters) throws Exception { ListStateDescriptor descriptor = new ListStateDescriptor<>( "pending-txn-rows", Types.POJO(TxnEvent.class)); pendingRows = getRuntimeContext().getListState(descriptor); } @Override public void processElement(TxnEvent event, Context ctx, Collector out) throws Exception { switch (event.type) { case TXN_BEGIN: pendingRows.clear(); break; case DATA: pendingRows.add(event); break; case TXN_END: List rows = new ArrayList<>(); for (TxnEvent row : pendingRows.get()) { rows.add(row); } int partition = ctx.getCurrentKey(); for (int i = 0; i < rows.size(); i++) { TxnEvent row = rows.get(i); DefaultStarRocksRowData rowData = new DefaultStarRocksRowData( null, row.database, row.table, row.json); rowData.setSourcePartition(partition); // Mark the last row as transaction end if (i == rows.size() - 1) { rowData.setTransactionEnd(true); } out.collect(rowData); } pendingRows.clear(); break; } } } // ===================================================== // 3. Main Program // ===================================================== public static void main(String[] args) throws Exception { MultipleParameterTool params = MultipleParameterTool.fromArgs(args); String jdbcUrl = params.get("jdbcUrl", "jdbc:mysql://127.0.0.1:9030"); String loadUrl = params.get("loadUrl", "127.0.0.1:8030"); String userName = params.get("userName", "root"); String password = params.get("password", ""); StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.enableCheckpointing(10_000); // Checkpoint is for recovery only; does not affect commit cycle // --- Source --- // Replace with an actual Kafka Source that deserializes into TxnEvent DataStream events = env.addSource(/* KafkaSource or MockTxnEventSource */); // --- Step 1: Assemble complete transactions per partition --- // TransactionAssembler emits rows only after TXN_END. // Multiple closed source txns accumulate in the sink buffer between flushes. DataStream rows = events .keyBy(e -> e.partition) .process(new TransactionAssembler()); // --- Step 2: Partition-affinity routing to sink --- // keyBy(sourcePartition) routes same-partition data to the same sink subtask. // The connector uses per-partition regions internally, so even when multiple // partitions land on the same sink subtask, transaction boundaries are tracked // independently per partition via PartitionCommitTracker. DataStream partitionedRows = rows .keyBy(DefaultStarRocksRowData::getSourcePartition); // --- Step 3: Configure the connector --- // sink.transaction.multi-table.enabled=true activates per-partition region // tracking inside StreamLoadManagerV2: each partition's regions are switched // independently when its txnEnd arrives, and commit triggers only when all // active partitions have been switched. StarRocksSinkOptions options = StarRocksSinkOptions.builder() .withProperty("jdbc-url", jdbcUrl) .withProperty("load-url", loadUrl) .withProperty("database-name", "*") // Wildcard for dynamic multi-table routing .withProperty("table-name", "*") // Wildcard for dynamic multi-table routing .withProperty("username", userName) .withProperty("password", password) .withProperty("sink.version", "V2") // Required: V2 .withProperty("sink.semantic", "at-least-once") .withProperty("sink.transaction.multi-table.enabled", "true") // Enable multi-table txn .withProperty("sink.buffer-flush.interval-ms", "1000") // ~1s data freshness .withProperty("sink.properties.format", "json") .withProperty("sink.properties.strip_outer_array", "true") .build(); // Optional: per-table stream load properties StreamLoadTableProperties orderItemsProps = StreamLoadTableProperties.builder() .database("test") .table("order_items") .addProperty("format", "json") .addProperty("strip_outer_array", "true") .addProperty("ignore_json_size", "true") .build(); options.addTableProperties(orderItemsProps); // --- Step 4: Create and attach the sink --- // Standard connector API — no custom SinkFunction needed. // addSink on the keyBy'd stream ensures partition affinity. SinkFunction sink = SinkFunctionFactory.createSinkFunction(options); partitionedRows.addSink(sink); env.execute("WriteMultipleTablesWithTransaction"); } } ``` ##### Data Flow Topology[​](#data-flow-topology "Direct link to Data Flow Topology") ```text Kafka (60 partitions) | v keyBy(partition) ———— Ensures same-partition events go to the same subtask | v TransactionAssembler (KeyedProcessFunction) | Buffers DATA events | On TXN_END: emits all rows (last row has transactionEnd=true) | Every row carries sourcePartition | v keyBy(sourcePartition) ———— Ensures same-partition rows go to the same sink subtask | v StarRocksDynamicSinkFunctionV2 (via SinkFunctionFactory.createSinkFunction) | | +——————————— StreamLoadManagerV2 (multi-table txn mode) —————————-——+ | | | | | Per-partition, per-table regions: | | | Region(P0, orders), Region(P0, order_items) | | | Region(P2, orders), Region(P2, order_items) | | | | | | Each region tracks: | | | - activeChunk / inactiveChunks | | | - lastSwitchTimeMs (for miniInterval batching) | | | - activeChunkCleanBoundary (true iff last task event is txnEnd)| | | | | | write(partition, db, table, row) [task thread] | | | -> routes to Region(partition, db, table) | | | -> write0 sets activeChunkCleanBoundary = false | | | -> In multi-table mode write0 does NOT switchChunk, so | | | activeChunk only freezes at a txnEnd boundary | | | | | | setCommitAllowed(partition, txnEnd=true) [task thread] | | | -> region.tryMiniIntervalSwitch(): | | | sets activeChunkCleanBoundary = true | | | if (now - lastSwitchTimeMs >= miniInterval | | | && activeChunk has data): switchChunkForCommit | | | else: data batches into activeChunk with subsequent | | | completed source transactions (N:1 mapping) | | | -> PartitionCommitTracker.onTxnEnd(partition) | | | | | | SharedTransactionCoordinator: | | | -> eagerly opens shared txn before any flush | | | -> all autonomous flushes use the shared label | | | -> recycles idle txn at 80% of server timeout | | | | | | Manager thread (every scanningFrequency): | | | -> tryForceCleanSwitch per region: | | | if cleanBoundary && has data && miniInterval elapsed | | | -> switchChunkForCommit (source-idle fallback) | | | -> tryStartTimerDrivenCommit: | | | if commitInterval elapsed && hasDataLoaded | | | -> set commitInFlight = true | | | -> autonomous flush: drain inactiveChunks via streamLoad | | | (never touches activeChunk in multi-table mode) | | | | | | Manager thread (commitInFlight=true): | | | -> triggerLoadIfNeeded per region (HTTP /api/transaction/load) | | | -> wait all loads complete | | | -> unified commit via SharedTransactionCoordinator | | | -> reset tracker; open next shared txn | | +———————————————————————————————————————————————————————————————————+ | v StarRocks (test.orders + test.order_items) ``` #### How It Works[​](#how-it-works "Direct link to How It Works") ##### Chunk Lifecycle and `miniInterval` Batching[​](#chunk-lifecycle-and-miniinterval-batching "Direct link to chunk-lifecycle-and-miniinterval-batching") Each `(partition, table)` region has one `activeChunk` (currently accepting writes) and a FIFO of `inactiveChunks` (frozen data, pending HTTP load). In multi-table transaction mode, the **only** way data moves from `activeChunk` into `inactiveChunks` is via `switchChunkForCommit`, which is called from exactly three sites: 1. **Task thread, on txnEnd** — via `region.tryMiniIntervalSwitch()` inside `setCommitAllowed(partition, true)`. This is the common path. 2. **Manager thread, source-idle fallback** — via `region.tryForceCleanSwitch()` on every scan cycle, for regions whose `activeChunk` has been sitting idle at a clean transaction boundary. 3. **Manager thread, savepoint/recycle** — force-switch all regions after verifying every region is at a clean boundary. To avoid one HTTP load per source transaction in high-throughput CDC, the task thread only performs a switch when at least `miniSwitchIntervalMs` has elapsed since the previous switch on the same region. `miniSwitchIntervalMs` is computed as `min(1000 ms, max(100 ms, commitInterval / 10))`, so a 1-second commit interval batches at 100 ms while a 30-second interval caps batching at 1 second. Within a miniInterval window, multiple completed source transactions accumulate into the same `activeChunk` (N:1 mapping) and are frozen together on the next switch. Each region carries two fields that drive these decisions: * `lastSwitchTimeMs` — epoch ms of the most recent switch. Initially 0, so the very first txnEnd after region creation always triggers a switch. * `activeChunkCleanBoundary` — `true` if the most recent task-thread event on this region was either an `onTxnEnd` or a `switchChunk`. `false` after any `write()`. The manager thread's `tryForceCleanSwitch` only runs when this flag is `true`, so it can never freeze partial source-transaction data. ##### Task Thread — Write and txnEnd[​](#task-thread--write-and-txnend "Direct link to Task Thread — Write and txnEnd") ```text invoke(record) [Flink task thread] | | if record is a data row: | write(partition, db, table, row) | region.write(row) → addRow(); cleanBoundary = false | | if record carries transactionEnd=true: | setCommitAllowed(partition, true) | for each region owned by this partition: | region.tryMiniIntervalSwitch(): | cleanBoundary = true // always (txnEnd observed) | if (now - lastSwitchTimeMs >= miniInterval | && activeChunk has data): | switchChunkForCommit() // freezes activeChunk | partitionTracker.onTxnEnd(partition) // safety bookkeeping only ``` The task thread is the sole serializer of `write()` and `setCommitAllowed()` events, which is why both the `cleanBoundary` mark and the conditional switch must run here (not deferred to the manager thread): `cleanBoundary` must reflect the most recent task-thread event at all times, or the manager's idle-fallback could race a write and freeze partial data. ##### 6.3 Manager Thread — Time-Driven Commit[​](#63-manager-thread--time-driven-commit "Direct link to 6.3 Manager Thread — Time-Driven Commit") The manager thread runs a scan loop at `scanningFrequency`. Each iteration: 1. **Ensure shared transaction**: open a new one (eager) or proactively recycle the current one if it is approaching the StarRocks server-side timeout (80% of `timeout` header, default 480 s). 2. **Source-idle fallback**: call `region.tryForceCleanSwitch()` on every region to freeze any `activeChunk` that is clean and has been idle for at least `miniInterval`. This handles the "source paused after a few txnEnds" case where the task thread stopped before issuing a fresh switch. 3. **Time-driven commit trigger**: call `tryStartTimerDrivenCommit()`, which sets `commitInFlight=true` if **both** conditions hold: * `now - lastCommitTimeMs >= commitInterval` (the configured `sink.buffer-flush.interval-ms`). * There is data to commit — either `txnCoordinator.hasDataLoaded()` is true or at least one region still has pending inactiveChunks. 4. **Autonomous flush**: drain any region whose `inactiveChunks` is non-empty via the `FlushAndCommitStrategy`. Multi-table mode's `flush()` only streams out already-frozen inactive chunks — it **never** touches `activeChunk`. This preserves the invariant that every chunk reaching StarRocks under the shared label comes from completed source transactions. When `commitInFlight=true` the main loop enters `processMultiTableCommit`, which waits for in-flight loads, triggers loads for any remaining inactive chunks, runs the unified commit via `SharedTransactionCoordinator`, updates `lastCommitTimeMs`, resets the tracker, and opens a new shared transaction for the next cycle. ##### Shared Transaction Coordination[​](#shared-transaction-coordination "Direct link to Shared Transaction Coordination") All tables within the same commit cycle share a single StarRocks transaction managed by `SharedTransactionCoordinator`: 1. **Eager transaction opening**: A shared transaction is opened eagerly before any autonomous flush, so all HTTP loads use the shared label. This eliminates the data-loss window where an independent-label flush could be orphaned when a shared transaction later overwrites the label. 2. **Unified commit**: After all regions' data is loaded, a single `commit` is executed for the shared label. Multi-table transactions skip the `prepare` step because StarRocks does not support `TXN_PREPARE` in multi-table mode. 3. **Idle transaction recycling**: If the shared transaction remains open longer than 80% of the StarRocks server-side timeout (default: 480s for a 600s timeout), it is proactively recycled (commit-or-rollback + reopen) to prevent server-side timeout errors. Recycle fails fast if any region has in-progress transaction data (clean-boundary violation) or any partition has written data without ever receiving a txnEnd. ##### 6.5 PartitionCommitTracker (Safety Bookkeeping)[​](#65-partitioncommittracker-safety-bookkeeping "Direct link to 6.5 PartitionCommitTracker (Safety Bookkeeping)") In the current design, commit timing is driven entirely by the commit interval and the per-region clean-boundary flags. `PartitionCommitTracker` is reduced to an informational/safety aid: * `onWrite(partition)` registers a partition as `ACTIVE` on first write. * `onTxnEnd(partition)` transitions the partition to `TXN_END_SEEN` (sticky — subsequent writes do **not** demote it back to `ACTIVE`). * `getPartitionsWithoutTxnEnd()` lists partitions that have written data but never received a txnEnd. Used by savepoint and recycle to fail fast on upstream contract violations (a source transaction that never closed). * `reset()` clears all partitions at the end of a commit cycle. The tracker no longer drives switch/commit decisions, does not track a `SWITCHED` state, and does not manage pending txnEnd signals. Partitions that stop producing data after a commit are simply cleared by `reset()`; if they resume later, the next `onWrite` re-registers them. ##### Autonomous Flush with Shared Labels[​](#autonomous-flush-with-shared-labels "Direct link to Autonomous Flush with Shared Labels") When a region's `inactiveChunks` becomes non-empty, the manager thread's autonomous-flush loop streams it to StarRocks via `/api/transaction/load` under the **current shared label**. Because the shared transaction is always opened before any data is loaded, there is no window where data could be loaded under an independent (orphaned) label. In multi-table mode, `flush()` never triggers a `switchChunk` — it only drains already-frozen inactive chunks — so the invariant "every chunk reaching StarRocks under the shared label comes from completed source transactions" holds unconditionally. ##### Safety Guarantees[​](#safety-guarantees "Direct link to Safety Guarantees") | Guarantee | Mechanism | | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | switchChunk does not split source transactions | `switchChunkForCommit` is only called at a clean transaction boundary (on txnEnd or when `activeChunkCleanBoundary` is `true`). | | Commit never includes partial source transactions | Every chunk reaching StarRocks originates from `switchChunkForCommit`. Autonomous flush never switches `activeChunk` in multi-table mode. | | Per-partition isolation | Each `(partition, table)` has its own region. One partition's switch never affects another's data. | | Within-partition ordering | `keyBy(sourcePartition)` routes same-partition rows to the same sink subtask. | | Task thread is non-blocking | `tryMiniIntervalSwitch` is O(regions-in-partition). HTTP work happens asynchronously on the manager thread. | | Source-idle data visibility | Manager thread's `tryForceCleanSwitch` freezes clean `activeChunk`s after `miniInterval` of idleness, so data remains visible within `commitInterval + miniInterval` even if the source pauses. | | Autonomous flushes are transaction-safe | Every load uses the shared label. Every frozen chunk is from completed source transactions. | | Idle transactions do not timeout | Shared transactions are recycled at 80% of server timeout. Recycle fails fast on in-progress data. | | Per-partition independent commit | A partition with a completed source transaction commits on the next commit interval, independent of other partitions' in-progress transactions. | ##### N:1 Transaction Mapping[​](#n1-transaction-mapping "Direct link to N:1 Transaction Mapping") Multiple source transactions can accumulate in a single StarRocks transaction via the miniInterval batching mechanism: ```text Source txn K1 (3 rows) -> write -> activeChunk -> txnEnd (1st switch) Source txn K2 (2 rows) -> write -> activeChunk -> txnEnd (inside miniInterval: no switch) Source txn K3 (4 rows) -> write -> activeChunk -> txnEnd (inside miniInterval: no switch) (miniInterval elapsed → next switch batches K2+K3) -> ... commitInterval elapsed -> commit(label=A) -> K1 + K2 + K3 atomically committed to StarRocks ``` Because the commit decision is time-driven (not tied to a specific txnEnd), the connector amortizes HTTP-load and begin/commit overhead across many small source transactions without any configuration changes. For a CDC source emitting 100 txnEnds per second with `commitInterval=1 s`, `miniInterval=100 ms`, the connector issues at most ~10 load calls per second instead of ~100. #### Limitations[​](#limitations "Direct link to Limitations") * **Requires `sink.version=V2`**: V1 does not support transaction stream load. * **At-least-once only**: Failed retries may produce duplicate writes. Multi-table mode guarantees all tables within the same batch succeed or fail together, but does not provide global exactly-once. For PRIMARY KEY tables, duplicate writes are idempotent (upsert). * **All tables must be in the same database**: StarRocks multi-table transactions are database-scoped; cross-database transactions are not supported. * **Transaction scope is per sink subtask + per partition**: Each sink subtask maintains its own StarRocks transaction independently. Atomicity is guaranteed **within a single source transaction** (all rows for one txnEnd on one partition, across all tables that partition writes to). Data visibility across **different** source partitions can interleave: once partition P0's source transaction has fully arrived and the commit interval has elapsed, P0's data is committed even if partition P1's source transaction is still in progress. Applications that require cross-partition atomicity at the StarRocks level must either use a single source partition or coordinate commits upstream. * **Data visibility latency**: Governed by `sink.buffer-flush.interval-ms` and the internal `miniInterval = min(1000, max(100, commitInterval/10))`. In a continuously flowing CDC stream, an individual row becomes visible in StarRocks within roughly `commitInterval + miniInterval` of its source commit. During a source pause, previously-committed data remains visible while any row between the last switch and the pause becomes visible after one more `miniInterval` (handled by the manager-thread clean-boundary fallback). * **Depends on StarRocks cluster transaction settings**: Monitor running txn limits, prepared timeout (default 600s), and label retention. Ensure `sink.buffer-flush.interval-ms` is significantly shorter than the StarRocks transaction timeout. * **activeChunk memory growth under large source transactions**: Because multi-table mode disables chunk-size-triggered internal switching (to preserve the clean-transaction-boundary invariant), `activeChunk` can grow until the next txnEnd arrives. `sink.transaction.multi-table.buffer-size` (soft) and `2 × buffer-size` (hard, via `blockIfCacheFull`) only bound the *flushable* part of the buffer — frozen chunks and all-clean partitions. Bytes that cannot leave the buffer before a txnEnd (the in-progress transaction itself, plus completed rows of sibling tables in the same partition that the lockstep rule holds back) are never waited on: once the buffer is at the hard cap and nothing is flushable, the writer logs a WARN (`Write-block cap ... reached with nothing flushable`) and keeps buffering until txnEnd, after which the partition is switched and drained. So a single source transaction may exceed `2 × buffer-size`; its size is bounded only by `sink.transaction.multi-table.max-txn-bytes` (default unlimited) and the TaskManager heap. Size the heap for the largest expected source transaction times the number of sink subtasks per TaskManager, set `max-txn-bytes` if you prefer a clear failure over heap pressure, and split unbounded transactions (bulk bootstraps) upstream. At switch time a frozen chunk larger than the load body limit — the smaller of `sink.chunk-limit` (3 GB by default) and `sink.transaction.multi-table.buffer-size` — is re-cut into several load requests under the same shared label, so no single `/api/transaction/load` body exceeds that limit, except for a single row that is itself larger than it. * **Cross-database writes are rejected**: Multi-table transactions validate that all regions belong to the same database. Writing to tables in different databases within the same commit cycle will throw an error. * **Incompatible with merge commit**: `sink.properties.enable_merge_commit=true` cannot be combined with `sink.transaction.multi-table.enabled=true`. Merge commit routes writes through `MergeCommitManager`, which lacks the partition-aware `write(int, ...)` / `setCommitAllowed(int, ...)` hooks that multi-table mode relies on for transaction boundaries. The connector fails fast at validation time if both are enabled. #### Monitoring and Troubleshooting[​](#monitoring-and-troubleshooting "Direct link to Monitoring and Troubleshooting") Recommended metrics: * **On the Flink Side** * Checkpoint success rate * Checkpoint duration * Sink flush/commit latency * **On the StarRocks Side** * Running/prepared txn count * Txn timeout occurrences * Label conflicts ##### Common issues[​](#common-issues "Direct link to Common issues") ###### `transaction not existed`[​](#transaction-not-existed "Direct link to transaction-not-existed") * Cause: StarRocks transaction timeout * Solution: The connector automatically recycles idle transactions at 80% of server timeout. If this still occurs, check if prepared timeout is too short or flush interval is too large. ###### `too many running txns`[​](#too-many-running-txns "Direct link to too-many-running-txns") * Cause: There are excessive concurrent transactions. * Solution: Reduce sink parallelism or increase the value of StarRocks FE configuration `max_running_txn_num_per_db`. ###### `Transaction start failed`[​](#transaction-start-failed "Direct link to transaction-start-failed") * Cause: The `beginTransaction` HTTP call failed * Solution: Verify load-url connectivity and the StarRocks version (v4.0 or later is required). ###### High data visibility latency[​](#high-data-visibility-latency "Direct link to High data visibility latency") * Cause: Commit conditions are not met. * Solution: Verify upstream data has correct `transactionEnd=true` markers; expect up to `commitInterval + miniInterval` latency per row. If latency exceeds this budget, check the manager thread is not stuck in a recycle or in-flight load (see `StarRocks-Sink-Manager` logs). ###### `Write-block cap ... reached with nothing flushable` (WARN)[​](#write-block-cap--reached-with-nothing-flushable-warn "Direct link to write-block-cap--reached-with-nothing-flushable-warn") * Cause: A source transaction, plus completed rows of sibling tables held back until its txnEnd, exceeds `2 × buffer-size`; the writer keeps buffering until txnEnd instead of blocking. * Solution: Expected under a large source transaction; the data is committed once its txnEnd arrives. Watch TaskManager heap. Set `sink.transaction.multi-table.max-txn-bytes` if you prefer a hard failure, or split the transaction upstream. ###### `would exceed sink.transaction.multi-table.max-txn-bytes`[​](#would-exceed-sinktransactionmulti-tablemax-txn-bytes "Direct link to would-exceed-sinktransactionmulti-tablemax-txn-bytes") * Cause: A source transaction is larger than the configured hard limit. * Solution: Raise `max-txn-bytes` (or set it to `0` for unlimited) or split the source transaction upstream. ###### Cross-database write error[​](#cross-database-write-error "Direct link to Cross-database write error") * Cause: Tables in different databases are in same commit cycle. * Solution: Ensure all tables written in the same job belong to the same StarRocks database. #### Best Practices[​](#best-practices "Direct link to Best Practices") 1. **TransactionAssembler contract**: * Emit rows only after the source transaction is fully closed. * The last row must have `setTransactionEnd(true)`. * Every row must have `setSourcePartition(partition)`. * All rows must be emitted synchronously within a single `processElement()` call. 2. **keyBy before sink is mandatory**: `rows.keyBy(DefaultStarRocksRowData::getSourcePartition).addSink(sink)` — omitting this breaks within-partition transaction ordering. 3. **Checkpoint is decoupled from commit**: Checkpoint interval can be set to a large value (for example, 60 seconds) for fault recovery. Data visibility is governed by `sink.buffer-flush.interval-ms` (for example, 1000ms). 4. **Keep routing strategies stable**: Avoid single transactions writing to an excessive number of distinct tables, which increases transaction duration and failure probability. 5. **Fault injection testing before production**: Kill TaskManagers / introduce network jitter and verify data correctness after checkpoint recovery. --- ### Continuously load data from Apache Flink® StarRocks provides a self-developed connector named StarRocks Connector for Apache Flink® (Flink connector for short) to help you load data into a StarRocks table by using Flink. The basic principle is to accumulate the data and then load it all at a time into StarRocks through [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). The Flink connector supports DataStream API, Table API & SQL, and Python API. It has a higher and more stable performance than [flink-connector-jdbc](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/jdbc/) provided by Apache Flink®. > **NOTICE** > > Loading data into StarRocks tables with Flink connector needs SELECT and INSERT privileges on the target StarRocks table. If you do not have these privileges, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant these privileges to the user that you use to connect to your StarRocks cluster. #### Version requirements[​](#version-requirements "Direct link to Version requirements") | Connector | Flink | StarRocks | Java | Scala | | --------- | ----------------------------- | ------------- | ---- | --------- | | 1.2.16 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.15 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.14 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.12 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.11 | 1.15,1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | #### Obtain Flink connector[​](#obtain-flink-connector "Direct link to Obtain Flink connector") You can obtain the Flink connector JAR file in the following ways: * Directly download the compiled Flink connector JAR file. * Add the Flink connector as a dependency in your Maven project and then download the JAR file. * Compile the source code of the Flink connector into a JAR file by yourself. The naming format of the Flink connector JAR file is as follows: * Since Flink 1.15, it's `flink-connector-starrocks-${connector_version}_flink-${flink_version}.jar`. For example, if you install Flink 1.15 and you want to use Flink connector 1.2.7, you can use `flink-connector-starrocks-1.2.7_flink-1.15.jar`. * Prior to Flink 1.15, it's `flink-connector-starrocks-${connector_version}_flink-${flink_version}_${scala_version}.jar`. For example, if you install Flink 1.14 and Scala 2.12 in your environment, and you want to use Flink connector 1.2.7, you can use `flink-connector-starrocks-1.2.7_flink-1.14_2.12.jar`. > **NOTICE** > > In general, the latest version of the Flink connector only maintains compatibility with the three most recent versions of Flink. ##### Download the compiled Jar file[​](#download-the-compiled-jar-file "Direct link to Download the compiled Jar file") Directly download the corresponding version of the Flink connector Jar file from the [Maven Central Repository](https://repo1.maven.org/maven2/com/starrocks). ##### Maven Dependency[​](#maven-dependency "Direct link to Maven Dependency") In your Maven project's `pom.xml` file, add the Flink connector as a dependency according to the following format. Replace `flink_version`, `scala_version`, and `connector_version` with the respective versions. * In Flink 1.15 and later ```xml com.starrocks flink-connector-starrocks ${connector_version}_flink-${flink_version} ``` * In versions earlier than Flink 1.15 ```xml com.starrocks flink-connector-starrocks ${connector_version}_flink-${flink_version}_${scala_version} ``` ##### Compile by yourself[​](#compile-by-yourself "Direct link to Compile by yourself") 1. Download the [Flink connector source code](https://github.com/StarRocks/starrocks-connector-for-apache-flink). 2. Execute the following command to compile the source code of Flink connector into a JAR file. Note that `flink_version` is replaced with the corresponding Flink version. ```bash sh build.sh ``` For example, if the Flink version in your environment is 1.16, you need to execute the following command: ```bash sh build.sh 1.16 ``` 3. Go to the `target/` directory to find the Flink connector JAR file, such as `flink-connector-starrocks-1.2.7_flink-1.16-SNAPSHOT.jar`, generated upon compilation. > **NOTE** > > The name of Flink connector which is not formally released contains the `SNAPSHOT` suffix. #### Connect to StarRocks[​](#connect-to-starrocks "Direct link to Connect to StarRocks") Set the StarRocks connection options to addresses of the FE: * `jdbc-url`: `jdbc:mysql://:`. The query port defaults to `9030`. * `load-url`: `:`. The HTTP port defaults to `8030`. For ``, use an address that stays the same when an FE restarts or is replaced, such as a DNS name or a load balancer in front of the FEs, not the IP address of an FE. If the address stops leading to an FE, the Flink job can no longer load data. The Flink TaskManagers must be able to reach these FE ports, and also the HTTP port (default `8040`) of every BE or CN, because the FE redirects each load request to a BE or CN. #### Options[​](#options "Direct link to Options") ##### General Options[​](#general-options "Direct link to General Options") ###### connector[​](#connector "Direct link to connector") * **Required**: Yes * **Default value**: NONE * **Description**: The connector that you want to use. The value must be "starrocks". ###### jdbc-url[​](#jdbc-url "Direct link to jdbc-url") * **Required**: Yes * **Default value**: NONE * **Description**: The address that is used to connect to the MySQL server of the FE. You can specify multiple addresses, which must be separated by a comma (,). Format: `jdbc:mysql://:,:,:`. ###### load-url[​](#load-url "Direct link to load-url") * **Required**: Yes * **Default value**: NONE * **Description**: The address that is used to connect to the HTTP server of the FE. You can specify multiple addresses, which must be separated by a semicolon (;). Format: `:;:`. ###### database-name[​](#database-name "Direct link to database-name") * **Required**: Yes * **Default value**: NONE * **Description**: The name of the StarRocks database into which you want to load data. ###### table-name[​](#table-name "Direct link to table-name") * **Required**: Yes * **Default value**: NONE * **Description**: The name of the table that you want to use to load data into StarRocks. ###### username[​](#username "Direct link to username") * **Required**: Yes * **Default value**: NONE * **Description**: The username of the account that you want to use to load data into StarRocks. The account needs [SELECT and INSERT privileges](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) on the target StarRocks table. ###### password[​](#password "Direct link to password") * **Required**: Yes * **Default value**: NONE * **Description**: The password of the preceding account. ###### sink.version[​](#sinkversion "Direct link to sink.version") * **Required**: No * **Default value**: AUTO * **Description**: The interface used to load data. This parameter is supported from Flink connector version 1.2.4 onwards. Valid Values: * `V1`: Use [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) interface to load data. Connectors before 1.2.4 only support this mode. * `V2`: Use [Stream Load transaction](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) interface to load data. It requires StarRocks to be at least version 2.4. Recommends `V2` because it optimizes the memory usage and provides a more stable exactly-once implementation. * `AUTO`: If the version of StarRocks supports transaction Stream Load, will choose `V2` automatically, otherwise choose `V1` ###### sink.label-prefix[​](#sinklabel-prefix "Direct link to sink.label-prefix") * **Required**: No * **Default value**: NONE * **Description**: The label prefix used by Stream Load. Recommend to configure it if you are using exactly-once with connector 1.2.8 and later. See [exactly-once usage notes](#exactly-once). ###### sink.semantic[​](#sinksemantic "Direct link to sink.semantic") * **Required**: No * **Default value**: at-least-once * **Description**: The semantic guaranteed by sink. Valid values: **at-least-once** and **exactly-once**. ###### sink.buffer-flush.max-bytes[​](#sinkbuffer-flushmax-bytes "Direct link to sink.buffer-flush.max-bytes") * **Required**: No * **Default value**: 94371840(90M) * **Description**: The maximum size of data that can be accumulated in memory before being sent to StarRocks at a time. The maximum value ranges from 64 MB to 10 GB. Setting this parameter to a larger value can improve loading performance but may increase loading latency. This parameter only takes effect when `sink.semantic` is set to `at-least-once`. If `sink.semantic` is set to `exactly-once`, the data in memory is flushed when a Flink checkpoint is triggered. In this circumstance, this parameter does not take effect. ###### sink.buffer-flush.max-rows[​](#sinkbuffer-flushmax-rows "Direct link to sink.buffer-flush.max-rows") * **Required**: No * **Default value**: 500000 * **Description**: The maximum number of rows that can be accumulated in memory before being sent to StarRocks at a time. This parameter is available only when `sink.version` is `V1` and `sink.semantic` is `at-least-once`. Valid values: 64000 to 5000000. ###### sink.buffer-flush.interval-ms[​](#sinkbuffer-flushinterval-ms "Direct link to sink.buffer-flush.interval-ms") * **Required**: No * **Default value**: 300000 * **Description**: The interval at which data is flushed. This parameter is available only when `sink.semantic` is `at-least-once`. Unit: ms. Valid value range: * For versions earlier than v1.2.14: \[1000, 3600000] * For v1.2.14 and later: (0, 3600000]. ###### sink.max-retries[​](#sinkmax-retries "Direct link to sink.max-retries") * **Required**: No * **Default value**: 3 * **Description**: The number of times that the system retries to perform the Stream Load job. This parameter is available only when you set `sink.version` to `V1`. Valid values: 0 to 10. ###### sink.connect.timeout-ms[​](#sinkconnecttimeout-ms "Direct link to sink.connect.timeout-ms") * **Required**: No * **Default value**: 30000 * **Description**: The timeout for establishing HTTP connection. Valid values: 100 to 60000. Unit: ms. Before Flink connector v1.2.9, the default value is `1000`. ###### sink.socket.timeout-ms[​](#sinksockettimeout-ms "Direct link to sink.socket.timeout-ms") * **Required**: No * **Default value**: -1 * **Description**: Supported since 1.2.10. The time duration for which the HTTP client waits for data. Unit: ms. The default value `-1` means there is no timeout. ###### sink.sanitize-error-log[​](#sinksanitize-error-log "Direct link to sink.sanitize-error-log") * **Required**: No * **Default value**: false * **Description**: Supported since 1.2.12. Whether to sanitize sensitive data in the error log for production security. When this item is set to `true`, sensitive row data and column values in Stream Load error logs are redacted in both the connector and SDK logs. The value defaults to `false` for backward compatibility. ###### sink.wait-for-continue.timeout-ms[​](#sinkwait-for-continuetimeout-ms "Direct link to sink.wait-for-continue.timeout-ms") * **Required**: No * **Default value**: 10000 * **Description**: Supported since 1.2.7. The timeout for waiting response of HTTP 100-continue from the FE. Valid values: `3000` to `60000`. Unit: ms ###### sink.ignore.update-before[​](#sinkignoreupdate-before "Direct link to sink.ignore.update-before") * **Required**: No * **Default value**: true * **Description**: Supported since version 1.2.8. Whether to ignore `UPDATE_BEFORE` records from Flink when loading data to Primary Key tables. If this parameter is set to false, the record is treated as a delete operation to StarRocks table. ###### sink.json.columns-from-flink-schema[​](#sinkjsoncolumns-from-flink-schema "Direct link to sink.json.columns-from-flink-schema") * **Required**: No * **Default value**: false * **Description**: Supported since 1.2.16. Whether the Stream Load `columns` header is derived from the Flink table schema when the format is `json`. By default no header is sent for `json`, so the server declares every table column and a column the payload does not carry is stored as NULL even when it has a DEFAULT. When `true`, the Flink table's columns are sent as the header, so a column the Flink schema does not declare is left off and the server applies its DEFAULT. Requires a Flink schema and `sink.properties.format=json`, and cannot be combined with `sink.properties.columns` or `sink.properties.jsonpaths`. ###### sink.parallelism[​](#sinkparallelism "Direct link to sink.parallelism") * **Required**: No * **Default value**: NONE * **Description**: The parallelism of loading. Only available for Flink SQL. If this parameter is not specified, Flink planner decides the parallelism. **In the scenario of multi-parallelism, users need to guarantee data is written in the correct order.** ###### sink.properties.\*[​](#sinkproperties "Direct link to sink.properties.*") * **Required**: No * **Default value**: NONE * **Description**: The parameters that are used to control Stream Load behavior. For example, the parameter `sink.properties.format` specifies the format used for Stream Load, such as CSV or JSON. For a list of supported parameters and their descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ###### sink.properties.format[​](#sinkpropertiesformat "Direct link to sink.properties.format") * **Required**: No * **Default value**: csv * **Description**: The format used for Stream Load. The Flink connector will transform each batch of data to the format before sending them to StarRocks. Valid values: `csv` and `json`. ###### sink.properties.column\_separator[​](#sinkpropertiescolumn_separator "Direct link to sink.properties.column_separator") * **Required**: No * **Default value**: \t * **Description**: The column separator for CSV-formatted data. ###### sink.properties.row\_delimiter[​](#sinkpropertiesrow_delimiter "Direct link to sink.properties.row_delimiter") * **Required**: No * **Default value**: \n * **Description**: The row delimiter for CSV-formatted data. ###### sink.properties.max\_filter\_ratio[​](#sinkpropertiesmax_filter_ratio "Direct link to sink.properties.max_filter_ratio") * **Required**: No * **Default value**: 0 * **Description**: The maximum error tolerance of the Stream Load. It's the maximum percentage of data records that can be filtered out due to inadequate data quality. Valid values: `0` to `1`. Default value: `0`. See [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) for details. ###### sink.properties.partial\_update[​](#sinkpropertiespartial_update "Direct link to sink.properties.partial_update") * **Required**: NO * **Default value**: `FALSE` * **Description**: Whether to use partial updates. Valid values: `TRUE` and `FALSE`. Default value: `FALSE`, indicating to disable this feature. ###### sink.properties.partial\_update\_mode[​](#sinkpropertiespartial_update_mode "Direct link to sink.properties.partial_update_mode") * **Required**: NO * **Default value**: `row` * **Description**: Specifies the mode for partial updates. Valid values: `row` and `column`. * The value `row` (default) means partial updates in row mode, which is more suitable for real-time updates with many columns and small batches. * The value `column` means partial updates in column mode, which is more suitable for batch updates with few columns and many rows. In such scenarios, enabling the column mode offers faster update speeds. For example, in a table with 100 columns, if only 10 columns (10% of the total) are updated for all rows, the update speed of the column mode is 10 times faster. ###### sink.properties.strict\_mode[​](#sinkpropertiesstrict_mode "Direct link to sink.properties.strict_mode") * **Required**: No * **Default value**: false * **Description**: Specifies whether to enable the strict mode for Stream Load. It affects the loading behavior when there are unqualified rows, such as inconsistent column values. Valid values: `true` and `false`. Default value: `false`. See [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) for details. ###### sink.properties.compression[​](#sinkpropertiescompression "Direct link to sink.properties.compression") * **Required**: No * **Default value**: NONE * **Description**: The compression algorithm used for Stream Load. Valid values: `lz4_frame`. Compression for the JSON format requires Flink connector 1.2.10+ and StarRocks v3.2.7+. Compression for the CSV format only requires Flink connector 1.2.11+. ###### sink.properties.prepared\_timeout[​](#sinkpropertiesprepared_timeout "Direct link to sink.properties.prepared_timeout") * **Required**: No * **Default value**: NONE * **Description**: Supported since 1.2.12 and only effective when `sink.version` is set to `V2`. Requires StarRocks 3.5.4 or later. Sets the timeout in seconds for the Transaction Stream Load phase from `PREPARED` to `COMMITTED`. Typically, only needed for exactly-once; at-least-once usually does not require setting this (the connector defaults to 300s). If not set in exactly-once, StarRocks FE configuration `prepared_transaction_default_timeout_second` (default 86400s) applies. See [StarRocks Transaction timeout management](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md#transaction-timeout-management). ###### sink.publish-timeout.ms[​](#sinkpublish-timeoutms "Direct link to sink.publish-timeout.ms") * **Required**: No * **Default value**: -1 * **Description**: Supported since 1.2.14 and only effective when `sink.version` is set to `V2`. Timeout in milliseconds for the Publish phase. If a transaction stays in COMMITTED status longer than this timeout, the system will consider it as successful. The default value `-1` means using StarRocks server-side default behavior. When Merge Commit is enabled, the default timeout is 10000 ms. ##### Merge Commit options[​](#merge-commit-options "Direct link to Merge Commit options") Supported from v1.2.14 onwards. Merge Commit allows the system to merge data from multiple subtasks into a single Stream Load transaction for better performance. You can enable this feature by setting `sink.properties.enable_merge_commit` to `true`. For more details about the merge commit feature in StarRocks, see [Merge Commit parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#merge-commit-parameters). The following Stream Load properties are used to control the Merge Commit behavior: ###### sink.properties.enable\_merge\_commit[​](#sinkpropertiesenable_merge_commit "Direct link to sink.properties.enable_merge_commit") * **Required**: No * **Default value**: false * **Description**: Whether to enable Merge Commit. ###### sink.properties.merge\_commit\_interval\_ms[​](#sinkpropertiesmerge_commit_interval_ms "Direct link to sink.properties.merge_commit_interval_ms") * **Required**: Yes (when Merge Commit is enabled) * **Default value**: NONE * **Description**: The Merge Commit time window in milliseconds. The system merges loading requests received within this window into a single transaction. A larger value improves merging efficiency but increases latency. This property must be set when `enable_merge_commit` is set to `true`. ###### sink.properties.merge\_commit\_parallel[​](#sinkpropertiesmerge_commit_parallel "Direct link to sink.properties.merge_commit_parallel") * **Required**: No * **Default value**: 3 * **Description**: The degree of parallelism for the loading plan created for each Merge Commit-enabled transaction. It is different from `sink.parallelism` which controls the parallelism of the Flink sink operator. ###### sink.properties.merge\_commit\_async[​](#sinkpropertiesmerge_commit_async "Direct link to sink.properties.merge_commit_async") * **Required**: No * **Default value**: true * **Description**: The server's return mode for Merge Commit. The default value is `true` (asynchronous), overriding the system default behavior (synchronous) for better throughput. In the asynchronous mode, the server returns immediately after receiving the data. The connector leverages Flink's checkpoint mechanism to ensure no data loss under the asynchronous mode, providing at-least-once guarantee. In most cases, you do not need to change this value. ###### sink.merge-commit.max-concurrent-requests[​](#sinkmerge-commitmax-concurrent-requests "Direct link to sink.merge-commit.max-concurrent-requests") * **Required**: No * **Default value**: Integer.MAX\_VALUE * **Description**: The maximum number of concurrent Stream Load requests. Set this property to `0` to ensure in-order (serial) loading, which is useful for Primary Key tables. A negative value is treated as `Integer.MAX_VALUE` (unlimited concurrency). ###### sink.merge-commit.chunk.size[​](#sinkmerge-commitchunksize "Direct link to sink.merge-commit.chunk.size") * **Required**: No * **Default value**: 20971520 * **Description**: The maximum size of data (in bytes) accumulated in a chunk before it is flushed and sent to StarRocks via a Stream Load request. A larger value improves throughput but increases memory usage and latency; a smaller value reduces memory usage and latency but may lower throughput. When `max-concurrent-requests` is set to `0` (in-order mode), the default value of this property is changed to 500 MB because only one request runs at a time so a larger batch maximizes throughput. #### Data type mapping between Flink and StarRocks[​](#data-type-mapping-between-flink-and-starrocks "Direct link to Data type mapping between Flink and StarRocks") | Flink data type | StarRocks data type | | ------------------------------------- | ------------------- | | BOOLEAN | BOOLEAN | | TINYINT | TINYINT | | SMALLINT | SMALLINT | | INTEGER | INTEGER | | BIGINT | BIGINT | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | DECIMAL | DECIMAL | | BINARY | INT | | CHAR | STRING | | VARCHAR | STRING | | STRING | STRING | | DATE | DATE | | TIMESTAMP\_WITHOUT\_TIME\_ZONE(N) | DATETIME | | TIMESTAMP\_WITH\_LOCAL\_TIME\_ZONE(N) | DATETIME | | ARRAY`<`T`>` | ARRAY`<`T`>` | | MAP`<`KT,VT`>` | JSON STRING | | ROW`<`arg T...`>` | JSON STRING | #### Usage notes[​](#usage-notes "Direct link to Usage notes") ##### Exactly Once[​](#exactly-once "Direct link to Exactly Once") * If you want sink to guarantee exactly-once semantics, we recommend you to upgrade StarRocks to 2.5 or later, and Flink connector to 1.2.4 or later * Since Flink connector 1.2.4, the exactly-once is redesigned based on [Stream Load transaction interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) provided by StarRocks since 2.4. Compared to the previous implementation based on non-transactional Stream Load non-transactional interface, the new implementation reduces memory usage and checkpoint overhead, thereby enhancing real-time performance and stability of loading. * If the version of StarRocks is earlier than 2.4 or the version of Flink connector is earlier than 1.2.4, the sink will automatically choose the implementation based on Stream Load non-transactional interface. * Configurations to guarantee exactly-once * The value of `sink.semantic` needs to be `exactly-once`. * If the version of Flink connector is 1.2.8 and later, it is recommended to specify the value of `sink.label-prefix`. Note that the label prefix must be unique among all types of loading in StarRocks, such as Flink jobs, Routine Load, and Broker Load. * If the label prefix is specified, the Flink connector will use the label prefix to clean up lingering transactions that may be generated in some Flink failure scenarios, such as the Flink job fails when a checkpoint is still in progress. These lingering transactions are generally in `PREPARED` status if you use `SHOW PROC '/transactions//running';` to view them in StarRocks. When the Flink job restores from checkpoint, the Flink connector will find these lingering transactions according to the label prefix and some information in checkpoint, and abort them. The Flink connector can not abort them when the Flink job exits because of the two-phase-commit mechanism to implement the exactly-once. When the Flink job exits, the Flink connector has not received the notification from Flink checkpoint coordinator whether the transactions should be included in a successful checkpoint, and it may lead to data loss if these transactions are aborted anyway. You can have an overview about how to achieve end-to-end exactly-once in Flink in this [blogpost](https://flink.apache.org/2018/02/28/an-overview-of-end-to-end-exactly-once-processing-in-apache-flink-with-apache-kafka-too/). * If the label prefix is not specified, lingering transactions will be cleaned up by StarRocks only after they time out. However the number of running transactions can reach the limitation of StarRocks `max_running_txn_num_per_db` if Flink jobs fail frequently before transactions time out. You can set a smaller timeout for `PREPARED` transactions to make them expired faster when the label prefix is not specified. See the following about how to set the prepared timeout. * If you are certain that the Flink job will eventually recover from checkpoint or savepoint after a long downtime because of stop or continuous failover, please adjust the following StarRocks configurations accordingly, to avoid data loss. * Adjust `PREPARED` transaction timeout. See the following about how to set the timeout. The timeout needs to be larger than the downtime of the Flink job. Otherwise, the lingering transactions that are included in a successful checkpoint may be aborted because of timeout before you restart the Flink job, which leads to data loss. Note that when you set a larger value to this configuration, it is better to specify the value of `sink.label-prefix` so that the lingering transactions can be cleaned according to the label prefix and some information in checkpoint, instead of due to timeout (which may cause data loss). * `label_keep_max_second` and `label_keep_max_num`: StarRocks FE configurations, default values are `259200` and `1000` respectively. For details, see [FE configurations](https://docs.starrocks.io/docs/loading/loading_introduction/loading_considerations.md#fe-configurations). The value of `label_keep_max_second` needs to be larger than the downtime of the Flink job. Otherwise, the Flink connector can not check the state of transactions in StarRocks by using the transaction labels saved in the Flink's savepoint or checkpoint and figure out whether these transactions are committed or not, which may eventually lead to data loss. * How to set the timeout for PREPARED transactions * For Connector 1.2.12+ and StarRocks 3.5.4+, you can set the timeout by configuring the connector parameter `sink.properties.prepared_timeout`. By default, the value is not set, and it falls back to the StarRocks FE's global configuration `prepared_transaction_default_timeout_second` (default value is `86400`). * For other versions of Connector or StarRocks, you can set the timeout by configuring the StarRocks FE's global configuration `prepared_transaction_default_timeout_second` (default value is `86400`). ##### Flush Policy[​](#flush-policy "Direct link to Flush Policy") The Flink connector will buffer the data in memory, and flush them in batch to StarRocks via Stream Load. How the flush is triggered is different between at-least-once and exactly-once. For at-least-once, the flush will be triggered when any of the following conditions are met: * the bytes of buffered rows reaches the limit `sink.buffer-flush.max-bytes` * the number of buffered rows reaches the limit `sink.buffer-flush.max-rows`. (Only valid for sink version V1) * the elapsed time since the last flush reaches the limit `sink.buffer-flush.interval-ms` * a checkpoint is triggered For exactly-once, the flush only happens when a checkpoint is triggered. ##### Merge Commit[​](#merge-commit "Direct link to Merge Commit") Merge Commit helps scale throughput without proportionally increasing StarRocks transaction overhead. Without Merge Commit, each Flink sink subtask maintains its own Stream Load transaction, so increasing `sink.parallelism` leads to more concurrent transactions and higher I/O and Compaction costs on StarRocks. Conversely, keeping parallelism low limits the pipeline's overall capacity. With Merge Commit is enabled, data from multiple sink subtasks is merged into a single transaction within each Merge window. This allows you to increase `sink.parallelism` for higher throughput without increasing the number of transactions. For configuration examples, see [Load data with merge commit](#load-data-with-merge-commit). Below are some important notes when using Merge Commit: * **Single parallelism has no benefit** If the Flink sink parallelism is 1, enabling Merge Commit provides no benefit since there is only one subtask sending data. It may even introduce additional latency due to the Merge Commit time window on the server side. * **Only at-least-once semantic** Merge Commit only guarantees at-least-once semantic. It does not support exactly-once semantic. Do not set `sink.semantic` to `exactly-once` when Merge Commit is enabled. * **Ordering for Primary Key tables** By default, `sink.merge-commit.max-concurrent-requests` is `Integer.MAX_VALUE`, which means a single sink subtask may send multiple Stream Load requests concurrently. This can cause out-of-order loading, which may be problematic for Primary Key tables. To ensure in-order loading, set `sink.merge-commit.max-concurrent-requests` to `0`, which, however, will reduce throughput. Alternatively, you can use Conditional Update to prevent newer data from being overwritten by older data. For configuration examples, see [In-order loading for Primary Key tables](#in-order-loading-for-primary-key-tables). * **End-to-end loading latency** The total loading latency consists of two parts: * **Connector batching latency**: Controlled by `sink.buffer-flush.interval-ms` and `sink.merge-commit.chunk.size`. Data is flushed from the connector when either the chunk size limit is reached or the flush interval elapses, whichever comes first. The maximum connector-side latency is `sink.buffer-flush.interval-ms`. A smaller `sink.buffer-flush.interval-ms` reduces connector-side latency but sends data in smaller batches. * **StarRocks merge window**: Controlled by `sink.properties.merge_commit_interval_ms`. The system waits for this duration to merge requests from multiple subtasks into a single transaction. A larger value improves merging efficiency (more requests will be merged into one transaction) but increases server-side latency. * As a general guideline, set `sink.buffer-flush.interval-ms` to be smaller than or equal to `sink.properties.merge_commit_interval_ms`, so that each subtask can flush at least once within each Merge window. For example, if `merge_commit_interval_ms` is `10000` (10s), you could set `sink.buffer-flush.interval-ms` to `5000` (5 seconds) or less. * **Tuning `sink.parallelism` and `sink.properties.merge_commit_parallel`** These two parameters control parallelism at different layers and should be tuned independently: * `sink.parallelism` controls the number of Flink sink subtasks. Each subtask buffers and sends data to StarRocks. Increase this value when Flink sink operators are CPU- or memory-bound — you can monitor Flink's per-operator CPU and memory usage to determine whether more subtasks are needed. * `sink.properties.merge_commit_parallel` controls the degree of parallelism for the loading plan that StarRocks creates for each Merge Commit transaction. Increase this value when StarRocks becomes the bottleneck. You can monitor the StarRocks metrics [merge\_commit\_pending\_total](https://docs.starrocks.io/docs/administration/management/monitoring/metrics.md#merge_commit_pending_total) (number of pending Merge Commit tasks) and [merge\_commit\_pending\_bytes](https://docs.starrocks.io/docs/administration/management/monitoring/metrics.md#merge_commit_pending_bytes) (bytes held by pending tasks) to determine whether more parallelism is needed on the StarRocks side — sustained high values indicate that the loading plan cannot keep up with incoming data. * **Relationship between `sink.merge-commit.chunk.size` and `sink.buffer-flush.max-bytes`**: * `sink.merge-commit.chunk.size` controls the maximum data size per individual Stream Load request (per chunk). When data in a chunk reaches this size, it is flushed immediately. * `sink.buffer-flush.max-bytes` controls the total memory limit for all cached data across all tables. When the total cached data exceeds this limit, the connector will evict chunks early to free memory. * Therefore, `sink.buffer-flush.max-bytes` should be set larger than `sink.merge-commit.chunk.size` to allow at least one full chunk to be accumulated. In general, `sink.buffer-flush.max-bytes` should be several times larger than `sink.merge-commit.chunk.size`, especially when there are multiple tables or high concurrency. ##### Let server-side DEFAULT values apply with the JSON format[​](#let-server-side-default-values-apply-with-the-json-format "Direct link to Let server-side DEFAULT values apply with the JSON format") With the `json` format, the connector does not send a `columns` header by default. The server therefore includes all table columns in the load. If a column is missing from the JSON payload, the server treats it as an explicit `NULL`, which overrides any `DEFAULT` value defined for that column. For example, if a table contains `ingest_ts DATETIME DEFAULT CURRENT_TIMESTAMP`, the column is loaded as `NULL` unless every row provides a value for `ingest_ts`. To allow server-side `DEFAULT` values to take effect, set `sink.json.columns-from-flink-schema=true`. The connector then uses the Flink table schema to generate the `columns` header, similar to how it handles `csv` when the Flink and StarRocks schemas differ. A StarRocks column that is not defined in the Flink table schema is omitted from the `columns` header, allowing the server to apply its `DEFAULT` value. Therefore, the Flink table definition determines which columns are included in the load: * **Column omitted from the Flink schema:** The connector does not send the column, so StarRocks applies its `DEFAULT` value. * **Column defined in the Flink schema:** The connector sends the value from the row, including `NULL` if the value is null. This option requires a Flink table schema and therefore applies to Flink SQL and sinks created with a schema. For `sink.version=V1`, you must also set `sink.properties.strip_outer_array=true`. The V1 sink sends each batch as a JSON array and does not set this property automatically, unlike V2. The raw `String` DataStream sink does not have a schema and therefore rejects `sink.json.columns-from-flink-schema=true` at startup. For this sink, set `sink.properties.columns` explicitly instead. ##### Monitoring load metrics[​](#monitoring-load-metrics "Direct link to Monitoring load metrics") The Flink connector provides the following metrics to monitor loading. | Metric | Type | Description | | ------------------------ | ------- | ------------------------------------------------------------------ | | totalFlushBytes | counter | successfully flushed bytes. | | totalFlushRows | counter | number of rows successfully flushed. | | totalFlushSucceededTimes | counter | number of times that the data is successfully flushed. | | totalFlushFailedTimes | counter | number of times that the data fails to be flushed. | | totalFilteredRows | counter | number of rows filtered, which is also included in totalFlushRows. | #### Examples[​](#examples "Direct link to Examples") The following examples show how to use the Flink connector to load data into a StarRocks table with Flink SQL or Flink DataStream. ##### Preparations[​](#preparations "Direct link to Preparations") ###### Create a StarRocks table[​](#create-a-starrocks-table "Direct link to Create a StarRocks table") Create a database `test` and create a Primary Key table `score_board`. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Set up Flink environment[​](#set-up-flink-environment "Direct link to Set up Flink environment") * Download Flink binary [Flink 1.15.2](https://archive.apache.org/dist/flink/flink-1.15.2/flink-1.15.2-bin-scala_2.12.tgz), and unzip it to directory `flink-1.15.2`. * Download [Flink connector 1.2.7](https://repo1.maven.org/maven2/com/starrocks/flink-connector-starrocks/1.2.7_flink-1.15/flink-connector-starrocks-1.2.7_flink-1.15.jar), and put it into the directory `flink-1.15.2/lib`. * Run the following commands to start a Flink cluster: ```shell cd flink-1.15.2 ./bin/start-cluster.sh ``` ###### Network configuration[​](#network-configuration "Direct link to Network configuration") Ensure that the machine where Flink is located can access the FE nodes of the StarRocks cluster via the [`http_port`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#http_port) (default: `8030`) and [`query_port`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#query_port) (default: `9030`), and the BE nodes via the [`be_http_port`](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#be_http_port) (default: `8040`). ##### Run with Flink SQL[​](#run-with-flink-sql "Direct link to Run with Flink SQL") * Run the following command to start a Flink SQL client. ```shell ./bin/sql-client.sh ``` * Create a Flink table `score_board`, and insert values into the table via Flink SQL Client. Note you must define the primary key in the Flink DDL if you want to load data into a Primary Key table of StarRocks. It's optional for other types of StarRocks tables. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '' ); INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 100); ``` ##### Run with Flink DataStream[​](#run-with-flink-datastream "Direct link to Run with Flink DataStream") There are several ways to implement a Flink DataStream job according to the type of the input records, such as a CSV Java `String`, a JSON Java `String` or a custom Java object. * The input records are CSV-format `String`. See [LoadCsvRecords](https://github.com/StarRocks/starrocks-connector-for-apache-flink/tree/cd8086cfedc64d5181785bdf5e89a847dc294c1d/examples/src/main/java/com/starrocks/connector/flink/examples/datastream) for a complete example. ```java /** * Generate CSV-format records. Each record has three values separated by "\t". * These values will be loaded to the columns `id`, `name`, and `score` in the StarRocks table. */ String[] records = new String[]{ "1\tstarrocks-csv\t100", "2\tflink-csv\t100" }; DataStream source = env.fromElements(records); /** * Configure the connector with the required properties. * You also need to add properties "sink.properties.format" and "sink.properties.column_separator" * to tell the connector the input records are CSV-format, and the column separator is "\t". * You can also use other column separators in the CSV-format records, * but remember to modify the "sink.properties.column_separator" correspondingly. */ StarRocksSinkOptions options = StarRocksSinkOptions.builder() .withProperty("jdbc-url", jdbcUrl) .withProperty("load-url", loadUrl) .withProperty("database-name", "test") .withProperty("table-name", "score_board") .withProperty("username", "root") .withProperty("password", "") .withProperty("sink.properties.format", "csv") .withProperty("sink.properties.column_separator", "\t") .build(); // Create the sink with the options. SinkFunction starRockSink = StarRocksSink.sink(options); source.addSink(starRockSink); ``` * The input records are JSON-format `String`. See [LoadJsonRecords](https://github.com/StarRocks/starrocks-connector-for-apache-flink/tree/cd8086cfedc64d5181785bdf5e89a847dc294c1d/examples/src/main/java/com/starrocks/connector/flink/examples/datastream) for a complete example. ```java /** * Generate JSON-format records. * Each record has three key-value pairs corresponding to the columns `id`, `name`, and `score` in the StarRocks table. */ String[] records = new String[]{ "{\"id\":1, \"name\":\"starrocks-json\", \"score\":100}", "{\"id\":2, \"name\":\"flink-json\", \"score\":100}", }; DataStream source = env.fromElements(records); /** * Configure the connector with the required properties. * You also need to add properties "sink.properties.format" and "sink.properties.strip_outer_array" * to tell the connector the input records are JSON-format and to strip the outermost array structure. */ StarRocksSinkOptions options = StarRocksSinkOptions.builder() .withProperty("jdbc-url", jdbcUrl) .withProperty("load-url", loadUrl) .withProperty("database-name", "test") .withProperty("table-name", "score_board") .withProperty("username", "root") .withProperty("password", "") .withProperty("sink.properties.format", "json") .withProperty("sink.properties.strip_outer_array", "true") .build(); // Create the sink with the options. SinkFunction starRockSink = StarRocksSink.sink(options); source.addSink(starRockSink); ``` * The input records are custom Java objects. See [LoadCustomJavaRecords](https://github.com/StarRocks/starrocks-connector-for-apache-flink/tree/cd8086cfedc64d5181785bdf5e89a847dc294c1d/examples/src/main/java/com/starrocks/connector/flink/examples/datastream) for a complete example. * In this example, the input record is a simple POJO `RowData`. ```java public static class RowData { public int id; public String name; public int score; public RowData() {} public RowData(int id, String name, int score) { this.id = id; this.name = name; this.score = score; } } ``` * The main program is as follows: ```java // Generate records which use RowData as the container. RowData[] records = new RowData[]{ new RowData(1, "starrocks-rowdata", 100), new RowData(2, "flink-rowdata", 100), }; DataStream source = env.fromElements(records); // Configure the connector with the required properties. StarRocksSinkOptions options = StarRocksSinkOptions.builder() .withProperty("jdbc-url", jdbcUrl) .withProperty("load-url", loadUrl) .withProperty("database-name", "test") .withProperty("table-name", "score_board") .withProperty("username", "root") .withProperty("password", "") .build(); /** * The Flink connector will use a Java object array (Object[]) to represent a row to be loaded into the StarRocks table, * and each element is the value for a column. * You need to define the schema of the Object[] which matches that of the StarRocks table. */ TableSchema schema = TableSchema.builder() .field("id", DataTypes.INT().notNull()) .field("name", DataTypes.STRING()) .field("score", DataTypes.INT()) // When the StarRocks table is a Primary Key table, you must specify notNull(), for example, DataTypes.INT().notNull(), for the primary key `id`. .primaryKey("id") .build(); // Transform the RowData to the Object[] according to the schema. RowDataTransformer transformer = new RowDataTransformer(); // Create the sink with the schema, options, and transformer. SinkFunction starRockSink = StarRocksSink.sink(schema, options, transformer); source.addSink(starRockSink); ``` * The `RowDataTransformer` in the main program is defined as follows: ```java private static class RowDataTransformer implements StarRocksSinkRowBuilder { /** * Set each element of the object array according to the input RowData. * The schema of the array matches that of the StarRocks table. */ @Override public void accept(Object[] internalRow, RowData rowData) { internalRow[0] = rowData.id; internalRow[1] = rowData.name; internalRow[2] = rowData.score; // When the StarRocks table is a Primary Key table, you need to set the last element to indicate whether the data loading is an UPSERT or DELETE operation. internalRow[internalRow.length - 1] = StarRocksSinkOP.UPSERT.ordinal(); } } ``` ##### Synchronize data with Flink CDC 3.0 (with schema change supported)[​](#synchronize-data-with-flink-cdc-30-with-schema-change-supported "Direct link to Synchronize data with Flink CDC 3.0 (with schema change supported)") [Flink CDC 3.0](https://nightlies.apache.org/flink/flink-cdc-docs-stable) framework can be used to easily build a streaming ELT pipeline from CDC sources (such as MySQL and Kafka) to StarRocks. The pipeline can synchronize whole database, merged sharding tables, and schema changes from sources to StarRocks. Since v1.2.9, the Flink connector for StarRocks is integrated into this framework as [StarRocks Pipeline Connector](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.1/docs/connectors/pipeline-connectors/starrocks/). The StarRocks Pipeline Connector supports: * Automatic creation of databases and tables * Schema change synchronization * Full and incremental data synchronization For quick start, see [Streaming ELT from MySQL to StarRocks using Flink CDC 3.0 with StarRocks Pipeline Connector](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.4/docs/get-started/quickstart/mysql-to-starrocks/). It is advised to use StarRocks v3.2.1 and later versions to enable [fast\_schema\_evolution](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md#set-fast-schema-evolution). It will improve the speed of adding or dropping columns and reduce resource usage. #### Best practices[​](#best-practices "Direct link to Best practices") ##### Load data to a Primary Key table[​](#load-data-to-a-primary-key-table "Direct link to Load data to a Primary Key table") This section will show how to load data to a StarRocks Primary Key table to achieve partial updates and conditional updates. You can see [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md) for the introduction of those features. These examples use Flink SQL. ###### Preparations[​](#preparations-1 "Direct link to Preparations") Create a database `test` and create a Primary Key table `score_board` in StarRocks. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Partial update[​](#partial-update "Direct link to Partial update") This example will show how to load data only to columns `id` and `name`. 1. Insert two data rows into the StarRocks table `score_board` in MySQL client. ```sql mysql> INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 100); mysql> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | flink | 100 | +------+-----------+-------+ 2 rows in set (0.02 sec) ``` 2. Create a Flink table `score_board` in Flink SQL client. * Define the DDL which only includes the columns `id` and `name`. * Set the option `sink.properties.partial_update` to `true` which tells the Flink connector to perform partial updates. * If the Flink connector version `<=` 1.2.7, you also need to set the option `sink.properties.columns` to `id,name,__op` to tells the Flink connector which columns need to be updated. Note that you need to append the field `__op` at the end. The field `__op` indicates that the data loading is an UPSERT or DELETE operation, and its values are set by the connector automatically. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.partial_update' = 'true', -- only for Flink connector version <= 1.2.7 'sink.properties.columns' = 'id,name,__op' ); ``` 3. Insert two data rows into the Flink table. The primary keys of the data rows are as same as these of rows in the StarRocks table. but the values in the column `name` are modified. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks-update'), (2, 'flink-update'); ``` 4. Query the StarRocks table in MySQL client. ```sql mysql> select * from score_board; +------+------------------+-------+ | id | name | score | +------+------------------+-------+ | 1 | starrocks-update | 100 | | 2 | flink-update | 100 | +------+------------------+-------+ 2 rows in set (0.02 sec) ``` You can see that only values for `name` change, and the values for `score` do not change. ###### Conditional update[​](#conditional-update "Direct link to Conditional update") This example will show how to do conditional update according to the value of column `score`. The update for an `id` takes effect only when the new value for `score` is has a greater or equal to the old value. 1. Insert two data rows into the StarRocks table in MySQL client. ```sql mysql> INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 100); mysql> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | flink | 100 | +------+-----------+-------+ 2 rows in set (0.02 sec) ``` 2. Create a Flink table `score_board` in the following ways: * Define the DDL including all of columns. * Set the option `sink.properties.merge_condition` to `score` to tell the connector to use the column `score` as the condition. * Set the option `sink.version` to `V1` or `V2`. Both support conditional update. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.merge_condition' = 'score', 'sink.version' = 'V1' ); ``` 3. Insert two data rows into the Flink table. The primary keys of the data rows are as same as these of rows in the StarRocks table. The first data row has a smaller value in the column `score`, and the second data row has a larger value in the column `score`. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks-update', 99), (2, 'flink-update', 101); ``` 4. Query the StarRocks table in MySQL client. ```sql mysql> select * from score_board; +------+--------------+-------+ | id | name | score | +------+--------------+-------+ | 1 | starrocks | 100 | | 2 | flink-update | 101 | +------+--------------+-------+ 2 rows in set (0.03 sec) ``` You can see that only the values of the second data row change, and the values of the first data row do not change. ##### Load data with Merge Commit[​](#load-data-with-merge-commit "Direct link to Load data with Merge Commit") This section shows how to use Merge Commit to improve loading throughput when you have multiple Flink sink subtasks writing to the same StarRocks table. These examples use Flink SQL and StarRocks v3.4.0 or later. ###### Preparations[​](#preparations-2 "Direct link to Preparations") Create a database `test` and create a Primary Key table `score_board` in StarRocks. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Basic configuration[​](#basic-configuration "Direct link to Basic configuration") This Flink SQL enables merge commit with a 10-second merge window. Data from all sink subtasks is merged into a single transaction within each window. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.enable_merge_commit' = 'true', 'sink.properties.merge_commit_interval_ms' = '10000', 'sink.buffer-flush.interval-ms' = '5000' ); ``` Insert data into the Flink table. The data will be loaded into StarRocks via merge commit. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 95), (3, 'spark', 90); ``` ###### In-order loading for Primary Key tables[​](#in-order-loading-for-primary-key-tables "Direct link to In-order loading for Primary Key tables") By default, a single sink subtask may send multiple Stream Load requests concurrently, which can cause out-of-order loading. For Primary Key tables where data ordering matters, there are two approaches to handle this issue. **Method 1: Use `sink.merge-commit.max-concurrent-requests`** Set `sink.merge-commit.max-concurrent-requests` to `0` to ensure each subtask sends requests one at a time. This guarantees in-order loading but may reduce throughput. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.enable_merge_commit' = 'true', 'sink.properties.merge_commit_interval_ms' = '10000', 'sink.buffer-flush.interval-ms' = '5000', 'sink.merge-commit.max-concurrent-requests' = '0' ); INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 95), (3, 'spark', 90); ``` **Method 2: Use Conditional Update** If you want to keep concurrent requests for higher throughput but still prevent older data from overwriting newer data, you can use [Conditional Update](#conditional-update). Set `sink.properties.merge_condition` to a column (for example, a version or timestamp column) so that an update only takes effect when the incoming value is greater than or equal to the existing value. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.enable_merge_commit' = 'true', 'sink.properties.merge_commit_interval_ms' = '10000', 'sink.buffer-flush.interval-ms' = '5000', 'sink.properties.merge_condition' = 'score' ); INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 95), (3, 'spark', 90); ``` With this configuration, concurrent requests are allowed (default `sink.merge-commit.max-concurrent-requests` is `Integer.MAX_VALUE`), but an update to a row only takes effect when the new `score` is greater than or equal to the existing `score`. This prevents newer data from being overwritten by older data even under out-of-order loading. ##### Load data into columns of BITMAP type[​](#load-data-into-columns-of-bitmap-type "Direct link to Load data into columns of BITMAP type") [`BITMAP`](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/BITMAP.md) is often used to accelerate count distinct, such as counting UV, see [Use Bitmap for exact Count Distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_bitmap.md). Here we take the counting of UV as an example to show how to load data into columns of the `BITMAP` type. 1. Create a StarRocks Aggregate table in MySQL client. In the database `test`, create an Aggregate table `page_uv` where the column `visit_users` is defined as the `BITMAP` type and configured with the aggregate function `BITMAP_UNION`. ```sql CREATE TABLE `test`.`page_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` BITMAP BITMAP_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`); ``` 2. Create a Flink table in Flink SQL client. The column `visit_user_id` in the Flink table is of `BIGINT` type, and we want to load this column to the column `visit_users` of `BITMAP` type in the StarRocks table. So when defining the DDL of the Flink table, note that: * Because Flink does not support `BITMAP`, you need to define a column `visit_user_id` as `BIGINT` type to represent the column `visit_users` of `BITMAP` type in the StarRocks table. * You need to set the option `sink.properties.columns` to `page_id,visit_date,user_id,visit_users=to_bitmap(visit_user_id)`, which tells the connector the column mapping between the Flink table and StarRocks table. Also you need to use [`to_bitmap`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/to_bitmap.md) function to tell the connector to convert the data of `BIGINT` type into `BITMAP` type. ```sql CREATE TABLE `page_uv` ( `page_id` INT, `visit_date` TIMESTAMP, `visit_user_id` BIGINT ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'page_uv', 'username' = 'root', 'password' = '', 'sink.properties.columns' = 'page_id,visit_date,visit_user_id,visit_users=to_bitmap(visit_user_id)' ); ``` 3. Load data into Flink table in Flink SQL client. ```sql INSERT INTO `page_uv` VALUES (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 13), (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 23), (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 33), (1, CAST('2020-06-23 02:30:30' AS TIMESTAMP), 13), (2, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 23); ``` 4. Calculate page UVs from the StarRocks table in MySQL client. ```sql MySQL [test]> SELECT `page_id`, COUNT(DISTINCT `visit_users`) FROM `page_uv` GROUP BY `page_id`; +---------+-----------------------------+ | page_id | count(DISTINCT visit_users) | +---------+-----------------------------+ | 2 | 1 | | 1 | 3 | +---------+-----------------------------+ 2 rows in set (0.05 sec) ``` ##### Load data into columns of HLL type[​](#load-data-into-columns-of-hll-type "Direct link to Load data into columns of HLL type") [`HLL`](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/HLL.md) can be used for approximate count distinct, see [Use HLL for approximate count distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_HLL.md). Here we take the counting of UV as an example to show how to load data into columns of the `HLL` type. 1. Create a StarRocks Aggregate table In the database `test`, create an Aggregate table `hll_uv` where the column `visit_users` is defined as the `HLL` type and configured with the aggregate function `HLL_UNION`. ```sql CREATE TABLE `hll_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` HLL HLL_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`); ``` 2. Create a Flink table in Flink SQL client. The column `visit_user_id` in the Flink table is of `BIGINT` type, and we want to load this column to the column `visit_users` of `HLL` type in the StarRocks table. So when defining the DDL of the Flink table, note that: * Because Flink does not support `BITMAP`, you need to define a column `visit_user_id` as `BIGINT` type to represent the column `visit_users` of `HLL` type in the StarRocks table. * You need to set the option `sink.properties.columns` to `page_id,visit_date,user_id,visit_users=hll_hash(visit_user_id)` which tells the connector the column mapping between Flink table and StarRocks table. Also you need to use [`hll_hash`](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_hash.md) function to tell the connector to convert the data of `BIGINT` type into `HLL` type. ```sql CREATE TABLE `hll_uv` ( `page_id` INT, `visit_date` TIMESTAMP, `visit_user_id` BIGINT ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'hll_uv', 'username' = 'root', 'password' = '', 'sink.properties.columns' = 'page_id,visit_date,visit_user_id,visit_users=hll_hash(visit_user_id)' ); ``` 3. Load data into Flink table in Flink SQL client. ```sql INSERT INTO `hll_uv` VALUES (3, CAST('2023-07-24 12:00:00' AS TIMESTAMP), 78), (4, CAST('2023-07-24 13:20:10' AS TIMESTAMP), 2), (3, CAST('2023-07-24 12:30:00' AS TIMESTAMP), 674); ``` 4. Calculate page UVs from the StarRocks table in MySQL client. ```sql mysql> SELECT `page_id`, COUNT(DISTINCT `visit_users`) FROM `hll_uv` GROUP BY `page_id`; **+---------+-----------------------------+ | page_id | count(DISTINCT visit_users) | +---------+-----------------------------+ | 3 | 2 | | 4 | 1 | +---------+-----------------------------+ 2 rows in set (0.04 sec) ``` --- ### Load data from HDFS StarRocks provides the following options for loading data from HDFS: * Synchronous loading using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md)+[`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) * Asynchronous loading using [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) * Continuous asynchronous loading using [Pipe](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md) Each of these options has its own advantages, which are detailed in the following sections. In most cases, we recommend that you use the INSERT+`FILES()` method, which is much easier to use. However, the INSERT+`FILES()` method currently supports only the Parquet, ORC, and CSV file formats. Therefore, if you need to load data of other file formats such as JSON, or perform data changes such as DELETE during data loading, you can resort to Broker Load. If you need to load a large number of data files with a significant data volume in total (for example, more than 100 GB or even 1 TB), we recommend that you use the Pipe method. Pipe can split the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job and minimizes the need for retries due to data errors. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Make source data ready[​](#make-source-data-ready "Direct link to Make source data ready") Make sure the source data you want to load into StarRocks is properly stored in your HDFS cluster. This topic assumes that you want to load `/user/amber/user_behavior_ten_million_rows.parquet` from HDFS into StarRocks. ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ##### Gather authentication details[​](#gather-authentication-details "Direct link to Gather authentication details") You can use the simple authentication method to establish connections with your HDFS cluster. To use simple authentication, you need to gather the username and password of the account that you can use to access the NameNode of the HDFS cluster. #### Use INSERT+FILES()[​](#use-insertfiles "Direct link to Use INSERT+FILES()") This method is available from v3.1 onwards and currently supports only the Parquet, ORC, and CSV (from v3.3.0 onwards) file formats. ##### Advantages of INSERT+FILES()[​](#advantages-of-insertfiles "Direct link to Advantages of INSERT+FILES()") [`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) can read the file stored in cloud storage based on the path-related properties you specify, infer the table schema of the data in the file, and then return the data from the file as data rows. With `FILES()`, you can: * Query the data directly from HDFS using [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). * Create and load a table using [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) (CTAS). * Load the data into an existing table using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). ##### Typical examples[​](#typical-examples "Direct link to Typical examples") ###### Querying directly from HDFS using SELECT[​](#querying-directly-from-hdfs-using-select "Direct link to Querying directly from HDFS using SELECT") Querying directly from HDFS using SELECT+`FILES()` can give a good preview of the content of a dataset before you create a table. For example: * Get a preview of the dataset without storing the data. * Query for the min and max values and decide what data types to use. * Check for `NULL` values. The following example queries the data file `/user/amber/user_behavior_ten_million_rows.parquet` stored in the HDFS cluster: ```sql SELECT * FROM FILES ( "path" = "hdfs://:/user/amber/user_behavior_ten_million_rows.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ) LIMIT 3; ``` The system returns the following query result: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 543711 | 829192 | 2355072 | pv | 2017-11-27 08:22:37 | | 543711 | 2056618 | 3645362 | pv | 2017-11-27 10:16:46 | | 543711 | 1165492 | 3645362 | pv | 2017-11-27 10:17:00 | +--------+---------+------------+--------------+---------------------+ ``` > **NOTE** > > Notice that the column names as returned above are provided by the Parquet file. ###### Creating and loading a table using CTAS[​](#creating-and-loading-a-table-using-ctas "Direct link to Creating and loading a table using CTAS") This is a continuation of the previous example. The previous query is wrapped in CREATE TABLE AS SELECT (CTAS) to automate the table creation using schema inference. This means StarRocks will infer the table schema, create the table you want, and then load the data into the table. The column names and types are not required to create a table when using the `FILES()` table function with Parquet files as the Parquet format includes the column names. > **NOTE** > > The syntax of CREATE TABLE when using schema inference does not allow setting the number of replicas, so set it before creating the table. The example below is for a system with three replicas: > > ```sql > ADMIN SET FRONTEND CONFIG ('default_replication_num' = "3"); > > ``` Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Use CTAS to create a table and load the data of the data file `/user/amber/user_behavior_ten_million_rows.parquet` into the table: ```sql CREATE TABLE user_behavior_inferred AS SELECT * FROM FILES ( "path" = "hdfs://:/user/amber/user_behavior_ten_million_rows.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ); ``` After creating the table, you can view its schema by using [DESCRIBE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md): ```sql DESCRIBE user_behavior_inferred; ``` The system returns the following query result: ```plain +--------------+-----------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+-----------+------+-------+---------+-------+ | UserID | bigint | YES | true | NULL | | | ItemID | bigint | YES | true | NULL | | | CategoryID | bigint | YES | true | NULL | | | BehaviorType | varbinary | YES | false | NULL | | | Timestamp | varbinary | YES | false | NULL | | +--------------+-----------+------+-------+---------+-------+ ``` Query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_inferred LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+--------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+--------+------------+--------------+---------------------+ | 84 | 56257 | 1879194 | pv | 2017-11-26 05:56:23 | | 84 | 108021 | 2982027 | pv | 2017-12-02 05:43:00 | | 84 | 390657 | 1879194 | pv | 2017-11-28 11:20:30 | +--------+--------+------------+--------------+---------------------+ ``` ###### Loading into an existing table using INSERT[​](#loading-into-an-existing-table-using-insert "Direct link to Loading into an existing table using INSERT") You may want to customize the table that you are inserting into, for example, the: * column data type, nullable setting, or default values * key types and columns * data partitioning and bucketing > **NOTE** > > Creating the most efficient table structure requires knowledge of how the data will be used and the content of the columns. This topic does not cover table design. For information about table design, see [Table types](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md). In this example, we are creating a table based on knowledge of how the table will be queried and the data in the Parquet file. The knowledge of the data in the Parquet file can be gained by querying the file directly in HDFS. * Since a query of the dataset in HDFS indicates that the `Timestamp` column contains data that matches a VARBINARY data type, the column type is specified in the following DDL. * By querying the data in HDFS, you can find that there are no `NULL` values in the dataset, so the DDL does not set any columns as nullable. * Based on knowledge of the expected query types, the sort key and bucketing column are set to the column `UserID`. Your use case might be different for this data, so you might decide to use `ItemID` in addition to or instead of `UserID` for the sort key. Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from HDFS): ```sql CREATE TABLE user_behavior_declared ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` Display the schema so that you can compare it with the inferred schema produced by the `FILES()` table function: ```sql DESCRIBE user_behavior_declared; ``` ```plaintext +--------------+----------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+----------------+------+-------+---------+-------+ | UserID | int | NO | true | NULL | | | ItemID | int | NO | false | NULL | | | CategoryID | int | NO | false | NULL | | | BehaviorType | varchar(65533) | NO | false | NULL | | | Timestamp | varbinary | NO | false | NULL | | +--------------+----------------+------+-------+---------+-------+ 5 rows in set (0.00 sec) ``` tip Compare the schema you just created with the schema inferred earlier using the `FILES()` table function. Look at: * data types * nullable * key fields To better control the schema of the destination table and for better query performance, we recommend that you specify the table schema by hand in production environments. After creating the table, you can load it with INSERT INTO SELECT FROM FILES(): ```sql INSERT INTO user_behavior_declared SELECT * FROM FILES ( "path" = "hdfs://:/user/amber/user_behavior_ten_million_rows.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ); ``` After the load is complete, you can query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_declared LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 107 | 1568743 | 4476428 | pv | 2017-11-25 14:29:53 | | 107 | 470767 | 1020087 | pv | 2017-11-25 14:32:31 | | 107 | 358238 | 1817004 | pv | 2017-11-25 14:43:23 | +--------+---------+------------+--------------+---------------------+ ``` ###### Check load progress[​](#check-load-progress "Direct link to Check load progress") You can query the progress of INSERT jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. Example: ```sql SELECT * FROM information_schema.loads ORDER BY JOB_ID DESC; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'insert_0d86c3f9-851f-11ee-9c3e-00163e044958' \G *************************** 1. row *************************** JOB_ID: 10214 LABEL: insert_0d86c3f9-851f-11ee-9c3e-00163e044958 DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: INSERT PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):300; max_filter_ratio:0.0 CREATE_TIME: 2023-11-17 15:58:14 ETL_START_TIME: 2023-11-17 15:58:14 ETL_FINISH_TIME: 2023-11-17 15:58:14 LOAD_START_TIME: 2023-11-17 15:58:14 LOAD_FINISH_TIME: 2023-11-17 15:58:18 JOB_DETAILS: {"All backends":{"0d86c3f9-851f-11ee-9c3e-00163e044958":[10120]},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":311710786,"InternalTableLoadRows":10000000,"ScanBytes":581574034,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"0d86c3f9-851f-11ee-9c3e-00163e044958":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` > **NOTE** > > INSERT is a synchronous command. If an INSERT job is still running, you need to open another session to check its execution status. #### Use Broker Load[​](#use-broker-load "Direct link to Use Broker Load") An asynchronous Broker Load process handles making the connection to HDFS, pulling the data, and storing the data in StarRocks. This method supports the following file formats: * Parquet * ORC * CSV * JSON (supported from v3.2.3 onwards) ##### Advantages of Broker Load[​](#advantages-of-broker-load "Direct link to Advantages of Broker Load") * Broker Load runs in the background and clients do not need to stay connected for the job to continue. * Broker Load is preferred for long-running jobs, with the default timeout spanning 4 hours. * In addition to Parquet and ORC file format, Broker Load supports CSV file format and JSON file format (JSON file format is supported from v3.2.3 onwards). ##### Data flow[​](#data-flow "Direct link to Data flow") ![Workflow of Broker Load](/assets/images/broker_load_how-to-work_en-bb36de70866e6366b2b21808f0f77be8.png) 1. The user creates a load job. 2. The frontend (FE) creates a query plan and distributes the plan to the backend nodes (BEs) or compute nodes (CNs). 3. The BEs or CNs pull the data from the source and load the data into StarRocks. ##### Typical example[​](#typical-example "Direct link to Typical example") Create a table, start a load process that pulls the data file `/user/amber/user_behavior_ten_million_rows.parquet` from HDFS, and verify the progress and success of the data loading. ###### Create a database and a table[​](#create-a-database-and-a-table "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table has the same schema as the Parquet file that you want to load from HDFS): ```sql CREATE TABLE user_behavior ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Broker Load[​](#start-a-broker-load "Direct link to Start a Broker Load") Run the following command to start a Broker Load job that loads data from the data file `/user/amber/user_behavior_ten_million_rows.parquet` to the `user_behavior` table: ```sql LOAD LABEL user_behavior ( DATA INFILE("hdfs://:/user/amber/user_behavior_ten_million_rows.parquet") INTO TABLE user_behavior FORMAT AS "parquet" ) WITH BROKER ( "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ) PROPERTIES ( "timeout" = "72000" ); ``` This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. * `BROKER`: The connection details for the source. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Check load progress[​](#check-load-progress-1 "Direct link to Check load progress") You can query the progress of Broker Load jobs from the `information_schema.loads` view. This feature is supported from v3.1 onwards. ```sql SELECT * FROM information_schema.loads; ``` For information about the fields provided in the `loads` view, see [Information Schema](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md)). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'user_behavior'; ``` In the output below there are two entries for the load job `user_behavior`: * The first record shows a state of `CANCELLED`. Scroll to `ERROR_MSG`, and you can see that the job has failed due to `listPath failed`. * The second record shows a state of `FINISHED`, which means that the job has succeeded. ```plaintext JOB_ID|LABEL |DATABASE_NAME|STATE |PROGRESS |TYPE |PRIORITY|SCAN_ROWS|FILTERED_ROWS|UNSELECTED_ROWS|SINK_ROWS|ETL_INFO|TASK_INFO |CREATE_TIME |ETL_START_TIME |ETL_FINISH_TIME |LOAD_START_TIME |LOAD_FINISH_TIME |JOB_DETAILS |ERROR_MSG |TRACKING_URL|TRACKING_SQL|REJECTED_RECORD_PATH| ------+-------------------------------------------+-------------+---------+-------------------+------+--------+---------+-------------+---------------+---------+--------+----------------------------------------------------+-------------------+-------------------+-------------------+-------------------+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+------------+------------+--------------------+ 10121|user_behavior |mydatabase |CANCELLED|ETL:N/A; LOAD:N/A |BROKER|NORMAL | 0| 0| 0| 0| |resource:N/A; timeout(s):72000; max_filter_ratio:0.0|2023-08-10 14:59:30| | | |2023-08-10 14:59:34|{"All backends":{},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":0,"InternalTableLoadRows":0,"ScanBytes":0,"ScanRows":0,"TaskNumber":0,"Unfinished backends":{}} |type:ETL_RUN_FAIL; msg:listPath failed| | | | 10106|user_behavior |mydatabase |FINISHED |ETL:100%; LOAD:100%|BROKER|NORMAL | 86953525| 0| 0| 86953525| |resource:N/A; timeout(s):72000; max_filter_ratio:0.0|2023-08-10 14:50:15|2023-08-10 14:50:19|2023-08-10 14:50:19|2023-08-10 14:50:19|2023-08-10 14:55:10|{"All backends":{"a5fe5e1d-d7d0-4826-ba99-c7348f9a5f2f":[10004]},"FileNumber":1,"FileSize":1225637388,"InternalTableLoadBytes":2710603082,"InternalTableLoadRows":86953525,"ScanBytes":1225637388,"ScanRows":86953525,"TaskNumber":1,"Unfinished backends":{"a5| | | | | ``` After you confirm that the load job has finished, you can check a subset of the destination table to see if the data has been successfully loaded. Example: ```sql SELECT * from user_behavior LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 142 | 2869980 | 2939262 | pv | 2017-11-25 03:43:22 | | 142 | 2522236 | 1669167 | pv | 2017-11-25 15:14:12 | | 142 | 3031639 | 3607361 | pv | 2017-11-25 15:19:25 | +--------+---------+------------+--------------+---------------------+ ``` #### Use Pipe[​](#use-pipe "Direct link to Use Pipe") Starting from v3.2, StarRocks provides the Pipe loading method, which currently supports only the Parquet and ORC file formats. ##### Advantages of Pipe[​](#advantages-of-pipe "Direct link to Advantages of Pipe") Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading.During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that HDFS uses LastModifiedTime as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Data flow[​](#data-flow "Direct link to Data flow") ![Pipe data flow](/assets/images/pipe_data_flow-2a4dc0b44a06c987d9afc0ecf632f5d9.png) Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading. During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that HDFS uses `LastModifiedTime` as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Data flow[​](#data-flow-1 "Direct link to Data flow") ![Pipe data flow](/assets/images/pipe_data_flow-2a4dc0b44a06c987d9afc0ecf632f5d9.png) ##### Differences between Pipe and INSERT+FILES()[​](#differences-between-pipe-and-insertfiles "Direct link to Differences between Pipe and INSERT+FILES()") A Pipe job is split into one or more transactions based on the size and number of rows in each data file. Users can query the intermediate results during the loading process. In contrast, an INSERT+`FILES()` job is processed as a single transaction, and users are unable to view the data during the loading process. ##### File loading sequence[​](#file-loading-sequence "Direct link to File loading sequence") For each Pipe job, StarRocks maintains a file queue, from which it fetches and loads data files as micro-batches. Pipe does not ensure that the data files are loaded in the same order as they are uploaded. Therefore, newer data may be loaded prior to older data. ##### Typical example[​](#typical-example-1 "Direct link to Typical example") ###### Create a database and a table[​](#create-a-database-and-a-table-1 "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from HDFS): ```sql CREATE TABLE user_behavior_replica ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Pipe job[​](#start-a-pipe-job "Direct link to Start a Pipe job") Run the following command to start a Pipe job that loads data from the data file `/user/amber/user_behavior_ten_million_rows.parquet` to the `user_behavior_replica` table: ```sql CREATE PIPE user_behavior_replica PROPERTIES ( "AUTO_INGEST" = "TRUE" ) AS INSERT INTO user_behavior_replica SELECT * FROM FILES ( "path" = "hdfs://:/user/amber/user_behavior_ten_million_rows.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ); ``` This job has four main sections: * `pipe_name`: The name of the pipe. The pipe name must be unique within the database to which the pipe belongs. * `INSERT_SQL`: The INSERT INTO SELECT FROM FILES statement that is used to load data from the specified source data file to the destination table. * `PROPERTIES`: A set of optional parameters that specify how to execute the pipe. These include `AUTO_INGEST`, `POLL_INTERVAL`, `BATCH_SIZE`, and `BATCH_FILES`. Specify these properties in the `"key" = "value"` format. For detailed syntax and parameter descriptions, see [CREATE PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md). ###### Check load progress[​](#check-load-progress-2 "Direct link to Check load progress") * Query the progress of Pipe jobs by using [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md). ```sql SHOW PIPES; ``` If you have submitted multiple load jobs, you can filter on the `NAME` associated with the job. Example: ```sql SHOW PIPES WHERE NAME = 'user_behavior_replica' \G *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10252 PIPE_NAME: user_behavior_replica STATE: RUNNING TABLE_NAME: mydatabase.user_behavior_replica LOAD_STATUS: {"loadedFiles":1,"loadedBytes":132251298,"loadingFiles":0,"lastLoadedTime":"2023-11-17 16:13:22"} LAST_ERROR: NULL CREATED_TIME: 2023-11-17 16:13:15 1 row in set (0.00 sec) ``` * Query the progress of Pipe jobs from the [`pipes`](https://docs.starrocks.io/docs/sql-reference/information_schema/pipes.md) view in the StarRocks Information Schema. ```sql SELECT * FROM information_schema.pipes; ``` If you have submitted multiple load jobs, you can filter on the `PIPE_NAME` associated with the job. Example: ```sql SELECT * FROM information_schema.pipes WHERE pipe_name = 'user_behavior_replica' \G *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10252 PIPE_NAME: user_behavior_replica STATE: RUNNING TABLE_NAME: mydatabase.user_behavior_replica LOAD_STATUS: {"loadedFiles":1,"loadedBytes":132251298,"loadingFiles":0,"lastLoadedTime":"2023-11-17 16:13:22"} LAST_ERROR: CREATED_TIME: 2023-11-17 16:13:15 1 row in set (0.00 sec) ``` ###### Check file status[​](#check-file-status "Direct link to Check file status") You can query the load status of the files loaded from the [`pipe_files`](https://docs.starrocks.io/docs/sql-reference/information_schema/pipe_files.md) view in the StarRocks Information Schema. ```sql SELECT * FROM information_schema.pipe_files; ``` If you have submitted multiple load jobs, you can filter on the `PIPE_NAME` associated with the job. Example: ```sql SELECT * FROM information_schema.pipe_files WHERE pipe_name = 'user_behavior_replica' \G *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10252 PIPE_NAME: user_behavior_replica FILE_NAME: hdfs://172.26.195.67:9000/user/amber/user_behavior_ten_million_rows.parquet FILE_VERSION: 1700035418838 FILE_SIZE: 132251298 LAST_MODIFIED: 2023-11-15 08:03:38 LOAD_STATE: FINISHED STAGED_TIME: 2023-11-17 16:13:16 START_LOAD_TIME: 2023-11-17 16:13:17 FINISH_LOAD_TIME: 2023-11-17 16:13:22 ERROR_MSG: 1 row in set (0.02 sec) ``` ###### Manage Pipes[​](#manage-pipes "Direct link to Manage Pipes") You can alter, suspend or resume, drop, or query the pipes you have created and retry to load specific data files. For more information, see [ALTER PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/ALTER_PIPE.md), [SUSPEND or RESUME PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SUSPEND_or_RESUME_PIPE.md), [DROP PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/DROP_PIPE.md), [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md), and [RETRY FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/RETRY_FILE.md). --- ### Load data using INSERT This topic describes how to load data into StarRocks by using a SQL statement - INSERT. Similar to MySQL and many other database management systems, StarRocks supports loading data to an internal table with INSERT. You can insert one or more rows directly with the VALUES clause to test a function or a DEMO. You can also insert data defined by the results of a query into an internal table from an [external table](https://docs.starrocks.io/docs/data_source/External_table.md). From StarRocks v3.1 onwards, you can directly load data from files on cloud storage using the INSERT command and the table function [FILES()](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md). StarRocks v2.4 further supports overwriting data into a table by using INSERT OVERWRITE. The INSERT OVERWRITE statement integrates the following operations to implement the overwriting function: 1. Creates temporary partitions according to the partitions that store the original data. 2. Inserts data into the temporary partitions. 3. Swaps the original partitions with the temporary partitions. > **NOTE** > > If you need to verify the data before overwriting it, instead of using INSERT OVERWRITE, you can follow the above procedures to overwrite your data and verify it before swapping the partitions. From v3.4.0 onwards, StarRocks supports a new semantic - Dynamic Overwrite for INSERT OVERWRITE with partitioned tables. For more information, see [Dynamic Overwrite](#dynamic-overwrite). #### Precautions[​](#precautions "Direct link to Precautions") * You can cancel a synchronous INSERT transaction only by pressing the **Ctrl** and **C** keys from your MySQL client. * You can submit an asynchronous INSERT task using [SUBMIT TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md). * As for the current version of StarRocks, the INSERT transaction fails by default if the data of any rows does not comply with the schema of the table. For example, the INSERT transaction fails if the length of a field in any row exceeds the length limit for the mapping field in the table. You can set the session variable `enable_insert_strict` to `false` to allow the transaction to continue by filtering out the rows that mismatch the table. * If you execute the INSERT statement frequently to load small batches of data into StarRocks, excessive data versions are generated. It severely affects query performance. We recommend that, in production, you should not load data with the INSERT command too often or use it as a routine for data loading on a daily basis. If your application or analytic scenario demand solutions to loading streaming data or small data batches separately, we recommend you use Apache Kafka® as your data source and load the data via Routine Load. * If you execute the INSERT OVERWRITE statement, StarRocks creates temporary partitions for the partitions which store the original data, inserts new data into the temporary partitions, and [swaps the original partitions with the temporary partitions](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md#use-a-temporary-partition-to-replace-the-current-partition). All these operations are executed in the FE Leader node. Hence, if the FE Leader node crashes while executing INSERT OVERWRITE command, the whole load transaction will fail, and the temporary partitions will be truncated. #### Preparation[​](#preparation "Direct link to Preparation") ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ##### Create objects[​](#create-objects "Direct link to Create objects") Create a database named `load_test`, and create a table `insert_wiki_edit` as the destination table and a table `source_wiki_edit` as the source table. > **NOTE** > > Examples demonstrated in this topic are based on the table `insert_wiki_edit` and the table `source_wiki_edit`. If you prefer working with your own tables and data, you can skip the preparation and move on to the next step. ```sql CREATE DATABASE IF NOT EXISTS load_test; USE load_test; CREATE TABLE insert_wiki_edit ( event_time DATETIME, channel VARCHAR(32) DEFAULT '', user VARCHAR(128) DEFAULT '', is_anonymous TINYINT DEFAULT '0', is_minor TINYINT DEFAULT '0', is_new TINYINT DEFAULT '0', is_robot TINYINT DEFAULT '0', is_unpatrolled TINYINT DEFAULT '0', delta INT DEFAULT '0', added INT DEFAULT '0', deleted INT DEFAULT '0' ) DUPLICATE KEY( event_time, channel, user, is_anonymous, is_minor, is_new, is_robot, is_unpatrolled ) PARTITION BY RANGE(event_time)( PARTITION p06 VALUES LESS THAN ('2015-09-12 06:00:00'), PARTITION p12 VALUES LESS THAN ('2015-09-12 12:00:00'), PARTITION p18 VALUES LESS THAN ('2015-09-12 18:00:00'), PARTITION p24 VALUES LESS THAN ('2015-09-13 00:00:00') ) DISTRIBUTED BY HASH(user); CREATE TABLE source_wiki_edit ( event_time DATETIME, channel VARCHAR(32) DEFAULT '', user VARCHAR(128) DEFAULT '', is_anonymous TINYINT DEFAULT '0', is_minor TINYINT DEFAULT '0', is_new TINYINT DEFAULT '0', is_robot TINYINT DEFAULT '0', is_unpatrolled TINYINT DEFAULT '0', delta INT DEFAULT '0', added INT DEFAULT '0', deleted INT DEFAULT '0' ) DUPLICATE KEY( event_time, channel,user, is_anonymous, is_minor, is_new, is_robot, is_unpatrolled ) PARTITION BY RANGE(event_time)( PARTITION p06 VALUES LESS THAN ('2015-09-12 06:00:00'), PARTITION p12 VALUES LESS THAN ('2015-09-12 12:00:00'), PARTITION p18 VALUES LESS THAN ('2015-09-12 18:00:00'), PARTITION p24 VALUES LESS THAN ('2015-09-13 00:00:00') ) DISTRIBUTED BY HASH(user); ``` > **NOTICE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). #### Insert data via INSERT INTO VALUES[​](#insert-data-via-insert-into-values "Direct link to Insert data via INSERT INTO VALUES") You can append one or more rows to a specific table by using INSERT INTO VALUES command. Multiple rows are separated by comma (,). For detailed instructions and parameter references, see [SQL Reference - INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). > **CAUTION** > > Inserting data via INSERT INTO VALUES merely applies to the situation when you need to verify a DEMO with a small dataset. It is not recommended for a massive testing or production environment. To load mass data into StarRocks, see [Loading options](https://docs.starrocks.io/docs/loading/loading_introduction.md) for other options that suit your scenarios. The following example inserts two rows into the data source table `source_wiki_edit` with the label `insert_load_wikipedia`. Label is the unique identification label for each data load transaction within the database. ```sql INSERT INTO source_wiki_edit WITH LABEL insert_load_wikipedia VALUES ("2015-09-12 00:00:00","#en.wikipedia","AustinFF",0,0,0,0,0,21,5,0), ("2015-09-12 00:00:00","#ca.wikipedia","helloSR",0,1,0,1,0,3,23,0); ``` #### Insert data via INSERT INTO SELECT[​](#insert-data-via-insert-into-select "Direct link to Insert data via INSERT INTO SELECT") You can load the result of a query on a data source table into the target table via INSERT INTO SELECT command. INSERT INTO SELECT command performs ETL operations on the data from the data source table, and loads the data into an internal table in StarRocks. The data source can be one or more internal or external tables, or even data files on cloud storage. The target table MUST be an internal table in StarRocks. For detailed instructions and parameter references, see [SQL Reference - INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). ##### Insert data from an internal or external table into an internal table[​](#insert-data-from-an-internal-or-external-table-into-an-internal-table "Direct link to Insert data from an internal or external table into an internal table") > **NOTE** > > Inserting data from an external table is identical to inserting data from an internal table. For simplicity, we only demonstrate how to insert data from an internal table in the following examples. * The following example inserts the data from the source table to the target table `insert_wiki_edit`. ```sql INSERT INTO insert_wiki_edit WITH LABEL insert_load_wikipedia_1 SELECT * FROM source_wiki_edit; ``` * The following example inserts the data from the source table to the `p06` and `p12` partitions of the target table `insert_wiki_edit`. If no partition is specified, the data will be inserted into all partitions. Otherwise, the data will be inserted only into the specified partition(s). ```sql INSERT INTO insert_wiki_edit PARTITION(p06, p12) WITH LABEL insert_load_wikipedia_2 SELECT * FROM source_wiki_edit; ``` Query the target table to make sure there is data in them. ```plain MySQL > select * from insert_wiki_edit; +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | event_time | channel | user | is_anonymous | is_minor | is_new | is_robot | is_unpatrolled | delta | added | deleted | +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | 2015-09-12 00:00:00 | #en.wikipedia | AustinFF | 0 | 0 | 0 | 0 | 0 | 21 | 5 | 0 | | 2015-09-12 00:00:00 | #ca.wikipedia | helloSR | 0 | 1 | 0 | 1 | 0 | 3 | 23 | 0 | +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ 2 rows in set (0.00 sec) ``` If you truncate the `p06` and `p12` partitions, the data will not be returned in a query. ```plain MySQL > TRUNCATE TABLE insert_wiki_edit PARTITION(p06, p12); Query OK, 0 rows affected (0.01 sec) MySQL > select * from insert_wiki_edit; Empty set (0.00 sec) ``` * The following example inserts the `event_time` and `channel` columns from the source table to the target table `insert_wiki_edit`. Default values are used in the columns that are not specified here. ```sql INSERT INTO insert_wiki_edit WITH LABEL insert_load_wikipedia_3 ( event_time, channel ) SELECT event_time, channel FROM source_wiki_edit; ``` note From v3.3.1, specifying a column list in the INSERT INTO statement on a Primary Key table will perform Partial Updates (instead of Full Upsert in earlier versions). If the column list is not specified, the system will perform Full Upsert. ##### Insert data directly from files in an external source using FILES()[​](#insert-data-directly-from-files-in-an-external-source-using-files "Direct link to Insert data directly from files in an external source using FILES()") From v3.1 onwards, StarRocks supports directly loading data from files on cloud storage using the INSERT command and the [FILES()](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) function, thereby you do not need to create an external catalog or file external table first. Besides, FILES() can automatically infer the table schema of the files, greatly simplifying the process of data loading. The following example inserts data rows from the Parquet file **parquet/insert\_wiki\_edit\_append.parquet** within the AWS S3 bucket `inserttest` into the table `insert_wiki_edit`: ```plain INSERT INTO insert_wiki_edit SELECT * FROM FILES( "path" = "s3://inserttest/parquet/insert_wiki_edit_append.parquet", "format" = "parquet", "aws.s3.access_key" = "XXXXXXXXXX", "aws.s3.secret_key" = "YYYYYYYYYY", "aws.s3.region" = "us-west-2" ); ``` #### Overwrite data via INSERT OVERWRITE VALUES[​](#overwrite-data-via-insert-overwrite-values "Direct link to Overwrite data via INSERT OVERWRITE VALUES") You can overwrite a specific table with one or more rows by using INSERT OVERWRITE VALUES command. Multiple rows are separated by comma (,). For detailed instructions and parameter references, see [SQL Reference - INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). > **CAUTION** > > Overwriting data via INSERT OVERWRITE VALUES merely applies to the situation when you need to verify a DEMO with a small dataset. It is not recommended for a massive testing or production environment. To load mass data into StarRocks, see [Loading options](https://docs.starrocks.io/docs/loading/loading_introduction.md) for other options that suit your scenarios. Query the source table and the target table to make sure there is data in them. ```plain MySQL > SELECT * FROM source_wiki_edit; +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | event_time | channel | user | is_anonymous | is_minor | is_new | is_robot | is_unpatrolled | delta | added | deleted | +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | 2015-09-12 00:00:00 | #ca.wikipedia | helloSR | 0 | 1 | 0 | 1 | 0 | 3 | 23 | 0 | | 2015-09-12 00:00:00 | #en.wikipedia | AustinFF | 0 | 0 | 0 | 0 | 0 | 21 | 5 | 0 | +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ 2 rows in set (0.02 sec) MySQL > SELECT * FROM insert_wiki_edit; +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | event_time | channel | user | is_anonymous | is_minor | is_new | is_robot | is_unpatrolled | delta | added | deleted | +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | 2015-09-12 00:00:00 | #ca.wikipedia | helloSR | 0 | 1 | 0 | 1 | 0 | 3 | 23 | 0 | | 2015-09-12 00:00:00 | #en.wikipedia | AustinFF | 0 | 0 | 0 | 0 | 0 | 21 | 5 | 0 | +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ 2 rows in set (0.01 sec) ``` The following example overwrites the source table `source_wiki_edit` with two new rows. ```sql INSERT OVERWRITE source_wiki_edit WITH LABEL insert_load_wikipedia_ow VALUES ("2015-09-12 00:00:00","#cn.wikipedia","GELongstreet",0,0,0,0,0,36,36,0), ("2015-09-12 00:00:00","#fr.wikipedia","PereBot",0,1,0,1,0,17,17,0); ``` #### Overwrite data via INSERT OVERWRITE SELECT[​](#overwrite-data-via-insert-overwrite-select "Direct link to Overwrite data via INSERT OVERWRITE SELECT") You can overwrite a table with the result of a query on a data source table via INSERT OVERWRITE SELECT command. INSERT OVERWRITE SELECT statement performs ETL operations on the data from one or more internal or external tables, and overwrites an internal table with the data For detailed instructions and parameter references, see [SQL Reference - INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). > **NOTE** > > Loading data from an external table is identical to loading data from an internal table. For simplicity, we only demonstrate how to overwrite the target table with the data from an internal table in the following examples. Query the source table and the target table to make sure that they hold different rows of data. ```plain MySQL > SELECT * FROM source_wiki_edit; +---------------------+---------------+--------------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | event_time | channel | user | is_anonymous | is_minor | is_new | is_robot | is_unpatrolled | delta | added | deleted | +---------------------+---------------+--------------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | 2015-09-12 00:00:00 | #cn.wikipedia | GELongstreet | 0 | 0 | 0 | 0 | 0 | 36 | 36 | 0 | | 2015-09-12 00:00:00 | #fr.wikipedia | PereBot | 0 | 1 | 0 | 1 | 0 | 17 | 17 | 0 | +---------------------+---------------+--------------+--------------+----------+--------+----------+----------------+-------+-------+---------+ 2 rows in set (0.02 sec) MySQL > SELECT * FROM insert_wiki_edit; +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | event_time | channel | user | is_anonymous | is_minor | is_new | is_robot | is_unpatrolled | delta | added | deleted | +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | 2015-09-12 00:00:00 | #en.wikipedia | AustinFF | 0 | 0 | 0 | 0 | 0 | 21 | 5 | 0 | | 2015-09-12 00:00:00 | #ca.wikipedia | helloSR | 0 | 1 | 0 | 1 | 0 | 3 | 23 | 0 | +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ 2 rows in set (0.01 sec) ``` * The following example overwrites the table `insert_wiki_edit` with the data from the source table. ```sql INSERT OVERWRITE insert_wiki_edit WITH LABEL insert_load_wikipedia_ow_1 SELECT * FROM source_wiki_edit; ``` * The following example overwrites the `p06` and `p12` partitions of the table `insert_wiki_edit` with the data from the source table. ```sql INSERT OVERWRITE insert_wiki_edit PARTITION(p06, p12) WITH LABEL insert_load_wikipedia_ow_2 SELECT * FROM source_wiki_edit; ``` Query the target table to make sure there is data in them. ```plain MySQL > select * from insert_wiki_edit; +---------------------+---------------+--------------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | event_time | channel | user | is_anonymous | is_minor | is_new | is_robot | is_unpatrolled | delta | added | deleted | +---------------------+---------------+--------------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | 2015-09-12 00:00:00 | #fr.wikipedia | PereBot | 0 | 1 | 0 | 1 | 0 | 17 | 17 | 0 | | 2015-09-12 00:00:00 | #cn.wikipedia | GELongstreet | 0 | 0 | 0 | 0 | 0 | 36 | 36 | 0 | +---------------------+---------------+--------------+--------------+----------+--------+----------+----------------+-------+-------+---------+ 2 rows in set (0.01 sec) ``` If you truncate the `p06` and `p12` partitions, the data will not be returned in a query. ```plain MySQL > TRUNCATE TABLE insert_wiki_edit PARTITION(p06, p12); Query OK, 0 rows affected (0.01 sec) MySQL > select * from insert_wiki_edit; Empty set (0.00 sec) ``` note For tables that use the `PARTITION BY column` strategy, INSERT OVERWRITE supports creating new partitions in the destination table by specifying the value of the partition key. Existing partitions are overwritten as usual. The following example creates the partitioned table `activity`, and creates a new partition in the table while inserting data into it: ```sql CREATE TABLE activity ( id INT NOT NULL, dt VARCHAR(10) NOT NULL ) ENGINE=OLAP DUPLICATE KEY(`id`) PARTITION BY (`id`, `dt`) DISTRIBUTED BY HASH(`id`); INSERT OVERWRITE activity PARTITION(id='4', dt='2022-01-01') WITH LABEL insert_activity_auto_partition VALUES ('4', '2022-01-01'); ``` * The following example overwrites the target table `insert_wiki_edit` with the `event_time` and `channel` columns from the source table. The default value is assigned to the columns into which no data is overwritten. ```sql INSERT OVERWRITE insert_wiki_edit WITH LABEL insert_load_wikipedia_ow_3 ( event_time, channel ) SELECT event_time, channel FROM source_wiki_edit; ``` ##### Dynamic Overwrite[​](#dynamic-overwrite "Direct link to Dynamic Overwrite") From v3.4.0 onwards, StarRocks supports a new semantic - Dynamic Overwrite for INSERT OVERWRITE with partitioned tables. Currently, the default behavior of INSERT OVERWRITE is as follows: * When overwriting a partitioned table as a whole (that is, without specifying the PARTITION clause), new data records will replace the data in their corresponding partitions. If there are partitions that are not involved, they will be truncated while the others are overwritten. * When overwriting an empty partitioned table (that is, with no partitions in it) and specifying the PARTITION clause, the system returns an error `ERROR 1064 (HY000): Getting analyzing error. Detail message: Unknown partition 'xxx' in table 'yyy'`. * When overwriting a partitioned table and specifying a non-existent partition in the PARTITION clause, the system returns an error `ERROR 1064 (HY000): Getting analyzing error. Detail message: Unknown partition 'xxx' in table 'yyy'`. * When overwriting a partitioned table with data records that do not match any of the specified partitions in the PARTITION clause, the system either returns an error `ERROR 1064 (HY000): Insert has filtered data in strict mode` (if the strict mode is enabled) or filters the unqualified data records (if the strict mode is disabled). The behavior of the new Dynamic Overwrite semantic is much different: When overwriting a partitioned table as a whole, new data records will replace the data in their corresponding partitions. If there are partitions that are not involved, they will be left alone, instead of being truncated or deleted. And if there are new data records correspond to a non-existent partition, the system will create the partition. The Dynamic Overwrite semantic is disabled by default. To enable it, you need to set the system variable `dynamic_overwrite` to `true`. Enable Dynamic Overwrite in the current session: ```sql SET dynamic_overwrite = true; ``` You can also set it in the hint of the INSERT OVERWRITE statement to allow it take effect for the statement only:. Example: ```sql INSERT /*+set_var(dynamic_overwrite = true)*/ OVERWRITE insert_wiki_edit SELECT * FROM source_wiki_edit; ``` #### Insert data into a table with generated columns[​](#insert-data-into-a-table-with-generated-columns "Direct link to Insert data into a table with generated columns") A generated column is a special column whose value is derived from a pre-defined expression or evaluation based on other columns. Generated columns are especially useful when your query requests involve evaluations of expensive expressions, for example, querying a certain field from a JSON value, or calculating ARRAY data. StarRocks evaluates the expression and stores the results in the generated columns while data is being loaded into the table, thereby avoiding the expression evaluation during queries and improving the query performance. You can load data into a table with generated columns using INSERT. The following example creates a table `insert_generated_columns` and inserts a row into it. The table contains two generated columns: `avg_array` and `get_string`. `avg_array` calculates the average value of ARRAY data in `data_array`, and `get_string` extracts the strings from the JSON path `a` in `data_json`. ```sql CREATE TABLE insert_generated_columns ( id INT(11) NOT NULL COMMENT "ID", data_array ARRAY NOT NULL COMMENT "ARRAY", data_json JSON NOT NULL COMMENT "JSON", avg_array DOUBLE NULL AS array_avg(data_array) COMMENT "Get the average of ARRAY", get_string VARCHAR(65533) NULL AS get_json_string(json_string(data_json), '$.a') COMMENT "Extract JSON string" ) ENGINE=OLAP PRIMARY KEY(id) DISTRIBUTED BY HASH(id); INSERT INTO insert_generated_columns VALUES (1, [1,2], parse_json('{"a" : 1, "b" : 2}')); ``` > **NOTE** > > Directly loading data into generated columns is not supported. You can query the table to check the data within it. ```plain mysql> SELECT * FROM insert_generated_columns; +------+------------+------------------+-----------+------------+ | id | data_array | data_json | avg_array | get_string | +------+------------+------------------+-----------+------------+ | 1 | [1,2] | {"a": 1, "b": 2} | 1.5 | 1 | +------+------------+------------------+-----------+------------+ 1 row in set (0.02 sec) ``` #### INSERT data with PROPERTIES[​](#insert-data-with-properties "Direct link to INSERT data with PROPERTIES") From v3.4.0 onwards, INSERT statements support configuring PROPERTIES, which can serve a wide variety of purposes. PROPERTIES overrides their corresponding variables. ##### Enable strict mode[​](#enable-strict-mode "Direct link to Enable strict mode") From v3.4.0 onwards, you can enable strict mode and set `max_filter_ratio` for INSERT from FILES(). Strict mode for INSERT from FILES() has the same behavior as that of other loading methods. If you want to load a dataset with some unqualified rows, you either filter these unqualified rows or load them and assign NULL values to the unqualified columns. You can achieve them by using the properties `strict_mode` and `max_filter_ratio`. * To filter the unqualified rows: set `strict_mode` to `true`, and `max_filter_ratio` to a desired value. * To load all unqualified rows with NULL values: set `strict_mode` to `false`. The following example inserts data rows from the Parquet file **parquet/insert\_wiki\_edit\_append.parquet** within the AWS S3 bucket `inserttest` into the table `insert_wiki_edit`, enables strict mode to filter the unqualified data records, and tolerates at most 10% of error data: ```sql INSERT INTO insert_wiki_edit PROPERTIES( "strict_mode" = "true", "max_filter_ratio" = "0.1" ) SELECT * FROM FILES( "path" = "s3://inserttest/parquet/insert_wiki_edit_append.parquet", "format" = "parquet", "aws.s3.access_key" = "XXXXXXXXXX", "aws.s3.secret_key" = "YYYYYYYYYY", "aws.s3.region" = "us-west-2" ); ``` note `strict_mode` and `max_filter_ratio` are supported only for INSERT from FILES(). INSERT from tables does not support these properties. ##### Set timeout duration[​](#set-timeout-duration "Direct link to Set timeout duration") From v3.4.0 onwards, you can set the timeout duration for INSERT statements with properties. The following example inserts the data from the source table `source_wiki_edit` to the target table `insert_wiki_edit` with the timeout duration set to `2` seconds. ```sql INSERT INTO insert_wiki_edit PROPERTIES( "timeout" = "2" ) SELECT * FROM source_wiki_edit; ``` note From v3.4.0 onwards, you can also set the INSERT timeout duration using the system variable `insert_timeout`, which applies to operations involving INSERT (for example, UPDATE, DELETE, CTAS, materialized view refresh, statistics collection, and PIPE). In versions earlier than v3.4.0, the corresponding variable is `query_timeout`. ##### Match column by name[​](#match-column-by-name "Direct link to Match column by name") By default, INSERT matches the columns in the source and the target tables by their positions, that is, the mapping of the columns in the statement. The following example explicitly matches each column in the source and target tables by their positions: ```sql INSERT INTO insert_wiki_edit ( event_time, channel, user ) SELECT event_time, channel, user FROM source_wiki_edit; ``` The column mapping will change if you changed the order of `channel` and `user` in either the column list or the SELECT statement. ```sql INSERT INTO insert_wiki_edit ( event_time, channel, user ) SELECT event_time, user, channel FROM source_wiki_edit; ``` Here, the ingested data are probably not what you want, because `channel` in the target table `insert_wiki_edit` will be filled with data from `user` in the source table `source_wiki_edit`. By adding `BY NAME` clause in the INSERT statement, the system will detect the column names in the source and the target tables, and match the columns with the same name. note * You cannot specify the column list if `BY NAME` is specified. * If `BY NAME` is not specified, the system matches the columns by the position of the columns in the column list and the SELECT statement. The following example matches each column in the source and target tables by their names: ```sql INSERT INTO insert_wiki_edit BY NAME SELECT event_time, user, channel FROM source_wiki_edit; ``` In this case, changing the order of `channel` and `user` will not change the column mapping. #### Load data asynchronously using INSERT[​](#load-data-asynchronously-using-insert "Direct link to Load data asynchronously using INSERT") Loading data with INSERT submits a synchronous transaction, which may fail because of session interruption or timeout. You can submit an asynchronous INSERT transaction using [SUBMIT TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md). This feature is supported since StarRocks v2.5. * The following example asynchronously inserts the data from the source table to the target table `insert_wiki_edit`. ```sql SUBMIT TASK AS INSERT INTO insert_wiki_edit SELECT * FROM source_wiki_edit; ``` * The following example asynchronously overwrites the table `insert_wiki_edit` with the data from the source table. ```sql SUBMIT TASK AS INSERT OVERWRITE insert_wiki_edit SELECT * FROM source_wiki_edit; ``` * The following example asynchronously overwrites the table `insert_wiki_edit` with the data from the source table, and extends the query timeout to `100000` seconds using hint. ```sql SUBMIT /*+set_var(insert_timeout=100000)*/ TASK AS INSERT OVERWRITE insert_wiki_edit SELECT * FROM source_wiki_edit; ``` * The following example asynchronously overwrites the table `insert_wiki_edit` with the data from the source table, and specifies the task name as `async`. ```sql SUBMIT TASK async AS INSERT OVERWRITE insert_wiki_edit SELECT * FROM source_wiki_edit; ``` You can check the status of an asynchronous INSERT task by querying the metadata view `task_runs` in Information Schema. The following example checks the status of the INSERT task `async`. ```sql SELECT * FROM information_schema.task_runs WHERE task_name = 'async'; ``` #### Check the INSERT job status[​](#check-the-insert-job-status "Direct link to Check the INSERT job status") ##### Check via the result[​](#check-via-the-result "Direct link to Check via the result") A synchronous INSERT transaction returns different status in accordance with the result of the transaction. * **Transaction succeeds** StarRocks returns the following if the transaction succeeds: ```plain Query OK, 2 rows affected (0.05 sec) {'label':'insert_load_wikipedia', 'status':'VISIBLE', 'txnId':'1006'} ``` * **Transaction fails** If all rows of data fail to be loaded into the target table, the INSERT transaction fails. StarRocks returns the following if the transaction fails: ```plain ERROR 1064 (HY000): Insert has filtered data in strict mode, tracking_url=http://x.x.x.x:yyyy/api/_load_error_log?file=error_log_9f0a4fd0b64e11ec_906bbede076e9d08 ``` You can locate the problem by checking the log with `tracking_url`. ##### Check via Information Schema[​](#check-via-information-schema "Direct link to Check via Information Schema") You can use the [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) statement to query the results of one or more load jobs from the `loads` table in the `information_schema` database. This feature is supported from v3.1 onwards. Example 1: Query the results of load jobs executed on the `load_test` database, sort the results by creation time (`CREATE_TIME`) in descending order, and only return the top result. ```sql SELECT * FROM information_schema.loads WHERE database_name = 'load_test' ORDER BY create_time DESC LIMIT 1\G ``` Example 2: Query the result of the load job (whose label is `insert_load_wikipedia`) executed on the `load_test` database: ```sql SELECT * FROM information_schema.loads WHERE database_name = 'load_test' and label = 'insert_load_wikipedia'\G ``` The return is as follows: ```plain *************************** 1. row *************************** JOB_ID: 21319 LABEL: insert_load_wikipedia DATABASE_NAME: load_test STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: INSERT PRIORITY: NORMAL SCAN_ROWS: 0 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 2 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):300; max_filter_ratio:0.0 CREATE_TIME: 2023-08-09 10:42:23 ETL_START_TIME: 2023-08-09 10:42:23 ETL_FINISH_TIME: 2023-08-09 10:42:23 LOAD_START_TIME: 2023-08-09 10:42:23 LOAD_FINISH_TIME: 2023-08-09 10:42:24 JOB_DETAILS: {"All backends":{"5ebf11b5-365e-11ee-9e4a-7a563fb695da":[10006]},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":175,"InternalTableLoadRows":2,"ScanBytes":0,"ScanRows":0,"TaskNumber":1,"Unfinished backends":{"5ebf11b5-365e-11ee-9e4a-7a563fb695da":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL 1 row in set (0.01 sec) ``` For information about the fields in the return results, see [Information Schema > loads](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). ##### Check via curl command[​](#check-via-curl-command "Direct link to Check via curl command") You can check the INSERT transaction status by using curl command. Launch a terminal, and execute the following command: ```bash curl --location-trusted -u : \ http://:/api//_load_info?label= ``` The following example checks the status of the transaction with label `insert_load_wikipedia`. ```bash curl --location-trusted -u : \ http://x.x.x.x:8030/api/load_test/_load_info?label=insert_load_wikipedia ``` > **NOTE** > > If you use an account for which no password is set, you need to input only `:`. The return is as follows: ```plain { "jobInfo":{ "dbName":"load_test", "tblNames":[ "source_wiki_edit" ], "label":"insert_load_wikipedia", "state":"FINISHED", "failMsg":"", "trackingUrl":"" }, "status":"OK", "msg":"Success" } ``` #### Configuration[​](#configuration "Direct link to Configuration") You can set the following configuration items for INSERT transaction: * **FE configuration** | FE configuration | Description | | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | insert\_load\_default\_timeout\_second | Default timeout for INSERT transaction. Unit: second. If the current INSERT transaction is not completed within the time set by this parameter, it will be canceled by the system and the status will be CANCELLED. As for current version of StarRocks, you can only specify a uniform timeout for all INSERT transactions using this parameter, and you cannot set a different timeout for a specific INSERT transaction. The default is 3600 seconds (1 hour). If the INSERT transaction cannot be completed within the specified time, you can extend the timeout by adjusting this parameter. | * **Session variables** | Session variable | Description | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_insert\_strict | Switch value to control if the INSERT transaction is tolerant of invalid data rows. When it is set to `true`, the transaction fails if any of the data rows is invalid. When it is set to `false`, the transaction succeeds when at least one row of data has been loaded correctly, and the label will be returned. The default is `true`. You can set this variable with `SET enable_insert_strict = {true or false};` command. | | insert\_timeout | Timeout for the INSERT statement. Unit: second. You can set this variable with the `SET insert_timeout = xxx;` command. | --- ### Introduction You can import semi-structured data (for example, JSON) by using stream load or routine load. #### Use Scenarios[​](#use-scenarios "Direct link to Use Scenarios") * Stream Load: For JSON data stored in text files, use stream load to import. * Routine Load: For JSON data in Kafka, use routine load to import. ##### Stream Load Import[​](#stream-load-import "Direct link to Stream Load Import") Sample data: ```json { "id": 123, "city" : "beijing"}, { "id": 456, "city" : "shanghai"}, ... ``` Example: ```shell curl -v --location-trusted -u : \ -H "format: json" -H "jsonpaths: [\"$.id\", \"$.city\"]" \ -T example.json \ http://FE_HOST:HTTP_PORT/api/DATABASE/TABLE/_stream_load ``` The `format: json` parameter allows you to execute the format of the imported data. `jsonpaths` is used to execute the corresponding data import path. Related parameters: * `jsonpaths`: Select the JSON path for each column * `json_root`: Select the column where the JSON starts to be parsed * `strip_outer_array`: Crop the outermost array field * `strict_mode`: Strictly filter for column type conversion during import When the JSON data schema and StarRocks data schema are not exactly the same, modify the `Jsonpath`. Sample data: ```json {"k1": 1, "k2": 2} ``` Import example: ```bash curl -v --location-trusted -u : \ -H "format: json" -H "jsonpaths: [\"$.k2\", \"$.k1\"]" \ -H "columns: k2, tmp_k1, k1 = tmp_k1 * 100" \ -T example.json \ http://127.0.0.1:8030/api/db1/tbl1/_stream_load ``` The ETL operation of multiplying k1 by 100 is performed during the import, and the column is matched with the original data by `Jsonpath`. The import results are as follows: ```plain +------+------+ | k1 | k2 | +------+------+ | 100 | 2 | +------+------+ ``` For missing columns, if the column definition is nullable, then `NULL` will be added, or the default value can be added by `ifnull`. Sample data: ```json [ {"k1": 1, "k2": "a"}, {"k1": 2}, {"k1": 3, "k2": "c"}, ] ``` Import Example-1: ```shell curl -v --location-trusted -u : \ -H "format: json" -H "strip_outer_array: true" \ -T example.json \ http://127.0.0.1:8030/api/db1/tbl1/_stream_load ``` The import results are as follows: ```plain +------+------+ | k1 | k2 | +------+------+ | 1 | a | +------+------+ | 2 | NULL | +------+------+ | 3 | c | +------+------+ ``` Import Example-2: ```shell curl -v --location-trusted -u : \ -H "format: json" -H "strip_outer_array: true" \ -H "jsonpaths: [\"$.k1\", \"$.k2\"]" \ -H "columns: k1, tmp_k2, k2 = ifnull(tmp_k2, 'x')" \ -T example.json \ http://127.0.0.1:8030/api/db1/tbl1/_stream_load ``` The import results are as follows: ```plain +------+------+ | k1 | k2 | +------+------+ | 1 | a | +------+------+ | 2 | x | +------+------+ | 3 | c | +------+------+ ``` ##### Routine Load Import[​](#routine-load-import "Direct link to Routine Load Import") Similar to stream load, the message content of Kafka data sources is treated as a complete JSON data. 1. If a message contains multiple rows of data in array format, all rows will be imported and Kafka's offset will only be incremented by 1. 2. If a JSON in Array format represents multiple rows of data, but the parsing of the JSON fails due to a JSON format error, the error row will only be incremented by 1 (given that the parsing fails, StarRocks cannot actually determine how many rows of data it contains, and can only record the error data as one row). ##### Use Canal to import StarRocks from MySQL with incremental sync binlogs[​](#use-canal-to-import-starrocks-from-mysql-with-incremental-sync-binlogs "Direct link to Use Canal to import StarRocks from MySQL with incremental sync binlogs") [Canal](https://github.com/alibaba/canal) is an open-source MySQL binlog synchronization tool from Alibaba, through which we can synchronize MySQL data to Kafka. The data is generated in JSON format in Kafka. Here is a demonstration of how to use routine load to synchronize data in Kafka for incremental data synchronization with MySQL. * In MySQL we have a data table with the following table creation statement. ```sql CREATE TABLE `query_record` ( `query_id` varchar(64) NOT NULL, `conn_id` int(11) DEFAULT NULL, `fe_host` varchar(32) DEFAULT NULL, `user` varchar(32) DEFAULT NULL, `start_time` datetime NOT NULL, `end_time` datetime DEFAULT NULL, `time_used` double DEFAULT NULL, `state` varchar(16) NOT NULL, `error_message` text, `sql` text NOT NULL, `database` varchar(128) NOT NULL, `profile` longtext, `plan` longtext, PRIMARY KEY (`query_id`), KEY `idx_start_time` (`start_time`) USING BTREE ) ENGINE=InnoDB DEFAULT CHARSET=utf8 ``` * Prerequisite: Make sure MySQL has binlog enabled and the format is ROW. ```bash [mysqld] log-bin=mysql-bin # Enable binlog binlog-format=ROW # Select ROW mode server_id=1 # MySQL replication need to be defined, and do not duplicate canal's slaveId ``` * Create an account and grant privileges to the secondary MySQL server: ```sql CREATE USER canal IDENTIFIED BY 'canal'; GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%'; -- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%'; FLUSH PRIVILEGES; ``` * Then download and install Canal. ```bash wget https://github.com/alibaba/canal/releases/download/canal-1.0.17/canal.deployer-1.0.17.tar.gz mkdir /tmp/canal tar zxvf canal.deployer-$version.tar.gz -C /tmp/canal ``` * Modify the configuration (MySQL related). `$ vi conf/example/instance.properties` ```bash ## mysql serverId canal.instance.mysql.slaveId = 1234 #position info, need to change to your own database information canal.instance.master.address = 127.0.0.1:3306 canal.instance.master.journal.name = canal.instance.master.position = canal.instance.master.timestamp = #canal.instance.standby.address = #canal.instance.standby.journal.name = #canal.instance.standby.position = #canal.instance.standby.timestamp = #username/password, need to change to your own database information canal.instance.dbUsername = canal canal.instance.dbPassword = canal canal.instance.defaultDatabaseName = canal.instance.connectionCharset = UTF-8 #table regex canal.instance.filter.regex = .\*\\\\..\* # Select the name of the table to be synchronized and the partition name of the kafka target. canal.mq.dynamicTopic=databasename.query_record canal.mq.partitionHash= databasename.query_record:query_id ``` * Modify the configuration (Kafka related). `$ vi /usr/local/canal/conf/canal.properties` ```bash # Available options: tcp(by default), kafka, RocketMQ canal.serverMode = kafka # ... # kafka/rocketmq Cluster Configuration: 192.168.1.117:9092,192.168.1.118:9092,192.168.1.119:9092 canal.mq.servers = 127.0.0.1:6667 canal.mq.retries = 0 # This value can be increased in flagMessage mode, but do not exceed the maximum size of the MQ message. canal.mq.batchSize = 16384 canal.mq.maxRequestSize = 1048576 # In flatMessage mode, please change this value to a larger value, 50-200 is recommended. canal.mq.lingerMs = 1 canal.mq.bufferMemory = 33554432 # Canal's batch size with a default value of 50K. Please do not exceed 1M due to Kafka's maximum message size limit (under 900K) canal.mq.canalBatchSize = 50 # Timeout of `Canal get`, in milliseconds. Empty indicates unlimited timeout. canal.mq.canalGetTimeout = 100 # Whether the object is in flat json format canal.mq.flatMessage = false canal.mq.compressionType = none canal.mq.acks = all # Whether Kafka message delivery uses transactions canal.mq.transaction = false ``` * Initiation `bin/startup.sh` The corresponding synchronization log is shown in `logs/example/example.log` and in Kafka, with the following format: ```json { "data": [{ "query_id": "3c7ebee321e94773-b4d79cc3f08ca2ac", "conn_id": "34434", "fe_host": "172.26.34.139", "user": "zhaoheng", "start_time": "2020-10-19 20:40:10.578", "end_time": "2020-10-19 20:40:10", "time_used": "1.0", "state": "FINISHED", "error_message": "", "sql": "COMMIT", "database": "", "profile": "", "plan": "" }, { "query_id": "7ff2df7551d64f8e-804004341bfa63ad", "conn_id": "34432", "fe_host": "172.26.34.139", "user": "zhaoheng", "start_time": "2020-10-19 20:40:10.566", "end_time": "2020-10-19 20:40:10", "time_used": "0.0", "state": "FINISHED", "error_message": "", "sql": "COMMIT", "database": "", "profile": "", "plan": "" }, { "query_id": "3a4b35d1c1914748-be385f5067759134", "conn_id": "34440", "fe_host": "172.26.34.139", "user": "zhaoheng", "start_time": "2020-10-19 20:40:10.601", "end_time": "1970-01-01 08:00:00", "time_used": "-1.0", "state": "RUNNING", "error_message": "", "sql": " SELECT SUM(length(lo_custkey)), SUM(length(c_custkey)) FROM lineorder_str INNER JOIN customer_str ON lo_custkey=c_custkey;", "database": "ssb", "profile": "", "plan": "" }], "database": "center_service_lihailei", "es": 1603111211000, "id": 122, "isDdl": false, "mysqlType": { "query_id": "varchar(64)", "conn_id": "int(11)", "fe_host": "varchar(32)", "user": "varchar(32)", "start_time": "datetime(3)", "end_time": "datetime", "time_used": "double", "state": "varchar(16)", "error_message": "text", "sql": "text", "database": "varchar(128)", "profile": "longtext", "plan": "longtext" }, "old": null, "pkNames": ["query_id"], "sql": "", "sqlType": { "query_id": 12, "conn_id": 4, "fe_host": 12, "user": 12, "start_time": 93, "end_time": 93, "time_used": 8, "state": 12, "error_message": 2005, "sql": 2005, "database": 12, "profile": 2005, "plan": 2005 }, "table": "query_record", "ts": 1603111212015, "type": "INSERT" } ``` Add `json_root` and `strip_outer_array = true` to import data from `data`. ```sql create routine load manual.query_job on query_record columns (query_id,conn_id,fe_host,user,start_time,end_time,time_used,state,error_message,`sql`,`database`,profile,plan) PROPERTIES ( "format"="json", "json_root"="$.data", "desired_concurrent_number"="1", "strip_outer_array" ="true", "max_error_number"="1000" ) FROM KAFKA ( "kafka_broker_list"= "172.26.92.141:9092", "kafka_topic" = "databasename.query_record" ); ``` This completes the near real-time synchronization of data from MySQL to StarRocks. View status and error messages of the import job by `show routine load`. --- ### Load data from Apache Kafka #### [📄️ Load data using Kafka connector](https://docs.starrocks.io/docs/loading/kafka/Kafka-connector-starrocks.md) [How to use the StarRocks Kafka connector (Kafka Connect sink) to continuously load data from Apache Kafka topics into StarRocks with at-least-once semantics.](https://docs.starrocks.io/docs/loading/kafka/Kafka-connector-starrocks.md) #### [📄️ Load data using Routine Load](https://docs.starrocks.io/docs/loading/kafka/RoutineLoad.md) [How to create and manage Routine Load jobs in StarRocks to continuously stream data from Apache Kafka topics into StarRocks tables.](https://docs.starrocks.io/docs/loading/kafka/RoutineLoad.md) --- ### Load data using Kafka connector StarRocks provides a self-developed connector named Apache Kafka® connector (StarRocks Connector for Apache Kafka®, Kafka connector for short), as a sink connector, that continuously consumes messages from Kafka and loads them into StarRocks. The Kafka connector guarantees at-least-once semantics. The Kafka connector can seamlessly integrate with Kafka Connect, which allows StarRocks better integrated with the Kafka ecosystem. It is a wise choice if you want to load real-time data into StarRocks. Compared with Routine Load, it is recommended to use the Kafka connector in the following scenarios: * Compared with Routine Load which only supports loading data in CSV, JSON, and Avro formats, Kafka connector can load data in more formats, such as Protobuf. As long as data can be converted into JSON and CSV formats using Kafka Connect's converters, data can be loaded into StarRocks via the Kafka connector. * Customize data transformation, such as Debezium-formatted CDC data. * Load data from multiple Kafka topics. * Load data from Confluent Cloud. * Need finer control over load batch sizes, parallelism, and other parameters to achieve a balance between load speed and resource utilization. #### Preparations[​](#preparations "Direct link to Preparations") ##### Version requirements[​](#version-requirements "Direct link to Version requirements") | Connector | Kafka | StarRocks | Java | | --------- | --------- | ------------- | ---- | | 1.0.6 | 3.4+/4.0+ | 2.5 and later | 8 | | 1.0.5 | 3.4 | 2.5 and later | 8 | | 1.0.4 | 3.4 | 2.5 and later | 8 | | 1.0.3 | 3.4 | 2.5 and later | 8 | ##### Set up Kafka environment[​](#set-up-kafka-environment "Direct link to Set up Kafka environment") Both self-managed Apache Kafka clusters and Confluent Cloud are supported. * For a self-managed Apache Kafka cluster, you can refer to [Apache Kafka quickstart](https://kafka.apache.org/quickstart) to quickly deploy a Kafka cluster. Kafka Connect is already integrated into Kafka. * For Confluent Cloud, make sure that you have a Confluent account and have created a cluster. ##### Download Kafka connector[​](#download-kafka-connector "Direct link to Download Kafka connector") Submit the Kafka connector into Kafka Connect: * Self-managed Kafka cluster: Download [starrocks-connector-for-kafka-x.y.z-with-dependencies.jar](https://github.com/StarRocks/starrocks-connector-for-kafka/releases). * Confluent Cloud: Currently, the Kafka connector is not uploaded to Confluent Hub. You need to download [starrocks-connector-for-kafka-x.y.z-with-dependencies.jar](https://github.com/StarRocks/starrocks-connector-for-kafka/releases), package it into a ZIP file and upload the ZIP file to Confluent Cloud. ##### Network configuration[​](#network-configuration "Direct link to Network configuration") Ensure that the machine where Kafka is located can access the FE nodes of the StarRocks cluster via the [`http_port`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#http_port) (default: `8030`) and [`query_port`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#query_port) (default: `9030`), and the BE nodes via the [`be_http_port`](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#be_http_port) (default: `8040`). #### Usage[​](#usage "Direct link to Usage") This section uses a self-managed Kafka cluster as an example to explain how to configure the Kafka connector and the Kafka Connect, and then run the Kafka Connect to load data into StarRocks. ##### Prepare a dataset[​](#prepare-a-dataset "Direct link to Prepare a dataset") Suppose that JSON-format data exists in the topic `test` in a Kafka cluster. ```json {"id":1,"city":"New York"} {"id":2,"city":"Los Angeles"} {"id":3,"city":"Chicago"} ``` ##### Create a table[​](#create-a-table "Direct link to Create a table") Create the table `test_tbl` in the database `example_db` in the StarRocks cluster according to the keys of the JSON-format data. ```sql CREATE DATABASE example_db; USE example_db; CREATE TABLE test_tbl (id INT, city STRING); ``` ##### Configure Kafka connector and Kafka Connect, and then run Kafka Connect to load data[​](#configure-kafka-connector-and-kafka-connect-and-then-run-kafka-connect-to-load-data "Direct link to Configure Kafka connector and Kafka Connect, and then run Kafka Connect to load data") ###### Run Kafka Connect in standalone mode[​](#run-kafka-connect-in-standalone-mode "Direct link to Run Kafka Connect in standalone mode") 1. Configure the Kafka connector. In the **config** directory under the Kafka installation directory, create the configuration file **connect-StarRocks-sink.properties** for the Kafka connector, and configure the following parameters. For more parameters and descriptions, see [Parameters](#Parameters). info * In this example, the Kafka connector provided by StarRocks is a sink connector that can continuously consume data from Kafka and load data into StarRocks. * If the source data is CDC data, such as data in Debezium format, and the StarRocks table is a Primary Key table, you also need to [configure `transform`](#load-debezium-formatted-cdc-data) in the configuration file **connect-StarRocks-sink.properties** for the Kafka connector provided by StarRocks, to synchronize the source data changes to the Primary Key table. ```yaml name=starrocks-kafka-connector connector.class=com.starrocks.connector.kafka.StarRocksSinkConnector topics=test key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=false # The HTTP URL of the FE in your StarRocks cluster. The default port is 8030. starrocks.http.url=192.168.xxx.xxx:8030 # If the Kafka topic name is different from the StarRocks table name, you need to configure the mapping relationship between them. starrocks.topic2table.map=test:test_tbl # Enter the StarRocks username. starrocks.username=user1 # Enter the StarRocks password. starrocks.password=123456 starrocks.database.name=example_db sink.properties.strip_outer_array=true ``` 2. Configure and run the Kafka Connect. 1. Configure the Kafka Connect. In the configuration file **config/connect-standalone.properties** in the **config** directory, configure the following parameters. For more parameters and descriptions, see [Running Kafka Connect](https://kafka.apache.org/documentation.html#connect_running). ```yaml # The addresses of Kafka brokers. Multiple addresses of Kafka brokers need to be separated by commas (,). # Note that this example uses PLAINTEXT as the security protocol to access the Kafka cluster. If you are using other security protocol to access the Kafka cluster, you need to configure the relevant information in this file. bootstrap.servers=:9092 offset.storage.file.filename=/tmp/connect.offsets offset.flush.interval.ms=10000 key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=false # The absolute path of starrocks-connector-for-kafka-x.y.z-with-dependencies.jar. plugin.path=/home/kafka-connect/starrocks-kafka-connector ``` 2. Run the Kafka Connect. ```bash CLASSPATH=/home/kafka-connect/starrocks-kafka-connector/* bin/connect-standalone.sh config/connect-standalone.properties config/connect-starrocks-sink.properties ``` ###### Run Kafka Connect in distributed mode[​](#run-kafka-connect-in-distributed-mode "Direct link to Run Kafka Connect in distributed mode") 1. Configure and run the Kafka Connect. 1. Configure the Kafka Connect. In the configuration file `config/connect-distributed.properties` in the **config** directory, configure the following parameters. For more parameters and descriptions, refer to [Running Kafka Connect](https://kafka.apache.org/documentation.html#connect_running). ```yaml # The addresses of Kafka brokers. Multiple addresses of Kafka brokers need to be separated by commas (,). # Note that this example uses PLAINTEXT as the security protocol to access the Kafka cluster. If you are using other security protocol to access the Kafka cluster, you need to configure the relevant information in this file. bootstrap.servers=:9092 offset.storage.file.filename=/tmp/connect.offsets offset.flush.interval.ms=10000 key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=false # The absolute path of starrocks-connector-for-kafka-x.y.z-with-dependencies.jar. plugin.path=/home/kafka-connect/starrocks-kafka-connector ``` 2. Run the Kafka Connect. ```bash CLASSPATH=/home/kafka-connect/starrocks-kafka-connector/* bin/connect-distributed.sh config/connect-distributed.properties ``` 2. Configure and create the Kafka connector. Note that in distributed mode, you need to configure and create the Kafka connector through the REST API. For parameters and descriptions, see [Parameters](#Parameters). info * In this example, the Kafka connector provided by StarRocks is a sink connector that can continuously consume data from Kafka and load data into StarRocks. * If the source data is CDC data, such as data in Debezium format, and the StarRocks table is a Primary Key table, you also need to [configure `transform`](#load-debezium-formatted-cdc-data) in the configuration file **connect-StarRocks-sink.properties** for the Kafka connector provided by StarRocks, to synchronize the source data changes to the Primary Key table. ```shell curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ "name":"starrocks-kafka-connector", "config":{ "connector.class":"com.starrocks.connector.kafka.StarRocksSinkConnector", "topics":"test", "key.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter":"org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable":"true", "value.converter.schemas.enable":"false", "starrocks.http.url":"192.168.xxx.xxx:8030", "starrocks.topic2table.map":"test:test_tbl", "starrocks.username":"user1", "starrocks.password":"123456", "starrocks.database.name":"example_db", "sink.properties.strip_outer_array":"true" } }' ``` ###### Query StarRocks table[​](#query-starrocks-table "Direct link to Query StarRocks table") Query the target StarRocks table `test_tbl`. ```mysql MySQL [example_db]> select * from test_tbl; +------+-------------+ | id | city | +------+-------------+ | 1 | New York | | 2 | Los Angeles | | 3 | Chicago | +------+-------------+ 3 rows in set (0.01 sec) ``` The data is successfully loaded when the above result is returned. #### Parameters[​](#parameters "Direct link to Parameters") ##### name[​](#name "Direct link to name") **Required**: YES
**Default value**:
**Description**: Name for this Kafka connector. It must be globally unique among all Kafka connectors within this Kafka Connect cluster. For example, starrocks-kafka-connector. ##### connector.class[​](#connectorclass "Direct link to connector.class") **Required**: YES
**Default value**:
**Description**: Class used by this Kafka connector's sink. Set the value to `com.starrocks.connector.kafka.StarRocksSinkConnector`. ##### topics[​](#topics "Direct link to topics") **Required**:
**Default value**:
**Description**: One or more topics to subscribe to, where each topic corresponds to a StarRocks table. By default, StarRocks assumes that the topic name matches the name of the StarRocks table. So StarRocks determines the target StarRocks table by using the topic name. Please choose either to fill in `topics` or `topics.regex` (below), but not both. However, if the StarRocks table name is not the same as the topic name, then use the optional `starrocks.topic2table.map` parameter (below) to specify the mapping from topic name to table name. ##### topics.regex[​](#topicsregex "Direct link to topics.regex") **Required**:
**Default value**: **Description**: Regular expression to match the one or more topics to subscribe to. For more description, see `topics`. Please choose either to fill in `topics.regex` or `topics` (above), but not both.
##### starrocks.topic2table.map[​](#starrockstopic2tablemap "Direct link to starrocks.topic2table.map") **Required**: NO
**Default value**:
**Description**: The mapping of the StarRocks table name and the topic name when the topic name is different from the StarRocks table name. The format is `:,:,...`. ##### starrocks.http.url[​](#starrockshttpurl "Direct link to starrocks.http.url") **Required**: YES
**Default value**:
**Description**: The HTTP URL of the FE in your StarRocks cluster. The format is `:,:,...`. Multiple addresses are separated by commas (,). For example, `192.168.xxx.xxx:8030,192.168.xxx.xxx:8030`. ##### starrocks.database.name[​](#starrocksdatabasename "Direct link to starrocks.database.name") **Required**: YES
**Default value**:
**Description**: The name of StarRocks database. ##### starrocks.username[​](#starrocksusername "Direct link to starrocks.username") **Required**: YES
**Default value**:
**Description**: The username of your StarRocks cluster account. The user needs the [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) privilege on the StarRocks table. ##### starrocks.password[​](#starrockspassword "Direct link to starrocks.password") **Required**: YES
**Default value**:
**Description**: The password of your StarRocks cluster account. ##### key.converter[​](#keyconverter "Direct link to key.converter") **Required**: NO
**Default value**: Key converter used by Kafka Connect cluster
**Description**: This parameter specifies the key converter for the sink connector (Kafka-connector-starrocks), which is used to deserialize the keys of Kafka data. The default key converter is the one used by Kafka Connect cluster. ##### value.converter[​](#valueconverter "Direct link to value.converter") **Required**: NO
**Default value**: Value converter used by Kafka Connect cluster
**Description**: This parameter specifies the value converter for the sink connector (Kafka-connector-starrocks), which is used to deserialize the values of Kafka data. The default value converter is the one used by Kafka Connect cluster. ##### key.converter.schema.registry.url[​](#keyconverterschemaregistryurl "Direct link to key.converter.schema.registry.url") **Required**: NO
**Default value**:
**Description**: Schema registry URL for the key converter. ##### value.converter.schema.registry.url[​](#valueconverterschemaregistryurl "Direct link to value.converter.schema.registry.url") **Required**: NO
**Default value**:
**Description**: Schema registry URL for the value converter. ##### tasks.max[​](#tasksmax "Direct link to tasks.max") **Required**: NO
**Default value**: 1
**Description**: The upper limit for the number of task threads that the Kafka connector can create, which is usually the same as the number of CPU cores on the worker nodes in the Kafka Connect cluster. You can tune this parameter to control load performance. ##### bufferflush.maxbytes[​](#bufferflushmaxbytes "Direct link to bufferflush.maxbytes") **Required**: NO
**Default value**: 94371840(90M)
**Description**: The maximum size of data that can be accumulated in memory before being sent to StarRocks at a time. The maximum value ranges from 64 MB to 10 GB. Keep in mind that the Stream Load SDK buffer may create multiple Stream Load jobs to buffer data. Therefore, the threshold mentioned here refers to the total data size. ##### bufferflush.intervalms[​](#bufferflushintervalms "Direct link to bufferflush.intervalms") **Required**: NO
**Default value**: 1000
**Description**: Interval for sending a batch of data which controls the load latency. Range: \[1000, 3600000]. ##### connect.timeoutms[​](#connecttimeoutms "Direct link to connect.timeoutms") **Required**: NO
**Default value**: 1000
**Description**: Timeout for connecting to the HTTP URL. Range: \[100, 60000]. ##### sink.properties.\*[​](#sinkproperties "Direct link to sink.properties.*") **Required**:
**Default value**:
**Description**: Stream Load parameters o control load behavior. For example, the parameter `sink.properties.format` specifies the format used for Stream Load, such as CSV or JSON. For a list of supported parameters and their descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ##### sink.properties.format[​](#sinkpropertiesformat "Direct link to sink.properties.format") **Required**: NO
**Default value**: json
**Description**: The format used for Stream Load. The Kafka connector will transform each batch of data to the format before sending them to StarRocks. Valid values: `csv` and `json`. For more information, see [CSV parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#csv-parameters) and [JSON parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#json-parameters). ##### sink.properties.partial\_update[​](#sinkpropertiespartial_update "Direct link to sink.properties.partial_update") **Required**: NO
**Default value**: `FALSE`
**Description**: Whether to use partial updates. Valid values: `TRUE` and `FALSE`. Default value: `FALSE`, indicating to disable this feature. ##### sink.properties.partial\_update\_mode[​](#sinkpropertiespartial_update_mode "Direct link to sink.properties.partial_update_mode") **Required**: NO
**Default value**: `row`
**Description**: Specifies the mode for partial updates. Valid values: `row` and `column`. * The value `row` (default) means partial updates in row mode, which is more suitable for real-time updates with many columns and small batches. * The value `column` means partial updates in column mode, which is more suitable for batch updates with few columns and many rows. In such scenarios, enabling the column mode offers faster update speeds. For example, in a table with 100 columns, if only 10 columns (10% of the total) are updated for all rows, the update speed of the column mode is 10 times faster. #### Usage Notes[​](#usage-notes "Direct link to Usage Notes") ##### Flush Policy[​](#flush-policy "Direct link to Flush Policy") The Kafka connector will buffer the data in memory, and flush them in batch to StarRocks via Stream Load. The flush will be triggered when any of the following conditions are met: * The bytes of buffered rows reaches the limit `bufferflush.maxbytes`. * The elapsed time since the last flush reaches the limit `bufferflush.intervalms`. * The interval at which the connector tries committing offsets for tasks is reached. The interval is controlled by the Kafka Connect configuration [`offset.flush.interval.ms`](https://docs.confluent.io/platform/current/connect/references/allconfigs.html), and the default values is `60000`. For lower data latency, adjust these configurations in the Kafka connector settings. However, more frequent flushes will increase CPU and I/O usage. ##### Limits[​](#limits "Direct link to Limits") * It is not supported to flatten a single message from a Kafka topic into multiple data rows and load into StarRocks. * The sink of the Kafka connector provided by StarRocks guarantees at-least-once semantics. #### Best practices[​](#best-practices "Direct link to Best practices") ##### Load Debezium-formatted CDC data[​](#load-debezium-formatted-cdc-data "Direct link to Load Debezium-formatted CDC data") Debezium is a popular Change Data Capture (CDC) tool that supports monitoring data changes in various database systems and streaming these changes to Kafka. The following example demonstrates how to configure and use the Kafka connector to write PostgreSQL changes to a **Primary Key table** in StarRocks. ###### Step 1: Install and start Kafka[​](#step-1-install-and-start-kafka "Direct link to Step 1: Install and start Kafka") > **NOTE** > > You can skip this step if you have your own Kafka environment. 1. [Download](https://dlcdn.apache.org/kafka/) the latest Kafka release from the official site and extract the package. ```bash tar -xzf kafka_2.13-3.7.0.tgz cd kafka_2.13-3.7.0 ``` 2. Start the Kafka environment. Generate a Kafka cluster UUID. ```bash KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)" ``` Format the log directories. ```bash bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties ``` Start the Kafka server. ```bash bin/kafka-server-start.sh config/kraft/server.properties ``` ###### Step 2: Configure PostgreSQL[​](#step-2-configure-postgresql "Direct link to Step 2: Configure PostgreSQL") 1. Make sure the PostgreSQL user is granted `REPLICATION` privileges. 2. Adjust PostgreSQL configuration. Set `wal_level` to `logical` in **postgresql.conf**. ```properties wal_level = logical ``` Restart the PostgreSQL server to apply changes. ```bash pg_ctl restart ``` 3. Prepare the dataset. Create a table and insert test data. ```sql CREATE TABLE customers ( id int primary key , first_name varchar(65533) NULL, last_name varchar(65533) NULL , email varchar(65533) NULL ); INSERT INTO customers VALUES (1,'a','a','a@a.com'); ``` 4. Verify the CDC log messages in Kafka. ```json { "schema": { "type": "struct", "fields": [ { "type": "struct", "fields": [ { "type": "int32", "optional": false, "field": "id" }, { "type": "string", "optional": true, "field": "first_name" }, { "type": "string", "optional": true, "field": "last_name" }, { "type": "string", "optional": true, "field": "email" } ], "optional": true, "name": "test.public.customers.Value", "field": "before" }, { "type": "struct", "fields": [ { "type": "int32", "optional": false, "field": "id" }, { "type": "string", "optional": true, "field": "first_name" }, { "type": "string", "optional": true, "field": "last_name" }, { "type": "string", "optional": true, "field": "email" } ], "optional": true, "name": "test.public.customers.Value", "field": "after" }, { "type": "struct", "fields": [ { "type": "string", "optional": false, "field": "version" }, { "type": "string", "optional": false, "field": "connector" }, { "type": "string", "optional": false, "field": "name" }, { "type": "int64", "optional": false, "field": "ts_ms" }, { "type": "string", "optional": true, "name": "io.debezium.data.Enum", "version": 1, "parameters": { "allowed": "true,last,false,incremental" }, "default": "false", "field": "snapshot" }, { "type": "string", "optional": false, "field": "db" }, { "type": "string", "optional": true, "field": "sequence" }, { "type": "string", "optional": false, "field": "schema" }, { "type": "string", "optional": false, "field": "table" }, { "type": "int64", "optional": true, "field": "txId" }, { "type": "int64", "optional": true, "field": "lsn" }, { "type": "int64", "optional": true, "field": "xmin" } ], "optional": false, "name": "io.debezium.connector.postgresql.Source", "field": "source" }, { "type": "string", "optional": false, "field": "op" }, { "type": "int64", "optional": true, "field": "ts_ms" }, { "type": "struct", "fields": [ { "type": "string", "optional": false, "field": "id" }, { "type": "int64", "optional": false, "field": "total_order" }, { "type": "int64", "optional": false, "field": "data_collection_order" } ], "optional": true, "name": "event.block", "version": 1, "field": "transaction" } ], "optional": false, "name": "test.public.customers.Envelope", "version": 1 }, "payload": { "before": null, "after": { "id": 1, "first_name": "a", "last_name": "a", "email": "a@a.com" }, "source": { "version": "2.5.3.Final", "connector": "postgresql", "name": "test", "ts_ms": 1714283798721, "snapshot": "false", "db": "postgres", "sequence": "[\"22910216\",\"22910504\"]", "schema": "public", "table": "customers", "txId": 756, "lsn": 22910504, "xmin": null }, "op": "c", "ts_ms": 1714283798790, "transaction": null } } ``` ###### Step 3: Configure StarRocks[​](#step-3-configure-starrocks "Direct link to Step 3: Configure StarRocks") Create a Primary Key table in StarRocks with the same schema as the source table in PostgreSQL. ```sql CREATE TABLE `customers` ( `id` int(11) COMMENT "", `first_name` varchar(65533) NULL COMMENT "", `last_name` varchar(65533) NULL COMMENT "", `email` varchar(65533) NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY hash(id) buckets 1 PROPERTIES ( "bucket_size" = "4294967296", "in_memory" = "false", "enable_persistent_index" = "true", "replicated_storage" = "true", "fast_schema_evolution" = "true" ); ``` ###### Step 4: Install connector[​](#step-4-install-connector "Direct link to Step 4: Install connector") 1. Download the connectors and extract the packages in the **plugins** directory. ```bash mkdir plugins tar -zxvf debezium-debezium-connector-postgresql-2.5.3.zip -C plugins mv starrocks-connector-for-kafka-x.y.z-with-dependencies.jar plugins ``` This directory is the value of the configuration item `plugin.path` in **config/connect-standalone.properties**. ```properties plugin.path=/path/to/kafka_2.13-3.7.0/plugins ``` 2. Configure the PostgreSQL source connector in **pg-source.properties**. ```json { "name": "inventory-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "plugin.name": "pgoutput", "database.hostname": "localhost", "database.port": "5432", "database.user": "postgres", "database.password": "", "database.dbname" : "postgres", "topic.prefix": "test" } } ``` 3. Configure the StarRocks sink connector in **sr-sink.properties**. ```json { "name": "starrocks-kafka-connector", "config": { "connector.class": "com.starrocks.connector.kafka.StarRocksSinkConnector", "tasks.max": "1", "topics": "test.public.customers", "starrocks.http.url": "172.26.195.69:28030", "starrocks.database.name": "test", "starrocks.username": "root", "starrocks.password": "StarRocks@123", "sink.properties.strip_outer_array": "true", "connect.timeoutms": "3000", "starrocks.topic2table.map": "test.public.customers:customers", "transforms": "addfield,unwrap", "transforms.addfield.type": "com.starrocks.connector.kafka.transforms.AddOpFieldForDebeziumRecord", "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", "transforms.unwrap.drop.tombstones": "true", "transforms.unwrap.delete.handling.mode": "rewrite" } } ``` > **NOTE** > > * If the StarRocks table is not a Primary Key table, you do not need to specify the `addfield` transform. > * The unwrap transform is provided by Debezium and is used to unwrap Debezium's complex data structure based on the operation type. For more information, see [New Record State Extraction](https://debezium.io/documentation/reference/stable/transformations/event-flattening.html). 4. Configure Kafka Connect. Configure the following configuration items in the Kafka Connect configuration file **config/connect-standalone.properties**. ```properties # The addresses of Kafka brokers. Multiple addresses of Kafka brokers need to be separated by commas (,). # Note that this example uses PLAINTEXT as the security protocol to access the Kafka cluster. # If you use other security protocol to access the Kafka cluster, configure the relevant information in this part. bootstrap.servers=:9092 offset.storage.file.filename=/tmp/connect.offsets key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=false # The absolute path of starrocks-connector-for-kafka-x.y.z-with-dependencies.jar. plugin.path=/home/kafka-connect/starrocks-kafka-connector # Parameters that control the flush policy. For more information, see the Usage Note section. offset.flush.interval.ms=10000 bufferflush.maxbytes = xxx bufferflush.intervalms = xxx ``` For descriptions of more parameters, see [Running Kafka Connect](https://kafka.apache.org/documentation.html#connect_running). ###### Step 5: Start Kafka Connect in Standalone Mode[​](#step-5-start-kafka-connect-in-standalone-mode "Direct link to Step 5: Start Kafka Connect in Standalone Mode") Run Kafka Connect in standalone mode to initiate the connectors. ```bash bin/connect-standalone.sh config/connect-standalone.properties config/pg-source.properties config/sr-sink.properties ``` ###### Step 6: Verify data ingestion[​](#step-6-verify-data-ingestion "Direct link to Step 6: Verify data ingestion") Test the following operations and ensure the data is correctly ingested into StarRocks. ###### INSERT[​](#insert "Direct link to INSERT") * In PostgreSQL: ```plain postgres=# insert into customers values (2,'b','b','b@b.com'); INSERT 0 1 postgres=# select * from customers; id | first_name | last_name | email ----+------------+-----------+--------- 1 | a | a | a@a.com 2 | b | b | b@b.com (2 rows) ``` * In StarRocks: ```plain MySQL [test]> select * from customers; +------+------------+-----------+---------+ | id | first_name | last_name | email | +------+------------+-----------+---------+ | 1 | a | a | a@a.com | | 2 | b | b | b@b.com | +------+------------+-----------+---------+ 2 rows in set (0.01 sec) ``` ###### UPDATE[​](#update "Direct link to UPDATE") * In PostgreSQL: ```plain postgres=# update customers set email='c@c.com'; UPDATE 2 postgres=# select * from customers; id | first_name | last_name | email ----+------------+-----------+--------- 1 | a | a | c@c.com 2 | b | b | c@c.com (2 rows) ``` * In StarRocks: ```plain MySQL [test]> select * from customers; +------+------------+-----------+---------+ | id | first_name | last_name | email | +------+------------+-----------+---------+ | 1 | a | a | c@c.com | | 2 | b | b | c@c.com | +------+------------+-----------+---------+ 2 rows in set (0.00 sec) ``` ###### DELETE[​](#delete "Direct link to DELETE") * In PostgreSQL: ```plain postgres=# delete from customers where id=1; DELETE 1 postgres=# select * from customers; id | first_name | last_name | email ----+------------+-----------+--------- 2 | b | b | c@c.com (1 row) ``` * In StarRocks: ```plain MySQL [test]> select * from customers; +------+------------+-----------+---------+ | id | first_name | last_name | email | +------+------------+-----------+---------+ | 2 | b | b | c@c.com | +------+------------+-----------+---------+ 1 row in set (0.00 sec) ``` --- ### Load data using Routine Load tip Try Routine Load out in this [Quick Start](https://docs.starrocks.io/docs/quick_start/routine-load.md) This topic introduces how to create a Routine Load job to stream Kafka messages (events) into StarRocks, and familiarizes you with some basic concepts about Routine Load. To continuously load messages of a stream into StarRocks, you can store the message stream in a Kafka topic, and create a Routine Load job to consume the messages. The Routine Load job persists in StarRocks, generates a series of load tasks to consume the messages in all or part of the partitions in the topic, and loads the messages into StarRocks. A Routine Load job supports exactly-once delivery semantics to guarantee the data loaded into StarRocks is neither lost nor duplicated. Routine Load supports data transformation at data loading and supports data changes made by UPSERT and DELETE operations during data loading. For more information, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md) and [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md). You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. #### Supported data formats[​](#supported-data-formats "Direct link to Supported data formats") Routine Load now supports consuming CSV, JSON, and Avro (supported since v3.0.1) formatted data from a Kafka cluster. > **NOTE** > > For CSV data, take note of the following points: > > * You can use a UTF-8 string, such as a comma (,), tab, or pipe (|), whose length does not exceed 50 bytes as a text delimiter. > * Null values are denoted by using `\N`. For example, a data file consists of three columns, and a record from that data file holds data in the first and third columns but no data in the second column. In this situation, you need to use `\N` in the second column to denote a null value. This means the record must be compiled as `a,\N,b` instead of `a,,b`. `a,,b` denotes that the second column of the record holds an empty string. #### Basic concepts[​](#basic-concepts "Direct link to Basic concepts") ![routine load](/assets/images/4.5.2-1-a6d6d07699a1f32c703d4b80921e2e44.png) ##### Terminology[​](#terminology "Direct link to Terminology") * **Load job** A Routine Load job is a long-running job. As long as its status is RUNNING, a load job continuously generates one or multiple concurrent load tasks which consume the messages in a topic of a Kafka cluster and load the data into StarRocks. * **Load task** A load job is split into multiple load tasks by certain rules. A load task is the basic unit of data loading. As an individual event, a load task implements the load mechanism based on [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md). Multiple load tasks concurrently consume the messages from different partitions of a topic, and load the data into StarRocks. ##### Workflow[​](#workflow "Direct link to Workflow") 1. **Create a Routine Load job.** To load data from Kafka, you need to create a Routine Load job by running the [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md) statement. The FE parses the statement, and creates the job according to the properties you have specified. 2. **The FE splits the job into multiple load tasks.** The FE split the job into multiple load tasks based on certain rules. Each load task is an individual transaction. The splitting rules are as follows: * The FE calculates the actual concurrent number of the load tasks according to the desired concurrent number `desired_concurrent_number`, the partition number in the Kafka topic, and the number of the BE nodes that are alive. * The FE splits the job into load tasks based on the actual concurrent number calculated, and arranges the tasks in the task queue. Each Kafka topic consists of multiple partitions. The relation between the topic partition and the load task is as follows: * A partition is uniquely assigned to a load task, and all messages from the partition are consumed by the load task. * A load task can consume messages from one or more partitions. * All partitions are distributed evenly among load tasks. 3. **Multiple load tasks run concurrently to consume the messages from multiple Kafka topic partitions, and load the data into StarRocks** 1. **The FE schedules and submits load tasks**: the FE schedules the load tasks in the queue on a timely basis, and assigns them to selected Coordinator BE nodes. The interval between load tasks is defined by the configuration item `max_batch_interval`. The FE distributes the load tasks evenly to all BE nodes. See [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md#examples) for more information about `max_batch_interval`. 2. The Coordinator BE starts the load task, consumes messages in partitions, parses and filters the data. A load task lasts until the pre-defined amount of messages are consumed or the pre-defined time limit is reached. The message batch size and time limit are defined in the FE configurations `max_routine_load_batch_size` and `routine_load_task_consume_second`. For detailed information, see [FE Configuration](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md). The Coordinator BE then distributes the messages to the Executor BEs. The Executor BEs write the messages to disks. > **NOTE** > > StarRocks supports access to Kafka via security protocols including SASL\_SSL, SAS\_PLAINTEXT, SSL, and PLAINTEXT. This topic uses connecting to Kafka via PLAINTEXT as an example. If you need to connect to Kafka via other security protocols, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). 4. **The FE generates new load tasks to load data continuously.** After the Executor BEs has written the data to disks, the Coordinator BE reports the result of the load task to the FE. Based on the result, the FE then generates new load tasks to load the data continuously. Or the FE retries the failed tasks to make sure the data loaded into StarRocks is neither lost nor duplicated. #### Create a Routine Load job[​](#create-a-routine-load-job "Direct link to Create a Routine Load job") The following three examples describe how to consume CSV-format, JSON-format and Avro-format data in Kafka, and load the data into StarRocks by creating a Routine Load job. For detailed syntax and parameter descriptions, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### Load CSV-format data[​](#load-csv-format-data "Direct link to Load CSV-format data") This section describes how to create a Routine Load job to consume CSV-format data in a Kafka cluster, and load the data into StarRocks. ###### Prepare a dataset[​](#prepare-a-dataset "Direct link to Prepare a dataset") Suppose there is a CSV-format dataset in the topic `ordertest1` in a Kafka cluster. Every message in the dataset includes six fields: order ID, payment date, customer name, nationality, gender, and price. ```plain 2020050802,2020-05-08,Johann Georg Faust,Deutschland,male,895 2020050802,2020-05-08,Julien Sorel,France,male,893 2020050803,2020-05-08,Dorian Grey,UK,male,1262 2020050901,2020-05-09,Anna Karenina",Russia,female,175 2020051001,2020-05-10,Tess Durbeyfield,US,female,986 2020051101,2020-05-11,Edogawa Conan,japan,male,8924 ``` ###### Create a table[​](#create-a-table "Direct link to Create a table") According to the fields of CSV-format data, create the table `example_tbl1` in the database `example_db`. The following example creates a table with 5 fields excluding the field of customer gender in the CSV-format data. ```sql CREATE TABLE example_db.example_tbl1 ( `order_id` bigint NOT NULL COMMENT "Order ID", `pay_dt` date NOT NULL COMMENT "Payment date", `customer_name` varchar(26) NULL COMMENT "Customer name", `nationality` varchar(26) NULL COMMENT "Nationality", `price`double NULL COMMENT "Price" ) ENGINE=OLAP DUPLICATE KEY (order_id,pay_dt) DISTRIBUTED BY HASH(`order_id`); ``` > **NOTICE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). ###### Submit a Routine Load job[​](#submit-a-routine-load-job "Direct link to Submit a Routine Load job") Execute the following statement to submit a Routine Load job named `example_tbl1_ordertest1` to consume the messages in the topic `ordertest1` and load the data into the table `example_tbl1`. The load task consumes the messages from the initial offset in the specified partitions of the topic. ```sql CREATE ROUTINE LOAD example_db.example_tbl1_ordertest1 ON example_tbl1 COLUMNS TERMINATED BY ",", COLUMNS (order_id, pay_dt, customer_name, nationality, temp_gender, price) PROPERTIES ( "desired_concurrent_number" = "5" ) FROM KAFKA ( "kafka_broker_list" = ":,:", "kafka_topic" = "ordertest1", "kafka_partitions" = "0,1,2,3,4", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` After submitting the load job, you can execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job. * **load job name** There could be multiple load job on a table. Therefore, we recommend you name a load job with the corresponding Kafka topic and the time when the load job is submitted. It helps you distinguish the load job on each table. * **Column separator** The property `COLUMN TERMINATED BY` defines the column separator of the CSV-format data. The default is `\t`. * **Kafka topic partition and offset** You can specify the properties `kafka_partitions` and `kafka_offsets` to specify the partitions and offsets to consume the messages. For example, if you want the load job to consume messages from the Kafka partitions `"0,1,2,3,4"` of the topic `ordertest1` all with the initial offsets, you can specify the properties as follows: If you want the load job to consume messages from the Kafka partitions `"0,1,2,3,4"`and you need to specify a separate starting offset for each partition, you can configure as follows: ```sql "kafka_partitions" ="0,1,2,3,4", "kafka_offsets" = "OFFSET_BEGINNING, OFFSET_END, 1000, 2000, 3000" ``` You can also set the default offsets of all partitions with the property `property.kafka_default_offsets`. ```sql "kafka_partitions" ="0,1,2,3,4", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ``` For detailed information, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). * **Data mapping and transformation** To specify the mapping and transformation relationship between the CSV-format data, and the StarRocks table, you need to use the `COLUMNS` parameter. **Data mapping:** * StarRocks extracts the columns in the CSV-format data and maps them **in sequence** onto the fields declared in the `COLUMNS` parameter. * StarRocks extracts the fields declared in the `COLUMNS` parameter and maps them **by name** onto the columns of StarRocks table. **Data transformation:** And because the example excludes the column of customer gender from the CSV-format data, the field `temp_gender` in `COLUMNS` parameter is used as a placeholder for this field. The other fields are mapped to columns of the StarRocks table `example_tbl1` directly. For more information about data transformation, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md). > **NOTE** > > You do not need to specify the `COLUMNS` parameter if the names, number, and order of the columns in the CSV-format data completely correspond to those of the StarRocks table. * **Task concurrency** When there are many Kafka topic partitions and enough BE nodes, you can accelerate the loading by increasing the task concurrency. To increase the actual load task concurrency, you can increase the desired load task concurrency `desired_concurrent_number` when you create a routine load job. You can also set the dynamic configuration item of FE `max_routine_load_task_concurrent_num` ( default maximum load task currency ) to a larger value. For more information about `max_routine_load_task_concurrent_num`, please see [FE configuration items](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md). The actual task concurrency is defined by the minimum value among the number of BE nodes that are alive, the number of the pre-specified Kafka topic partitions, and the values of `desired_concurrent_number` and `max_routine_load_task_concurrent_num`. In the example, the number of BE nodes that are alive is `5`, the number of the pre-specified Kafka topic partitions is `5`, and the value of `max_routine_load_task_concurrent_num` is `5`. To increase the actual load task concurrency, you can increase the `desired_concurrent_number` from the default value `3` to `5`. For more about the properties, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### Load JSON-format data[​](#load-json-format-data "Direct link to Load JSON-format data") This section describes how to create a Routine Load job to consume JSON-format data in a Kafka cluster, and load the data into StarRocks. ###### Prepare a dataset[​](#prepare-a-dataset-1 "Direct link to Prepare a dataset") Suppose there is a JSON-format dataset in the topic `ordertest2` in a Kafka cluster. The dataset includes six keys: commodity ID, customer name, nationality, payment time, and price. Besides, you want to transform the payment time column into the DATE type, and load it into the `pay_dt` column in the StarRocks table. ```json {"commodity_id": "1", "customer_name": "Mark Twain", "country": "US","pay_time": 1589191487,"price": 875} {"commodity_id": "2", "customer_name": "Oscar Wilde", "country": "UK","pay_time": 1589191487,"price": 895} {"commodity_id": "3", "customer_name": "Antoine de Saint-Exupéry","country": "France","pay_time": 1589191487,"price": 895} ``` > **CAUTION** Each JSON object in a row must be in one Kafka message, otherwise a JSON parsing error is returned. ###### Create a table[​](#create-a-table-1 "Direct link to Create a table") According to the keys of the JSON-format data, create the table `example_tbl2` in the database `example_db`. ```sql CREATE TABLE `example_tbl2` ( `commodity_id` varchar(26) NULL COMMENT "Commodity ID", `customer_name` varchar(26) NULL COMMENT "Customer name", `country` varchar(26) NULL COMMENT "Country", `pay_time` bigint(20) NULL COMMENT "Payment time", `pay_dt` date NULL COMMENT "Payment date", `price`double SUM NULL COMMENT "Price" ) ENGINE=OLAP AGGREGATE KEY(`commodity_id`,`customer_name`,`country`,`pay_time`,`pay_dt`) DISTRIBUTED BY HASH(`commodity_id`); ``` > **NOTICE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). ###### Submit a Routine Load job[​](#submit-a-routine-load-job-1 "Direct link to Submit a Routine Load job") Execute the following statement to submit a Routine Load job named `example_tbl2_ordertest2` to consume the messages in the topic `ordertest2` and load the data into the table `example_tbl2`. The load task consumes the messages from the initial offset in the specified partitions of the topic. ```sql CREATE ROUTINE LOAD example_db.example_tbl2_ordertest2 ON example_tbl2 COLUMNS(commodity_id, customer_name, country, pay_time, price, pay_dt=from_unixtime(pay_time, '%Y%m%d')) PROPERTIES ( "desired_concurrent_number" = "5", "format" = "json", "jsonpaths" = "[\"$.commodity_id\",\"$.customer_name\",\"$.country\",\"$.pay_time\",\"$.price\"]" ) FROM KAFKA ( "kafka_broker_list" =":,:", "kafka_topic" = "ordertest2", "kafka_partitions" ="0,1,2,3,4", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` After submitting the load job, you can execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job. * **Data format** You need to specify `"format" = "json"` in the clause `PROPERTIES` to define that the data format is JSON. * **Data mapping and transformation** To specify the mapping and transformation relationship between the JSON-format data, and the StarRocks table, you need to specify the parameter `COLUMNS` and property`jsonpaths`. The order of fields specified in the `COLUMNS` parameter must match that of the JSON-format data, and the name of fields must match that of the StarRocks table. The property `jsonpaths` is used to extract the required fields from the JSON data. These fields are then named by the property `COLUMNS`. Because the example needs to transform the payment time field to the DATE data type, and load the data into the `pay_dt` column in the StarRocks table, you need to use the from\_unixtime function. The other fields are mapped to fields of the table `example_tbl2` directly. **Data mapping:** * StarRocks extracts the `name` and `code` keys of JSON-format data and maps them onto the keys declared in the `jsonpaths` property. * StarRocks extracts the keys declared in the `jsonpaths` property and maps them **in sequence** onto the fields declared in the `COLUMNS` parameter. * StarRocks extracts the fields declared in the `COLUMNS` parameter and maps them **by name** onto the columns of StarRocks table. **Data transformation**: * Because the example needs to transform the key `pay_time` to the DATE data type, and load the data into the `pay_dt` column in the StarRocks table, you need to use the from\_unixtime function in `COLUMNS` parameter. The other fields are mapped to fields of the table `example_tbl2` directly. * And because the example excludes the column of customer gender from the JSON-format data, the field `temp_gender` in `COLUMNS` parameter is used as a placeholder for this field. The other fields are mapped to columns of the StarRocks table `example_tbl1` directly. For more information about data transformation, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md). > **NOTE** > > You do not need to specify the `COLUMNS` parameter if the names and number of the keys in the JSON object completely match those of fields in the StarRocks table. ##### Load Avro-format data[​](#load-avro-format-data "Direct link to Load Avro-format data") Since v3.0.1, StarRocks supports loading Avro data by using Routine Load. ###### Prepare a dataset[​](#prepare-a-dataset-2 "Direct link to Prepare a dataset") ###### Avro schema[​](#avro-schema "Direct link to Avro schema") 1. Create the following Avro schema file `avro_schema.avsc`: ```json { "type": "record", "name": "sensor_log", "fields" : [ {"name": "id", "type": "long"}, {"name": "name", "type": "string"}, {"name": "checked", "type" : "boolean"}, {"name": "data", "type": "double"}, {"name": "sensor_type", "type": {"type": "enum", "name": "sensor_type_enum", "symbols" : ["TEMPERATURE", "HUMIDITY", "AIR-PRESSURE"]}} ] } ``` 2. Register the Avro schema in the [Schema Registry](https://docs.confluent.io/cloud/current/get-started/schema-registry.html#create-a-schema). ###### Avro data[​](#avro-data "Direct link to Avro data") Prepare the Avro data and send it to the Kafka topic `topic_0`. ###### Create a table[​](#create-a-table-2 "Direct link to Create a table") According to the fields of Avro data, create a table `sensor_log` in the target database `example_db` in the StarRocks cluster. The column names of the table must match the field names in the Avro data. For the data type mapping between the table columns and the Avro data fields, see \[Data types mapping]\(#Data types mapping). ```sql CREATE TABLE example_db.sensor_log ( `id` bigint NOT NULL COMMENT "sensor id", `name` varchar(26) NOT NULL COMMENT "sensor name", `checked` boolean NOT NULL COMMENT "checked", `data` double NULL COMMENT "sensor data", `sensor_type` varchar(26) NOT NULL COMMENT "sensor type" ) ENGINE=OLAP DUPLICATE KEY (id) DISTRIBUTED BY HASH(`id`); ``` > **NOTICE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). ###### Submit a Routine Load job[​](#submit-a-routine-load-job-2 "Direct link to Submit a Routine Load job") Execute the following statement to submit a Routine Load job named `sensor_log_load_job` to consume the Avro messages in the Kafka topic `topic_0` and load the data into the table `sensor_log` in the database `sensor`. The load job consumes the messages from the initial offset in the specified partitions of the topic. ```sql CREATE ROUTINE LOAD example_db.sensor_log_load_job ON sensor_log PROPERTIES ( "format" = "avro" ) FROM KAFKA ( "kafka_broker_list" = ":,:,...", "confluent.schema.registry.url" = "http://172.xx.xxx.xxx:8081", "kafka_topic" = "topic_0", "kafka_partitions" = "0,1,2,3,4,5", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` * Data Format You need to specify `"format = "avro"` in the clause `PROPERTIES` to define that the data format is Avro. * Schema Registry You need to configure `confluent.schema.registry.url` to specify the URL of the Schema Registry where the Avro schema is registered. StarRocks retrieves the Avro schema by using this URL. The format is as follows: ```plaintext confluent.schema.registry.url = http[s]://[:@][:] ``` * Data mapping and transformation To specify the mapping and transformation relationship between the Avro-format data and the StarRocks table, you need to specify the parameter `COLUMNS` and property `jsonpaths`. The order of fields specified in the `COLUMNS` parameter must match that of the fields in the property `jsonpaths`, and the names of fields must match these of the StarRocks table. The property `jsonpaths` is used to extract the required fields from the Avro data. These fields are then named by the property `COLUMNS`. For more information about data transformation, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md). > NOTE > > You do not need to specify the `COLUMNS` parameter if the names and number of the fields in the Avro record completely match those of columns in the StarRocks table. After submitting the load job, you can execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job. ###### Data types mapping[​](#data-types-mapping "Direct link to Data types mapping") The data type mapping between the Avro data fields you want to load and the StarRocks table columns is as follows: ###### Primitive types[​](#primitive-types "Direct link to Primitive types") | Avro | StarRocks | | ------- | --------- | | nul | NULL | | boolean | BOOLEAN | | int | INT | | long | BIGINT | | float | FLOAT | | double | DOUBLE | | bytes | STRING | | string | STRING | ###### Complex types[​](#complex-types "Direct link to Complex types") | Avro | StarRocks | | -------------- | -------------------------------------------------------------------------- | | record | STRUCT, or load the entire RECORD or its subfields into StarRocks as JSON. | | enums | STRING | | arrays | ARRAY | | maps | MAP or JSON | | union(T, null) | NULLABLE(T) | | fixed | STRING | ###### Limits[​](#limits "Direct link to Limits") * Currently, StarRocks does not support schema evolution. * Each Kafka message must only contain a single Avro data record. ##### Access source message metadata[​](#access-source-message-metadata "Direct link to Access source message metadata") When loading JSON- or Avro-format data, you can populate destination columns from a message's Kafka/Pulsar metadata — topic, partition, offset, timestamp, key, and headers — instead of from the message payload. You declare the metadata in an `INCLUDE METADATA (...)` clause that binds each metadata key to an alias; the alias is an ordinary source column you reference from `COLUMNS`. This is useful for auditing (which topic/partition/offset a row came from), event-time processing (using the message timestamp), and routing on a header value. ###### Syntax[​](#syntax "Direct link to Syntax") ```sql INCLUDE METADATA ( [AS ] [, [AS ] ...] ) ``` `INCLUDE METADATA` is a load property; place it among the other load properties (such as `COLUMNS` and `WHERE`), in any order, before the `PROPERTIES` and `FROM` clauses. `AS ` is optional. If it is omitted, the alias defaults to ``. The alias must be unique within the clause, and it must not collide with a payload field, a destination-table column, or a reserved column name. ###### Metadata keys[​](#metadata-keys "Direct link to Metadata keys") The supported keys depend on the data source. | Source | Key | Type | Description | | ------ | ----------------- | ---------------------- | ------------------------------------------------------------------------------------------------- | | KAFKA | `TOPIC` | VARCHAR | Topic name. | | KAFKA | `PARTITION` | INT | Partition number. | | KAFKA | `OFFSET` | BIGINT | Message offset within the partition. | | KAFKA | `TIMESTAMP_MS` | BIGINT | Record timestamp in milliseconds since epoch. `NULL` when the broker reports no timestamp. | | KAFKA | `KEY` | VARCHAR | Message key as raw bytes. `NULL` when the message has no key. | | KAFKA | `HEADERS` | MAP\ | All headers as a map. On duplicate keys, the last value wins. | | PULSAR | `TOPIC` | VARCHAR | The logical topic the job consumes (a partitioned topic's `-partition-N` suffix is not included). | | PULSAR | `PARTITION` | INT | Partition index, parsed from the per-message topic name. `NULL` for a non-partitioned topic. | | PULSAR | `KEY` | VARCHAR | Partition key. `NULL` when the message has no key. | | PULSAR | `MESSAGE_ID` | VARCHAR | Message ID. | | PULSAR | `PUBLISH_TIME_MS` | BIGINT | Publish time in milliseconds since epoch. | | PULSAR | `EVENT_TIME_MS` | BIGINT | Event time in milliseconds since epoch. `NULL` when the producer did not set it. | | PULSAR | `PROPERTIES` | MAP\ | All properties as a map. On duplicate keys, the last value wins. | To read a single header/property value, use `element_at(, '')` over the `HEADERS`/`PROPERTIES` map: the last value wins on a duplicate key, and the result is `NULL` when the key is absent. Header/property values are raw bytes placed into VARCHAR as-is (no UTF-8 validation). ###### Usage notes[​](#usage-notes "Direct link to Usage notes") * `INCLUDE METADATA` is available for `format = json` (Kafka and Pulsar) and `format = avro` (Kafka only; Pulsar Routine Load does not support Avro). It is not supported for CSV, where one message can expand into many rows and per-message metadata would be ambiguous. * A metadata alias is an ordinary source column: reference it anywhere in the `COLUMNS` expressions. If a payload field has the same name as a metadata key, specify a different alias with `AS ` to avoid ambiguity. * `OFFSET` is Kafka-only and `MESSAGE_ID` is Pulsar-only; using a key unsupported by the source raises an error that lists the keys supported for that source. * `HEADERS`/`PROPERTIES` is a `MAP\`. The source headers/properties are an ordered list that may repeat a key; duplicates collapse into the map with the last value winning (an `element_at(map, 'name')` lookup is likewise last-wins, and returns `NULL` when the key is absent). Values are raw bytes stored in VARCHAR as-is — there is no UTF-8 validation or decoding. ###### Example[​](#example "Direct link to Example") Load the order payload field `order_id` together with the source topic, partition, offset, the message timestamp converted to a `DATETIME`, and a `trace-id` header: ```sql CREATE TABLE example_db.orders_with_meta ( order_id BIGINT, src_topic VARCHAR(256), src_partition INT, src_offset BIGINT, msg_time DATETIME, trace_id VARCHAR(128) ) ENGINE = OLAP DUPLICATE KEY(order_id) DISTRIBUTED BY HASH(order_id); CREATE ROUTINE LOAD example_db.orders_with_meta_job ON orders_with_meta INCLUDE METADATA ( TOPIC AS m_topic, PARTITION AS m_partition, OFFSET AS m_offset, TIMESTAMP_MS AS m_timestamp, HEADERS AS m_headers ), COLUMNS ( order_id, src_topic = m_topic, src_partition = m_partition, src_offset = m_offset, msg_time = from_unixtime(m_timestamp / 1000), trace_id = element_at(m_headers, 'trace-id') ) PROPERTIES ( "format" = "json", "jsonpaths" = "[\"$.order_id\"]" ) FROM KAFKA ( "kafka_broker_list" = ":,...", "kafka_topic" = "topic_orders", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` A metadata alias may be used inside an expression (as with `from_unixtime(m_timestamp / 1000)` above); only payload columns are listed in `jsonpaths`. #### Check a load job and task[​](#check-a-load-job-and-task "Direct link to Check a load job and task") ##### Check a load job[​](#check-a-load-job "Direct link to Check a load job") Execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job `example_tbl2_ordertest2`. StarRocks returns the execution state `State`, the statistical information (including the total rows consumed and the total rows loaded) `Statistics`, and the progress of the load job `progress`. If the state of the load job is automatically changed to **PAUSED**, it is possibly because the number of error rows has exceeded the threshold. For detailed instructions on setting this threshold, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). You can check the files `ReasonOfStateChanged` and `ErrorLogUrls` to identify and troubleshoot the problem. Having fixed the problem, you can then execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume the **PAUSED** load job. If the state of the load job is **CANCELLED**, it is possibly because the load job encounters an exception (such as the table has been dropped). You can check the files `ReasonOfStateChanged` and `ErrorLogUrls` to identify and troubleshoot the problem. However, you cannot resume a **CANCELLED** load job. ```sql MySQL [example_db]> SHOW ROUTINE LOAD FOR example_tbl2_ordertest2 \G *************************** 1. row *************************** Id: 63013 Name: example_tbl2_ordertest2 CreateTime: 2022-08-10 17:09:00 PauseTime: NULL EndTime: NULL DbName: default_cluster:example_db TableName: example_tbl2 State: RUNNING DataSourceType: KAFKA CurrentTaskNum: 3 JobProperties: {"partitions":"*","partial_update":"false","columnToColumnExpr":"commodity_id,customer_name,country,pay_time,pay_dt=from_unixtime(`pay_time`, '%Y%m%d'),price","maxBatchIntervalS":"20","whereExpr":"*","dataFormat":"json","timezone":"Asia/Shanghai","format":"json","json_root":"","strict_mode":"false","jsonpaths":"[\"$.commodity_id\",\"$.customer_name\",\"$.country\",\"$.pay_time\",\"$.price\"]","desireTaskConcurrentNum":"3","maxErrorNum":"0","strip_outer_array":"false","currentTaskConcurrentNum":"3","maxBatchRows":"200000"} DataSourceProperties: {"topic":"ordertest2","currentKafkaPartitions":"0,1,2,3,4","brokerList":":,:"} CustomProperties: {"kafka_default_offsets":"OFFSET_BEGINNING"} Statistic: {"receivedBytes":230,"errorRows":0,"committedTaskNum":1,"loadedRows":2,"loadRowsRate":0,"abortedTaskNum":0,"totalRows":2,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":522} Progress: {"0":"1","1":"OFFSET_ZERO","2":"OFFSET_ZERO","3":"OFFSET_ZERO","4":"OFFSET_ZERO"} ReasonOfStateChanged: ErrorLogUrls: OtherMsg: ``` > **CAUTION** > > You cannot check a load job that has stopped or has not yet started. ##### Check a load task[​](#check-a-load-task "Direct link to Check a load task") Execute the [SHOW ROUTINE LOAD TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD_TASK.md) statement to check the load tasks of the load job `example_tbl2_ordertest2`, such as how many tasks are currently running, the Kafka topic partitions that are consumed and the consumption progress `DataSourceProperties`, and the corresponding Coordinator BE node `BeId`. ```sql MySQL [example_db]> SHOW ROUTINE LOAD TASK WHERE JobName = "example_tbl2_ordertest2" \G *************************** 1. row *************************** TaskId: 18c3a823-d73e-4a64-b9cb-b9eced026753 TxnId: -1 TxnStatus: UNKNOWN JobId: 63013 CreateTime: 2022-08-10 17:09:05 LastScheduledTime: 2022-08-10 17:47:27 ExecuteStartTime: NULL Timeout: 60 BeId: -1 DataSourceProperties: {"1":0,"4":0} Message: there is no new data in kafka, wait for 20 seconds to schedule again *************************** 2. row *************************** TaskId: f76c97ac-26aa-4b41-8194-a8ba2063eb00 TxnId: -1 TxnStatus: UNKNOWN JobId: 63013 CreateTime: 2022-08-10 17:09:05 LastScheduledTime: 2022-08-10 17:47:26 ExecuteStartTime: NULL Timeout: 60 BeId: -1 DataSourceProperties: {"2":0} Message: there is no new data in kafka, wait for 20 seconds to schedule again *************************** 3. row *************************** TaskId: 1a327a34-99f4-4f8d-8014-3cd38db99ec6 TxnId: -1 TxnStatus: UNKNOWN JobId: 63013 CreateTime: 2022-08-10 17:09:26 LastScheduledTime: 2022-08-10 17:47:27 ExecuteStartTime: NULL Timeout: 60 BeId: -1 DataSourceProperties: {"0":2,"3":0} Message: there is no new data in kafka, wait for 20 seconds to schedule again ``` #### Pause a load job[​](#pause-a-load-job "Direct link to Pause a load job") You can execute the [PAUSE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/PAUSE_ROUTINE_LOAD.md) statement to pause a load job. The state of the load job will be **PAUSED** after the statement is executed. However, it has not stopped. You can execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume it. You can also check its status with the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. The following example pauses the load job `example_tbl2_ordertest2`: ```sql PAUSE ROUTINE LOAD FOR example_tbl2_ordertest2; ``` #### Resume a load job[​](#resume-a-load-job "Direct link to Resume a load job") You can execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume a paused load job. The state of the load job will be **NEED\_SCHEDULE** temporarily (because the load job is being re-scheduled), and then become **RUNNING**. You can check its status with the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. The following example resumes the paused load job `example_tbl2_ordertest2`: ```sql RESUME ROUTINE LOAD FOR example_tbl2_ordertest2; ``` #### Alter a load job[​](#alter-a-load-job "Direct link to Alter a load job") Before altering a load job, you must pause it with the [PAUSE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/PAUSE_ROUTINE_LOAD.md) statement. Then you can execute the [ALTER ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/ALTER_ROUTINE_LOAD.md). After altering it, you can execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume it, and check its status with the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. Suppose the number of the BE nodes that are alive increases to `6` and the Kafka topic partitions to be consumed is `"0,1,2,3,4,5,6,7"`. If you want to increase the actual load task concurrency, you can execute the following statement to increase the number of desired task concurrency `desired_concurrent_number` to `6` (greater than or equal to the number of BE nodes that are alive), and specify the Kafka topic partitions and initial offsets. > **NOTE** > > Because the actual task concurrency is determined by the minimum value of multiple parameters, you must make sure that the value of the FE dynamic parameter `max_routine_load_task_concurrent_num` is greater than or equal to `6`. ```sql ALTER ROUTINE LOAD FOR example_tbl2_ordertest2 PROPERTIES ( "desired_concurrent_number" = "6" ) FROM kafka ( "kafka_partitions" = "0,1,2,3,4,5,6,7", "kafka_offsets" = "OFFSET_BEGINNING,OFFSET_BEGINNING,OFFSET_BEGINNING,OFFSET_BEGINNING,OFFSET_END,OFFSET_END,OFFSET_END,OFFSET_END" ); ``` #### Stop a load job[​](#stop-a-load-job "Direct link to Stop a load job") You can execute the [STOP ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/STOP_ROUTINE_LOAD.md) statement to stop a load job. The state of the load job will be **STOPPED** after the statement is executed, and you cannot resume a stopped load job. You cannot check the status of a stopped load job with the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. The following example stops the load job `example_tbl2_ordertest2`: ```sql STOP ROUTINE LOAD FOR example_tbl2_ordertest2; ``` --- ### Continuously load data from Apache® Pulsar™ Experimental feature [Advice on use of experimental features](https://docs.starrocks.io/docs/maturity/features.md) As of StarRocks version 2.5, Routine Load supports continuously loading data from Apache® Pulsar™. Pulsar is distributed, open source pub-sub messaging and streaming platform with a store-compute separation architecture. Loading data from Pulsar via Routine Load is similar to loading data from Apache Kafka. This topic uses CSV-formatted data as an example to introduce how to load data from Apache Pulsar via Routine Load. #### Supported data file formats[​](#supported-data-file-formats "Direct link to Supported data file formats") Routine Load supports consuming CSV and JSON formatted data from a Pulsar cluster. > NOTE > > As for data in CSV format, StarRocks supports UTF-8 encoded strings within 50 bytes as column separators. Commonly used column separators include comma (,), tab and pipe (|). #### Pulsar-related concepts[​](#pulsar-related-concepts "Direct link to Pulsar-related concepts") **[Topic](https://pulsar.apache.org/docs/2.10.x/concepts-messaging/#topics)** Topics in Pulsar are named channels for transmitting messages from producers to consumers. Topics in Pulsar are divided into partitioned topics and non-partitioned topics. * **[Partitioned topics](https://pulsar.apache.org/docs/2.10.x/concepts-messaging/#partitioned-topics)** are a special type of topic that are handled by multiple brokers, thus allowing for higher throughput. A partitioned topic is actually implemented as N internal topics, where N is the number of partitions. * **Non-partitioned topics** are a normal type of topic that are served only by a single broker, which limits the maximum throughput of the topic. **[Message ID](https://pulsar.apache.org/docs/2.10.x/concepts-messaging/#messages)** The message ID of a message is assigned by [BookKeeper instances](https://pulsar.apache.org/docs/2.10.x/concepts-architecture-overview/#apache-bookkeeper) as soon as the message is persistently stored. Message ID indicates a message' s specific position in a ledger and is unique within a Pulsar cluster. Pulsar supports consumers specifying the initial position through consumer.*seek*(*messageId*). But compared to the Kafka consumer offset which is a long integer value, the message ID consists of four parts: `ledgerId:entryID:partition-index:batch-index`. Therefore, you can't get the Message ID directly from the message. As a result, at present, **Routine Load does not support specifying initial position when loading data from Pulsar, but only supports consuming data from the beginning or end of a partition.** **[Subscription](https://pulsar.apache.org/docs/2.10.x/concepts-messaging/#subscriptions)** A subscription is a named configuration rule that determines how messages are delivered to consumers. Pulsar also supports consumers simultaneously subscribing to multiple topics. A topic can have multiple subscriptions. The type of a subscription is defined when a consumer connects to it, and the type can be changed by restarting all consumers with a different configuration. Four subscription types are available in Pulsar: * `exclusive` (default)*:* Only a single consumer is allowed to attach to the subscription. Only one customer is allowed to consume messages. * `shared`: Multiple consumers can attach to the same subscription. Messages are delivered in a round robin distribution across consumers, and any given message is delivered to only one consumer. * `failover`: Multiple consumers can attach to the same subscription. A master consumer is picked for a non-partitioned topic or each partition of a partitioned topic and receives messages. When the master consumer disconnects, all (non-acknowledged and subsequent) messages are delivered to the next consumer in line. * `key_shared`: Multiple consumers can attach to the same subscription. Messages are delivered in a distribution across consumers and message with same key or same ordering key are delivered to only one consumer. > Note: > > Currently Routine Load uses exclusive type. #### Create a Routine Load job[​](#create-a-routine-load-job "Direct link to Create a Routine Load job") The following examples describe how to consume CSV-formatted messages in Pulsar, and load the data into StarRocks by creating a Routine Load job. For detailed instruction and reference, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ```sql CREATE ROUTINE LOAD load_test.routine_wiki_edit_1 ON routine_wiki_edit COLUMNS TERMINATED BY ",", ROWS TERMINATED BY "\n", COLUMNS (order_id, pay_dt, customer_name, nationality, temp_gender, price) WHERE event_time > "2022-01-01 00:00:00", PROPERTIES ( "desired_concurrent_number" = "1", "max_batch_interval" = "15000", "max_error_number" = "1000" ) FROM PULSAR ( "pulsar_service_url" = "pulsar://localhost:6650", "pulsar_topic" = "persistent://tenant/namespace/topic-name", "pulsar_subscription" = "load-test", "pulsar_partitions" = "load-partition-0,load-partition-1", "pulsar_initial_positions" = "POSITION_EARLIEST,POSITION_LATEST", "property.auth.token" = "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUJzdWIiOiJqaXV0aWFuY2hlbiJ9.lulGngOC72vE70OW54zcbyw7XdKSOxET94WT_hIqD5Y" ); ``` When Routine Load is created to consume data from Pulsar, most input parameters except for `data_source_properties` are the same as consuming data from Kafka . For descriptions about parameters except data\_source\_properties `data_source_properties` , see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). The parameters related to `data_source_properties` and their descriptions are as follows: | **Parameter** | **Required** | **Description** | | ---------------------------------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | pulsar\_service\_url | Yes | The URL that is used to connect to the Pulsar cluster. Format: `"pulsar://ip:port"` or `"pulsar://service:port"`.Example: `"pulsar_service_url" = "pulsar://``localhost:6650``"` | | pulsar\_topic | Yes | Subscribed topic. Example: "pulsar\_topic" = "persistent://tenant/namespace/topic-name" | | pulsar\_subscription | Yes | Subscription configured for the topic.Example: `"pulsar_subscription" = "my_subscription"` | | pulsar\_partitions, pulsar\_initial\_positions | No | `pulsar_partitions` : Subscribed partitions in the topic.`pulsar_initial_positions`: initial positions of partitions specified by `pulsar_partitions`. The initial positions must correspond to the partitions in `pulsar_partitions`. Valid values:`POSITION_EARLIEST` (Default value): Subscription starts from the earliest available message in the partition. `POSITION_LATEST`: Subscription starts from the latest available message in the partition.Note:If `pulsar_partitions` is not specified, the topic's all partitions are subscribed.If both `pulsar_partitions` and `property.pulsar_default_initial_position` are specified, the `pulsar_partitions` value overrides `property.pulsar_default_initial_position` value.If neither `pulsar_partitions` nor `property.pulsar_default_initial_position` is specified, subscription starts from the latest available message in the partition.Example:`"pulsar_partitions" = "my-partition-0,my-partition-1,my-partition-2,my-partition-3", "pulsar_initial_positions" = "POSITION_EARLIEST,POSITION_EARLIEST,POSITION_LATEST,POSITION_LATEST"` | Routine Load supports the following custom parameters for Pulsar. | Parameter | Required | Description | | ------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | property.pulsar\_default\_initial\_position | No | The default initial positions when the topic's partitions are subscribed. The parameter takes effect when `pulsar_initial_positions` is not specified. Its valid values are the same as the valid values of `pulsar_initial_positions`.Example: `"``property.pulsar_default_initial_position" = "POSITION_EARLIEST"` | | property.auth.token | No | If Pulsar enables authenticating clients using security tokens, you need the token string to verify your identity.Example: `"p``roperty.auth.token" = "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUJzdWIiOiJqaXV0aWFuY2hlbiJ9.lulGngOC72vE70OW54zcbyw7XdKSOxET94WT_hIqD"` | #### Check a load job and task[​](#check-a-load-job-and-task "Direct link to Check a load job and task") ##### Check a load job[​](#check-a-load-job "Direct link to Check a load job") Execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job `routine_wiki_edit_1`. StarRocks returns the execution state `State`, the statistical information (including the total rows consumed and the total rows loaded) `Statistics`, and the progress of the load job `progress`. When you check a Routine Load job that consumes data from Pulsar, most returned parameters except for `progress` are the same as consuming data from Kafka. `progress` refers to backlog, that is the number of unacked messages in a partition. ```plaintext MySQL [load_test] > SHOW ROUTINE LOAD for routine_wiki_edit_1 \G *************************** 1. row *************************** Id: 10142 Name: routine_wiki_edit_1 CreateTime: 2022-06-29 14:52:55 PauseTime: 2022-06-29 17:33:53 EndTime: NULL DbName: default_cluster:test_pulsar TableName: test1 State: PAUSED DataSourceType: PULSAR CurrentTaskNum: 0 JobProperties: {"partitions":"*","rowDelimiter":"'\n'","partial_update":"false","columnToColumnExpr":"*","maxBatchIntervalS":"10","whereExpr":"*","timezone":"Asia/Shanghai","format":"csv","columnSeparator":"','","json_root":"","strict_mode":"false","jsonpaths":"","desireTaskConcurrentNum":"3","maxErrorNum":"10","strip_outer_array":"false","currentTaskConcurrentNum":"0","maxBatchRows":"200000"} DataSourceProperties: {"serviceUrl":"pulsar://localhost:6650","currentPulsarPartitions":"my-partition-0,my-partition-1","topic":"persistent://tenant/namespace/topic-name","subscription":"load-test"} CustomProperties: {"auth.token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUJzdWIiOiJqaXV0aWFuY2hlbiJ9.lulGngOC72vE70OW54zcbyw7XdKSOxET94WT_hIqD"} Statistic: {"receivedBytes":5480943882,"errorRows":0,"committedTaskNum":696,"loadedRows":66243440,"loadRowsRate":29000,"abortedTaskNum":0,"totalRows":66243440,"unselectedRows":0,"receivedBytesRate":2400000,"taskExecuteTimeMs":2283166} Progress: {"my-partition-0(backlog): 100","my-partition-1(backlog): 0"} ReasonOfStateChanged: ErrorLogUrls: OtherMsg: 1 row in set (0.00 sec) ``` ##### Check a load task[​](#check-a-load-task "Direct link to Check a load task") Execute the [SHOW ROUTINE LOAD TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD_TASK.md) statement to check the load tasks of the load job `routine_wiki_edit_1`, such as how many tasks are running, the Kafka topic partitions that are consumed and the consumption progress `DataSourceProperties`, and the corresponding Coordinator BE node `BeId`. ```sql MySQL [example_db]> SHOW ROUTINE LOAD TASK WHERE JobName = "routine_wiki_edit_1" \G ``` #### Alter a load job[​](#alter-a-load-job "Direct link to Alter a load job") Before altering a load job, you must pause it by using the [PAUSE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/PAUSE_ROUTINE_LOAD.md) statement. Then you can execute the [ALTER ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/ALTER_ROUTINE_LOAD.md). After altering it, you can execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume it, and check its status by using the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. When Routine Load is used to consume data from Pulsar, most returned parameters except for `data_source_properties` are the same as consuming data from Kafka. **Take note of the following points**: * Among the `data_source_properties` related parameters, only `pulsar_partitions`, `pulsar_initial_positions`, and custom Pulsar parameters `property.pulsar_default_initial_position` and `property.auth.token` are currently supported to be modified. The parameters `pulsar_service_url`, `pulsar_topic`, and `pulsar_subscription` cannot be modified. * If you need to modify the partition to be consumed and the matched initilal position, you need to make sure that you specify the partition using `pulsar_partitions` when you create the Routine Load job, and only the intial position `pulsar_initial_positions` of the specified partition can be modified. * If you specify only Topic `pulsar_topic` when creating a Routine Load job, but not partitions `pulsar_partitions`, you can modify the starting position of all partitions under topic via `pulsar_default_initial_position`. --- ### Change data through loading [Primary Key tables](https://docs.starrocks.io/docs/table_design/table_types/primary_key_table.md) provided by StarRocks allow you to make data changes to StarRocks tables by running [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md), [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md), or [Routine Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md) jobs. These data changes include inserts, updates, and deletions. However, Primary Key tables do not support changing data by using [Spark Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SPARK_LOAD.md) or [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). StarRocks also supports partial updates and conditional updates. You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. This topic uses CSV data as an example to describe how to make data changes to a StarRocks table through loading. The data file formats that are supported vary depending on the loading method of your choice. > **NOTE** > > For CSV data, you can use a UTF-8 string, such as a comma (,), tab, or pipe (|), whose length does not exceed 50 bytes as a text delimiter. #### Implementation[​](#implementation "Direct link to Implementation") Primary Key tables provided by StarRocks support UPSERT and DELETE operations and does not distinguish INSERT operations from UPDATE operations. When you create a load job, StarRocks supports adding a field named `__op` to the job creation statement or command. The `__op` field is used to specify the type of operation you want to perform. > **NOTE** > > When you create a table, you do not need to add a column named `__op` to that table. The method of defining the `__op` field varies depending on the loading method of your choice: * If you choose Stream Load, define the `__op` field by using the `columns` parameter. * If you choose Broker Load, define the `__op` field by using the SET clause. * If you choose Routine Load, define the `__op` field by using the `COLUMNS` column. You can decide whether to add the `__op` field based on the data changes you want to make. If you do not add the `__op` field, the operation type defaults to UPSERT. The major data change scenarios are as follows: * If the data file you want to load involves only UPSERT operations, you do not need to add the `__op` field. * If the data file you want to load involves only DELETE operations, you must add the `__op` field and specify the operation type as DELETE. * If the data file you want to load involves both UPSERT and DELETE operations, you must add the `__op` field and make sure that the data file contains a column whose values are `0` or `1`. A value of `0` indicates an UPSERT operation, and a value of `1` indicates a DELETE operation. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * Make sure that each row in your data file has the same number of columns. * The columns that involve data changes must include the primary key column. #### Basic operations[​](#basic-operations "Direct link to Basic operations") This section provides examples of how to make data changes to a StarRocks table through loading. For detailed syntax and parameter descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md), [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md), and [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### UPSERT[​](#upsert "Direct link to UPSERT") If the data file you want to load involves only UPSERT operations, you do not need to add the `__op` field. > **NOTE** > > If you add the `__op` field: > > * You can specify the operation type as UPSERT. > > * You can leave the `__op` field empty, because the operation type defaults to UPSERT. ###### Data examples[​](#data-examples "Direct link to Data examples") 1. Prepare a data file. a. Create a CSV file named `example1.csv` in your local file system. The file consists of three columns, which represent user ID, user name, and user score in sequence. ```plain 101,Lily,100 102,Rose,100 ``` b. Publish the data of `example1.csv` to `topic1` of your Kafka cluster. 2. Prepare a StarRocks table. a. Create a Primary Key table named `table1` in your StarRocks database `test_db`. The table consists of three columns: `id`, `name`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `table1` ( `id` int(11) NOT NULL COMMENT "user ID", `name` varchar(65533) NOT NULL COMMENT "user name", `score` int(11) NOT NULL COMMENT "user score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` > **NOTE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). b. Insert a record into `table1`. ```sql INSERT INTO table1 VALUES (101, 'Lily',80); ``` ###### Load data[​](#load-data "Direct link to Load data") Run a load job to update the record whose `id` is `101` in `example1.csv` to `table1` and insert the record whose `id` is `102` in `example1.csv` into `table1`. * Run a Stream Load job. * If you do not want to include the `__op` field, run the following command: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "label:label1" \ -H "column_separator:," \ -T example1.csv -XPUT \ http://:/api/test_db/table1/_stream_load ``` * If you want to include the `__op` field, run the following command: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "label:label2" \ -H "column_separator:," \ -H "columns:__op ='upsert'" \ -T example1.csv -XPUT \ http://:/api/test_db/table1/_stream_load ``` * Run a Broker Load job. * If you do not want to include the `__op` field, run the following command: ```sql LOAD LABEL test_db.label1 ( data infile("hdfs://:/example1.csv") into table table1 columns terminated by "," format as "csv" ) WITH BROKER; ``` * If you want to include the `__op` field, run the following command: ```sql LOAD LABEL test_db.label2 ( data infile("hdfs://:/example1.csv") into table table1 columns terminated by "," format as "csv" set (__op = 'upsert') ) WITH BROKER; ``` * Run a Routine Load job. * If you do not want to include the `__op` field, run the following command: ```sql CREATE ROUTINE LOAD test_db.table1 ON table1 COLUMNS TERMINATED BY ",", COLUMNS (id, name, score) PROPERTIES ( "desired_concurrent_number" = "3", "max_batch_interval" = "20", "max_batch_rows"= "250000", "max_error_number" = "1000" ) FROM KAFKA ( "kafka_broker_list" =":", "kafka_topic" = "test1", "property.kafka_default_offsets" ="OFFSET_BEGINNING" ); ``` * If you want to include the `__op` field, run the following command: ```sql CREATE ROUTINE LOAD test_db.table1 ON table1 COLUMNS TERMINATED BY ",", COLUMNS (id, name, score, __op ='upsert') PROPERTIES ( "desired_concurrent_number" = "3", "max_batch_interval" = "20", "max_batch_rows"= "250000", "max_error_number" = "1000" ) FROM KAFKA ( "kafka_broker_list" =":", "kafka_topic" = "test1", "property.kafka_default_offsets" ="OFFSET_BEGINNING" ); ``` ###### Query data[​](#query-data "Direct link to Query data") After the load is complete, query the data of `table1` to verify that the load is successful: ```sql SELECT * FROM table1; +------+------+-------+ | id | name | score | +------+------+-------+ | 101 | Lily | 100 | | 102 | Rose | 100 | +------+------+-------+ 2 rows in set (0.02 sec) ``` As shown in the preceding query result, the record whose `id` is `101` in `example1.csv` has been updated to `table1`, and the record whose `id` is `102` in `example1.csv` has been inserted into `table1`. ##### DELETE[​](#delete "Direct link to DELETE") If the data file you want to load involves only DELETE operations, you must add the `__op` field and specify the operation type as DELETE. ###### Data examples[​](#data-examples-1 "Direct link to Data examples") 1. Prepare a data file. a. Create a CSV file named `example2.csv` in your local file system. The file consists of three columns, which represent user ID, user name, and user score in sequence. ```plain 101,Jack,100 ``` b. Publish the data of `example2.csv` to `topic2` of your Kafka cluster. 2. Prepare a StarRocks table. a. Create a Primary Key table named `table2` in your StarRocks table `test_db`. The table consists of three columns: `id`, `name`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `table2` ( `id` int(11) NOT NULL COMMENT "user ID", `name` varchar(65533) NOT NULL COMMENT "user name", `score` int(11) NOT NULL COMMENT "user score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` > **NOTE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). b. Insert two records into `table2`. ```sql INSERT INTO table2 VALUES (101, 'Jack', 100), (102, 'Bob', 90); ``` ###### Load data[​](#load-data-1 "Direct link to Load data") Run a load job to delete the record whose `id` is `101` in `example2.csv` from `table2`. * Run a Stream Load job. ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "label:label3" \ -H "column_separator:," \ -H "columns:__op='delete'" \ -T example2.csv -XPUT \ http://:/api/test_db/table2/_stream_load ``` * Run a Broker Load job. ```sql LOAD LABEL test_db.label3 ( data infile("hdfs://:/example2.csv") into table table2 columns terminated by "," format as "csv" set (__op = 'delete') ) WITH BROKER; ``` * Run a Routine Load job. ```sql CREATE ROUTINE LOAD test_db.table2 ON table2 COLUMNS(id, name, score, __op = 'delete') PROPERTIES ( "desired_concurrent_number" = "3", "max_batch_interval" = "20", "max_batch_rows"= "250000", "max_error_number" = "1000" ) FROM KAFKA ( "kafka_broker_list" =":", "kafka_topic" = "test2", "property.kafka_default_offsets" ="OFFSET_BEGINNING" ); ``` ###### Query data[​](#query-data-1 "Direct link to Query data") After the load is complete, query the data of `table2` to verify that the load is successful: ```sql SELECT * FROM table2; +------+------+-------+ | id | name | score | +------+------+-------+ | 102 | Bob | 90 | +------+------+-------+ 1 row in set (0.00 sec) ``` As shown in the preceding query result, the record whose `id` is `101` in `example2.csv` has been deleted from `table2`. ##### UPSERT and DELETE[​](#upsert-and-delete "Direct link to UPSERT and DELETE") If the data file you want to load involves both UPSERT and DELETE operations, you must add the `__op` field and make sure that the data file contains a column whose values are `0` or `1`. A value of `0` indicates an UPSERT operation, and a value of `1` indicates a DELETE operation. ###### Data examples[​](#data-examples-2 "Direct link to Data examples") 1. Prepare a data file. a. Create a CSV file named `example3.csv` in your local file system. The file consists of four columns, which represent user ID, user name, user score, and operation type in sequence. ```plain 101,Tom,100,1 102,Sam,70,0 103,Stan,80,0 ``` b. Publish the data of `example3.csv` to `topic3` of your Kafka cluster. 2. Prepare a StarRocks table. a. Create a Primary Key table named `table3` in your StarRocks database `test_db`. The table consists of three columns: `id`, `name`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `table3` ( `id` int(11) NOT NULL COMMENT "user ID", `name` varchar(65533) NOT NULL COMMENT "user name", `score` int(11) NOT NULL COMMENT "user score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` > **NOTE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). b. Insert two records into `table3`. ```sql INSERT INTO table3 VALUES (101, 'Tom', 100), (102, 'Sam', 90); ``` ###### Load data[​](#load-data-2 "Direct link to Load data") Run a load job to delete the record whose `id` is `101` in `example3.csv` from `table3`, update the record whose `id` is `102` in `example3.csv` to `table3`, and insert the record whose `id` is `103` in `example3.csv` into `table3`. * Run a Stream Load job: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "label:label4" \ -H "column_separator:," \ -H "columns: id, name, score, temp, __op = temp" \ -T example3.csv -XPUT \ http://:/api/test_db/table3/_stream_load ``` > **NOTE** > > In the preceding example, the fourth column that represents the operation type in `example3.csv` is temporarily named as `temp` and the `__op` field is mapped onto the `temp` column by using the `columns` parameter. As such, StarRocks can decide whether to perform an UPSERT or DELETE operation depending on the value in the fourth column of `example3.csv` is `0` or `1`. * Run a Broker Load job: ```bash LOAD LABEL test_db.label4 ( data infile("hdfs://:/example1.csv") into table table1 columns terminated by "," format as "csv" (id, name, score, temp) set (__op=temp) ) WITH BROKER; ``` * Run a Routine Load job: ```sql CREATE ROUTINE LOAD test_db.table3 ON table3 COLUMNS(id, name, score, temp, __op = temp) PROPERTIES ( "desired_concurrent_number" = "3", "max_batch_interval" = "20", "max_batch_rows"= "250000", "max_error_number" = "1000" ) FROM KAFKA ( "kafka_broker_list" = ":", "kafka_topic" = "test3", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` ###### Query data[​](#query-data-2 "Direct link to Query data") After the load is complete, query the data of `table3` to verify that the load is successful: ```sql SELECT * FROM table3; +------+------+-------+ | id | name | score | +------+------+-------+ | 102 | Sam | 70 | | 103 | Stan | 80 | +------+------+-------+ 2 rows in set (0.01 sec) ``` As shown in the preceding query result, the record whose `id` is `101` in `example3.csv` has been deleted from `table3`, the record whose `id` is `102` in `example3.csv` has been updated to `table3`, and the record whose `id` is `103` in `example3.csv` has been inserted into `table3`. #### Partial updates[​](#partial-updates "Direct link to Partial updates") Primary Key tables also support partial updates, and provide two modes of partial updates, row mode and column mode, for different data update scenarios. These two modes of partial updates can minimize the overhead of partial updates as much as possible while guaranteeing query performance, ensuring real-time updates. Row mode is more suitable for real-time update scenarios involving many columns and small batches. Column mode is suitable for batch processing update scenarios involving a few columns and a large number of rows. > **NOTICE** > > When you perform a partial update, if the row to be updated does not exist, StarRocks inserts a new row, and fills default values in fields that are empty because no data updates are inserted into them. This section uses CSV as an example to describe how to perform partial updates. ##### Data examples[​](#data-examples-3 "Direct link to Data examples") 1. Prepare a data file. a. Create a CSV file named `example4.csv` in your local file system. The file consists of two columns, which represent user ID and user name in sequence. ```plain 101,Lily 102,Rose 103,Alice ``` b. Publish the data of `example4.csv` to `topic4` of your Kafka cluster. 2. Prepare a StarRocks table. a. Create a Primary Key table named `table4` in your StarRocks database `test_db`. The table consists of three columns: `id`, `name`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `table4` ( `id` int(11) NOT NULL COMMENT "user ID", `name` varchar(65533) NOT NULL COMMENT "user name", `score` int(11) NOT NULL COMMENT "user score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` > **NOTE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). b. Insert a record into `table4`. ```sql INSERT INTO table4 VALUES (101, 'Tom',80); ``` ##### Load data[​](#load-data-3 "Direct link to Load data") Run a load to update the data in the two columns of `example4.csv` to the `id` and `name` columns of `table4`. * Run a Stream Load job: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "label:label7" -H "column_separator:," \ -H "partial_update:true" \ -H "columns:id,name" \ -T example4.csv -XPUT \ http://:/api/test_db/table4/_stream_load ``` > **NOTE** > > If you choose Stream Load, you must set the `partial_update` parameter to `true` to enable the partial update feature. The default is partial updates in row mode. If you need to perform partial updates in column mode, you need to set `partial_update_mode` to `column`. Additionally, you must use the `columns` parameter to specify the columns you want to update. * Run a Broker Load job: ```sql LOAD LABEL test_db.table4 ( data infile("hdfs://:/example4.csv") into table table4 format as "csv" (id, name) ) WITH BROKER PROPERTIES ( "partial_update" = "true" ); ``` > **NOTE** > > If you choose Broker Load, you must set the `partial_update` parameter to `true` to enable the partial update feature. The default is partial updates in row mode. If you need to perform partial updates in column mode, you need to set `partial_update_mode` to `column`. Additionally, you must use the `column_list` parameter to specify the columns you want to update. * Run a Routine Load job: ```sql CREATE ROUTINE LOAD test_db.table4 on table4 COLUMNS (id, name), COLUMNS TERMINATED BY ',' PROPERTIES ( "partial_update" = "true" ) FROM KAFKA ( "kafka_broker_list" =":", "kafka_topic" = "test4", "property.kafka_default_offsets" ="OFFSET_BEGINNING" ); ``` > **NOTE** > > * If you choose Routine Load, you must set the `partial_update` parameter to `true` to enable the partial update feature. Additionally, you must use the `COLUMNS` parameter to specify the columns you want to update. > * Routine Load only supports partial updates in row modes and does not support partial updates in column mode. ##### Query data[​](#query-data-3 "Direct link to Query data") After the load is complete, query the data of `table4` to verify that the load is successful: ```sql SELECT * FROM table4; +------+-------+-------+ | id | name | score | +------+-------+-------+ | 102 | Rose | 0 | | 101 | Lily | 80 | | 103 | Alice | 0 | +------+-------+-------+ 3 rows in set (0.01 sec) ``` As shown in the preceding query result, the record whose `id` is `101` in `example4.csv` has been updated to `table4`, and the records whose `id` are `102` and `103` in `example4.csv` have been Inserted into `table4`. #### Conditional updates[​](#conditional-updates "Direct link to Conditional updates") From StarRocks v2.5 onwards, Primary Key tables support conditional updates. You can specify a non-primary key column as the condition to determine whether updates can take effect. As such, the update from a source record to a destination record takes effect only when the source data record has a greater or equal value than the destination data record in the specified column. The conditional update feature is designed to resolve data disorder. If the source data is disordered, you can use this feature to ensure that new data will not be overwritten by old data. > **NOTICE** > > * You cannot specify different columns as update conditions for the same batch of data. > * DELETE operations do not support conditional updates. > * In versions earlier than v3.1.3, partial updates and conditional updates cannot be used simultaneously. From v3.1.3 onwards, StarRocks supports using partial updates with conditional updates. ##### Data examples[​](#data-examples-4 "Direct link to Data examples") 1. Prepare a data file. a. Create a CSV file named `example5.csv` in your local file system. The file consists of three columns, which represent user ID, version, and user score in sequence. ```plain 101,1,100 102,3,100 ``` b. Publish the data of `example5.csv` to `topic5` of your Kafka cluster. 2. Prepare a StarRocks table. a. Create a Primary Key table named `table5` in your StarRocks database `test_db`. The table consists of three columns: `id`, `version`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `table5` ( `id` int(11) NOT NULL COMMENT "user ID", `version` int NOT NULL COMMENT "version", `score` int(11) NOT NULL COMMENT "user score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` > **NOTE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). b. Insert a record into `table5`. ```sql INSERT INTO table5 VALUES (101, 2, 80), (102, 2, 90); ``` ##### Load data[​](#load-data-4 "Direct link to Load data") Run a load to update the records whose `id` values are `101` and `102`, respectively, from `example5.csv` into `table5`, and specify that the updates take effect only when the `version` value in each of the two records is greater or equal to their current `version` values. * Run a Stream Load job: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "label:label10" \ -H "column_separator:," \ -H "merge_condition:version" \ -T example5.csv -XPUT \ http://:/api/test_db/table5/_stream_load ``` * Run a Insert Load job: ```sql INSERT INTO test_db.table5 properties("merge_condition" = "version") VALUES (101, 2, 70), (102, 3, 100); ``` * Run a Routine Load job: ```sql CREATE ROUTINE LOAD test_db.table5 on table5 COLUMNS (id, version, score), COLUMNS TERMINATED BY ',' PROPERTIES ( "merge_condition" = "version" ) FROM KAFKA ( "kafka_broker_list" =":", "kafka_topic" = "topic5", "property.kafka_default_offsets" ="OFFSET_BEGINNING" ); ``` * Run a Broker Load job: ```sql LOAD LABEL test_db.table5 ( DATA INFILE ("s3://xxx.csv") INTO TABLE table5 COLUMNS TERMINATED BY "," FORMAT AS "CSV" ) WITH BROKER PROPERTIES ( "merge_condition" = "version" ); ``` ##### Query data[​](#query-data-4 "Direct link to Query data") After the load is complete, query the data of `table5` to verify that the load is successful: ```sql SELECT * FROM table5; +------+------+-------+ | id | version | score | +------+------+-------+ | 101 | 2 | 80 | | 102 | 3 | 100 | +------+------+-------+ 2 rows in set (0.02 sec) ``` As shown in the preceding query result, the record whose `id` is `101` in `example5.csv` is not updated to `table5`, and the record whose `id` is `102` in `example5.csv` has been Inserted into `table5`. --- ### Loading options Data loading is the process of cleansing and transforming raw data from various data sources based on your business requirements and loading the resulting data into StarRocks to facilitate analysis. StarRocks provides a variety of options for data loading: * Loading methods: Insert, Stream Load, Broker Load, Pipe, Routine Load, and Spark Load * Ecosystem tools: StarRocks Connector for Apache Kafka® (Kafka connector for short), StarRocks Connector for Apache Spark™ (Spark connector for short), StarRocks Connector for Apache Flink® (Flink connector for short), and other tools such as SMT, DataX, CloudCanal, and Kettle Connector * API: Stream Load transaction interface These options each have its own advantages and support its own set of data source systems to pull from. This topic provides an overview of these options, along with comparisons between them to help you determine the loading option of your choice based on your data source, business scenario, data volume, data file format, and loading frequency. #### Introduction to loading options[​](#introduction-to-loading-options "Direct link to Introduction to loading options") This section mainly describes the characteristics and business scenarios of the loading options available in StarRocks. ![Loading options overview](/assets/images/loading_intro_overview-01dc28ae3c1adabb8349da9fbfa0140f.png) note In the following sections, "batch" or "batch loading" refers to the loading of a large amount of data from a specified source all at a time into StarRocks, whereas "stream" or "streaming" refers to the continuous loading of data in real time. #### Loading methods[​](#loading-methods "Direct link to Loading methods") ##### [Insert](https://docs.starrocks.io/docs/loading/InsertInto.md)[​](#insert "Direct link to insert") **Business scenario:** * INSERT INTO VALUES: Append to an internal table with small amounts of data. * INSERT INTO SELECT: * INSERT INTO SELECT FROM ``: Append to a table with the result of a query on an internal or external table. * INSERT INTO SELECT FROM FILES(): Append to a table with the result of a query on data files in remote storage. note For AWS S3, this feature is supported from v3.1 onwards. For HDFS, Microsoft Azure Storage, Google GCS, and S3-compatible storage (such as MinIO), this feature is supported from v3.2 onwards. **File format:** * INSERT INTO VALUES: SQL * INSERT INTO SELECT: * INSERT INTO SELECT FROM ``: StarRocks tables * INSERT INTO SELECT FROM FILES(): Parquet and ORC **Data volume:** Not fixed (The data volume varies based on the memory size.) ##### [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md)[​](#stream-load "Direct link to stream-load") **Business scenario:** Batch load data from a local file system. **File format:** CSV and JSON **Data volume:** 10 GB or less ##### [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md)[​](#broker-load "Direct link to broker-load") **Business scenario:** * Batch load data from HDFS or cloud storage like AWS S3, Microsoft Azure Storage, Google GCS, and S3-compatible storage (such as MinIO). * Batch load data from a local file system or NAS. **File format:** CSV, Parquet, ORC, and JSON (supported since v3.2.3) **Data volume:** Dozens of GB to hundreds of GB ##### [Pipe](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md)[​](#pipe "Direct link to pipe") **Business scenario:** Batch load or stream data from HDFS or AWS S3. note This loading method is supported from v3.2 onwards. **File format:** Parquet and ORC **Data volume:** 100 GB to 1 TB or more ##### [Routine Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md)[​](#routine-load "Direct link to routine-load") **Business scenario:** Stream data from Kafka. **File format:** CSV, JSON, and Avro (supported since v3.0.1) **Data volume:** MBs to GBs of data as mini-batches ##### [Spark Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SPARK_LOAD.md)[​](#spark-load "Direct link to spark-load") **Business scenario:** Batch load data of Apache Hive™ tables stored in HDFS by using Spark clusters. **File format:** CSV, Parquet (supported since v2.0), and ORC (supported since v2.0) **Data volume:** Dozens of GB to TBs #### Ecosystem tools[​](#ecosystem-tools "Direct link to Ecosystem tools") ##### [Kafka connector](https://docs.starrocks.io/docs/loading/kafka/Kafka-connector-starrocks.md)[​](#kafka-connector "Direct link to kafka-connector") **Business scenario:** Stream data from Kafka. ##### [Spark connector](https://docs.starrocks.io/docs/loading/spark/Spark-connector-starrocks.md)[​](#spark-connector "Direct link to spark-connector") **Business scenario:** Batch load data from Spark. ##### [Flink connector](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks.md)[​](#flink-connector "Direct link to flink-connector") **Business scenario:** Stream data from Flink. ##### [SMT](https://docs.starrocks.io/docs/integrations/loading_tools/SMT.md)[​](#smt "Direct link to smt") **Business scenario:** Load data from data sources such as MySQL, PostgreSQL, SQL Server, Oracle, Hive, ClickHouse, and TiDB through Flink. ##### [DataX](https://docs.starrocks.io/docs/integrations/loading_tools/DataX-starrocks-writer.md)[​](#datax "Direct link to datax") **Business scenario:** Synchronize data between various heterogeneous data sources, including relational databases (for example, MySQL and Oracle), HDFS, and Hive. ##### [CloudCanal](https://docs.starrocks.io/docs/integrations/loading_tools/CloudCanal.md)[​](#cloudcanal "Direct link to cloudcanal") **Business scenario:** Migrate or synchronize data from source databases (for example, MySQL, Oracle, and PostgreSQL) to StarRocks. ##### [Kettle Connector](https://github.com/StarRocks/starrocks-connector-for-kettle)[​](#kettle-connector "Direct link to kettle-connector") **Business scenario:** Integrate with Kettle. By combining Kettle's robust data processing and transformation capabilities with StarRocks's high-performance data storage and analytical abilities, more flexible and efficient data processing workflows can be achieved. #### API[​](#api "Direct link to API") ##### [Stream Load transaction interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md)[​](#stream-load-transaction-interface "Direct link to stream-load-transaction-interface") **Business scenario:** Implement two-phase commit (2PC) for transactions that are run to load data from external systems such as Flink and Kafka, while improving the performance of highly concurrent stream loads. This feature is supported from v2.4 onwards. **File format:** CSV and JSON **Data volume:** 10 GB or less #### Choice of loading options[​](#choice-of-loading-options "Direct link to Choice of loading options") This section lists the loading options available for common data sources, helping you choose the option that best suits your situation. ##### Object storage[​](#object-storage "Direct link to Object storage") | **Data source** | **Available loading options** | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | AWS S3 | - (Batch) INSERT INTO SELECT FROM FILES() (supported since v3.1)
- (Batch) Broker Load
- (Batch or streaming) Pipe (supported since v3.2)See [Load data from AWS S3](https://docs.starrocks.io/docs/loading/objectstorage/s3.md). | | Microsoft Azure Storage | - (Batch) INSERT INTO SELECT FROM FILES() (supported since v3.2)
- (Batch) Broker LoadSee [Load data from Microsoft Azure Storage](https://docs.starrocks.io/docs/loading/objectstorage/azure.md). | | Google GCS | - (Batch) INSERT INTO SELECT FROM FILES() (supported since v3.2)
- (Batch) Broker LoadSee [Load data from GCS](https://docs.starrocks.io/docs/loading/objectstorage/gcs.md). | | S3-compatible storage (such as MinIO) | - (Batch) INSERT INTO SELECT FROM FILES() (supported since v3.2)
- (Batch) Broker LoadSee [Load data from MinIO](https://docs.starrocks.io/docs/loading/objectstorage/minio.md). | ##### Local file system (including NAS)[​](#local-file-system-including-nas "Direct link to Local file system (including NAS)") | **Data source** | **Available loading options** | | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Local file system (including NAS) | - (Batch) Stream Load
- (Batch) Broker LoadSee [Load data from a local file system](https://docs.starrocks.io/docs/loading/StreamLoad.md). | ##### HDFS[​](#hdfs "Direct link to HDFS") | **Data source** | **Available loading options** | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | HDFS | - (Batch) INSERT INTO SELECT FROM FILES() (supported since v3.2)
- (Batch) Broker Load
- (Batch or streaming) Pipe (supported since v3.2)See [Load data from HDFS](https://docs.starrocks.io/docs/loading/hdfs_load.md). | ##### Flink, Kafka, and Spark[​](#flink-kafka-and-spark "Direct link to Flink, Kafka, and Spark") | **Data source** | **Available loading options** | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Apache Flink® | - [Flink connector](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks.md)
- [Stream Load transaction interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) | | Apache Kafka® | - (Streaming) [Kafka connector](https://docs.starrocks.io/docs/loading/kafka/Kafka-connector-starrocks.md)
- (Streaming) [Routine Load](https://docs.starrocks.io/docs/loading/kafka/RoutineLoad.md)
- [Stream Load transaction interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) **NOTE**
If the source data requires multi-table joins and extract, transform and load (ETL) operations, you can use Flink to read and pre-process the data and then use [Flink connector](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks.md) to load the data into StarRocks. | | Apache Spark™ | - [Spark connector](https://docs.starrocks.io/docs/loading/spark/Spark-connector-starrocks.md)
- [Spark Load](https://docs.starrocks.io/docs/loading/spark/SparkLoad.md) | ##### Data lakes[​](#data-lakes "Direct link to Data lakes") | **Data source** | **Available loading options** | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Apache Hive™ | - (Batch) Create a [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md) and then use [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table).
- (Batch) [Spark Load](https://docs.starrocks.io/docs/loading/SparkLoad/). | | Apache Iceberg | (Batch) Create an [Iceberg catalog](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md) and then use [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table). | | Apache Hudi | (Batch) Create a [Hudi catalog](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md) and then use [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table). | | Delta Lake | (Batch) Create a [Delta Lake catalog](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md) and then use [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table). | | Elasticsearch | (Batch) Create an [Elasticsearch catalog](https://docs.starrocks.io/docs/data_source/catalog/elasticsearch_catalog.md) and then use [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table). | | Apache Paimon | (Batch) Create a [Paimon catalog](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog.md) and then use [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table). | Note that StarRocks provides [unified catalogs](https://docs.starrocks.io/docs/data_source/catalog/unified_catalog/) from v3.2 onwards to help you handle tables from Hive, Iceberg, Hudi, and Delta Lake data sources as a unified data source without ingestion. ##### Internal and external databases[​](#internal-and-external-databases "Direct link to Internal and external databases") | **Data source** | **Available loading options** | | ---------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | StarRocks | (Batch) Create a [StarRocks external table](https://docs.starrocks.io/docs/data_source/External_table.md#starrocks-external-table) and then use [INSERT INTO VALUES](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-via-insert-into-values) to insert a few data records or [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table) to insert the data of a table.
**NOTE**
StarRocks external tables only support data writes. They do not support data reads. | | MySQL | - (Batch) Create a [JDBC catalog](https://docs.starrocks.io/docs/data_source/catalog/jdbc_catalog.md) (recommended) or a [MySQL external table](https://docs.starrocks.io/docs/data_source/External_table.md#deprecated-mysql-external-table) and then use [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table).
- (Streaming) Use [SMT, Flink CDC connector, Flink, and Flink connector](https://docs.starrocks.io/docs/loading/Flink_cdc_load.md). | | Other databases such as Oracle, PostgreSQL, SQL Server, ClickHouse, and TiDB | - (Batch) Create a [JDBC catalog](https://docs.starrocks.io/docs/data_source/catalog/jdbc_catalog.md) (recommended) or a [JDBC external table](https://docs.starrocks.io/docs/data_source/External_table.md#external-table-for-a-jdbc-compatible-database) and then use [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table).
- (Streaming) Use [SMT, Flink CDC connector, Flink, and Flink connector](https://docs.starrocks.io/docs/loading/loading_tools.md). | --- ### Feature Support: Data Loading and Unloading This document outlines the features of various data loading and unloading methods supported by StarRocks. #### File format[​](#file-format "Direct link to File format") ##### Loading file formats[​](#loading-file-formats "Direct link to Loading file formats") | | Data Source | File Format | | | | | | | | -------------------- | -------------------------------------------- | ----------- | --------------- | --------------- | --------------- | ---------------- | --------------- | --------------- | | | | CSV | JSON \[3] | Parquet | ORC | Avro | ProtoBuf | Thrift | | Stream Load | Local file systems, applications, connectors | Yes | Yes | To be supported | To be supported | To be supported | | | | INSERT from FILES | HDFS, S3, OSS, Azure, GCS, NFS(NAS) \[5] | Yes (v3.3+) | To be supported | Yes (v3.1+) | Yes (v3.1+) | Yes (v3.4.4+) | To be supported | | | Broker Load | | Yes | Yes (v3.2.3+) | Yes | Yes | To be supported | | | | Routine Load | Kafka | Yes | Yes | To be supported | To be supported | Yes (v3.0+) \[1] | To be supported | To be supported | | Spark Load | | Yes | To be supported | Yes | Yes | To be supported | | | | Connectors | Flink, Spark | Yes | Yes | To be supported | To be supported | To be supported | | | | Kafka Connector \[2] | Kafka | Yes (v3.0+) | | To be supported | To be supported | Yes (v3.0+) | | To be supported | | PIPE \[4] | Consistent with INSERT from FILES | | | | | | | | note \[1], \[2]: Schema Registry is required. \[3]: JSON supports a variety of CDC formats. For details about the JSON CDC formats supported by StarRocks, see [JSON CDC format](#json-cdc-formats). \[4]: Currently, only INSERT from FILES is supported for loading with PIPE. \[5]: You need to mount a NAS device as NFS under the same directory of each BE or CN node to access the files in NFS via the `file://` protocol. ###### JSON CDC formats[​](#json-cdc-formats "Direct link to JSON CDC formats") | | Stream Load | Routine Load | Broker Load | INSERT from FILES | Kafka Connector \[1] | | -------- | --------------- | --------------- | --------------- | ----------------- | -------------------- | | Debezium | To be supported | To be supported | To be supported | To be supported | Yes (v3.0+) | | Canal | To be supported | | | | | | Maxwell | | | | | | note \[1]: You must configure the `transforms` parameter while loading Debezium CDC format data into Primary Key tables in StarRocks. ##### Unloading file formats[​](#unloading-file-formats "Direct link to Unloading file formats") | | Target | | File format | | | | | ------------------- | -------------------- | ---------------------------------------- | --------------- | --------------- | --------------- | --------------- | | | Table format | Remote storage | CSV | JSON | Parquet | ORC | | INSERT INTO FILES | N/A | HDFS, S3, OSS, Azure, GCS, NFS(NAS) \[3] | Yes (v3.3+) | To be supported | Yes (v3.2+) | Yes (v3.3+) | | INSERT INTO Catalog | Hive | HDFS, S3, OSS, Azure, GCS | Yes (v3.3+) | To be supported | Yes (v3.2+) | Yes (v3.3+) | | | Iceberg | HDFS, S3, OSS, Azure, GCS | To be supported | To be supported | Yes (v3.2+) | To be supported | | | Hudi/Delta | | To be supported | | | | | EXPORT | N/A | HDFS, S3, OSS, Azure, GCS | Yes \[1] | To be supported | To be supported | To be supported | | PIPE | To be supported \[2] | | | | | | note \[1]: Configuring Broker process is supported. \[2]: Currently, unloading data using PIPE is not supported. \[3]: You need to mount a NAS device as NFS under the same directory of each BE or CN node to access the files in NFS via the `file://` protocol. #### File format-related parameters[​](#file-format-related-parameters "Direct link to File format-related parameters") ##### Loading file format-related parameters[​](#loading-file-format-related-parameters "Direct link to Loading file format-related parameters") | File format | Parameter | Loading method | | | | | | ----------- | ------------------- | -------------- | ----------------- | ---------------- | ---------------- | --------------- | | | | Stream Load | INSERT from FILES | Broker Load | Routine Load | Spark Load | | CSV | column\_separator | Yes | Yes (v3.3+) | Yes \[1] | | | | | row\_delimiter | Yes | | Yes \[2] (v3.1+) | Yes \[3] (v2.2+) | To be supported | | | enclose | Yes (v3.0+) | | Yes (v3.0+) | Yes (v3.0+) | To be supported | | | escape | | | | | | | | skip\_header | | To be supported | | | | | | trim\_space | | Yes (v3.0+) | | | | | JSON | jsonpaths | Yes | To be supported | Yes (v3.2.3+) | Yes | To be supported | | | strip\_outer\_array | | | | | | | | json\_root | | | | | | | | ignore\_json\_size | | | | To be supported | | note \[1]: The corresponding parameter is `COLUMNS TERMINATED BY`. \[2]: The corresponding parameter is `ROWS TERMINATED BY`. \[3]: The corresponding parameter is `ROWS TERMINATED BY`. ##### Unloading file format-related parameters[​](#unloading-file-format-related-parameters "Direct link to Unloading file format-related parameters") | File format | Parameter | Unloading method | | | ----------- | -------------------- | ----------------- | ------ | | | | INSERT INTO FILES | EXPORT | | CSV | column\_separator | Yes (v3.3+) | Yes | | | line\_delimiter \[1] | | | | | enclose | Yes | No | | | escape | Yes | No | | | include\_header | Yes | No | note \[1]: The corresponding parameter in data loading is `row_delimiter`. #### Compression formats[​](#compression-formats "Direct link to Compression formats") ##### Loading compression formats[​](#loading-compression-formats "Direct link to Loading compression formats") | File format | Compression format | Loading method | | | | | | ----------- | ---------------------------------------------------------------- | ------------------ | --------------- | ----------------- | --------------- | --------------- | | | | Stream Load | Broker Load | INSERT from FILES | Routine Load | Spark Load | | CSV | - deflate
- bzip2
- gzip
- lz4\_frame
- zstd | Yes \[1] | Yes \[2] | To be supported | To be supported | To be supported | | JSON | | Yes (v3.2.7+) \[3] | To be supported | N/A | To be supported | N/A | | Parquet | - gzip
- lz4
- snappy
- zlib
- zstd | N/A | Yes \[4] | | To be supported | Yes \[4] | | ORC | | | | | | | note \[1]: Currently, only when loading CSV files with Stream Load can you specify the compression format by using `format=gzip`, indicating gzip-compressed CSV files. `deflate` and `bzip2` formats are also supported. \[2]: Broker Load does not support specifying the compression format of CSV files by using the parameter `format`. Broker Load identifies the compression format by using the suffix of the file. The suffix of gzip-compressed files is `.gz`, and that of the zstd-compressed files is `.zst`. Besides, other `format`-related parameters, such as `trim_space` and `enclose`, are also not supported. \[3]: Supports specifying the compression format by using `compression = gzip`. \[4]: Supported by Arrow Library. You do not need to configure the `compression` parameter. ##### Unloading compression formats[​](#unloading-compression-formats "Direct link to Unloading compression formats") | File format | Compression format | Unloading method | | | | | | ----------- | ---------------------------------------------------------------- | ----------------- | ------------------- | --------------- | --------------- | --------------- | | | | INSERT INTO FILES | INSERT INTO Catalog | | | EXPORT | | | | | Hive | Iceberg | Hudi/Delta | | | CSV | - deflate
- bzip2
- gzip
- lz4\_frame
- zstd | To be supported | To be supported | To be supported | To be supported | To be supported | | JSON | N/A | N/A | N/A | N/A | N/A | N/A | | Parquet | - gzip
- lz4
- snappy
- zstd | Yes (v3.2+) | Yes (v3.2+) | Yes (v3.2+) | To be supported | N/A | | ORC | | | | | | | #### Credentials[​](#credentials "Direct link to Credentials") ##### Loading - Authentication[​](#loading---authentication "Direct link to Loading - Authentication") | Authentication | Loading method | | | | | | -------------------------------------------------------- | -------------- | ------------------------------------------- | ------------------------------------- | ------------------ | ---------------------- | | | Stream Load | INSERT from FILES | Broker Load | Routine Load | External Catalog | | Single Kerberos | N/A | Yes (v3.1+) | Yes \[1] (versions earlier than v2.5) | Yes \[2] (v3.1.4+) | Yes | | Kerberos Ticket Granting Ticket (TGT) | N/A | To be supported | | | Yes (v3.1.10+/v3.2.1+) | | Single KDC Multiple Kerberos | N/A | | | | | | Basic access authentications (Access Key pair, IAM Role) | N/A | Yes (HDFS and S3-compatible object storage) | | Yes \[3] | Yes | note \[1]: For HDFS, StarRocks supports both simple authentication and Kerberos authentication. \[2]: When the security protocol is set to `sasl_plaintext` or `sasl_ssl`, both SASL and GSSAPI (Kerberos) authentications are supported. \[3]: When the security protocol is set to `sasl_plaintext` or `sasl_ssl`, both SASL and PLAIN authentications are supported. ##### Unloading - Authentication[​](#unloading---authentication "Direct link to Unloading - Authentication") | | INSERT INTO FILES | EXPORT | | --------------- | ----------------- | --------------- | | Single Kerberos | To be supported | To be supported | #### Loading - Other parameters and features[​](#loading---other-parameters-and-features "Direct link to Loading - Other parameters and features") | Parameter and feature | Loading method | | | | | | | | | -------------------------------------------- | -------------- | ----------------- | ------------------------- | ---------------------- | --------------- | --------------- | --------------- | - | | | Stream Load | INSERT from FILES | INSERT from SELECT/VALUES | Broker Load | PIPE | Routine Load | Spark Load | | | partial\_update | Yes (v3.0+) | Yes \[1] (v3.3+) | | Yes (v3.0+) | N/A | Yes (v3.0+) | To be supported | | | partial\_update\_mode | Yes (v3.1+) | To be supported | | Yes (v3.1+) | N/A | To be supported | To be supported | | | COLUMNS FROM PATH | N/A | Yes (v3.2+) | N/A | Yes | N/A | N/A | Yes | | | timezone or session variable time\_zone \[2] | Yes \[3] | Yes \[4] | Yes \[4] | Yes \[4] | To be supported | Yes \[4] | To be supported | | | Time accuracy - Microsecond | Yes | Yes | Yes | Yes (v3.1.11+/v3.2.6+) | To be supported | Yes | Yes | | note \[1]: From v3.3 onwards, StarRocks supports Partial Updates in Row mode for INSERT INTO by specifying the column list. \[2]: Setting the time zone by the parameter or the session variable will affect the results returned by functions such as strftime(), alignment\_timestamp(), and from\_unixtime(). \[3]: Only the parameter `timezone` is supported. \[4]: Only the session variable `time_zone` is supported. #### Unloading - Other parameters and features[​](#unloading---other-parameters-and-features "Direct link to Unloading - Other parameters and features") | Parameter and feature | INSERT INTO FILES | EXPORT | | --------------------------- | ----------------- | --------------- | | target\_max\_file\_size | Yes (v3.2+) | To be supported | | single | | | | Partitioned\_by | | | | Session variable time\_zone | To be supported | | | Time accuracy - Microsecond | To be supported | To be supported | --- ### Loading concepts This topic introduces common concepts and information about data loading. #### Privileges[​](#privileges "Direct link to Privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. #### Labeling[​](#labeling "Direct link to Labeling") You can load data into StarRocks by running load jobs. Each load job has a unique label that is specified by the user or automatically generated by StarRocks to identify the job. Each label can be used only for one load job. After a load job is complete, its label cannot be reused for any other load jobs. Only the labels of failed load jobs can be reused. #### Atomicity[​](#atomicity "Direct link to Atomicity") All the loading methods provided by StarRocks guarantee atomicity. Atomicity means that the qualified data within a load job must be all successfully loaded or none of the qualified data is successfully loaded. It never happens that some of the qualified data is loaded while the other data is not. Note that the qualified data does not include the data that is filtered out due to quality issues such as data type conversion errors. #### Protocols[​](#protocols "Direct link to Protocols") StarRocks supports two communication protocols that can be used to submit load jobs: MySQL and HTTP. Of all the loading methods supported by StarRocks, only Stream Load uses HTTP, whereas all the others use MySQL. #### Data types[​](#data-types "Direct link to Data types") StarRocks supports loading data of all data types. You only need to take note of the limits on the loading of a few specific data types. For more information, see [Data types](https://docs.starrocks.io/docs/sql-reference/data-types). #### Strict mode[​](#strict-mode "Direct link to Strict mode") Strict mode is an optional property that you can configure for data loads. It affects the loading behavior and the final loaded data. For details, see [Strict mode](https://docs.starrocks.io/docs/loading/strict_mode.md). #### Loading modes[​](#loading-modes "Direct link to Loading modes") StarRocks supports two loading modes: synchronous loading mode and asynchronous loading mode. note If you load data by using external programs, you must choose a loading mode that best suits your business requirements before you decide the loading method of your choice. ##### Synchronous loading[​](#synchronous-loading "Direct link to Synchronous loading") In synchronous loading mode, after you submit a load job, StarRocks synchronously runs the job to load data, and returns the result of the job after the job finishes. You can check whether the job is successful based on the job result. StarRocks provides two loading methods that support synchronous loading: [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) and [INSERT](https://docs.starrocks.io/docs/loading/InsertInto.md). The process of synchronous loading is as follows: 1. Create a load job. 2. View the job result returned by StarRocks. 3. Check whether the job is successful based on the job result. If the job result indicates a load failure, you can retry the job. ##### Asynchronous loading[​](#asynchronous-loading "Direct link to Asynchronous loading") In asynchronous loading mode, after you submit a load job, StarRocks immediately returns the job creation result. * If the result indicates a job creation success, StarRocks asynchronously runs the job. However, that does not mean that the data has been successfully loaded. You must use statements or commands to check the status of the job. Then, you can determine whether the data is successfully loaded based on the job status. * If the result indicates a job creation failure, you can determine whether you need to retry the job based on the failure information. tip You can set different write quorum for tables, that is, how many replicas are required to return loading success before StarRocks can determine the loading task is successful. You can specify write quorum by adding the property `write_quorum` when you [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md), or add this property to an existing table using [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md). StarRocks provides four loading methods that support asynchronous loading: [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md), [Pipe](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md), [Routine Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md), and [Spark Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SPARK_LOAD.md). The process of asynchronous loading is as follows: 1. Create a load job. 2. View the job creation result returned by StarRocks and determine whether the job is successfully created. * If the job creation succeeds, go to Step 3. * If the job creation fails, return to Step 1. 3. Use statements or commands to check the status of the job until the job status shows **FINISHED** or **CANCELLED**. ###### Workflow of Broker Load or Spark Load[​](#workflow-of-broker-load-or-spark-load "Direct link to Workflow of Broker Load or Spark Load") The workflow of a Broker Load or Spark Load job consists of five stages, as shown in the following figure. ![Broker Load or Spark Load overflow](/assets/images/4.1-1-5e37ffe979abb28c1b29aed2870659f7.png) The workflow is described as follows: 1. **PENDING** The job is in queue waiting to be scheduled by an FE. 2. **ETL** The FE pre-processes the data, including cleansing, partitioning, sorting, and aggregation. Only a Spark Load job has the ETL stage. A Broker Load job skips this stage. 3. **LOADING** The FE cleanses and transforms the data, and then sends the data to the BEs or CNs. After all data is loaded, the data is in queue waiting to take effect. At this time, the status of the job remains **LOADING**. 4. **FINISHED** When loading finishes and all data involved takes effect, the status of the job becomes **FINISHED**. At this time, the data can be queried. **FINISHED** is a final job state. 5. **CANCELLED** Before the status of the job becomes **FINISHED**, you can cancel the job at any time. Additionally, StarRocks can automatically cancel the job in case of load errors. After the job is canceled, the status of the job becomes **CANCELLED**, and all data updates made before the cancellation are reverted. **CANCELLED** is also a final job state. ###### Workflow of Pipe[​](#workflow-of-pipe "Direct link to Workflow of Pipe") The workflow of a Pipe job is described as follows: 1. The job is submitted to an FE from a MySQL client. 2. The FE splits the data files stored in the specified path based on their number or size, breaking down job into smaller, sequential tasks. The tasks enter a queue, waiting to be scheduled, after they are created. 3. The FE obtains the tasks from the queue, and invokes the INSERT INTO SELECT FROM FILES statement to execute each task. 4. The data loading finishes: * If `"AUTO_INGEST" = "FALSE"` is specified for the job at job creation, the job finishes after the data of all the data files stored in the specified path is loaded. * If `"AUTO_INGEST" = "TRUE"` is specified for the job at job creation, the FE will continue to monitor changes to the data files and automatically loads new or updated data from the data files into the destination StarRocks table. ###### Workflow of Routine Load[​](#workflow-of-routine-load "Direct link to Workflow of Routine Load") The workflow of a Routine Load job is described as follows: 1. The job is submitted to an FE from a MySQL client. 2. The FE splits the job into multiple tasks. Each task is engineered to load data from multiple partitions. 3. The FE distributes the tasks to specified BEs or CNs. 4. The BEs or CNs execute the tasks, and report to the FE after they finish the tasks. 5. The FE generates subsequent tasks, retries failed tasks if there are any, or suspends task scheduling based on the reports from the BEs. --- ### Considerations This topic describes some system limits and configurations that you need to consider before you run data loads. #### Memory limits[​](#memory-limits "Direct link to Memory limits") StarRocks provides parameters for you to limit the memory usage for each load job, thereby reducing memory consumption, especially in high concurrency scenarios. However, do not specify an excessively low memory usage limit. If the memory usage limit is excessively low, data may be frequently flushed from memory to disk because the memory usage for load jobs reaches the specified limit. We recommend that you specify a proper memory usage limit based on your business scenario. The parameters that are used to limit memory usage vary for each loading method. For more information, see [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md), [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md), [Routine Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md), [Spark Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SPARK_LOAD.md), and [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). Note that a load job usually runs on multiple BEs or CNs. Therefore, the parameters limit the memory usage of each load job on each involved BE or CN rather than the total memory usage of the load job on all involved BEs or CNs. StarRocks also provides parameters for you to limit the total memory usage of all load jobs that run on each individual BE or CN. For more information, see the "[System configurations](#system-configurations)" section below. #### System configurations[​](#system-configurations "Direct link to System configurations") This section describes some parameter configurations that are applicable to all of the loading methods provided by StarRocks. ##### FE configurations[​](#fe-configurations "Direct link to FE configurations") You can configure the following parameters in the configuration file **fe.conf** of each FE: * `max_load_timeout_second` and `min_load_timeout_second` These parameters specify the maximum timeout period and minimum timeout period of each load job. The timeout periods are measured in seconds. The default maximum timeout period spans 3 days, and the default minimum timeout period spans 1 second. The maximum timeout period and minimum timeout period that you specify must fall within the range of 1 second to 3 days. These parameters are valid for both synchronous load jobs and asynchronous load jobs. * `desired_max_waiting_jobs` This parameter specifies the maximum number of load jobs that can be held waiting in queue. The default value is **1024** (100 in v2.4 and earlier, and 1024 in v2.5 and later). When the number of load jobs in the **PENDING** state on an FE reaches the maximum number that you specify, the FE rejects new load requests. This parameter is valid only for asynchronous load jobs. * `max_running_txn_num_per_db` This parameter specifies the maximum number of ongoing load transactions that are allowed in each database of your StarRocks cluster. A load job can contain one or more transactions. The default value is **100**. When the number of load transactions running in a database reaches the maximum number that you specify, the subsequent load jobs that you submit are not scheduled. In this situation, if you submit a synchronous load job, the job is rejected. If you submit an asynchronous load job, the job is held waiting in queue. note StarRocks counts all load jobs together and does not distinguish between synchronous load jobs and asynchronous load jobs. * `label_keep_max_second` This parameter specifies the retention period of the history records for load jobs that have finished and are in the **FINISHED** or **CANCELLED** state. The default retention period spans 3 days. This parameter is valid for both synchronous load jobs and asynchronous load jobs. ##### BE/CN configurations[​](#becn-configurations "Direct link to BE/CN configurations") You can configure the following parameters in the configuration file **be.conf** of each BE or the configuration file **cn.conf** of each CN: * `write_buffer_size` This parameter specifies the maximum memory block size. The default size is 100 MB. The loaded data is first written to a memory block on the BE or CN. When the amount of data that is loaded reaches the maximum memory block size that you specify, the data is flushed to disk. You must specify a proper maximum memory block size based on your business scenario. * If the maximum memory block size is exceedingly small, a large number of small files may be generated on the BE or CN. In this case, query performance degrades. You can increase the maximum memory block size to reduce the number of files generated. * If the maximum memory block size is exceedingly large, remote procedure calls (RPCs) may time out. In this case, you can adjust the value of this parameter based on your business needs. * `streaming_load_rpc_max_alive_time_sec` The waiting timeout period for each Writer process. The default value is 1200 seconds. During the data loading process, StarRocks starts a Writer process to receive data from and write data to each tablet. If a Writer process does not receive any data within the waiting timeout period that you specify, StarRocks stops the Writer process. When your StarRocks cluster processes data at low speeds, a Writer process may not receive the next batch of data within a long period of time and therefore reports a "TabletWriter add batch with unknown id" error. In this case, you can increase the value of this parameter. * `load_process_max_memory_limit_bytes` and `load_process_max_memory_limit_percent` These parameters specify the maximum amount of memory that can be consumed for all load jobs on each individual BE or CN. StarRocks identifies the smaller memory consumption among the values of the two parameters as the final memory consumption that is allowed. * `load_process_max_memory_limit_bytes`: specifies the maximum memory size. The default maximum memory size is 100 GB. * `load_process_max_memory_limit_percent`: specifies the maximum memory usage. The default value is 30%. This parameter differs from the `mem_limit` parameter. The `mem_limit` parameter specifies the total maximum memory usage of your StarRocks cluster, and the default value is 90% x 90%. If the memory capacity of the machine on which the BE or CN resides is M, the maximum amount of memory that can be consumed for load jobs is calculated as follows: `M x 90% x 90% x 30%`. ##### System variable configurations[​](#system-variable-configurations "Direct link to System variable configurations") You can configure the following [system variable](https://docs.starrocks.io/docs/sql-reference/System_variable.md): * `insert_timeout` The INSERT timeout duration. Unit: seconds. Value range: `1` to `259200`. Default value: `14400`. This variable will act on all operations involving INSERT jobs (for example, UPDATE, DELETE, CTAS, materialized view refresh, statistics collection, and PIPE) in the current connection. --- ### Troubleshooting Data Loading This guide is designed to help DBAs and operation engineers monitor the status of data load jobs through SQL interfaces—without relying on external monitoring systems. It also provides guidance on identifying performance bottlenecks and troubleshooting anomalies during load operations. #### Terminology[​](#terminology "Direct link to Terminology") **Load Job:** A continuous data load process, such as a **Routine Load Job** or **Pipe Job**. **Load Task:** A one-time data load process, usually corresponding to a single load transaction. Examples include **Broker Load**, **Stream Load**, **Spark Load**, and **INSERT INTO**. Routine Load jobs and Pipe jobs continuously generate tasks to perform data ingestion. #### Observe load jobs[​](#observe-load-jobs "Direct link to Observe load jobs") There are two ways to observe load jobs: * Using SQL statements **[SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md)** and **[SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md)**. * Using system views **[information\_schema.routine\_load\_jobs](https://docs.starrocks.io/docs/sql-reference/information_schema/routine_load_jobs.md)** and **[information\_schema.pipes](https://docs.starrocks.io/docs/sql-reference/information_schema/pipes.md)**. #### Observe load tasks[​](#observe-load-tasks "Direct link to Observe load tasks") Load tasks can also be monitored in two ways: * Using SQL statements **[SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md)** and **[SHOW ROUTINE LOAD TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD_TASK.md)**. * Using system views **[information\_schema.loads](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md)** and **statistics.loads\_history**. ##### SQL statements[​](#sql-statements "Direct link to SQL statements") The **SHOW** statements display both ongoing and recently completed load tasks for the current database, providing a quick overview of task status. The information retrieved is a subset of the **statistics.loads\_history** system view. SHOW LOAD statements return information of Broker Load, Insert Into, and Spark Load tasks, and SHOW ROUTINE LOAD TASK statements return Routine Load task information. ##### System views[​](#system-views "Direct link to System views") ###### information\_schema.loads[​](#information_schemaloads "Direct link to information_schema.loads") The **information\_schema.loads** system view stores information about recent load tasks, including active and recently completed ones. StarRocks periodically synchronizes the data to the **statistics.loads\_history** system table for persistent storage. **information\_schema.loads** provides the following fields: | Field | Description | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ID | Globally unique identifier. | | LABEL | Label of the load job. | | PROFILE\_ID | The ID of the Profile, which can be analyzed via `ANALYZE PROFILE`. | | DB\_NAME | The database to which the target table belongs. | | TABLE\_NAME | The target table. | | USER | The user who initiates the load job. | | WAREHOUSE | The warehouse to which the load job belongs. | | STATE | The state of the load job. Valid values:- `PENDING`/`BEGIN`: The load job is created.
- `QUEUEING`/`BEFORE_LOAD`: The load job is in the queue waiting to be scheduled.
- `LOADING`: The load job is running.
- `PREPARING`: The transaction is being pre-committed.
- `PREPARED`: The transaction has been pre-committed.
- `COMMITED`: The transaction has been committed.
- `FINISHED`: The load job succeeded.
- `CANCELLED`: The load job failed. | | PROGRESS | The progress of the ETL stage and LOADING stage of the load job. | | TYPE | The type of the load job. For Broker Load, the return value is `BROKER`. For INSERT, the return value is `INSERT`. For Stream Load, the return value is `STREAM`. For Routine Load Load, the return value is `ROUTINE`. | | PRIORITY | The priority of the load job. Valid values: `HIGHEST`, `HIGH`, `NORMAL`, `LOW`, and `LOWEST`. | | SCAN\_ROWS | The number of data rows that are scanned. | | SCAN\_BYTES | The number of bytes that are scanned. | | FILTERED\_ROWS | The number of data rows that are filtered out due to inadequate data quality. | | UNSELECTED\_ROWS | The number of data rows that are filtered out due to the conditions specified in the WHERE clause. | | SINK\_ROWS | The number of data rows that are loaded. | | RUNTIME\_DETAILS | Load runtime metadata. For details, see [RUNTIME\_DETAILS](#runtime_details). | | CREATE\_TIME | The time at which the load job was created. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | LOAD\_START\_TIME | The start time of the LOADING stage of the load job. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | LOAD\_COMMIT\_TIME | The time at which the loading transaction was committed. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | LOAD\_FINISH\_TIME | The end time of the LOADING stage of the load job. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | PROPERTIES | The static properties of the load job. For details, see [PROPERTIES](#properties). | | ERROR\_MSG | The error message of the load job. If the load job did not encounter any error, `NULL` is returned. | | TRACKING\_SQL | The SQL statement that can be used to query the tracking log of the load job. A SQL statement is returned only when the load job involves unqualified data rows. If the load job does not involve any unqualified data rows, `NULL` is returned. | | REJECTED\_RECORD\_PATH | The path from which you can access all the unqualified data rows that are filtered out in the load job. The number of unqualified data rows logged is determined by the `log_rejected_record_num` parameter configured in the load job. You can use the `wget` command to access the path. If the load job does not involve any unqualified data rows, `NULL` is returned. | ###### RUNTIME\_DETAILS[​](#runtime_details "Direct link to RUNTIME_DETAILS") * Universal metrics: | Metric | Description | | -------- | ---------------------------------------------- | | load\_id | Globally unique ID of the load execution plan. | | txn\_id | Load transaction ID. | * Specific metrics for Broker Load, INSERT INTO, and Spark Load: | Metric | Description | | -------------------- | ----------------------------------------------------------------------------------------------------------------- | | etl\_info | ETL Details. This field is only valid for Spark Load jobs. For other types of load jobs, the value will be empty. | | etl\_start\_time | The start time of the ETL stage of the load job. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | etl\_start\_time | The end time of the ETL stage of the load job. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | unfinished\_backends | List of BEs with unfinished executions. | | backends | List of BEs participating in execution. | | file\_num | Number of files read. | | file\_size | Total size of files read. | | task\_num | Number of subtasks. | * Specific metrics for Routine Load: | Metric | Description | | --------------------- | ----------------------------------------------------------------------------------- | | schedule\_interval | The interval for Routine Load to be scheduled. | | wait\_slot\_time | Time elapsed while the Routine Load task waits for execution slots. | | check\_offset\_time | Time consumed when checking offset information during Routine Load task scheduling. | | consume\_time | Time consumed by the Routine Load task to read upstream data. | | plan\_time | Time for generating the execution plan. | | commit\_publish\_time | Time consumed to execute the COMMIT RPC. | * Specific metrics for Stream Load: | Metric | Description | | ------------------------- | ---------------------------------------- | | timeout | Timeout for load tasks. | | begin\_txn\_ms | Time consumed to begin the transaction. | | plan\_time\_ms | Time for generating the execution plan. | | receive\_data\_time\_ms | Time for receiving data. | | commit\_publish\_time\_ms | Time consumed to execute the COMMIT RPC. | | client\_ip | Client IP address. | ###### PROPERTIES[​](#properties "Direct link to PROPERTIES") * Specific properties for Broker Load, INSERT INTO, and Spark Load: | Property | Description | | ------------------ | -------------------------------------------------------------------------------- | | timeout | Timeout for load tasks. | | max\_filter\_ratio | Maximum ratio of data rows that are filtered out due to inadequate data quality. | * Specific properties for Routine Load: | Property | Description | | --------- | ------------------------------------------------- | | job\_name | Routine Load job name. | | task\_num | Number of subtasks actually executed in parallel. | | timeout | Timeout for load tasks. | ###### statistics.loads\_history[​](#statisticsloads_history "Direct link to statistics.loads_history") The **statistics.loads\_history** system view stores load records for the last three months by default. DBAs can adjust the retention period by modifying the `partition_ttl` of the view. **statistics.loads\_history** has the consistent schema with **information\_schema.loads**. #### Identify loading performance issues with Load Profiles[​](#identify-loading-performance-issues-with-load-profiles "Direct link to Identify loading performance issues with Load Profiles") A **Load Profile** records execution details of all worker nodes involved in a data load. It helps you quickly pinpoint performance bottlenecks in the StarRocks cluster. ##### Enable Load Profiles[​](#enable-load-profiles "Direct link to Enable Load Profiles") StarRocks provides multiple methods to enable Load Profiles, depending on the type of load: ###### For Broker Load and INSERT INTO[​](#for-broker-load-and-insert-into "Direct link to For Broker Load and INSERT INTO") Enable Load Profiles for Broker Load and INSERT INTO at session level: ```sql SET enable_profile = true; ``` By default, profiles are automatically enabled for long-running jobs (longer than 300 seconds). You can customize this threshold by: ```sql SET big_query_profile_threshold = 60s; ``` note When `big_query_profile_threshold` is set to its default value `0`, the default behavior is to disable Query Profiling for queries. However, for load tasks, profiles are automatically recorded for tasks with execution times exceeding 300 seconds. StarRocks also supports **Runtime Profiles**, which periodically (every 30 seconds) report execution metrics of long-running load jobs. You can customize the report interval by: ```sql SET runtime_profile_report_interval = 60; ``` note `runtime_profile_report_interval` specifies only the minimum report interval for load tasks. The actual report interval is dynamically adjusted and may exceed this value. ###### For Stream Load and Routine Load[​](#for-stream-load-and-routine-load "Direct link to For Stream Load and Routine Load") Enable Load Profiles for Stream Load and Routine Load at table level: ```sql ALTER TABLE SET ("enable_load_profile" = "true"); ``` Stream Load typically has high QPS, so StarRocks allows sampling for Load Profile collection to avoid performance degradation from extensive profiling. You can adjust the collection interval by configuring the FE parameter `load_profile_collect_interval_second`. This setting only applies to Load Profiles enabled via table properties. The default value is `0`. ```sql ADMIN SET FRONTEND CONFIG ("load_profile_collect_interval_second"="30"); ``` StarRocks also allows collecting profiles only from load jobs that exceed a certain time threshold. You can adjust this threshold by configuring the FE parameter `stream_load_profile_collect_threshold_second`. The default value is `0`. ```sql ADMIN SET FRONTEND CONFIG ("stream_load_profile_collect_threshold_second"="10"); ``` ##### Analyze Load Profiles[​](#analyze-load-profiles "Direct link to Analyze Load Profiles") The structure of Load Profiles is identical to that of Query Profiles. For detailed instructions, see [Query Tuning Recipes](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_tuning_recipes.md). You can analyze Load Profiles by executing [ANALYZE PROFILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/ANALYZE_PROFILE.md). For detailed instructions, see [Analyze text-based Profiles](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_text_based_analysis.md). Profiles provide detailed operator metrics. Key components include the `OlapTableSink` operator and the `LoadChannel` operator. ###### OlapTableSink operator[​](#olaptablesink-operator "Direct link to OlapTableSink operator") | Metric | Description | | ----------------- | --------------------------------------------------------------------------------------------------------------- | | IndexNum | Number of synchronous materialized views of the target table. | | ReplicatedStorage | Whether single leader replication is enabled. | | TxnID | Load transaction ID. | | RowsRead | Number of data rows read from the upstream operator. | | RowsFiltered | The number of data rows that are filtered out due to inadequate data quality. | | RowsReturned | The number of data rows that are loaded. | | RpcClientSideTime | Total time consumed for data writing RPC from client-side statistics. | | RpcServerSideTime | Total time consumed for data writing RPC from server-side statistics. | | PrepareDataTime | Time consumed for data format conversion and data quality checks. | | SendDataTime | Local time consumed for sending data, including data serialization, compression, and writing to the send queue. | tip * The significant variance between the maximum and minimum values of `PushChunkNum` in `OLAP_TABLE_SINK` indicates data skew in the upstream operator, which may cause write performance bottlenecks. * `RpcClientSideTime` equals the sum of `RpcServerSideTime`, Network transmission time, and RPC framework processing time. If the difference between `RpcClientSideTime` and `RpcServerSideTime` is significant, consider to enable data compression to reduce transmission time. * If `RpcServerSideTime` accounts for a significant portion of the time spent, further analysis can be conducted using `LoadChannel` Profile. ###### LoadChannel operator[​](#loadchannel-operator "Direct link to LoadChannel operator") | Metric | Description | | ------------------- | -------------------------------------------------------------------------------------------------------- | | Address | IP address or FQDN of the BE node. | | LoadMemoryLimit | Memory Limit for loading. | | PeakMemoryUsage | Peak memory usage for loading. | | OpenCount | The number of times the channel is opened, reflecting the sink's total concurrency. | | OpenTime | Total time consumed for the opening channel. | | AddChunkCount | Number of loading chunks, that is, the number of calls to `TabletsChannel::add_chunk`. | | AddRowNum | The number of data rows that are loaded. | | AddChunkTime | Total time consumed by loading chunks, that is, the total execution time of `TabletsChannel::add_chunk`. | | WaitFlushTime | Total time spent by `TabletsChannel::add_chunk` waiting for MemTable flush. | | WaitWriterTime | Total time spent by `TabletsChannel::add_chunk` waiting for Async Delta Writer execution. | | WaitReplicaTime | Total time spent by `TabletsChannel::add_chunk` waiting for synchronization from replicas. | | PrimaryTabletsNum | Number of primary tablets. | | SecondaryTabletsNum | Number of secondary tablets. | tip If `WaitFlushTime` takes an extended period, it may indicate insufficient resources for the flush thread. Consider adjusting the BE configuration `flush_thread_num_per_store`. #### Best practices[​](#best-practices "Direct link to Best practices") ##### Diagnose Broker Load performance bottleneck[​](#diagnose-broker-load-performance-bottleneck "Direct link to Diagnose Broker Load performance bottleneck") 1. Load data using Broker Load: ```sql LOAD LABEL click_bench.hits_1713874468 ( DATA INFILE ("s3://test-data/benchmark_data/query_data/click_bench/hits.tbl*") INTO TABLE hits COLUMNS TERMINATED BY "\t" (WatchID,JavaEnable,Title,GoodEvent,EventTime,EventDate,CounterID,ClientIP,RegionID,UserID,CounterClass,OS,UserAgent,URL,Referer,IsRefresh,RefererCategoryID,RefererRegionID,URLCategoryID,URLRegionID,ResolutionWidth,ResolutionHeight,ResolutionDepth,FlashMajor,FlashMinor,FlashMinor2,NetMajor,NetMinor,UserAgentMajor,UserAgentMinor,CookieEnable,JavascriptEnable,IsMobile,MobilePhone,MobilePhoneModel,Params,IPNetworkID,TraficSourceID,SearchEngineID,SearchPhrase,AdvEngineID,IsArtifical,WindowClientWidth,WindowClientHeight,ClientTimeZone,ClientEventTime,SilverlightVersion1,SilverlightVersion2,SilverlightVersion3,SilverlightVersion4,PageCharset,CodeVersion,IsLink,IsDownload,IsNotBounce,FUniqID,OriginalURL,HID,IsOldCounter,IsEvent,IsParameter,DontCountHits,WithHash,HitColor,LocalEventTime,Age,Sex,Income,Interests,Robotness,RemoteIP,WindowName,OpenerName,HistoryLength,BrowserLanguage,BrowserCountry,SocialNetwork,SocialAction,HTTPError,SendTiming,DNSTiming,ConnectTiming,ResponseStartTiming,ResponseEndTiming,FetchTiming,SocialSourceNetworkID,SocialSourcePage,ParamPrice,ParamOrderID,ParamCurrency,ParamCurrencyID,OpenstatServiceName,OpenstatCampaignID,OpenstatAdID,OpenstatSourceID,UTMSource,UTMMedium,UTMCampaign,UTMContent,UTMTerm,FromTag,HasGCLID,RefererHash,URLHash,CLID) ) WITH BROKER ( "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "" ) ``` 2. Use **SHOW PROFILELIST** to retrieve the list of runtime profiles. ```sql MySQL [click_bench]> SHOW PROFILELIST; +--------------------------------------+---------------------+----------+---------+----------------------------------------------------------------------------------------------------------------------------------+ | QueryId | StartTime | Time | State | Statement | +--------------------------------------+---------------------+----------+---------+----------------------------------------------------------------------------------------------------------------------------------+ | 3df61627-f82b-4776-b16a-6810279a79a3 | 2024-04-23 20:28:26 | 11s850ms | Running | LOAD LABEL click_bench.hits_1713875306 (DATA INFILE ("s3://test-data/benchmark_data/query_data/click_bench/hits.tbl*" ... | +--------------------------------------+---------------------+----------+---------+----------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) ``` 3. Use **ANALYZE PROFILE** to view the Runtime Profile. ```sql MySQL [click_bench]> ANALYZE PROFILE FROM '3df61627-f82b-4776-b16a-6810279a79a3'; +-------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Summary | | Attention: The transaction of the statement will be aborted, and no data will be actually inserted!!! | | Attention: Profile is not identical!!! | | QueryId: 3df61627-f82b-4776-b16a-6810279a79a3 | | Version: default_profile-70fe819 | | State: Running | | Legend: ⏳ for blocked; 🚀 for running; ✅ for finished | | TotalTime: 31s832ms | | ExecutionTime: 30s1ms [Scan: 28s885ms (96.28%), Network: 0ns (0.00%), ResultDeliverTime: 7s613ms (25.38%), ScheduleTime: 145.701ms (0.49%)] | | FrontendProfileMergeTime: 3.838ms | | QueryPeakMemoryUsage: 141.367 MB, QueryAllocatedMemoryUsage: 82.422 GB | | Top Most Time-consuming Nodes: | | 1. FILE_SCAN (id=0) 🚀 : 28s902ms (85.43%) | | 2. OLAP_TABLE_SINK 🚀 : 4s930ms (14.57%) | | Top Most Memory-consuming Nodes: | | Progress (finished operator/all operator): 0.00% | | NonDefaultVariables: | | big_query_profile_threshold: 0s -> 60s | | enable_adaptive_sink_dop: false -> true | | enable_profile: false -> true | | sql_mode_v2: 32 -> 34 | | use_compute_nodes: -1 -> 0 | | Fragment 0 | | │ BackendNum: 3 | | │ InstancePeakMemoryUsage: 128.541 MB, InstanceAllocatedMemoryUsage: 82.422 GB | | │ PrepareTime: 2.304ms | | └──OLAP_TABLE_SINK | | │ TotalTime: 4s930ms (14.57%) [CPUTime: 4s930ms] | | │ OutputRows: 14.823M (14823424) | | │ PartitionType: RANDOM | | │ Table: hits | | └──FILE_SCAN (id=0) 🚀 | | Estimates: [row: ?, cpu: ?, memory: ?, network: ?, cost: ?] | | TotalTime: 28s902ms (85.43%) [CPUTime: 17.038ms, ScanTime: 28s885ms] | | OutputRows: 14.823M (14823424) | | Progress (processed rows/total rows): ? | | Detail Timers: [ScanTime = IOTaskExecTime + IOTaskWaitTime] | | IOTaskExecTime: 25s612ms [min=19s376ms, max=28s804ms] | | IOTaskWaitTime: 63.192ms [min=20.946ms, max=91.668ms] | | | +-------------------------------------------------------------------------------------------------------------------------------------------------------------+ 40 rows in set (0.04 sec) ``` The profile shows that the `FILE_SCAN` section took nearly 29 seconds, accounting for approximately 90% of the total 32-second duration. This indicates that reading data from object storage is currently the bottleneck in the loading process. ##### Diagnose Stream Load Performance[​](#diagnose-stream-load-performance "Direct link to Diagnose Stream Load Performance") 1. Enable Load Profile for the target table. ```sql mysql> ALTER TABLE duplicate_200_column_sCH SET('enable_load_profile'='true'); Query OK, 0 rows affected (0.00 sec) ``` 2. Use **SHOW PROFILELIST** to retrieve the list of profiles. ```sql mysql> SHOW PROFILELIST; +--------------------------------------+---------------------+----------+----------+-----------+ | QueryId | StartTime | Time | State | Statement | +--------------------------------------+---------------------+----------+----------+-----------+ | 90481df8-afaf-c0fd-8e91-a7889c1746b6 | 2024-09-19 10:43:38 | 9s571ms | Finished | | | 9c41a13f-4d7b-2c18-4eaf-cdeea3facba5 | 2024-09-19 10:43:37 | 10s664ms | Finished | | | 5641cf37-0af4-f116-46c6-ca7cce149886 | 2024-09-19 10:43:20 | 13s88ms | Finished | | | 4446c8b3-4dc5-9faa-dccb-e1a71ab3519e | 2024-09-19 10:43:20 | 13s64ms | Finished | | | 48469b66-3866-1cd9-9f3b-17d786bb4fa7 | 2024-09-19 10:43:20 | 13s85ms | Finished | | | bc441907-e779-bc5a-be8e-992757e4d992 | 2024-09-19 10:43:19 | 845ms | Finished | | +--------------------------------------+---------------------+----------+----------+-----------+ ``` 3. Use **ANALYZE PROFILE** to view the Profile. ```sql mysql> ANALYZE PROFILE FROM '90481df8-afaf-c0fd-8e91-a7889c1746b6'; +-----------------------------------------------------------+ | Explain String | +-----------------------------------------------------------+ | Load: | | Summary: | | - Query ID: 90481df8-afaf-c0fd-8e91-a7889c1746b6 | | - Start Time: 2024-09-19 10:43:38 | | - End Time: 2024-09-19 10:43:48 | | - Query Type: Load | | - Load Type: STREAM_LOAD | | - Query State: Finished | | - StarRocks Version: main-d49cb08 | | - Sql Statement | | - Default Db: ingestion_db | | - NumLoadBytesTotal: 799008 | | - NumRowsAbnormal: 0 | | - NumRowsNormal: 280 | | - Total: 9s571ms | | - numRowsUnselected: 0 | | Execution: | | Fragment 0: | | - Address: 172.26.93.218:59498 | | - InstanceId: 90481df8-afaf-c0fd-8e91-a7889c1746b7 | | - TxnID: 1367 | | - ReplicatedStorage: true | | - AutomaticPartition: false | | - InstanceAllocatedMemoryUsage: 12.478 MB | | - InstanceDeallocatedMemoryUsage: 10.745 MB | | - InstancePeakMemoryUsage: 9.422 MB | | - MemoryLimit: -1.000 B | | - RowsProduced: 280 | | - AllocAutoIncrementTime: 348ns | | - AutomaticBucketSize: 0 | | - BytesRead: 0.000 B | | - CloseWaitTime: 9s504ms | | - IOTaskExecTime: 0ns | | - IOTaskWaitTime: 0ns | | - IndexNum: 1 | | - NumDiskAccess: 0 | | - OpenTime: 15.639ms | | - PeakMemoryUsage: 0.000 B | | - PrepareDataTime: 583.480us | | - ConvertChunkTime: 44.670us | | - ValidateDataTime: 109.333us | | - RowsFiltered: 0 | | - RowsRead: 0 | | - RowsReturned: 280 | | - RowsReturnedRate: 12.049K (12049) /sec | | - RpcClientSideTime: 28s396ms | | - RpcServerSideTime: 28s385ms | | - RpcServerWaitFlushTime: 0ns | | - ScanTime: 9.841ms | | - ScannerQueueCounter: 1 | | - ScannerQueueTime: 3.272us | | - ScannerThreadsInvoluntaryContextSwitches: 0 | | - ScannerThreadsTotalWallClockTime: 0ns | | - MaterializeTupleTime(*): 0ns | | - ScannerThreadsSysTime: 0ns | | - ScannerThreadsUserTime: 0ns | | - ScannerThreadsVoluntaryContextSwitches: 0 | | - SendDataTime: 2.452ms | | - PackChunkTime: 1.475ms | | - SendRpcTime: 1.617ms | | - CompressTime: 0ns | | - SerializeChunkTime: 880.424us | | - WaitResponseTime: 0ns | | - TotalRawReadTime(*): 0ns | | - TotalReadThroughput: 0.000 B/sec | | DataSource: | | - DataSourceType: FileDataSource | | - FileScanner: | | - CastChunkTime: 0ns | | - CreateChunkTime: 227.100us | | - FileReadCount: 3 | | - FileReadTime: 253.765us | | - FillTime: 6.892ms | | - MaterializeTime: 133.637us | | - ReadTime: 0ns | | - ScannerTotalTime: 9.292ms | +-----------------------------------------------------------+ 76 rows in set (0.00 sec) ``` #### Appendix[​](#appendix "Direct link to Appendix") ##### Useful SQL for Operations[​](#useful-sql-for-operations "Direct link to Useful SQL for Operations") note This section only applies to shared-nothing clusters. ###### query the throughput per minute[​](#query-the-throughput-per-minute "Direct link to query the throughput per minute") ```sql -- overall select date_trunc('minute', load_finish_time) as t,count(*) as tpm,sum(SCAN_BYTES) as scan_bytes,sum(sink_rows) as sink_rows from _statistics_.loads_history group by t order by t desc limit 10; -- table select date_trunc('minute', load_finish_time) as t,count(*) as tpm,sum(SCAN_BYTES) as scan_bytes,sum(sink_rows) as sink_rows from _statistics_.loads_history where table_name = 't' group by t order by t desc limit 10; ``` ###### Query RowsetNum and SegmentNum of a table[​](#query-rowsetnum-and-segmentnum-of-a-table "Direct link to Query RowsetNum and SegmentNum of a table") ```sql -- overall select * from information_schema.be_tablets t, information_schema.tables_config c where t.table_id = c.table_id order by num_segment desc limit 5; select * from information_schema.be_tablets t, information_schema.tables_config c where t.table_id = c.table_id order by num_rowset desc limit 5; -- table select * from information_schema.be_tablets t, information_schema.tables_config c where t.table_id = c.table_id and table_name = 't' order by num_segment desc limit 5; select * from information_schema.be_tablets t, information_schema.tables_config c where t.table_id = c.table_id and table_name = 't' order by num_rowset desc limit 5; ``` * High RowsetNum (>100) indicates too frequent loads. You may consider to reduce frequency or increase Compaction threads. * High SegmentNum (>100) indicates excessive segments per load. You may consider increase Compaction threads or adopt the random distribution strategy for the table. ###### Check data skew[​](#check-data-skew "Direct link to Check data skew") ###### Data skew across nodes[​](#data-skew-across-nodes "Direct link to Data skew across nodes") ```sql -- overall SELECT tbt.be_id, sum(tbt.DATA_SIZE) FROM information_schema.tables_config tb JOIN information_schema.be_tablets tbt ON tb.TABLE_ID = tbt.TABLE_ID group by be_id; -- table SELECT tbt.be_id, sum(tbt.DATA_SIZE) FROM information_schema.tables_config tb JOIN information_schema.be_tablets tbt ON tb.TABLE_ID = tbt.TABLE_ID WHERE tb.table_name = 't' group by be_id; ``` If you detected node-level skew, you may consider to use a higher-cardinality column as the distribution key or adopt the random distribution strategy for the table. ###### Data skew across tablets[​](#data-skew-across-tablets "Direct link to Data skew across tablets") ```sql select tablet_id,t.data_size,num_row,visible_version,num_version,num_rowset,num_segment,PARTITION_NAME from information_schema.partitions_meta m, information_schema.be_tablets t where t.partition_id = m.partition_id and m.partition_name = 'att' and m.table_name='att' order by t.data_size desc; ``` ##### Common monitoring metrics for loading[​](#common-monitoring-metrics-for-loading "Direct link to Common monitoring metrics for loading") ###### BE Load[​](#be-load "Direct link to BE Load") These metrics are available under the **BE Load** category in Grafana. If you cannot find this category, verify that you are using the [latest Grafana dashboard template](https://docs.starrocks.io/docs/administration/management/monitoring.md#125-configure-dashboard). ###### ThreadPool[​](#threadpool "Direct link to ThreadPool") These metrics help analyze the status of thread pools — for example, whether tasks are being backlogged, or how long they spend pending. Currently, there are four monitored thread pools: * `async_delta_writer` * `memtable_flush` * `segment_replicate_sync` * `segment_flush` Each thread pool includes the following metrics: | Name | Description | | ----------- | --------------------------------------------------------------------------------------------------------- | | **rate** | Task processing rate. | | **pending** | Time tasks spend waiting in the queue. | | **execute** | Task execution time. | | **total** | Maximum number of threads available in the pool. | | **util** | Pool utilization over a given period; due to sampling inaccuracy, it may exceed 100% when heavily loaded. | | **count** | Instantaneous number of tasks in the queue. | note * A reliable indicator for backlog is whether **pending duration** keeps increasing. **workers util** and **queue count** are necessary but not sufficient indicators. * If a backlog occurs, use **rate** and **execute duration** to determine whether it is due to increased load or slower processing. * **workers util** helps assess how busy the pool is, which can guide tuning efforts. ###### LoadChannel::add\_chunks[​](#loadchanneladd_chunks "Direct link to LoadChannel::add_chunks") These metrics help analyze the behavior of `LoadChannel::add_chunks` after receiving a `BRPC tablet_writer_add_chunks` request. | Name | Description | | ------------------ | --------------------------------------------------------------------------------------- | | **rate** | Processing rate of `add_chunks` requests. | | **execute** | Average execution time of `add_chunks`. | | **wait\_memtable** | Average wait time for the primary replica’s MemTable flush. | | **wait\_writer** | Average wait time for the primary replica’s async delta writer to perform write/commit. | | **wait\_replica** | Average wait time for secondary replicas to complete segment flush. | note * The **latency** metric equals the sum of `wait_memtable`, `wait_writer`, and `wait_replica`. * A high waiting ratio indicates downstream bottlenecks, which should be further analyzed. ###### Async Delta Writer[​](#async-delta-writer "Direct link to Async Delta Writer") These metrics help analyze the behavior of the **async delta writer**. | Name | Description | | ------------------ | ------------------------------------------------- | | **rate** | Processing rate of write/commit tasks. | | **pending** | Time spent waiting in the thread pool queue. | | **execute** | Average time to process a single task. | | **wait\_memtable** | Average time waiting for MemTable flush. | | **wait\_replica** | Average time waiting for segment synchronization. | note * The total time per task (from the upstream perspective) equals **pending** plus **execute**. * **execute** further includes **wait\_memtable** plus **wait\_replica**. * A high **pending** time may indicate that **execute** is slow or the thread pool is undersized. * If **wait** occupies a large portion of **execute**, downstream stages are the bottleneck; otherwise, the bottleneck is likely within the writer’s logic itself. ###### MemTable Flush[​](#memtable-flush "Direct link to MemTable Flush") These metrics analyze **MemTable flush** performance. | Name | Description | | --------------- | -------------------------------------------- | | **rate** | Flush rate of MemTables. | | **memory-size** | Amount of in-memory data flushed per second. | | **disk-size** | Amount of disk data written per second. | | **execute** | Task execution time. | | **io** | I/O time of the flush task. | note * By comparing **rate** and **size**, you can determine whether the workload is changing or if massive imports are occurring — for example, a small **rate** but large **size** indicates a massive import. * The compression ratio can be estimated using `memory-size / disk-size`. * You can also assess if I/O is the bottleneck by checking the proportion of **io** time in **execute**. ###### Segment Replicate Sync[​](#segment-replicate-sync "Direct link to Segment Replicate Sync") | Name | Description | | ----------- | -------------------------------------------- | | **rate** | Rate of segment synchronization. | | **execute** | Time to synchronize a single tablet replica. | ###### Segment Flush[​](#segment-flush "Direct link to Segment Flush") These metrics analyze **segment flush** performance. | Name | Description | | ----------- | --------------------------------------- | | **rate** | Segment flush rate. | | **size** | Amount of disk data flushed per second. | | **execute** | Task execution time. | | **io** | I/O time of the flush task. | note * By comparing **rate** and **size**, you can determine whether the workload is changing or if large imports are occurring — for example, a small **rate** but large **size** indicates a massive import. * You can also assess if I/O is the bottleneck by checking the proportion of **io** time in **execute**. --- ### Load data using tools StarRocks and its ecosystem partners offer the following tools to help you seamlessly integrate StarRocks with external databases. #### [SMT](https://docs.starrocks.io/docs/integrations/loading_tools/SMT.md)[​](#smt "Direct link to smt") SMT (StarRocks Migration Tool) is a data migration tool provided by StarRocks, designed to optimize complex data loading pipelines: source databases (such as MySQL, Oracle, PostgreSQL) ---> Flink ---> destination StarRocks clusters. Its main functions are as follows: * Simplifies table creation in StarRocks: Generates statements to create tables in StarRocks based on information from external databases and the target StarRocks cluster. * Simplifies the full or incremental data synchronization process in the data pipeline: Generates SQL statements that can be run in Flink's SQL client to submit Flink jobs for synchronizing data. The following flowchart illustrates the process of loading data from the source database MySQL through Flink into StarRocks. ![img](/assets/images/load_tools-8e88eab0e5c9d2a228fcd668c63bc3e1.png) #### [DataX](https://docs.starrocks.io/docs/integrations/loading_tools/DataX-starrocks-writer.md)[​](#datax "Direct link to datax") DataX is a tool for offline data synchronization, and is open-sourced by Alibaba. DataX can synchronize data between various heterogeneous data sources, including relational databases (MySQL, Oracle, etc.), HDFS, and Hive. DataX provides the StarRocks Writer plugin to synchronize data from data sources supported by DataX to StarRocks. #### [CloudCanal](https://docs.starrocks.io/docs/integrations/loading_tools/CloudCanal.md)[​](#cloudcanal "Direct link to cloudcanal") CloudCanal Community Edition is a free data migration and synchronization platform published by [ClouGence Co., Ltd](https://www.bladepipe.com/) that integrates Schema Migration, Full Data Migration, verification, Correction, and real-time Incremental Synchronization. You can directly add StarRocks as a data source in CloudCanal's visual interface and create tasks to automatically migrate or synchronize data from source databases (e.g., MySQL, Oracle, PostgreSQL) to StarRocks. #### [Kettle connector](https://github.com/StarRocks/starrocks-connector-for-kettle)[​](#kettle-connector "Direct link to kettle-connector") Kettle is an ETL (Extract, Transform, Load) tool with a visual graphical interface, which allows users to build data processing workflows by dragging components and configuring parameters. This intuitive method greatly simplifies the process of data processing and loading, enabling users to handle data more conveniently. Additionally, Kettle provides a rich library of components, allowing users to select suitable components according to their needs and perform various complex data processing tasks. StarRocks offers the Kettle Connector to integrate with Kettle. By combining Kettle's robust data processing and transformation capabilities with StarRocks's high-performance data storage and analytical abilities, more flexible and efficient data processing workflows can be achieved. --- ### alibaba --- ### Load data from Microsoft Azure Storage StarRocks provides the following options for loading data from Azure: * Synchronous loading using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md)+[`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) * Asynchronous loading using [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) Each of these options has its own advantages, which are detailed in the following sections. In most cases, we recommend that you use the INSERT+`FILES()` method, which is much easier to use. However, the INSERT+`FILES()` method currently supports only the Parquet, ORC, and CSV file formats. Therefore, if you need to load data of other file formats such as JSON, or [perform data changes such as DELETE during data loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md), you can resort to Broker Load. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Make source data ready[​](#make-source-data-ready "Direct link to Make source data ready") Make sure that the source data you want to load into StarRocks is properly stored in a container within your Azure storage account. In this topic, suppose you want to load the data of a Parquet-formatted sample dataset (`user_behavior_ten_million_rows.parquet`) stored in the root directory of a container (`starrocks-container`) within an Azure Data Lake Storage Gen2 (ADLS Gen2) storage account (`starrocks`). ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ##### Gather authentication details[​](#gather-authentication-details "Direct link to Gather authentication details") The examples in this topic use the Shared Key authentication method. To ensure that you have permission to read data from ADLS Gen2, we recommend that you read [Azure Data Lake Storage Gen2 > Shared Key (access key of storage account)](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_azure_storage.md#shared-key-access-key-of-storage-account-1) to understand the authentication parameters that you need to configure. In a nutshell, if you practice Shared Key authentication, you need to gather the following information: * The username of your ADLS Gen2 storage account * The shared key of your ADLS Gen2 storage account For information about all the authentication methods available, see [Authenticate to Azure cloud storage](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_azure_storage.md). #### Use INSERT+FILES()[​](#use-insertfiles "Direct link to Use INSERT+FILES()") This method is available from v3.2 onwards and currently supports only the Parquet, ORC, and CSV (from v3.3.0 onwards) file formats. ##### Advantages of INSERT+FILES()[​](#advantages-of-insertfiles "Direct link to Advantages of INSERT+FILES()") `FILES()` can read the file stored in cloud storage based on the path-related properties you specify, infer the table schema of the data in the file, and then return the data from the file as data rows. With `FILES()`, you can: * Query the data directly from Azure using [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). * Create and load a table using [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) (CTAS). * Load the data into an existing table using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). ##### Typical examples[​](#typical-examples "Direct link to Typical examples") ###### Querying directly from Azure using SELECT[​](#querying-directly-from-azure-using-select "Direct link to Querying directly from Azure using SELECT") Querying directly from Azure using SELECT+`FILES()` can give a good preview of the content of a dataset before you create a table. For example: * Get a preview of the dataset without storing the data. * Query for the min and max values and decide what data types to use. * Check for `NULL` values. The following example queries the sample dataset `user_behavior_ten_million_rows.parquet` stored in the container `starrocks-container` within your storage account `starrocks`: ```sql SELECT * FROM FILES ( "path" = "abfss://starrocks-container@starrocks.dfs.core.windows.net/user_behavior_ten_million_rows.parquet", "format" = "parquet", "azure.adls2.storage_account" = "starrocks", "azure.adls2.shared_key" = "xxxxxxxxxxxxxxxxxx" ) LIMIT 3; ``` The system returns a query result similar to the following: ```plain +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 543711 | 829192 | 2355072 | pv | 2017-11-27 08:22:37 | | 543711 | 2056618 | 3645362 | pv | 2017-11-27 10:16:46 | | 543711 | 1165492 | 3645362 | pv | 2017-11-27 10:17:00 | +--------+---------+------------+--------------+---------------------+ ``` > **NOTE** > > Notice that the column names as returned above are provided by the Parquet file. ###### Creating and loading a table using CTAS[​](#creating-and-loading-a-table-using-ctas "Direct link to Creating and loading a table using CTAS") This is a continuation of the previous example. The previous query is wrapped in CREATE TABLE AS SELECT (CTAS) to automate the table creation using schema inference. This means StarRocks will infer the table schema, create the table you want, and then load the data into the table. The column names and types are not required to create a table when using the `FILES()` table function with Parquet files as the Parquet format includes the column names. > **NOTE** > > The syntax of CREATE TABLE when using schema inference does not allow setting the number of replicas. If you are using a StarRocks shared-nothing cluster, set the number of replicas before creating the table. The example below is for a system with three replicas: > > ```sql > ADMIN SET FRONTEND CONFIG ('default_replication_num' = "3"); > > ``` Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Use CTAS to create a table and load the data of the sample dataset `user_behavior_ten_million_rows.parquet`, which is stored in the container `starrocks-container` within your storage account `starrocks`, into the table: ```sql CREATE TABLE user_behavior_inferred AS SELECT * FROM FILES ( "path" = "abfss://starrocks-container@starrocks.dfs.core.windows.net/user_behavior_ten_million_rows.parquet", "format" = "parquet", "azure.adls2.storage_account" = "starrocks", "azure.adls2.shared_key" = "xxxxxxxxxxxxxxxxxx" ); ``` After creating the table, you can view its schema by using [DESCRIBE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md): ```sql DESCRIBE user_behavior_inferred; ``` The system returns the following query result: ```plain +--------------+-----------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+-----------+------+-------+---------+-------+ | UserID | bigint | YES | true | NULL | | | ItemID | bigint | YES | true | NULL | | | CategoryID | bigint | YES | true | NULL | | | BehaviorType | varbinary | YES | false | NULL | | | Timestamp | varbinary | YES | false | NULL | | +--------------+-----------+------+-------+---------+-------+ ``` Query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_inferred LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plain +--------+--------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+--------+------------+--------------+---------------------+ | 84 | 162325 | 2939262 | pv | 2017-12-02 05:41:41 | | 84 | 232622 | 4148053 | pv | 2017-11-27 04:36:10 | | 84 | 595303 | 903809 | pv | 2017-11-26 08:03:59 | +--------+--------+------------+--------------+---------------------+ ``` ###### Loading into an existing table using INSERT[​](#loading-into-an-existing-table-using-insert "Direct link to Loading into an existing table using INSERT") You may want to customize the table that you are inserting into, for example, the: * column data type, nullable setting, or default values * key types and columns * data partitioning and bucketing > **NOTE** > > Creating the most efficient table structure requires knowledge of how the data will be used and the content of the columns. This topic does not cover table design. For information about table design, see [Table types](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md). In this example, we are creating a table based on knowledge of how the table will be queried and the data in the Parquet file. The knowledge of the data in the Parquet file can be gained by querying the file directly in Azure. * Since a query of the dataset in Azure indicates that the `Timestamp` column contains data that matches a VARBINARY data type, the column type is specified in the following DDL. * By querying the data in Azure, you can find that there are no `NULL` values in the dataset, so the DDL does not set any columns as nullable. * Based on knowledge of the expected query types, the sort key and bucketing column are set to the column `UserID`. Your use case might be different for this data, so you might decide to use `ItemID` in addition to or instead of `UserID` for the sort key. Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from Azure): ```sql CREATE TABLE user_behavior_declared ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` Display the schema so that you can compare it with the inferred schema produced by the `FILES()` table function: ```sql DESCRIBE user_behavior_declared; ``` ```plaintext +--------------+----------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+----------------+------+-------+---------+-------+ | UserID | int | NO | true | NULL | | | ItemID | int | NO | false | NULL | | | CategoryID | int | NO | false | NULL | | | BehaviorType | varchar(65533) | NO | false | NULL | | | Timestamp | varbinary | NO | false | NULL | | +--------------+----------------+------+-------+---------+-------+ 5 rows in set (0.00 sec) ``` tip Compare the schema you just created with the schema inferred earlier using the `FILES()` table function. Look at: * data types * nullable * key fields To better control the schema of the destination table and for better query performance, we recommend that you specify the table schema by hand in production environments. After creating the table, you can load it with INSERT INTO SELECT FROM FILES(): ```sql INSERT INTO user_behavior_declared SELECT * FROM FILES ( "path" = "abfss://starrocks-container@starrocks.dfs.core.windows.net/user_behavior_ten_million_rows.parquet", "format" = "parquet", "azure.adls2.storage_account" = "starrocks", "azure.adls2.shared_key" = "xxxxxxxxxxxxxxxxxx" ); ``` After the load is complete, you can query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_declared LIMIT 3; ``` The system returns a query result similar to the following, indicating that the data has been successfully loaded: ```plain +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 142 | 2869980 | 2939262 | pv | 2017-11-25 03:43:22 | | 142 | 2522236 | 1669167 | pv | 2017-11-25 15:14:12 | | 142 | 3031639 | 3607361 | pv | 2017-11-25 15:19:25 | +--------+---------+------------+--------------+---------------------+ ``` ###### Check load progress[​](#check-load-progress "Direct link to Check load progress") You can query the progress of INSERT jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. Example: ```sql SELECT * FROM information_schema.loads ORDER BY JOB_ID DESC; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'insert_f3fc2298-a553-11ee-92f4-00163e0842bd' \G *************************** 1. row *************************** JOB_ID: 10193 LABEL: insert_f3fc2298-a553-11ee-92f4-00163e0842bd DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: INSERT PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):300; max_filter_ratio:0.0 CREATE_TIME: 2023-12-28 15:37:38 ETL_START_TIME: 2023-12-28 15:37:38 ETL_FINISH_TIME: 2023-12-28 15:37:38 LOAD_START_TIME: 2023-12-28 15:37:38 LOAD_FINISH_TIME: 2023-12-28 15:39:35 JOB_DETAILS: {"All backends":{"f3fc2298-a553-11ee-92f4-00163e0842bd":[10120]},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":581730322,"InternalTableLoadRows":10000000,"ScanBytes":581574034,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"f3fc2298-a553-11ee-92f4-00163e0842bd":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` > **NOTE** > > INSERT is a synchronous command. If an INSERT job is still running, you need to open another session to check its execution status. #### Use Broker Load[​](#use-broker-load "Direct link to Use Broker Load") An asynchronous Broker Load process handles making the connection to Azure, pulling the data, and storing the data in StarRocks. This method supports the following file formats: * Parquet * ORC * CSV * JSON (supported from v3.2.3 onwards) ##### Advantages of Broker Load[​](#advantages-of-broker-load "Direct link to Advantages of Broker Load") * Broker Load runs in the background and clients don't need to stay connected for the job to continue. * Broker Load is preferred for long running jobs, the default timeout is 4 hours. * In addition to Parquet and ORC file format, Broker Load supports CSV file format and JSON file format (JSON file format is supported from v3.2.3 onwards). ##### Data flow[​](#data-flow "Direct link to Data flow") ![Workflow of Broker Load](/assets/images/broker_load_how-to-work_en-bb36de70866e6366b2b21808f0f77be8.png) 1. The user creates a load job. 2. The frontend (FE) creates a query plan and distributes the plan to the backend nodes (BEs) or compute nodes (CNs). 3. The BEs or CNs pull the data from the source and load the data into StarRocks. ##### Typical example[​](#typical-example "Direct link to Typical example") Create a table, start a load process that pulls the sample dataset `user_behavior_ten_million_rows.parquet` from Azure, and verify the progress and success of the data loading. ###### Create a database and a table[​](#create-a-database-and-a-table "Direct link to Create a database and a table") Connect to your StarRocks cluster. Then, create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from Azure): ```sql CREATE TABLE user_behavior ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Broker Load[​](#start-a-broker-load "Direct link to Start a Broker Load") Run the following command to start a Broker Load job that loads data from the sample dataset `user_behavior_ten_million_rows.parquet` to the `user_behavior` table: ```sql LOAD LABEL user_behavior ( DATA INFILE("abfss://starrocks-container@starrocks.dfs.core.windows.net/user_behavior_ten_million_rows.parquet") INTO TABLE user_behavior FORMAT AS "parquet" ) WITH BROKER ( "azure.adls2.storage_account" = "starrocks", "azure.adls2.shared_key" = "xxxxxxxxxxxxxxxxxx" ) PROPERTIES ( "timeout" = "3600" ); ``` This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. * `BROKER`: The connection details for the source. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Check load progress[​](#check-load-progress-1 "Direct link to Check load progress") You can query the progress of Broker Load jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. ```sql SELECT * FROM information_schema.loads \G ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'user_behavior' \G *************************** 1. row *************************** JOB_ID: 10250 LABEL: user_behavior DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: BROKER PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):3600; max_filter_ratio:0.0 CREATE_TIME: 2023-12-28 16:15:19 ETL_START_TIME: 2023-12-28 16:15:25 ETL_FINISH_TIME: 2023-12-28 16:15:25 LOAD_START_TIME: 2023-12-28 16:15:25 LOAD_FINISH_TIME: 2023-12-28 16:16:31 JOB_DETAILS: {"All backends":{"6a8ef4c0-1009-48c9-8d18-c4061d2255bf":[10121]},"FileNumber":1,"FileSize":132251298,"InternalTableLoadBytes":311710786,"InternalTableLoadRows":10000000,"ScanBytes":132251298,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"6a8ef4c0-1009-48c9-8d18-c4061d2255bf":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` After you confirm that the load job has finished, you can check a subset of the destination table to see if the data has been successfully loaded. Example: ```sql SELECT * from user_behavior LIMIT 3; ``` The system returns a query result similar to the following, indicating that the data has been successfully loaded: ```plain +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 142 | 2869980 | 2939262 | pv | 2017-11-25 03:43:22 | | 142 | 2522236 | 1669167 | pv | 2017-11-25 15:14:12 | | 142 | 3031639 | 3607361 | pv | 2017-11-25 15:19:25 | +--------+---------+------------+--------------+---------------------+ ``` --- ### Load data from GCS StarRocks provides the following options for loading data from GCS: * Synchronous loading using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md)+[`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) * Asynchronous loading using [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) Each of these options has its own advantages, which are detailed in the following sections. In most cases, we recommend that you use the INSERT+`FILES()` method, which is much easier to use. However, the INSERT+`FILES()` method currently supports only the Parquet, ORC, and CSV file formats. Therefore, if you need to load data of other file formats such as JSON, or [perform data changes such as DELETE during data loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md), you can resort to Broker Load. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Make source data ready[​](#make-source-data-ready "Direct link to Make source data ready") Make sure the source data you want to load into StarRocks is properly stored in a GCS bucket. You may also consider where the data and the database are located, because data transfer costs are much lower when your bucket and your StarRocks cluster are located in the same region. In this topic, we provide you with a sample dataset in a GCS bucket, `gs://starrocks-samples/user_behavior_ten_million_rows.parquet`. You can access that dataset with any valid credentials as the object is readable by any GCP user. ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ##### Gather authentication details[​](#gather-authentication-details "Direct link to Gather authentication details") The examples in this topic use service account-based authentication. To practice IAM user-based authentication, you need to gather information about the following GCS resources: * The GCS bucket that stores your data. * The GCS object key (object name) if accessing a specific object in the bucket. Note that the object key can include a prefix if your GCS objects are stored in sub-folders. * The GCS region to which the GCS bucket belongs. * The `private_ key_id`, `private_key`, and `client_email` of your Google Cloud service account For information about all the authentication methods available, see [Authenticate to Google Cloud Storage](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_gcs.md). #### Use INSERT+FILES()[​](#use-insertfiles "Direct link to Use INSERT+FILES()") This method is available from v3.2 onwards and currently supports only the Parquet, ORC, and CSV (from v3.3.0 onwards) file formats. ##### Advantages of INSERT+FILES()[​](#advantages-of-insertfiles "Direct link to Advantages of INSERT+FILES()") `FILES()` can read the file stored in cloud storage based on the path-related properties you specify, infer the table schema of the data in the file, and then return the data from the file as data rows. With `FILES()`, you can: * Query the data directly from GCS using [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). * Create and load a table using [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) (CTAS). * Load the data into an existing table using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). ##### Typical examples[​](#typical-examples "Direct link to Typical examples") ###### Querying directly from GCS using SELECT[​](#querying-directly-from-gcs-using-select "Direct link to Querying directly from GCS using SELECT") Querying directly from GCS using SELECT+`FILES()` can give a good preview of the content of a dataset before you create a table. For example: * Get a preview of the dataset without storing the data. * Query for the min and max values and decide what data types to use. * Check for `NULL` values. The following example queries the sample dataset `gs://starrocks-samples/user_behavior_ten_million_rows.parquet`: ```sql SELECT * FROM FILES ( "path" = "gs://starrocks-samples/user_behavior_ten_million_rows.parquet", "format" = "parquet", "gcp.gcs.service_account_email" = "sampledatareader@xxxxx-xxxxxx-000000.iam.gserviceaccount.com", "gcp.gcs.service_account_private_key_id" = "baaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "gcp.gcs.service_account_private_key" = "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----" ) LIMIT 3; ``` > **NOTE** > > Substitute the credentials in the above command with your own credentials. Any valid service account email, key, and secret can be used, as the object is readable by any GCP authenticated user. The system returns a query result similar to the following: ```plain +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 543711 | 829192 | 2355072 | pv | 2017-11-27 08:22:37 | | 543711 | 2056618 | 3645362 | pv | 2017-11-27 10:16:46 | | 543711 | 1165492 | 3645362 | pv | 2017-11-27 10:17:00 | +--------+---------+------------+--------------+---------------------+ ``` > **NOTE** > > Notice that the column names as returned above are provided by the Parquet file. ###### Creating and loading a table using CTAS[​](#creating-and-loading-a-table-using-ctas "Direct link to Creating and loading a table using CTAS") This is a continuation of the previous example. The previous query is wrapped in CREATE TABLE AS SELECT (CTAS) to automate the table creation using schema inference. This means StarRocks will infer the table schema, create the table you want, and then load the data into the table. The column names and types are not required to create a table when using the `FILES()` table function with Parquet files as the Parquet format includes the column names. > **NOTE** > > The syntax of CREATE TABLE when using schema inference does not allow setting the number of replicas. If you are using a StarRocks shared-nothing cluster, set the number of replicas before creating the table. The example below is for a system with three replicas: > > ```sql > ADMIN SET FRONTEND CONFIG ('default_replication_num' = "3"); > > ``` Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Use CTAS to create a table and load the data of the sample dataset `gs://starrocks-samples/user_behavior_ten_million_rows.parquet` into the table: ```sql CREATE TABLE user_behavior_inferred AS SELECT * FROM FILES ( "path" = "gs://starrocks-samples/user_behavior_ten_million_rows.parquet", "format" = "parquet", "gcp.gcs.service_account_email" = "sampledatareader@xxxxx-xxxxxx-000000.iam.gserviceaccount.com", "gcp.gcs.service_account_private_key_id" = "baaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "gcp.gcs.service_account_private_key" = "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----" ); ``` > **NOTE** > > Substitute the credentials in the above command with your own credentials. Any valid service account email, key, and secret can be used, as the object is readable by any GCP authenticated user. After creating the table, you can view its schema by using [DESCRIBE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md): ```sql DESCRIBE user_behavior_inferred; ``` The system returns a query result similar to the following: ```plain +--------------+-----------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+-----------+------+-------+---------+-------+ | UserID | bigint | YES | true | NULL | | | ItemID | bigint | YES | true | NULL | | | CategoryID | bigint | YES | true | NULL | | | BehaviorType | varbinary | YES | false | NULL | | | Timestamp | varbinary | YES | false | NULL | | +--------------+-----------+------+-------+---------+-------+ ``` Query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_inferred LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plain +--------+--------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+--------+------------+--------------+---------------------+ | 84 | 162325 | 2939262 | pv | 2017-12-02 05:41:41 | | 84 | 232622 | 4148053 | pv | 2017-11-27 04:36:10 | | 84 | 595303 | 903809 | pv | 2017-11-26 08:03:59 | +--------+--------+------------+--------------+---------------------+ ``` ###### Loading into an existing table using INSERT[​](#loading-into-an-existing-table-using-insert "Direct link to Loading into an existing table using INSERT") You may want to customize the table that you are inserting into, for example, the: * column data type, nullable setting, or default values * key types and columns * data partitioning and bucketing > **NOTE** > > Creating the most efficient table structure requires knowledge of how the data will be used and the content of the columns. This topic does not cover table design. For information about table design, see [Table types](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md). In this example, we are creating a table based on knowledge of how the table will be queried and the data in the Parquet file. The knowledge of the data in the Parquet file can be gained by querying the file directly in GCS. * Since a query of the dataset in GCS indicates that the `Timestamp` column contains data that matches a VARBINARY data type, the column type is specified in the following DDL. * By querying the data in GCS, you can find that there are no `NULL` values in the dataset, so the DDL does not set any columns as nullable. * Based on knowledge of the expected query types, the sort key and bucketing column are set to the column `UserID`. Your use case might be different for this data, so you might decide to use `ItemID` in addition to or instead of `UserID` for the sort key. Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from GCS): ```sql CREATE TABLE user_behavior_declared ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` Display the schema so that you can compare it with the inferred schema produced by the `FILES()` table function: ```sql DESCRIBE user_behavior_declared; ``` ```plaintext +--------------+----------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+----------------+------+-------+---------+-------+ | UserID | int | NO | true | NULL | | | ItemID | int | NO | false | NULL | | | CategoryID | int | NO | false | NULL | | | BehaviorType | varchar(65533) | NO | false | NULL | | | Timestamp | varbinary | NO | false | NULL | | +--------------+----------------+------+-------+---------+-------+ 5 rows in set (0.00 sec) ``` tip Compare the schema you just created with the schema inferred earlier using the `FILES()` table function. Look at: * data types * nullable * key fields To better control the schema of the destination table and for better query performance, we recommend that you specify the table schema by hand in production environments. After creating the table, you can load it with INSERT INTO SELECT FROM FILES(): ```sql INSERT INTO user_behavior_declared SELECT * FROM FILES ( "path" = "gs://starrocks-samples/user_behavior_ten_million_rows.parquet", "format" = "parquet", "gcp.gcs.service_account_email" = "sampledatareader@xxxxx-xxxxxx-000000.iam.gserviceaccount.com", "gcp.gcs.service_account_private_key_id" = "baaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "gcp.gcs.service_account_private_key" = "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----" ); ``` > **NOTE** > > Substitute the credentials in the above command with your own credentials. Any valid service account email, key, and secret can be used, as the object is readable by any GCP authenticated user. After the load is complete, you can query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_declared LIMIT 3; ``` The system returns a query result similar to the following, indicating that the data has been successfully loaded: ```plain +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 142 | 2869980 | 2939262 | pv | 2017-11-25 03:43:22 | | 142 | 2522236 | 1669167 | pv | 2017-11-25 15:14:12 | | 142 | 3031639 | 3607361 | pv | 2017-11-25 15:19:25 | +--------+---------+------------+--------------+---------------------+ ``` ###### Check load progress[​](#check-load-progress "Direct link to Check load progress") You can query the progress of INSERT jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. Example: ```sql SELECT * FROM information_schema.loads ORDER BY JOB_ID DESC; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'insert_f3fc2298-a553-11ee-92f4-00163e0842bd' \G *************************** 1. row *************************** JOB_ID: 10193 LABEL: insert_f3fc2298-a553-11ee-92f4-00163e0842bd DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: INSERT PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):300; max_filter_ratio:0.0 CREATE_TIME: 2023-12-28 15:37:38 ETL_START_TIME: 2023-12-28 15:37:38 ETL_FINISH_TIME: 2023-12-28 15:37:38 LOAD_START_TIME: 2023-12-28 15:37:38 LOAD_FINISH_TIME: 2023-12-28 15:39:35 JOB_DETAILS: {"All backends":{"f3fc2298-a553-11ee-92f4-00163e0842bd":[10120]},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":581730322,"InternalTableLoadRows":10000000,"ScanBytes":581574034,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"f3fc2298-a553-11ee-92f4-00163e0842bd":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` > **NOTE** > > INSERT is a synchronous command. If an INSERT job is still running, you need to open another session to check its execution status. #### Use Broker Load[​](#use-broker-load "Direct link to Use Broker Load") An asynchronous Broker Load process handles making the connection to GCS, pulling the data, and storing the data in StarRocks. This method supports the following file formats: * Parquet * ORC * CSV * JSON (supported from v3.2.3 onwards) ##### Advantages of Broker Load[​](#advantages-of-broker-load "Direct link to Advantages of Broker Load") * Broker Load runs in the background and clients don't need to stay connected for the job to continue. * Broker Load is preferred for long running jobs, the default timeout is 4 hours. * In addition to Parquet and ORC file format, Broker Load supports CSV file format and JSON file format (JSON file format is supported from v3.2.3 onwards). ##### Data flow[​](#data-flow "Direct link to Data flow") ![Workflow of Broker Load](/assets/images/broker_load_how-to-work_en-bb36de70866e6366b2b21808f0f77be8.png) 1. The user creates a load job. 2. The frontend (FE) creates a query plan and distributes the plan to the backend nodes (BEs) or compute nodes (CNs). 3. The BEs or CNs pull the data from the source and load the data into StarRocks. ##### Typical example[​](#typical-example "Direct link to Typical example") Create a table, start a load process that pulls the sample dataset `gs://starrocks-samples/user_behavior_ten_million_rows.parquet` from GCS, and verify the progress and success of the data loading. ###### Create a database and a table[​](#create-a-database-and-a-table "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table has the same schema as the Parquet file that you want to load from GCS): ```sql CREATE TABLE user_behavior ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Broker Load[​](#start-a-broker-load "Direct link to Start a Broker Load") Run the following command to start a Broker Load job that loads data from the sample dataset `gs://starrocks-samples/user_behavior_ten_million_rows.parquet` to the `user_behavior` table: ```sql LOAD LABEL user_behavior ( DATA INFILE("gs://starrocks-samples/user_behavior_ten_million_rows.parquet") INTO TABLE user_behavior FORMAT AS "parquet" ) WITH BROKER ( "gcp.gcs.service_account_email" = "sampledatareader@xxxxx-xxxxxx-000000.iam.gserviceaccount.com", "gcp.gcs.service_account_private_key_id" = "baaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "gcp.gcs.service_account_private_key" = "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----" ) PROPERTIES ( "timeout" = "72000" ); ``` > **NOTE** > > Substitute the credentials in the above command with your own credentials. Any valid service account email, key, and secret can be used, as the object is readable by any GCP authenticated user. This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. * `BROKER`: The connection details for the source. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Check load progress[​](#check-load-progress-1 "Direct link to Check load progress") You can query the progress of INSERT jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. ```sql SELECT * FROM information_schema.loads; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'user_behavior'; ``` In the output below there are two entries for the load job `user_behavior`: * The first record shows a state of `CANCELLED`. Scroll to `ERROR_MSG`, and you can see that the job has failed due to `listPath failed`. * The second record shows a state of `FINISHED`, which means that the job has succeeded. ```plain JOB_ID|LABEL |DATABASE_NAME|STATE |PROGRESS |TYPE |PRIORITY|SCAN_ROWS|FILTERED_ROWS|UNSELECTED_ROWS|SINK_ROWS|ETL_INFO|TASK_INFO |CREATE_TIME |ETL_START_TIME |ETL_FINISH_TIME |LOAD_START_TIME |LOAD_FINISH_TIME |JOB_DETAILS |ERROR_MSG |TRACKING_URL|TRACKING_SQL|REJECTED_RECORD_PATH| ------+-------------------------------------------+-------------+---------+-------------------+------+--------+---------+-------------+---------------+---------+--------+----------------------------------------------------+-------------------+-------------------+-------------------+-------------------+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+------------+------------+--------------------+ 10121|user_behavior |mydatabase |CANCELLED|ETL:N/A; LOAD:N/A |BROKER|NORMAL | 0| 0| 0| 0| |resource:N/A; timeout(s):72000; max_filter_ratio:0.0|2023-08-10 14:59:30| | | |2023-08-10 14:59:34|{"All backends":{},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":0,"InternalTableLoadRows":0,"ScanBytes":0,"ScanRows":0,"TaskNumber":0,"Unfinished backends":{}} |type:ETL_RUN_FAIL; msg:listPath failed| | | | 10106|user_behavior |mydatabase |FINISHED |ETL:100%; LOAD:100%|BROKER|NORMAL | 86953525| 0| 0| 86953525| |resource:N/A; timeout(s):72000; max_filter_ratio:0.0|2023-08-10 14:50:15|2023-08-10 14:50:19|2023-08-10 14:50:19|2023-08-10 14:50:19|2023-08-10 14:55:10|{"All backends":{"a5fe5e1d-d7d0-4826-ba99-c7348f9a5f2f":[10004]},"FileNumber":1,"FileSize":1225637388,"InternalTableLoadBytes":2710603082,"InternalTableLoadRows":86953525,"ScanBytes":1225637388,"ScanRows":86953525,"TaskNumber":1,"Unfinished backends":{"a5| | | | | ``` After you confirm that the load job has finished, you can check a subset of the destination table to see if the data has been successfully loaded. Example: ```sql SELECT * from user_behavior LIMIT 3; ``` The system returns a query result similar to the following, indicating that the data has been successfully loaded: ```plain +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 142 | 2869980 | 2939262 | pv | 2017-11-25 03:43:22 | | 142 | 2522236 | 1669167 | pv | 2017-11-25 15:14:12 | | 142 | 3031639 | 3607361 | pv | 2017-11-25 15:19:25 | +--------+---------+------------+--------------+---------------------+ ``` --- ### huawei --- ### Load data from MinIO StarRocks provides the following options for loading data from MinIO: * Synchronous loading using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md)+[`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) * Asynchronous loading using [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) Each of these options has its own advantages, which are detailed in the following sections. In most cases, we recommend that you use the INSERT+`FILES()` method, which is much easier to use. However, the INSERT+`FILES()` method currently supports only the Parquet, ORC, and CSV file formats. Therefore, if you need to load data of other file formats such as JSON, or [perform data changes such as DELETE during data loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md), you can resort to Broker Load. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Make source data ready[​](#make-source-data-ready "Direct link to Make source data ready") Make sure the source data you want to load into StarRocks is properly stored in a MinIO bucket. You may also consider where the data and the database are located, because data transfer costs are much lower when your bucket and your StarRocks cluster are located in the same region. In this topic, we provide you with a sample dataset. You can download this with `curl`: ```bash curl -O https://starrocks-examples.s3.amazonaws.com/user_behavior_ten_million_rows.parquet ``` Load the Parquet file into your MinIO system and note the bucket name. The examples in this guide use a bucket name of `/starrocks`. ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ##### Gather connection details[​](#gather-connection-details "Direct link to Gather connection details") In a nutshell, to use MinIO Access Key authentication you need to gather the following information: * The bucket that stores your data * The object key (object name) if accessing a specific object in the bucket * The MinIO endpoint * The access key and secret key used as access credentials. ![MinIO access key](/assets/images/MinIO-create-f8a2e63a6a2e6a718cc0b46a115a019d.png) #### Use INSERT+FILES()[​](#use-insertfiles "Direct link to Use INSERT+FILES()") This method is available from v3.1 onwards and currently supports only the Parquet, ORC, and CSV (from v3.3.0 onwards) file formats. ##### Advantages of INSERT+FILES()[​](#advantages-of-insertfiles "Direct link to Advantages of INSERT+FILES()") [`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) can read the file stored in cloud storage based on the path-related properties you specify, infer the table schema of the data in the file, and then return the data from the file as data rows. With `FILES()`, you can: * Query the data directly from MinIO using [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). * Create and load a table using [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) (CTAS). * Load the data into an existing table using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). ##### Typical examples[​](#typical-examples "Direct link to Typical examples") ###### Querying directly from MinIO using SELECT[​](#querying-directly-from-minio-using-select "Direct link to Querying directly from MinIO using SELECT") Querying directly from MinIO using SELECT+`FILES()` can give a good preview of the content of a dataset before you create a table. For example: * Get a preview of the dataset without storing the data. * Query for the min and max values and decide what data types to use. * Check for `NULL` values. The following example queries the sample dataset previously added to your MinIO system. tip The highlighted section of the command includes the settings that you may need to change: * Set the `endpoint` and `path` to match your MinIO system. * If your MinIO system uses SSL set `enable_ssl` to `true`. * Substitute your MinIO access key and secret for `AAA` and `BBB`. ```sql SELECT * FROM FILES ( "aws.s3.endpoint" = "http://minio:9000", "path" = "s3://starrocks/user_behavior_ten_million_rows.parquet", "aws.s3.enable_ssl" = "false", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "format" = "parquet", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "false", "aws.s3.enable_path_style_access" = "true" ) LIMIT 3; ``` The system returns the following query result: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 543711 | 829192 | 2355072 | pv | 2017-11-27 08:22:37 | | 543711 | 2056618 | 3645362 | pv | 2017-11-27 10:16:46 | | 543711 | 1165492 | 3645362 | pv | 2017-11-27 10:17:00 | +--------+---------+------------+--------------+---------------------+ 3 rows in set (0.41 sec) ``` info Notice that the column names returned above are provided by the Parquet file. ###### Creating and loading a table using CTAS[​](#creating-and-loading-a-table-using-ctas "Direct link to Creating and loading a table using CTAS") This is a continuation of the previous example. The previous query is wrapped in CREATE TABLE AS SELECT (CTAS) to automate the table creation using schema inference. This means StarRocks will infer the table schema, create the table you want, and then load the data into the table. The column names and types are not required to create a table when using the `FILES()` table function with Parquet files as the Parquet format includes the column names. note The syntax of CREATE TABLE when using schema inference does not allow setting the number of replicas, so set it before creating the table. The example below is for a system with a single replica: ```sql ADMIN SET FRONTEND CONFIG ('default_replication_num' = '1'); ``` Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Use CTAS to create a table and load the data of the sample dataset previously added to your MinIO system. tip The highlighted section of the command includes the settings that you may need to change: * Set the `endpoint` and `path` to match your MinIO system. * If your MinIO system uses SSL set `enable_ssl` to `true`. * Substitute your MinIO access key and secret key for `AAA` and `BBB`. ```sql CREATE TABLE user_behavior_inferred AS SELECT * FROM FILES ( "aws.s3.endpoint" = "http://minio:9000", "path" = "s3://starrocks/user_behavior_ten_million_rows.parquet", "aws.s3.enable_ssl" = "false", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "format" = "parquet", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "false", "aws.s3.enable_path_style_access" = "true" ); ``` ```plaintext Query OK, 10000000 rows affected (3.17 sec) {'label':'insert_a5da3ff5-9ee4-11ee-90b0-02420a060004', 'status':'VISIBLE', 'txnId':'17'} ``` After creating the table, you can view its schema by using [DESCRIBE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md): ```sql DESCRIBE user_behavior_inferred; ``` The system returns the following query result: ```plaintext +--------------+------------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+------------------+------+-------+---------+-------+ | UserID | bigint | YES | true | NULL | | | ItemID | bigint | YES | true | NULL | | | CategoryID | bigint | YES | true | NULL | | | BehaviorType | varchar(1048576) | YES | false | NULL | | | Timestamp | varchar(1048576) | YES | false | NULL | | +--------------+------------------+------+-------+---------+-------+ ``` Query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_inferred LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+--------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+--------+------------+--------------+---------------------+ | 58 | 158350 | 2355072 | pv | 2017-11-27 13:06:51 | | 58 | 158590 | 3194735 | pv | 2017-11-27 02:21:04 | | 58 | 215073 | 3002561 | pv | 2017-11-30 10:55:42 | +--------+--------+------------+--------------+---------------------+ ``` ###### Loading into an existing table using INSERT[​](#loading-into-an-existing-table-using-insert "Direct link to Loading into an existing table using INSERT") You may want to customize the table that you are inserting into, for example, the: * column data type, nullable setting, or default values * key types and columns * data partitioning and bucketing tip Creating the most efficient table structure requires knowledge of how the data will be used and the content of the columns. This topic does not cover table design. For information about table design, see [Table types](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md). In this example, we are creating a table based on knowledge of how the table will be queried and the data in the Parquet file. The knowledge of the data in the Parquet file can be gained by querying the file directly in MinIO. * Since a query of the dataset in MinIO indicates that the `Timestamp` column contains data that matches a `datetime` data type, the column type is specified in the following DDL. * By querying the data in MinIO, you can find that there are no `NULL` values in the dataset, so the DDL does not set any columns as nullable. * Based on knowledge of the expected query types, the sort key and bucketing column are set to the column `UserID`. Your use case might be different for this data, so you might decide to use `ItemID` in addition to or instead of `UserID` for the sort key. Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from MinIO): ```sql CREATE TABLE user_behavior_declared ( UserID int(11) NOT NULL, ItemID int(11) NOT NULL, CategoryID int(11) NOT NULL, BehaviorType varchar(65533) NOT NULL, Timestamp datetime NOT NULL ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID) PROPERTIES ( 'replication_num' = '1' ); ``` Display the schema so that you can compare it with the inferred schema produced by the `FILES()` table function: ```sql DESCRIBE user_behavior_declared; ``` ```plaintext +--------------+----------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+----------------+------+-------+---------+-------+ | UserID | int | NO | true | NULL | | | ItemID | int | NO | false | NULL | | | CategoryID | int | NO | false | NULL | | | BehaviorType | varchar(65533) | NO | false | NULL | | | Timestamp | datetime | NO | false | NULL | | +--------------+----------------+------+-------+---------+-------+ 5 rows in set (0.00 sec) ``` tip Compare the schema you just created with the schema inferred earlier using the `FILES()` table function. Look at: * data types * nullable * key fields To better control the schema of the destination table and for better query performance, we recommend that you specify the table schema by hand in production environments. Having a `datetime` data type for the timestamp field is more efficient than using a `varchar`. After creating the table, you can load it with INSERT INTO SELECT FROM FILES(): tip The highlighted section of the command includes the settings that you may need to change: * Set the `endpoint` and `path` to match your MinIO system. * If your MinIO system uses SSL set `enable_ssl` to `true`. * Substitute your MinIO access key and secret key for `AAA` and `BBB`. ```sql INSERT INTO user_behavior_declared SELECT * FROM FILES ( "aws.s3.endpoint" = "http://minio:9000", "path" = "s3://starrocks/user_behavior_ten_million_rows.parquet", "aws.s3.enable_ssl" = "false", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "format" = "parquet", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "false", "aws.s3.enable_path_style_access" = "true" ); ``` After the load is complete, you can query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_declared LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 58 | 4309692 | 1165503 | pv | 2017-11-25 14:06:52 | | 58 | 181489 | 1165503 | pv | 2017-11-25 14:07:22 | | 58 | 3722956 | 1165503 | pv | 2017-11-25 14:09:28 | +--------+---------+------------+--------------+---------------------+ ``` ###### Check load progress[​](#check-load-progress "Direct link to Check load progress") You can query the progress of INSERT jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. Example: ```sql SELECT * FROM information_schema.loads ORDER BY JOB_ID DESC; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'insert_e3b882f5-7eb3-11ee-ae77-00163e267b60' \G *************************** 1. row *************************** JOB_ID: 10243 LABEL: insert_e3b882f5-7eb3-11ee-ae77-00163e267b60 DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: INSERT PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):300; max_filter_ratio:0.0 CREATE_TIME: 2023-11-09 11:56:01 ETL_START_TIME: 2023-11-09 11:56:01 ETL_FINISH_TIME: 2023-11-09 11:56:01 LOAD_START_TIME: 2023-11-09 11:56:01 LOAD_FINISH_TIME: 2023-11-09 11:56:44 JOB_DETAILS: {"All backends":{"e3b882f5-7eb3-11ee-ae77-00163e267b60":[10142]},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":311710786,"InternalTableLoadRows":10000000,"ScanBytes":581574034,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"e3b882f5-7eb3-11ee-ae77-00163e267b60":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` tip INSERT is a synchronous command. If an INSERT job is still running, you need to open another session to check its execution status. ##### Compare the table sizes on disk[​](#compare-the-table-sizes-on-disk "Direct link to Compare the table sizes on disk") This query compares the table with the inferred schema and the one where the schema is declared. Because the inferred schema has nullable columns and a varchar for the timestamp the data length is larger: ```sql SELECT TABLE_NAME, TABLE_ROWS, AVG_ROW_LENGTH, DATA_LENGTH FROM information_schema.tables WHERE TABLE_NAME like 'user_behavior%'\G ``` ```plaintext *************************** 1. row *************************** TABLE_NAME: user_behavior_declared TABLE_ROWS: 10000000 AVG_ROW_LENGTH: 10 DATA_LENGTH: 102562516 *************************** 2. row *************************** TABLE_NAME: user_behavior_inferred TABLE_ROWS: 10000000 AVG_ROW_LENGTH: 17 DATA_LENGTH: 176803880 2 rows in set (0.04 sec) ``` #### Use Broker Load[​](#use-broker-load "Direct link to Use Broker Load") An asynchronous Broker Load process handles making the connection to MinIO, pulling the data, and storing the data in StarRocks. This method supports the following file formats: * Parquet * ORC * CSV * JSON (supported from v3.2.3 onwards) ##### Advantages of Broker Load[​](#advantages-of-broker-load "Direct link to Advantages of Broker Load") * Broker Load runs in the background and clients do not need to stay connected for the job to continue. * Broker Load is preferred for long-running jobs, with the default timeout spanning 4 hours. * In addition to Parquet and ORC file format, Broker Load supports CSV file format and JSON file format (JSON file format is supported from v3.2.3 onwards). ##### Data flow[​](#data-flow "Direct link to Data flow") ![Workflow of Broker Load](/assets/images/broker_load_how-to-work_en-bb36de70866e6366b2b21808f0f77be8.png) 1. The user creates a load job. 2. The frontend (FE) creates a query plan and distributes the plan to the backend nodes (BEs) or compute nodes (CNs). 3. The BEs or CNs pull the data from the source and load the data into StarRocks. ##### Typical example[​](#typical-example "Direct link to Typical example") Create a table, start a load process that pulls the sample dataset previously loaded to your MinIO system. ###### Create a database and a table[​](#create-a-database-and-a-table "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table has the same schema as the Parquet file that you want to load from MinIO): ```sql CREATE TABLE user_behavior ( UserID int(11) NOT NULL, ItemID int(11) NOT NULL, CategoryID int(11) NOT NULL, BehaviorType varchar(65533) NOT NULL, Timestamp datetime NOT NULL ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID) PROPERTIES ( 'replication_num' = '1' ); ``` ###### Start a Broker Load[​](#start-a-broker-load "Direct link to Start a Broker Load") Run the following command to start a Broker Load job that loads data from the sample dataset `user_behavior_ten_million_rows.parquet` to the `user_behavior` table: tip The highlighted section of the command includes the settings that you may need to change: * Set the `endpoint` and `DATA INFILE` to match your MinIO system. * If your MinIO system uses SSL set `enable_ssl` to `true`. * Substitute your MinIO access key and secret for `AAA` and `BBB`. ```sql LOAD LABEL UserBehavior ( DATA INFILE("s3://starrocks/user_behavior_ten_million_rows.parquet") INTO TABLE user_behavior ) WITH BROKER ( "aws.s3.endpoint" = "http://minio:9000", "aws.s3.enable_ssl" = "false", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "false", "aws.s3.enable_path_style_access" = "true" ) PROPERTIES ( "timeout" = "72000" ); ``` This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. * `BROKER`: The connection details for the source. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Check load progress[​](#check-load-progress-1 "Direct link to Check load progress") You can query the progress of Broker Load jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. ```sql SELECT * FROM information_schema.loads; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'UserBehavior'\G ``` ```plaintext *************************** 1. row *************************** JOB_ID: 10176 LABEL: userbehavior DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: BROKER PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):72000; max_filter_ratio:0.0 CREATE_TIME: 2023-12-19 23:02:41 ETL_START_TIME: 2023-12-19 23:02:44 ETL_FINISH_TIME: 2023-12-19 23:02:44 LOAD_START_TIME: 2023-12-19 23:02:44 LOAD_FINISH_TIME: 2023-12-19 23:02:46 JOB_DETAILS: {"All backends":{"4aeec563-a91e-4c1e-b169-977b660950d1":[10004]},"FileNumber":1,"FileSize":132251298,"InternalTableLoadBytes":311710786,"InternalTableLoadRows":10000000,"ScanBytes":132251298,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"4aeec563-a91e-4c1e-b169-977b660950d1":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL 1 row in set (0.02 sec) ``` After you confirm that the load job has finished, you can check a subset of the destination table to see if the data has been successfully loaded. Example: ```sql SELECT * from user_behavior LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 142 | 2869980 | 2939262 | pv | 2017-11-25 03:43:22 | | 142 | 2522236 | 1669167 | pv | 2017-11-25 15:14:12 | | 142 | 3031639 | 3607361 | pv | 2017-11-25 15:19:25 | +--------+---------+------------+--------------+---------------------+ ``` --- ### Load data from AWS S3 StarRocks provides the following options for loading data from AWS S3: * Synchronous loading using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md)+[`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) * Asynchronous loading using [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) * Continuous asynchronous loading using [Pipe](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md) Each of these options has its own advantages, which are detailed in the following sections. In most cases, we recommend that you use the INSERT+`FILES()` method, which is much easier to use. However, the INSERT+`FILES()` method currently supports only the Parquet, ORC, and CSV file formats. Therefore, if you need to load data of other file formats such as JSON, or perform data changes such as DELETE during data loading, you can resort to Broker Load. If you need to load a large number of data files with a significant data volume in total (for example, more than 100 GB or even 1 TB), we recommend that you use the Pipe method. Pipe can split the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job and minimizes the need for retries due to data errors. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Make source data ready[​](#make-source-data-ready "Direct link to Make source data ready") Make sure the source data you want to load into StarRocks is properly stored in an S3 bucket. You may also consider where the data and the database are located, because data transfer costs are much lower when your bucket and your StarRocks cluster are located in the same region. In this topic, we provide you with a sample dataset in an S3 bucket, `s3://starrocks-examples/user-behavior-10-million-rows.parquet`. You can access that dataset with any valid credentials as the object is readable by any AWS authenticated user. ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ##### Gather authentication details[​](#gather-authentication-details "Direct link to Gather authentication details") The examples in this topic use IAM user-based authentication. To ensure that you have permission to read data from AWS S3, we recommend that you read [Preparation for IAM user-based authentication](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md) and follow the instructions to create an IAM user with proper [IAM policies](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md) configured. In a nutshell, if you practice IAM user-based authentication, you need to gather information about the following AWS resources: * The S3 bucket that stores your data. * The S3 object key (object name) if accessing a specific object in the bucket. Note that the object key can include a prefix if your S3 objects are stored in sub-folders. * The AWS region to which the S3 bucket belongs. * The access key and secret key used as access credentials. For information about all the authentication methods available, see [Authenticate to AWS resources](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md). #### Use INSERT+FILES()[​](#use-insertfiles "Direct link to Use INSERT+FILES()") This method is available from v3.1 onwards and currently supports only the Parquet, ORC, and CSV (from v3.3.0 onwards) file formats. ##### Advantages of INSERT+FILES()[​](#advantages-of-insertfiles "Direct link to Advantages of INSERT+FILES()") [`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) can read the file stored in cloud storage based on the path-related properties you specify, infer the table schema of the data in the file, and then return the data from the file as data rows. With `FILES()`, you can: * Query the data directly from S3 using [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). * Create and load a table using [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) (CTAS). * Load the data into an existing table using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). ##### Typical examples[​](#typical-examples "Direct link to Typical examples") ###### Querying directly from S3 using SELECT[​](#querying-directly-from-s3-using-select "Direct link to Querying directly from S3 using SELECT") Querying directly from S3 using SELECT+`FILES()` can give a good preview of the content of a dataset before you create a table. For example: * Get a preview of the dataset without storing the data. * Query for the min and max values and decide what data types to use. * Check for `NULL` values. The following example queries the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows.parquet`: ```sql SELECT * FROM FILES ( "path" = "s3://starrocks-examples/user-behavior-10-million-rows.parquet", "format" = "parquet", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ) LIMIT 3; ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. The system returns the following query result: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 1 | 2576651 | 149192 | pv | 2017-11-25 01:21:25 | | 1 | 3830808 | 4181361 | pv | 2017-11-25 07:04:53 | | 1 | 4365585 | 2520377 | pv | 2017-11-25 07:49:06 | +--------+---------+------------+--------------+---------------------+ ``` > **NOTE** > > Notice that the column names as returned above are provided by the Parquet file. ###### Creating and loading a table using CTAS[​](#creating-and-loading-a-table-using-ctas "Direct link to Creating and loading a table using CTAS") This is a continuation of the previous example. The previous query is wrapped in CREATE TABLE AS SELECT (CTAS) to automate the table creation using schema inference. This means StarRocks will infer the table schema, create the table you want, and then load the data into the table. The column names and types are not required to create a table when using the `FILES()` table function with Parquet files as the Parquet format includes the column names. > **NOTE** > > The syntax of CREATE TABLE when using schema inference does not allow setting the number of replicas, so set it before creating the table. The example below is for a system with one replica: > > ```sql > ADMIN SET FRONTEND CONFIG ('default_replication_num' = "1"); > > ``` Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Use CTAS to create a table and load the data of the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows.parquet` into the table: ```sql CREATE TABLE user_behavior_inferred AS SELECT * FROM FILES ( "path" = "s3://starrocks-examples/user-behavior-10-million-rows.parquet", "format" = "parquet", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ); ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. After creating the table, you can view its schema by using [DESCRIBE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md): ```sql DESCRIBE user_behavior_inferred; ``` The system returns the following query result: ```plain +--------------+------------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+------------------+------+-------+---------+-------+ | UserID | bigint | YES | true | NULL | | | ItemID | bigint | YES | true | NULL | | | CategoryID | bigint | YES | true | NULL | | | BehaviorType | varchar(1048576) | YES | false | NULL | | | Timestamp | varchar(1048576) | YES | false | NULL | | +--------------+------------------+------+-------+---------+-------+ ``` Query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_inferred LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 225586 | 3694958 | 1040727 | pv | 2017-12-01 00:58:40 | | 225586 | 3726324 | 965809 | pv | 2017-12-01 02:16:02 | | 225586 | 3732495 | 1488813 | pv | 2017-12-01 00:59:46 | +--------+---------+------------+--------------+---------------------+ ``` ###### Loading into an existing table using INSERT[​](#loading-into-an-existing-table-using-insert "Direct link to Loading into an existing table using INSERT") You may want to customize the table that you are inserting into, for example, the: * column data type, nullable setting, or default values * key types and columns * data partitioning and bucketing > **NOTE** > > Creating the most efficient table structure requires knowledge of how the data will be used and the content of the columns. This topic does not cover table design. For information about table design, see [Table types](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md). In this example, we are creating a table based on knowledge of how the table will be queried and the data in the Parquet file. The knowledge of the data in the Parquet file can be gained by querying the file directly in S3. * Since a query of the dataset in S3 indicates that the `Timestamp` column contains data that matches a VARCHAR data type, and StarRocks can cast from VARCHAR to DATETIME, the data type is changed to DATETIME in the following DDL. * By querying the data in S3, you can find that there are no `NULL` values in the dataset, so the DDL could also set all columns as non-nullable. * Based on knowledge of the expected query types, the sort key and bucketing column are set to the column `UserID`. Your use case might be different for this data, so you might decide to use `ItemID` in addition to, or instead of, `UserID` for the sort key. Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand: ```sql CREATE TABLE user_behavior_declared ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp datetime ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` Display the schema so that you can compare it with the inferred schema produced by the `FILES()` table function: ```sql DESCRIBE user_behavior_declared; ``` ```plaintext +--------------+----------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+----------------+------+-------+---------+-------+ | UserID | int | YES | true | NULL | | | ItemID | int | YES | false | NULL | | | CategoryID | int | YES | false | NULL | | | BehaviorType | varchar(65533) | YES | false | NULL | | | Timestamp | datetime | YES | false | NULL | | +--------------+----------------+------+-------+---------+-------+ ``` tip Compare the schema you just created with the schema inferred earlier using the `FILES()` table function. Look at: * data types * nullable * key fields To better control the schema of the destination table and for better query performance, we recommend that you specify the table schema by hand in production environments. After creating the table, you can load it with INSERT INTO SELECT FROM FILES(): ```sql INSERT INTO user_behavior_declared SELECT * FROM FILES ( "path" = "s3://starrocks-examples/user-behavior-10-million-rows.parquet", "format" = "parquet", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ); ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. After the load is complete, you can query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_declared LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 393529 | 3715112 | 883960 | pv | 2017-12-02 02:45:44 | | 393529 | 2650583 | 883960 | pv | 2017-12-02 02:45:59 | | 393529 | 3715112 | 883960 | pv | 2017-12-02 03:00:56 | +--------+---------+------------+--------------+---------------------+ ``` ###### Check load progress[​](#check-load-progress "Direct link to Check load progress") You can query the progress of INSERT jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. Example: ```sql SELECT * FROM information_schema.loads ORDER BY JOB_ID DESC; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'insert_e3b882f5-7eb3-11ee-ae77-00163e267b60' \G *************************** 1. row *************************** JOB_ID: 10243 LABEL: insert_e3b882f5-7eb3-11ee-ae77-00163e267b60 DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: INSERT PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):300; max_filter_ratio:0.0 CREATE_TIME: 2023-11-09 11:56:01 ETL_START_TIME: 2023-11-09 11:56:01 ETL_FINISH_TIME: 2023-11-09 11:56:01 LOAD_START_TIME: 2023-11-09 11:56:01 LOAD_FINISH_TIME: 2023-11-09 11:56:44 JOB_DETAILS: {"All backends":{"e3b882f5-7eb3-11ee-ae77-00163e267b60":[10142]},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":311710786,"InternalTableLoadRows":10000000,"ScanBytes":581574034,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"e3b882f5-7eb3-11ee-ae77-00163e267b60":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` > **NOTE** > > INSERT is a synchronous command. If an INSERT job is still running, you need to open another session to check its execution status. #### Use Broker Load[​](#use-broker-load "Direct link to Use Broker Load") An asynchronous Broker Load process handles making the connection to S3, pulling the data, and storing the data in StarRocks. This method supports the following file formats: * Parquet * ORC * CSV * JSON (supported from v3.2.3 onwards) ##### Advantages of Broker Load[​](#advantages-of-broker-load "Direct link to Advantages of Broker Load") * Broker Load runs in the background and clients do not need to stay connected for the job to continue. * Broker Load is preferred for long-running jobs, with the default timeout spanning 4 hours. * In addition to Parquet and ORC file format, Broker Load supports CSV file format and JSON file format (JSON file format is supported from v3.2.3 onwards). ##### Data flow[​](#data-flow "Direct link to Data flow") ![Workflow of Broker Load](/assets/images/broker_load_how-to-work_en-bb36de70866e6366b2b21808f0f77be8.png) 1. The user creates a load job. 2. The frontend (FE) creates a query plan and distributes the plan to the backend nodes (BEs) or compute nodes (CNs). 3. The BEs or CNs pull the data from the source and load the data into StarRocks. ##### Typical example[​](#typical-example "Direct link to Typical example") Create a table, start a load process that pulls the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows.parquet` from S3, and verify the progress and success of the data loading. ###### Create a database and a table[​](#create-a-database-and-a-table "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table has the same schema as the Parquet file that you want to load from AWS S3): ```sql CREATE TABLE user_behavior ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp datetime ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Broker Load[​](#start-a-broker-load "Direct link to Start a Broker Load") Run the following command to start a Broker Load job that loads data from the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows.parquet` to the `user_behavior` table: ```sql LOAD LABEL user_behavior ( DATA INFILE("s3://starrocks-examples/user-behavior-10-million-rows.parquet") INTO TABLE user_behavior FORMAT AS "parquet" ) WITH BROKER ( "aws.s3.enable_ssl" = "true", "aws.s3.use_instance_profile" = "false", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ) PROPERTIES ( "timeout" = "72000" ); ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. * `BROKER`: The connection details for the source. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Check load progress[​](#check-load-progress-1 "Direct link to Check load progress") You can query the progress of the Broker Load job from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'user_behavior'; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). This record shows a state of `LOADING`, and the progress is 39%. If you see something similar, then run the command again until you see a state of `FINISHED`. ```plaintext JOB_ID: 10466 LABEL: user_behavior DATABASE_NAME: mydatabase STATE: LOADING PROGRESS: ETL:100%; LOAD:39% TYPE: BROKER PRIORITY: NORMAL SCAN_ROWS: 4620288 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 4620288 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):72000; max_filter_ratio:0.0 CREATE_TIME: 2024-02-28 22:11:36 ETL_START_TIME: 2024-02-28 22:11:41 ETL_FINISH_TIME: 2024-02-28 22:11:41 LOAD_START_TIME: 2024-02-28 22:11:41 LOAD_FINISH_TIME: NULL JOB_DETAILS: {"All backends":{"2fb97223-b14c-404b-9be1-83aa9b3a7715":[10004]},"FileNumber":1,"FileSize":136901706,"InternalTableLoadBytes":144032784,"InternalTableLoadRows":4620288,"ScanBytes":143969616,"ScanRows":4620288,"TaskNumber":1,"Unfinished backends":{"2fb97223-b14c-404b-9be1-83aa9b3a7715":[10004]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` After you confirm that the load job has finished, you can check a subset of the destination table to see if the data has been successfully loaded. Example: ```sql SELECT * from user_behavior LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 34 | 856384 | 1029459 | pv | 2017-11-27 14:43:27 | | 34 | 5079705 | 1029459 | pv | 2017-11-27 14:44:13 | | 34 | 4451615 | 1029459 | pv | 2017-11-27 14:45:52 | +--------+---------+------------+--------------+---------------------+ ``` #### Use Pipe[​](#use-pipe "Direct link to Use Pipe") Starting from v3.2, StarRocks provides the Pipe loading method, which currently supports only the Parquet and ORC file formats. ##### Advantages of Pipe[​](#advantages-of-pipe "Direct link to Advantages of Pipe") Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading.During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that object storage like AWS S3 uses ETag as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Data flow[​](#data-flow "Direct link to Data flow") ![Pipe data flow](/assets/images/pipe_data_flow-2a4dc0b44a06c987d9afc0ecf632f5d9.png) Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading.During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that object storage like AWS S3 uses `ETag` as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Differences between Pipe and INSERT+FILES()[​](#differences-between-pipe-and-insertfiles "Direct link to Differences between Pipe and INSERT+FILES()") A Pipe job is split into one or more transactions based on the size and number of rows in each data file. Users can query the intermediate results during the loading process. In contrast, an INSERT+`FILES()` job is processed as a single transaction, and users are unable to view the data during the loading process. ##### File loading sequence[​](#file-loading-sequence "Direct link to File loading sequence") For each Pipe job, StarRocks maintains a file queue, from which it fetches and loads data files as micro-batches. Pipe does not ensure that the data files are loaded in the same order as they are uploaded. Therefore, newer data may be loaded prior to older data. ##### Typical example[​](#typical-example-1 "Direct link to Typical example") ###### Create a database and a table[​](#create-a-database-and-a-table-1 "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from AWS S3): ```sql CREATE TABLE user_behavior_from_pipe ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp datetime ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Pipe job[​](#start-a-pipe-job "Direct link to Start a Pipe job") Run the following command to start a Pipe job that loads data from the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows/` to the `user_behavior_from_pipe` table. This pipe job uses both micro batches, and continuous loading (described above) pipe-specific features. The other examples in this guide load a single Parquet file with 10 million rows. For the pipe example, the same dataset is split into 57 separate files, and these are all stored in one S3 folder. Note in the `CREATE PIPE` command below the `path` is the URI for an S3 folder and rather than providing a filename the URI ends in `/*`. By setting `AUTO_INGEST` and specifying a folder rather than an individual file the pipe job will poll the S3 folder for new files and ingest them as they are added to the folder. ```sql CREATE PIPE user_behavior_pipe PROPERTIES ( "AUTO_INGEST" = "TRUE" ) AS INSERT INTO user_behavior_from_pipe SELECT * FROM FILES ( "path" = "s3://starrocks-examples/user-behavior-10-million-rows/*", "format" = "parquet", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ); ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. This job has four main sections: * `pipe_name`: The name of the pipe. The pipe name must be unique within the database to which the pipe belongs. * `INSERT_SQL`: The INSERT INTO SELECT FROM FILES statement that is used to load data from the specified source data file to the destination table. * `PROPERTIES`: A set of optional parameters that specify how to execute the pipe. These include `AUTO_INGEST`, `POLL_INTERVAL`, `BATCH_SIZE`, and `BATCH_FILES`. Specify these properties in the `"key" = "value"` format. For detailed syntax and parameter descriptions, see [CREATE PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md). ###### Check load progress[​](#check-load-progress-2 "Direct link to Check load progress") * Query the progress of the Pipe job by using [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md) in the current database to which the Pipe job belongs. ```sql SHOW PIPES WHERE NAME = 'user_behavior_pipe' \G ``` The following result is returned: tip In the output shown below the pipe is in the `RUNNING` state. A pipe will stay in the `RUNNING` state until you manually stop it. The output also shows the number of files loaded (57) and the last time that a file was loaded. ```sql *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10476 PIPE_NAME: user_behavior_pipe STATE: RUNNING TABLE_NAME: mydatabase.user_behavior_from_pipe LOAD_STATUS: {"loadedFiles":57,"loadedBytes":295345637,"loadingFiles":0,"lastLoadedTime":"2024-02-28 22:14:19"} LAST_ERROR: NULL CREATED_TIME: 2024-02-28 22:13:41 1 row in set (0.02 sec) ``` * Query the progress of the Pipe job from the [`pipes`](https://docs.starrocks.io/docs/sql-reference/information_schema/pipes.md) view in the StarRocks Information Schema. ```sql SELECT * FROM information_schema.pipes WHERE pipe_name = 'user_behavior_replica' \G ``` The following result is returned: tip Some of the queries in this guide end in `\G` instead of a semicolon (`;`). This causes the MySQL client to output the results in vertical format. If you are using DBeaver or another client you may need to use a semicolon (`;`) rather than `\G`. ```sql *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10217 PIPE_NAME: user_behavior_replica STATE: RUNNING TABLE_NAME: mydatabase.user_behavior_replica LOAD_STATUS: {"loadedFiles":1,"loadedBytes":132251298,"loadingFiles":0,"lastLoadedTime":"2023-11-09 15:35:42"} LAST_ERROR: CREATED_TIME: 9891-01-15 07:51:45 1 row in set (0.01 sec) ``` ###### Check file status[​](#check-file-status "Direct link to Check file status") You can query the load status of the files loaded from the [`pipe_files`](https://docs.starrocks.io/docs/sql-reference/information_schema/pipe_files.md) view in the StarRocks Information Schema. ```sql SELECT * FROM information_schema.pipe_files WHERE pipe_name = 'user_behavior_replica' \G ``` The following result is returned: ```sql *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10217 PIPE_NAME: user_behavior_replica FILE_NAME: s3://starrocks-examples/user-behavior-10-million-rows.parquet FILE_VERSION: e29daa86b1120fea58ad0d047e671787-8 FILE_SIZE: 132251298 LAST_MODIFIED: 2023-11-06 13:25:17 LOAD_STATE: FINISHED STAGED_TIME: 2023-11-09 15:35:02 START_LOAD_TIME: 2023-11-09 15:35:03 FINISH_LOAD_TIME: 2023-11-09 15:35:42 ERROR_MSG: 1 row in set (0.03 sec) ``` ###### Manage Pipe jobs[​](#manage-pipe-jobs "Direct link to Manage Pipe jobs") You can alter, suspend or resume, drop, or query the pipes you have created and retry to load specific data files. For more information, see [ALTER PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/ALTER_PIPE.md), [SUSPEND or RESUME PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SUSPEND_or_RESUME_PIPE.md), [DROP PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/DROP_PIPE.md), [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md), and [RETRY FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/RETRY_FILE.md). --- ### s3_compatible --- ### tencent --- ### Load data from Apache Spark #### [📄️ Load data in bulk using Spark Load](https://docs.starrocks.io/docs/loading/spark/SparkLoad.md) [How to use Spark Load for bulk data import into StarRocks using external Spark resources for pre-processing, suited for TB-scale initial data migration.](https://docs.starrocks.io/docs/loading/spark/SparkLoad.md) #### [📄️ Load data using Spark connector (recommended)](https://docs.starrocks.io/docs/loading/spark/Spark-connector-starrocks.md) [How to use the StarRocks Spark connector to load data from Apache Spark (DataFrames or SQL) into StarRocks via Stream Load, supporting batch and structured...](https://docs.starrocks.io/docs/loading/spark/Spark-connector-starrocks.md) --- ### Load data using Spark connector (recommended) StarRocks provides a self-developed connector named StarRocks Connector for Apache Spark™ (Spark connector for short) to help you load data into a StarRocks table by using Spark. The basic principle is to accumulate the data and then load it all at a time into StarRocks through [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). The Spark connector is implemented based on Spark DataSource V2. A DataSource can be created by using Spark DataFrames or Spark SQL. And both batch and structured streaming modes are supported. > **NOTICE** > > Only users with the SELECT and INSERT privileges on a StarRocks table can load data into this table. You can follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant these privileges to a user. #### Version requirements[​](#version-requirements "Direct link to Version requirements") | Spark connector | Spark | StarRocks | Java | Scala | | --------------- | ------------------ | ------------- | ---- | ----- | | 1.1.4 | 4.0, 4.1 | 2.5 and later | 17 | 2.13 | | 1.1.4 | 3.3, 3.4, 3.5 | 2.5 and later | 8 | 2.12 | | 1.1.3 | 3.2, 3.3, 3.4, 3.5 | 2.5 and later | 8 | 2.12 | | 1.1.2 | 3.2, 3.3, 3.4, 3.5 | 2.5 and later | 8 | 2.12 | | 1.1.1 | 3.2, 3.3, or 3.4 | 2.5 and later | 8 | 2.12 | | 1.1.0 | 3.2, 3.3, or 3.4 | 2.5 and later | 8 | 2.12 | > **NOTICE** > > * Please see [Upgrade Spark connector](#upgrade-spark-connector) for behavior changes among different versions of the Spark connector. > * The Spark connector does not provide MySQL JDBC driver since version 1.1.1, and you need import the driver to the spark classpath manually. You can find the driver on [MySQL site](https://dev.mysql.com/downloads/connector/j/) or [Maven Central](https://repo1.maven.org/maven2/mysql/mysql-connector-java/). #### Obtain Spark connector[​](#obtain-spark-connector "Direct link to Obtain Spark connector") You can obtain the Spark connector JAR file in the following ways: * Directly download the compiled Spark Connector JAR file. * Add the Spark connector as a dependency in your Maven project and then download the JAR file. * Compile the source code of the Spark Connector into a JAR file by yourself. The naming format of the Spark connector JAR file is `starrocks-spark-connector-${spark_version}_${scala_version}-${connector_version}.jar`. For example, if you install Spark 3.2 and Scala 2.12 in your environment and you want to use Spark connector 1.1.0, you can use `starrocks-spark-connector-3.2_2.12-1.1.0.jar`. > **NOTICE** > > In general, the latest version of the Spark connector only maintains compatibility with the three most recent versions of Spark. ##### Download the compiled Jar file[​](#download-the-compiled-jar-file "Direct link to Download the compiled Jar file") Directly download the corresponding version of the Spark connector JAR from the [Maven Central Repository](https://repo1.maven.org/maven2/com/starrocks). ##### Maven Dependency[​](#maven-dependency "Direct link to Maven Dependency") 1. In your Maven project's `pom.xml` file, add the Spark connector as a dependency according to the following format. Replace `spark_version`, `scala_version`, and `connector_version` with the respective versions. ```xml com.starrocks starrocks-spark-connector-${spark_version}_${scala_version} ${connector_version} ``` 2. For example, if the version of Spark in your environment is 3.2, the version of Scala is 2.12, and you choose Spark connector 1.1.0, you need to add the following dependency: ```xml com.starrocks starrocks-spark-connector-3.2_2.12 1.1.0 ``` ##### Compile by yourself[​](#compile-by-yourself "Direct link to Compile by yourself") 1. Download the [Spark connector package](https://github.com/StarRocks/starrocks-connector-for-apache-spark). 2. Execute the following command to compile the source code of Spark connector into a JAR file. Note that `spark_version` is replaced with the corresponding Spark version. ```bash sh build.sh ``` For example, if the Spark version in your environment is 3.2, you need to execute the following command: ```bash sh build.sh 3.2 ``` 3. Go to the `target/` directory to find the Spark connector JAR file, such as `starrocks-spark-connector-3.2_2.12-1.1.0-SNAPSHOT.jar` , generated upon compilation. > **NOTE** > > The name of Spark connector which is not formally released contains the `SNAPSHOT` suffix. #### Parameters[​](#parameters "Direct link to Parameters") ##### `starrocks.fe.http.url`[​](#starrocksfehttpurl "Direct link to starrocksfehttpurl") * **Required**: YES * **Default value**: None * **Description**: The HTTP URL of the FE in your StarRocks cluster. You can specify multiple URLs, which must be separated by a comma (,). Format: `:,:`. Since version 1.1.1, you can also add `http://` prefix to the URL, such as `http://:,http://:`. ##### `starrocks.fe.jdbc.url`[​](#starrocksfejdbcurl "Direct link to starrocksfejdbcurl") * **Required**: YES * **Default value**: None * **Description**: The address that is used to connect to the MySQL server of the FE. Format: `jdbc:mysql://:`. ##### `starrocks.table.identifier`[​](#starrockstableidentifier "Direct link to starrockstableidentifier") * **Required**: YES * **Default value**: None * **Description**: The name of the StarRocks table. Format: `.`. ##### `starrocks.user`[​](#starrocksuser "Direct link to starrocksuser") * **Required**: YES * **Default value**: None * **Description**: The username of your StarRocks cluster account. The user needs the [SELECT and INSERT privileges](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) on the StarRocks table. ##### `starrocks.password`[​](#starrockspassword "Direct link to starrockspassword") * **Required**: YES * **Default value**: None * **Description**: The password of your StarRocks cluster account. ##### `starrocks.write.label.prefix`[​](#starrockswritelabelprefix "Direct link to starrockswritelabelprefix") * **Required**: NO * **Default value**: `spark-` * **Description**: The label prefix used by Stream Load. ##### `starrocks.write.enable.transaction-stream-load`[​](#starrockswriteenabletransaction-stream-load "Direct link to starrockswriteenabletransaction-stream-load") * **Required**: NO * **Default value**: `TRUE` * **Description**: Whether to use [Stream Load transaction interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) to load data. It requires StarRocks v2.5 or later. This feature can load more data in a transaction with less memory usage, and improve performance. note Since 1.1.1, this parameter takes effect only when the value of `starrocks.write.max.retries` is non-positive because Stream Load transaction interface does not support retry. ##### `starrocks.write.buffer.size`[​](#starrockswritebuffersize "Direct link to starrockswritebuffersize") * **Required**: NO * **Default value**: `104857600` * **Description**: The maximum size of data that can be accumulated in memory before being sent to StarRocks at a time. Setting this parameter to a larger value can improve loading performance but may increase loading latency. ##### `starrocks.write.buffer.rows`[​](#starrockswritebufferrows "Direct link to starrockswritebufferrows") * **Required**: NO * **Default value**: Integer.MAX\_VALUE * **Description**: Supported since version 1.1.1. The maximum number of rows that can be accumulated in memory before being sent to StarRocks at a time. ##### `starrocks.write.flush.interval.ms`[​](#starrockswriteflushintervalms "Direct link to starrockswriteflushintervalms") * **Required**: NO * **Default value**: `300000` * **Description**: The interval at which data is sent to StarRocks. This parameter is used to control the loading latency. ##### `starrocks.write.max.retries`[​](#starrockswritemaxretries "Direct link to starrockswritemaxretries") * **Required**: NO * **Default value**: `3` * **Description**: Supported since version 1.1.1. The number of times that the connector retries to perform the Stream Load for the same batch of data if the load fails. note Because Stream Load transaction interface does not support retry. If this parameter is positive, the connector always use Stream Load interface and ignore the value of `starrocks.write.enable.transaction-stream-load`. ##### `starrocks.write.retry.interval.ms`[​](#starrockswriteretryintervalms "Direct link to starrockswriteretryintervalms") * **Required**: NO * **Default value**: `10000` * **Description**: Supported since version 1.1.1. The interval to retry the Stream Load for the same batch of data if the load fails. ##### `starrocks.write.use_bitmap_hash64`[​](#starrockswriteuse_bitmap_hash64 "Direct link to starrockswriteuse_bitmap_hash64") * **Required**: NO * **Default value**: `false` * **Description**: Supported since version 1.1.3. Whether to use the 64-bit hash function to generate bitmap. The default is to use 32-bit hash function. ##### `starrocks.columns`[​](#starrockscolumns "Direct link to starrockscolumns") * **Required**: NO * **Default value**: None * **Description**: The StarRocks table column into which you want to load data. You can specify multiple columns, which must be separated by commas (,), for example, `"col0,col1,col2"`. ##### `starrocks.column.types`[​](#starrockscolumntypes "Direct link to starrockscolumntypes") * **Required**: NO * **Default value**: None * **Description**: Supported since version 1.1.1. Customize the column data types for Spark instead of using the defaults inferred from the StarRocks table and the [default mapping](#data-type-mapping-between-spark-and-starrocks). The parameter value is a schema in DDL format same as the output of Spark [StructType#toDDL](https://github.com/apache/spark/blob/master/sql/api/src/main/scala/org/apache/spark/sql/types/StructType.scala#L449) , such as `col0 INT, col1 STRING, col2 BIGINT`. Note that you only need to specify columns that need customization. One use case is to load data into columns of [BITMAP](#load-data-into-columns-of-bitmap-type) or [HLL](#load-data-into-columns-of-hll-type) type. ##### `starrocks.write.properties.*`[​](#starrockswriteproperties "Direct link to starrockswriteproperties") * **Required**: NO * **Default value**: None * **Description**: The parameters that are used to control Stream Load behavior. For example, the parameter `starrocks.write.properties.format` specifies the format of the data to be loaded, such as CSV or JSON. For a list of supported parameters and their descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ##### `starrocks.write.properties.format`[​](#starrockswritepropertiesformat "Direct link to starrockswritepropertiesformat") * **Required**: NO * **Default value**: `CSV` * **Description**: The file format based on which the Spark connector transforms each batch of data before the data is sent to StarRocks. Valid values: CSV and JSON. ##### `starrocks.write.properties.row_delimiter`[​](#starrockswritepropertiesrow_delimiter "Direct link to starrockswritepropertiesrow_delimiter") * **Required**: NO * **Default value**: `\n` * **Description**: The row delimiter for CSV-formatted data. ##### `starrocks.write.properties.column_separator`[​](#starrockswritepropertiescolumn_separator "Direct link to starrockswritepropertiescolumn_separator") * **Required**: NO * **Default value**: `\t` * **Description**: The column separator for CSV-formatted data. ##### `starrocks.write.properties.partial_update`[​](#starrockswritepropertiespartial_update "Direct link to starrockswritepropertiespartial_update") * **Required**: NO * **Default value**: `FALSE` * **Description**: Whether to use partial updates. Valid values: `TRUE` and `FALSE`. Default value: `FALSE`, indicating to disable this feature. ##### `starrocks.write.properties.partial_update_mode`[​](#starrockswritepropertiespartial_update_mode "Direct link to starrockswritepropertiespartial_update_mode") * **Required**: NO * **Default value**: `row` * **Description**: Specifies the mode for partial updates. Valid values: `row` and `column`. * The value `row` (default) means partial updates in row mode, which is more suitable for real-time updates with many columns and small batches. * The value `column` means partial updates in column mode, which is more suitable for batch updates with few columns and many rows. In such scenarios, enabling the column mode offers faster update speeds. For example, in a table with 100 columns, if only 10 columns (10% of the total) are updated for all rows, the update speed of the column mode is 10 times faster. ##### `starrocks.write.num.partitions`[​](#starrockswritenumpartitions "Direct link to starrockswritenumpartitions") * **Required**: NO * **Default value**: None * **Description**: The number of partitions into which Spark can write data in parallel. When the data volume is small, you can reduce the number of partitions to lower the loading concurrency and frequency. The default value for this parameter is determined by Spark. However, this method may cause Spark Shuffle cost. ##### `starrocks.write.partition.columns`[​](#starrockswritepartitioncolumns "Direct link to starrockswritepartitioncolumns") * **Required**: NO * **Default value**: None * **Description**: The partitioning columns in Spark. The parameter takes effect only when `starrocks.write.num.partitions` is specified. If this parameter is not specified, all columns being written are used for partitioning. ##### `starrocks.timezone`[​](#starrockstimezone "Direct link to starrockstimezone") * **Required**: NO * **Default value**: Default timezone of JVM * **Description**: Supported since 1.1.1. The timezone used to convert Spark `TimestampType` to StarRocks `DATETIME`. The default is the timezone of JVM returned by `ZoneId#systemDefault()`. The format can be a timezone name such as `Asia/Shanghai`, or a zone offset such as `+08:00`. ##### `starrocks.write.socket.timeout.ms`[​](#starrockswritesockettimeoutms "Direct link to starrockswritesockettimeoutms") * **Required**: NO * **Default value**: `-1` * **Description**: Supported since 1.1.3. The time duration for which the HTTP client waits for data. Unit: ms. The default value `-1` means there is no timeout. ##### `starrocks.write.properties.compression`[​](#starrockswritepropertiescompression "Direct link to starrockswritepropertiescompression") * **Required**: NO * **Default value**: None * **Description**: Supported since 1.1.3. The compression algorithm used for Stream Load. Valid values: `lz4_frame`. Compression for JSON format requires the database v3.2.7 or later. Compression for CSV format has no requirement for the database version. #### Data type mapping between Spark and StarRocks[​](#data-type-mapping-between-spark-and-starrocks "Direct link to Data type mapping between Spark and StarRocks") * The default data type mapping is as follows: | Spark data type | StarRocks data type | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | BooleanType | BOOLEAN | | ByteType | TINYINT | | ShortType | SMALLINT | | IntegerType | INT | | LongType | BIGINT | | StringType | LARGEINT | | FloatType | FLOAT | | DoubleType | DOUBLE | | DecimalType | DECIMAL | | StringType | CHAR | | StringType | VARCHAR | | StringType | STRING | | StringType | JSON | | DateType | DATE | | TimestampType | DATETIME | | ArrayType | ARRAY
**NOTE:**
**Supported since version 1.1.1**. For detailed steps, see [Load data into columns of ARRAY type](#load-data-into-columns-of-array-type). | | MapType | MAP
**NOTE:**
**Supported since version 1.1.3**. For detailed steps, see [Load nested columns](#load-nested-columns-struct-array-and-map). | | StructType | STRUCT
**NOTE:**
**Supported since version 1.1.3**. For detailed steps, see [Load nested columns](#load-nested-columns-struct-array-and-map). | * You can also customize the data type mapping. For example, a StarRocks table contains BITMAP and HLL columns, but Spark does not support the two data types. You need to customize the corresponding data types in Spark. For detailed steps, see load data into [BITMAP](#load-data-into-columns-of-bitmap-type) and [HLL](#load-data-into-columns-of-hll-type) columns. **BITMAP and HLL are supported since version 1.1.1**. #### Upgrade Spark connector[​](#upgrade-spark-connector "Direct link to Upgrade Spark connector") ##### Upgrade from version 1.1.0 to 1.1.1[​](#upgrade-from-version-110-to-111 "Direct link to Upgrade from version 1.1.0 to 1.1.1") * Since 1.1.1, the Spark connector does not provide `mysql-connector-java` which is the official JDBC driver for MySQL, because of the limitations of the GPL license used by `mysql-connector-java`. However, the Spark connector still needs the MySQL JDBC driver to connect to StarRocks for the table metadata, so you need to add the driver to the Spark classpath manually. You can find the driver on [MySQL site](https://dev.mysql.com/downloads/connector/j/) or [Maven Central](https://repo1.maven.org/maven2/mysql/mysql-connector-java/). * Since 1.1.1, the connector uses Stream Load interface by default rather than Stream Load transaction interface in version 1.1.0. If you still want to use Stream Load transaction interface, you can set the option `starrocks.write.max.retries` to `0`. Please see the description of `starrocks.write.enable.transaction-stream-load` and `starrocks.write.max.retries` for details. #### Examples[​](#examples "Direct link to Examples") The following examples show how to use the Spark connector to load data into a StarRocks table with Spark DataFrames or Spark SQL. The Spark DataFrames supports both Batch and Structured Streaming modes. For more examples, see [Spark Connector Examples](https://github.com/StarRocks/starrocks-connector-for-apache-spark/tree/main/src/test/java/com/starrocks/connector/spark/examples). ##### Preparations[​](#preparations "Direct link to Preparations") ###### Create a StarRocks table[​](#create-a-starrocks-table "Direct link to Create a StarRocks table") Create a database `test` and create a Primary Key table `score_board`. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Network configuration[​](#network-configuration "Direct link to Network configuration") Ensure that the machine where Spark is located can access the FE nodes of the StarRocks cluster via the `http_port` (default: `8030`) and `query_port` (default: `9030`), and the BE nodes via the `be_http_port` (default: `8040`). ###### Set up your Spark environment[​](#set-up-your-spark-environment "Direct link to Set up your Spark environment") Note that the following examples are run in Spark 3.2.4 and use `spark-shell`, `pyspark` and `spark-sql`. Before running the examples, make sure to place the Spark connector JAR file in the `$SPARK_HOME/jars` directory. ##### Load data with Spark DataFrames[​](#load-data-with-spark-dataframes "Direct link to Load data with Spark DataFrames") The following two examples explain how to load data with Spark DataFrames Batch or Structured Streaming mode. ###### Batch[​](#batch "Direct link to Batch") Construct data in memory and load data into the StarRocks table. 1. You can write the spark application using Scala or Python. For Scala, run the following code snippet in `spark-shell`: ```scala // 1. Create a DataFrame from a sequence. val data = Seq((1, "starrocks", 100), (2, "spark", 100)) val df = data.toDF("id", "name", "score") // 2. Write to StarRocks by configuring the format as "starrocks" and the following options. // You need to modify the options according your own environment. df.write.format("starrocks") .option("starrocks.fe.http.url", "127.0.0.1:8030") .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") .option("starrocks.table.identifier", "test.score_board") .option("starrocks.user", "root") .option("starrocks.password", "") .mode("append") .save() ``` For Python, run the following code snippet in `pyspark`: ```python from pyspark.sql import SparkSession spark = SparkSession \ .builder \ .appName("StarRocks Example") \ .getOrCreate() # 1. Create a DataFrame from a sequence. data = [(1, "starrocks", 100), (2, "spark", 100)] df = spark.sparkContext.parallelize(data) \ .toDF(["id", "name", "score"]) # 2. Write to StarRocks by configuring the format as "starrocks" and the following options. # You need to modify the options according your own environment. df.write.format("starrocks") \ .option("starrocks.fe.http.url", "127.0.0.1:8030") \ .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") \ .option("starrocks.table.identifier", "test.score_board") \ .option("starrocks.user", "root") \ .option("starrocks.password", "") \ .mode("append") \ .save() ``` 2. Query data in the StarRocks table. ```sql MySQL [test]> SELECT * FROM `score_board`; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | spark | 100 | +------+-----------+-------+ 2 rows in set (0.00 sec) ``` ###### Structured Streaming[​](#structured-streaming "Direct link to Structured Streaming") Construct a streaming read of data from a CSV file and load data into the StarRocks table. 1. In the directory `csv-data`, create a CSV file `test.csv` with the following data: ```csv 3,starrocks,100 4,spark,100 ``` 2. You can write the Spark application using Scala or Python. For Scala, run the following code snippet in `spark-shell`: ```scala import org.apache.spark.sql.types.StructType // 1. Create a DataFrame from CSV. val schema = (new StructType() .add("id", "integer") .add("name", "string") .add("score", "integer") ) val df = (spark.readStream .option("sep", ",") .schema(schema) .format("csv") // Replace it with your path to the directory "csv-data". .load("/path/to/csv-data") ) // 2. Write to StarRocks by configuring the format as "starrocks" and the following options. // You need to modify the options according your own environment. val query = (df.writeStream.format("starrocks") .option("starrocks.fe.http.url", "127.0.0.1:8030") .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") .option("starrocks.table.identifier", "test.score_board") .option("starrocks.user", "root") .option("starrocks.password", "") // replace it with your checkpoint directory .option("checkpointLocation", "/path/to/checkpoint") .outputMode("append") .start() ) ``` For Python, run the following code snippet in `pyspark`: ```python from pyspark.sql import SparkSession from pyspark.sql.types import IntegerType, StringType, StructType, StructField spark = SparkSession \ .builder \ .appName("StarRocks SS Example") \ .getOrCreate() # 1. Create a DataFrame from CSV. schema = StructType([ StructField("id", IntegerType()), StructField("name", StringType()), StructField("score", IntegerType()) ]) df = ( spark.readStream .option("sep", ",") .schema(schema) .format("csv") # Replace it with your path to the directory "csv-data". .load("/path/to/csv-data") ) # 2. Write to StarRocks by configuring the format as "starrocks" and the following options. # You need to modify the options according your own environment. query = ( df.writeStream.format("starrocks") .option("starrocks.fe.http.url", "127.0.0.1:8030") .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") .option("starrocks.table.identifier", "test.score_board") .option("starrocks.user", "root") .option("starrocks.password", "") # replace it with your checkpoint directory .option("checkpointLocation", "/path/to/checkpoint") .outputMode("append") .start() ) ``` 3. Query data in the StarRocks table. ```sql MySQL [test]> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 4 | spark | 100 | | 3 | starrocks | 100 | +------+-----------+-------+ 2 rows in set (0.67 sec) ``` ##### Load data with Spark SQL[​](#load-data-with-spark-sql "Direct link to Load data with Spark SQL") The following example explains how to load data with Spark SQL by using the `INSERT INTO` statement in the [Spark SQL CLI](https://spark.apache.org/docs/latest/sql-distributed-sql-engine-spark-sql-cli.html). 1. Execute the following SQL statement in the `spark-sql`: ```sql -- 1. Create a table by configuring the data source as `starrocks` and the following options. -- You need to modify the options according your own environment. CREATE TABLE `score_board` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.score_board", "starrocks.user"="root", "starrocks.password"="" ); -- 2. Insert two rows into the table. INSERT INTO `score_board` VALUES (5, "starrocks", 100), (6, "spark", 100); ``` 2. Query data in the StarRocks table. ```sql MySQL [test]> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 6 | spark | 100 | | 5 | starrocks | 100 | +------+-----------+-------+ 2 rows in set (0.00 sec) ``` #### Best Practices[​](#best-practices "Direct link to Best Practices") ##### Load data to Primary Key table[​](#load-data-to-primary-key-table "Direct link to Load data to Primary Key table") This section will show how to load data to StarRocks Primary Key table to achieve partial updates, and conditional updates. You can see [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md) for the detailed introduction of these features. These examples use Spark SQL. ###### Preparations[​](#preparations-1 "Direct link to Preparations") Create a database `test` and create a Primary Key table `score_board` in StarRocks. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Partial updates[​](#partial-updates "Direct link to Partial updates") This example will show how to only update data in the column `name` through loading: 1. Insert initial data to StarRocks table in MySQL client. ```sql mysql> INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'spark', 100); mysql> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | spark | 100 | +------+-----------+-------+ 2 rows in set (0.02 sec) ``` 2. Create a Spark table `score_board` in Spark SQL client. * Set the option `starrocks.write.properties.partial_update` to `true` which tells the connector to do partial update. * Set the option `starrocks.columns` to `"id,name"` to tell the connector which columns to write. ```sql CREATE TABLE `score_board` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.score_board", "starrocks.user"="root", "starrocks.password"="", "starrocks.write.properties.partial_update"="true", "starrocks.columns"="id,name" ); ``` 3. Insert data into the table in Spark SQL client, and only update the column `name`. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks-update'), (2, 'spark-update'); ``` 4. Query the StarRocks table in MySQL client. You can see that only values for `name` change, and the values for `score` does not change. ```sql mysql> select * from score_board; +------+------------------+-------+ | id | name | score | +------+------------------+-------+ | 1 | starrocks-update | 100 | | 2 | spark-update | 100 | +------+------------------+-------+ 2 rows in set (0.02 sec) ``` ###### Conditional updates[​](#conditional-updates "Direct link to Conditional updates") This example will show how to do conditional updates according to the values of column `score`. The update for an `id` takes effect only when the new value for `score` is has a greater or equal to the old value. 1. Insert initial data to StarRocks table in MySQL client. ```sql mysql> INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'spark', 100); mysql> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | spark | 100 | +------+-----------+-------+ 2 rows in set (0.02 sec) ``` 2. Create a Spark table `score_board` in the following ways. * Set the option `starrocks.write.properties.merge_condition` to `score` which tells the connector to use the column `score` as the condition. * Make sure that the Spark connector use Stream Load interface to load data, rather than Stream Load transaction interface, because the latter does not support this feature. ```sql CREATE TABLE `score_board` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.score_board", "starrocks.user"="root", "starrocks.password"="", "starrocks.write.properties.merge_condition"="score" ); ``` 3. Insert data to the table in Spark SQL client, and update the row whose `id` is 1 with a smaller score value, and the row whose `id` is 2 with a larger score value. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks-update', 99), (2, 'spark-update', 101); ``` 4. Query the StarRocks table in MySQL client. You can see that only the row whose `id` is 2 changes, and the row whose `id` is 1 does not change. ```sql mysql> select * from score_board; +------+--------------+-------+ | id | name | score | +------+--------------+-------+ | 1 | starrocks | 100 | | 2 | spark-update | 101 | +------+--------------+-------+ 2 rows in set (0.03 sec) ``` ##### Load data into columns of BITMAP type[​](#load-data-into-columns-of-bitmap-type "Direct link to Load data into columns of BITMAP type") [`BITMAP`](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/BITMAP.md) is often used to accelerate count distinct, such as counting UV, see [Use Bitmap for exact Count Distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_bitmap.md). Here we take the counting of UV as an example to show how to load data into columns of the `BITMAP` type. **`BITMAP` is supported since version 1.1.1**. 1. Create a StarRocks Aggregate table. In the database `test`, create an Aggregate table `page_uv` where the column `visit_users` is defined as the `BITMAP` type and configured with the aggregate function `BITMAP_UNION`. ```sql CREATE TABLE `test`.`page_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` BITMAP BITMAP_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`); ``` 2. Create a Spark table. The schema of the Spark table is inferred from the StarRocks table, and the Spark does not support the `BITMAP` type. So you need to customize the corresponding column data type in Spark, for example as `BIGINT`, by configuring the option `"starrocks.column.types"="visit_users BIGINT"`. When using Stream Load to ingest data, the connector uses the [`to_bitmap`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/to_bitmap.md) function to convert the data of `BIGINT` type into `BITMAP` type. Run the following DDL in `spark-sql`: ```sql CREATE TABLE `page_uv` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.page_uv", "starrocks.user"="root", "starrocks.password"="", "starrocks.column.types"="visit_users BIGINT" ); ``` 3. Load data into StarRocks table. Run the following DML in `spark-sql`: ```sql INSERT INTO `page_uv` VALUES (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 13), (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 23), (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 33), (1, CAST('2020-06-23 02:30:30' AS TIMESTAMP), 13), (2, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 23); ``` 4. Calculate page UVs from the StarRocks table. ```sql MySQL [test]> SELECT `page_id`, COUNT(DISTINCT `visit_users`) FROM `page_uv` GROUP BY `page_id`; +---------+-----------------------------+ | page_id | count(DISTINCT visit_users) | +---------+-----------------------------+ | 2 | 1 | | 1 | 3 | +---------+-----------------------------+ 2 rows in set (0.01 sec) ``` note The connector uses [`to_bitmap`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/to_bitmap.md) function to convert data of the `TINYINT`, `SMALLINT`, `INTEGER`, and `BIGINT` types in Spark to the `BITMAP` type in StarRocks, and uses [`bitmap_hash`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_hash.md) or [`bitmap_hash64`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_hash64.md) function for other Spark data types. ##### Load data into columns of HLL type[​](#load-data-into-columns-of-hll-type "Direct link to Load data into columns of HLL type") [`HLL`](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/HLL.md) can be used for approximate count distinct, see [Use HLL for approximate count distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_HLL.md). Here we take the counting of UV as an example to show how to load data into columns of the `HLL` type. **`HLL` is supported since version 1.1.1**. 1. Create a StarRocks Aggregate table. In the database `test`, create an Aggregate table `hll_uv` where the column `visit_users` is defined as the `HLL` type and configured with the aggregate function `HLL_UNION`. ```sql CREATE TABLE `hll_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` HLL HLL_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`); ``` 2. Create a Spark table. The schema of the Spark table is inferred from the StarRocks table, and the Spark does not support the `HLL` type. So you need to customize the corresponding column data type in Spark, for example as `BIGINT`, by configuring the option `"starrocks.column.types"="visit_users BIGINT"`. When using Stream Load to ingest data, the connector uses the [`hll_hash`](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_hash.md) function to convert the data of `BIGINT` type into `HLL` type. Run the following DDL in `spark-sql`: ```sql CREATE TABLE `hll_uv` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.hll_uv", "starrocks.user"="root", "starrocks.password"="", "starrocks.column.types"="visit_users BIGINT" ); ``` 3. Load data into StarRocks table. Run the following DML in `spark-sql`: ```sql INSERT INTO `hll_uv` VALUES (3, CAST('2023-07-24 12:00:00' AS TIMESTAMP), 78), (4, CAST('2023-07-24 13:20:10' AS TIMESTAMP), 2), (3, CAST('2023-07-24 12:30:00' AS TIMESTAMP), 674); ``` 4. Calculate page UVs from the StarRocks table. ```sql MySQL [test]> SELECT `page_id`, COUNT(DISTINCT `visit_users`) FROM `hll_uv` GROUP BY `page_id`; +---------+-----------------------------+ | page_id | count(DISTINCT visit_users) | +---------+-----------------------------+ | 4 | 1 | | 3 | 2 | +---------+-----------------------------+ 2 rows in set (0.01 sec) ``` ##### Load data into columns of ARRAY type[​](#load-data-into-columns-of-array-type "Direct link to Load data into columns of ARRAY type") The following example explains how to load data into columns of the [`ARRAY`](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/Array.md) type. 1. Create a StarRocks table. In the database `test`, create a Primary Key table `array_tbl` that includes one `INT` column and two `ARRAY` columns. ```sql CREATE TABLE `array_tbl` ( `id` INT NOT NULL, `a0` ARRAY, `a1` ARRAY> ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`) ; ``` 2. Write data to StarRocks. Because some versions of StarRocks does not provide the metadata of `ARRAY` column, the connector can not infer the corresponding Spark data type for this column. However, you can explicitly specify the corresponding Spark data type of the column in the option `starrocks.column.types`. In this example, you can configure the option as `a0 ARRAY,a1 ARRAY>`. Run the following codes in `spark-shell`: ```scala val data = Seq( | (1, Seq("hello", "starrocks"), Seq(Seq(1, 2), Seq(3, 4))), | (2, Seq("hello", "spark"), Seq(Seq(5, 6, 7), Seq(8, 9, 10))) | ) val df = data.toDF("id", "a0", "a1") df.write .format("starrocks") .option("starrocks.fe.http.url", "127.0.0.1:8030") .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") .option("starrocks.table.identifier", "test.array_tbl") .option("starrocks.user", "root") .option("starrocks.password", "") .option("starrocks.column.types", "a0 ARRAY,a1 ARRAY>") .mode("append") .save() ``` 3. Query data in the StarRocks table. ```sql MySQL [test]> SELECT * FROM `array_tbl`; +------+-----------------------+--------------------+ | id | a0 | a1 | +------+-----------------------+--------------------+ | 1 | ["hello","starrocks"] | [[1,2],[3,4]] | | 2 | ["hello","spark"] | [[5,6,7],[8,9,10]] | +------+-----------------------+--------------------+ 2 rows in set (0.01 sec) ``` #### Load nested columns (STRUCT, ARRAY, and MAP)[​](#load-nested-columns-struct-array-and-map "Direct link to Load nested columns (STRUCT, ARRAY, and MAP)") Loading nested columns is supported since version 1.1.3. The Spark connector supports writing StarRocks columns of type `STRUCT`, `ARRAY`, and `MAP`. You must declare the StarRocks column types using the `starrocks.column.types` option so the connector can serialize the data correctly. ##### Nested data type mapping[​](#nested-data-type-mapping "Direct link to Nested data type mapping") Nested values are serialized as JSON-compatible strings before being sent to StarRocks via Stream Load: * `STRUCT` columns are serialized as a JSON object: `{"field1": value1, "field2": value2}`. * `ARRAY` columns are serialized as a JSON array: `[value1, value2, ...]`. * `MAP` columns are serialized as a JSON object: `{"key1": value1, "key2": value2}`. ##### Example[​](#example "Direct link to Example") Given the following StarRocks table: ```sql CREATE TABLE nested_tbl ( id INT, info STRUCT, tags ARRAY, attributes MAP ) ENGINE=OLAP DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 4; ``` Write data from Spark: ```scala import org.apache.spark.sql.types._ import org.apache.spark.sql.Row val schema = StructType(Seq( StructField("id", IntegerType), StructField("info", StructType(Seq( StructField("type", StringType), StructField("phone", LongType) ))), StructField("tags", ArrayType(StringType)), StructField("attributes", MapType(StringType, StringType)) )) val data = Seq( Row(1, Row("admin", 123456789L), Seq("spark", "starrocks"), Map("env" -> "prod")) ) val df = spark.createDataFrame(spark.sparkContext.parallelize(data), schema) val columnTypes = "info STRUCT, " + "tags ARRAY, " + "attributes MAP" df.write.format("starrocks") .option("starrocks.fenodes", "127.0.0.1:8030") .option("starrocks.table.identifier", "test.nested_tbl") .option("starrocks.user", "root") .option("starrocks.password", "") .option("starrocks.column.types", columnTypes) .mode("append") .save() ``` --- ### Load data in bulk using Spark Load This load uses external Apache Spark™ resources to pre-process imported data, which improves import performance and saves compute resources. It is mainly used for **initial migration** and **large data import** into StarRocks (data volume up to TB level). Spark load is an **asynchronous** import method that requires users to create Spark-type import jobs via the MySQL protocol and view the import results using `SHOW LOAD`. > **NOTICE** > > * Only users with the INSERT privilege on a StarRocks table can load data into this table. You can follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the required privilege. > * Spark Load can not be used to load data into a Primary Key table. #### Terminology explanation[​](#terminology-explanation "Direct link to Terminology explanation") * **Spark ETL**: Mainly responsible for ETL of data in the import process, including global dictionary construction (BITMAP type), partitioning, sorting, aggregation, etc. * **Broker**: Broker is an independent stateless process. It encapsulates the file system interface and provides StarRocks with the ability to read files from remote storage systems. * **Global Dictionary**: Saves the data structure that maps data from the original value to the encoded value. The original value can be any data type, while the encoded value is an integer. The global dictionary is mainly used in scenarios where exact count distinct is precomputed. #### Fundamentals[​](#fundamentals "Direct link to Fundamentals") The user submits a Spark type import job through the MySQL client;the FE records the metadata and returns the submission result. The execution of the spark load task is divided into the following main phases. 1. The user submits the spark load job to the FE. 2. The FE schedules the submission of the ETL task to the Apache Spark™ cluster for execution. 3. The Apache Spark™ cluster executes the ETL task that includes global dictionary construction (BITMAP type), partitioning, sorting, aggregation, etc. 4. After the ETL task is completed, the FE gets the data path of each preprocessed slice and schedules the relevant BE to execute the Push task. 5. The BE reads data through Broker process from HDFS and converts it into StarRocks storage format. > If you choose not to use Broker process, the BE reads data from HDFS directly. 6. The FE schedules the effective version and completes the import job. The following diagram illustrates the main flow of spark load. ![Spark load](/assets/images/4.3.2-1-7104f83d68f711e7dd819b5f11391eb7.png) *** #### Global Dictionary[​](#global-dictionary "Direct link to Global Dictionary") ##### Applicable Scenarios[​](#applicable-scenarios "Direct link to Applicable Scenarios") Currently, the BITMAP column in StarRocks is implemented using the Roaringbitmap, which only has integer to be the input data type. So if you want to implement precomputation for the BITMAP column in the import process, then you need to convert the input data type to integer. In the existing import process of StarRocks, the data structure of the global dictionary is implemented based on the Hive table, which saves the mapping from the original value to the encoded value. ##### Build Process[​](#build-process "Direct link to Build Process") 1. Read the data from the upstream data source and generate a temporary Hive table, named `hive-table`. 2. Extract the values of the de-emphasized fields of `hive-table` to generate a new Hive table named `distinct-value-table`. 3. Create a new global dictionary table named `dict-table` with one column for the original values and one column for the encoded values. 4. Left join between `distinct-value-table` and `dict-table`, and then use the window function to encode this set. Finally both the original value and the encoded value of the de-duplicated column are written back to `dict-table`. 5. Join between `dict-table` and `hive-table` to finish the job of replacing the original value in `hive-table` with the integer encoded value. 6. `hive-table` will be read by the next time data pre-processing, and then imported into StarRocks after calculation. #### Data Pre-processing[​](#data-pre-processing "Direct link to Data Pre-processing") The basic process of data pre-processing is as follows: 1. Read data from the upstream data source (HDFS file or Hive table). 2. Complete field mapping and calculation for the read data, then generate `bucket-id` based on the partition information. 3. Generate RollupTree based on the Rollup metadata of StarRocks table. 4. Iterate through the RollupTree and perform hierarchical aggregation operations. The Rollup of the next hierarchy can be calculated from the Rollup of the previous hierarchy. 5. Each time the aggregation calculation is completed, the data is bucketed according to `bucket-id` and then written to HDFS. 6. The subsequent Broker process will pull the files from HDFS and import them into the StarRocks BE node. #### Basic Operations[​](#basic-operations "Direct link to Basic Operations") ##### Configuring ETL Clusters[​](#configuring-etl-clusters "Direct link to Configuring ETL Clusters") Apache Spark™ is used as an external computational resource in StarRocks for ETL work. There may be other external resources added to StarRocks, such as Spark/GPU for query, HDFS/S3 for external storage, MapReduce for ETL, etc. Therefore, we introduce `Resource Management` to manage these external resources used by StarRocks. Before submitting a Apache Spark™ import job, configure the Apache Spark™ cluster for performing ETL tasks. The syntax for operation is as follows: ```sql -- create Apache Spark™ resource CREATE EXTERNAL RESOURCE resource_name PROPERTIES ( type = spark, spark_conf_key = spark_conf_value, working_dir = path, broker = broker_name, broker.property_key = property_value ); -- drop Apache Spark™ resource DROP RESOURCE resource_name; -- show resources SHOW RESOURCES SHOW PROC "/resources"; -- privileges GRANT USAGE_PRIV ON RESOURCE resource_name TO user_identityGRANT USAGE_PRIV ON RESOURCE resource_name TO ROLE role_name; REVOKE USAGE_PRIV ON RESOURCE resource_name FROM user_identityREVOKE USAGE_PRIV ON RESOURCE resource_name FROM ROLE role_name; ``` * Create resource **For example**: ```sql -- yarn cluster mode CREATE EXTERNAL RESOURCE "spark0" PROPERTIES ( "type" = "spark", "spark.master" = "yarn", "spark.submit.deployMode" = "cluster", "spark.jars" = "xxx.jar,yyy.jar", "spark.files" = "/tmp/aaa,/tmp/bbb", "spark.executor.memory" = "1g", "spark.yarn.queue" = "queue0", "spark.hadoop.yarn.resourcemanager.address" = "127.0.0.1:9999", "spark.hadoop.fs.defaultFS" = "hdfs://127.0.0.1:10000", "working_dir" = "hdfs://127.0.0.1:10000/tmp/starrocks", "broker" = "broker0", "broker.username" = "user0", "broker.password" = "password0" ); -- yarn HA cluster mode CREATE EXTERNAL RESOURCE "spark1" PROPERTIES ( "type" = "spark", "spark.master" = "yarn", "spark.submit.deployMode" = "cluster", "spark.hadoop.yarn.resourcemanager.ha.enabled" = "true", "spark.hadoop.yarn.resourcemanager.ha.rm-ids" = "rm1,rm2", "spark.hadoop.yarn.resourcemanager.hostname.rm1" = "host1", "spark.hadoop.yarn.resourcemanager.hostname.rm2" = "host2", "spark.hadoop.fs.defaultFS" = "hdfs://127.0.0.1:10000", "working_dir" = "hdfs://127.0.0.1:10000/tmp/starrocks", "broker" = "broker1" ); ``` `resource-name` is the name of the Apache Spark™ resource configured in StarRocks. `PROPERTIES` includes parameters relating to the Apache Spark™ resource, as follows: > **Note** > > For detailed description of Apache Spark™ resource PROPERTIES, please see [CREATE RESOURCE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/CREATE_RESOURCE.md) * Spark related parameters: * `type`: Resource type, required, currently only supports `spark`. * `spark.master`: Required, currently only supports `yarn`. * `spark.submit.deployMode`: The deployment mode of the Apache Spark™ program, required, currently supports both `cluster` and `client`. * `spark.hadoop.fs.defaultFS`: Required if master is yarn. * Parameters related to yarn resource manager, required. * one ResourceManager on a single node `spark.hadoop.yarn.resourcemanager.address`: Address of the single point resource manager. * ResourceManager HA > You can choose to specify ResourceManager's hostname or address. * `spark.hadoop.yarn.resourcemanager.ha.enabled`: Enable the resource manager HA, set to `true`. * `spark.hadoop.yarn.resourcemanager.ha.rm-ids`: list of resource manager logical ids. * `spark.hadoop.yarn.resourcemanager.hostname.rm-id`: For each rm-id, specify the hostname corresponding to the resource manager. * `spark.hadoop.yarn.resourcemanager.address.rm-id`: For each rm-id, specify `host:port` for the client to submit jobs to. * `*working_dir`: The directory used by ETL. Required if Apache Spark™ is used as an ETL resource. For example: `hdfs://host:port/tmp/starrocks`. * Broker related parameters: * `broker`: Broker name. Required if Apache Spark™ is used as an ETL resource. You need to use the `ALTER SYSTEM ADD BROKER` command to complete the configuration in advance. * `broker.property_key`: Information (e.g.authentication information) to be specified when Broker process reads the intermediate file generated by the ETL. **Precaution**: The above is a description of parameters for loading through Broker process. If you intend to load data without Broker process, the following should be noted. * You do not need to specify `broker`. * If you need to configure user authentication, and HA for NameNode nodes, you need to configure the parameters in the hdfs-site.xml file in the HDFS cluster, see [broker\_properties](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#hdfs) for descriptions of parameters. and you need to move the **hdfs-site.xml** file under **$FE\_HOME/conf** for each FE and **$BE\_HOME/conf** for each BE. > Note > > If the HDFS file can only be accessed by a specific user, you still need to specify the HDFS username in `broker.name` and the user password in `broker.password`. * View resources Regular accounts can only view resources to which they have `USAGE-PRIV` access. The root and admin accounts can view all resources. * Resource Permissions Resource permissions are managed through `GRANT REVOKE`, which currently only supports `USAGE-PRIV` permissions. You can give `USAGE-PRIV` permissions to a user or a role. ```sql -- Grant access to spark0 resources to user0 GRANT USAGE_PRIV ON RESOURCE "spark0" TO "user0"@"%"; -- Grant access to spark0 resources to role0 GRANT USAGE_PRIV ON RESOURCE "spark0" TO ROLE "role0"; -- Grant access to all resources to user0 GRANT USAGE_PRIV ON RESOURCE* TO "user0"@"%"; -- Grant access to all resources to role0 GRANT USAGE_PRIV ON RESOURCE* TO ROLE "role0"; -- Revoke the use privileges of spark0 resources from user user0 REVOKE USAGE_PRIV ON RESOURCE "spark0" FROM "user0"@"%"; ``` ##### Configuring Spark Client[​](#configuring-spark-client "Direct link to Configuring Spark Client") Configure the Spark client for FE so that the latter can submit Spark tasks by executing the `spark-submit` command. It is recommended to use the official version of Spark2 2.4.5 or above [spark download address](https://archive.apache.org/dist/spark/). After downloading, please use the following steps to complete the configuration. * Configure `SPARK-HOME` Place the Spark client in a directory on the same machine as the FE, and configure `spark_home_default_dir` in the FE configuration file to this directory, which by default is the `lib/spark2x` path in the FE root directory, and cannot be empty. * **Configure SPARK dependency package** To configure the dependency package, zip and archive all jar files in the jars folder under the Spark client, and configure the `spark_resource_path` item in the FE configuration to this zip file. If this configuration is empty, the FE will try to find the `lib/spark2x/jars/spark-2x.zip` file in the FE root directory. If the FE fails to find it, it will report an error. When the spark load job is submitted, the archived dependency files will be uploaded to the remote repository. The default repository path is under the `working_dir/{cluster_id}` directory named with `--spark-repository--{resource-name}`, which means that a resource in the cluster corresponds to a remote repository. The directory structure is referenced as follows: ```bash ---spark-repository--spark0/ |---archive-1.0.0/ | |\---lib-990325d2c0d1d5e45bf675e54e44fb16-spark-dpp-1.0.0\-jar-with-dependencies.jar | |\---lib-7670c29daf535efe3c9b923f778f61fc-spark-2x.zip |---archive-1.1.0/ | |\---lib-64d5696f99c379af2bee28c1c84271d5-spark-dpp-1.1.0\-jar-with-dependencies.jar | |\---lib-1bbb74bb6b264a270bc7fca3e964160f-spark-2x.zip |---archive-1.2.0/ | |-... ``` In addition to the spark dependencies (named `spark-2x.zip` by default), the FE also uploads the DPP dependencies to the remote repository. If all the dependencies submitted by the spark load already exist in the remote repository, then there is no need to upload the dependencies again, saving the time of repeatedly uploading a large number of files each time. ##### Configuring YARN Client[​](#configuring-yarn-client "Direct link to Configuring YARN Client") Configure the yarn client for FE so that the FE can execute yarn commands to get the status of the running application or kill it.It is recommended to use the official version of Hadoop2 2.5.2 or above ([hadoop download address](https://archive.apache.org/dist/hadoop/common/)). After downloading, please use the following steps to complete the configuration: * **Configure the YARN executable path** Place the downloaded yarn client in a directory on the same machine as the FE, and configure the `yarn_client_path` item in the FE configuration file to the binary executable file of yarn, which by default is the `lib/yarn-client/hadoop/bin/yarn` path in the FE root directory. * **Configure the path to the configuration file needed to generate YARN (optional)** When the FE goes through the yarn client to get the status of the application, or to kill the application, by default StarRocks generates the configuration file required to execute the yarn command in the `lib/yarn-config` path of the FE root directory This path can be modified by configuring the `yarn_config_dir` entry in the FE configuration file, which currently includes `core-site.xml` and `yarn-site.xml`. ##### Create Import Job[​](#create-import-job "Direct link to Create Import Job") **Syntax:** ```sql LOAD LABEL load_label (data_desc, ...) WITH RESOURCE resource_name [resource_properties] [PROPERTIES (key1=value1, ... )] * load_label: db_name.label_name * data_desc: DATA INFILE ('file_path', ...) [NEGATIVE] INTO TABLE tbl_name [PARTITION (p1, p2)] [COLUMNS TERMINATED BY separator ] [(col1, ...)] [COLUMNS FROM PATH AS (col2, ...)] [SET (k1=f1(xx), k2=f2(xx))] [WHERE predicate] DATA FROM TABLE hive_external_tbl [NEGATIVE] INTO TABLE tbl_name [PARTITION (p1, p2)] [SET (k1=f1(xx), k2=f2(xx))] [WHERE predicate] * resource_properties: (key2=value2, ...) ``` **Example 1**: The case where the upstream data source is HDFS ```sql LOAD LABEL db1.label1 ( DATA INFILE("hdfs://abc.com:8888/user/starrocks/test/ml/file1") INTO TABLE tbl1 COLUMNS TERMINATED BY "," (tmp_c1,tmp_c2) SET ( id=tmp_c2, name=tmp_c1 ), DATA INFILE("hdfs://abc.com:8888/user/starrocks/test/ml/file2") INTO TABLE tbl2 COLUMNS TERMINATED BY "," (col1, col2) where col1 > 1 ) WITH RESOURCE 'spark0' ( "spark.executor.memory" = "2g", "spark.shuffle.compress" = "true" ) PROPERTIES ( "timeout" = "3600" ); ``` **Example 2**: The case where the upstream data source is Hive. * Step 1: Create a new hive resource ```sql CREATE EXTERNAL RESOURCE hive0 PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:8080" ); ``` * Step 2: Create a new hive external table ```sql CREATE EXTERNAL TABLE hive_t1 ( k1 INT, K2 SMALLINT, k3 varchar(50), uuid varchar(100) ) ENGINE=hive PROPERTIES ( "resource" = "hive0", "database" = "tmp", "table" = "t1" ); ``` * Step 3: Submit the load command, requiring that the columns in the imported StarRocks table exist in the hive external table. ```sql LOAD LABEL db1.label1 ( DATA FROM TABLE hive_t1 INTO TABLE tbl1 SET ( uuid=bitmap_dict(uuid) ) ) WITH RESOURCE 'spark0' ( "spark.executor.memory" = "2g", "spark.shuffle.compress" = "true" ) PROPERTIES ( "timeout" = "3600" ); ``` Introduction to the parameters in the Spark load: * **Label** Label of the import job. Each import job has a Label that is unique within the database, following the same rules as broker load. * **Data description class parameters** Currently, supported data sources are CSV and Hive table. Other rules are the same as broker load. * **Import Job Parameters** Import job parameters refer to the parameters belonging to the `opt_properties` section of the import statement. These parameters are applicable to the entire import job. The rules are the same as broker load. * **Spark Resource Parameters** Spark resources need to be configured into StarRocks in advance and users need to be given USAGE-PRIV permissions before they can apply the resources to Spark load. Spark resource parameters can be set when the user has a temporary need, such as adding resources for a job and modifying Spark configs. The setting only takes effect on this job and does not affect the existing configurations in the StarRocks cluster. ```sql WITH RESOURCE 'spark0' ( "spark.driver.memory" = "1g", "spark.executor.memory" = "3g" ) ``` * **Import when the data source is Hive** Currently, to use a Hive table in the import process, you need to create an external table of the `Hive` type and then specify its name when submitting the import command. * **Import process to build a global dictionary** In the load command, you can specify the required fields for building the global dictionary in the following format: `StarRocks field name=bitmap_dict(hive table field name)` Note that currently **the global dictionary is only supported when the upstream data source is a Hive table**. * **Load binary type data** Since v2.5.17, Spark Load supports the bitmap\_from\_binary function, which can convert binary data into bitmap data. If the column type of the Hive table or HDFS file is binary and the corresponding column in the StarRocks table is a bitmap-type aggregate column, you can specify the fields in the load command in the following format, `StarRocks field name=bitmap_from_binary(Hive table field name)`. This eliminates the need for building a global dictionary. #### Viewing Import Jobs[​](#viewing-import-jobs "Direct link to Viewing Import Jobs") The Spark load import is asynchronous, as is the broker load. The user must record the label of the import job and use it in the `SHOW LOAD` command to view the import results. The command to view the import is common to all import methods. The example is as follows. Refer to Broker Load for a detailed explanation of returned parameters.The differences are as follows. ```sql mysql> show load order by createtime desc limit 1\G *************************** 1. row *************************** JobId: 76391 Label: label1 State: FINISHED Progress: ETL:100%; LOAD:100% Type: SPARK EtlInfo: unselected.rows=4; dpp.abnorm.ALL=15; dpp.norm.ALL=28133376 TaskInfo: cluster:cluster0; timeout(s):10800; max_filter_ratio:5.0E-5 ErrorMsg: N/A CreateTime: 2019-07-27 11:46:42 EtlStartTime: 2019-07-27 11:46:44 EtlFinishTime: 2019-07-27 11:49:44 LoadStartTime: 2019-07-27 11:49:44 LoadFinishTime: 2019-07-27 11:50:16 URL: http://1.1.1.1:8089/proxy/application_1586619723848_0035/ JobDetails: {"ScannedRows":28133395,"TaskNumber":1,"FileNumber":1,"FileSize":200000} ``` * **State** The current stage of the imported job. PENDING: The job is committed. ETL: Spark ETL is committed. LOADING: The FE schedule an BE to execute push operation. FINISHED: The push is completed and the version is effective. There are two final stages of the import job – `CANCELLED` and `FINISHED`, both indicating the load job is completed. `CANCELLED` indicates import failure and `FINISHED` indicates import success. * **Progress** Description of the import job progress. There are two types of progress –ETL and LOAD, which correspond to the two phases of the import process, ETL and LOADING. * The range of progress for LOAD is 0~100%. `LOAD progress = the number of currently completed tablets of all replications imports / the total number of tablets of this import job * 100%`. * If all tables have been imported, the LOAD progress is 99%, and changes to 100% when the import enters the final validation phase. * The import progress is not linear. If there is no change in progress for a period of time, it does not mean that the import is not executing. * **Type** The type of the import job. SPARK for spark load. * **CreateTime/EtlStartTime/EtlFinishTime/LoadStartTime/LoadFinishTime** These values represent the time when the import was created, when the ETL phase started, when the ETL phase completed, when the LOADING phase started, and when the entire import job was completed. * **JobDetails** Displays the detailed running status of the job, including the number of imported files, total size (in bytes), number of subtasks, number of raw rows being processed, etc. For example: ```json {"ScannedRows":139264,"TaskNumber":1,"FileNumber":1,"FileSize":940754064} ``` * **URL** You can copy the input to your browser to access the web interface of the corresponding application. ##### View Apache Spark™ Launcher commit logs[​](#view-apache-spark-launcher-commit-logs "Direct link to View Apache Spark™ Launcher commit logs") Sometimes users need to view the detailed logs generated during a Apache Spark™ job commit. By default, the logs are saved in the path `log/spark_launcher_log` in the FE root directory named as `spark-launcher-{load-job-id}-{label}.log`. The logs are saved in this directory for a period of time and will be erased when the import information in FE metadata is cleaned up. The default retention time is 3 days. ##### Cancel Import[​](#cancel-import "Direct link to Cancel Import") When the Spark load job status is not `CANCELLED` or `FINISHED`, it can be cancelled manually by the user by specifying the Label of the import job. *** #### Related System Configurations[​](#related-system-configurations "Direct link to Related System Configurations") **FE Configuration:** The following configuration is the system-level configuration of Spark load, which applies to all Spark load import jobs. The configuration values can be adjusted mainly by modifying `fe.conf`. * enable-spark-load: Enable Spark load and resource creation with a default value of false. * spark-load-default-timeout-second: The default timeout for the job is 259200 seconds (3 days). * spark-home-default-dir: The Spark client path (`fe/lib/spark2x`). * spark-resource-path: The path to the packaged S park dependency file (empty by default). * spark-launcher-log-dir: The directory where the commit log of the Spark client is stored (`fe/log/spark-launcher-log`). * yarn-client-path: The path to the yarn binary executable (`fe/lib/yarn-client/hadoop/bin/yarn`). * yarn-config-dir: Yarn's configuration file path (`fe/lib/yarn-config`). *** #### Best Practices[​](#best-practices "Direct link to Best Practices") The most suitable scenario for using Spark load is when the raw data is in the file system (HDFS) and the data volume is in the tens of GB to TB level. Use Stream Load or Broker Load for smaller data volumes. For the full spark load import example, refer to the demo on github: #### FAQs[​](#faqs "Direct link to FAQs") * `Error: When running with master 'yarn' either HADOOP-CONF-DIR or YARN-CONF-DIR must be set in the environment.` Using Spark Load without configuring the `HADOOP-CONF-DIR` environment variable in `spark-env.sh` of the Spark client. * `Error: Cannot run program "xxx/bin/spark-submit": error=2, No such file or directory` The `spark_home_default_dir` configuration item does not specify the Spark client root directory when using Spark Load. * `Error: File xxx/jars/spark-2x.zip does not exist.` The `spark-resource-path` configuration item does not point to the packed zip file when using Spark load. * `Error: yarn client does not exist in path: xxx/yarn-client/hadoop/bin/yarn` The yarn-client-path configuration item does not specify the yarn executable when using Spark load. * `ERROR: Cannot execute hadoop-yarn/bin/... /libexec/yarn-config.sh` When using Hadoop with CDH, you need to configure the `HADOOP_LIBEXEC_DIR` environment variable. Since `hadoop-yarn` and hadoop directories are different, the default `libexec` directory will look for `hadoop-yarn/bin/... /libexec`, while `libexec` is in the hadoop directory. The \`\`\`yarn application status\`\` command to get the Spark task status reported an error causing the import job to fail. --- ### SQL Transaction Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) Start a simple SQL transaction to commit multiple DML statements in a batch. #### Overview[​](#overview "Direct link to Overview") From v3.5.0, StarRocks supports SQL transactions to assure the atomicity of the updated tables when manipulating data within multiple tables. A transaction consists of multiple SQL statements that are processed within the same atomic unit. The statements in the transaction are either applied or undone together, thus guaranteeing the ACID (atomicity, consistency, isolation, and durability) properties of the transaction. Currently, the SQL transaction in StarRocks supports the following operations: * INSERT INTO * UPDATE * DELETE note * INSERT OVERWRITE is not supported currently. * Multiple INSERT statements against the same table within a transaction are supported only in shared-data clusters from v4.0 onwards. * UPDATE and DELETE are supported only in shared-data clusters from v4.0 onwards. From v4.0 onwards, within one SQL transaction: * **Multiple INSERT statements** against the one table are supported. * **Only one UPDATE *OR* DELETE** statement against one table is allowed. * **An UPDATE *OR* DELETE** statement **after** INSERT statements against the same table is **not allowed**. The ACID properties of the transaction are guaranteed only on the limited READ COMMITTED isolation level, that is: * A statement operates only on data that was committed before the statement began. * Two successive statements within the same transaction may operate on different data if another transaction is committed between the execution of the first and the second statements. * Data changes brought by preceding DML statements are invisible to subsequent statements within the same transaction. A transaction is associated with a single session. Multiple sessions cannot share the same transaction. #### Usage[​](#usage "Direct link to Usage") 1. A transaction must be started by executing a START TRANSACTION statement. StarRocks also supports the synonym BEGIN. ```sql { START TRANSACTION | BEGIN [ WORK ] } ``` 2. After starting the transaction, you can define multiple DML statements in the transaction. For detailed information, see [Usage notes](#usage-notes). 3. A transaction must be ended explicitly by executing `COMMIT` or `ROLLBACK`. * To apply (commit) the transaction, use the following syntax: ```sql COMMIT [ WORK ] ``` * To undo (roll back) the transaction, use the following syntax: ```sql ROLLBACK [ WORK ] ``` #### Example[​](#example "Direct link to Example") 1. Create the demo table `desT` in a shared-data cluster, and load data into it. note If you want to try this example in a shared-nothing cluster, you must skip Step 3 and define only one INSERT statement in Step 4. ```sql CREATE TABLE desT ( k int, v int ) PRIMARY KEY(k); INSERT INTO desT VALUES (1,1), (2,2), (3,3); ``` 2. Start a transaction. ```sql START TRANSACTION; ``` Or ```sql BEGIN WORK; ``` 3. Define an UPDATE or DELETE statement. ```sql UPDATE desT SET v = v + 1 WHERE k = 1, ``` Or ```sql DELETE FROM desT WHERE k = 1; ``` 4. Define multiple INSERT statements. ```sql -- Insert data with specified values. INSERT INTO desT VALUES (4,4); -- Insert data from a native table to another. INSERT INTO desT SELECT * FROM srcT; -- Insert data from remote storage. INSERT INTO desT SELECT * FROM FILES( "path" = "s3://inserttest/parquet/srcT.parquet", "format" = "parquet", "aws.s3.access_key" = "XXXXXXXXXX", "aws.s3.secret_key" = "YYYYYYYYYY", "aws.s3.region" = "us-west-2" ); ``` 5. Apply or undo the transaction. * To apply the SQL statements in the transaction. ```sql COMMIT WORK; ``` * To undo the SQL statements in the transaction. ```sql ROLLBACK WORK; ``` #### Usage notes[​](#usage-notes "Direct link to Usage notes") * Currently, StarRocks supports SELECT, INSERT, UPDATE, and DELETE statements in SQL transactions. UPDATE and DELETE are supported only in shared-data clusters from v4.0 onwards. * SELECT statements against the tables whose data have been changed in the same transaction are not allowed. * Multiple INSERT statements against the same table within a transaction are supported only in shared-data clusters from v4.0 onwards. * Within a transaction, you can only define one UPDATE or DELETE statement against each table, and it must precede the INSERT statements. * Subsequent DML statements cannot read the uncommitted changes brought by preceding statements within the same transaction. For example, the target table of the preceding INSERT statement cannot be the source table of subsequent statements. Otherwise, the system returns an error. * For the same reason, a partial column update (an INSERT that writes only a subset of a Primary Key table's columns) against a table that was already modified earlier in the same transaction is not allowed. A partial update must implicitly read the table's other columns to complete each row, which would read the preceding statements' uncommitted changes. Otherwise, the system returns an error. * All target tables of the DML statements in a transaction must be within the same database. Cross-database operations are not allowed. * Currently, INSERT OVERWRITE is not supported. * Nesting transactions are not allowed. You cannot specify BEGIN WORK within a BEGIN-COMMIT/ROLLBACK pair. * If the session where an on-going transaction belongs is terminated or closed, the transaction is automatically rolled back. * StarRock only supports limited READ COMMITTED for Transaction Isolation Level as described above. * Write conflict checks are not supported. When two transactions write to the same table simultaneously, both transactions can be committed successfully. The visibility (order) of the data changes depends on the execution order of the COMMIT WORK statements. --- ### Load data using Stream Load transaction interface From v2.4 onwards, StarRocks provides a Stream Load transaction interface to implement two-phase commit (2PC) for transactions that are run to load data from external systems such as Apache Flink® and Apache Kafka®. The Stream Load transaction interface helps improve the performance of highly concurrent stream loads. From v4.0 onwards, the Stream Load transaction interface supports Multi-table Transaction, that is, loading data into multiple tables within the same database. This topic describes the Stream Load transaction interface and how to load data into StarRocks by using this interface. #### Description[​](#description "Direct link to Description") The Stream Load transaction interface supports using an HTTP protocol-compatible tool or language to call API operations. This topic uses curl as an example to explain how to use this interface. This interface provides various features, such as transaction management, data write, transaction pre-commit, transaction deduplication, and transaction timeout management. note Stream Load supports CSV and JSON file formats. This method is recommended if you want to load data from a small number of files whose individual sizes do not exceed 10 GB. Stream Load does not support Parquet file format. If you need to load data from Parquet files, use [INSERT+files()](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-directly-from-files-in-an-external-source-using-files). ##### Transaction management[​](#transaction-management "Direct link to Transaction management") The Stream Load transaction interface provides the following API operations, which are used to manage transactions: * `/api/transaction/begin`: starts a new transaction. * `/api/transaction/prepare`: pre-commits the current transaction and make data changes temporarily persistent. After you pre-commit a transaction, you can proceed to commit or roll back the transaction. If your cluster crashes after a transaction is pre-committed, you can still proceed to commit the transaction after the cluster is restored. * `/api/transaction/commit`: commits the current transaction to make data changes persistent. * `/api/transaction/rollback`: rolls back the current transaction to abort data changes. > **NOTE** > > After the transaction is pre-committed, do not continue to write data using the transaction. If you continue to write data using the transaction, your write request returns errors. The following diagram shows the relationship between transaction states and operations: ```mermaid stateDiagram-v2 direction LR [*] --> PREPARE : begin PREPARE --> PREPARED : prepare PREPARE --> ABORTED : rollback PREPARED --> COMMITTED : commit PREPARED --> ABORTED : rollback ``` ##### Data write[​](#data-write "Direct link to Data write") The Stream Load transaction interface provides the `/api/transaction/load` operation, which is used to write data. You can call this operation multiple times within one transaction. From v4.0 onwards, you can call `/api/transaction/load` operations on different tables to load data into multiple tables within the same database. ##### Transaction deduplication[​](#transaction-deduplication "Direct link to Transaction deduplication") The Stream Load transaction interface carries over the labeling mechanism of StarRocks. You can bind a unique label to each transaction to achieve at-most-once guarantees for transactions. ##### Transaction timeout management[​](#transaction-timeout-management "Direct link to Transaction timeout management") When you begin a transaction, you can use the `timeout` field in the HTTP request header to specify a timeout period (in seconds) for the transaction from `PREPARE` to `PREPARED` state. If the transaction has not been prepared after this period, it will be automatically aborted. If this field is not specified, the default value is determined by the FE configuration [`stream_load_default_timeout_second`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#stream_load_default_timeout_second) (Default: 600 seconds). When you begin a transaction, you can also use the `idle_transaction_timeout` field in the HTTP request header to specify a timeout period (in seconds) within which the transaction can stay idle. If no data is written within this period, the transaction will be automatically rolled back. When you prepare a transaction, you can use the `prepared_timeout` field in the HTTP request header to specify a timeout period (in seconds) for the transaction from `PREPARED` to `COMMITTED` state. If the transaction has not been committed after this period, it will be automatically aborted. If this field is not specified, the default value is determined by the FE configuration [`prepared_transaction_default_timeout_second`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#prepared_transaction_default_timeout_second) (Default: 86400 seconds). `prepared_timeout` is supported from v3.5.4 onwards. #### Benefits[​](#benefits "Direct link to Benefits") The Stream Load transaction interface brings the following benefits: * **Exactly-once semantics** A transaction is split into two phases, pre-commit and commit, which make it easy to load data across systems. For example, this interface can guarantee exactly-once semantics for data loads from Flink. * **Improved load performance** If you run a load job by using a program, the Stream Load transaction interface allows you to merge multiple mini-batches of data on demand and then send them all at once within one transaction by calling the `/api/transaction/commit` operation. As such, fewer data versions need to be loaded, and load performance is improved. #### Limits[​](#limits "Direct link to Limits") The Stream Load transaction interface has the following limits: * **Single-database multi-table** transactions are supported from v4.0 onwards. Support for **multi-database multi-table** transactions is in development. * Only **concurrent data writes from one client** are supported. Support for **concurrent data writes from multiple clients** is in development. * The `/api/transaction/load` operation can be called multiple times within one transaction. In this case, the parameter settings (except `table`) specified for all of the `/api/transaction/load` operations that are called must be the same. * When you load CSV-formatted data by using the Stream Load transaction interface, make sure that each data record in your data file ends with a row delimiter. #### Precautions[​](#precautions "Direct link to Precautions") * If the `/api/transaction/begin`, `/api/transaction/load`, or `/api/transaction/prepare` operation that you have called returns errors, the transaction fails and is automatically rolled back. * When calling the `/api/transaction/begin` operation to start a new transaction, you must specify a label. Note that the subsequent `/api/transaction/load`, `/api/transaction/prepare`, and `/api/transaction/commit` operations must use the same label as the `/api/transaction/begin` operation. * If you use the label of an ongoing transaction to call the `/api/transaction/begin` operation to start a new transaction, the previous transaction will fail and be rolled back. * If you use a multi-table transaction to load data into different tables, you must specify the parameter `-H "transaction_type:multi"` for all operations involved in the transaction. * The default column separator and row delimiter that StarRocks supports for CSV-formatted data are `\t` and `\n`. If your data file does not use the default column separator or row delimiter, you must use `"column_separator: "` or `"row_delimiter: "` to specify the column separator or row delimiter that is actually used in your data file when calling the `/api/transaction/load` operation. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ###### Check network configuration[​](#check-network-configuration "Direct link to Check network configuration") Make sure that the machine on which the data you want to load resides can access the FE and BE nodes of the StarRocks cluster via the [`http_port`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#http_port) (default: `8030`) and [`be_http_port`](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#be_http_port) (default: `8040`) , respectively. #### Basic operations[​](#basic-operations "Direct link to Basic operations") ##### Prepare sample data[​](#prepare-sample-data "Direct link to Prepare sample data") This topic uses CSV-formatted data as an example. 1. In the `/home/disk1/` path of your local file system, create a CSV file named `example1.csv`. The file consists of three columns, which represent the user ID, user name, and user score in sequence. ```plain 1,Lily,23 2,Rose,23 3,Alice,24 4,Julia,25 ``` 2. In your StarRocks database `test_db`, create a Primary Key table named `table1`. The table consists of three columns: `id`, `name`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `table1` ( `id` int(11) NOT NULL COMMENT "user ID", `name` varchar(65533) NULL COMMENT "user name", `score` int(11) NOT NULL COMMENT "user score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`) BUCKETS 10; ``` ##### Start a transaction[​](#start-a-transaction "Direct link to Start a transaction") ###### Syntax[​](#syntax "Direct link to Syntax") ```bash curl --location-trusted -u : -H "label:" \ -H "Expect:100-continue" \ [-H "transaction_type:multi"]\ # Optional. Initiates a multi-table transaction. -H "db:" -H "table:" \ -XPOST http://:/api/transaction/begin ``` > **NOTE** > > Specify `-H "transaction_type:multi"` in the command if you want to load data into different tables within the transaction. ###### Example[​](#example "Direct link to Example") ```bash curl --location-trusted -u :<123456> -H "label:streamload_txn_example1_table1" \ -H "Expect:100-continue" \ -H "db:test_db" -H "table:table1" \ -XPOST http://:/api/transaction/begin ``` > **NOTE** > > For this example, `streamload_txn_example1_table1` is specified as the label of the transaction. ###### Return result[​](#return-result "Direct link to Return result") * If the transaction is successfully started, the following result is returned: ```bash { "Status": "OK", "Message": "", "Label": "streamload_txn_example1_table1", "TxnId": 9032, "BeginTxnTimeMs": 0 } ``` * If the transaction is bound to a duplicate label, the following result is returned: ```bash { "Status": "LABEL_ALREADY_EXISTS", "ExistingJobStatus": "RUNNING", "Message": "Label [streamload_txn_example1_table1] has already been used." } ``` * If errors other than duplicate label occur, the following result is returned: ```bash { "Status": "FAILED", "Message": "" } ``` ##### Write data[​](#write-data "Direct link to Write data") ###### Syntax[​](#syntax-1 "Direct link to Syntax") ```bash curl --location-trusted -u : -H "label:" \ -H "Expect:100-continue" \ [-H "transaction_type:multi"]\ # Optional. Loads data via a multi-table transaction. -H "db:" -H "table:" \ -T \ -XPUT http://:/api/transaction/load ``` > **NOTE** > > * When calling the `/api/transaction/load` operation, you must use `` to specify the save path of the data file you want to load. > * You can call `/api/transaction/load` operations with different `table` parameter values to load data into different tables within the same database. In this case, you must specify `-H "transaction_type:multi"` in the command. ###### Example[​](#example-1 "Direct link to Example") ```bash curl --location-trusted -u :<123456> -H "label:streamload_txn_example1_table1" \ -H "Expect:100-continue" \ -H "db:test_db" -H "table:table1" \ -T /home/disk1/example1.csv \ -H "column_separator: ," \ -XPUT http://:/api/transaction/load ``` > **NOTE** > > For this example, the column separator used in the data file `example1.csv` is commas (`,`) instead of StarRocks‘s default column separator (`\t`). Therefore, when calling the `/api/transaction/load` operation, you must use `"column_separator: "` to specify commas (`,`) as the column separator. ###### Return result[​](#return-result-1 "Direct link to Return result") * If the data write is successful, the following result is returned: ```bash { "TxnId": 1, "Seq": 0, "Label": "streamload_txn_example1_table1", "Status": "OK", "Message": "", "NumberTotalRows": 5265644, "NumberLoadedRows": 5265644, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 10737418067, "LoadTimeMs": 418778, "StreamLoadPutTimeMs": 68, "ReceivedDataTimeMs": 38964, } ``` * If the transaction is considered unknown, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "TXN_NOT_EXISTS" } ``` * If the transaction is considered in an invalid state, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "Transcation State Invalid" } ``` * If errors other than unknown transaction and invalid status occur, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "" } ``` ##### Pre-commit a transaction[​](#pre-commit-a-transaction "Direct link to Pre-commit a transaction") ###### Syntax[​](#syntax-2 "Direct link to Syntax") ```bash curl --location-trusted -u : -H "label:" \ -H "Expect:100-continue" \ [-H "transaction_type:multi"]\ # Optional. Pre-commits a multi-table transaction. -H "db:" \ [-H "prepared_timeout:"] \ -XPOST http://:/api/transaction/prepare ``` > **NOTE** > > Specify `-H "transaction_type:multi"` in the command if the transaction you want to pre-commit is a multi-table transaction. ###### Example[​](#example-2 "Direct link to Example") ```bash curl --location-trusted -u :<123456> -H "label:streamload_txn_example1_table1" \ -H "Expect:100-continue" \ -H "db:test_db" \ -H "prepared_timeout:300" \ -XPOST http://:/api/transaction/prepare ``` > **NOTE** > > The `prepared_timeout` field is optional. If it is not specified, the default value is determined by the FE configuration [`prepared_transaction_default_timeout_second`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#prepared_transaction_default_timeout_second) (Default: 86400 seconds). `prepared_timeout` is supported from v3.5.4 onwards. ###### Return result[​](#return-result-2 "Direct link to Return result") * If the pre-commit is successful, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "OK", "Message": "", "NumberTotalRows": 5265644, "NumberLoadedRows": 5265644, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 10737418067, "LoadTimeMs": 418778, "StreamLoadPutTimeMs": 68, "ReceivedDataTimeMs": 38964, "WriteDataTimeMs": 417851 "CommitAndPublishTimeMs": 1393 } ``` * If the transaction is considered not existent, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "Transcation Not Exist" } ``` * If the pre-commit times out, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "commit timeout", } ``` * If errors other than non-existent transaction and pre-commit timeout occur, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "publish timeout" } ``` ##### Commit a transaction[​](#commit-a-transaction "Direct link to Commit a transaction") ###### Syntax[​](#syntax-3 "Direct link to Syntax") ```bash curl --location-trusted -u : -H "label:" \ -H "Expect:100-continue" \ [-H "transaction_type:multi"]\ # Optional. Commits a multi-table transaction. -H "db:" \ -XPOST http://:/api/transaction/commit ``` > **NOTE** > > Specify `-H "transaction_type:multi"` in the command if the transaction you want to commit is a multi-table transaction. ###### Example[​](#example-3 "Direct link to Example") ```bash curl --location-trusted -u :<123456> -H "label:streamload_txn_example1_table1" \ -H "Expect:100-continue" \ -H "db:test_db" \ -XPOST http://:/api/transaction/commit ``` ###### Return result[​](#return-result-3 "Direct link to Return result") * If the commit is successful, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "OK", "Message": "", "NumberTotalRows": 5265644, "NumberLoadedRows": 5265644, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 10737418067, "LoadTimeMs": 418778, "StreamLoadPutTimeMs": 68, "ReceivedDataTimeMs": 38964, "WriteDataTimeMs": 417851 "CommitAndPublishTimeMs": 1393 } ``` * If the transaction has already been committed, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "OK", "Message": "Transaction already commited", } ``` * If the transaction is considered not existent, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "Transcation Not Exist" } ``` * If the commit times out, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "commit timeout", } ``` * If the data publish times out, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "publish timeout", "CommitAndPublishTimeMs": 1393 } ``` * If errors other than non-existent transaction and timeout occur, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "" } ``` ##### Roll back a transaction[​](#roll-back-a-transaction "Direct link to Roll back a transaction") ###### Syntax[​](#syntax-4 "Direct link to Syntax") ```bash curl --location-trusted -u : -H "label:" \ -H "Expect:100-continue" \ [-H "transaction_type:multi"]\ # Optional. Rolls back a multi-table transaction. -H "db:" \ -XPOST http://:/api/transaction/rollback ``` > **NOTE** > > Specify `-H "transaction_type:multi"` in the command if the transaction you want to roll back is a multi-table transaction. ###### Example[​](#example-4 "Direct link to Example") ```bash curl --location-trusted -u :<123456> -H "label:streamload_txn_example1_table1" \ -H "Expect:100-continue" \ -H "db:test_db" \ -XPOST http://:/api/transaction/rollback ``` ###### Return result[​](#return-result-4 "Direct link to Return result") * If the rollback is successful, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "OK", "Message": "" } ``` * If the transaction is considered not existent, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "Transcation Not Exist" } ``` * If errors other than not existent transaction occur, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "" } ``` #### References[​](#references "Direct link to References") For information about the suitable application scenarios and supported data file formats of Stream Load and about how Stream Load works, see [Loading from a local file system via Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md#loading-from-a-local-file-system-via-stream-load). For information about the syntax and parameters for creating Stream Load jobs, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). --- ### Load data from a local file system StarRocks provides two methods of loading data from a local file system: * Synchronous loading using [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) * Asynchronous loading using [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) Each of these options has its own advantages: * Stream Load supports CSV and JSON file formats. This method is recommended if you want to load data from a small number of files whose individual sizes do not exceed 10 GB. * Broker Load supports Parquet, ORC, CSV, and JSON file formats (JSON file format is supported from v3.2.3 onwards). This method is recommended if you want to load data from a large number of files whose individual sizes exceed 10 GB, or if the files are stored in a network attached storage (NAS) device. **Using Broker Load to load data from a local file system is supported from v2.5 onwards.** For CSV data, take note of the following points: * You can use a UTF-8 string, such as a comma (,), tab, or pipe (|), whose length does not exceed 50 bytes as a text delimiter. * Null values are denoted by using `\N`. For example, a data file consists of three columns, and a record from that data file holds data in the first and third columns but no data in the second column. In this situation, you need to use `\N` in the second column to denote a null value. This means the record must be compiled as `a,\N,b` instead of `a,,b`. `a,,b` denotes that the second column of the record holds an empty string. Stream Load and Broker Load both support data transformation at data loading and supports data changes made by UPSERT and DELETE operations during data loading. For more information, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md) and [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md). #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ###### Check network configuration[​](#check-network-configuration "Direct link to Check network configuration") Make sure that the machine on which the data you want to load resides can access the FE and BE nodes of the StarRocks cluster via the [`http_port`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#http_port) (default: `8030`) and [`be_http_port`](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#be_http_port) (default: `8040`) , respectively. #### Loading from a local file system via Stream Load[​](#loading-from-a-local-file-system-via-stream-load "Direct link to Loading from a local file system via Stream Load") Stream Load is an HTTP PUT-based synchronous loading method. After you submit a load job, StarRocks synchronously runs the job, and returns the result of the job after the job finishes. You can determine whether the job is successful based on the job result. > **NOTICE** > > After you load data into a StarRocks table by using Stream Load, the data of the materialized views that are created on that table is also updated. ##### How it works[​](#how-it-works "Direct link to How it works") You can submit a load request on your client to an FE according to HTTP, and the FE then uses an HTTP redirect to forward the load request to a specific BE or CN. You can also directly submit a load request on your client to a BE or CN of your choice. note If you submit load requests to an FE, the FE uses a polling mechanism to decide which BE or CN will serve as a coordinator to receive and process the load requests. The polling mechanism helps achieve load balancing within your StarRocks cluster. Therefore, we recommend that you send load requests to an FE. The BE or CN that receives the load request runs as the Coordinator BE or CN to split data based on the used schema into portions and assign each portion of the data to the other involved BEs or CNs. After the load finishes, the Coordinator BE or CN returns the result of the load job to your client. Note that if you stop the Coordinator BE or CN during the load, the load job fails. The following figure shows the workflow of a Stream Load job. ![Workflow of Stream Load](/assets/images/4.2-1-5e9185a6b2919c81f4697cceffd7734b.png) ##### Limits[​](#limits "Direct link to Limits") Stream Load does not support loading the data of a CSV file that contains a JSON-formatted column. ##### Typical example[​](#typical-example "Direct link to Typical example") This section uses curl as an example to describe how to load the data of a CSV or JSON file from your local file system into StarRocks. For detailed syntax and parameter descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). Note that in StarRocks some literals are used as reserved keywords by the SQL language. Do not directly use these keywords in SQL statements. If you want to use such a keyword in an SQL statement, enclose it in a pair of backticks (\`). See [Keywords](https://docs.starrocks.io/docs/sql-reference/sql-statements/keywords.md). ###### Load CSV data[​](#load-csv-data "Direct link to Load CSV data") ###### Prepare datasets[​](#prepare-datasets "Direct link to Prepare datasets") In your local file system, create a CSV file named `example1.csv`. The file consists of three columns, which represent the user ID, user name, and user score in sequence. ```plain 1,Lily,23 2,Rose,23 3,Alice,24 4,Julia,25 ``` ###### Create a database and a table[​](#create-a-database-and-a-table "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a Primary Key table named `table1`. The table consists of three columns: `id`, `name`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `table1` ( `id` int(11) NOT NULL COMMENT "user ID", `name` varchar(65533) NULL COMMENT "user name", `score` int(11) NOT NULL COMMENT "user score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` note Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). ###### Start a Stream Load[​](#start-a-stream-load "Direct link to Start a Stream Load") Run the following command to load the data of `example1.csv` into `table1`: ```bash curl --location-trusted -u : -H "label:123" \ -H "Expect:100-continue" \ -H "column_separator:," \ -H "columns: id, name, score" \ -T example1.csv -XPUT \ http://:/api/mydatabase/table1/_stream_load ``` note * If you use an account for which no password is set, you need to input only `:`. * You can use [SHOW FRONTENDS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_FRONTENDS.md) to view the IP address and HTTP port of the FE node. `example1.csv` consists of three columns, which are separated by commas (,) and can be mapped in sequence onto the `id`, `name`, and `score` columns of `table1`. Therefore, you need to use the `column_separator` parameter to specify the comma (,) as the column separator. You also need to use the `columns` parameter to temporarily name the three columns of `example1.csv` as `id`, `name`, and `score`, which are mapped in sequence onto the three columns of `table1`. After the load is complete, you can query `table1` to verify that the load is successful: ```sql SELECT * FROM table1; +------+-------+-------+ | id | name | score | +------+-------+-------+ | 1 | Lily | 23 | | 2 | Rose | 23 | | 3 | Alice | 24 | | 4 | Julia | 25 | +------+-------+-------+ 4 rows in set (0.00 sec) ``` ###### Load JSON data[​](#load-json-data "Direct link to Load JSON data") Since v3.2.7, Stream Load supports compressing JSON data during transmission, reducing network bandwidth overhead. Users can specify different compression algorithms using parameters `compression` and `Content-Encoding`. Supported compression algorithms including GZIP, BZIP2, LZ4\_FRAME, and ZSTD. For the syntax, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ###### Prepare datasets[​](#prepare-datasets-1 "Direct link to Prepare datasets") In your local file system, create a JSON file named `example2.json`. The file consists of two columns, which represent city ID and city name in sequence. ```json {"name": "Beijing", "code": 2} ``` ###### Create a database and a table[​](#create-a-database-and-a-table-1 "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a Primary Key table named `table2`. The table consists of two columns: `id` and `city`, of which `id` is the primary key. ```sql CREATE TABLE `table2` ( `id` int(11) NOT NULL COMMENT "city ID", `city` varchar(65533) NULL COMMENT "city name" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` note Since v2.5.7, StarRocks can set the number of(BUCKETS) automatically when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). ###### Start a Stream Load[​](#start-a-stream-load-1 "Direct link to Start a Stream Load") Run the following command to load the data of `example2.json` into `table2`: ```bash curl -v --location-trusted -u : -H "strict_mode: true" \ -H "Expect:100-continue" \ -H "format: json" -H "jsonpaths: [\"$.name\", \"$.code\"]" \ -H "columns: city,tmp_id, id = tmp_id * 100" \ -T example2.json -XPUT \ http://:/api/mydatabase/table2/_stream_load ``` note * If you use an account for which no password is set, you need to input only `:`. * You can use [SHOW FRONTENDS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_FRONTENDS.md) to view the IP address and HTTP port of the FE node. `example2.json` consists of two keys, `name` and `code`, which are mapped onto the `id` and `city` columns of `table2`, as shown in the following figure. ![JSON - Column Mapping](/assets/images/4.2-2-c642e32999d9076ae9633ff470fc8764.png) The mappings shown in the preceding figure are described as follows: * StarRocks extracts the `name` and `code` keys of `example2.json` and maps them onto the `name` and `code` fields declared in the `jsonpaths` parameter. * StarRocks extracts the `name` and `code` fields declared in the `jsonpaths` parameter and **maps them in sequence** onto the `city` and `tmp_id` fields declared in the `columns` parameter. * StarRocks extracts the `city` and `tmp_id` fields declared in the `columns` parameter and **maps them by name** onto the `city` and `id` columns of `table2`. note In the preceding example, the value of `code` in `example2.json` is multiplied by 100 before it is loaded into the `id` column of `table2`. For detailed mappings between `jsonpaths`, `columns`, and the columns of the StarRocks table, see the "Column mappings" section in [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). After the load is complete, you can query `table2` to verify that the load is successful: ```sql SELECT * FROM table2; +------+--------+ | id | city | +------+--------+ | 200 | Beijing| +------+--------+ 4 rows in set (0.01 sec) ``` ###### Merge Stream Load requests[​](#merge-stream-load-requests "Direct link to Merge Stream Load requests") Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) From v3.4.0, the system supports merging multiple Stream Load requests. warning Note that the Merge Commit optimization is suitable for the scenario with **concurrent** Stream Load jobs on a single table. It is not recommended if the concurrency is one. Meanwhile, think twice before setting `merge_commit_async` to `false` and `merge_commit_interval_ms` to a large value because they may cause load performance degradation. Merge Commit is an optimization for Stream Load, designed for high concurrency, small-batch (from KB to tens of MB) real-time loading scenarios. In earlier versions, each Stream Load request would generate a transaction and a data version, which led to the following issues in high concurrency loading scenarios: * Excessive data versions impact query performance, and limiting the number of versions may cause `too many versions` errors. * Data version merging through Compaction increases resource consumption. * It generates small files, increasing IOPS and I/O latency. And in shared-data clusters, this also raises cloud object storage costs. * Leader FE node, as the transaction manager, may become a single point of bottleneck. Merge Commit mitigates these issues by merging multiple concurrent Stream Load requests within a time window into a single transaction. This reduces the number of transactions and versions generated by high concurrency requests, thereby improving loading performance. Merge Commit supports both synchronous and asynchronous modes. Each mode has advantages and disadvantages. You can choose based on your use cases. * **Synchronous mode** The server returns only after the merged transaction is committed, ensuring the loading is successful and visible. * **Asynchronous mode** The server returns immediately after receiving the data. This mode does not ensure the loading is successful. | **Mode** | **Advantages** | **Disadvantages** | | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Synchronous | - Ensures data persistence and visibility upon request return.
- Guarantees that multiple sequential loading requests from the same client are executed in order. | Each loading request from the client is blocked until the server closes the merge window. It may reduce the data processing capability of a single client if the window is excessively large. | | Asynchronous | Allows a single client to send subsequent loading requests without waiting for the server to close the merge window, improving loading throughput. | - Does not guarantee data persistence or visibility upon return. The client must later verify the transaction status.
- Does not guarantee that multiple sequential loading requests from the same client are executed in order. | ###### Start a Stream Load[​](#start-a-stream-load-2 "Direct link to Start a Stream Load") * Run the following command to start a Stream Load job with Merge Commit enabled in synchronous mode, and set the merging window to `5000` milliseconds and degree of parallelism to `2`: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "column_separator:," \ -H "columns: id, name, score" \ -H "enable_merge_commit:true" \ -H "merge_commit_interval_ms:5000" \ -H "merge_commit_parallel:2" \ -T example1.csv -XPUT \ http://:/api/mydatabase/table1/_stream_load ``` * Run the following command to start a Stream Load job with Merge Commit enabled in asynchronous mode, and set the merging window to `60000` milliseconds and degree of parallelism to `2`: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "column_separator:," \ -H "columns: id, name, score" \ -H "enable_merge_commit:true" \ -H "merge_commit_async:true" \ -H "merge_commit_interval_ms:60000" \ -H "merge_commit_parallel:2" \ -T example1.csv -XPUT \ http://:/api/mydatabase/table1/_stream_load ``` note * Merge Commit only supports merging **homogeneous** loading requests into a single database and table. "Homogeneous" indicates that the Stream Load parameters are identical, including: common parameters, JSON format parameters, CSV format parameters, `opt_properties`, and Merge Commit parameters. * For loading CSV-formatted data, you must ensure that each row ends with a line separator. `skip_header` is not supported. * The server automatically generates labels for transactions. They will be ignored if specified. * Merge Commit merges multiple loading requests into a single transaction. If one request contains data quality issues, all requests in the transaction will fail. ###### Check Stream Load progress[​](#check-stream-load-progress "Direct link to Check Stream Load progress") After a load job is complete, StarRocks returns the result of the job in JSON format. For more information, see the "Return value" section in [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). Stream Load does not allow you to query the result of a load job by using the SHOW LOAD statement. ###### Cancel a Stream Load job[​](#cancel-a-stream-load-job "Direct link to Cancel a Stream Load job") Stream Load does not allow you to cancel a load job. If a load job times out or encounters errors, StarRocks automatically cancels the job. ##### Parameter configurations[​](#parameter-configurations "Direct link to Parameter configurations") This section describes some system parameters that you need to configure if you choose the loading method Stream Load. These parameter configurations take effect on all Stream Load jobs. * `streaming_load_max_mb`: the maximum size of each data file you want to load. The default maximum size is 10 GB. For more information, see [Configure BE or CN dynamic parameters](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md). We recommend that you do not load more than 10 GB of data at a time. If the size of a data file exceeds 10 GB, we recommend that you split the data file into small files that each are less than 10 GB in size and then load these files one by one. If you cannot split a data file greater than 10 GB, you can increase the value of this parameter based on the file size. After you increase the value of this parameter, the new value can take effect only after you restart the BEs or CNs of your StarRocks cluster. Additionally, system performance may deteriorate, and the costs of retries in the event of load failures also increase. note When you load the data of a JSON file, take note of the following points: * The size of each JSON object in the file cannot exceed 4 GB. If any JSON object in the file exceeds 4 GB, StarRocks throws an error "This parser can't support a document that big." * By default, the JSON body in an HTTP request cannot exceed 100 MB. If the JSON body exceeds 100 MB, StarRocks throws an error "The size of this batch exceed the max size \[104857600] of json type data data \[8617627793]. Set ignore\_json\_size to skip check, although it may lead huge memory consuming." To prevent this error, you can add `"ignore_json_size:true"` in the HTTP request header to ignore the check on the JSON body size. * `stream_load_default_timeout_second`: the timeout period of each load job. The default timeout period is 600 seconds. For more information, see [Configure FE dynamic parameters](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#configure-fe-dynamic-parameters). If many of the load jobs that you create time out, you can increase the value of this parameter based on the calculation result that you obtain from the following formula: **Timeout period of each load job > Amount of data to be loaded/Average loading speed** For example, if the size of the data file that you want to load is 10 GB and the average loading speed of your StarRocks cluster is 100 MB/s, set the timeout period to more than 100 seconds. note **Average loading speed** in the preceding formula is the average loading speed of your StarRocks cluster. It varies depending on the disk I/O and the number of BEs or CNs in your StarRocks cluster. Stream Load also provides the `timeout` parameter, which allows you to specify the timeout period of an individual load job. For more information, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ##### Usage notes[​](#usage-notes "Direct link to Usage notes") If a field is missing for a record in the data file you want to load and the column onto which the field is mapped in your StarRocks table is defined as `NOT NULL`, StarRocks automatically fills a `NULL` value in the mapping column of your StarRocks table during the load of the record. You can also use the `ifnull()` function to specify the default value that you want to fill. For example, if the field that represents city ID in the preceding `example2.json` file is missing and you want to fill an `x` value in the mapping column of `table2`, you can specify `"columns: city, tmp_id, id = ifnull(tmp_id, 'x')"`. #### Loading from a local file system via Broker Load[​](#loading-from-a-local-file-system-via-broker-load "Direct link to Loading from a local file system via Broker Load") In addition to Stream Load, you can also use Broker Load to load data from a local file system. This feature is supported from v2.5 onwards. Broker Load is an asynchronous loading method. After you submit a load job, StarRocks asynchronously runs the job and does not immediately return the job result. You need to query the job result by hand. See [Check Broker Load progress](#check-broker-load-progress). ##### Limits[​](#limits-1 "Direct link to Limits") * Currently Broker Load supports loading from a local file system only through a single broker whose version is v2.5 or later. * Highly concurrent queries against a single broker may cause issues such as timeout and OOM. To mitigate the impact, you can use the `pipeline_dop` variable (see [System variable](https://docs.starrocks.io/docs/sql-reference/System_variable.md#pipeline_dop)) to set the query parallelism for Broker Load. For queries against a single broker, we recommend that you set `pipeline_dop` to a value smaller than `16`. ##### Typical example[​](#typical-example-1 "Direct link to Typical example") Broker Load supports loading from a single data file to a single table, loading from multiple data files to a single table, and loading from multiple data files to multiple tables. This section uses loading from multiple data files to a single table as an example. Note that in StarRocks some literals are used as reserved keywords by the SQL language. Do not directly use these keywords in SQL statements. If you want to use such a keyword in an SQL statement, enclose it in a pair of backticks (\`). See [Keywords](https://docs.starrocks.io/docs/sql-reference/sql-statements/keywords.md). ###### Prepare datasets[​](#prepare-datasets-2 "Direct link to Prepare datasets") Use the CSV file format as an example. Log in to your local file system, and create two CSV files, `file1.csv` and `file2.csv`, in a specific storage location (for example, `/home/disk1/business/`). Both files consist of three columns, which represent the user ID, user name, and user score in sequence. * `file1.csv` ```plain 1,Lily,21 2,Rose,22 3,Alice,23 4,Julia,24 ``` * `file2.csv` ```plain 5,Tony,25 6,Adam,26 7,Allen,27 8,Jacky,28 ``` ###### Create a database and a table[​](#create-a-database-and-a-table-2 "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a Primary Key table named `mytable`. The table consists of three columns: `id`, `name`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `mytable` ( `id` int(11) NOT NULL COMMENT "User ID", `name` varchar(65533) NULL DEFAULT "" COMMENT "User name", `score` int(11) NOT NULL DEFAULT "0" COMMENT "User score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`) PROPERTIES("replication_num"="1"); ``` ###### Start a Broker Load[​](#start-a-broker-load "Direct link to Start a Broker Load") Run the following command to start a Broker Load job that loads data from all data files (`file1.csv` and `file2.csv`) stored in the `/home/disk1/business/` path of your local file system to the StarRocks table `mytable`: ```sql LOAD LABEL mydatabase.label_local ( DATA INFILE("file:///home/disk1/business/csv/*") INTO TABLE mytable COLUMNS TERMINATED BY "," (id, name, score) ) WITH BROKER "sole_broker" PROPERTIES ( "timeout" = "3600" ); ``` This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Check Broker Load progress[​](#check-broker-load-progress "Direct link to Check Broker Load progress") In v3.0 and earlier, use the [SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md) statement or the curl command to view the progress of Broker Load jobs. In v3.1 and later, you can view the progress of Broker Load jobs from the [`information_schema.loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view: ```sql SELECT * FROM information_schema.loads; ``` If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'label_local'; ``` After you confirm that the load job has finished, you can query table to see if the data has been successfully loaded. Example: ```sql SELECT * FROM mytable; +------+-------+-------+ | id | name | score | +------+-------+-------+ | 3 | Alice | 23 | | 5 | Tony | 25 | | 6 | Adam | 26 | | 1 | Lily | 21 | | 2 | Rose | 22 | | 4 | Julia | 24 | | 7 | Allen | 27 | | 8 | Jacky | 28 | +------+-------+-------+ 8 rows in set (0.07 sec) ``` ###### Cancel a Broker Load job[​](#cancel-a-broker-load-job "Direct link to Cancel a Broker Load job") When a load job is not in the **CANCELLED** or **FINISHED** stage, you can use the [CANCEL LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/CANCEL_LOAD.md) statement to cancel the job. For example, you can execute the following statement to cancel a load job, whose label is `label_local`, in the database `mydatabase`: ```sql CANCEL LOAD FROM mydatabase WHERE LABEL = "label_local"; ``` #### Loading from NAS via Broker Load[​](#loading-from-nas-via-broker-load "Direct link to Loading from NAS via Broker Load") There are two ways to load data from NAS by using Broker Load: * Consider NAS as a local file system, and run a load job with a broker. See the previous section "[Loading from a local system via Broker Load](#loading-from-a-local-file-system-via-broker-load)". * (Recommended) Consider NAS as a cloud storage system, and run a load job without a broker. This section introduces the second way. Detailed operations are as follows: 1. Mount your NAS device to the same path on all the BE or CN nodes and FE nodes of your StarRocks cluster. As such, all BEs or CNs can access the NAS device like they access their own locally stored files. 2. Use Broker Load to load data from the NAS device to the destination StarRocks table. Example: ```sql LOAD LABEL test_db.label_nas ( DATA INFILE("file:///home/disk1/sr/*") INTO TABLE mytable COLUMNS TERMINATED BY "," ) WITH BROKER PROPERTIES ( "timeout" = "3600" ); ``` This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. Note that `DATA INFILE` in the declaration is used to specify the mount point folder path of the NAS device, as shown in the above example in which `file:///` is the prefix and `/home/disk1/sr` is the mount point folder path. * `BROKER`: You do not need to specify the broker name. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). After you submit a job, you can view the load progress or cancel the job as needed. For detailed operations, see "[Check Broker Load progress](#check-broker-load-progress)" and "[Cancel a Broker Load job](#cancel-a-broker-load-job) in this topic. --- ### Strict mode Strict mode is an optional property that you can configure for data loads. It affects the loading behavior and the final loaded data. This topic introduces what strict mode is and how to set strict mode. #### Understand strict mode[​](#understand-strict-mode "Direct link to Understand strict mode") During data loading, the data types of the source columns may not be completely consistent with the data types of the destination columns. In such cases, StarRocks performs conversions on the source column values that have inconsistent data types. Data conversions may fail due to various issues such as unmatched field data types and field length overflows. Source column values that fail to be properly converted are unqualified column values, and source rows that contain unqualified column values are referred to as "unqualified rows". Strict mode is used to control whether to filter out unqualified rows during data loading. Strict mode works as follows: * If strict mode is enabled, StarRocks loads only qualified rows. It filters out unqualified rows and returns details about the unqualified rows. * If strict mode is disabled, StarRocks converts unqualified column values into `NULL` and loads unqualified rows that contain these `NULL` values together with qualified rows. Note the following points: * In actual business scenarios, both qualified and unqualified rows may contain `NULL` values. If the destination columns do not allow `NULL` values, StarRocks reports errors and filters out the rows that contain `NULL` values. * The maximum percentage of unqualified rows that can be filtered out for a load job is controlled by an optional job property `max_filter_ratio`. note The `max_filter_ratio` property for INSERT is supported from v3.4.0. For example, you want to load four rows that hold `\N` (`\N` denotes a `NULL` value), `abc`, `2000`, and `1` values respectively in a column from a CSV-formatted data file into a StarRocks table, and the data type of the destination StarRocks table column is TINYINT \[-128, 127]. * The source column value `\N` is processed into `NULL` upon conversion to TINYINT. > **NOTE** > > `\N` is always processed into `NULL` upon conversion regardless of the destination data type. * The source column value `abc` is processed into `NULL`, because its data type is not TINYINT and the conversion fails. * The source column value `2000` is processed into `NULL`, because it is beyond the range supported by TINYINT and the conversion fails. * The source column value `1` can be properly converted to a TINYINT-type value `1`. If strict mode is disabled, StarRocks loads all the four rows. If strict mode is enabled, StarRocks loads only the rows that hold `\N` or `1` and filters out the rows that hold `abc` or `2000`. The rows filtered out are counted against the maximum percentage of rows that can be filtered out due to inadequate data quality as specified by the `max_filter_ratio` parameter. ##### Final loaded data with strict mode disabled[​](#final-loaded-data-with-strict-mode-disabled "Direct link to Final loaded data with strict mode disabled") | Source column value | Column value upon conversion to TINYINT | Load result when destination column allows NULL values | Load result when destination column does not allow NULL values | | ------------------- | --------------------------------------- | ------------------------------------------------------ | -------------------------------------------------------------- | | \N | NULL | The value `NULL` is loaded. | An error is reported. | | abc | NULL | The value `NULL` is loaded. | An error is reported. | | 2000 | NULL | The value `NULL` is loaded. | An error is reported. | | 1 | 1 | The value `1` is loaded. | The value `1` is loaded. | ##### Final loaded data with strict mode enabled[​](#final-loaded-data-with-strict-mode-enabled "Direct link to Final loaded data with strict mode enabled") | Source column value | Column value upon conversion to TINYINT | Load result when destination column allows NULL values | Load result when destination column does not allow NULL values | | ------------------- | --------------------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------- | | \N | NULL | The value `NULL` is loaded. | An error is reported. | | abc | NULL | The value `NULL` is not allowed and therefore is filtered out. | An error is reported. | | 2000 | NULL | The value `NULL` is not allowed and therefore is filtered out. | An error is reported. | | 1 | 1 | The value `1` is loaded. | The value `1` is loaded. | #### Set strict mode[​](#set-strict-mode "Direct link to Set strict mode") You can use the `strict_mode` parameter to set strict mode for the load job. Valid values are `true` and `false`. The default value is `false`. The value `true` enables strict mode, and the value `false` disables strict mode. Note the `strict_mode` parameter is supported for INSERT from v3.4.0, with the default value `true`. Now, except Stream Load, for all other loading methods, `strict_mode` is set the same way in PROPERTIES clause. You can also use the `enable_insert_strict` session variable to set strict mode. Valid values are `true` and `false`. The default value is `true`. The value `true` enables strict mode, and the value `false` disables strict mode. note From v3.4.0 onwards, when `enable_insert_strict` is set to `true`, the system loads only qualified rows. It filters out unqualified rows and returns details about the unqualified rows. Instead, in versions earlier than v3.4.0, when `enable_insert_strict` is set to `true`, the INSERT jobs fails when there is an unqualified rows. Examples are as follows: ##### Stream Load[​](#stream-load "Direct link to Stream Load") ```bash curl --location-trusted -u : \ -H "strict_mode: {true | false}" \ -T -XPUT \ http://:/api///_stream_load ``` For detailed syntax and parameters about Stream Load, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ##### Broker Load[​](#broker-load "Direct link to Broker Load") ```sql LOAD LABEL [.] ( DATA INFILE (""[, "" ...]) INTO TABLE ) WITH BROKER ( "username" = "", "password" = "" ) PROPERTIES ( "strict_mode" = "{true | false}" ) ``` The preceding code snippet uses HDFS as an example. For detailed syntax and parameters about Broker Load, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ##### Routine Load[​](#routine-load "Direct link to Routine Load") ```sql CREATE ROUTINE LOAD [.] ON PROPERTIES ( "strict_mode" = "{true | false}" ) FROM KAFKA ( "kafka_broker_list" =":[,:...]", "kafka_topic" = "" ) ``` The preceding code snippet uses Apache Kafka® as an example. For detailed syntax and parameters about Routine Load, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### Spark Load[​](#spark-load "Direct link to Spark Load") ```sql LOAD LABEL [.] ( DATA INFILE (""[, "" ...]) INTO TABLE ) WITH RESOURCE ( "spark.executor.memory" = "3g", "broker.username" = "", "broker.password" = "" ) PROPERTIES ( "strict_mode" = "{true | false}" ) ``` The preceding code snippet uses HDFS as an example. For detailed syntax and parameters about Spark Load, see [SPARK LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SPARK_LOAD.md). ##### INSERT[​](#insert "Direct link to INSERT") ```sql INSERT INTO [.] PROPERTIES( "strict_mode" = "{true | false}" ) ``` For detailed syntax and parameters about INSERT, see [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). --- ## Maturity ### Versions and Maturity #### [📄️ Beta and experimental features](https://docs.starrocks.io/docs/maturity/features.md) [Explains feature maturity levels: Experimental, Beta, Generally Available (GA), Deprecated, and End-of-Life, and what each means for production use.](https://docs.starrocks.io/docs/maturity/features.md) #### [📄️ Version naming](https://docs.starrocks.io/docs/maturity/release_numbering.md) [StarRocks version naming conventions: MAJOR.MINOR.PATCH format, prerelease designations, and how to interpret version numbers.](https://docs.starrocks.io/docs/maturity/release_numbering.md) --- ### Beta and experimental features Features have five potential maturity levels: * Experimental * Beta * Generally Available (GA) * Deprecated * Removed Most features are GA, and if there is no label in the documentation to indicate that a feature is Experimental, Beta, or Deprecated—then the feature is GA. #### Experimental features[​](#experimental-features "Direct link to Experimental features") * **Stability**: Possibly buggy, with minor known issues. * **Maturity**: Low * **Interface**: The interface may be changed in the future. This includes command syntax, configuration parameters, defaults, feature removal, etc. * **Availability**: Experimental features are off by default, and need to be allowed by setting a parameter with SQL or in a configuration file. * **Production readiness**: Experimental features should not be used in production. * **Support**: Please open a [GitHub issue](https://github.com/StarRocks/starrocks/issues) or ask questions in [Slack](https://docs.starrocks.io/join/) and the Engineering team will try to help you. #### Beta features[​](#beta-features "Direct link to Beta features") * **Stability**: Well tested. May be not good for corner cases. * **Maturity**: Core functionality is complete, may not be performance-optimized. * **Interface**: The interface may be changed in the future. May be not backward compatible. * **Availability**: Beta features are off by default, and need to be allowed by setting a parameter with SQL or in a configuration file. * **Production readiness**: Beta features are not recommended for production use. * **Support**: Please open a [GitHub issue](https://github.com/StarRocks/starrocks/issues) or ask questions in [Slack](https://docs.starrocks.io/join/) and the Engineering team will try to help you. #### GA features[​](#ga-features "Direct link to GA features") * **Stability**: Comprehensively tested. * **Maturity**: High. * **Interface**: Stable API. * **Availability**: GA features are on by default. * **Production readiness**: Production ready. * **Support**: The support team provides support to customers. Open-source community members should open a [GitHub issue](https://github.com/StarRocks/starrocks/issues) or ask questions in [Slack](https://docs.starrocks.io/join/) and the Engineering team will try to help you. #### Deprecated features[​](#deprecated-features "Direct link to Deprecated features") Some features are deprecated—marked for removal, because they are replaced with other features or the features were not being used. Generally when a feature is deprecated we will suggest an alternative in the documentation. --- ### Version naming Purpose: To provide a unified and clear explanation of the current naming conventions for StarRocks software packages. #### Numbering[​](#numbering "Direct link to Numbering") The version numbering format for released versions is in the form of `MAJOR.MINOR.PATCH[-PRERELEASE]`, where PRERELEASE indicates the type and number of the prerelease version. Detailed explanations are as follows: ##### MAJOR[​](#major "Direct link to MAJOR") The major version is incremented when incompatible API changes are made or there are major adjustments to the product's overall strategy. ##### MINOR[​](#minor "Direct link to MINOR") The minor version is incremented when functionality is added in a backward-compatible manner. This typically refers to the addition of new features or improvements that do not break the API of the current major version or the main behavioral patterns of the software. ##### PATCH[​](#patch "Direct link to PATCH") The patch version is incremented when backward-compatible bug fixes are made. This is usually for bug fixes or minor improvements that do not affect the software's main functions or API. ##### PRERELEASE[​](#prerelease "Direct link to PRERELEASE") Adding a prerelease identifier to the version indicates that the build is for testing by early adopters. The prerelease identifier will indicate the version number of the prerelease, such as `rc01`, `rc02`, etc. Currently, the only prerelease tag in use is `rc` followed by a two-digit number such as `rc01`. note Release Candidates are only built for the initial minor and major versions. For example, `3.4.0-rc01` or `4.0.0-rc01`. #### Version examples[​](#version-examples "Direct link to Version examples") * `3.4.0`: The official release version. * `3.4.0-rc01`: The first prerelease version of `3.4.0`. For instance, version `3.3` also had a second prerelease version `3.3.0-rc02`. * `3.4.1`, `3.4.2`: Bug fix versions. #### Software package naming rules[​](#software-package-naming-rules "Direct link to Software package naming rules") The naming convention for software packages is: `StarRocks-x.y.z[-rcxx]{-os}{-arch}.tar.gz`. (That is, the main form is `Product-version-os-arch`) 1. x.y.z: follows the three-digit version numbering rule. `-rcxx` is an optional prerelease version number, such as `-rc01`, `-rc02`. 2. OS includes: `ubuntu`, `centos`. Mandatory. 3. ARCH includes: `amd64`. Mandatory. (There is no community edition for `arm64` yet.) ###### Example software package names[​](#example-software-package-names "Direct link to Example software package names") 1. `StarRocks-3.4.0-rc01-ubuntu-amd64.tar.gz`: The prerelease version of 3.4.0, built for Ubuntu Linux. 2. `StarRocks-3.4.1-centos-amd64.tar.gz`: The PATCH revision built for Red Hat Enterprise Linux / CentOS. --- ## Quick_start ### Quick Start These Quick Start guides will help you get going with a small StarRocks environment. The clusters that you will launch will be suitable for learning how StarRocks works, but are not meant for analyzing large datasets or performance testing. For scalable deployments please see [deploying StarRocks](https://docs.starrocks.io/docs/deployment.md) #### [📄️ Claude + StarRocks MCP](https://docs.starrocks.io/docs/quick_start/MCP.md) [Analyze complex data with Claude and the StarRocks MCP server on a shared-data cluster.](https://docs.starrocks.io/docs/quick_start/MCP.md) #### [📄️ Deploy StarRocks with Docker](https://docs.starrocks.io/docs/quick_start/shared-nothing.md) [StarRocks in Docker: Query real data with JOINs](https://docs.starrocks.io/docs/quick_start/shared-nothing.md) #### [📄️ Separate storage and compute](https://docs.starrocks.io/docs/quick_start/shared-data.md) [Quick start guide for deploying StarRocks in shared-data mode with separate compute and storage.](https://docs.starrocks.io/docs/quick_start/shared-data.md) #### [📄️ Apache Iceberg Lakehouse](https://docs.starrocks.io/docs/quick_start/iceberg.md) [Quick start guide for querying Apache Iceberg data lakes with StarRocks.](https://docs.starrocks.io/docs/quick_start/iceberg.md) #### [📄️ Apache Hudi Lakehouse](https://docs.starrocks.io/docs/quick_start/hudi.md) [Quick start guide for querying Apache Hudi data lakes with StarRocks.](https://docs.starrocks.io/docs/quick_start/hudi.md) #### [📄️ Kafka routine load StarRocks using shared-data storage](https://docs.starrocks.io/docs/quick_start/routine-load.md) [Quick start guide for streaming data from Kafka into StarRocks using Routine Load with shared-data storage.](https://docs.starrocks.io/docs/quick_start/routine-load.md) #### [📄️ StarRocks with Helm](https://docs.starrocks.io/docs/quick_start/helm.md) [Quick start guide for deploying StarRocks on Kubernetes using Helm.](https://docs.starrocks.io/docs/quick_start/helm.md) --- ### StarRocks with Helm #### Goals[​](#goals "Direct link to Goals") The goals of this quickstart are: * Deploy the StarRocks Kubernetes Operator and a StarRocks cluster with Helm * Configure a password for the StarRocks database user `root` * Provide for high-availability with three FEs and three BEs * Store metadata in persistent storage * Store data in persistent storage * Allow MySQL clients to connect from outside the Kubernetes cluster * Allow loading data from outside the Kubernetes cluster using Stream Load * Load some public datasets * Query the data tip The datasets and queries are the same as the ones used in the Basic Quick Start. The main difference here is deploying with Helm and the StarRocks Operator. The data used is provided by NYC OpenData and the National Centers for Environmental Information. Both of these datasets are large, and because this tutorial is intended to help you get exposed to working with StarRocks we are not going to load data for the past 120 years. You can run this with a GKE Kubernetes cluster built on three e2-standard-4 machines (or similar) with 80GB disk. For larger deployments, we have other documentation and will provide that later. There is a lot of information in this document, and it is presented with step-by-step content at the beginning, and the technical details at the end. This is done to serve these purposes in this order: 1. Get the system deployed with Helm. 2. Allow the reader to load data in StarRocks and analyze that data. 3. Explain the basics of data transformation during loading. *** #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") ##### Kubernetes environment[​](#kubernetes-environment "Direct link to Kubernetes environment") The Kubernetes environment used while writing this guide consists of three nodes with four vCPUS, and 16GB RAM each (GCP `e2-standard-4` machines). The Kubernetes cluster was deployed with this `gcloud` command: tip This command is for your reference, if you are using AWS, Azure, or any other Kubernetes provider you will need to modify this for your environment. In Google Cloud you will need to specify your own project and an appropriate location. ```bash gcloud container --project enterprise-demo-422514 \ clusters create ee-docs \ --location=southamerica-west1-b \ --machine-type e2-standard-4 --disk-size 80 --num-nodes 3 ``` ##### Helm[​](#helm "Direct link to Helm") Helm is a package manager for Kubernetes that simplifies the deployment and management of applications. In this lab you will use Helm to deploy the CelerData Enterprise Edition Kubernetes operator and the sample StarRocks cluster. [Install helm](https://helm.sh/docs/intro/quickstart/) ##### SQL client[​](#sql-client "Direct link to SQL client") You can use the SQL client provided in the Kubernetes environment, or use one on your system. This guide uses the `mysql CLI` Many MySQL-compatible clients will work. ##### curl[​](#curl "Direct link to curl") `curl` is used to issue the data load job to StarRocks, and to download the datasets. Check to see if you have it installed by running `curl` or `curl.exe` at your OS prompt. If curl is not installed, [get curl here](https://curl.se/). *** #### Terminology[​](#terminology "Direct link to Terminology") ##### FE[​](#fe "Direct link to FE") Frontend nodes are responsible for metadata management, client connection management, query planning, and query scheduling. Each FE stores and maintains a complete copy of metadata in its memory, which guarantees indiscriminate services among the FEs. ##### BE[​](#be "Direct link to BE") Backend nodes are responsible for both data storage and executing query plans. *** #### Add the StarRocks Helm chart repo[​](#add-the-starrocks-helm-chart-repo "Direct link to Add the StarRocks Helm chart repo") The Helm Chart contains the definitions of the StarRocks Operator and the custom resource StarRocksCluster. 1. Add the Helm Chart Repo. ```bash helm repo add starrocks https://starrocks.github.io/starrocks-kubernetes-operator ``` 2. Update the Helm Chart Repo to the latest version. ```bash helm repo update ``` 3. View the Helm Chart Repo that you added. ```bash helm search repo starrocks ``` ```text NAME CHART VERSION APP VERSION DESCRIPTION starrocks/kube-starrocks 1.9.7 3.2-latest kube-starrocks includes two subcharts, operator... starrocks/operator 1.9.7 1.9.7 A Helm chart for StarRocks operator starrocks/starrocks 1.9.7 3.2-latest A Helm chart for StarRocks cluster starrocks/warehouse 1.9.7 3.2-latest Warehouse is currently a feature of the StarRoc... ``` *** #### Download the data[​](#download-the-data "Direct link to Download the data") Download these two datasets to your machine. ##### New York City crash data[​](#new-york-city-crash-data "Direct link to New York City crash data") ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/datasets/NYPD_Crash_Data.csv ``` ##### Weather data[​](#weather-data "Direct link to Weather data") ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/datasets/72505394728.csv ``` *** #### Create a Helm values file[​](#create-a-helm-values-file "Direct link to Create a Helm values file") The goals for this quick start are: 1. Configure a password for the StarRocks database user `root` 2. Provide for high-availability with three FEs and three BEs 3. Store metadata in persistent storage 4. Store data in persistent storage 5. Allow MySQL clients to connect from outside the Kubernetes cluster 6. Allow loading data from outside the Kubernetes cluster using Stream Load The Helm chart provides options to satisfy all of these goals, but they are not configured by default. The rest of this section covers the configuration needed to meet all of these goals. A complete values spec will be provided, but first read the details for each of the six sections and then copy the full spec. ##### 1. Password for the database user[​](#1-password-for-the-database-user "Direct link to 1. Password for the database user") This bit of YAML instructs the StarRocks operator to set the password for the database user `root` to the value of the `password` key of the Kubernetes secret \`starrocks-root-pass. ```yaml starrocks: initPassword: enabled: true # Set a password secret, for example: # kubectl create secret generic starrocks-root-pass --from-literal=password='g()()dpa$$word' passwordSecret: starrocks-root-pass ``` * Task: Create the Kubernetes secret ```bash kubectl create secret generic starrocks-root-pass --from-literal=password='g()()dpa$$word' ``` ##### 2. High Availability with 3 FEs and 3 BEs[​](#2-high-availability-with-3-fes-and-3-bes "Direct link to 2. High Availability with 3 FEs and 3 BEs") By setting `starrocks.starrockFESpec.replicas` to 3, and `starrocks.starrockBeSpec.replicas` to 3 you will have enough FEs and BEs for high availability. Setting the CPU and memory requests low allows the pods to be created in a small Kubernetes environment. ```yaml starrocks: starrocksFESpec: replicas: 3 resources: requests: cpu: 1 memory: 1Gi starrocksBeSpec: replicas: 3 resources: requests: cpu: 1 memory: 2Gi ``` ##### 3. Store metadata in persistent storage[​](#3-store-metadata-in-persistent-storage "Direct link to 3. Store metadata in persistent storage") Setting a value for `starrocks.starrocksFESpec.storageSpec.name` to anything other than `""` causes: * Persistent storage to be used * the value of `starrocks.starrocksFESpec.storageSpec.name` to be used as the prefix for all storage volumes for the service. By setting the value to `fe` these PVs will be created for FE 0: * `fe-meta-kube-starrocks-fe-0` * `fe-log-kube-starrocks-fe-0` ```yaml starrocks: starrocksFESpec: storageSpec: name: fe ``` ##### 4. Store data in persistent storage[​](#4-store-data-in-persistent-storage "Direct link to 4. Store data in persistent storage") Setting a value for `starrocks.starrocksBeSpec.storageSpec.name` to anything other than `""` causes: * Persistent storage to be used * the value of `starrocks.starrocksBeSpec.storageSpec.name` to be used as the prefix for all storage volumes for the service. By setting the value to `be` these PVs will be created for BE 0: * `be-data-kube-starrocks-be-0` * `be-log-kube-starrocks-be-0` Setting the `storageSize` to 15Gi reduces the storage from the default of 1Ti to fit smaller quotas for storage. ```yaml starrocks: starrocksBeSpec: storageSpec: name: be storageSize: 15Gi ``` ##### 5. LoadBalancer for MySQL clients[​](#5-loadbalancer-for-mysql-clients "Direct link to 5. LoadBalancer for MySQL clients") By default, access to the FE service is through cluster IPs. To allow external access, `service.type` is set to `LoadBalancer` ```yaml starrocks: starrocksFESpec: service: type: LoadBalancer ``` ##### 6. LoadBalancer for external data loading[​](#6-loadbalancer-for-external-data-loading "Direct link to 6. LoadBalancer for external data loading") Stream Load requires external access to both FEs and BEs. The requests are sent to the FE and then the FE assigns a BE to process the upload. To allow the `curl` command to be redirected to the BE the `starroclFeProxySpec` needs to be enabled and set to type `LoadBalancer`. ```yaml starrocks: starrocksFeProxySpec: enabled: true service: type: LoadBalancer ``` ##### The complete values file[​](#the-complete-values-file "Direct link to The complete values file") The above snippets combined provide a full values file. Save this to `my-values.yaml`: ```yaml starrocks: initPassword: enabled: true # Set a password secret, for example: # kubectl create secret generic starrocks-root-pass --from-literal=password='g()()dpa$$word' passwordSecret: starrocks-root-pass starrocksFESpec: replicas: 3 service: type: LoadBalancer resources: requests: cpu: 1 memory: 1Gi storageSpec: name: fe starrocksBeSpec: replicas: 3 resources: requests: cpu: 1 memory: 2Gi storageSpec: name: be storageSize: 15Gi starrocksFeProxySpec: enabled: true service: type: LoadBalancer ``` #### Set the StarRocks root database user password[​](#set-the-starrocks-root-database-user-password "Direct link to Set the StarRocks root database user password") To load data from outside of the Kubernetes cluster the StarRocks database will be exposed externally. You should set a password for the StarRocks database user `root`. The operator will apply the password to the FE and BE nodes. ```bash kubectl create secret generic starrocks-root-pass --from-literal=password='g()()dpa$$word' ``` ```text secret/starrocks-root-pass created ``` *** #### Deploy the operator and StarRocks cluster[​](#deploy-the-operator-and-starrocks-cluster "Direct link to Deploy the operator and StarRocks cluster") ```bash helm install -f my-values.yaml starrocks starrocks/kube-starrocks ``` ```text NAME: starrocks LAST DEPLOYED: Wed Jun 26 20:25:09 2024 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Thank you for installing kube-starrocks-1.9.7 kube-starrocks chart. It will install both operator and starrocks cluster, please wait for a few minutes for the cluster to be ready. Please see the values.yaml for more operation information: https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/helm-charts/charts/kube-starrocks/values.yaml ``` #### Check the status of the StarRocks cluster[​](#check-the-status-of-the-starrocks-cluster "Direct link to Check the status of the StarRocks cluster") You can check the progress with these commands: ```bash kubectl --namespace default get starrockscluster -l "cluster=kube-starrocks" ``` ```text NAME PHASE FESTATUS BESTATUS CNSTATUS FEPROXYSTATUS kube-starrocks reconciling reconciling reconciling reconciling ``` ```bash kubectl get pods ``` note The `kube-starrocks-initpwd` pod will go through `error` and `CrashLoopBackOff` states as it attempts to connect to the FE and BE pods to set the StarRocks root password. You should ignore these errors and wait for a status of `Completed` for this pod. ```text NAME READY STATUS RESTARTS AGE kube-starrocks-be-0 0/1 Running 0 20s kube-starrocks-be-1 0/1 Running 0 20s kube-starrocks-be-2 0/1 Running 0 20s kube-starrocks-fe-0 1/1 Running 0 66s kube-starrocks-fe-1 0/1 Running 0 65s kube-starrocks-fe-2 0/1 Running 0 66s kube-starrocks-fe-proxy-56f8998799-d4qmt 1/1 Running 0 20s kube-starrocks-initpwd-m84br 0/1 CrashLoopBackOff 3 (50s ago) 92s kube-starrocks-operator-54ffcf8c5c-xsjc8 1/1 Running 0 92s ``` ```bash kubectl get pvc ``` ```text NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE be-data-kube-starrocks-be-0 Bound pvc-4ae0c9d8-7f9a-4147-ad74-b22569165448 15Gi RWO standard-rwo 82s be-data-kube-starrocks-be-1 Bound pvc-28b4dbd1-0c8f-4b06-87e8-edec616cabbc 15Gi RWO standard-rwo 82s be-data-kube-starrocks-be-2 Bound pvc-c7232ea6-d3d9-42f1-bfc1-024205a17656 15Gi RWO standard-rwo 82s be-log-kube-starrocks-be-0 Bound pvc-6193c43d-c74f-4d12-afcc-c41ace3d5408 1Gi RWO standard-rwo 82s be-log-kube-starrocks-be-1 Bound pvc-c01f124a-014a-439a-99a6-6afe95215bf0 1Gi RWO standard-rwo 82s be-log-kube-starrocks-be-2 Bound pvc-136df15f-4d2e-43bc-a1c0-17227ce3fe6b 1Gi RWO standard-rwo 82s fe-log-kube-starrocks-fe-0 Bound pvc-7eac524e-d286-4760-b21c-d9b6261d976f 5Gi RWO standard-rwo 2m23s fe-log-kube-starrocks-fe-1 Bound pvc-38076b78-71e8-4659-b8e7-6751bec663f6 5Gi RWO standard-rwo 2m23s fe-log-kube-starrocks-fe-2 Bound pvc-4ccfee60-02b7-40ba-a22e-861ea29dac74 5Gi RWO standard-rwo 2m23s fe-meta-kube-starrocks-fe-0 Bound pvc-5130c9ff-b797-4f79-a1d2-4214af860d70 10Gi RWO standard-rwo 2m23s fe-meta-kube-starrocks-fe-1 Bound pvc-13545330-63be-42cf-b1ca-3ed6f96a8c98 10Gi RWO standard-rwo 2m23s fe-meta-kube-starrocks-fe-2 Bound pvc-609cadd4-c7b7-4cf9-84b0-a75678bb3c4d 10Gi RWO standard-rwo 2m23s ``` ##### Verify that the cluster is healthy[​](#verify-that-the-cluster-is-healthy "Direct link to Verify that the cluster is healthy") tip These are the same commands as above, but show the desired state. ```bash kubectl --namespace default get starrockscluster -l "cluster=kube-starrocks" ``` ```text NAME PHASE FESTATUS BESTATUS CNSTATUS FEPROXYSTATUS kube-starrocks running running running running ``` ```bash kubectl get pods ``` tip The system is ready when all of the pods except for `kube-starrocks-initpwd` show `1/1` in the `READY` column. The `kube-starrocks-initpwd` pod should show `0/1` and a `STATUS` of `Completed`. ```text NAME READY STATUS RESTARTS AGE kube-starrocks-be-0 1/1 Running 0 57s kube-starrocks-be-1 1/1 Running 0 57s kube-starrocks-be-2 1/1 Running 0 57s kube-starrocks-fe-0 1/1 Running 0 103s kube-starrocks-fe-1 1/1 Running 0 102s kube-starrocks-fe-2 1/1 Running 0 103s kube-starrocks-fe-proxy-56f8998799-d4qmt 1/1 Running 0 57s kube-starrocks-initpwd-m84br 0/1 Completed 4 2m9s kube-starrocks-operator-54ffcf8c5c-xsjc8 1/1 Running 0 2m9s ``` The `EXTERNAL-IP` addresses in the highlighted lines will be used to provide SQL client and Stream Load access from outside the Kubernetes cluster. ```bash kubectl get services ``` ```bash NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-starrocks-be-search ClusterIP None 9050/TCP 78s kube-starrocks-be-service ClusterIP 34.118.228.231 9060/TCP,8040/TCP,9050/TCP,8060/TCP 78s kube-starrocks-fe-proxy-service LoadBalancer 34.118.230.176 34.176.12.205 8080:30241/TCP 78s kube-starrocks-fe-search ClusterIP None 9030/TCP 2m4s kube-starrocks-fe-service LoadBalancer 34.118.226.82 34.176.215.97 8030:30620/TCP,9020:32461/TCP,9030:32749/TCP,9010:30911/TCP 2m4s kubernetes ClusterIP 34.118.224.1 443/TCP 8h ``` tip Store the `EXTERNAL-IP` addresses from the highlighted lines in environment variables so that you have them handy: ```text export MYSQL_IP=`kubectl get services kube-starrocks-fe-service --output jsonpath='{.status.loadBalancer.ingress[0].ip}'` ``` ```text export FE_PROXY=`kubectl get services kube-starrocks-fe-proxy-service --output jsonpath='{.status.loadBalancer.ingress[0].ip}'`:8080 ``` *** ##### Connect to StarRocks with a SQL client[​](#connect-to-starrocks-with-a-sql-client "Direct link to Connect to StarRocks with a SQL client") tip If you are using a client other than the mysql CLI, open that now. This command will run the `mysql` command in a Kubernetes pod: ```sql kubectl exec --stdin --tty kube-starrocks-fe-0 -- \ mysql -P9030 -h127.0.0.1 -u root --prompt="StarRocks > " ``` If you have the mysql CLI installed locally, you can use it instead of the one in the Kubernetes cluster: ```sql mysql -P9030 -h $MYSQL_IP -u root --prompt="StarRocks > " -p ``` *** #### Create some tables[​](#create-some-tables "Direct link to Create some tables") ```bash mysql -P9030 -h $MYSQL_IP -u root --prompt="StarRocks > " -p ``` ##### Create a database[​](#create-a-database "Direct link to Create a database") Type these two lines in at the `StarRocks > `prompt and press enter after each: ```sql CREATE DATABASE IF NOT EXISTS quickstart; USE quickstart; ``` ##### Create two tables[​](#create-two-tables "Direct link to Create two tables") ###### Crashdata[​](#crashdata "Direct link to Crashdata") The crash dataset contains many more fields than these, the schema has been trimmed down to include only the fields that might be useful to answer questions about the impact weather has on driving conditions. ```sql CREATE TABLE IF NOT EXISTS crashdata ( CRASH_DATE DATETIME, BOROUGH STRING, ZIP_CODE STRING, LATITUDE INT, LONGITUDE INT, LOCATION STRING, ON_STREET_NAME STRING, CROSS_STREET_NAME STRING, OFF_STREET_NAME STRING, CONTRIBUTING_FACTOR_VEHICLE_1 STRING, CONTRIBUTING_FACTOR_VEHICLE_2 STRING, COLLISION_ID INT, VEHICLE_TYPE_CODE_1 STRING, VEHICLE_TYPE_CODE_2 STRING ); ``` ###### Weatherdata[​](#weatherdata "Direct link to Weatherdata") Similar to the crash data, the weather dataset has many more columns (a total of 125 columns) and only the ones that are expected to answer the questions are included in the database. ```sql CREATE TABLE IF NOT EXISTS weatherdata ( DATE DATETIME, NAME STRING, HourlyDewPointTemperature STRING, HourlyDryBulbTemperature STRING, HourlyPrecipitation STRING, HourlyPresentWeatherType STRING, HourlyPressureChange STRING, HourlyPressureTendency STRING, HourlyRelativeHumidity STRING, HourlySkyConditions STRING, HourlyVisibility STRING, HourlyWetBulbTemperature STRING, HourlyWindDirection STRING, HourlyWindGustSpeed STRING, HourlyWindSpeed STRING ); ``` Exit from the MySQL client, or open a new shell to run commands at the command line to upload data. ```sql exit ``` #### Upload data[​](#upload-data "Direct link to Upload data") There are many ways to load data into StarRocks. For this tutorial, the simplest way is to use curl and StarRocks Stream Load. Upload the two datasets that you downloaded earlier. tip Open a new shell as these curl commands are run at the operating system prompt, not in the `mysql` client. The commands refer to the datasets that you downloaded, so run them from the directory where you downloaded the files. Since this is a new shell, run the export commands again: ```bash export MYSQL_IP=`kubectl get services kube-starrocks-fe-service --output jsonpath='{.status.loadBalancer.ingress[0].ip}'` export FE_PROXY=`kubectl get services kube-starrocks-fe-proxy-service --output jsonpath='{.status.loadBalancer.ingress[0].ip}'`:8080 ``` You will be prompted for a password. Use the password that you added to the Kubernetes secret `starrocks-root-pass`. If you used the command provided, the password is `g()()dpa$$word`. The `curl` commands look complex, but they are explained in detail at the end of the tutorial. For now, we recommend running the commands and running some SQL to analyze the data, and then reading about the data loading details at the end. ```bash curl --location-trusted -u root \ -T ./NYPD_Crash_Data.csv \ -H "label:crashdata-0" \ -H "column_separator:," \ -H "skip_header:1" \ -H "enclose:\"" \ -H "max_filter_ratio:1" \ -H "columns:tmp_CRASH_DATE, tmp_CRASH_TIME, CRASH_DATE=str_to_date(concat_ws(' ', tmp_CRASH_DATE, tmp_CRASH_TIME), '%m/%d/%Y %H:%i'),BOROUGH,ZIP_CODE,LATITUDE,LONGITUDE,LOCATION,ON_STREET_NAME,CROSS_STREET_NAME,OFF_STREET_NAME,NUMBER_OF_PERSONS_INJURED,NUMBER_OF_PERSONS_KILLED,NUMBER_OF_PEDESTRIANS_INJURED,NUMBER_OF_PEDESTRIANS_KILLED,NUMBER_OF_CYCLIST_INJURED,NUMBER_OF_CYCLIST_KILLED,NUMBER_OF_MOTORIST_INJURED,NUMBER_OF_MOTORIST_KILLED,CONTRIBUTING_FACTOR_VEHICLE_1,CONTRIBUTING_FACTOR_VEHICLE_2,CONTRIBUTING_FACTOR_VEHICLE_3,CONTRIBUTING_FACTOR_VEHICLE_4,CONTRIBUTING_FACTOR_VEHICLE_5,COLLISION_ID,VEHICLE_TYPE_CODE_1,VEHICLE_TYPE_CODE_2,VEHICLE_TYPE_CODE_3,VEHICLE_TYPE_CODE_4,VEHICLE_TYPE_CODE_5" \ -XPUT http://$FE_PROXY/api/quickstart/crashdata/_stream_load ``` ```text Enter host password for user 'root': { "TxnId": 2, "Label": "crashdata-0", "Status": "Success", "Message": "OK", "NumberTotalRows": 423726, "NumberLoadedRows": 423725, "NumberFilteredRows": 1, "NumberUnselectedRows": 0, "LoadBytes": 96227746, "LoadTimeMs": 2483, "BeginTxnTimeMs": 42, "StreamLoadPlanTimeMs": 122, "ReadDataTimeMs": 1610, "WriteDataTimeMs": 2253, "CommitAndPublishTimeMs": 65, "ErrorURL": "http://kube-starrocks-be-2.kube-starrocks-be-search.default.svc.cluster.local:8040/api/_load_error_log?file=error_log_5149e6f80de42bcb_eab2ea77276de4ba" } ``` ```bash curl --location-trusted -u root \ -T ./72505394728.csv \ -H "label:weather-0" \ -H "column_separator:," \ -H "skip_header:1" \ -H "enclose:\"" \ -H "max_filter_ratio:1" \ -H "columns: STATION, DATE, LATITUDE, LONGITUDE, ELEVATION, NAME, REPORT_TYPE, SOURCE, HourlyAltimeterSetting, HourlyDewPointTemperature, HourlyDryBulbTemperature, HourlyPrecipitation, HourlyPresentWeatherType, HourlyPressureChange, HourlyPressureTendency, HourlyRelativeHumidity, HourlySkyConditions, HourlySeaLevelPressure, HourlyStationPressure, HourlyVisibility, HourlyWetBulbTemperature, HourlyWindDirection, HourlyWindGustSpeed, HourlyWindSpeed, Sunrise, Sunset, DailyAverageDewPointTemperature, DailyAverageDryBulbTemperature, DailyAverageRelativeHumidity, DailyAverageSeaLevelPressure, DailyAverageStationPressure, DailyAverageWetBulbTemperature, DailyAverageWindSpeed, DailyCoolingDegreeDays, DailyDepartureFromNormalAverageTemperature, DailyHeatingDegreeDays, DailyMaximumDryBulbTemperature, DailyMinimumDryBulbTemperature, DailyPeakWindDirection, DailyPeakWindSpeed, DailyPrecipitation, DailySnowDepth, DailySnowfall, DailySustainedWindDirection, DailySustainedWindSpeed, DailyWeather, MonthlyAverageRH, MonthlyDaysWithGT001Precip, MonthlyDaysWithGT010Precip, MonthlyDaysWithGT32Temp, MonthlyDaysWithGT90Temp, MonthlyDaysWithLT0Temp, MonthlyDaysWithLT32Temp, MonthlyDepartureFromNormalAverageTemperature, MonthlyDepartureFromNormalCoolingDegreeDays, MonthlyDepartureFromNormalHeatingDegreeDays, MonthlyDepartureFromNormalMaximumTemperature, MonthlyDepartureFromNormalMinimumTemperature, MonthlyDepartureFromNormalPrecipitation, MonthlyDewpointTemperature, MonthlyGreatestPrecip, MonthlyGreatestPrecipDate, MonthlyGreatestSnowDepth, MonthlyGreatestSnowDepthDate, MonthlyGreatestSnowfall, MonthlyGreatestSnowfallDate, MonthlyMaxSeaLevelPressureValue, MonthlyMaxSeaLevelPressureValueDate, MonthlyMaxSeaLevelPressureValueTime, MonthlyMaximumTemperature, MonthlyMeanTemperature, MonthlyMinSeaLevelPressureValue, MonthlyMinSeaLevelPressureValueDate, MonthlyMinSeaLevelPressureValueTime, MonthlyMinimumTemperature, MonthlySeaLevelPressure, MonthlyStationPressure, MonthlyTotalLiquidPrecipitation, MonthlyTotalSnowfall, MonthlyWetBulb, AWND, CDSD, CLDD, DSNW, HDSD, HTDD, NormalsCoolingDegreeDay, NormalsHeatingDegreeDay, ShortDurationEndDate005, ShortDurationEndDate010, ShortDurationEndDate015, ShortDurationEndDate020, ShortDurationEndDate030, ShortDurationEndDate045, ShortDurationEndDate060, ShortDurationEndDate080, ShortDurationEndDate100, ShortDurationEndDate120, ShortDurationEndDate150, ShortDurationEndDate180, ShortDurationPrecipitationValue005, ShortDurationPrecipitationValue010, ShortDurationPrecipitationValue015, ShortDurationPrecipitationValue020, ShortDurationPrecipitationValue030, ShortDurationPrecipitationValue045, ShortDurationPrecipitationValue060, ShortDurationPrecipitationValue080, ShortDurationPrecipitationValue100, ShortDurationPrecipitationValue120, ShortDurationPrecipitationValue150, ShortDurationPrecipitationValue180, REM, BackupDirection, BackupDistance, BackupDistanceUnit, BackupElements, BackupElevation, BackupEquipment, BackupLatitude, BackupLongitude, BackupName, WindEquipmentChangeDate" \ -XPUT http://$FE_PROXY/api/quickstart/weatherdata/_stream_load ``` ```text Enter host password for user 'root': { "TxnId": 4, "Label": "weather-0", "Status": "Success", "Message": "OK", "NumberTotalRows": 22931, "NumberLoadedRows": 22931, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 15558550, "LoadTimeMs": 404, "BeginTxnTimeMs": 1, "StreamLoadPlanTimeMs": 7, "ReadDataTimeMs": 157, "WriteDataTimeMs": 372, "CommitAndPublishTimeMs": 23 } ``` #### Connect with a MySQL client[​](#connect-with-a-mysql-client "Direct link to Connect with a MySQL client") Connect with a MySQL client if you are not connected. Remember to use the external IP address of the `kube-starrocks-fe-service` service and the password that you configured in the Kubernetes secret `starrocks-root-pass`. ```bash mysql -P9030 -h $MYSQL_IP -u root --prompt="StarRocks > " -p ``` #### Answer some questions[​](#answer-some-questions "Direct link to Answer some questions") These queries can be run in your SQL client. All of the queries use the `quickstart` database. ```sql USE quickstart; ``` ###### How many crashes are there per hour in NYC?[​](#how-many-crashes-are-there-per-hour-in-nyc "Direct link to How many crashes are there per hour in NYC?") ```sql SELECT COUNT(*), date_trunc("hour", crashdata.CRASH_DATE) AS Time FROM crashdata GROUP BY Time ORDER BY Time ASC LIMIT 200; ``` Here is part of the output. Note that I am looking closer at January 6th and 7th as this is Monday and Tuesday of a non-holiday week. Looking at New Years Day is probably not indicative of a normal morning during rush-hour traffic. ```plaintext | 14 | 2014-01-06 06:00:00 | | 16 | 2014-01-06 07:00:00 | | 43 | 2014-01-06 08:00:00 | | 44 | 2014-01-06 09:00:00 | | 21 | 2014-01-06 10:00:00 | | 28 | 2014-01-06 11:00:00 | | 34 | 2014-01-06 12:00:00 | | 31 | 2014-01-06 13:00:00 | | 35 | 2014-01-06 14:00:00 | | 36 | 2014-01-06 15:00:00 | | 33 | 2014-01-06 16:00:00 | | 40 | 2014-01-06 17:00:00 | | 35 | 2014-01-06 18:00:00 | | 23 | 2014-01-06 19:00:00 | | 16 | 2014-01-06 20:00:00 | | 12 | 2014-01-06 21:00:00 | | 17 | 2014-01-06 22:00:00 | | 14 | 2014-01-06 23:00:00 | | 10 | 2014-01-07 00:00:00 | | 4 | 2014-01-07 01:00:00 | | 1 | 2014-01-07 02:00:00 | | 3 | 2014-01-07 03:00:00 | | 2 | 2014-01-07 04:00:00 | | 6 | 2014-01-07 06:00:00 | | 16 | 2014-01-07 07:00:00 | | 41 | 2014-01-07 08:00:00 | | 37 | 2014-01-07 09:00:00 | | 33 | 2014-01-07 10:00:00 | ``` It looks like about 40 accidents on a Monday or Tuesday morning during rush hour traffic, and around the same at 17:00 hours. ###### What is the average temperature in NYC?[​](#what-is-the-average-temperature-in-nyc "Direct link to What is the average temperature in NYC?") ```sql SELECT avg(HourlyDryBulbTemperature), date_trunc("hour", weatherdata.DATE) AS Time FROM weatherdata GROUP BY Time ORDER BY Time ASC LIMIT 100; ``` Output: Note that this is data from 2014, NYC has not been this cold lately. ```plaintext +-------------------------------+---------------------+ | avg(HourlyDryBulbTemperature) | Time | +-------------------------------+---------------------+ | 25 | 2014-01-01 00:00:00 | | 25 | 2014-01-01 01:00:00 | | 24 | 2014-01-01 02:00:00 | | 24 | 2014-01-01 03:00:00 | | 24 | 2014-01-01 04:00:00 | | 24 | 2014-01-01 05:00:00 | | 25 | 2014-01-01 06:00:00 | | 26 | 2014-01-01 07:00:00 | ``` ###### Is it safe to drive in NYC when visibility is poor?[​](#is-it-safe-to-drive-in-nyc-when-visibility-is-poor "Direct link to Is it safe to drive in NYC when visibility is poor?") Let's look at the number of crashes when visibility is poor (between 0 and 1.0 miles). To answer this question use a JOIN across the two tables on the DATETIME column. ```sql SELECT COUNT(DISTINCT c.COLLISION_ID) AS Crashes, truncate(avg(w.HourlyDryBulbTemperature), 1) AS Temp_F, truncate(avg(w.HourlyVisibility), 2) AS Visibility, max(w.HourlyPrecipitation) AS Precipitation, date_format((date_trunc("hour", c.CRASH_DATE)), '%d %b %Y %H:%i') AS Hour FROM crashdata c LEFT JOIN weatherdata w ON date_trunc("hour", c.CRASH_DATE)=date_trunc("hour", w.DATE) WHERE w.HourlyVisibility BETWEEN 0.0 AND 1.0 GROUP BY Hour ORDER BY Crashes DESC LIMIT 100; ``` The highest number of crashes in a single hour during low visibility is 129. There are multiple things to consider: * February 3rd 2014 was a Monday * 8AM is rush hour * It was raining (0.12 inches or precipitation that hour) * The temperature is 32 degrees F (the freezing point for water) * Visibility is bad at 0.25 miles, normal for NYC is 10 miles ```plaintext +---------+--------+------------+---------------+-------------------+ | Crashes | Temp_F | Visibility | Precipitation | Hour | +---------+--------+------------+---------------+-------------------+ | 129 | 32 | 0.25 | 0.12 | 03 Feb 2014 08:00 | | 114 | 32 | 0.25 | 0.12 | 03 Feb 2014 09:00 | | 104 | 23 | 0.33 | 0.03 | 09 Jan 2015 08:00 | | 96 | 26.3 | 0.33 | 0.07 | 01 Mar 2015 14:00 | | 95 | 26 | 0.37 | 0.12 | 01 Mar 2015 15:00 | | 93 | 35 | 0.75 | 0.09 | 18 Jan 2015 09:00 | | 92 | 31 | 0.25 | 0.12 | 03 Feb 2014 10:00 | | 87 | 26.8 | 0.5 | 0.09 | 01 Mar 2015 16:00 | | 85 | 55 | 0.75 | 0.20 | 23 Dec 2015 17:00 | | 85 | 20 | 0.62 | 0.01 | 06 Jan 2015 11:00 | | 83 | 19.6 | 0.41 | 0.04 | 05 Mar 2015 13:00 | | 80 | 20 | 0.37 | 0.02 | 06 Jan 2015 10:00 | | 76 | 26.5 | 0.25 | 0.06 | 05 Mar 2015 09:00 | | 71 | 26 | 0.25 | 0.09 | 05 Mar 2015 10:00 | | 71 | 24.2 | 0.25 | 0.04 | 05 Mar 2015 11:00 | ``` ###### What about driving in icy conditions?[​](#what-about-driving-in-icy-conditions "Direct link to What about driving in icy conditions?") Water vapor can desublimate to ice at 40 degrees F; this query looks at temps between 0 and 40 degrees F. ```sql SELECT COUNT(DISTINCT c.COLLISION_ID) AS Crashes, truncate(avg(w.HourlyDryBulbTemperature), 1) AS Temp_F, truncate(avg(w.HourlyVisibility), 2) AS Visibility, max(w.HourlyPrecipitation) AS Precipitation, date_format((date_trunc("hour", c.CRASH_DATE)), '%d %b %Y %H:%i') AS Hour FROM crashdata c LEFT JOIN weatherdata w ON date_trunc("hour", c.CRASH_DATE)=date_trunc("hour", w.DATE) WHERE w.HourlyDryBulbTemperature BETWEEN 0.0 AND 40.5 GROUP BY Hour ORDER BY Crashes DESC LIMIT 100; ``` The results for freezing temperatures suprised me a little, I did not expect too much traffic on a Sunday morning in the city on a cold January day.A quick look at [weather.com](https://weather.com/storms/winter/news/northeast-storm-rain-snow-wind) showed that there was a big storm with many crashes that day, just like what can be seen in the data. ```plaintext +---------+--------+------------+---------------+-------------------+ | Crashes | Temp_F | Visibility | Precipitation | Hour | +---------+--------+------------+---------------+-------------------+ | 192 | 34 | 1.5 | 0.09 | 18 Jan 2015 08:00 | | 170 | 21 | NULL | | 21 Jan 2014 10:00 | | 145 | 19 | NULL | | 21 Jan 2014 11:00 | | 138 | 33.5 | 5 | 0.02 | 18 Jan 2015 07:00 | | 137 | 21 | NULL | | 21 Jan 2014 09:00 | | 129 | 32 | 0.25 | 0.12 | 03 Feb 2014 08:00 | | 114 | 32 | 0.25 | 0.12 | 03 Feb 2014 09:00 | | 104 | 23 | 0.7 | 0.04 | 09 Jan 2015 08:00 | | 98 | 16 | 8 | 0.00 | 06 Mar 2015 08:00 | | 96 | 26.3 | 0.33 | 0.07 | 01 Mar 2015 14:00 | ``` Drive carefully! ```sql exit ``` #### Cleanup[​](#cleanup "Direct link to Cleanup") Run this command if you are finished and would like to remove the StarRocks cluster and the StarRocks operator. ```bash helm delete starrocks ``` *** #### Summary[​](#summary "Direct link to Summary") In this tutorial you: * Deployed StarRocks with Helm and the StarRocks Operator * Loaded crash data provided by New York City and weather data provided by NOAA * Analyzed the data using SQL JOINs to find out that driving in low visibility or icy streets is a bad idea There is more to learn; we intentionally glossed over the data transformation done during the Stream Load. The details on that are in the notes on the curl commands below. *** #### Notes on the curl commands[​](#notes-on-the-curl-commands "Direct link to Notes on the curl commands") StarRocks Stream Load and curl take many arguments. Only the ones used in this tutorial are described here, the rest will be linked to in the more information section. ###### `--location-trusted`[​](#--location-trusted "Direct link to --location-trusted") This configures curl to pass credentials to any redirected URLs. ###### `-u root`[​](#-u-root "Direct link to -u-root") The username used to log in to StarRocks ###### `-T filename`[​](#-t-filename "Direct link to -t-filename") T is for transfer, the filename to transfer. ###### `label:name-num`[​](#labelname-num "Direct link to labelname-num") The label to associate with this Stream Load job. The label must be unique, so if you run the job multiple times you can add a number and keep incrementing that. ###### `column_separator:,`[​](#column_separator "Direct link to column_separator") If you load a file that uses a single `,` then set it as shown above, if you use a different delimiter then set that delimiter here. Common choices are `\t`, `,`, and `|`. ###### `skip_header:1`[​](#skip_header1 "Direct link to skip_header1") Some CSV files have a single header row with all of the column names listed, and some add a second line with datatypes. Set skip\_header to `1` or `2` if you have one or two header lines, and set it to `0` if you have none. ###### `enclose:\"`[​](#enclose "Direct link to enclose") It is common to enclose strings that contain embedded commas with double-quotes. The sample datasets used in this tutorial have geo locations that contain commas and so the enclose setting is set to `\"`. Remember to escape the `"` with a `\`. ###### `max_filter_ratio:1`[​](#max_filter_ratio1 "Direct link to max_filter_ratio1") This allows some errors in the data. Ideally this would be set to `0` and the job would fail with any errors. It is set to `1` to allow all rows to fail during debugging. ###### `columns:`[​](#columns "Direct link to columns") The mapping of CSV file columns to StarRocks table columns. You will notice that there are many more columns in the CSV files than columns in the table. Any columns that are not included in the table are skipped. You will also notice that there is some transformation of data included in the `columns:` line for the crash dataset. It is very common to find dates and times in CSV files that do not conform to standards. This is the logic for converting the CSV data for the time and date of the crash to a DATETIME type: ###### The columns line[​](#the-columns-line "Direct link to The columns line") This is the beginning of one data record. The date is in `MM/DD/YYYY` format, and the time is `HH:MI`. Since DATETIME is generally `YYYY-MM-DD HH:MI:SS` we need to transform this data. ```plaintext 08/05/2014,9:10,BRONX,10469,40.8733019,-73.8536375,"(40.8733019, -73.8536375)", ``` This is the beginning of the `columns:` parameter: ```bash -H "columns:tmp_CRASH_DATE, tmp_CRASH_TIME, CRASH_DATE=str_to_date(concat_ws(' ', tmp_CRASH_DATE, tmp_CRASH_TIME), '%m/%d/%Y %H:%i') ``` This instructs StarRocks to: * Assign the content of the first column of the CSV file to `tmp_CRASH_DATE` * Assign the content of the second column of the CSV file to `tmp_CRASH_TIME` * `concat_ws()` concatenates `tmp_CRASH_DATE` and `tmp_CRASH_TIME` together with a space between them * `str_to_date()` creates a DATETIME from the concatenated string * store the resulting DATETIME in the column `CRASH_DATE` *** #### More information[​](#more-information "Direct link to More information") Default [`values.yaml`](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/helm-charts/charts/kube-starrocks/values.yaml) [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) The [Motor Vehicle Collisions - Crashes](https://data.cityofnewyork.us/Public-Safety/Motor-Vehicle-Collisions-Crashes/h9gi-nx95) dataset is provided by New York City subject to these [terms of use](https://www.nyc.gov/home/terms-of-use.page) and [privacy policy](https://www.nyc.gov/home/privacy-policy.page). The [Local Climatological Data](https://www.ncdc.noaa.gov/cdo-web/datatools/lcd)(LCD) is provided by NOAA with this [disclaimer](https://www.noaa.gov/disclaimer) and this [privacy policy](https://www.noaa.gov/protecting-your-privacy). [Helm](https://helm.sh/) is a package manager for Kubernetes. A [Helm Chart](https://helm.sh/docs/topics/charts/) is a Helm package and contains all of the resource definitions necessary to run an application on a Kubernetes cluster. [`starrocks-kubernetes-operator` and `kube-starrocks` Helm Chart](https://github.com/StarRocks/starrocks-kubernetes-operator). --- ### Apache Hudi Lakehouse #### Overview[​](#overview "Direct link to Overview") * Deploy Object Storage, Apache Spark, Hudi, and StarRocks using Docker compose * Load a tiny dataset into Hudi with Apache Spark * Configure StarRocks to access the Hive Metastore using an external catalog * Query the data with StarRocks where the data sits ![DLA](/assets/images/1.1-8-dla-c67d601d709b092317fa11eb64ac1783.png) In addition to efficient analytics of local data, StarRocks can work as the compute engine to analyze data stored in data lakes such as Apache Hudi, Apache Iceberg, and Delta Lake. One of the key features of StarRocks is its external catalog, which acts as the linkage to an externally maintained metastore. This functionality provides users with the capability to query external data sources seamlessly, eliminating the need for data migration. As such, users can analyze data from different systems such as HDFS and Amazon S3, in various file formats such as Parquet, ORC, and CSV, etc. The preceding figure shows a data lake analytics scenario where StarRocks is responsible for data computing and analysis, and the data lake is responsible for data storage, organization, and maintenance. Data lakes allow users to store data in open storage formats and use flexible schemas to produce reports on "single source of truth" for various BI, AI, ad-hoc, and reporting use cases. StarRocks fully leverages the advantages of its vectorization engine and CBO, significantly improving the performance of data lake analytics. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") ##### StarRocks `demo` repository[​](#starrocks-demo-repository "Direct link to starrocks-demo-repository") Clone the [StarRocks demo repository](https://github.com/StarRocks/demo/) to your local machine. All the steps in this guide will be run from the `demo/documentation-samples/hudi/` directory in the directory where you cloned the `demo` GitHub repo. ##### Docker[​](#docker "Direct link to Docker") * Docker Setup: For Mac, Please follow the steps as defined in [Install Docker Desktop on Mac](https://docs.docker.com/desktop/setup/install/mac-install/). For running Spark-SQL queries, please ensure at least 5 GB memory and 4 CPUs are allocated to Docker (See Docker → Preferences → Advanced). Otherwise, spark-SQL queries could be killed because of memory issues. * 20 GB free disk space assigned to Docker ##### SQL client[​](#sql-client "Direct link to SQL client") You can use the SQL client provided in the Docker environment, or use one on your system. Many MySQL compatible clients will work. #### Configuration[​](#configuration "Direct link to Configuration") Change directory into `demo/documentation-samples/hudi` and look at the files. This is not a tutorial on Hudi, so not every configuration file will be described; but it is important for the reader to know where to look to see how things are configured. In the `hudi/` directory you will find the `docker-compose.yml` file which is used to launch and configure the services in Docker. Here is a list of those services and a brief description: ##### Docker services[​](#docker-services "Direct link to Docker services") | Service | Responsibilities | | ------------------------ | ------------------------------------------------------------------- | | **`starrocks-fe`** | Metadata management, client connections, query plans and scheduling | | **`starrocks-be`** | Running query plans | | **`metastore_db`** | Postgres DB used to store the Hive metadata | | **`hive_metastore`** | Provides the Apache Hive metastore | | **`minio`** and **`mc`** | MinIO Object Storage and MinIO command line client | | **`spark-hudi`** | Distributed computing and Transactional data lake platform | ##### Configuration files[​](#configuration-files "Direct link to Configuration files") In the `hudi/conf/` directory you will find configuration files that get mounted in the `spark-hudi` container. ###### `core-site.xml`[​](#core-sitexml "Direct link to core-sitexml") This file contains the object storage related settings. Links for this and other items in More information at the end of this document. ###### `spark-defaults.conf`[​](#spark-defaultsconf "Direct link to spark-defaultsconf") Settings for Hive, MinIO, and Spark SQL. ###### `hudi-defaults.conf`[​](#hudi-defaultsconf "Direct link to hudi-defaultsconf") Default file used to silence warnings in the `spark-shell`. ###### `hadoop-metrics2-hbase.properties`[​](#hadoop-metrics2-hbaseproperties "Direct link to hadoop-metrics2-hbaseproperties") Empty file used to silence warnings in the `spark-shell`. ###### `hadoop-metrics2-s3a-file-system.properties`[​](#hadoop-metrics2-s3a-file-systemproperties "Direct link to hadoop-metrics2-s3a-file-systemproperties") Empty file used to silence warnings in the `spark-shell`. #### Bringing up Demo Cluster[​](#bringing-up-demo-cluster "Direct link to Bringing up Demo Cluster") This demo system consists of StarRocks, Hudi, MinIO, and Spark services. Run Docker compose to bring up the cluster: ```bash docker compose up --detach --wait --wait-timeout 60 ``` ```plaintext [+] Running 8/8 ✔ Network hudi Created 0.0s ✔ Container hudi-starrocks-fe-1 Healthy 0.1s ✔ Container hudi-minio-1 Healthy 0.1s ✔ Container hudi-metastore_db-1 Healthy 0.1s ✔ Container hudi-starrocks-be-1 Healthy 0.0s ✔ Container hudi-mc-1 Healthy 0.0s ✔ Container hudi-hive-metastore-1 Healthy 0.0s ✔ Container hudi-spark-hudi-1 Healthy 0.1s ``` tip With many containers running, `docker compose ps` output is easier to read if you pipe it to `jq`: ```bash docker compose ps --format json | \ jq '{Service: .Service, State: .State, Status: .Status}' ``` ```json { "Service": "hive-metastore", "State": "running", "Status": "Up About a minute (healthy)" } { "Service": "mc", "State": "running", "Status": "Up About a minute" } { "Service": "metastore_db", "State": "running", "Status": "Up About a minute" } { "Service": "minio", "State": "running", "Status": "Up About a minute" } { "Service": "spark-hudi", "State": "running", "Status": "Up 33 seconds (healthy)" } { "Service": "starrocks-be", "State": "running", "Status": "Up About a minute (healthy)" } { "Service": "starrocks-fe", "State": "running", "Status": "Up About a minute (healthy)" } ``` #### Configure MinIO[​](#configure-minio "Direct link to Configure MinIO") When you run the Spark commands you will set the basepath for the table being created to an `s3a` URI: ```java val basePath = "s3a://huditest/hudi_coders" ``` In this step you will create the bucket `huditest` in MinIO. The MinIO console is running on port `9000`. ##### Authenticate to MinIO[​](#authenticate-to-minio "Direct link to Authenticate to MinIO") Open a browser to and authenticate. The username and password are specified in `docker-compose.yml`; they are `admin` and `password`. ##### Create a bucket[​](#create-a-bucket "Direct link to Create a bucket") In the left navigation select **Buckets**, and then **Create Bucket +**. Name the bucket `huditest` and select **Create Bucket** ![Create bucket huditest](/assets/images/hudi-test-bucket-7e936058aca4001f97f613ee96f6c7d7.png) #### Create and populate a table, then sync it to Hive[​](#create-and-populate-a-table-then-sync-it-to-hive "Direct link to Create and populate a table, then sync it to Hive") tip Run this command, and any other `docker compose` commands, from the directory containing the `docker-compose.yml` file. Open `spark-shell` in the `spark-hudi` service ```bash docker compose exec spark-hudi spark-shell ``` note There will be warnings when `spark-shell` starts about illegal reflective access. You can ignore these warnings. Run these commands at the `scala>` prompt to: * Configure this Spark session to load, process, and write data * Create a dataframe and write that to a Hudi table * Sync to the Hive Metastore ```scala import org.apache.spark.sql.functions._ import org.apache.spark.sql.types._ import org.apache.spark.sql.Row import org.apache.spark.sql.SaveMode._ import org.apache.hudi.DataSourceReadOptions._ import org.apache.hudi.DataSourceWriteOptions._ import org.apache.hudi.config.HoodieWriteConfig._ import scala.collection.JavaConversions._ val schema = StructType( Array( StructField("language", StringType, true), StructField("users", StringType, true), StructField("id", StringType, true) )) val rowData= Seq(Row("Java", "20000", "a"), Row("Python", "100000", "b"), Row("Scala", "3000", "c")) val df = spark.createDataFrame(rowData,schema) val databaseName = "hudi_sample" val tableName = "hudi_coders_hive" val basePath = "s3a://huditest/hudi_coders" df.write.format("hudi"). option(org.apache.hudi.config.HoodieWriteConfig.TABLE_NAME, tableName). option(RECORDKEY_FIELD_OPT_KEY, "id"). option(PARTITIONPATH_FIELD_OPT_KEY, "language"). option(PRECOMBINE_FIELD_OPT_KEY, "users"). option("hoodie.datasource.write.hive_style_partitioning", "true"). option("hoodie.datasource.hive_sync.enable", "true"). option("hoodie.datasource.hive_sync.mode", "hms"). option("hoodie.datasource.hive_sync.database", databaseName). option("hoodie.datasource.hive_sync.table", tableName). option("hoodie.datasource.hive_sync.partition_fields", "language"). option("hoodie.datasource.hive_sync.partition_extractor_class", "org.apache.hudi.hive.MultiPartKeysValueExtractor"). option("hoodie.datasource.hive_sync.metastore.uris", "thrift://hive-metastore:9083"). mode(Overwrite). save(basePath) System.exit(0) ``` note You will see a warning: ```java WARN org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not found at path s3a://huditest/hudi_coders/.hoodie/metadata ``` This can be ignored, the file will be created automatically during this `spark-shell` session. There will also be a warning: ```bash 78184 [main] WARN org.apache.hadoop.fs.s3a.S3ABlockOutputStream - Application invoked the Syncable API against stream writing to hudi_coders/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-0-0. This is unsupported ``` This warning informs you that syncing a log file that is open for writes is not supported when using object storage. The file will only be synced when it is closed. See [Stack Overflow](https://stackoverflow.com/a/74886836/10424890). The final command in the above spark-shell session should exit the container, if it doesn't press enter and it will exit. #### Configure StarRocks[​](#configure-starrocks "Direct link to Configure StarRocks") ##### Connect to StarRocks[​](#connect-to-starrocks "Direct link to Connect to StarRocks") Connect to StarRocks with the provided MySQL client provided by the `starrocks-fe` service, or use your favorite SQL client and configure it to connect using the MySQL protocol on `localhost:9030`. ```bash docker compose exec starrocks-fe \ mysql -P 9030 -h 127.0.0.1 -u root --prompt="StarRocks > " ``` ##### Create the linkage between StarRocks and Hudi[​](#create-the-linkage-between-starrocks-and-hudi "Direct link to Create the linkage between StarRocks and Hudi") There is a link at the end of this guide with more information on external catalogs. The external catalog created in this step acts as the linkage to the Hive Metastore (HMS) running in Docker. ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://hive-metastore:9083", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "admin", "aws.s3.secret_key" = "password", "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "http://minio:9000" ); ``` ```plaintext Query OK, 0 rows affected (0.59 sec) ``` ##### Use the new catalog[​](#use-the-new-catalog "Direct link to Use the new catalog") ```sql SET CATALOG hudi_catalog_hms; ``` ```plaintext Query OK, 0 rows affected (0.01 sec) ``` ##### Navigate to the data inserted with Spark[​](#navigate-to-the-data-inserted-with-spark "Direct link to Navigate to the data inserted with Spark") ```sql SHOW DATABASES; ``` ```plaintext +--------------------+ | Database | +--------------------+ | default | | hudi_sample | | information_schema | +--------------------+ 2 rows in set (0.40 sec) ``` ```sql USE hudi_sample; ``` ```plaintext Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed ``` ```sql SHOW TABLES; ``` ```plaintext +-----------------------+ | Tables_in_hudi_sample | +-----------------------+ | hudi_coders_hive | +-----------------------+ 1 row in set (0.07 sec) ``` ##### Query the data in Hudi with StarRocks[​](#query-the-data-in-hudi-with-starrocks "Direct link to Query the data in Hudi with StarRocks") Run this query twice, the first time may take around five seconds to complete as data is not yet cached in StarRocks. The second query will be very quick. ```sql SELECT * from hudi_coders_hive\G ``` tip Some of the SQL queries in the StarRocks documentation end with `\G` instead of a semicolon. The `\G` causes the mysql CLI to render the query results vertically. Many SQL clients do not interpret vertical formatting output, so you should replace `\G` with `;` if you are not using the mysql CLI. ```plaintext *************************** 1. row *************************** _hoodie_commit_time: 20240208165522561 _hoodie_commit_seqno: 20240208165522561_0_0 _hoodie_record_key: c _hoodie_partition_path: language=Scala _hoodie_file_name: bb29249a-b69d-4c32-843b-b7142d8dc51c-0_0-27-1221_20240208165522561.parquet language: Scala users: 3000 id: c *************************** 2. row *************************** _hoodie_commit_time: 20240208165522561 _hoodie_commit_seqno: 20240208165522561_2_0 _hoodie_record_key: a _hoodie_partition_path: language=Java _hoodie_file_name: 12fc14aa-7dc4-454c-b710-1ad0556c9386-0_2-27-1223_20240208165522561.parquet language: Java users: 20000 id: a *************************** 3. row *************************** _hoodie_commit_time: 20240208165522561 _hoodie_commit_seqno: 20240208165522561_1_0 _hoodie_record_key: b _hoodie_partition_path: language=Python _hoodie_file_name: 51977039-d71e-4dd6-90d4-0c93656dafcf-0_1-27-1222_20240208165522561.parquet language: Python users: 100000 id: b 3 rows in set (0.15 sec) ``` #### Summary[​](#summary "Direct link to Summary") This tutorial exposed you to the use of a StarRocks external catalog to show you that you can query your data where it sits using the Hudi external catalog. Many other integrations are available using Iceberg, Delta Lake, and JDBC catalogs. In this tutorial you: * Deployed StarRocks and a Hudi/Spark/MinIO environment in Docker * Loaded a tiny dataset into Hudi with Apache Spark * Configured a StarRocks external catalog to provide access to the Hudi catalog * Queried the data with SQL in StarRocks without copying the data from the data lake #### More information[​](#more-information "Direct link to More information") [StarRocks Catalogs](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md) [Apache Hudi quickstart](https://hudi.apache.org/docs/quick-start-guide/) (includes Spark) [Apache Hudi S3 configuration](https://hudi.apache.org/docs/s3_hoodie/) [Apache Spark configuration docs](https://spark.apache.org/docs/latest/configuration.html) --- ### Apache Iceberg Lakehouse This guide will get you up and running with Apache Iceberg™ using StarRocks™, including sample code to highlight some powerful features. ##### Docker-Compose[​](#docker-compose "Direct link to Docker-Compose") The fastest way to get started is to use a docker-compose file that uses the `starrocks/fe-ubuntu` and `starrocks/be-ubuntu` images which contain a local StarRocks cluster with a configured Iceberg catalog. To use this, you'll need to install the Docker CLI. Once you have Docker installed, save the yaml below into a file named docker-compose.yml: ```yml services: starrocks-fe: image: starrocks/fe-ubuntu:4.0-latest hostname: starrocks-fe container_name: starrocks-fe user: root command: | bash /opt/starrocks/fe/bin/start_fe.sh --host_type FQDN ports: - 8030:8030 - 9020:9020 - 9030:9030 networks: iceberg_net: environment: - AWS_ACCESS_KEY_ID=admin - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 healthcheck: test: 'mysql -u root -h starrocks-fe -P 9030 -e "SHOW FRONTENDS\G" |grep "Alive: true"' interval: 10s timeout: 5s retries: 3 starrocks-be: image: starrocks/be-ubuntu:4.0-latest command: - /bin/bash - -c - | ulimit -n 65535; echo "# Enable data cache" >> /opt/starrocks/be/conf/be.conf echo "block_cache_enable = true" >> /opt/starrocks/be/conf/be.conf echo "block_cache_mem_size = 536870912" >> /opt/starrocks/be/conf/be.conf echo "block_cache_disk_size = 1073741824" >> /opt/starrocks/be/conf/be.conf sleep 15s mysql --connect-timeout 2 -h starrocks-fe -P 9030 -u root -e "ALTER SYSTEM ADD BACKEND \"starrocks-be:9050\";" bash /opt/starrocks/be/bin/start_be.sh ports: - 8040:8040 hostname: starrocks-be container_name: starrocks-be user: root depends_on: starrocks-fe: condition: service_healthy healthcheck: test: 'mysql -u root -h starrocks-fe -P 9030 -e "SHOW BACKENDS\G" |grep "Alive: true"' interval: 10s timeout: 5s retries: 3 networks: iceberg_net: environment: - HOST_TYPE=FQDN - AWS_EC2_METADATA_DISABLED=true rest: image: apache/iceberg-rest-fixture container_name: iceberg-rest networks: iceberg_net: aliases: - iceberg-rest.minio ports: - 8181:8181 environment: - AWS_ACCESS_KEY_ID=admin - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 - CATALOG_WAREHOUSE=s3://warehouse/ - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO - CATALOG_S3_ENDPOINT=http://minio:9000 minio: image: minio/minio:RELEASE.2024-10-29T16-01-48Z container_name: minio environment: - MINIO_ROOT_USER=admin - MINIO_ROOT_PASSWORD=password - MINIO_DOMAIN=minio networks: iceberg_net: aliases: - warehouse.minio ports: - 9001:9001 - 9000:9000 command: ["server", "/data", "--console-address", ":9001"] mc: depends_on: - minio image: minio/mc:RELEASE.2024-10-29T15-34-59Z container_name: mc networks: iceberg_net: environment: - AWS_ACCESS_KEY_ID=admin - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 entrypoint: > /bin/sh -c " until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done; /usr/bin/mc rm -r --force minio/warehouse; /usr/bin/mc mb minio/warehouse; /usr/bin/mc policy set public minio/warehouse; tail -f /dev/null " networks: iceberg_net: ``` Next, start up the docker containers with this command: ```plain docker compose up --detach --wait --wait-timeout 400 ``` You can then run any of the following commands to start a StarRocks session. ```bash docker exec -it starrocks-fe \ mysql -P 9030 -h 127.0.0.1 -u root --prompt="StarRocks > " ``` ##### Adding and Using a Catalog[​](#adding-and-using-a-catalog "Direct link to Adding and Using a Catalog") ```sql CREATE EXTERNAL CATALOG 'demo' COMMENT "External catalog to Apache Iceberg on MinIO" PROPERTIES ( "type"="iceberg", "iceberg.catalog.type"="rest", "iceberg.catalog.uri"="http://iceberg-rest:8181", "iceberg.catalog.warehouse"="warehouse", "aws.s3.access_key"="admin", "aws.s3.secret_key"="password", "aws.s3.endpoint"="http://minio:9000", "aws.s3.enable_path_style_access"="true" ); ``` ```sql SHOW CATALOGS\G ``` ```sql *************************** 1. row *************************** Catalog: default_catalog Type: Internal Comment: An internal catalog contains this cluster's self-managed tables. *************************** 2. row *************************** Catalog: demo Type: Iceberg Comment: External catalog to Apache Iceberg on MinIO 2 rows in set (0.00 sec) ``` ```sql SET CATALOG demo; ``` ##### Creating and using a database[​](#creating-and-using-a-database "Direct link to Creating and using a database") ```sql CREATE DATABASE nyc; ``` ```sql USE nyc; ``` ##### Creating a table[​](#creating-a-table "Direct link to Creating a table") ```sql CREATE TABLE demo.nyc.taxis ( trip_id bigint, trip_distance float, fare_amount double, store_and_fwd_flag string, vendor_id bigint ) PARTITION BY (vendor_id); ``` ##### Writing Data to a Table[​](#writing-data-to-a-table "Direct link to Writing Data to a Table") ```sql INSERT INTO demo.nyc.taxis VALUES (1000371, 1.8, 15.32, 'N', 1), (1000372, 2.5, 22.15, 'N', 2), (1000373, 0.9, 9.01, 'N', 2), (1000374, 8.4, 42.13, 'Y', 1); ``` ##### Reading Data from a Table[​](#reading-data-from-a-table "Direct link to Reading Data from a Table") ```sql SELECT * FROM demo.nyc.taxis; ``` ##### Verify that the data is stored in object storage[​](#verify-that-the-data-is-stored-in-object-storage "Direct link to Verify that the data is stored in object storage") When you added and used the external catalog, Starrocks started using MinIO as the object store for the `demo.nyc.taxis` table. If you navigate to and then navigate through the Object Browser menu to `warehouse/nyc/taxis/` you can confirm that StarRocks is using MinIO for the storage. tip The username and password for MinIO are in the docker-compose.yml file. You will be prompted to change the password to something better, just ignore this advice for the tutorial. ![img](/assets/images/MinIO-Iceberg-data-8ade61c31be69444bd02b00acafe263c.png) ##### Next Steps[​](#next-steps "Direct link to Next Steps") ###### Adding Iceberg to StarRocks[​](#adding-iceberg-to-starrocks "Direct link to Adding Iceberg to StarRocks") If you already have a StarRocks 3.2.0, or later, environment, it comes with the Iceberg 1.6.0 included. No additional downloads or jars are needed. ###### Learn More[​](#learn-more "Direct link to Learn More") Now that you're up and running with Iceberg and StarRocks, check out the [StarRocks-Iceberg docs](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md) to learn more! --- ### Claude + StarRocks MCP Stand up a small StarRocks cluster on object storage, load a real public dataset, and ask **Claude** plain-English questions about it through the **StarRocks MCP server**. Claude discovers the schema, writes the SQL (including multi-table joins), and renders charts. All data lives in object storage on **MinIO**. This tutorial covers: * Running StarRocks in shared-data mode (1 FE + 1 CN) and MinIO in Docker * Creating an S3 (MinIO) storage volume for separate storage and compute * Loading the Olist Brazilian E-Commerce dataset (8 related tables) * Wiring the StarRocks and AIStor MCP servers to Claude * Asking Claude plain-English questions and rendering charts The dataset is the **Olist Brazilian E-Commerce** set — 8 related tables, good for genuinely complex joins. Everything runs on **1 FE + 1 CN**, on a laptop or a free cloud tier. There is a lot of information in this document, and it is presented with the step by step content at the beginning and the reference material at the end. This is done so that you can stand up the environment and start asking questions first, and read the supporting details afterward. *** #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") ##### Docker[​](#docker "Direct link to Docker") * [Docker](https://www.docker.com/get-started/) (Docker Desktop or engine + compose) * \~4 GB RAM assigned to Docker ##### MySQL client[​](#mysql-client "Direct link to MySQL client") A MySQL client (e.g. `mysql`) to run the SQL files. This is provided by the StarRocks FE server, so all of the SQL commands in this guide are run with `docker compose exec`. ##### uv[​](#uv "Direct link to uv") [`uv`](https://docs.astral.sh/uv/) runs the StarRocks MCP server. ##### Claude Code or Claude Desktop[​](#claude-code-or-claude-desktop "Direct link to Claude Code or Claude Desktop") [Claude Code](https://claude.com/product/claude-code) or Claude Desktop connects the MCP servers. tip This demo includes steps using Claude Code. Other LLMs work with MCP servers, but this demo has only been tested with Claude Code. Please open an issue and let us know what you experience with other LLMs. ##### The Olist dataset[​](#the-olist-dataset "Direct link to The Olist dataset") The **Olist dataset** is downloaded automatically when you load the data via `kagglehub` (no Kaggle account needed): . note Apple Silicon: StarRocks leans on AVX2 (x86); use an ARM build or expect slower emulation. *** #### Terminology[​](#terminology "Direct link to Terminology") ##### MCP[​](#mcp "Direct link to MCP") The Model Context Protocol (MCP) is an open protocol that lets an AI assistant such as Claude discover and call external tools. This tutorial wires up two MCP servers: * The **StarRocks MCP server** (`mcp-server-starrocks`) exposes tools that let Claude read the schema and run SQL against StarRocks. * The **AIStor MCP server** (`aistor`) exposes tools that let Claude work with the MinIO object storage where the data lives — for example, browsing buckets and inspecting the objects StarRocks writes. ##### FE[​](#fe "Direct link to FE") Frontend nodes are responsible for metadata management, client connection management, query planning, and query scheduling. ##### CN[​](#cn "Direct link to CN") Compute Nodes are responsible for executing query plans in shared-data deployments. *** #### Install the prerequisites[​](#install-the-prerequisites "Direct link to Install the prerequisites") ##### `uv`[​](#uv-1 "Direct link to uv-1") ```bash # macOS / Linux curl -LsSf https://astral.sh/uv/install.sh | sh # Windows (PowerShell) powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" # Or via Homebrew / pipx / pip brew install uv # pipx install uv # pip install uv ``` See the [official `uv` installation guide](https://docs.astral.sh/uv/getting-started/installation/) for other options. After installing, verify it is on your `PATH`: ```bash uv --version ``` ##### Docker[​](#docker-1 "Direct link to Docker") ##### Claude Code[​](#claude-code "Direct link to Claude Code") *** #### Clone the demo repo[​](#clone-the-demo-repo "Direct link to Clone the demo repo") Clone : ```bash gh repo clone StarRocks/demo ``` or ```bash git clone git@github.com:StarRocks/demo.git ``` note This directory contains just enough to get you going. It deliberately contains *no* canned queries, no expected answers, and no walkthrough — so that everything Claude shows is reasoned from the live schema, not recalled from material in the checkout. *** #### Bring up StarRocks and MinIO[​](#bring-up-starrocks-and-minio "Direct link to Bring up StarRocks and MinIO") ```bash cd demo/documentation-samples/MCP docker compose up --detach --wait --wait-timeout 120 ``` A frontend (FE), a compute node (CN), and MinIO come up locally. Check for healthy status on the MinIO, FE, and CN services: ```bash docker compose ps -a --format "table {{.Service}}\t{{.Status}}" ``` tip If the CN is not reporting healthy just wait a few seconds and check again — it is the last service to start. *** #### Create a bucket in MinIO[​](#create-a-bucket-in-minio "Direct link to Create a bucket in MinIO") Open the MinIO console at (login `miniouser` / `M!n10R0cks`), click **Create Bucket**, and create the bucket `my-starrocks-bucket`. *** #### Create the storage volume[​](#create-the-storage-volume "Direct link to Create the storage volume") The file `storage_volume.sql` creates a StarRocks storage volume in the bucket you created in the previous step and sets it as the default. The contents of the file are explained in detail at the end of this tutorial; for now, run it: ```bash docker compose exec -T starrocks-fe \ mysql -P9030 -h127.0.0.1 -uroot < storage_volume.sql ``` This must succeed (and the volume must be the default) before any `CREATE TABLE`. ```sql *************************** 1. row *************************** Name: s3_volume Type: S3 IsDefault: true Location: s3://my-starrocks-bucket/ Params: {"aws.s3.access_key":"******","aws.s3.secret_key":"******","aws.s3.endpoint":"minio:9000","aws.s3.region":"us-east-1","aws.s3.use_instance_profile":"false","aws.s3.use_web_identity_token_file":"false","aws.s3.use_aws_sdk_default_behavior":"false"} Enabled: true Comment: ``` *** #### Create the tables[​](#create-the-tables "Direct link to Create the tables") ```bash docker compose exec -T starrocks-fe \ mysql -h127.0.0.1 -P9030 -uroot < olist_schema.sql ``` tip If you see an error `The specified bucket does not exist` you may have skipped part of the previous step. Open the MinIO UI and create the bucket specified above. tip This is a good time to try out the MCP server. Start Claude Code from the current directory and allow the two MCP servers (`aistor` and `mcp-server-starrocks`), then ask Claude to list the databases and describe the schema in the `olist` DB. *** #### Load the data[​](#load-the-data "Direct link to Load the data") Download the Olist CSVs from Kaggle with **kagglehub** — anonymous, no Kaggle account or API token required. It caches the files locally and prints the folder they landed in. Download the data and capture the folder it landed in as `CSV_DIR`. kagglehub prints a version warning to stdout, so take the last line (the path) with `tail -n1`: ```bash export CSV_DIR="$(uv run --with "kagglehub==0.3.12" python \ -c "import kagglehub; print(kagglehub.dataset_download('olistbr/brazilian-ecommerce'))" \ | tail -n1)" echo "$CSV_DIR" # sanity check: should be a .../brazilian-ecommerce/versions/N path ``` note The `kagglehub==0.3.12` pin is deliberate: newer releases (1.0.x) pull a `kagglesdk` build that fails to import (`ModuleNotFoundError: kagglesdk.competitions.legacy`). 0.3.12 downloads public datasets anonymously and works fine. kagglehub caches the files, so re-running is cheap. Then run the loader (`CSV_DIR` is already exported): ```bash FE_HTTP_PORT=8040 bash load_olist.sh ``` `FE_HTTP_PORT=8040` posts Stream Load straight to the CN, avoiding a `starrocks-cn` hostname redirect (no `/etc/hosts` edit / no sudo needed). The script prints a row-count check at the end. *** #### Wire the MCP servers to Claude[​](#wire-the-mcp-servers-to-claude "Direct link to Wire the MCP servers to Claude") ```bash cp .env.example .env ``` `.mcp.json` defines two MCP servers — `mcp-server-starrocks` (run from GitHub via `uv`, so there's **nothing to install by hand**) and `aistor` (run via Docker) — both reading `.env`. Launch Claude Code from this directory (or add the same block to Claude Desktop's config). note **Approve the servers on first launch.** Project-scoped servers from `.mcp.json` are *not* trusted automatically — Claude Code prompts *"New MCP servers found — approve?"* the first time you launch here. Accept it. (The pre-approval lives in `.claude/settings.local.json`, which is gitignored, so it is **absent on a fresh clone** — that's expected; approving via the prompt recreates it.) **Confirm the connection with `/mcp`.** Run `/mcp` inside Claude Code and check that both `mcp-server-starrocks` and `aistor` show **connected**. This is the authoritative status check — if a server isn't connected, its tools won't load no matter how you ask. **First launch is slow.** The first `uv run` downloads the server's dependencies (pyarrow, kaleido, …, ~115 MB), so the StarRocks server can take a couple of minutes to come up the first time. Subsequent launches are fast. Once `/mcp` shows both servers connected, confirm the tools are available (e.g. ask *"What tools does the StarRocks MCP server provide?"*). The StarRocks cluster is reachable over the MySQL protocol as `root` (no password) at `localhost:9030`, database `olist`. *** #### Ask questions[​](#ask-questions "Direct link to Ask questions") Ask Claude plain-English questions about the data — start by having it orient itself, e.g. *"What tables are in this database, and how do they relate to each other?"*, then explore from there. Claude inspects the live schema and writes the SQL itself. ##### Suggested questions[​](#suggested-questions "Direct link to Suggested questions") These are the questions asked in the demo video, in order. Each one builds on the last — start with the first to let Claude orient itself, then work down: * *What databases and tables are in here?* * *How do these tables fit together?* * *Where does this data actually live?* * *Plot orders by customer state.* * *Does it cost more to ship things farther?* * *Add a second line — freight per kilo — to separate weight from distance.* * *How did you build that query?* * *Do late deliveries hurt our reviews?* * *Are you using canned answers, or did every number come from the live tables?* tip **Charts:** ask for an interactive chart in **HTML** format and the StarRocks MCP server writes it to `DEMO_Output/` (set by `STARROCKS_CHART_OUTPUT_DIR` in `.env`), with a static preview shown inline — open the HTML file in a browser to hover / zoom / pan. *** #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") * **StarRocks MCP tools never load / server "still connecting":** the project servers were never approved. Run `/mcp` — if they aren't **connected**, relaunch Claude Code from this directory and accept the *"New MCP servers found — approve?"* prompt (or add `"enabledMcpjsonServers": ["mcp-server-starrocks", "aistor"]` to `.claude/settings.local.json`). You do **not** need to install the server separately — `.mcp.json` runs it from GitHub via `uv`. * **First launch takes a couple of minutes:** the first `uv run` downloads the server's deps (~115 MB). This is expected once, not a failure. * **`mysql` client fails with `Authentication plugin 'mysql_native_password' cannot be loaded`:** a Homebrew `mysql`-client quirk, not a StarRocks problem. Query through the container instead: `docker compose exec -T starrocks-fe mysql -h127.0.0.1 -P9030 -uroot -e "SHOW DATABASES;"`. * **`CREATE TABLE` hangs / times out:** the CN's AWS SDK is probing the EC2 metadata service. The compose file already sets `AWS_EC2_METADATA_DISABLED=true` on `starrocks-cn`; if you use your own compose file, add it. * **Stream Load redirect errors:** use `FE_HTTP_PORT=8040` (as above) to post to the CN directly, or add `127.0.0.1 starrocks-cn` to `/etc/hosts`. * **`order_reviews` rejects rows:** the free-text comment columns contain embedded newlines; use the LEAN variant noted in `load_olist.sh`. *** #### Summary[​](#summary "Direct link to Summary") In this tutorial you: * Deployed StarRocks shared-data (1 FE + 1 CN) and MinIO in Docker * Created an S3 (MinIO) storage volume and set it as the default * Loaded the Olist Brazilian E-Commerce dataset (8 related tables) * Wired the StarRocks and AIStor MCP servers to Claude * Asked Claude plain-English questions and let it discover the schema, write the SQL, and render charts The build/setup material — narrated walkthrough and the video script — is maintained separately so this environment stays free of canned answers: * **StarRocks MCP server:** *** #### What's in here[​](#whats-in-here "Direct link to What's in here") The demo files live in the `documentation-samples/MCP` directory of the [StarRocks/demo](https://github.com/StarRocks/demo) repo: | File | What it is | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `CLAUDE.md` | Instructions to Claude Code to show its work, rely on the schema and not outside information, etc. You should read this file. | | `.claude/settings.json` | Content is `"autoMemoryEnabled": false` to prevent Claude from remembering the questions (and answers) asked in previous sessions. You can remove this if you like; it is in here to prevent the author's questions from biasing your experimentation. | | `docker-compose.yml` | StarRocks shared-data quick start (1 FE + 1 CN + MinIO), patched for laptop/Docker use. | | `storage_volume.sql` | Creates the S3 (MinIO) storage volume and sets it as default. | | `olist_schema.sql` | DDL for the 8 (+1 optional) Olist tables. | | `load_olist.sh` | Stream Load for the Olist CSVs + a row-count check. | | `.mcp.json` | Wires the StarRocks and AIStor MCP servers to Claude. | | `.env.example` | Credentials/endpoints template — `cp` to `.env` and edit if needed. | *** #### Notes on `storage_volume.sql`[​](#notes-on-storage_volumesql "Direct link to notes-on-storage_volumesql") `storage_volume.sql` does two things: it raises a timeout, then creates and activates the storage volume. First it increases the timeout for tablet creation and creates the storage volume: ```sql -- default is 10 s, too tight for object-store tablet creation ADMIN SET FRONTEND CONFIG ('tablet_create_timeout_second'='60'); CREATE STORAGE VOLUME s3_volume TYPE = S3 LOCATIONS = ("s3://my-starrocks-bucket/") PROPERTIES ( "enabled" = "true", "aws.s3.endpoint" = "minio:9000", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.use_instance_profile" = "false", "aws.s3.use_aws_sdk_default_behavior" = "false" ); ``` It then sets the new storage volume as the default and shows its details: ```sql SET s3_volume AS DEFAULT STORAGE VOLUME; -- REQUIRED before CREATE DATABASE/TABLE DESC STORAGE VOLUME s3_volume\G -- verify: enabled=true, IsDefault=true ``` *** #### More information[​](#more-information "Direct link to More information") * StarRocks shared-data quick start — [shared-data](https://docs.starrocks.io/docs/quick_start/shared-data.md) * StarRocks MCP server — * AIStor MCP server — * Olist dataset — *** #### Credits & license[​](#credits--license "Direct link to Credits & license") * **Olist Brazilian E-Commerce** dataset via Kaggle, licensed **CC BY-NC-SA 4.0 (non-commercial)** — used here for educational demonstration; credit retained to Olist. Confirm terms before any commercial reuse. * StarRocks and the StarRocks MCP server are open source (Apache-2.0 / project license). * MinIO and `mcp-server-aistor` © MinIO, Inc. --- ### Kafka routine load StarRocks using shared-data storage #### About Routine Load[​](#about-routine-load "Direct link to About Routine Load") Routine load is a method using Apache Kafka, or in this lab, Redpanda, to continuously stream data into StarRocks. The data is streamed into a Kafka topic, and a Routine Load job consumes the data into StarRocks. More details on Routine Load are provided at the end of the lab. #### About shared-data[​](#about-shared-data "Direct link to About shared-data") In systems that separate storage from compute, data is stored in low-cost reliable remote storage systems such as Amazon S3, Google Cloud Storage, Azure Blob Storage, and other S3-compatible storage like MinIO. Hot data is cached locally and when the cache is hit, the query performance is comparable to that of storage-compute coupled architecture. Compute nodes (CN) can be added or removed on demand within seconds. This architecture reduces storage costs, ensures better resource isolation, and provides elasticity and scalability. This tutorial covers: * Running StarRocks, Redpanda, and MinIO with Docker Compose * Using MinIO as the StarRocks storage layer * Configuring StarRocks for shared-data * Adding a Routine Load job to consume data from Redpanda The data used is synthetic. There is a lot of information in this document, and it is presented with step-by-step content at the beginning, and the technical details at the end. This is done to serve these purposes in this order: 1. Configure Routine Load. 2. Allow the reader to load data in a shared-data deployment and analyze that data. 3. Provide the configuration details for shared-data deployments. *** #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") ##### Docker[​](#docker "Direct link to Docker") * [Docker](https://docs.docker.com/engine/install/) * 4 GB RAM assigned to Docker * 10 GB free disk space assigned to Docker ##### SQL client[​](#sql-client "Direct link to SQL client") You can use the SQL client provided in the Docker environment, or use one on your system. Many MySQL-compatible clients will work, and this guide covers the configuration of DBeaver and MySQL Workbench. ##### curl[​](#curl "Direct link to curl") `curl` is used to download the Compose file and the script to generate the data. Check to see if you have it installed by running `curl` or `curl.exe` at your OS prompt. If curl is not installed, [get curl here](https://curl.se/). ##### Python[​](#python "Direct link to Python") Python 3 and the Python client for Apache Kafka, `kafka-python`, are required. * [Python](https://www.python.org/) * [`kafka-python`](https://pypi.org/project/kafka-python/) *** #### Terminology[​](#terminology "Direct link to Terminology") ##### FE[​](#fe "Direct link to FE") Frontend nodes are responsible for metadata management, client connection management, query planning, and query scheduling. Each FE stores and maintains a complete copy of metadata in its memory, which guarantees indiscriminate services among the FEs. ##### CN[​](#cn "Direct link to CN") Compute Nodes are responsible for executing query plans in shared-data deployments. ##### BE[​](#be "Direct link to BE") Backend nodes are responsible for both data storage and executing query plans in shared-nothing deployments. note This guide does not use BEs, this information is included here so that you understand the difference between BEs and CNs. *** #### Launch StarRocks[​](#launch-starrocks "Direct link to Launch StarRocks") To run StarRocks with shared-data using Object Storage you need: * A frontend engine (FE) * A compute node (CN) * Object Storage This guide uses MinIO, which is S3 compatible Object Storage provider. MinIO is provided under the GNU Affero General Public License. ##### Download the lab files[​](#download-the-lab-files "Direct link to Download the lab files") ###### `docker-compose.yml`[​](#docker-composeyml "Direct link to docker-composeyml") ```bash mkdir routineload cd routineload curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/routine-load-shared-data/docker-compose.yml ``` ###### `gen.py`[​](#genpy "Direct link to genpy") `gen.py` is a script that uses the Python client for Apache Kafka to publish (produce) data to a Kafka topic. The script has been written with the address and port of the Redpanda container. ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/routine-load-shared-data/gen.py ``` #### Start StarRocks, MinIO, and Redpanda[​](#start-starrocks-minio-and-redpanda "Direct link to Start StarRocks, MinIO, and Redpanda") ```bash docker compose up --detach --wait --wait-timeout 120 ``` Check the progress of the services. It should take 30 seconds or more for the containers to become healthy. The `routineload-minio_mc-1` container will not show a health indicator, and it will exit once it is done configuring MinIO with the access key that StarRocks will use. Wait for `routineload-minio_mc-1` to exit with a `0` code and the rest of the services to be `Healthy`. Run `docker compose ps` until the services are healthy: ```bash docker compose ps ``` ```plaintext WARN[0000] /Users/droscign/routineload/docker-compose.yml: `version` is obsolete [+] Running 6/7 ✔ Network routineload_default Crea... 0.0s ✔ Container minio Healthy 5.6s ✔ Container redpanda Healthy 3.6s ✔ Container redpanda-console Healt... 1.1s ⠧ Container routineload-minio_mc-1 Waiting 23.1s ✔ Container starrocks-fe Healthy 11.1s ✔ Container starrocks-cn Healthy 23.0s container routineload-minio_mc-1 exited (0) ``` *** #### Examine MinIO credentials[​](#examine-minio-credentials "Direct link to Examine MinIO credentials") In order to use MinIO for Object Storage with StarRocks, StarRocks needs a MinIO access key. The access key was generated during the startup of the Docker services. To help you better understand the way that StarRocks connects to MinIO you should verify that the key exists. ##### Open the MinIO web UI[​](#open-the-minio-web-ui "Direct link to Open the MinIO web UI") Browse to The username and password are specified in the Docker compose file, and are `miniouser` and `miniopassword`. You should see that there is one access key. The Key is `AAAAAAAAAAAAAAAAAAAA`, you cannot see the secret in the MinIO Console, but it is in the Docker compose file and is `BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB`: ![View the MinIO access key](/assets/images/MinIO-view-key-9df9610b4c42313f682f7ac6ecb1df07.png) *** ##### Create a bucket for your data[​](#create-a-bucket-for-your-data "Direct link to Create a bucket for your data") When you create a storage volume in StarRocks you will specify the `LOCATION` for the data: ```sh LOCATIONS = ("s3://my-starrocks-bucket/") ``` Open and add a bucket for the storage volume. Name the bucket `my-starrocks-bucket`. Accept the defaults for the three listed options. *** #### SQL Clients[​](#sql-clients "Direct link to SQL Clients") These three clients are tested with this tutorial, you only need one: * mysql CLI: You can run this from the Docker environment or your machine. * [DBeaver](https://dbeaver.io/download/) is available as a community version and a Pro version. * [MySQL Workbench](https://dev.mysql.com/downloads/workbench/) ###### Configuring the client[​](#configuring-the-client "Direct link to Configuring the client") * mysql CLI * DBeaver * MySQL Workbench The easiest way to use the mysql CLI is to run it from the StarRocks container `starrocks-fe`: ```bash docker compose exec starrocks-fe \ mysql -P 9030 -h 127.0.0.1 -u root --prompt="StarRocks > " ``` tip All `docker compose` commands must be run from the directory containing the `docker-compose.yml` file. If you would like to install the mysql CLI expand **mysql client install** below: mysql client install * macOS: If you use Homebrew and do not need MySQL Server run `brew install mysql-client@8.0` to install the CLI. * Linux: Check your repository system for the `mysql` client. For example, `yum install mariadb`. * Microsoft Windows: Install the [MySQL Community Server](https://dev.mysql.com/downloads/mysql/) and run the provided client, or run `mysql` from WSL. - Install [DBeaver](https://dbeaver.io/download/), and add a connection: ![Add a connection](/assets/images/DBeaver-1-58907bd9df44bba1e40282214e4a056c.png) - Configure the port, IP, and username. Test the connection, and click Finish if the test succeeds: ![Configure and test](/assets/images/DBeaver-2-8e4bdc09135b4738944d7016bfb1af94.png) * Install the [MySQL Workbench](https://dev.mysql.com/downloads/workbench/), and add a connection. * Configure the port, IP, and username and then test the connection: ![Connection settings](/assets/images/Workbench-1-9fd3a3126d5f83478e51891287e412b9.png) * You will see warnings from the Workbench as it is checking for a specific MySQL version. You can ignore the warnings and when prompted, you can configure Workbench to stop displaying the warnings: ![Ignore warning](/assets/images/Workbench-2-efba18763e56101091746837ae666b5e.png) *** #### StarRocks configuration for shared-data[​](#starrocks-configuration-for-shared-data "Direct link to StarRocks configuration for shared-data") At this point you have StarRocks running, and you have MinIO running. The MinIO access key is used to connect StarRocks and Minio. This is the part of the `FE` configuration that specifies that the StarRocks deployment will use shared data. This was added to the file `fe.conf` when Docker Compose created the deployment. ```sh # enable the shared data run mode run_mode = shared_data cloud_native_storage_type = S3 ``` info You can verify these settings by running this command from the `quickstart` directory and looking at the end of the file: ```sh docker compose exec starrocks-fe \ cat /opt/starrocks/fe/conf/fe.conf ``` ::: ##### Connect to StarRocks with a SQL client[​](#connect-to-starrocks-with-a-sql-client "Direct link to Connect to StarRocks with a SQL client") tip Run this command from the directory containing the `docker-compose.yml` file. If you are using a client other than the mysql CLI, open that now. ```sql docker compose exec starrocks-fe \ mysql -P9030 -h127.0.0.1 -uroot --prompt="StarRocks > " ``` ###### Examine the storage volumes[​](#examine-the-storage-volumes "Direct link to Examine the storage volumes") ```sql SHOW STORAGE VOLUMES; ``` tip There should be no storage volumes, you will create one next. ```sh Empty set (0.04 sec) ``` ###### Create a shared-data storage volume[​](#create-a-shared-data-storage-volume "Direct link to Create a shared-data storage volume") Earlier you created a bucket in MinIO named `my-starrocks-volume`, and you verified that MinIO has an access key named `AAAAAAAAAAAAAAAAAAAA`. The following SQL will create a storage volume in the MionIO bucket using the access key and secret. ```sql CREATE STORAGE VOLUME s3_volume TYPE = S3 LOCATIONS = ("s3://my-starrocks-bucket/") PROPERTIES ( "enabled" = "true", "aws.s3.endpoint" = "minio:9000", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.use_instance_profile" = "false", "aws.s3.use_aws_sdk_default_behavior" = "false" ); ``` Now you should see a storage volume listed, earlier it was an empty set: ```text SHOW STORAGE VOLUMES; ``` ```text +----------------+ | Storage Volume | +----------------+ | s3_volume | +----------------+ 1 row in set (0.02 sec) ``` View the details of the storage volume and note that this is nott yet the default volume, and that it is configured to use your bucket: ```text DESC STORAGE VOLUME s3_volume\G ``` tip Some of the SQL in this document, and many other documents in the StarRocks documentation, and with `\G` instead of a semicolon. The `\G` causes the mysql CLI to render the query results vertically. Many SQL clients do not interpret vertical formatting output, so you should replace `\G` with `;`. ```sh *************************** 1. row *************************** Name: s3_volume Type: S3 IsDefault: false Location: s3://my-starrocks-bucket/ Params: {"aws.s3.access_key":"******","aws.s3.secret_key":"******","aws.s3.endpoint":"minio:9000","aws.s3.region":"us-east-1","aws.s3.use_instance_profile":"false","aws.s3.use_web_identity_token_file":"false","aws.s3.use_aws_sdk_default_behavior":"false"} Enabled: true Comment: 1 row in set (0.02 sec) ``` #### Set the default storage volume[​](#set-the-default-storage-volume "Direct link to Set the default storage volume") ```text SET s3_volume AS DEFAULT STORAGE VOLUME; ``` ```text DESC STORAGE VOLUME s3_volume\G ``` ```sh *************************** 1. row *************************** Name: s3_volume Type: S3 IsDefault: true Location: s3://my-starrocks-bucket/ Params: {"aws.s3.access_key":"******","aws.s3.secret_key":"******","aws.s3.endpoint":"minio:9000","aws.s3.region":"us-east-1","aws.s3.use_instance_profile":"false","aws.s3.use_web_identity_token_file":"false","aws.s3.use_aws_sdk_default_behavior":"false"} Enabled: true Comment: 1 row in set (0.02 sec) ``` *** #### Create a table[​](#create-a-table "Direct link to Create a table") These SQL commands are run in your SQL client. ```sql CREATE DATABASE IF NOT EXISTS quickstart; ``` Verify that the database `quickstart` is using the storage volume `s3_volume`: ```text SHOW CREATE DATABASE quickstart \G ``` ```sh *************************** 1. row *************************** Database: quickstart Create Database: CREATE DATABASE `quickstart` PROPERTIES ("storage_volume" = "s3_volume") ``` ```sql USE quickstart; ``` ```sql CREATE TABLE site_clicks ( `uid` bigint NOT NULL COMMENT "uid", `site` string NOT NULL COMMENT "site url", `vtime` bigint NOT NULL COMMENT "vtime" ) DISTRIBUTED BY HASH(`uid`) PROPERTIES("replication_num"="1"); ``` *** ##### Open the Redpanda Console[​](#open-the-redpanda-console "Direct link to Open the Redpanda Console") There will be no topics yet, a topic will be created in the next step. ##### Publish data to a Redpanda topic[​](#publish-data-to-a-redpanda-topic "Direct link to Publish data to a Redpanda topic") From a command shell in the `routineload/` folder run this command to generate data: ```python python gen.py 5 ``` tip On your system, you might need to use `python3` in place of `python` in the command. If you are missing `kafka-python` try: ```text pip install kafka-python ``` or ```text pip3 install kafka-python ``` ```plaintext b'{ "uid": 6926, "site": "https://docs.starrocks.io/", "vtime": 1718034793 } ' b'{ "uid": 3303, "site": "https://www.starrocks.io/product/community", "vtime": 1718034793 } ' b'{ "uid": 227, "site": "https://docs.starrocks.io/", "vtime": 1718034243 } ' b'{ "uid": 7273, "site": "https://docs.starrocks.io/", "vtime": 1718034794 } ' b'{ "uid": 4666, "site": "https://www.starrocks.io/", "vtime": 1718034794 } ' ``` ##### Verify in the Redpanda Console[​](#verify-in-the-redpanda-console "Direct link to Verify in the Redpanda Console") Navigate to in the Redpanda Console, and you will see one topic named `test2`. Select that topic and then the **Messages** tab and you will see five messages matching the output of `gen.py`. #### Consume the messages[​](#consume-the-messages "Direct link to Consume the messages") In StarRocks you will create a Routine Load job to: 1. Consume the messages from the Redpanda topic `test2` 2. Load those messages into the table `site_clicks` StarRocks is configured to use MinIO for storage, so the data inserted into the `site_clicks` table will be stored in MinIO. ##### Create a Routine Load job[​](#create-a-routine-load-job "Direct link to Create a Routine Load job") Run this command in the SQL client to create the Routine Load job, the command will be explained in detail at the end of the lab. ```sql CREATE ROUTINE LOAD quickstart.clicks ON site_clicks PROPERTIES ( "format" = "JSON", "jsonpaths" ="[\"$.uid\",\"$.site\",\"$.vtime\"]" ) FROM KAFKA ( "kafka_broker_list" = "redpanda:29092", "kafka_topic" = "test2", "kafka_partitions" = "0", "kafka_offsets" = "OFFSET_BEGINNING" ); ``` ##### Verify the Routine Load job[​](#verify-the-routine-load-job "Direct link to Verify the Routine Load job") ```sql SHOW ROUTINE LOAD\G ``` Verify the three highlighted lines: 1. The state should be `RUNNING` 2. The topic should be `test2` and the broker should be `redpanda:2092` 3. The statistics should show either 0 or 5 loaded rows depending on how soon you ran the `SHOW ROUTINE LOAD` command. If there are 0 loaded rows run it again. ```sql *************************** 1. row *************************** Id: 10078 Name: clicks CreateTime: 2024-06-12 15:51:12 PauseTime: NULL EndTime: NULL DbName: quickstart TableName: site_clicks State: RUNNING DataSourceType: KAFKA CurrentTaskNum: 1 JobProperties: {"partitions":"*","partial_update":"false","columnToColumnExpr":"*","maxBatchIntervalS":"10","partial_update_mode":"null","whereExpr":"*","dataFormat":"json","timezone":"Etc/UTC","format":"json","log_rejected_record_num":"0","taskTimeoutSecond":"60","json_root":"","maxFilterRatio":"1.0","strict_mode":"false","jsonpaths":"[\"$.uid\",\"$.site\",\"$.vtime\"]","taskConsumeSecond":"15","desireTaskConcurrentNum":"5","maxErrorNum":"0","strip_outer_array":"false","currentTaskConcurrentNum":"1","maxBatchRows":"200000"} DataSourceProperties: {"topic":"test2","currentKafkaPartitions":"0","brokerList":"redpanda:29092"} CustomProperties: {"group.id":"clicks_ea38a713-5a0f-4abe-9b11-ff4a241ccbbd"} Statistic: {"receivedBytes":0,"errorRows":0,"committedTaskNum":0,"loadedRows":0,"loadRowsRate":0,"abortedTaskNum":0,"totalRows":0,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":1} Progress: {"0":"OFFSET_ZERO"} TimestampProgress: {} ReasonOfStateChanged: ErrorLogUrls: TrackingSQL: OtherMsg: LatestSourcePosition: {} 1 row in set (0.00 sec) ``` ```sql SHOW ROUTINE LOAD\G ``` ```sql *************************** 1. row *************************** Id: 10076 Name: clicks CreateTime: 2024-06-12 18:40:53 PauseTime: NULL EndTime: NULL DbName: quickstart TableName: site_clicks State: RUNNING DataSourceType: KAFKA CurrentTaskNum: 1 JobProperties: {"partitions":"*","partial_update":"false","columnToColumnExpr":"*","maxBatchIntervalS":"10","partial_update_mode":"null","whereExpr":"*","dataFormat":"json","timezone":"Etc/UTC","format":"json","log_rejected_record_num":"0","taskTimeoutSecond":"60","json_root":"","maxFilterRatio":"1.0","strict_mode":"false","jsonpaths":"[\"$.uid\",\"$.site\",\"$.vtime\"]","taskConsumeSecond":"15","desireTaskConcurrentNum":"5","maxErrorNum":"0","strip_outer_array":"false","currentTaskConcurrentNum":"1","maxBatchRows":"200000"} DataSourceProperties: {"topic":"test2","currentKafkaPartitions":"0","brokerList":"redpanda:29092"} CustomProperties: {"group.id":"clicks_a9426fee-45bb-403a-a1a3-b3bc6c7aa685"} Statistic: {"receivedBytes":372,"errorRows":0,"committedTaskNum":1,"loadedRows":5,"loadRowsRate":0,"abortedTaskNum":0,"totalRows":5,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":519} Progress: {"0":"4"} TimestampProgress: {"0":"1718217035111"} ReasonOfStateChanged: ErrorLogUrls: TrackingSQL: OtherMsg: LatestSourcePosition: {"0":"5"} 1 row in set (0.00 sec) ``` *** #### Verify that data is stored in MinIO[​](#verify-that-data-is-stored-in-minio "Direct link to Verify that data is stored in MinIO") Open MinIO and verify that there are objects stored under `my-starrocks-bucket`. *** #### Query the data from StarRocks[​](#query-the-data-from-starrocks "Direct link to Query the data from StarRocks") ```sql USE quickstart; SELECT * FROM site_clicks; ``` ```sql +------+--------------------------------------------+------------+ | uid | site | vtime | +------+--------------------------------------------+------------+ | 4607 | https://www.starrocks.io/blog | 1718031441 | | 1575 | https://www.starrocks.io/ | 1718031523 | | 2398 | https://docs.starrocks.io/ | 1718033630 | | 3741 | https://www.starrocks.io/product/community | 1718030845 | | 4792 | https://www.starrocks.io/ | 1718033413 | +------+--------------------------------------------+------------+ 5 rows in set (0.07 sec) ``` #### Publish additional data[​](#publish-additional-data "Direct link to Publish additional data") Running `gen.py` again will publish another five records to Redpanda. ```bash python gen.py 5 ``` ##### Verify that data is added[​](#verify-that-data-is-added "Direct link to Verify that data is added") Since the Routine Load job runs on a schedule (every 10 seconds by default), the data will be loaded within a few seconds. ```sql SELECT * FROM site_clicks; ``` ```text +------+--------------------------------------------+------------+ | uid | site | vtime | +------+--------------------------------------------+------------+ | 6648 | https://www.starrocks.io/blog | 1718205970 | | 7914 | https://www.starrocks.io/ | 1718206760 | | 9854 | https://www.starrocks.io/blog | 1718205676 | | 1186 | https://www.starrocks.io/ | 1718209083 | | 3305 | https://docs.starrocks.io/ | 1718209083 | | 2288 | https://www.starrocks.io/blog | 1718206759 | | 7879 | https://www.starrocks.io/product/community | 1718204280 | | 2666 | https://www.starrocks.io/ | 1718208842 | | 5801 | https://www.starrocks.io/ | 1718208783 | | 8409 | https://www.starrocks.io/ | 1718206889 | +------+--------------------------------------------+------------+ 10 rows in set (0.02 sec) ``` *** #### Configuration details[​](#configuration-details "Direct link to Configuration details") Now that you have experienced using StarRocks with shared-data it is important to understand the configuration. ##### CN configuration[​](#cn-configuration "Direct link to CN configuration") The CN configuration used here is the default, as the CN is designed for shared-data use. The default configuration is shown below. You do not need to make any changes. ```bash sys_log_level = INFO # ports for admin, web, heartbeat service be_port = 9060 be_http_port = 8040 heartbeat_service_port = 9050 brpc_port = 8060 starlet_port = 9070 ``` ##### FE configuration[​](#fe-configuration "Direct link to FE configuration") The FE configuration is slightly different from the default as the FE must be configured to expect that data is stored in Object Storage rather than on local disks on BE nodes. The `docker-compose.yml` file generates the FE configuration in the `command`. ```plaintext # enable shared data, set storage type, set endpoint run_mode = shared_data cloud_native_storage_type = S3 ``` note This config file does not contain the default entries for an FE, only the shared-data configuration is shown. The non-default FE configuration settings: note Many configuration parameters are prefixed with `s3_`. This prefix is used for all Amazon S3 compatible storage types (for example: S3, GCS, and MinIO). When using Azure Blob Storage the prefix is `azure_`. ###### `run_mode=shared_data`[​](#run_modeshared_data "Direct link to run_modeshared_data") This enables shared-data use. ###### `cloud_native_storage_type=S3`[​](#cloud_native_storage_types3 "Direct link to cloud_native_storage_types3") This specifies whether S3 compatible storage or Azure Blob Storage is used. For MinIO this is always S3. ##### Details of `CREATE storage volume`[​](#details-of-create-storage-volume "Direct link to details-of-create-storage-volume") ```sql CREATE STORAGE VOLUME s3_volume TYPE = S3 LOCATIONS = ("s3://my-starrocks-bucket/") PROPERTIES ( "enabled" = "true", "aws.s3.endpoint" = "minio:9000", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.use_instance_profile" = "false", "aws.s3.use_aws_sdk_default_behavior" = "false" ); ``` ###### `aws_s3_endpoint=minio:9000`[​](#aws_s3_endpointminio9000 "Direct link to aws_s3_endpointminio9000") The MinIO endpoint, including port number. ###### `aws_s3_path=starrocks`[​](#aws_s3_pathstarrocks "Direct link to aws_s3_pathstarrocks") The bucket name. ###### `aws_s3_access_key=AAAAAAAAAAAAAAAAAAAA`[​](#aws_s3_access_keyaaaaaaaaaaaaaaaaaaaa "Direct link to aws_s3_access_keyaaaaaaaaaaaaaaaaaaaa") The MinIO access key. ###### `aws_s3_secret_key=BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB`[​](#aws_s3_secret_keybbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb "Direct link to aws_s3_secret_keybbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb") The MinIO access key secret. ###### `aws_s3_use_instance_profile=false`[​](#aws_s3_use_instance_profilefalse "Direct link to aws_s3_use_instance_profilefalse") When using MinIO an access key is used, and so instance profiles are not used with MinIO. ###### `aws_s3_use_aws_sdk_default_behavior=false`[​](#aws_s3_use_aws_sdk_default_behaviorfalse "Direct link to aws_s3_use_aws_sdk_default_behaviorfalse") When using MinIO this parameter is always set to false. *** #### Notes on the Routine Load command[​](#notes-on-the-routine-load-command "Direct link to Notes on the Routine Load command") StarRocks Routine Load takes many arguments. Only the ones used in this tutorial are described here, the rest will be linked to in the more information section. ```sql CREATE ROUTINE LOAD quickstart.clicks ON site_clicks PROPERTIES ( "format" = "JSON", "jsonpaths" ="[\"$.uid\",\"$.site\",\"$.vtime\"]" ) FROM KAFKA ( "kafka_broker_list" = "redpanda:29092", "kafka_topic" = "test2", "kafka_partitions" = "0", "kafka_offsets" = "OFFSET_BEGINNING" ); ``` ##### Parameters[​](#parameters "Direct link to Parameters") ```text CREATE ROUTINE LOAD quickstart.clicks ON site_clicks ``` The parameters for `CREATE ROUTINE LOAD ON` are: * database\_name.job\_name * table\_name `database_name` is optional. In this lab, it is `quickstart` and is specified. `job_name` is required, and is `clicks` `table_name` is required, and is `site_clicks` ##### Job properties[​](#job-properties "Direct link to Job properties") ###### Property `format`[​](#property-format "Direct link to property-format") ```text "format" = "JSON", ``` In this case, the data is in JSON format, so the property is set to `JSON`. The other valid formats are: `CSV`, `JSON`, and `Avro`. `CSV` is the default. ###### Property `jsonpaths`[​](#property-jsonpaths "Direct link to property-jsonpaths") ```text "jsonpaths" ="[\"$.uid\",\"$.site\",\"$.vtime\"]" ``` The names of the fields that you want to load from JSON-formatted data. The value of this parameter is a valid JsonPath expression. More information is available at the end of this page. ##### Data source properties[​](#data-source-properties "Direct link to Data source properties") ###### `kafka_broker_list`[​](#kafka_broker_list "Direct link to kafka_broker_list") ```text "kafka_broker_list" = "redpanda:29092", ``` Kafka's broker connection information. The format is `:`. Multiple brokers are separated by commas. ###### `kafka_topic`[​](#kafka_topic "Direct link to kafka_topic") ```text "kafka_topic" = "test2", ``` The Kafka topic to consume from. ###### `kafka_partitions` and `kafka_offsets`[​](#kafka_partitions-and-kafka_offsets "Direct link to kafka_partitions-and-kafka_offsets") ```text "kafka_partitions" = "0", "kafka_offsets" = "OFFSET_BEGINNING" ``` These properties are presented together as there is one `kafka_offset` required for each `kafka_partitions` entry. `kafka_partitions` is a list of one or more partitions to consume. If this property is not set, then all partitions are consumed. `kafka_offsets` is a list of offsets, one for each partition listed in `kafka_partitions`. In this case the value is `OFFSET_BEGINNING` which causes all of the data to be consumed. The default is to only consume new data. *** #### Summary[​](#summary "Direct link to Summary") In this tutorial you: * Deployed StarRocks, Reedpanda, and Minio in Docker * Created a Routine Load job to consume data from a Kafka topic * Learned how to configure a StarRocks Storage Volume that uses MinIO #### More information[​](#more-information "Direct link to More information") [StarRocks Architecture](https://docs.starrocks.io/docs/introduction/Architecture.md) The sample used for this lab is very simple. Routine Load has many more options and capabilities. [learn more](https://docs.starrocks.io/docs/loading/kafka/RoutineLoad.md). [JSONPath](https://goessner.net/articles/JsonPath/) --- ### Separate storage and compute In systems that separate storage from compute data is stored in low-cost reliable remote storage systems such as Amazon S3, Google Cloud Storage, Azure Blob Storage, and other S3-compatible storage like MinIO. Hot data is cached locally and When the cache is hit, the query performance is comparable to that of storage-compute coupled architecture. Compute nodes (CN) can be added or removed on demand within seconds. This architecture reduces storage cost, ensures better resource isolation, and provides elasticity and scalability. This tutorial covers: * Running StarRocks in Docker containers * Using MinIO for Object Storage * Configuring StarRocks for shared-data * Loading two public datasets * Analyzing the data with SELECT and JOIN * Basic data transformation (the **T** in ETL) The data used is provided by NYC OpenData and the National Centers for Environmental Information at NOAA. Both of these datasets are very large, and because this tutorial is intended to help you get exposed to working with StarRocks we are not going to load data for the past 120 years. You can run the Docker image and load this data on a machine with 4 GB RAM assigned to Docker. For larger fault-tolerant and scalable deployments we have other documentation and will provide that later. There is a lot of information in this document, and it is presented with the step by step content at the beginning, and the technical details at the end. This is done to serve these purposes in this order: 1. Allow the reader to load data in a shared-data deployment and analyze that data. 2. Provide the configuration details for shared-data deployments. 3. Explain the basics of data transformation during loading. *** #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") ##### Docker[​](#docker "Direct link to Docker") * [Docker](https://docs.docker.com/engine/install/) * 4 GB RAM assigned to Docker * 10 GB free disk space assigned to Docker ##### SQL client[​](#sql-client "Direct link to SQL client") You can use the SQL client provided in the Docker environment, or use one on your system. Many MySQL compatible clients will work, and this guide covers the configuration of DBeaver and MySQL Workbench. ##### curl[​](#curl "Direct link to curl") `curl` is used to issue the data load job to StarRocks, and to download the datasets. Check to see if you have it installed by running `curl` or `curl.exe` at your OS prompt. If curl is not installed, [get curl here](https://curl.se/). ##### `/etc/hosts`[​](#etchosts "Direct link to etchosts") The ingest method used in this guide is Stream Load. Stream Load connects to the FE service to start the ingest job. The FE then assigns the job to a backend node, the CN in this guide. In order for the ingest job to connect to the CN the name of the CN must be available to your operating system. Add this line to `/etc/hosts`: ```bash 127.0.0.1 starrocks-cn ``` *** #### Terminology[​](#terminology "Direct link to Terminology") ##### FE[​](#fe "Direct link to FE") Frontend nodes are responsible for metadata management, client connection management, query planning, and query scheduling. Each FE stores and maintains a complete copy of metadata in its memory, which guarantees indiscriminate services among the FEs. ##### CN[​](#cn "Direct link to CN") Compute Nodes are responsible for executing query plans in shared-data deployments. ##### BE[​](#be "Direct link to BE") Backend nodes are responsible for both data storage and executing query plans in shared-nothing deployments. note This guide does not use BEs, this information is included here so that you understand the difference between BEs and CNs. *** #### Edit your hosts file[​](#edit-your-hosts-file "Direct link to Edit your hosts file") The ingest method used in this guide is Stream Load. Stream Load connects to the FE service to start the ingest job. The FE then assigns the job to a backend node—the CN in this guide. In order for the ingest job to connect to the CN, the name of the CN must be available to your operating system. Add this line to `/etc/hosts`: ```bash 127.0.0.1 starrocks-cn ``` #### Download the lab files[​](#download-the-lab-files "Direct link to Download the lab files") There are three files to download: * The Docker Compose file that deploys the StarRocks and MinIO environment * New York City crash data * Weather data This guide uses MinIO, which is S3 compatible Object Storage provided under the GNU Affero General Public License. ##### Create a directory to store the lab files[​](#create-a-directory-to-store-the-lab-files "Direct link to Create a directory to store the lab files") ```bash mkdir quickstart cd quickstart ``` ##### Download the Docker Compose file[​](#download-the-docker-compose-file "Direct link to Download the Docker Compose file") ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/docker-compose.yml ``` ##### Download the data[​](#download-the-data "Direct link to Download the data") Download these two datasets: ###### New York City crash data[​](#new-york-city-crash-data "Direct link to New York City crash data") ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/datasets/NYPD_Crash_Data.csv ``` ###### Weather data[​](#weather-data "Direct link to Weather data") ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/datasets/72505394728.csv ``` *** #### Deploy StarRocks and MinIO[​](#deploy-starrocks-and-minio "Direct link to Deploy StarRocks and MinIO") ```bash docker compose up --detach --wait --wait-timeout 120 ``` It should take around 30 seconds for the FE, CN, and MinIO services to become healthy. The `quickstart-minio_mc-1` container will show a status of `Waiting` and also an exit code. An exit code of `0` indicates success. ```bash [+] Running 4/5 ✔ Network quickstart_default Created 0.0s ✔ Container minio Healthy 6.8s ✔ Container starrocks-fe Healthy 29.3s ⠼ Container quickstart-minio_mc-1 Waiting 29.3s ✔ Container starrocks-cn Healthy 29.2s container quickstart-minio_mc-1 exited (0) ``` *** #### MinIO[​](#minio "Direct link to MinIO") This quick start uses MinIO for shared storage. ##### Verify the MinIO credentials[​](#verify-the-minio-credentials "Direct link to Verify the MinIO credentials") To use MinIO for Object Storage with StarRocks, StarRocks needs a MinIO access key. The access key was generated during the startup of the Docker services. To help you better understand the way that StarRocks connects to MinIO you should verify that the key exists. Browse to The username and password are specified in the Docker compose file, and are `miniouser` and `miniopassword`. You should see that there is one access key. The Key is `AAAAAAAAAAAAAAAAAAAA`, you cannot see the secret in the MinIO Console, but it is in the Docker compose file and is `BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB`: ![View the MinIO access key](/assets/images/MinIO-view-key-9df9610b4c42313f682f7ac6ecb1df07.png) tip If there are no access keys showing in the MinIO web UI, check the logs of the `minio_mc` service: ```bash docker compose logs minio_mc ``` Try rerunning the `minio_mc` pod: ```bash docker compose run minio_mc ``` ##### Create a bucket for your data[​](#create-a-bucket-for-your-data "Direct link to Create a bucket for your data") When you create a storage volume in StarRocks you will specify the `LOCATION` for the data: ```sh LOCATIONS = ("s3://my-starrocks-bucket/") ``` Open and add a bucket for the storage volume. Name the bucket `my-starrocks-bucket`. Accept the defaults for the three listed options. *** #### SQL Clients[​](#sql-clients "Direct link to SQL Clients") These three clients are tested with this tutorial, you only need one: * mysql CLI: You can run this from the Docker environment or your machine. * [DBeaver](https://dbeaver.io/download/) is available as a community version and a Pro version. * [MySQL Workbench](https://dev.mysql.com/downloads/workbench/) ###### Configuring the client[​](#configuring-the-client "Direct link to Configuring the client") * mysql CLI * DBeaver * MySQL Workbench The easiest way to use the mysql CLI is to run it from the StarRocks container `starrocks-fe`: ```bash docker compose exec starrocks-fe \ mysql -P 9030 -h 127.0.0.1 -u root --prompt="StarRocks > " ``` tip All `docker compose` commands must be run from the directory containing the `docker-compose.yml` file. If you would like to install the mysql CLI expand **mysql client install** below: mysql client install * macOS: If you use Homebrew and do not need MySQL Server run `brew install mysql-client@8.0` to install the CLI. * Linux: Check your repository system for the `mysql` client. For example, `yum install mariadb`. * Microsoft Windows: Install the [MySQL Community Server](https://dev.mysql.com/downloads/mysql/) and run the provided client, or run `mysql` from WSL. - Install [DBeaver](https://dbeaver.io/download/), and add a connection: ![Add a connection](/assets/images/DBeaver-1-58907bd9df44bba1e40282214e4a056c.png) - Configure the port, IP, and username. Test the connection, and click Finish if the test succeeds: ![Configure and test](/assets/images/DBeaver-2-8e4bdc09135b4738944d7016bfb1af94.png) * Install the [MySQL Workbench](https://dev.mysql.com/downloads/workbench/), and add a connection. * Configure the port, IP, and username and then test the connection: ![Connection settings](/assets/images/Workbench-1-9fd3a3126d5f83478e51891287e412b9.png) * You will see warnings from the Workbench as it is checking for a specific MySQL version. You can ignore the warnings and when prompted, you can configure Workbench to stop displaying the warnings: ![Ignore warning](/assets/images/Workbench-2-efba18763e56101091746837ae666b5e.png) *** #### StarRocks configuration for shared-data[​](#starrocks-configuration-for-shared-data "Direct link to StarRocks configuration for shared-data") At this point you have StarRocks running, and you have MinIO running. The MinIO access key is used to connect StarRocks and Minio. This is the part of the `FE` configuration that specifies that the StarRocks deployment will use shared data. This was added to the file `fe.conf` when Docker Compose created the deployment. ```sh # enable the shared data run mode run_mode = shared_data cloud_native_storage_type = S3 ``` info You can verify these settings by running this command from the `quickstart` directory and looking at the end of the file: ```sh docker compose exec starrocks-fe \ cat /opt/starrocks/fe/conf/fe.conf ``` ::: ##### Connect to StarRocks with a SQL client[​](#connect-to-starrocks-with-a-sql-client "Direct link to Connect to StarRocks with a SQL client") tip Run this command from the directory containing the `docker-compose.yml` file. If you are using a client other than the MySQL Command-Line Client, open that now. ```sql docker compose exec starrocks-fe \ mysql -P9030 -h127.0.0.1 -uroot --prompt="StarRocks > " ``` ###### Examine the storage volumes[​](#examine-the-storage-volumes "Direct link to Examine the storage volumes") ```sql SHOW STORAGE VOLUMES; ``` tip There should be no storage volumes, you will create one next. ```sh Empty set (0.04 sec) ``` ###### Create a shared-data storage volume[​](#create-a-shared-data-storage-volume "Direct link to Create a shared-data storage volume") Earlier you created a bucket in MinIO named `my-starrocks-volume`, and you verified that MinIO has an access key named `AAAAAAAAAAAAAAAAAAAA`. The following SQL will create a storage volume in the MionIO bucket using the access key and secret. ```sql CREATE STORAGE VOLUME s3_volume TYPE = S3 LOCATIONS = ("s3://my-starrocks-bucket/") PROPERTIES ( "enabled" = "true", "aws.s3.endpoint" = "minio:9000", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.use_instance_profile" = "false", "aws.s3.use_aws_sdk_default_behavior" = "false" ); ``` Now you should see a storage volume listed, earlier it was an empty set: ```text SHOW STORAGE VOLUMES; ``` ```text +----------------+ | Storage Volume | +----------------+ | s3_volume | +----------------+ 1 row in set (0.02 sec) ``` View the details of the storage volume and note that this is nott yet the default volume, and that it is configured to use your bucket: ```text DESC STORAGE VOLUME s3_volume\G ``` tip Some of the SQL in this document, and many other documents in the StarRocks documentation, and with `\G` instead of a semicolon. The `\G` causes the mysql CLI to render the query results vertically. Many SQL clients do not interpret vertical formatting output, so you should replace `\G` with `;`. ```sh *************************** 1. row *************************** Name: s3_volume Type: S3 IsDefault: false Location: s3://my-starrocks-bucket/ Params: {"aws.s3.access_key":"******","aws.s3.secret_key":"******","aws.s3.endpoint":"minio:9000","aws.s3.region":"us-east-1","aws.s3.use_instance_profile":"false","aws.s3.use_web_identity_token_file":"false","aws.s3.use_aws_sdk_default_behavior":"false"} Enabled: true Comment: 1 row in set (0.02 sec) ``` #### Set the default storage volume[​](#set-the-default-storage-volume "Direct link to Set the default storage volume") ```text SET s3_volume AS DEFAULT STORAGE VOLUME; ``` ```text DESC STORAGE VOLUME s3_volume\G ``` ```sh *************************** 1. row *************************** Name: s3_volume Type: S3 IsDefault: true Location: s3://my-starrocks-bucket/ Params: {"aws.s3.access_key":"******","aws.s3.secret_key":"******","aws.s3.endpoint":"minio:9000","aws.s3.region":"us-east-1","aws.s3.use_instance_profile":"false","aws.s3.use_web_identity_token_file":"false","aws.s3.use_aws_sdk_default_behavior":"false"} Enabled: true Comment: 1 row in set (0.02 sec) ``` #### Create a database[​](#create-a-database "Direct link to Create a database") ```text CREATE DATABASE IF NOT EXISTS quickstart; ``` Verify that the database `quickstart` is using the storage volume `s3_volume`: ```text SHOW CREATE DATABASE quickstart \G ``` ```sh *************************** 1. row *************************** Database: quickstart Create Database: CREATE DATABASE `quickstart` PROPERTIES ("storage_volume" = "s3_volume") ``` *** #### Create some tables[​](#create-some-tables "Direct link to Create some tables") ##### Create a database[​](#create-a-database "Direct link to Create a database") Type these two lines in at the `StarRocks > `prompt and press enter after each: ```sql CREATE DATABASE IF NOT EXISTS quickstart; USE quickstart; ``` ##### Create two tables[​](#create-two-tables "Direct link to Create two tables") ###### Crashdata[​](#crashdata "Direct link to Crashdata") The crash dataset contains many more fields than these, the schema has been trimmed down to include only the fields that might be useful to answer questions about the impact weather has on driving conditions. ```sql CREATE TABLE IF NOT EXISTS crashdata ( CRASH_DATE DATETIME, BOROUGH STRING, ZIP_CODE STRING, LATITUDE INT, LONGITUDE INT, LOCATION STRING, ON_STREET_NAME STRING, CROSS_STREET_NAME STRING, OFF_STREET_NAME STRING, CONTRIBUTING_FACTOR_VEHICLE_1 STRING, CONTRIBUTING_FACTOR_VEHICLE_2 STRING, COLLISION_ID INT, VEHICLE_TYPE_CODE_1 STRING, VEHICLE_TYPE_CODE_2 STRING ); ``` ###### Weatherdata[​](#weatherdata "Direct link to Weatherdata") Similar to the crash data, the weather dataset has many more columns (a total of 125 columns) and only the ones that are expected to answer the questions are included in the database. ```sql CREATE TABLE IF NOT EXISTS weatherdata ( DATE DATETIME, NAME STRING, HourlyDewPointTemperature STRING, HourlyDryBulbTemperature STRING, HourlyPrecipitation STRING, HourlyPresentWeatherType STRING, HourlyPressureChange STRING, HourlyPressureTendency STRING, HourlyRelativeHumidity STRING, HourlySkyConditions STRING, HourlyVisibility STRING, HourlyWetBulbTemperature STRING, HourlyWindDirection STRING, HourlyWindGustSpeed STRING, HourlyWindSpeed STRING ); ``` *** #### Load two datasets[​](#load-two-datasets "Direct link to Load two datasets") There are many ways to load data into StarRocks. For this tutorial the simplest way is to use curl and StarRocks Stream Load. tip Run these curl commands from the directory where you downloaded the dataset. You will be prompted for a password. You probably have not assigned a password to the MySQL `root` user, so just hit enter. The `curl` commands look complex, but they are explained in detail at the end of the tutorial. For now, we recommend running the commands and running some SQL to analyze the data, and then reading about the data loading details at the end. ##### New York City collision data - Crashes[​](#new-york-city-collision-data---crashes "Direct link to New York City collision data - Crashes") ```bash curl --location-trusted -u root \ -T ./NYPD_Crash_Data.csv \ -H "label:crashdata-0" \ -H "column_separator:," \ -H "skip_header:1" \ -H "enclose:\"" \ -H "max_filter_ratio:1" \ -H "columns:tmp_CRASH_DATE, tmp_CRASH_TIME, CRASH_DATE=str_to_date(concat_ws(' ', tmp_CRASH_DATE, tmp_CRASH_TIME), '%m/%d/%Y %H:%i'),BOROUGH,ZIP_CODE,LATITUDE,LONGITUDE,LOCATION,ON_STREET_NAME,CROSS_STREET_NAME,OFF_STREET_NAME,NUMBER_OF_PERSONS_INJURED,NUMBER_OF_PERSONS_KILLED,NUMBER_OF_PEDESTRIANS_INJURED,NUMBER_OF_PEDESTRIANS_KILLED,NUMBER_OF_CYCLIST_INJURED,NUMBER_OF_CYCLIST_KILLED,NUMBER_OF_MOTORIST_INJURED,NUMBER_OF_MOTORIST_KILLED,CONTRIBUTING_FACTOR_VEHICLE_1,CONTRIBUTING_FACTOR_VEHICLE_2,CONTRIBUTING_FACTOR_VEHICLE_3,CONTRIBUTING_FACTOR_VEHICLE_4,CONTRIBUTING_FACTOR_VEHICLE_5,COLLISION_ID,VEHICLE_TYPE_CODE_1,VEHICLE_TYPE_CODE_2,VEHICLE_TYPE_CODE_3,VEHICLE_TYPE_CODE_4,VEHICLE_TYPE_CODE_5" \ -XPUT http://localhost:8030/api/quickstart/crashdata/_stream_load ``` Here is the output of the above command. The first highlighted section shown what you should expect to see (OK and all but one row inserted). One row was filtered out because it does not contain the correct number of columns. ```bash Enter host password for user 'root': { "TxnId": 2, "Label": "crashdata-0", "Status": "Success", "Message": "OK", "NumberTotalRows": 423726, "NumberLoadedRows": 423725, "NumberFilteredRows": 1, "NumberUnselectedRows": 0, "LoadBytes": 96227746, "LoadTimeMs": 1013, "BeginTxnTimeMs": 21, "StreamLoadPlanTimeMs": 63, "ReadDataTimeMs": 563, "WriteDataTimeMs": 870, "CommitAndPublishTimeMs": 57, "ErrorURL": "http://starrocks-cn:8040/api/_load_error_log?file=error_log_da41dd88276a7bfc_739087c94262ae9f" }% ``` If there was an error the output provides a URL to see the error messages. The error message also contains the backend node that the Stream Load job was assigned to (`starrocks-cn`). Because you added an entry for `starrocks-cn` to the `/etc/hosts` file, you should be able to navigate to it and read the error message. Expand the summary for the content seen while developing this tutorial: Reading error messages in the browser ```bash Error: Value count does not match column count. Expect 29, but got 32. Column delimiter: 44,Row delimiter: 10.. Row: 09/06/2015,14:15,,,40.6722269,-74.0110059,"(40.6722269, -74.0110059)",,,"R/O 1 BEARD ST. ( IKEA'S 09/14/2015,5:30,BRONX,10473,40.814551,-73.8490955,"(40.814551, -73.8490955)",TORRY AVENUE ,NORTON AVENUE ,,0,0,0,0,0,0,0,0,Driver Inattention/Distraction,Unspecified,,,,3297457,PASSENGER VEHICLE,PASSENGER VEHICLE,,, ``` ##### Weather data[​](#weather-data-1 "Direct link to Weather data") Load the weather dataset in the same manner as you loaded the crash data. ```bash curl --location-trusted -u root \ -T ./72505394728.csv \ -H "label:weather-0" \ -H "column_separator:," \ -H "skip_header:1" \ -H "enclose:\"" \ -H "max_filter_ratio:1" \ -H "columns: STATION, DATE, LATITUDE, LONGITUDE, ELEVATION, NAME, REPORT_TYPE, SOURCE, HourlyAltimeterSetting, HourlyDewPointTemperature, HourlyDryBulbTemperature, HourlyPrecipitation, HourlyPresentWeatherType, HourlyPressureChange, HourlyPressureTendency, HourlyRelativeHumidity, HourlySkyConditions, HourlySeaLevelPressure, HourlyStationPressure, HourlyVisibility, HourlyWetBulbTemperature, HourlyWindDirection, HourlyWindGustSpeed, HourlyWindSpeed, Sunrise, Sunset, DailyAverageDewPointTemperature, DailyAverageDryBulbTemperature, DailyAverageRelativeHumidity, DailyAverageSeaLevelPressure, DailyAverageStationPressure, DailyAverageWetBulbTemperature, DailyAverageWindSpeed, DailyCoolingDegreeDays, DailyDepartureFromNormalAverageTemperature, DailyHeatingDegreeDays, DailyMaximumDryBulbTemperature, DailyMinimumDryBulbTemperature, DailyPeakWindDirection, DailyPeakWindSpeed, DailyPrecipitation, DailySnowDepth, DailySnowfall, DailySustainedWindDirection, DailySustainedWindSpeed, DailyWeather, MonthlyAverageRH, MonthlyDaysWithGT001Precip, MonthlyDaysWithGT010Precip, MonthlyDaysWithGT32Temp, MonthlyDaysWithGT90Temp, MonthlyDaysWithLT0Temp, MonthlyDaysWithLT32Temp, MonthlyDepartureFromNormalAverageTemperature, MonthlyDepartureFromNormalCoolingDegreeDays, MonthlyDepartureFromNormalHeatingDegreeDays, MonthlyDepartureFromNormalMaximumTemperature, MonthlyDepartureFromNormalMinimumTemperature, MonthlyDepartureFromNormalPrecipitation, MonthlyDewpointTemperature, MonthlyGreatestPrecip, MonthlyGreatestPrecipDate, MonthlyGreatestSnowDepth, MonthlyGreatestSnowDepthDate, MonthlyGreatestSnowfall, MonthlyGreatestSnowfallDate, MonthlyMaxSeaLevelPressureValue, MonthlyMaxSeaLevelPressureValueDate, MonthlyMaxSeaLevelPressureValueTime, MonthlyMaximumTemperature, MonthlyMeanTemperature, MonthlyMinSeaLevelPressureValue, MonthlyMinSeaLevelPressureValueDate, MonthlyMinSeaLevelPressureValueTime, MonthlyMinimumTemperature, MonthlySeaLevelPressure, MonthlyStationPressure, MonthlyTotalLiquidPrecipitation, MonthlyTotalSnowfall, MonthlyWetBulb, AWND, CDSD, CLDD, DSNW, HDSD, HTDD, NormalsCoolingDegreeDay, NormalsHeatingDegreeDay, ShortDurationEndDate005, ShortDurationEndDate010, ShortDurationEndDate015, ShortDurationEndDate020, ShortDurationEndDate030, ShortDurationEndDate045, ShortDurationEndDate060, ShortDurationEndDate080, ShortDurationEndDate100, ShortDurationEndDate120, ShortDurationEndDate150, ShortDurationEndDate180, ShortDurationPrecipitationValue005, ShortDurationPrecipitationValue010, ShortDurationPrecipitationValue015, ShortDurationPrecipitationValue020, ShortDurationPrecipitationValue030, ShortDurationPrecipitationValue045, ShortDurationPrecipitationValue060, ShortDurationPrecipitationValue080, ShortDurationPrecipitationValue100, ShortDurationPrecipitationValue120, ShortDurationPrecipitationValue150, ShortDurationPrecipitationValue180, REM, BackupDirection, BackupDistance, BackupDistanceUnit, BackupElements, BackupElevation, BackupEquipment, BackupLatitude, BackupLongitude, BackupName, WindEquipmentChangeDate" \ -XPUT http://localhost:8030/api/quickstart/weatherdata/_stream_load ``` *** #### Verify that data is stored in MinIO[​](#verify-that-data-is-stored-in-minio "Direct link to Verify that data is stored in MinIO") Open MinIO and verify that you have entries below `my-starrocks-bucket/` tip The folder names below `my-starrocks-bucket/` are generated when you load the data. You should see a single directory below `my-starrocks-bucket`, and then two more below that. In those directories you will find the data, metadata, or schema entries. ![MinIO object browser](/assets/images/MinIO-data-1c9d4bdab9cdda10f4526ba5c41673c9.png) *** #### Answer some questions[​](#answer-some-questions "Direct link to Answer some questions") These queries can be run in your SQL client. All of the queries use the `quickstart` database. ```sql USE quickstart; ``` ###### How many crashes are there per hour in NYC?[​](#how-many-crashes-are-there-per-hour-in-nyc "Direct link to How many crashes are there per hour in NYC?") ```sql SELECT COUNT(*), date_trunc("hour", crashdata.CRASH_DATE) AS Time FROM crashdata GROUP BY Time ORDER BY Time ASC LIMIT 200; ``` Here is part of the output. Note that I am looking closer at January 6th and 7th as this is Monday and Tuesday of a non-holiday week. Looking at New Years Day is probably not indicative of a normal morning during rush-hour traffic. ```plaintext | 14 | 2014-01-06 06:00:00 | | 16 | 2014-01-06 07:00:00 | | 43 | 2014-01-06 08:00:00 | | 44 | 2014-01-06 09:00:00 | | 21 | 2014-01-06 10:00:00 | | 28 | 2014-01-06 11:00:00 | | 34 | 2014-01-06 12:00:00 | | 31 | 2014-01-06 13:00:00 | | 35 | 2014-01-06 14:00:00 | | 36 | 2014-01-06 15:00:00 | | 33 | 2014-01-06 16:00:00 | | 40 | 2014-01-06 17:00:00 | | 35 | 2014-01-06 18:00:00 | | 23 | 2014-01-06 19:00:00 | | 16 | 2014-01-06 20:00:00 | | 12 | 2014-01-06 21:00:00 | | 17 | 2014-01-06 22:00:00 | | 14 | 2014-01-06 23:00:00 | | 10 | 2014-01-07 00:00:00 | | 4 | 2014-01-07 01:00:00 | | 1 | 2014-01-07 02:00:00 | | 3 | 2014-01-07 03:00:00 | | 2 | 2014-01-07 04:00:00 | | 6 | 2014-01-07 06:00:00 | | 16 | 2014-01-07 07:00:00 | | 41 | 2014-01-07 08:00:00 | | 37 | 2014-01-07 09:00:00 | | 33 | 2014-01-07 10:00:00 | ``` It looks like about 40 accidents on a Monday or Tuesday morning during rush hour traffic, and around the same at 17:00 hours. ###### What is the average temperature in NYC?[​](#what-is-the-average-temperature-in-nyc "Direct link to What is the average temperature in NYC?") ```sql SELECT avg(HourlyDryBulbTemperature), date_trunc("hour", weatherdata.DATE) AS Time FROM weatherdata GROUP BY Time ORDER BY Time ASC LIMIT 100; ``` Output: Note that this is data from 2014, NYC has not been this cold lately. ```plaintext +-------------------------------+---------------------+ | avg(HourlyDryBulbTemperature) | Time | +-------------------------------+---------------------+ | 25 | 2014-01-01 00:00:00 | | 25 | 2014-01-01 01:00:00 | | 24 | 2014-01-01 02:00:00 | | 24 | 2014-01-01 03:00:00 | | 24 | 2014-01-01 04:00:00 | | 24 | 2014-01-01 05:00:00 | | 25 | 2014-01-01 06:00:00 | | 26 | 2014-01-01 07:00:00 | ``` ###### Is it safe to drive in NYC when visibility is poor?[​](#is-it-safe-to-drive-in-nyc-when-visibility-is-poor "Direct link to Is it safe to drive in NYC when visibility is poor?") Let's look at the number of crashes when visibility is poor (between 0 and 1.0 miles). To answer this question use a JOIN across the two tables on the DATETIME column. ```sql SELECT COUNT(DISTINCT c.COLLISION_ID) AS Crashes, truncate(avg(w.HourlyDryBulbTemperature), 1) AS Temp_F, truncate(avg(w.HourlyVisibility), 2) AS Visibility, max(w.HourlyPrecipitation) AS Precipitation, date_format((date_trunc("hour", c.CRASH_DATE)), '%d %b %Y %H:%i') AS Hour FROM crashdata c LEFT JOIN weatherdata w ON date_trunc("hour", c.CRASH_DATE)=date_trunc("hour", w.DATE) WHERE w.HourlyVisibility BETWEEN 0.0 AND 1.0 GROUP BY Hour ORDER BY Crashes DESC LIMIT 100; ``` The highest number of crashes in a single hour during low visibility is 129. There are multiple things to consider: * February 3rd 2014 was a Monday * 8AM is rush hour * It was raining (0.12 inches or precipitation that hour) * The temperature is 32 degrees F (the freezing point for water) * Visibility is bad at 0.25 miles, normal for NYC is 10 miles ```plaintext +---------+--------+------------+---------------+-------------------+ | Crashes | Temp_F | Visibility | Precipitation | Hour | +---------+--------+------------+---------------+-------------------+ | 129 | 32 | 0.25 | 0.12 | 03 Feb 2014 08:00 | | 114 | 32 | 0.25 | 0.12 | 03 Feb 2014 09:00 | | 104 | 23 | 0.33 | 0.03 | 09 Jan 2015 08:00 | | 96 | 26.3 | 0.33 | 0.07 | 01 Mar 2015 14:00 | | 95 | 26 | 0.37 | 0.12 | 01 Mar 2015 15:00 | | 93 | 35 | 0.75 | 0.09 | 18 Jan 2015 09:00 | | 92 | 31 | 0.25 | 0.12 | 03 Feb 2014 10:00 | | 87 | 26.8 | 0.5 | 0.09 | 01 Mar 2015 16:00 | | 85 | 55 | 0.75 | 0.20 | 23 Dec 2015 17:00 | | 85 | 20 | 0.62 | 0.01 | 06 Jan 2015 11:00 | | 83 | 19.6 | 0.41 | 0.04 | 05 Mar 2015 13:00 | | 80 | 20 | 0.37 | 0.02 | 06 Jan 2015 10:00 | | 76 | 26.5 | 0.25 | 0.06 | 05 Mar 2015 09:00 | | 71 | 26 | 0.25 | 0.09 | 05 Mar 2015 10:00 | | 71 | 24.2 | 0.25 | 0.04 | 05 Mar 2015 11:00 | ``` ###### What about driving in icy conditions?[​](#what-about-driving-in-icy-conditions "Direct link to What about driving in icy conditions?") Water vapor can desublimate to ice at 40 degrees F; this query looks at temps between 0 and 40 degrees F. ```sql SELECT COUNT(DISTINCT c.COLLISION_ID) AS Crashes, truncate(avg(w.HourlyDryBulbTemperature), 1) AS Temp_F, truncate(avg(w.HourlyVisibility), 2) AS Visibility, max(w.HourlyPrecipitation) AS Precipitation, date_format((date_trunc("hour", c.CRASH_DATE)), '%d %b %Y %H:%i') AS Hour FROM crashdata c LEFT JOIN weatherdata w ON date_trunc("hour", c.CRASH_DATE)=date_trunc("hour", w.DATE) WHERE w.HourlyDryBulbTemperature BETWEEN 0.0 AND 40.5 GROUP BY Hour ORDER BY Crashes DESC LIMIT 100; ``` The results for freezing temperatures suprised me a little, I did not expect too much traffic on a Sunday morning in the city on a cold January day.A quick look at [weather.com](https://weather.com/storms/winter/news/northeast-storm-rain-snow-wind) showed that there was a big storm with many crashes that day, just like what can be seen in the data. ```plaintext +---------+--------+------------+---------------+-------------------+ | Crashes | Temp_F | Visibility | Precipitation | Hour | +---------+--------+------------+---------------+-------------------+ | 192 | 34 | 1.5 | 0.09 | 18 Jan 2015 08:00 | | 170 | 21 | NULL | | 21 Jan 2014 10:00 | | 145 | 19 | NULL | | 21 Jan 2014 11:00 | | 138 | 33.5 | 5 | 0.02 | 18 Jan 2015 07:00 | | 137 | 21 | NULL | | 21 Jan 2014 09:00 | | 129 | 32 | 0.25 | 0.12 | 03 Feb 2014 08:00 | | 114 | 32 | 0.25 | 0.12 | 03 Feb 2014 09:00 | | 104 | 23 | 0.7 | 0.04 | 09 Jan 2015 08:00 | | 98 | 16 | 8 | 0.00 | 06 Mar 2015 08:00 | | 96 | 26.3 | 0.33 | 0.07 | 01 Mar 2015 14:00 | ``` Drive carefully! *** #### Configuring StarRocks for shared-data[​](#configuring-starrocks-for-shared-data "Direct link to Configuring StarRocks for shared-data") Now that you have experienced using StarRocks with shared-data it is important to understand the configuration. ##### CN configuration[​](#cn-configuration "Direct link to CN configuration") The CN configuration used here is the default, as the CN is designed for shared-data use. The default configuration is shown below. You do not need to make any changes. ```bash sys_log_level = INFO # ports for admin, web, heartbeat service be_port = 9060 be_http_port = 8040 heartbeat_service_port = 9050 brpc_port = 8060 starlet_port = 9070 ``` ##### FE configuration[​](#fe-configuration "Direct link to FE configuration") The FE configuration is slightly different from the default as the FE must be configured to expect that data is stored in Object Storage rather than on local disks on BE nodes. The `docker-compose.yml` file generates the FE configuration in the `command`. ```plaintext # enable shared data, set storage type, set endpoint run_mode = shared_data cloud_native_storage_type = S3 ``` note This config file does not contain the default entries for an FE, only the shared-data configuration is shown. The non-default FE configuration settings: note Many configuration parameters are prefixed with `s3_`. This prefix is used for all Amazon S3 compatible storage types (for example: S3, GCS, and MinIO). When using Azure Blob Storage the prefix is `azure_`. ###### `run_mode=shared_data`[​](#run_modeshared_data "Direct link to run_modeshared_data") This enables shared-data use. ###### `cloud_native_storage_type=S3`[​](#cloud_native_storage_types3 "Direct link to cloud_native_storage_types3") This specifies whether S3 compatible storage or Azure Blob Storage is used. For MinIO this is always S3. ##### Details of `CREATE storage volume`[​](#details-of-create-storage-volume "Direct link to details-of-create-storage-volume") ```sql CREATE STORAGE VOLUME s3_volume TYPE = S3 LOCATIONS = ("s3://my-starrocks-bucket/") PROPERTIES ( "enabled" = "true", "aws.s3.endpoint" = "minio:9000", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.use_instance_profile" = "false", "aws.s3.use_aws_sdk_default_behavior" = "false" ); ``` ###### `aws_s3_endpoint=minio:9000`[​](#aws_s3_endpointminio9000 "Direct link to aws_s3_endpointminio9000") The MinIO endpoint, including port number. ###### `aws_s3_path=starrocks`[​](#aws_s3_pathstarrocks "Direct link to aws_s3_pathstarrocks") The bucket name. ###### `aws_s3_access_key=AAAAAAAAAAAAAAAAAAAA`[​](#aws_s3_access_keyaaaaaaaaaaaaaaaaaaaa "Direct link to aws_s3_access_keyaaaaaaaaaaaaaaaaaaaa") The MinIO access key. ###### `aws_s3_secret_key=BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB`[​](#aws_s3_secret_keybbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb "Direct link to aws_s3_secret_keybbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb") The MinIO access key secret. ###### `aws_s3_use_instance_profile=false`[​](#aws_s3_use_instance_profilefalse "Direct link to aws_s3_use_instance_profilefalse") When using MinIO an access key is used, and so instance profiles are not used with MinIO. ###### `aws_s3_use_aws_sdk_default_behavior=false`[​](#aws_s3_use_aws_sdk_default_behaviorfalse "Direct link to aws_s3_use_aws_sdk_default_behaviorfalse") When using MinIO this parameter is always set to false. ##### Configuring FQDN mode[​](#configuring-fqdn-mode "Direct link to Configuring FQDN mode") The command to start the FE is also changed. The FE service command in the Docker Compose file has the option `--host_type FQDN` added. By setting `host_type` to `FQDN` the Stream Load job is forwarded to the fully qualified domain name of the CN pod, rather than the IP address. This is done because the IP address is in a range assigned to the Docker environment, and is not typically available from the host machine. These three changes allow traffic between the host network and the CN: * setting `--host_type` to `FQDN` * exposing the CN port 8040 to the host network * adding an entry to the hosts file for `starrocks-cn` pointing to `127.0.0.1` *** #### Summary[​](#summary "Direct link to Summary") In this tutorial you: * Deployed StarRocks and Minio in Docker * Created a MinIO access key * Configured a StarRocks Storage Volume that uses MinIO * Loaded crash data provided by New York City and weather data provided by NOAA * Analyzed the data using SQL JOINs to find out that driving in low visibility or icy streets is a bad idea There is more to learn; we intentionally glossed over the data transform done during the Stream Load. The details on that are in the notes on the curl commands below. #### Notes on the curl commands[​](#notes-on-the-curl-commands "Direct link to Notes on the curl commands") StarRocks Stream Load and curl take many arguments. Only the ones used in this tutorial are described here, the rest will be linked to in the more information section. ###### `--location-trusted`[​](#--location-trusted "Direct link to --location-trusted") This configures curl to pass credentials to any redirected URLs. ###### `-u root`[​](#-u-root "Direct link to -u-root") The username used to log in to StarRocks ###### `-T filename`[​](#-t-filename "Direct link to -t-filename") T is for transfer, the filename to transfer. ###### `label:name-num`[​](#labelname-num "Direct link to labelname-num") The label to associate with this Stream Load job. The label must be unique, so if you run the job multiple times you can add a number and keep incrementing that. ###### `column_separator:,`[​](#column_separator "Direct link to column_separator") If you load a file that uses a single `,` then set it as shown above, if you use a different delimiter then set that delimiter here. Common choices are `\t`, `,`, and `|`. ###### `skip_header:1`[​](#skip_header1 "Direct link to skip_header1") Some CSV files have a single header row with all of the column names listed, and some add a second line with datatypes. Set skip\_header to `1` or `2` if you have one or two header lines, and set it to `0` if you have none. ###### `enclose:\"`[​](#enclose "Direct link to enclose") It is common to enclose strings that contain embedded commas with double-quotes. The sample datasets used in this tutorial have geo locations that contain commas and so the enclose setting is set to `\"`. Remember to escape the `"` with a `\`. ###### `max_filter_ratio:1`[​](#max_filter_ratio1 "Direct link to max_filter_ratio1") This allows some errors in the data. Ideally this would be set to `0` and the job would fail with any errors. It is set to `1` to allow all rows to fail during debugging. ###### `columns:`[​](#columns "Direct link to columns") The mapping of CSV file columns to StarRocks table columns. You will notice that there are many more columns in the CSV files than columns in the table. Any columns that are not included in the table are skipped. You will also notice that there is some transformation of data included in the `columns:` line for the crash dataset. It is very common to find dates and times in CSV files that do not conform to standards. This is the logic for converting the CSV data for the time and date of the crash to a DATETIME type: ###### The columns line[​](#the-columns-line "Direct link to The columns line") This is the beginning of one data record. The date is in `MM/DD/YYYY` format, and the time is `HH:MI`. Since DATETIME is generally `YYYY-MM-DD HH:MI:SS` we need to transform this data. ```plaintext 08/05/2014,9:10,BRONX,10469,40.8733019,-73.8536375,"(40.8733019, -73.8536375)", ``` This is the beginning of the `columns:` parameter: ```bash -H "columns:tmp_CRASH_DATE, tmp_CRASH_TIME, CRASH_DATE=str_to_date(concat_ws(' ', tmp_CRASH_DATE, tmp_CRASH_TIME), '%m/%d/%Y %H:%i') ``` This instructs StarRocks to: * Assign the content of the first column of the CSV file to `tmp_CRASH_DATE` * Assign the content of the second column of the CSV file to `tmp_CRASH_TIME` * `concat_ws()` concatenates `tmp_CRASH_DATE` and `tmp_CRASH_TIME` together with a space between them * `str_to_date()` creates a DATETIME from the concatenated string * store the resulting DATETIME in the column `CRASH_DATE` #### More information[​](#more-information "Direct link to More information") [StarRocks table design](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md) [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) The [Motor Vehicle Collisions - Crashes](https://data.cityofnewyork.us/Public-Safety/Motor-Vehicle-Collisions-Crashes/h9gi-nx95) dataset is provided by New York City subject to these [terms of use](https://www.nyc.gov/home/terms-of-use.page) and [privacy policy](https://www.nyc.gov/home/privacy-policy.page). The [Local Climatological Data](https://www.ncdc.noaa.gov/cdo-web/datatools/lcd)(LCD) is provided by NOAA with this [disclaimer](https://www.noaa.gov/disclaimer) and this [privacy policy](https://www.noaa.gov/protecting-your-privacy). --- ### Deploy StarRocks with Docker This tutorial covers: * Running StarRocks in a single Docker container * Loading two public datasets including basic transformation of the data * Analyzing the data with SELECT and JOIN * Basic data transformation (the **T** in ETL) #### Follow along with the video if you prefer[​](#follow-along-with-the-video-if-you-prefer "Direct link to Follow along with the video if you prefer") [StarRocks in 5 - Getting Started With StarRocks on Docker](https://www.youtube.com/embed/h7F4U6xEA5M) The data used is provided by NYC OpenData and the National Centers for Environmental Information. Both of these datasets are very large, and because this tutorial is intended to help you get exposed to working with StarRocks we are not going to load data for the past 120 years. You can run the Docker image and load this data on a machine with 4 GB RAM assigned to Docker. For larger fault-tolerant and scalable deployments we have other documentation and will provide that later. There is a lot of information in this document, and it is presented with the step by step content at the beginning, and the technical details at the end. This is done to serve these purposes in this order: 1. Allow the reader to load data in StarRocks and analyze that data. 2. Explain the basics of data transformation during loading. *** #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") ##### Docker[​](#docker "Direct link to Docker") * [Docker](https://docs.docker.com/engine/install/) * 4 GB RAM assigned to Docker * 10 GB free disk space assigned to Docker ##### SQL client[​](#sql-client "Direct link to SQL client") You can use the SQL client provided in the Docker environment, or use one on your system. Many MySQL compatible clients will work, and this guide covers the configuration of DBeaver and MySQL Workbench. ##### curl[​](#curl "Direct link to curl") `curl` is used to issue the data load job to StarRocks, and to download the datasets. Check to see if you have it installed by running `curl` or `curl.exe` at your OS prompt. If curl is not installed, [get curl here](https://curl.se/download.html). *** #### Terminology[​](#terminology "Direct link to Terminology") ##### FE[​](#fe "Direct link to FE") Frontend nodes are responsible for metadata management, client connection management, query planning, and query scheduling. Each FE stores and maintains a complete copy of metadata in its memory, which guarantees indiscriminate services among the FEs. ##### BE[​](#be "Direct link to BE") Backend nodes are responsible for both data storage and executing query plans. *** #### Launch StarRocks[​](#launch-starrocks "Direct link to Launch StarRocks") ```bash docker pull starrocks/allin1-ubuntu docker run -p 9030:9030 -p 8030:8030 -p 8040:8040 -itd \ --name quickstart starrocks/allin1-ubuntu ``` *** #### SQL clients[​](#sql-clients "Direct link to SQL clients") These three clients are tested with this tutorial, you only need one: * mysql CLI: You can run this from the Docker environment or your machine. * [DBeaver](https://dbeaver.io/download/) is available as a community version and a Pro version. * [MySQL Workbench](https://dev.mysql.com/downloads/workbench/) ###### Configuring the client[​](#configuring-the-client "Direct link to Configuring the client") * mysql CLI * DBeaver * MySQL Workbench The easiest way to use the mysql CLI is to run it from the StarRocks container `starrocks-fe`: ```bash docker exec -it quickstart \ mysql -P 9030 -h 127.0.0.1 -u root --prompt="StarRocks > " ``` If you would like to install the mysql CLI expand **mysql client install** below: mysql client install * macOS: If you use Homebrew and do not need MySQL Server run `brew install mysql-client@8.0` to install the CLI. * Linux: Check your repository system for the `mysql` client. For example, `yum install mariadb`. * Microsoft Windows: Install the [MySQL Community Server](https://dev.mysql.com/downloads/mysql/) and run the provided client, or run `mysql` from WSL. - Install [DBeaver](https://dbeaver.io/download/), and add a connection: ![Add a connection](/assets/images/DBeaver-1-58907bd9df44bba1e40282214e4a056c.png) - Configure the port, IP, and username. Test the connection, and click Finish if the test succeeds: ![Configure and test](/assets/images/DBeaver-2-8e4bdc09135b4738944d7016bfb1af94.png) * Install the [MySQL Workbench](https://dev.mysql.com/downloads/workbench/), and add a connection. * Configure the port, IP, and username and then test the connection: ![Connection settings](/assets/images/Workbench-1-9fd3a3126d5f83478e51891287e412b9.png) * You will see warnings from the Workbench as it is checking for a specific MySQL version. You can ignore the warnings and when prompted, you can configure Workbench to stop displaying the warnings: ![Ignore warning](/assets/images/Workbench-2-efba18763e56101091746837ae666b5e.png) *** #### Download the data[​](#download-the-data "Direct link to Download the data") Download these two datasets to your machine. You can download them to the host machine where you are running Docker, they do not need to be downloaded inside the container. ##### New York City crash data[​](#new-york-city-crash-data "Direct link to New York City crash data") ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/datasets/NYPD_Crash_Data.csv ``` ##### Weather data[​](#weather-data "Direct link to Weather data") ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/datasets/72505394728.csv ``` *** ##### Connect to StarRocks with a SQL client[​](#connect-to-starrocks-with-a-sql-client "Direct link to Connect to StarRocks with a SQL client") tip If you are using a client other than the mysql CLI, open that now. This command will run the `mysql` command in the Docker container: ```sql docker exec -it quickstart \ mysql -P 9030 -h 127.0.0.1 -u root --prompt="StarRocks > " ``` *** #### Create some tables[​](#create-some-tables "Direct link to Create some tables") ##### Create a database[​](#create-a-database "Direct link to Create a database") Type these two lines in at the `StarRocks > `prompt and press enter after each: ```sql CREATE DATABASE IF NOT EXISTS quickstart; USE quickstart; ``` ##### Create two tables[​](#create-two-tables "Direct link to Create two tables") ###### Crashdata[​](#crashdata "Direct link to Crashdata") The crash dataset contains many more fields than these, the schema has been trimmed down to include only the fields that might be useful to answer questions about the impact weather has on driving conditions. ```sql CREATE TABLE IF NOT EXISTS crashdata ( CRASH_DATE DATETIME, BOROUGH STRING, ZIP_CODE STRING, LATITUDE INT, LONGITUDE INT, LOCATION STRING, ON_STREET_NAME STRING, CROSS_STREET_NAME STRING, OFF_STREET_NAME STRING, CONTRIBUTING_FACTOR_VEHICLE_1 STRING, CONTRIBUTING_FACTOR_VEHICLE_2 STRING, COLLISION_ID INT, VEHICLE_TYPE_CODE_1 STRING, VEHICLE_TYPE_CODE_2 STRING ); ``` ###### Weatherdata[​](#weatherdata "Direct link to Weatherdata") Similar to the crash data, the weather dataset has many more columns (a total of 125 columns) and only the ones that are expected to answer the questions are included in the database. ```sql CREATE TABLE IF NOT EXISTS weatherdata ( DATE DATETIME, NAME STRING, HourlyDewPointTemperature STRING, HourlyDryBulbTemperature STRING, HourlyPrecipitation STRING, HourlyPresentWeatherType STRING, HourlyPressureChange STRING, HourlyPressureTendency STRING, HourlyRelativeHumidity STRING, HourlySkyConditions STRING, HourlyVisibility STRING, HourlyWetBulbTemperature STRING, HourlyWindDirection STRING, HourlyWindGustSpeed STRING, HourlyWindSpeed STRING ); ``` *** #### Load two datasets[​](#load-two-datasets "Direct link to Load two datasets") There are many ways to load data into StarRocks. For this tutorial the simplest way is to use curl and StarRocks Stream Load. tip Open a new shell as these curl commands are run at the operating system prompt, not in the `mysql` client. The commands refer to the datasets that you downloaded, so run them from the directory where you downloaded the files. You will be prompted for a password. You probably have not assigned a password to the MySQL `root` user, so just hit enter. The `curl` commands look complex, but they are explained in detail at the end of the tutorial. For now, we recommend running the commands and running some SQL to analyze the data, and then reading about the data loading details at the end. ##### New York City collision data - Crashes[​](#new-york-city-collision-data---crashes "Direct link to New York City collision data - Crashes") ```bash curl --location-trusted -u root \ -T ./NYPD_Crash_Data.csv \ -H "label:crashdata-0" \ -H "column_separator:," \ -H "skip_header:1" \ -H "enclose:\"" \ -H "max_filter_ratio:1" \ -H "columns:tmp_CRASH_DATE, tmp_CRASH_TIME, CRASH_DATE=str_to_date(concat_ws(' ', tmp_CRASH_DATE, tmp_CRASH_TIME), '%m/%d/%Y %H:%i'),BOROUGH,ZIP_CODE,LATITUDE,LONGITUDE,LOCATION,ON_STREET_NAME,CROSS_STREET_NAME,OFF_STREET_NAME,NUMBER_OF_PERSONS_INJURED,NUMBER_OF_PERSONS_KILLED,NUMBER_OF_PEDESTRIANS_INJURED,NUMBER_OF_PEDESTRIANS_KILLED,NUMBER_OF_CYCLIST_INJURED,NUMBER_OF_CYCLIST_KILLED,NUMBER_OF_MOTORIST_INJURED,NUMBER_OF_MOTORIST_KILLED,CONTRIBUTING_FACTOR_VEHICLE_1,CONTRIBUTING_FACTOR_VEHICLE_2,CONTRIBUTING_FACTOR_VEHICLE_3,CONTRIBUTING_FACTOR_VEHICLE_4,CONTRIBUTING_FACTOR_VEHICLE_5,COLLISION_ID,VEHICLE_TYPE_CODE_1,VEHICLE_TYPE_CODE_2,VEHICLE_TYPE_CODE_3,VEHICLE_TYPE_CODE_4,VEHICLE_TYPE_CODE_5" \ -XPUT http://localhost:8030/api/quickstart/crashdata/_stream_load ``` Here is the output of the preceding command. The first highlighted section shows what you should expect to see (OK and all but one row inserted). One row was filtered out because it does not contain the correct number of columns. ```bash Enter host password for user 'root': { "TxnId": 2, "Label": "crashdata-0", "Status": "Success", "Message": "OK", "NumberTotalRows": 423726, "NumberLoadedRows": 423725, "NumberFilteredRows": 1, "NumberUnselectedRows": 0, "LoadBytes": 96227746, "LoadTimeMs": 1013, "BeginTxnTimeMs": 21, "StreamLoadPlanTimeMs": 63, "ReadDataTimeMs": 563, "WriteDataTimeMs": 870, "CommitAndPublishTimeMs": 57, "ErrorURL": "http://127.0.0.1:8040/api/_load_error_log?file=error_log_da41dd88276a7bfc_739087c94262ae9f" }% ``` If there was an error the output provides a URL to see the error messages. Open this in a browser to find out what happened. Expand the detail to see a sample error message: Reading error messages in the browser ```bash Error: Target column count: 29 doesn't match source value column count: 32. Column separator: ',', Row delimiter: '\n'. Row: 09/06/2015,14:15,,,40.6722269,-74.0110059,"(40.6722269, -74.0110059)",,,"R/O 1 BEARD ST. ( IKEA'S 09/14/2015,5:30,BRONX,10473,40.814551,-73.8490955,"(40.814551, -73.8490955)",TORRY AVENUE ,NORTON AVENUE ,,0,0,0,0,0,0,0,0,Driver Inattention/Distraction,Unspecified,,,,3297457,PASSENGER VEHICLE,PASSENGER VEHICLE,,, ``` ##### Weather data[​](#weather-data-1 "Direct link to Weather data") Load the weather dataset in the same manner as you loaded the crash data. ```bash curl --location-trusted -u root \ -T ./72505394728.csv \ -H "label:weather-0" \ -H "column_separator:," \ -H "skip_header:1" \ -H "enclose:\"" \ -H "max_filter_ratio:1" \ -H "columns: STATION, DATE, LATITUDE, LONGITUDE, ELEVATION, NAME, REPORT_TYPE, SOURCE, HourlyAltimeterSetting, HourlyDewPointTemperature, HourlyDryBulbTemperature, HourlyPrecipitation, HourlyPresentWeatherType, HourlyPressureChange, HourlyPressureTendency, HourlyRelativeHumidity, HourlySkyConditions, HourlySeaLevelPressure, HourlyStationPressure, HourlyVisibility, HourlyWetBulbTemperature, HourlyWindDirection, HourlyWindGustSpeed, HourlyWindSpeed, Sunrise, Sunset, DailyAverageDewPointTemperature, DailyAverageDryBulbTemperature, DailyAverageRelativeHumidity, DailyAverageSeaLevelPressure, DailyAverageStationPressure, DailyAverageWetBulbTemperature, DailyAverageWindSpeed, DailyCoolingDegreeDays, DailyDepartureFromNormalAverageTemperature, DailyHeatingDegreeDays, DailyMaximumDryBulbTemperature, DailyMinimumDryBulbTemperature, DailyPeakWindDirection, DailyPeakWindSpeed, DailyPrecipitation, DailySnowDepth, DailySnowfall, DailySustainedWindDirection, DailySustainedWindSpeed, DailyWeather, MonthlyAverageRH, MonthlyDaysWithGT001Precip, MonthlyDaysWithGT010Precip, MonthlyDaysWithGT32Temp, MonthlyDaysWithGT90Temp, MonthlyDaysWithLT0Temp, MonthlyDaysWithLT32Temp, MonthlyDepartureFromNormalAverageTemperature, MonthlyDepartureFromNormalCoolingDegreeDays, MonthlyDepartureFromNormalHeatingDegreeDays, MonthlyDepartureFromNormalMaximumTemperature, MonthlyDepartureFromNormalMinimumTemperature, MonthlyDepartureFromNormalPrecipitation, MonthlyDewpointTemperature, MonthlyGreatestPrecip, MonthlyGreatestPrecipDate, MonthlyGreatestSnowDepth, MonthlyGreatestSnowDepthDate, MonthlyGreatestSnowfall, MonthlyGreatestSnowfallDate, MonthlyMaxSeaLevelPressureValue, MonthlyMaxSeaLevelPressureValueDate, MonthlyMaxSeaLevelPressureValueTime, MonthlyMaximumTemperature, MonthlyMeanTemperature, MonthlyMinSeaLevelPressureValue, MonthlyMinSeaLevelPressureValueDate, MonthlyMinSeaLevelPressureValueTime, MonthlyMinimumTemperature, MonthlySeaLevelPressure, MonthlyStationPressure, MonthlyTotalLiquidPrecipitation, MonthlyTotalSnowfall, MonthlyWetBulb, AWND, CDSD, CLDD, DSNW, HDSD, HTDD, NormalsCoolingDegreeDay, NormalsHeatingDegreeDay, ShortDurationEndDate005, ShortDurationEndDate010, ShortDurationEndDate015, ShortDurationEndDate020, ShortDurationEndDate030, ShortDurationEndDate045, ShortDurationEndDate060, ShortDurationEndDate080, ShortDurationEndDate100, ShortDurationEndDate120, ShortDurationEndDate150, ShortDurationEndDate180, ShortDurationPrecipitationValue005, ShortDurationPrecipitationValue010, ShortDurationPrecipitationValue015, ShortDurationPrecipitationValue020, ShortDurationPrecipitationValue030, ShortDurationPrecipitationValue045, ShortDurationPrecipitationValue060, ShortDurationPrecipitationValue080, ShortDurationPrecipitationValue100, ShortDurationPrecipitationValue120, ShortDurationPrecipitationValue150, ShortDurationPrecipitationValue180, REM, BackupDirection, BackupDistance, BackupDistanceUnit, BackupElements, BackupElevation, BackupEquipment, BackupLatitude, BackupLongitude, BackupName, WindEquipmentChangeDate" \ -XPUT http://localhost:8030/api/quickstart/weatherdata/_stream_load ``` *** #### Answer some questions[​](#answer-some-questions "Direct link to Answer some questions") These queries can be run in your SQL client. All of the queries use the `quickstart` database. ```sql USE quickstart; ``` ###### How many crashes are there per hour in NYC?[​](#how-many-crashes-are-there-per-hour-in-nyc "Direct link to How many crashes are there per hour in NYC?") ```sql SELECT COUNT(*), date_trunc("hour", crashdata.CRASH_DATE) AS Time FROM crashdata GROUP BY Time ORDER BY Time ASC LIMIT 200; ``` Here is part of the output. Note that I am looking closer at January 6th and 7th as this is Monday and Tuesday of a non-holiday week. Looking at New Years Day is probably not indicative of a normal morning during rush-hour traffic. ```plaintext | 14 | 2014-01-06 06:00:00 | | 16 | 2014-01-06 07:00:00 | | 43 | 2014-01-06 08:00:00 | | 44 | 2014-01-06 09:00:00 | | 21 | 2014-01-06 10:00:00 | | 28 | 2014-01-06 11:00:00 | | 34 | 2014-01-06 12:00:00 | | 31 | 2014-01-06 13:00:00 | | 35 | 2014-01-06 14:00:00 | | 36 | 2014-01-06 15:00:00 | | 33 | 2014-01-06 16:00:00 | | 40 | 2014-01-06 17:00:00 | | 35 | 2014-01-06 18:00:00 | | 23 | 2014-01-06 19:00:00 | | 16 | 2014-01-06 20:00:00 | | 12 | 2014-01-06 21:00:00 | | 17 | 2014-01-06 22:00:00 | | 14 | 2014-01-06 23:00:00 | | 10 | 2014-01-07 00:00:00 | | 4 | 2014-01-07 01:00:00 | | 1 | 2014-01-07 02:00:00 | | 3 | 2014-01-07 03:00:00 | | 2 | 2014-01-07 04:00:00 | | 6 | 2014-01-07 06:00:00 | | 16 | 2014-01-07 07:00:00 | | 41 | 2014-01-07 08:00:00 | | 37 | 2014-01-07 09:00:00 | | 33 | 2014-01-07 10:00:00 | ``` It looks like about 40 accidents on a Monday or Tuesday morning during rush hour traffic, and around the same at 17:00 hours. ###### What is the average temperature in NYC?[​](#what-is-the-average-temperature-in-nyc "Direct link to What is the average temperature in NYC?") ```sql SELECT avg(HourlyDryBulbTemperature), date_trunc("hour", weatherdata.DATE) AS Time FROM weatherdata GROUP BY Time ORDER BY Time ASC LIMIT 100; ``` Output: Note that this is data from 2014, NYC has not been this cold lately. ```plaintext +-------------------------------+---------------------+ | avg(HourlyDryBulbTemperature) | Time | +-------------------------------+---------------------+ | 25 | 2014-01-01 00:00:00 | | 25 | 2014-01-01 01:00:00 | | 24 | 2014-01-01 02:00:00 | | 24 | 2014-01-01 03:00:00 | | 24 | 2014-01-01 04:00:00 | | 24 | 2014-01-01 05:00:00 | | 25 | 2014-01-01 06:00:00 | | 26 | 2014-01-01 07:00:00 | ``` ###### Is it safe to drive in NYC when visibility is poor?[​](#is-it-safe-to-drive-in-nyc-when-visibility-is-poor "Direct link to Is it safe to drive in NYC when visibility is poor?") Let's look at the number of crashes when visibility is poor (between 0 and 1.0 miles). To answer this question use a JOIN across the two tables on the DATETIME column. ```sql SELECT COUNT(DISTINCT c.COLLISION_ID) AS Crashes, truncate(avg(w.HourlyDryBulbTemperature), 1) AS Temp_F, truncate(avg(w.HourlyVisibility), 2) AS Visibility, max(w.HourlyPrecipitation) AS Precipitation, date_format((date_trunc("hour", c.CRASH_DATE)), '%d %b %Y %H:%i') AS Hour FROM crashdata c LEFT JOIN weatherdata w ON date_trunc("hour", c.CRASH_DATE)=date_trunc("hour", w.DATE) WHERE w.HourlyVisibility BETWEEN 0.0 AND 1.0 GROUP BY Hour ORDER BY Crashes DESC LIMIT 100; ``` The highest number of crashes in a single hour during low visibility is 129. There are multiple things to consider: * February 3rd 2014 was a Monday * 8AM is rush hour * It was raining (0.12 inches or precipitation that hour) * The temperature is 32 degrees F (the freezing point for water) * Visibility is bad at 0.25 miles, normal for NYC is 10 miles ```plaintext +---------+--------+------------+---------------+-------------------+ | Crashes | Temp_F | Visibility | Precipitation | Hour | +---------+--------+------------+---------------+-------------------+ | 129 | 32 | 0.25 | 0.12 | 03 Feb 2014 08:00 | | 114 | 32 | 0.25 | 0.12 | 03 Feb 2014 09:00 | | 104 | 23 | 0.33 | 0.03 | 09 Jan 2015 08:00 | | 96 | 26.3 | 0.33 | 0.07 | 01 Mar 2015 14:00 | | 95 | 26 | 0.37 | 0.12 | 01 Mar 2015 15:00 | | 93 | 35 | 0.75 | 0.09 | 18 Jan 2015 09:00 | | 92 | 31 | 0.25 | 0.12 | 03 Feb 2014 10:00 | | 87 | 26.8 | 0.5 | 0.09 | 01 Mar 2015 16:00 | | 85 | 55 | 0.75 | 0.20 | 23 Dec 2015 17:00 | | 85 | 20 | 0.62 | 0.01 | 06 Jan 2015 11:00 | | 83 | 19.6 | 0.41 | 0.04 | 05 Mar 2015 13:00 | | 80 | 20 | 0.37 | 0.02 | 06 Jan 2015 10:00 | | 76 | 26.5 | 0.25 | 0.06 | 05 Mar 2015 09:00 | | 71 | 26 | 0.25 | 0.09 | 05 Mar 2015 10:00 | | 71 | 24.2 | 0.25 | 0.04 | 05 Mar 2015 11:00 | ``` ###### What about driving in icy conditions?[​](#what-about-driving-in-icy-conditions "Direct link to What about driving in icy conditions?") Water vapor can desublimate to ice at 40 degrees F; this query looks at temps between 0 and 40 degrees F. ```sql SELECT COUNT(DISTINCT c.COLLISION_ID) AS Crashes, truncate(avg(w.HourlyDryBulbTemperature), 1) AS Temp_F, truncate(avg(w.HourlyVisibility), 2) AS Visibility, max(w.HourlyPrecipitation) AS Precipitation, date_format((date_trunc("hour", c.CRASH_DATE)), '%d %b %Y %H:%i') AS Hour FROM crashdata c LEFT JOIN weatherdata w ON date_trunc("hour", c.CRASH_DATE)=date_trunc("hour", w.DATE) WHERE w.HourlyDryBulbTemperature BETWEEN 0.0 AND 40.5 GROUP BY Hour ORDER BY Crashes DESC LIMIT 100; ``` The results for freezing temperatures suprised me a little, I did not expect too much traffic on a Sunday morning in the city on a cold January day.A quick look at [weather.com](https://weather.com/storms/winter/news/northeast-storm-rain-snow-wind) showed that there was a big storm with many crashes that day, just like what can be seen in the data. ```plaintext +---------+--------+------------+---------------+-------------------+ | Crashes | Temp_F | Visibility | Precipitation | Hour | +---------+--------+------------+---------------+-------------------+ | 192 | 34 | 1.5 | 0.09 | 18 Jan 2015 08:00 | | 170 | 21 | NULL | | 21 Jan 2014 10:00 | | 145 | 19 | NULL | | 21 Jan 2014 11:00 | | 138 | 33.5 | 5 | 0.02 | 18 Jan 2015 07:00 | | 137 | 21 | NULL | | 21 Jan 2014 09:00 | | 129 | 32 | 0.25 | 0.12 | 03 Feb 2014 08:00 | | 114 | 32 | 0.25 | 0.12 | 03 Feb 2014 09:00 | | 104 | 23 | 0.7 | 0.04 | 09 Jan 2015 08:00 | | 98 | 16 | 8 | 0.00 | 06 Mar 2015 08:00 | | 96 | 26.3 | 0.33 | 0.07 | 01 Mar 2015 14:00 | ``` Drive carefully! *** #### Summary[​](#summary "Direct link to Summary") In this tutorial you: * Deployed StarRocks in Docker * Loaded crash data provided by New York City and weather data provided by NOAA * Analyzed the data using SQL JOINs to find out that driving in low visibility or icy streets is a bad idea There is more to learn; we intentionally glossed over the data transformation done during the Stream Load. The details on that are in the notes on the curl commands below. *** #### Notes on the curl commands[​](#notes-on-the-curl-commands "Direct link to Notes on the curl commands") StarRocks Stream Load and curl take many arguments. Only the ones used in this tutorial are described here, the rest will be linked to in the more information section. ###### `--location-trusted`[​](#--location-trusted "Direct link to --location-trusted") This configures curl to pass credentials to any redirected URLs. ###### `-u root`[​](#-u-root "Direct link to -u-root") The username used to log in to StarRocks ###### `-T filename`[​](#-t-filename "Direct link to -t-filename") T is for transfer, the filename to transfer. ###### `label:name-num`[​](#labelname-num "Direct link to labelname-num") The label to associate with this Stream Load job. The label must be unique, so if you run the job multiple times you can add a number and keep incrementing that. ###### `column_separator:,`[​](#column_separator "Direct link to column_separator") If you load a file that uses a single `,` then set it as shown above, if you use a different delimiter then set that delimiter here. Common choices are `\t`, `,`, and `|`. ###### `skip_header:1`[​](#skip_header1 "Direct link to skip_header1") Some CSV files have a single header row with all of the column names listed, and some add a second line with datatypes. Set skip\_header to `1` or `2` if you have one or two header lines, and set it to `0` if you have none. ###### `enclose:\"`[​](#enclose "Direct link to enclose") It is common to enclose strings that contain embedded commas with double-quotes. The sample datasets used in this tutorial have geo locations that contain commas and so the enclose setting is set to `\"`. Remember to escape the `"` with a `\`. ###### `max_filter_ratio:1`[​](#max_filter_ratio1 "Direct link to max_filter_ratio1") This allows some errors in the data. Ideally this would be set to `0` and the job would fail with any errors. It is set to `1` to allow all rows to fail during debugging. ###### `columns:`[​](#columns "Direct link to columns") The mapping of CSV file columns to StarRocks table columns. You will notice that there are many more columns in the CSV files than columns in the table. Any columns that are not included in the table are skipped. You will also notice that there is some transformation of data included in the `columns:` line for the crash dataset. It is very common to find dates and times in CSV files that do not conform to standards. This is the logic for converting the CSV data for the time and date of the crash to a DATETIME type: ###### The columns line[​](#the-columns-line "Direct link to The columns line") This is the beginning of one data record. The date is in `MM/DD/YYYY` format, and the time is `HH:MI`. Since DATETIME is generally `YYYY-MM-DD HH:MI:SS` we need to transform this data. ```plaintext 08/05/2014,9:10,BRONX,10469,40.8733019,-73.8536375,"(40.8733019, -73.8536375)", ``` This is the beginning of the `columns:` parameter: ```bash -H "columns:tmp_CRASH_DATE, tmp_CRASH_TIME, CRASH_DATE=str_to_date(concat_ws(' ', tmp_CRASH_DATE, tmp_CRASH_TIME), '%m/%d/%Y %H:%i') ``` This instructs StarRocks to: * Assign the content of the first column of the CSV file to `tmp_CRASH_DATE` * Assign the content of the second column of the CSV file to `tmp_CRASH_TIME` * `concat_ws()` concatenates `tmp_CRASH_DATE` and `tmp_CRASH_TIME` together with a space between them * `str_to_date()` creates a DATETIME from the concatenated string * store the resulting DATETIME in the column `CRASH_DATE` *** #### More information[​](#more-information "Direct link to More information") [StarRocks table design](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md) [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) The [Motor Vehicle Collisions - Crashes](https://data.cityofnewyork.us/Public-Safety/Motor-Vehicle-Collisions-Crashes/h9gi-nx95) dataset is provided by New York City subject to these [terms of use](https://www.nyc.gov/home/terms-of-use.page) and [privacy policy](https://www.nyc.gov/home/privacy-policy.page). The [Local Climatological Data](https://www.ncdc.noaa.gov/cdo-web/datatools/lcd)(LCD) is provided by NOAA with this [disclaimer](https://www.noaa.gov/disclaimer) and this [privacy policy](https://www.noaa.gov/protecting-your-privacy). --- ## Release 2.5 ### StarRocks version 2.5 #### 2.5.22[​](#2522 "Direct link to 2.5.22") Release date: June 20, 2024 ##### Improvements[​](#improvements "Direct link to Improvements") * Optimized a partition check logic used for building query execution plan, significantly reducing the time consumption of complex queries that involve multiple tables. [#46781](https://github.com/StarRocks/starrocks/pull/46781) ##### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") Fixed the following issues: * Function Call does not handle child errors correctly. [#42590](https://github.com/StarRocks/starrocks/pull/42590) * The internal data statistics were not cleaned up regularly, causing inaccurate estimated information and thereby inefficient query plans. This will cause a drop in query performance and a surge in memory usage. [#45839](https://github.com/StarRocks/starrocks/pull/45839) * Using a stale column histogram may lead to the Division by Zero exception. [#45614](https://github.com/StarRocks/starrocks/pull/45614) #### 2.5.21[​](#2521 "Direct link to 2.5.21") Release date: May 15, 2024 ##### Improvements[​](#improvements-1 "Direct link to Improvements") * Optimized the usage of database locks for materialized view refresh to prevent deadlock. [#42801](https://github.com/StarRocks/starrocks/pull/42801) * Both `s3a://` and `s3://` can be used to access data in AWS S3. [#42460](https://github.com/StarRocks/starrocks/pull/42460) ##### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") Fixed the following issues: * Schema change may cause issues in prefix index sorting, leading to incorrect results for queries based on prefix indexes. [#44941](https://github.com/StarRocks/starrocks/pull/44941) * After a Routine Load task is paused due to Kafka cluster abnormalities, the background still attempts to connect to this abnormal Kafka cluster, which prevents other Routine Load tasks in this StarRocks cluster from consuming normal Kafka messages. [#45029](https://github.com/StarRocks/starrocks/pull/45029) * When querying views in `information_schema`, the database lock is held for an unexpectedly long time, which prolongs the overall query time. [#45392](https://github.com/StarRocks/starrocks/pull/45392) * Enabling Query Cache may cause BEs to crash if the SQL query contains a HAVING clause. This issue can be resolved by disabling Query Cache using `set enable_query_cache=false`. [#43823](https://github.com/StarRocks/starrocks/pull/43823) * When Query Cache is enabled, some queries may return an error message `All slotIds should be remapped`. [#42861](https://github.com/StarRocks/starrocks/pull/42861) #### 2.5.20[​](#2520 "Direct link to 2.5.20") Release date: March 22, 2024 ##### Improvements[​](#improvements-2 "Direct link to Improvements") * `replace_if_not_null` supports BITMAP columns in an Aggregate table. Users can specify `replace_if_not_null` as the aggregate function for BITMAP columns in an Aggregate table. [#42104](https://github.com/StarRocks/starrocks/pull/42104) * G1 Garbage Collector is used for JDK 9 and later by default. [#41374](https://github.com/StarRocks/starrocks/pull/41374) ##### Parameter Changes[​](#parameter-changes "Direct link to Parameter Changes") * The default value of the BE parameter `update_compaction_size_threshold` is changed from 256 MB to 64 MB to accelerate compaction. [#42776](https://github.com/StarRocks/starrocks/pull/42776) ##### Bug Fixes[​](#bug-fixes-2 "Direct link to Bug Fixes") Fixed the following issues: * Synchronizing data using StarRocks external tables encounters the error "commit and publish txn failed". The synchronization succeeds after a retry but the same copy of data is loaded twice. [#25165](https://github.com/StarRocks/starrocks/pull/25165) * RPC transmit resources are temporarily unavailable due to GC issues. [#41636](https://github.com/StarRocks/starrocks/pull/41636) * array\_agg() in v2.5 processes NULLs in a different way than it does in v2.3. As a result, the query result is incorrect after an upgrade from v2.3 to v2.5. [#42639](https://github.com/StarRocks/starrocks/pull/42639) * The Sink Operator in a query unexpectedly exits, which causes BEs to crash. [#38662](https://github.com/StarRocks/starrocks/pull/38662) * Executing the DELETE command on an Aggregate table results in a race for accessing tablet metadata, which causes BEs to crash. [#42174](https://github.com/StarRocks/starrocks/pull/42174) * The MemTracker encounters the Use-After-Free issue during UDF calling, which causes BEs to crash. [#41710](https://github.com/StarRocks/starrocks/pull/41710) * The unnest() function does not support aliases. [#42138](https://github.com/StarRocks/starrocks/pull/42138) #### 2.5.19[​](#2519 "Direct link to 2.5.19") Release date: February 8, 2024 ##### New features[​](#new-features "Direct link to New features") * Added Bitmap value processing functions: serialize, deserialize, and serializeToString. [#40162](https://github.com/StarRocks/starrocks/pull/40162/files) ##### Improvements[​](#improvements-3 "Direct link to Improvements") * Supports automatic activation of inactive materialized views when refreshing these materialized views. [#38521](https://github.com/StarRocks/starrocks/pull/38521) * Optimized BE log printing to prevent too many irrelevant logs. [#22820](https://github.com/StarRocks/starrocks/pull/22820) [#36187](https://github.com/StarRocks/starrocks/pull/36187) * Supports using [Hive UDFs](https://docs.starrocks.io/docs/integrations/hive_bitmap_udf/) to process and load Bitmap data into StarRocks and export Bitmap data from StarRocks to Hive. [#40165](https://github.com/StarRocks/starrocks/pull/40165) [#40168](https://github.com/StarRocks/starrocks/pull/40168) * Added date formats `yyyy-MM-ddTHH:mm` and `yyyy-MM-dd HH:mm` to support TIMESTAMP partition fields in Apache Iceberg tables. [#39986](https://github.com/StarRocks/starrocks/pull/39986) ##### Bug Fixes[​](#bug-fixes-3 "Direct link to Bug Fixes") Fixed the following issues: * Running a Spark Load job that has no PROPERTIES specified causes null pointer exceptions (NPEs). [#38765](https://github.com/StarRocks/starrocks/pull/38765) * INSERT INTO SELECT occasionally encounters the error "timeout by txn manager". [#36688](https://github.com/StarRocks/starrocks/pull/36688) * The memory consumption of PageCache exceeds the threshold specified by the BE dynamic parameter `storage_page_cache_limit` in certain circumstances. [#37740](https://github.com/StarRocks/starrocks/pull/37740) * After a table is dropped and then re-created with the same table name, refreshing asynchronous materialized views created on that table fails. [#38008](https://github.com/StarRocks/starrocks/pull/38008) [#38982](https://github.com/StarRocks/starrocks/pull/38982) * Writing data to S3 buckets using SELECT INTO occasionally encounters the error "The tablet write operation update metadata take a long time". [#38443](https://github.com/StarRocks/starrocks/pull/38443) * Some operations during data loading may encounter "reached timeout". [#36746](https://github.com/StarRocks/starrocks/pull/36746) * The DECIMAL type returned by SHOW CREATE TABLE is inconsistent with that specified in CREATE TABLE. [#39297](https://github.com/StarRocks/starrocks/pull/39297) * If partition columns in external tables contain null values, queries against those tables will cause BEs to crash. [#38888](https://github.com/StarRocks/starrocks/pull/38888) * When deleting data from a Duplicate Key table, if the condition in the WHERE clause of the DELETE statement has a leading space, the deleted data can still be queried using SELECT. [#39797](https://github.com/StarRocks/starrocks/pull/39797) * Loading `array` data from ORC files into StarRocks (`array`) may cause BEs to crash. [#39233](https://github.com/StarRocks/starrocks/pull/39233) * Querying Hive catalogs may be stuck and even expire. [#39863](https://github.com/StarRocks/starrocks/pull/39863) * Partitions cannot be dynamically created if hour-level partitions are specified in the PARTITION BY clause. [#40256](https://github.com/StarRocks/starrocks/pull/40256) * The error message "failed to call frontend service" is returned during loading from Apache Flink. [#40710](https://github.com/StarRocks/starrocks/pull/40710) #### 2.5.18[​](#2518 "Direct link to 2.5.18") Release date: Jan 10, 2024 ##### New Features[​](#new-features-1 "Direct link to New Features") * Users can set or modify session variables when they [CREATE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/CREATE_MATERIALIZED_VIEW/#parameters) or [ALTER](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/ALTER_MATERIALIZED_VIEW/) asynchronous materialized views. [#37401](https://github.com/StarRocks/starrocks/pull/37401) ##### Improvements[​](#improvements-4 "Direct link to Improvements") * When using JDK, the default GC algorithm is G1. [#37498](https://github.com/StarRocks/starrocks/pull/37498) * The result returned by the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/SHOW_ROUTINE_LOAD/) statement now includes the timestamps of consumption messages from each partition. [#36222](https://github.com/StarRocks/starrocks/pull/36222) ##### Behavior Change[​](#behavior-change "Direct link to Behavior Change") * Added the session variable `enable_materialized_view_for_insert`, which controls whether materialized views rewrite the queries in INSERT INTO SELECT statements. The default value is `false`. [#37505](https://github.com/StarRocks/starrocks/pull/37505) * Added the session variable `enable_strict_order_by`. When this variable is set to the default value `TRUE`, an error is reported for such a query pattern: Duplicate alias is used in different expressions of the query and this alias is also a sorting field in ORDER BY, for example, `select distinct t1.* from tbl1 t1 order by t1.k1;`. The logic is the same as that in v2.3 and earlier. When this variable is set to `FALSE`, a loose deduplication mechanism is used, which processes such queries as valid SQL queries. [#37910](https://github.com/StarRocks/starrocks/pull/37910) ##### Parameter Change[​](#parameter-change "Direct link to Parameter Change") * Added session variables `transaction_read_only` and `tx_read_only` to specify the transaction access mode, which are compatible with MySQL versions 5.7.20 and above. [#37249](https://github.com/StarRocks/starrocks/pull/37249) * Added the FE configuration item `routine_load_unstable_threshold_second`. [#36222](https://github.com/StarRocks/starrocks/pull/36222) * Added the FE configuration item `http_worker_threads_num`, which specifies the number of threads for HTTP server to deal with HTTP requests. The default value is `0`. If the value for this parameter is set to a negative value or 0, the actual thread number is twice the number of CPU cores. [#37530](https://github.com/StarRocks/starrocks/pull/37530) * Added the BE configuration item `pindex_major_compaction_limit_per_disk` to configure the maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. The default value is `1`. [#37695](https://github.com/StarRocks/starrocks/pull/37695) ##### Bug Fixes[​](#bug-fixes-4 "Direct link to Bug Fixes") Fixed the following issues: * Using NaN (Not a Number) columns as ORDER BY columns may cause BEs to crash. [#30759](https://github.com/StarRocks/starrocks/pull/30759) * Failure to update primary key indexes may cause the error "get\_applied\_rowsets failed". [#27488](https://github.com/StarRocks/starrocks/pull/27488) * Hive metadata in [Hive catalogs](https://docs.starrocks.io/docs/2.5/data_source/catalog/hive_catalog/) is not automatically refreshed when new fields are added to Hive tables. [#37668](https://github.com/StarRocks/starrocks/pull/37668) * When `SELECT ... FROM ... INTO OUTFILE` is executed to export data into CSV files, the error "Unmatched number of columns" is reported if the FROM clause contains multiple constants. [#38045](https://github.com/StarRocks/starrocks/pull/38045) * In some cases, `bitmap_to_string` may return incorrect result due to data type overflow. [#37405](https://github.com/StarRocks/starrocks/pull/37405) #### 2.5.17[​](#2517 "Direct link to 2.5.17") Release date: December 19, 2023 ##### New Features[​](#new-features-2 "Direct link to New Features") * Added a new metric `max_tablet_rowset_num` for setting the maximum allowed number of rowsets. This metric helps detect possible compaction issues and thus reduces the occurrences of the error "too many versions". [#36539](https://github.com/StarRocks/starrocks/pull/36539) * Added the [subdivide\_bitmap](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/subdivide_bitmap/) function. [#35817](https://github.com/StarRocks/starrocks/pull/35817) ##### Improvements[​](#improvements-5 "Direct link to Improvements") * The result returned by the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/SHOW_ROUTINE_LOAD/) statement provides a new field `OtherMsg`, which shows information about the last failed task. [#35806](https://github.com/StarRocks/starrocks/pull/35806) * The default retention period of trash files is changed to 1 day from the original 3 days. [#37113](https://github.com/StarRocks/starrocks/pull/37113) * Optimized the performance of persistent index update when compaction is performed on all rowsets of a Primary Key table, which reduces disk read I/O. [#36819](https://github.com/StarRocks/starrocks/pull/36819) * Optimized the logic used to compute compaction scores for Primary Key tables, thereby aligning the compaction scores for Primary Key tables within a more consistent range with the other three table types. [#36534](https://github.com/StarRocks/starrocks/pull/36534) * Queries on MySQL external tables and the external tables within JDBC catalogs support including keywords in the WHERE clause. [#35917](https://github.com/StarRocks/starrocks/pull/35917) * Added the bitmap\_from\_binary function to Spark Load to support loading Binary data. [#36050](https://github.com/StarRocks/starrocks/pull/36050) * The bRPC expiration time is shortened from 1 hour to the duration specified by the session variable [`query_timeout`](https://docs.starrocks.io/zh/docs/3.2/reference/System_variable/#query_timeout). This prevents query failures caused by RPC request expiration. [#36778](https://github.com/StarRocks/starrocks/pull/36778) ##### Parameter Change[​](#parameter-change-1 "Direct link to Parameter Change") * Added a BE configuration item `enable_stream_load_verbose_log` is added. The default value is `false`. With this parameter set to `true`, StarRocks can record the HTTP requests and responses for Stream Load jobs, making troubleshooting easier. [#36113](https://github.com/StarRocks/starrocks/pull/36113) * Added a BE static parameter `update_compaction_per_tablet_min_interval_seconds` becomes mutable. [#36819](https://github.com/StarRocks/starrocks/pull/36819) ##### Bug Fixes[​](#bug-fixes-5 "Direct link to Bug Fixes") Fixed the following issues: * Queries fail during hash joins, causing BEs to crash. [#32219](https://github.com/StarRocks/starrocks/pull/32219) * The FE performance plunges after the FE configuration item `enable_collect_query_detail_info` is set to `true`. [#35945](https://github.com/StarRocks/starrocks/pull/35945) * Errors may be thrown if large amounts of data are loaded into a Primary Key table with persistent index enabled. [#34352](https://github.com/StarRocks/starrocks/pull/34352) * The starrocks\_be process may exit unexpectedly when `./agentctl.sh stop be` is used to stop a BE. [#35108](https://github.com/StarRocks/starrocks/pull/35108) * The [array\_distinct](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_distinct/) function occasionally causes the BEs to crash. [#36377](https://github.com/StarRocks/starrocks/pull/36377) * Deadlocks may occur when users refresh materialized views. [#35736](https://github.com/StarRocks/starrocks/pull/35736) * In some scenarios, dynamic partitioning may encounter an error, which causes FE start failures. [#36846](https://github.com/StarRocks/starrocks/pull/36846) #### 2.5.16[​](#2516 "Direct link to 2.5.16") Release date: December 1, 2023 ##### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") Fixed the following issues: * Global Runtime Filter may cause BEs to crash in certain scenarios. [#35776](https://github.com/StarRocks/starrocks/pull/35776) #### 2.5.15[​](#2515 "Direct link to 2.5.15") Release date: November 29, 2023 ##### Improvements[​](#improvements-6 "Direct link to Improvements") * Added slow request logs to track slow requests. [#33908](https://github.com/StarRocks/starrocks/pull/33908) * Optimized the performance of using Spark Load to read Parquet and ORC files when there are a large number of files. [#34787](https://github.com/StarRocks/starrocks/pull/34787) * Optimized the performance of some Bitmap-related operations, including: * Optimized nested loop joins. [#340804](https://github.com/StarRocks/starrocks/pull/34804) [#35003](https://github.com/StarRocks/starrocks/pull/35003) * Optimized the `bitmap_xor` function. [#34069](https://github.com/StarRocks/starrocks/pull/34069) * Supports Copy on Write to optimize Bitmap performance and reduce memory consumption. [#34047](https://github.com/StarRocks/starrocks/pull/34047) ##### Parameter Change[​](#parameter-change-2 "Direct link to Parameter Change") * The FE dynamic parameter `enable_new_publish_mechanism` is changed to a static parameter. You must restart the FE after you modify the parameter settings. [#35338](https://github.com/StarRocks/starrocks/pull/35338) ##### Bug Fixes[​](#bug-fixes-7 "Direct link to Bug Fixes") * If a filtering condition is specified in a Broker Load job, BEs may crash during the data loading in certain circumstances. [#29832](https://github.com/StarRocks/starrocks/pull/29832) * Failures in replaying replica operations may cause FEs to crash. [#32295](https://github.com/StarRocks/starrocks/pull/32295) * Setting the FE parameter `recover_with_empty_tablet` to `true` may cause FEs to crash. [#33071](https://github.com/StarRocks/starrocks/pull/33071) * The error "get\_applied\_rowsets failed, tablet updates is in error state: tablet:18849 actual row size changed after compaction" is returned for queries. [#33246](https://github.com/StarRocks/starrocks/pull/33246) * A query that contains a window function may cause BEs to crash. [#33671](https://github.com/StarRocks/starrocks/pull/33671) * Running `show proc '/statistic'` may cause a deadlock. [#34237](https://github.com/StarRocks/starrocks/pull/34237/files) * Errors may be thrown if large amounts of data are loaded into a Primary Key table with persistent index enabled. [#34566](https://github.com/StarRocks/starrocks/pull/34566) * After StarRocks is upgraded from v2.4 or earlier to a later version, compaction scores may rise unexpectedly. [#34618](https://github.com/StarRocks/starrocks/pull/34618) * If `INFORMATION_SCHEMA` is queried by using the database driver MariaDB ODBC, the `CATALOG_NAME` column returned in the `schemata` view holds only `null` values. [#34627](https://github.com/StarRocks/starrocks/pull/34627) * If schema changes are being executed while a Stream Load job is in the **PREPARED** state, a portion of the source data to be loaded by the job is lost. [#34381](https://github.com/StarRocks/starrocks/pull/34381) * Including two or more slashes (`/`) at the end of the HDFS storage path causes the backup and restore of the data from HDFS to fail. [#34601](https://github.com/StarRocks/starrocks/pull/34601) * Running a loading task or a query may cause the FEs to hang. [#34569](https://github.com/StarRocks/starrocks/pull/34569) #### 2.5.14[​](#2514 "Direct link to 2.5.14") Release date: November 14, 2023 ##### Improvements[​](#improvements-7 "Direct link to Improvements") * The `COLUMNS` table in the system database `INFORMATION_SCHEMA` can display ARRAY, MAP, and STRUCT columns. [#33431](https://github.com/StarRocks/starrocks/pull/33431) ##### Parameter change[​](#parameter-change-3 "Direct link to Parameter change") ###### System variables[​](#system-variables "Direct link to System variables") * Added a session variable `cbo_decimal_cast_string_strict`, which controls how the CBO converts data from the DECIMAL type to the STRING type. If this variable is set to `true`, the logic built in v2.5.x and later versions prevails and the system implements strict conversion (namely, the system truncates the generated string and fills 0s based on the scale length). If this variable is set to `false`, the logic built in versions earlier than v2.5.x prevails and the system processes all valid digits to generate a string. The default value is `true`. [#34208](https://github.com/StarRocks/starrocks/pull/34208) * Added a session variable `cbo_eq_base_type`, which specifies the data type used for data comparison between DECIMAL-type data and STRING-type data. The default value is `VARCHAR`, and DECIMAL is also a valid value. [#34208](https://github.com/StarRocks/starrocks/pull/34208) ##### Bug Fixes[​](#bug-fixes-8 "Direct link to Bug Fixes") Fixed the following issues: * The error `java.lang.IllegalStateException: null` is reported if the ON condition is nested with a subquery. [#30876](https://github.com/StarRocks/starrocks/pull/30876) * The result of COUNT(*) is inconsistent among replicas if COUNT(*) is run immediately after `INSERT INTO SELECT ... LIMIT` is successfully executed. [#24435](https://github.com/StarRocks/starrocks/pull/24435) * BE may crash for specific data types if the target data type specified in the cast() function is the same as the original data type. [#31465](https://github.com/StarRocks/starrocks/pull/31465) * An error is reported if specific path formats are used during data loading via Broker Load: `msg:Fail to parse columnsFromPath, expected: [rec_dt]`. [#32721](https://github.com/StarRocks/starrocks/issues/32721) * During an upgrade to 3.x, if some column types are also upgraded (for example, Decimal is upgraded to Decimal v3), BEs crash when Compaction is performed on tables with specific characteristics. [#31626](https://github.com/StarRocks/starrocks/pull/31626) * When data is loaded by using Flink Connector, the load job is suspended unexpectedly if there are highly concurrent load jobs and both the number of HTTP and Scan threads have reached their upper limits. [#32251](https://github.com/StarRocks/starrocks/pull/32251) * BEs crash when libcurl is invoked. [#31667](https://github.com/StarRocks/starrocks/pull/31667) * Adding BITMAP columns to a Primary Key table fails with the following error: `Analyze columnDef error: No aggregate function specified for 'userid'`. [#31763](https://github.com/StarRocks/starrocks/pull/31763) * Long-time, frequent data loading into a Primary Key table with persistent index enabled may cause BEs to crash. [#33220](https://github.com/StarRocks/starrocks/pull/33220) * The query result is incorrect when Query Cache is enabled. [#32778](https://github.com/StarRocks/starrocks/pull/32778) * Specifying a nullable Sort Key when creating a Primary Key table causes compaction to fail. [#29225](https://github.com/StarRocks/starrocks/pull/29225) * The error "StarRocks planner use long time 10000 ms in logical phase" occasionally occurs for complex Join queries. [#34177](https://github.com/StarRocks/starrocks/pull/34177) #### 2.5.13[​](#2513 "Direct link to 2.5.13") Release date: September 28, 2023 ##### Improvements[​](#improvements-8 "Direct link to Improvements") * Window functions COVAR\_SAMP, COVAR\_POP, CORR, VARIANCE, VAR\_SAMP, STD, and STDDEV\_SAMP now support the ORDER BY clause and Window clause. [#30786](https://github.com/StarRocks/starrocks/pull/30786) * An error instead of NULL is returned if a decimal overflow occurs during queries on the DECIMAL type data. [#30419](https://github.com/StarRocks/starrocks/pull/30419) * Executing SQL commands with invalid comments now returns results consistent with MySQL. [#30210](https://github.com/StarRocks/starrocks/pull/30210) * Rowsets corresponding to tablets that have been deleted are cleaned up, reducing the memory usage during BE startup. [#30625](https://github.com/StarRocks/starrocks/pull/30625) ##### Bug Fixes[​](#bug-fixes-9 "Direct link to Bug Fixes") Fixed the following issues: * An error "Set cancelled by MemoryScratchSinkOperator" occurs when users read data from StarRocks using the Spark Connector or Flink Connector. [#30702](https://github.com/StarRocks/starrocks/pull/30702) [#30751](https://github.com/StarRocks/starrocks/pull/30751) * An error "java.lang.IllegalStateException: null" occurs during queries with an ORDER BY clause that includes aggregate functions. [#30108](https://github.com/StarRocks/starrocks/pull/30108) * FEs fail to restart when there are inactive materialized views. [#30015](https://github.com/StarRocks/starrocks/pull/30015) * Performing INSERT OVERWRITE operations on duplicate partitions corrupts the metadata, leading to FE restart failures. [#27545](https://github.com/StarRocks/starrocks/pull/27545) * An error "java.lang.NullPointerException: null" occurs when users modify columns that do not exist in a Primary Key table. [#30366](https://github.com/StarRocks/starrocks/pull/30366) * An error "get TableMeta failed from TNetworkAddress" occurs when users load data into a partitioned StarRocks external table. [#30124](https://github.com/StarRocks/starrocks/pull/30124) * In certain scenarios, an error occurs when users load data via CloudCanal. [#30799](https://github.com/StarRocks/starrocks/pull/30799) * An error "current running txns on db xxx is 200, larger than limit 200" occurs when users load data via the Flink Connector or perform DELETE and INSERT operations. [#18393](https://github.com/StarRocks/starrocks/pull/18393) * Asynchronous materialized views which use HAVING clauses that include aggregate functions cannot rewrite queries properly. [#29976](https://github.com/StarRocks/starrocks/pull/29976) #### 2.5.12[​](#2512 "Direct link to 2.5.12") Release date: September 4, 2023 ##### Improvements[​](#improvements-9 "Direct link to Improvements") * Comments in an SQL are retained in the Audit Log. [#29747](https://github.com/StarRocks/starrocks/pull/29747) * Added CPU and memory statistics of INSERT INTO SELECT to the Audit Log. [#29901](https://github.com/StarRocks/starrocks/pull/29901) ##### Bug Fixes[​](#bug-fixes-10 "Direct link to Bug Fixes") Fixed the following issues: * When Broker Load is used to load data, the NOT NULL attribute of some fields may cause BEs to crash or cause the "msg:mismatched row count" error. [#29832](https://github.com/StarRocks/starrocks/pull/29832) * Queries against ORC-formatted files fail because the bugfix ORC-1304 ([apache/orc#1299](https://github.com/apache/orc/pull/1299)) from Apache ORC is not merged. [#29804](https://github.com/StarRocks/starrocks/pull/29804) * Restoring Primary Key tables causes metadata inconsistency after BEs are restarted. [#30135](https://github.com/StarRocks/starrocks/pull/30135) #### 2.5.11[​](#2511 "Direct link to 2.5.11") Release date: August 28, 2023 ##### Improvements[​](#improvements-10 "Direct link to Improvements") * Supports implicit conversions for all compound predicates and for all expressions in the WHERE clause. You can enable or disable implicit conversions by using the [session variable](https://docs.starrocks.io/docs/sql-reference/System_variable/#enable_strict_type) `enable_strict_type`. The default value is `false`. [#21870](https://github.com/StarRocks/starrocks/pull/21870) * Optimized the prompt returned if users do not specify `hive.metastore.uri` when they create an Iceberg Catalog. The error prompt is more accurate. [#16543](https://github.com/StarRocks/starrocks/issues/16543) * Added more prompts in the error message `xxx too many versions xxx`. [#28397](https://github.com/StarRocks/starrocks/pull/28397) * Dynamic partitioning further supports the partitioning unit to be `year`. [#28386](https://github.com/StarRocks/starrocks/pull/28386) ##### Bug Fixes[​](#bug-fixes-11 "Direct link to Bug Fixes") Fixed the following issues: * When data is loaded into tables with multiple replicas, a large number of invalid log records are written if some partitions of the tables are empty. [#28824](https://github.com/StarRocks/starrocks/issues/28824) * The DELETE operation fails if the field in the WHERE condition is a BITMAP or HLL field. [#28592](https://github.com/StarRocks/starrocks/pull/28592) * Manually refreshing an asynchronous materialized view via a synchronous call (SYNC MODE) results in multiple INSERT OVERWRITE records in the `information_schema.task_runs` table. [#28060](https://github.com/StarRocks/starrocks/pull/28060) * If CLONE operations are triggered on tablets in an ERROR state, disk usage increases. [#28488](https://github.com/StarRocks/starrocks/pull/28488) * When Join Reorder is enabled, the query result is incorrect if the column to query is a constant. [#29239](https://github.com/StarRocks/starrocks/pull/29239) * During tablet migration between SSDs and HDDs, if the FE sends excessive migration tasks to BEs, BEs will encounter OOM issues. [#29055](https://github.com/StarRocks/starrocks/pull/29055) * The security vulnerability in `/apache_hdfs_broker/lib/log4j-1.2.17.jar`. [#28866](https://github.com/StarRocks/starrocks/pull/28866) * During data queries through Hive Catalog, if a partitioning column and an OR operator are used in the WHERE clause, the query result is incorrect. [#28876](https://github.com/StarRocks/starrocks/pull/28876) * The error "java.util.ConcurrentModificationException: null" occasionally occurs during data queries. [#29296](https://github.com/StarRocks/starrocks/pull/29296) * FEs cannot be restarted if the base table of an asynchronous materialized view is dropped. [#29318](https://github.com/StarRocks/starrocks/pull/29318) * For an asynchronous materialized view that is created across databases, the Leader FE occasionally encounters a deadlock when data is being written into base tables of this materialized view. [#29432](https://github.com/StarRocks/starrocks/pull/29432) #### 2.5.10[​](#2510 "Direct link to 2.5.10") Release date: August 7, 2023 ##### New features[​](#new-features-3 "Direct link to New features") * Supports aggregate functions [COVAR\_SAMP](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/covar_samp/), [COVAR\_POP](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/covar_pop/), and [CORR](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/corr/). * Supports the following [window functions](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function/): COVAR\_SAMP, COVAR\_POP, CORR, VARIANCE, VAR\_SAMP, STD, and STDDEV\_SAMP. ##### Improvements[​](#improvements-11 "Direct link to Improvements") * Optimized the scheduling logic of TabletChecker to prevent the checker from repeatedly scheduling tablets that are not repaired. [#27648](https://github.com/StarRocks/starrocks/pull/27648) * When Schema Change and Routine Load occur simultaneously, Routine Load jobs may fail if Schema Change completes first. The error message reported in this situation is optimized. [#28425](https://github.com/StarRocks/starrocks/pull/28425) * Users are prohibited from defining NOT NULL columns when they create external tables (If NOT NULL columns are defined, errors will occur after an upgrade and the table must be created again). External catalogs are recommended starting from v2.3.0 to replace external tables. [#25485](https://github.com/StarRocks/starrocks/pull/25441) * Added an error message when Broker Load retries encounter an error. This facilitates troubleshooting and debugging during data loading. [#21982](https://github.com/StarRocks/starrocks/pull/21982) * Supports large-scale data writes when a load job involves both UPSERT and DELETE operations. [#17264](https://github.com/StarRocks/starrocks/pull/17264) * Optimized query rewrite using materialized views. [#27934](https://github.com/StarRocks/starrocks/pull/27934) [#25542](https://github.com/StarRocks/starrocks/pull/25542) [#22300](https://github.com/StarRocks/starrocks/pull/22300) [#27557](https://github.com/StarRocks/starrocks/pull/27557) [#22300](https://github.com/StarRocks/starrocks/pull/22300) [#26957](https://github.com/StarRocks/starrocks/pull/26957) [#27728](https://github.com/StarRocks/starrocks/pull/27728) [#27900](https://github.com/StarRocks/starrocks/pull/27900) ##### Bug Fixes[​](#bug-fixes-12 "Direct link to Bug Fixes") Fixed the following issues: * When CAST is used to convert a string into an array, the result may be incorrect if the input includes constants. [#19793](https://github.com/StarRocks/starrocks/pull/19793) * SHOW TABLET returns incorrect results if it contains ORDER BY and LIMIT. [#23375](https://github.com/StarRocks/starrocks/pull/23375) * Outer join and Anti join rewrite errors for materialized views. [#28028](https://github.com/StarRocks/starrocks/pull/28028) * Incorrect table-level scan statistics in FE cause inaccurate metrics for table queries and loading. [#27779](https://github.com/StarRocks/starrocks/pull/27779) * `An exception occurred when using the current long link to access metastore. msg: Failed to get next notification based on last event id: 707602` is reported in FE logs if event listener is configured on the HMS to incrementally update Hive metadata. [#21056](https://github.com/StarRocks/starrocks/pull/21056) * The query result is not stable if the sort key is modified for a partitioned table. [#27850](https://github.com/StarRocks/starrocks/pull/27850) * Data loaded using Spark Load may be distributed to the wrong buckets if the bucketing column is a DATE, DATETIME, or DECIMAL column. [#27005](https://github.com/StarRocks/starrocks/pull/27005) * The regex\_replace function may cause BEs to crash in some scenarios. [#27117](https://github.com/StarRocks/starrocks/pull/27117) * BE crashes if the input of the sub\_bitmap function is not a BITMAP value. [#27982](https://github.com/StarRocks/starrocks/pull/27982) * "Unknown error" is returned for a query when Join Reorder is enabled. [#27472](https://github.com/StarRocks/starrocks/pull/27472) * Inaccurate estimation of average row size causes Primary Key partial updates to occupy excessively large memory. [#27485](https://github.com/StarRocks/starrocks/pull/27485) * Some INSERT jobs return `[42000][1064] Dict Decode failed, Dict can't take cover all key :0` if low-cardinality optimization is enabled. [#26463](https://github.com/StarRocks/starrocks/pull/26463) * If users specify `"hadoop.security.authentication" = "simple"` in their Broker Load jobs created to load data from HDFS, the job fails. [#27774](https://github.com/StarRocks/starrocks/pull/27774) * Modifying the refresh mode of materialized views causes inconsistent metadata between the leader FE and follower FE. [#28082](https://github.com/StarRocks/starrocks/pull/28082) [#28097](https://github.com/StarRocks/starrocks/pull/28097) * Passwords are not hidden when SHOW CREATE CATALOG and SHOW RESOURCES are used to query specific information. [#28059](https://github.com/StarRocks/starrocks/pull/28059) * FE memory leak caused by blocked LabelCleaner threads. [#28311](https://github.com/StarRocks/starrocks/pull/28311) #### 2.5.9[​](#259 "Direct link to 2.5.9") Release date: July 19, 2023 ##### New features[​](#new-features-4 "Direct link to New features") * Queries that contain a different type of join than the materialized view can be rewritten. [#25099](https://github.com/StarRocks/starrocks/pull/25099) ##### Improvements[​](#improvements-12 "Direct link to Improvements") * StarRocks external tables whose destination cluster is the current StarRocks cluster cannot be created. [#25441](https://github.com/StarRocks/starrocks/pull/25441) * If the queried fields are not included in the output columns of a materialized view but are included in the predicate of the materialized view, the query can still be rewritten. [#23028](https://github.com/StarRocks/starrocks/issues/23028) * Added a new field `table_id` to the table `tables_config` in the database `Information_schema`. You can join `tables_config` with `be_tablets` on the column `table_id` to query the names of the database and table to which a tablet belongs. [#24061](https://github.com/StarRocks/starrocks/pull/24061) ##### Bug Fixes[​](#bug-fixes-13 "Direct link to Bug Fixes") Fixed the following issues: * Count Distinct result is incorrect for Duplicate Key tables. [#24222](https://github.com/StarRocks/starrocks/pull/24222) * BEs may crash if the Join key is a large BINARY column. [#25084](https://github.com/StarRocks/starrocks/pull/25084) * The INSERT operation hangs if the length of CHAR data in a STRUCT to be inserted exceeds the maximum CHAR length defined in the STRUCT column. [#25942](https://github.com/StarRocks/starrocks/pull/25942) * The result of coalesce() is incorrect. [#26250](https://github.com/StarRocks/starrocks/pull/26250) * The version number for a tablet is inconsistent between the BE and FE after data is restored. [#26518](https://github.com/StarRocks/starrocks/pull/26518/files) * Partitions cannot be automatically created for recovered tables. [#26813](https://github.com/StarRocks/starrocks/pull/26813) #### 2.5.8[​](#258 "Direct link to 2.5.8") Release date: June 30, 2023 ##### Improvements[​](#improvements-13 "Direct link to Improvements") * Optimized the error message reported when partitions are added to a non-partitioned table. [#25266](https://github.com/StarRocks/starrocks/pull/25266) * Optimized the [auto tablet distribution policy](https://docs.starrocks.io/docs/2.5/table_design/Data_distribution/#determine-the-number-of-tablets) for tables. [#24543](https://github.com/StarRocks/starrocks/pull/24543) * Optimized the default comments in the CREATE TABLE statement. [#24803](https://github.com/StarRocks/starrocks/pull/24803) * Optimized the manual refreshing of asynchronous materialized views. Supports using the REFRESH MATERIALIZED VIEW WITH SYNC MODE syntax to synchronously invoke materialized view refresh tasks. [#25910](https://github.com/StarRocks/starrocks/pull/25910) ##### Bug Fixes[​](#bug-fixes-14 "Direct link to Bug Fixes") Fixed the following issues: * The COUNT result of an asynchronous materialized view may be inaccurate if the materialized view is built on Union results. [#24460](https://github.com/StarRocks/starrocks/issues/24460) * "Unknown error" is reported when users attempt to forcibly reset the root password. [#25492](https://github.com/StarRocks/starrocks/pull/25492) * Inaccurate error message is displayed when INSERT OVERWRITE is executed on a cluster with less than three alive BEs. [#25314](https://github.com/StarRocks/starrocks/pull/25314) #### 2.5.7[​](#257 "Direct link to 2.5.7") Release date: June 14, 2023 ##### New features[​](#new-features-5 "Direct link to New features") * Inactive materialized views can be manually activated using `ALTER MATERIALIZED VIEW ACTIVE`. You can use this SQL command to activate materialized views whose base tables were dropped and then recreated. For more information, see [ALTER MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/ALTER_MATERIALIZED_VIEW/). [#24001](https://github.com/StarRocks/starrocks/pull/24001) * StarRocks can automatically set an appropriate number of tablets when you create a table or add a partition, eliminating the need for manual operations. For more information, see [Determine the number of tablets](https://docs.starrocks.io/docs/2.5/table_design/Data_distribution/#determine-the-number-of-tablets). [#10614](https://github.com/StarRocks/starrocks/pull/10614) ##### Improvements[​](#improvements-14 "Direct link to Improvements") * Optimized the I/O concurrency of Scan nodes used in external table queries, which reduces memory usage and improves the stability of data loading from external tables. [#23617](https://github.com/StarRocks/starrocks/pull/23617) [#23624](https://github.com/StarRocks/starrocks/pull/23624) [#23626](https://github.com/StarRocks/starrocks/pull/23626) * Optimized the error message for Broker Load jobs. The error message contains retry information and the name of erroneous files. [#18038](https://github.com/StarRocks/starrocks/pull/18038) [#21982](https://github.com/StarRocks/starrocks/pull/21982) * Optimized the error message returned when CREATE TABLE times out and added parameter tuning tips. [#24510](https://github.com/StarRocks/starrocks/pull/24510) * Optimized the error message returned when ALTER TABLE fails because the table status is not Normal. [#24381](https://github.com/StarRocks/starrocks/pull/24381) * Ignores full-width spaces in the CREATE TABLE statement. [#23885](https://github.com/StarRocks/starrocks/pull/23885) * Optimized the Broker access timeout to increase the success rate of Broker Load jobs. [#22699](https://github.com/StarRocks/starrocks/pull/22699) * For Primary Key tables, the `VersionCount` field returned by SHOW TABLET contains Rowsets that are in the Pending state. [#23847](https://github.com/StarRocks/starrocks/pull/23847) * Optimized the Persistent Index policy. [#22140](https://github.com/StarRocks/starrocks/pull/22140) ##### Bug Fixes[​](#bug-fixes-15 "Direct link to Bug Fixes") Fixed the following issues: * When users load Parquet data into StarRocks, DATETIME values overflow during type conversion, causing data errors. [#22356](https://github.com/StarRocks/starrocks/pull/22356) * Bucket information is lost after Dynamic Partitioning is disabled. [#22595](https://github.com/StarRocks/starrocks/pull/22595) * Using unsupported properties in the CREATE TABLE statement causes null pointer exceptions (NPEs). [#23859](https://github.com/StarRocks/starrocks/pull/23859) * Table permission filtering in `information_schema` becomes ineffective. As a result, users can view tables they do not have permission to. [#23804](https://github.com/StarRocks/starrocks/pull/23804) * Information returned by SHOW TABLE STATUS is incomplete. [#24279](https://github.com/StarRocks/starrocks/issues/24279) * A schema change sometimes may be hung if data loading occurs simultaneously with the schema change. [#23456](https://github.com/StarRocks/starrocks/pull/23456) * RocksDB WAL flush blocks the brpc worker from processing bthreads, which interrupts high-frequency data loading into Primary Key tables. [#22489](https://github.com/StarRocks/starrocks/pull/22489) * TIME-type columns that are not supported in StarRocks can be successfully created. [#23474](https://github.com/StarRocks/starrocks/pull/23474) * Materialized view Union rewrite fails. [#22922](https://github.com/StarRocks/starrocks/pull/22922) #### 2.5.6[​](#256 "Direct link to 2.5.6") Release date: May 19, 2023 ##### Improvements[​](#improvements-15 "Direct link to Improvements") * Optimized the error message reported when INSERT INTO ... SELECT expires due to a small `thrift_server_max_worker_thread` value. [#21964](https://github.com/StarRocks/starrocks/pull/21964) * Tables created using CTAS have three replicas by default, which is consistent with the default replica number for common tables. [#22854](https://github.com/StarRocks/starrocks/pull/22854) ##### Bug Fixes[​](#bug-fixes-16 "Direct link to Bug Fixes") * Truncating partitions fails because the TRUNCATE operation is case-sensitive to partition names. [#21809](https://github.com/StarRocks/starrocks/pull/21809) * Decommissioning BE fails due to the failure in creating temporary partitions for materialized views. [#22745](https://github.com/StarRocks/starrocks/pull/22745) * Dynamic FE parameters that require an ARRAY value cannot be set to an empty array. [#22225](https://github.com/StarRocks/starrocks/pull/22225) * Materialized views with the `partition_refresh_number` property specified may fail to completely refresh. [#21619](https://github.com/StarRocks/starrocks/pull/21619) * SHOW CREATE TABLE masks cloud credential information, which causes incorrect credential information in memory. [#21311](https://github.com/StarRocks/starrocks/pull/21311) * Predicates cannot take effect on some ORC files that are queried via external tables. [#21901](https://github.com/StarRocks/starrocks/pull/21901) * The min-max filter cannot properly handle lower- and upper-case letters in column names. [#22626](https://github.com/StarRocks/starrocks/pull/22626) * Late materialization causes errors in querying complex data types (STRUCT or MAP). [#22862](https://github.com/StarRocks/starrocks/pull/22862) * The issue that occurs when restoring a Primary Key table. [#23384](https://github.com/StarRocks/starrocks/pull/23384) #### 2.5.5[​](#255 "Direct link to 2.5.5") Release date: April 28, 2023 ##### New features[​](#new-features-6 "Direct link to New features") Added a metric to monitor the tablet status of Primary Key tables: * Added the FE metric `err_state_metric`. * Added the `ErrorStateTabletNum` column to the output of `SHOW PROC '/statistic/'` to display the number of **err\_state** tablets. * Added the `ErrorStateTablets` column to the output of `SHOW PROC '/statistic//'` to display the IDs of **err\_state** tablets. For more information, see [SHOW PROC](https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/SHOW_PROC/). ##### Improvements[​](#improvements-16 "Direct link to Improvements") * Optimized the disk balancing speed when multiple BEs are added. [# 19418](https://github.com/StarRocks/starrocks/pull/19418) * Optimized the inference of `storage_medium`. When BEs use both SSD and HDD as storage devices, if the property `storage_cooldown_time` is specified, StarRocks sets `storage_medium` to `SSD`. Otherwise, StarRocks sets `storage_medium` to `HDD`. [#18649](https://github.com/StarRocks/starrocks/pull/18649) * Optimized the performance of Unique Key tables by forbidding the collection of statistics from value columns. [#19563](https://github.com/StarRocks/starrocks/pull/19563) ##### Bug Fixes[​](#bug-fixes-17 "Direct link to Bug Fixes") * For Colocation tables, the replica status can be manually specified as `bad` by using statements like `ADMIN SET REPLICA STATUS PROPERTIES ("tablet_id" = "10003", "backend_id" = "10001", "status" = "bad");`. If the number of BEs is less than or equal to the number of replicas, the corrupted replica cannot be repaired. [# 17876](https://github.com/StarRocks/starrocks/issues/17876) * After a BE is started, its process exists but the BE port cannot be enabled. [# 19347](https://github.com/StarRocks/starrocks/pull/19347) * Wrong results are returned for aggregate queries whose subquery is nested with a window function. [# 19725](https://github.com/StarRocks/starrocks/issues/19725) * `auto_refresh_partitions_limit` does not take effect when the materialized view (MV) is refreshed for the first time. As a result, all the partitions are refreshed. [# 19759](https://github.com/StarRocks/starrocks/issues/19759) * An error occurs when querying a CSV Hive external table whose array data is nested with complex data such as MAP and STRUCT. [# 20233](https://github.com/StarRocks/starrocks/pull/20233) * Queries that use Spark connector time out. [# 20264](https://github.com/StarRocks/starrocks/pull/20264) * If one replica of a two-replica table is corrupted, the table cannot recover. [# 20681](https://github.com/StarRocks/starrocks/pull/20681) * Query failure caused by MV query rewrite failure. [# 19549](https://github.com/StarRocks/starrocks/issues/19549) * The metric interface expires due to database lock. [# 20790](https://github.com/StarRocks/starrocks/pull/20790) * Wrong results are returned for Broadcast Join. [# 20952](https://github.com/StarRocks/starrocks/issues/20952) * NPE is returned when an unsupported data type is used in CREATE TABLE. [# 20999](https://github.com/StarRocks/starrocks/issues/20999) * The issue caused by using window\_funnel() with the Query Cache feature. [# 21474](https://github.com/StarRocks/starrocks/issues/21474) * Optimization plan selection takes an unexpectedly long time after the CTE is rewritten. [# 16515](https://github.com/StarRocks/starrocks/pull/16515) #### 2.5.4[​](#254 "Direct link to 2.5.4") Release date: April 4, 2023 ##### Improvements[​](#improvements-17 "Direct link to Improvements") * Optimized the performance of rewriting queries on materialized views during query planning. The amount of time taken for query planning is reduced by about 70%. [#19579](https://github.com/StarRocks/starrocks/pull/19579) * Optimized the type inference logic. If a query like `SELECT sum(CASE WHEN XXX);` contains a constant `0`, such as `SELECT sum(CASE WHEN k1 = 1 THEN v1 ELSE 0 END) FROM test;`, pre-aggregation is automatically enabled to accelerate the query. [#19474](https://github.com/StarRocks/starrocks/pull/19474) * Supports using `SHOW CREATE VIEW` to view the creation statement of a materialized view. [#19999](https://github.com/StarRocks/starrocks/pull/19999) * Supports transmitting packets that are 2 GB or larger in size for a single bRPC request between BE nodes. [#20283](https://github.com/StarRocks/starrocks/pull/20283) [#20230](https://github.com/StarRocks/starrocks/pull/20230) * Supports using [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/SHOW_CREATE_CATALOG/) to query the creation statement of an external catalog. ##### Bug Fixes[​](#bug-fixes-18 "Direct link to Bug Fixes") The following bugs are fixed: * After queries on materialized views are rewritten, the global dictionary for low-cardinality optimization does not take effect. [#19615](https://github.com/StarRocks/starrocks/pull/19615) * If a query on materialized views fails to be rewritten, the query fails. [#19774](https://github.com/StarRocks/starrocks/pull/19774) * If a materialized view is created based on a Primary Key or Unique Key table, queries on that materialized view cannot be rewritten. [#19600](https://github.com/StarRocks/starrocks/pull/19600) * The column names of materialized views are case-sensitive. However, when you create a table, the table is successfully created without an error message even if column names are incorrect in the `PROPERTIES` of the table creation statement, and moreover the rewriting of queries on materialized views created on that table fails. [#19780](https://github.com/StarRocks/starrocks/pull/19780) * After a query on materialized views is rewritten, the query plan ma contain partition column-based, invalid predicates, which affect query performance. [#19784](https://github.com/StarRocks/starrocks/pull/19784) * When data is loaded into a newly created partition, queries on materialized views may fail to be rewritten. [#20323](https://github.com/StarRocks/starrocks/pull/20323) * Configuring `"storage_medium" = "SSD"` at the creation of materialized views causes the refresh of the materialized views to fail. [#19539](https://github.com/StarRocks/starrocks/pull/19539) [#19626](https://github.com/StarRocks/starrocks/pull/19626) * Concurrent compaction may happen on Primary Key tables. [#19692](https://github.com/StarRocks/starrocks/pull/19692) * Compaction does not occur promptly after a large number of DELETE operations. [#19623](https://github.com/StarRocks/starrocks/pull/19623) * If the expression of a statement contains multiple low-cardinality columns, the expression may fail to be properly rewritten. As a result, the global dictionary for low-cardinality optimization does not take effect. [#20161](https://github.com/StarRocks/starrocks/pull/20161) #### 2.5.3[​](#253 "Direct link to 2.5.3") Release date: March 10, 2023 ##### Improvements[​](#improvements-18 "Direct link to Improvements") * Optimized query rewrite for materialized views (MVs). * Supports rewriting queries with Outer Join and Cross Join. [#18629](https://github.com/StarRocks/starrocks/pull/18629) * Optimized the data scan logic for MVs, further accelerating the rewritten queries. [#18629](https://github.com/StarRocks/starrocks/pull/18629) * Enhanced rewrite capabilities for single-table aggregate queries. [#18629](https://github.com/StarRocks/starrocks/pull/18629) * Enhanced rewrite capabilities in View Delta scenarios, which is when the queried tables are a subset of the MV's base tables. [#18800](https://github.com/StarRocks/starrocks/pull/18800) * Optimized the performance and memory usage when the window function RANK() is used as a filter or a sort key. [#17553](https://github.com/StarRocks/starrocks/issues/17553) ##### Bug Fixes[​](#bug-fixes-19 "Direct link to Bug Fixes") The following bugs are fixed: * Errors caused by null literals `[]` in ARRAY data. [#18563](https://github.com/StarRocks/starrocks/pull/18563) * Misuse of the low-cardinality optimization dictionary in some complex query scenarios. The dictionary mapping check is now added before applying the dictionary. [#17318](https://github.com/StarRocks/starrocks/pull/17318) * In a single BE environment, Local Shuffle causes GROUP BY to produce duplicate results. [#17845](https://github.com/StarRocks/starrocks/pull/17845) * Misuses of partition-related PROPERTIES for a non-partitioned MV may cause the MV refresh to fail. The partition PROPERTIES check is now performed when users create an MV. [#18741](https://github.com/StarRocks/starrocks/pull/18741) * Errors in parsing Parquet Repetition columns. [#17626](https://github.com/StarRocks/starrocks/pull/17626) [#17788](https://github.com/StarRocks/starrocks/pull/17788) [#18051](https://github.com/StarRocks/starrocks/pull/18051) * The obtained column's nullable information is incorrect. Solution: When CTAS is used to create a Primary Key table, only the primary key columns are non-nullable; non-primary key columns are nullable. [#16431](https://github.com/StarRocks/starrocks/pull/16431) * Some issues caused by deleting data from Primary Key tables. [#18768](https://github.com/StarRocks/starrocks/pull/18768) #### 2.5.2[​](#252 "Direct link to 2.5.2") Release date: February 21, 2023 ##### New Features[​](#new-features-7 "Direct link to New Features") * Supports using the Instance Profile and Assumed Role-based credential methods to access AWS S3 and AWS Glue. [#15958](https://github.com/StarRocks/starrocks/pull/15958) * Supports the following bit functions: bit\_shift\_left, bit\_shift\_right, and bit\_shift\_right\_logical. [#14151](https://github.com/StarRocks/starrocks/pull/14151) ##### Improvements[​](#improvements-19 "Direct link to Improvements") * Optimized the memory release logic, which significantly reduces peak memory usage when a query contains a large number of aggregate queries. [#16913](https://github.com/StarRocks/starrocks/pull/16913) * Reduced the memory usage of sorting. The memory consumption is halved when a query involves window functions or sorting. [#16937](https://github.com/StarRocks/starrocks/pull/16937) [#17362](https://github.com/StarRocks/starrocks/pull/17362) [#17408](https://github.com/StarRocks/starrocks/pull/17408) ##### Bug Fixes[​](#bug-fixes-20 "Direct link to Bug Fixes") The following bugs are fixed: * Apache Hive external tables that contain MAP and ARRAY data cannot be refreshed. [#17548](https://github.com/StarRocks/starrocks/pull/17548) * Superset cannot identify column types of materialized views. [#17686](https://github.com/StarRocks/starrocks/pull/17686) * BI connectivity fails because SET GLOBAL/SESSION TRANSACTION cannot be parsed. [#17295](https://github.com/StarRocks/starrocks/pull/17295) * The bucket number of dynamic partitioned tables in a Colocate Group cannot be modified and an error message is returned. [#17418](https://github.com/StarRocks/starrocks/pull/17418/) * Potential issues caused by a failure in the Prepare stage. [#17323](https://github.com/StarRocks/starrocks/pull/17323) ##### Behavior Change[​](#behavior-change-1 "Direct link to Behavior Change") * Changed the default value of `enable_experimental_mv` from `false` to `true`, which means asynchronous materialized view is enabled by default. * Added CHARACTER to the reserved keyword list. [#17488](https://github.com/StarRocks/starrocks/pull/17488) #### 2.5.1[​](#251 "Direct link to 2.5.1") Release date: February 5, 2023 ##### Improvements[​](#improvements-20 "Direct link to Improvements") * Asynchronous materialized views created based on external catalogs support query rewrite. [#11116](https://github.com/StarRocks/starrocks/issues/11116) [#15791](https://github.com/StarRocks/starrocks/issues/15791) * Allows users to specify a collection period for automatic CBO statistics collection, which prevents cluster performance jitter caused by automatic full collection. [#14996](https://github.com/StarRocks/starrocks/pull/14996) * Added Thrift server queue. Requests that cannot be processed immediately during INSERT INTO SELECT can be pending in the Thrift server queue, preventing requests from being rejected. [#14571](https://github.com/StarRocks/starrocks/pull/14571) * Deprecated the FE parameter `default_storage_medium`. If `storage_medium` is not explicitly specified when users create a table, the system automatically infers the storage medium of the table based on BE disk type. For more information, see description of `storage_medium` in [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/CREATE_VIEW/). [#14394](https://github.com/StarRocks/starrocks/pull/14394) ##### Bug Fixes[​](#bug-fixes-21 "Direct link to Bug Fixes") The following bugs are fixed: * Null pointer exception (NPE) caused by SET PASSWORD. [#15247](https://github.com/StarRocks/starrocks/pull/15247) * JSON data with empty keys cannot be parsed. [#16852](https://github.com/StarRocks/starrocks/pull/16852) * Data of invalid types can be successfully converted into ARRAY data. [#16866](https://github.com/StarRocks/starrocks/pull/16866) * Nested Loop Join cannot be interrupted when an exception occurs. [#16875](https://github.com/StarRocks/starrocks/pull/16875) ##### Behavior Change[​](#behavior-change-2 "Direct link to Behavior Change") * Deprecated the FE parameter `default_storage_medium`. The storage medium of a table is automatically inferred by the system. [#14394](https://github.com/StarRocks/starrocks/pull/14394) #### 2.5.0[​](#250 "Direct link to 2.5.0") Release date: January 22, 2023 ##### New Features[​](#new-features-8 "Direct link to New Features") * Supports querying Merge On Read tables using [Hudi catalogs](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog/) and [Hudi external tables](https://docs.starrocks.io/docs/data_source/External_table#deprecated-hudi-external-table). [#6780](https://github.com/StarRocks/starrocks/pull/6780) * Supports querying STRUCT and MAP data using [Hive catalogs](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog/), Hudi catalogs, and [Iceberg catalogs](https://docs.starrocks.io/docs/data_source/catalog/iceberg_catalog/). [#10677](https://github.com/StarRocks/starrocks/issues/10677) * Provides [Data Cache](https://docs.starrocks.io/docs/data_source/data_cache/) to improve access performance of hot data stored in external storage systems, such as HDFS. [#11597](https://github.com/StarRocks/starrocks/pull/11579) * Supports creating [Delta Lake catalogs](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog/), which allow direct queries on data from Delta Lake. [#11972](https://github.com/StarRocks/starrocks/issues/11972) * Hive, Hudi, and Iceberg catalogs are compatible with AWS Glue. [#12249](https://github.com/StarRocks/starrocks/issues/12249) * Supports creating [file external tables](https://docs.starrocks.io/docs/data_source/file_external_table/), which allow direct queries on Parquet and ORC files from HDFS and object stores. [#13064](https://github.com/StarRocks/starrocks/pull/13064) * Supports creating materialized views based on Hive, Hudi, Iceberg catalogs, and materialized views. For more information, see [Materialized view](https://docs.starrocks.io/docs/using_starrocks/Materialized_view/). [#11116](https://github.com/StarRocks/starrocks/issues/11116) [#11873](https://github.com/StarRocks/starrocks/pull/11873) * Supports conditional updates for tables that use the Primary Key table. For more information, see [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables/). [#12159](https://github.com/StarRocks/starrocks/pull/12159) * Supports [Query Cache](https://docs.starrocks.io/docs/using_starrocks/query_cache/), which stores intermediate computation results of queries, improving the QPS and reduces the average latency of highly-concurrent, simple queries. [#9194](https://github.com/StarRocks/starrocks/pull/9194) * Supports specifying the priority of Broker Load jobs. For more information, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/BROKER_LOAD/) [#11029](https://github.com/StarRocks/starrocks/pull/11029) * Supports specifying the number of replicas for data loading for StarRocks native tables. For more information, see [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/CREATE_TABLE/). [#11253](https://github.com/StarRocks/starrocks/pull/11253) * Supports [query queues](https://docs.starrocks.io/docs/administration/query_queues/). [#12594](https://github.com/StarRocks/starrocks/pull/12594) * Supports isolating compute resources occupied by data loading, thereby limiting the resource consumption of data loading tasks. For more information, see [Resource group](https://docs.starrocks.io/docs/administration/resource_group/). [#12606](https://github.com/StarRocks/starrocks/pull/12606) * Supports specifying the following data compression algorithms for StarRocks native tables: LZ4, Zstd, Snappy, and Zlib. For more information, see [Data compression](https://docs.starrocks.io/docs/table_design/data_compression/). [#10097](https://github.com/StarRocks/starrocks/pull/10097) [#12020](https://github.com/StarRocks/starrocks/pull/12020) * Supports [user-defined variables](https://docs.starrocks.io/docs/reference/user_defined_variables/). [#10011](https://github.com/StarRocks/starrocks/pull/10011) * Supports [lambda expression](https://docs.starrocks.io/docs/sql-reference/sql-functions/Lambda_expression/) and the following higher-order functions: [array\_map](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_map/), [array\_sum](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_sum/), and [array\_sortby](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_sortby/). [#9461](https://github.com/StarRocks/starrocks/pull/9461) [#9806](https://github.com/StarRocks/starrocks/pull/9806) [#10323](https://github.com/StarRocks/starrocks/pull/10323) [#14034](https://github.com/StarRocks/starrocks/pull/14034) * Provides the QUALIFY clause that filters the results of [window functions](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function/). [#13239](https://github.com/StarRocks/starrocks/pull/13239) * Supports using the result returned by the uuid() and uuid\_numeric() functions as the default value of a column when you create a table. For more information, see [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/CREATE_TABLE/). [#11155](https://github.com/StarRocks/starrocks/pull/11155) * Supports the following functions: [map\_size](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_size/), [map\_keys](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_keys/), [map\_values](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_values/), [max\_by](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/max_by/), [sub\_bitmap](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/sub_bitmap/), [bitmap\_to\_base64](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_to_base64/), [host\_name](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/host_name/), and [date\_slice](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_slice/). [#11299](https://github.com/StarRocks/starrocks/pull/11299) [#11323](https://github.com/StarRocks/starrocks/pull/11323) [#12243](https://github.com/StarRocks/starrocks/pull/12243) [#11776](https://github.com/StarRocks/starrocks/pull/11776) [#12634](https://github.com/StarRocks/starrocks/pull/12634) [#14225](https://github.com/StarRocks/starrocks/pull/14225) ##### Improvements[​](#improvements-21 "Direct link to Improvements") * Optimized the metadata access performance when you query external data using [Hive catalogs](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog/), [Hudi catalogs](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog/), and [Iceberg catalogs](https://docs.starrocks.io/docs/data_source/catalog/iceberg_catalog/). [#11349](https://github.com/StarRocks/starrocks/issues/11349) * Supports querying ARRAY data using [Elasticsearch external tables](https://docs.starrocks.io/docs/data_source/External_table#deprecated-elasticsearch-external-table). [#9693](https://github.com/StarRocks/starrocks/pull/9693) * Optimized the following aspects of materialized views: * Asynchronous materialized views support automatic and transparent query rewrite based on the SPJG-type materialized views. For more information, see [Materialized view](https://docs.starrocks.io/docs/using_starrocks/Materialized_view#rewrite-and-accelerate-queries-with-the-asynchronous-materialized-view). [#13193](https://github.com/StarRocks/starrocks/issues/13193) * Asynchronous materialized views support multiple async refresh mechanisms. For more information, see [Materialized view](https://docs.starrocks.io/docs/using_starrocks/Materialized_view#manually-refresh-an-asynchronous-materialized-view). [#12712](https://github.com/StarRocks/starrocks/pull/12712) [#13171](https://github.com/StarRocks/starrocks/pull/13171) [#13229](https://github.com/StarRocks/starrocks/pull/13229) [#12926](https://github.com/StarRocks/starrocks/pull/12926) * The efficiency of refreshing materialized views is improved. [#13167](https://github.com/StarRocks/starrocks/issues/13167) * Optimized the following aspects of data loading: * Optimized loading performance in multi-replica scenarios by supporting the "single leader replication" mode. Data loading gains a one-fold performance lift. For more information about "single leader replication", see `replicated_storage` in [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/CREATE_TABLE/). [#10138](https://github.com/StarRocks/starrocks/pull/10138) * Broker Load and Spark Load no longer need to depend on brokers for data loading when only one HDFS cluster or one Kerberos user is configured. However, if you have multiple HDFS clusters or multiple Kerberos users, you still need to deploy a broker. For more information, see [Load data from HDFS or cloud storage](https://docs.starrocks.io/docs/loading/BrokerLoad/) and [Bulk load using Apache Spark™](https://docs.starrocks.io/docs/loading/SparkLoad/). [#9049](https://github.com/starrocks/starrocks/pull/9049) [#9228](https://github.com/StarRocks/starrocks/pull/9228) * Optimized the performance of Broker Load when a large number of small ORC files are loaded. [#11380](https://github.com/StarRocks/starrocks/pull/11380) * Reduced the memory usage when you load data into Primary Key tables. * Optimized the `information_schema` database and the `tables` and `columns` tables within. Adds a new table `table_config`. For more information, see [Information Schema](https://docs.starrocks.io/docs/reference/overview-pages/information_schema/). [#10033](https://github.com/StarRocks/starrocks/pull/10033) * Optimized data backup and restore: * Supports backing up and restoring data from multiple tables in a database at a time. For more information, see [Backup and restore data](https://docs.starrocks.io/docs/administration/Backup_and_restore/). [#11619](https://github.com/StarRocks/starrocks/issues/11619) * Supports backing up and restoring data from Primary Key tables. For more information, see Backup and restore. [#11885](https://github.com/StarRocks/starrocks/pull/11885) * Optimized the following functions: * Added an optional parameter for the [time\_slice](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/time_slice/) function, which is used to determine whether the beginning or end of the time interval is returned. [#11216](https://github.com/StarRocks/starrocks/pull/11216) * Added a new mode `INCREASE` for the [window\_funnel](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/window_funnel/) function to avoid computing duplicate timestamps. [#10134](https://github.com/StarRocks/starrocks/pull/10134) * Supports specifying multiple arguments in the [unnest](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/unnest/) function. [#12484](https://github.com/StarRocks/starrocks/pull/12484) * lead() and lag() functions support querying HLL and BITMAP data. For more information, see [Window function](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function/). [#12108](https://github.com/StarRocks/starrocks/pull/12108) * The following ARRAY functions support querying JSON data: [array\_agg](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_agg/), [array\_sort](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_sort/), [array\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_concat/), [array\_slice](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_slice/), and [reverse](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/reverse/). [#13155](https://github.com/StarRocks/starrocks/pull/13155) * Optimized the use of some functions. The `current_date`, `current_timestamp`, `current_time`, `localtimestamp`, and `localtime` functions can be executed without using `()`, for example, you can directly run `select current_date;`. [# 14319](https://github.com/StarRocks/starrocks/pull/14319) * Removed some redundant information from FE logs. [# 15374](https://github.com/StarRocks/starrocks/pull/15374) ##### Bug Fixes[​](#bug-fixes-22 "Direct link to Bug Fixes") The following bugs are fixed: * The append\_trailing\_char\_if\_absent() function may return an incorrect result when the first argument is empty. [#13762](https://github.com/StarRocks/starrocks/pull/13762) * After a table is restored using the RECOVER statement, the table does not exist. [#13921](https://github.com/StarRocks/starrocks/pull/13921) * The result returned by the SHOW CREATE MATERIALIZED VIEW statement does not contain the database and catalog specified in the query statement when the materialized view was created. [#12833](https://github.com/StarRocks/starrocks/pull/12833) * Schema change jobs in the `waiting_stable` state cannot be canceled. [#12530](https://github.com/StarRocks/starrocks/pull/12530) * Running the `SHOW PROC '/statistic';` command on a Leader FE and non-Leader FE returns different results. [#12491](https://github.com/StarRocks/starrocks/issues/12491) * The position of the ORDER BY clause is incorrect in the result returned by SHOW CREATE TABLE. [# 13809](https://github.com/StarRocks/starrocks/pull/13809) * When users use Hive Catalog to query Hive data, if the execution plan generated by FE does not contain partition IDs, BEs fail to query Hive partition data. [# 15486](https://github.com/StarRocks/starrocks/pull/15486). ##### Behavior Change[​](#behavior-change-3 "Direct link to Behavior Change") * Changed the default value of the `AWS_EC2_METADATA_DISABLED` parameter to `False`, which means that the metadata of Amazon EC2 is obtained to access AWS resources. * Renamed session variable `is_report_success` to `enable_profile`, which can be queried using the SHOW VARIABLES statement. * Added four reserved keywords: `CURRENT_DATE`, `CURRENT_TIME`, `LOCALTIME`, and `LOCALTIMESTAMP`. [# 14319](https://github.com/StarRocks/starrocks/pull/14319) * The maximum length of table and database names can be up to 1023 characters. [# 14929](https://github.com/StarRocks/starrocks/pull/14929) [# 15020](https://github.com/StarRocks/starrocks/pull/15020) * BE configuration items `enable_event_based_compaction_framework` and `enable_size_tiered_compaction_strategy` are set to `true` by default, which significantly reduces compaction overheads when there are a large number of tablets or a single tablet has large data volume. ##### Upgrade Notes[​](#upgrade-notes "Direct link to Upgrade Notes") * You can upgrade your cluster to 2.5.0 from 2.0.x, 2.1.x, 2.2.x, 2.3.x, or 2.4.x. However, if you need to perform a rollback, we recommend that you roll back only to 2.4.x. --- ## Release 3.0 ### StarRocks version 3.0 #### 3.0.9[​](#309 "Direct link to 3.0.9") Release date: January 2, 2024 ##### New features[​](#new-features "Direct link to New features") * Added the [percentile\_disc](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/percentile_disc/) function. [#36352](https://github.com/StarRocks/starrocks/pull/36352) * Added a new metric `max_tablet_rowset_num` for setting the maximum allowed number of rowsets. This metric helps detect possible compaction issues and thus reduces the occurrences of the error "too many versions". [#36539](https://github.com/StarRocks/starrocks/pull/36539) ##### Improvements[​](#improvements "Direct link to Improvements") * A new value option `GROUP_CONCAT_LEGACY` is added to the session variable [sql\_mode](https://docs.starrocks.io/docs/sql-reference/System_variable/#sql_mode) to provide compatibility with the implementation logic of the [group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat/) function in versions earlier than v2.5. [#36150](https://github.com/StarRocks/starrocks/pull/36150) * When using JDK, the default GC algorithm is G1. [#37386](https://github.com/StarRocks/starrocks/pull/37386) * The `be_tablets` view in the `information_schema` database provides a new field `INDEX_DISK`, which records the disk usage (measured in bytes) of persistent indexes [#35615](https://github.com/StarRocks/starrocks/pull/35615) * Queries on MySQL external tables and the external tables within JDBC catalogs support including keywords in the WHERE clause. [#35917](https://github.com/StarRocks/starrocks/pull/35917) * Supports updates onto the specified partitions of an automatically partitioned table. If the specified partitions do not exist, an error is returned. [#34777](https://github.com/StarRocks/starrocks/pull/34777) * The Primary Key table size returned by the [SHOW DATA](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATA/) statement includes the sizes of **.cols** files (these are files related to partial column updates and generated columns) and persistent index files. [#34898](https://github.com/StarRocks/starrocks/pull/34898) * Optimized the performance of persistent index update when compaction is performed on all rowsets of a Primary Key table, which reduces disk read I/O. [#36819](https://github.com/StarRocks/starrocks/pull/36819) * When the string on the right side of the LIKE operator within the WHERE clause does not include `%` or `_`, the LIKE operator is converted into the `=` operator. [#37515](https://github.com/StarRocks/starrocks/pull/37515) * Optimized the logic used to compute compaction scores for Primary Key tables, thereby aligning the compaction scores for Primary Key tables within a more consistent range with the other three table types. [#36534](https://github.com/StarRocks/starrocks/pull/36534) * The result returned by the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD_TASK/) statement now includes the timestamps of consumption messages from each partition. [#36222](https://github.com/StarRocks/starrocks/pull/36222) * Optimized the performance of some Bitmap-related operations, including: * Optimized nested loop joins. [#340804](https://github.com/StarRocks/starrocks/pull/34804) [#35003](https://github.com/StarRocks/starrocks/pull/35003) * Optimized the `bitmap_xor` function. [#34069](https://github.com/StarRocks/starrocks/pull/34069) * Supports Copy on Write to optimize Bitmap performance and reduce memory consumption. [#34047](https://github.com/StarRocks/starrocks/pull/34047) ##### Behavior Change[​](#behavior-change "Direct link to Behavior Change") * Added the session variable `enable_materialized_view_for_insert`, which controls whether materialized views rewrite the queries in INSERT INTO SELECT statements. The default value is `false`. [#37505](https://github.com/StarRocks/starrocks/pull/37505) * Changed the FE configuration item `enable_new_publish_mechanism` to a static parameter from a dynamic one. You must restart the FE after you modify the parameter settings. [#35338](https://github.com/StarRocks/starrocks/pull/35338) * Changed the default retention period of trash files to 1 day from the original 3 days. [#37113](https://github.com/StarRocks/starrocks/pull/37113) ##### Parameter Change[​](#parameter-change "Direct link to Parameter Change") ###### Session variables[​](#session-variables "Direct link to Session variables") * Added session variable `cbo_decimal_cast_string_strict`, which controls how the CBO converts data from the DECIMAL type to the STRING type. If this variable is set to `true`, the logic built in v2.5.x and later versions prevails and the system implements strict conversion (namely, the system truncates the generated string and fills 0s based on the scale length). If this variable is set to `false`, the logic built in versions earlier than v2.5.x prevails and the system processes all valid digits to generate a string. The default value is `true`. [#34208](https://github.com/StarRocks/starrocks/pull/34208) * Added session variables `transaction_read_only` and `tx_read_only` to specify the transaction access mode, which are compatible with MySQL versions 5.7.20 and above. [#37249](https://github.com/StarRocks/starrocks/pull/37249) ###### FE Parameters[​](#fe-parameters "Direct link to FE Parameters") * Added the FE configuration item `routine_load_unstable_threshold_second`. [#36222](https://github.com/StarRocks/starrocks/pull/36222) * Added the FE configuration item `http_worker_threads_num`, which specifies the number of threads for HTTP server to deal with HTTP requests. The default value is `0`. If the value for this parameter is set to a negative value or 0, the actual thread number is twice the number of CPU cores. [#37530](https://github.com/StarRocks/starrocks/pull/37530) * Added the FE configuration item `default_mv_refresh_immediate`, which specifies whether to immediately refresh the materialized view after the materialized view is created. The default value is `true`. [#37093](https://github.com/StarRocks/starrocks/pull/37093) ###### BE Parameters[​](#be-parameters "Direct link to BE Parameters") * Added the BE configuration item `enable_stream_load_verbose_log`. The default value is `false`. With this parameter set to `true`, StarRocks can record the HTTP requests and responses for Stream Load jobs, making troubleshooting easier. [#36113](https://github.com/StarRocks/starrocks/pull/36113) * Added the BE configuration item `pindex_major_compaction_limit_per_disk` to configure the maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. The default value is `1`. [#37694](https://github.com/StarRocks/starrocks/pull/37694) * Added BE configuration items to specify the timeout duration for connecting to object storage: * `object_storage_connect_timeout_ms`: Timeout duration to establish socket connections with object storage. The default value is `-1`, which means to use the default timeout duration of the SDK configurations. * `object_storage_request_timeout_ms`: Timeout duration to establish HTTP connections with object storage. The default value is `-1`, which means to use the default timeout duration of the SDK configurations. ##### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") Fixed the following issues: * In some cases, BEs may crash when a Catalog is used to read ORC external tables. [#27971](https://github.com/StarRocks/starrocks/pull/27971) * The BEs crash if users create persistent indexes in the event of data corruption. [#30841](https://github.com/StarRocks/starrocks/pull/30841) * BEs occasionally crash after a Bitmap index is added. [#26463](https://github.com/StarRocks/starrocks/pull/26463) * Failures in replaying replica operations may cause FEs to crash. [#32295](https://github.com/StarRocks/starrocks/pull/32295) * Setting the FE parameter `recover_with_empty_tablet` to `true` may cause FEs to crash. [#33071](https://github.com/StarRocks/starrocks/pull/33071) * Queries fail during hash joins, causing BEs to crash. [#32219](https://github.com/StarRocks/starrocks/pull/32219) * In a StarRocks shared-nothing cluster, queries against Iceberg or Hive tables may cause BEs to crash. [#34682](https://github.com/StarRocks/starrocks/pull/34682) * The error "get\_applied\_rowsets failed, tablet updates is in error state: tablet:18849 actual row size changed after compaction" is returned for queries. [#33246](https://github.com/StarRocks/starrocks/pull/33246) * Running `show proc '/statistic'` may cause a deadlock. [#34237](https://github.com/StarRocks/starrocks/pull/34237/files) * The FE performance plunges after the FE configuration item `enable_collect_query_detail_info` is set to `true`. [#35945](https://github.com/StarRocks/starrocks/pull/35945) * Errors may be thrown if large amounts of data are loaded into a Primary Key table with persistent index enabled. [#34352](https://github.com/StarRocks/starrocks/pull/34352) * After StarRocks is upgraded from v2.4 or earlier to a later version, compaction scores may rise unexpectedly. [#34618](https://github.com/StarRocks/starrocks/pull/34618) * If `INFORMATION_SCHEMA` is queried by using the database driver MariaDB ODBC, the `CATALOG_NAME` column returned in the `schemata` view holds only `null` values. [#34627](https://github.com/StarRocks/starrocks/pull/34627) * FEs crash due to the abnormal data loaded and cannot restart. [#34590](https://github.com/StarRocks/starrocks/pull/34590) * If schema changes are being executed while a Stream Load job is in the **PREPARED** state, a portion of the source data to be loaded by the job is lost. [#34381](https://github.com/StarRocks/starrocks/pull/34381) * Including two or more slashes (`/`) at the end of the HDFS storage path causes the backup and restore of the data from HDFS to fail. [#34601](https://github.com/StarRocks/starrocks/pull/34601) * The `partition_live_number` property added by using the ALTER TABLE statement does not take effect. [#34842](https://github.com/StarRocks/starrocks/pull/34842) * The [array\_distinct](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_distinct/) function occasionally causes the BEs to crash. [#36377](https://github.com/StarRocks/starrocks/pull/36377) * Deadlocks may occur when users refresh materialized views. [#35736](https://github.com/StarRocks/starrocks/pull/35736) * Global Runtime Filter may cause BEs to crash in certain scenarios. [#35776](https://github.com/StarRocks/starrocks/pull/35776) * In some cases, `bitmap_to_string` may return incorrect result due to data type overflow. [#37405](https://github.com/StarRocks/starrocks/pull/37405) #### 3.0.8[​](#308 "Direct link to 3.0.8") Release date: November 17, 2023 ##### Improvements[​](#improvements-1 "Direct link to Improvements") * The `COLUMNS` view in the system database `INFORMATION_SCHEMA` can display ARRAY, MAP, and STRUCT columns. [#33431](https://github.com/StarRocks/starrocks/pull/33431) ##### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") Fixed the following issues: * When `show proc '/current_queries';` is being executed and meanwhile a query begins to be executed, BEs may crash. [#34316](https://github.com/StarRocks/starrocks/pull/34316) * When data is continuously loaded into a Primary Key table with a sort key specified at a high frequency, compaction failures may occur. [#26486](https://github.com/StarRocks/starrocks/pull/26486) * If a filtering condition is specified in a Broker Load job, BEs may crash during the data loading in certain circumstances. [#29832](https://github.com/StarRocks/starrocks/pull/29832) * An unknown error is reported when SHOW GRANTS is executed. [#30100](https://github.com/StarRocks/starrocks/pull/30100) * BE may crash for specific data types if the target data type specified in the `cast()` function is the same as the original data type. [#31465](https://github.com/StarRocks/starrocks/pull/31465) * `DATA_TYPE` and `COLUMN_TYPE` for BINARY or VARBINARY data types are displayed as `unknown` in the `information_schema.columns` view. [#32678](https://github.com/StarRocks/starrocks/pull/32678) * Long-time, frequent data loading into a Primary Key table with persistent index enabled may cause BEs to crash. [#33220](https://github.com/StarRocks/starrocks/pull/33220) * The query result is incorrect when Query Cache is enabled. [#32778](https://github.com/StarRocks/starrocks/pull/32778) * After a cluster is restarted, the data in a restored table may be inconsistent with the data in that table before being backed up. [#33567](https://github.com/StarRocks/starrocks/pull/33567) * If RESTORE is executed and meanwhile Compaction takes place, it may cause BEs to crash. [#32902](https://github.com/StarRocks/starrocks/pull/32902) #### 3.0.7[​](#307 "Direct link to 3.0.7") Release date: October 18, 2023 ##### Improvements[​](#improvements-2 "Direct link to Improvements") * Window functions COVAR\_SAMP, COVAR\_POP, CORR, VARIANCE, VAR\_SAMP, STD, and STDDEV\_SAMP now support the ORDER BY clause and Window clause. [#30786](https://github.com/StarRocks/starrocks/pull/30786) * The Publish phase of a load job that writes data into a Primary Key table is changed from asynchronous mode to synchronous mode. As such, the data loaded can be queried immediately after the load job finishes. [#27055](https://github.com/StarRocks/starrocks/pull/27055) * An error instead of NULL is returned if a decimal overflow occurs during queries on the DECIMAL type data. [#30419](https://github.com/StarRocks/starrocks/pull/30419) * Executing SQL commands with invalid comments now returns results consistent with MySQL. [#30210](https://github.com/StarRocks/starrocks/pull/30210) * For a StarRocks table that uses RANGE partitioning with only one partitioning column or expression partitioning, SQL predicates containing partition column expressions can also be used for partition pruning. [#30421](https://github.com/StarRocks/starrocks/pull/30421) ##### Bug Fixes[​](#bug-fixes-2 "Direct link to Bug Fixes") Fixed the following issues: * Concurrently creating and deleting databases and tables can, in certain cases, result in the table not being found and further leads to the failure of data loading into that table. [#28985](https://github.com/StarRocks/starrocks/pull/28985) * Using UDFs may lead to memory leaks in certain cases. [#29467](https://github.com/StarRocks/starrocks/pull/29467) [#29465](https://github.com/StarRocks/starrocks/pull/29465) * If the ORDER BY clause contains aggregate functions, an error "java.lang.IllegalStateException: null" is returned. [#30108](https://github.com/StarRocks/starrocks/pull/30108) * If users run queries against data stored in Tencent COS by using their Hive catalog which consists of multiple levels, the query results will be incorrect. [#30363](https://github.com/StarRocks/starrocks/pull/30363) * If some subcfields of the STRUCT in ARRAY\ type data are missing, the data length is incorrect when default values are filled in the missing subcfields during queries, which causes BEs to crash. * The version of Berkeley DB Java Edition is upgraded to avoid security vulnerabilities.[#30029](https://github.com/StarRocks/starrocks/pull/30029) * If users load data into a Primary Key table on which truncate operations and queries are concurrently performed, an error "java.lang.NullPointerException" is thrown in certain cases. [#30573](https://github.com/StarRocks/starrocks/pull/30573) * If the Schema Change execution time is too long, it may fail because the tablet of the specified version is garbage-collected. [#31376](https://github.com/StarRocks/starrocks/pull/31376) * If users use CloudCanal to load data into table columns that are set to `NOT NULL` but have no default value specified, an error "Unsupported dataFormat value is : \N" is thrown. [#30799](https://github.com/StarRocks/starrocks/pull/30799) * In StarRocks shared-data clusters, the information of table keys is not recorded in `information_schema.COLUMNS`. As a result, DELETE operations cannot be performed when data is loaded by using Flink Connector. [#31458](https://github.com/StarRocks/starrocks/pull/31458) * During the upgrade, if the types of certain columns are also upgraded (for example, from Decimal type to Decimal v3 type), compaction on certain tables with specific characteristics may cause BEs to crash. [#31626](https://github.com/StarRocks/starrocks/pull/31626) * When data is loaded by using Flink Connector, the load job is suspended unexpectedly if there are highly concurrent load jobs and both the number of HTTP threads and the number of Scan threads have reached their upper limits. [#32251](https://github.com/StarRocks/starrocks/pull/32251) * BEs crash when libcurl is invoked. [#31667](https://github.com/StarRocks/starrocks/pull/31667) * An error occurs when a column of BITMAP type is added to a Primary Key table. [#31763](https://github.com/StarRocks/starrocks/pull/31763) #### 3.0.6[​](#306 "Direct link to 3.0.6") Release date: September 12, 2023 ##### Behavior Change[​](#behavior-change-1 "Direct link to Behavior Change") * When using the [group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat/) function, you must use the SEPARATOR keyword to declare the separator. ##### New Features[​](#new-features-1 "Direct link to New Features") * The aggregate function [group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat/) supports the DISTINCT keyword and the ORDER BY clause. [#28778](https://github.com/StarRocks/starrocks/pull/28778) * Data in partitions can be automatically cooled down over time. (This feature is not supported for [list partitioning](https://docs.starrocks.io/docs/table_design/list_partitioning/).) [#29335](https://github.com/StarRocks/starrocks/pull/29335) [#29393](https://github.com/StarRocks/starrocks/pull/29393) ##### Improvements[​](#improvements-3 "Direct link to Improvements") * Supports implicit conversions for all compound predicates and for all expressions in the WHERE clause. You can enable or disable implicit conversions by using the [session variable](https://docs.starrocks.io/docs/sql-reference/System_variable/) `enable_strict_type`. The default value of this session variable is `false`. [#21870](https://github.com/StarRocks/starrocks/pull/21870) * Unifies the logic between FEs and BEs in converting strings to integers. [#29969](https://github.com/StarRocks/starrocks/pull/29969) ##### Bug Fixes[​](#bug-fixes-3 "Direct link to Bug Fixes") * If `enable_orc_late_materialization` is set to `true`, an unexpected result is returned when a Hive catalog is used to query STRUCT-type data in ORC files. [#27971](https://github.com/StarRocks/starrocks/pull/27971) * During data queries through Hive Catalog, if a partitioning column and an OR operator are specified in the WHERE clause, the query result is incorrect. [#28876](https://github.com/StarRocks/starrocks/pull/28876) * The values returned by the RESTful API action `show_data` for cloud-native tables are incorrect. [#29473](https://github.com/StarRocks/starrocks/pull/29473) * If the [shared-data cluster](https://docs.starrocks.io/docs/deployment/shared_data/azure/) stores data in Azure Blob Storage and a table is created, the FE fails to start after the cluster is rolled back to version 3.0. [#29433](https://github.com/StarRocks/starrocks/pull/29433) * A user has no permission when querying a table in the Iceberg catalog even if the user is granted permission on that table. [#29173](https://github.com/StarRocks/starrocks/pull/29173) * The `Default` field values returned by the [SHOW FULL COLUMNS](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_FULL_COLUMNS/) statement for columns of the [BITMAP](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/BITMAP/) or [HLL](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/HLL/) data type are incorrect. [#29510](https://github.com/StarRocks/starrocks/pull/29510) * Modifying the FE dynamic parameter `max_broker_load_job_concurrency` using the `ADMIN SET FRONTEND CONFIG` command does not take effect. * The FE may fail to start when a materialized view is being refreshed while its refresh strategy is being modified. [#29964](https://github.com/StarRocks/starrocks/pull/29964) [#29720](https://github.com/StarRocks/starrocks/pull/29720) * The error `unknown error` is returned when `select count(distinct(int+double)) from table_name` is executed. [#29691](https://github.com/StarRocks/starrocks/pull/29691) * After a Primary Key table is restored, metadata errors occur and cause metadata inconsistencies occur if a BE is restarted. [#30135](https://github.com/StarRocks/starrocks/pull/30135) #### 3.0.5[​](#305 "Direct link to 3.0.5") Release date: August 16, 2023 ##### New Features[​](#new-features-2 "Direct link to New Features") * Supports aggregate functions [COVAR\_SAMP](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/covar_samp/), [COVAR\_POP](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/covar_pop/), and [CORR](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/corr/). * Supports the following [window functions](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function/): COVAR\_SAMP, COVAR\_POP, CORR, VARIANCE, VAR\_SAMP, STD, and STDDEV\_SAMP. ##### Improvements[​](#improvements-4 "Direct link to Improvements") * Added more prompts in the error message `xxx too many versions xxx`. [#28397](https://github.com/StarRocks/starrocks/pull/28397) * Dynamic partitioning further supports the partitioning unit to be year. [#28386](https://github.com/StarRocks/starrocks/pull/28386) * The partitioning field is case-insensitive when expression partitioning is used at table creation and [INSERT OVERWRITE is used to overwrite data in a specific partition](https://docs.starrocks.io/docs/table_design/expression_partitioning#load-data-into-partitions). [#28309](https://github.com/StarRocks/starrocks/pull/28309) ##### Bug Fixes[​](#bug-fixes-4 "Direct link to Bug Fixes") Fixed the following issues: * Incorrect table-level scan statistics in FE cause inaccurate metrics for table queries and loading. [#27779](https://github.com/StarRocks/starrocks/pull/27779) * The query result is not stable if the sort key is modified for a partitioned table. [#27850](https://github.com/StarRocks/starrocks/pull/27850) * The version number for a tablet is inconsistent between the BE and FE after data is restored. [#26518](https://github.com/StarRocks/starrocks/pull/26518/files) * If the bucket number is not specified when users create a Colocation table, the number will be inferred as 0, which causes failures in adding new partitions. [#27086](https://github.com/StarRocks/starrocks/pull/27086) * When the SELECT result set of INSERT INTO SELECT is empty, the load job status returned by SHOW LOAD is `CANCELED`. [#26913](https://github.com/StarRocks/starrocks/pull/26913) * BEs may crash when the input values of the sub\_bitmap function are not of the BITMAP type. [#27982](https://github.com/StarRocks/starrocks/pull/27982) * BEs may crash when the AUTO\_INCREMENT column is being updated. [#27199](https://github.com/StarRocks/starrocks/pull/27199) * Outer join and Anti join rewrite errors for materialized views. [#28028](https://github.com/StarRocks/starrocks/pull/28028) * Inaccurate estimation of average row size causes Primary Key partial updates to occupy excessively large memory. [#27485](https://github.com/StarRocks/starrocks/pull/27485) * Activating an inactive materialized view may cause a FE to crash. [#27959](https://github.com/StarRocks/starrocks/pull/27959) * Queries can not be rewritten to materialized views created based on external tables in a Hudi catalog. [#28023](https://github.com/StarRocks/starrocks/pull/28023) * The data of a Hive table can still be queried even after the table is dropped and the metadata cache is manually updated. [#28223](https://github.com/StarRocks/starrocks/pull/28223) * Manually refreshing an asynchronous materialized view via a synchronous call results in multiple INSERT OVERWRITE records in the `information_schema.task_runs` table. [#28060](https://github.com/StarRocks/starrocks/pull/28060) * FE memory leak caused by blocked LabelCleaner threads. [#28311](https://github.com/StarRocks/starrocks/pull/28311) #### 3.0.4[​](#304 "Direct link to 3.0.4") Release date: July 18, 2023 ##### New Feature[​](#new-feature "Direct link to New Feature") Queries can be rewritten even when the queries contain a different type of join than the materialized view. [#25099](https://github.com/StarRocks/starrocks/pull/25099) ##### Improvements[​](#improvements-5 "Direct link to Improvements") * Optimized the manual refreshing of asynchronous materialized views. Supports using the REFRESH MATERIALIZED VIEW WITH SYNC MODE syntax to synchronously invoke materialized view refresh tasks. [#25910](https://github.com/StarRocks/starrocks/pull/25910) * If the queried fields are not included in the output columns of a materialized view but are included in the predicate of the materialized view, the query can still be rewritten to benefit from the materialized view. [#23028](https://github.com/StarRocks/starrocks/issues/23028) * [When the SQL dialect (`sql_dialect`) is set to `trino`](https://docs.starrocks.io/docs/sql-reference/System_variable/), table aliases are not case-sensitive. [#26094](https://github.com/StarRocks/starrocks/pull/26094) [#25282](https://github.com/StarRocks/starrocks/pull/25282) * Added a new field `table_id` to the table `Information_schema.tables_config`. You can join the table `tables_config` with the table `be_tablets` on the column `table_id` in the database `Information_schema` to query the names of the database and table to which a tablet belongs. [#24061](https://github.com/StarRocks/starrocks/pull/24061) ##### Bug Fixes[​](#bug-fixes-5 "Direct link to Bug Fixes") Fixed the following issues: * If a query that contains the sum aggregate function is rewritten to directly obtain query results from a single-table materialized view, the values in sum() field may be incorrect due to type inference issues. [#25512](https://github.com/StarRocks/starrocks/pull/25512) * An error occurs when SHOW PROC is used to view information about tablets in a StarRocks shared-data cluster. * The INSERT operation hangs when the length of CHAR data in a STRUCT to be inserted exceeds the maximum length. [#25942](https://github.com/StarRocks/starrocks/pull/25942) * Some data rows queried fail to be returned for INSERT INTO SELECT with FULL JOIN. [#26603](https://github.com/StarRocks/starrocks/pull/26603) * An error `ERROR xxx: Unknown table property xxx` occurs when the ALTER TABLE statement is used to modify the table's property `default.storage_medium`. [#25870](https://github.com/StarRocks/starrocks/issues/25870) * An error occurs when Broker Load is used to load empty files. [#26212](https://github.com/StarRocks/starrocks/pull/26212) * Decommissioning a BE sometimes hangs. [#26509](https://github.com/StarRocks/starrocks/pull/26509) #### 3.0.3[​](#303 "Direct link to 3.0.3") Release date: June 28, 2023 ##### Improvements[​](#improvements-6 "Direct link to Improvements") * Metadata synchronization of StarRocks external tables has been changed to occur during data loading. [#24739](https://github.com/StarRocks/starrocks/pull/24739) * Users can specify partitions when they run INSERT OVERWRITE on tables whose partitions are automatically created. For more information, see [Automatic partitioning](https://docs.starrocks.io/docs/table_design/expression_partitioning/). [#25005](https://github.com/StarRocks/starrocks/pull/25005) * Optimized the error message reported when partitions are added to a non-partitioned table. [#25266](https://github.com/StarRocks/starrocks/pull/25266) ##### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") Fixed the following issues: * The min/max filter gets the wrong Parquet field when the Parquet file contains complex data types. [#23976](https://github.com/StarRocks/starrocks/pull/23976) * Load tasks are still queuing even when the related database or table has been dropped. [#24801](https://github.com/StarRocks/starrocks/pull/24801) * There is a low probability that an FE restart may cause BEs to crash. [#25037](https://github.com/StarRocks/starrocks/pull/25037) * Load and query jobs occasionally freeze when the variable `enable_profile` is set to `true`. [#25060](https://github.com/StarRocks/starrocks/pull/25060) * Inaccurate error message is displayed when INSERT OVERWRITE is executed on a cluster with less than three alive BEs. [#25314](https://github.com/StarRocks/starrocks/pull/25314) #### 3.0.2[​](#302 "Direct link to 3.0.2") Release date: June 13, 2023 ##### Improvements[​](#improvements-7 "Direct link to Improvements") * Predicates in a UNION query can be pushed down after the query is rewritten by an asynchronous materialized view. [#23312](https://github.com/StarRocks/starrocks/pull/23312) * Optimized the auto tablet distribution policy for tables. [#24543](https://github.com/StarRocks/starrocks/pull/24543) * Removed the dependency of NetworkTime on system clocks, which fixes incorrect NetworkTime caused by inconsistent system clocks across servers. [#24858](https://github.com/StarRocks/starrocks/pull/24858) ##### Bug Fixes[​](#bug-fixes-7 "Direct link to Bug Fixes") Fixed the following issues: * A schema change sometimes may be hung if data loading occurs simultaneously with the schema change. [#23456](https://github.com/StarRocks/starrocks/pull/23456) * Queries encounter an error when the session variable `pipeline_profile_level` is set to `0`. [#23873](https://github.com/StarRocks/starrocks/pull/23873) * CREATE TABLE encounters an error when `cloud_native_storage_type` is set to `S3`. * LDAP authentication succeeds even when no password is used. [#24862](https://github.com/StarRocks/starrocks/pull/24862) * CANCEL LOAD fails if the table involved in the load job does not exist. [#24922](https://github.com/StarRocks/starrocks/pull/24922) ##### Upgrade Notes[​](#upgrade-notes "Direct link to Upgrade Notes") If your system has a database named `starrocks`, change it to another name using ALTER DATABASE RENAME before the upgrade. This is because `starrocks` is the name of a default system database that stores privilege information. #### 3.0.1[​](#301 "Direct link to 3.0.1") Release date: June 1, 2023 ##### New Features[​](#new-features-3 "Direct link to New Features") * \[Preview] Supports spilling intermediate computation results of large operators to disks to reduce the memory consumption of large operators. For more information, see [Spill to disk](https://docs.starrocks.io/docs/administration/spill_to_disk/). * [Routine Load](https://docs.starrocks.io/docs/loading/RoutineLoad#load-avro-format-data) supports loading Avro data. * Supports [Microsoft Azure Storage](https://docs.starrocks.io/docs/integrations/authenticate_to_azure_storage/) (including Azure Blob Storage and Azure Data Lake Storage). ##### Improvements[​](#improvements-8 "Direct link to Improvements") * Shared-data clusters support using StarRocks external tables to synchronize data with another StarRocks cluster. * Added `load_tracking_logs` to [Information Schema](https://docs.starrocks.io/docs/sql-reference/information_schema/load_tracking_logs/) to record recent loading errors. * Ignores special characters in CREATE TABLE statements. [#23885](https://github.com/StarRocks/starrocks/pull/23885) ##### Bug Fixes[​](#bug-fixes-8 "Direct link to Bug Fixes") Fixed the following issues: * Information returned by SHOW CREATE TABLE is incorrect for Primary Key tables. [#24237](https://github.com/StarRocks/starrocks/issues/24237) * BEs may crash during a Routine Load job. [#20677](https://github.com/StarRocks/starrocks/issues/20677) * Null pointer exception (NPE) occurs if you specify unsupported properties when creating a partitioned table. [#21374](https://github.com/StarRocks/starrocks/issues/21374) * Information returned by SHOW TABLE STATUS is incomplete. [#24279](https://github.com/StarRocks/starrocks/issues/24279) ##### Upgrade Notes[​](#upgrade-notes-1 "Direct link to Upgrade Notes") If your system has a database named `starrocks`, change it to another name using ALTER DATABASE RENAME before the upgrade. This is because `starrocks` is the name of a default system database that stores privilege information. #### 3.0.0[​](#300 "Direct link to 3.0.0") Release date: April 28, 2023 ##### New Features[​](#new-features-4 "Direct link to New Features") ###### System architecture[​](#system-architecture "Direct link to System architecture") * **Decouple storage and compute.** StarRocks now supports data persistence into S3-compatible object storage, enhancing resource isolation, reducing storage costs, and making compute resources more scalable. Local disks are used as hot data cache for boosting query performance. The query performance of the new shared-data architecture is comparable to the classic architecture (shared-nothing) when local disk cache is hit. For more information, see [Deploy and use shared-data StarRocks](https://docs.starrocks.io/docs/deployment/shared_data/s3/). ###### Storage engine and data ingestion[​](#storage-engine-and-data-ingestion "Direct link to Storage engine and data ingestion") * The [AUTO\_INCREMENT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/auto_increment/) attribute is supported to provide globally unique IDs, which simplifies data management. * Automatic partitioning and partitioning expressions are supported, which makes partition creation easier to use and more flexible. * Primary Key tables support more complete [UPDATE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/UPDATE/) and [DELETE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DELETE/#primary-key-tables) syntax, including the use of CTEs and references to multiple tables. * Added Load Profile for Broker Load and INSERT INTO jobs. You can view the details of a load job by querying the load profile. The usage is the same as [Analyze query profile](https://docs.starrocks.io/docs/administration/query_profile_overview/). ###### Data Lake Analytics[​](#data-lake-analytics "Direct link to Data Lake Analytics") * \[Preview] Supports Presto/Trino compatible dialect. Presto/Trino's SQL can be automatically rewritten into StarRocks' SQL pattern. For more information, see [the system variable](https://docs.starrocks.io/docs/reference/System_variable/) `sql_dialect`. * \[Preview] Supports [JDBC catalogs](https://docs.starrocks.io/docs/data_source/catalog/jdbc_catalog/). * Supports using [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG/) to manually switch between catalogs in the current session. ###### Privileges and security[​](#privileges-and-security "Direct link to Privileges and security") * Provides a new privilege system with full RBAC functionalities, supporting role inheritance and default roles. For more information, see [Overview of privileges](https://docs.starrocks.io/docs/administration/privilege_overview/). * Provides more privilege management objects and more fine-grained privileges. For more information, see [Privileges supported by StarRocks](https://docs.starrocks.io/docs/administration/privilege_item/). ###### Query engine[​](#query-engine "Direct link to Query engine") * Allows more queries on joined tables to benefit from the [query cache](https://docs.starrocks.io/docs/using_starrocks/query_cache/). For example, the query cache now supports Broadcast Join and Bucket Shuffle Join. * Supports [Global UDFs](https://docs.starrocks.io/docs/sql-reference/sql-functions/JAVA_UDF/). * Dynamic adaptive parallelism: StarRocks can automatically adjust the `pipeline_dop` parameter for query concurrency. ###### SQL reference[​](#sql-reference "Direct link to SQL reference") * Added the following privilege-related SQL statements: [SET DEFAULT ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_DEFAULT_ROLE/), [SET ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_ROLE/), [SHOW ROLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_ROLES/), and [SHOW USERS](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_USERS/). * Added the following semi-structured data analysis functions: [map\_apply](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_apply/), [map\_from\_arrays](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_from_arrays/). * [array\_agg](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_agg/) supports ORDER BY. * Window functions [lead](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function#lead) and [lag](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function#lag) support IGNORE NULLS. * Added string functions [replace](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/replace/), [hex\_decode\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/hex_decode_binary/), and [hex\_decode\_string()](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/hex_decode_string/). * Added encryption functions [base64\_decode\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/base64_decode_binary/) and [base64\_decode\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/base64_decode_string/). * Added math functions [sinh](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/sinh/), [cosh](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cosh/), and [tanh](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/tanh/). * Added utility function [current\_role](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/current_role/). ##### Improvements[​](#improvements-9 "Direct link to Improvements") ###### Deployment[​](#deployment "Direct link to Deployment") * Updated Docker image and the related [Docker deployment document](https://docs.starrocks.io/docs/quick_start/deploy_with_docker/) for version 3.0. [#20623](https://github.com/StarRocks/starrocks/pull/20623) [#21021](https://github.com/StarRocks/starrocks/pull/21021) ###### Storage engine and data ingestion[​](#storage-engine-and-data-ingestion-1 "Direct link to Storage engine and data ingestion") * Supports more CSV parameters for data ingestion, including SKIP\_HEADER, TRIM\_SPACE, ENCLOSE, and ESCAPE. See [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD/), [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD/), and [ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD/). * The primary key and sort key are decoupled in [Primary Key tables](https://docs.starrocks.io/docs/table_design/table_types/primary_key_table/). The sort key can be separately specified in `ORDER BY` when you create a table. * Optimized the memory usage of data ingestion into Primary Key tables in scenarios such as large-volume ingestion, partial updates, and persistent primary indexes. * Supports creating asynchronous INSERT tasks. For more information, see [INSERT](https://docs.starrocks.io/docs/loading/InsertInto#load-data-asynchronously-using-insert) and [SUBMIT TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK/). [#20609](https://github.com/StarRocks/starrocks/issues/20609) ###### Materialized view[​](#materialized-view "Direct link to Materialized view") * Optimized the rewriting capabilities of [materialized views](https://docs.starrocks.io/docs/using_starrocks/Materialized_view/), including: * Supports rewrite of View Delta Join, Outer Join, and Cross Join. * Optimized SQL rewrite of Union with partition. * Improved materialized view building capabilities: supporting CTE, select \*, and Union. * Optimized the information returned by [SHOW MATERIALIZED VIEWS](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/SHOW_MATERIALIZED_VIEW/). * Supports adding MV partitions in batches, which improves the efficiency of partition addition during materialized view building. [#21167](https://github.com/StarRocks/starrocks/pull/21167) ###### Query engine[​](#query-engine-1 "Direct link to Query engine") * All operators are supported in the pipeline engine. Non-pipeline code will be removed in later versions. * Improved [Big Query Positioning](https://docs.starrocks.io/docs/administration/monitor_manage_big_queries/) and added big query log. [SHOW PROCESSLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROCESSLIST/) supports viewing CPU and memory information. * Optimized Outer Join Reorder. * Optimized error messages in the SQL parsing stage, providing more accurate error positioning and clearer error messages. ###### Data Lake Analytics[​](#data-lake-analytics-1 "Direct link to Data Lake Analytics") * Optimized metadata statistics collection. * Supports using [SHOW CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_CREATE_TABLE/) to view the creation statements of the tables that are managed by an external catalog and are stored in Apache Hive™, Apache Iceberg, Apache Hudi, or Delta Lake. ##### Bug Fixes[​](#bug-fixes-9 "Direct link to Bug Fixes") * Some URLs in the license header of StarRocks' source file cannot be accessed. [#2224](https://github.com/StarRocks/starrocks/issues/2224) * An unknown error is returned during SELECT queries. [#19731](https://github.com/StarRocks/starrocks/issues/19731) * Supports SHOW/SET CHARACTER. [#17480](https://github.com/StarRocks/starrocks/issues/17480) * When the loaded data exceeds the field length supported by StarRocks, the error message returned is not correct. [#14](https://github.com/StarRocks/DataX/issues/14) * Supports `show full fields from 'table'`. [#17233](https://github.com/StarRocks/starrocks/issues/17233) * Partition pruning causes MV rewrites to fail. [#14641](https://github.com/StarRocks/starrocks/issues/14641) * MV rewrite fails when the CREATE MATERIALIZED VIEW statement contains `count(distinct)` and `count(distinct)` is applied to the DISTRIBUTED BY column. [#16558](https://github.com/StarRocks/starrocks/issues/16558) * FEs fail to start when a VARCHAR column is used as the partitioning column of a materialized view. [#19366](https://github.com/StarRocks/starrocks/issues/19366) * Window functions [LEAD](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function#lead) and [LAG](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function#lag) incorrectly handle IGNORE NULLS. [#21001](https://github.com/StarRocks/starrocks/pull/21001) * Adding temporary partitions conflicts with automatic partition creation. [#21222](https://github.com/StarRocks/starrocks/issues/21222) ##### Behavior Change[​](#behavior-change-2 "Direct link to Behavior Change") * The new role-based access control (RBAC) system supports the previous privileges and roles. However, the syntax of related statements such as [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT/) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE/) is changed. * Renamed SHOW MATERIALIZED VIEW as [SHOW MATERIALIZED VIEWS](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/SHOW_MATERIALIZED_VIEW/). * Added the following [Reserved keywords](https://docs.starrocks.io/docs/sql-reference/sql-statements/keywords/): AUTO\_INCREMENT, CURRENT\_ROLE, DEFERRED, ENCLOSE, ESCAPE, IMMEDIATE, PRIVILEGES, SKIP\_HEADER, TRIM\_SPACE, VARBINARY. ##### Upgrade Notes[​](#upgrade-notes-2 "Direct link to Upgrade Notes") You can upgrade from v2.5 to v3.0 or downgrade from v3.0 to v2.5. > In theory, an upgrade from a version earlier than v2.5 is also supported. To ensure system availability, we recommend that you first upgrade your cluster to v2.5 and then to v3.0. Take note of the following points when you perform a downgrade from v3.0 to v2.5. ###### BDBJE[​](#bdbje "Direct link to BDBJE") StarRocks upgrades the BDB library in v3.0. However, BDBJE cannot be rolled back. You must use BDB library of v3.0 after a downgrade. Perform the following steps: 1. After you replace the FE package with a v2.5 package, copy `fe/lib/starrocks-bdb-je-18.3.13.jar` of v3.0 to the `fe/lib` directory of v2.5. 2. Delete `fe/lib/je-7.*.jar`. ###### Privilege system[​](#privilege-system "Direct link to Privilege system") The new RBAC privilege system is used by default after you upgrade to v3.0. You can only downgrade to v2.5. After a downgrade, run [ALTER SYSTEM CREATE IMAGE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM/) to create a new image and wait for the new image to be synchronized to all follower FEs. If you do not run this command, some of the downgrade operations may fail. This command is supported from 2.5.3 and later. For details about the differences between the privilege system of v2.5 and v3.0, see "Upgrade notes" in [Privileges supported by StarRocks](https://docs.starrocks.io/docs/administration/privilege_item/). --- ## Release 3.1 ### StarRocks version 3.1 #### 3.1.17[​](#3117 "Direct link to 3.1.17") Release Date: January 3, 2025 ##### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") Fixed the following issues: * Cross-cluster Data Migration Tool caused the Follower FE to crash during data synchronization and commit, due to not accounting for the deletion of partitions in the target cluster. [#54061](https://github.com/StarRocks/starrocks/pull/54061) * BE in the target cluster might crash when synchronizing tables with DELETE operations using Cross-cluster Data Migration Tool. [#54081](https://github.com/StarRocks/starrocks/pull/54081) * A bug in the BDBJE handshake mechanism where Leader FE would reject reconnection attempts from Follower FE when connection is being re-established, causing Follower FE nodes to exit. [#50412](https://github.com/StarRocks/starrocks/pull/50412) * Duplicate memory statistics in FE leads to excessive memory usage. [#53055](https://github.com/StarRocks/starrocks/pull/53055) * The statuses of asynchronous materialized view refresh tasks are inconsistent across multiple FE nodes, which lead to inaccurate states of the materialized view during queries. [#54236](https://github.com/StarRocks/starrocks/pull/54236) #### 3.1.16[​](#3116 "Direct link to 3.1.16") Release date: December 16, 2024 ##### Improvements[​](#improvements "Direct link to Improvements") * Optimized table-related statistics. [#50316](https://github.com/StarRocks/starrocks/pull/50316) ##### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") Fixed the following issues: * Insufficient granularity in error code handling for disk full scenarios caused the BE to mistakenly identify disk errors and delete data. [#51411](https://github.com/StarRocks/starrocks/pull/51411) * Stream Load failures when submitted using HTTP 1.0. [#53010](https://github.com/StarRocks/starrocks/pull/53010) [#53008](https://github.com/StarRocks/starrocks/pull/53008) * Routine Load tasks were canceled due to expired transactions (now tasks are canceled only if the database or table no longer exists and paused when transactions expired). [#50334](https://github.com/StarRocks/starrocks/pull/50334) * Unloading data using `EXPORT` with Broker to `file://` resulted in a file rename error, causing the export to fail. [#52544](https://github.com/StarRocks/starrocks/pull/52544) * If the join condition in an equal-join is an expression based on a low-cardinality column, the system may incorrectly push down a Runtime Filter predicate, leading to a BE crash. [#50690](https://github.com/StarRocks/starrocks/pull/50690) #### 3.1.15[​](#3115 "Direct link to 3.1.15") Release date: September 4, 2024 ##### Bug Fixes[​](#bug-fixes-2 "Direct link to Bug Fixes") Fixed the following issues: * During query rewrite with asynchronous materialized views, `count(*)` on certain tables returns NULL. [#49288](https://github.com/StarRocks/starrocks/pull/49288) * `partition_linve_nubmer` does not take effect. [#49213](https://github.com/StarRocks/starrocks/pull/49213) * FE throws a tablet exception: BE disk offline, and cannot migrate tablets. [#47833](https://github.com/StarRocks/starrocks/pull/47833) #### 3.1.14[​](#3114 "Direct link to 3.1.14") Release date: July 29, 2024 ##### Improvements[​](#improvements-1 "Direct link to Improvements") * Stream Load now supports using `\t` and `\n` as row and column delimiters. Users do not need to convert them to their hexadecimal ASCII codes. [#47302](https://github.com/StarRocks/starrocks/pull/47302) ##### Bug Fixes[​](#bug-fixes-3 "Direct link to Bug Fixes") Fixed the following issues: * Frequent INSERT and UPDATE operations on Primary Key tables may cause write and query delays in the database. [#47838](https://github.com/StarRocks/starrocks/pull/47838) * When a Primary Key table encounters data persistence failures, the persistent index may fail to capture the error, leading to data loss and reporting the error "Insert found duplicate key". [#48045](https://github.com/StarRocks/starrocks/pull/48045) * Materialized views may report insufficient permissions when refreshed. [#47561](https://github.com/StarRocks/starrocks/pull/47561) * Materialized view reports the error "For input string" when refreshed. [#46131](https://github.com/StarRocks/starrocks/pull/46131) * During materialized view refresh, the lock is held excessively long, causing the Leader FE to be restarted by the deadlock detection script. [#48256](https://github.com/StarRocks/starrocks/pull/48256) * Queries against views with the IN clause in its definition may return inaccurate results. [#47484](https://github.com/StarRocks/starrocks/pull/47484) * Global Runtime Filter causes incorrect results. [#48496](https://github.com/StarRocks/starrocks/pull/48496) * MySQL protocol `COM_CHANGE_USER` does not support `conn_attr`. [#47796](https://github.com/StarRocks/starrocks/pull/47796) ##### Behavior Changes[​](#behavior-changes "Direct link to Behavior Changes") * When users create a non-partitioned table without specifying the bucket number, the minimum bucket number the system sets for the table is `16` (instead of `2` based on the formula `2*BE or CN count`). If users want to set a smaller bucket number when creating a small table, they must set it explicitly. [#47005](https://github.com/StarRocks/starrocks/pull/47005) #### 3.1.13[​](#3113 "Direct link to 3.1.13") Release date: June 26, 2024 ##### Improvements[​](#improvements-2 "Direct link to Improvements") * The Broker process supports access to Tencent Cloud COS Posix buckets. Users can load data from COS Posix buckets using Broker Load or unload data to COS Posix buckets using the SELECT INTO OUTFILE statement. [#46597](https://github.com/StarRocks/starrocks/pull/46597) * Supports viewing comments of Hive tables in Hive Catalogs using SHOW CREATE TABLE. [#37686](https://github.com/StarRocks/starrocks/pull/37686) * Optimized the evaluation time of Conjunct in WHERE clauses, such as multiple LIKE clauses on the same column or CASE WHEN expressions. [#46914](https://github.com/StarRocks/starrocks/pull/46914) ##### Bug Fixes[​](#bug-fixes-4 "Direct link to Bug Fixes") Fixed the following issues: * DELETE statements fail in shared-data clusters if there are excessive number of partitions to be deleted. [#46229](https://github.com/StarRocks/starrocks/pull/46229) #### 3.1.12[​](#3112 "Direct link to 3.1.12") Release date: May 30, 2024 ##### New Features[​](#new-features "Direct link to New Features") * Flink connector supports reading complex data types ARRAY, MAP, and STRUCT from StarRocks. [#42932](https://github.com/StarRocks/starrocks/pull/42932) [#347](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/347) ##### Improvements[​](#improvements-3 "Direct link to Improvements") * Previously, when BE failed to communicate with FE via RPC, FE would return a generic error message: `call frontend service failed reason=xxx`, making it unclear what the specific issue was. The error messages are now optimized to include specific reasons, such as timeout or server busy. [#44153](https://github.com/StarRocks/starrocks/pull/44153) * Improved error messages to indicate specific issues during data loading, such as the number of error data rows exceeding limits, mismatched column numbers, invalid column names, and no data in any partition. ##### Security[​](#security "Direct link to Security") * Upgraded Kafka client dependency to v3.4.0 to fix the CVE-2023-25194 security issue. [#45382](https://github.com/StarRocks/starrocks/pull/45382) ##### Bug Fixes[​](#bug-fixes-5 "Direct link to Bug Fixes") Fixed the following issues: * If a materialized view definition includes multiple self-joins of the same table and incremental refreshes by partitions based on that table, incorrect results would occur due to wrong partition selection. [#45936](https://github.com/StarRocks/starrocks/pull/45936) * FEs crash when a Bitmap index is created in a materialized view in shared-data clusters. [#45665](https://github.com/StarRocks/starrocks/pull/45665) * BEs crash due to null pointer issues when FE follower is connected via ODBC and CREATE TABLE is executed. [#45043](https://github.com/StarRocks/starrocks/pull/45043) * Querying `information_schema.task_runs` fails frequently when many asynchronous tasks exist. [#45520](https://github.com/StarRocks/starrocks/pull/45520) * When a SQL statement contains multiple COUNT DISTINCT and includes LIMIT, LIMIT is wrongly processed, resulting in inconsistent data returned each time the statement is executed. [#44749](https://github.com/StarRocks/starrocks/pull/44749) * Queries with ORDER BY LIMIT clauses on Duplicate Key tables and Aggregate tables produce incorrect results. [#45037](https://github.com/StarRocks/starrocks/pull/45037) #### 3.1.11[​](#3111 "Direct link to 3.1.11") Release date: April 28, 2024 ##### Behavior Changes[​](#behavior-changes-1 "Direct link to Behavior Changes") * Users are not allowed to drop views in the system database `information_schema` using DROP TABLE. [#43556](https://github.com/StarRocks/starrocks/pull/43556) * Users are not allowed to specify duplicate keys in the ORDER BY clause when creating a Primary Key table. [#43374](https://github.com/StarRocks/starrocks/pull/43374) ##### Improvements[​](#improvements-4 "Direct link to Improvements") * Queries on Parquet-formatted Iceberg v2 tables support equality deletes. ##### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") Fixed the following issues: * When a user queries data from an external table in an external catalog, access to this table is denied even when the user has the SELECT privilege on this table. SHOW GRANTS also shows that the user has this privilege. [#44061](https://github.com/StarRocks/starrocks/pull/44061) * `str_to_map` may cause BEs to crash. [#43930](https://github.com/StarRocks/starrocks/pull/43930) * When a Routine Load job is going on, running `show proc '/routine_loads'` is stuck due to deadlock. [#44249](https://github.com/StarRocks/starrocks/pull/44249) * Persistent Index of Primary Key tables may cause BEs to crash due to issues in concurrency control. [#43720](https://github.com/StarRocks/starrocks/pull/43720) * The `pending_task_run_count` displayed on the page of `leaderFE_IP:8030` is incorrect. The displayed number is the sum of Pending and Running tasks, not Pending tasks. In addition, the information of the metric `refresh_pending` cannot be displayed using `followerFE_IP:8030`. [#43052](https://github.com/StarRocks/starrocks/pull/43052) * Querying `information_schema.task_runs` fails frequently. [#43052](https://github.com/StarRocks/starrocks/pull/43052) * Some SQL queries that contain CTEs may encounter the `Invalid plan: PhysicalTopNOperator` error. [#44185](https://github.com/StarRocks/starrocks/pull/44185) #### 3.1.10 (Yanked)[​](#3110-yanked "Direct link to 3.1.10 (Yanked)") tip This version has been taken offline due to privilege issues in querying external tables in external catalogs such as Hive and Iceberg. * **Problem**: When a user queries data from an external table in an external catalog, access to this table is denied even when the user has the SELECT privilege on this table. SHOW GRANTS also shows that the user has this privilege. * **Impact scope**: This problem only affects queries on external tables in external catalogs. Other queries are not affected. * **Temporary workaround**: The query succeeds after the SELECT privilege on this table is granted to the user again. But `SHOW GRANTS` will return duplicate privilege entries. After an upgrade to v3.1.11, users can run `REVOKE` to remove one of the privilege entries. Release date: March 29, 2024 ##### New Features[​](#new-features-1 "Direct link to New Features") * Primary Key tables support Size-tiered Compaction. [#42474](https://github.com/StarRocks/starrocks/pull/42474) * Added a pattern-matching function `regexp_extract_all`. [#42178](https://github.com/StarRocks/starrocks/pull/42178) ##### Behavior Changes[​](#behavior-changes-2 "Direct link to Behavior Changes") * When null values in JSON data are evaluated based on the `IS NULL` operator, they are considered NULL values following SQL language. For example, `true` is returned for `SELECT parse_json('{"a": null}') -> 'a' IS NULL` (before this behavior change, `false` is returned). [#42815](https://github.com/StarRocks/starrocks/pull/42815) ##### Improvements[​](#improvements-5 "Direct link to Improvements") * When Broker Load is used to load data from ORC files that contain TIMESTAMP-type data, StarRocks supports retaining microseconds in the timestamps when converting the timestamps to match its own DATETIME data type. [#42348](https://github.com/StarRocks/starrocks/pull/42348) ##### Bug Fixes[​](#bug-fixes-7 "Direct link to Bug Fixes") Fixed the following issues: * In shared-data mode, the garbage collection and thread eviction mechanisms for handling persistent indexes created on Primary Key tables cannot take effect on CN nodes. As a result, obsolete data cannot be deleted. [#42241](https://github.com/StarRocks/starrocks/pull/42241) * When users query ORC files by using Hive catalogs, the query results may be incorrect because StarRocks used to read ORC files from Hive based on mapping by position. To resolve this issue, users can set the session variable `orc_use_column_names` to `true`, which specifies to read ORC files from Hive based on mapping by column name. [#42905](https://github.com/StarRocks/starrocks/pull/42905) * When LDAP authentication for the AD system is adopted, logins without passwords are allowed. [#42476](https://github.com/StarRocks/starrocks/pull/42476) * When disk device names end with digits, the values of monitoring metrics remain 0s because the disk device names may be invalid after such digits are removed. [#42741](https://github.com/StarRocks/starrocks/pull/42741) #### 3.1.9[​](#319 "Direct link to 3.1.9") Release date: March 8, 2024 ##### New Features[​](#new-features-2 "Direct link to New Features") * Cloud-native Primary Key tables in shared-data clusters support Size-tiered Compaction to reduce write I/O amplification for the loading of a large number of small-sized files. [#41610](https://github.com/StarRocks/starrocks/pull/41610) * Added the view `information_schema.partitions_meta`, which records detailed metadata of partitions. [#41101](https://github.com/StarRocks/starrocks/pull/41101) * Added the view `sys.fe_memory_usage`, which records the memory usage for StarRocks. [#41083](https://github.com/StarRocks/starrocks/pull/41083) ##### Behavior Changes[​](#behavior-changes-3 "Direct link to Behavior Changes") * The logic of dynamic partitioning is changed. Now partition columns of the DATE type do not support hour-level data. Note that partition columns of the DATETIME type still support hour-level data. [#40328](https://github.com/StarRocks/starrocks/pull/40328) * The user who can refresh materialized views is changed from the `root` user to the user who creates the materialized views. This change does not affect existing materialized views. [#40698](https://github.com/StarRocks/starrocks/pull/40698) * By default, when comparing columns of constant and string types, StarRocks compares them as strings. Users can use the session variable `cbo_eq_base_type` to adjust the default rule used for the comparison. For example, users can set `cbo_eq_base_type` to `decimal`, and StarRocks then compares the columns as numeric values. [#41712](https://github.com/StarRocks/starrocks/pull/41712) ##### Improvements[​](#improvements-6 "Direct link to Improvements") * StarRocks supports using the parameter `s3_compatible_fs_list` to specify which S3-compatible object storage can be accessed via AWS SDK, and supports using the parameter `fallback_to_hadoop_fs_list` to specify non-S3-compatible object storage that require access via HDFS Schema (this method necessitates the use of vendor-provided JAR packages). [#41612](https://github.com/StarRocks/starrocks/pull/41612) * The compatibility with Trino's SQL statement syntax is optimized to support converting the following functions of Trino: `current_catalog`, `current_schema`, `to_char`, `from_hex`, `to_date`, `to_timestamp`, and `index`. [#41505](https://github.com/StarRocks/starrocks/pull/41505) [#41270](https://github.com/StarRocks/starrocks/pull/41270) [#40838](https://github.com/StarRocks/starrocks/pull/40838) * A new session variable `cbo_materialized_view_rewrite_related_mvs_limit` is added to control the maximum number of candidate materialized views allowed during query planning. The default value of this session variable is `64`. This session variable helps mitigate the excessive resource consumption caused by a large number of candidate materialized views for a query during the query planning. [#39829](https://github.com/StarRocks/starrocks/pull/39829) * The `agg_type` of BITMAP-type columns in an Aggregate table can be set to `replace_if_not_null` to support updates only to a few columns of the table. [#42102](https://github.com/StarRocks/starrocks/pull/42102) * The session variable `cbo_eq_base_type` is optimized to support specifying the implicit conversion rule applied to the comparison of data that contains both string and numeric data types. By default, such data is compared as strings. [#40619](https://github.com/StarRocks/starrocks/pull/41712) * More DATE-type data (for example, "%Y-%m-%e %H:%i") can be recognized to better support partition expressions for Iceberg tables. [#40474](https://github.com/StarRocks/starrocks/pull/40474) * The JDBC connector supports the TIME data type. [#31940](https://github.com/StarRocks/starrocks/pull/31940) * The `path` parameter in the SQL statement for creating a file external table supports wildcards (`*`). However, like the `DATA INFILE` parameter in the SQL statement for creating a Broker Load job, the `path` parameter supports using wildcards (`*`) to match at most one level of directory or file. [#40844](https://github.com/StarRocks/starrocks/pull/40844) * A new internal SQL log file is added to record log data related to statistics and materialized views. [#40682](https://github.com/StarRocks/starrocks/pull/40682) ##### Bug Fixes[​](#bug-fixes-8 "Direct link to Bug Fixes") Fixed the following issues: * "Analyze Error" is thrown if inconsistent letter cases are assigned to the names or aliases of tables or views queried in the creation of a Hive view. [#40921](https://github.com/StarRocks/starrocks/pull/40921) * I/O usage reaches the upper limit if persistent indexes are created on Primary Key tables. [#39959](https://github.com/StarRocks/starrocks/pull/39959) * In shared-data clusters, the primary key index directory is deleted every 5 hours. [#40745](https://github.com/StarRocks/starrocks/pull/40745) * After a table for which list partitioning is enabled is truncated or its partitions are truncated, queries based on the partitioning keys of the table return no data. [#40495](https://github.com/StarRocks/starrocks/pull/40495) * After users execute ALTER TABLE COMPACT by hand, the memory usage statistics for compaction operations are abnormal. [#41150](https://github.com/StarRocks/starrocks/pull/41150) * During data migration between clusters, if only some columns are updated in column mode, the destination cluster may crash. [#40692](https://github.com/StarRocks/starrocks/pull/40692) * The SQL blacklist may not take effect if the submitted SQL statement contains multiple spaces or newline characters. [#40457](https://github.com/StarRocks/starrocks/pull/40457) #### 3.1.8[​](#318 "Direct link to 3.1.8") Release date: February 5, 2024 ##### New Features[​](#new-features-3 "Direct link to New Features") * StarRocks Community provides the StarRocks Cross-cluster Data Migration Tool, which supports migrating data from a shared-nothing cluster to either another shared-nothing cluster or a shared-data cluster. * Supports creating synchronous materialized views with the WHERE clause specified. * Added metrics that show memory usage of the data cache to MemTracker. [#39600](https://github.com/StarRocks/starrocks/pull/39600) ##### Parameter Change[​](#parameter-change "Direct link to Parameter Change") * Added a BE configuration item, `lake_pk_compaction_max_input_rowsets`, which controls the maximum number of input rowsets allowed in a Primary Key table compaction task in a shared-data StarRocks cluster. This helps optimize resource consumption for compaction tasks. [#39611](https://github.com/StarRocks/starrocks/pull/39611) ##### Improvements[​](#improvements-7 "Direct link to Improvements") * Supports ORDER BY and INDEX clauses in CTAS statements. [#38886](https://github.com/StarRocks/starrocks/pull/38886) * Supports equality deletes on ORC-formatted Iceberg v2 tables. [#37419](https://github.com/StarRocks/starrocks/pull/37419) * Supports setting the `datacache.partition_duration` property for cloud-native tables created with the list partitioning strategy. This property controls the validity period of the data cache and can be dynamically configured. [#35681](https://github.com/StarRocks/starrocks/pull/35681) [#38509](https://github.com/StarRocks/starrocks/pull/38509) * Optimized the BE configuration item `update_compaction_per_tablet_min_interval_seconds`. This parameter is originally used only to control the frequency of compaction tasks on Primary Key tables. After the optimization, it can also be used to control the frequency of major compaction tasks on Primary Key table indexes. [#39640](https://github.com/StarRocks/starrocks/pull/39640) * Parquet Reader supports converting INT32-type data in Parquet-formatted data to DATETIME-type data and storing the resulting data to StarRocks. [#39808](https://github.com/StarRocks/starrocks/pull/39808) ##### Bug Fixes[​](#bug-fixes-9 "Direct link to Bug Fixes") Fixed the following issues: * Using NaN (Not a Number) columns as ORDER BY columns may cause BEs to crash. [#30759](https://github.com/StarRocks/starrocks/pull/30759) * Failure to update primary key indexes may cause the error "get\_applied\_rowsets failed". [#27488](https://github.com/StarRocks/starrocks/pull/27488) * The resources occupied by compaction\_state\_cache are not recycled after compaction task failures. [#38499](https://github.com/StarRocks/starrocks/pull/38499) * If partition columns in external tables contain null values, queries against those tables will cause BEs to crash. [#38888](https://github.com/StarRocks/starrocks/pull/38888) * After a table is dropped and then re-created with the same table name, refreshing asynchronous materialized views created on that table fails. [#38008](https://github.com/StarRocks/starrocks/pull/38008) * Refreshing asynchronous materialized views created on empty Iceberg tables fail. [#24068](https://starrocks.atlassian.net/browse/SR-24068) #### 3.1.7[​](#317 "Direct link to 3.1.7") Release date: January 12, 2024 ##### New Features[​](#new-features-4 "Direct link to New Features") * Added a new function, `unnest_bitmap`. [#38136](https://github.com/StarRocks/starrocks/pull/38136) * Supports conditional updates for [Broker Load](https://docs.starrocks.io/docs/3.1/sql-reference/sql-statements/data-manipulation/BROKER_LOAD/#opt_properties). [#37400](https://github.com/StarRocks/starrocks/pull/37400) ##### Behavior Change[​](#behavior-change "Direct link to Behavior Change") * Added the session variable `enable_materialized_view_for_insert`, which controls whether materialized views rewrite the queries in INSERT INTO SELECT statements. The default value is `false`. [#37505](https://github.com/StarRocks/starrocks/pull/37505) * The FE dynamic parameter `enable_new_publish_mechanism` is changed to a static parameter. You must restart the FE after you modify the parameter settings. [#35338](https://github.com/StarRocks/starrocks/pull/35338) * Added the session variable `enable_strict_order_by`. When this variable is set to the default value `TRUE`, an error is reported for such a query pattern: Duplicate alias is used in different expressions of the query and this alias is also a sorting field in ORDER BY, for example, `select distinct t1.* from tbl1 t1 order by t1.k1;`. The logic is the same as that in v2.3 and earlier. When this variable is set to `FALSE`, a loose deduplication mechanism is used, which processes such queries as valid SQL queries. [#37910](https://github.com/StarRocks/starrocks/pull/37910) ##### Parameter Change[​](#parameter-change-1 "Direct link to Parameter Change") * Added the FE configuration item `routine_load_unstable_threshold_second`. [#36222](https://github.com/StarRocks/starrocks/pull/36222) * Added the FE configuration item `http_worker_threads_num`, which specifies the number of threads for HTTP server to deal with HTTP requests. The default value is `0`. If the value for this parameter is set to a negative value or `0`, the actual thread number is twice the number of CPU cores. [#37530](https://github.com/StarRocks/starrocks/pull/37530) * Added the BE configuration item `pindex_major_compaction_limit_per_disk` to configure the maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. The default value is `1`. [#36681](https://github.com/StarRocks/starrocks/pull/36681) * Added session variables `transaction_read_only` and `tx_read_only` to specify the transaction access mode, which are compatible with MySQL versions 5.7.20 and above. [#37249](https://github.com/StarRocks/starrocks/pull/37249) * Added the FE configuration item `default_mv_refresh_immediate`, which specifies whether to immediately refresh the materialized view after the materialized view is created. The default value is `true`. [#37093](https://github.com/StarRocks/starrocks/pull/37093) * Added a new BE configuration item `lake_enable_vertical_compaction_fill_data_cache`, which specifies whether to allow compaction tasks to cache data on local disks in a shared-data cluster. The default value is `false`. [#37296](https://github.com/StarRocks/starrocks/pull/37296) ##### Improvements[​](#improvements-8 "Direct link to Improvements") * INSERT INTO FILE() SELECT FROM supports reading BINARY-type data from tables and exporting the data to Parquet-formatted files in remote storage. [#36797](https://github.com/StarRocks/starrocks/pull/36797) * Asynchronous materialized views support dynamically setting the `datacache.partition_duration` property, which controls the validity period of the hot data in the data cache. [#35681](https://github.com/StarRocks/starrocks/pull/35681) * Wen using JDK, the default GC algorithm is G1. [#37386](https://github.com/StarRocks/starrocks/pull/37386) * The `date_trunc`, `adddate`, and `time_slice` functions support setting the `interval` parameter to values that are accurate to the millisecond and microsecond. [#36386](https://github.com/StarRocks/starrocks/pull/36386) * When the string on the right side of the LIKE operator within the WHERE clause does not include `%` or `_`, the LIKE operator is converted into the `=` operator. [#37515](https://github.com/StarRocks/starrocks/pull/37515) * A new field `LatestSourcePosition` is added to the return result of [SHOW ROUTINE LOAD](https://docs.starrocks.io/zh/docs/3.1/sql-reference/sql-statements/data-manipulation/SHOW_ROUTINE_LOAD/) to record the position of the latest message in each partition of the Kafka topic, helping check the latencies of data loading. [#38298](https://github.com/StarRocks/starrocks/pull/38298) * Added a new resource group property, `spill_mem_limit_threshold`, to control the memory usage threshold (percentage) at which a resource group triggers the spilling of intermediate results when the system variable `spill_mode` is set to `auto`. The valid range is (0, 1). The default value is `1`, indicating the threshold does not take effect. [#37707](https://github.com/StarRocks/starrocks/pull/37707) * The result returned by the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/3.1/sql-reference/sql-statements/data-manipulation/SHOW_ROUTINE_LOAD/) statement now includes the timestamps of consumption messages from each partition. [#36222](https://github.com/StarRocks/starrocks/pull/36222) * The scheduling policy for Routine Load is optimized, so that slow tasks do not block the execution of the other normal tasks. [#37638](https://github.com/StarRocks/starrocks/pull/37638) ##### Bug Fixes[​](#bug-fixes-10 "Direct link to Bug Fixes") Fixed the following issues: * The execution of [ANALYZE TABLE](https://docs.starrocks.io/docs/3.1/sql-reference/sql-statements/data-definition/ANALYZE_TABLE/) gets stuck occasionally. [#36836](https://github.com/StarRocks/starrocks/pull/36836) * The memory consumption by PageCache exceeds the threshold specified by the BE dynamic parameter `storage_page_cache_limit` in certain circumstances. [#37740](https://github.com/StarRocks/starrocks/pull/37740) * Hive metadata in [Hive catalogs](https://docs.starrocks.io/docs/3.1/data_source/catalog/hive_catalog/) is not automatically refreshed when new fields are added to Hive tables. [#37668](https://github.com/StarRocks/starrocks/pull/37668) * In some cases, `bitmap_to_string` may return incorrect results due to data type overflow. [#37405](https://github.com/StarRocks/starrocks/pull/37405) * Executing the DELETE statement on an empty table returns "ERROR 1064 (HY000): Index: 0, Size: 0". [#37461](https://github.com/StarRocks/starrocks/pull/37461) * When the FE dynamic parameter `enable_sync_publish` is set to `TRUE`, queries on data that is written after the BEs crash and then restart may fail. [#37398](https://github.com/StarRocks/starrocks/pull/37398) * The value of the `TABLE_CATALOG` field in `views` of the StarRocks Information Schema is `null`. [#37570](https://github.com/StarRocks/starrocks/pull/37570) * When `SELECT ... FROM ... INTO OUTFILE` is executed to export data into CSV files, the error "Unmatched number of columns" is reported if the FROM clause contains multiple constants. [#38045](https://github.com/StarRocks/starrocks/pull/38045) #### 3.1.6[​](#316 "Direct link to 3.1.6") Release date: December 18, 2023 ##### New Features[​](#new-features-5 "Direct link to New Features") * Added the [now(p)](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/now/) function to return the current date and time with the specified fractional seconds precision (accurate to the microsecond). If `p` is not specified, this function returns only date and time accurate to the second. [#36676](https://github.com/StarRocks/starrocks/pull/36676) * Added a new metric `max_tablet_rowset_num` for setting the maximum allowed number of rowsets. This metric helps detect possible compaction issues and thus reduces the occurrences of the error "too many versions". [#36539](https://github.com/StarRocks/starrocks/pull/36539) * Supports obtaining heap profiles by using a command line tool, making troubleshooting easier.[#35322](https://github.com/StarRocks/starrocks/pull/35322) * Supports creating asynchronous materialized views with common table expressions (CTEs). [#36142](https://github.com/StarRocks/starrocks/pull/36142) * Added the following bitmap functions: [subdivide\_bitmap](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/subdivide_bitmap/), [bitmap\_from\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_from_binary/), and [bitmap\_to\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_to_binary/). [#35817](https://github.com/StarRocks/starrocks/pull/35817) [#35621](https://github.com/StarRocks/starrocks/pull/35621) * Optimized the logic used to compute compaction scores for Primary Key tables, thereby aligning the compaction scores for Primary Key tables within a more consistent range with the other three table types. [#36534](https://github.com/StarRocks/starrocks/pull/36534) ##### Parameter Change[​](#parameter-change-2 "Direct link to Parameter Change") * The default retention period of trash files is changed to 1 day from the original 3 days. [#37113](https://github.com/StarRocks/starrocks/pull/37113) * A new BE configuration item `enable_stream_load_verbose_log` is added. The default value is `false`. With this parameter set to `true`, StarRocks can record the HTTP requests and responses for Stream Load jobs, making troubleshooting easier. [#36113](https://github.com/StarRocks/starrocks/pull/36113) * A new BE configuration item `enable_lazy_delta_column_compaction` is added. The default value is `true`, indicating that StarRocks does not perform frequent compaction operations on delta columns. [#36654](https://github.com/StarRocks/starrocks/pull/36654) * A new FE configuration item `enable_mv_automatic_active_check` is added to control whether the system automatically checks and re-activates the asynchronous materialized views that are set inactive because their base tables (views) had undergone Schema Change or had been dropped and re-created. The default value is `true`. [#36463](https://github.com/StarRocks/starrocks/pull/36463) ##### Improvements[​](#improvements-9 "Direct link to Improvements") * A new value option `GROUP_CONCAT_LEGACY` is added to the session variable [sql\_mode](https://docs.starrocks.io/docs/reference/System_variable/#sql_mode) to provide compatibility with the implementation logic of the [group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat/) function in versions earlier than v2.5. [#36150](https://github.com/StarRocks/starrocks/pull/36150) * The Primary Key table size returned by the [SHOW DATA](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/SHOW_DATA/) statement includes the sizes of **.cols** files (these are files related to partial column updates and generated columns) and persistent index files. [#34898](https://github.com/StarRocks/starrocks/pull/34898) * Queries on MySQL external tables and the external tables within JDBC catalogs support including keywords in the WHERE clause. [#35917](https://github.com/StarRocks/starrocks/pull/35917) * Plugin loading failures will no longer cause an error or cause an FE start failure. Instead, the FE can properly start, and the error status of the plug-in can be queried using [SHOW PLUGINS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/SHOW_PLUGINS/). [#36566](https://github.com/StarRocks/starrocks/pull/36566) * Dynamic partitioning supports random distribution. [#35513](https://github.com/StarRocks/starrocks/pull/35513) * The result returned by the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/SHOW_ROUTINE_LOAD/) statement provides a new field `OtherMsg`, which shows information about the last failed task. [#35806](https://github.com/StarRocks/starrocks/pull/35806) * The authentication information `aws.s3.access_key` and `aws.s3.access_secret` for AWS S3 in Broker Load jobs are hidden in audit logs. [#36571](https://github.com/StarRocks/starrocks/pull/36571) * The `be_tablets` view in the `information_schema` database provides a new field `INDEX_DISK`, which records the disk usage (measured in bytes) of persistent indexes [#35615](https://github.com/StarRocks/starrocks/pull/35615) ##### Bug Fixes[​](#bug-fixes-11 "Direct link to Bug Fixes") Fixed the following issues: * The BEs crash if users create persistent indexes in the event of data corruption. [#30841](https://github.com/StarRocks/starrocks/pull/30841) * If users create an asynchronous materialized view that contains nested queries, the error "resolve partition column failed" is reported. [#26078](https://github.com/StarRocks/starrocks/issues/26078) * If users create an asynchronous materialized view on a base table whose data is corrupted, the error "Unexpected exception: null" is reported. [#30038](https://github.com/StarRocks/starrocks/pull/30038) * If users run a query that contains a window function, the SQL error "\[1064] \[42000]: Row count of const column reach limit: 4294967296" is reported. [#33561](https://github.com/StarRocks/starrocks/pull/33561) * The FE performance plunges after the FE configuration item `enable_collect_query_detail_info` is set to `true`. [#35945](https://github.com/StarRocks/starrocks/pull/35945) * In the StarRocks shared-data mode, the error "Reduce your request rate" may be reported when users attempt to delete files from object storage. [#35566](https://github.com/StarRocks/starrocks/pull/35566) * Deadlocks may occur when users refresh materialized views. [#35736](https://github.com/StarRocks/starrocks/pull/35736) * After the DISTINCT window operator pushdown feature is enabled, errors are reported if SELECT DISTINCT operations are performed on the complex expressions of the columns computed by window functions. [#36357](https://github.com/StarRocks/starrocks/pull/36357) * The BEs crash if the source data file is in ORC format and contains nested arrays. [#36127](https://github.com/StarRocks/starrocks/pull/36127) * Some S3-compatible object storage returns duplicate files, causing the BEs to crash. [#36103](https://github.com/StarRocks/starrocks/pull/36103) * The [array\_distinct](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_distinct/) function occasionally causes the BEs to crash. [#36377](https://github.com/StarRocks/starrocks/pull/36377) * Global Runtime Filter may cause BEs to crash in certain scenarios. [#35776](https://github.com/StarRocks/starrocks/pull/35776) #### 3.1.5[​](#315 "Direct link to 3.1.5") Release date: November 28, 2023 ##### New features[​](#new-features-6 "Direct link to New features") * The CN nodes of a StarRocks shared-data cluster now support data export. [#34018](https://github.com/StarRocks/starrocks/pull/34018) ##### Improvements[​](#improvements-10 "Direct link to Improvements") * The [`COLUMNS`](https://docs.starrocks.io/docs/reference/information_schema/columns/) view in the system database `INFORMATION_SCHEMA` can display ARRAY, MAP, and STRUCT columns. [#33431](https://github.com/StarRocks/starrocks/pull/33431) * Supports queries against Parquet, ORC, and CSV formatted files that are compressed by using LZO and stored in [Hive](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog/). [#30923](https://github.com/StarRocks/starrocks/pull/30923) [#30721](https://github.com/StarRocks/starrocks/pull/30721) * Supports updates onto the specified partitions of an automatically partitioned table. If the specified partitions do not exist, an error is returned. [#34777](https://github.com/StarRocks/starrocks/pull/34777) * Supports automatic refresh of materialized views when Swap, Drop, or Schema Change operations are performed on the tables and views (including the other tables and materialized views associated with these views) on which these materialized views are created. [#32829](https://github.com/StarRocks/starrocks/pull/32829) * Optimized the performance of some Bitmap-related operations, including: * Optimized nested loop joins. [#340804](https://github.com/StarRocks/starrocks/pull/34804) [#35003](https://github.com/StarRocks/starrocks/pull/35003) * Optimized the `bitmap_xor` function. [#34069](https://github.com/StarRocks/starrocks/pull/34069) * Supports Copy on Write to optimize Bitmap performance and reduce memory consumption. [#34047](https://github.com/StarRocks/starrocks/pull/34047) ##### Bug Fixes[​](#bug-fixes-12 "Direct link to Bug Fixes") Fixed the following issues: * If a filtering condition is specified in a Broker Load job, BEs may crash during the data loading in certain circumstances. [#29832](https://github.com/StarRocks/starrocks/pull/29832) * An unknown error is reported when SHOW GRANTS is executed. [#30100](https://github.com/StarRocks/starrocks/pull/30100) * When data is loaded into a table that uses expression-based automatic partitioning, the error "Error: The row create partition failed since Runtime error: failed to analyse partition value" may be thrown. [#33513](https://github.com/StarRocks/starrocks/pull/33513) * The error "get\_applied\_rowsets failed, tablet updates is in error state: tablet:18849 actual row size changed after compaction" is returned for queries. [#33246](https://github.com/StarRocks/starrocks/pull/33246) * In a StarRocks shared-nothing cluster, queries against Iceberg or Hive tables may cause BEs to crash. [#34682](https://github.com/StarRocks/starrocks/pull/34682) * In a StarRocks shared-nothing cluster, if multiple partitions are automatically created during data loading, the data loaded may occasionally be written to unmatched partitions. [#34731](https://github.com/StarRocks/starrocks/pull/34731) * Long-time, frequent data loading into a Primary Key table with persistent index enabled may cause BEs to crash. [#33220](https://github.com/StarRocks/starrocks/pull/33220) * The error "Exception: java.lang.IllegalStateException: null" is returned for queries. [#33535](https://github.com/StarRocks/starrocks/pull/33535) * When `show proc '/current_queries';` is being executed and meanwhile a query begins to be executed, BEs may crash. [#34316](https://github.com/StarRocks/starrocks/pull/34316) * Errors may be thrown if large amounts of data are loaded into a Primary Key table with persistent index enabled. [#34352](https://github.com/StarRocks/starrocks/pull/34352) * After StarRocks is upgraded from v2.4 or earlier to a later version, compaction scores may rise unexpectedly. [#34618](https://github.com/StarRocks/starrocks/pull/34618) * If `INFORMATION_SCHEMA` is queried by using the database driver MariaDB ODBC, the `CATALOG_NAME` column returned in the `schemata` view holds only `null` values. [#34627](https://github.com/StarRocks/starrocks/pull/34627) * FEs crash due to the abnormal data loaded and cannot restart. [#34590](https://github.com/StarRocks/starrocks/pull/34590) * If schema changes are being executed while a Stream Load job is in the **PREPARED** state, a portion of the source data to be loaded by the job is lost. [#34381](https://github.com/StarRocks/starrocks/pull/34381) * Including two or more slashes (`/`) at the end of the HDFS storage path causes the backup and restore of the data from HDFS to fail. [#34601](https://github.com/StarRocks/starrocks/pull/34601) * Setting the session variable `enable_load_profile` to `true` makes Stream Load jobs prone to fail. [#34544](https://github.com/StarRocks/starrocks/pull/34544) * Performing partial updates in column mode onto a Primary Key table causes some tablets of the table to show data inconsistencies between their replicas. [#34555](https://github.com/StarRocks/starrocks/pull/34555) * The `partition_live_number` property added by using the ALTER TABLE statement does not take effect. [#34842](https://github.com/StarRocks/starrocks/pull/34842) * FEs fail to start and report the error "failed to load journal type 118". [#34590](https://github.com/StarRocks/starrocks/pull/34590) * Setting the FE parameter `recover_with_empty_tablet` to `true` may cause FEs to crash. [#33071](https://github.com/StarRocks/starrocks/pull/33071) * Failures in replaying replica operations may cause FEs to crash. [#32295](https://github.com/StarRocks/starrocks/pull/32295) ##### Parameter Change[​](#parameter-change-3 "Direct link to Parameter Change") ###### FE/BE Parameters[​](#febe-parameters "Direct link to FE/BE Parameters") * Added an FE configuration item [`enable_statistics_collect_profile`](https://docs.starrocks.io/docs/administration/FE_configuration#enable_statistics_collect_profile), which controls whether to generate profiles for statistics queries. The default value is `false`. [#33815](https://github.com/StarRocks/starrocks/pull/33815) * The FE configuration item [`mysql_server_version`](https://docs.starrocks.io/docs/administration/FE_configuration#mysql_server_version) is now mutable. The new setting can take effect for the current session without requiring an FE restart. [#34033](https://github.com/StarRocks/starrocks/pull/34033) * Added a BE/CN configuration item [`update_compaction_ratio_threshold`](https://docs.starrocks.io/docs/administration/BE_configuration#update_compaction_ratio_threshold), which controls the maximum proportion of data that a compaction can merge for a Primary Key table in a StarRocks shared-data cluster. The default value is `0.5`. We recommend shrinking this value if a single tablet becomes excessively large. For a StarRocks shared-nothing cluster, the proportion of data that a compaction can merge for a Primary Key table is still automatically adjusted. [#35129](https://github.com/StarRocks/starrocks/pull/35129) ###### System Variables[​](#system-variables "Direct link to System Variables") * Added a session variable `cbo_decimal_cast_string_strict`, which controls how the CBO converts data from the DECIMAL type to the STRING type. If this variable is set to `true`, the logic built in v2.5.x and later versions prevails and the system implements strict conversion (namely, the system truncates the generated string and fills 0s based on the scale length). If this variable is set to `false`, the logic built in versions earlier than v2.5.x prevails and the system processes all valid digits to generate a string. The default value is `true`. [#34208](https://github.com/StarRocks/starrocks/pull/34208) * Added a session variable `cbo_eq_base_type`, which specifies the data type used for data comparison between DECIMAL-type data and STRING-type data. The default value is `VARCHAR`, and `DECIMAL` is also a valid value. [#34208](https://github.com/StarRocks/starrocks/pull/34208) * Added a session variable `big_query_profile_second_threshold`. When the session variable [`enable_profile`](https://docs.starrocks.io/docs/reference/System_variable#enable_profile) is set to `false` and the amount of time taken by a query exceeds the threshold specified by the `big_query_profile_second_threshold` variable, a profile is generated for that query. [#33825](https://github.com/StarRocks/starrocks/pull/33825) #### 3.1.4[​](#314 "Direct link to 3.1.4") Release date: November 2, 2023 ##### New Features[​](#new-features-7 "Direct link to New Features") * Supports sort keys for Primary Key tables created in shared-data StarRocks clusters. * Supports using the str2date function to specify partition expressions for asynchronous materialized views. This helps facilitate incremental updates and query rewrites of asynchronous materialized views created on tables that reside in external catalogs and use the STRING-type data as their partitioning expressions. [#29923](https://github.com/StarRocks/starrocks/pull/29923) [#31964](https://github.com/StarRocks/starrocks/pull/31964) * Added a new session variable `enable_query_tablet_affinity`, which controls whether to direct multiple queries against the same tablet to a fixed replica. This session variable is set to `false` by default. [#33049](https://github.com/StarRocks/starrocks/pull/33049) * Added the utility function `is_role_in_session`, which is used to check whether the specified roles are activated in the current session. It supports checking nested roles granted to a user. [#32984](https://github.com/StarRocks/starrocks/pull/32984) * Supports setting resource group-level query queue, which is controlled by the global variable `enable_group_level_query_queue` (default value: `false`). When the global-level or resource group-level resource consumption reaches a predefined threshold, new queries are placed in queue, and will be run when both the global-level resource consumption and the resource group-level resource consumption fall below their thresholds. * Users can set `concurrency_limit` for each resource group to limit the maximum number of concurrent queries allowed per BE. * Users can set `max_cpu_cores` for each resource group to limit the maximum CPU consumption allowed per BE. * Added two parameters, `plan_cpu_cost_range` and `plan_mem_cost_range`, for resource group classifiers. * `plan_cpu_cost_range`: the CPU consumption range estimated by the system. The default value `NULL` indicates no limit is imposed. * `plan_mem_cost_range`: the memory consumption range estimated by the system. The default value `NULL` indicates no limit is imposed. ##### Improvements[​](#improvements-11 "Direct link to Improvements") * Window functions COVAR\_SAMP, COVAR\_POP, CORR, VARIANCE, VAR\_SAMP, STD, and STDDEV\_SAMP now support the ORDER BY clause and Window clause. [#30786](https://github.com/StarRocks/starrocks/pull/30786) * An error instead of NULL is returned if a decimal overflow occurs during queries on the DECIMAL type data. [#30419](https://github.com/StarRocks/starrocks/pull/30419) * The number of concurrent queries allowed in a query queue is now managed by the leader FE. Each follower FE notifies of the leader FE when a query starts and finishes. If the number of concurrent queries reaches the global-level or resource group-level `concurrency_limit`, new queries are rejected or placed in queue. ##### Bug Fixes[​](#bug-fixes-13 "Direct link to Bug Fixes") Fixed the following issues: * Spark or Flink may report data read errors due to inaccurate memory usage statistics. [#30702](https://github.com/StarRocks/starrocks/pull/30702) [#30751](https://github.com/StarRocks/starrocks/pull/30751) * Memory usage statistics for Metadata Cache are inaccurate. [#31978](https://github.com/StarRocks/starrocks/pull/31978) * BEs crash when libcurl is invoked. [#31667](https://github.com/StarRocks/starrocks/pull/31667) * When StarRocks materialized views created on Hive views are refreshed, an error "java.lang.ClassCastException: com.starrocks.catalog.HiveView cannot be cast to com.starrocks.catalog.HiveMetaStoreTable" is returned. [#31004](https://github.com/StarRocks/starrocks/pull/31004) * If the ORDER BY clause contains aggregate functions, an error "java.lang.IllegalStateException: null" is returned. [#30108](https://github.com/StarRocks/starrocks/pull/30108) * In shared-data StarRocks clusters, the information of table keys is not recorded in `information_schema.COLUMNS`. As a result, DELETE operations cannot be performed when data is loaded by using Flink Connector. [#31458](https://github.com/StarRocks/starrocks/pull/31458) * When data is loaded by using Flink Connector, the load job is suspended unexpectedly if there are highly concurrent load jobs and both the number of HTTP threads and the number of Scan threads have reached their upper limits. [#32251](https://github.com/StarRocks/starrocks/pull/32251) * When a field of only a few bytes is added, executing SELECT COUNT(\*) before the data change finishes returns an error that reads "error: invalid field name". [#33243](https://github.com/StarRocks/starrocks/pull/33243) * Query results are incorrect after the query cache is enabled. [#32781](https://github.com/StarRocks/starrocks/pull/32781) * Queries fail during hash joins, causing BEs to crash. [#32219](https://github.com/StarRocks/starrocks/pull/32219) * `DATA_TYPE` and `COLUMN_TYPE` for BINARY or VARBINARY data types are displayed as `unknown` in the `information_schema.columns` view. [#32678](https://github.com/StarRocks/starrocks/pull/32678) ##### Behavior Change[​](#behavior-change-1 "Direct link to Behavior Change") * From v3.1.4 onwards, persistent indexing is enabled by default for Primary Key tables created in new StarRocks clusters (this does not apply to existing StarRocks clusters whose versions are upgraded to v3.1.4 from an earlier version). [#33374](https://github.com/StarRocks/starrocks/pull/33374) * A new FE parameter `enable_sync_publish` which is set to `true` by default is added. When this parameter is set to `true`, the Publish phase of a data load into a Primary Key table returns the execution result only after the Apply task finishes. As such, the data loaded can be queried immediately after the load job returns a success message. However, setting this parameter to `true` may cause data loads into Primary Key tables to take a longer time. (Before this parameter is added, the Apply task is asynchronous with the Publish phase.) [#27055](https://github.com/StarRocks/starrocks/pull/27055) #### 3.1.3 (Yanked)[​](#313-yanked "Direct link to 3.1.3 (Yanked)") tip This version has been taken offline. Release date: September 25, 2023 ##### New Features[​](#new-features-8 "Direct link to New Features") * Primary Key tables created in shared-data StarRocks clusters support index persistence onto local disks in the same way as they do in shared-nothing StarRocks clusters. * The aggregate function [group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat/) supports the DISTINCT keyword and the ORDER BY clause. [#28778](https://github.com/StarRocks/starrocks/pull/28778) * [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/STREAM_LOAD/), [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/BROKER_LOAD/), [Kafka Connector](https://docs.starrocks.io/docs/loading/Kafka-connector-starrocks/), [Flink Connector](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks/), and [Spark Connector](https://docs.starrocks.io/docs/loading/Spark-connector-starrocks/) support partial updates in column mode on a Primary Key table. [#28288](https://github.com/StarRocks/starrocks/pull/28288) * Data in partitions can be automatically cooled down over time. (This feature is not supported for [list partitioning](https://docs.starrocks.io/docs/table_design/list_partitioning/).) [#29335](https://github.com/StarRocks/starrocks/pull/29335) [#29393](https://github.com/StarRocks/starrocks/pull/29393) ##### Improvements[​](#improvements-12 "Direct link to Improvements") Executing SQL commands with invalid comments now returns results consistent with MySQL. [#30210](https://github.com/StarRocks/starrocks/pull/30210) ##### Bug Fixes[​](#bug-fixes-14 "Direct link to Bug Fixes") Fixed the following issues: * If the [BITMAP](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/BITMAP/) or [HLL](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/HLL/) data type is specified in the WHERE clause of a [DELETE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/DELETE/) statement to be executed, the statement cannot be properly executed. [#28592](https://github.com/StarRocks/starrocks/pull/28592) * After a follower FE is restarted, CpuCores statistics are not up-to-date, resulting in query performance degradation. [#28472](https://github.com/StarRocks/starrocks/pull/28472) [#30434](https://github.com/StarRocks/starrocks/pull/30434) * The execution cost of the [to\_bitmap()](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/to_bitmap/) function is incorrectly calculated. As a result, an inappropriate execution plan is selected for the function after materialized views are rewritten. [#29961](https://github.com/StarRocks/starrocks/pull/29961) * In certain use cases of the shared-data architecture, after a follower FE is restarted, queries submitted to the follower FE return an error that reads "Backend node not found. Check if any backend node is down". [#28615](https://github.com/StarRocks/starrocks/pull/28615) * If data is continuously loaded into a table that is being altered by using the [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/ALTER_TABLE/) statement, an error "Tablet is in error state" may be thrown. [#29364](https://github.com/StarRocks/starrocks/pull/29364) * Modifying the FE dynamic parameter `max_broker_load_job_concurrency` using the `ADMIN SET FRONTEND CONFIG` command does not take effect. [#29964](https://github.com/StarRocks/starrocks/pull/29964) [#29720](https://github.com/StarRocks/starrocks/pull/29720) * BEs crash if the time unit in the [date\_diff()](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_diff/) function is a constant but the dates are not constants. [#29937](https://github.com/StarRocks/starrocks/issues/29937) * In the shared-data architecture, automatic partitioning does not take effect after asynchronous load is enabled. [#29986](https://github.com/StarRocks/starrocks/issues/29986) * If users create a Primary Key table by using the [CREATE TABLE LIKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/CREATE_TABLE_LIKE/) statement, an error `Unexpected exception: Unknown properties: {persistent_index_type=LOCAL}` is thrown. [#30255](https://github.com/StarRocks/starrocks/pull/30255) * Restoring Primary Key tables causes metadata inconsistency after BEs are restarted. [#30135](https://github.com/StarRocks/starrocks/pull/30135) * If users load data into a Primary Key table on which truncate operations and queries are concurrently performed, an error "java.lang.NullPointerException" is thrown in certain cases. [#30573](https://github.com/StarRocks/starrocks/pull/30573) * If predicate expressions are specified in materialized view creation statements, the refresh results of those materialized views are incorrect. [#29904](https://github.com/StarRocks/starrocks/pull/29904) * After users upgrade their StarRocks cluster to v3.1.2, the storage volume properties of the tables created before the upgrade are reset to `null`. [#30647](https://github.com/StarRocks/starrocks/pull/30647) * If checkpointing and restoration are concurrently performed on tablet metadata, some tablet replicas will be lost and cannot be retrieved. [#30603](https://github.com/StarRocks/starrocks/pull/30603) * If users use CloudCanal to load data into table columns that are set to `NOT NULL` but have no default value specified, an error "Unsupported dataFormat value is : \N" is thrown. [#30799](https://github.com/StarRocks/starrocks/pull/30799) ##### Behavior Change[​](#behavior-change-2 "Direct link to Behavior Change") * When using the [group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat/) function, users must use the SEPARATOR keyword to declare the separator. * The default value of the session variable [`group_concat_max_len`](https://docs.starrocks.io/docs/reference/System_variable#group_concat_max_len) which controls the default maximum length of the string returned by the [group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat/) function is changed from unlimited to `1024`. #### 3.1.2[​](#312 "Direct link to 3.1.2") Release date: August 25, 2023 ##### Bug Fixes[​](#bug-fixes-15 "Direct link to Bug Fixes") Fixed the following issues: * If a user specifies which database is to be connected by default and the user only has permissions on tables in the database but does not have permissions on the database, an error stating that the user does not have permissions on the database is thrown. [#29767](https://github.com/StarRocks/starrocks/pull/29767) * The values returned by the RESTful API action `show_data` for cloud-native tables are incorrect. [#29473](https://github.com/StarRocks/starrocks/pull/29473) * BEs crash if queries are canceled while the [array\_agg()](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_agg/) function is being run. [#29400](https://github.com/StarRocks/starrocks/issues/29400) * The `Default` field values returned by the [SHOW FULL COLUMNS](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/SHOW_FULL_COLUMNS/) statement for columns of the [BITMAP](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/BITMAP/) or [HLL](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/HLL/) data type are incorrect. [#29510](https://github.com/StarRocks/starrocks/pull/29510) * If the [array\_map()](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_map/) function in queries involves multiple tables, the queries fail due to pushdown strategy issues. [#29504](https://github.com/StarRocks/starrocks/pull/29504) * Queries against ORC-formatted files fail because the bugfix ORC-1304 ([apache/orc#1299](https://github.com/apache/orc/pull/1299)) from Apache ORC is not merged. [#29804](https://github.com/StarRocks/starrocks/pull/29804) ##### Behavior Change[​](#behavior-change-3 "Direct link to Behavior Change") For a newly deployed StarRocks v3.1 cluster, you must have the USAGE privilege on the destination external catalog if you want to run [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/SET_CATALOG/) to switch to that catalog. You can use [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT/) to grant the required privileges. For a v3.1 cluster upgraded from an earlier version, you can run SET CATALOG with inherited privilege. #### 3.1.1[​](#311 "Direct link to 3.1.1") Release date: August 18, 2023 ##### New Features[​](#new-features-9 "Direct link to New Features") * Supports Azure Blob Storage for [shared-data clusters](https://docs.starrocks.io/docs/deployment/shared_data/s3/). * Supports List partitioning for [shared-data clusters](https://docs.starrocks.io/docs/deployment/shared_data/s3/). * Supports aggregate functions [COVAR\_SAMP](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/covar_samp/), [COVAR\_POP](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/covar_pop/), and [CORR](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/corr/). * Supports the following [window functions](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function/): COVAR\_SAMP, COVAR\_POP, CORR, VARIANCE, VAR\_SAMP, STD, and STDDEV\_SAMP. ##### Improvements[​](#improvements-13 "Direct link to Improvements") Supports implicit conversions for all compound predicates and for all expressions in the WHERE clause. You can enable or disable implicit conversions by using the [session variable](https://docs.starrocks.io/docs/reference/System_variable/) `enable_strict_type`. The default value of this session variable is `false`. ##### Bug Fixes[​](#bug-fixes-16 "Direct link to Bug Fixes") Fixed the following issues: * When data is loaded into tables that have multiple replicas, a large number of invalid log records are written if some partitions of the tables are empty. [#28824](https://github.com/StarRocks/starrocks/issues/28824) * Inaccurate estimation of average row size causes partial updates in column mode on Primary Key tables to occupy excessively large memory. [#27485](https://github.com/StarRocks/starrocks/pull/27485) * If clone operations are triggered on tablets in an ERROR state, disk usage increases. [#28488](https://github.com/StarRocks/starrocks/pull/28488) * Compaction causes cold data to be written to the local cache. [#28831](https://github.com/StarRocks/starrocks/pull/28831) #### 3.1.0[​](#310 "Direct link to 3.1.0") Release date: August 7, 2023 ##### New Features[​](#new-features-10 "Direct link to New Features") ###### Shared-data cluster[​](#shared-data-cluster "Direct link to Shared-data cluster") * Added support for Primary Key tables, on which persistent indexes cannot be enabled. * Supports the [AUTO\_INCREMENT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/auto_increment/) column attribute, which enables a globally unique ID for each data row and thus simplifies data management. * Supports [automatically creating partitions during loading and using partitioning expressions to define partitioning rules](https://docs.starrocks.io/docs/table_design/data_distribution/expression_partitioning/), thereby making partition creation easier to use and more flexible. * Supports [abstraction of storage volumes](https://docs.starrocks.io/docs/deployment/shared_data/s3#use-your-shared-data-starrocks-cluster), in which users can configure storage location and authentication information, in shared-data StarRocks clusters. Users can directly reference an existing storage volume when creating a database or table, making authentication configuration easier. ###### Data Lake analytics[​](#data-lake-analytics "Direct link to Data Lake analytics") * Supports accessing views created on tables within [Hive catalogs](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog/). * Supports accessing Parquet-formatted Iceberg v2 tables. * Supports [sinking data to Parquet-formatted Iceberg tables](https://docs.starrocks.io/docs/data_source/catalog/iceberg_catalog#sink-data-to-an-iceberg-table). * \[Preview] Supports accessing data stored in Elasticsearch by using [Elasticsearch catalogs](https://docs.starrocks.io/docs/data_source/catalog/elasticsearch_catalog/). This simplifies the creation of Elasticsearch external tables. * \[Preview] Supports performing analytics on streaming data stored in Apache Paimon by using [Paimon catalogs](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog/). ###### Storage engine, data ingestion, and query[​](#storage-engine-data-ingestion-and-query "Direct link to Storage engine, data ingestion, and query") * Upgraded automatic partitioning to [expression partitioning](https://docs.starrocks.io/docs/table_design/expression_partitioning/). Users only need to use a simple partition expression (either a time function expression or a column expression) to specify a partitioning method at table creation, and StarRocks will automatically create partitions based on the data characteristics and the rule defined in the partition expression during data loading. This method of partition creation is suitable for most scenarios and is more flexible and user-friendly. * Supports [list partitioning](https://docs.starrocks.io/docs/table_design/list_partitioning/). Data is partitioned based on a list of values predefined for a particular column, which can accelerate queries and manage clearly categorized data more efficiently. * Added a new table named `loads` to the `Information_schema` database. Users can query the results of [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/BROKER_LOAD/) and [Insert](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/INSERT/) jobs from the `loads` table. * Supports logging the unqualified data rows that are filtered out by [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/STREAM_LOAD/), [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/BROKER_LOAD/), and [Spark Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/SPARK_LOAD/) jobs. Users can use the `log_rejected_record_num` parameter in their load job to specify the maximum number of data rows that can be logged. * Supports [random bucketing](https://docs.starrocks.io/docs/table_design/Data_distribution#how-to-choose-the-bucketing-columns). With this feature, users do not need to configure bucketing columns at table creation, and StarRocks will randomly distribute the data loaded into it to buckets. Using this feature together with the capability of automatically setting the number of buckets (`BUCKETS`) that StarRocks has provided since v2.5.7, users no longer need to consider bucket configurations, and table creation statements are greatly simplified. In big data and high performance-demanding scenarios, however, we recommend that users continue using hash bucketing, because this way they can use bucket pruning to accelerate queries. * Supports using the table function FILES() in [INSERT INTO](https://docs.starrocks.io/docs/loading/InsertInto/) to directly load the data of Parquet- or ORC-formatted data files stored in AWS S3. The FILES() function can automatically infer the table schema, which relieves the need to create external catalogs or file external tables before data loading and therefore greatly simplifies the data loading process. * Supports [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns/). With the generated column feature, StarRocks can automatically generate and store the values of column expressions and automatically rewrite queries to improve query performance. * Supports loading data from Spark to StarRocks by using [Spark connector](https://docs.starrocks.io/docs/loading/Spark-connector-starrocks/). Compared to [Spark Load](https://docs.starrocks.io/docs/loading/SparkLoad/), the Spark connector provides more comprehensive capabilities. Users can define a Spark job to perform ETL operations on the data, and the Spark connector serves as the sink in the Spark job. * Supports loading data into columns of the [MAP](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/Map/) and [STRUCT](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/STRUCT/) data types, and supports nesting Fast Decimal values in ARRAY, MAP, and STRUCT. ###### SQL reference[​](#sql-reference "Direct link to SQL reference") * Added the following storage volume-related statements: [CREATE STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/CREATE_STORAGE_VOLUME/), [ALTER STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/ALTER_STORAGE_VOLUME/), [DROP STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/DROP_STORAGE_VOLUME/), [SET DEFAULT STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/SET_DEFAULT_STORAGE_VOLUME/), [DESC STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/DESC_STORAGE_VOLUME/), [SHOW STORAGE VOLUMES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/SHOW_STORAGE_VOLUMES/). * Supports altering table comments using [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/ALTER_TABLE/). [#21035](https://github.com/StarRocks/starrocks/pull/21035) * Added the following functions: * Struct functions: [struct (row)](https://docs.starrocks.io/docs/sql-reference/sql-functions/struct-functions/row/), [named\_struct](https://docs.starrocks.io/docs/sql-reference/sql-functions/struct-functions/named_struct/) * Map functions: [str\_to\_map](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/str_to_map/), [map\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_concat/), [map\_from\_arrays](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_from_arrays/), [element\_at](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/element_at/), [distinct\_map\_keys](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/distinct_map_keys/), [cardinality](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/cardinality/) * Higher-order Map functions: [map\_filter](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_filter/), [map\_apply](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_apply/), [transform\_keys](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/transform_keys/), [transform\_values](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/transform_values/) * Array functions: [array\_agg](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_agg/) supports `ORDER BY`, [array\_generate](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_generate/), [element\_at](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/element_at/), [cardinality](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/cardinality/) * Higher-order Array functions: [all\_match](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/all_match/), [any\_match](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/any_match/) * Aggregate functions: [min\_by](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/min_by/), [percentile\_disc](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/percentile_disc/) * Table functions: [FILES](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files/), [generate\_series](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/generate_series/) * Date functions: [next\_day](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/next_day/), [previous\_day](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/previous_day/), [last\_day](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/last_day/), [makedate](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/makedate/), [date\_diff](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_diff/) * Bitmap functions: [bitmap\_subset\_limit](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_subset_limit/), [bitmap\_subset\_in\_range](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_subset_in_range/) * Math functions: [cosine\_similarity](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cos_similarity/), [cosine\_similarity\_norm](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cos_similarity_norm/) ###### Privileges and security[​](#privileges-and-security "Direct link to Privileges and security") Added [privilege items](https://docs.starrocks.io/docs/administration/privilege_item#storage-volume) related to storage volumes and [privilege items](https://docs.starrocks.io/docs/administration/privilege_item#catalog) related to external catalogs, and supports using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT/) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE/) to grant and revoke these privileges. ##### Improvements[​](#improvements-14 "Direct link to Improvements") ###### Shared-data cluster[​](#shared-data-cluster-1 "Direct link to Shared-data cluster") Optimized the data cache in shared-data StarRocks clusters. The optimized data cache allows for specifying the range of hot data. It can also prevent queries against cold data from occupying the local disk cache, thereby ensuring the performance of queries against hot data. ###### Materialized view[​](#materialized-view "Direct link to Materialized view") * Optimized the creation of an asynchronous materialized view: * Supports random bucketing. If users do not specify bucketing columns, StarRocks adopts random bucketing by default. * Supports using `ORDER BY` to specify a sort key. * Supports specifying attributes such as `colocate_group`, `storage_medium`, and `storage_cooldown_time`. * Supports using session variables. Users can configure these variables by using the `properties("session." = "")` syntax to flexibly adjust view refreshing strategies. * Enables the spill feature for all asynchronous materialized views and implements a query timeout duration of 1 hour by default. * Supports creating materialized views based on views. This makes materialized views easier to use in data modeling scenarios, because users can flexibly use views and materialized views based on their varying needs to implement layered modeling. * Optimized query rewrite with asynchronous materialized views: * Supports Stale Rewrite, which allows materialized views that are not refreshed within a specified time interval to be used for query rewrite regardless of whether the base tables of the materialized views are updated. Users can specify the time interval by using the `mv_rewrite_staleness_second` property at materialized view creation. * Supports rewriting View Delta Join queries against materialized views that are created on Hive catalog tables (a primary key and a foreign key must be defined). * Optimized the mechanism for rewriting queries that contain union operations, and supports rewriting queries that contain joins or functions such as COUNT DISTINCT and time\_slice. * Optimized the refreshing of asynchronous materialized views: * Optimized the mechanism for refreshing materialized views that are created on Hive catalog tables. StarRocks now can perceive partition-level data changes, and refreshes only the partitions with data changes during each automatic refresh. * Supports using the `REFRESH MATERIALIZED VIEW WITH SYNC MODE` syntax to synchronously invoke materialized view refresh tasks. * Enhanced the use of asynchronous materialized views: * Supports using `ALTER MATERIALIZED VIEW {ACTIVE | INACTIVE}` to enable or disable a materialized view. Materialized views that are disabled (in the `INACTIVE` state) cannot be refreshed or used for query rewrite, but can be directly queried. * Supports using `ALTER MATERIALIZED VIEW SWAP WITH` to swap two materialized views. Users can create a new materialized view and then perform an atomic swap with an existing materialized view to implement schema changes on the existing materialized view. * Optimized synchronous materialized views: * Supports direct queries against synchronous materialized views using SQL hints `[_SYNC_MV_]`, allowing for walking around issues that some queries cannot be properly rewritten in rare circumstances. * Supports more expressions, such as `CASE-WHEN`, `CAST`, and mathematical operations, which make materialized views suitable for more business scenarios. ###### Data Lake analytics[​](#data-lake-analytics-1 "Direct link to Data Lake analytics") * Optimized metadata caching and access for Iceberg to improve Iceberg data query performance. * Optimized the data cache to further improve data lake analytics performance. ###### Storage engine, data ingestion, and query[​](#storage-engine-data-ingestion-and-query-1 "Direct link to Storage engine, data ingestion, and query") * Announced the general availability of the [spill](https://docs.starrocks.io/docs/3.1/administration/management/resource_management/spill_to_disk/) feature, which supports spilling the intermediate computation results of some blocking operators to disk. With the spill feature enabled, when a query contains aggregate, sort, or join operators, StarRocks can cache the intermediate computation results of the operators to disk to reduce memory consumption, thereby minimizing query failures caused by memory limits. * Supports pruning on cardinality-preserving joins. If users maintain a large number of tables which are organized in the star schema (for example, SSB) or the snowflake schema (for example, TCP-H) but they query only a small number of these tables, this feature helps prune unnecessary tables to improve the performance of joins. * Supports partial updates in column mode. Users can enable the column mode when they perform partial updates on Primary Key tables by using the [UPDATE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/UPDATE/) statement. The column mode is suitable for updating a small number of columns but a large number of rows, and can improve the updating performance by up to 10 times. * Optimized the collection of statistics for the CBO. This reduces the impact of statistics collection on data ingestion and increases statistics collection performance. * Optimized the merge algorithm to increase the overall performance by up to 2 times in permutation scenarios. * Optimized the query logic to reduce dependency on database locks. * Dynamic partitioning further supports the partitioning unit to be year. [#28386](https://github.com/StarRocks/starrocks/pull/28386) ###### SQL reference[​](#sql-reference-1 "Direct link to SQL reference") * Conditional functions case, coalesce, if, ifnull, and nullif support the ARRAY, MAP, STRUCT, and JSON data types. * The following Array functions support nested types MAP, STRUCT, and ARRAY: * array\_agg * array\_contains, array\_contains\_all, array\_contains\_any * array\_slice, array\_concat * array\_length, array\_append, array\_remove, array\_position * reverse, array\_distinct, array\_intersect, arrays\_overlap * array\_sortby * The following Array functions support the Fast Decimal data type: * array\_agg * array\_append, array\_remove, array\_position, array\_contains * array\_length * array\_max, array\_min, array\_sum, array\_avg * arrays\_overlap, array\_difference * array\_slice, array\_distinct, array\_sort, reverse, array\_intersect, array\_concat * array\_sortby, array\_contains\_all, array\_contains\_any ##### Bug Fixes[​](#bug-fixes-17 "Direct link to Bug Fixes") Fixed the following issues: * Requests to reconnect to Kafka for Routine Load jobs cannot be properly processed. [#23477](https://github.com/StarRocks/starrocks/issues/23477) * For SQL queries that involve multiple tables and contain a `WHERE` clause, if these SQL queries have the same semantics but the order of the tables in each SQL query is different, some of these SQL queries may fail to be rewritten to benefit from the related materialized views. [#22875](https://github.com/StarRocks/starrocks/issues/22875) * Duplicate records are returned for queries that contain a `GROUP BY` clause. [#19640](https://github.com/StarRocks/starrocks/issues/19640) * Invoking the lead() or lag() function may cause BE crashes. [#22945](https://github.com/StarRocks/starrocks/issues/22945) * Rewriting partial partition queries based on materialized views that are created on external catalog tables fail. [#19011](https://github.com/StarRocks/starrocks/issues/19011) * SQL statements that contain both a backward slash (`\`) and a semicolon (`;`) cannot be properly parsed. [#16552](https://github.com/StarRocks/starrocks/issues/16552) * A table cannot be truncated if a materialized view created on the table is removed. [#19802](https://github.com/StarRocks/starrocks/issues/19802) ##### Behavior Change[​](#behavior-change-4 "Direct link to Behavior Change") * The `storage_cache_ttl` parameter is deleted from the table creation syntax used for shared-data StarRocks clusters. Now the data in the local cache is evicted based on the LRU algorithm. * The BE configuration items `disable_storage_page_cache` and `alter_tablet_worker_count` and the FE configuration item `lake_compaction_max_tasks` are changed from immutable parameters to mutable parameters. * The default value of the BE configuration item `block_cache_checksum_enable` is changed from `true` to `false`. * The default value of the BE configuration item `enable_new_load_on_memory_limit_exceeded` is changed from `false` to `true`. * The default value of the FE configuration item `max_running_txn_num_per_db` is changed from `100` to `1000`. * The default value of the FE configuration item `http_max_header_size` is changed from `8192` to `32768`. * The default value of the FE configuration item `tablet_create_timeout_second` is changed from `1` to `10`. * The default value of the FE configuration item `max_routine_load_task_num_per_be` is changed from `5` to `16`, and error information will be returned if a large number of Routine Load tasks are created. * The FE configuration item `quorom_publish_wait_time_ms` is renamed as `quorum_publish_wait_time_ms`, and the FE configuration item `async_load_task_pool_size` is renamed as `max_broker_load_job_concurrency`. * The BE configuration item `routine_load_thread_pool_size` is deprecated. Now the routine load thread pool size per BE node is controlled only by the FE configuration item `max_routine_load_task_num_per_be`. * The BE configuration item `txn_commit_rpc_timeout_ms` and the system variable `tx_visible_wait_timeout` are deprecated. * The FE configuration items `max_broker_concurrency` and `load_parallel_instance_num` are deprecated. * The FE configuration item `max_routine_load_job_num` is deprecated. Now StarRocks dynamically infers the maximum number of Routine Load tasks supported by each individual BE node based on the `max_routine_load_task_num_per_be` parameter and provides suggestions on task failures. * The CN configuration item `thrift_port` is renamed as `be_port`. * Two new Routine Load job properties, `task_consume_second` and `task_timeout_second`, are added to control the maximum amount of time to consume data and the timeout duration for individual load tasks within a Routine Load job, making job adjustment more flexible. If users do not specify these two properties in their Routine Load job, the FE configuration items `routine_load_task_consume_second` and `routine_load_task_timeout_second` prevail. * The session variable `enable_resource_group` is deprecated because the [Resource Group](https://docs.starrocks.io/docs/administration/resource_group/) feature is enabled by default since v3.1.0. * Two new reserved keywords, COMPACTION and TEXT, are added. --- ## Release 3.2 ### StarRocks version 3.2 #### 3.2.16[​](#3216 "Direct link to 3.2.16") Release Date: April 30, 2025 ##### Improvements[​](#improvements "Direct link to Improvements") * Stream Load task scheduling now supports BE node blacklist. Nodes in the blacklist will be excluded from task scheduling. [#57919](https://github.com/StarRocks/starrocks/pull/57919) ##### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") Fixed the following issues: * Create tablet timeout. [#55808](https://github.com/StarRocks/starrocks/pull/55808) * Authentication information is lost when creating views via the `files()` function. [#56606](https://github.com/StarRocks/starrocks/pull/56606) * Optimizer failed to correctly handle constant comparisons when processing empty sets, leading to query failure. [#57735](https://github.com/StarRocks/starrocks/pull/57735) * Pre-aggregation strategy caused BE crashes when handling data overflow. [#58022](https://github.com/StarRocks/starrocks/pull/58022) * Attempting to delete associated materialized views after some partitions of the base table were deleted may cause exceptions, resulting in failure of the delete operation. [#58037](https://github.com/StarRocks/starrocks/pull/58037) * Defect in priority evaluation logic when loading Tablets for primary key tables, resulting in data loss due to incorrect version recognition. [#58404](https://github.com/StarRocks/starrocks/pull/58404) #### 3.2.15[​](#3215 "Direct link to 3.2.15") Release date: February 14, 2025 ##### New Features[​](#new-features "Direct link to New Features") * Window functions support `max_by` and `min_by`. [#54961](https://github.com/StarRocks/starrocks/pull/54961) ##### Improvements[​](#improvements-1 "Direct link to Improvements") * Added StarClient timeout parameters. [#54496](https://github.com/StarRocks/starrocks/pull/54496) * star\_client\_read\_timeout\_seconds * star\_client\_list\_timeout\_seconds * star\_client\_write\_timeout\_seconds * Tables with List partitioning strategies support partition pruning for DELETE statements. [#55400](https://github.com/StarRocks/starrocks/pull/55400) ##### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") Fixed the following issues: * Stream Load fails when a node with an Alive status of false was scheduled. [#55371](https://github.com/StarRocks/starrocks/pull/55371) * An error is returned during partial updates on Primary Key tables with Stream Load. [#53403](https://github.com/StarRocks/starrocks/pull/55430) * bRPC error persists after BE node restart. [#40229](https://github.com/StarRocks/starrocks/pull/40229) #### 3.2.14[​](#3214 "Direct link to 3.2.14") Release date: January 8, 2025 ##### Improvements[​](#improvements-2 "Direct link to Improvements") * Supports collecting statistics of Paimon tables. [#52858](https://github.com/StarRocks/starrocks/pull/52858) * Included node information and histogram metrics in JSON metrics. [#53735](https://github.com/StarRocks/starrocks/pull/53735) ##### Bug Fixes[​](#bug-fixes-2 "Direct link to Bug Fixes") Fixed the following issues: * The score of the Primary Key table index was not updated in the Commit phase. [#41737](https://github.com/StarRocks/starrocks/pull/41737) * Incorrect execution plans for `max(count(distinct))` when low-cardinality optimization is enabled. [#53403](https://github.com/StarRocks/starrocks/pull/53403) * When the List partition column has NULL values, queries against the Min/Max value of the partition column will lead to incorrect partition pruning. [#53235](https://github.com/StarRocks/starrocks/pull/53235) * Upload retries fail when backing up data to HDFS. [#53679](https://github.com/StarRocks/starrocks/pull/53679) #### 3.2.13[​](#3213 "Direct link to 3.2.13") Release date: December 13, 2024 ##### Improvements[​](#improvements-3 "Direct link to Improvements") * Supports setting a time range within which Base Compaction is forbidden for a specific table. [#50120](https://github.com/StarRocks/starrocks/pull/50120) ##### Bug Fixes[​](#bug-fixes-3 "Direct link to Bug Fixes") Fixed the following issues: * The `loadRowsRate` field returned `0` after executing SHOW ROUTINE LOAD. [#52151](https://github.com/StarRocks/starrocks/pull/52151) * The `Files()` function read columns that were not queried. [#52210](https://github.com/StarRocks/starrocks/pull/52210) * Prometheus failed to parse materialized view metrics with special characters in their names. (Now materialized view metrics support tags.) [#52782](https://github.com/StarRocks/starrocks/pull/52782) * The `array_map` function caused BE to crash. [#52909](https://github.com/StarRocks/starrocks/pull/52909) * Metadata Cache issues caused BE to crash. [#52968](https://github.com/StarRocks/starrocks/pull/52968) * Routine Load tasks were canceled due to expired transactions. (Now tasks are canceled only if the database or table no longer exists). [#50334](https://github.com/StarRocks/starrocks/pull/50334) * Stream Load failures when submitted using HTTP 1.0. [#53010](https://github.com/StarRocks/starrocks/pull/53010) [#53008](https://github.com/StarRocks/starrocks/pull/53008) * Issues related to Glue and S3 integration: [#48433](https://github.com/StarRocks/starrocks/pull/48433) * Some error messages did not display the root cause. * Error messages for writing to a Hive partitioned table with the partition column of type STRING when Glue was used as the metadata service. * Dropping Hive tables failed without proper error messages when the user lacked sufficient permissions. * The `storage_cooldown_time` property for materialized views did not take effect when set to `maximum`. [#52079](https://github.com/StarRocks/starrocks/pull/52079) #### 3.2.12[​](#3212 "Direct link to 3.2.12") Release date: October 23, 2024 ##### Improvements[​](#improvements-4 "Direct link to Improvements") * Optimized memory allocation and statistics in BE for certain complex query scenarios to avoid OOM. [#51382](https://github.com/StarRocks/starrocks/pull/51382) * Optimized memory usage in FE in Schema Change scenarios. [#50855](https://github.com/StarRocks/starrocks/pull/50855) * Optimized the job status display when querying the system-defined view `information_schema.routine_load_jobs` from Follower FE nodes. [#51763](https://github.com/StarRocks/starrocks/pull/51763) * Supports Backup and Restore of with the List partitioned tables. [#51993](https://github.com/StarRocks/starrocks/pull/51993) ##### Bug Fixes[​](#bug-fixes-4 "Direct link to Bug Fixes") Fixed the following issues: * The error message was lost after writing to Hive failed. [#33167](https://github.com/StarRocks/starrocks/pull/33167) * The `array_map` function causes a crash when excessive constant parameters are used. [#51244](https://github.com/StarRocks/starrocks/pull/51244) * Special characters in the PARTITION BY columns of expression partitioned tables cause FE CheckPoint failures. [#51677](https://github.com/StarRocks/starrocks/pull/51677) * Accessing the system-defined view `information_schema.fe_locks` causes a crash. [#51742](https://github.com/StarRocks/starrocks/pull/51742) * Querying generated columns causes an error. [#51755](https://github.com/StarRocks/starrocks/pull/51755) * Optimize Table fails when the table name contains special characters. [#51755](https://github.com/StarRocks/starrocks/pull/51755) * Tablets could not be balanced in certain scenarios. [#51828](https://github.com/StarRocks/starrocks/pull/51828) ##### Behavior Changes[​](#behavior-changes "Direct link to Behavior Changes") * Supports dynamic modification of Backup and Restore-related parameters.[#52111](https://github.com/StarRocks/starrocks/pull/52111) #### 3.2.11[​](#3211 "Direct link to 3.2.11") Release date: September 9, 2024 ##### Improvements[​](#improvements-5 "Direct link to Improvements") * Supports masking authentication information for Files() and PIPE. [#47629](https://github.com/StarRocks/starrocks/pull/47629) * Support automatic inference for the STRUCT type when reading Parquet files through Files(). [#50481](https://github.com/StarRocks/starrocks/pull/50481) ##### Bug Fixes[​](#bug-fixes-5 "Direct link to Bug Fixes") Fixed the following issues: * An error is returned for equi-join queries because they failed to be rewritten by the global dictionary. [#50690](https://github.com/StarRocks/starrocks/pull/50690) * The error "version has been compacted" caused by an infinite loop on the FE side during Tablet Clone. [#50561](https://github.com/StarRocks/starrocks/pull/50561) * Incorrect scheduling for unhealthy replica repairs after distributing data based on labels. [#50331](https://github.com/StarRocks/starrocks/pull/50331) * An error in the statistics collection log: "Unknown column '%s' in '%s." [#50785](https://github.com/StarRocks/starrocks/pull/50785) * Incorrect timezone usage when reading complex types like TIMESTAMP from Parquet files via Files(). [#50448](https://github.com/StarRocks/starrocks/pull/50448) ##### Behavior Changes[​](#behavior-changes-1 "Direct link to Behavior Changes") * When downgrading StarRocks from v3.3.x to v3.2.11, the system will ignore it if there is incompatible metadata. [#49636](https://github.com/StarRocks/starrocks/pull/49636) #### 3.2.10[​](#3210 "Direct link to 3.2.10") Release date: August 23, 2024 ##### Improvements[​](#improvements-6 "Direct link to Improvements") * Files() will automatically convert `BYTE_ARRAY` data with a `logical_type` of `JSON` in Parquet files to the JSON type in StarRocks. [#49385](https://github.com/StarRocks/starrocks/pull/49385) * Optimized error messages for Files() when Access Key ID and Secret Access Key are missing. [#49090](https://github.com/StarRocks/starrocks/pull/49090) * `information_schema.columns` supports the `GENERATION_EXPRESSION` field. [#49734](https://github.com/StarRocks/starrocks/pull/49734) ##### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") Fixed the following issues: * Downgrading a v3.3 shared-data cluster to v3.2 after setting the Primary Key table property `"persistent_index_type" = "CLOUD_NATIVE"` causes a crash. [#48149](https://github.com/StarRocks/starrocks/pull/48149) * Exporting data to CSV files using SELECT INTO OUTFILE may cause data inconsistency. [#48052](https://github.com/StarRocks/starrocks/pull/48052) * Queries encounter failures during concurrent query execution. [#48180](https://github.com/StarRocks/starrocks/pull/48180) * Queries would hang due to a timeout in the Plan phase without exiting. [#48405](https://github.com/StarRocks/starrocks/pull/48405) * After disabling index compression for Primary Key tables in older versions and then upgrading to v3.2.9, accessing `page_off` information causes an array out-of-bounds crash. [#48230](https://github.com/StarRocks/starrocks/pull/48230) * BE crash caused by concurrent execution of ADD/DROP COLUMN operations. [#49355](https://github.com/StarRocks/starrocks/pull/49355) * Queries against negative `TINYINT` values in ORC format files return `None` on the aarch64 architecture. [#49517](https://github.com/StarRocks/starrocks/pull/49517) * If the disk write operation fails, failures of `l0` snapshots for Primary Key Persistent Index may cause data loss. [#48045](https://github.com/StarRocks/starrocks/pull/48045) * Partial Update in Column mode for Primary Key tables fails under scenarios with large-volume data updates. [#49054](https://github.com/StarRocks/starrocks/pull/49054) * BE crash caused by Fast Schema Evolution when downgrading a v3.3.0 shared-data cluster to v3.2.9. [#42737](https://github.com/StarRocks/starrocks/pull/42737) * `partition_linve_nubmer` does not take effect. [#49213](https://github.com/StarRocks/starrocks/pull/49213) * The conflict between index persistence and compaction in Primary Key tables could cause clone failures. [#49341](https://github.com/StarRocks/starrocks/pull/49341) * Modifications of `partition_line_number` using ALTER TABLE do not take effect. [#49437](https://github.com/StarRocks/starrocks/pull/49437) * Rewrite of CTE distinct grouping sets generates an invalid plan. [#48765](https://github.com/StarRocks/starrocks/pull/48765) * RPC failures polluted the thread pool. [#49619](https://github.com/StarRocks/starrocks/pull/49619) * authentication failure issues when loading files from AWS S3 via PIPE. [#49837](https://github.com/StarRocks/starrocks/pull/49837) ##### Behavior Changes[​](#behavior-changes-2 "Direct link to Behavior Changes") * Added a check for the `meta` directory in the FE startup script. If the directory does not exist, it will be automatically created. [#48940](https://github.com/StarRocks/starrocks/pull/48940) * Added a memory limit parameter `load_process_max_memory_hard_limit_ratio` for data loading. If memory usage exceeds the limit, subsequent loading tasks will fail. [#48495](https://github.com/StarRocks/starrocks/pull/48495) #### 3.2.9[​](#329 "Direct link to 3.2.9") Release date: July 11, 2024 ##### New Features[​](#new-features-1 "Direct link to New Features") * Paimon tables now support DELETE Vectors. [#45866](https://github.com/StarRocks/starrocks/issues/45866) * Supports Column-level access control through Apache Ranger. [#47702](https://github.com/StarRocks/starrocks/pull/47702) * Stream Load can automatically convert JSON strings into STRUCT/MAP/ARRAY types during loading. [#45406](https://github.com/StarRocks/starrocks/pull/45406) * JDBC Catalog now supports Oracle and SQL Server. [#35691](https://github.com/StarRocks/starrocks/issues/35691) ##### Improvements[​](#improvements-7 "Direct link to Improvements") * Improved privilege management by restricting `user_admin` role users from resetting the password of the root user. [#47801](https://github.com/StarRocks/starrocks/pull/47801) * Stream Load now supports using `\t` and `\n` as row and column delimiters. Users do not need to convert them to their hexadecimal ASCII codes. [#47302](https://github.com/StarRocks/starrocks/pull/47302) * Optimized memory usage during data loading. [#47047](https://github.com/StarRocks/starrocks/pull/47047) * Supports masking authentication information for the Files() function in audit logs. [#46893](https://github.com/StarRocks/starrocks/pull/46893) * Hive tables now support the `skip.header.line.count` property. [#47001](https://github.com/StarRocks/starrocks/pull/47001) * JDBC Catalog supports more data types. [#47618](https://github.com/StarRocks/starrocks/pull/47618) ##### Behavior Changes[​](#behavior-changes-3 "Direct link to Behavior Changes") * Changed the value inheritance order of the `JAVA_OPTS` parameters. If versions other than JDK\_9 or JDK\_11 are used, users need to configure `JAVA_OPTS` directly. [#47495](https://github.com/StarRocks/starrocks/pull/47495) * When users create a non-partitioned table without specifying the bucket number, the minimum bucket number the system sets for the table is `16` (instead of `2` based on the formula `2*BE or CN count`). If users want to set a smaller bucket number when creating a small table, they must set it explicitly. [#47005](https://github.com/StarRocks/starrocks/pull/47005) * When users create a partitioned table without specifying the bucket number, if the number of partitions exceeds 5, the rule for setting the bucket count is changed to `max(2*BE or CN count, bucket number calculated based on the largest historical partition data volume)`. The previous rule was to calculate the bucket number based on the largest historical partition data volume. [#47949](https://github.com/StarRocks/starrocks/pull/47949) ##### Bug Fixes[​](#bug-fixes-7 "Direct link to Bug Fixes") Fixed the following issues: * BE crash caused by ALTER TABLE ADD COLUMN after upgrading a shared-data cluster from v3.2.x to v3.3.0 and then rolling it back. [#47826](https://github.com/StarRocks/starrocks/pull/47826) * Tasks initiated through SUBMIT TASK showed a Running status indefinitely in the QueryDetail interface. [#47619](https://github.com/StarRocks/starrocks/pull/47619) * Forwarding queries to the FE Leader node caused a null pointer exception. [#47559](https://github.com/StarRocks/starrocks/pull/47559) * SHOW MATERIALIZED VIEWS with WHERE conditions caused a null pointer exception. [#47811](https://github.com/StarRocks/starrocks/pull/47811) * Vertical Compaction fails for Primary Key tables in shared-data clusters. [#47192](https://github.com/StarRocks/starrocks/pull/47192) * Improper handling of I/O Error when sinking data to Hive or Iceberg tables. [#46979](https://github.com/StarRocks/starrocks/pull/46979) * Table properties do not take effect when whitespaces are added to their values. [#47119](https://github.com/StarRocks/starrocks/pull/47119) * BE crash caused by concurrent migration and Index Compaction operations on Primary Key tables. [#46675](https://github.com/StarRocks/starrocks/pull/46675) #### 3.2.8[​](#328 "Direct link to 3.2.8") Release date: June 7, 2024 ##### New Features[​](#new-features-2 "Direct link to New Features") * **[Supports adding labels on BEs](https://docs.starrocks.io/docs/3.2/administration/management/resource_management/be_label/)**: Supports adding labels on BEs based on information such as the racks and data centers where BEs are located. It ensures even data distribution among racks and data centers, and facilitates disaster recovery in case of power failures in certain racks or faults in data centers. [#38833](https://github.com/StarRocks/starrocks/pull/38833) ##### Bug Fixes[​](#bug-fixes-8 "Direct link to Bug Fixes") Fixed the following issues: * An error is returned when users DELETE data rows from tables that use the expression partitioning method with str2date. [#45939](https://github.com/StarRocks/starrocks/pull/45939) * BEs in the destination cluster crash when the StarRocks Cross-cluster Data Migration Tool fails to retrieve the Schema information from the source cluster. [#46068](https://github.com/StarRocks/starrocks/pull/46068) * The error `Multiple entries with same key` is returned to queries with non-deterministic functions. [#46602](https://github.com/StarRocks/starrocks/pull/46602) #### 3.2.7[​](#327 "Direct link to 3.2.7") Release date: May 24, 2024 ##### New Features[​](#new-features-3 "Direct link to New Features") * Stream Load supports data compression during transmission, reducing network bandwidth overhead. Users can specify different compression algorithms using parameters `compression` and `Content-Encoding`. Supported compression algorithms including GZIP, BZIP2, LZ4\_FRAME, and ZSTD. [#43732](https://github.com/StarRocks/starrocks/pull/43732) * Optimized the garbage collection (GC) mechanism in shared-data clusters. Supports manual compaction for tables or partitions stored in object storage. [#39532](https://github.com/StarRocks/starrocks/issues/39532) * Flink connector supports reading complex data types ARRAY, MAP, and STRUCT from StarRocks. [#42932](https://github.com/StarRocks/starrocks/pull/42932) [#347](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/347) * Supports populating Data Cache asynchronously during queries, reducing the impact of populating cache on query performance. [#40489](https://github.com/StarRocks/starrocks/pull/40489) * ANALYZE TABLE supports collecting histograms for external tables, effectively addressing data skews. For more information, see [CBO statistics](https://docs.starrocks.io/docs/3.2/using_starrocks/Cost_based_optimizer/#collect-statistics-of-hiveiceberghudi-tables). [#42693](https://github.com/StarRocks/starrocks/pull/42693) * Lateral Join with [UNNEST](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/array-functions/unnest/) supports LEFT JOIN. [#43973](https://github.com/StarRocks/starrocks/pull/43973) * Query Pool supports configuring memory usage threshold that triggers spilling via BE static parameter `query_pool_spill_mem_limit_threshold`. Once the threshold is reached, intermediate results of queries will be spilled to disks to reduce memory usage, thus avoiding OOM. * Supports creating asynchronous materialized views based on Hive views. ##### Improvements[​](#improvements-8 "Direct link to Improvements") * Optimized the error message returned for Broker Load tasks when there is no data under the specified HDFS paths. [#43839](https://github.com/StarRocks/starrocks/pull/43839) * Optimized the error message returned when the Files function is used to read data from AWS S3 without Access Key and Secret Key specified. [#42450](https://github.com/StarRocks/starrocks/pull/42450) * Optimized the error message returned for Broker Load tasks that load no data to any partitions. [#44292](https://github.com/StarRocks/starrocks/pull/44292) * Optimized the error message returned for INSERT INTO SELECT tasks when the column count of the destination table does not match that in the SELECT statement. [#44331](https://github.com/StarRocks/starrocks/pull/44331) ##### Bug Fixes[​](#bug-fixes-9 "Direct link to Bug Fixes") Fixed the following issues: * Concurrent read or write of the BITMAP-type data may cause BE to crash. [#44167](https://github.com/StarRocks/starrocks/pull/44167) * Primary key indexes may cause BE to crash. [#43793](https://github.com/StarRocks/starrocks/pull/43793) [#43569](https://github.com/StarRocks/starrocks/pull/43569) [#44034](https://github.com/StarRocks/starrocks/pull/44034) * Under high query concurrency scenarios, the str\_to\_map function may cause BE to crash. [#43901](https://github.com/StarRocks/starrocks/pull/43901) * When the Masking policy of Apache Ranger is used, an error is returned when table aliases are specified in queries. [#44445](https://github.com/StarRocks/starrocks/pull/44445) * In shared-data clusters, query execution cannot be routed to a backup node when the current node encounters exceptions. The corresponding error message is optimized for this issue. [#43489](https://github.com/StarRocks/starrocks/pull/43489) * Memory information is incorrect in the container environment. [#43225](https://github.com/StarRocks/starrocks/issues/43225) * An exception is thrown when INSERT tasks are canceled. [#44239](https://github.com/StarRocks/starrocks/pull/44239) * Expression-based dynamic partitions cannot be automatically created. [#44163](https://github.com/StarRocks/starrocks/pull/44163) * Creating partitions may cause FE deadlock. [#44974](https://github.com/StarRocks/starrocks/pull/44974) #### 3.2.6[​](#326 "Direct link to 3.2.6") Release date: April 18, 2024 ##### Bug Fixes[​](#bug-fixes-10 "Direct link to Bug Fixes") Fixed the following issue: * The privileges of external tables cannot be found due to incompatibility issues. [#44030](https://github.com/StarRocks/starrocks/pull/44030) #### 3.2.5 (Yanked)[​](#325-yanked "Direct link to 3.2.5 (Yanked)") Release date: April 12, 2024 tip This version has been taken offline due to privilege issues in querying external tables in external catalogs such as Hive and Iceberg. * **Problem**: When a user queries data from an external table in an external catalog, access to this table is denied even when the user has the SELECT privilege on this table. SHOW GRANTS also shows that the user has this privilege. * **Impact scope**: This problem only affects queries on external tables in external catalogs. Other queries are not affected. * **Temporary workaround**: The query succeeds after the SELECT privilege on this table is granted to the user again. But `SHOW GRANTS` will return duplicate privilege entries. After an upgrade to v3.2.6, users can run `REVOKE` to remove one of the privilege entries. ##### New Features[​](#new-features-4 "Direct link to New Features") * Supports the [dict\_mapping](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/dict-functions/dict_mapping/) column property, which can significantly facilitate the loading process during the construction of a global dictionary, accelerating the exact COUNT DISTINCT calculation. ##### Behavior Changes[​](#behavior-changes-4 "Direct link to Behavior Changes") * When null values in JSON data are evaluated based on the `IS NULL` operator, they are considered NULL values following SQL language. For example, `true` is returned for `SELECT parse_json('{"a": null}') -> 'a' IS NULL` (before this behavior change, `false` is returned). [#42765](https://github.com/StarRocks/starrocks/pull/42765) ##### Improvements[​](#improvements-9 "Direct link to Improvements") * Optimized the column type unionization rules for automatic schema detection in the FILES table function. When columns with the same name but different types exist in separate files, FILES will attempt to merge them by selecting the type with the larger granularity as the final type. For example, if there are columns with the same name but of types FLOAT and INT respectively, FILES will return DOUBLE as the final type. [#40959](https://github.com/StarRocks/starrocks/pull/40959) * Primary Key tables support Size-tiered Compaction to reduce the I/O amplification. [#41130](https://github.com/StarRocks/starrocks/pull/41130) * When Broker Load is used to load data from ORC files that contain TIMESTAMP-type data, StarRocks supports retaining microseconds in the timestamps when converting the timestamps to match its own DATETIME data type. [#42179](https://github.com/StarRocks/starrocks/pull/42179) * Optimized the error messages for Routine Load. [#41306](https://github.com/StarRocks/starrocks/pull/41306) * Optimized the error messages when the FILES table function is used to convert invalid data types. [#42717](https://github.com/StarRocks/starrocks/pull/42717) ##### Bug Fixes[​](#bug-fixes-11 "Direct link to Bug Fixes") Fixed the following issues: * FEs fail to start after system-defined views are dropped. Dropping system-defined views is now prohibited. [#43552](https://github.com/StarRocks/starrocks/pull/43552) * BEs crash when duplicate sort key columns exist in Primary Key tables. Duplicate sort key columns are now prohibited. [#43206](https://github.com/StarRocks/starrocks/pull/43206) * An error, instead of NULL, is returned when the input value of the to\_json() function is NULL. [#42171](https://github.com/StarRocks/starrocks/pull/42171) * In shared-data mode, the garbage collection and thread eviction mechanisms for handling persistent indexes created on Primary Key tables cannot take effect on CN nodes. As a result, obsolete data cannot be deleted. [#41955](https://github.com/StarRocks/starrocks/pull/41955) * In shared-data mode, an error is returned when users modify the `enable_persistent_index` property of a Primary Key table. [#42890](https://github.com/StarRocks/starrocks/pull/42890) * In shared-data mode, NULL values are given to columns that are not supposed to be changed when users update a Primary Key table with partial updates in column mode. [#42355](https://github.com/StarRocks/starrocks/pull/42355) * Queries cannot be rewritten with asynchronous materialized views created on logical views. [#42173](https://github.com/StarRocks/starrocks/pull/42173) * CNs crash when the Cross-cluster Data Migration Tool is used to migrate Primary Key tables to a shared-data cluster. [#42260](https://github.com/StarRocks/starrocks/pull/42260) * The partition ranges of the external catalog-based asynchronous materialized views are not consecutive. [#41957](https://github.com/StarRocks/starrocks/pull/41957) #### 3.2.4 (Yanked)[​](#324-yanked "Direct link to 3.2.4 (Yanked)") Release date: March 12, 2024 tip This version has been taken offline due to privilege issues in querying external tables in external catalogs such as Hive and Iceberg. * **Problem**: When a user queries data from an external table in an external catalog, access to this table is denied even when the user has the SELECT privilege on this table. SHOW GRANTS also shows that the user has this privilege. * **Impact scope**: This problem only affects queries on external tables in external catalogs. Other queries are not affected. * **Temporary workaround**: The query succeeds after the SELECT privilege on this table is granted to the user again. But `SHOW GRANTS` will return duplicate privilege entries. After an upgrade to v3.2.6, users can run `REVOKE` to remove one of the privilege entries. ##### New Features[​](#new-features-5 "Direct link to New Features") * Cloud-native Primary Key tables in shared-data clusters support Size-tiered Compaction to reduce the write I/O amplification. [#41034](https://github.com/StarRocks/starrocks/pull/41034) * Added the date function `milliseconds_diff`. [#38171](https://github.com/StarRocks/starrocks/pull/38171) * Added the session variable `catalog`, which specifies the catalog to which the session belongs. [#41329](https://github.com/StarRocks/starrocks/pull/41329) * Supports [setting user-defined variables in hints](https://docs.starrocks.io/docs/3.2/administration/Query_planning/#user-defined-variable-hint). [#40746](https://github.com/StarRocks/starrocks/pull/40746) * Supports CREATE TABLE LIKE in Hive catalogs. [#37685](https://github.com/StarRocks/starrocks/pull/37685) * Added the view `information_schema.partitions_meta`, which records detailed metadata of partitions. [#39265](https://github.com/StarRocks/starrocks/pull/39265) * Added the view `sys.fe_memory_usage`, which records the memory usage for StarRocks. [#40464](https://github.com/StarRocks/starrocks/pull/40464) ##### Behavior Changes[​](#behavior-changes-5 "Direct link to Behavior Changes") * `cbo_decimal_cast_string_strict` is used to control how CBO converts data from the DECIMAL type to the STRING type. The default value `true` indicates that the logic built in v2.5.x and later versions prevails and the system implements strict conversion (namely, the system truncates the generated string and fills 0s based on the scale length). The DECIMAL type is not strictly filled in earlier versions, causing different results when comparing the DECIMAL type and the STRING type. [#40619](https://github.com/StarRocks/starrocks/pull/40619) * The default value of the Iceberg Catalog parameter `enable_iceberg_metadata_cache` has been changed to `false`. From v3.2.1 to v3.2.3, this parameter is set to `true` by default, regardless of what metastore service is used. In v3.2.4 and later, if the Iceberg cluster uses AWS Glue as metastore, this parameter still defaults to `true`. However, if the Iceberg cluster uses other metastore service such as Hive metastore, this parameter defaults to `false`. [#41826](https://github.com/StarRocks/starrocks/pull/41826) * The user who can refresh materialized views is changed from the `root` user to the user who creates the materialized views. This change does not affect existing materialized views. [#40670](https://github.com/StarRocks/starrocks/pull/40670) * By default, when comparing columns of constant and string types, StarRocks compares them as strings. Users can use the session variable `cbo_eq_base_type` to adjust the rule used for the comparison. For example, users can set `cbo_eq_base_type` to `decimal`, and StarRocks then compares the columns as numeric values. [#40619](https://github.com/StarRocks/starrocks/pull/40619) ##### Improvements[​](#improvements-10 "Direct link to Improvements") * Shared-data StarRocks clusters support the Partitioned Prefix feature for S3-compatible object storage systems. When this feature is enabled, StarRocks stores the data into multiple, uniformly prefixed partitions (sub-paths) under the bucket. This improves the read and write efficiency on data files in S3-compatible object storages. [#41627](https://github.com/StarRocks/starrocks/pull/41627) * StarRocks supports using the parameter `s3_compatible_fs_list` to specify which S3-compatible object storage can be accessed via AWS SDK, and supports using the parameter `fallback_to_hadoop_fs_list` to specify non-S3-compatible object storages that require access via HDFS Schema (this method requires the use of vendor-provided JAR packages). [#41123](https://github.com/StarRocks/starrocks/pull/41123) * Optimized compatibility with Trino. Supports syntax conversion from the following Trino functions: current\_catalog, current\_schema, to\_char, from\_hex, to\_date, to\_timestamp, and index. [#41217](https://github.com/StarRocks/starrocks/pull/41217) [#41319](https://github.com/StarRocks/starrocks/pull/41319) [#40803](https://github.com/StarRocks/starrocks/pull/40803) * Optimized the query rewrite logic of materialized views. StarRocks can rewrite queries with materialized views created upon logical views. [#42173](https://github.com/StarRocks/starrocks/pull/42173) * Improved the efficiency of converting the STRING type to the DATETIME type by 35% to 40%. [#41464](https://github.com/StarRocks/starrocks/pull/41464) * The `agg_type` of BITMAP-type columns in an Aggregate table can be set to `replace_if_not_null` in order to support updates only to a few columns of the table. [#42034](https://github.com/StarRocks/starrocks/pull/42034) * Improved the Broker Load performance when loading small ORC files. [#41765](https://github.com/StarRocks/starrocks/pull/41765) * The tables with hybrid row-column storage support Schema Change. [#40851](https://github.com/StarRocks/starrocks/pull/40851) * The tables with hybrid row-column storage support complex types including BITMAP, HLL, JSON, ARRAY, MAP, and STRUCT. [#41476](https://github.com/StarRocks/starrocks/pull/41476) * A new internal SQL log file is added to record log data related to statistics and materialized views. [#40453](https://github.com/StarRocks/starrocks/pull/40453) ##### Bug Fixes[​](#bug-fixes-12 "Direct link to Bug Fixes") Fixed the following issues: * "Analyze Error" is thrown if inconsistent letter cases are assigned to the names or aliases of tables or views queried in the creation of a Hive view. [#40921](https://github.com/StarRocks/starrocks/pull/40921) * I/O usage reaches the upper limit if persistent indexes are created on Primary Key tables. [#39959](https://github.com/StarRocks/starrocks/pull/39959) * In shared-data clusters, primary key index directories are deleted every 5 hours. [#40745](https://github.com/StarRocks/starrocks/pull/40745) * After users execute ALTER TABLE COMPACT by hand, the memory usage statistics for compaction operations are abnormal. [#41150](https://github.com/StarRocks/starrocks/pull/41150) * Retries of the Publish phase may hang for Primary Key tables. [#39890](https://github.com/StarRocks/starrocks/pull/39890) #### 3.2.3[​](#323 "Direct link to 3.2.3") Release date: February 8, 2024 ##### New Features[​](#new-features-6 "Direct link to New Features") * \[Preview] Supports hybrid row-column storage for tables. It allows better performance for high concurrency, low-latency point lookups against Primary Key tables and partial data updates. Currently, this feature does not support modification via ALTER TABLE, changing Sort Key, and partial updates in column mode. * Supports backing up and restoring asynchronous materialized views. * Broker Load supports loading JSON-type data. * Supports query rewrite using asynchronous materialized views created upon views. Queries against a view can be rewritten based on materialized views that are created upon that view. * Supports CREATE OR REPLACE PIPE. [#37658](https://github.com/StarRocks/starrocks/pull/37658) ##### Behavior Changes[​](#behavior-changes-6 "Direct link to Behavior Changes") * Added the session variable `enable_strict_order_by`. When this variable is set to the default value `TRUE`, an error is reported for such a query pattern: Duplicate alias is used in different expressions of the query and this alias is also a sorting field in ORDER BY, for example, `select distinct t1.* from tbl1 t1 order by t1.k1;`. The logic is the same as that in v2.3 and earlier. When this variable is set to `FALSE`, a loose deduplication mechanism is used, which processes such queries as valid SQL queries. [#37910](https://github.com/StarRocks/starrocks/pull/37910) * Added the session variable `enable_materialized_view_for_insert`, which controls whether materialized views rewrite the queries in INSERT INTO SELECT statements. The default value is `false`. [#37505](https://github.com/StarRocks/starrocks/pull/37505) * When a single query is executed within the Pipeline framework, its memory limit is now constrained by the variable `query_mem_limit` instead of `exec_mem_limit`. Setting the value of `query_mem_limit` to `0` indicates no limit. [#34120](https://github.com/StarRocks/starrocks/pull/34120) ##### Parameter Changes[​](#parameter-changes "Direct link to Parameter Changes") * Added the FE configuration item `http_worker_threads_num`, which specifies the number of threads for HTTP server to deal with HTTP requests. The default value is `0`. If the value for this parameter is set to a negative value or `0`, the actual thread number is twice the number of CPU cores. [#37530](https://github.com/StarRocks/starrocks/pull/37530) * Added the BE configuration item `lake_pk_compaction_max_input_rowsets`, which controls the maximum number of input rowsets allowed in a Primary Key table compaction task in a shared-data StarRocks cluster. This helps optimize resource consumption for compaction tasks. [#39611](https://github.com/StarRocks/starrocks/pull/39611) * Added the session variable `connector_sink_compression_codec`, which specifies the compression algorithm used for writing data into Hive tables or Iceberg tables, or exporting data with Files(). Valid algorithms include GZIP, BROTLI, ZSTD, and LZ4. [#37912](https://github.com/StarRocks/starrocks/pull/37912) * Added the FE configuration item `routine_load_unstable_threshold_second`. [#36222](https://github.com/StarRocks/starrocks/pull/36222) * Added the BE configuration item `pindex_major_compaction_limit_per_disk` to configure the maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. The default value is `1`. [#36681](https://github.com/StarRocks/starrocks/pull/36681) * Added the BE configuration item `enable_lazy_delta_column_compaction`. The default value is `true`, indicating that StarRocks does not perform frequent compaction operations on delta columns. [#36654](https://github.com/StarRocks/starrocks/pull/36654) * Added the FE configuration item `default_mv_refresh_immediate`, which specifies whether to immediately refresh the materialized view after the materialized view is created. The default value is `true`. [#37093](https://github.com/StarRocks/starrocks/pull/37093) * Changed the default value of the FE configuration item `default_mv_refresh_partition_num`to `1`. This indicates that when multiple partitions need to be updated during a materialized view refresh, the task will be split in batches, refreshing only one partition at a time. This helps reduce resource consumption during each refresh. [#36560](https://github.com/StarRocks/starrocks/pull/36560) * Changed the default value of the BE/CN configuration item `starlet_use_star_cache` to `true`. This indicates that Data Cache is enabled by default in shared-data clusters. If, before upgrading, you have manually configured the BE/CN configuration item `starlet_cache_evict_high_water` to `X`, you must configure the BE/CN configuration item `starlet_star_cache_disk_size_percent` to `(1.0 - X) * 100`. For example, if you have set `starlet_cache_evict_high_water` to `0.3` before upgrading, you must set `starlet_star_cache_disk_size_percent` to `70`. This ensures that both file data cache and Data Cache will not exceed the disk capacity limit. [#38200](https://github.com/StarRocks/starrocks/pull/38200) ##### Improvements[​](#improvements-11 "Direct link to Improvements") * Added date formats `yyyy-MM-ddTHH:mm` and `yyyy-MM-dd HH:mm` to support TIMESTAMP partition fields in Apache Iceberg tables. [#39986](https://github.com/StarRocks/starrocks/pull/39986) * Added Data Cache-related metrics to the monitoring API. [#40375](https://github.com/StarRocks/starrocks/pull/40375) * Optimized BE log printing to prevent too many irrelevant logs. [#22820](https://github.com/StarRocks/starrocks/pull/22820) [#36187](https://github.com/StarRocks/starrocks/pull/36187) * Added the field `storage_medium` to the view `information_schema.be_tablets`. [#37070](https://github.com/StarRocks/starrocks/pull/37070) * Supports `SET_VAR` in multiple sub-queries. [#36871](https://github.com/StarRocks/starrocks/pull/36871) * A new field `LatestSourcePosition` is added to the return result of SHOW ROUTINE LOAD to record the position of the latest message in each partition of the Kafka topic, helping check the latencies of data loading. [#38298](https://github.com/StarRocks/starrocks/pull/38298) * When the string on the right side of the LIKE operator within the WHERE clause does not include `%` or `_`, the LIKE operator is converted into the `=` operator. [#37515](https://github.com/StarRocks/starrocks/pull/37515) * The default retention period of trash files is changed to 1 day from the original 3 days. [#37113](https://github.com/StarRocks/starrocks/pull/37113) * Supports collecting statistics from Iceberg tables with Partition Transform. [#39907](https://github.com/StarRocks/starrocks/pull/39907) * The scheduling policy for Routine Load is optimized, so that slow tasks do not block the execution of the other normal tasks. [#37638](https://github.com/StarRocks/starrocks/pull/37638) ##### Bug Fixes[​](#bug-fixes-13 "Direct link to Bug Fixes") Fixed the following issues: * The execution of ANALYZE TABLE gets stuck occasionally. [#36836](https://github.com/StarRocks/starrocks/pull/36836) * The memory consumption by PageCache exceeds the threshold specified by the BE dynamic parameter `storage_page_cache_limit` in certain circumstances. [#37740](https://github.com/StarRocks/starrocks/pull/37740) * Hive metadata in Hive catalogs is not automatically refreshed when new fields are added to Hive tables. [#37549](https://github.com/StarRocks/starrocks/pull/37549) * In some cases, `bitmap_to_string` may return incorrect results due to data type overflow. [#37405](https://github.com/StarRocks/starrocks/pull/37405) * When `SELECT ... FROM ... INTO OUTFILE` is executed to export data into CSV files, the error "Unmatched number of columns" is reported if the FROM clause contains multiple constants. [#38045](https://github.com/StarRocks/starrocks/pull/38045) * In some cases, querying semi-structured data in tables may cause BEs to crash. [#40208](https://github.com/StarRocks/starrocks/pull/40208) #### 3.2.2[​](#322 "Direct link to 3.2.2") Release date: December 30, 2023 ##### Bug Fixes[​](#bug-fixes-14 "Direct link to Bug Fixes") Fixed the following issue: * When StarRocks is upgraded from v3.1.2 or earlier to v3.2, FEs may fail to restart. [#38172](https://github.com/StarRocks/starrocks/pull/38172) #### 3.2.1[​](#321 "Direct link to 3.2.1") Release date: December 21, 2023 ##### New Features[​](#new-features-7 "Direct link to New Features") ###### Data Lake Analytics[​](#data-lake-analytics "Direct link to Data Lake Analytics") * Supports reading [Hive Catalog](https://docs.starrocks.io/docs/3.2/data_source/catalog/hive_catalog/) tables and file external tables in Avro, SequenceFile, and RCFile formats through Java Native Interface (JNI). ###### Materialized View[​](#materialized-view "Direct link to Materialized View") * Added a view `object_dependencies` to the database `sys`. It contains the lineage information of asynchronous materialized views. [#35060](https://github.com/StarRocks/starrocks/pull/35060) * Supports creating synchronous materialized views with the WHERE clause. * Supports partition-level incremental refresh for asynchronous materialized views created upon Iceberg catalogs. * \[Preview] Supports creating asynchronous materialized views based on tables in a Paimon catalog with partition-level refresh. ###### Query and SQL functions[​](#query-and-sql-functions "Direct link to Query and SQL functions") * Supports the prepared statement. It allows better performance for processing high concurrency point lookup queries. It also prevents SQL injection effectively. * Supports the following Bitmap functions: [subdivide\_bitmap](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/bitmap-functions/subdivide_bitmap/), [bitmap\_from\_binary](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/bitmap-functions/bitmap_from_binary/), and [bitmap\_to\_binary](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/bitmap-functions/bitmap_to_binary/). * Supports the Array function [array\_unique\_agg](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/array-functions/array_unique_agg/). ###### Monitoring and alerts[​](#monitoring-and-alerts "Direct link to Monitoring and alerts") * Added a new metric `max_tablet_rowset_num` for setting the maximum allowed number of rowsets. This metric helps detect possible compaction issues and thus reduces the occurrences of the error "too many versions". [#36539](https://github.com/StarRocks/starrocks/pull/36539) ##### Parameter change[​](#parameter-change "Direct link to Parameter change") * A new BE configuration item `enable_stream_load_verbose_log` is added. The default value is `false`. With this parameter set to `true`, StarRocks can record the HTTP requests and responses for Stream Load jobs, making troubleshooting easier. [#36113](https://github.com/StarRocks/starrocks/pull/36113) ##### Improvements[​](#improvements-12 "Direct link to Improvements") * Upgraded the default GC algorithm in JDK8 to G1. [#37268](https://github.com/StarRocks/starrocks/pull/37268) * A new value option `GROUP_CONCAT_LEGACY` is added to the session variable [sql\_mode](https://docs.starrocks.io/docs/3.2/sql-reference/System_variable/#sql_mode) to provide compatibility with the implementation logic of the [group\_concat](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/string-functions/group_concat/) function in versions earlier than v2.5. [#36150](https://github.com/StarRocks/starrocks/pull/36150) * The authentication information `aws.s3.access_key` and `aws.s3.access_secret` for [AWS S3 in Broker Load jobs](https://docs.starrocks.io/docs/3.2/loading/s3/) are hidden in audit logs. [#36571](https://github.com/StarRocks/starrocks/pull/36571) * The `be_tablets` view in the `information_schema` database provides a new field `INDEX_DISK`, which records the disk usage (measured in bytes) of persistent indexes. [#35615](https://github.com/StarRocks/starrocks/pull/35615) * The result returned by the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/3.2/sql-reference/sql-statements/data-manipulation/SHOW_ROUTINE_LOAD/) statement provides a new field `OtherMsg`, which shows information about the last failed task. [#35806](https://github.com/StarRocks/starrocks/pull/35806) ##### Bug Fixes[​](#bug-fixes-15 "Direct link to Bug Fixes") Fixed the following issues: * The BEs crash if users create persistent indexes in the event of data corruption.[#30841](https://github.com/StarRocks/starrocks/pull/30841) * The [array\_distinct](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/array-functions/array_distinct/) function occasionally causes the BEs to crash. [#36377](https://github.com/StarRocks/starrocks/pull/36377) * After the DISTINCT window operator pushdown feature is enabled, errors are reported if SELECT DISTINCT operations are performed on the complex expressions of the columns computed by window functions. [#36357](https://github.com/StarRocks/starrocks/pull/36357) * Some S3-compatible object storage returns duplicate files, causing the BEs to crash. [#36103](https://github.com/StarRocks/starrocks/pull/36103) #### 3.2.0[​](#320 "Direct link to 3.2.0") Release date: December 1, 2023 ##### New Features[​](#new-features-8 "Direct link to New Features") ###### Shared-data cluster[​](#shared-data-cluster "Direct link to Shared-data cluster") * Supports persisting indexes of [Primary Key tables](https://docs.starrocks.io/docs/3.2/table_design/table_types/primary_key_table/) to local disks. * Supports even distribution of Data Cache among multiple local disks. ###### Materialized View[​](#materialized-view-1 "Direct link to Materialized View") **Asynchronous materialized view** * The Query Dump file can include information of asynchronous materialized views. * The Spill to Disk feature is enabled by default for the refresh tasks of asynchronous materialized views, reducing memory consumption. ###### Data Lake Analytics[​](#data-lake-analytics-1 "Direct link to Data Lake Analytics") * Supports creating and dropping databases and managed tables in [Hive catalogs](https://docs.starrocks.io/docs/3.2/data_source/catalog/hive_catalog/), and supports exporting data to Hive's managed tables using INSERT or INSERT OVERWRITE. * Supports [Unified Catalog](https://docs.starrocks.io/docs/3.2/data_source/catalog/unified_catalog/), with which users can access different table formats (Hive, Iceberg, Hudi, and Delta Lake) that share a common metastore like Hive metastore or AWS Glue. * Supports collecting statistics of Hive and Iceberg tables using ANALYZE TABLE, and storing the statistics in StarRocks, thus facilitating optimization of query plans and accelerating subsequent queries. * Supports Information Schema for external tables, providing additional convenience for interactions between external systems (such as BI tools) and StarRocks. ###### Storage engine, data ingestion, and export[​](#storage-engine-data-ingestion-and-export "Direct link to Storage engine, data ingestion, and export") * Added the following features of loading with the table function [FILES()](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/table-functions/files/): * Loading Parquet and ORC format data from Azure or GCP. * Extracting the value of a key/value pair from the file path as the value of a column using the parameter `columns_from_path`. * Loading complex data types including ARRAY, JSON, MAP, and STRUCT. * Supports unloading data from StarRocks to Parquet-formatted files stored in AWS S3 or HDFS by using INSERT INTO FILES. For detailed instructions, see [Unload data using INSERT INTO FILES](https://docs.starrocks.io/docs/3.2/unloading/unload_using_insert_into_files/). * Supports [manual optimization of table structure and data distribution strategy](https://docs.starrocks.io/docs/3.2/table_design/Data_distribution#optimize-data-distribution-after-table-creation-since-32) used in an existing table to optimize the query and loading performance. You can set a new bucket key, bucket number, or sort key for a table. You can also set a different bucket number for specific partitions. * Supports continuous data loading from [AWS S3](https://docs.starrocks.io/docs/3.2/loading/s3/#use-pipe) or [HDFS](https://docs.starrocks.io/docs/3.2/loading/hdfs_load/#use-pipe) using the PIPE method. * When PIPE detects new or modifications in a remote storage directory, it can automatically load the new or modified data into the destination table in StarRocks. While loading data, PIPE automatically splits a large loading task into smaller, serialized tasks, enhancing stability in large-scale data ingestion scenarios and reducing the cost of error retries. ###### Query[​](#query "Direct link to Query") * Supports [HTTP SQL API](https://docs.starrocks.io/docs/3.2/reference/HTTP_API/SQL/), enabling users to access StarRocks data via HTTP and execute SELECT, SHOW, EXPLAIN, or KILL operations. * Supports Runtime Profile and text-based Profile analysis commands (SHOW PROFILELIST, ANALYZE PROFILE, EXPLAIN ANALYZE) to allow users to directly analyze profiles via MySQL clients, facilitating bottleneck identification and discovery of optimization opportunities. ###### SQL reference[​](#sql-reference "Direct link to SQL reference") Added the following functions: * String functions: substring\_index, url\_extract\_parameter, url\_encode, url\_decode, and translate * Date functions: dayofweek\_iso, week\_iso, quarters\_add, quarters\_sub, milliseconds\_add, milliseconds\_sub, date\_diff, jodatime\_format, str\_to\_jodatime, to\_iso8601, to\_tera\_date, and to\_tera\_timestamp * Pattern matching function: regexp\_extract\_all * hash function: xx\_hash3\_64 * Aggregate functions: approx\_top\_k * Window functions: cume\_dist, percent\_rank and session\_number * Utility functions: get\_query\_profile and is\_role\_in\_session ###### Privileges and security[​](#privileges-and-security "Direct link to Privileges and security") StarRocks supports access control through [Apache Ranger](https://docs.starrocks.io/docs/3.2/administration/ranger_plugin/), providing a higher level of data security and allowing the reuse of existing services of external data sources. After integrating with Apache Ranger, StarRocks enables the following access control methods: * When accessing internal tables, external tables, or other objects in StarRocks, access control can be enforced based on the access policies configured for the StarRocks Service in Ranger. * When accessing an external catalog, access control can also leverage the corresponding Ranger service of the original data source (such as Hive Service) to control access (currently, access control for exporting data to Hive is not yet supported). For more information, see [Manage permissions with Apache Ranger](https://docs.starrocks.io/docs/3.2/administration/ranger_plugin/). ##### Improvements[​](#improvements-13 "Direct link to Improvements") ###### Data Lake Analytics[​](#data-lake-analytics-2 "Direct link to Data Lake Analytics") * Optimized ORC Reader: * Optimized the ORC Column Reader, resulting in nearly a two-fold performance improvement for VARCHAR and CHAR data reading. * Optimized the decompression performance of ORC files in Zlib compression format. * Optimized Parquet Reader: * Supports adaptive I/O merging, allowing adaptive merging of columns with and without predicates based on filtering effects, thus reducing I/O. * Optimized Dict Filter for faster predicate rewriting. Supports STRUCT sub-columns, and on-demand dictionary column decoding. * Optimized Dict Decode performance. * Optimized late materialization performance. * Supports caching file footers to avoid repeated computation overhead. * Supports decompression of Parquet files in lzo compression format. * Optimized CSV Reader: * Optimized the Reader performance. * Supports decompression of CSV files in Snappy and lzo compression formats. * Optimized the performance of the count calculation. * Optimized Iceberg Catalog capabilities: * Supports collecting column statistics from Manifest files to accelerate queries. * Supports collecting NDV (number of distinct values) from Puffin files to accelerate queries. * Supports partition pruning. * Reduced Iceberg metadata memory consumption to enhance stability in scenarios with large metadata volume or high query concurrency. ###### Materialized View[​](#materialized-view-2 "Direct link to Materialized View") **Asynchronous materialized view** * Supports automatic refresh for an asynchronous materialized view created upon views or materialized views when schema changes occur on the views, materialized views, or their base tables. * Data consistency: * Added the property `query_rewrite_consistency` for asynchronous materialized view creation. This property defines the query rewrite rules based on the consistency check. * Add the property `force_external_table_query_rewrite` for external catalog-based asynchronous materialized view creation. This property defines whether to allow force query rewrite for asynchronous materialized views created upon external catalogs. * For detailed information, see [CREATE MATERIALIZED VIEW](https://docs.starrocks.io/docs/3.2/sql-reference/sql-statements/data-definition/CREATE_MATERIALIZED_VIEW/). * Added a consistency check for materialized views' partitioning key. * When users create an asynchronous materialized view with window functions that include a PARTITION BY expression, the partitioning column of the window function must match that of the materialized view. ###### Storage engine, data ingestion, and export[​](#storage-engine-data-ingestion-and-export-1 "Direct link to Storage engine, data ingestion, and export") * Optimized the persistent index for Primary Key tables by improving memory usage logic while reducing I/O read and write amplification. [#24875](https://github.com/StarRocks/starrocks/pull/24875) [#27577](https://github.com/StarRocks/starrocks/pull/27577) [#28769](https://github.com/StarRocks/starrocks/pull/28769) * Supports data re-distribution across local disks for Primary Key tables. * Partitioned tables support automatic cooldown based on the partition time range and cooldown time. Compared to the original cooldown logic, it is more convenient to perform hot and cold data management on the partition level. For more information, see [Specify initial storage medium, automatic storage cooldown time, replica number](https://docs.starrocks.io/docs/3.2/sql-reference/sql-statements/data-definition/CREATE_TABLE#specify-initial-storage-medium-automatic-storage-cooldown-time-replica-number). * The Publish phase of a load job that writes data into a Primary Key table is changed from asynchronous mode to synchronous mode. As such, the data loaded can be queried immediately after the load job finishes. For more information, see [enable\_sync\_publish](https://docs.starrocks.io/docs/3.2/administration/FE_configuration#enable_sync_publish). * Supports Fast Schema Evolution, which is controlled by the table property [`fast_schema_evolution`](https://docs.starrocks.io/docs/3.2/sql-reference/sql-statements/data-definition/CREATE_TABLE#set-fast-schema-evolution). After this feature is enabled, the execution efficiency of adding or dropping columns is significantly improved. This mode is disabled by default (Default value is `false`). You cannot modify this property for existing tables using ALTER TABLE. * [Supports dynamically adjusting the number of tablets to create](https://docs.starrocks.io/docs/3.2/table_design/Data_distribution#set-the-number-of-buckets) according to cluster information and the size of the data for **Duplicate Key** tables created with the Radom Bucketing strategy. ###### Query[​](#query-1 "Direct link to Query") * Optimized StarRocks' compatibility with Metabase and Superset. Supports integrating them with external catalogs. ###### SQL Reference[​](#sql-reference-1 "Direct link to SQL Reference") * [array\_agg](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/array-functions/array_agg/) supports the keyword DISTINCT. * INSERT, UPDATE, and DELETE operations now support `SET_VAR`. [#35283](https://github.com/StarRocks/starrocks/pull/35283) ###### Others[​](#others "Direct link to Others") * Added the session variable `large_decimal_underlying_type = "panic"|"double"|"decimal"` to set the rules to deal with DECIMAL type overflow. `panic` indicates returning an error immediately, `double` indicates converting the data to DOUBLE type, and `decimal` indicates converting the data to DECIMAL(38,s). ##### Developer tools[​](#developer-tools "Direct link to Developer tools") * Supports Trace Query Profile for asynchronous materialized views, which can be used to analyze its transparent rewrite. ##### Behavior Change[​](#behavior-change "Direct link to Behavior Change") To be updated. ##### Parameter Change[​](#parameter-change-1 "Direct link to Parameter Change") ###### FE Parameters[​](#fe-parameters "Direct link to FE Parameters") * Added the following FE configuration items: * `catalog_metadata_cache_size` * `enable_backup_materialized_view` * `enable_colocate_mv_index` * `enable_fast_schema_evolution` * `json_file_size_limit` * `lake_enable_ingest_slowdown` * `lake_ingest_slowdown_threshold` * `lake_ingest_slowdown_ratio` * `lake_compaction_score_upper_bound` * `mv_auto_analyze_async` * `primary_key_disk_schedule_time` * `statistic_auto_collect_small_table_rows` * `stream_load_task_keep_max_num` * `stream_load_task_keep_max_second` * Removed FE configuration item `enable_pipeline_load`. * Default value modifications: * The default value of `enable_sync_publish` is changed from `false` to `true`. * The default value of `enable_persistent_index_by_default` is changed from `false` to `true`. ###### BE Parameters[​](#be-parameters "Direct link to BE Parameters") * Data Cache-related configuration changes. * Added `datacache_enable` to replace `block_cache_enable`. * Added `datacache_mem_size` to replace `block_cache_mem_size`. * Added `datacache_disk_size` to replace `block_cache_disk_size`. * Added `datacache_disk_path` to replace `block_cache_disk_path`. * Added `datacache_meta_path` to replace `block_cache_meta_path`. * Added `datacache_block_size` to replace `block_cache_block_size`. * Added `datacache_checksum_enable` to replace `block_cache_checksum_enable`. * Added `datacache_direct_io_enable` to replace `block_cache_direct_io_enable`. * Added `datacache_max_concurrent_inserts` to replace `block_cache_max_concurrent_inserts`. * Added `datacache_max_flying_memory_mb`. * Added `datacache_engine` to replace `block_cache_engine`. * Removed `block_cache_max_parcel_memory_mb`. * Removed `block_cache_report_stats`. * Removed `block_cache_lru_insertion_point`. After renaming Block Cache to Data Cache, StarRocks has introduced a new set of BE parameters prefixed with `datacache` to replace the original parameters prefixed with `block_cache`. After upgrade to v3.2, the original parameters will still be effective. Once enabled, the new parameters will override the original ones. The mixed usage of new and original parameters is not supported, as it may result in some configurations not taking effect. In the future, StarRocks plans to deprecate the original parameters with the `block_cache` prefix, so we recommend you use the new parameters with the `datacache` prefix. * Added the following BE configuration items: * `spill_max_dir_bytes_ratio` * `streaming_agg_limited_memory_size` * `streaming_agg_chunk_buffer_size` * Removed the following BE configuration items: * Dynamic parameter `tc_use_memory_min` * Dynamic parameter `tc_free_memory_rate` * Dynamic parameter `tc_gc_period` * Static parameter `tc_max_total_thread_cache_byte` * Default value modifications: * The default value of `disable_column_pool` is changed from `false` to `true`. * The default value of `thrift_port` is changed from `9060` to `0`. * The default value of `enable_load_colocate_mv` is changed from `false` to `true`. * The default value of `enable_pindex_minor_compaction` is changed from `false` to `true`. ###### System Variables[​](#system-variables "Direct link to System Variables") * Added the following session variables: * `enable_per_bucket_optimize` * `enable_write_hive_external_table` * `hive_temp_staging_dir` * `spill_revocable_max_bytes` * `thrift_plan_protocol` * Removed the following session variables: * `enable_pipeline_query_statistic` * `enable_deliver_batch_fragments` * Renamed the following session variables: * `enable_scan_block_cache` is renamed as `enable_scan_datacache`. * `enable_populate_block_cache` is renamed as `enable_populate_datacache`. ###### Reserved Keywords[​](#reserved-keywords "Direct link to Reserved Keywords") Added reserved keywords `OPTIMIZE` and `PREPARE`. ##### Bug Fixes[​](#bug-fixes-16 "Direct link to Bug Fixes") Fixed the following issues: * BEs crash when libcurl is invoked. [#31667](https://github.com/StarRocks/starrocks/pull/31667) * Schema Change may fail if it takes an excessively long period of time, because the specified tablet version is handled by garbage collection. [#31376](https://github.com/StarRocks/starrocks/pull/31376) * Failed to access the Parquet files in MinIO via file external tables. [#29873](https://github.com/StarRocks/starrocks/pull/29873) * The ARRAY, MAP, and STRUCT type columns are not correctly displayed in `information_schema.columns`. [#33431](https://github.com/StarRocks/starrocks/pull/33431) * An error is reported if specific path formats are used during data loading via Broker Load: `msg:Fail to parse columnsFromPath, expected: [rec_dt]`. [#32720](https://github.com/StarRocks/starrocks/pull/32720) * `DATA_TYPE` and `COLUMN_TYPE` for BINARY or VARBINARY data types are displayed as `unknown` in the `information_schema.columns` view. [#32678](https://github.com/StarRocks/starrocks/pull/32678) * Complex queries that involve many unions, expressions, and SELECT columns can result in a sudden surge in the bandwidth or CPU usage within an FE node. * The refresh of asynchronous materialized view may occasionally encounter deadlock. [#35736](https://github.com/StarRocks/starrocks/pull/35736) ##### Upgrade Notes[​](#upgrade-notes "Direct link to Upgrade Notes") * Optimization on **Random Bucketing** is disabled by default. To enable it, you need to add the property `bucket_size` when creating tables. This allows the system to dynamically adjust the number of tablets based on cluster information and the size of loaded data. Please note that once this optimization is enabled, if you need to roll back your cluster to v3.1 or earlier, you must delete tables with this optimization enabled and manually execute a metadata checkpoint (by executing `ALTER SYSTEM CREATE IMAGE`). Otherwise, the rollback will fail. * Starting from v3.2.0, StarRocks has disabled non-Pipeline queries. Therefore, before upgrading your cluster to v3.2, you need to globally enable the Pipeline engine (by adding the configuration `enable_pipeline_engine=true` in the FE configuration file **fe.conf**). Failure to do so will result in errors for non-Pipeline queries. --- ## Release 3.3 ### StarRocks version 3.3 warning * After upgrading StarRocks to v3.3, DO NOT downgrade it directly to v3.2.0, v3.2.1, or v3.2.2, otherwise it will cause metadata loss. You must downgrade the cluster to v3.2.3 or later to prevent the issue. * After upgrading StarRocks to v3.3.9, you can only downgrade it to v3.2.11 or later. #### 3.3.22[​](#3322 "Direct link to 3.3.22") Release Date: January 27, 2026 ##### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") The following issues have been fixed: * CVE-2025-27818. [#67335](https://github.com/StarRocks/starrocks/pull/67335) * SIGSEGV crash in CN when querying non-partitioned Iceberg tables with DATE/TIME predicates on ARM64/Graviton architectures. [#66864](https://github.com/StarRocks/starrocks/pull/66864) * Deadlock caused by lock ordering issues when closing `LocalTabletsChannel` and `LakeTabletsChannel`. [#66748](https://github.com/StarRocks/starrocks/pull/66748) * Potential BE crash when executing `CACHE SELECT` queries with filter conditions. [#67375](https://github.com/StarRocks/starrocks/pull/67375) * Issue where the Multicast Sink Operator could get stuck in the `OUTPUT_FULL` state if an upstream operator (for example, LIMIT) finished early, causing the query to hang. [#67153](https://github.com/StarRocks/starrocks/pull/67153) * Potential Segfault caused by failure to invalidate cache pointers when `ObjectColumn` is resized or moved. [#66957](https://github.com/StarRocks/starrocks/pull/66957) * Potential errors or OOM issues when Java UDFs handle Nullable columns containing all NULLs. [#67025](https://github.com/StarRocks/starrocks/pull/67025) * BE crash caused by the optimization logic of Ranking Window Functions when `PARTITION BY` and `ORDER BY` are missing. [#67081](https://github.com/StarRocks/starrocks/pull/67081) * Incorrect result issue where `COUNT(DISTINCT)` was not correctly rewritten to `multi_distinct_count` when queried alongside non-distinct aggregations (like SUM) on a single-bucket table. [#66767](https://github.com/StarRocks/starrocks/pull/66767) * Incorrect results when `regexp_replace` processes multiple rows with `enable_hyperscan_vec` enabled. [#67380](https://github.com/StarRocks/starrocks/pull/67380) * Potential incorrect results with Sorted Streaming Aggregate in shared-data mode. [#67376](https://github.com/StarRocks/starrocks/pull/67376) * Query failure where the optimizer generated access paths using the old column name after the column was renamed. [#67533](https://github.com/StarRocks/starrocks/pull/67533) * Incorrect Bitmap column type propagation in the rewrite rule from `bitmap_to_array` to `unnest_bitmap`. [#66855](https://github.com/StarRocks/starrocks/pull/66855) * Dependency derivation error in Low Cardinality optimization logic by adopting the Union-Find algorithm to correctly handle column relationships. [#66724](https://github.com/StarRocks/starrocks/pull/66724) * "Compute node not found" error in Short-circuit Read under shared-data clusters by adding a fallback mechanism to non-short-circuit mode. [#67323](https://github.com/StarRocks/starrocks/pull/67323) * "Version not found" error during Replication publishing caused by FE Replicas not updating the minimum readable version. [#67538](https://github.com/StarRocks/starrocks/pull/67538) * Logic error in physical partition comparison during replication transactions to ensure deterministic comparison using ID order. [#67616](https://github.com/StarRocks/starrocks/pull/67616) * Inaccurate statistics for counters like scan rows in Cloud Native Tables. [#67307](https://github.com/StarRocks/starrocks/pull/67307) * Issue where expired Tablets were not cleaned up from the scheduler, causing the scheduling queue to pile up. [#66718](https://github.com/StarRocks/starrocks/pull/66718) * Inaccurate SQL statements displayed in the Profile when multiple statements are submitted. [#67097](https://github.com/StarRocks/starrocks/pull/67097) * Issue where the transaction ID was empty in `publish_version` logs in the new FE. [#66732](https://github.com/StarRocks/starrocks/pull/66732) * Performance issue caused by unnecessary Protobuf message copying after `set_allocated`. [#67844](https://github.com/StarRocks/starrocks/pull/67844) #### 3.3.21[​](#3321 "Direct link to 3.3.21") Release Date: December 25, 2025 ##### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") The following issues have been fixed: * Logic errors when the `trim` function handles specific Unicode whitespace characters (for example, `\u1680`) and performance issues caused by reserved memory calculation. [#66428](https://github.com/StarRocks/starrocks/pull/66428) [#66477](https://github.com/StarRocks/starrocks/pull/66477) * Foreign key constraints are lost after FE restart due to table loading order. [#66474](https://github.com/StarRocks/starrocks/pull/66474) * Security vulnerabilities CVE-2025-66566 and CVE-2025-12183 in `lz4-java`, and potential crashes in upstream. [#66453](https://github.com/StarRocks/starrocks/pull/66453) [#66362](https://github.com/StarRocks/starrocks/pull/66362) [#67075](https://github.com/StarRocks/starrocks/pull/67075) * Incorrect results when Join is used with window functions in Group Execution mode. [#66441](https://github.com/StarRocks/starrocks/pull/66441) * System continues attempting to fetch metadata for a deleted warehouse, causing `SHOW LOAD` or SQL execution failures. [#66436](https://github.com/StarRocks/starrocks/pull/66436) * `PartitionColumnMinMaxRewriteRule` optimization incorrectly returns an empty set instead of NULL when the Scan input for aggregation is fully filtered. [#66356](https://github.com/StarRocks/starrocks/pull/66356) * Rowset IDs are not properly released when Rowset Commit or Compaction fails, preventing disk space reclamation. [#66301](https://github.com/StarRocks/starrocks/pull/66301) * Missing scan statistics (for example, scanned rows/bytes) in audit logs when a high-selectivity filter causes the Scan to end early (EOS). [#66280](https://github.com/StarRocks/starrocks/pull/66280) * BE continues to respond to heartbeats as Alive after entering the crash handling process (for example, SIGSEGV), causing the FE to continue dispatching queries and reporting errors [#66212](https://github.com/StarRocks/starrocks/pull/66212) * BE crash caused by multiple calls to `set_collector` of the Runtime Filter due to Local TopN pushdown optimization. [#66199](https://github.com/StarRocks/starrocks/pull/66199) * Load task failures caused by un-initialized `rssid` when column-mode Partial Update is used with Conditional Update. [#66139](https://github.com/StarRocks/starrocks/pull/66139) * Race condition when submitting drivers in Colocate Execution Group, leading to BE crashes. [#66099](https://github.com/StarRocks/starrocks/pull/66099) * `MemoryScratchSinkOperator` remains in a pending state and cannot be cancelled after `RecordBatchQueue` is closed (For example, triggered by SparkSQL Limit), causing queries to hang. [#66041](https://github.com/StarRocks/starrocks/pull/66041) * Use-after-free issue caused by a race condition when accessing `_num_pipelines` in the `ExecutionGroup` countdown logic. [#65940](https://github.com/StarRocks/starrocks/pull/65940) * Incorrect calculation logic for query error rate monitoring metrics (Internal/Analysis/Timeout error rate), resulting in negative values. [#65891](https://github.com/StarRocks/starrocks/pull/65891) * Null Pointer Exception (NPE) caused by unset `ConnectContext` when executing tasks as an LDAP user. [#65843](https://github.com/StarRocks/starrocks/pull/65843) * Performance issues where Filesystem Cache lookups for the same Key fail because object reference comparison is used instead of value comparison. [#65823](https://github.com/StarRocks/starrocks/pull/65823) * Tablet-related files are not correctly cleaned up after a snapshot load failure due to incorrect status variable checking. [#65709](https://github.com/StarRocks/starrocks/pull/65709) * Compression type and level configured in FE are not correctly propagated to BE or persisted during table creation or Schema Change, resulting in default compression settings. [#65673](https://github.com/StarRocks/starrocks/pull/65673) * `COM_STMT_EXECUTE` has no audit logs by default, and Profile information is incorrectly merged into the Prepare stage. [#65448](https://github.com/StarRocks/starrocks/pull/65448) * Delete Vector CRC32 check failures during cluster upgrade/downgrade scenarios due to version incompatibility. [#65442](https://github.com/StarRocks/starrocks/pull/65442) * BE crash caused by improper handling of Nullable properties when `UnionConstSourceOperator` merges Union to Values. [#65429](https://github.com/StarRocks/starrocks/pull/65429) * Concurrent load tasks fail during the Commit phase if the target tablet is dropped during an ALTER TABLE operation. [#65396](https://github.com/StarRocks/starrocks/pull/65396) * Inaccurate error log information when authentication fails due to incorrect context setting if users are switched in the HTTP SQL interface. [#65371](https://github.com/StarRocks/starrocks/pull/65371) * Statistics collection issues after `INSERT OVERWRITE`, including failure to collect statistics for temporary partitions and inaccurate row count statistics at transaction commit preventing collection trigger. [#65327](https://github.com/StarRocks/starrocks/pull/65327), [#65298](https://github.com/StarRocks/starrocks/pull/65298), [#65225](https://github.com/StarRocks/starrocks/pull/65225) * `HttpConnectContext` related to SQL is not released in TCP connection reuse scenarios because subsequent non-SQL HTTP requests overwrite the context. [#65203](https://github.com/StarRocks/starrocks/pull/65203) * Potential version loss caused by incomplete tablet loading when RocksDB iteration times out during BE startup metadata loading. [#65146](https://github.com/StarRocks/starrocks/pull/65146) * BE crash caused by out-of-bounds access when processing compression parameters in the `percentile_approx_weighted` function. [#64838](https://github.com/StarRocks/starrocks/pull/64838) * Missing Query Profile logs for queries forwarded from Follower to Leader. [#64395](https://github.com/StarRocks/starrocks/pull/64395) * BE crash caused by lax size checks during LZ4 encoding of large string columns during spilling. [#61495](https://github.com/StarRocks/starrocks/pull/61495) * `MERGING-EXCHANGE` operator crash caused by Ranking window function optimization generating an empty `ORDER BY` when there is no `PARTITION BY` and `GROUP BY`. [#67081](https://github.com/StarRocks/starrocks/pull/67081) * Unstable results or errors in the Low Cardinality column rewrite logic due to dependence on the Set iteration order. [#66724](https://github.com/StarRocks/starrocks/pull/66724) #### 3.3.20[​](#3320 "Direct link to 3.3.20") Release Date: November 18, 2025 ##### Bug Fixes[​](#bug-fixes-2 "Direct link to Bug Fixes") The following issues have been fixed: * CVE-2024-47561. [#64193](https://github.com/StarRocks/starrocks/pull/64193) * CVE-2025-59419. [#64142](https://github.com/StarRocks/starrocks/pull/64142) * Incorrect row count for lake Primary Key tables. [#64007](https://github.com/StarRocks/starrocks/pull/64007) * Window function with IGNORE NULLS flags can not be consolidated with its counterpart without IGNORE NULLS flag. [#63958](https://github.com/StarRocks/starrocks/pull/63958) * ASAN error in `PartitionedSpillerWriter::_remove_partition`. [#63903](https://github.com/StarRocks/starrocks/pull/63903) * Wrong results for sorted aggregation in shared-data clusters. [#63849](https://github.com/StarRocks/starrocks/pull/63849) * NPE when creating a partitioned materialized view. [#63830](https://github.com/StarRocks/starrocks/pull/63830) * Partitioned Spill crash when removing partitions. [#63825](https://github.com/StarRocks/starrocks/pull/63825) * NPE when removing expired load jobs in FE. [#63820](https://github.com/StarRocks/starrocks/pull/63820) * A potential deadlock during initialization of `ExceptionStackContext`. [#63776](https://github.com/StarRocks/starrocks/pull/63776) * Degraded scan performance caused by the profitless simplification of CASE WHEN with complex functions. [#63732](https://github.com/StarRocks/starrocks/pull/63732) * Materialized view rewrite failures caused by type mismatch. [#63659](https://github.com/StarRocks/starrocks/pull/63659) * Materialized view rewrite throws `IllegalStateException` under certain plans. [#63655](https://github.com/StarRocks/starrocks/pull/63655) * LZ4 compression and decompression errors cannot be perceived. [#63629](https://github.com/StarRocks/starrocks/pull/63629) * Stability issue caused by incorrect overflow detection when casting LARGEINT to DECIMAL128 at sign-edge cases (for example, INT128\_MIN) [#63559](https://github.com/StarRocks/starrocks/pull/63559) * `date_trunc` partition pruning with combined predicates that mistakenly produced EMPTYSET. [#63464](https://github.com/StarRocks/starrocks/pull/63464) * Incomplete `Left Join` results caused by ARRAY low-cardinality optimization. [#63419](https://github.com/StarRocks/starrocks/pull/63419) * An issue caused by the aggregate intermediate type uses `ARRAY`. [#63371](https://github.com/StarRocks/starrocks/pull/63371) * Metadata inconsistency in partial updates based on auto-increment columns. [#63370](https://github.com/StarRocks/starrocks/pull/63370) * Incompatible Bitmap index reuse for Fast Schema Evolution in shared-data clusters. [#63315](https://github.com/StarRocks/starrocks/pull/63315) * Unnecessary CN deregistration during pod restart/upgrade. [#63085](https://github.com/StarRocks/starrocks/pull/63085) * Profiles showing SQL as `omit` for returns of the PREPARE/EXECUTE statements. [#62988](https://github.com/StarRocks/starrocks/pull/62988) #### 3.3.19[​](#3319 "Direct link to 3.3.19") Release Date: October 14, 2025 ##### Bug Fixes[​](#bug-fixes-3 "Direct link to Bug Fixes") The following issues have been fixed: * `UserProperty` had lower priority than Session Variables. [#63173](https://github.com/StarRocks/starrocks/pull/63173) * Materialized view refresh failures that could occur when the Hive base table was dropped and recreated. [#63072](https://github.com/StarRocks/starrocks/pull/63072) * Issues with the aggregation pushdown rewrite rule. [#63060](https://github.com/StarRocks/starrocks/pull/63060) * Inconsistencies between null columns and data columns in Boolean extraction functions for JSON. [#63054](https://github.com/StarRocks/starrocks/pull/63054) * Issues when getting partition columns in Delta Lake format tables. [#62953](https://github.com/StarRocks/starrocks/pull/62953) * Lack of colocation support for materialized views in shared-data clusters. [#62941](https://github.com/StarRocks/starrocks/pull/62941) * Projection mapping errors in view-based materialized view rewrite. [#62918](https://github.com/StarRocks/starrocks/pull/62918) * SQL syntax errors in histogram statistics when Most Common Values (MCV) contained single quotes. [#62853](https://github.com/StarRocks/starrocks/pull/62853) * `KILL ANALYZE` did not work. [#62842](https://github.com/StarRocks/starrocks/pull/62842) * CVE-2025-58056 vulnerability. [#62801](https://github.com/StarRocks/starrocks/pull/62801) * Executing `SHOW CREATE ROUTINE LOAD` without specifying a database causes wrong results. [#62745](https://github.com/StarRocks/starrocks/pull/62745) * Data loss caused by incorrectly skipping CSV headers in `files()`. [#62719](https://github.com/StarRocks/starrocks/pull/62719) * Version check failures when Replication and Compaction transactions were committed together. [#62663](https://github.com/StarRocks/starrocks/pull/62663) * Materialized view refresh is skipped because the materialized view version map is not cleared after a failed restore job. [#62634](https://github.com/StarRocks/starrocks/pull/62634) * Issues caused by case-sensitive partition column validation in the materialized view analyzer. [#62598](https://github.com/StarRocks/starrocks/pull/62598) #### 3.3.18[​](#3318 "Direct link to 3.3.18") Release Date: August 28, 2025 ##### Bug Fixes[​](#bug-fixes-4 "Direct link to Bug Fixes") The following issues have been fixed: * BE crashes when `LakePersistentIndex` initialization failed due to cleanup of `_memtable`. [#62279](https://github.com/StarRocks/starrocks/pull/62279) * A concurrency issue caused by missing locks when retrieving the maximum Tablet version in the replication transaction manager. [#62238](https://github.com/StarRocks/starrocks/pull/62238) * A hang issue in the phased scheduler, which waited indefinitely during synchronous Profile collection (after the fix, the system correctly terminates Profile collection when scheduling errors occur). [#62140](https://github.com/StarRocks/starrocks/pull/62140) * Exception handling issues in low-cardinality optimization under the `ALLOW_THROW_EXCEPTION` mode (after the fix, exceptions in expression evaluation are properly caught and returned). [#62098](https://github.com/StarRocks/starrocks/pull/62098) * FThe system failed to compute nested CTE statistics outside of the memo during table pruning when `enable_rbo_table_prune` was set to `false`. [#62070](https://github.com/StarRocks/starrocks/pull/62070) * CVE-2025-55163 issue. [#62041](https://github.com/StarRocks/starrocks/pull/62041) * An issue where `split_morsel_queue` nested inside `partition_morsel_queue` failed to correctly receive the Tablet Schema. [#62034](https://github.com/StarRocks/starrocks/pull/62034) * Incorrect handling of `NULL` arrays during Parquet writes, which could cause data inconsistency or crashes (after the fix, the system ensures the `split` function can correctly handle `NULL` input strings). [#61999](https://github.com/StarRocks/starrocks/pull/61999) * Failure when creating materialized views using `CASE WHEN` expressions due to incompatible return types of VARCHAR (after the fix, the system ensures consistency before and after refresh). [#61996](https://github.com/StarRocks/starrocks/pull/61996) * A concurrency safety issue caused by long operations holding shard-level locks while calculating compression scores. [#61899](https://github.com/StarRocks/starrocks/pull/61899) * An incomplete table pruning issue in CBO caused by pruning logic not considering all relevant predicates. [#61881](https://github.com/StarRocks/starrocks/pull/61881) #### 3.3.17[​](#3317 "Direct link to 3.3.17") Release Date: July 30, 2025 ##### Bug Fixes[​](#bug-fixes-5 "Direct link to Bug Fixes") The following issues have been fixed: * Upgraded HttpClient5 to 5.4.3. [#61298](https://github.com/StarRocks/starrocks/pull/61298) * Incorrect `cpu_core_used_permille` limit in resource groups. [#61177](https://github.com/StarRocks/starrocks/pull/61177) * Conflict between ALTER jobs and partition creation tasks. [#61167](https://github.com/StarRocks/starrocks/pull/61167) * NPE caused by missing `globalStateMgr` in `ConnectContext`. [#60880](https://github.com/StarRocks/starrocks/pull/60880) * Partition creation failed when partition names matched case-insensitively but had different values. [#60909](https://github.com/StarRocks/starrocks/pull/60909) * Lock competition caused by synchronous access to partition statistics. [#61041](https://github.com/StarRocks/starrocks/pull/61041) * ANALYZE tasks stuck in `pending` state after FE restart. [#61113](https://github.com/StarRocks/starrocks/pull/61113) * Issue with JIT (Just-In-Time) compilation in BE. [#61060](https://github.com/StarRocks/starrocks/pull/61060) * Leader address issue in Starmgr. [#61016](https://github.com/StarRocks/starrocks/pull/61016) * CVE vulnerabilities in Broker. [#60908](https://github.com/StarRocks/starrocks/pull/60908) * Actual number of JDBC connections exceeded `jdbc_connection_pool_size` limit. [#61004](https://github.com/StarRocks/starrocks/pull/61004) * CVE-2022-41404 vulnerability. [#59689](https://github.com/StarRocks/starrocks/pull/59689) * CVEs related to Parquet and HttpClient5. [#58750](https://github.com/StarRocks/starrocks/pull/58750) * Partition not removed from `_partition_map` when physical partition ID was empty. [#60842](https://github.com/StarRocks/starrocks/pull/60842) * Missing version check in shared-data clusters. [#59422](https://github.com/StarRocks/starrocks/pull/59422) * Transaction log missing when publishing logs in batches in shared-data clusters. [#60949](https://github.com/StarRocks/starrocks/pull/60949) * Concurrent publishing of the same transaction when Batch Publish is enabled in shared-data clusters. [#57574](https://github.com/StarRocks/starrocks/pull/57574) * Statistics overwrite issue caused by lack of semi-synchronous mode. [#60897](https://github.com/StarRocks/starrocks/pull/60897) * Inaccurate `maxInstantTime` used for filtering Hudi files when retrieving latest merged file slices. [#60927](https://github.com/StarRocks/starrocks/pull/60927) * TaskRun state incompatible with earlier versions. [#60438](https://github.com/StarRocks/starrocks/pull/60438) * CVE-2025-52999 vulnerability. [#60795](https://github.com/StarRocks/starrocks/pull/60795) * Vulnerability caused by `log4j-1.2.17-cloudera6` in Broker. [#59579](https://github.com/StarRocks/starrocks/pull/59579) * BE crash when loading OOM partitions. [#60778](https://github.com/StarRocks/starrocks/pull/60778) * Base Compaction tasks blocking other compaction tasks. [#60711](https://github.com/StarRocks/starrocks/pull/60711) * Inefficient handling of error string truncation. [#60878](https://github.com/StarRocks/starrocks/pull/60878) * Materialized view rewrite failed in multi-FE environments. [#60841](https://github.com/StarRocks/starrocks/pull/60841) * INSERT OVERWRITE failed on manually created partitions. [#60750](https://github.com/StarRocks/starrocks/pull/60750) * Issue caused by using random distribution in aggregate keys. [#60702](https://github.com/StarRocks/starrocks/pull/60702) * Crash caused by low cardinality rewrite in `multi_distinct_count`. [#60664](https://github.com/StarRocks/starrocks/pull/60664) * Issue with Pivot resolving fields. [#60748](https://github.com/StarRocks/starrocks/pull/60748) * Upgraded `hudi-common` to 1.0.2. [#59501](https://github.com/StarRocks/starrocks/pull/59501) * BE crash when CLONE and DROP TABLE run concurrently. [#61359](https://github.com/StarRocks/starrocks/pull/61359) #### 3.3.16[​](#3316 "Direct link to 3.3.16") Release Date: July 4, 2025 ##### Improvements[​](#improvements "Direct link to Improvements") * Optimized error logs when creating Hive tables with duplicate names. [#60076](https://github.com/StarRocks/starrocks/pull/60076) * Added the FE parameter `slow_lock_print_stack` to prevent process stalls in large clusters when printing thread stacks. [#59967](https://github.com/StarRocks/starrocks/pull/59967) * Reduced unnecessary locks during tablet scheduling. [#59744](https://github.com/StarRocks/starrocks/pull/59744) ##### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") Fixed the following issues: * SplitOR fails to prune scan columns. [#60223](https://github.com/StarRocks/starrocks/pull/60223) * Incorrect query plan for null-aware left anti joins. [#60119](https://github.com/StarRocks/starrocks/pull/60119) * Incorrect query results when rewriting queries with materialized views due to missing NULL partitions. [#60087](https://github.com/StarRocks/starrocks/pull/60087) * Partition pruning errors when tables contain empty partitions. [#60162](https://github.com/StarRocks/starrocks/pull/60162) * Refresh errors on Iceberg external tables when using partition expressions based on `str2date`. [#60089](https://github.com/StarRocks/starrocks/pull/60089) * Unexpected behavior caused by materialized view schema changes. [#60079](https://github.com/StarRocks/starrocks/pull/60079) * Issues related to low-cardinality global dictionaries in UNION operators. [#60075](https://github.com/StarRocks/starrocks/pull/60075) * Incorrect partition ranges for temporary partitions created using the START END syntax. [#60014](https://github.com/StarRocks/starrocks/pull/60014) * Lock issues with SUBMIT TASK. [#60026](https://github.com/StarRocks/starrocks/pull/60026) * Partial updates fail on Primary Key tables under certain conditions. [#60052](https://github.com/StarRocks/starrocks/pull/60052) * Crashes caused by BE failing to create directories due to a lack of permissions to access storage paths. [#60028](https://github.com/StarRocks/starrocks/pull/60028) * Cache failures due to cache key duplication in concurrent scenarios. [#60053](https://github.com/StarRocks/starrocks/pull/60053) * Hive table metadata background refresh failure in Unified Catalog. [#55215](https://github.com/StarRocks/starrocks/pull/55215) * Query failures caused by incorrect return types of CASE WHEN. [#59972](https://github.com/StarRocks/starrocks/pull/59972) * Query failures when Delta Lake tables UNION themselves. [#60030](https://github.com/StarRocks/starrocks/pull/60030) * Partition creation failure when writing to multiple tables within the same transaction. [#59954](https://github.com/StarRocks/starrocks/pull/59954) * Queries could return empty results instead of errors when tablet versions were updated during execution. [#53060](https://github.com/StarRocks/starrocks/pull/53060) * Queries against modified columns in a table return null after upgrading to v3.4. [#59941](https://github.com/StarRocks/starrocks/pull/59941) * Authentication information is printed in logs. [#59907](https://github.com/StarRocks/starrocks/pull/59907) * Metadata refresh failures for external tables in Hive Catalog. [#54596](https://github.com/StarRocks/starrocks/pull/54596) * CACHE SELECT failures for tables after schema changes. [#59812](https://github.com/StarRocks/starrocks/pull/59812) * Broker Load could not recover after FE Leader shifts. [#59732](https://github.com/StarRocks/starrocks/pull/59732) * Stream Load failures when the target table name contains Chinese characters. [#59722](https://github.com/StarRocks/starrocks/pull/59722) * Incorrect query results in external tables due to search key hash collisions (affecting Iceberg/Delta/Paimon). [#59781](https://github.com/StarRocks/starrocks/pull/59781) #### 3.3.15[​](#3315 "Direct link to 3.3.15") Release Date: Jun 20, 2025 ##### Bug Fixes[​](#bug-fixes-7 "Direct link to Bug Fixes") Fixed the following issues: * Missing double quotes for string parameters in statistics INSERT statements. [#59713](https://github.com/StarRocks/starrocks/pull/59713) * Downgrade failure caused by Rollup tasks. [#59735](https://github.com/StarRocks/starrocks/pull/59735) * Incorrect function parameters in the result of `SHOW CREATE VIEW`. [#59714](https://github.com/StarRocks/starrocks/pull/59714) * A security issue where SQL statements with syntax errors exposed sensitive information in the Audit Log. [#59442](https://github.com/StarRocks/starrocks/pull/59442) * Error "Query version not found". [#59194](https://github.com/StarRocks/starrocks/pull/59194) * Failure to change data distribution using the `ALTER TABLE` statement. [#59360](https://github.com/StarRocks/starrocks/pull/59360) * An issue where root user processes were still visible when admin protection was enabled. [#59435](https://github.com/StarRocks/starrocks/pull/59435) * Failure of `INSERT OVERWRITE` into Hive. [#59469](https://github.com/StarRocks/starrocks/pull/59469) * Missing Tablet ID in the `max_tablet_rowset_num` log item. [#59467](https://github.com/StarRocks/starrocks/pull/59467) * An error caused by misconfigured Persistent Index parameters on a Duplicate table. [#56040](https://github.com/StarRocks/starrocks/pull/56040) * TaskRun history being archived on FE Follower nodes. [#59393](https://github.com/StarRocks/starrocks/pull/59393) * External catalog-based materialized view refresh errors. [#59369](https://github.com/StarRocks/starrocks/pull/59369) * Missing minimum version in Tablet information on shared-data clusters. [#59373](https://github.com/StarRocks/starrocks/pull/59373) * Abnormal maximum column unique ID in native tables of shared-data clusters due to version compatibility logic errors. [#59190](https://github.com/StarRocks/starrocks/pull/59190) * Materialized view refresh failure on Iceberg catalogs when the source Iceberg table is dropped and recreated, and manual refresh also fails after the materialized view is set to active. [#59287](https://github.com/StarRocks/starrocks/pull/59287) * Contamination of parameters in materialized view refresh tasks. [#59052](https://github.com/StarRocks/starrocks/pull/59052) * Data loss caused by Persistent Index when loading snapshot fails. [#59247](https://github.com/StarRocks/starrocks/pull/59247) * Issues caused when subcolumns of STRUCT appear in multiple predicates. [#59216](https://github.com/StarRocks/starrocks/pull/59216) * Query failure after renaming columns. [#59178](https://github.com/StarRocks/starrocks/pull/59178) * Loading failure due to multiple Stream Load requests. [#59181](https://github.com/StarRocks/starrocks/pull/59181) * Inability to refresh Hive table-based materialized views at the partition level in Unified Catalog. [#59139](https://github.com/StarRocks/starrocks/pull/59139) * Incorrect UNION plan causing FE out-of-memory (OOM). [#59030](https://github.com/StarRocks/starrocks/pull/59030) * Version loss during data loading. [#59006](https://github.com/StarRocks/starrocks/pull/59006) * Predicate loss when queries are rewritten to synchronous materialized views. [#58831](https://github.com/StarRocks/starrocks/pull/58831) * Issues with BITMAP/HLL/PERCENTILE data types in window functions. [#58776](https://github.com/StarRocks/starrocks/pull/58776) * Metadata changes to the external tables in Hive Catalog cannot be refreshed. [#54596](https://github.com/StarRocks/starrocks/pull/54596) ##### Behavior Changes[​](#behavior-changes "Direct link to Behavior Changes") * Introduced FE configuration parameter `task_runs_max_history_number` to control the number of historical TaskRuns retained in the `information_schema.task_runs` view, reducing memory usage. [#59161](https://github.com/StarRocks/starrocks/pull/59161) #### 3.3.14[​](#3314 "Direct link to 3.3.14") Release Date: May 14, 2025 ##### Improvements[​](#improvements-1 "Direct link to Improvements") * Optimized error messages for regex parsing failures. [#57904](https://github.com/StarRocks/starrocks/pull/57904) * Fixed security vulnerabilities [SNYK-JAVA-ORGJSON-5488379](https://security.snyk.io/vuln/SNYK-JAVA-ORGJSON-5488379) and [SNYK-JAVA-ORGJSON-5962464](https://security.snyk.io/vuln/SNYK-JAVA-ORGJSON-5962464). [#58425](https://github.com/StarRocks/starrocks/pull/58425) ##### Bug Fixes[​](#bug-fixes-8 "Direct link to Bug Fixes") Fixed the following issues: * Issues with the JSON data type in `first_value`/`last_value`/`lead`/`lag` window functions. [#58697](https://github.com/StarRocks/starrocks/pull/58697) * Deadlock caused by table-level locks from base tables during materialized view writes (after the bug fix, DB-level locks are used). [#58615](https://github.com/StarRocks/starrocks/pull/58615) * INSERT tasks hang when the target table is deleted. [#58603](https://github.com/StarRocks/starrocks/pull/58603) * Failure to change active/inactive state of materialized views with List partitions. [#58575](https://github.com/StarRocks/starrocks/pull/58575) * Incorrect `streaming_load_current_processing` metric. [#58565](https://github.com/StarRocks/starrocks/pull/58565) * Data version update errors caused by continuous loading and replica clone tasks. [#58513](https://github.com/StarRocks/starrocks/pull/58513) * Failed to refresh materialized views on external tables. [#58506](https://github.com/StarRocks/starrocks/pull/58506) * Incorrect `if()` results on ARM architecture. [#58455](https://github.com/StarRocks/starrocks/pull/58455) * Materialized view rewriting generated incorrect query plans. [#58487](https://github.com/StarRocks/starrocks/pull/58487) * Iceberg table metadata did not refresh automatically. [#58490](https://github.com/StarRocks/starrocks/pull/58490) * Incorrect query plan generated by `group_concat`. [#57908](https://github.com/StarRocks/starrocks/pull/57908) * Mass Tablet load failures caused by unhandled exceptions during loading. [#58393](https://github.com/StarRocks/starrocks/pull/58393) * Constant folding failed due to type mismatches while pruning List partitions with generated columns (after the bug fix, an implicit cast rule was added). [#54543](https://github.com/StarRocks/starrocks/pull/54543) * Mismatch between aggregate function return type and original column type (after the bug fix, the column type is `cast` to the function output type). [#58407](https://github.com/StarRocks/starrocks/pull/58407) * `broadcast_row_limit` set to 0 or below failed to prevent BROADCAST JOIN generation. [#58307](https://github.com/StarRocks/starrocks/pull/58307) * Broker Load used BE nodes that had already been blacklisted. [#58350](https://github.com/StarRocks/starrocks/pull/58350) * Asynchronous tasks persist in the background and cannot be dropped after manually cancelling materialized view refresh tasks. [#58310](https://github.com/StarRocks/starrocks/pull/58310) * Failed to create expression partitions with month or year granularity. [#58182](https://github.com/StarRocks/starrocks/pull/58182) * `ngram_search` generated invalid query plans. [#58190](https://github.com/StarRocks/starrocks/pull/58190) #### 3.3.13[​](#3313 "Direct link to 3.3.13") Release Date: April 22, 2025 ##### Improvements[​](#improvements-2 "Direct link to Improvements") * Added memory consumption metrics for queries in FE in audit logs and the QueryDetail interface. [#57731](https://github.com/StarRocks/starrocks/pull/57731) * Optimized the strategy for concurrent creation of expression partitions. [#57899](https://github.com/StarRocks/starrocks/pull/57899) * Added monitoring metrics for the number of active FE nodes. [#57857](https://github.com/StarRocks/starrocks/pull/57857) * The `information_schema.task_runs` view supports pushdown of the LIMIT clause. [#57404](https://github.com/StarRocks/starrocks/pull/57404) * Fixed several CVE issues. [#57705](https://github.com/StarRocks/starrocks/pull/57705) [#57620](https://github.com/StarRocks/starrocks/pull/57620) * Primary Key tables support retry during the PUBLISH stage, enhancing system disaster recovery capabilities. [#57354](https://github.com/StarRocks/starrocks/pull/57354) * Reduced memory consumption of Flat JSON. [#57357](https://github.com/StarRocks/starrocks/pull/57357) * The `information_schema.routine_load_jobs` view adds the `timestamp_progress` column, consistent with the SHOW ROUTINE LOAD statement return. [#57123](https://github.com/StarRocks/starrocks/pull/57123) * Disallowed unauthorized behaviors from StarRocks to LDAP. [#57131](https://github.com/StarRocks/starrocks/pull/57131) * Supports returning an error when the schema of an AVRO file does not match the schema of the Hive table. [#57296](https://github.com/StarRocks/starrocks/pull/57296) * Materialized views support the `excluded_refresh_tables` property. [#56428](https://github.com/StarRocks/starrocks/pull/56428) ##### Bug Fixes[​](#bug-fixes-9 "Direct link to Bug Fixes") Fixed the following issues: * Flat JSON does not support the `get_json_bool` function. [#58077](https://github.com/StarRocks/starrocks/pull/58077) * SHOW AUTHENTICATION statement returns the password. [#58072](https://github.com/StarRocks/starrocks/pull/58072) * The `percentile_count` function returns incorrect values. [#58038](https://github.com/StarRocks/starrocks/pull/58038) * Issues caused by spilling strategies. [#58022](https://github.com/StarRocks/starrocks/pull/58022) * After a BE is blacklisted, Stream Load still dispatches tasks to the BE, causing task failures. [#57919](https://github.com/StarRocks/starrocks/pull/57919) * Issues when using the `cast` function with semi-structured data types. [#57804](https://github.com/StarRocks/starrocks/pull/57804) * The `array_map` function returns incorrect values. [#57756](https://github.com/StarRocks/starrocks/pull/57756) * In the scenario of a single tablet, using multiple `distinct` functions on the same column with a single-column GROUP BY clause leads to incorrect query results. [#57690](https://github.com/StarRocks/starrocks/pull/57690) * MIN/MAX values in the profiles of big queries are inaccurate. [#57655](https://github.com/StarRocks/starrocks/pull/57655) * Non-partitioned materialized views based on Delta Lake data cannot rewrite queries. [#57686](https://github.com/StarRocks/starrocks/pull/57686) * A Routine Load deadlock issue. [#57430](https://github.com/StarRocks/starrocks/pull/57430) * Predicate pushdown issues with DATE/DATETIME columns. [#57576](https://github.com/StarRocks/starrocks/pull/57576) * An issue when the `percentile_disc` function has an empty input. [#57572](https://github.com/StarRocks/starrocks/pull/57572) * When modifying the bucket distribution of a table with the statement `ALTER TABLE {table} PARTITIONS (p1, p1) DISTRIBUTED BY ...`, specifying duplicate partition names could result in failure to delete internally generated temporary partitions. [#57005](https://github.com/StarRocks/starrocks/pull/57005) * ALTER TABLE MODIFY COLUMN fails with expression partitioned tables based on `str2date` function. [#57487](https://github.com/StarRocks/starrocks/pull/57487) * CACHE SELECT issue with semi-structured columns. [#57448](https://github.com/StarRocks/starrocks/pull/57448) * Upgrade compatibility issue caused by `hadoop-lib`. [#57436](https://github.com/StarRocks/starrocks/pull/57436) * Case sensitivity error issues when creating partitions. [#54867](https://github.com/StarRocks/starrocks/pull/54867) * Some columns generate incorrect sort keys during updates. [#57375](https://github.com/StarRocks/starrocks/pull/57375) * Unknown issues caused by nested window functions . [#57216](https://github.com/StarRocks/starrocks/pull/57216) #### 3.3.12[​](#3312 "Direct link to 3.3.12") Release date: April 3, 2025 ##### New Features[​](#new-features "Direct link to New Features") * Supports the `percentile_approx_weighted` function. [#56654](https://github.com/StarRocks/starrocks/pull/56654) * Supports modifying properties of Hive Catalog and Hudi Catalog. [#56212](https://github.com/StarRocks/starrocks/pull/56212) * Paimon Catalog supports manifest cache. [#55788](https://github.com/StarRocks/starrocks/pull/55788) * Supports `SHOW PARTITIONS` for tables in Paimon Catalog. [#55785](https://github.com/StarRocks/starrocks/pull/55785) * Supports statistics collection for Paimon Catalog. [#55757](https://github.com/StarRocks/starrocks/pull/55757) ##### Improvements[​](#improvements-3 "Direct link to Improvements") * Various improvements and bug fixes related to statistics. [#57147](https://github.com/StarRocks/starrocks/pull/57147) [#57238](https://github.com/StarRocks/starrocks/pull/57238) [#57170](https://github.com/StarRocks/starrocks/pull/57170) [#57154](https://github.com/StarRocks/starrocks/pull/57154) [#57124](https://github.com/StarRocks/starrocks/pull/57124) [#57047](https://github.com/StarRocks/starrocks/pull/57047) [#56956](https://github.com/StarRocks/starrocks/pull/56956) [#57031](https://github.com/StarRocks/starrocks/pull/57031) [#56904](https://github.com/StarRocks/starrocks/pull/56904) [#56950](https://github.com/StarRocks/starrocks/pull/56950) [#56671](https://github.com/StarRocks/starrocks/pull/56671) [#55922](https://github.com/StarRocks/starrocks/pull/55922) * Optimized error messages when table creation fails. [#57055](https://github.com/StarRocks/starrocks/pull/57055) * Enhanced retry mechanism for Broker Load. [#56987](https://github.com/StarRocks/starrocks/pull/56987) * Improved performance of `array_generate`. [#57252](https://github.com/StarRocks/starrocks/pull/57252) * Aborted ongoing Compaction tasks for deleted partitions. [#56943](https://github.com/StarRocks/starrocks/pull/56943) * Optimized error messages when `ALTER TABLE` fails. [#57054](https://github.com/StarRocks/starrocks/pull/57054) * Removed unnecessary reverse step from `array_agg()` to improve performance. [#56958](https://github.com/StarRocks/starrocks/pull/56958) * Added checksum verification for replicas in Primary Key tables. [#56519](https://github.com/StarRocks/starrocks/pull/56519) * Masked sensitive information in the `FILES` function output. [#56684](https://github.com/StarRocks/starrocks/pull/56684) * Reduced noisy logs related to materialized views. [#56672](https://github.com/StarRocks/starrocks/pull/56672) * Upgraded Iceberg version to 1.7.1. [#55271](https://github.com/StarRocks/starrocks/pull/55271) ##### Bug Fixes[​](#bug-fixes-10 "Direct link to Bug Fixes") * `INSERT INTO FILES` did not support CSV delimiter conversion. [#57126](https://github.com/StarRocks/starrocks/pull/57126) * Issues with Iceberg REST Catalog. [#55416](https://github.com/StarRocks/starrocks/pull/55416) * Predicate was lost during rewrite for view-based materialized views. [#57153](https://github.com/StarRocks/starrocks/pull/57153) * Paimon Catalog failed to read tables with schema changes. [#56796](https://github.com/StarRocks/starrocks/pull/56796) * Timezone conversion issue in Paimon Catalog. [#56879](https://github.com/StarRocks/starrocks/pull/56879) * `SHOW MATERIALIZED VIEWS` did not display `default_catalog` information. [#56362](https://github.com/StarRocks/starrocks/pull/56362) * In Trino dialect mode, time strings containing 'T' were not accepted. (Solution: replaced `parse_datetime` with `str_to_jodatime`.) [#56565](https://github.com/StarRocks/starrocks/pull/56565) * Incorrect result of `first_value` function. [#56467](https://github.com/StarRocks/starrocks/pull/56467) * Incorrect result of `concat_ws` function. [#56384](https://github.com/StarRocks/starrocks/pull/56384) ##### Behavior Changes[​](#behavior-changes-1 "Direct link to Behavior Changes") * Added authentication to the FE Profile interface. [#56914](https://github.com/StarRocks/starrocks/pull/56914) * Changed default value of session variable `big_query_profile_threshold` from `0` to `30`. [#56520](https://github.com/StarRocks/starrocks/pull/56520) #### 3.3.11[​](#3311 "Direct link to 3.3.11") Release date: March 7, 2025 ##### New Features[​](#new-features-1 "Direct link to New Features") * Window functions support `max_by` and `min_by`. [#54961](https://github.com/StarRocks/starrocks/pull/54961) ##### Improvements[​](#improvements-4 "Direct link to Improvements") * `Files` supports exporting JSON type data into Parquet files. [#56406](https://github.com/StarRocks/starrocks/pull/56406) * Optimized Data Cache WarmUp performance for cloud-native tables in shared-data clusters. [#56190](https://github.com/StarRocks/starrocks/pull/56190) * Supports parsing `AT TIME ZONE` expressions and the `from_iso8601_timestamp` function in Trino. [#56311](https://github.com/StarRocks/starrocks/pull/56311) [#55573](https://github.com/StarRocks/starrocks/pull/55573) * Partial Updates for Primary Key tables within shared-data clusters supports Condition Updates. [#56132](https://github.com/StarRocks/starrocks/pull/56132) * Extended support for statistics collection across all types of SQL statements. [#56257](https://github.com/StarRocks/starrocks/pull/56257) * Supports configuring the maximum number of returned rows for `SHOW PROC '/transaction'`. [#55933](https://github.com/StarRocks/starrocks/pull/55933) * Supports creating asynchronous materialized views on Oracle-type JDBC Catalog tables. [#55372](https://github.com/StarRocks/starrocks/pull/55372) * MemTracker on BE WebUI supports pagination with 25 rows per page. [#56206](https://github.com/StarRocks/starrocks/pull/56206) * Supports pushdown for subfields of complex types in table functions. [#55425](https://github.com/StarRocks/starrocks/pull/55425) * Supports LDAP login for MariaDB clients. [#55720](https://github.com/StarRocks/starrocks/pull/55720) * Upgraded Paimon version to 1.0.1. [#54796](https://github.com/StarRocks/starrocks/pull/54796) [#55760](https://github.com/StarRocks/starrocks/pull/55760) * Eliminates unnecessary `unnest` computations during query execution to reduce overhead. [#55431](https://github.com/StarRocks/starrocks/pull/55431) * Supports enabling Compaction for source clusters that are in the shared-data mode during cross-cluster synchronization. [#54787](https://github.com/StarRocks/starrocks/pull/54787) * Brings high-cost operations like DECIMAL division forward in topN computations to reduce overhead. [#55417](https://github.com/StarRocks/starrocks/pull/55417) * Optimized performance under ARM architecture. [#55072](https://github.com/StarRocks/starrocks/pull/55072) [#55510](https://github.com/StarRocks/starrocks/pull/55510) * For Hive table-based materialized views, StarRocks will perform checks and refreshes on the updated partitions only instead of full table refreshes if the base table was dropped and recreated. [#45118](https://github.com/StarRocks/starrocks/pull/45118) * DELETE operations support partition pruning. [#55400](https://github.com/StarRocks/starrocks/pull/55400) * Optimized priority strategy for collecting internal table statistics to improve efficiency when there are excessive tables. [#55446](https://github.com/StarRocks/starrocks/pull/55446) * When data loading involves multiple partitions, StarRocks merges transaction logs to improve loading performance. [#55143](https://github.com/StarRocks/starrocks/pull/55143) * Optimized error messages for SQL Translation. [#55327](https://github.com/StarRocks/starrocks/pull/55327) * Added a session variable `parallel_merge_late_materialization_mode` to control parallel merge behavior. [#55082](https://github.com/StarRocks/starrocks/pull/55082) * Optimized error messages for generated columns. [#54949](https://github.com/StarRocks/starrocks/pull/54949) * Optimized performance of `SHOW MATERIALIZED VIEWS`. [#54374](https://github.com/StarRocks/starrocks/pull/54374) ##### Bug Fixes[​](#bug-fixes-11 "Direct link to Bug Fixes") Fixed the following issues: * FE does not support casting constant TIME data types into DATETIME. [#55804](https://github.com/StarRocks/starrocks/pull/55804) * Stream Load transaction interface does not support the `starrocks_fe_table_load_rows` and `starrocks_fe_table_load_bytes` metrics. [#44991](https://github.com/StarRocks/starrocks/pull/44991) * Changes to automatic statistics collection do not take effect. [#56173](https://github.com/StarRocks/starrocks/pull/56173) * Materialized views in abnormal states caused issues with `SHOW MATERIALIZED VIEWS`. [#55995](https://github.com/StarRocks/starrocks/pull/55995) * Text-based materialized view rewrite does not work across different databases. [#56001](https://github.com/StarRocks/starrocks/pull/56001) * Metadata compatibility issues in JDBC Catalogs. [#55993](https://github.com/StarRocks/starrocks/pull/55993) * Issues of handling the JSON data type in JDBC Catalogs. [#56008](https://github.com/StarRocks/starrocks/pull/56008) * Incorrect Sort Key settings during Schema Change. [#55902](https://github.com/StarRocks/starrocks/pull/55902) * Credential information leak issue in Broker Load. [#55358](https://github.com/StarRocks/starrocks/pull/55358) * An error caused by pushing down LIMIT before predicates in CTE. [#55768](https://github.com/StarRocks/starrocks/pull/55768) * An error caused by table schema changes in Stream Load. [#55773](https://github.com/StarRocks/starrocks/pull/55773) * A privilege issue due to the execution plan of DELETE statements containing SELECT. [#55695](https://github.com/StarRocks/starrocks/pull/55695) * An issue caused by not aborting compaction tasks when shutting down CN. [#55503](https://github.com/StarRocks/starrocks/pull/55503) * Follower FE nodes unable to fetch updated loading statistics. [#55758](https://github.com/StarRocks/starrocks/pull/55758) * Incorrect capacity statistics for the spill directory. [#55703](https://github.com/StarRocks/starrocks/pull/55703) * Failed to create materialized views due to lack of sufficient partition checks for base tables with list partitions. [#55673](https://github.com/StarRocks/starrocks/pull/55673) * An issue caused by missing metadata locks in ALTER TABLE. [#55605](https://github.com/StarRocks/starrocks/pull/55605) * An error in `SHOW CREATE TABLE` caused by constraints. [#55592](https://github.com/StarRocks/starrocks/pull/55592) * OOM due to large ARRAY in Nestloop Join. [#55603](https://github.com/StarRocks/starrocks/pull/55603) * Lock issue with DROP PARTITION. [#55549](https://github.com/StarRocks/starrocks/pull/55549) * An issue with min/max window functions due to not supporting string types. [#55537](https://github.com/StarRocks/starrocks/pull/55537) * Parser performance degraded. [#54830](https://github.com/StarRocks/starrocks/pull/54830) * Column name case sensitivity issue during partial updates. [#55442](https://github.com/StarRocks/starrocks/pull/55442) * Import failures when Stream Load is scheduled on nodes with an "Alive" state of false. [#55371](https://github.com/StarRocks/starrocks/pull/55371) * Incorrect output column order in materialized views containing ORDER BY. [#55355](https://github.com/StarRocks/starrocks/pull/55355) * BE crashes due to disk failure. [#55042](https://github.com/StarRocks/starrocks/pull/55042) * Incorrect query results caused by Query Cache. [#55287](https://github.com/StarRocks/starrocks/pull/55287) * Parquet Writer fails to convert time zone when writing TIMESTAMP type with time zones. [#55194](https://github.com/StarRocks/starrocks/pull/55194) * Loading tasks hang due to ALTER job timeout. [#55207](https://github.com/StarRocks/starrocks/pull/55207) * An error caused by `date_format` function when input is in milliseconds. [#54854](https://github.com/StarRocks/starrocks/pull/54854) * Materialized view rewrite failure caused by Partition Key being of DATE type. [#54804](https://github.com/StarRocks/starrocks/pull/54804) ##### Behavior Changes[​](#behavior-changes-2 "Direct link to Behavior Changes") * Added authentication to the `query_detail` interface in FE. [#55919](https://github.com/StarRocks/starrocks/pull/55919) * The UUID type in Iceberg now maps to BINARY. [#54978](https://github.com/StarRocks/starrocks/pull/54978) * Uses changed row count instead of the visible time of partitions to determine if statistics need to be recollected. [#55373](https://github.com/StarRocks/starrocks/pull/55373) #### 3.3.10 (Yanked)[​](#3310-yanked "Direct link to 3.3.10 (Yanked)") Release date: February 21, 2025 tip This version has been taken offline due to metadata loss issues in **shared-data clusters**. * **Problem**: When there are committed compaction transactions that are not yet been published during a shift of Leader FE node in a shared-data cluster, metadata loss may occur after the shift. * **Impact scope**: This problem only affects shared-data clusters. Shared-nothing clusters are unaffected. * **Temporary workaround**: When the Publish task is returned with an error, you can execute `SHOW PROC 'compactions'` to check if there are any partitions that have two compaction transactions with empty `FinishTime`. You can execute `ALTER TABLE DROP PARTITION FORCE` to drop the partitions to avoid Publish tasks getting hang. #### 3.3.9[​](#339 "Direct link to 3.3.9") Release date: January 12, 2025 ##### New Features[​](#new-features-2 "Direct link to New Features") * Supports the translation of Trino SQL into StarRocks SQL. [#54185](https://github.com/StarRocks/starrocks/pull/54185) ##### Improvements[​](#improvements-5 "Direct link to Improvements") * Corrected FE node names starting with `bdbje_reset_election_group` to enhance clarity. [#54399](https://github.com/StarRocks/starrocks/pull/54399) * Implemented vectorization for the `IF` function on ARM architectures. [#53093](https://github.com/StarRocks/starrocks/pull/53093) * `ALTER SYSTEM CREATE IMAGE` supports creating an image for StarManager. [#54370](https://github.com/StarRocks/starrocks/pull/54370) * Supports deleting cloud-native indexes of Primary Key tables in shared-data clusters. [#53971](https://github.com/StarRocks/starrocks/pull/53971) * Enforced the refresh of materialized views when the `FORCE` keyword is specified. [#52081](https://github.com/StarRocks/starrocks/pull/52081) * Supports specifying hints in `CACHE SELECT`. [#54697](https://github.com/StarRocks/starrocks/pull/54697) * Supports loading compressed CSV files using the `FILES()` function. Supported compression formats include gzip, bz2, lz4, deflate, and zstd. [#54626](https://github.com/StarRocks/starrocks/pull/54626) * Supports assigning multiple values to the same column in an `UPDATE` statement. [#54534](https://github.com/StarRocks/starrocks/pull/54534) ##### Bug Fixes[​](#bug-fixes-12 "Direct link to Bug Fixes") Fixed the following issues: * Unexpected errors when refreshing materialized views built on JDBC catalogs. [#54487](https://github.com/StarRocks/starrocks/pull/54487) * Instability in results when a Delta Lake table joins itself. [#54473](https://github.com/StarRocks/starrocks/pull/54473) * Upload retries fail when backing up data to HDFS. [#53679](https://github.com/StarRocks/starrocks/pull/53679) * BFD initialization errors on the aarch64 architecture. [#54372](https://github.com/StarRocks/starrocks/pull/54372) * Sensitive information recorded in BE logs. [#54677](https://github.com/StarRocks/starrocks/pull/54677) * Errors in Compaction-related metrics in profiles. [#54678](https://github.com/StarRocks/starrocks/pull/54678) * BE crashes caused by creating tables with nested `TIME` types. [#54601](https://github.com/StarRocks/starrocks/pull/54601) * Query plan errors for `LIMIT` queries with subquery TOP-N. [#54507](https://github.com/StarRocks/starrocks/pull/54507) ##### Downgrade notes[​](#downgrade-notes "Direct link to Downgrade notes") * Clusters can be downgraded from v3.3.9 only to v3.2.11 and later. #### 3.3.8[​](#338 "Direct link to 3.3.8") Release date: January 3, 2025 ##### Improvements[​](#improvements-6 "Direct link to Improvements") * Added a cluster idle API to assist in determining cluster status. [#53850](https://github.com/StarRocks/starrocks/pull/53850) * Included node information and histogram metrics in JSON metrics. [#53735](https://github.com/StarRocks/starrocks/pull/53735) * Optimized the MemTable for Primary Key tables in shared-data clusters. [#54178](https://github.com/StarRocks/starrocks/pull/54178) * Optimized memory usage and statistics for Primary Key tables in shared-data clusters. [#54358](https://github.com/StarRocks/starrocks/pull/54358) * Introduced a limit on the number of partitions scanned per node for queries requiring full-table or large-scale partition scans, enhancing system stability by reducing scanning pressure on individual BE or CN nodes. [#53747](https://github.com/StarRocks/starrocks/pull/53747) * Supports collecting statistics of Paimon tables. [#52858](https://github.com/StarRocks/starrocks/pull/52858) * Supports configuration of S3 client request timeout for shared-data clusters. [#54211](https://github.com/StarRocks/starrocks/pull/54211) ##### Bug Fixes[​](#bug-fixes-13 "Direct link to Bug Fixes") Fixed the following issues: * BE crashes caused by inconsistencies in the DelVec of Primary Key tables. [#53460](https://github.com/StarRocks/starrocks/pull/53460) * Issues with lock release of Primary Key tables in shared-data clusters. [#53878](https://github.com/StarRocks/starrocks/pull/53878) * Errors of UDFs nested in functions are not returned in query failures. [#44297](https://github.com/StarRocks/starrocks/pull/44297) * Transactions are blocked at the Decommission phase because they depend on the original replicas. [#49349](https://github.com/StarRocks/starrocks/pull/49349) * Queries against Delta Lake tables use relative paths instead of filenames for file retrieval. [#53949](https://github.com/StarRocks/starrocks/pull/53949) * An error is returned when querying Delta Lake Shallow Clone tables. [#54044](https://github.com/StarRocks/starrocks/pull/54044) * Case sensitivity issues when reading Paimon using JNI. [#54041](https://github.com/StarRocks/starrocks/pull/54041) * An error is returned during `INSERT OVERWRITE` operations on Hive tables created in Hive. [#53792](https://github.com/StarRocks/starrocks/pull/53792) * `SHOW TABLE STATUS` command does not validate view privileges. [#53811](https://github.com/StarRocks/starrocks/pull/53811) * Missing FE metrics. [#53058](https://github.com/StarRocks/starrocks/pull/53058) * Memory leaks in `INSERT` tasks. [#53809](https://github.com/StarRocks/starrocks/pull/53809) * Concurrency issues caused by missing write locks in replication tasks. [#54061](https://github.com/StarRocks/starrocks/pull/54061) * `partition_ttl` of tables in the `statistics` database does not take effect. [#54398](https://github.com/StarRocks/starrocks/pull/54398) * Query Cache-related issues: * Crashes when Query Cache is enabled with Group Execution. [#54363](https://github.com/StarRocks/starrocks/pull/54363) * Runtime Filter crashes. [#54305](https://github.com/StarRocks/starrocks/pull/54305) * Issues with materialized view Union Rewrite. [#54293](https://github.com/StarRocks/starrocks/pull/54293) * Missing padding in string updates for partial updates in Primary Key tables. [#54182](https://github.com/StarRocks/starrocks/pull/54182) * Incorrect execution plans for `max(count(distinct))` when low-cardinality optimization is enabled. [#53403](https://github.com/StarRocks/starrocks/pull/53403) * Issues with changing the `excluded_refresh_tables` parameter of materialized views. [#53394](https://github.com/StarRocks/starrocks/pull/53394) ##### Behavior Changes[​](#behavior-changes-3 "Direct link to Behavior Changes") * Changed the default value of `persistent_index_type` for Primary Key tables in shared-data clusters to `CLOUD_NATIVE`, that is, enabled Persistent Index by default. [#52209](https://github.com/StarRocks/starrocks/pull/52209) #### 3.3.7[​](#337 "Direct link to 3.3.7") Release date: November 29, 2024 ##### New Features[​](#new-features-3 "Direct link to New Features") * Added a new Materialized View parameter, `excluded_refresh_tables`, exclude tables that need to be refreshed. [#50926](https://github.com/StarRocks/starrocks/pull/50926) ##### Improvements[​](#improvements-7 "Direct link to Improvements") * Rewrote `unnest(bitmap_to_array)` as `unnest_bitmap` to improve performance. [#52870](https://github.com/StarRocks/starrocks/pull/52870) * Reduced the write and delete operations of Txn logs. [#42542](https://github.com/StarRocks/starrocks/pull/42542) ##### Bug Fixes[​](#bug-fixes-14 "Direct link to Bug Fixes") Fixed the following issues: * Failure to connect Power BI to external tables. [#52977](https://github.com/StarRocks/starrocks/pull/52977) * Misleading FE Thrift RPC failure messages in logs. [#52706](https://github.com/StarRocks/starrocks/pull/52706) * Routine Load tasks were canceled due to expired transactions (now tasks are canceled only if the database or table no longer exists). [#50334](https://github.com/StarRocks/starrocks/pull/50334) * Stream Load failures when submitted using HTTP 1.0. [#53010](https://github.com/StarRocks/starrocks/pull/53010) [#53008](https://github.com/StarRocks/starrocks/pull/53008) * Integer overflow of partition IDs. [#52965](https://github.com/StarRocks/starrocks/pull/52965) * Hive Text Reader failed to recognize the last empty element. [#52990](https://github.com/StarRocks/starrocks/pull/52990) * Issues caused by `array_map` in Join conditions. [#52911](https://github.com/StarRocks/starrocks/pull/52911) * Metadata cache issues under high concurrency scenarios. [#52968](https://github.com/StarRocks/starrocks/pull/52968) * The whole materialized view was refreshed when a partition was dropped from the base table. [#52740](https://github.com/StarRocks/starrocks/pull/52740) #### 3.3.6[​](#336 "Direct link to 3.3.6") Release date: November 18, 2024 ##### Improvements[​](#improvements-8 "Direct link to Improvements") * Optimized internal repair logic for Primary Key tables. [#52707](https://github.com/StarRocks/starrocks/pull/52707) * Optimized the internal implementation of histograms of statistics. [#52400](https://github.com/StarRocks/starrocks/pull/52400) * Supports adjusting log level via the FE configuration item `sys_log_warn_modules` to reduce Hudi Catalog logging. [#52709](https://github.com/StarRocks/starrocks/pull/52709) * Supports constant folding in the `yearweek` function. [#52714](https://github.com/StarRocks/starrocks/pull/52714) * Avoided push-down for Lambda functions. [#52655](https://github.com/StarRocks/starrocks/pull/52655) * Divided the Query Error metric into three: Internal Error Rate, Analysis Error Rate, and Timeout Rate. [#52646](https://github.com/StarRocks/starrocks/pull/52646) * Avoided constant expressions being extracted as common expressions within `array_map`. [#52541](https://github.com/StarRocks/starrocks/pull/52541) * Optimized the Text-based Rewrite of materialized views. [#52498](https://github.com/StarRocks/starrocks/pull/52498) ##### Bug Fixes[​](#bug-fixes-15 "Direct link to Bug Fixes") Fixed the following issues: * The `unique_constraints` and `foreign_constraints` parameters were incomplete in SHOW CREATE TABLE for cloud-native tables in shared-data clusters. [#52804](https://github.com/StarRocks/starrocks/pull/52804) * Some materialized views were activated even when `enable_mv_automatic_active_check` was set to `false`. [#52799](https://github.com/StarRocks/starrocks/pull/52799) * Memory usage is not reducing after stale memory flush. [#52613](https://github.com/StarRocks/starrocks/pull/52613) * Resource leak caused by Hudi file-system views. [#52738](https://github.com/StarRocks/starrocks/pull/52738) * Concurrent Publish and Update operations on Primary Key tables may cause issues. [#52687](https://github.com/StarRocks/starrocks/pull/52687) * Failures to terminate queries on clients. [#52185](https://github.com/StarRocks/starrocks/pull/52185) * Multi-column List partitions cannot be pushed down. [#51036](https://github.com/StarRocks/starrocks/pull/51036) * Incorrect result due to the lack of `hasnull` property in ORC files. [#52555](https://github.com/StarRocks/starrocks/pull/52555) * An issue caused by using uppercase column names in ORDER BY during table creation. [#52513](https://github.com/StarRocks/starrocks/pull/52513) * An error was returned after running `ALTER TABLE PARTITION (*) SET ("storage_cooldown_ttl" = "xxx")`. [#52482](https://github.com/StarRocks/starrocks/pull/52482) ##### Behavior Changes[​](#behavior-changes-4 "Direct link to Behavior Changes") * In earlier versions, scale-in operations would fail if there were insufficient replicas for views in the `_statistics_` database. Starting from v3.3.6, if nodes are scaled in to 3 or more, view replicas are set to 3; if there is only 1 node after the scale-in, view replicas are set to 1, allowing for successful scale-in. [#51799](https://github.com/StarRocks/starrocks/pull/51799) Affected views include: * `column_statistics` * `histogram_statistics` * `table_statistic_v1` * `external_column_statistics` * `external_histogram_statistics` * `pipe_file_list` * `loads_history` * `task_run_history` * New Primary Key tables no longer allow `__op` as a column name, even if `allow_system_reserved_names` is set to `true`. Existing tables are unaffected. [#52621](https://github.com/StarRocks/starrocks/pull/52621) * Expression-partitioned tables cannot have partition names modified. [#52557](https://github.com/StarRocks/starrocks/pull/52557) * Deprecated FE parameters `heartbeat_mgr_blocking_queue_size` and `profile_process_threads_num`. [#52236](https://github.com/StarRocks/starrocks/pull/52236) * Enabled persistent index on object storage by default for Primary Key tables in shared-data clusters. [#52209](https://github.com/StarRocks/starrocks/pull/52209) * Disallowed manual changes to bucketing methods for tables with the random bucketing method. [#52120](https://github.com/StarRocks/starrocks/pull/52120) * Backup and Restore-related parameter changes: [#52111](https://github.com/StarRocks/starrocks/pull/52111) * `make_snapshot_worker_count` supports dynamic configuration. * `release_snapshot_worker_count` supports dynamic configuration. * `upload_worker_count` supports dynamic configuration. Its default value is changed from `1` to the number of CPU cores on the machine where the BE resides. * `download_worker_count` supports dynamic configuration. Its default value is changed from `1` to the number of CPU cores on the machine where the BE resides. * The return type of `SELECT @@autocommit` has changed from BOOLEAN to BIGINT. [#51946](https://github.com/StarRocks/starrocks/pull/51946) * Added a new FE configuration item, `max_bucket_number_per_partition`, to control the maximum number of buckets per partition. [#47852](https://github.com/StarRocks/starrocks/pull/47852) * Enabled memory usage checks by default for Primary Key tables. [#52393](https://github.com/StarRocks/starrocks/pull/52393) * Optimized loading strategy to reduce loading speed when Compaction tasks cannot be completed on time. [#52269](https://github.com/StarRocks/starrocks/pull/52269) #### 3.3.5[​](#335 "Direct link to 3.3.5") Release date: October 23, 2024 ##### New Features[​](#new-features-4 "Direct link to New Features") * Supports millisecond and microsecond precision in the DATETIME type. * Resource groups support CPU hard isolation. ##### Improvements[​](#improvements-9 "Direct link to Improvements") * Optimized performance and extraction strategy for Flat JSON. [#50696](https://github.com/StarRocks/starrocks/pull/50696) * Reduced memory usage for the following ARRAY functions: * array\_contains/array\_position [#50912](https://github.com/StarRocks/starrocks/pull/50912) * array\_filter [#51363](https://github.com/StarRocks/starrocks/pull/51363) * array\_match [#51377](https://github.com/StarRocks/starrocks/pull/51377) * array\_map [#51244](https://github.com/StarRocks/starrocks/pull/51244) * Optimized error messages when loading `Null` values into List partition keys with the `Not Null` attribute. [#51086](https://github.com/StarRocks/starrocks/pull/51086) * Optimized error messages for Files() when authentication fails in the Files function. [#51697](https://github.com/StarRocks/starrocks/pull/51697) * Optimized internal statistics for `INSERT OVERWRITE`. [#50417](https://github.com/StarRocks/starrocks/pull/50417) * Shared-data clusters support garbage collection (GC) for persistent index files. [#51684](https://github.com/StarRocks/starrocks/pull/51684) * Added FE logs to help diagnose FE out-of-memory (OOM) issues. [#51528](https://github.com/StarRocks/starrocks/pull/51528) * Supports recovering metadata from the metadata directory of FE. [#51040](https://github.com/StarRocks/starrocks/pull/51040) ##### Bug Fixes[​](#bug-fixes-16 "Direct link to Bug Fixes") Fixed the following issues: * A deadlock issue caused by PIPE exceptions. [#50841](https://github.com/StarRocks/starrocks/pull/50841) * Dynamic partition creation failures block subsequent partition creation. [#51440](https://github.com/StarRocks/starrocks/pull/51440) * An error is returned for `UNION ALL` queries with `ORDER BY`. [#51647](https://github.com/StarRocks/starrocks/pull/51647) * CTE in UPDATE statements causes hints to be ignored. [#51458](https://github.com/StarRocks/starrocks/pull/51458) * The `load_finish_time` field in the system-defined view `statistics.loads_history` does not update as expected after a loading task is completed. [#51174](https://github.com/StarRocks/starrocks/pull/51174) * UDTF mishandles multibyte UTF-8 characters. [#51232](https://github.com/StarRocks/starrocks/pull/51232) ##### Behavior Changes[​](#behavior-changes-5 "Direct link to Behavior Changes") * Modified the return content of the `EXPLAIN` statement. After the change, the return content is equivalent to `EXPLAIN COST`. You can configure the level of details returned by `EXPLAIN` using the dynamic FE parameter `query_detail_explain_level`. The default value is `COSTS`, with other valid values being `NORMAL` and `VERBOSE`. [#51439](https://github.com/StarRocks/starrocks/pull/51439) #### 3.3.4[​](#334 "Direct link to 3.3.4") Release date: September 30, 2024 ##### New Features[​](#new-features-5 "Direct link to New Features") * Supports creating asynchronous materialized views on List Partition tables. [#46680](https://github.com/StarRocks/starrocks/pull/46680) [#46808](https://github.com/StarRocks/starrocks/pull/46808/files) * List Partition tables now support Nullable partition columns. [#47797](https://github.com/StarRocks/starrocks/pull/47797) * Supports viewing external file schema information using `DESC FILES()`. [#50527](https://github.com/StarRocks/starrocks/pull/50527) * Supports viewing replication task metrics via `SHOW PROC '/replications'`. [#50483](https://github.com/StarRocks/starrocks/pull/50483) ##### Improvements[​](#improvements-10 "Direct link to Improvements") * Optimized data recycling performance for `TRUNCATE TABLE` in shared-data clusters. [#49975](https://github.com/StarRocks/starrocks/pull/49975) * Supports intermediate result spilling for CTE operators. [#47982](https://github.com/StarRocks/starrocks/pull/47982) * Supports adaptive phased scheduling to alleviate OOM issues caused by complex queries. [#47868](https://github.com/StarRocks/starrocks/pull/47868) * Supports predicate pushdown for STRING-type date or datatime columns in specific scenarios. [#50643](https://github.com/StarRocks/starrocks/pull/50643) * Supports COUNT DISTINCT computation on constant semi-structured data. [#48273](https://github.com/StarRocks/starrocks/pull/48273) * Added a new FE parameter `lake_enable_balance_tablets_between_workers` to enable tablet balancing for tables in shared-date clusters. [#50843](https://github.com/StarRocks/starrocks/pull/50843) * Enhanced query rewrite capabilities for generated columns. [#50398](https://github.com/StarRocks/starrocks/pull/50398) * Partial Update now supports automatically populating columns with default values of `CURRENT_TIMESTAMP`. [#50287](https://github.com/StarRocks/starrocks/pull/50287) ##### Bug Fixes[​](#bug-fixes-17 "Direct link to Bug Fixes") Fixed the following issues: * The error "version has been compacted" caused by an infinite loop on the FE side during Tablet Clone. [#50561](https://github.com/StarRocks/starrocks/pull/50561) * ISO- formatted DATETIME types cannot be pushed down. [#49358](https://github.com/StarRocks/starrocks/pull/49358) * In concurrent scenarios, data still existed after the tablet was deleted. [#50382](https://github.com/StarRocks/starrocks/pull/50382) * Incorrect results returned by the `yearweek` function. [#51065](https://github.com/StarRocks/starrocks/pull/51065) * An issue with low cardinality dictionaries in ARRAY during CTE queries. [#51148](https://github.com/StarRocks/starrocks/pull/51148) * After FE restarts, partition TTL-related parameters were lost for materialized views. [#51028](https://github.com/StarRocks/starrocks/pull/51028) * Data loss in columns defined with `CURRENT_TIMESTAMP` after upgrading. [#50911](https://github.com/StarRocks/starrocks/pull/50911) * A stack overflow caused by the `array_distinct` function. [#51017](https://github.com/StarRocks/starrocks/pull/51017) * Activation failures for materialized views after upgrading due to changes in default field lengths. You can avoid such issues by setting `enable_active_materialized_view_schema_strict_check` to `false`. [#50869](https://github.com/StarRocks/starrocks/pull/50869) * Resource group property `cpu_weight` can be set to a negative value. [#51005](https://github.com/StarRocks/starrocks/pull/51005) * Incorrect statistics for disk capacity information. [#50669](https://github.com/StarRocks/starrocks/pull/50669) * Constant fold in the `replace` function. [#50828](https://github.com/StarRocks/starrocks/pull/50828) ##### Behavior Changes[​](#behavior-changes-6 "Direct link to Behavior Changes") * Changed the default replica number for external catalog-based materialized views from `1` to the value of the FE parameter `default_replication_num` (Default value: `3`). [#50931](https://github.com/StarRocks/starrocks/pull/50931) #### 3.3.3[​](#333 "Direct link to 3.3.3") Release date: September 5, 2024 ##### New Features[​](#new-features-6 "Direct link to New Features") * Supports user-level variables. [#48477](https://github.com/StarRocks/starrocks/pull/48477) * Supports Delta Lake Catalog metadata cache with manual and periodic refresh strategies. [#46526](https://github.com/StarRocks/starrocks/pull/46526) [#49069](https://github.com/StarRocks/starrocks/pull/49069) * Supports loading JSON types from Parquet files. [#49385](https://github.com/StarRocks/starrocks/pull/49385) * JDBC SQL Server Catalog supports queries with LIMIT. [#48248](https://github.com/StarRocks/starrocks/pull/48248) * Shared-data clusters support Partial Updates with INSERT INTO. [#49336](https://github.com/StarRocks/starrocks/pull/49336) ##### Improvements[​](#improvements-11 "Direct link to Improvements") * Optimized error messages for loading: * When memory limits are reached during loading, the IP of the corresponding BE node is returned for easier troubleshooting. [#49335](https://github.com/StarRocks/starrocks/pull/49335) * Detailed messages are provided when CSV data is loaded to target table columns that are not long enough. [#49713](https://github.com/StarRocks/starrocks/pull/49713) * Specific node information is provided when Kerberos authentication fails in Broker Load. [#46085](https://github.com/StarRocks/starrocks/pull/46085) * Optimized the partitioning mechanism during data loading to reduce memory usage in the initial stage. [#47976](https://github.com/StarRocks/starrocks/pull/47976) * Optimized memory usage for shared-nothing clusters by limiting metadata memory usage to avoid issues when there are too many Tablets or Segment files. [#49170](https://github.com/StarRocks/starrocks/pull/49170) * Optimized the performance of queries using `max(partition_column)`. [#49391](https://github.com/StarRocks/starrocks/pull/49391) * Partition pruning is used to optimize query performance when the partition column is a generated column (a column that is calculated based on a native column in the table), and the query predicate filter condition includes the native column. [#48692](https://github.com/StarRocks/starrocks/pull/48692) * Supports masking authentication information for Files() and PIPE. [#47629](https://github.com/StarRocks/starrocks/pull/47629) * Introduced a new statement `show proc '/global_current_queries'` to view queries running on all FE nodes. `show proc '/current_queries'` only shows queries running on the current FE node. [#49826](https://github.com/StarRocks/starrocks/pull/49826) ##### Bug Fixes[​](#bug-fixes-18 "Direct link to Bug Fixes") Fixed the following issues: * The source cluster's BE nodes were mistakenly added to the current cluster when exporting data to the destination cluster via StarRocks external tables. [#49323](https://github.com/StarRocks/starrocks/pull/49323) * TINYINT data type returned NULL when StarRocks reads ORC files using `select * from files` from clusters deployed on aarch64 machines. [#49517](https://github.com/StarRocks/starrocks/pull/49517) * Stream Load fails when loading JSON files containing large Integer types. [#49927](https://github.com/StarRocks/starrocks/pull/49927) * Incorrect schema is returned due to improper handling of invisible characters when users load CSV files with Files(). [#49718](https://github.com/StarRocks/starrocks/pull/49718) * An issue with temporary partition replacement in tables with multiple partition columns. [#49764](https://github.com/StarRocks/starrocks/pull/49764) ##### Behavior Changes[​](#behavior-changes-7 "Direct link to Behavior Changes") * Introduced a new parameter `object_storage_rename_file_request_timeout_ms` to better accommodate backup scenarios with cloud object storage. This parameter will be used as the backup timeout, with a default value of 30 seconds. [#49706](https://github.com/StarRocks/starrocks/pull/49706) * `to_json`, `CAST(AS MAP)`, and `STRUCT AS JSON` will return NULL instead of throwing an error by default when the conversion fails. You can allow errors by setting the system variable `sql_mode` to `ALLOW_THROW_EXCEPTION`. [#50157](https://github.com/StarRocks/starrocks/pull/50157) #### 3.3.2[​](#332 "Direct link to 3.3.2") Release date: August 8, 2024 ##### New Features[​](#new-features-7 "Direct link to New Features") * Supports renaming columns within StarRocks internal tables. [#47851](https://github.com/StarRocks/starrocks/pull/47851) * Supports reading Iceberg views. Currently, only Iceberg views created through StarRocks are supported. [#46273](https://github.com/StarRocks/starrocks/issues/46273) * \[Experimental] Supports adding and removing fields of STRUCT-type data. [#46452](https://github.com/StarRocks/starrocks/issues/46452) * Supports specifying the compression level for ZSTD compression format during table creation. [#46839](https://github.com/StarRocks/starrocks/issues/46839) * Added the following FE dynamic parameters to limit table boundaries. [#47896](https://github.com/StarRocks/starrocks/pull/47869) Including: * `auto_partition_max_creation_number_per_load` * `max_partition_number_per_table` * `max_bucket_number_per_partition` * `max_column_number_per_table` * Supports runtime optimization of table data distribution, ensuring optimization tasks do not conflict with DML operations on the table. [#43747](https://github.com/StarRocks/starrocks/pull/43747) * Added an observability interface for the global hit rate of Data Cache. [#48450](https://github.com/StarRocks/starrocks/pull/48450) * Added the SQL function array\_repeat. [#47862](https://github.com/StarRocks/starrocks/pull/47862) ##### Improvements[​](#improvements-12 "Direct link to Improvements") * Optimized the error messages for Routine Load failures due to Kafka authentication failures. [#46136](https://github.com/StarRocks/starrocks/pull/46136) [#47649](https://github.com/StarRocks/starrocks/pull/47649) * Stream Load supports using `\t` and `\n` as row and column delimiters. Users do not need to convert them to their hexadecimal ASCII codes. [#47302](https://github.com/StarRocks/starrocks/pull/47302) * Optimized the asynchronous statistics collection method for write operators, addressing the issue of increased latency when there are many import tasks. [#48162](https://github.com/StarRocks/starrocks/pull/48162) * Added the following BE dynamic parameters to control resource hard limits during loading, reducing the impact on BE stability when writing a large number of tablets. [#48495](https://github.com/StarRocks/starrocks/pull/48495) Including: * `load_process_max_memory_hard_limit_ratio` * `enable_new_load_on_memory_limit_exceeded` * Added consistency checks for Column IDs within the same table to prevent Compaction errors. [#48498](https://github.com/StarRocks/starrocks/pull/48628) * Supports persisting PIPE metadata to prevent metadata loss due to FE restarts. [#48852](https://github.com/StarRocks/starrocks/pull/48852) ##### Bug Fixes[​](#bug-fixes-19 "Direct link to Bug Fixes") Fixed the following issues: * The process could not end when creating a dictionary from an FE Follower. [#47802](https://github.com/StarRocks/starrocks/pull/47802) * Inconsistent information returned by the SHOW PARTITIONS command in shared-data clusters and shared-nothing clusters. [#48647](https://github.com/StarRocks/starrocks/pull/48647) * Data errors caused by incorrect type handling when loading data from JSON fields to `ARRAY` columns. [#48387](https://github.com/StarRocks/starrocks/pull/48387) * The `query_id` column in `information_schema.task_runs` cannot be queried. [#48876](https://github.com/StarRocks/starrocks/pull/48879) * During Backup, multiple requests for the same operation are submitted to different Brokers, causing request errors. [#48856](https://github.com/StarRocks/starrocks/pull/48856) * Downgrading to versions earlier than v3.1.11 or v3.2.4 causes Primary Key table index decompression failures, leading to query errors. [#48659](https://github.com/StarRocks/starrocks/pull/48659) ##### Downgrade Notes[​](#downgrade-notes-1 "Direct link to Downgrade Notes") If you have used the renaming column feature, you must rename the columns to their original names before downgrading your cluster to an earlier version. You can check the audit log of your cluster after upgrading to identify any `ALTER TABLE RENAME COLUMN` operations and the original names of the columns. #### 3.3.1 (Yanked)[​](#331-yanked "Direct link to 3.3.1 (Yanked)") Release date: July 18, 2024 tip This version has been taken offline due to compatibility issues in Primary Key tables. * **Problem**: After the cluster is upgraded from versions earlier than v3.1.11 and v3.2.4 to v3.3.1, index decompression failures will lead to failures of queries against Primary Key tables. * **Impact scope**: This problem only affects queries against Primary Key tables. * **Temporary workaround**: You can downgrade the cluster to v3.3.0 or earlier to avoid this issue. It will be fixed in v3.3.2. ##### New Features[​](#new-features-8 "Direct link to New Features") * \[Preview] Supports temporary tables. * \[Preview] JDBC Catalog supports Oracle and SQL Server. * \[Preview] Unified Catalog supports Kudu. * INSERT INTO on Primary Key tables supports Partial Updates by specifying the column list. * User-defined variables support the ARRAY type. [#42631](https://github.com/StarRocks/starrocks/pull/42613) * Stream Load supports converting JSON-type data and loading it into columns of STRUCT/MAP/ARRAY types. [#45406](https://github.com/StarRocks/starrocks/pull/45406) * Supports global dictionary cache. * Supports deleting partitions in batch. [#44744](https://github.com/StarRocks/starrocks/issues/44744) * Supports managing column-level permissions in Apache Ranger. (Column-level permissions for materialized views and views must be set under the table object.) [#47702](https://github.com/StarRocks/starrocks/pull/47702) * Supports Partial Updates in Column mode For Primary Key tables in shared-data clusters. [#46516](https://github.com/StarRocks/starrocks/issues/46516) * Stream Load supports data compression during transmission, reducing network bandwidth overhead. Users can specify different compression algorithms using parameters `compression` and `Content-Encoding`. Supported compression algorithms including GZIP, BZIP2, LZ4\_FRAME, and ZSTD. [#43732](https://github.com/StarRocks/starrocks/pull/43732) ##### Improvements[​](#improvements-13 "Direct link to Improvements") * Optimized the IdChain hashcode implementation to reduce the FE restart time. [#47599](https://github.com/StarRocks/starrocks/pull/47599) * Improved error messages for the `csv.trim_space` parameter in the FILES() function, checking for illegal characters and providing reasonable prompts. [#44740](https://github.com/StarRocks/starrocks/pull/44740) * Stream Load supports using `\t` and `\n` as row and column delimiters. Users do not need to convert them to their hexadecimal ASCII codes. [#47302](https://github.com/StarRocks/starrocks/pull/47302) ##### Bug Fixes[​](#bug-fixes-20 "Direct link to Bug Fixes") Fixed the following issues: * Schema Change failures due to file location changes caused by Tablet migration during the Schema Change process. [#45517](https://github.com/StarRocks/starrocks/pull/45517) * Cross-cluster Data Migration Tool fails to create tables in the target cluster due to control characters such as `\`, `\r` in the default values of fields. [#47861](https://github.com/StarRocks/starrocks/pull/47861) * Persistent bRPC failures after BE restarts. [#40229](https://github.com/StarRocks/starrocks/pull/40229) * The `user_admin` role can change the root password using the ALTER USER command. [#47801](https://github.com/StarRocks/starrocks/pull/47801) * Primary key index write failures cause data write errors. [#48045](https://github.com/StarRocks/starrocks/pull/48045) ##### Behavior Changes[​](#behavior-changes-8 "Direct link to Behavior Changes") * Intermediate result spilling is enabled by default when sinking data to Hive and Iceberg. [#47118](https://github.com/StarRocks/starrocks/pull/47118) * Changed the default value of the BE configuration item `max_cumulative_compaction_num_singleton_deltas` to `500`. [#47621](https://github.com/StarRocks/starrocks/pull/47621) * When users create a partitioned table without specifying the bucket number, if the number of partitions exceeds 5, the rule for setting the bucket count is changed to `max(2*BE or CN count, bucket number calculated based on the largest historical partition data volume)`. The previous rule was to calculate the bucket number based on the largest historical partition data volume). [#47949](https://github.com/StarRocks/starrocks/pull/47949) * Specifying a column list in the INSERT INTO statement on a Primary Key table will perform Partial Updates instead of Full Upsert in earlier versions. ##### Downgrade notes[​](#downgrade-notes-2 "Direct link to Downgrade notes") To downgrade a cluster from v3.3.1 or later to v3.2, users must clean all temporary tables in the cluster by following these steps: 1. Disallow users to create new temporary tables: ```sql ADMIN SET FRONTEND CONFIG("enable_experimental_temporary_table"="false"); ``` 2. Check if there are any temporary tables in the cluster: ```sql SELECT * FROM information_schema.temp_tables; ``` 3. If there are temporary tables in the system, clean them up using the following command (the SYSTEM-level OPERATE privilege is required): ```sql CLEAN TEMPORARY TABLE ON SESSION 'session'; ``` #### 3.3.0[​](#330 "Direct link to 3.3.0") Release date: June 21, 2024 ##### New Features and Improvements[​](#new-features-and-improvements "Direct link to New Features and Improvements") ###### Shared-data Cluster[​](#shared-data-cluster "Direct link to Shared-data Cluster") * Optimized the performance of Schema Evolution in shared-data clusters, reducing the time consumption of DDL changes to a sub-second level. For more information, see [Schema Evolution](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE/#set-fast-schema-evolution). * To satisfy the requirement for data migration from shared-nothing clusters to shared-data clusters, the community officially released the [StarRocks Data Migration Tool](https://docs.starrocks.io/docs/administration/data_migration_tool/). It can also be used for data synchronization and disaster recovery between shared-nothing clusters. * \[Preview] AWS Express One Zone Storage can be used as storage volumes, significantly improving read and write performance. For more information, see [CREATE STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME/#properties). * Optimized the garbage collection (GC) mechanism in shared-data clusters. Supports manual compaction for data in object storage. For more information, see [Manual Compaction](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE/#manual-compaction-from-31). * Optimized the Publish execution of Compaction transactions for Primary Key tables in shared-data clusters, reducing I/O and memory overhead by avoiding reading primary key indexes. * Supports Internal Parallel Scan within tablets. This optimizes query performance in scenarios where there are very few buckets in the table, which limits query parallelism to the number of tablets. Users can enable the Parallel Scan feature by setting the following system variables: ```sql SET GLOBAL enable_lake_tablet_internal_parallel = true; SET GLOBAL tablet_internal_parallel_mode = "force_split"; ``` ###### Data Lake Analytics[​](#data-lake-analytics "Direct link to Data Lake Analytics") * **Data Cache enhancements** * Added the [Data Cache Warmup](https://docs.starrocks.io/docs/data_source/data_cache_warmup/) command CACHE SELECT to fetch hotspot data from data lakes, which speeds up queries and minimizes resource usage. CACHE SELECT can work with SUBMIT TASK to achieve periodic cache warmup. This feature supports both tables in external catalogs and internal tables in shared-data clusters. * Added metrics and monitoring methods to enhance the [observability of Data Cache](https://docs.starrocks.io/docs/data_source/data_cache_observe/). * **Parquet reader performance enhancements** * Optimized Page Index, significantly reducing the data scan size. * Reduced the occurrence of reading unnecessary pages when Page Index is used. * Uses SIMD to accelerate the computation to determine whether data rows are empty. * **ORC reader performance enhancements** * Uses column ID for predicate pushdown to read ORC files after Schema Change. * Optimized the processing logic for ORC tiny stripes. * **Iceberg table format enhancements** * Significantly improved the metadata access performance of the Iceberg Catalog by refactoring the parallel Scan logic. Resolved the single-threaded I/O bottleneck in the native Iceberg SDK when handling large volumes of metadata files. As a result, queries with metadata bottlenecks now experience more than a 10-fold performance increase. * Queries on Parquet-formatted Iceberg v2 tables support [equality deletes](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_catalog/#usage-notes). * **\[Experimental] Paimon Catalog enhancements** * Materialized views created based on the Paimon external tables now support automatic query rewriting. * Optimized Scan Range scheduling for queries against the Paimon Catalog, improving I/O concurrency. * Support for querying Paimon system tables. * Paimon external tables now support DELETE Vectors, enhancing query efficiency in update and delete scenarios. * **[Enhancements in collecting external table statistics](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer/#collect-statistics-of-hiveiceberghudi-tables)** * ANALYZE TABLE can be used to collect histograms of external tables, which helps prevent data skews. * Supports collecting statistics of STRUCT subfields. * **Table sink enhancements** * The performance of the Sink operator is doubled compared to Trino. * Data can be sunk to Textfile- and ORC-formatted tables in [Hive catalogs](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog/) and storage systems such as HDFS and cloud storage like AWS S3. * \[Preview] Supports Alibaba Cloud [MaxCompute catalogs](https://docs.starrocks.io/docs/data_source/catalog/maxcompute_catalog/), with which you can query data from MaxCompute without ingestion and directly transform and load the data from MaxCompute by using INSERT INTO. * \[Experimental] Supports ClickHouse Catalog. * \[Experimental] Supports [Kudu Catalog](https://docs.starrocks.io/docs/data_source/catalog/kudu_catalog/). ###### Performance Improvement and Query Optimization[​](#performance-improvement-and-query-optimization "Direct link to Performance Improvement and Query Optimization") * **Optimized performance on ARM.** * Significantly optimized performance for ARM architecture instruction sets. Performance tests under AWS Graviton instances showed that the ARM architecture was 11% faster than the x86 architecture in the SSB 100G test, 39% faster in the Clickbench test, 13% faster in the TPC-H 100G test, and 35% faster in the TPC-DS 100G test. * **Spill to Disk is in GA.** Optimized the memory usage of complex queries and improved spill scheduling, allowing large queries to run stably without OOM. * \[Preview] Supports [spilling intermediate results to object storage](https://docs.starrocks.io/docs/administration/management/resource_management/spill_to_disk/#preview-spill-intermediate-result-to-object-storage). * **Supports more indexes.** * \[Preview] Supports [full-text inverted index](https://docs.starrocks.io/docs/table_design/indexes/inverted_index/) to accelerate full-text searches. * \[Preview] Supports [N-Gram bloom filter index](https://docs.starrocks.io/docs/table_design/indexes/Ngram_Bloom_Filter_Index/) to speed up `LIKE` queries and the computation speed of `ngram_search` and `ngram_search_case_insensitive` functions. * Improved the performance and memory usage of Bitmap functions. Added the capability to export Bitmap data to Hive by using [Hive Bitmap UDFs](https://docs.starrocks.io/docs/sql-reference/sql-functions/hive_bitmap_udf/). * **\[Preview] Supports [Flat JSON](https://docs.starrocks.io/docs/using_starrocks/Flat_json/).** This feature automatically detects JSON data during data loading, extracts common fields from the JSON data, and stores these fields in a columnar manner. This improves JSON query performance, comparable to querying STRUCT data. * **\[Preview] Optimized global dictionary.** Provides a dictionary object to store the mapping of key-value pairs from a dictionary table in the BE memory. A new `dictionary_get()` function is now used to directly query the dictionary object in the BE memory, accelerating the speed of querying the dictionary table compared to using the `dict_mapping()` function. Furthermore, the dictionary object can also serve as a dimension table. Dimension values can be obtained by directly querying the dictionary object using `dictionary_get()`, resulting in faster query speeds than the original method of performing JOIN operations on the dimension table to obtain dimension values. * \[Preview] Supports Colocate Group Execution. Significantly reduces memory usage for executing Join and Agg operators on the colocate tables, which ensures that large queries can be executed more stably. * Optimized the performance of CodeGen. JIT is enabled by default, which achieves a 5X performance improvement for complex expression calculations. * Supports using vectorization technology to implement regular expression matching, which reduces the CPU consumption of the `regexp_replace` function. * Optimized Broadcast Join so that the Broadcast Join operation can be terminated in advance when the right table is empty. * Optimized Shuffle Join in scenarios of data skew to prevent OOM. * When an aggregate query contains `Limit`, multiple Pipeline threads can share the `Limit` condition to prevent compute resource consumption. ###### Storage Optimization and Cluster Management[​](#storage-optimization-and-cluster-management "Direct link to Storage Optimization and Cluster Management") * **[Enhanced flexibility of range partitioning](https://docs.starrocks.io/docs/table_design/Data_distribution/#range-partitioning).** Three time functions can be used as partitioning columns. These functions convert timestamps or strings in the partitioning columns into date values and then the data can be partitioned based on the converted date values. * **FE memory observability.** Provides detailed memory usage metrics for each module within the FE to better manage resources. * **[Optimized metadata locks in FE](https://docs.starrocks.io/docs/administration/management/FE_configuration/#lock_manager_enabled).** Provides Lock manager to achieve centralized management for metadata locks in FE. For example, it can refine the granularity of metadata lock from the database level to the table level, which improves load and query concurrency. In a scenario of 100 concurrent load jobs on a small dataset, the load time can be reduced by 35%. * **[Supports adding labels on BEs](https://docs.starrocks.io/docs/administration/management/resource_management/be_label/).** Supports adding labels on BEs based on information such as the racks and data centers where BEs are located. It ensures even data distribution among racks and data centers, and facilitates disaster recovery in case of power failures in certain racks or faults in data centers. * **[Optimized the sort key](https://docs.starrocks.io/docs/table_design/indexes/Prefix_index_sort_key/#usage-notes).** Duplicate Key tables, Aggregate tables, and Unique Key tables all support specifying sort keys through the `ORDER BY` clause. * **\[Experimental] Optimized the storage efficiency of non-string scalar data.** This type of data supports dictionary encoding, reducing storage space usage by 12%. * **Supports size-tiered compaction for Primary Key tables.** Reduces write I/O and memory overhead during compaction. This improvement is supported in both shared-data and shared-nothing clusters. You can use the BE configuration item `enable_pk_size_tiered_compaction_strategy` to control whether to enable this feature (enabled by default). * **Optimized read I/O for persistent indexes in Primary Key tables.** Supports reading persistent indexes by a smaller granularity (page) and improves the persistent index's bloom filter. This improvement is supported in both shared-data and shared-nothing clusters. * Supports for IPv6. StarRocks now supports deployment on IPv6 networks. ###### Materialized Views[​](#materialized-views "Direct link to Materialized Views") * **Supports view-based query rewrite.** With this feature enabled, queries against views can be rewritten to materialized views created upon those views. For more information, see [View-based materialized view rewrite](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views/#view-based-materialized-view-rewrite). * **Supports text-based query rewrite.** With this feature enabled, queries (or their sub-queries) that have the same abstract syntax trees (AST) as the materialized views can be transparently rewritten. For more information, see [Text-based materialized view rewrite](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views/#text-based-materialized-view-rewrite). * **\[Preview] Supports setting transparent rewrite mode for queries directly against the materialized view.** When the `transparent_mv_rewrite_mode` property is enabled, StarRocks will automatically rewrite queries to materialized views. It will merge data from refreshed materialized view partitions with the raw data corresponding to the unrefreshed partitions using an automatic UNION operation. This mode is suitable for modeling scenarios where data consistency must be maintained while also aiming to control refresh frequency and reduce refresh costs. For more information, see [CREATE MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW/#parameters-1). * Supports aggregation pushdown for materialized view query rewrite: When the `enable_materialized_view_agg_pushdown_rewrite` variable is enabled, users can use single-table asynchronous materialized views with [Aggregation Rollup](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views/#aggregation-rollup-rewrite) to accelerate multi-table join scenarios. Aggregate functions will be pushed down to the Scan Operator during query execution and rewritten by the materialized view before the Join Operator is executed, significantly improving query efficiency. For more information, see [Aggregation pushdown](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views/#aggregation-pushdown). * **Supports a new property to control materialized view rewrite.** Users can set the `enable_query_rewrite` property to `false` to disable query rewrite based on a specific materialized view, reducing query rewrite overhead. If a materialized view is used only for direct query after modeling and not for query rewrite, users can disable query rewrite for this materialized view. For more information, see [CREATE MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW/#parameters-1). * **Optimized the cost of materialized view rewrite.** Supports specifying the number of candidate materialized views and enhanced the filter algorithms. Introduced materialized view plan cache to reduce the time consumption of the Optimizer at the query rewrite phase. For more information, see `cbo_materialized_view_rewrite_related_mvs_limit`. * **Optimized materialized views created upon Iceberg catalogs.** Materialized views based on Iceberg catalogs now support incremental refresh triggered by partition updates and partition alignment for Iceberg tables using Partition Transforms. For more information, see [Data lake query acceleration with materialized views](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views/#choose-a-suitable-refresh-strategy). * **Enhanced the observability of materialized views.** Improved the monitoring and management of materialized views for better system insights. For more information, see [Metrics for asynchronous materialized views](https://docs.starrocks.io/docs/administration/management/monitoring/metrics/#metrics-for-asynchronous-materialized-views). * **Improved the efficiency of large-scale materialized view refresh.** Supports global FIFO scheduling, optimized the cascading refresh strategy for nested materialized views, and fixed some issues that occur in high-frequency refresh scenarios. * **Supports refresh triggered by multiple fact tables.** Materialized views created upon multiple fact tables now support partition-level incremental refresh when data in any of the fact tables is updated, increasing data management flexibility. For more information, see [Align partitions with multiple base tables](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/create_partitioned_materialized_view/#align-partitions-with-multiple-base-tables). ###### SQL Functions[​](#sql-functions "Direct link to SQL Functions") * DATETIME fields support microsecond precision. The new time unit is supported in related time functions and during data loading. * Added the following functions: * [String functions](https://docs.starrocks.io/docs/category/string-1/): crc32, url\_extract\_host, ngram\_search * Array functions: [array\_contains\_seq](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_contains_seq/) * Date and time functions: [yearweek](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/yearweek/) * Math functions: [cbrt](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cbrt/) ###### Ecosystem Support[​](#ecosystem-support "Direct link to Ecosystem Support") * \[Experimental] Provides [ClickHouse SQL Rewriter](https://github.com/StarRocks/SQLTransformer), a new tool for converting the syntax in ClickHouse to the syntax in StarRocks. * The Flink connector v1.2.9 provided by StarRocks is integrated with the Flink CDC 3.0 framework, which can build a streaming ELT pipeline from CDC data sources to StarRocks. The pipeline can synchronize the entire database, sharded tables, and schema changes in the sources to StarRocks. For more information, see [Synchronize data with Flink CDC 3.0 (with schema change supported)](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks/#synchronize-data-with-flink-cdc-30-with-schema-change-supported). ##### Behavior and Parameter Changes[​](#behavior-and-parameter-changes "Direct link to Behavior and Parameter Changes") ###### Table Creation and Data Distribution[​](#table-creation-and-data-distribution "Direct link to Table Creation and Data Distribution") * Users must specify Distribution Key when creating a colocate table using CTAS. [#45537](https://github.com/StarRocks/starrocks/pull/45537) * When users create a non-partitioned table without specifying the bucket number, the minimum bucket number the system sets for the table is `16` (instead of `2` based on the formula `2*BE or CN count`). If users want to set a smaller bucket number when creating a small table, they must set it explicitly. [#47005](https://github.com/StarRocks/starrocks/pull/47005) ###### Loading and Unloading[​](#loading-and-unloading "Direct link to Loading and Unloading") * `__op` is reserved by StarRocks for special purposes and creating columns with names prefixed by `__op` is forbidden by default. You can allow this such name format by setting FE configuration `allow_system_reserved_names` to `true`. Please note that creating such columns in Primary Key tables may result in undefined behaviors. [#46239](https://github.com/StarRocks/starrocks/pull/46239) * During Routine Load jobs, if the time duration that StarRocks cannot consume data exceeds the threshold specified in the FE configuration `routine_load_unstable_threshold_second` (Default value is `3600`, that is one hour), the status of the job will become `UNSTABLE`, but the job will continue. [#36222](https://github.com/StarRocks/starrocks/pull/36222) * The default value of the FE configuration `enable_automatic_bucket` is changed from `false` to `true`. When this item is set to `true`, the system will automatically set `bucket_size` for newly created tables, thus enabling automatic bucketing, which is the optimized random bucketing feature. However, in v3.2, setting `enable_automatic_bucket` to `true` will take effect. Instead, the system only enables automatic bucketing when `bucket_size` is specified. This will prevent risks when users downgrade StarRocks from v3.3 to v3.2. ###### Query and Semi-structured Data[​](#query-and-semi-structured-data "Direct link to Query and Semi-structured Data") * When a single query is executed within the Pipeline framework, the memory limit is no longer restricted by `exec_mem_limit` but is only limited by `query_mem_limit`. A value of `0` for `query_mem_limit` indicates no limit. [#34120](https://github.com/StarRocks/starrocks/pull/34120) * NULL values in JSON is treated as SQL NULL values when they are executed by IS NULL and IS NOT NULL operators. For example, `parse_json('{"a": null}') -> 'a' IS NULL` returns `1`, and `parse_json('{"a": null}') -> 'a' IS NOT NULL` returns `0`. [#42765](https://github.com/StarRocks/starrocks/pull/42765) [#42909](https://github.com/StarRocks/starrocks/pull/42909) * A new session variable `cbo_decimal_cast_string_strict` is added to control how CBO converts data from the DECIMAL type to the STRING type. If this variable is set to `true`, the logic built in v2.5.x and later versions prevails and the system implements strict conversion (namely, the system truncates the generated string and fills 0s based on the scale length). If this variable is set to `false`, the logic built in versions earlier than v2.5.x prevails and the system processes all valid digits to generate a string. The default value is `true`. [#34208](https://github.com/StarRocks/starrocks/pull/34208) * The default value of `cbo_eq_base_type` is changed from `varchar` to `decimal`, indicating that the system will compare the DECIMAL-type data with strings as numerical values instead of strings. [#43443](https://github.com/StarRocks/starrocks/pull/43443) ###### Others[​](#others "Direct link to Others") * JDK 11 or later is required from StarRocks v3.3.0 onwards. * The default value of the materialized view property `partition_refresh_num` has been changed from `-1` to `1`. When a partitioned materialized view needs to be refreshed, instead of refreshing all partitions in a single task, the new behavior will incrementally refresh one partition at a time. This change is intended to prevent excessive resource consumption caused by the original behavior. The default behavior can be adjusted using the FE configuration `default_mv_partition_refresh_number`. * Originally, the database consistency checker was scheduled based on GMT+8 time zone. Database consistency checker is scheduled based on the local time zone now. [#45748](https://github.com/StarRocks/starrocks/issues/45748) * By default, Data Cache is enabled to accelerate data lake queries. Users can manually disable it by executing `SET enable_scan_datacache = false`. * If users want to re-use the cached data in Data Cache after downgrading a shared-data cluster from v3.3 to v3.2.8 and earlier, they need to manually rename the Blockfile in the directory **starlet\_cache** by changing the file name format from `blockfile_{n}.{version}` to `blockfile_{n}`, that is, to remove the suffix of version information. For more information, refer to the [Data Cache Usage Notes](https://docs.starrocks.io/docs/using_starrocks/caching/block_cache/#usage-notes). v3.2.9 and later versions are compatible with the file name format in v3.3, so users do not need to perform this operation manually. * Supports dynamically modifying FE parameter `sys_log_level`. [#45062](https://github.com/StarRocks/starrocks/issues/45062) * The default value of the Hive Catalog property `metastore_cache_refresh_interval_sec` is changed from `7200` (two hours) to `60` (one minute). [#46681](https://github.com/StarRocks/starrocks/pull/46681) ##### Bug Fixes[​](#bug-fixes-21 "Direct link to Bug Fixes") Fixed the following issues: * Query results are incorrect when queries are rewritten to materialized views created by using UNION ALL. [#42949](https://github.com/StarRocks/starrocks/issues/42949) * Extra columns are read when queries with predicates are rewritten to materialized views during query execution. [#45272](https://github.com/StarRocks/starrocks/issues/45272) * The results of functions `next_day` and `previous_day` are incorrect. [#45343](https://github.com/StarRocks/starrocks/issues/45343) * Schema change fails because of replica migration. [#45384](https://github.com/StarRocks/starrocks/issues/45384) * Restoring a table with full-text inverted index causes BEs to crash. [#45010](https://github.com/StarRocks/starrocks/issues/45010) * Duplicate data rows are returned when an Iceberg catalog is used to query data. [#44753](https://github.com/StarRocks/starrocks/issues/44753) * Low cardinality dictionary optimization does not take effect on `ARRAY`-type columns in Aggregate tables. [#44702](https://github.com/StarRocks/starrocks/issues/44702) * Query results are incorrect when queries are rewritten to materialized views created by using UNION ALL. [#42949](https://github.com/StarRocks/starrocks/issues/42949) * If BEs are compiled with ASAN, BEs crash when the cluster is started and the `be.warning` log shows `dict_func_expr == nullptr`. [#44551](https://github.com/StarRocks/starrocks/issues/44551) * Query results are incorrect when aggregate queries are performed on single-replica tables. [#43223](https://github.com/StarRocks/starrocks/issues/43223) * View Delta Join rewrite fails. [#43788](https://github.com/StarRocks/starrocks/issues/43788) * BEs crash after the column type is modified from VARCHAR to DECIMAL. [#44406](https://github.com/StarRocks/starrocks/issues/44406) * When a table with List partitioning is queried by using a not-equal operator, partitions are incorrectly pruned, resulting in wrong query results. [#42907](https://github.com/StarRocks/starrocks/issues/42907) * Leader FE's heap size increases quickly as many Stream Load jobs using non-transactional interface finishes. [#43715](https://github.com/StarRocks/starrocks/issues/43715) ##### Downgrade notes[​](#downgrade-notes-3 "Direct link to Downgrade notes") To downgrade a cluster from v3.3.0 or later to v3.2, users must follow these steps: 1. Ensure that all ALTER TABLE SCHEMA CHANGE transactions initiated in the v3.3 cluster are either completed or canceled before downgrading. 2. Clear all transaction history by executing the following command: ```sql ADMIN SET FRONTEND CONFIG ("history_job_keep_max_second" = "0"); ``` 3. Verify that there are no remaining historical records by running the following command: ```sql SHOW PROC '/jobs//schema_change'; ``` 4. If you want to downgrade the cluster to a patch version earlier than v3.2.8 or v3.1.14, you must drop all asynchronous materialized views you have created using `PROPERTIES('compression' = 'lz4')`. 5. Execute the following command to create an image file for your metadata: ```sql ALTER SYSTEM CREATE IMAGE; ``` 6. After the new image file is transmitted to the directory **meta/image** of all FE nodes, you can first downgrade a Follower FE node. If no error is returned, you can then downgrade other nodes in the cluster. --- ## Release 3.4 ### StarRocks version 3.4 #### 3.4.10[​](#3410 "Direct link to 3.4.10") Release Date: January 12, 2026 ##### Improvements[​](#improvements "Direct link to Improvements") * Supports pushing down GROUP BY expressions to scan operators and rewriting through materialized views, further improving query performance [#66546](https://github.com/StarRocks/starrocks/pull/66546) * Added a configuration switch for the Hudi library internal metadata table, allowing users to disable it when encountering performance issues. [#67581](https://github.com/StarRocks/starrocks/pull/67581) ##### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") The following issues have been fixed: * CVE-2025-12183 and CVE-2025-66566. [#66373](https://github.com/StarRocks/starrocks/pull/66373) [#66480](https://github.com/StarRocks/starrocks/pull/66480) * In multi-statement submission scenarios, the SQL/statement information recorded in the Profile may be incorrect, leading to unreliable troubleshooting and performance analysis data. [#67119](https://github.com/StarRocks/starrocks/pull/67119) * Java UDF/UDAF parameter conversion may take an abnormal path when the input column is "all NULL and nullable", causing Java heap memory to balloon abnormally and potentially triggering OOM. [#67105](https://github.com/StarRocks/starrocks/pull/67105) * When there is no `PARTITION BY`/`GROUP BY` and the window function is a ranking type (`row_number`/`rank`/`dense_rank`), the optimizer may generate an invalid execution plan (TOP-N with empty ORDER BY + MERGING-EXCHANGE), causing BE to crash. [#67085](https://github.com/StarRocks/starrocks/pull/67085) * After operations like resizing, deserialization, or filtering on `Object`/JSON columns, the internal pointer cache may still point to old addresses (dangling pointers), returning `nullptr` when reading object values and causing segmentation faults/data corruption. [#66990](https://github.com/StarRocks/starrocks/pull/66990) * `trim()` may trigger vector out-of-bounds operations (for example, underflow on empty slices) with specific Unicode whitespace characters/boundary inputs, causing BE internal errors or crashes. [#66484](https://github.com/StarRocks/starrocks/pull/66484) * `trim()` has improper buffer reservation length calculation, potentially causing insufficient reservation and frequent expansions, triggering exception paths, and leading to BE internal errors in severe cases. [#66489](https://github.com/StarRocks/starrocks/pull/66489) * After table value function rewrite (from `bitmap_to_array` to `unnest_bitmap`), the projection column type may be incorrectly inferred (bitmap mistaken as `ARRAY``<``BIGINT``>`), leading to type inconsistency risks in subsequent plans/execution. [#66986](https://github.com/StarRocks/starrocks/pull/66986) * When BE triggers crash handling on fatal signals (for example, SIGSEGV), the heartbeat service may still briefly return success, causing FE to mistakenly believe the BE is alive (until heartbeat timeout), potentially scheduling queries to the crashed node during this period. [#66250](https://github.com/StarRocks/starrocks/pull/66250) * Race condition exists between initial submission and dynamic driver addition (`submit_next_driver`) in Execution Groups, potentially causing "driver already blocked yet added to schedule" assertion failures and BE crashes. [#66111](https://github.com/StarRocks/starrocks/pull/66111) * When pushing down DISTINCT + LIMIT predicates, the global LIMIT may be incorrectly applied before the Exchange node, causing data to be truncated prematurely and result sets to miss some rows. [#66129](https://github.com/StarRocks/starrocks/pull/66129) * When using ExecutionGroup (group execution) mode, if JOIN is followed by window functions, data may be out of order or duplicated, leading to incorrect results. [#66458](https://github.com/StarRocks/starrocks/pull/66458) * In audit logs and query statistics, statistics like scan row count may be missing or inaccurate in some high-selectivity filtering scenarios, causing monitoring and troubleshooting information to be distorted. [#66422](https://github.com/StarRocks/starrocks/pull/66422) * When CASE-WHEN nesting is deep and each layer has many branches, the expression tree node count may explode exponentially, causing FE OOM. [#66379](https://github.com/StarRocks/starrocks/pull/66379) * The `percentile_approx_weighted` function may access the wrong parameter position when obtaining the compression factor from const parameters, causing BE crashes. [#65217](https://github.com/StarRocks/starrocks/pull/65217) * During BE startup, when loading tablet metadata, if RocksDB iteration times out, it may discard already-loaded tablets and retry from the beginning; in cross-disk migration scenarios, this may lead to version loss. [#65445](https://github.com/StarRocks/starrocks/pull/65445) [#65427](https://github.com/StarRocks/starrocks/pull/65427) * Stream Load may fail during transaction commit due to invalid tablet references (for example, deleted during ALTER). [#65986](https://github.com/StarRocks/starrocks/pull/65986) * When rowset COMMIT or Compaction COMMIT fails on Primary Key tables, the rowset ID is not released, causing files to not be garbage-collected and disk space to leak. [#66336](https://github.com/StarRocks/starrocks/pull/66336) * DELETE statements during partition pruning may attempt materialized view rewrite preparation, potentially blocking or failing DELETE due to table lock order or deadlock issues. [#65818](https://github.com/StarRocks/starrocks/pull/65818) [#65820](https://github.com/StarRocks/starrocks/pull/65820) * When the same table is referenced multiple times in a query, scan nodes may concurrently allocate partition IDs, potentially causing ID conflicts and partition mapping confusion, leading to incorrect query results. [#65608](https://github.com/StarRocks/starrocks/pull/65608) * When column-mode partial update is used together with conditional update, loading may fail with "invalid rssid" errors. [#66217](https://github.com/StarRocks/starrocks/pull/66217) * When concurrent transactions create temporary partitions with the same partition values but different transaction IDs, "Duplicate values" errors may cause automatic partition creation to fail. [#66203](https://github.com/StarRocks/starrocks/pull/66203) [#66398](https://github.com/StarRocks/starrocks/pull/66398) * Clone task checks the wrong status variable during cleanup after `_finish_clone_primary` failure, potentially preventing cleanup logic from executing correctly. [#65765](https://github.com/StarRocks/starrocks/pull/65765) * When DROP and CLONE tasks execute concurrently on the same tablet, the only replica may be deleted by DROP, causing query failures. [#66271](https://github.com/StarRocks/starrocks/pull/66271) * In Spilling scenarios, very large string encoding may cause BE crashes due to buffer reservation errors or type overflows. [#65373](https://github.com/StarRocks/starrocks/pull/65373) * When using CACHE SELECT functionality, if column iterators are not properly sought or the schema is incorrectly reordered, assertion failures or data confusion may occur, causing BE crashes. [#66276](https://github.com/StarRocks/starrocks/pull/66276) * External table (file format schema detection) sampling scans may experience range index out-of-bounds, causing BE crashes or reading incorrect data. [#65931](https://github.com/StarRocks/starrocks/pull/65931) * When materialized views are based on VIEW + JOIN scenarios and the view name matches a base table name (but in different databases), partition expression parsing may fail, causing materialized view creation errors. [#66315](https://github.com/StarRocks/starrocks/pull/66315) * When materialized views refresh on multi-level partitioned base tables, only parent partition metadata (ID/Version) is checked, not sub-partition changes, causing materialized view not to refresh after sub-partition data updates. [#66108](https://github.com/StarRocks/starrocks/pull/66108) * After the Iceberg table snapshot expiration, the partition `last_updated_at` may be null, causing materialized views depending on that table to fail to track partition changes correctly and skip refreshes. [#66044](https://github.com/StarRocks/starrocks/pull/66044) * When queries contain the same table multiple times with different partition predicates, materialized view compensation (MVCompensation) may confuse partition information, leading to incorrect rewrites. [#66416](https://github.com/StarRocks/starrocks/pull/66416) * Text-based materialized view rewrite after AST cache hits does not refresh metadata, potentially using outdated tablet information and causing query failures or data inconsistencies. [#66583](https://github.com/StarRocks/starrocks/pull/66583) * Low cardinality optimization has bugs in disabled column propagation logic, potentially causing incorrect column disabling and wrong query results. [#66771](https://github.com/StarRocks/starrocks/pull/66771) * PRIMARY KEY tables with low cardinality optimization enabled may crash or produce incorrect data due to incompatible global dictionary collection logic. [#66739](https://github.com/StarRocks/starrocks/pull/66739) * Nested CTEs with partial inlining and partial reuse scenarios may have overly strict optimizer checks, rejecting valid plans. [#66703](https://github.com/StarRocks/starrocks/pull/66703) * After merging UNION to constants (VALUES), output column nullability may be incorrectly set, causing downstream operators to crash or produce incorrect results. [#65454](https://github.com/StarRocks/starrocks/pull/65454) * Partition column min/max rewrite optimization may generate invalid TOP-N in scenarios without PARTITION BY/ORDER BY, causing BE crashes or incorrect results. [#66498](https://github.com/StarRocks/starrocks/pull/66498) * Non-deterministic functions (e.g., `now()`) are incorrectly pushed down to lower operators, potentially causing results to be inconsistent across different operators/shards. [#66391](https://github.com/StarRocks/starrocks/pull/66391) * Foreign key constraints are lost after FE restart because `MaterializedView. onCreate()` does not trigger constraint rebuilding and registration. [#66615](https://github.com/StarRocks/starrocks/pull/66615) * When materialized views contain `colocate_with` property, metadata is not written to Edit Log, causing follower FE to be unaware of colocate relationships and query performance degradation. [#65840](https://github.com/StarRocks/starrocks/pull/65840) [#65405](https://github.com/StarRocks/starrocks/pull/65405) * After a warehouse is deleted, `SHOW LOAD` or `information_schema.loads` queries may fail; original sessions cannot execute any SQL (including switching warehouses). [#66464](https://github.com/StarRocks/starrocks/pull/66464) * If the tablet statistics report is untimely, table cardinality estimation may incorrectly use sample-type statistics in some edge cases, with cardinality of 1, causing execution plans to deviate significantly. [#65655](https://github.com/StarRocks/starrocks/pull/65655) * Tablet statistics reporting timing issues may cause partition row counts to be 0, rendering table cardinality estimation completely ineffective. [#65266](https://github.com/StarRocks/starrocks/pull/65266) * FE's locking order on TransactionState during `createPartition` is opposite to Gson serialization, potentially causing deadlocks. [#65792](https://github.com/StarRocks/starrocks/pull/65792) * DELETE VECTOR CRC32 may fail validation in ABA upgrade/downgrade scenarios due to version incompatibilities, causing query errors. [#65436](https://github.com/StarRocks/starrocks/pull/65436) [#65421](https://github.com/StarRocks/starrocks/pull/65421) [#65475](https://github.com/StarRocks/starrocks/pull/65475) [#65483](https://github.com/StarRocks/starrocks/pull/65483) * `map_agg` aggregate function may trigger crashes on specific inputs. [#67460](https://github.com/StarRocks/starrocks/pull/67460) * When `flat_path` is empty, calling `substr(1)` triggers `std::out_of_range` exception, causing BE crashes. [#65386](https://github.com/StarRocks/starrocks/pull/65386) * In shared-data clusters, compression configurations do not take effect correctly during table creation and schema changes. [#65778](https://github.com/StarRocks/starrocks/pull/65778) * When adding columns with default values, concurrent INSERTs may fail due to invalid column references. [#66107](https://github.com/StarRocks/starrocks/pull/66107) [#65968](https://github.com/StarRocks/starrocks/pull/65968) * Segment iterator selection order is inconsistent during scan initialization between shared-data and shared-nothing scenarios, potentially causing inconsistent scan behavior. [#65782](https://github.com/StarRocks/starrocks/pull/65782) [#61171](https://github.com/StarRocks/starrocks/pull/61171) * merge\_condition is not supported when merge commit is enabled, causing partial update scenarios to fail. [#65278](https://github.com/StarRocks/starrocks/pull/65278) * Image journal ID retrieval logic is incorrect, potentially causing cluster snapshot functionality abnormalities. [#65989](https://github.com/StarRocks/starrocks/pull/65989) * LDAP users trigger NPE in TaskRun scenarios due to null `ConnectContext. get()`, causing task failures. [#65877](https://github.com/StarRocks/starrocks/pull/65877) * When ANALYZE statements execute on follower FE, RPC timeout still uses `query_timeout` instead of `statistic_collect_query_timeout`, potentially causing timeouts too early or too late. [#66785](https://github.com/StarRocks/starrocks/pull/66785) * `MemoryScratchSinkOperator` cannot properly finish pending\_finish after RecordBatchQueue shutdown, causing tasks to hang. [#66095](https://github.com/StarRocks/starrocks/pull/66095) * Query error rate metric calculations use the wrong variables, potentially producing negative values or inaccuracies. [#65901](https://github.com/StarRocks/starrocks/pull/65901) * Load profile counters may be updated repeatedly, causing inflated statistics. [#65352](https://github.com/StarRocks/starrocks/pull/65352) * In multi-task deployment (deploy more tasks) scenarios, the Profile collection thread context is not switched correctly, causing some metrics to be lost. [#65733](https://github.com/StarRocks/starrocks/pull/65733) * Local/Lake TabletsChannel lifecycle management has circular lock waiting risks: deadlocks may occur on specific close/deregister paths, affecting import/write task availability. [#66820](https://github.com/StarRocks/starrocks/pull/66820) * Filesystem instance cache (filesystem cache) causes query performance to drop significantly and cannot recover after capacity is set to 0 (due to cache key and instance mismatch). [#65979](https://github.com/StarRocks/starrocks/pull/65979) #### 3.4.9[​](#349 "Direct link to 3.4.9") Release Date: November 24, 2025 ##### Behavior Changes[​](#behavior-changes "Direct link to Behavior Changes") * Changed the return type of `json_extract` in the Trino dialect from STRING to JSON. This may cause incompatibility in CAST, UNNEST, and type check logic. [#59718](https://github.com/StarRocks/starrocks/pull/59718) * The metric that reports “connections per user” under `/metrics` now requires admin authentication. Without authentication, only total connection counts are exposed, preventing information leakage of all usernames via metrics. [#64635](https://github.com/StarRocks/starrocks/pull/64635) * Removed the deprecated system variable `analyze_mv`. Materialized view refresh no longer automatically triggers ANALYZE jobs, avoiding large numbers of background statistics tasks. This changes expectations for users relying on legacy behavior. [#64863](https://github.com/StarRocks/starrocks/pull/64863) * Changed the overflow detection logic of casting from LARGEINT to DECIMAL128 on x86. `INT128_MIN * 1` is no longer considered an overflow to ensure consistent casting semantics for extreme values. [#63559](https://github.com/StarRocks/starrocks/pull/63559) * Added a configurable table-level lock timeout to `finishTransaction`. If a table lock cannot be acquired within the timeout, finishing the transaction will fail for this round and be retried later, rather than blocking indefinitely. The final result is unchanged, but the lock behavior is more explicit. [#63981](https://github.com/StarRocks/starrocks/pull/63981) ##### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") The following issues have been fixed: * During BE start, if loading tablet metadata from RocksDB times out, RocksDB may restart loading from the beginning and accidentally pick up stale tablet entries, risking data version loss. [#65146](https://github.com/StarRocks/starrocks/pull/65146) * Data corruption issues related to CRC32C checksum for delete-vectors of Lake Primary Key tables. [#65006](https://github.com/StarRocks/starrocks/pull/65006) [#65354](https://github.com/StarRocks/starrocks/pull/65354) [#65442](https://github.com/StarRocks/starrocks/pull/65442) [#65354](https://github.com/StarRocks/starrocks/pull/65354) * When the internal `flat_path` string is empty because the JSON hyper extraction path is `$` or all paths are skipped, calling `substr` will throw an exception and cause BE crash. [#65260](https://github.com/StarRocks/starrocks/pull/65260) * When spilling large strings to disk, insufficient length checks, using 32‑bit attachment sizes, and issues in the BlockReader could cause crashes. [#65373](https://github.com/StarRocks/starrocks/pull/65373) * When multiple HTTP requests reuse the same TCP connection, if a non‑ExecuteSQL request arrives after an ExecuteSQL request, the `HttpConnectContext` cannot be unregistered at channel close, causing HTTP context leaks. [#65203](https://github.com/StarRocks/starrocks/pull/65203) * Primitive value loss issue under certain circumstances when JSON data is being flattened. [#64939](https://github.com/StarRocks/starrocks/pull/64939) [#64703](https://github.com/StarRocks/starrocks/pull/64703) * Crash in `ChunkAccumulator` when chunks are appended with incompatible JSON schemas. [#64894](https://github.com/StarRocks/starrocks/pull/64894) * In `AsyncFlushOutputStream`, asynchronous I/O tasks may attempt to access a destroyed `MemTracker`, resulting in use‑after‑free crashes. [#64735](https://github.com/StarRocks/starrocks/pull/64735) * Concurrent Compaction tasks against the same Lake Primary Key table lack integrity checks, which could leave metadata in an inconsistent state after a failed publish. [#65005](https://github.com/StarRocks/starrocks/pull/65005) * When spilling Hash Joins, if the build side’s `set_finishing` task failed, it only recorded the status in the spiller, allowing the Probe side to continue, and eventually causing a crash or an indefinite loop. [#65027](https://github.com/StarRocks/starrocks/pull/65027) * During tablet migration, if the only newest replica is marked as DECOMMISSION, the version of the target replica is outdated and stuck at VERSION\_INCOMPLETE. [#62942](https://github.com/StarRocks/starrocks/pull/62942) * Use-after-free issue because the relevant Block Group is not released when `PartitionedSpillerWriter` is removing partitions. [#63903](https://github.com/StarRocks/starrocks/pull/63903) [#63825](https://github.com/StarRocks/starrocks/pull/63825) * BE crash caused by MorselQueue's failure to get splits. [#62753](https://github.com/StarRocks/starrocks/pull/62753) * In shared-data clusters, Sorted-by-key Scans with multiple I/O tasks could produce wrong results in sort-based aggregations. [#63849](https://github.com/StarRocks/starrocks/pull/63849) * On ARM, reading Parquet columns for certain Hive external tables could crash in LZ4 conversion when copying NULL bitmaps because the destination null buffer pointer was stale due to out-of-order execution. [#63294](https://github.com/StarRocks/starrocks/pull/63294) #### 3.4.8[​](#348 "Direct link to 3.4.8") Release Date: September 30, 2025 ##### Behavior Changes[​](#behavior-changes-1 "Direct link to Behavior Changes") * By setting the default value of `enable_lake_tablet_internal_parallel` to `true`, Parallel Scan for Cloud-native tables in shared-data clusters is enabled by default to increase per‑query internal parallelism. It may raise peak resource usage. [#62159](https://github.com/StarRocks/starrocks/pull/62159) ##### Bug Fixes[​](#bug-fixes-2 "Direct link to Bug Fixes") The following issues have been fixed: * Delta Lake partition column names were forcibly converted to lowercase, causing a mismatch with the actual column names. [#62953](https://github.com/StarRocks/starrocks/pull/62953) * The Iceberg manifest cache eviction race could trigger a NullPointerException (NPE). [#](https://github.com/StarRocks/starrocks/pull/63052)[#63043](https://github.com/StarRocks/starrocks/pull/63043) * Uncaught generic exceptions during the Iceberg scan phase interrupted scan range submission and produced no metrics. [#62994](https://github.com/StarRocks/starrocks/pull/62994) * Complex multi-layer projected views used in materialized view rewrite produced invalid plans or missing column statistics. [#62918](https://github.com/StarRocks/starrocks/pull/62918) [#62198](https://github.com/StarRocks/starrocks/pull/62198) * Case mismatch of partition columns in the Hive table-based materialized view was incorrectly rejected. [#62598](https://github.com/StarRocks/starrocks/pull/62598) * Materialized view refresh used only the creator’s default role, causing an insufficient privilege issue. [#62396](https://github.com/StarRocks/starrocks/pull/62396) * Case-insensitive conflicts in partition names of list-partitioned materialized views led to duplicate name errors. [#62389](https://github.com/StarRocks/starrocks/pull/62389) * Residual version mapping after failed materialized view restores caused subsequent incremental refresh to be skipped, returning empty results. [#62634](https://github.com/StarRocks/starrocks/pull/62634) * Abnormal partitions after materialized view restores caused FE restart NullPointerException. [#62563](https://github.com/StarRocks/starrocks/pull/62563) * Non-global aggregation queries incorrectly applied the aggregation pushdown rewrite, producing invalid plans. [#63060](https://github.com/StarRocks/starrocks/pull/63060) * The tablet deletion state was only updated in memory and not persisted, so GC still treated it as running and skipped reclamation. [#63623](https://github.com/StarRocks/starrocks/pull/63623) * Concurrent query and drop tablet led to early delvec cleanup and "no delete vector found" errors. [#63291](https://github.com/StarRocks/starrocks/pull/63291) * An issue with base and cumulative compaction for the Primary Key index sharing the same `max_rss_rowid`. [#63277](https://github.com/StarRocks/starrocks/pull/63277) * Possible BE crash when LakePersistentIndex destructor runs after a failed initialization. [#62279](https://github.com/StarRocks/starrocks/pull/62279) * Graceful shutdown of Publish thread pool silently discarded queued tasks without marking failures, creating version holes and a false "all succeeded" impression. [#62417](https://github.com/StarRocks/starrocks/pull/62417) * The newly cloned replica on a newly added BE during rebalance was immediately judged redundant and removed, preventing data migration to the new node. [#62542](https://github.com/StarRocks/starrocks/pull/62542) * Missing lock when reading the tablet's maximum version caused inconsistent replication transaction decisions. [#62238](https://github.com/StarRocks/starrocks/pull/62238) * A combination of `date_trunc` equality and raw column range predicate was reduced to a point interval, returning empty result sets (for example, `date_trunc('month', dt)='2025-09-01' AND dt>'2025-09-23'`). [#63464](https://github.com/StarRocks/starrocks/pull/63464) * Pushdown of non-deterministic predicates (random/time functions) produced inconsistent results. [#63495](https://github.com/StarRocks/starrocks/pull/63495) * Missing consumer node after CTE reuse decision produced incomplete execution plans. [#62784](https://github.com/StarRocks/starrocks/pull/62784) * Type mismatch crashes when table functions and low-cardinality dictionary encoding coexist. [#62466](https://github.com/StarRocks/starrocks/pull/62466) [#62292](https://github.com/StarRocks/starrocks/pull/62292) * Oversized CSV split into parallel fragments caused every fragment to skip header rows, leading to data loss. [#62719](https://github.com/StarRocks/starrocks/pull/62719) * `SHOW CREATE ROUTINE LOAD` without explicit DB returned job from another database with the same name. [#62745](https://github.com/StarRocks/starrocks/pull/62745) * NullPointerException when `sameLabelJobs` became null during concurrent load job cleanup. [#63042](https://github.com/StarRocks/starrocks/pull/63042) * BE decommission blocked even when all tablets were already in the recycle bin. [#62781](https://github.com/StarRocks/starrocks/pull/62781) * `OPTIMIZE TABLE` task stuck in PENDING after thread pool rejection. [#62300](https://github.com/StarRocks/starrocks/pull/62300) * Dirty tablet metadata cleanup used GTID arguments in the wrong order. [62275](https://github.com/StarRocks/starrocks/pull/62275) #### 3.4.7[​](#347 "Direct link to 3.4.7") Release Date: September 1, 2025 ##### Bug Fixes[​](#bug-fixes-3 "Direct link to Bug Fixes") The following issues have been fixed: * Routine Load jobs did not serialize `max_filter_ratio`. [#61755](https://github.com/StarRocks/starrocks/pull/61755) * In Stream Load, the `now(precision)` function lost the precision parameter. [#61721](https://github.com/StarRocks/starrocks/pull/61721) * In Audit Log, the Scan Rows result for `INSERT INTO SELECT` statements was inaccurate. [#61381](https://github.com/StarRocks/starrocks/pull/61381) * After upgrading the cluster to v3.4.5, the `fslib read iops` metric increased compared to before the upgrade. [#61724](https://github.com/StarRocks/starrocks/pull/61724) * Queries against SQLServer using JDBC Catalog often got stuck. [#61719](https://github.com/StarRocks/starrocks/pull/61719) #### 3.4.6[​](#346 "Direct link to 3.4.6") Release Date: August 7, 2025 ##### Improvements[​](#improvements-1 "Direct link to Improvements") * When exporting data to Parquet files using `INSERT INTO FILES`, you can now specify the Parquet version via the [`parquet.version`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md#parquetversion) property to improve compatibility with other tools when reading the exported files. [#60843](https://github.com/StarRocks/starrocks/pull/60843) ##### Bug Fixes[​](#bug-fixes-4 "Direct link to Bug Fixes") The following issues have been fixed: * Loading jobs failed due to overly coarse lock granularity in `TableMetricsManager`. [#58911](https://github.com/StarRocks/starrocks/pull/58911) * Case sensitivity issue in column names when loading Parquet data via `FILES()`. [#61059](https://github.com/StarRocks/starrocks/pull/61059) * Cache did not take effect after upgrading a shared-data cluster from v3.3 to v3.4 or later. [#60973](https://github.com/StarRocks/starrocks/pull/60973) * A division-by-zero error occurred when the partition ID was null, causing a BE crash. [#60842](https://github.com/StarRocks/starrocks/pull/60842) * Broker Load jobs failed during BE scaling. [#60224](https://github.com/StarRocks/starrocks/pull/60224) ##### Behavior Changes[​](#behavior-changes-2 "Direct link to Behavior Changes") * The `keyword` column in the `information_schema.keywords` view has been renamed to `word` to align with the MySQL definition. [#60863](https://github.com/StarRocks/starrocks/pull/60863) #### 3.4.5[​](#345 "Direct link to 3.4.5") Release Date: July 10, 2025 ##### Improvements[​](#improvements-2 "Direct link to Improvements") * Enhanced observability of loading job execution: Unified the runtime information of loading tasks into the `information_schema.loads` view. Users can view the execution details of all INSERT, Broker Load, Stream Load, and Routine Load subtasks in this view. Additional fields have been added to help users better understand the status of loading tasks and the association with parent jobs (PIPES, Routine Load Jobs). * Support modifying `kafka_broker_list` via the `ALTER ROUTINE LOAD` statement. ##### Bug Fixes[​](#bug-fixes-5 "Direct link to Bug Fixes") The following issues have been fixed: * Under high-frequency loading scenarios, Compaction could be delayed. [#59998](https://github.com/StarRocks/starrocks/pull/59998) * Querying Iceberg external tables via Unified Catalog would throw an error: `not support getting unified metadata table factory`. [#59412](https://github.com/StarRocks/starrocks/pull/59412) * When using `DESC FILES()` to view CSV files in remote storage, incorrect results were returned because the system mistakenly inferred `xinf` as the FLOAT type. [#59574](https://github.com/StarRocks/starrocks/pull/59574) * `INSERT INTO` could cause BE to crash when encountering empty partitions. [#59553](https://github.com/StarRocks/starrocks/pull/59553) * When StarRocks reads Equality Delete files in Iceberg, it could still access deleted data if the data had already been removed from the Iceberg table. [#59709](https://github.com/StarRocks/starrocks/pull/59709) * Query failures caused by renaming columns. [#59178](https://github.com/StarRocks/starrocks/pull/59178) ##### Behavior Changes[​](#behavior-changes-3 "Direct link to Behavior Changes") * The default value of the BE configuration item `skip_pk_preload` has been changed from `false` to `true`. As a result, the system will skip preloading Primary Key Indexes for Primary Key tables to reduce the likelihood of `Reached Timeout` errors. This change may increase query latency for operations that require loading Primary Key Indexes. #### 3.4.4[​](#344 "Direct link to 3.4.4") Release Date: June 10, 2025 ##### Improvements[​](#improvements-3 "Direct link to Improvements") * Storage Volume now supports ADLS2 using Managed Identity as the credential. [#58454](https://github.com/StarRocks/starrocks/pull/58454) * For [partitions based on complex time function expressions](https://docs.starrocks.io/docs/table_design/data_distribution/expression_partitioning/#partitioning-based-on-a-complex-time-function-expression-since-v34), partition pruning works well for partitions based on most DATETIME-related functions * Supports loading Avro data files from Azure using the `FILES` function. [#58131](https://github.com/StarRocks/starrocks/pull/58131) * When Routine Load encounters invalid JSON data, the consumed partition and offset information is logged in the error log to facilitate troubleshooting. [#55772](https://github.com/StarRocks/starrocks/pull/55772) ##### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") The following issues have been fixed: * Concurrent queries accessing the same partition in a partitioned table caused Hive Metastore to hang. [#58089](https://github.com/StarRocks/starrocks/pull/58089) * Abnormal termination of `INSERT` tasks caused the job to remain in the `QUEUEING` state. [#58603](https://github.com/StarRocks/starrocks/pull/58603) * After upgrading the cluster from v3.4.0 to v3.4.2, a large number of tablet replicas encounter exceptions. [#58518](https://github.com/StarRocks/starrocks/pull/58518) * FE OOM caused by incorrect `UNION` execution plans. [#59040](https://github.com/StarRocks/starrocks/pull/59040) * Invalid database IDs during partition recycling could cause FE startup to fail. [#59666](https://github.com/StarRocks/starrocks/pull/59666) * After a failed FE CheckPoint operation, the process could not exit properly, resulting in blocking. [#58602](https://github.com/StarRocks/starrocks/pull/58602) #### 3.4.3[​](#343 "Direct link to 3.4.3") Release Date: April 30, 2025 ##### Improvements[​](#improvements-4 "Direct link to Improvements") * Routine Load and Stream Load support the use of Lambda expressions in the `columns` parameter for complex column data extraction. `array_filter`/`map_filter` can be used to filter and extract ARRAY/MAP data. Complex filtering and extraction of JSON data can be achieved by combining the `cast` function to convert JSON array/JSON object to ARRAY and MAP types. For example, `COLUMNS (js, col=array_filter(i -> json_query(i, '$.type')=='t1', cast(js as Array))[1])` can extract the first JSON object from the JSON array `js` where `type` is `t1`. [#58149](https://github.com/StarRocks/starrocks/pull/58149) * Supports converting JSON objects to MAP type using the `cast` function, combined with `map_filter` to extract items from the JSON object that meet specific conditions. For example, `map_filter((k, v) -> json_query(v, '$.type') == 't1', cast(js AS MAP))` can extract the JSON object from `js` where `type` is `t1`. [#58045](https://github.com/StarRocks/starrocks/pull/58045) * LIMIT is now supported when querying the `information_schema.task_runs` view. [#57404](https://github.com/StarRocks/starrocks/pull/57404) ##### Bug Fixes[​](#bug-fixes-7 "Direct link to Bug Fixes") The following issues have been fixed: * Queries against ORC format Hive tables are returned with an error `OrcChunkReader::lazy_seek_to failed. reason = bad read in RleDecoderV2: :readByte`. [#57454](https://github.com/StarRocks/starrocks/pull/57454) * RuntimeFilter from the upper layer could not be pushed down when querying Iceberg tables that contain Equality Delete files. [#57651](https://github.com/StarRocks/starrocks/pull/57651) * Enabling the spill-to-disk pre-aggregation strategy causes queries to fail. [#58022](https://github.com/StarRocks/starrocks/pull/58022) * Queries are returned with an error `ConstantRef-cmp-ConstantRef not supported here, null != 111 should be eliminated earlier`. [#57735](https://github.com/StarRocks/starrocks/pull/57735) * Query timeout with the `query_queue_pending_timeout_second` parameter while the Query Queue feature is not enabled. [#57719](https://github.com/StarRocks/starrocks/pull/57719) #### 3.4.2[​](#342 "Direct link to 3.4.2") Release Date: April 10, 2025 ##### Improvements[​](#improvements-5 "Direct link to Improvements") * FE supports graceful shutdown to improve system availability. When exiting FE via `./stop_fe.sh -g`, FE will first return a 500 status code to the front-end Load Balancer via the `/api/health` API to indicate that it is preparing to shut down, allowing the Load Balancer to switch to other available FE nodes. Meanwhile, FE will continue to run ongoing queries until they finish or timeout (default timeout: 60 seconds). [#56823](https://github.com/StarRocks/starrocks/pull/56823) ##### Bug Fixes[​](#bug-fixes-8 "Direct link to Bug Fixes") The following issues have been fixed: * Partition pruning might not work if the partition column is a generated column. [#54543](https://github.com/StarRocks/starrocks/pull/54543) * Incorrect parameter handling in the `concat` function could cause a BE crash during query execution. [#57522](https://github.com/StarRocks/starrocks/pull/57522) * The `ssl_enable` property did not take effect when using Broker Load to load data. [#57229](https://github.com/StarRocks/starrocks/pull/57229) * When NULL values exist, querying subfields of STRUCT-type columns could cause a BE crash. [#56496](https://github.com/StarRocks/starrocks/pull/56496) * When modifying the bucket distribution of a table with the statement `ALTER TABLE {table} PARTITIONS (p1, p1) DISTRIBUTED BY ...`, specifying duplicate partition names could result in failure to delete internally generated temporary partitions. [#57005](https://github.com/StarRocks/starrocks/pull/57005) * In a shared-data cluster, running `SHOW PROC '/current_queries'` resulted in the error "Error 1064 (HY000): Sending collect query statistics request fails". [#56597](https://github.com/StarRocks/starrocks/pull/56597) * Running `INSERT OVERWRITE` loading tasks in parallel caused the error "ConcurrentModificationException: null", resulting in loading failure. [#56557](https://github.com/StarRocks/starrocks/pull/56557) * After upgrading from v2.5.21 to v3.1.17, running multiple Broker Load tasks concurrently could cause exceptions. [#56512](https://github.com/StarRocks/starrocks/pull/56512) ##### Behavior Changes[​](#behavior-changes-4 "Direct link to Behavior Changes") * The default value of the BE configuration item `avro_ignore_union_type_tag` has been changed to `true`, enabling the direct parsing of `["NULL", "STRING"]` as STRING type data, which better aligns with typical user requirements. [#57553](https://github.com/StarRocks/starrocks/pull/57553) * The default value of the session variable `big_query_profile_threshold` has been changed from 0 to 30 (seconds). [#57177](https://github.com/StarRocks/starrocks/pull/57177) * A new FE configuration item `enable_mv_refresh_collect_profile` has been added to control whether to collect Profile information during materialized view refresh. The default value is `false` (previously, the system collected Profile by default). [#56971](https://github.com/StarRocks/starrocks/pull/56971) #### 3.4.1 (Yanked)[​](#341-yanked "Direct link to 3.4.1 (Yanked)") Release Date: March 12, 2025 tip This version has been taken offline due to metadata loss issues in **shared-data clusters**. * **Problem**: When there are committed compaction transactions that are not yet been published during a shift of Leader FE node in a shared-data cluster, metadata loss may occur after the shift. * **Impact scope**: This problem only affects shared-data clusters. Shared-nothing clusters are unaffected. * **Temporary workaround**: When the Publish task is returned with an error, you can execute `SHOW PROC 'compactions'` to check if there are any partitions that have two compaction transactions with empty `FinishTime`. You can execute `ALTER TABLE DROP PARTITION FORCE` to drop the partitions to avoid Publish tasks getting hang. ##### New Features and Enhancements[​](#new-features-and-enhancements "Direct link to New Features and Enhancements") * Data lake analytics supports Deletion Vector in Delta Lake. * Supports secure views. By creating a secure view, you can prevent users without the SELECT privilege on the referenced base tables from querying the view (even if they have the SELECT privilege on the view). * Supports for Sketch HLL ([`ds_hll_count_distinct`](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/ds_hll_count_distinct/)). Compared to `approx_count_distinct`, this function provides higher-precision approximate deduplication. * Shared-data clusters support automatic snapshot creation for cluster recovery. * Storage Volume in the shared-data clusters supports Azure Data Lake Storage Gen2. * Supports SSL authentication for connections to StarRocks via the MySQL protocol, ensuring that data transmitted between the client and the StarRocks cluster cannot be read by unauthorized users. ##### Bug Fixes[​](#bug-fixes-9 "Direct link to Bug Fixes") The following issues have been fixed: * An issue where OLAP views affected the materialized view processing logic. [#52989](https://github.com/StarRocks/starrocks/pull/52989) * Write transactions would fail if one replica was not found, regardless of how many replicas had successfully committed. (After the fix, the transaction succeeds as long as the majority replicas succeed. [#55212](https://github.com/StarRocks/starrocks/pull/55212) * Stream Load fails when a node with an Alive status of false was scheduled. [#55371](https://github.com/StarRocks/starrocks/pull/55371) * Files in cluster snapshots were mistakenly deleted. [#56338](https://github.com/StarRocks/starrocks/pull/56338) ##### Behavior Changes[​](#behavior-changes-5 "Direct link to Behavior Changes") * Graceful shutdown is now enabled by default (previously it was disabled). The default value of the related BE/CN parameter `loop_count_wait_fragments_finish` has been changed to `2`, meaning that the system will wait up to 20 seconds for running queries to complete. [#56002](https://github.com/StarRocks/starrocks/pull/56002) #### 3.4.0[​](#340 "Direct link to 3.4.0") Release date: January 24, 2025 ##### Data Lake Analytics[​](#data-lake-analytics "Direct link to Data Lake Analytics") * Optimized Iceberg V2 query performance and lowered memory usage by reducing repeated reads of delete-files. * Supports column mapping for Delta Lake tables, allowing queries against data after Delta Schema Evolution. For more information, see [Delta Lake catalog - Feature support](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog/#feature-support). * Data Cache related improvements: * Introduces a Segmented LRU (SLRU) Cache eviction strategy, which significantly defends against cache pollution from occasional large queries, improves cache hit rate, and reduces fluctuations in query performance. In simulated test cases with large queries, SLRU-based query performance can be improved by 70% or even higher. For more information, see [Data Cache - Cache replacement policies](https://docs.starrocks.io/docs/data_source/data_cache/#cache-replacement-policies). * Unified the Data Cache instance used in both shared-data architecture and data lake query scenarios to simplify the configuration and improve resource utilization. For more information, see [Data Cache](https://docs.starrocks.io/docs/using_starrocks/caching/block_cache/). * Provides an adaptive I/O strategy optimization for Data Cache, which flexibly routes some query requests to remote storage based on the cache disk's load and performance, thereby enhancing overall access throughput. * Supports persistence of data in Data Cache in Data Lake query scenarios. The previously cached data can be reused after BE restarts to reduce query performance fluctuations. * Supports automatic collection of external table statistics through automatic ANALYZE tasks triggered by queries. It can provide more accurate NDV information compared to metadata files, thereby optimizing the query plan and improving query performance. For more information, see [Query-triggered collection](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer/#query-triggered-collection). * Provides Time Travel query capability for Iceberg, allowing data to be read from a specified BRANCH or TAG by specifying TIMESTAMP or VERSION. * Supports asynchronous delivery of query fragments for data lake queries. It avoids the restriction that FE must obtain all files to be queried before BE can execute a query, thus allowing FE to fetch query files and BE to execute queries in parallel, and reducing the overall latency of data lake queries involving a large number of files that are not in the cache. Meanwhile, it reduces the memory load on FE due to caching the file list and improves query stability. (Currently, the optimization for Hudi and Delta Lake is implemented, while the optimization for Iceberg is still under development.) ##### Performance Improvement and Query Optimization[​](#performance-improvement-and-query-optimization "Direct link to Performance Improvement and Query Optimization") * \[Experimental] Offers a preliminary Query Feedback feature for automatic optimization of slow queries. The system will collect the execution details of slow queries, automatically analyze its query plan for potential opportunities for optimization, and generate a tailored optimization guide for the query. If CBO generates the same bad plan for subsequent identical queries, the system will locally optimize this query plan based on the guide. For more information, see [Query Feedback](https://docs.starrocks.io/docs/using_starrocks/query_feedback/). * \[Experimental] Supports Python UDFs, offering more convenient function customization compared to Java UDFs. For more information, see [Python UDF](https://docs.starrocks.io/docs/sql-reference/sql-functions/Python_UDF/). * Enables the pushdown of multi-column OR predicates, allowing queries with multi-column OR conditions (for example, `a = xxx OR b = yyy`) to utilize certain column indexes, thus reducing data read volume and improving query performance. * Optimized TPC-DS query performance by roughly 20% under the TPC-DS 1TB Iceberg dataset. Optimization methods include table pruning and aggregated column pruning using primary and foreign keys, and aggregation pushdown. ##### Shared-data Enhancements[​](#shared-data-enhancements "Direct link to Shared-data Enhancements") * Supports Query Cache, aligning the shared-nothing architecture. * Supports synchronous materialized views, aligning the shared-nothing architecture. ##### Storage Engine[​](#storage-engine "Direct link to Storage Engine") * Unified all partitioning methods into the expression partitioning and supported multi-level partitioning, where each level can be any expression. For more information, see [Expression Partitioning](https://docs.starrocks.io/docs/table_design/data_distribution/expression_partitioning/). * \[Preview] Supports all native aggregate functions in Aggregate tables. By introducing a generic aggregate function state storage framework, all native aggregate functions supported by StarRocks can be used to define an Aggregate table. * Supports vector indexes, enabling fast approximate nearest neighbor searches (ANNS) of large-scale, high-dimensional vectors, which are commonly required in deep learning and machine learning scenarios. Currently, StarRocks supports two types of vector indexes: IVFPQ and HNSW. ##### Loading[​](#loading "Direct link to Loading") * INSERT OVERWRITE now supports a new semantic - Dynamic Overwrite. When this semantic is enabled, the ingested data will either create new partitions or overwrite existing partitions that correspond to the new data records. Partitions not involved will not be truncated or deleted. This semantic is especially useful when users want to recover data in specific partitions without specifying the partition names. For more information, see [Dynamic Overwrite](https://docs.starrocks.io/docs/loading/InsertInto/#dynamic-overwrite). * Optimized the data ingestion with INSERT from FILES to replace Broker Load as the preferred loading method: * FILES now supports listing files in remote storage, and providing basic statistics of the files. For more information, see [FILES - list\_files\_only](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files/#list_files_only). * INSERT now supports matching columns by name, which is especially useful when users load data from numerous columns with identical names. (The default behavior matches columns by their position.) For more information, see [Match column by name](https://docs.starrocks.io/docs/loading/InsertInto/#match-column-by-name). * INSERT supports specifying PROPERTIES, aligning with other loading methods. Users can specify `strict_mode`, `max_filter_ratio`, and `timeout` for INSERT operations to control and behavior and quality of the data ingestion. For more information, see [INSERT - PROPERTIES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT/#properties). * INSERT from FILES supports pushing down the target table schema check to the Scan stage of FILES to infer a more accurate source data schema. For more information, see see [Push down target table schema check](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files/#push-down-target-table-schema-check). * FILES supports unionizing files with different schema. The schema of Parquet and ORC files are unionized based on the column names, and that of CSV files are unionized based on the position (order) of the columns. When there are mismatched columns, users can choose to fill the columns with NULL or return an error by specifying the property `fill_mismatch_column_with`. For more information, see [Union files with different schema](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files/#union-files-with-different-schema). * FILES supports inferring the STRUCT type data from Parquet files. (In earlier versions, STRUCT data is inferred as STRING type.) For more information, see [Infer STRUCT type from Parquet](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files/#infer-struct-type-from-parquet). * Supports merging multiple concurrent Stream Load requests into a single transaction and committing data in a batch, thus improving the throughput of real-time data ingestion. It is designed for high concurrency, small-batch (from KB to tens of MB) real-time loading scenarios. It can reduce the excessive data versions caused by frequent loading operations, resource consumption during Compaction, and IOPS and I/O latency brought by excessive small files. ##### Others[​](#others "Direct link to Others") * Optimized the graceful exit process of BE and CN by accurately displaying the status of BE or CN nodes during a graceful exit as `SHUTDOWN`. * Optimized log printing to avoid excessive disk space being occupied. * Shared-nothing clusters now support backing up and restoring more objects: logical view, external catalog metadata, and partitions created with expression partitioning and list partitioning strategies. * \[Preview] Supports CheckPoint on Follower FE to avoid excessive memory on Leader FE during CheckPoint, thereby improving the stability of Leader FE. ##### Behavior Changes[​](#behavior-changes-6 "Direct link to Behavior Changes") * Because the Data Cache instance used in both shared-data architecture and data lake query scenarios is now unified, there will be the following behavior changes after the upgrade to v3.4.0: * BE configuration item `datacache_disk_path` is now deprecated. The data will be cached under the directory `${storage_root_path}/datacache`. If you want to allocate a dedicated disk for data cache, you can manually point the directory to the directory mentioned above using a symlink. * Cached data in the shared-data cluster will be automatically migrated to `${storage_root_path}/datacache` and can be re-used after the upgrade. * The behavior changes of `datacache_disk_size`: * When `datacache_disk_size` is `0` (Default), the automatic adjustment of cache capacity is enabled (consistent with the behavior before the upgrade). * When `datacache_disk_size` is set to a value greater than `0`, the system will pick a larger value between `datacache_disk_size` and `starlet_star_cache_disk_size_percent` as the cache capacity. * From v3.4.0 onwards, `insert_timeout` applies to operations involved INSERT (for example, UPDATE, DELETE, CTAS, materialized view refresh, statistics collection, and PIPE), replacing `query_timeout`. * From v3.4.0 onwards, the default value of `mysql_server_version` is changed to `8.0.33`. ##### Downgrade Notes[​](#downgrade-notes "Direct link to Downgrade Notes") * Clusters can be downgraded from v3.4.0 only to v3.3.9 and later. --- ## Release 3.5 ### StarRocks version 3.5 warning **Upgrade Notes** * JDK 17 or later is required from StarRocks v3.5.0 onwards. * To upgrade a cluster from v3.4 or earlier, you must upgrade the version of JDK that StarRocks depends, and remove the options that are incompatible with JDK 17 in the configuration item `JAVA_OPTS` in the FE configuration file **fe.conf**, for example, options that involve CMS and GC. The default value of `JAVA_OPTS` in the v3.5 configuration file is recommended. * For clusters using external catalogs, you need to add `--add-opens=java.base/java.util=ALL-UNNAMED` to the `JAVA_OPTS` configuration item in the BE configuration file **be.conf**. * For clusters using Java UDFs, you need to add `--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED` to the `JAVA_OPTS` configuration item in the BE configuration file **be.conf**. * In addition, as of v3.5.0, StarRocks no longer provides JVM configurations for specific JDK versions. All versions of JDK use `JAVA_OPTS`. * It is recommended to upgrade the cluster to v3.4.10 or later before upgrading it to v3.5. Otherwise, you must manually disable low cardinality optimization during the gray-scale upgrade by executing the following statement: ```sql SET GLOBAL cbo_enable_low_cardinality_optimize=false; ``` **Downgrade Notes** * After upgrading StarRocks to v3.5, DO NOT downgrade it directly to v3.4.0 ～ v3.4.5, otherwise it will cause metadata incompatibility. You must downgrade the cluster to v3.4.6 or later to prevent the issue. * After upgrading StarRocks to v3.5.2 or later, DO NOT downgrade it to v3.5.0 & v3.5.1, otherwise it will cause FE crash. #### 3.5.21[​](#3521 "Direct link to 3.5.21") Release date: August 28, 2026 ##### Behavior Changes[​](#behavior-changes "Direct link to Behavior Changes") * Reverted the 3.5.20 change that cached Iceberg REST vended-credential tables and kept their credentials fresh: on `branch-3.5`, that caching caused `INSERT OVERWRITE` transactions against a slow Iceberg REST catalog to stay `COMMITTED` but not `VISIBLE` for several seconds, because a REST metadata refresh could now run while the planner held a lock that transaction publishing also needs. The caching behavior remains in place on the 4.x line. [#77039](https://github.com/StarRocks/starrocks/pull/77039) * GIN (inverted) indexes on Primary Key tables now read from the correct segment after a column-mode partial update, instead of serving stale index data from the unmodified base segment. [#76271](https://github.com/StarRocks/starrocks/pull/76271) * Audit logs for statements forwarded to the leader FE now record the leader-resolved, fully qualified table relations (with CTEs excluded), instead of the follower's unresolved names. [#76387](https://github.com/StarRocks/starrocks/pull/76387) * `ARRAY`/`MAP` constructor expressions now reject a result whose flattened size exceeds 4 GB instead of silently wrapping around and returning corrupted values. [#76419](https://github.com/StarRocks/starrocks/pull/76419) * Iceberg REST catalogs using OAuth2 client-credentials now self-heal their background token-refresh session after a prolonged failure, instead of leaving the catalog permanently unable to refresh its access token. [#76457](https://github.com/StarRocks/starrocks/pull/76457) * External scan contexts (for example, an abandoned Spark/Flink connector read) now properly cancel their pipeline fragments when reaped as expired, instead of leaving them running. [#76535](https://github.com/StarRocks/starrocks/pull/76535) * External scan plans (Spark/Flink connector reads) now set `query_delivery_timeout`, so their `QueryContext` no longer waits indefinitely for fragments that will never arrive. [#76536](https://github.com/StarRocks/starrocks/pull/76536) * `array_difference()` on integer arrays no longer overflows in 32-bit precision before widening to `BIGINT`, fixing incorrect results when the true difference falls outside the `INT` range. [#76569](https://github.com/StarRocks/starrocks/pull/76569) * Division expressions with a non-constant divisor (for example, `10 DIV c`) are no longer treated as monotonic, fixing incorrect ZoneMap-based pruning that could return an empty or wrong result. [#76744](https://github.com/StarRocks/starrocks/pull/76744) * The configured compression codec now applies to the synthetic null/offset sub-columns of flat JSON, `ARRAY`, `MAP`, and `STRUCT` columns, instead of always writing them as uncompressed raw pages. [#76949](https://github.com/StarRocks/starrocks/pull/76949) * Load quorum selection no longer picks a replica in `DECOMMISSION` state as the load primary. [#77035](https://github.com/StarRocks/starrocks/pull/77035) * Optimizer rules that rebuild a logical window operator now preserve its `inputIsBinary` flag, keeping the binary-input merge optimization for ranking-window pre-aggregation intact. [#77058](https://github.com/StarRocks/starrocks/pull/77058) ##### Improvements[​](#improvements "Direct link to Improvements") * Materialized views no longer force a full-partition refresh when manually set from `INACTIVE` to `ACTIVE`; only the metadata version map is cleared during a schema change. [#57371](https://github.com/StarRocks/starrocks/pull/57371) * Improved error messages for large-column-capacity-limit checks by removing internal diagnostics (raw pointers, operator dumps) from the user-facing error, and fixed a typo in the shared status string. [#76303](https://github.com/StarRocks/starrocks/pull/76303) ##### Bug fixes[​](#bug-fixes "Direct link to Bug fixes") The following issues have been fixed: * Guarded against cyclic view definitions: an `ALTER VIEW` that closes a reference cycle (`v1` -> `v2` -> `v1`) is now rejected with a clear error instead of causing a later `SELECT` to recurse forever and crash with `StackOverflowError`. [#75033](https://github.com/StarRocks/starrocks/pull/75033) * Forbade pushing an aggregate down through a CASE expression that has a non-null constant ELSE branch, fixing a planning abort (`IllegalStateException`) that could occur once the rule fired. [#75037](https://github.com/StarRocks/starrocks/pull/75037) * `CTEAnchor` now prunes correctly when its child is a `ValueOperator`. [#64491](https://github.com/StarRocks/starrocks/pull/64491) * Fixed predicate conversion for Paimon: an AND-combined predicate now retains its convertible not-null branch instead of the whole conversion returning null. [#66038](https://github.com/StarRocks/starrocks/pull/66038) * The Arrow Flight prepared-statement schema no longer reports every view column as nullable regardless of its actual `NOT NULL` definition, and a related regression that produced wrong nullability for `GROUP BY ROLLUP`/`CUBE`/`GROUPING SETS` key columns has also been fixed. [#75684](https://github.com/StarRocks/starrocks/pull/75684) [#76149](https://github.com/StarRocks/starrocks/pull/76149) * Fixed two statistics bugs: an OR-predicate statistics estimate that always clamped the merged `nullsFraction` to `1` regardless of the real value, and a column-statistics cache load failure under `ERROR_IF_OVERFLOW` when a column's persisted min/max is an empty string. [#75864](https://github.com/StarRocks/starrocks/pull/75864) [#76684](https://github.com/StarRocks/starrocks/pull/76684) * Fixed a bRPC stub cache timer leak that leaked memory over time. [#75973](https://github.com/StarRocks/starrocks/pull/75973) * `UNNEST` output struct pruning now uses its input array's subfield group instead of the output's own, fixing a mismatch between the struct type BE materializes and the one FE declares. [#76002](https://github.com/StarRocks/starrocks/pull/76002) * Fixed Arrow Flight Prepared Statement forwarding sending the wrong action-type string when a request is forwarded to a different FE, which caused every ADBC client using prepared statements behind a load balancer to fail. [#76310](https://github.com/StarRocks/starrocks/pull/76310) * Hive `getTable()` now reconnects before falling back to `get_table_req()`, fixing an intermittent `out of sequence response` / `Unknown table` error when querying an Iceberg table through a Hive metastore catalog. [#76456](https://github.com/StarRocks/starrocks/pull/76456) * Made catalog-drop existence checks atomic under the write lock, fixing a check-then-act race that could persist a redundant drop record when two drops of the same catalog ran concurrently. [#76778](https://github.com/StarRocks/starrocks/pull/76778) * Fixed `PipeObservable` emitting a source event instead of a sink event on a deferred sink notification, which could leave a driver blocked on `OUTPUT_FULL` unresponsive. [#76782](https://github.com/StarRocks/starrocks/pull/76782) * `dictionary_get()` no longer rejects a non-NULL key when its input column's cached `has_null` flag is stale. [#76881](https://github.com/StarRocks/starrocks/pull/76881) * Added the missing `arrow-compression` module for Arrow Flight SQL, restoring LZ4/ZSTD codec support for compressed Arrow IPC clients. [#76921](https://github.com/StarRocks/starrocks/pull/76921) * Fixed an OOM in streaming pre-aggregation under memory pressure with spill enabled. [#76702](https://github.com/StarRocks/starrocks/pull/76702) * `Set` operators are no longer placed in colocate execution groups, fixing a hang caused by their branches being terminated by a plain local-exchange sink instead of a grouped-execution sink. [#77025](https://github.com/StarRocks/starrocks/pull/77025) * Stopped the query cache from storing incomplete per-tablet results for an aggregation with a `LIMIT`, which could return wrong results from the cache. [#77066](https://github.com/StarRocks/starrocks/pull/77066) * An insert-overwrite failure is no longer journaled against a table that was concurrently dropped, fixing an FE crash on journal replay. [#77212](https://github.com/StarRocks/starrocks/pull/77212) * Checkpoint-thread-created thread pools are no longer registered in the metrics registry. [#77367](https://github.com/StarRocks/starrocks/pull/77367) * Stopped shipping test-scope jars in the `java-extensions` reader libraries. [#77752](https://github.com/StarRocks/starrocks/pull/77752) * Resolved tablet backend IDs outside the `TabletInvertedIndex` write lock, fixing an FE deadlock between materialized-view refresh/insert-overwrite commit and tablet force-delete. [#78102](https://github.com/StarRocks/starrocks/pull/78102) * Several BE/CN crashes: a crash during graceful shutdown when a pending brpc closure ran after `SinkBuffer` was destroyed; a null-pointer crash in `SparseRangeIterator::has_more()` on an empty physical-split tablet; a `bad_weak_ptr` abort from an unscheduled global runtime-filter timer in `PipelineDriver`'s destructor; a SIGSEGV in the native Parquet reader on an incomplete nested lake schema; a heap-use-after-free with a multi-character CSV delimiter that straddles a buffer expansion; and a heap-buffer-overflow building the error message for a raw JSON value in `SimdJsonConverter`. [#73202](https://github.com/StarRocks/starrocks/pull/73202) [#75985](https://github.com/StarRocks/starrocks/pull/75985) [#76252](https://github.com/StarRocks/starrocks/pull/76252) [#76455](https://github.com/StarRocks/starrocks/pull/76455) [#76718](https://github.com/StarRocks/starrocks/pull/76718) [#76752](https://github.com/StarRocks/starrocks/pull/76752) * Several dependency CVEs: excluded an unused `avro-ipc` dependency that bundled a vulnerable jQuery 1.4.2; bumped Netty to 4.1.136.Final; excluded vulnerable Jetty jars and bumped pgjdbc to 42.7.12; bumped Apache Thrift to 0.24.0; and bumped Apache HttpCore to 5.4.3. [#76270](https://github.com/StarRocks/starrocks/pull/76270) [#76555](https://github.com/StarRocks/starrocks/pull/76555) [#76783](https://github.com/StarRocks/starrocks/pull/76783) [#76922](https://github.com/StarRocks/starrocks/pull/76922) [#77753](https://github.com/StarRocks/starrocks/pull/77753) #### 3.5.20[​](#3520 "Direct link to 3.5.20") Release date: July 23, 2026 ##### Behavior Changes[​](#behavior-changes-1 "Direct link to Behavior Changes") * `CREATE DATABASE IF NOT EXISTS` on Iceberg REST catalogs now succeeds silently when the database already exists, instead of raising an error. [#75017](https://github.com/StarRocks/starrocks/pull/75017) * Iceberg REST catalogs with vended credentials now cache `Table` objects and keep their credentials refreshed on access, instead of bypassing the cache and re-fetching from the REST catalog/Lake Formation on every `getTable()` call, which could trigger AWS `Rate exceeded` errors. [#75431](https://github.com/StarRocks/starrocks/pull/75431) * GIN inverted-index-accelerated `NOT MATCH` predicates no longer return rows with a `NULL` value, matching SQL three-valued-logic semantics. [#75578](https://github.com/StarRocks/starrocks/pull/75578) ##### Improvements[​](#improvements-1 "Direct link to Improvements") * Added the FE metric `txn_max_committed_pending_publish_ms`, a per-database gauge reporting the longest time a committed transaction has been pending publish, to help diagnose stuck or lagging version publishing. [#75025](https://github.com/StarRocks/starrocks/pull/75025) * Enforced the query memory limit when a column is upgraded (widened) during window-function aggregation in `Analytor`, instead of letting it grow unbounded. [#75821](https://github.com/StarRocks/starrocks/pull/75821) * Removed useless per-rowid seeks in the array-column offsets-only read path used by `array_length()`/`cardinality()`. [#75861](https://github.com/StarRocks/starrocks/pull/75861) ##### Bug fixes[​](#bug-fixes-1 "Direct link to Bug fixes") The following issues have been fixed: * Several wrong-result issues: `EliminateSortColumnWithEqualityPredicateRule` dropping the global `LIMIT` under concurrency; `SplitJoinORToUnionRule` producing duplicate rows for a null-safe-equal (`<=>`) `JOIN ON p1 OR p2`; JIT codegen truncating `LARGEINT` literals `>= 2^64` to 64 bits; `array_map`/`transform` silently dropping `NULL` rows when all non-null input arrays were empty; nested dictionary expressions rebuilt inconsistently across exchange fragments causing dict-decode failures; and a `LIKE` pattern with the `_` wildcard returning wrong rows on a GIN inverted index. [#74983](https://github.com/StarRocks/starrocks/pull/74983) [#75038](https://github.com/StarRocks/starrocks/pull/75038) [#75137](https://github.com/StarRocks/starrocks/pull/75137) [#75141](https://github.com/StarRocks/starrocks/pull/75141) [#75246](https://github.com/StarRocks/starrocks/pull/75246) [#75551](https://github.com/StarRocks/starrocks/pull/75551) * Join-reorder column pruning could drop a column still referenced by a predicate, causing a `missing statistic of col` planning error, and `JoinTuningGuide` could lose `predicateCommonOperators` when rebuilding a join, failing plan validation. [#74791](https://github.com/StarRocks/starrocks/pull/74791) [#75773](https://github.com/StarRocks/starrocks/pull/75773) * Sync materialized view/rollup rewrite could lose a rollup column when a query aggregated the same base column twice (e.g. `min(c)` and `max(c)`), and async materialized view rewrite could serve stale results after an Iceberg base table's `rollback_to_snapshot`. [#75528](https://github.com/StarRocks/starrocks/pull/75528) [#75924](https://github.com/StarRocks/starrocks/pull/75924) * `PARTITION-TOP-N` could rewrite its partition-by column to a dictionary slot that no longer existed, failing with a `slot_id not found` error. [#75956](https://github.com/StarRocks/starrocks/pull/75956) * An NPE collecting view tables when a `SECURITY INVOKER` view's stored definition contains a CTE. [#74813](https://github.com/StarRocks/starrocks/pull/74813) * Three FE metadata-lock correctness races around `DROP PERSISTENT INDEX`, `RestoreJob` post-restore handling, and related unlocked paths. [#74968](https://github.com/StarRocks/starrocks/pull/74968) * A race between FE EOS-cancel and BE stage-2 deploy could mark a fully successful query as canceled. [#75009](https://github.com/StarRocks/starrocks/pull/75009) * `ApplyTuningGuideRule` could throw `UnsupportedOperationException` when an earlier rewrite produced an `OptExpression` with an immutable input list. [#70785](https://github.com/StarRocks/starrocks/pull/70785) * BE/CN crashes: a null `driver_executor` when a cancel RPC arrives before pipeline start; a use-after-free in the spill partition-sort-sink cancel path; a heap-use-after-free in `OrderedPartitionExchanger` for a skew-hinted window function at DOP>1; an `NLJoin` crash from a build-side column nullability mismatch; a `StructColumn` field-count mismatch in `UNNEST` output; a crash loop reading a flat-JSON column that changed from `NOT NULL` to nullable during compaction; an uncaught memory-allocation exception in `NLJoinProbeOperator`; a crash in primary-key auto-increment partial-update apply; and a crash rewriting predicates inside an `array_map` lambda during scan-predicate pushdown. [#75030](https://github.com/StarRocks/starrocks/pull/75030) [#75140](https://github.com/StarRocks/starrocks/pull/75140) [#75279](https://github.com/StarRocks/starrocks/pull/75279) [#75343](https://github.com/StarRocks/starrocks/pull/75343) [#75445](https://github.com/StarRocks/starrocks/pull/75445) [#75680](https://github.com/StarRocks/starrocks/pull/75680) [#75788](https://github.com/StarRocks/starrocks/pull/75788) [#76119](https://github.com/StarRocks/starrocks/pull/76119) [#76380](https://github.com/StarRocks/starrocks/pull/76380) * `histogram()` crashed (or silently mis-bucketed values) on a non-positive `bucket_num` instead of raising a clear error, and `bar()` could grow an unbounded string for a negative or huge `width` argument, exhausting BE memory. [#75041](https://github.com/StarRocks/starrocks/pull/75041) [#75143](https://github.com/StarRocks/starrocks/pull/75143) * A query using `unnest` over array columns could exceed `query_mem_limit` and get the BE OOM-killed instead of failing just that query. [#75179](https://github.com/StarRocks/starrocks/pull/75179) * A second-order SQL injection in the `information_schema.task_runs` `TASK_NAME`/`QUERY_ID` predicate lookup. [#75520](https://github.com/StarRocks/starrocks/pull/75520) * `SHOW CREATE ROUTINE LOAD` could emit a spurious leading comma before the first load-desc clause, and an unescaped `jsonpaths` value, producing non-runnable DDL. [#75522](https://github.com/StarRocks/starrocks/pull/75522) [#75755](https://github.com/StarRocks/starrocks/pull/75755) * Shared-data (lake) `SHOW PARTITIONS` and `information_schema.partitions_meta` reported every physical partition's bucket count as the table-level default instead of its own bucket count. [#75734](https://github.com/StarRocks/starrocks/pull/75734) * Several dependency CVEs by upgrading `jackson-databind` and Netty. [#75373](https://github.com/StarRocks/starrocks/pull/75373) [#76555](https://github.com/StarRocks/starrocks/pull/76555) * Batched `TabletInvertedIndex` write-lock acquisition in `markTabletsForceDelete`, reducing lock churn when force-deleting many tablets at once. [#75616](https://github.com/StarRocks/starrocks/pull/75616) * Batched tablet inverted-index writes in the insert-overwrite path. [#75923](https://github.com/StarRocks/starrocks/pull/75923) * Skipped an unnecessary remote `clear_parent_path` call when a load spill never used remote storage. [#76224](https://github.com/StarRocks/starrocks/pull/76224) * A null-padding size mismatch for missing columns in `ParquetScanner` so padded rows match the actual per-batch chunk size instead of the whole Parquet/Arrow batch size. [#75981](https://github.com/StarRocks/starrocks/pull/75981) * Vulnerable, stale transitive dependencies (old BouncyCastle, OkHttp 2.x, Tomcat, and others) that previously shipped alongside their fixed counterparts. [#76097](https://github.com/StarRocks/starrocks/pull/76097) #### 3.5.19[​](#3519 "Direct link to 3.5.19") Release date: June 26, 2026 ##### Behavior Changes[​](#behavior-changes-2 "Direct link to Behavior Changes") * `parse_json` now respects `ALLOW_THROW_EXCEPTION`: malformed JSON fails the query instead of silently producing `NULL`, mirroring the earlier `get_json_string` change. [#74976](https://github.com/StarRocks/starrocks/pull/74976) * `FILES()` and Broker Load now honor the Parquet `isAdjustedToUTC=false` flag for `INT64` timestamps, so such timestamps are no longer shifted by the session time zone. [#73674](https://github.com/StarRocks/starrocks/pull/73674) * `SHOW FUNCTIONS` now surfaces the `isolation` property (`shared` or `isolated`) for Java UDFs and UDAFs. [#75255](https://github.com/StarRocks/starrocks/pull/75255) * The non-reserved keywords `FLOOR` and `CEIL` are now allowed as column names. [#75241](https://github.com/StarRocks/starrocks/pull/75241) ##### Improvements[​](#improvements-2 "Direct link to Improvements") * Supports loading Arrow `LARGE_LIST` and `FIXED_SIZE_LIST` columns into `JSON` columns via `FILES()` and Broker Load. [#73714](https://github.com/StarRocks/starrocks/pull/73714) [#73718](https://github.com/StarRocks/starrocks/pull/73718) * Added an opt-in `statistics_large_string_column_merge_threshold` to isolate wide `CHAR`/`VARCHAR` columns into dedicated statistics collection. [#73258](https://github.com/StarRocks/starrocks/pull/73258) * Optimized `base64_to_bitmap` for constant inputs and hardened it against invalid base64-encoded bitmaps. [#74684](https://github.com/StarRocks/starrocks/pull/74684) * Added new metrics for lake vacuum batch size and retry counts, and gauges for `CatalogRecycleBin` size. [#74112](https://github.com/StarRocks/starrocks/pull/74112) [#74440](https://github.com/StarRocks/starrocks/pull/74440) * Supports auditing a statement twice. [#73896](https://github.com/StarRocks/starrocks/pull/73896) ##### Bug fixes[​](#bug-fixes-2 "Direct link to Bug fixes") The following issues have been fixed: * Several wrong-result and planning issues: low-cardinality dictionary translation for expressions where `f(NULL)` is not `NULL`; a `Multiple entries with same key` error from common-subexpression elimination of commutative `AND`/`OR`; an `AGGREGATE has mismatch types` error; a compound predicate with an always-false nested branch under `UNION` returning no rows; and an off-by-one that dropped a row in `RANK` TopN at a chunk boundary. [#69376](https://github.com/StarRocks/starrocks/pull/69376) [#72823](https://github.com/StarRocks/starrocks/pull/72823) [#74159](https://github.com/StarRocks/starrocks/pull/74159) [#74218](https://github.com/StarRocks/starrocks/pull/74218) [#75045](https://github.com/StarRocks/starrocks/pull/75045) * Materialized view rewrite issues that could return incorrect results: aggregate MVs with a `HAVING` clause rewriting queries without (or with weaker) `HAVING`, and `avg(DISTINCT x)` being rewritten through a `sum`/`count` MV. [#73610](https://github.com/StarRocks/starrocks/pull/73610) [#75071](https://github.com/StarRocks/starrocks/pull/75071) * Incorrect window-function results when `enable_push_down_pre_agg_with_rank` split a window count into local pre-aggregation and global analytic merge, and an empty window operator generated after pushing down distinct aggregation. [#74453](https://github.com/StarRocks/starrocks/pull/74453) [#74810](https://github.com/StarRocks/starrocks/pull/74810) * Partition TopN losing a child operator's output column, and silently swallowing sort or pre-aggregation errors and returning wrong or partial results. [#72848](https://github.com/StarRocks/starrocks/pull/72848) [#74693](https://github.com/StarRocks/starrocks/pull/74693) * Iceberg equality-delete rows with `NULL` identity-column values were not applied. [#67321](https://github.com/StarRocks/starrocks/pull/67321) * A spurious strict-mode cast overflow error raised from undefined data in `NULL` rows. [#74903](https://github.com/StarRocks/starrocks/pull/74903) * Decimal scale could be lost when a column is entirely `NULL`. [#73789](https://github.com/StarRocks/starrocks/pull/73789) * BE crashes in `to_base64` (stack overflow), JSON load of nested types via partial append, local partition TopN with a non-nullable aggregate result, partitioned join (out-of-bounds from inaccurate memory accounting), runtime profile serialization (counter min/max race), JIT compilation failure (use-after-free of `LLVMContext`), invalid JIT IR for `CASE WHEN` with mixed float/integer types, and partial column updates under schema drift. [#70623](https://github.com/StarRocks/starrocks/pull/70623) [#73715](https://github.com/StarRocks/starrocks/pull/73715) [#69752](https://github.com/StarRocks/starrocks/pull/69752) [#74315](https://github.com/StarRocks/starrocks/pull/74315) [#72904](https://github.com/StarRocks/starrocks/pull/72904) [#74396](https://github.com/StarRocks/starrocks/pull/74396) [#74382](https://github.com/StarRocks/starrocks/pull/74382) [#74005](https://github.com/StarRocks/starrocks/pull/74005) * An out-of-bounds read and potential oversized allocation in `split`, `split_part`, and `str_to_map` when the input ends with a truncated UTF-8 byte. [#75068](https://github.com/StarRocks/starrocks/pull/75068) * A memory leak from the UDAF context cache and inflated query-pool memory accounting in `OlapTableSink`. [#74025](https://github.com/StarRocks/starrocks/pull/74025) [#73807](https://github.com/StarRocks/starrocks/pull/73807) * Unexpected backend process restarts. [#74424](https://github.com/StarRocks/starrocks/pull/74424) * Materialized view issues: a slot-nullability crash for MVs defined with `FULL OUTER JOIN` under late materialization, an NPE refreshing nested MVs, a duplicated warehouse property in `SHOW CREATE MATERIALIZED VIEW`, and a vector ANN query polluting a shared table schema and breaking unrelated statements. [#72621](https://github.com/StarRocks/starrocks/pull/72621) [#73644](https://github.com/StarRocks/starrocks/pull/73644) [#69418](https://github.com/StarRocks/starrocks/pull/69418) [#74785](https://github.com/StarRocks/starrocks/pull/74785) * Querying Paimon tables whose `DATE` partition column contains `NULL` values. [#73950](https://github.com/StarRocks/starrocks/pull/73950) * Reading Hudi MOR tables with `char`/`varchar` columns when `hudi_mor_force_jni_reader` is enabled. [#58521](https://github.com/StarRocks/starrocks/pull/58521) * Nested `INT96` timestamps (inside `ARRAY`, `MAP`, or `STRUCT`) were shifted by the session time zone during `FILES()`/Broker Load. [#74868](https://github.com/StarRocks/starrocks/pull/74868) * Incorrect bytes-read statistics in the audit log for connector scans, and incremental connector scan ranges being assigned to driver sequences absent from the deployed fragment, which could drop part of the scan. [#73799](https://github.com/StarRocks/starrocks/pull/73799) [#74674](https://github.com/StarRocks/starrocks/pull/74674) * Meta scan could fail after schema changes such as `ADD COLUMN`, which could fail background statistics collection. [#72901](https://github.com/StarRocks/starrocks/pull/72901) * Slow broker RPCs held the per-job Routine Load write lock and blocked admin RPCs and `SHOW ROUTINE LOAD`. [#73591](https://github.com/StarRocks/starrocks/pull/73591) * `ALTER ROUTINE LOAD` persisted an invalid statement for reserved-keyword table names, which could drop the load clause on FE restart. [#74188](https://github.com/StarRocks/starrocks/pull/74188) * `GRANT`/`REVOKE` on the `public` role did not invalidate cached merged privileges, leaving stale authorization. [#73717](https://github.com/StarRocks/starrocks/pull/73717) * A race allowing concurrent operations to observe torn state during table and materialized-view `RENAME` and `SWAP`, and a data race on `MaterializedIndexMeta` schema-update tracking. [#74100](https://github.com/StarRocks/starrocks/pull/74100) [#74412](https://github.com/StarRocks/starrocks/pull/74412) * Database-level UDFs were missing on FE followers after `RESTORE ... AS `. [#74313](https://github.com/StarRocks/starrocks/pull/74313) * Queries could become unkillable when a coordinator held its lock during external resource cleanup. [#72830](https://github.com/StarRocks/starrocks/pull/72830) * A permanent version hole on non-primary-key replicas could cause queries to fail with `version not found`. [#74408](https://github.com/StarRocks/starrocks/pull/74408) * Force-killed `SUBMIT TASK` runs disappeared from task-run history (and session-prefixed task-run timeouts are now honored), and an illegal running-to-running edit log could wedge subsequent task runs. [#74146](https://github.com/StarRocks/starrocks/pull/74146) [#73882](https://github.com/StarRocks/starrocks/pull/73882) * `ADMIN SHOW REPLICA STATUS` emitted a misaligned row for missing replicas, which could hang or disconnect the client. [#74393](https://github.com/StarRocks/starrocks/pull/74393) * `CatalogRecycleBin` halted all deletions in shared-data mode when cluster snapshots kept failing, causing unbounded FE memory growth. [#74379](https://github.com/StarRocks/starrocks/pull/74379) * An NPE in statistics calculation when a partition is dropped concurrently, and zero row counts written into partition statistics after `INSERT OVERWRITE` corrupting cardinality estimates. [#73711](https://github.com/StarRocks/starrocks/pull/73711) [#74801](https://github.com/StarRocks/starrocks/pull/74801) * Colocate tablets with all replicas on dead BEs were reported as healthy when `tablet_sched_disable_colocate_balance` is enabled. [#73550](https://github.com/StarRocks/starrocks/pull/73550) * An `IllegalMonitorStateException` from a lock mismatch in the tablet checker could abort a checker round. [#74596](https://github.com/StarRocks/starrocks/pull/74596) * Reduced lock contention by narrowing several full-database `WRITE` locks to table-scoped locks in shared-nothing mode, and skipped unnecessary locking in `TabletInvertedIndex.deleteTablets` for empty input. [#74523](https://github.com/StarRocks/starrocks/pull/74523) [#73955](https://github.com/StarRocks/starrocks/pull/73955) * A race between transaction begin and autovacuum could delete a still-needed transaction log and permanently wedge publishing in shared-data mode; decorrelated jitter was also added to lake vacuum retry backoff. [#74906](https://github.com/StarRocks/starrocks/pull/74906) [#74108](https://github.com/StarRocks/starrocks/pull/74108) * Added `lake_vacuum_enable_task_timeout` to abort BE vacuum tasks once the FE caller's timeout elapses. [#74694](https://github.com/StarRocks/starrocks/pull/74694) * A crash from a missing null check when reading `gtid` during a data-rewriting schema change. [#74855](https://github.com/StarRocks/starrocks/pull/74855) * A thread-name race produced noisy warnings during BE data directory load. [#73862](https://github.com/StarRocks/starrocks/pull/73862) * An `IllegalStateException` during parallel profile collection for external-table queries when `enable_profile` is on. [#74746](https://github.com/StarRocks/starrocks/pull/74746) * `ALTER TABLE ... MODIFY COLUMN ... AFTER` a nonexistent column raised an internal NPE instead of a clean error. [#75073](https://github.com/StarRocks/starrocks/pull/75073) * Query hangs and operator stalls from missing notifications when a distinct aggregate source finishes and on missed operator state transitions, and sort merge provider errors now propagate to the fragment context. [#74055](https://github.com/StarRocks/starrocks/pull/74055) [#74557](https://github.com/StarRocks/starrocks/pull/74557) [#73337](https://github.com/StarRocks/starrocks/pull/73337) * FE dropped the connection (client `ERROR 2013`) instead of returning a proper error when connecting to a database the user lacks privileges on. [#70072](https://github.com/StarRocks/starrocks/pull/70072) * Prepared statements with a microsecond `DATETIME` parameter failed with `Invalid date type: DECIMAL(6,0)`. [#74141](https://github.com/StarRocks/starrocks/pull/74141) * The audit log recorded `ReturnRows=0` for `SELECT ... INTO OUTFILE`. [#74467](https://github.com/StarRocks/starrocks/pull/74467) * `DATETIME_PRECISION` was always `NULL` in `information_schema.COLUMNS`, which broke type mapping for some MySQL-protocol clients. [#74623](https://github.com/StarRocks/starrocks/pull/74623) * CTAS did not preserve an explicitly declared `VARCHAR(N)` length in the new table's schema. [#73498](https://github.com/StarRocks/starrocks/pull/73498) * `enable_statistic_collect_on_first_load` now allows a table-level setting to override the global configuration. [#74794](https://github.com/StarRocks/starrocks/pull/74794) * A typo in the `azure_adls2_oauth2_client_endpoint` configuration field name. [#74581](https://github.com/StarRocks/starrocks/pull/74581) * Assertion name lookup in assert-num-rows. [#74178](https://github.com/StarRocks/starrocks/pull/74178) * Several dependency CVEs by upgrading libthrift, Tomcat, and Netty, and excluding a vulnerable jline transitive dependency. [#73243](https://github.com/StarRocks/starrocks/pull/73243) [#73797](https://github.com/StarRocks/starrocks/pull/73797) [#74668](https://github.com/StarRocks/starrocks/pull/74668) [#75066](https://github.com/StarRocks/starrocks/pull/75066) #### 3.5.18[​](#3518 "Direct link to 3.5.18") Release date: June 5, 2026 ##### Behavior Changes[​](#behavior-changes-3 "Direct link to Behavior Changes") * `SHOW` statements are now allowed inside explicit transactions. [#72954](https://github.com/StarRocks/starrocks/pull/72954) * `get_json_string` now respects `ALLOW_THROW_EXCEPTION` when handling JSON parsing errors. [#73199](https://github.com/StarRocks/starrocks/pull/73199) * `IGNORE NULLS` is now preserved in view definitions when the window function argument is an expression. [#69971](https://github.com/StarRocks/starrocks/pull/69971) * Ranger row filter and masking policies are now correctly applied to Hive views and to base tables expanded from Hive view definitions. [#73265](https://github.com/StarRocks/starrocks/pull/73265) * Hive partition statistics are no longer automatically refreshed per partition. Existing cached stats are preserved while a table-level asynchronous refresh updates the cache in batches. [#73563](https://github.com/StarRocks/starrocks/pull/73563) ##### Improvements[​](#improvements-3 "Direct link to Improvements") * Supports caching Java UDAF class-level initialization so shared UDAFs can reuse loaded classes and generated stubs across aggregator and window-function instances. [#72038](https://github.com/StarRocks/starrocks/pull/72038) * Supports Paimon time types and improves Paimon materialized view handling. [#58292](https://github.com/StarRocks/starrocks/pull/58292) * Added an Avro schema cache for shadowed `PartitionData` during partition load. [#72215](https://github.com/StarRocks/starrocks/pull/72215) * Added a configurable FE write timeout `mysql_send_packet_timeout_ms` for the MySQL result send path to prevent indefinitely blocked result sending to slow clients. [#73646](https://github.com/StarRocks/starrocks/pull/73646) * Optimized `CatalogRecycleBin` adjusted recycle timestamp lookup. [#72128](https://github.com/StarRocks/starrocks/pull/72128) * Reduced metadata and lock overhead in load balancing, compaction scheduling, consistency checks, and StarMgr metadata synchronization paths. [#73555](https://github.com/StarRocks/starrocks/pull/73555) [#72218](https://github.com/StarRocks/starrocks/pull/72218) [#72178](https://github.com/StarRocks/starrocks/pull/72178) [#72108](https://github.com/StarRocks/starrocks/pull/72108) * Improved diagnostics for filesystem copy failures and Parquet broker load errors by surfacing the underlying cause and file/column/row context. [#73414](https://github.com/StarRocks/starrocks/pull/73414) [#73236](https://github.com/StarRocks/starrocks/pull/73236) * Reduced external catalog and information schema metadata overhead by deferring JDBC REMARKS fetching, avoiding redundant Paimon snapshot lookups, and pushing down `table_name` predicates for `information_schema.tables_config`. [#73488](https://github.com/StarRocks/starrocks/pull/73488) [#72892](https://github.com/StarRocks/starrocks/pull/72892) [#73210](https://github.com/StarRocks/starrocks/pull/73210) * Simplified the scalar-function merge implementation by using `merge()` directly. [#69575](https://github.com/StarRocks/starrocks/pull/69575) ##### Bug fixes[​](#bug-fixes-3 "Direct link to Bug fixes") The following issues have been fixed: * Empty `ALTER TABLE` statements could be parsed as OPTIMIZE clauses, and replaying malformed OPTIMIZE jobs could clear a table's default distribution. [#73352](https://github.com/StarRocks/starrocks/pull/73352) * Decimal-valued unit counters in runtime profiles could cause query progress parsing failures and noisy FE warnings. [#73683](https://github.com/StarRocks/starrocks/pull/73683) * Concurrent `SegmentFlushTask` race in `DeltaWriter::commit()` and loss of `merge_condition` during normal rowset commit. [#73371](https://github.com/StarRocks/starrocks/pull/73371) [#72542](https://github.com/StarRocks/starrocks/pull/72542) * Crashes, hangs, or unsafe cleanup in `SinkBuffer` graceful exit, `PipelineTimerTask`, runtime filter workers, spillable hash join probe, `information_schema.warehouse_queries`, lake vacuum, HTTP connection unregister paths, and query queue timeout handling. [#73202](https://github.com/StarRocks/starrocks/pull/73202) [#73082](https://github.com/StarRocks/starrocks/pull/73082) [#72058](https://github.com/StarRocks/starrocks/pull/72058) [#72626](https://github.com/StarRocks/starrocks/pull/72626) [#72397](https://github.com/StarRocks/starrocks/pull/72397) [#72019](https://github.com/StarRocks/starrocks/pull/72019) [#73088](https://github.com/StarRocks/starrocks/pull/73088) [#72006](https://github.com/StarRocks/starrocks/pull/72006) [#65802](https://github.com/StarRocks/starrocks/pull/65802) * Materialized view issues involving JDBC SQL Server tables, lost index properties, cached plan context memory leaks, Paimon tables, and incorrect shuffle distribution after MV rewrite. [#72962](https://github.com/StarRocks/starrocks/pull/72962) [#69187](https://github.com/StarRocks/starrocks/pull/69187) [#72300](https://github.com/StarRocks/starrocks/pull/72300) [#58292](https://github.com/StarRocks/starrocks/pull/58292) [#71075](https://github.com/StarRocks/starrocks/pull/71075) * Query planning and rewrite issues in Spark connector external scans, `INSERT OVERWRITE` re-planning, aggregation spill with small LIMIT, and generated columns produced by `UNNEST`. [#73225](https://github.com/StarRocks/starrocks/pull/73225) [#72832](https://github.com/StarRocks/starrocks/pull/72832) [#72705](https://github.com/StarRocks/starrocks/pull/72705) [#72027](https://github.com/StarRocks/starrocks/pull/72027) * Paimon Primary Key columns could be incorrectly marked as non-nullable when querying external catalogs. [#71660](https://github.com/StarRocks/starrocks/pull/71660) * Primary Key and tablet metadata issues including partial tablet schema short-key mismatch, rowset metadata cache warmup deadlock, disk data cache expansion failure, Azure filesystem client cache issues in Starlet, and colocate-heavy cluster-balance performance issues in StarOS. [#70586](https://github.com/StarRocks/starrocks/pull/70586) [#71459](https://github.com/StarRocks/starrocks/pull/71459) [#58206](https://github.com/StarRocks/starrocks/pull/58206) [#73145](https://github.com/StarRocks/starrocks/pull/73145) [#72391](https://github.com/StarRocks/starrocks/pull/72391) * Locker rollback and unlock-order issues during partial intensive-lock acquisition. [#72789](https://github.com/StarRocks/starrocks/pull/72789) [#72423](https://github.com/StarRocks/starrocks/pull/72423) * Dependency CVEs and broker dependency regressions. [#72905](https://github.com/StarRocks/starrocks/pull/72905) [#72797](https://github.com/StarRocks/starrocks/pull/72797) [#72184](https://github.com/StarRocks/starrocks/pull/72184) [#72191](https://github.com/StarRocks/starrocks/pull/72191) * JNI local-reference leaks in JDBC scanner initialization. [#72913](https://github.com/StarRocks/starrocks/pull/72913) * Arrow dictionary values in Parquet scanner and Apache Parquet namespace ambiguity during scanner builds. [#71855](https://github.com/StarRocks/starrocks/pull/71855) [#72284](https://github.com/StarRocks/starrocks/pull/72284) * NPE in Iceberg `getPartitionLastUpdatedTime` when the snapshot is expired. [#68925](https://github.com/StarRocks/starrocks/pull/68925) #### 3.5.17[​](#3517 "Direct link to 3.5.17") Release date: May 13, 2026 ##### Behavior Changes[​](#behavior-changes-4 "Direct link to Behavior Changes") * `SHOW CREATE TABLE` and `DESC` now show Primary Keys for Paimon tables. [#70535](https://github.com/StarRocks/starrocks/pull/70535) * Disallowed INSERT into insert-only ACID Hive tables in Hive catalogs. [#71460](https://github.com/StarRocks/starrocks/pull/71460) * `START_TIME` and `END_TIME` in Profile are now displayed using the session time zone. [#71429](https://github.com/StarRocks/starrocks/pull/71429) ##### Improvements[​](#improvements-4 "Direct link to Improvements") * Supports `csv.enclose` and `csv.escape` in `INSERT INTO FILES` CSV export. [#71589](https://github.com/StarRocks/starrocks/pull/71589) * Added query relation information (directly queried tables and viewa) to audit logs. [#71596](https://github.com/StarRocks/starrocks/pull/71596) * Made the FE configuration `star_mgr_meta_sync_interval_sec` runtime mutable. [#71675](https://github.com/StarRocks/starrocks/pull/71675) * Reduced metadata and lock overhead in table metadata and row-count paths. [#72053](https://github.com/StarRocks/starrocks/pull/72053) [#72042](https://github.com/StarRocks/starrocks/pull/72042) [#71672](https://github.com/StarRocks/starrocks/pull/71672) * Improved build and dependency hygiene by merging the broker builder into the FE build and removing WildFly OpenSSL. [#71823](https://github.com/StarRocks/starrocks/pull/71823) [#71908](https://github.com/StarRocks/starrocks/pull/71908) ##### Bug fixes[​](#bug-fixes-4 "Direct link to Bug fixes") The following issues have been fixed: * Wrong results for local-shuffle aggregate queries with OFFSET. [#71997](https://github.com/StarRocks/starrocks/pull/71997) * Incorrect Join output properties after Exchange shuffle columns are pruned. [#72003](https://github.com/StarRocks/starrocks/pull/72003) * Several dependency CVE issues. [#71762](https://github.com/StarRocks/starrocks/pull/71762) [#71914](https://github.com/StarRocks/starrocks/pull/71914) * Oracle JDBC NLS format handling issue. [#71412](https://github.com/StarRocks/starrocks/pull/71412) * Missing Iceberg column statistics in manifest data file cache. [#71913](https://github.com/StarRocks/starrocks/pull/71913) * Missing Hive partition directory before INSERT OVERWRITE commit. [#71810](https://github.com/StarRocks/starrocks/pull/71810) * Aggregate-join-pushdown materialized view rewrite and min/max optimization issues on Iceberg base tables. [#71856](https://github.com/StarRocks/starrocks/pull/71856) [#71863](https://github.com/StarRocks/starrocks/pull/71863) * Race conditions in `ConnectorSinkPassthroughExchanger` and `LoadChannel::get_load_replica_status`. [#71848](https://github.com/StarRocks/starrocks/pull/71848) [#71843](https://github.com/StarRocks/starrocks/pull/71843) * Credential redaction issue in INSERT FILES operations. [#71245](https://github.com/StarRocks/starrocks/pull/71245) * Incorrect `reverse(DecimalV3)` results. [#71834](https://github.com/StarRocks/starrocks/pull/71834) * Missing JNI exception handling checks in Java UDF code. [#71734](https://github.com/StarRocks/starrocks/pull/71734) * Incorrect short-circuit checks in `EventScheduler`. [#71740](https://github.com/StarRocks/starrocks/pull/71740) * Incorrect Arrow Flight column name for empty result sets. [#71534](https://github.com/StarRocks/starrocks/pull/71534) * Batch publish deadlock caused by partition version gaps. [#71483](https://github.com/StarRocks/starrocks/pull/71483) * Repeated Apply attachments in scalar-subquery plans. [#71155](https://github.com/StarRocks/starrocks/pull/71155) #### 3.5.16[​](#3516 "Direct link to 3.5.16") Release date: April 20, 2026 ##### Improvements[​](#improvements-5 "Direct link to Improvements") * Added clearer warning logs for swallowed exceptions in `WarehouseManager`. [#71215](https://github.com/StarRocks/starrocks/pull/71215) * Supports pausing Routine Load jobs on non-retryable errors. [#71161](https://github.com/StarRocks/starrocks/pull/71161) * Added thread names to the utility that prints all thread stacks. [#69366](https://github.com/StarRocks/starrocks/pull/69366) * Supports constant folding for `regexp_replace` in FE. [#70804](https://github.com/StarRocks/starrocks/pull/70804) * Supports showing column comments for PostgreSQL external tables, and added an option to let `information_schema.tables` fetch full metadata such as comments from external catalogs. [#70520](https://github.com/StarRocks/starrocks/pull/70520) [#70197](https://github.com/StarRocks/starrocks/pull/70197) * Added automatic query plan dumping on query exceptions. [#70387](https://github.com/StarRocks/starrocks/pull/70387) * Improved cloud-native tablet metadata fetch and repair efficiency. [#70492](https://github.com/StarRocks/starrocks/pull/70492) [#70386](https://github.com/StarRocks/starrocks/pull/70386) * Added batch tablet deletion in FE to reduce write lock contention. [#70052](https://github.com/StarRocks/starrocks/pull/70052) * Added logs for update compaction suspension, and added Iceberg metadata-table and time-travel query metrics. [#70538](https://github.com/StarRocks/starrocks/pull/70538) [#70825](https://github.com/StarRocks/starrocks/pull/70825) [#70788](https://github.com/StarRocks/starrocks/pull/70788) ##### Bug fixes[​](#bug-fixes-5 "Direct link to Bug fixes") The following issues have been fixed: * `be_tablets.DATA_SIZE` reports rowset column data bytes inaccurately. [#70735](https://github.com/StarRocks/starrocks/pull/70735) * An outdated Maven repository for broker builds. [#71533](https://github.com/StarRocks/starrocks/pull/71533) * Incorrect thread-pool resizing when updating `parallel_clone_task_per_path`. [#71484](https://github.com/StarRocks/starrocks/pull/71484) * Several use-after-free issues. [#71513](https://github.com/StarRocks/starrocks/pull/71513) [#71276](https://github.com/StarRocks/starrocks/pull/71276) [#71083](https://github.com/StarRocks/starrocks/pull/71083) [#62917](https://github.com/StarRocks/starrocks/pull/62917) [#69926](https://github.com/StarRocks/starrocks/pull/69926) [#69968](https://github.com/StarRocks/starrocks/pull/69968) * Resource group user classifier validation is not aligned with `CREATE USER`. [#71470](https://github.com/StarRocks/starrocks/pull/71470) * “no queryable replica” issues on follower FEs by syncing StarMgr journal replay. [#71263](https://github.com/StarRocks/starrocks/pull/71263) * Multiple dependency CVEs. [#71256](https://github.com/StarRocks/starrocks/pull/71256) [#71017](https://github.com/StarRocks/starrocks/pull/71017) [#70862](https://github.com/StarRocks/starrocks/pull/70862) * `VARCHAR` length is not preserved after reduce-cast with global variables. [#70269](https://github.com/StarRocks/starrocks/pull/70269) * Special-character escaping in equality predicates on `information_schema.tables`. [#71273](https://github.com/StarRocks/starrocks/pull/71273) * `UpdateTabletSchemaTask` signature collisions across alter jobs. [#71242](https://github.com/StarRocks/starrocks/pull/71242) * Issue with shared-object mutation in `PushDownAggregateRewriter` for `CASE WHEN` and `IF` expressions. [#71309](https://github.com/StarRocks/starrocks/pull/71309) * Stopped inactive materialized view schedulers correctly and added missing leader checks in TaskManager scheduler callbacks. [#71265](https://github.com/StarRocks/starrocks/pull/71265) [#71156](https://github.com/StarRocks/starrocks/pull/71156) * `NaN` row-count estimation for MCV-only histograms. [#71241](https://github.com/StarRocks/starrocks/pull/71241) * Packaging issues caused by a missing `s3-transfer-manager` dependency in the AWS SDK. [#71230](https://github.com/StarRocks/starrocks/pull/71230) * Thread-local `ConnectContext` pollution after leader forwarding. [#71141](https://github.com/StarRocks/starrocks/pull/71141) * Orphaned delvec entries when write and compaction transactions are published in the same batch. [#71001](https://github.com/StarRocks/starrocks/pull/71001) [#71049](https://github.com/StarRocks/starrocks/pull/71049) [#71107](https://github.com/StarRocks/starrocks/pull/71107) * Missing partition predicates in short-circuit point lookups. [#71124](https://github.com/StarRocks/starrocks/pull/71124) * Potential hash-table data loss during aggregation spill `set_finishing`. [#70851](https://github.com/StarRocks/starrocks/pull/70851) * Query-progress HTTP loopback records from `current_queries`. [#71032](https://github.com/StarRocks/starrocks/pull/71032) * Primary Key tablet rowset metadata loss caused by a GC race during disk re-migration (A→B→A). [#70727](https://github.com/StarRocks/starrocks/pull/70727) * DB read-lock leaks in `SharedDataStorageVolumeMgr`. [#70987](https://github.com/StarRocks/starrocks/pull/70987) * Incorrect `NullColumn` sharing in `NullableColumn`, `BinaryColumn`, and `locate()`. [#66037](https://github.com/StarRocks/starrocks/pull/66037) [#70957](https://github.com/StarRocks/starrocks/pull/70957) * Race conditions in global runtime-filter readiness checks and corrected metric overflow caused by `ACCUMULATED` macro truncation. [#70920](https://github.com/StarRocks/starrocks/pull/70920) [#70889](https://github.com/StarRocks/starrocks/pull/70889) * Generated-column is not displayed in `DESC` and `SHOW CREATE TABLE`. [#70037](https://github.com/StarRocks/starrocks/pull/70037) * An issue with load spill directory cleanup timing, an ASAN crash in memory table spiller workgroup handling, and CN crashes when scanning empty tablets with physical split enabled. [#70778](https://github.com/StarRocks/starrocks/pull/70778) [#64379](https://github.com/StarRocks/starrocks/pull/64379) [#70281](https://github.com/StarRocks/starrocks/pull/70281) * Incorrect `Content-Length` handling when `proxy_pass_request_body` is off. [#70821](https://github.com/StarRocks/starrocks/pull/70821) * Issues with connector scan retry state handling and multiple schema pushdown issues for `INSERT INTO BY NAME ... FROM FILES()`. [#70762](https://github.com/StarRocks/starrocks/pull/70762) [#70774](https://github.com/StarRocks/starrocks/pull/70774) [#70622](https://github.com/StarRocks/starrocks/pull/70622) [#70621](https://github.com/StarRocks/starrocks/pull/70621) * Invalid conjunct pushdown in MySQL and JDBC scan nodes that caused BE predicate type errors. [#70694](https://github.com/StarRocks/starrocks/pull/70694) * Incomplete and partially written Iceberg manifest cache entries, and bypassed catalog caching when vended credentials are enabled. [#70675](https://github.com/StarRocks/starrocks/pull/70675) [#70652](https://github.com/StarRocks/starrocks/pull/70652) [#69434](https://github.com/StarRocks/starrocks/pull/69434) * Ubuntu runtime dependency issues by installing `libssl-dev`. [#70688](https://github.com/StarRocks/starrocks/pull/70688) * User authentication strings are not masked in audit logs and SQL redaction. [#70360](https://github.com/StarRocks/starrocks/pull/70360) * External materialized view refresh issues for Iceberg-like connectors. [#70589](https://github.com/StarRocks/starrocks/pull/70589) [#70523](https://github.com/StarRocks/starrocks/pull/70523) * `array_map` crashes on null literal arrays and BE crashes when a child iterator is exhausted in `MaskMergeIterator`. [#70629](https://github.com/StarRocks/starrocks/pull/70629) [#70539](https://github.com/StarRocks/starrocks/pull/70539) * `starlet` configuration updates were incorrectly captured through `std::call_once`. [#70482](https://github.com/StarRocks/starrocks/pull/70482) * Robustness issue with Iceberg materialized view refresh when snapshot timestamps are non-monotonic. [#70382](https://github.com/StarRocks/starrocks/pull/70382) * Issues that forced materialized view refresh is not supported, and duplicated partition names during materialized view refresh. [#70381](https://github.com/StarRocks/starrocks/pull/70381) [#70354](https://github.com/StarRocks/starrocks/pull/70354) * Incorrect distribution-column handling across partitions in `OlapTableSink`. [#70310](https://github.com/StarRocks/starrocks/pull/70310) * File-existence checks are not cached across tablet metadata versions during missing-file validation. [#70364](https://github.com/StarRocks/starrocks/pull/70364) * Issue with `dataVersion` handling during RESTORE, and incorrect materialized view rewrite logic. [#70373](https://github.com/StarRocks/starrocks/pull/70373) [#69751](https://github.com/StarRocks/starrocks/pull/69751) * Negative `query_pool` memory accounting during ingestion, and high FE OOM risk under high concurrency. [#70228](https://github.com/StarRocks/starrocks/pull/70228) [#68444](https://github.com/StarRocks/starrocks/pull/68444) * Incorrect SLF4J parameterized logging. [#70330](https://github.com/StarRocks/starrocks/pull/70330) * `AuditEventProcessor` exiting on `OutOfMemoryException`. [#70206](https://github.com/StarRocks/starrocks/pull/70206) * Adjusted handling of column-mode partial updates for Primary Key tables; the initial corruption fix was reverted in this release cycle for follow-up work. [#69652](https://github.com/StarRocks/starrocks/pull/69652) * Equality and deduplication issues in `InformationFunction`. [#70464](https://github.com/StarRocks/starrocks/pull/70464) * `brpc` connection retries when exceptions are wrapped in `NoSuchElementException`. [#70203](https://github.com/StarRocks/starrocks/pull/70203) * Lock-free materialized view rewrite does not fallback to live metadata. [#70475](https://github.com/StarRocks/starrocks/pull/70475) * Issue with `JoinHashTable::merge_ht()` that it does not skip dummy rows for expression-based join-key columns. [#70465](https://github.com/StarRocks/starrocks/pull/70465) #### 3.5.15[​](#3515 "Direct link to 3.5.15") Release Date: March 26, 2026 ##### Behavior Changes[​](#behavior-changes-5 "Direct link to Behavior Changes") * Improved `sql_mode` handling: when `DIVISION_BY_ZERO` or `FAIL_PARSE_DATE` mode is set, division by zero and date parse failures in `str_to_date`/`str2date` now return an error instead of being silently ignored. [#70004](https://github.com/StarRocks/starrocks/pull/70004) * When `sql_mode` is set to `FORBID_INVALID_DATE`, invalid dates in `INSERT VALUES` clauses are now correctly rejected instead of being bypassed. [#69803](https://github.com/StarRocks/starrocks/pull/69803) * Expression partition generated columns are now hidden from `DESC` and `SHOW CREATE TABLE` output. [#69793](https://github.com/StarRocks/starrocks/pull/69793) * Client ID is no longer included in audit logs. [#69383](https://github.com/StarRocks/starrocks/pull/69383) * The `FORCE` option for `REFRESH EXTERNAL TABLE` has been reverted and is no longer supported. [#70428](https://github.com/StarRocks/starrocks/pull/70428) * Aligned the backslash escape sequence usage in LIKE predicates with MySQL. Users need to specify four backslashes `\\\\` in sequence in a LIKE predicate to match a literal backslash `\` in the data, and eight backslashes `\\\\\\\\` in the predicate to match two consecutive literal backslashes `\\`. ##### Improvements[​](#improvements-6 "Direct link to Improvements") * Allowed disabling split and reverse scan ranges for descending TopN by setting `desc_hint_split_range` to `0` or less. [#70307](https://github.com/StarRocks/starrocks/pull/70307) * `information_schema` now shows comments for external catalog tables. [#70197](https://github.com/StarRocks/starrocks/pull/70197) * Added `EXPLAIN` and `EXPLAIN ANALYZE` support for `INSERT` statements in Trino dialect. [#70174](https://github.com/StarRocks/starrocks/pull/70174) * Added configurable parameters for `CatalogRecycleBin` to control recycle bin behavior. [#69838](https://github.com/StarRocks/starrocks/pull/69838) * Improved `ADMIN REPAIR TABLE` and `SHOW TABLET STATUS` to provide better repair and status information. [#69656](https://github.com/StarRocks/starrocks/pull/69656) * Blacklisted queries are now excluded from error metrics. [#69621](https://github.com/StarRocks/starrocks/pull/69621) * Added support for `SHOW TABLET STATUS` for cloud-native tablets in shared-data deployments. [#69616](https://github.com/StarRocks/starrocks/pull/69616) * Reduced overhead of Primary Key tablet statistics collection in shared-data clusters. [#69548](https://github.com/StarRocks/starrocks/pull/69548) * Added support for dynamic configuration of the execution state report thread pool size. [#69142](https://github.com/StarRocks/starrocks/pull/69142) ##### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") Fixed the following bugs: * Data version not set when restoring a tablet. [#70373](https://github.com/StarRocks/starrocks/pull/70373) * Table comment not set when creating a Hive table. [#70318](https://github.com/StarRocks/starrocks/pull/70318) * Constant folding with double precision arithmetic producing `INF` instead of returning an error. [#70309](https://github.com/StarRocks/starrocks/pull/70309) * Iceberg materialized view refresh failing when snapshot timestamps are non-monotonic. [#70382](https://github.com/StarRocks/starrocks/pull/70382) * `toIcebergTable` function using `common` instead of `comment` in property mapping. [#70267](https://github.com/StarRocks/starrocks/pull/70267) * Root user not correctly bypassing Ranger permission checks in all scenarios. [#70254](https://github.com/StarRocks/starrocks/pull/70254) * `AuditEventProcessor` thread exiting unexpectedly when an `OutOfMemoryException` occurs. [#70206](https://github.com/StarRocks/starrocks/pull/70206) * Out-of-bounds access in `cal_new_base_version` during schema change publish. [#70132](https://github.com/StarRocks/starrocks/pull/70132) * Partition predicates pruned unexpectedly due to type mismatch in boundary comparison. [#70097](https://github.com/StarRocks/starrocks/pull/70097) * `str_to_date` losing microsecond precision in BE runtime. [#70068](https://github.com/StarRocks/starrocks/pull/70068) * Crash in join spill process when `set_callback_function` is called. [#70030](https://github.com/StarRocks/starrocks/pull/70030) * DCHECK failure in `DeltaWriter::close()` when called from a bthread context. [#69960](https://github.com/StarRocks/starrocks/pull/69960) * Use-after-free race condition in `AsyncDeltaWriter` close/finish lifecycle. [#69940](https://github.com/StarRocks/starrocks/pull/69940) * Journal replay not awaited in `changeCatalogDb` on follower FE, causing consistency issues. [#69834](https://github.com/StarRocks/starrocks/pull/69834) * Race condition causing missed write transaction finished editlog. [#69899](https://github.com/StarRocks/starrocks/pull/69899) * Several known CVEs addressed. [#69863](https://github.com/StarRocks/starrocks/pull/69863) * Incorrect LIKE pattern matching with backslash escape sequences. [#69775](https://github.com/StarRocks/starrocks/pull/69775) * Expression analysis failing after renaming a partition column. [#69771](https://github.com/StarRocks/starrocks/pull/69771) * Use-after-free crash in `AsyncDeltaWriter::close`. [#69770](https://github.com/StarRocks/starrocks/pull/69770) * Potential bugs in `PartitionColumnMinMaxRewriteRule` caused by incorrect `Partition.hasStorageData` results. [#69751](https://github.com/StarRocks/starrocks/pull/69751) * Duplicated CSV compression suffix in file sink output file names. [#69749](https://github.com/StarRocks/starrocks/pull/69749) * Lake `capture_tablet_and_rowsets` operation accessible without experimental config flag. [#69748](https://github.com/StarRocks/starrocks/pull/69748) * Corrupted cache for Primary Key SST tables. [#69693](https://github.com/StarRocks/starrocks/pull/69693) * Use-after-free in `AsyncFlushOutputStream`. [#69688](https://github.com/StarRocks/starrocks/pull/69688) * Incorrect retention clock reset and incomplete scan in `disableRecoverPartitionWithSameName`. [#69677](https://github.com/StarRocks/starrocks/pull/69677) * Tablet info not fetched correctly based on run mode in `SchemaBeTabletsScanner`. [#69645](https://github.com/StarRocks/starrocks/pull/69645) * Incorrect minimum partition pruning with shadow partitions. [#69641](https://github.com/StarRocks/starrocks/pull/69641) * Different transactions publishing the same version after graceful exit. [#69639](https://github.com/StarRocks/starrocks/pull/69639) * Iterator undefined behavior in `get_column_values` when `rssid` is not found. [#69617](https://github.com/StarRocks/starrocks/pull/69617) * `KILL ANALYZE` statement sometimes not stopping a running `ANALYZE TABLE` operation. [#69592](https://github.com/StarRocks/starrocks/pull/69592) * Materialized view force refresh bugs for partition tables. [#69488](https://github.com/StarRocks/starrocks/pull/69488) #### 3.5.14[​](#3514 "Direct link to 3.5.14") Release Date: March 5, 2026 ##### Improvements[​](#improvements-7 "Direct link to Improvements") * Added SST read/write failure metrics for Primary Key index in Lake tables. [#69513](https://github.com/StarRocks/starrocks/pull/69513) * Added a counter metric for "segment file not found" errors. [#69543](https://github.com/StarRocks/starrocks/pull/69543) * Extracted range predicates from scalar-subquery containing `convert_tz`. [#69055](https://github.com/StarRocks/starrocks/pull/69055) * Supports complex type for Paimon tables. [#66784](https://github.com/StarRocks/starrocks/pull/66784) * Deferred remote load Spill Directory removal. [#68803](https://github.com/StarRocks/starrocks/pull/68803) * Supports repairing cloud-native tables. [#67108](https://github.com/StarRocks/starrocks/pull/67108) * Supports inserting ARRAY type to Hive table in CSV format. [#67355](https://github.com/StarRocks/starrocks/pull/67355) ##### Bug Fixes[​](#bug-fixes-7 "Direct link to Bug Fixes") The following issues have been fixed: * Unexpected behavior caused by exceptions of `RowGroupWriter`. [#69568](https://github.com/StarRocks/starrocks/pull/69568) * Sort key not including newly added key columns after schema change on Aggregate Key/Unique Key tables. [#69529](https://github.com/StarRocks/starrocks/pull/69529) * Mertic value `g_publish_version_failed_tasks` does not reflect the real situation during the `resource_busy` state. [#69526](https://github.com/StarRocks/starrocks/pull/69526) * Rowset files are removed when moving Primary Key tablets to trash. [#69438](https://github.com/StarRocks/starrocks/pull/69438) * Lock leak in `addPartitions` caused by name-based table lookup after concurrent SWAP. [#69284](https://github.com/StarRocks/starrocks/pull/69284) * `isInternalCancelError` used `equals` instead of `startsWith`. [#69523](https://github.com/StarRocks/starrocks/pull/69523) * Pipeline blocks or crashes when `_writer->Close()` throws an exception other than `ParquetStatusException`. [#69492](https://github.com/StarRocks/starrocks/pull/69492) * A Hadoop-client lib bug. [#69503](https://github.com/StarRocks/starrocks/pull/69503) * Success is mistakenly returned while write operations fails. [#69473](https://github.com/StarRocks/starrocks/pull/69473) * CVE-2025-67721. [#69138](https://github.com/StarRocks/starrocks/pull/69138) * Issue with RuntimeFilter with low-cardinality optimization in share-data clusters. [#64669](https://github.com/StarRocks/starrocks/pull/64669) * Materialized view tablet meta inconsistency between FE leader and follower. [#69428](https://github.com/StarRocks/starrocks/pull/69428) * Rollup handler's active transaction ID was not considered in `computeMinActiveTxnId`. [#69285](https://github.com/StarRocks/starrocks/pull/69285) * Arrow Flight Proxy issue with multiple FE. [#68300](https://github.com/StarRocks/starrocks/pull/68300) * Concurrency bug of function field. [#69315](https://github.com/StarRocks/starrocks/pull/69315) * `DROP FUNCTION IF EXISTS` ignored `ifExists` flag. [#69216](https://github.com/StarRocks/starrocks/pull/69216) * Lacking case-insensitive username normalization for LDAP authentication. [#67966](https://github.com/StarRocks/starrocks/pull/67966) * Certain kinds of partitions cannot be written. [#68221](https://github.com/StarRocks/starrocks/pull/68221) * Projection loss in materialized view rewrite due to shared mutable state. [#69063](https://github.com/StarRocks/starrocks/pull/69063) * Issue with case-insensitive partition lookup in query table copy. [#69173](https://github.com/StarRocks/starrocks/pull/69173) * All-null value handling bug in synchronous materialized views. [#69136](https://github.com/StarRocks/starrocks/pull/69136) * `mv onReload` issues when visiting external catalogs. [#68926](https://github.com/StarRocks/starrocks/pull/68926) * DISTINCT ORDER BY alias issues for duplicated constants. [#69014](https://github.com/StarRocks/starrocks/pull/69014) * Incorrect query results after modifying CHAR column length in shared-data clusters. [#68808](https://github.com/StarRocks/starrocks/pull/68808) * Issue with Azure ABFS/WASB FileSystem cache key. [#68901](https://github.com/StarRocks/starrocks/pull/68901) * Incorrect predicate rewrite for OUTER JOIN with constant-side column reference. [#67072](https://github.com/StarRocks/starrocks/pull/67072) * `IllegalArgumentException` comparator transitivity violation. [#68743](https://github.com/StarRocks/starrocks/pull/68743) * Issue caused by the query lifetime being shorter than the fragment in `report_fragment`. [#67219](https://github.com/StarRocks/starrocks/pull/67219) * Low-cardinality rewrite NPE caused by shared `DecodeInfo`. [#68799](https://github.com/StarRocks/starrocks/pull/68799) * Missing `pcu_upt_cnt` metric. [#68845](https://github.com/StarRocks/starrocks/pull/68845) * JSON-flatten array/object conflict on identical paths. [#68804](https://github.com/StarRocks/starrocks/pull/68804) * `ClonExpr` nullable bug. [#68800](https://github.com/StarRocks/starrocks/pull/68800) #### 3.5.13[​](#3513 "Direct link to 3.5.13") Release Date: February 13, 2026 ##### Improvements[​](#improvements-8 "Direct link to Improvements") * Added an FE configuration `enable_table_metrics_collect` to control the collection of table-level metrics. [#68691](https://github.com/StarRocks/starrocks/pull/68691) * Supports setting the default Warehouse for Merge Commit at user level. [#68616](https://github.com/StarRocks/starrocks/pull/68616) ##### Bug fixes[​](#bug-fixes-8 "Direct link to Bug fixes") The following issues have been fixed: * Issue with source partition checking in replication transactions. [#68883](https://github.com/StarRocks/starrocks/pull/68883) * Used labels were not identified when labels were specified in BEGIN TRANSACTION. [#68660](https://github.com/StarRocks/starrocks/pull/68660) * JOIN ON clause bug with CTE scope. [#68809](https://github.com/StarRocks/starrocks/pull/68809) * Overlapping range partitions can be created when an explicit lower bound is provided. [#68255](https://github.com/StarRocks/starrocks/pull/68255) * Incorrect parser logic when SQL dialect downgrades from Trino to StarRocks. [#68725](https://github.com/StarRocks/starrocks/pull/68725) * Issue with pruning projection columns. [#68242](https://github.com/StarRocks/starrocks/pull/68242) * Issue with subquery scope check. [#68415](https://github.com/StarRocks/starrocks/pull/68415) * Unmatched type cast in the function analyzer. [#66749](https://github.com/StarRocks/starrocks/pull/66749) * Incorrect candidate materialized view selection logic. [#68571](https://github.com/StarRocks/starrocks/pull/68571) * The Thrift `accept` thread exits on exception. [#68644](https://github.com/StarRocks/starrocks/pull/68644) * Inaccurate Iceberg data file size estimation. [#68787](https://github.com/StarRocks/starrocks/pull/68787) * Lake table memory leak issue. [#68678](https://github.com/StarRocks/starrocks/pull/68678) * Deadlock when the HMS connection pool is full. [#68033](https://github.com/StarRocks/starrocks/pull/68033) * Iceberg delete column nullability issue. [#68649](https://github.com/StarRocks/starrocks/pull/68649) * Materialized views hold large external tables. [#68171](https://github.com/StarRocks/starrocks/pull/68171) * Iceberg table cache memory limit issue. [#67769](https://github.com/StarRocks/starrocks/pull/67769) * Wrong timeout parameter is used for PocoHttpClient. [#68765](https://github.com/StarRocks/starrocks/pull/68765) * BE compile failure with Clang. [#68805](https://github.com/StarRocks/starrocks/pull/68805) * Materialized view was reloaded multiple times during startup. [#62351](https://github.com/StarRocks/starrocks/pull/62351) * CVE-2025-27821. [#68529](https://github.com/StarRocks/starrocks/pull/68529) * Variadic functions return incorrect date values in certain scenarios. [#67947](https://github.com/StarRocks/starrocks/pull/67947) #### 3.5.12[​](#3512 "Direct link to 3.5.12") Release Date: January 22, 2026 ##### Improvements[​](#improvements-9 "Direct link to Improvements") * Added a cleaner for BrpcStubCache to clean up unused connections. [#61417](https://github.com/StarRocks/starrocks/pull/61417) * Supports batch processing for statistics delete (for dropped tables) and Edit Log write requests. [#67896](https://github.com/StarRocks/starrocks/pull/67896) * Preserves SQL comments in Audit Logs when encryption is required. [#63298](https://github.com/StarRocks/starrocks/pull/63298) * Added the `warehouse_name` label to the materialized view metrics. [#67715](https://github.com/StarRocks/starrocks/pull/67715) * Improved identifier wrapping for JDBC table and column names. [#67853](https://github.com/StarRocks/starrocks/pull/67853) * Added the `CLIENT_FACTORY` property to the Iceberg JDBC catalog. [#67613](https://github.com/StarRocks/starrocks/pull/67613) ##### Bug fixes[​](#bug-fixes-9 "Direct link to Bug fixes") The following issues have been fixed: * Variadic functions return wrong dates when mixing DATE and DATETIME types. [#67947](https://github.com/StarRocks/starrocks/pull/67947) * `NormalizePredicateRule` oscillation on non-deterministic expressions. [#67923](https://github.com/StarRocks/starrocks/pull/67923) * Low cardinality bugs with the Lambda function. [#67843](https://github.com/StarRocks/starrocks/pull/67843) * Subfield expression does not collect children subfields. [#67850](https://github.com/StarRocks/starrocks/pull/67850) * NPE in RBO Join reorder when child statistics are missing. [#67693](https://github.com/StarRocks/starrocks/pull/67693) * BE crash due to MemTable finalize failed. [#67787](https://github.com/StarRocks/starrocks/pull/67787) * Temporary partitions are not cleaned up after FE restart for dynamic overwrite. [#67629](https://github.com/StarRocks/starrocks/pull/67629) * Inaccurate I/O statistics of Compaction. [#67524](https://github.com/StarRocks/starrocks/pull/67524) * Incorrect logic in physical partition comparison across clusters during replication transaction. [#67616](https://github.com/StarRocks/starrocks/pull/67616) * Issue with SQL Server and Oracle identifier symbol handling. [#67965](https://github.com/StarRocks/starrocks/pull/67965) * NPE in the Iceberg metadata table query due to missing configuration propagation. [#67151](https://github.com/StarRocks/starrocks/pull/67151) * Issue with the `f``iles()` schema detection for empty Parquet or ORC files. [#67762](https://github.com/StarRocks/starrocks/pull/67762) * Inaccurate value of metrics in Profile caused by UNION ALL on Hive tables. [#67912](https://github.com/StarRocks/starrocks/pull/67912) * Lacking support for data retrieval from Arrow Flight proxy for FE queries. [#67794](https://github.com/StarRocks/starrocks/pull/67794) * SIGSEGV crash during automatic partition creation caused by a race condition in `OlapTableSink::is_full()`. [#67566](https://github.com/StarRocks/starrocks/pull/67566) #### 3.5.11[​](#3511 "Direct link to 3.5.11") Release date: January 5, 2026 ##### Improvements[​](#improvements-10 "Direct link to Improvements") * Supports Arrow Flight data retrieval from inaccessible nodes. [#66348](https://github.com/StarRocks/starrocks/pull/66348) * Logs the cause (including the triggering process information) in the SIGTERM handler. [#66737](https://github.com/StarRocks/starrocks/pull/66737) * Added an FE configuration `enable_statistic_collect_on_update` to control whether UPDATE statements can trigger automatic statistics collection. [#66794](https://github.com/StarRocks/starrocks/pull/66794) * Supports configuring `networkaddress.cache.ttl`. [#66723](https://github.com/StarRocks/starrocks/pull/66723) * Improve the “no rows imported” error message. [#66624](https://github.com/StarRocks/starrocks/pull/66624) [#66535](https://github.com/StarRocks/starrocks/pull/66535) * Optimized `deltaRows` with lazy evaluation for large partition tables. [#66381](https://github.com/StarRocks/starrocks/pull/66381) * Optimized materialized view rewrite performance. [#66623](https://github.com/StarRocks/starrocks/pull/66623) * Supports single-tablet `ResultSink` optimization in shared-data clusters. [#66517](https://github.com/StarRocks/starrocks/pull/66517) * `rewrite``_``simple``_``agg``_``to``_``meta``_``scan` is enabled by default. [#64698](https://github.com/StarRocks/starrocks/pull/64698) * Supports pushing down GROUP BY expressions and materialized view rewrite. [#66507](https://github.com/StarRocks/starrocks/pull/66507) * Add overloaded `newMessage` methods to improve materialized view logs. [#66367](https://github.com/StarRocks/starrocks/pull/66367) ##### Bug Fixes[​](#bug-fixes-10 "Direct link to Bug Fixes") The following issues have been fixed: * A Publish Compaction crash when the input rowset is not found. [#67154](https://github.com/StarRocks/starrocks/pull/67154) * Significant CPU overhead and lock contention caused by repetitive invocation of `update_segment_cache_size` when querying tables with a large number of columns. [#66714](https://github.com/StarRocks/starrocks/pull/66714) * `MulticastSinkOperator` stuck in the `OUTPUT_FULL` state. [#67153](https://github.com/StarRocks/starrocks/pull/67153) * A “column not found” issue in the skew join hint. [#66929](https://github.com/StarRocks/starrocks/pull/66929) * The growth of all tablets continues unabated, and the sum of pending and running tablets is not the total number of tablets. [#66718](https://github.com/StarRocks/starrocks/pull/66718) * Transactions in the Compaction map built during Leader startup cannot be accessed by CompactionScheduler and will never be removed from the map. [#66533](https://github.com/StarRocks/starrocks/pull/66533) * Delta Lake table refresh does not take effect. [#67156](https://github.com/StarRocks/starrocks/pull/67156) * CN crash at queries against non-partitioned Iceberg tables with DATE predicates. [#66864](https://github.com/StarRocks/starrocks/pull/66864) * Statements in Profiles cannot be correctly displayed when multiple statements are submitted. [#67097](https://github.com/StarRocks/starrocks/pull/67097) * Missing dictionary information during collection because Meta Reader does not support reading from Delta column group files. [#66995](https://github.com/StarRocks/starrocks/pull/66995) * Potential Java heap OOM in Java UDAF. [#67025](https://github.com/StarRocks/starrocks/pull/67025) * BE crash due to the incorrect logic of ranking window optimization without PARTITION BY and ORDER BY. [#67081](https://github.com/StarRocks/starrocks/pull/67081) * Misleading log level for timezone cache miss. [#66817](https://github.com/StarRocks/starrocks/pull/66817) * Crash and incorrect results caused by the incorrect `can_use_bf` checking when merging runtime filters. [#67021](https://github.com/StarRocks/starrocks/pull/67021) * Issue about pushing down runtime bitset filter with other OR predicates. [#66996](https://github.com/StarRocks/starrocks/pull/66996) * Patch critical fix from lz4. [#67053](https://github.com/StarRocks/starrocks/pull/67053) * AsyncTaskQueue deadlock issue. [#66791](https://github.com/StarRocks/starrocks/pull/66791) * Cache inconsistency in ObjectColumn. [#66957](https://github.com/StarRocks/starrocks/pull/66957) * RewriteUnnestBitmapRule causes wrong output column types. [#66855](https://github.com/StarRocks/starrocks/pull/66855) * Data races and data loss when there are WRITE or FLUSH tasks after FINISH tasks in the Delta Writer. [#66943](https://github.com/StarRocks/starrocks/pull/66943) * Invalid load channel and misleading internal errors caused by reopened load channels that were previously aborted. [#66793](https://github.com/StarRocks/starrocks/pull/66793) * Bugs of Arrow Flight SQL. [#65889](https://github.com/StarRocks/starrocks/pull/65889) * Issues when querying renamed columns with MetaScan. [#66819](https://github.com/StarRocks/starrocks/pull/66819) * Hash column is not removed before flushing chunk in partitionwise spillable aggregation when skew elimination is off. [#66839](https://github.com/StarRocks/starrocks/pull/66839) * BOOLEAN type default values were not correctly handled when stored as string literals. [#66818](https://github.com/StarRocks/starrocks/pull/66818) * decimal2decimal cast unexpectedly returns the input column as the result directly. [#66773](https://github.com/StarRocks/starrocks/pull/66773) * NPE in query planning during schema change. [#66811](https://github.com/StarRocks/starrocks/pull/66811) * LocalTabletsChannel and LakeTabletsChannel deadlock. [#66748](https://github.com/StarRocks/starrocks/pull/66748) * `publish_version` log shows empty `txn_ids` with new FE. [#66732](https://github.com/StarRocks/starrocks/pull/66732) * Incorrect behavior of the FE configuration `statistic_collect_query_timeout`. [#66363](https://github.com/StarRocks/starrocks/pull/66363) * UPDATE statements do not support statistics collection. [#66443](https://github.com/StarRocks/starrocks/pull/66443) * Case rewrite errors related to low cardinality. [#66724](https://github.com/StarRocks/starrocks/pull/66724) * Statistics query failure when the column list is empty. [#66138](https://github.com/StarRocks/starrocks/pull/66138) * Usage/record mismatch when switching warehouse via hint. [#66677](https://github.com/StarRocks/starrocks/pull/66677) * `ANALYZE TABLE` statements lack ExecTimeout. [#66361](https://github.com/StarRocks/starrocks/pull/66361) * `array_map` returns wrong results from constant unary expressions. [#66514](https://github.com/StarRocks/starrocks/pull/66514) * Foreign key constraints are lost after FE restart. [#66474](https://github.com/StarRocks/starrocks/pull/66474) * `max(not null string)` on empty table throws `std::length_error`. [#66554](https://github.com/StarRocks/starrocks/pull/66554) * Concurrency issue between Primary Key index Compaction and Apply. [#66282](https://github.com/StarRocks/starrocks/pull/66282) * Improper behavior of `EXPLAIN `. [#66542](https://github.com/StarRocks/starrocks/pull/66542) * Issue when sinking DECIMAL128 to Iceberg table column. [#66071](https://github.com/StarRocks/starrocks/pull/66071) * JSON length check issue for JSON → CHAR/VARCHAR when the target length equals the minimum. [#66628](https://github.com/StarRocks/starrocks/pull/66628) * An expression children count error. [#66511](https://github.com/StarRocks/starrocks/pull/66511) #### 3.5.10[​](#3510 "Direct link to 3.5.10") Release date: December 15, 2025 ##### Improvements[​](#improvements-11 "Direct link to Improvements") * Supports dumping plan node IDs in BE crash logs to speed up locating problematic operators. [#66454](https://github.com/StarRocks/starrocks/pull/66454) * Optimized scans on the system views in `information_schema` to reduce the overhead. [#66200](https://github.com/StarRocks/starrocks/pull/66200) * Added two histogram metrics (`slow_lock_held_time_ms` and `slow_lock_wait_time_ms`) to provide better observability for slow lock scenarios and distinguish between long-held locks and high lock contention. [#66027](https://github.com/StarRocks/starrocks/pull/66027) * Optimized replica lock handling in tablet report and clone flows by switching the lock from database level to table level, reducing lock contention and improving scheduling efficiency. [#61939](https://github.com/StarRocks/starrocks/pull/61939) * Avoided outputting columns in BE storage, and pushed down predicate computation to BE storage. [#60462](https://github.com/StarRocks/starrocks/pull/60462) * Improved query profile accuracy when deploying scan ranges in background threads. [#62223](https://github.com/StarRocks/starrocks/pull/62223) * Improved profile accounting when deploying additional tasks, so CPU time is not repetitively counted. [#62186](https://github.com/StarRocks/starrocks/pull/62186) * Added more detailed error messages when a referenced partition does not exist, making failures easier to diagnose. [#65674](https://github.com/StarRocks/starrocks/pull/65674) * Made sample-type cardinality estimation more robust in corner cases to improve row-count estimates. [#65599](https://github.com/StarRocks/starrocks/pull/65599) * Added a partition filter when loading statistics to prevent INSERT OVERWRITE from reading stale partition statistics. [#65578](https://github.com/StarRocks/starrocks/pull/65578) * Splited pipeline CPU `execution_time` metrics into separate series for queries and loads, improving observability by workload type. [#65535](https://github.com/StarRocks/starrocks/pull/65535) * Supported `enable_statistic_collect_on_first_load` at table granularity for finer-grained control over statistics collection on the first load. [#65463](https://github.com/StarRocks/starrocks/pull/65463) * Renamed the S3-dependent unit test from `PocoClientTest` to an S3-specific name to better reflect its dependency and intent. [#65524](https://github.com/StarRocks/starrocks/pull/65524) ##### Bug Fixes[​](#bug-fixes-11 "Direct link to Bug Fixes") The following issues have been fixed: * libhdfs crashes when StarRocks is started with an incompatible JDK. [#65882](https://github.com/StarRocks/starrocks/pull/65882) * Incorrect query results caused by `PartitionColumnMinMaxRewriteRule`. [#66356](https://github.com/StarRocks/starrocks/pull/66356) * Rewrite issues due to the materialized view metadata not refreshed when resolving materialized views by AST keys. [#66472](https://github.com/StarRocks/starrocks/pull/66472) * The trim function crashes or produces wrong results when trimming specific Unicode whitespace characters. [#66428](https://github.com/StarRocks/starrocks/pull/66428), [#66477](https://github.com/StarRocks/starrocks/pull/66477) * Failures in load metadata and SQL execution that still referenced a deleted warehouse. [#66436](https://github.com/StarRocks/starrocks/pull/66436) * Wrong results when group execution Join is combined with window functions. [#66441](https://github.com/StarRocks/starrocks/pull/66441) * A possible FE null pointer in `resetDecommStatForSingleReplicaTabletUnlocked`. [#66034](https://github.com/StarRocks/starrocks/pull/66034) * Missing Join runtime filter pushdown optimization in shared-data clusters for `LakeDataSource`. [#66354](https://github.com/StarRocks/starrocks/pull/66354) * Parameters are inconsistent for runtime filter transmit options (timeouts, HTTP RPC limits, etc.) because they are not forwarded to receivers. [#66393](https://github.com/StarRocks/starrocks/pull/66393) * Automatic partition creation fails when partition values already exist. [#66167](https://github.com/StarRocks/starrocks/pull/66167) * Inaccurate scan statistics in audit logs when predicates have high selectivity. [#66280](https://github.com/StarRocks/starrocks/pull/66280) * Incorrect query results because non-deterministic functions are pushed below operators. [#66323](https://github.com/StarRocks/starrocks/pull/66323) * Exponential growth in the number of expressions caused by CASE WHEN. [#66324](https://github.com/StarRocks/starrocks/pull/66324) * Materialized view compensation bugs when the same table appears multiple times in a query with different partition predicates. [#66369](https://github.com/StarRocks/starrocks/pull/66369) * BE becomes unresponsive when using fork in subprocesses. [#66334](https://github.com/StarRocks/starrocks/pull/66334) * CVE-2025-66566 and CVE-2025-12183. [#66453](https://github.com/StarRocks/starrocks/pull/66453), [#66362](https://github.com/StarRocks/starrocks/pull/66362) * Errors caused by nested CTE reuse. [#65800](https://github.com/StarRocks/starrocks/pull/65800) * Issues due to the lack of validation on conflicting schema-change clauses. [#66208](https://github.com/StarRocks/starrocks/pull/66208) * Improper rowset GC behavior when rowset commit fails. [#66301](https://github.com/StarRocks/starrocks/pull/66301) * A potential use-after-free when counting down pipelines. [#65940](https://github.com/StarRocks/starrocks/pull/65940) * The `W``arehouse` field is NULL in `information_schema.loads` for Stream Load. [#66202](https://github.com/StarRocks/starrocks/pull/66202) * Issues with materialized view creation when the referenced view has the same name as its base table. [#66274](https://github.com/StarRocks/starrocks/pull/66274) * The global dictionary is not updated correctly under some cases. [#66194](https://github.com/StarRocks/starrocks/pull/66194) * Incorrect query profile logging for queries forwarded from Follower nodes. [#64395](https://github.com/StarRocks/starrocks/pull/64395) * BE crash when caching SELECT results and reordering schema. [#65850](https://github.com/StarRocks/starrocks/pull/65850) * Shadow partitions are dropped when dropping partitions by expression. [#66171](https://github.com/StarRocks/starrocks/pull/66171) * DROP tasks run when a CLONE task exists for the same tablet. [#65780](https://github.com/StarRocks/starrocks/pull/65780) * Stability and observability issues because RocksDB log file options were not properly set. [#66166](https://github.com/StarRocks/starrocks/pull/66166) * Incorrect materialized view compensation that could produce NULL results. [#66216](https://github.com/StarRocks/starrocks/pull/66216) * BE reported as alive even after receiving `SIGSEGV`. [#66212](https://github.com/StarRocks/starrocks/pull/66212) * Bugs in Iceberg scans. [#65658](https://github.com/StarRocks/starrocks/pull/65658) * Regression coverage and stability issues for Iceberg view SQL test cases. [#66126](https://github.com/StarRocks/starrocks/pull/66126) * Unexpected behavior because `set_collector` is invoked repetitively. [#66199](https://github.com/StarRocks/starrocks/pull/66199) * Ingestion failures when column-mode partial updates are used together with conditional updates. [#66139](https://github.com/StarRocks/starrocks/pull/66139) * Temporary partition value conflicts under concurrent transactions. [#66025](https://github.com/StarRocks/starrocks/pull/66025) * An Iceberg table cache bug where Guava LocalCache could retain stale entries even when `cache.size() == 0`, causing refresh to be ineffective and queries to return outdated tables. [#65917](https://github.com/StarRocks/starrocks/pull/65917) * Incorrect format placeholder in `LargeInPredicateException`, causing the actual number of LargeInPredicate occurrences to be incorrectly reported in the error message. [#66152](https://github.com/StarRocks/starrocks/pull/66152) * NullPointerException in `ConnectScheduler’s` timeout checker when connectContext is null. [#66136](https://github.com/StarRocks/starrocks/pull/66136) * Crashes caused by unhandled exceptions thrown from threadpool tasks. [#66114](https://github.com/StarRocks/starrocks/pull/66114) * Data loss when pushing down DISTINCT LIMIT in certain plans. [#66109](https://github.com/StarRocks/starrocks/pull/66109) * `multi_distinct_count` not updating `distinct_size` after the underlying hash set is converted to a two-level hash set, which could lead to incorrect distinct counts. [#65916](https://github.com/StarRocks/starrocks/pull/65916) * A race condition when an exec group submits the next driver which could trigger `Check failed: !driver->is_in_blocked() `and abort the BE process. [#66099](https://github.com/StarRocks/starrocks/pull/66099) * INSERT failures when running ALTER TABLE ADD COLUMN with a default value concurrently with INSERT, due to mismatched types for the newly added column’s default expression. [#65968](https://github.com/StarRocks/starrocks/pull/65968) * An issue where `MemoryScratchSinkOperator` could remain in `pending_finish` after `RecordBatchQueue` was shut down when SparkSQL exited early, causing the pipeline to hang. [#66041](https://github.com/StarRocks/starrocks/pull/66041) * A core dump when reading Parquet files that contain empty row groups. [#65928](https://github.com/StarRocks/starrocks/pull/65928) * Recursive calls and potential stack overflow at high DOP because the event scheduler’s readiness check is complicated. [#66016](https://github.com/StarRocks/starrocks/pull/66016) * Asynchronous materialized view refresh skips updates when the Iceberg base table contains expired snapshots. [#65969](https://github.com/StarRocks/starrocks/pull/65969) * Potential issues in predicate reuse and rewrite because the optimizer relies solely on hashCode to distinguish differences in predicates. [#65999](https://github.com/StarRocks/starrocks/pull/65999) * In the refresh of an asynchronous materialized view with multi-level partitioned base tables, only the parent partition metadata was checked, while sub-partition updates were skipped. [#65596](https://github.com/StarRocks/starrocks/pull/65596) * Statistics collection issues where AVG(ARRAY\_LENGTH(...)) could return NULL for empty result sets. [#65788](https://github.com/StarRocks/starrocks/pull/65788) * Runtime profile counters are not correctly updating or clearing their min/max values during incremental updates on both BE and FE. [#65869](https://github.com/StarRocks/starrocks/pull/65869) * Incorrect logic to obtain the image journal ID when creating a cluster snapshot to ensure the snapshot uses the correct log position. [#65970](https://github.com/StarRocks/starrocks/pull/65970) * Results are misreported when the cleanup fails due to incorrect status checking logic in file cleanup error handling. [#65709](https://github.com/StarRocks/starrocks/pull/65709) * A possible infinite loop in certain plans isVariable() in DictMappingOperator. [#65743](https://github.com/StarRocks/starrocks/pull/65743) * Failures and missing audit/profile data because ConnectContext is not passed into scan-range deployment threads. [#63544](https://github.com/StarRocks/starrocks/pull/63544) * A use-after-free issue in the local Primary Key index manager when the storage engine is stopped. [#65534](https://github.com/StarRocks/starrocks/pull/65534) * Statistics collection issues for INSERT OVERWRITE with dynamic overwrite. [#65657](https://github.com/StarRocks/starrocks/pull/65657) * Concurrency issues caused by coarse-grained locks in DiskAndTabletLoadReBalancer. [#65557](https://github.com/StarRocks/starrocks/pull/65557) * Slow locks cannot be detected and reported correctly for the lack of slow-lock detection for critical locks. [#65559](https://github.com/StarRocks/starrocks/pull/65559) * NullPointerException when replaying upsert transaction state after the target database has been dropped. [#65595](https://github.com/StarRocks/starrocks/pull/65595) * Stale statistics were used because outdated partition statistics are not dropped after statistics collection triggered by INSERT OVERWRITE. [#65586](https://github.com/StarRocks/starrocks/pull/65586) * Data race in partition ID allocation that could lead to ID conflicts under concurrency. [#65608](https://github.com/StarRocks/starrocks/pull/65608) * Missing tablet IDs when retrieving initial tablet metadata. [#65550](https://github.com/StarRocks/starrocks/pull/65550) * Incorrect record information for PREPARE/EXECUTE statements in audit and profile logs. [#65448](https://github.com/StarRocks/starrocks/pull/65448) * Potential crashes because the non–thread-safe has\_output function is called from multiple threads. [#65514](https://github.com/StarRocks/starrocks/pull/65514) * MemTable finalize tasks cannot be properly tracked because the `memtable_finalize_task_total` counter metric is lacking. [#65548](https://github.com/StarRocks/starrocks/pull/65548) * Query ID collisions in Arrow Flight, causing multiple queries no longer share the same query ID. [#65558](https://github.com/StarRocks/starrocks/pull/65558) * Lock conflicts for `TabletChecker.doCheck()` with other operations. [#65237](https://github.com/StarRocks/starrocks/pull/65237) * Scan behavior is inconsistent between shared-data and shared-nothing clusters, causing query semantics to differ. [#61100](https://github.com/StarRocks/starrocks/pull/61100) #### 3.5.9[​](#359 "Direct link to 3.5.9") Release date: November 26, 2025 ##### Improvements[​](#improvements-12 "Direct link to Improvements") * Added transaction latency metrics to FE for observing timing across transaction stages. [#64948](https://github.com/StarRocks/starrocks/pull/64948) * Supports overwriting S3 unpartitioned Hive tables to simplify full-table rewrites in data lake scenarios. [#65340](https://github.com/StarRocks/starrocks/pull/65340) * Introduced CacheOptions to provide finer-grained control over tablet metadata caching. [#65222](https://github.com/StarRocks/starrocks/pull/65222) * Supports sample statistics collection for INSERT OVERWRITE to ensure statistics stay consistent with the latest data. [#65363](https://github.com/StarRocks/starrocks/pull/65363) * Optimized the statistics collection strategy after INSERT OVERWRITE to avoid missing or incorrect statistics due to asynchronous tablet reports. [#65327](https://github.com/StarRocks/starrocks/pull/65327) * Introduced a retention period for partitions dropped or replaced by INSERT OVERWRITE or materialized view refresh operations, keeping them in the recycle bin for a while to improve recoverability. [#64779](https://github.com/StarRocks/starrocks/pull/64779) ##### Bug Fixes[​](#bug-fixes-12 "Direct link to Bug Fixes") The following issues have been fixed: * Lock contention and concurrency issues related to `LocalMetastore.truncateTable()`. [#65191](https://github.com/StarRocks/starrocks/pull/65191) * Lock contention and replica check performance issues related to TabletChecker. [#65312](https://github.com/StarRocks/starrocks/pull/65312) * Incorrect error logging when changing user via HTTP SQL. [#65371](https://github.com/StarRocks/starrocks/pull/65371) * Checksum failures caused by DelVec CRC32 upgrade compatibility issues. [#65442](https://github.com/StarRocks/starrocks/pull/65442) * Tablet metadata load failures caused by RocksDB iteration timeout. [#65146](https://github.com/StarRocks/starrocks/pull/65146) * When the internal `flat_path` string is empty because the JSON hyper extraction path is `$` or all paths are skipped, calling `substr` will throw an exception and cause BE crash. [#65260](https://github.com/StarRocks/starrocks/pull/65260) * The PREPARED flag in fragment execution is not correctly set. [#65423](https://github.com/StarRocks/starrocks/pull/65423) * Inaccurate write and flush metrics caused by duplicated load profile counters. [#65252](https://github.com/StarRocks/starrocks/pull/65252) * When multiple HTTP requests reuse the same TCP connection, if a non‑ExecuteSQL request arrives after an ExecuteSQL request, the `HttpConnectContext` cannot be unregistered at channel close, causing HTTP context leaks. [#65203](https://github.com/StarRocks/starrocks/pull/65203) * MySQL 8.0 schema introspection errors (Fixed by adding session variables `default_authentication_plugin` and `authentication_policy`). [#65330](https://github.com/StarRocks/starrocks/pull/65330) * SHOW ANALYZE STATUS errors caused by unnecessary statistics collection for temporary partitions created after partition overwrite operations. [#65298](https://github.com/StarRocks/starrocks/pull/65298) * Global Runtime Filter race in the Event Scheduler. [#65200](https://github.com/StarRocks/starrocks/pull/65200) * Data Cache is aggressively disabled because the minimum Data Cache disk size constraint is too large. [#64909](https://github.com/StarRocks/starrocks/pull/64909) * An aarch64 build issue related to the `gold` linker automatic fallback. [#65156](https://github.com/StarRocks/starrocks/pull/65156) #### 3.5.8[​](#358 "Direct link to 3.5.8") Release date: November 10, 2025 ##### Improvements[​](#improvements-13 "Direct link to Improvements") * Upgraded Arrow to 19.0.1 to support the Parquet legacy list to include nested, complex files. [#64238](https://github.com/StarRocks/starrocks/pull/64238) * FILES() supports legacy Parquet LIST encodings. [#64160](https://github.com/StarRocks/starrocks/pull/64160) * Automatically determine the Partial Update mode based on the session variable and the number of inserted columns. [#62091](https://github.com/StarRocks/starrocks/pull/62091) * Applied low-cardinality optimization on analytic operators above table functions. [#63378](https://github.com/StarRocks/starrocks/pull/63378) * Added configurable table lock timeout to `finishTransaction` to avoid blocking. [#63981](https://github.com/StarRocks/starrocks/pull/63981) * Shared-data clusters support table-level scan metrics attribution. [#62832](https://github.com/StarRocks/starrocks/pull/62832) * Window functions LEAD/LAG/FIRST\_VALUE/LAST\_VALUE now accept ARRAY type arguments. [#63547](https://github.com/StarRocks/starrocks/pull/63547) * Supports constant folding for several array functions to improve predicate pushdown and join simplification. [#63692](https://github.com/StarRocks/starrocks/pull/63692) * Supports batched API to optimize `tabletNum` retrieval for a given node via `SHOW PROC /backends/{id}`. Added an FE configuration item `enable_collect_tablet_num_in_show_proc_backend_disk_path` (Default: `true`). [#64013](https://github.com/StarRocks/starrocks/pull/64013) * Ensured `INSERT ... SELECT` reads the freshest metadata by refreshing external tables before planning. [#64026](https://github.com/StarRocks/starrocks/pull/64026) * Added `capacity_limit_reached` checks to table functions, NL-join probe, and hash-join probe to avoid constructing overflowing columns. [#64009](https://github.com/StarRocks/starrocks/pull/64009) * Added FE configuration item `collect_stats_io_tasks_per_connector_operator` (Default: `4`) for setting the maximum number of tasks to collect statistics for external tables. [#64016](https://github.com/StarRocks/starrocks/pull/64016) * Updated the default partition size for sample collection from 1000 to 300. [#64022](https://github.com/StarRocks/starrocks/pull/64022) * Increased lock table slots to 256 and added `rid` to slow-lock logs. [#63945](https://github.com/StarRocks/starrocks/pull/63945) * Improved robustness of Gson deserialization in the presence of legacy data. [#63555](https://github.com/StarRocks/starrocks/pull/63555) * Reduced metadata lock scope for FILES() schema pushdown to cut lock contention and planning latency. [#63796](https://github.com/StarRocks/starrocks/pull/63796) * Added Task Run execute timeout checker by introducing an FE configuration item `task_runs_timeout_second`, and refined cancellation logics for overdue runs. [#63842](https://github.com/StarRocks/starrocks/pull/63842) * Ensured `REFRESH MATERIALIZED VIEW ... FORCE` always refreshes target partitions (even in inconsistent or corrupted cases). [#63844](https://github.com/StarRocks/starrocks/pull/63844) ##### Bug Fixes[​](#bug-fixes-13 "Direct link to Bug Fixes") The following issues have been fixed: * An exception when parsing the Nullable (Decimal) type of ClickHouse. [#64195](https://github.com/StarRocks/starrocks/pull/64195) * An issue with tablet migration and Primary Key index lookup concurrency. [#64164](https://github.com/StarRocks/starrocks/pull/64164) * Lack of FINISHED status in materialized view refresh. [#64191](https://github.com/StarRocks/starrocks/pull/64191) * Schema Change Publish does not retry in shared-data clusters. [#64093](https://github.com/StarRocks/starrocks/pull/64093) * Wrong row count statistics on Primary Key tables in Data Lake. [#64007](https://github.com/StarRocks/starrocks/pull/64007) * When tablet creation times out in shared-data clusters, node information cannot be returned. [#63963](https://github.com/StarRocks/starrocks/pull/63963) * Corrupted Lake DataCache cannot be cleared. [#63182](https://github.com/StarRocks/starrocks/pull/63182) * Window function with IGNORE NULLS flags can not be consolidated with its counterpart without iIGNORE NULLS flag. [#63958](https://github.com/StarRocks/starrocks/pull/63958) * Table compaction cannot be scheduled again after FE restart if the compaction was previously aborted. [#63881](https://github.com/StarRocks/starrocks/pull/63881) * Tasks fail to be scheduled if FE restarts frequently. [#63966](https://github.com/StarRocks/starrocks/pull/63966) * An issue with GCS error codes. [#64066](https://github.com/StarRocks/starrocks/pull/64066) * Instability issue with StarMgr gRPC executor. [#63828](https://github.com/StarRocks/starrocks/pull/63828) * Deadlock when creating an exclusive work group. [#63893](https://github.com/StarRocks/starrocks/pull/63893) * Cache for Iceberg tables is not properly invalidated. [#63971](https://github.com/StarRocks/starrocks/pull/63971) * Wrong results for sorted aggregation in shared-data clusters. [#63849](https://github.com/StarRocks/starrocks/pull/63849) * ASAN error in `PartitionedSpillerWriter::_remove_partition`. [#63903](https://github.com/StarRocks/starrocks/pull/63903) * BE crash when failing to get splits from morsel queue. [#62753](https://github.com/StarRocks/starrocks/pull/62753) * A bug with aggregate push-down type cast in materialized view rewrite. [#63875](https://github.com/StarRocks/starrocks/pull/63875) * NPE when removing expired load jobs in FE. [#63820](https://github.com/StarRocks/starrocks/pull/63820) * Partitioned Spill crash when removing partitions. [#63825](https://github.com/StarRocks/starrocks/pull/63825) * Materialized view rewrite throws `IllegalStateException` under certain plans. [#63655](https://github.com/StarRocks/starrocks/pull/63655) * NPE when creating a partitioned materialized view. [#63830](https://github.com/StarRocks/starrocks/pull/63830) #### 3.5.7[​](#357 "Direct link to 3.5.7") Release date: October 21, 2025 ##### Improvements[​](#improvements-14 "Direct link to Improvements") * Improved memory statistics accuracy for Scan operators by introducing retry backoff under heavy memory contention scenarios. [#63788](https://github.com/StarRocks/starrocks/pull/63788) * Optimized materialized view bucketing inference by leveraging existing tablet distribution to prevent excessive bucket creation. [#63367](https://github.com/StarRocks/starrocks/pull/63367) * Revised the Iceberg table caching mechanism to enhance consistency and reduce cache invalidation risks during frequent metadata updates. [#63388](https://github.com/StarRocks/starrocks/pull/63388) * Added the `querySource` field to `QueryDetail` and `AuditEvent` for better traceability of query origins across APIs and schedulers. [#63480](https://github.com/StarRocks/starrocks/pull/63480) * Enhanced Persistent Index diagnostics by printing detailed context when duplicate keys are detected in MemTable writes. [#63560](https://github.com/StarRocks/starrocks/pull/63560) * Reduced lock contention in materialized view operations by refining lock granularity and sequencing in concurrent scenarios. [#63481](https://github.com/StarRocks/starrocks/pull/63481) ##### Bug Fixes[​](#bug-fixes-14 "Direct link to Bug Fixes") The following issues have been fixed: * Materialized view rewrite failures caused by type mismatch. [#63659](https://github.com/StarRocks/starrocks/pull/63659) * `regexp_extract_all` has wrong behavior and lacks support for `pos=0`. [#63626](https://github.com/StarRocks/starrocks/pull/63626) * Degraded scan performance caused by the profitless simplification of CASE WHEN with complex functions. [#63732](https://github.com/StarRocks/starrocks/pull/63732) * Incorrect DCG data reading when partial updates switch from column mode to row mode. [#61529](https://github.com/StarRocks/starrocks/pull/61529) * A potential deadlock during initialization of `ExceptionStackContext`. [#63776](https://github.com/StarRocks/starrocks/pull/63776) * Crashes in Parquet numeric conversion for ARM architecture machines. [#63294](https://github.com/StarRocks/starrocks/pull/63294) * An issue caused by the aggregate intermediate type uses `ARRAY`. [#63371](https://github.com/StarRocks/starrocks/pull/63371) * Stability issue caused by incorrect overflow detection when casting LARGEINT to DECIMAL128 at sign-edge cases (for example, INT128\_MIN) [#63559](https://github.com/StarRocks/starrocks/pull/63559) * LZ4 compression and decompression errors cannot be perceived. [#63629](https://github.com/StarRocks/starrocks/pull/63629) * `ClassCastException` when querying tables partitioned by `FROM_UNIXTIME` on INT-type columns. [#63684](https://github.com/StarRocks/starrocks/pull/63684) * Tablets cannot be repaired after a balance-triggered migration when the only valid source replica is marked `DECOMMISSION`. [#62942](https://github.com/StarRocks/starrocks/pull/62942) * Profiles lost SQL statements and Planner Trace when the PREPARE statement is used. [#63519](https://github.com/StarRocks/starrocks/pull/63519) * The `extract_number`, `extract_bool`, and `extract_string` functions are not exception-safe. [#63575](https://github.com/StarRocks/starrocks/pull/63575) * Shutdown tablets cannot be garbage-collected properly. [#63595](https://github.com/StarRocks/starrocks/pull/63595) * Profiles showing SQL as `omit` for returns of the PREPARE/EXECUTE statements. [#62988](https://github.com/StarRocks/starrocks/pull/62988) * `date_trunc` partition pruning with combined predicates that mistakenly produced EMPTYSET. [#63464](https://github.com/StarRocks/starrocks/pull/63464) * Crashes in release builds due to the CHECK in NullableColumn. [#63553](https://github.com/StarRocks/starrocks/pull/63553) #### 3.5.6[​](#356 "Direct link to 3.5.6") Release date: September 22, 2025 ##### Improvements[​](#improvements-15 "Direct link to Improvements") * A decommissioned BE will be forcibly dropped when all its tablets are in the recycle bin, to avoid the decommission being blocked by those tablets. [#62781](https://github.com/StarRocks/starrocks/pull/62781) * Vacuum metrics will be updated when Vacuum succeeds. [#62540](https://github.com/StarRocks/starrocks/pull/62540) * Added thread pool metrics to the fragment instance execution state report, including active threads, queue count, and running threads. [#63067](https://github.com/StarRocks/starrocks/pull/63067) * Supports S3 path-style access in shared-data clusters to improve compatibility with MinIO and other S3-compatible storage systems. You can enable this feature by setting `aws.s3.enable_path_style_access` to `true` when creating a storage volume. [#62591](https://github.com/StarRocks/starrocks/pull/62591) * Supports resetting the starting point of the AUTO\_INCREMENT value via `ALTER TABLE`` `` AUTO_INCREMENT`` = 10000;`. [#62767](https://github.com/StarRocks/starrocks/pull/62767) * Supports using Distinguished Name (DN) in Group Provider for group matching, improving the user group solution for LDAP/Microsoft Active Directory environments. [#62711](https://github.com/StarRocks/starrocks/pull/62711) * Supports Azure Workload Identity authentication for Azure Data Lake Storage Gen2. [#62754](https://github.com/StarRocks/starrocks/pull/62754) * Added transaction error messages to the `information_schema.``loads` view to aid failure diagnosis. [#61364](https://github.com/StarRocks/starrocks/pull/61364) * Supports reusing common expressions for complex CASE WHEN expressions in Scan predicates to reduce repetitive computation. [#62779](https://github.com/StarRocks/starrocks/pull/62779) * Uses the REFRESH (instead of ALTER) privilege on the materialized view to execute REFRESH statements. [#62636](https://github.com/StarRocks/starrocks/pull/62636) * Disabled low-cardinality optimization on Lake tables by default to avoid potential issues. [#62586](https://github.com/StarRocks/starrocks/pull/62586) * Enabled tablet balancing between workers by default in shared-data clusters. [#62661](https://github.com/StarRocks/starrocks/pull/62661) * Supports reusing expressions in outer-join WHERE predicates to reduce repetitive computation. [#62139](https://github.com/StarRocks/starrocks/pull/62139) * Added Clone metrics in FE. [#62421](https://github.com/StarRocks/starrocks/pull/62421) * Added Clone metrics in BE. [#62479](https://github.com/StarRocks/starrocks/pull/62479) * Added an FE configuration item `enable_statistic_cache_refresh_after_write` to disable statistics-cache lazy refresh by default. [#62518](https://github.com/StarRocks/starrocks/pull/62518) * Masked credential information in SUBMIT TASK for better security. [#62311](https://github.com/StarRocks/starrocks/pull/62311) * `json_extract` in the Trino dialect returns a JSON type. [#59718](https://github.com/StarRocks/starrocks/pull/59718) * Supports ARRAY type in `null_or_empty`. [#62207](https://github.com/StarRocks/starrocks/pull/62207) * Adjusted the size limit for the Iceberg manifest cache. [#61966](https://github.com/StarRocks/starrocks/pull/61966) * Added a remote file-cache limit for Hive. [#62288](https://github.com/StarRocks/starrocks/pull/62288) ##### Bug Fixes[​](#bug-fixes-15 "Direct link to Bug Fixes") The following issues have been fixed: * Secondary replicas hang indefinitely due to negative timeout values, which cause incorrect timestamp comparisons. [#62805](https://github.com/StarRocks/starrocks/pull/62805) * PublishTask may be blocked when TransactionState is REPLICATION. [#61664](https://github.com/StarRocks/starrocks/pull/61664) * Incorrect repair mechanism for Hive tables that have been dropped and recreated during materialized view refresh. [#63072](https://github.com/StarRocks/starrocks/pull/63072) * Incorrect execution plans were generated after the materialized view aggregation push‑down rewrite. [#63060](https://github.com/StarRocks/starrocks/pull/63060) * ANALYZE PROFILE failures caused by PlanTuningGuide producing unrecognized strings (null explainString) in the query profiles. [#63024](https://github.com/StarRocks/starrocks/pull/63024) * Inappropriate return type of `hour_from_unixtime` and incorrect rewrite rule of `CAST`. [#63006](https://github.com/StarRocks/starrocks/pull/63006) * NPE in Iceberg manifest cache under data races. [#63043](https://github.com/StarRocks/starrocks/pull/63043) * Shared-data clusters lack support for colocation in materialized views. [#62941](https://github.com/StarRocks/starrocks/pull/62941) * Iceberg table Scan Exception during Scan Range deployment.[ #62994](https://github.com/StarRocks/starrocks/pull/62994) * Incorrect execution plans were generated for view-based rewrite. [#62918](https://github.com/StarRocks/starrocks/pull/62918) * Errors and disrupted tasks due to Compute Nodes are not gracefully shut down on exit. [#62916](https://github.com/StarRocks/starrocks/pull/62916) * NPE when Stream Load execution status updates. [#62921](https://github.com/StarRocks/starrocks/pull/62921) * An issue with statistics when the column name and the name in the PARTITION BY clause differ in case. [#62953](https://github.com/StarRocks/starrocks/pull/62953) * Wrong results are returned when the `LEAST` function is used as a predicate. [#62826](https://github.com/StarRocks/starrocks/pull/62826) * Invalid ProjectOperator above the table-pruning frontier CTEConsumer. [#62914](https://github.com/StarRocks/starrocks/pull/62914) * Redundant replica handling after Clone. [#62542](https://github.com/StarRocks/starrocks/pull/62542) * Failed to collect Stream Load profiles. [#62802](https://github.com/StarRocks/starrocks/pull/62802) * Ineffective disk rebalancing caused by improper BE selection. [#62776](https://github.com/StarRocks/starrocks/pull/62776) * A potential NPE crash in LocalTabletsChannel when a missing `tablet_id` leads to a null delta writer. [#62861](https://github.com/StarRocks/starrocks/pull/62861) * KILL ANALYZE does not take effect. [#62842](https://github.com/StarRocks/starrocks/pull/62842) * SQL syntax errors in histogram stats when MCV values contain single quotes. [#62853](https://github.com/StarRocks/starrocks/pull/62853) * Incorrect output format of metrics for Prometheus. [#62742](https://github.com/StarRocks/starrocks/pull/62742) * NPE when querying `information_schema.analyze_status` after the database is dropped. [#62796](https://github.com/StarRocks/starrocks/pull/62796) * CVE-2025-58056. [#62801](https://github.com/StarRocks/starrocks/pull/62801) * When SHOW CREATE ROUTINE LOAD is executed, wrong results are returned because the database is considered null if not specified. [#62745](https://github.com/StarRocks/starrocks/pull/62745) * Data loss caused by incorrectly skipping CSV headers in `files()`. [#62719](https://github.com/StarRocks/starrocks/pull/62719) * NPE when replaying batch-transaction upserts. [#62715](https://github.com/StarRocks/starrocks/pull/62715) * Publish being incorrectly reported as successful during graceful shutdown in shared-nothing clusters. [#62417](https://github.com/StarRocks/starrocks/pull/62417) * Crash in asynchronous delta writer due to a null pointer. [#62626](https://github.com/StarRocks/starrocks/pull/62626) * Materialized view refresh is skipped because the materialized view version map is not cleared after a failed restore job. [#62634](https://github.com/StarRocks/starrocks/pull/62634) * Issues caused by case-sensitive partition column validation in the materialized view analyzer. [#62598](https://github.com/StarRocks/starrocks/pull/62598) * Duplicate IDs for statements with syntax errors. [#62258](https://github.com/StarRocks/starrocks/pull/62258) * StatisticsExecutor status is overridden due to redundant state assignment in CancelableAnalyzeTask. [#62538](https://github.com/StarRocks/starrocks/pull/62538) * Incorrect error messages produced by statistics collection. [#62533](https://github.com/StarRocks/starrocks/pull/62533) * Premature throttling caused by insufficient default maximum connections for external users. [#62523](https://github.com/StarRocks/starrocks/pull/62523) * A potential NPE in materialized view backup and restore operations. [#62514](https://github.com/StarRocks/starrocks/pull/62514) * Incorrect `http_workers_num` metric. [#62457](https://github.com/StarRocks/starrocks/pull/62457) * The runtime filter fails to locate the corresponding execution group during construction. [#62465](https://github.com/StarRocks/starrocks/pull/62465) * Tedious results on Scan Node caused by simplifying CASE WHEN with complex functions. [#62505](https://github.com/StarRocks/starrocks/pull/62505) * `gmtime` is not thread-safe. [#60483](https://github.com/StarRocks/starrocks/pull/60483) * An issue with getting Hive partitions with escaped strings. [#59032](https://github.com/StarRocks/starrocks/pull/59032) #### 3.5.5[​](#355 "Direct link to 3.5.5") Release date: September 5, 2025 ##### Improvements[​](#improvements-16 "Direct link to Improvements") * Added a new system variable `enable_drop_table_check_mv_dependency` (default: `false`). When set to `true`, if the object to be dropped is referenced by a downstream materialized view, the system prevents the execution of `DROP TABLE` / `DROP VIEW` / `DROP MATERIALIZED VIEW`. The error message lists the dependent materialized views and suggests checking the `sys.object_dependencies` view for details. [#61584](https://github.com/StarRocks/starrocks/pull/61584) * Logs now include the Linux distribution and CPU architecture of the build, to facilitate issue reproduction and troubleshooting. Log format: `... build distro arch `. [#62017](https://github.com/StarRocks/starrocks/pull/62017) * Persisted per-Tablet index and incremental column group file sizes are now cached, replacing on-demand directory scans. This accelerates Tablet status reporting in BE and reduces latency under high I/O scenarios. [#61901](https://github.com/StarRocks/starrocks/pull/61901) * Downgraded several high-frequency INFO logs in FE and BE to VLOG, and aggregated task submission logs, significantly reducing redundant storage-related logs and log volume under heavy load. [#62121](https://github.com/StarRocks/starrocks/pull/62121) * Improved query performance for External Catalog metadata through `information_schema` by pushing table filters before calling `getTable`, avoiding per-table RPCs. [#62404](https://github.com/StarRocks/starrocks/pull/62404) ##### Bug Fixes[​](#bug-fixes-16 "Direct link to Bug Fixes") The following issues have been fixed: * NullPointerException when fetching partition-level column statistics during the Plan stage due to missing data. [#61935](https://github.com/StarRocks/starrocks/pull/61935) * Fixed Parquet write issues with non-empty NULL arrays, and corrected `SPLIT(NULL, …)` behavior to consistently return NULL, preventing data corruption and runtime errors. [#61999](https://github.com/StarRocks/starrocks/pull/61999) * Failure when creating materialized views using `CASE WHEN` expressions due to incompatible VARCHAR type returns (fixed by ensuring consistency before and after refresh, and introducing a new FE configuration `transform_type_prefer_string_for_varchar` to prefer STRING and avoid length mismatch). [#61996](https://github.com/StarRocks/starrocks/pull/61996) * Statistics for nested CTEs could not be computed outside of memo when `enable_rbo_table_prune` was `false`. [#62070](https://github.com/StarRocks/starrocks/pull/62070) * In Audit Logs, inaccurate Scan Rows results for INSERT INTO SELECT statements. [#61381](https://github.com/StarRocks/starrocks/pull/61381) * ExceptionInInitializerError/NullPointerException during initialization caused FE startup failure when Query Queue v2 was enabled. [#62161](https://github.com/StarRocks/starrocks/pull/62161) * BE crash when `LakePersistentIndex` initialization failed and `_memtable` cleanup was triggered. [#62279](https://github.com/StarRocks/starrocks/pull/62279) * Permission issues during materialized view refresh due to creator roles not being activated (fixed by adding FE configuration `mv_use_creator_based_authorization`. When set to `false`, materialized views are refreshed as root, for compatibility with LDAP-authenticated clusters). [#62396](https://github.com/StarRocks/starrocks/pull/62396) * Materialized view refresh failures caused by case-sensitive List partition table names (fixed by enforcing case-insensitive uniqueness checks on partition names, aligning with OLAP table semantics). [#62389](https://github.com/StarRocks/starrocks/pull/62389) #### 3.5.4[​](#354 "Direct link to 3.5.4") Release Date: August 22, 2025 ##### Improvements[​](#improvements-17 "Direct link to Improvements") * Added logs to clarify the reason that tablets cannot be repaired.  [#61959](https://github.com/StarRocks/starrocks/pull/61959) * Optimized DROP PARTITION information in logs.  [#61787](https://github.com/StarRocks/starrocks/pull/61787) * Assigned a large but configurable row count to tables with unknown stats for statistical estimation.  [#61332](https://github.com/StarRocks/starrocks/pull/61332) * Added balance statistic according to label location.  [#61905](https://github.com/StarRocks/starrocks/pull/61905) * Added colocate group balance statistics to improve cluster monitoring. [#61736](https://github.com/StarRocks/starrocks/pull/61736) * Skipped the Publish waiting phase when the number of healthy replicas exceeds the default replica count. [#61820](https://github.com/StarRocks/starrocks/pull/61820) * Included the tablet information collection time in the tablet report. [#61643](https://github.com/StarRocks/starrocks/pull/61643) * Supports writing Starlet files with tags. [ #61605](https://github.com/StarRocks/starrocks/pull/61605) * Supports viewing cluster balance statistics via SHOW PROC.  [#61578](https://github.com/StarRocks/starrocks/pull/61578) * Bumped librdkafka to 2.11.0 to support Kafka 4.0 and removed deprecated configurations.  [#61698](https://github.com/StarRocks/starrocks/pull/61698) * Added `prepared_timeout` configuration to Stream Load Transaction Interface.  [#61539](https://github.com/StarRocks/starrocks/pull/61539) * Upgraded StarOS to v3.5‑rc3.  [#61685](https://github.com/StarRocks/starrocks/pull/61685) ##### Bug Fixes[​](#bug-fixes-17 "Direct link to Bug Fixes") The following issues have been fixed: * Incorrect Dict version of random distribution tables. [#61933](https://github.com/StarRocks/starrocks/pull/61933) * Incorrect query context in context conditions. [#61929](https://github.com/StarRocks/starrocks/pull/61929) * Publish failures caused by synchronous Publish for shadow tablets during ALTER operations. [#61887](https://github.com/StarRocks/starrocks/pull/61887) * CVE‑2025‑55163 issue.  [#62041](https://github.com/StarRocks/starrocks/pull/62041) * Memory leak in real-time data ingestion from Apache Kafka.  [#61698](https://github.com/StarRocks/starrocks/pull/61698) * Incorrect count of rebuild files in the lake persistent index. [#61859](https://github.com/StarRocks/starrocks/pull/61859) * Statistics collection on generated expression columns causes cross-database query errors. [#61829](https://github.com/StarRocks/starrocks/pull/61829) * Query Cache misaligns in shared-nothing clusters, causing inconsistent results. [#61783](https://github.com/StarRocks/starrocks/pull/61783) * High memory usage in CatalogRecycleBin due to retaining deleted partition information.[#61582](https://github.com/StarRocks/starrocks/pull/61582) * SQL Server JDBC connections fail when the timeout exceeds 65,535 milliseconds. [#61719](https://github.com/StarRocks/starrocks/pull/61719) * Security Integration fails to encrypt passwords, exposing sensitive information. [#60666](https://github.com/StarRocks/starrocks/pull/60666) * `MIN()` and `MAX()` functions on Iceberg partition columns return NULL unexpectedly.  [#61858](https://github.com/StarRocks/starrocks/pull/61858) * Other predicates of Join containing non‑push‑down subfields were incorrectly rewritten.  [#61868](https://github.com/StarRocks/starrocks/pull/61868) * QueryContext cancellation can lead to a use‑after‑free situation.  [#61897](https://github.com/StarRocks/starrocks/pull/61897) * CBO’s table pruning overlooks other predicates.  [#61881](https://github.com/StarRocks/starrocks/pull/61881) * Partial Updates in `COLUMN_UPSERT_MODE` may overwrite auto-increment columns with zero.  [#61341](https://github.com/StarRocks/starrocks/pull/61341) * JDBC TIME type conversion uses an incorrect timezone offset that leads to wrong time values. [#61783](https://github.com/StarRocks/starrocks/pull/61783) * `max_filter_ratio` was not being serialized in Routine Load jobs. [#61755](https://github.com/StarRocks/starrocks/pull/61755) * Precision loss in the `now(precision)` function in Stream Load. [#61721](https://github.com/StarRocks/starrocks/pull/61721) * Cancelling a query may result in a “query id not found” error. [#61667](https://github.com/StarRocks/starrocks/pull/61667) * LDAP authentication may miss PartialResultException, causing incomplete query results.[ #60667](https://github.com/StarRocks/starrocks/pull/60667) * Paimon Timestamp timezone conversion issue when the query condition contains DATETIME.[ #60473](https://github.com/StarRocks/starrocks/pull/60473) #### 3.5.3[​](#353 "Direct link to 3.5.3") Release Date: August 11, 2025 ##### Improvements[​](#improvements-18 "Direct link to Improvements") * Lake Compaction adds Segment write time statistics. [#60891](https://github.com/StarRocks/starrocks/pull/60891) * Disable inline mode for Data Cache writes to avoid performance degradation. [#60530](https://github.com/StarRocks/starrocks/pull/60530) * Iceberg metadata scan supports shared file I/O. [#61012](https://github.com/StarRocks/starrocks/pull/61012) * Support termination of all PENDING ANALYZE tasks. [#61118](https://github.com/StarRocks/starrocks/pull/61118) * Force reuse when there are too many CTE nodes to avoid excessive optimization time. [#60983](https://github.com/StarRocks/starrocks/pull/60983) * Added `BALANCE` type to cluster balance results. [#61081](https://github.com/StarRocks/starrocks/pull/61081) * Optimized materialized view rewrite for external tables. [#61037](https://github.com/StarRocks/starrocks/pull/61037) * Default value of system variable `enable_materialized_view_agg_pushdown_rewrite` is changed to `true`, enabling aggregation pushdown for materialized view queries by default. [#60976](https://github.com/StarRocks/starrocks/pull/60976) * Optimized partition statistics lock competition. [#61041](https://github.com/StarRocks/starrocks/pull/61041) ##### Bug Fixes[​](#bug-fixes-18 "Direct link to Bug Fixes") The following issues have been fixed: * Inconsistent Chunk column size after column pruning. [#61271](https://github.com/StarRocks/starrocks/pull/61271) * Synchronous execution of partition statistics loading may cause deadlocks. [#61300](https://github.com/StarRocks/starrocks/pull/61300) * Crash when `array_map` processes constant array columns. [#61309](https://github.com/StarRocks/starrocks/pull/61309) * Setting an auto-increment column to NULL results in the system mistakenly rejecting valid data within the same Chunk. [#61255](https://github.com/StarRocks/starrocks/pull/61255) * The actual number of JDBC connections may exceed the `jdbc_connection_pool_size` limit. [#61038](https://github.com/StarRocks/starrocks/pull/61038) * FQDN mode did not use IP addresses as cache map keys. [#61203](https://github.com/StarRocks/starrocks/pull/61203) * Array column cloning error during array comparison. [#61036](https://github.com/StarRocks/starrocks/pull/61036) * Deploying serialized thread pool blockage led to query performance degradation. [#61150](https://github.com/StarRocks/starrocks/pull/61150) * OK hbResponse not synchronized after heartbeat retry counter reset. [#61249](https://github.com/StarRocks/starrocks/pull/61249) * Incorrect result for the `hour_from_unixtime` function. [#61206](https://github.com/StarRocks/starrocks/pull/61206) * Conflicts between ALTER TABLE jobs and partition creation. [#60890](https://github.com/StarRocks/starrocks/pull/60890) * Cache does not take effect after upgrading from v3.3 to v3.4 or later. [#60973](https://github.com/StarRocks/starrocks/pull/60973) * Vector index metric `hit_count` is not set. [#61102](https://github.com/StarRocks/starrocks/pull/61102) * Stream Load transactions fail to find the coordinator node. [#60154](https://github.com/StarRocks/starrocks/pull/60154) * BE crashes when loading OOM partitions. [#60778](https://github.com/StarRocks/starrocks/pull/60778) * INSERT OVERWRITE failed on manually created partitions. [#60750](https://github.com/StarRocks/starrocks/pull/60750) * Partition creation failed when partition names matched case-insensitively but had different values. [#60909](https://github.com/StarRocks/starrocks/pull/60909) * The system does not support PostgreSQL UUID type. [#61021](https://github.com/StarRocks/starrocks/pull/61021) * Case sensitivity issue with column names when loading Parquet data via `FILES()`. [#61059](https://github.com/StarRocks/starrocks/pull/61059) #### 3.5.2[​](#352 "Direct link to 3.5.2") Release Date: July 18, 2025 ##### Improvements[​](#improvements-19 "Direct link to Improvements") * Collected NDV (number of distinct values) statistics for ARRAY columns to improve query plan accuracy. [#60623](https://github.com/StarRocks/starrocks/pull/60623) * Disabled replica balancing for Colocate tables and tablet scheduling in Shared-data clusters to reduce unnecessary log output. [#60737](https://github.com/StarRocks/starrocks/pull/60737) * Optimized Catalog access workflow: FE now delays accessing external data sources asynchronously at startup to prevent hanging due to external service unavailability. [#60614](https://github.com/StarRocks/starrocks/pull/60614) * Added session variable `enable_predicate_expr_reuse` to control predicate pushdown. [#60603](https://github.com/StarRocks/starrocks/pull/60603) * Supports a retry mechanism when fetching Kafka partition information fails. [#60513](https://github.com/StarRocks/starrocks/pull/60513) * Removed the restriction requiring exact mapping of partition columns between materialized views and base tables. [#60565](https://github.com/StarRocks/starrocks/pull/60565) * Supports building Runtime In-Filters to enhance aggregation performance by filtering data during aggregation. [#59288](https://github.com/StarRocks/starrocks/pull/59288) ##### Bug Fixes[​](#bug-fixes-19 "Direct link to Bug Fixes") Fixed the following issues: * COUNT DISTINCT queries crash due to low-cardinality optimization for multiple columns. [#60664](https://github.com/StarRocks/starrocks/pull/60664) * Incorrect matching of global UDFs when multiple functions share the same name. [#60550](https://github.com/StarRocks/starrocks/pull/60550) * Null pointer exception (NPE) issue during Stream Load import. [#60755](https://github.com/StarRocks/starrocks/pull/60755) * Null pointer exception (NPE) issue when starting FE during a recovery from a cluster snapshot. [#60604](https://github.com/StarRocks/starrocks/pull/60604) * BE crash caused by column mode mismatch when processing short-circuit queries with out-of-order values. [#60466](https://github.com/StarRocks/starrocks/pull/60466) * Session variables set via PROPERTIES in SUBMIT TASK statements did not take effect. [#60584](https://github.com/StarRocks/starrocks/pull/60584) * Incorrect results for `SELECT min/max` queries under specific conditions. [#60601](https://github.com/StarRocks/starrocks/pull/60601) * Incorrect bucket pruning when the left side of a predicate is a function, leading to incorrect query results. [#60467](https://github.com/StarRocks/starrocks/pull/60467) * Crash for queries against a non-existent `query_id` via Arrow Flight SQL. [#60497](https://github.com/StarRocks/starrocks/pull/60497) ##### Behavior Changes[​](#behavior-changes-6 "Direct link to Behavior Changes") * The default value of `lake_compaction_allow_partial_success` is set to `true`. Compaction operations can now be marked as successful even if partially completed, preventing blockage of subsequent compaction tasks. [#60643](https://github.com/StarRocks/starrocks/pull/60643) #### 3.5.1[​](#351 "Direct link to 3.5.1") Release Date: July 1, 2025 ##### New Features[​](#new-features "Direct link to New Features") * \[Experimental] Starting from v3.5.1, StarRocks introduces a high-performance data transfer channel based on the Apache Arrow Flight SQL protocol, comprehensively optimizing the data import channel and significantly improving transfer efficiency. This solution establishes a fully columnar data transfer pipeline from the StarRocks columnar execution engine to the client, eliminating the frequent row-column conversions and serialization overhead typically seen in traditional JDBC and ODBC interfaces, and achieving true zero-copy, low-latency, and high-throughput data transfer capabilities. [#57956](https://github.com/StarRocks/starrocks/pull/57956) * Java Scalar UDFs (user-defined functions) now support ARRAY and MAP types as input parameters. [#55356](https://github.com/StarRocks/starrocks/pull/55356) * **Cross-node data cache sharing**: Enables nodes to share cached external table data of data lakes across compute nodes via the network. If a local cache miss occurs, the system first attempts to fetch data from the caches of other nodes within the same cluster. Only if all caches miss will it re-fetch data from remote storage. This feature effectively reduces performance jitter caused by cache invalidation during elastic scaling and ensures stable query performance. A new FE configuration parameter `enable_trace_historical_node` controls this behavior (Default: `false`). [#57083](https://github.com/StarRocks/starrocks/pull/57083) * **Storage Volume adds native support for Google Cloud Storage (GCS)**: You can now use GCS as a backend storage volume and manage and access GCS resources through the native SDK. [#58815](https://github.com/StarRocks/starrocks/pull/58815) ##### Improvements[​](#improvements-20 "Direct link to Improvements") * Optimized error messages when creating Hive external tables fails. [#60076](https://github.com/StarRocks/starrocks/pull/60076) * Optimized `count(1)` query performance using the `file_record_count` in Iceberg metadata. [#60022](https://github.com/StarRocks/starrocks/pull/60022) * Refined the Compaction scheduling logic to avoid delayed scheduling when all subtasks succeed. [#59998](https://github.com/StarRocks/starrocks/pull/59998) * Added `JAVA_OPTS="--add-opens=java.base/java.util=ALL-UNNAMED"` to BE and CN after upgrading to JDK 17. [#59947](https://github.com/StarRocks/starrocks/pull/59947) * Supports modifying the `kafka_broker_list` property via the ALTER ROUTINE LOAD command when Kafka Broker endpoints change. [#59787](https://github.com/StarRocks/starrocks/pull/59787) * Supports reducing build dependencies of the Docker base image through parameters. [#59772](https://github.com/StarRocks/starrocks/pull/59772) * Supports accessing Azure using Managed Identity authentication. [#59657](https://github.com/StarRocks/starrocks/pull/59657) * Improved error messages when querying external data via `Files()` function with duplicate path column names. [#59597](https://github.com/StarRocks/starrocks/pull/59597) * Optimized LIMIT pushdown logic. [#59265](https://github.com/StarRocks/starrocks/pull/59265) ##### Bug Fixes[​](#bug-fixes-20 "Direct link to Bug Fixes") Fixed the following issues: * Partition pruning issue when queries include Max and Min aggregations and empty partitions. [#60162](https://github.com/StarRocks/starrocks/pull/60162) * Incorrect query results when rewriting queries with materialized views due to missing NULL partitions. [#60087](https://github.com/StarRocks/starrocks/pull/60087) * Refresh errors on Iceberg external tables when using partition expressions based on `str2date`. [#60089](https://github.com/StarRocks/starrocks/pull/60089) * Incorrect partition range when creating temporary partitions using the START END syntax. [#60014](https://github.com/StarRocks/starrocks/pull/60014) * Incorrect display of Routine Load metrics on non-leader FE nodes. [#59985](https://github.com/StarRocks/starrocks/pull/59985) * BE/CN crashes when executing queries containing `COUNT(*)` window functions. [#60003](https://github.com/StarRocks/starrocks/pull/60003) * Stream Load failures when the target table name contains Chinese characters. [#59722](https://github.com/StarRocks/starrocks/pull/59722) * Overall loading failures to triple-replica tables when loading to a secondary replica fails. [#59762](https://github.com/StarRocks/starrocks/pull/59762) * Missing parameters in SHOW CREATE VIEW output. [#59714](https://github.com/StarRocks/starrocks/pull/59714) ##### Behavior Changes[​](#behavior-changes-7 "Direct link to Behavior Changes") * Some FE metrics include the `is_leader` label. [#59883](https://github.com/StarRocks/starrocks/pull/59883) #### 3.5.0[​](#350 "Direct link to 3.5.0") Release Date: June 13, 2025 ##### Shared-data Enhancement[​](#shared-data-enhancement "Direct link to Shared-data Enhancement") * Shared-data clusters support generated columns. [#53526](https://github.com/StarRocks/starrocks/pull/53526) * Cloud-native Primary Key tables in shared-data clusters support rebuilding specific indexes. The performance of the indexes is also optimized. [#53971](https://github.com/StarRocks/starrocks/pull/53971) [#54178](https://github.com/StarRocks/starrocks/pull/54178) * Optimized the execution logic of large-scale data loading operations to avoid generating too many small files in Rowset due to memory limitations. During the import, the system will merge the temporary data blocks to reduce the generation of small files, which improves the query performance after the import and also reduces the subsequent Compaction operations to improve the system resource utilization. [#53954](https://github.com/StarRocks/starrocks/issues/53954) ##### Data Lake Analytics[​](#data-lake-analytics "Direct link to Data Lake Analytics") * **\[Beta]** Supports creating Iceberg views in the Iceberg Catalog with Hive Metastore integration. And supports adding or modifying the dialect of the Iceberg view using the ALTER VIEW statement for better syntax compatibility with external systems. [#56120](https://github.com/StarRocks/starrocks/pull/56120) * Supports nested namespace for Iceberg REST Catalog. [#58016](https://github.com/StarRocks/starrocks/pull/58016) * Supports using `IcebergAwsClientFactory` to create AWS clients in Iceberg REST Catalog to offer vended credentials. [#58296](https://github.com/StarRocks/starrocks/pull/58296) * Parquet Reader supports filtering data with Bloom Filter. [#56445](https://github.com/StarRocks/starrocks/pull/56445) * Supports automatically creating global dictionaries for low-cardinality columns in Parquet-formatted Hive/Iceberg tables during queries. [#55167](https://github.com/StarRocks/starrocks/pull/55167) ##### Performance Improvement and Query Optimization[​](#performance-improvement-and-query-optimization "Direct link to Performance Improvement and Query Optimization") * Statistics optimization: * Supports Table Sample. Improved statistics accuracy and query performance by sampling data blocks in physical files. [#52787](https://github.com/StarRocks/starrocks/issues/52787) * Supports [recording the predicate columns in queries](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer/#predicate-column) for targeted statistics collection. [#53204](https://github.com/StarRocks/starrocks/issues/53204) * Supports partition-level cardinality estimation. The system reuses the system-defined view `_statistics_.column_statistics` to record the NDV of each partition. [#51513](https://github.com/StarRocks/starrocks/pull/51513) * Supports [multi-column Joint NDV collection](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer/#multi-column-joint-statistics) to optimize the query plan generated by CBO in the scenario where columns correlate with each other. [#56481](https://github.com/StarRocks/starrocks/pull/56481) [#56715](https://github.com/StarRocks/starrocks/pull/56715) [#56766](https://github.com/StarRocks/starrocks/pull/56766) [#56836](https://github.com/StarRocks/starrocks/pull/56836) * Supports using histograms to estimate the Join node cardinality and in\_predicate selectivity, thus improving the estimation accuracy in data skew. [#57874](https://github.com/StarRocks/starrocks/pull/57874) [#57639](https://github.com/StarRocks/starrocks/pull/57639) * Optimized [Query Feedback](https://docs.starrocks.io/docs/using_starrocks/query_feedback/). Queries with the identical structure but different parameter values will be categorized as the same type and share the same tuning guide for plan execution optimization. [#58306](https://github.com/StarRocks/starrocks/pull/58306) * Supports Runtime Bitset Filter as an alternative for optimization to Bloom Filter in specific scenarios. [#57157](https://github.com/StarRocks/starrocks/pull/57157) * Supports pushing down Join Runtime Filter to the storage layer. [#55124](https://github.com/StarRocks/starrocks/pull/55124) * Supports Pipeline Event Scheduler. [#54259](https://github.com/StarRocks/starrocks/pull/54259) ##### Partition Management[​](#partition-management "Direct link to Partition Management") * Supports using ALTER TABLE to [merge expression partitions based on time functions](https://docs.starrocks.io/docs/table_design/data_distribution/expression_partitioning/#merge-expression-partitions) for optimized storage efficiency and query performance. [#56840](https://github.com/StarRocks/starrocks/pull/56840) * Supports partition Time-to-live (TTL) for List-partitioned tables and materialized views. And supports the property `partition_retention_condition` in tables and materialized views to allow users to set data retention strategies for list partitions, thus achieving more flexible partition deletion strategies. [#53117](https://github.com/StarRocks/starrocks/issues/53117) * Supports using ALTER TABLE to [delete partitions specified by common partition expressions](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE/#drop-partitions), allowing users to flexibly delete partitions in batches. [#53118](https://github.com/StarRocks/starrocks/pull/53118) ##### Cluster Management[​](#cluster-management "Direct link to Cluster Management") * Upgraded FE compile target from Java 11 to Java 17 for better system stability and performance. [#53617](https://github.com/StarRocks/starrocks/pull/53617) [#57030](https://github.com/StarRocks/starrocks/pull/57030) ##### Security and Authentication[​](#security-and-authentication "Direct link to Security and Authentication") * Supports secure [connections encrypted by SSL](https://docs.starrocks.io/zh/docs/administration/user_privs/ssl_authentication/) based on the MySQL protocol. [#54877](https://github.com/StarRocks/starrocks/pull/54877) * Enhanced authentication using external systems: * Supports creating StarRocks users with [OAuth 2.0](https://docs.starrocks.io/docs/administration/user_privs/authentication/oauth2_authentication/) and [JSON Web Token (JWT)](https://docs.starrocks.io/docs/administration/user_privs/authentication/jwt_authentication/). * Supports [Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration/) to simplify the authentication process with external systems. Security Integration supports LDAP, OAuth 2.0, and JWT. [#55846](https://github.com/StarRocks/starrocks/pull/55846) * Supports [Group Provider](https://docs.starrocks.io/docs/administration/user_privs/group_provider/) to obtain the user group information from external authentication services. The group information can then be used in authentication and authorization. Group Provider supports acquiring group information from LDAP, operating systems, or files. Users can query the user group they belong to using the function `current_group()`. [#56670](https://github.com/StarRocks/starrocks/pull/56670) ##### Materialized Views[​](#materialized-views "Direct link to Materialized Views") * Supports creating materialized views with multiple partition columns to allow users to partition the data with a more flexible strategy. [#52576](https://github.com/StarRocks/starrocks/issues/52576) * Supports setting `query_rewrite_consistency` to `force_mv` to force the system to use the materialized view for query rewrite, thus keeping performance stability at the cost of data timeliness to a certain extent. [#53819](https://github.com/StarRocks/starrocks/pull/53819) ##### Loading and Unloading[​](#loading-and-unloading "Direct link to Loading and Unloading") * Supports pausing Routine Load jobs on JSON parse errors by setting the property `pause_on_json_parse_error` to `true`. [#56062](https://github.com/StarRocks/starrocks/pull/56062) * **\[Beta]** Supports [transactions with multiple SQL statements](https://docs.starrocks.io/docs/loading/SQL_transaction/) (currently, only INSERT is supported). Users can start, apply, or undo a transaction to guarantee the ACID (atomicity, consistency, isolation, and durability) properties of multiple loading operations. [#53978](https://github.com/StarRocks/starrocks/issues/53978) ##### Functions[​](#functions "Direct link to Functions") * Introduced the system variable `lower_upper_support_utf8` on the session and global level, enhancing the support for UTF-8 strings (especially non-ASCII characters) in case conversion functions such as `upper()` and `lower()`. [#56192](https://github.com/StarRocks/starrocks/pull/56192) * Added new functions: * [`field()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/field/) [#5533](https://github.com/StarRocks/starrocks/pull/55331) * [`ds_theta_count_distinct()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/ds_theta_count_distinct/) [#56960](https://github.com/StarRocks/starrocks/pull/56960) * [`array_flatten()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_flatten/) [#50080](https://github.com/StarRocks/starrocks/pull/50080) * [`inet_aton()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/inet_aton/) [#51883](https://github.com/StarRocks/starrocks/pull/51883) * [`percentile_approx_weight()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/percentile_approx_weight/) [#57410](https://github.com/StarRocks/starrocks/pull/57410) --- ## Release 4.0 ### StarRocks version 4.0 warning **Downgrade Notes** * After upgrading StarRocks to v4.0, DO NOT downgrade it directly to v3.5.0 & v3.5.1, otherwise it will cause metadata incompatibility and FE crash. You must downgrade the cluster to v3.5.2 or later to prevent these issues. * Before downgrading clusters from v4.0.2 to v4.0.1, v4.0.0, and v3.5.2~v3.5.10, execute the following statement: ```sql SET GLOBAL enable_rewrite_simple_agg_to_meta_scan=false; ``` After upgrading the cluster back to v4.0.2 and later, execute the following statement: ```sql SET GLOBAL enable_rewrite_simple_agg_to_meta_scan=true; ``` #### 4.0.16[​](#4016 "Direct link to 4.0.16") Release Date: September 30, 2026 ##### Behavior Changes[​](#behavior-changes "Direct link to Behavior Changes") * The default HTTP client for the AWS SDK is switched from Poco back to the SDK's own client: `enable_poco_client_for_aws_sdk` now defaults to `false`. Set it to `true` to restore the previous behavior. [#78375](https://github.com/StarRocks/starrocks/pull/78375) * jemalloc's dedicated oversize arena is enabled by default, so large allocations no longer land in the per-CPU arena of the calling thread. This improves reuse of large buffers and reduces fragmentation, and changes how BE and CN memory is distributed across arenas. [#78396](https://github.com/StarRocks/starrocks/pull/78396) * A shared-data transaction whose combined txn log was not fully written is now refused at commit instead of committing and then failing publish forever with `txn log list does not contain txn log of tablet`. `replication_num` no longer weakens the shared-data write quorum. [#78162](https://github.com/StarRocks/starrocks/pull/78162) * A shared-data partition now advances straight to the final version of a batch publish instead of through each intermediate version, so a query can no longer request an intermediate version that has no tablet metadata object, which surfaced as `The specified key does not exist`. [#79175](https://github.com/StarRocks/starrocks/pull/79175) * A shared-data publish that keeps failing now backs off and throttles its retry logging instead of emitting the `previous publish version task has not finished` message on every attempt, which could dominate the FE leader's CPU and log volume during an incident. [#79299](https://github.com/StarRocks/starrocks/pull/79299) * A shared-data load cancelled by a user error now reports that error instead of the derived `AsyncDeltaWriter has been closed`. [#78632](https://github.com/StarRocks/starrocks/pull/78632) * HDFS files are no longer `hsync`ed before they are closed. The `hsync` was a workaround for a crash that a separate change had already fixed, and it forced an `fsync()` on every DataNode replica on each close. [#78864](https://github.com/StarRocks/starrocks/pull/78864) * A Parquet `null_count` that under-reports NULLs is no longer taken at face value during decoding and pruning. Files written by older parquet-mr versions previously returned corrupted values or dropped rows. [#78329](https://github.com/StarRocks/starrocks/pull/78329) * List partition pruning of `!=`, `NOT IN`, and `IS NOT NULL` no longer discards a multi-value partition just because its value list also contains `NULL`. Queries on such a table, including joins that derive `IS NOT NULL` on the join key, return the non-null rows that were previously lost. [#78946](https://github.com/StarRocks/starrocks/pull/78946) * Batch DDL can now build date range partitions that start at year `0000`, which is a legal `DATE` value. Such a statement previously failed with `the range of the partition is invalid`. [#78746](https://github.com/StarRocks/starrocks/pull/78746) * Serialized datetime partition bounds now keep their microseconds. An operator-precedence error previously added the fractional seconds to the shift distance instead of the shifted value, corrupting the bound and potentially blocking metadata replay. [#78922](https://github.com/StarRocks/starrocks/pull/78922) * `max_partitions_in_one_batch` and the partition-number limit now count the partitions the request in hand is about to create. An oversized batch is rejected up front instead of being admitted and only failing the following batch. [#79379](https://github.com/StarRocks/starrocks/pull/79379) * Streaming aggregation tuning is now driven by the group-by cardinality instead of the rows left after `HAVING`, so a query whose `HAVING` clause filters most groups away no longer loses its local pre-aggregation. [#79064](https://github.com/StarRocks/starrocks/pull/79064) * Audit log rows now carry `scanBytes`, `scanRows`, `returnRows`, `cpuCostNs`, and `memCostBytes` for DML that targets an external Iceberg or Hive table or the `FILES()` table function, and for asynchronous materialized-view refreshes. These fields previously reported `0`. [#74072](https://github.com/StarRocks/starrocks/pull/74072) [#74954](https://github.com/StarRocks/starrocks/pull/74954) * Internal queries issued while filling `information_schema.materialized_views` and `SHOW MATERIALIZED VIEWS` are now bounded by the outer statement's `query_timeout` instead of `statistic_collect_query_timeout` (default one hour), so such a statement can no longer hang far past its own timeout. [#77585](https://github.com/StarRocks/starrocks/pull/77585) * SQL blacklist verification no longer holds a global read lock while matching, so an expensive pattern or a concurrent `ADD`/`DELETE SQLBLACKLIST` no longer stalls every query at the SQL entry point. [#78476](https://github.com/StarRocks/starrocks/pull/78476) * `paimon.option.cache-enabled = false` disables the Paimon metadata cache again. [#78271](https://github.com/StarRocks/starrocks/pull/78271) * The background metadata refresh for a Paimon catalog now visits only tables that were recently accessed and have changed, instead of enumerating and reloading the whole warehouse every cycle. A table that StarRocks has never queried is no longer refreshed. [#78057](https://github.com/StarRocks/starrocks/pull/78057) * A `VARCHAR` column of a Paimon table now reports the length declared in the Paimon schema instead of an incorrect one. [#68383](https://github.com/StarRocks/starrocks/pull/68383) * `ALTER TABLE ... RENAME` on an Iceberg REST catalog no longer fails with a `NullPointerException`. [#76498](https://github.com/StarRocks/starrocks/pull/76498) * Reading the `$partitions` metadata table of an Iceberg table with a `DECIMAL` partition field, or with a null `timestamptz` partition value, no longer fails with a `NullPointerException`. [#77742](https://github.com/StarRocks/starrocks/pull/77742) * A table with many stale partitions no longer reports a statistics health pinned near `1.0`, so auto-ANALYZE is triggered when it should be. Tables that were previously considered healthy may now be re-analyzed. [#76927](https://github.com/StarRocks/starrocks/pull/76927) * A column whose name contains a special character such as `#` no longer fails the low-cardinality dictionary collection query with a `ParsingException`; the generated alias is quoted. [#77132](https://github.com/StarRocks/starrocks/pull/77132) * A `GROUPING SETS` aggregation joined to another one on its group-by keys no longer fails at planning with `no executable plan for this sql`. [#77296](https://github.com/StarRocks/starrocks/pull/77296) * `encode_sort_key` now expands a constant argument once per row instead of writing the single stored element into every row buffer. [#77847](https://github.com/StarRocks/starrocks/pull/77847) * A statement forwarded to the Leader FE now keeps its original SQL text in the audit log and the query profile, instead of a form rebuilt from the AST. [#77982](https://github.com/StarRocks/starrocks/pull/77982) * A connection ID that wraps the 24-bit per-FE counter no longer collides with a live connection, which previously hid the older session and drifted the global and per-user connection accounting. [#78217](https://github.com/StarRocks/starrocks/pull/78217) * A column that is missing from the tablet schema no longer aborts the BE during predicate push-down; the predicate is simply not pushed down. [#79234](https://github.com/StarRocks/starrocks/pull/79234) * When `OlapTableSink` initialization fails, the partially initialized sink is no longer taken through the full close path, which could dereference state that was never created and abort the BE. The original initialization error is returned instead. [#79370](https://github.com/StarRocks/starrocks/pull/79370) * Short key index page geometry is now validated without 32-bit overflow, so a corrupted page footer is rejected instead of wrapping and causing a heap out-of-bounds write. [#79048](https://github.com/StarRocks/starrocks/pull/79048) * An HTTPS request whose TLS handshake times out is no longer treated as connected and retried on the same socket. A stalled request now fails after approximately one timeout instead of `connect_timeout` plus two receive timeouts. [#79369](https://github.com/StarRocks/starrocks/pull/79369) * An expression returning `ARRAY` or `STRUCT` over a low-cardinality column is no longer rewritten to operate on dictionary ids, which crashed the BE at fragment prepare. [#78488](https://github.com/StarRocks/starrocks/pull/78488) * `starrocks_fe_max_tablet_compaction_score` is reported correctly on a shared-nothing cluster after an FE restart; the active `get_tablets_info` response now carries the maximum tablet compaction score. [#78496](https://github.com/StarRocks/starrocks/pull/78496) * FE feature log files are written under `feature_log_dir` instead of the general log directory. [#77221](https://github.com/StarRocks/starrocks/pull/77221) * `be.conf` and `fe.conf` entries on a last line without a trailing newline are now read instead of silently dropped. [#78065](https://github.com/StarRocks/starrocks/pull/78065) ##### Improvements[​](#improvements "Direct link to Improvements") * Supported Fast Schema Evolution for materialized views. [#67915](https://github.com/StarRocks/starrocks/pull/67915) * Extended the grouping-sets push-down rule to `AVG` and `COUNT`, in addition to `MAX`, `MIN`, and `SUM`, which speeds up `ROLLUP`, `CUBE`, and `GROUPING SETS` queries with more than three levels. [#77832](https://github.com/StarRocks/starrocks/pull/77832) * Removed the process-wide lock in front of the lake segment metadata cache, leaving the sharded LRU cache to do its own locking. [#78260](https://github.com/StarRocks/starrocks/pull/78260) * Compaction and replication now drop the never-read `delete_predicate` when they archive a rowset into `compaction_inputs`, which reclaims tablet-metadata bloat at its source in shared-data clusters. [#76870](https://github.com/StarRocks/starrocks/pull/76870) * Added server-level spill metrics that aggregate spill activity across all spillable operators, broken down by local and remote storage backend. [#71827](https://github.com/StarRocks/starrocks/pull/71827) * Added the leader-FE gauge `max_journal_replay_lag`, which reports how far the furthest-behind follower or observer trails the leader in replayed journal ids. [#78382](https://github.com/StarRocks/starrocks/pull/78382) * The jemalloc runtime options a BE or CN was started with are now visible through the `jemalloc_conf` configuration item, and its mutable options such as `dirty_decay_ms` and `muzzy_decay_ms` can be changed at runtime without a restart. [#75344](https://github.com/StarRocks/starrocks/pull/75344) [#78333](https://github.com/StarRocks/starrocks/pull/78333) * Extended jemalloc observability: added dirty and muzzy page metrics, broke the metadata total down into extent metadata and the rtree, dumped `stats.mapped` and `stats.retained` in `be.INFO`, and let `/memz` take jemalloc's own statistics options so the per-arena section and the extent size-class breakdown are reachable from a running BE. [#78159](https://github.com/StarRocks/starrocks/pull/78159) [#78225](https://github.com/StarRocks/starrocks/pull/78225) [#78320](https://github.com/StarRocks/starrocks/pull/78320) [#79090](https://github.com/StarRocks/starrocks/pull/79090) * On arm64, a 4K-page jemalloc variant is built alongside the 64K one and selected at runtime, which avoids the memory overhead of 64K page granularity on the common 4K-page kernels. The jemalloc libraries now live in dedicated directories selected through `LD_LIBRARY_PATH` instead of runtime symlink manipulation. [#77886](https://github.com/StarRocks/starrocks/pull/77886) [#67667](https://github.com/StarRocks/starrocks/pull/67667) ##### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") The following issues have been fixed: * `SHOW PROFILELIST`, `ANALYZE PROFILE`, `get_query_profile()`, and `/api/profile` returned every cached profile, including other users' SQL text and plans, to any authenticated user. The new FE configuration `authorization_enable_query_profile_access_check` restricts a cached profile to the user who ran the query and to holders of `SYSTEM OPERATE`. [#79375](https://github.com/StarRocks/starrocks/pull/79375) * In Ranger mode, the column privilege check mishandled views and materialized views over external tables. [#56316](https://github.com/StarRocks/starrocks/pull/56316) * Addressed security vulnerabilities (CVE): upgraded Apache Ranger to 2.9.0 and Bouncy Castle to 1.85, Apache HttpClient5 to 5.6.4 and HttpCore to 5.4.3, Netty to 4.1.137.Final, ZooKeeper to 3.8.7 and 3.9.6, the broker's Tomcat to 9.0.121 and its commons-lang3 to 3.20.0, and aligned all `io.grpc` artifacts through `grpc-bom`. Test-scope jars are no longer shipped in the java-extensions reader lib directories. [#79331](https://github.com/StarRocks/starrocks/pull/79331) [#79538](https://github.com/StarRocks/starrocks/pull/79538) [#77753](https://github.com/StarRocks/starrocks/pull/77753) [#78287](https://github.com/StarRocks/starrocks/pull/78287) [#79707](https://github.com/StarRocks/starrocks/pull/79707) [#79719](https://github.com/StarRocks/starrocks/pull/79719) [#77748](https://github.com/StarRocks/starrocks/pull/77748) [#77171](https://github.com/StarRocks/starrocks/pull/77171) [#77752](https://github.com/StarRocks/starrocks/pull/77752) * Upgraded StarOS, starlet, and starcache to v4.0.8. [#79478](https://github.com/StarRocks/starrocks/pull/79478) [#77717](https://github.com/StarRocks/starrocks/pull/77717) * The authorization check, `INSERT` planning, view expansion, the privilege check, `inspect_mv_refresh_info`, and `FILES()` schema inference for `CREATE TABLE AS SELECT` all made external-catalog round trips while holding an FE metadata lock, so a slow metastore stalled every DDL waiting on the same lock. Those calls now happen outside the lock. [#79319](https://github.com/StarRocks/starrocks/pull/79319) [#79226](https://github.com/StarRocks/starrocks/pull/79226) [#78772](https://github.com/StarRocks/starrocks/pull/78772) * `PlannerMetaLocker` unlocked metadata it had never successfully locked when `lock()` failed part way through. [#74041](https://github.com/StarRocks/starrocks/pull/74041) * `FULL OUTER JOIN ... USING (c1, c2, ...)` returned the wrong values: the first `USING` column was correct and every later one carried the previous column's value. Adding `ORDER BY ` to the same query failed with an `IndexOutOfBoundsException`. [#79298](https://github.com/StarRocks/starrocks/pull/79298) * A `HAVING` predicate was lost when `GROUPING SETS`, `ROLLUP`, or `CUBE` was rewritten into a union, so the query returned rows it was supposed to filter out. [#79415](https://github.com/StarRocks/starrocks/pull/79415) * `GROUP BY ROLLUP(a, b, a)`, and `CUBE` or `GROUPING SETS` naming a column twice within one set, produced a grouping set that carried the same key twice. [#78261](https://github.com/StarRocks/starrocks/pull/78261) * Removing the aggregation above an aggregate-key table dropped the query's `LIMIT` before it reached the storage layer, so the BE read the whole table. [#79414](https://github.com/StarRocks/starrocks/pull/79414) * Splitting an `OR` join into a union lined up the branches inconsistently, because the union's output columns and the first branch were ordered by column id while the other branches followed the projection map's iteration order. [#79473](https://github.com/StarRocks/starrocks/pull/79473) * The skew-join rewrite replicated the preserved side of an outer join, so one input row could be returned several times and the result moved between runs. Salting is now skipped in that case. [#79168](https://github.com/StarRocks/starrocks/pull/79168) * A predicate under an `OR` whose branch was only partially covered by bitmap indexes was erased once its bitmap range had been applied, which produced false positives such as `(a = 1 AND b = 2) OR (a = 2 AND b = 3)` matching `(a = 1, b = 3)`. [#76275](https://github.com/StarRocks/starrocks/pull/76275) * A column read only by a predicate inside an `OR` subtree was pruned from the read set after GIN inverted index filtering, so the surviving predicate evaluated against a column that was never read. [#77659](https://github.com/StarRocks/starrocks/pull/77659) * A predicate-free `COUNT(*)` could be folded from tablet row counts that never covered the visible version, because the guard only checked that a collection cycle had finished, not that its RPCs had succeeded. [#77190](https://github.com/StarRocks/starrocks/pull/77190) * `SELECT DISTINCT` over a window aggregate left dropped columns in a projection after the distinct push-down below the window. [#77218](https://github.com/StarRocks/starrocks/pull/77218) * Ranking-window pre-aggregation was applied when the aggregate window and the rank-related window were not in the same sort group. [#77068](https://github.com/StarRocks/starrocks/pull/77068) * A window over an always-non-null aggregate such as `bitmap_union_count()` lost its declared nullability, so a `DISTINCT` or `GROUP BY` over it failed with `error nullablel column`. [#79009](https://github.com/StarRocks/starrocks/pull/79009) * An analytic frame starting at a `FOLLOWING` offset dropped the current row. [#77342](https://github.com/StarRocks/starrocks/pull/77342) * A subquery in a window's `PARTITION BY` or `ORDER BY` clause failed to plan with `slot type shouldn't be invalid`. [#77087](https://github.com/StarRocks/starrocks/pull/77087) * `Analytor::prepare()` filled its function list only at the very end, so an error part way through left the destructor reading past the end of an empty vector and killing the BE. [#77758](https://github.com/StarRocks/starrocks/pull/77758) * A merge-phase aggregate re-appended an already-replaced constant argument, which shifted every following constant argument one slot to the right. [#77069](https://github.com/StarRocks/starrocks/pull/77069) * `count()` over an array of decimal produced an `INVALID` result type and failed with `slot type shouldn't be invalid`. [#77092](https://github.com/StarRocks/starrocks/pull/77092) * The `if()` built for a multi-column `count(DISTINCT)` lost its decimal type, so the BE received an expression typed `DECIMAL(-1,-1)` and aborted. [#77346](https://github.com/StarRocks/starrocks/pull/77346) * Common plan fusion skipped the `_if` combinator fall-back check, and `GROUP BY` rules were applied to only some children of an expression, so `SELECT max(s) LIKE s FROM t` failed with `Invalid plan`. [#77099](https://github.com/StarRocks/starrocks/pull/77099) * `corr`, `covar_pop`, and `covar_samp` crashed the BE when the optimizer folded an argument into a constant after analysis. [#77216](https://github.com/StarRocks/starrocks/pull/77216) * `percentile_cont` merge tracked exhaustion by value instead of by position, which could abort the BE. [#77261](https://github.com/StarRocks/starrocks/pull/77261) * A global dictionary was used for aggregate-state columns and their subfields were pruned, both of which aborted the BE on a failed down-cast. [#77096](https://github.com/StarRocks/starrocks/pull/77096) [#77901](https://github.com/StarRocks/starrocks/pull/77901) * A JSON subfield aggregate on an aggregate table failed with a raw `NullPointerException`, because the synthetic subfield column has no aggregation type. [#77952](https://github.com/StarRocks/starrocks/pull/77952) * `CTEAnchor` was not pruned when its child was a value operator, which surfaced as `MultiCastPlanFragment don't support return result`. [#64491](https://github.com/StarRocks/starrocks/pull/64491) * Duplicating a plan threw `ClassCastException` on every `LogicalLimit`, because the duplicator treated a limit as a set operator. [#77692](https://github.com/StarRocks/starrocks/pull/77692) * UKFK constraints were collected conditionally during join reorder, so `enable_ukfk_join_reorder` could fail with a `NullPointerException`. [#77197](https://github.com/StarRocks/starrocks/pull/77197) * Planning a query over a colocate table could fail with `Cannot invoke "PhysicalPropertySet.equals(Object)" because "enforcedProperty" is null`, because colocate-group stability was re-evaluated while the optimizer ran. [#77091](https://github.com/StarRocks/starrocks/pull/77091) * A colocate `UNION`, `EXCEPT`, or `INTERSECT` could hang, because a branch whose source cannot local-shuffle stayed in the colocate execution group without a grouped exchange. [#77025](https://github.com/StarRocks/starrocks/pull/77025) * A non-positive ranking predicate was pushed into a TopN, which aborted the BE. [#77445](https://github.com/StarRocks/starrocks/pull/77445) * The global limit shared by parallel streaming-distinct instances truncated results incorrectly. [#77086](https://github.com/StarRocks/starrocks/pull/77086) * `DeferProjectAfterTopNRule` matched a scan's access paths by column name instead of column id, so a renamed column lost its subfield pruning. [#77987](https://github.com/StarRocks/starrocks/pull/77987) * An OLAP scan's complex type was narrowed even when the BE did not prune it by an access path, so downstream operators built columns from a type the BE never materialized. [#77894](https://github.com/StarRocks/starrocks/pull/77894) * Dead subfield slots pulled out of the scan by predicate expression reuse were not dropped, which aborted the BE. [#77456](https://github.com/StarRocks/starrocks/pull/77456) * `PruneComplexTypeUtil` disabled complex-type subfield pruning for every column in the statement when it met a single expression it could not describe, such as a `CASE WHEN` over a complex column. The bail-out is now scoped to the expression that caused it. [#79181](https://github.com/StarRocks/starrocks/pull/79181) * Scan heavy-expression result columns were not normalized to the slot's nullability, which crashed the BE. [#77242](https://github.com/StarRocks/starrocks/pull/77242) * The query cache could serve an entry built below a `TOPN_FILTER` that a `SortNode` produced, and its digest did not cover the scan schema, an ASOF join's temporal condition, a join predicate's common sub-expressions, or an ANN search specification, so unrelated queries could collide. [#77404](https://github.com/StarRocks/starrocks/pull/77404) [#77559](https://github.com/StarRocks/starrocks/pull/77559) [#77563](https://github.com/StarRocks/starrocks/pull/77563) [#77565](https://github.com/StarRocks/starrocks/pull/77565) [#77586](https://github.com/StarRocks/starrocks/pull/77586) * The query cache crashed the BE when it reused a lane for a partitioned hash join, and hung when a lane's join had an empty build side. [#77564](https://github.com/StarRocks/starrocks/pull/77564) [#77571](https://github.com/StarRocks/starrocks/pull/77571) * A view scan's projection was dropped when a partially rewritten view-based materialized view rewrite replaced the scan back with its inlined plan. [#77613](https://github.com/StarRocks/starrocks/pull/77613) * A materialized view refresh on a Paimon base table performed a full scan because the partition predicate was not pushed down. [#61813](https://github.com/StarRocks/starrocks/pull/61813) * `REFRESH MATERIALIZED VIEW ... FORCE` silently refreshed nothing when the materialized view had a `partition_retention_condition`. [#77671](https://github.com/StarRocks/starrocks/pull/77671) * The column min/max statistics cache was not invalidated by partition or schema DDL, so a stale entry could be served indefinitely. [#77441](https://github.com/StarRocks/starrocks/pull/77441) * Background auto-ANALYZE of an external table could fail with `primary key size exceed the limit` when the catalog, database, and table names made the encoded `table_uuid` primary key exceed the BE limit. The `table_uuid` is now hashed. [#75735](https://github.com/StarRocks/starrocks/pull/75735) * `CancelableAnalyzeTask.cancel()` could be won by more than one caller, because it read and wrote its state without an atomic compare-and-set. [#78593](https://github.com/StarRocks/starrocks/pull/78593) * Hive-family catalogs failed outright when the metastore or AWS Glue denied permission to read column statistics; they now degrade to empty statistics. Glue column-statistics requests also carry `catalogId`, so a cross-account deployment reads statistics from the configured catalog. [#78455](https://github.com/StarRocks/starrocks/pull/78455) [#78461](https://github.com/StarRocks/starrocks/pull/78461) * Elasticsearch external tables and Elasticsearch catalogs rejected clusters running version 8 or 9 with `Unsupported/Unknown ES Cluster version`, and concurrent queries could overwrite each other's request URL parameters, which surfaced as unrelated analysis errors such as `Column 'id' cannot be resolved`. [#78267](https://github.com/StarRocks/starrocks/pull/78267) [#79391](https://github.com/StarRocks/starrocks/pull/79391) * Predicates on a `DECIMAL` column of a Paimon table were converted using the StarRocks type family's default precision and scale instead of the column's Paimon `DecimalType`, so the literal could not represent the value. [#78760](https://github.com/StarRocks/starrocks/pull/78760) * Table references inside a connector view were qualified incorrectly when the Iceberg view metadata recorded no `default-catalog`, and when the view body referenced a CTE. [#77887](https://github.com/StarRocks/starrocks/pull/77887) * An Iceberg table whose metadata file is missing from storage could not be dropped, not even with `DROP TABLE IF EXISTS ... FORCE`. [#77786](https://github.com/StarRocks/starrocks/pull/77786) * `ALTER TABLE ... EXECUTE rewrite_data_files() WHERE ` on an Iceberg table silently dropped rows when the filter did not align with partition boundaries. [#77797](https://github.com/StarRocks/starrocks/pull/77797) * Azure Blob credential lookup failed when the endpoint was configured as a full URL, because the Hadoop configuration key embedded the scheme. Endpoints are now normalized to the filesystem authority. [#78241](https://github.com/StarRocks/starrocks/pull/78241) * The send and receive timeouts computed for an object-storage request were dropped when the pooled Poco session was reused, so only the connection timeout applied. [#78361](https://github.com/StarRocks/starrocks/pull/78361) * Querying a Parquet `DECIMAL` column stored as `FIXED_LEN_BYTE_ARRAY` through a connector scan could return corrupted values and silently drop rows, non-deterministically between runs of the same query. [#78685](https://github.com/StarRocks/starrocks/pull/78685) * A corrupt `DELTA_BINARY_PACKED` page header was thrown rather than returned, and since `Status` is not an exception type nothing caught it. [#77446](https://github.com/StarRocks/starrocks/pull/77446) * Parquet decoder scratch buffers were sized as variable-length arrays on the stack, which a large data page could overflow. [#77616](https://github.com/StarRocks/starrocks/pull/77616) * A JSON path with an array index such as `'[0].value.url'` threw a C++ exception for every row whose array was shorter than the index, which turned a large scan into a lock convoy inside the stack unwinder. The index is now bounds-checked. [#78569](https://github.com/StarRocks/starrocks/pull/78569) * `JsonColumn::append` read out of range and silently lost rows when flat-JSON schemas differed between segments. [#77090](https://github.com/StarRocks/starrocks/pull/77090) * The JSON key column was not rebuilt when a set operation deserialized its keys, which aborted the BE. [#77215](https://github.com/StarRocks/starrocks/pull/77215) * Flat JSON vertical compaction in shared-data mode used more memory than necessary, because the merger retained the previous result column and the root metadata did not account for its children. [#77682](https://github.com/StarRocks/starrocks/pull/77682) * Nested `array_map` and `array_sort` lambdas failed to plan or evaluate with `Input dependency cols check failed` or `Expr evaluate meet error: slot_id N not found`, because common sub expressions hoisted by an inner lambda escaped the scope that defined them. [#78645](https://github.com/StarRocks/starrocks/pull/78645) [#78769](https://github.com/StarRocks/starrocks/pull/78769) [#78915](https://github.com/StarRocks/starrocks/pull/78915) * `array_sort`'s lambda comparator could drive the sort out of bounds, and its comparator validation shared one error `Status` across drivers, which could double-free it. [#77687](https://github.com/StarRocks/starrocks/pull/77687) [#78314](https://github.com/StarRocks/starrocks/pull/78314) * `unnest` and `multi_unnest` read array offsets as signed 32-bit integers, so an offset of `2^31` or more came back negative and crashed the BE. [#78480](https://github.com/StarRocks/starrocks/pull/78480) [#78479](https://github.com/StarRocks/starrocks/pull/78479) * `substr`, `left`, `lpad`, `rpad`, and `translate` returned bytes from neighboring rows, and `locate` searched past the end of a value, when the input ended in a truncated UTF-8 lead byte. [#78722](https://github.com/StarRocks/starrocks/pull/78722) * `strpos` indexed out of bounds on an `INT32_MIN` instance argument. [#77260](https://github.com/StarRocks/starrocks/pull/77260) * `timestampdiff(MONTH, ...)`, `timestampdiff(YEAR, ...)`, `months_diff`, and `years_diff` returned `-1` instead of `0` when both arguments fell in the same month or year and the second was the earlier one. [#79037](https://github.com/StarRocks/starrocks/pull/79037) * `DateValue::weekday()` returned a negative index for an out-of-range Julian value, so `trunc_to_week()` read before the start of its lookup table. [#77310](https://github.com/StarRocks/starrocks/pull/77310) * The Teradata and Joda datetime parsers read past the end of the input string, which aborted the BE. [#77098](https://github.com/StarRocks/starrocks/pull/77098) * `generate_series` crashed on a step count that cannot be represented, such as `generate_series(0, -2147483648, -1)`. [#77103](https://github.com/StarRocks/starrocks/pull/77103) * Constant arguments to `tokenize` and `ngram_search` that crash the BE are now rejected. [#77102](https://github.com/StarRocks/starrocks/pull/77102) * `EXPLAIN` printed the wrong number of dictionary keys for a `dictionary_get` call with multiple keys, because the key count was inferred from the number of children. [#78706](https://github.com/StarRocks/starrocks/pull/78706) * A descending sort of an `ARRAY`, `MAP`, or `STRUCT` column placed NULLs inconsistently, because the run-sort and run-merge phases applied opposite NULL conventions. [#77679](https://github.com/StarRocks/starrocks/pull/77679) * `EXCEPT` and `INTERSECT` sized their serialization buffer from the widest row in the chunk, so a single fat row could ask the memory pool for hundreds of gigabytes. Keys are now serialized row by row in that case. [#77821](https://github.com/StarRocks/starrocks/pull/77821) * A nested-loop join evaluated conjuncts on an empty chunk, and its null-aware anti join lost probe rows when a joined chunk had no matched rows, so `NOT IN` queries silently dropped rows on a multi-BE cluster. [#77344](https://github.com/StarRocks/starrocks/pull/77344) [#77970](https://github.com/StarRocks/starrocks/pull/77970) * The BE read past the end of the output schema in `_switch_context` when a segment had to force-encode a string column to global dictionary ids. [#77660](https://github.com/StarRocks/starrocks/pull/77660) * `ALTER TABLE` that added an ordinary column and a generated column in one statement killed the BE on a table that already had data. [#77436](https://github.com/StarRocks/starrocks/pull/77436) * `ALTER TABLE ... ADD COLUMN` on an Aggregate table silently created a key column when neither an aggregate function nor the `KEY` keyword was given, changing the grain the table aggregates on. Such a statement is now rejected as ambiguous. [#78359](https://github.com/StarRocks/starrocks/pull/78359) * `PARTITION BY RANGE(())` with an empty argument list raised a raw `IndexOutOfBoundsException` instead of a parse error. [#75080](https://github.com/StarRocks/starrocks/pull/75080) * `USING GIN("IMP_LIB" = "builtin")` was accepted by `CREATE TABLE` and then broke every `INSERT` with `Can not get inverted imp type`, because the property key reached the BE with its original casing. [#77818](https://github.com/StarRocks/starrocks/pull/77818) * An unchecked `StatusOr` unwrap turned a recoverable error into an exception that escaped into the pipeline, a child expression error aborted the BE instead of being propagated, and an exception escaping the Thrift processor aborted the BE. [#77052](https://github.com/StarRocks/starrocks/pull/77052) [#77442](https://github.com/StarRocks/starrocks/pull/77442) [#78805](https://github.com/StarRocks/starrocks/pull/78805) * `GroupedExecutionSinkOperator::push_chunk` swallowed the status returned by the local exchanger, so spill failures and closed-exchanger states went unreported. [#77737](https://github.com/StarRocks/starrocks/pull/77737) * `SystemMetrics::update()` shared its `getline()` buffer across threads, which could double-free it. [#71040](https://github.com/StarRocks/starrocks/pull/71040) * BE and CN could crash inside jemalloc with unrelated stacks, because `malloc_usable_size` performed a full thread-specific-data fetch; the heap profile toggle also wrote `prof.thread_active_init` and `has_enable_heap_profile()` read the boolean `prof.active` node into an `int`, so profiling was reported as disabled and later profiles were damaged. [#77625](https://github.com/StarRocks/starrocks/pull/77625) [#78384](https://github.com/StarRocks/starrocks/pull/78384) [#78820](https://github.com/StarRocks/starrocks/pull/78820) * `System.print()` on a script's foreign objects printed `instance of ` instead of the object's text, because the `toString` bindings were registered as methods rather than getters. [#78877](https://github.com/StarRocks/starrocks/pull/78877) * An FE dropped with `ALTER SYSTEM DROP FOLLOWER` or `DROP OBSERVER` never exited and became a zombie that held elections forever. [#78215](https://github.com/StarRocks/starrocks/pull/78215) * `TabletInvertedIndex.markTabletsForceDelete()` resolved tablet backend ids while holding the index write lock, which could deadlock the FE against a replica deletion. [#78102](https://github.com/StarRocks/starrocks/pull/78102) * The tablet scheduler threw a `NullPointerException` on a disk that reported no storage medium. [#76966](https://github.com/StarRocks/starrocks/pull/76966) * Dropping the auto-increment map waited on dead BEs and CNs, so the operation blocked for the full 60-second timeout whenever a registered node was down. [#78561](https://github.com/StarRocks/starrocks/pull/78561) * `PrepareCollectMetaTask` threw a `NullPointerException` when the optimizer had no query id, which happens when materialized-view plan contexts are built on their own thread pool. [#77051](https://github.com/StarRocks/starrocks/pull/77051) * Removing a transaction from the dependency graph while its predecessors were still present corrupted the graph, so a later transaction on the same table could lose its dependency edge. [#78077](https://github.com/StarRocks/starrocks/pull/78077) * Three FE paths shipped a raw compute-node hostname to the BE instead of a resolved address, which made compute nodes resolve peer hostnames themselves and could flood the DNS server in containerized deployments. [#78390](https://github.com/StarRocks/starrocks/pull/78390) * The `getLoads` RPC returned every load record in a single message, so a busy cluster filled the leader log with `FE RPC failure ... MaxMessageSize reached` and the `loads_history` archive rolled back and permanently lost rows. The RPC is now paginated by job id. [#78898](https://github.com/StarRocks/starrocks/pull/78898) * A task run that migrated between the pending queue, the running map, the in-memory history, and the archived history while `task_runs` was being read could be reported twice. [#79438](https://github.com/StarRocks/starrocks/pull/79438) * `SHOW ALTER TABLE ROLLUP` and `SHOW ALTER MATERIALIZED VIEW` accepted `WHERE`, `ORDER BY`, and `LIMIT` and then ignored them; only `SHOW ALTER TABLE COLUMN` applied them. `FinishedTime` is also accepted in a `SHOW ALTER TABLE ROLLUP` predicate. [#79281](https://github.com/StarRocks/starrocks/pull/79281) [#79498](https://github.com/StarRocks/starrocks/pull/79498) * A client disconnect was handled inline on the XNIO I/O thread rather than on a worker, so killing the running query blocked the event loop. [#77590](https://github.com/StarRocks/starrocks/pull/77590) * Loading the SQL blacklist rebuilt and sorted its immutable snapshot once per rule, which is quadratic in the number of rules and ran at every FE start and checkpoint. [#78752](https://github.com/StarRocks/starrocks/pull/78752) * `AsyncTaskQueue` busy-spun between `getOutputs()` and `tryGetOutputs()` after the consumer thread was interrupted, because the interrupt was swallowed without clearing `hasMoreOutput`. [#79297](https://github.com/StarRocks/starrocks/pull/79297) * `TableKeeper` declared a `jdk.jshell` exception type that does not exist on a JRE, so reflection on the class failed with `NoClassDefFoundError`. [#79520](https://github.com/StarRocks/starrocks/pull/79520) * Loading into an expression-partitioned table through the multi-table transaction stream load interface failed at commit whenever the BE had to create a partition, with `automatic create partition failed. error: txn ... not exist`. [#78790](https://github.com/StarRocks/starrocks/pull/78790) * A completed merge-commit load could report less than full merge-window progress in `information_schema.loads`. [#78723](https://github.com/StarRocks/starrocks/pull/78723) * A `VARCHAR` column holding non-UTF-8 bytes made protobuf log an `invalid UTF-8 data` warning once per dictionary word on every tablet writer open RPC. [#78657](https://github.com/StarRocks/starrocks/pull/78657) * Data cache read counters were missing from the intermediate query statistics, so the audit log field `CacheHitRatio` and the FE cache-hit-ratio metrics under-reported in shared-data mode. [#77696](https://github.com/StarRocks/starrocks/pull/77696) * In combined txn log mode, a coordinator whose own end-of-stream arrived early could wait for the txn log collector indefinitely when the load was cancelled, and the compaction txn log that file bundling skips writing was not cached. [#79389](https://github.com/StarRocks/starrocks/pull/79389) [#77578](https://github.com/StarRocks/starrocks/pull/77578) * A corrupted local cache copy was re-read on every retry when a primary-key index SST compaction or a delvec page checksum hit corruption, so the operation failed the same way until the entry happened to be evicted. The corrupted entry is now dropped and the read retried. [#77481](https://github.com/StarRocks/starrocks/pull/77481) [#78086](https://github.com/StarRocks/starrocks/pull/78086) * A partial-update rewrite that degenerated into a copy dropped the source's encryption metadata, leaving a segment that no reader could open and that compaction could not rewrite. [#79053](https://github.com/StarRocks/starrocks/pull/79053) * A CN could crash with SIGSEGV during graceful shutdown when an in-flight load reached a StarOS-backed filesystem after `shutdown_staros_worker()` had released the global worker. [#79058](https://github.com/StarRocks/starrocks/pull/79058) * `AsyncFlushOutputStream::close()` satisfied its promise while still holding the stream's own mutex, so the I/O thread woke futexes on freed memory after the waiter had destroyed the stream. [#79182](https://github.com/StarRocks/starrocks/pull/79182) * Snapshot-protected and post-snapshot physical partitions that share a shard group could have the snapshot's shards deleted by StarMgr metadata sync, which made snapshot restoration unsafe and could surface as missing tablet errors. Shard groups already removed by the recycle bin are also ignored during meta sync. [#78800](https://github.com/StarRocks/starrocks/pull/78800) [#76604](https://github.com/StarRocks/starrocks/pull/76604) * Bundled tablet metadata readers probed a per-tablet object that does not exist in a bundled partition and paid an avoidable HTTP 404 first, because the partition marker was keyed differently by producers and readers. A BE counter for `NotFound` lake tablet metadata reads was added. [#78466](https://github.com/StarRocks/starrocks/pull/78466) [#78451](https://github.com/StarRocks/starrocks/pull/78451) * A shared-data partition with the highest compaction score in the cluster could be admitted on every scheduling round and never started, because candidates were not sorted by maximum tablet score. Runtime changes of `compact_threads` also lost tokens when the limiter shrank. [#77482](https://github.com/StarRocks/starrocks/pull/77482) [#78087](https://github.com/StarRocks/starrocks/pull/78087) * A lake alter job could get stuck after an FE failover, because replaying a pending online rewrite job did not restore its table-state reservation. [#78517](https://github.com/StarRocks/starrocks/pull/78517) * Rowset metadata cache entries kept their initial charge after lazy-loaded indexes had increased segment memory usage, so the cache under-reported its footprint. [#77893](https://github.com/StarRocks/starrocks/pull/77893) * Primary-key compaction estimated its read chunk size from compressed disk size and from rowset metadata that had not been loaded. [#78192](https://github.com/StarRocks/starrocks/pull/78192) * `Tablet::get_average_row_size()` walked `_rs_version_map` without holding `_meta_lock`, against the contract that map is declared with. [#79250](https://github.com/StarRocks/starrocks/pull/79250) * `CACHE SELECT` could crash the CN when the read schema contained complex-type columns and was reordered for predicate evaluation. [#77245](https://github.com/StarRocks/starrocks/pull/77245) * The `pipe_driver_queue_len` metric had been missing from the BE `/metrics` output since v3.5.0. [#77890](https://github.com/StarRocks/starrocks/pull/77890) * The profile dictionary carried a misspelled `ZoneMapIndexFiter` key that never matched what the BE emits, and the scan `SampleTime` counter reported a block count rather than elapsed time. [#78728](https://github.com/StarRocks/starrocks/pull/78728) [#78732](https://github.com/StarRocks/starrocks/pull/78732) * `BinaryColumnBase::replicate` overflowed a 32-bit integer when computing the total size, which crashed the BE. [#77545](https://github.com/StarRocks/starrocks/pull/77545) * The warning for `JAVA_OPTS_FOR_JDK_11` and `JAVA_OPTS_FOR_JDK_9_AND_LATER` understated that these variables override `JAVA_OPTS` outright and that support for them will be removed. [#78328](https://github.com/StarRocks/starrocks/pull/78328) * A runtime change of `sys_log_level` was written to the configuration but never applied, even though the item is documented as mutable. [#79200](https://github.com/StarRocks/starrocks/pull/79200) * `paimon-reader` did not inherit the managed `libthrift` version. [#78747](https://github.com/StarRocks/starrocks/pull/78747) #### 4.0.14[​](#4014 "Direct link to 4.0.14") Release Date: August 11, 2026 ##### Behavior Changes[​](#behavior-changes-1 "Direct link to Behavior Changes") * A static partition clause in an `INSERT` into an Iceberg or Hive table is now validated even when a target column list is given. A clause that names a column which is not a partition column of the table is rejected with a clear error instead of being silently ignored. Every `INSERT` shape that was accepted before is still accepted. [#76659](https://github.com/StarRocks/starrocks/pull/76659) * `ARRAY` and `MAP` constructors now fail with a `CapacityLimitExceed` error when the flattened result of a single chunk exceeds the addressable byte limit, instead of silently returning corrupted values that `CREATE TABLE AS SELECT`, `INSERT`, or a materialized view refresh could persist. For affected queries, reduce `chunk_size` or split the constructor expression. [#76419](https://github.com/StarRocks/starrocks/pull/76419) * Division expressions whose divisor is not constant (for example, `10 DIV c`) are no longer treated as monotonic, which stops unsafe Zone Map pruning from dropping rows that satisfy the predicate. Queries filtering on such an expression may return more rows than in earlier versions; the previous results were incorrect. Monotonic forms such as `c DIV 10` still benefit from the pruning. [#76744](https://github.com/StarRocks/starrocks/pull/76744) * On a column with a GIN inverted index, `NOT MATCH` no longer returns rows whose value is `NULL`, which brings it in line with SQL three-valued logic. Queries that depended on the previous behavior return fewer rows. [#75578](https://github.com/StarRocks/starrocks/pull/75578) * Under `"compression" = "zstd"`, flat JSON sub-columns and the synthetic null and offset sub-columns of `ARRAY`, `MAP`, and `STRUCT` columns are now compressed. They were previously written as raw pages, which could make a ZSTD table larger on disk than the same table compressed with LZ4. Only segments written after the upgrade are affected, so existing tables shrink gradually as their segments are rewritten. [#76949](https://github.com/StarRocks/starrocks/pull/76949) * The Iceberg partition cache is now bounded by memory instead of by entry count, through the new catalog property `iceberg_partition_cache_memory_usage_ratio` (default `0.1`), and its footprint is reported in `/api/memory_usage` and the minute-level memory logs. Under memory pressure the cache may hold fewer entries than before; raise the ratio to restore the previous footprint. [#76165](https://github.com/StarRocks/starrocks/pull/76165) * FE memory estimation now also counts the internal overhead of maps and collections — per-entry nodes and the backing table — instead of only the sampled keys, values, and elements. Weight-bounded caches such as the Iceberg metadata caches therefore report their true footprint and evict earlier for the same `*_memory_usage_ratio`, which lowers FE memory usage but may increase cache misses. [#75971](https://github.com/StarRocks/starrocks/pull/75971) * When the keep-alive GC reaps an expired external scan context — left behind by a Spark or Flink connector reader that died without calling `close_scanner` — it now cancels the corresponding pipeline fragment. The buffered scan memory is released within `keep_alive_min` plus one GC interval instead of staying pinned until `query_timeout`. [#76535](https://github.com/StarRocks/starrocks/pull/76535) * The audit log of a statement forwarded to the Leader FE now records the relations resolved by the Leader, so `QueriedRelations` holds fully qualified table names with CTE references excluded, matching what the Leader logs. A Follower falls back to local collection only when the statement ran locally or the Leader did not return the list, for example during a rolling upgrade. [#76387](https://github.com/StarRocks/starrocks/pull/76387) * The `Operation` column of `SHOW ALTER TABLE OPTIMIZE` now shows a readable description of the optimize operation instead of an internal object address such as `com.starrocks.sql.ast.OptimizeClause@b5dc069`. [#75948](https://github.com/StarRocks/starrocks/pull/75948) ##### Improvements[​](#improvements-1 "Direct link to Improvements") * Supported complex types (`ARRAY`, `MAP`, and `STRUCT`) in Paimon tables; querying such columns could previously crash the BE. [#66784](https://github.com/StarRocks/starrocks/pull/66784) * Added the `LAST_REFRESH_TIME` column to `information_schema.materialized_views` and `SHOW MATERIALIZED VIEWS`. It reports the data-freshness timestamp used by the `mv_rewrite_staleness_second` check, which is distinct from the existing `LAST_REFRESH_FINISHED_TIME` (when the refresh job finished running). [#71642](https://github.com/StarRocks/starrocks/pull/71642) * Added the mutable BE configuration item `object_storage_client_cache_size` (default `8`), which replaces the hard-coded capacity of the S3 and Azure Blob client caches. [#75851](https://github.com/StarRocks/starrocks/pull/75851) * Metadata refresh of a filesystem-backed external table triggered by `INSERT ... SELECT` no longer runs while FE internal metadata locks are held, so slow remote metadata access no longer stalls unrelated work on the same path. [#73391](https://github.com/StarRocks/starrocks/pull/73391) * Multi-statement (multi-table) transaction Stream Load now dispatches all per-table channels before waiting for them, instead of dispatching and waiting for each table in turn, which shortens commit time for CDC pipelines that write many tables under one label. [#76715](https://github.com/StarRocks/starrocks/pull/76715) * Reduced lock contention across the `INSERT OVERWRITE` code path by relaxing or shortening the table-level locks held in each phase. [#75828](https://github.com/StarRocks/starrocks/pull/75828) * Error messages for the large-column capacity limit no longer embed internal diagnostics such as the driver address and the operator chain, and the `Capaticy` typo in the shared status string is corrected. [#76303](https://github.com/StarRocks/starrocks/pull/76303) * Shared-data Primary Key publish now applies the per-segment delete vectors carried in `op_write.seg_delvecs`, so a BE running this version consumes such metadata correctly instead of leaving duplicate primary-key rows live. [#76474](https://github.com/StarRocks/starrocks/pull/76474) * Addressed security vulnerabilities (CVE): upgraded Thrift to 0.24.0 and Netty to 4.1.136.Final, bumped the PostgreSQL JDBC driver to 42.7.12, and removed vulnerable transitive dependencies that shipped alongside their fixed counterparts, namely `bcprov-jdk15on`, the end-of-life OkHttp 2.x line, `avro-ipc` (which bundles jQuery 1.4.2), and the Jetty client and security jars. [#76922](https://github.com/StarRocks/starrocks/pull/76922) [#76555](https://github.com/StarRocks/starrocks/pull/76555) [#76783](https://github.com/StarRocks/starrocks/pull/76783) [#76097](https://github.com/StarRocks/starrocks/pull/76097) [#76270](https://github.com/StarRocks/starrocks/pull/76270) ##### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") The following issues have been fixed: * The query cache could store an incomplete per-tablet result and serve it to later queries, returning wrong results. [#77066](https://github.com/StarRocks/starrocks/pull/77066) [#77404](https://github.com/StarRocks/starrocks/pull/77404) * `COUNT(DISTINCT)` on an Iceberg table partitioned by `bucket()` returned an over-count when `enable_bucket_aware_execution_on_lake` was enabled and the `GROUP BY` list was a superset of the bucket column. [#76601](https://github.com/StarRocks/starrocks/pull/76601) * JSON subfield pushdown returned wrong results when two subfield keys of the same column differed only in case (for example, `get_json_string(c, 'Campaign')` and `get_json_string(c, 'campaign')`), because the generated column names are resolved case-insensitively. Such collisions are now excluded from the pushdown. [#76594](https://github.com/StarRocks/starrocks/pull/76594) [#76593](https://github.com/StarRocks/starrocks/pull/76593) * `array_difference` on integer input computed each adjacent difference in the 32-bit input type before widening to the `BIGINT` result type, so a difference outside the `INT` range overflowed and returned a wrong value. [#76569](https://github.com/StarRocks/starrocks/pull/76569) * Aggregating a whole `STRUCT` column together with `ROLLUP`, `CUBE`, or `GROUPING SETS` failed at plan time with `StructType SlotRef must have an non-empty usedStructFiledPos`. [#76804](https://github.com/StarRocks/starrocks/pull/76804) * Rebuilding a logical window operator in the optimizer dropped the `inputIsBinary` flag, losing the binary-input merge behavior selected by ranking-window pre-aggregation. [#77058](https://github.com/StarRocks/starrocks/pull/77058) * A query could fail with `Expr evaluate meet error: slot_id N not found` because the partition-by columns of a PARTITION TOP-N node were rewritten to a dictionary slot that had already been decoded below the node. [#75956](https://github.com/StarRocks/starrocks/pull/75956) * Query planning, including `EXPLAIN`, aborted with an `IllegalStateException` when a contradictory range predicate such as `col > X AND col < X` collapsed to an empty value set and the same column was also referenced by a column-to-column join predicate. [#75011](https://github.com/StarRocks/starrocks/pull/75011) * Planning aborted when an aggregate over a multi-branch `CASE` with a non-null constant `ELSE` clause was considered for push-down below a join. [#75037](https://github.com/StarRocks/starrocks/pull/75037) * Selecting from views whose definitions form a cycle — which `ALTER VIEW` can create — failed with an opaque `Unknown error` caused by a `StackOverflowError`. Cyclic definitions are now detected and reported. [#75033](https://github.com/StarRocks/starrocks/pull/75033) * Grouping key columns of `GROUP BY ROLLUP`, `CUBE`, and `GROUPING SETS` were reported as non-nullable at analysis time, which produced an incorrect result schema for Arrow Flight SQL clients. [#76149](https://github.com/StarRocks/starrocks/pull/76149) * `array_contains` and `array_position` failed with `class com.starrocks.type.NullType cannot be cast to class com.starrocks.type.ArrayType` when the first argument was an untyped `NULL` literal. [#76970](https://github.com/StarRocks/starrocks/pull/76970) * For an `OR` predicate with more than 16 disjuncts, the merged null fraction of a column was always estimated as `1` instead of the average of the operands' null fractions, which distorted cardinality estimation. [#75864](https://github.com/StarRocks/starrocks/pull/75864) * Loading column statistics failed for a column whose values are all `NULL` when the global `sql_mode` includes `ERROR_IF_OVERFLOW`, because the stored empty min/max string could not be cast to the column type. [#76684](https://github.com/StarRocks/starrocks/pull/76684) * The FE scan-range heap-safety check ran once per physical partition instead of once per scan node, which burned minutes of FE CPU per plan attempt on a table with tens of thousands of physical partitions. [#76978](https://github.com/StarRocks/starrocks/pull/76978) * A predicate on a wildcard `CHAR` column, such as `CAST(json_col->'$.x' AS char)`, threw `std::length_error` because the column's declared length of `-1` was used for zero-padding. [#77444](https://github.com/StarRocks/starrocks/pull/77444) * The BE crashed when scan predicates were pulled up through an `array_map` lambda, because the rewrite descended into the lambda body. [#76380](https://github.com/StarRocks/starrocks/pull/76380) * The `STRUCT` output of `UNNEST` could be pruned narrower than the element type the BE materializes for the input array, when another consumer keeps that input array fully materialized. [#76002](https://github.com/StarRocks/starrocks/pull/76002) * Analyzing `__iceberg_transform_truncate` or `__iceberg_transform_bucket` mutated the shared builtin function object in place, stamping its wildcard decimal signature with the precision and scale of the first query analyzed. [#76777](https://github.com/StarRocks/starrocks/pull/76777) * Rewriting a query with two aggregates on the same base column of a synchronous materialized view or rollup, such as `min(c)` and `max(c)`, failed during cost estimation with `missing statistic of col: ... mv_min_c`. [#75528](https://github.com/StarRocks/starrocks/pull/75528) * A materialized view could serve stale results after `rollback_to_snapshot` on its Iceberg base table, because a negative staleness was accepted as within the `mv_rewrite_staleness_second` budget. [#75924](https://github.com/StarRocks/starrocks/pull/75924) * With `mv_rewrite_staleness_second` set and `query_rewrite_consistency = checked`, a chained partial refresh of one recently committed partition kept renewing the freshness of the whole materialized view while another partition lagged far beyond the tolerance. The staleness baseline is now the time the view was last confirmed fresh. [#76758](https://github.com/StarRocks/starrocks/pull/76758) * A materialized view defined on an un-partitioned Delta Lake or Kudu table was never used for query rewrite, even after a successful refresh. [#76359](https://github.com/StarRocks/starrocks/pull/76359) * The FE Iceberg manifest data-file cache could serve a cached file set that was missing a live data file while still passing the read-side completeness check, so scan planning silently dropped the file and the query returned a short result. [#76215](https://github.com/StarRocks/starrocks/pull/76215) * Queries on an Iceberg V1 table failed after a partition field was dropped, because a dropped V1 partition field is retained in the spec with a void transform and keeps the name of its source column. [#75149](https://github.com/StarRocks/starrocks/pull/75149) * An Iceberg REST catalog configured with an OAuth2 client credential failed every request until the catalog was recreated, once its background token refresh had exhausted its retry budget. The catalog now rebuilds its session and retries the request once, at most one rebuild per 60 seconds; catalogs using `jwt` security or a static token are not affected. [#76457](https://github.com/StarRocks/starrocks/pull/76457) * Closing an Iceberg incremental scan-range iterator while the executor thread was still consuming it, which happens on query cancellation, was unsafe. [#75953](https://github.com/StarRocks/starrocks/pull/75953) * Iceberg REST catalog vended credentials for GCS were ignored and FE metadata reads failed with `403 Forbidden`, because the gcs-connector 3.x configuration keys were renamed. [#75979](https://github.com/StarRocks/starrocks/pull/75979) * Querying a table in a Hive catalog intermittently failed with `out of sequence response` and then `Unknown table`, because the `getTable()` fallback reused the same Thrift connection after a read timeout without reconnecting. [#76456](https://github.com/StarRocks/starrocks/pull/76456) * A Paimon predicate that cannot be converted, such as a comparison on a `CASE` expression, discarded the whole conjunction instead of keeping the convertible conjuncts. [#66038](https://github.com/StarRocks/starrocks/pull/66038) * `BOOLEAN` columns were excluded from page-level predicate pushdown in Parquet files, because the Parquet column index min/max decoder had no `BOOLEAN` case. [#74752](https://github.com/StarRocks/starrocks/pull/74752) * A streaming pre-aggregation running with `enable_spill = true` could run out of memory, because the memory budget latched when the sink downgraded to limited-memory mode was computed only once and could be frozen at `0`. [#76702](https://github.com/StarRocks/starrocks/pull/76702) * The data load path and the column-mode partial update path could build a chunk whose `ARRAY` or string column exceeded the addressable size, leaving the offsets no longer describing the buffer they belong to. Both paths now bound and check the chunk capacity. [#77163](https://github.com/StarRocks/starrocks/pull/77163) * Columns missing from a Parquet file were null-padded to the whole Arrow batch size instead of the bounded chunk size, so a batch processed across several chunks produced unequal column lengths within one chunk and crashed. [#75981](https://github.com/StarRocks/starrocks/pull/75981) * A replica in `DECOMMISSION` state could be elected as the primary replica of a load. Because it is the replica most likely to be removed while the load is still running, the whole load then failed with `Fail to get tablet ...` instead of the write quorum absorbing a single-replica failure. Such a replica is now skipped unless no other healthy candidate exists. [#77035](https://github.com/StarRocks/starrocks/pull/77035) * The BE crashed during a load spill because `LoadChunkSpiller` used an unsynchronized null check as its initialization flag, letting a racing memtable-flush thread use a spiller whose serde was not ready yet. [#76098](https://github.com/StarRocks/starrocks/pull/76098) * A multi-character CSV delimiter that straddled a buffer expansion caused a heap use-after-free in the CSV reader, reachable through the `FILES()` table function, Broker Load, and the Hive text connector. [#76718](https://github.com/StarRocks/starrocks/pull/76718) * Loading malformed JSON crashed the BE with a heap buffer overflow, because the data-quality error message was built from an unbounded raw JSON pointer. [#76752](https://github.com/StarRocks/starrocks/pull/76752) * Cancelling a load crashed the BE or CN with a SIGSEGV when tracing was enabled through `jaeger_endpoint`, because the sink's `close_wait` was not idempotent. [#76869](https://github.com/StarRocks/starrocks/pull/76869) * Aborting a failed `INSERT INTO FILES()` threw a swallowed NPE, because the abort path looked up a database that a table function table does not have. [#75983](https://github.com/StarRocks/starrocks/pull/75983) * `INSERT OVERWRITE` garbage collection could journal a failed-state change for a table that had already been dropped, because the target table was resolved before the table write lock was acquired. [#77212](https://github.com/StarRocks/starrocks/pull/77212) * `SHOW CREATE ROUTINE LOAD` emitted the `jsonpaths` value without escaping its double quotes, producing DDL that cannot be parsed or replayed. [#75755](https://github.com/StarRocks/starrocks/pull/75755) * On a Primary Key table with file bundling, retrying an aggregate publish could probe remote storage for a version that exists only in the metadata cache and leave a dangling `prev_garbage_version`. Durable metadata is now read when calculating the new base version. [#75904](https://github.com/StarRocks/starrocks/pull/75904) * A shared-data publish could write a bundle tablet metadata file that was missing tablet pages, which left the partition's publish permanently stuck with `can not find tablet ... from shared tablet metadata`. Such a file is now refused, so the failure stays transient and retryable. [#76850](https://github.com/StarRocks/starrocks/pull/76850) * A CN crashed on a physical-split scan of an empty tablet, because `SparseRangeIterator::has_more()` was not null-safe and a transient lake segment-load failure was swallowed instead of being surfaced as a retryable error. [#75985](https://github.com/StarRocks/starrocks/pull/75985) * A CN crashed while building a segment for a materialized view carrying a legacy NGRAMBF index that older FE versions persisted without its `gram_num` property. [#76989](https://github.com/StarRocks/starrocks/pull/76989) * The BE crashed when the build side of an aggregation `IN` runtime filter was a constant column. [#74941](https://github.com/StarRocks/starrocks/pull/74941) * Cancelling a query could cause a use-after-free in the spillable hash-join build operator, because its `set_finishing` ran the spill-start path even when the runtime state was already cancelled. [#76633](https://github.com/StarRocks/starrocks/pull/76633) * A BE or CN could abort during shutdown with `std::bad_weak_ptr`, because the global runtime-filter timer of an abandoned pipeline driver was never unscheduled. [#76252](https://github.com/StarRocks/starrocks/pull/76252) * Memory leaked because the bRPC stub cache cleanup timer task was unscheduled but never removed. [#75973](https://github.com/StarRocks/starrocks/pull/75973) * A query's `ConnectContext`, together with the whole `ExecPlan` object graph it references, stayed pinned in the thread-local of a pooled query-deploy worker. [#76366](https://github.com/StarRocks/starrocks/pull/76366) * `PipeObservable::defer_notify_sink()` emitted a source event instead of a sink event, so a driver blocked on `OUTPUT_FULL` could stay blocked until an unrelated event arrived. [#76782](https://github.com/StarRocks/starrocks/pull/76782) * An exception thrown by a `ThreadPool` task was swallowed by default and the task was still counted as finished. [#76863](https://github.com/StarRocks/starrocks/pull/76863) * The BE kept running with the crashing flag set, and therefore kept reporting `SHUTDOWN` to the FE, when the fatal-signal handler stalled before re-raising the signal. The process is now forced to exit. [#76491](https://github.com/StarRocks/starrocks/pull/76491) * Dropping a catalog verified its existence under a read lock and removed it under a separate write lock, so two concurrent drops of the same catalog could both pass the check. [#76778](https://github.com/StarRocks/starrocks/pull/76778) * Java UDFs failed on JDK 21 and later with `NoSuchMethodException: java.nio.DirectByteBuffer.(long,int)`, because that private constructor was removed in JDK 21. [#75666](https://github.com/StarRocks/starrocks/pull/75666) * Behind a load balancer, every ADBC client that opens with a prepared statement failed, because a `CreatePreparedStatement` or `ClosePreparedStatement` forwarded to another FE carried the protobuf message name instead of the Flight SQL action type. [#76310](https://github.com/StarRocks/starrocks/pull/76310) * The Arrow LZ4 and ZSTD IPC codecs were missing from the FE classpath of the Arrow Flight SQL service. [#76921](https://github.com/StarRocks/starrocks/pull/76921) #### 4.0.13[​](#4013 "Direct link to 4.0.13") Release Date: July 16, 2026 ##### Behavior Changes[​](#behavior-changes-2 "Direct link to Behavior Changes") * The escape handling of `LIKE` predicates with constant operands (folded on the FE) now matches MySQL 8: `SELECT 'a\\b' LIKE 'a\\\\b'` returns `1` and `SELECT 'a\\b' LIKE 'a\\b'` returns `0`. Queries that relied on the previous non-MySQL escaping semantics will return different results. [#74814](https://github.com/StarRocks/starrocks/pull/74814) * `SHOW [FULL] FUNCTIONS` now always includes the `isolation` property (`shared` or `isolated`) in the Properties column of UDFs, so users can tell whether the property is set without recreating the function. [#75255](https://github.com/StarRocks/starrocks/pull/75255) * Iceberg REST catalogs with vended credentials use the table metadata cache again, reverting the earlier cache bypass that sent every `getTable()` to the REST catalog and caused AWS Lake Formation `Rate exceeded` failures. Cached tables now renew their credentials on every refresh cycle, and the table cache expiry for REST catalogs is additionally capped at 3000 seconds. [#75431](https://github.com/StarRocks/starrocks/pull/75431) ##### Improvements[​](#improvements-2 "Direct link to Improvements") * Added checksum protection for shared-data tablet metadata and transaction logs. [#74924](https://github.com/StarRocks/starrocks/pull/74924) * Supported combined transaction log / file bundling for `FRONTEND_STREAMING` loads. [#74460](https://github.com/StarRocks/starrocks/pull/74460) * Scoped shared-data schema-change job locks to the table to reduce lock contention. [#75087](https://github.com/StarRocks/starrocks/pull/75087) * Batch tablet force-delete marking now acquires the `TabletInvertedIndex` write lock once per batch instead of once per tablet. [#75616](https://github.com/StarRocks/starrocks/pull/75616) * Added an FE metric for the maximum pending-publish time of committed transactions. [#75025](https://github.com/StarRocks/starrocks/pull/75025) * Added a memory limit check for column upgrades in window operator processing. [#75821](https://github.com/StarRocks/starrocks/pull/75821) * Removed unnecessary per-row seeks in the offsets-only read path of array columns. [#75861](https://github.com/StarRocks/starrocks/pull/75861) * Foreground row-count estimation of Iceberg tables is now computed from manifest metadata without enumerating every data file. [#75280](https://github.com/StarRocks/starrocks/pull/75280) * Addressed security vulnerabilities (CVE): excluded the vulnerable `org.jline:jline` (jline-remote-telnet) from Hadoop transitive dependencies, and upgraded jackson-databind to 2.21.4. [#75066](https://github.com/StarRocks/starrocks/pull/75066) [#75373](https://github.com/StarRocks/starrocks/pull/75373) ##### Bug Fixes[​](#bug-fixes-2 "Direct link to Bug Fixes") The following issues have been fixed: * Incremental scan-range scheduling could recompute a different per-driver layout when reusing a deployed fragment instance, leaving part of the scan ranges unconsumed and losing rows (for example, INSERT from Hive). [#74674](https://github.com/StarRocks/starrocks/pull/74674) * INT96 timestamps nested inside ARRAY/MAP/STRUCT in Parquet files read via `FILES()` or Broker Load missed the session-timezone conversion and were returned shifted by the timezone offset (top-level INT96 columns were correct). [#74868](https://github.com/StarRocks/starrocks/pull/74868) * The audit log did not record the exported row count of `SELECT INTO OUTFILE`. [#74467](https://github.com/StarRocks/starrocks/pull/74467) * A strict cast could raise an overflow error from the underlying data of NULL rows, which should be ignored. [#74903](https://github.com/StarRocks/starrocks/pull/74903) * `parse_json` did not respect the `ALLOW_THROW_EXCEPTION` setting when handling invalid input. [#74976](https://github.com/StarRocks/starrocks/pull/74976) * First-load statistics collection could not be enabled per table while disabled globally: an explicitly set table property now takes precedence over the global configuration `enable_statistic_collect_on_first_load`. [#74794](https://github.com/StarRocks/starrocks/pull/74794) * Partial column updates on shared-data tables could crash the BE or silently corrupt data when the tablet schema drifted from the transaction schema. [#74005](https://github.com/StarRocks/starrocks/pull/74005) * Unexpected BE process restarts. [#74424](https://github.com/StarRocks/starrocks/pull/74424) * A BE crash (SIGFPE) in Iceberg `truncate`/`bucket` partition transforms when the width or bucket count is zero. [#74998](https://github.com/StarRocks/starrocks/pull/74998) * A BE crash caused by a null `driver_executor` in `FragmentContext::set_final_status`. [#75030](https://github.com/StarRocks/starrocks/pull/75030) * A race between transaction begin and autovacuum could delete a still-needed transaction log, permanently wedging the partition's publish ("Both txn\_log and corresponding tablet\_meta missing"). [#74906](https://github.com/StarRocks/starrocks/pull/74906) * `avg(DISTINCT x)` was incorrectly rewritten to use a sum/count materialized view, returning wrong results. [#75071](https://github.com/StarRocks/starrocks/pull/75071) * A boundary bug in TopN RANK sorting could produce incorrect results. [#75045](https://github.com/StarRocks/starrocks/pull/75045) * `split` / `split_part` / `str_to_map` with an empty delimiter could read out of bounds on invalid UTF-8 input. [#75068](https://github.com/StarRocks/starrocks/pull/75068) * `ALTER TABLE ... MODIFY COLUMN ... AFTER` a nonexistent column now returns a clear error message. [#75073](https://github.com/StarRocks/starrocks/pull/75073) * A BE crash (SIGFPE) in `mod()` / `pmod()` when computing the type's minimum value modulo -1. [#74980](https://github.com/StarRocks/starrocks/pull/74980) * `bar()` grew memory without bound on a negative or huge width (potential DoS); such inputs are now rejected with an error. [#75143](https://github.com/StarRocks/starrocks/pull/75143) * The transaction-state callback was not unregistered when a multi-statement stream load task was removed. [#75188](https://github.com/StarRocks/starrocks/pull/75188) * A crash when a query was cancelled during spill partition sorting. [#75140](https://github.com/StarRocks/starrocks/pull/75140) * The query memory limit was not enforced during table function execution. [#75179](https://github.com/StarRocks/starrocks/pull/75179) * Selecting a column named `floor` or `ceil` failed at parse time with a ClassCastException. [#75241](https://github.com/StarRocks/starrocks/pull/75241) * A heap-use-after-free in `OrderedPartitionExchanger` when the previous chunk was mutated downstream. [#75279](https://github.com/StarRocks/starrocks/pull/75279) * Three FE metadata-lock correctness races. [#74968](https://github.com/StarRocks/starrocks/pull/74968) * Load spilling could dereference a missing query context when recording spill metrics. [#75236](https://github.com/StarRocks/starrocks/pull/75236) * ADLS2 `ListPaths` on storage accounts without hierarchical namespace caused CN crashes and vacuum failures. [#75166](https://github.com/StarRocks/starrocks/pull/75166) * BE/CN JVM metrics emitted invalid Prometheus `# TYPE` lines. [#75240](https://github.com/StarRocks/starrocks/pull/75240) * JIT code generation truncated LARGEINT literals to 64 bits, producing wrong results. [#75137](https://github.com/StarRocks/starrocks/pull/75137) * A combined ALTER TABLE on an external Iceberg table re-executed already-queued actions. [#74036](https://github.com/StarRocks/starrocks/pull/74036) * A nested-loop join crash caused by a build-side column nullability mismatch. [#75343](https://github.com/StarRocks/starrocks/pull/75343) * Iceberg manifest column statistics are now cached selectively to avoid excessive FE memory usage. [#75395](https://github.com/StarRocks/starrocks/pull/75395) * `addPhysicalPartition` could create only one physical partition per call, making physical-partition backfill of random-distribution tables extremely slow. [#75430](https://github.com/StarRocks/starrocks/pull/75430) * A SQL injection vulnerability in the `information_schema.task_runs` predicate lookup. [#75520](https://github.com/StarRocks/starrocks/pull/75520) * A BE crash (SIGSEGV) when multiple UNNEST calls in one query referenced the same array column. [#75012](https://github.com/StarRocks/starrocks/pull/75012) * Incorrect translation of nested dictionary-encoded expressions across exchange nodes. [#75246](https://github.com/StarRocks/starrocks/pull/75246) * The join skew hint was lost when the optimizer duplicated expressions. [#68964](https://github.com/StarRocks/starrocks/pull/68964) * Collecting the tables referenced by a view did not skip CTE references. [#74813](https://github.com/StarRocks/starrocks/pull/74813) * A BE abort on `CAST(json AS STRUCT<...>)` when a struct field name is not a valid JSON path. [#75355](https://github.com/StarRocks/starrocks/pull/75355) * Temporary Parquet dictionary-code columns could leak to upper layers of the execution plan. [#74452](https://github.com/StarRocks/starrocks/pull/74452) * Incorrect results caused by the sort-column elimination optimization. [#74983](https://github.com/StarRocks/starrocks/pull/74983) * The UNNEST result struct type was not narrowed consistently with its input array element type. [#75445](https://github.com/StarRocks/starrocks/pull/75445) * A race between the FE EOS-cancel and the BE stage-2 deployment could mark a successfully finished query as cancelled. [#75009](https://github.com/StarRocks/starrocks/pull/75009) * Join reorder pruning could drop columns that predicates still referenced. [#74791](https://github.com/StarRocks/starrocks/pull/74791) * The global metadata lock was not released when a metadata image dump failed to acquire some database locks, blocking subsequent metadata operations. [#75488](https://github.com/StarRocks/starrocks/pull/75488) * `StringSearch::_pattern` was left uninitialized, and the `split_debug_symbol` log output was incorrect. [#75614](https://github.com/StarRocks/starrocks/pull/75614) * `LIKE` with the single-character wildcard `_` returned wrong results on columns with a GIN inverted index. [#75551](https://github.com/StarRocks/starrocks/pull/75551) * The segment-iterator vector could get positionally misaligned during shared-data primary key index rebuild. [#74887](https://github.com/StarRocks/starrocks/pull/74887) * With file bundling enabled, vacuum could delete a bundle file still referenced by sibling tablets because orphaned bundled segments were not flagged as shared; subsequent publishes then failed with "Object ... does not exist". [#75689](https://github.com/StarRocks/starrocks/pull/75689) * `histogram()` crashed on a non-positive bucket count; it now returns an error. [#75041](https://github.com/StarRocks/starrocks/pull/75041) * A compaction crash (CHECK failure in `JsonMergeIterator`) when a flat-JSON column changed from NOT NULL to nullable. [#75680](https://github.com/StarRocks/starrocks/pull/75680) * Common predicate operators were lost when the join tuning guide rebuilt a join. [#75773](https://github.com/StarRocks/starrocks/pull/75773) * An `UnsupportedOperationException` in `ApplyTuningGuideRule` caused by an immutable inputs list in `OptExpression`. [#70785](https://github.com/StarRocks/starrocks/pull/70785) * `SHOW PARTITIONS` and `partitions_meta` reported the logical-partition bucket count instead of the per-physical-partition count for shared-data tables. [#75734](https://github.com/StarRocks/starrocks/pull/75734) * A memory allocation exception in `NLJoinProbeOperator::pull_chunk` crashed the BE instead of failing the query. [#75788](https://github.com/StarRocks/starrocks/pull/75788) * `SHOW CREATE ROUTINE LOAD` emitted a spurious comma before the first load-property clause, making the output non-executable. [#75522](https://github.com/StarRocks/starrocks/pull/75522) * CTAS (`CREATE TABLE AS SELECT`) did not accept an `ENGINE` clause, making CTAS into a Unified catalog impossible. [#75771](https://github.com/StarRocks/starrocks/pull/75771) * OR predicates over null-safe-equal (`<=>`) join conditions were incorrectly rewritten to `UNION ALL`, returning wrong results. [#75038](https://github.com/StarRocks/starrocks/pull/75038) * A query cache normalization crash for tables with sub-partitions. [#75789](https://github.com/StarRocks/starrocks/pull/75789) * `array_map` / `transform` dropped NULL rows when all non-null input arrays were empty. [#75141](https://github.com/StarRocks/starrocks/pull/75141) * `regexp_extract_all` fell into an infinite loop on a zero-length capture group. [#75798](https://github.com/StarRocks/starrocks/pull/75798) * Flat-JSON subfield reads returned NULL for keys whose name contains `.`, which could also cause silent row loss when predicates were pushed down. [#75583](https://github.com/StarRocks/starrocks/pull/75583) * UNNEST output struct subfields were pruned by the output's own access group instead of the input array's, which could prune subfields that were still needed. [#76002](https://github.com/StarRocks/starrocks/pull/76002) * The FE Iceberg manifest data-file cache could serve an incomplete file set, silently dropping a data file from scan planning and under-counting query results. [#76215](https://github.com/StarRocks/starrocks/pull/76215) * The Iceberg table metadata cache was not invalidated after an INSERT commit, so subsequent queries could miss the latest snapshot. [#67230](https://github.com/StarRocks/starrocks/pull/67230) #### 4.0.12[​](#4012 "Direct link to 4.0.12") Release Date: June 25, 2026 ##### Behavior Changes[​](#behavior-changes-3 "Direct link to Behavior Changes") * When reading INT64 timestamps from Parquet files written with `isAdjustedToUTC=false` (timezone-naive), `SELECT FROM FILES()` and broker/stream LOAD no longer shift the values by the session timezone offset. Such timestamps are now read as wall-clock values, consistent with Trino, Spark, and Impala. Previously the values drifted whenever the session timezone was not UTC. [#73674](https://github.com/StarRocks/starrocks/pull/73674) * CTAS (`CREATE TABLE AS SELECT`) now preserves the declared `VARCHAR(N)` length when the source carries an explicit user length (a catalog column reference, `CAST AS VARCHAR(N)`, or a string literal), instead of widening it to `VARCHAR(1048576)`. This keeps the length constraint enforceable and aligns DDL with dbt schema contracts. Materialized view materialization still widens columns as before. [#73498](https://github.com/StarRocks/starrocks/pull/73498) * The Paimon connector now respects the session variable `connector_max_split_size` when calculating scan splits, instead of always using the default value, so tuning it now affects Paimon scan parallelism. [#71756](https://github.com/StarRocks/starrocks/pull/71756) ##### Improvements[​](#improvements-3 "Direct link to Improvements") * Optimized `base64_to_bitmap` by folding the conversion at constant-evaluation time for constant inputs. [#74684](https://github.com/StarRocks/starrocks/pull/74684) * `ngram_search` now supports a non-constant needle (the search term can be a column expression rather than only a constant). [#74675](https://github.com/StarRocks/starrocks/pull/74675) * The Arrow-to-JSON converter now supports `LARGE_LIST` and `FIXED_SIZE_LIST` types. [#73714](https://github.com/StarRocks/starrocks/pull/73714) * Added an opt-in option to isolate wide-string columns during statistics collection to reduce memory pressure. [#73258](https://github.com/StarRocks/starrocks/pull/73258) * `information_schema.COLUMNS` now populates the `DATETIME_PRECISION` field. [#74623](https://github.com/StarRocks/starrocks/pull/74623) * Relaxed database read locks to table-scoped intensive locks in `InformationSchemaDataSource` and `FrontendServiceImpl` to improve concurrency. [#73936](https://github.com/StarRocks/starrocks/pull/73936) [#73913](https://github.com/StarRocks/starrocks/pull/73913) * Narrowed database write locks to table-scoped intensive write locks for shared-nothing clusters, and scoped replica row-count updates to the table lock. [#74523](https://github.com/StarRocks/starrocks/pull/74523) [#74521](https://github.com/StarRocks/starrocks/pull/74521) * Moved the routine-load broker RPC out of the per-job write lock to reduce contention. [#73591](https://github.com/StarRocks/starrocks/pull/73591) * Deferred JDBC `REMARKS` fetching out of the `getTable()` hot path to speed up metadata access for JDBC catalogs. [#73488](https://github.com/StarRocks/starrocks/pull/73488) * Pushed down the `table_name` predicate for `information_schema.tables_config` queries. [#73210](https://github.com/StarRocks/starrocks/pull/73210) * Skipped per-replica scans on single-medium BEs in `BackendLoadStatistic`. [#73555](https://github.com/StarRocks/starrocks/pull/73555) * Added a write timeout to the MySQL channel result send path to prevent stuck connections. [#73646](https://github.com/StarRocks/starrocks/pull/73646) * Added catalog recycle bin size gauge metrics. [#74440](https://github.com/StarRocks/starrocks/pull/74440) * Added vacuum batch-size and retry-count metrics, and added decorrelated jitter to the lake vacuum retry backoff to reduce retry storms. [#74112](https://github.com/StarRocks/starrocks/pull/74112) [#74108](https://github.com/StarRocks/starrocks/pull/74108) * Upgraded third-party dependencies to address security vulnerabilities (CVE): Netty to 4.1.135.Final, Tomcat to 9.0.118, and Thrift to 0.23.0. [#74668](https://github.com/StarRocks/starrocks/pull/74668) [#73797](https://github.com/StarRocks/starrocks/pull/73797) [#73625](https://github.com/StarRocks/starrocks/pull/73625) ##### Bug Fixes[​](#bug-fixes-3 "Direct link to Bug Fixes") The following issues have been fixed: * Successfully committed multi-statement transaction stream loads were shown as `PREPARING` forever in `information_schema.loads` and `SHOW STREAM LOAD`. [#74386](https://github.com/StarRocks/starrocks/pull/74386) * Rows were silently dropped from `information_schema.loads` on clusters whose session timezone differs from Asia/Shanghai, because load times were exchanged as naive wall-clock strings across the BE/FE thrift boundary. [#73365](https://github.com/StarRocks/starrocks/pull/73365) * The `COMMIT` of an explicit transaction waited only `query_timeout` milliseconds (instead of seconds) for the database write lock due to a unit mismatch. [#73549](https://github.com/StarRocks/starrocks/pull/73549) * `current_timestamp` / `now()` column defaults were displayed as a frozen literal after `ALTER TABLE ... ADD COLUMN` and could be lost across FE restarts or edit-log replay. [#73455](https://github.com/StarRocks/starrocks/pull/73455) * Querying `sys.fe_memory_usage` / `sys.fe_locks` without the `OPERATE ON SYSTEM` privilege returned a misleading RPC-failure message instead of a clear access-denied error. [#73567](https://github.com/StarRocks/starrocks/pull/73567) * Automatic per-key Hive partition stats refresh could overload the Hive Metastore for tables with many partitions. [#73563](https://github.com/StarRocks/starrocks/pull/73563) * A null-pointer issue when reading the GTID during a schema change. [#74855](https://github.com/StarRocks/starrocks/pull/74855) * An empty analytic operator was not pruned after pushing down a distinct aggregation. [#74810](https://github.com/StarRocks/starrocks/pull/74810) * Zero row counts could corrupt partition statistics. [#74801](https://github.com/StarRocks/starrocks/pull/74801) * Vector index rewrite could pollute the shared table schema. [#74785](https://github.com/StarRocks/starrocks/pull/74785) * An `IllegalStateException` during parallel profile collection, fixed by making Tracers fork-aware. [#74746](https://github.com/StarRocks/starrocks/pull/74746) * BE vacuum tasks were not aborted once the FE caller's timeout elapsed. [#74694](https://github.com/StarRocks/starrocks/pull/74694) * Partition consumer errors in `ChunksPartitioner` were lost instead of being propagated. [#74693](https://github.com/StarRocks/starrocks/pull/74693) * A lock mismatch in `blockingAddTabletCtxToScheduler`. [#74596](https://github.com/StarRocks/starrocks/pull/74596) * A typo in the `azure_adls2_oauth2_client_endpoint` configuration field name. [#74581](https://github.com/StarRocks/starrocks/pull/74581) * Pipeline observers were not notified on missed operator state transitions. [#74557](https://github.com/StarRocks/starrocks/pull/74557) * The reported vacuum watermark was incorrect when retain-boundary metadata was gone. [#74429](https://github.com/StarRocks/starrocks/pull/74429) * A data race on `MaterializedIndexMeta` during `updateSchemaBackendId`. [#74412](https://github.com/StarRocks/starrocks/pull/74412) * A non-primary-key replica could get stuck with a permanent version hole; it now self-heals. [#74408](https://github.com/StarRocks/starrocks/pull/74408) * A use-after-free of `LLVMContext` when JIT compilation fails. [#74396](https://github.com/StarRocks/starrocks/pull/74396) * A column mismatch in the missing-replica row of `ADMIN SHOW REPLICA STATUS`. [#74393](https://github.com/StarRocks/starrocks/pull/74393) * Invalid JIT IR generated for `CASE WHEN` with mixed float/int WHEN and result types. [#74382](https://github.com/StarRocks/starrocks/pull/74382) * The `CatalogRecycleBin` was frozen when a cluster snapshot kept failing. [#74379](https://github.com/StarRocks/starrocks/pull/74379) * A partial update targeting a table modified earlier in the same explicit transaction is now rejected with a clear error. [#74344](https://github.com/StarRocks/starrocks/pull/74344) * Immutable-partition updates did not use the transaction's compute resource. [#74316](https://github.com/StarRocks/starrocks/pull/74316) * A potential out-of-bounds error caused by partitioned join. [#74315](https://github.com/StarRocks/starrocks/pull/74315) * Database-level UDFs were not restored to the renamed target database for FE followers. [#74313](https://github.com/StarRocks/starrocks/pull/74313) * Non-root compound predicates yielded `EOF` instead of `NotPushDown`. [#74218](https://github.com/StarRocks/starrocks/pull/74218) * Table names were not backquoted when persisting the routine load `origStmt`. [#74188](https://github.com/StarRocks/starrocks/pull/74188) * An assertion name lookup error in assert-num-rows. [#74178](https://github.com/StarRocks/starrocks/pull/74178) * Aggregation used type-mismatched aggregate functions. [#74159](https://github.com/StarRocks/starrocks/pull/74159) * Force-killed task runs were not archived, and session-prefixed task-run timeouts were not honored. [#74146](https://github.com/StarRocks/starrocks/pull/74146) * `RENAME` and `SWAP` (for tables and materialized views) now take the database write lock to avoid concurrent-modification issues. [#74100](https://github.com/StarRocks/starrocks/pull/74100) * Composite-rowset stats were not summed when batching op\_writes in primary-key multi-statement transactions. [#74059](https://github.com/StarRocks/starrocks/pull/74059) * The sink was not notified when a distinct aggregate source finished. [#74055](https://github.com/StarRocks/starrocks/pull/74055) * `pipe_file_list` was not recreated when `_statistics_` was dropped. [#73970](https://github.com/StarRocks/starrocks/pull/73970) * A crash in `TabletInvertedIndex.deleteTablets`, fixed by fast-path skipping empty input. [#73955](https://github.com/StarRocks/starrocks/pull/73955) * The task manager could write an illegal edit log for a task run. [#73882](https://github.com/StarRocks/starrocks/pull/73882) * A `set_thread_name` race on data directory load threads. [#73862](https://github.com/StarRocks/starrocks/pull/73862) * A race condition in `TabletSinkSender::_send_chunk_by_node`. [#73820](https://github.com/StarRocks/starrocks/pull/73820) * Incorrect memory accounting in `OlapTableSink`. [#73807](https://github.com/StarRocks/starrocks/pull/73807) * Incorrect connector bytes-read statistics. [#73799](https://github.com/StarRocks/starrocks/pull/73799) * `BACKUP ON (ALL FUNCTION)` / `(ALL EXTERNAL CATALOGS)` failures. [#73790](https://github.com/StarRocks/starrocks/pull/73790) * `NullableColumnUnaryFunction` lost the decimal scale when all values were null. [#73789](https://github.com/StarRocks/starrocks/pull/73789) * A FlatJSON crash when subwriters had no appends. [#73730](https://github.com/StarRocks/starrocks/pull/73730) * `LargeList` / `FixedSizeList` could not be converted to a JSON column during broker load. [#73718](https://github.com/StarRocks/starrocks/pull/73718) * Partial-append of nested types failed during JSON load. [#73715](https://github.com/StarRocks/starrocks/pull/73715) * An NPE in `StatisticsCalcUtils` when a partition is dropped concurrently. [#73711](https://github.com/StarRocks/starrocks/pull/73711) * Decimal-valued unit counters were not parsed in `RuntimeProfileParser`. [#73683](https://github.com/StarRocks/starrocks/pull/73683) * An NPE during nested materialized view refresh. [#73644](https://github.com/StarRocks/starrocks/pull/73644) * Multi-statement transactions were not handled correctly in lake `publish_log_version`. [#73423](https://github.com/StarRocks/starrocks/pull/73423) * An empty `ALTER TABLE` clause is now rejected, and `OPTIMIZE` replay was improved. [#73352](https://github.com/StarRocks/starrocks/pull/73352) * Disk cache overflow when Iceberg metadata entries are pinned. [#71651](https://github.com/StarRocks/starrocks/pull/71651) #### 4.0.11[​](#4011 "Direct link to 4.0.11") Release Date: June 5, 2026 ##### Behavior Changes[​](#behavior-changes-4 "Direct link to Behavior Changes") * `get_json_string` and the other `get_json_*` functions now return the JSON parse error instead of NULL when implicit VARCHAR-to-JSON parsing fails under `ALLOW_THROW_EXCEPTION`. The default behavior (returning NULL when the mode is disabled) is unchanged. [#73199](https://github.com/StarRocks/starrocks/pull/73199) * `pipeline_enable_large_column_checker` is now enabled by default. [#72798](https://github.com/StarRocks/starrocks/pull/72798) ##### Improvements[​](#improvements-4 "Direct link to Improvements") * Lake write-path load spill files now use a flat, single-level directory layout with the transaction ID baked into each filename, and are reclaimed by a txn-id-based vacuum pass. This moves bulk deletes off the write hot path and lets vacuum clean up spill files leaked by BE crashes. [#73064](https://github.com/StarRocks/starrocks/pull/73064) * SHOW statements (such as `SHOW GRANTS` and `SHOW WAREHOUSES`) are now allowed inside an explicit transaction, so BI/JDBC clients that automatically issue SHOW no longer break the transaction flow. [#72954](https://github.com/StarRocks/starrocks/pull/72954) * Java UDAF and UDTF now support STRUCT arguments and return types. [#72911](https://github.com/StarRocks/starrocks/pull/72911) * Scalar Java UDF now supports STRUCT arguments. [#72620](https://github.com/StarRocks/starrocks/pull/72620) * Java UDF now supports DATE and DATETIME types. [#72337](https://github.com/StarRocks/starrocks/pull/72337) * Java UDF now supports nested ARRAY/MAP types. [#72283](https://github.com/StarRocks/starrocks/pull/72283) * Added the FE configuration `deploy_serialization_min_thread_pool_size`. [#72274](https://github.com/StarRocks/starrocks/pull/72274) * Skipped redundant partition key expression building when an `add_partition_value` deduplication hit occurs. [#73156](https://github.com/StarRocks/starrocks/pull/73156) * Avoided a redundant `latestSnapshot()` call in `PaimonMetadata#getTableVersionRange`. [#72892](https://github.com/StarRocks/starrocks/pull/72892) * Deduplicated commutative AND/OR expressions in scalar operator common subexpression elimination. [#72823](https://github.com/StarRocks/starrocks/pull/72823) ##### Bug Fixes[​](#bug-fixes-4 "Direct link to Bug Fixes") The following issues have been fixed: * A memory leak introduced by the UDAF cache. [#74025](https://github.com/StarRocks/starrocks/pull/74025) * An incorrect implementation in aggregate combined functions. [#74169](https://github.com/StarRocks/starrocks/pull/74169) * An issue in shared-data combined txn log mode where the per-partition coordinator claim was not re-recorded on every sender's open, which could drop txn logs. [#73962](https://github.com/StarRocks/starrocks/pull/73962) * A read failure on Iceberg tables that use a custom `LocationProvider`, fixed by lazily initializing the `LocationProvider` in `SerializableTable`. [#73482](https://github.com/StarRocks/starrocks/pull/73482) * A serialization failure caused by the `de.javakaffee` `UnmodifiableCollectionsSerializer`, now replaced with a Java 17-compatible version. [#73458](https://github.com/StarRocks/starrocks/pull/73458) * `HdfsFsManager` copy error messages now include the underlying cause. [#73414](https://github.com/StarRocks/starrocks/pull/73414) * A concurrent `SegmentFlushTask` race in `DeltaWriter::commit()`. [#73371](https://github.com/StarRocks/starrocks/pull/73371) * Sort merge provider errors are now propagated to the fragment context instead of being lost. [#73337](https://github.com/StarRocks/starrocks/pull/73337) * An issue where Ranger row-filter/masking policies on Hive views were skipped, so policies on the view or its base tables were not applied. [#73265](https://github.com/StarRocks/starrocks/pull/73265) * Upgraded libthrift to 0.23.0 to address a security vulnerability (CVE). [#73243](https://github.com/StarRocks/starrocks/pull/73243) * An FE file-descriptor leak, fixed by reusing `HttpClient` instances. [#73239](https://github.com/StarRocks/starrocks/pull/73239) * Parquet broker load errors now include file/column/row context. [#73236](https://github.com/StarRocks/starrocks/pull/73236) * A slot lookup failure for output slots with an empty `col_name` in the Spark connector external scan. [#73225](https://github.com/StarRocks/starrocks/pull/73225) * A crash in `SinkBuffer` during graceful exit. [#73202](https://github.com/StarRocks/starrocks/pull/73202) * Query cache conflicts with local shuffle aggregation. [#73194](https://github.com/StarRocks/starrocks/pull/73194) * A use-after-free of the Hive partition descriptor across fragment teardown. [#73176](https://github.com/StarRocks/starrocks/pull/73176) * A thread-safety issue in lake vacuum, fixed by using `localtime_r`. [#73088](https://github.com/StarRocks/starrocks/pull/73088) * A race condition between `PipelineTimerTask` `doRun` and unscheduling during query context destruction. [#73082](https://github.com/StarRocks/starrocks/pull/73082) * Lock contention on read-only query-engine paths, reduced by relaxing DB locks. [#73067](https://github.com/StarRocks/starrocks/pull/73067) * An materialized view refresh failure with SQL Server tables in a JDBC catalog. [#72962](https://github.com/StarRocks/starrocks/pull/72962) * A JNI local-reference leak in `JDBCScanner::_init_jdbc_scanner`. [#72913](https://github.com/StarRocks/starrocks/pull/72913) * An issue where partition TopN could lose a child's output column. [#72848](https://github.com/StarRocks/starrocks/pull/72848) * An incorrect plan caused by not clearing `LambdaArgument.transformedOp` before INSERT OVERWRITE re-planning. [#72832](https://github.com/StarRocks/starrocks/pull/72832) * The coordinator lock was held during external resource cleanup. [#72830](https://github.com/StarRocks/starrocks/pull/72830) * `Locker` rollback is now exception-safe and the unlock order is fixed. [#72789](https://github.com/StarRocks/starrocks/pull/72789) * An incorrect byte order in `ColumnDict.merge`, now using unsigned byte order. [#72778](https://github.com/StarRocks/starrocks/pull/72778) * A stack-buffer-overflow when formatting into a temporary `std::string`. [#72728](https://github.com/StarRocks/starrocks/pull/72728) * The HAVING clause is now checked when disabling aggregation spill on a small LIMIT. [#72705](https://github.com/StarRocks/starrocks/pull/72705) * A hang caused by joining forwarded RPCs when draining the runtime\_filter worker. [#72626](https://github.com/StarRocks/starrocks/pull/72626) * Incorrect lazy-materialization slot nullability for a materialized view over an outer join. [#72621](https://github.com/StarRocks/starrocks/pull/72621) * `merge_condition` was not preserved when applying a normal rowset commit. [#72542](https://github.com/StarRocks/starrocks/pull/72542) * Lock contention in `TabletScheduler` / `TabletSchedCtx` hot paths during clone, reduced by relaxing DB locks. [#72475](https://github.com/StarRocks/starrocks/pull/72475) * `Locker` did not roll back a partial intensive-lock acquisition. [#72423](https://github.com/StarRocks/starrocks/pull/72423) * A spillable hash join probe crash. [#72397](https://github.com/StarRocks/starrocks/pull/72397) * COALESCE children are now cast to a common type in the JOIN USING transformer. [#72338](https://github.com/StarRocks/starrocks/pull/72338) * DB READ lock was held too broadly for single-table proc directories, now relaxed to per-table. [#72334](https://github.com/StarRocks/starrocks/pull/72334) * A memory leak when caching the materialized view plan context. [#72300](https://github.com/StarRocks/starrocks/pull/72300) * FSE-v2 did not set the schema for shared-data sorted schema change. [#72235](https://github.com/StarRocks/starrocks/pull/72235) * `ConsistencyChecker` held a DB READ lock too broadly in periodic scans, now relaxed to per-table READ. [#72218](https://github.com/StarRocks/starrocks/pull/72218) * A BE crash when querying `information_schema.warehouse_queries`. [#72019](https://github.com/StarRocks/starrocks/pull/72019) * A trailing `\r` was not stripped before the closing enclose in CRLF CSV inputs. [#71866](https://github.com/StarRocks/starrocks/pull/71866) * Paimon primary key columns were incorrectly marked as non-nullable when querying an external catalog. [#71660](https://github.com/StarRocks/starrocks/pull/71660) * A redundant double slash was created when constructing the JDBC URL if the URI already ended with a trailing slash, breaking strict drivers such as ClickHouse. [#70992](https://github.com/StarRocks/starrocks/pull/70992) #### 4.0.10[​](#4010 "Direct link to 4.0.10") Release Date: May 9, 2026 ##### Behavior Changes[​](#behavior-changes-5 "Direct link to Behavior Changes") * Cloud storage credentials are now redacted in error messages produced by `INSERT INTO FILES`, preventing accidental exposure of secrets in error logs and `SHOW LOAD` output. [#71245](https://github.com/StarRocks/starrocks/pull/71245) * StarRocks no longer permits queries against insert-only ACID Hive tables in Hive catalog. Previously such queries could silently return more rows than actually visible because INSERT OVERWRITE operations were not recognized. Affected tables now return an explicit error instead of incorrect results. [#71460](https://github.com/StarRocks/starrocks/pull/71460) ##### Improvements[​](#improvements-5 "Direct link to Improvements") * Added an Avro schema cache in Iceberg `PartitionData` construction to remove redundant Jackson `ObjectMapper` allocations during partition load on tables with many partitions. [#72215](https://github.com/StarRocks/starrocks/pull/72215) * Optimized `CatalogRecycleBin.getAdjustedRecycleTimestamp` to avoid rebuilding the table-id map on every call, reducing recycle-bin cleanup and tablet scheduling overhead. [#72128](https://github.com/StarRocks/starrocks/pull/72128) * `OlapTableSink.createLocation` now batches tablet-location lookups in shared-data mode, removing per-tablet StarOS RPCs that previously stalled the planner critical section. [#72041](https://github.com/StarRocks/starrocks/pull/72041) * Java UDAF instances are now loaded and initialized once per query and reused across pipeline driver instances, removing the linear driver-preparation overhead at high `pipeline_dop`. [#72038](https://github.com/StarRocks/starrocks/pull/72038) * Added BE metrics `starrocks_be_staros_shard_info_fallback_total` and `starrocks_be_staros_shard_info_fallback_failed_total` to track when the StarOS worker falls back to fetching shard info from `starmgr` because the local cache missed. [#71620](https://github.com/StarRocks/starrocks/pull/71620) * File-bundle writes now prefer a tablet-local aggregator so the bundled tablet metadata path does not require cross-node shard-info lookups. [#71613](https://github.com/StarRocks/starrocks/pull/71613) * Audit log entries now include the queried tables and views referenced by each query. [#71596](https://github.com/StarRocks/starrocks/pull/71596) * `INSERT INTO FILES` CSV export now supports `csv.enclose` and `csv.escape` properties for controlling field quoting and escaping. [#71589](https://github.com/StarRocks/starrocks/pull/71589) * Added LDAP direct bind authentication via DN pattern, removing the requirement for an admin search account in single-tenant LDAP setups. [#71559](https://github.com/StarRocks/starrocks/pull/71559) * Added the `starrocks_fe_tablet_num` metric for shared-data clusters to match the shared-nothing metric set. [#71444](https://github.com/StarRocks/starrocks/pull/71444) * `star_mgr_meta_sync_interval_sec` is now runtime-mutable via `ADMIN SET FRONTEND CONFIG`; the new interval takes effect on the next sync cycle without an FE restart. [#71675](https://github.com/StarRocks/starrocks/pull/71675) ##### Bug Fixes[​](#bug-fixes-5 "Direct link to Bug Fixes") The following issues have been fixed: * A race in shared-data combined txn log mode where INSERT into per-partition coordinator dispatch could classify legitimate txn logs as orphan and drop them, leaving the transaction stuck in non-VISIBLE state. [#72237](https://github.com/StarRocks/starrocks/pull/72237) * An issue where `_incremental_open_node_channel` channels in shared-data combined txn log mode silently dropped txn logs because the legacy "sender\_id == 0 collects all logs" rule did not apply to incremental channels. [#71992](https://github.com/StarRocks/starrocks/pull/71992) * An issue where `RuntimeProfile::to_thrift()` could crash BE with `std::bad_optional_access` when another thread reset counter min/max values during profile serialization. [#72904](https://github.com/StarRocks/starrocks/pull/72904) * An inconsistency in flat JSON merge results when one side contributed empty values. [#72973](https://github.com/StarRocks/starrocks/pull/72973) * An issue where `CREATE TABLE` for an Iceberg table failed with "Multiple entries with same key: format-version" when the user explicitly specified `format-version` in `PROPERTIES`. [#72828](https://github.com/StarRocks/starrocks/pull/72828) * A `CompactionScheduler.startCompaction` lock scope that held a DB-wide READ lock across single-table critical work, blocking concurrent DDL on other tables in the same database. Switched to IS on DB plus READ on the target table. [#72178](https://github.com/StarRocks/starrocks/pull/72178) * An issue where `StarMgrMetaSyncer.syncTableMetaInternal` and `syncTableColocationInfo` held DB READ/WRITE locks across external StarOS RPCs, freezing CREATE/DROP/ALTER/RENAME on every table in the database for the duration of each RPC. [#72108](https://github.com/StarRocks/starrocks/pull/72108) * An issue where `StarMgrMetaSyncer.getAllPartitionShardGroupId` held the DB READ lock for full iteration over all cloud-native tables and physical partitions, stalling FE threads waiting for the DB write lock on large catalogs. [#71614](https://github.com/StarRocks/starrocks/pull/71614) * A redundant DB READ lock in `getTableNamesViewWithLock`. The underlying `nameToTable` is a `ConcurrentHashMap`, so the enclosing lock added contention without correctness benefit. [#72042](https://github.com/StarRocks/starrocks/pull/72042) * A DB WRITE lock in the read-only `/api/{db}/{table}/_count` REST endpoint that was unnecessary for computing `proximateRowCount()`. [#72053](https://github.com/StarRocks/starrocks/pull/72053) * A batch publish deadlock caused by partition version gaps that operations like tablet split, schema change, and alter jobs reserved by advancing `nextVersion` without a matching publish. [#71483](https://github.com/StarRocks/starrocks/pull/71483) * A deadlock in shared-nothing mode when warming up the LRU cache for rowset metadata while the cache was full. [#71459](https://github.com/StarRocks/starrocks/pull/71459) * A `PipelineTimerTask` that could remain stuck in `waitUtilFinished` due to incorrect ordering between consumer registration and finished signaling. [#72058](https://github.com/StarRocks/starrocks/pull/72058) * A condition race in `ConnectorSinkPassthroughExchanger::accept` that crashed BE with SIGSEGV via out-of-bounds vector access on `_writer_count`. [#71848](https://github.com/StarRocks/starrocks/pull/71848) * A use-after-free in `LoadChannel::get_load_replica_status` caused by destruction of a temporary `shared_ptr`. [#71843](https://github.com/StarRocks/starrocks/pull/71843) * A use-after-free in the information schema sink due to a missing reference count increment in async RPC closure handling. [#71513](https://github.com/StarRocks/starrocks/pull/71513) * A BE crash in `reverse(DecimalV3)` caused by improper handling of decimal value width. [#71834](https://github.com/StarRocks/starrocks/pull/71834) * A BE crash when `UNNEST` produced columns whose define-expression carried an ARRAY type, which was incompatible with global dictionary generation downstream. [#72027](https://github.com/StarRocks/starrocks/pull/72027) * An NPE in FE when creating an Iceberg external table with invalid transform argument order such as `bucket(4, region)`; FE now returns a normal analyzer error. [#71917](https://github.com/StarRocks/starrocks/pull/71917) * An issue where Iceberg manifest data file cache entries were missing column statistics when the first query against a table did not request stats (for example `SELECT *`). [#71913](https://github.com/StarRocks/starrocks/pull/71913) * An issue where the Iceberg min/max optimization was silently skipped when the table was partitioned by `bucket(col, N)` because `PruneHDFSScanColumnRule` injected a placeholder materialized column. [#71863](https://github.com/StarRocks/starrocks/pull/71863) * An issue where `AggregateJoinPushDownRule` failed to rewrite materialized views over Iceberg base tables because `Table.getId()` was compared instead of identity, and connector-table ids can shift across plan rebuilds. [#71856](https://github.com/StarRocks/starrocks/pull/71856) * An issue where INSERT OVERWRITE into Hive dynamic partitions failed when the metastore listed a partition whose location no longer existed on the file system; the missing partition directory is now created before commit. [#71810](https://github.com/StarRocks/starrocks/pull/71810) * A Parquet scanner failure (`Illegal converting from arrow type(dictionary) ...`) when Arrow returned dictionary-typed columns, including dictionaries nested inside arrays, structs, and maps. [#71855](https://github.com/StarRocks/starrocks/pull/71855) * An issue where stale scan ranges from earlier batches persisted across `ColocatedBackendSelector.Assignment` incremental batches, causing files to be re-deployed and re-scanned. [#71789](https://github.com/StarRocks/starrocks/pull/71789) * An issue where `PruneShuffleColumnRule` did not update the Join `outputProperty` after pruning Exchange shuffle columns, leading to incorrect downstream distribution. [#72003](https://github.com/StarRocks/starrocks/pull/72003) * Incorrect shuffle distribution caused by a missing project node when `PushDownJoinOnExpressionToChildProject` was disabled during the first stage of multi-stage materialized view rewrite. [#71075](https://github.com/StarRocks/starrocks/pull/71075) * Duplicate `Apply` attachments in `ReplaceSubqueryRewriteRule` when predicate normalization made the same scalar-subquery placeholder appear multiple times. [#71155](https://github.com/StarRocks/starrocks/pull/71155) * A short-circuit issue in `EventScheduler` where a finished join probe could prevent the pipeline from transitioning to the finished state. [#71740](https://github.com/StarRocks/starrocks/pull/71740) * An issue where AWS assume-role configured via `aws.s3.iam_role_arn` was not applied to JNI scanners (RCFile / Avro / SequenceFile / Hudi), causing S3 403 errors. [#71422](https://github.com/StarRocks/starrocks/pull/71422) * An issue where Oracle JDBC predicate pushdown produced invalid SQL because date literals did not match the Oracle NLS format; literals are now emitted as `date '...'`. [#71412](https://github.com/StarRocks/starrocks/pull/71412) * An issue in shared-data mode where a follower FE forwarded DDL to the leader and waited only for FE journal replay, missing the StarMgr journal and producing "no queryable replica" errors for queries that immediately followed table creation. [#71263](https://github.com/StarRocks/starrocks/pull/71263) * An issue where `get_tablet_stats` for Primary Key tablets repeatedly reloaded the entire `TabletMetadata` for every segment via `get_del_vec_in_meta()`. [#71672](https://github.com/StarRocks/starrocks/pull/71672) * An Arrow Flight issue where empty result sets returned column names of `r` because the placeholder name was emitted instead of the actual schema. [#71534](https://github.com/StarRocks/starrocks/pull/71534) * An issue where `parallel_clone_task_per_path` updates did not include the store-path count when resizing the CLONE thread pool. [#71484](https://github.com/StarRocks/starrocks/pull/71484) * An issue where the resource group user classifier rejected digit-leading usernames that `CREATE USER` allowed. The classifier now uses the same validation rule as `CREATE USER`. [#71470](https://github.com/StarRocks/starrocks/pull/71470) * An issue where `HttpServerHandler.channelInactive` skipped `unregisterConnection` when `isRegistered()` was false, leaking connection-map entries for early-failing requests. [#72006](https://github.com/StarRocks/starrocks/pull/72006) * An issue where Java UDF JNI calls (`NewObject`, `NewArray`, `NewStringUTF`, etc.) did not check for exceptions or null returns, leading to silent failures or undefined behavior. [#71734](https://github.com/StarRocks/starrocks/pull/71734) * An issue where `be_tablets.DATA_SIZE` reported `total_disk_size` (including rowset-embedded indexes and the persistent PK index for lake PK tablets) instead of rowset column data bytes. [#70735](https://github.com/StarRocks/starrocks/pull/70735) * A noisy "Failed to batch drop tablets" warning printed by `StarMgrMetaSyncer` even when there were no shards to delete. [#72209](https://github.com/StarRocks/starrocks/pull/72209) * CVE-2026-42198 (pgjdbc) and CVE-2026-5598 (BouncyCastle): bumped `org.postgresql:postgresql` to 42.7.11 and BouncyCastle to 1.84. [#72797](https://github.com/StarRocks/starrocks/pull/72797) * CVE in netty: upgraded netty to 4.1.133.Final. [#72905](https://github.com/StarRocks/starrocks/pull/72905) * Cleaned broker CVEs by upgrading netty / jetty / awssdk / jackson dependencies in the broker. [#72184](https://github.com/StarRocks/starrocks/pull/72184) * Upgraded jetty-http to 9.4.58.v20250814 to address known CVEs in the previous jetty-http version. [#71762](https://github.com/StarRocks/starrocks/pull/71762) * Temporarily masked CVE-2026-2332 to unblock the build, since jetty 9.x is EOL and no upstream fix is published. [#71914](https://github.com/StarRocks/starrocks/pull/71914) #### 4.0.9[​](#409 "Direct link to 4.0.9") Release Date: April 16, 2026 ##### Behavior Changes[​](#behavior-changes-6 "Direct link to Behavior Changes") * When VARBINARY columns appear inside nested types (ARRAY, MAP, or STRUCT), StarRocks now correctly encodes the values in binary format in MySQL result sets. Previously, raw bytes were emitted directly, which could break text-protocol parsing for null bytes or non-printable characters. This change may affect downstream clients or tools that process VARBINARY data inside nested types. [#71346](https://github.com/StarRocks/starrocks/pull/71346) * Routine Load jobs now automatically pause when a non-retryable error is encountered, such as a row causing the Primary Key size limit to be exceeded. Previously, the job would retry indefinitely because such errors were not recognized as non-retryable by the FE transaction status handler. [#71161](https://github.com/StarRocks/starrocks/pull/71161) * `SHOW CREATE TABLE` and `DESC` statements now display the Primary Key columns for Paimon external tables. [#70535](https://github.com/StarRocks/starrocks/pull/70535) * Cloud-native tablet metadata fetch operations (such as `get_tablet_stats` and `get_tablet_metadatas`) now use a dedicated thread pool instead of the shared `UPDATE_TABLET_META_INFO` pool. This prevents metadata fetch contention from impacting repair and other tasks. The new thread pool size is configurable via a new BE parameter. [#70492](https://github.com/StarRocks/starrocks/pull/70492) ##### Improvements[​](#improvements-6 "Direct link to Improvements") * Added session variables to control the encoding behavior of VARBINARY values in MySQL protocol responses, providing fine-grained control over binary result encoding in client connections. [#71415](https://github.com/StarRocks/starrocks/pull/71415) * Added a `snapshot_meta.json` marker file to cluster snapshots to support integrity validation before snapshot restoration. [#71209](https://github.com/StarRocks/starrocks/pull/71209) * Added warning logs for silently swallowed exceptions in `WarehouseManager` to improve observability of silent failures. [#71215](https://github.com/StarRocks/starrocks/pull/71215) * Added metrics for Iceberg metadata table queries to support performance monitoring and diagnosis. [#70825](https://github.com/StarRocks/starrocks/pull/70825) * The `regexp_replace()` function now supports constant folding during FE query planning, reducing planning overhead for queries with constant string arguments. [#70804](https://github.com/StarRocks/starrocks/pull/70804) * Added categorized metrics for Iceberg time travel queries to improve monitoring and performance analysis. [#70788](https://github.com/StarRocks/starrocks/pull/70788) * Added log output when update compaction is suspended, improving visibility into compaction lifecycle. [#70538](https://github.com/StarRocks/starrocks/pull/70538) * `SHOW COLUMNS` now returns column comments for PostgreSQL external tables. [#70520](https://github.com/StarRocks/starrocks/pull/70520) * Added support for dumping query execution plans when a query encounters an exception, improving diagnosability of runtime failures. [#70387](https://github.com/StarRocks/starrocks/pull/70387) * Tablet deletion during DDL operations is now batched, reducing write lock contention on tablet metadata. [#70052](https://github.com/StarRocks/starrocks/pull/70052) * Added a Force Drop recovery mechanism for synchronous materialized views that are stuck in an error state and cannot be dropped through normal means. [#70029](https://github.com/StarRocks/starrocks/pull/70029) ##### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") The following issues have been fixed: * An issue where the profile `START_TIME` and `END_TIME` were not displayed in the session timezone. [#71429](https://github.com/StarRocks/starrocks/pull/71429) * A shared-object mutation bug in `PushDownAggregateRewriter` when processing CASE-WHEN/IF expressions, which could cause incorrect query results. [#71309](https://github.com/StarRocks/starrocks/pull/71309) * A use-after-free bug in `ThreadPool::do_submit` triggered when thread creation fails. [#71276](https://github.com/StarRocks/starrocks/pull/71276) * An issue where `information_schema.tables` did not properly escape special characters in equality predicates, causing incorrect results. [#71273](https://github.com/StarRocks/starrocks/pull/71273) * An issue where the materialized view scheduler continued to run after the materialized view became inactive. [#71265](https://github.com/StarRocks/starrocks/pull/71265) * Fixed a task signature collision in `UpdateTabletSchemaTask` across concurrent ALTER jobs that could cause schema update tasks to be skipped. [#71242](https://github.com/StarRocks/starrocks/pull/71242) * An issue where row count estimation produced NaN values for histograms that contained only MCV (Most Common Values) entries. [#71241](https://github.com/StarRocks/starrocks/pull/71241) * A missing dependency on the AWS S3 Transfer Manager in the AWS SDK integration. [#71230](https://github.com/StarRocks/starrocks/pull/71230) * An issue where `TaskManager` scheduler callbacks did not verify whether the current node is the leader, potentially causing duplicate task execution on follower nodes. [#71156](https://github.com/StarRocks/starrocks/pull/71156) * A thread-local context pollution issue where `ConnectContext` information was not cleared after a leader-forwarded request completed. [#71141](https://github.com/StarRocks/starrocks/pull/71141) * An issue where the partition predicate was missing in short-circuit point lookups, causing incorrect query results. [#71124](https://github.com/StarRocks/starrocks/pull/71124) * A NullPointerException when analyzing generated columns during Stream Load or Broker Load if a column referenced by the generated column expression was absent from the load schema. [#71116](https://github.com/StarRocks/starrocks/pull/71116) * A use-after-free bug in the error handling path of parallel segment and rowset loading. [#71083](https://github.com/StarRocks/starrocks/pull/71083) * An issue where delvec orphan entries were left behind when a write operation preceded compaction in the same publish batch. [#71049](https://github.com/StarRocks/starrocks/pull/71049) * An issue where queries appeared in the `current_queries` result via HTTP loopback when checking query progress internally. [#71032](https://github.com/StarRocks/starrocks/pull/71032) * CVE-2026-33870 and CVE-2026-33871. [#71017](https://github.com/StarRocks/starrocks/pull/71017) * A read lock leak in `SharedDataStorageVolumeMgr`. [#70987](https://github.com/StarRocks/starrocks/pull/70987) * An issue where the input and result columns of the `locate()` function shared the same NullColumn reference inside BinaryColumns, causing incorrect results. [#70957](https://github.com/StarRocks/starrocks/pull/70957) * An issue where safe tablet deletion checks were incorrectly applied during ALTER operations in share-nothing mode. [#70934](https://github.com/StarRocks/starrocks/pull/70934) * A race condition in `_all_global_rf_ready_or_timeout` that could prevent global runtime filters from being applied correctly. [#70920](https://github.com/StarRocks/starrocks/pull/70920) * An int32 overflow in the `ACCUMULATED` metric macro that caused metric values to silently overflow. [#70889](https://github.com/StarRocks/starrocks/pull/70889) * Incorrect aggregation results in dictionary-encoded merge GROUP BY queries. [#70866](https://github.com/StarRocks/starrocks/pull/70866) * CVE-2025-54920. [#70862](https://github.com/StarRocks/starrocks/pull/70862) * A potential data loss issue in aggregation spill caused by incorrect hash table state handling during `set_finishing`. [#70851](https://github.com/StarRocks/starrocks/pull/70851) * An issue where the `content-length` header was not reset when `proxy_pass_request_body` is disabled. [#70821](https://github.com/StarRocks/starrocks/pull/70821) * An issue where the spill directory for load operations was cleaned up in the object destructor rather than during `DeltaWriter::close()`, potentially causing premature deletion of spill data. [#70778](https://github.com/StarRocks/starrocks/pull/70778) * An issue where `INSERT INTO ... BY NAME` from `FILES()` did not correctly push down the schema for partial column sets. [#70774](https://github.com/StarRocks/starrocks/pull/70774) * An issue where connector scan nodes did not reset the scan range source on query retry, causing incorrect results upon retry. [#70762](https://github.com/StarRocks/starrocks/pull/70762) * A potential rowset metadata loss for Primary Key model tablets caused by a GC race during disk re-migration of the form A→B→A. [#70727](https://github.com/StarRocks/starrocks/pull/70727) * An issue where a query-scoped warehouse hint leaked the `ComputeResource` object in `ConnectContext`, potentially affecting subsequent queries on the same connection. [#70706](https://github.com/StarRocks/starrocks/pull/70706) * An issue where redundant conjuncts in `MySqlScanNode` and `JDBCScanNode` caused BE errors related to `VectorizedInPredicate` type mismatches. [#70694](https://github.com/StarRocks/starrocks/pull/70694) * A missing `libssl-dev` dependency in the Ubuntu runtime environment. [#70688](https://github.com/StarRocks/starrocks/pull/70688) * An issue where Iceberg manifest cache completeness was not validated on read, leading to incorrect scan results when the cache was partially populated. [#70675](https://github.com/StarRocks/starrocks/pull/70675) * A duplicate closure reference in `_tablet_multi_get_rpc` that could cause use-after-free. [#70657](https://github.com/StarRocks/starrocks/pull/70657) * Partial manifest cache writes in the Iceberg `ManifestReader` that could result in incomplete cache entries and incorrect scan behavior. [#70652](https://github.com/StarRocks/starrocks/pull/70652) * A crash in `array_map()` when processing arrays that contain null literal elements. [#70629](https://github.com/StarRocks/starrocks/pull/70629) * A stack overflow in the `to_base64()` function when processing large inputs. [#70623](https://github.com/StarRocks/starrocks/pull/70623) * An issue where `INSERT INTO ... BY NAME` from `FILES()` used positional column mapping instead of name-based mapping, causing data to be written to incorrect columns. [#70622](https://github.com/StarRocks/starrocks/pull/70622) * An issue where `NOT NULL` constraints were incorrectly pushed down into the schema inferred from `FILES()`, causing load failures for nullable columns. [#70621](https://github.com/StarRocks/starrocks/pull/70621) * An issue where precise external materialized view refresh did not fall back correctly for Iceberg-like connectors. [#70589](https://github.com/StarRocks/starrocks/pull/70589) * A `num_short_key_columns` mismatch when constructing a partial tablet schema, which could cause data read errors. [#70586](https://github.com/StarRocks/starrocks/pull/70586) * A BE crash that occurred when the child iterator was exhausted in `MaskMergeIterator`. [#70539](https://github.com/StarRocks/starrocks/pull/70539) * An issue where materialized view refresh jobs repeatedly refreshed partitions whose corresponding Iceberg snapshots had expired. [#70523](https://github.com/StarRocks/starrocks/pull/70523) * An issue where starlet configuration parameters could not be set. [#70482](https://github.com/StarRocks/starrocks/pull/70482) * An issue where the lock-free materialized view rewrite path incorrectly fell back to live metadata, causing inconsistent rewrite behavior. [#70475](https://github.com/StarRocks/starrocks/pull/70475) * An issue in `JoinHashTable::merge_ht` where dummy rows were not skipped for expression-based join key columns, causing incorrect join results. [#70465](https://github.com/StarRocks/starrocks/pull/70465) * An incorrect equality comparison in `InformationFunction` that could produce wrong results in certain queries. [#70464](https://github.com/StarRocks/starrocks/pull/70464) * A column type mismatch in the `__iceberg_transform_bucket` internal function. [#70443](https://github.com/StarRocks/starrocks/pull/70443) * An issue where Iceberg materialized view refresh failed when Iceberg snapshot timestamps were non-monotonic. [#70382](https://github.com/StarRocks/starrocks/pull/70382) * An issue where user authentication credentials were exposed in audit logs and SQL redaction output. [#70360](https://github.com/StarRocks/starrocks/pull/70360) * A CN crash that occurred when scanning an empty tablet with physical split enabled. [#70281](https://github.com/StarRocks/starrocks/pull/70281) * An issue where the VARCHAR column length was not preserved after a redundant CAST was eliminated during query optimization. [#70269](https://github.com/StarRocks/starrocks/pull/70269) * An issue where brpc connection retry logic did not correctly handle a wrapped `NoSuchElementException`, causing connection failures after the retry attempt. [#70203](https://github.com/StarRocks/starrocks/pull/70203) * An issue where null fractions for outer join columns were not preserved during statistics estimation, leading to suboptimal query plans. [#70144](https://github.com/StarRocks/starrocks/pull/70144) * A memory tracker leak in connector sink operations running on poller threads. [#70121](https://github.com/StarRocks/starrocks/pull/70121) #### 4.0.8[​](#408 "Direct link to 4.0.8") Release Date: March 25, 2026 ##### Behavior Changes[​](#behavior-changes-7 "Direct link to Behavior Changes") * Improved `sql_mode` handling: when `DIVISION_BY_ZERO` or `FAIL_PARSE_DATE` mode is set, division by zero and date parse failures in `str_to_date`/`str2date` now return an error instead of being silently ignored. [#70004](https://github.com/StarRocks/starrocks/pull/70004) * When `sql_mode` is set to `FORBID_INVALID_DATE`, invalid dates in `INSERT VALUES` clauses are now correctly rejected instead of being bypassed. [#69803](https://github.com/StarRocks/starrocks/pull/69803) * Expression partition generated columns are now hidden from `DESC` and `SHOW CREATE TABLE` output. [#69793](https://github.com/StarRocks/starrocks/pull/69793) * Client ID is no longer included in audit logs. [#69383](https://github.com/StarRocks/starrocks/pull/69383) ##### Improvements[​](#improvements-7 "Direct link to Improvements") * Added a configuration item `local_exchange_buffer_mem_limit_per_driver` to limit the local exchange buffer size to `dop * local_exchange_buffer_mem_limit_per_driver`. [#70393](https://github.com/StarRocks/starrocks/pull/70393) * Cached file existence check results across versions in `check_missing_files` to reduce redundant storage I/O. [#70364](https://github.com/StarRocks/starrocks/pull/70364) * Allowed disabling split and reverse scan ranges for descending TopN runtime filters when `desc_hint_split_range` is set to ≤ 0. [#70307](https://github.com/StarRocks/starrocks/pull/70307) * Added `EXPLAIN` and `EXPLAIN ANALYZE` support for `INSERT` statements in the Trino dialect. [#70174](https://github.com/StarRocks/starrocks/pull/70174) * Optimized Iceberg read performance when position deletes are present. [#69717](https://github.com/StarRocks/starrocks/pull/69717) * Optimized materialized view best-selector strategy based on distributed keys to improve materialized view selection accuracy. [#69679](https://github.com/StarRocks/starrocks/pull/69679) ##### Bug Fixes[​](#bug-fixes-7 "Direct link to Bug Fixes") The following issues have been fixed: * JDBC MySQL pushdown failing for unsupported cast operations. [#70415](https://github.com/StarRocks/starrocks/pull/70415) * Type mismatch issues in materialized view refresh. Added `mv_refresh_force_partition_type` configuration to force partition type in materialized view refresh. [#70381](https://github.com/StarRocks/starrocks/pull/70381) * `dataVersion` not set correctly when restoring from backup. [#70373](https://github.com/StarRocks/starrocks/pull/70373) * Duplicated partition names in materialized view refresh tasks. [#70354](https://github.com/StarRocks/starrocks/pull/70354) * Incorrect SLF4J parameterized logging using string concatenation instead of placeholder arguments. [#70330](https://github.com/StarRocks/starrocks/pull/70330) * Comment not set when creating Hive tables. [#70318](https://github.com/StarRocks/starrocks/pull/70318) * `FileSystemExpirationChecker` blocking on slow HDFS close operations. [#70311](https://github.com/StarRocks/starrocks/pull/70311) * Distribute column validation not applied across different partitions in `OlapTableSink`. [#70310](https://github.com/StarRocks/starrocks/pull/70310) * Constant folding producing INF instead of an error when double addition overflows. [#70309](https://github.com/StarRocks/starrocks/pull/70309) * Typo in Iceberg table creation: field `common` was used instead of `comment`. [#70267](https://github.com/StarRocks/starrocks/pull/70267) * Root user not bypassing all Ranger permission checks in some scenarios. [#70254](https://github.com/StarRocks/starrocks/pull/70254) * `query_pool` memory tracker going negative during data ingestion. [#70228](https://github.com/StarRocks/starrocks/pull/70228) * `AuditEventProcessor` thread exiting due to `OutOfMemoryException`. [#70206](https://github.com/StarRocks/starrocks/pull/70206) * `SplitTopNRule` not applying partition pruning correctly. [#70154](https://github.com/StarRocks/starrocks/pull/70154) * Out-of-bounds access in `cal_new_base_version` during schema change publish. [#70132](https://github.com/StarRocks/starrocks/pull/70132) * Materialied view rewrite ignoring dropped partitions from the base table. [#70130](https://github.com/StarRocks/starrocks/pull/70130) * Unexpected partition predicate pruning due to type mismatch in boundary comparisons. [#70097](https://github.com/StarRocks/starrocks/pull/70097) * `str_to_date` losing microsecond precision in BE runtime. [#70068](https://github.com/StarRocks/starrocks/pull/70068) * Join spill process crashing in `set_callback_function`. [#70030](https://github.com/StarRocks/starrocks/pull/70030) * Broker Load failing GCS authentication after `gcs-connector` upgrade to version 3.0.13. [#70012](https://github.com/StarRocks/starrocks/pull/70012) * DCHECK failure in `DeltaWriter::close()` when called from a bthread context. [#69960](https://github.com/StarRocks/starrocks/pull/69960) * Use-after-free race condition in `AsyncDeltaWriter` close/finish lifecycle. [#69940](https://github.com/StarRocks/starrocks/pull/69940) * Race condition causing write transaction edit log entry to be missed. [#69899](https://github.com/StarRocks/starrocks/pull/69899) * Known CVE vulnerabilities. [#69863](https://github.com/StarRocks/starrocks/pull/69863) * Follower FE not waiting for journal replay in `changeCatalogDb`. [#69834](https://github.com/StarRocks/starrocks/pull/69834) * Incorrect `LIKE` pattern matching with backslash escape sequences. [#69775](https://github.com/StarRocks/starrocks/pull/69775) * Expression analysis failure after renaming a partition column. [#69771](https://github.com/StarRocks/starrocks/pull/69771) * Use-after-free crash in `AsyncDeltaWriter::close`. [#69770](https://github.com/StarRocks/starrocks/pull/69770) * Crash in local partition TopN execution. [#69752](https://github.com/StarRocks/starrocks/pull/69752) * Incorrect behavior in `PartitionColumnMinMaxRewriteRule` caused by `Partition.hasStorageData`. [#69751](https://github.com/StarRocks/starrocks/pull/69751) * Duplicated CSV compression suffix in file sink output filenames. [#69749](https://github.com/StarRocks/starrocks/pull/69749) * `lake_capture_tablet_and_rowsets` not gated behind an experimental configuration flag. [#69748](https://github.com/StarRocks/starrocks/pull/69748) * Incorrect partition min pruning with shadow partitions. [#69641](https://github.com/StarRocks/starrocks/pull/69641) * Java UDTF/UDAF crashing when method parameters use generic types. [#69197](https://github.com/StarRocks/starrocks/pull/69197) * Per-query metadata not released after query planning, causing FE OOM during concurrent query execution. [#68444](https://github.com/StarRocks/starrocks/pull/68444) * Query-scope warehouse hint leaking `ComputeResource` in `ConnectContext`. [#70706](https://github.com/StarRocks/starrocks/pull/70706) * Lock-free materialized view rewrite incorrectly falling back to live metadata. [#70475](https://github.com/StarRocks/starrocks/pull/70475) * Duplicate closure reference in `_tablet_multi_get_rpc`. [#70657](https://github.com/StarRocks/starrocks/pull/70657) * Infinite recursion in `ReplaceColumnRefRewriter`. [#66974](https://github.com/StarRocks/starrocks/pull/66974) * `NOT NULL` constraint incorrectly pushed down to `FILES()` table function schema. [#70621](https://github.com/StarRocks/starrocks/pull/70621) * `num_short_key_columns` mismatch in partial tablet schema. [#70586](https://github.com/StarRocks/starrocks/pull/70586) * `COLUMN_UPSERT_MODE` checksum error in shared-data clusters. [#65320](https://github.com/StarRocks/starrocks/pull/65320) * Column type mismatch for `__iceberg_transform_bucket`. [#70443](https://github.com/StarRocks/starrocks/pull/70443) * Starlet configuration items not taking effect. [#70482](https://github.com/StarRocks/starrocks/pull/70482) * DCG data not read correctly when switching from column mode to row mode in partial update. [#61529](https://github.com/StarRocks/starrocks/pull/61529) #### 4.0.7[​](#407 "Direct link to 4.0.7") Release Date: March 12, 2026 ##### Behavior Change[​](#behavior-change "Direct link to Behavior Change") * Disallowed creating materialized views based on Iceberg views. [#69471](https://github.com/StarRocks/starrocks/pull/69471) * Fixed inconsistencies in multi-statement Stream Load transaction behavior. [#68542](https://github.com/StarRocks/starrocks/pull/68542) ##### Improvements[​](#improvements-8 "Direct link to Improvements") * Added fine-grained trace counters for `LakePersistentIndex` in the Publish phase. [#69640](https://github.com/StarRocks/starrocks/pull/69640) * Triggered early flush in `LakePersistentIndex` when rebuild row count exceeds the threshold. [#69698](https://github.com/StarRocks/starrocks/pull/69698) * Added `dump_lake_persistent_index_sst` operation to `meta_tool`. [#69682](https://github.com/StarRocks/starrocks/pull/69682) * Improved `REPAIR TABLE` functionality and `SHOW TABLET` status display. [#69656](https://github.com/StarRocks/starrocks/pull/69656) * Supports `ADMIN SHOW TABLET STATUS` for cloud-native tables. [#69616](https://github.com/StarRocks/starrocks/pull/69616) * Upgraded `hadoop-client` from 3.4.2 to 3.4.3. [#69503](https://github.com/StarRocks/starrocks/pull/69503) * Prevented crashes when deserialization mismatches occur. [#69481](https://github.com/StarRocks/starrocks/pull/69481) * Pushed down predicates to FE when querying `information_schema.loads`. [#69472](https://github.com/StarRocks/starrocks/pull/69472) * Optimized the SQL displayed for materialized view refresh TaskRuns. [#69437](https://github.com/StarRocks/starrocks/pull/69437) * Bypassed caching in `CachingIcebergCatalog` when vended credentials are enabled. [#69434](https://github.com/StarRocks/starrocks/pull/69434) * Uses `tryLock` with timeout in `canTxnFinished` to reduce lock contention. [#69427](https://github.com/StarRocks/starrocks/pull/69427) * Added a global readonly variable `@@run_mode`. [#69247](https://github.com/StarRocks/starrocks/pull/69247) * Uses Estimator to estimate cache entry weight in `DeltaLakeMetastore`. [#69244](https://github.com/StarRocks/starrocks/pull/69244) * Added resource share type support for AWS Glue `GetDatabases` API. [#69056](https://github.com/StarRocks/starrocks/pull/69056) * Extracted range predicates from scalar-subqueries containing `convert_tz`. [#69055](https://github.com/StarRocks/starrocks/pull/69055) * Added ByteBuffer Estimator. [#69042](https://github.com/StarRocks/starrocks/pull/69042) * Supports fast cancel for Lake DeltaWriter in shared-data clusters. [#68877](https://github.com/StarRocks/starrocks/pull/68877) * Supports an interface to add physical partitions for random distribution tables. [#68503](https://github.com/StarRocks/starrocks/pull/68503) * Gated SQL transactions behind the session variable `enable_sql_transaction` (default: true). [#63535](https://github.com/StarRocks/starrocks/pull/63535) * Added partition scan number limit when querying external tables. [#68480](https://github.com/StarRocks/starrocks/pull/68480) ##### Bug Fixes[​](#bug-fixes-8 "Direct link to Bug Fixes") The following issues have been fixed: * Incorrect value for the metric `g_publish_version_failed_tasks` when the resource is busy. [#69526](https://github.com/StarRocks/starrocks/pull/69526) * NPE in `IcebergCatalog.getPartitionLastUpdatedTime` when the snapshot has expired. [#68925](https://github.com/StarRocks/starrocks/pull/68925) * DCHECK failure in `DeltaWriter::close()` when called from bthread context. [#70057](https://github.com/StarRocks/starrocks/pull/70057) * Several use-after-free issues. [#69968](https://github.com/StarRocks/starrocks/pull/69968) * Use-after-free race in AsyncDeltaWriter close/finish lifecycle. [#69961](https://github.com/StarRocks/starrocks/pull/69961) * Corrupted cache for PK SST tables is not clraered. [#69693](https://github.com/StarRocks/starrocks/pull/69693) * `AsyncFlushOutputStream` use-after-free issue. [#69688](https://github.com/StarRocks/starrocks/pull/69688) * Retention clock reset issue and incomplete scan in `disableRecoverPartitionWithSameName`. [#69677](https://github.com/StarRocks/starrocks/pull/69677) * NPE in `StreamLoadMultiStmtTask.cancelAfterRestart` after deserialization. [#69662](https://github.com/StarRocks/starrocks/pull/69662) * Unnecessary RPCs and metadata queries caused by the incorrect logic of `SchemaBeTabletsScanner`. [#69645](https://github.com/StarRocks/starrocks/pull/69645) * Graceful exit caused different transactions to publish the same version. [#69639](https://github.com/StarRocks/starrocks/pull/69639) * `TabletUpdates::get_column_values` crashes with SIGSEGV when the Primary Key Index contains stale entries that point to rowsets that have been compacted. [#69617](https://github.com/StarRocks/starrocks/pull/69617) * `KILL ANALYZE` fails to stop `ANALYZE TABLE` tasks. [#69592](https://github.com/StarRocks/starrocks/pull/69592) * Unexpected behavior because not all exceptions of RowGroupWriter are caught. [#69568](https://github.com/StarRocks/starrocks/pull/69568) * TaskRun warehouse display issues after changing the warehouse for the materialized view. [#69567](https://github.com/StarRocks/starrocks/pull/69567) * Sort key does not include newly added key columns after schema change on aggregate and unique tables. [#69529](https://github.com/StarRocks/starrocks/pull/69529) * Issues caused by `isInternalCancelError` using `equals`. [#69523](https://github.com/StarRocks/starrocks/pull/69523) * TaskManager scheduling bugs after `ALTER MATERIALIZED VIEW`. [#69504](https://github.com/StarRocks/starrocks/pull/69504) * Pipeline will be blocked or crash because not all exceptions of `ParquetFileWriter::close` are caught. [#69492](https://github.com/StarRocks/starrocks/pull/69492) * Materialized view force refresh bugs for partitioned tables. [#69488](https://github.com/StarRocks/starrocks/pull/69488) * Incorrect status was returned when certain writers failed to flush data. [#69473](https://github.com/StarRocks/starrocks/pull/69473) * Rowset files were deleted when Primary Key tablets were moved to trash. [#69438](https://github.com/StarRocks/starrocks/pull/69438) * INSERT failure when the range of an automatic partition is enclosed by an existing merged partition. [#69429](https://github.com/StarRocks/starrocks/pull/69429) * Materialized view tablet meta inconsistency between FE leader and follower. [#69428](https://github.com/StarRocks/starrocks/pull/69428) * Concurrency bugs related to function fields. [#69315](https://github.com/StarRocks/starrocks/pull/69315) * Premature deletion of data because rollup handler's active transaction ID is not considered in `computeMinActiveTxnId`. [#69285](https://github.com/StarRocks/starrocks/pull/69285) * Lock leak in `addPartitions` caused by name-based table lookup after concurrent SWAP. [#69284](https://github.com/StarRocks/starrocks/pull/69284) * `DROP FUNCTION IF EXISTS` ignored the `ifExists` flag. [#69216](https://github.com/StarRocks/starrocks/pull/69216) * Inconsistent behavior between StarRocks and MySQL-compatible syntax when `CAST(... AS SIGNED)` in TypeParser. [#69181](https://github.com/StarRocks/starrocks/pull/69181) * Issue with case-insensitive partition lookup in query table copy. [#69173](https://github.com/StarRocks/starrocks/pull/69173) * Missing aggregate function when MIN/MAX stats rewrite failed. [#69149](https://github.com/StarRocks/starrocks/pull/69149) * CVE-2025-67721. [#69138](https://github.com/StarRocks/starrocks/pull/69138) * All-null value handling bug in synchronous materialized views. [#69136](https://github.com/StarRocks/starrocks/pull/69136) * Projection loss in materialized view rewrite due to shared mutable state. [#69063](https://github.com/StarRocks/starrocks/pull/69063) * Incorrect estimation of the Iceberg cache Weigher. [#69058](https://github.com/StarRocks/starrocks/pull/69058) * `FULL OUTER JOIN USING` issue with constant subqueries. [#69028](https://github.com/StarRocks/starrocks/pull/69028) * Issue with `DISTINCT ORDER BY` alias resolution for duplicated constants. [#69014](https://github.com/StarRocks/starrocks/pull/69014) * Issue when materialized view visits external catalog on reload. [#68926](https://github.com/StarRocks/starrocks/pull/68926) * NPE in Iceberg `getPartitions`. [#68907](https://github.com/StarRocks/starrocks/pull/68907) * The container were not properly included in the Azure ABFS/WASB FileSystem cache key. [#68901](https://github.com/StarRocks/starrocks/pull/68901) * Erroneous query results after modifying `CHAR` column length in shared-data clusters. [#68808](https://github.com/StarRocks/starrocks/pull/68808) * Case-insensitive issue with username in LDAP authentication. [#67966](https://github.com/StarRocks/starrocks/pull/67966) * Partitions could not be created after adding `storage_cooldown_ttl` to a table. [#60290](https://github.com/StarRocks/starrocks/pull/60290) #### 4.0.6[​](#406 "Direct link to 4.0.6") Release Date: February 14, 2026 ##### Improvements[​](#improvements-9 "Direct link to Improvements") * Support Partition Transforms with parentheses when creating Iceberg tables (for example, `PARTITION BY (bucket(k1, 3))`). [#68945](https://github.com/StarRocks/starrocks/pull/68945) * Removed the restriction that partition columns in Iceberg tables must be at the end of the column list; they can now be defined at any position. [#68340](https://github.com/StarRocks/starrocks/pull/68340) * Introduced host-level sorting for Iceberg table sink, controlled by the system variable `connector_sink_sort_scope` (Default: FILE), to organize data layout for better read performance. [#68121](https://github.com/StarRocks/starrocks/pull/68121) * Improved error messages for Iceberg partition transform functions (for example, `bucket`, `truncate`) when the argument count is incorrect. [#68349](https://github.com/StarRocks/starrocks/pull/68349) * Refactored table property handling to improve support for different file formats (ORC/Parquet) and compression codecs in Iceberg tables. [#68588](https://github.com/StarRocks/starrocks/pull/68588) * Added table-level query timeout configuration `table_query_timeout` for fine-grained control (Priority: Session `>` Table `>` Cluster). [#67547](https://github.com/StarRocks/starrocks/pull/67547) * Supports the `ADMIN SHOW AUTOMATED CLUSTER SNAPSHOT` statement to view automated snapshot status and schedule. [#68455](https://github.com/StarRocks/starrocks/pull/68455) * Supports displaying the original user-defined SQL with comments in `SHOW CREATE VIEW`. [#68040](https://github.com/StarRocks/starrocks/pull/68040) * Exposed Merge Commit-enabled Stream Load tasks in `information_schema.loads` for better observability. [#67879](https://github.com/StarRocks/starrocks/pull/67879) * Introduced FE memory estimation utility API `/api/memory_usage`. [#68287](https://github.com/StarRocks/starrocks/pull/68287) * Reduced unnecessary logging in `CatalogRecycleBin` during partition recycling. [#68533](https://github.com/StarRocks/starrocks/pull/68533) * Triggered refresh of related asynchronous materialized views when the base table undergoes Swap/Drop/Replace Partition operations. [#68430](https://github.com/StarRocks/starrocks/pull/68430) * Supports `VARBINARY` type for `count distinct`-like aggregate functions. [#68442](https://github.com/StarRocks/starrocks/pull/68442) * Enhanced expression statistics to propagate histogram MCV for semantics-safe expressions (for example, `cast(k as bigint) + 10`) to improve skew detection. [#68292](https://github.com/StarRocks/starrocks/pull/68292) ##### Bug Fixes[​](#bug-fixes-9 "Direct link to Bug Fixes") The following issues have been fixed: * Potential crashes in Skew Join V2 runtime filters. [#67611](https://github.com/StarRocks/starrocks/pull/67611) * Join predicate type mismatch (for example, INT = VARCHAR) caused by low-cardinality rewriting. [#68568](https://github.com/StarRocks/starrocks/pull/68568) * Issues in query queue allocation time and pending timeout logic. [#65802](https://github.com/StarRocks/starrocks/pull/65802) * `unique_id` conflict for Flat JSON extended columns after schema changes. [#68279](https://github.com/StarRocks/starrocks/pull/68279) * Concurrent partition access issues in `OlapTableSink.complete()`. [#68853](https://github.com/StarRocks/starrocks/pull/68853) * Incorrect metadata tracking when restoring manually downloaded cluster snapshots. [#68368](https://github.com/StarRocks/starrocks/pull/68368) * Double slashes in backup paths when the repository location ends with `/`. [#68764](https://github.com/StarRocks/starrocks/pull/68764) * OBS AK/SK credentials in the `SHOW CREATE CATALOG` output were not masked. [#65462](https://github.com/StarRocks/starrocks/pull/65462) #### 4.0.5[​](#405 "Direct link to 4.0.5") Release Date: February 3, 2026 ##### Improvements[​](#improvements-10 "Direct link to Improvements") * Bumped Paimon version to 1.3.1. [#67098](https://github.com/StarRocks/starrocks/pull/67098) * Restored missing optimizations in DP statistics estimation to reduce redundant calculations. [#67852](https://github.com/StarRocks/starrocks/pull/67852) * Improved pruning in DP Join reorder to skip expensive candidate plans earlier. [#67828](https://github.com/StarRocks/starrocks/pull/67828) * Optimized JoinReorderDP partition enumeration to reduce object allocation and added an atom count cap (≤ 62). [#67643](https://github.com/StarRocks/starrocks/pull/67643) * Optimized DP join reorder pruning and added checks to BitSet to reduce stream operation overhead. [#67644](https://github.com/StarRocks/starrocks/pull/67644) * Skipped predicate column statistics collection during DP statistics estimation to reduce CPU overhead. [#67663](https://github.com/StarRocks/starrocks/pull/67663) * Optimized correlated Join row count estimation to avoid repeatedly building `Statistics` objects. [#67773](https://github.com/StarRocks/starrocks/pull/67773) * Reduced memory allocations in `Statistics.getUsedColumns`. [#67786](https://github.com/StarRocks/starrocks/pull/67786) * Avoided redundant `Statistics` map copies when only row counts are updated. [#67777](https://github.com/StarRocks/starrocks/pull/67777) * Skipped aggregate pushdown logic when no aggregation exists in the query to reduce overhead. [#67603](https://github.com/StarRocks/starrocks/pull/67603) * Improved COUNT DISTINCT over windows, added support for fused multi-distinct aggregations, and optimized CTE generation. [#67453](https://github.com/StarRocks/starrocks/pull/67453) * Supports `map_agg` function in the Trino dialect. [#66673](https://github.com/StarRocks/starrocks/pull/66673) * Supports batching retrieval of LakeTablet location information during physical planning to reduce RPC calls in shared-data clusters. [#67325](https://github.com/StarRocks/starrocks/pull/67325) * Added a thread pool for Publish Version transactions to shared-nothing clusters to improve concurrency. [#67797](https://github.com/StarRocks/starrocks/pull/67797) * Optimized LocalMetastore locking granularity by replacing database-level locks with table-level locks. [#67658](https://github.com/StarRocks/starrocks/pull/67658) * Refactored MergeCommitTask lifecycle management and added support for task cancellation. [#67425](https://github.com/StarRocks/starrocks/pull/67425) * Supports intervals for automated cluster snapshots. [#67525](https://github.com/StarRocks/starrocks/pull/67525) * Automatically cleaned up unused `mem_pool` entries in MemTrackerManager. [#67347](https://github.com/StarRocks/starrocks/pull/67347) * Ignored `information_schema` queries during warehouse idle checks. [#67958](https://github.com/StarRocks/starrocks/pull/67958) * Supports dynamically enabling global shuffle for Iceberg table sinks based on data distribution. [#67442](https://github.com/StarRocks/starrocks/pull/67442) * Added Profile metrics for connector sink modules. [#67761](https://github.com/StarRocks/starrocks/pull/67761) * Improved the collection and display of load spill metrics in Profiles, distinguishing between local and remote I/O. [#67527](https://github.com/StarRocks/starrocks/pull/67527) * Changed Async-Profiler log level to Error to avoid repeating warning logs. [#67297](https://github.com/StarRocks/starrocks/pull/67297) * Notified Starlet during BE shutdown to report SHUTDOWN status to StarMgr. [#67461](https://github.com/StarRocks/starrocks/pull/67461) ##### Bug Fixes[​](#bug-fixes-10 "Direct link to Bug Fixes") The following issues have been fixed: * Lacking support for legal simple paths containing hyphens (`-`). [#67988](https://github.com/StarRocks/starrocks/pull/67988) * Runtime error when aggregate pushdown occurred on grouping keys involving JSON types. [#68142](https://github.com/StarRocks/starrocks/pull/68142) * Issue where JSON path rewrite rules incorrectly pruned partition columns referenced in partition predicates. [#67986](https://github.com/StarRocks/starrocks/pull/67986) * Type mismatch issue when rewriting simple aggregation using statistics. [#67829](https://github.com/StarRocks/starrocks/pull/67829) * Potential heap-buffer-overflow in partition Joins. [#67435](https://github.com/StarRocks/starrocks/pull/67435) * Duplicate `slot_ids` introduced when pushing down heavy expressions. [#67477](https://github.com/StarRocks/starrocks/pull/67477) * Division-by-zero error in ExecutionDAG fragment connection for lacking precondition checks. [#67918](https://github.com/StarRocks/starrocks/pull/67918) * Potential issues caused by fragment parallel prepare for single BE. [#67798](https://github.com/StarRocks/starrocks/pull/67798) * Operator terminates incorrectly for lacking `set_finished` method for RawValuesSourceOperator. [#67609](https://github.com/StarRocks/starrocks/pull/67609) * BE crash caused by unsupported DECIMAL256 type (precision > 38) in column aggregators. [#68134](https://github.com/StarRocks/starrocks/pull/68134) * Shared-data clusters lack support for Fast Schema Evolution v2 over DELETE operations by carrying `schema_key` in requests. [#67456](https://github.com/StarRocks/starrocks/pull/67456) * Shared-data clusters lack support for Fast Schema Evolution v2 over synchronous materialized views and traditional schema changes. [#67443](https://github.com/StarRocks/starrocks/pull/67443) * Vacuum might accidentally delete files when file bundling is disabled during FE downgrade. [#67849](https://github.com/StarRocks/starrocks/pull/67849) * Incorrect graceful exit handling in MySQLReadListener. [#67917](https://github.com/StarRocks/starrocks/pull/67917) #### 4.0.4[​](#404 "Direct link to 4.0.4") Release Date: January 16, 2026 ##### Improvements[​](#improvements-11 "Direct link to Improvements") * Supports Parallel Prepare for Operators and Drivers, and single-node batch fragment deployment to improve query scheduling performance. [#63956](https://github.com/StarRocks/starrocks/pull/63956) * Optimized `deltaRows` calculation with lazy evaluation for large partition tables. [#66381](https://github.com/StarRocks/starrocks/pull/66381) * Optimized Flat JSON processing with sequential iteration and improved path derivation. [#66941](https://github.com/StarRocks/starrocks/pull/66941) [#66850](https://github.com/StarRocks/starrocks/pull/66850) * Supports releasing Spill Operator memory earlier to reduce memory usage in group execution. [#66669](https://github.com/StarRocks/starrocks/pull/66669) * Optimized the logic to reduce string comparison overhead. [#66570](https://github.com/StarRocks/starrocks/pull/66570) * Improved skew detection in `GroupByCountDistinctDataSkewEliminateRule` and `SkewJoinOptimizeRule` to support histogram and NULL-based strategies. [#66640](https://github.com/StarRocks/starrocks/pull/66640) [#67100](https://github.com/StarRocks/starrocks/pull/67100) * Enhanced Column ownership management in Chunk using Move semantics to reduce Copy-On-Write overhead. [#66805](https://github.com/StarRocks/starrocks/pull/66805) * For shared-data clusters, added FE `TableSchemaService` and updated `MetaScanNode` to support Fast Schema Evolution v2 schema retrieval. [#66142](https://github.com/StarRocks/starrocks/pull/66142) [#66970](https://github.com/StarRocks/starrocks/pull/66970) * Supports multi-warehouse Backend resource statistics and parallelism (DOP) calculation for better resource isolation. [#66632](https://github.com/StarRocks/starrocks/pull/66632) * Supports configuring Iceberg split size via StarRocks session variable `connector_huge_file_size`. [#67044](https://github.com/StarRocks/starrocks/pull/67044) * Supports label-formatted statistics in `QueryDumpDeserializer`. [#66656](https://github.com/StarRocks/starrocks/pull/66656) * Added an FE configuration `lake_enable_fullvacuum` (Default: `false`) to allow disabling Full Vacuum in shared-data clusters. [#63859](https://github.com/StarRocks/starrocks/pull/63859) * Upgraded lz4 dependency to v1.10.0. [#67045](https://github.com/StarRocks/starrocks/pull/67045) * Added fallback logic for sample-type cardinality estimation when row count is 0. [#65599](https://github.com/StarRocks/starrocks/pull/65599) * Validated Strict Weak Ordering property for lambda comparator in `array_sort`. [#66951](https://github.com/StarRocks/starrocks/pull/66951) * Optimized error messages when fetching external table metadata (Delta/Hive/Hudi/Iceberg) fails, showing root causes. [#66916](https://github.com/StarRocks/starrocks/pull/66916) * Supports dumping pipeline status on query timeout and cancelling with `TIMEOUT` state in FE. [#66540](https://github.com/StarRocks/starrocks/pull/66540) * Displays matched rule index in SQL blacklist error messages. [#66618](https://github.com/StarRocks/starrocks/pull/66618) * Added labels to column statistics in `EXPLAIN` output. [#65899](https://github.com/StarRocks/starrocks/pull/65899) * Filtered out "cancel fragment" logs for normal query completions (for example, LIMIT reached). [#66506](https://github.com/StarRocks/starrocks/pull/66506) * Reduced Backend heartbeat failure logs when the warehouse is suspended. [#66733](https://github.com/StarRocks/starrocks/pull/66733) * Supports `IF EXISTS` in the `ALTER STORAGE VOLUME` syntax. [#66691](https://github.com/StarRocks/starrocks/pull/66691) ##### Bug Fix[​](#bug-fix "Direct link to Bug Fix") The following issues have been fixed: * Incorrect `DISTINCT` and `GROUP BY` results under Low Cardinality optimization due to missing `withLocalShuffle`. [#66768](https://github.com/StarRocks/starrocks/pull/66768) * Rewrite error for JSON v2 functions with Lambda expressions. [#66550](https://github.com/StarRocks/starrocks/pull/66550) * Incorrect application of Partition Join in Null-aware Left Anti Join within correlated subqueries. [#67038](https://github.com/StarRocks/starrocks/pull/67038) * Incorrect row count calculation in the Meta Scan rewrite rule. [#66852](https://github.com/StarRocks/starrocks/pull/66852) * Nullable property mismatched in Union Node when rewriting Meta Scan by statistics. [#67051](https://github.com/StarRocks/starrocks/pull/67051) * BE crash caused by optimization logic for Ranking window functions when `PARTITION BY` and `ORDER BY` are missing. [#67094](https://github.com/StarRocks/starrocks/pull/67094) * Potential wrong results in Group Execution Join with window functions. [#66441](https://github.com/StarRocks/starrocks/pull/66441) * Incorrect results from `PartitionColumnMinMaxRewriteRule` under specific filter conditions. [#66356](https://github.com/StarRocks/starrocks/pull/66356) * Incorrect Nullable property deduction in Union operations after aggregation. [#65429](https://github.com/StarRocks/starrocks/pull/65429) * Crash in `percentile_approx_weighted` when handling compression parameters. [#64838](https://github.com/StarRocks/starrocks/pull/64838) * Crash when spilling with large string encoding. [#61495](https://github.com/StarRocks/starrocks/pull/61495) * Crash triggered by multiple calls to `set_collector` when pushing down local TopN. [#66199](https://github.com/StarRocks/starrocks/pull/66199) * Dependency deduction error in LowCardinality rewrite logic. [#66795](https://github.com/StarRocks/starrocks/pull/66795) * Rowset ID leak when rowset commit fails. [#66301](https://github.com/StarRocks/starrocks/pull/66301) * Metacache lock contention. [#66637](https://github.com/StarRocks/starrocks/pull/66637) * Ingestion failure when column-mode partial update is used with conditional update. [#66139](https://github.com/StarRocks/starrocks/pull/66139) * Concurrent import failure caused by Tablet deletion during the ALTER operation. [#65396](https://github.com/StarRocks/starrocks/pull/65396) * Tablet metadata load error due to RocksDB iteration timeout. [#65146](https://github.com/StarRocks/starrocks/pull/65146) * Compression settings were not applied during table creation and Schema Change in shared-data clusters. [#65673](https://github.com/StarRocks/starrocks/pull/65673) * Delete Vector CRC32 compatibility issue during upgrade. [#65442](https://github.com/StarRocks/starrocks/pull/65442) * Status check logic error in file cleanup after clone task failure. [#65709](https://github.com/StarRocks/starrocks/pull/65709) * Abnormal statistics collection logic after `INSERT OVERWRITE`. [#65327](https://github.com/StarRocks/starrocks/pull/65327) [#65298](https://github.com/StarRocks/starrocks/pull/65298) [#65225](https://github.com/StarRocks/starrocks/pull/65225) * Foreign Key constraints were lost after FE restart. [#66474](https://github.com/StarRocks/starrocks/pull/66474) * Metadata retrieval error after Warehouse deletion. [#66436](https://github.com/StarRocks/starrocks/pull/66436) * Inaccurate Audit Log scan statistics under high selectivity filters. [#66280](https://github.com/StarRocks/starrocks/pull/66280) * Incorrect query error rate metrics calculation logic. [#65891](https://github.com/StarRocks/starrocks/pull/65891) * Potential MySQL connection leaks when tasks exit. [#66829](https://github.com/StarRocks/starrocks/pull/66829) * BE status was not updated immediately on the SIGSEGV crash. [#66212](https://github.com/StarRocks/starrocks/pull/66212) * NPE during LDAP user login. [#65843](https://github.com/StarRocks/starrocks/pull/65843) * Inaccurate error log when switching users in HTTP SQL requests. [#65371](https://github.com/StarRocks/starrocks/pull/65371) * HTTP context leaks during TCP connection reuse. [#65203](https://github.com/StarRocks/starrocks/pull/65203) * Missing QueryDetail in Profile logs for queries forwarded from Follower. [#64395](https://github.com/StarRocks/starrocks/pull/64395) * Missing Prepare/Execute details in Audit logs. [#65448](https://github.com/StarRocks/starrocks/pull/65448) * Crash caused by HyperLogLog memory allocation failures. [#66747](https://github.com/StarRocks/starrocks/pull/66747) * Issue with the `trim` function memory reservation. [#66477](https://github.com/StarRocks/starrocks/pull/66477) [#66428](https://github.com/StarRocks/starrocks/pull/66428) * CVE-2025-66566 and CVE-2025-12183. [#66453](https://github.com/StarRocks/starrocks/pull/66453) [#66362](https://github.com/StarRocks/starrocks/pull/66362) [#67053](https://github.com/StarRocks/starrocks/pull/67053) * Race condition in Exec Group driver submission. [#66099](https://github.com/StarRocks/starrocks/pull/66099) * Use-after-free risk in Pipeline countdown. [#65940](https://github.com/StarRocks/starrocks/pull/65940) * `MemoryScratchSinkOperator` hangs when the queue closes. [#66041](https://github.com/StarRocks/starrocks/pull/66041) * Filesystem cache key collision issue. [#65823](https://github.com/StarRocks/starrocks/pull/65823) * Wrong subtask count in `SHOW PROC '/compactions'`. [#67209](https://github.com/StarRocks/starrocks/pull/67209) * A unified JSON format is not returned in the Query Profile API. [#67077](https://github.com/StarRocks/starrocks/pull/67077) * Improper `getTable` exception handling that affects the materialized view check. [#67224](https://github.com/StarRocks/starrocks/pull/67224) * Inconsistent output of the `Extra` column from the `DESC` statement for native and cloud-native tables. [#67238](https://github.com/StarRocks/starrocks/pull/67238) * Race condition in single-node deployments. [#67215](https://github.com/StarRocks/starrocks/pull/67215) * Log leakage from third-party libraries. [#67129](https://github.com/StarRocks/starrocks/pull/67129) * Incorrect REST Catalog authentication logic that causes authentication failures. [#66861](https://github.com/StarRocks/starrocks/pull/66861) #### 4.0.3[​](#403 "Direct link to 4.0.3") Release Date: December 25, 2025 ##### Behavior Changes[​](#behavior-changes-8 "Direct link to Behavior Changes") * Previously, the FE configuration item `max_scalar_operator_flat_children` (default `10000`) limited the node count only of expressions built by `CASE WHEN`. It now limits every scalar expression that the optimizer rewrites, and the node count is recalculated after each rewrite instead of reusing a cached value. As a result, after an upgrade a query that ran before can be rejected in the analysis phase with `Expression too complex. Current nodes: N, Limit: M.`, typically when inlining a view, a CTE, or a subquery expands its expression tree. Simplify the expression, or increase `max_scalar_operator_flat_children`, which can be modified dynamically. [#66324](https://github.com/StarRocks/starrocks/pull/66324) ##### Improvements[​](#improvements-12 "Direct link to Improvements") * Supports `ORDER BY` clauses for STRUCT data types [#66035](https://github.com/StarRocks/starrocks/pull/66035) * Supports creating Iceberg views with properties and displaying properties in the output of `SHOW CREATE VIEW`. [#65938](https://github.com/StarRocks/starrocks/pull/65938) * Supports altering Iceberg table partition specs using `ALTER TABLE ADD/DROP PARTITION COLUMN`. [#65922](https://github.com/StarRocks/starrocks/pull/65922) * Supports `COUNT/SUM/AVG(DISTINCT)` aggregation over framed windows (for example, `ORDER BY`/`PARTITION BY`) with optimization options. [#65815](https://github.com/StarRocks/starrocks/pull/65815) * Optimized CSV parsing performance by using `memchr` for single-character delimiters. [#63715](https://github.com/StarRocks/starrocks/pull/63715) * Added an optimizer rule to push down Partial TopN to the Pre-Aggregation phase to reduce network overhead. [#61497](https://github.com/StarRocks/starrocks/pull/61497) * Enhanced Data Cache monitoring * Added new metrics for memory/disk quota and usage. [#66168](https://github.com/StarRocks/starrocks/pull/66168) * Added Page Cache statistics to the `api/datacache/stat` HTTP endpoint. [#66240](https://github.com/StarRocks/starrocks/pull/66240) * Added hit rate statistics for native tables. [#66198](https://github.com/StarRocks/starrocks/pull/66198) * Optimized Sort and Aggregation operators to support rapid memory release in OOM scenarios. [#66157](https://github.com/StarRocks/starrocks/pull/66157) * Added `TableSchemaService` in FE for shared-data clusters to allow CNs to fetch specific schemas on demand. [#66142](https://github.com/StarRocks/starrocks/pull/66142) * Optimized Fast Schema Evolution to retain history schemas until all dependent ingestion jobs are finished. [#65799](https://github.com/StarRocks/starrocks/pull/65799) * Enhanced `filterPartitionsByTTL` to properly handle NULL partition values to prevent all partitions from being filtered. [#65923](https://github.com/StarRocks/starrocks/pull/65923) * Optimized `FusedMultiDistinctState` to clear the associated MemPool upon reset. [#66073](https://github.com/StarRocks/starrocks/pull/66073) * Made `ICEBERG_CATALOG_SECURITY` property check case-insensitive in Iceberg REST Catalog. [#66028](https://github.com/StarRocks/starrocks/pull/66028) * Added HTTP endpoint `GET /service_id` to retrieve StarOS Service ID in shared-data clusters. [#65816](https://github.com/StarRocks/starrocks/pull/65816) * Replaced deprecated `metadata.broker.list` with `bootstrap.servers` in Kafka consumer configurations. [#65437](https://github.com/StarRocks/starrocks/pull/65437) * Added FE configuration `lake_enable_fullvacuum` (Default: false) to allow disabling the Full Vacuum Daemon. [#66685](https://github.com/StarRocks/starrocks/pull/66685) * Updated lz4 library to v1.10.0. [#67080](https://github.com/StarRocks/starrocks/pull/67080) ##### Bug Fixes[​](#bug-fixes-11 "Direct link to Bug Fixes") The following issues have been fixed: * `latest_cached_tablet_metadata` could cause versions to be incorrectly skipped during batch Publish. [#66558](https://github.com/StarRocks/starrocks/pull/66558) * Potential issues caused by `ClusterSnapshot` relative checks in `CatalogRecycleBin` when running in shared-nothing clusters. [#66501](https://github.com/StarRocks/starrocks/pull/66501) * BE crash when writing complex data types (ARRAY/MAP/STRUCT) to Iceberg tables during Spill operations. [#66209](https://github.com/StarRocks/starrocks/pull/66209) * Potential hang in Connector Chunk Sink when the writer's initialization or initial write fails. [#65951](https://github.com/StarRocks/starrocks/pull/65951) * Connector Chunk Sink bug where `PartitionChunkWriter` initialization failure caused a null pointer dereference during close. [#66097](https://github.com/StarRocks/starrocks/pull/66097) * Setting a non-existent system variable would silently succeed instead of reporting an error. [#66022](https://github.com/StarRocks/starrocks/pull/66022) * Bundle metadata parsing failure when Data Cache is corrupted. [#66021](https://github.com/StarRocks/starrocks/pull/66021) * MetaScan returned NULL instead of 0 for count columns when the result is empty. [#66010](https://github.com/StarRocks/starrocks/pull/66010) * `SHOW VERBOSE RESOURCE GROUP ALL` displays NULL instead of `default_mem_pool` for resource groups created in earlier versions. [#65982](https://github.com/StarRocks/starrocks/pull/65982) * A `RuntimeException` during query execution after disabling the `flat_json` table configuration. [#65921](https://github.com/StarRocks/starrocks/pull/65921) * Type mismatch issue in shared-data clusters caused by rewriting `min`/`max` statistics to MetaScan after Schema Change. [#65911](https://github.com/StarRocks/starrocks/pull/65911) * BE crash caused by ranking window optimization when `PARTITION BY` and `ORDER BY` are missing. [#67093](https://github.com/StarRocks/starrocks/pull/67093) * Incorrect `can_use_bf` check when merging runtime filters, which could lead to wrong results or crashes. [#67062](https://github.com/StarRocks/starrocks/pull/67062) * Pushing down runtime bitset filters into nested OR predicates causes incorrect results. [#67061](https://github.com/StarRocks/starrocks/pull/67061) * Potential data race and data loss issues caused by write or flush operations after the DeltaWriter has finished. [#66966](https://github.com/StarRocks/starrocks/pull/66966) * Execution error caused by mismatched nullable properties when rewriting simple aggregation to MetaScan. [#67068](https://github.com/StarRocks/starrocks/pull/67068) * Incorrect row count calculation in the MetaScan rewrite rule. [#66967](https://github.com/StarRocks/starrocks/pull/66967) * Versions might be incorrectly skipped during batch Publish due to inconsistent cached tablet metadata. [#66575](https://github.com/StarRocks/starrocks/pull/66575) * Improper error handling for memory allocation failures in HyperLogLog operations. [#66827](https://github.com/StarRocks/starrocks/pull/66827) #### 4.0.2[​](#402 "Direct link to 4.0.2") Release Date: December 4, 2025 ##### New Features[​](#new-features "Direct link to New Features") * Introduced a new resource group attribute, `mem_pool`, allowing multiple resource groups to share the same memory pool and enforce a joint memory limit for the pool. This feature is backward compatible. `default_mem_pool` is used if `mem_pool` is not specified. [#64112](https://github.com/StarRocks/starrocks/pull/64112) ##### Improvements[​](#improvements-13 "Direct link to Improvements") * Reduced remote storage access during Vacuum after File Bundling is enabled. [#65793](https://github.com/StarRocks/starrocks/pull/65793) * The File Bundling feature caches the latest tablet metadata. [#65640](https://github.com/StarRocks/starrocks/pull/65640) * Improved safety and stability for long-string scenarios. [#65433](https://github.com/StarRocks/starrocks/pull/65433) [#65148](https://github.com/StarRocks/starrocks/pull/65148) * Optimized the `SplitTopNAggregateRule` logic to avoid performance regression. [#65478](https://github.com/StarRocks/starrocks/pull/65478) * Applied the Iceberg/DeltaLake table statistics collection strategy to other external data sources to avoid collecting statistics when the table is a single table. [#65430](https://github.com/StarRocks/starrocks/pull/65430) * Added Page Cache metrics to the Data Cache HTTP API `api/datacache/app_stat`. [#65341](https://github.com/StarRocks/starrocks/pull/65341) * Supports ORC file splitting to enable parallel scanning of a single large ORC file. [#65188](https://github.com/StarRocks/starrocks/pull/65188) * Added selectivity estimation for IF predicates in the optimizer. [#64962](https://github.com/StarRocks/starrocks/pull/64962) * Supports constant evaluation of `hour`, `minute`, and `second` for `DATE` and `DATETIME` types in the FE. [#64953](https://github.com/StarRocks/starrocks/pull/64953) * Enabled rewrite of simple aggregation to MetaScan by default. [#64698](https://github.com/StarRocks/starrocks/pull/64698) * Improved multiple-replica assignment handling in shared-data clusters for enhanced reliability. [#64245](https://github.com/StarRocks/starrocks/pull/64245) * Exposes cache hit ratio in audit logs and metrics. [#63964](https://github.com/StarRocks/starrocks/pull/63964) * Estimates per-bucket distinct counts for histograms using HyperLogLog or sampling to provide more accurate NDV for predicates and joins. [#58516](https://github.com/StarRocks/starrocks/pull/58516) * Supports FULL OUTER JOIN USING with SQL-standard semantics. [#65122](https://github.com/StarRocks/starrocks/pull/65122) * Prints memory information when Optimizer times out for diagnostics. [#65206](https://github.com/StarRocks/starrocks/pull/65206) ##### Bug Fixes[​](#bug-fixes-12 "Direct link to Bug Fixes") The following issues have been fixed: * DECIMAL56 `mod`-related issue. [#65795](https://github.com/StarRocks/starrocks/pull/65795) * Issue related to Iceberg scan range handling. [#65658](https://github.com/StarRocks/starrocks/pull/65658) * MetaScan rewrite issues on temporary partitions and random buckets. [#65617](https://github.com/StarRocks/starrocks/pull/65617) * `JsonPathRewriteRule` uses the wrong table after transparent materialized view rewrite. [#65597](https://github.com/StarRocks/starrocks/pull/65597) * Materialized view refresh failures when `partition_retention_condition` referenced generated columns. [#65575](https://github.com/StarRocks/starrocks/pull/65575) * Iceberg min/max value typing issue. [#65551](https://github.com/StarRocks/starrocks/pull/65551) * Issue with queries against `information_schema.tables` and `views` across different databases when `enable_evaluate_schema_scan_rule` is set to `true`. [#65533](https://github.com/StarRocks/starrocks/pull/65533) * Integer overflow in JSON array comparison. [#64981](https://github.com/StarRocks/starrocks/pull/64981) * MySQL Reader does not support SSL. [#65291](https://github.com/StarRocks/starrocks/pull/65291) * ARM build issue caused by SVE build incompatibility. [#65268](https://github.com/StarRocks/starrocks/pull/65268) * Queries based on bucket-aware execution may get stuck for bucketed Iceberg tables. [#65261](https://github.com/StarRocks/starrocks/pull/65261) * Robust error propagation and memory safety issues for the lack of memory limit checks in OLAP table scan. [#65131](https://github.com/StarRocks/starrocks/pull/65131) ##### Behavior Changes[​](#behavior-changes-9 "Direct link to Behavior Changes") * When a materialized view is inactivated, the system recursively inactivates its dependent materialized views. [#65317](https://github.com/StarRocks/starrocks/pull/65317) * Uses the original materialized view query SQL (including comments/formatting) when generating SHOW CREATE output. [#64318](https://github.com/StarRocks/starrocks/pull/64318) #### 4.0.1[​](#401 "Direct link to 4.0.1") Release Date: November 17, 2025 ##### Improvements[​](#improvements-14 "Direct link to Improvements") * Optimized TaskRun session variable handling to process known variables only. [#64150](https://github.com/StarRocks/starrocks/pull/64150) * Supports collecting statistics of Iceberg and Delta Lake tables from metadata by default. [#64140](https://github.com/StarRocks/starrocks/pull/64140) * Supports collecting statistics of Iceberg tables with bucket and truncate partition transform. [#64122](https://github.com/StarRocks/starrocks/pull/64122) * Supports inspecting FE `/proc` profile for debugging. [#63954](https://github.com/StarRocks/starrocks/pull/63954) * Enhanced OAuth2 and JWT authentication support for Iceberg REST catalogs. [#63882](https://github.com/StarRocks/starrocks/pull/63882) * Improved bundle tablet metadata validation and recovery handling. [#63949](https://github.com/StarRocks/starrocks/pull/63949) * Improved scan-range memory estimation logic. [#64158](https://github.com/StarRocks/starrocks/pull/64158) ##### Bug Fixes[​](#bug-fixes-13 "Direct link to Bug Fixes") The following issues have been fixed: * Transaction logs were deleted when publishing bundle tablets. [#64030](https://github.com/StarRocks/starrocks/pull/64030) * The join algorithm cannot guarantee the sort property because, after joining, the sort property is not reset. [#64086](https://github.com/StarRocks/starrocks/pull/64086) * Issues related to transparent materialized view rewrite. [#63962](https://github.com/StarRocks/starrocks/pull/63962) ##### Behavior Changes[​](#behavior-changes-10 "Direct link to Behavior Changes") * Added the property `enable_iceberg_table_cache` to Iceberg Catalogs to optionally disable Iceberg table cache and allow it always to read the latest data. [#64082](https://github.com/StarRocks/starrocks/pull/64082) * Ensured `INSERT ... SELECT` reads the freshest metadata by refreshing external tables before planning. [#64026](https://github.com/StarRocks/starrocks/pull/64026) * Increased lock table slots to 256 and added `rid` to slow-lock logs. [#63945](https://github.com/StarRocks/starrocks/pull/63945) * Temporarily disabled `shared_scan` due to incompatibility with event-based scheduling. [#63543](https://github.com/StarRocks/starrocks/pull/63543) * Changed the default Hive Catalog cache TTL to 24 hours and removed unused parameters. [#63459](https://github.com/StarRocks/starrocks/pull/63459) * Automatically determine the Partial Update mode based on the session variable and the number of inserted columns. [#62091](https://github.com/StarRocks/starrocks/pull/62091) #### 4.0.0[​](#400 "Direct link to 4.0.0") Release date: October 17, 2025 ##### Data Lake Analytics[​](#data-lake-analytics "Direct link to Data Lake Analytics") * Unified Page Cache and Data Cache for BE metadata, and adopted an adaptive strategy for scaling. [#61640](https://github.com/StarRocks/starrocks/issues/61640) * Optimized metadata file parsing for Iceberg statistics to avoid repetitive parsing. [#59955](https://github.com/StarRocks/starrocks/pull/59955) * Optimized COUNT/MIN/MAX queries against Iceberg metadata by efficiently skipping over data file scans, significantly improving aggregation query performance on large partitioned tables and reducing resource consumption. [#60385](https://github.com/StarRocks/starrocks/pull/60385) * Supports compaction for Iceberg tables via procedure `rewrite_data_files`. * Supports Iceberg tables with hidden partitions, including creating, writing, and reading the tables. [#58914](https://github.com/StarRocks/starrocks/issues/58914) * Supports setting sort keys when creating Iceberg tables. * Optimizes sink performance for Iceberg tables. * Iceberg Sink supports spilling large operators, global shuffle, and local sorting to optimize memory usage and address small file issues. [#61963](https://github.com/StarRocks/starrocks/pull/61963) * Iceberg Sink optimizes local sorting based on Spill Partition Writer to improve write efficiency. [#62096](https://github.com/StarRocks/starrocks/pull/62096) * Iceberg Sink supports global shuffle for partitions to further reduce small files. [#62123](https://github.com/StarRocks/starrocks/pull/62123) * Enhanced bucket-aware execution for Iceberg tables to improve concurrency and distribution capabilities of bucketed tables. [#61756](https://github.com/StarRocks/starrocks/pull/61756) * Supports the TIME data type in the Paimon catalog. [#58292](https://github.com/StarRocks/starrocks/pull/58292) * Upgraded Iceberg version to 1.10.0. [#63667](https://github.com/StarRocks/starrocks/pull/63667) ##### Security and Authentication[​](#security-and-authentication "Direct link to Security and Authentication") * In scenarios where JWT authentication and the Iceberg REST Catalog are used, StarRocks supports the passthrough of user login information to Iceberg via the REST Session Catalog for subsequent data access authentication. [#59611](https://github.com/StarRocks/starrocks/pull/59611) [#58850](https://github.com/StarRocks/starrocks/pull/58850) * Supports vended credentials for the Iceberg catalog. * Supports granting StarRocks internal roles to external groups obtained via Group Provider. [#63385](https://github.com/StarRocks/starrocks/pull/63385) [#63258](https://github.com/StarRocks/starrocks/pull/63258) * Added REFRESH privilege to external tables to control the permission to refresh them. [#63385](https://github.com/StarRocks/starrocks/pull/62636) ##### Storage Optimization and Cluster Management[​](#storage-optimization-and-cluster-management "Direct link to Storage Optimization and Cluster Management") * Introduced  the File Bundling optimization for the cloud-native table in shared-data clusters to automatically bundle the data files generated by loading, Compaction, or Publish operations, thereby reducing the API cost caused by high-frequency access to the external storage system. File Bundling is enabled by default for tables created in v4.0 or later. [#58316](https://github.com/StarRocks/starrocks/issues/58316) * Supports Multi-Table Write-Write Transaction to allow users to control the atomic submission of INSERT, UPDATE, and DELETE operations. The transaction supports Stream Load and INSERT INTO interfaces, effectively guaranteeing cross-table consistency in ETL and real-time write scenarios. [#61362](https://github.com/StarRocks/starrocks/issues/61362) * Supports Kafka 4.0 for Routine Load. * Supports full-text inverted indexes on Primary Key tables in shared-nothing clusters. * Supports modifying aggregate keys of Aggregate tables. [#62253](https://github.com/StarRocks/starrocks/issues/62253) * Supports enabling case-insensitive processing on names of catalogs, databases, tables, views, and materialized views. [#61136](https://github.com/StarRocks/starrocks/pull/61136) * Supports blacklisting Compute Nodes in shared-data clusters. [#60830](https://github.com/StarRocks/starrocks/pull/60830) * Supports global connection ID. [#57256](https://github.com/StarRocks/starrocks/pull/57276) * Added the `recyclebin_catalogs` metadata view to Information Schema to display recoverable deleted metadata. [#51007](https://github.com/StarRocks/starrocks/pull/51007) ##### Query and Performance Improvement[​](#query-and-performance-improvement "Direct link to Query and Performance Improvement") * Supports DECIMAL256 data type, expanding the upper limit of precision from 38 to 76 bits. Its 256-bit storage provides better adaptability to high-precision financial and scientific computing scenarios, effectively mitigating DECIMAL128's precision overflow problem in very large aggregations and high-order operations. [#59645](https://github.com/StarRocks/starrocks/issues/59645) * Improved the performance for basic operators.[#61691](https://github.com/StarRocks/starrocks/issues/61691) [#61632](https://github.com/StarRocks/starrocks/pull/61632) [#62585](https://github.com/StarRocks/starrocks/pull/62585) [#61405](https://github.com/StarRocks/starrocks/pull/61405) [#61429](https://github.com/StarRocks/starrocks/pull/61429) * Optimized the performance of the JOIN and AGG operators. [#61691](https://github.com/StarRocks/starrocks/issues/61691) * \[Preview] Introduced SQL Plan Manager to allow users to bind a query plan to a query, thereby preventing the query plan from changing due to system state changes (mainly data updates and statistics updates), thus stabilizing query performance. [#56310](https://github.com/StarRocks/starrocks/issues/56310) * Introduced Partition-wise Spillable Aggregate/Distinct operators to replace the original Spill implementation based on sorted aggregation, significantly improving aggregation performance and reducing read/write overhead in complex and high-cardinality GROUP BY scenarios. [#60216](https://github.com/StarRocks/starrocks/pull/60216) * Flat JSON V2: * Supports configuring Flat JSON on the table level. [#57379](https://github.com/StarRocks/starrocks/pull/57379) * Enhance JSON columnar storage by retaining the V1 mechanism while adding page- and segment-level indexes (ZoneMaps, Bloom filters), predicate pushdown with late materialization, dictionary encoding, and integration of a low-cardinality global dictionary to significantly boost execution efficiency. [#60953](https://github.com/StarRocks/starrocks/issues/60953) * Supports an adaptive ZoneMap index creation strategy for the STRING data type. [#61960](https://github.com/StarRocks/starrocks/issues/61960) * Enhanced query observability: * Optimized EXPLAIN ANALYZE output to display the execution metrics by group and by operator for better readability. [#63326](https://github.com/StarRocks/starrocks/pull/63326) * `QueryDetailActionV2` and `QueryProfileActionV2` now support JSON format, enhancing cross-FE query capabilities. [#63235](https://github.com/StarRocks/starrocks/pull/63235) * Supports retrieving Query Profile information across all FEs. [#61345](https://github.com/StarRocks/starrocks/pull/61345) * SHOW PROCESSLIST statements display Catalog, Query ID, and other information. [#62552](https://github.com/StarRocks/starrocks/pull/62552) * Enhanced query queue and process monitoring, supporting display of Running/Pending statuses.[#62261](https://github.com/StarRocks/starrocks/pull/62261) * Materialized view rewrites consider the distribution and sort keys of the original table, improving the selection of optimal materialized views. [#62830](https://github.com/StarRocks/starrocks/pull/62830) ##### Functions and SQL Syntax[​](#functions-and-sql-syntax "Direct link to Functions and SQL Syntax") * Added the following functions: * `bitmap_hash64` [#56913](https://github.com/StarRocks/starrocks/pull/56913) * `bool_or` [#57414](https://github.com/StarRocks/starrocks/pull/57414) * `strpos` [#57278](https://github.com/StarRocks/starrocks/pull/57287) * `to_datetime` and `to_datetime_ntz` [#60637](https://github.com/StarRocks/starrocks/pull/60637) * `regexp_count` [#57182](https://github.com/StarRocks/starrocks/pull/57182) * `tokenize` [#58965](https://github.com/StarRocks/starrocks/pull/58965) * `format_bytes` [#61535](https://github.com/StarRocks/starrocks/pull/61535) * `encode_sort_key` [#61781](https://github.com/StarRocks/starrocks/pull/61781) * `column_size` and `column_compressed_size` [#62481](https://github.com/StarRocks/starrocks/pull/62481) * Provides the following syntactic extensions: * Supports IF NOT EXISTS keywords in CREATE ANALYZE FULL TABLE. [#59789](https://github.com/StarRocks/starrocks/pull/59789) * Supports EXCLUDE clauses in SELECT. [#57411](https://github.com/StarRocks/starrocks/pull/57411/files) * Supports FILTER clauses in aggregate functions, improving readability and execution efficiency of conditional aggregations. [#58937](https://github.com/StarRocks/starrocks/pull/58937) ##### Behavior Changes[​](#behavior-changes-11 "Direct link to Behavior Changes") * Adjust the logic of the materialized view parameter `auto_partition_refresh_number` to limit the number of partitions to refresh regardless of auto refresh or manual refresh. [#62301](https://github.com/StarRocks/starrocks/pull/62301) * Flat JSON is enabled by default. [#62097](https://github.com/StarRocks/starrocks/pull/62097) * The default value of the system variable `enable_materialized_view_agg_pushdown_rewrite` is set to `true`, indicating that aggregation pushdown for materialized view query rewrite is enabled by default. [#60976](https://github.com/StarRocks/starrocks/pull/60976) * Changed the type of some columns in `information_schema.materialized_views` to better align with the corresponding data. [#60054](https://github.com/StarRocks/starrocks/pull/60054) * The `split_part` function returns NULL when the delimiter is not matched. [#56967](https://github.com/StarRocks/starrocks/pull/56967) * Use STRING to replace fixed-length CHAR in CTAS/CREATE MATERIALIZED VIEW to avoid deducing the wrong column length, which may cause materialized view refresh failures. [#63114](https://github.com/StarRocks/starrocks/pull/63114) [#62476](https://github.com/StarRocks/starrocks/pull/62476) * Data Cache-related configurations are simplified. [#61640](https://github.com/StarRocks/starrocks/issues/61640) * `datacache_mem_size` and `datacache_disk_size` are now effective. * `storage_page_cache_limit`, `block_cache_mem_size`, `block_cache_disk_size` are deprecated. * Added new catalog properties (`remote_file_cache_memory_ratio` for Hive, and `iceberg_data_file_cache_memory_usage_ratio` and `iceberg_delete_file_cache_memory_usage_ratio` for Iceberg) to limit the memory resources used for Hive and Iceberg metadata cache, and set the default values to `0.1` (10%). Adjust the metadata cache TTL to 24 hours. [#63459](https://github.com/StarRocks/starrocks/pull/63459) [#63373](https://github.com/StarRocks/starrocks/pull/63373) [#61966](https://github.com/StarRocks/starrocks/pull/61966) [#62288](https://github.com/StarRocks/starrocks/pull/62288) * SHOW DATA DISTRIBUTION now will not merge the statistics of all materialized indexes with the same bucket sequence number. It only shows data distribution at the materialized index level. [#59656](https://github.com/StarRocks/starrocks/pull/59656) * The default bucket size for automatic bucket tables is changed from 4GB to 1GB to improve performance and resource utilization. [#63168](https://github.com/StarRocks/starrocks/pull/63168) * The system determines the Partial Update mode based on the corresponding session variable and the number of columns in the INSERT statement. [#62091](https://github.com/StarRocks/starrocks/pull/62091) * Optimized the `fe_tablet_schedules` view in the Information Schema. [#62073](https://github.com/StarRocks/starrocks/pull/62073) [#59813](https://github.com/StarRocks/starrocks/pull/59813) * Renamed the `TABLET_STATUS` column to `SCHEDULE_REASON`, the `CLONE_SRC` column to `SRC_BE_ID`, and the `CLONE_DEST` column to `DEST_BE_ID`. * The data types of the `CREATE_TIME`, `SCHEDULE_TIME` and `FINISH_TIME` columns have been changed from `DOUBLE` to `DATETIME`. * The `is_leader` label has been added to some FE metrics. [#63004](https://github.com/StarRocks/starrocks/pull/63004) * Shared-data clusters using Microsoft Azure Blob Storage and Data Lake Storage Gen 2 as object storage will experience Data Cache failure after being upgraded to v4.0. The system will automatically reload the cache. --- ## Release 4.1 ### StarRocks version 4.1 danger **Container Image Issue (v4.1.0)** Due to an unstable load order issue in the v4.1.0 container image, BE processes may fail to start reliably in container environments. **Container environment users should NOT upgrade to v4.1.0.** Please wait for v4.1.1, which includes the fix ([#71825](https://github.com/StarRocks/starrocks/pull/71825)). warning **Downgrade Notes** * After upgrading StarRocks to v4.1, DO NOT downgrade to any v4.0 version below v4.0.6. Due to internal changes in data layout introduced in v4.1 (related to tablet splitting and distribution mechanisms), clusters upgraded to v4.1 may generate metadata and storage structures that are not fully compatible with earlier versions. As a result, downgrade from v4.1 is only supported to v4.0.6 or later. Downgrading to versions prior to v4.0.6 is not supported. This limitation is due to backward compatibility constraints in how earlier versions interpret tablet layout and distribution metadata. #### 4.1.6[​](#416 "Direct link to 4.1.6") Release Date: September 30, 2026 ##### Behavior Changes[​](#behavior-changes "Direct link to Behavior Changes") * The default HTTP client for the AWS SDK is switched from Poco back to the SDK's own client: `enable_poco_client_for_aws_sdk` now defaults to `false`. Set it to `true` to restore the previous behavior. [#78375](https://github.com/StarRocks/starrocks/pull/78375) * Online optimize table is now opt-in: the FE configuration `enable_online_optimize_table` defaults to `false`, so `ALTER TABLE ... DISTRIBUTED BY ...` again blocks writes unless the config is turned on. [#78206](https://github.com/StarRocks/starrocks/pull/78206) * jemalloc's dedicated oversize arena is enabled by default, so large allocations no longer land in the per-CPU arena of the calling thread. This improves reuse of large buffers and reduces fragmentation, and changes how BE and CN memory is distributed across arenas. [#78396](https://github.com/StarRocks/starrocks/pull/78396) * Shared-data tablet metadata and transaction log files are now written with an Adler-32 checksum by default (`lake_enable_protobuf_file_checksum`), so corruption is detected on read. Readers already verified the checksum whenever a file carried one, so no downgrade step is needed. [#75334](https://github.com/StarRocks/starrocks/pull/75334) * A shared-data primary-key load that mixes `UPSERT` and `DELETE` rows for the same key now preserves their in-transaction order by default (`lake_enable_pk_preserve_txn_delete_order`). [#76211](https://github.com/StarRocks/starrocks/pull/76211) * Multi-table transactions now participate in shared-data batch publish instead of publishing one transaction at a time, which removes the serialization that CDC micro-batches fanning out to many tables used to hit. [#77029](https://github.com/StarRocks/starrocks/pull/77029) * A shared-data partition now advances straight to the final version of a batch publish instead of through each intermediate version, so a query can no longer request an intermediate version that has no tablet metadata object, which surfaced as `The specified key does not exist`. [#79175](https://github.com/StarRocks/starrocks/pull/79175) * A shared-data publish that keeps failing now backs off and throttles its retry logging instead of emitting the `previous publish version task has not finished` message on every attempt, which could dominate the FE leader's CPU and log volume during an incident. [#79299](https://github.com/StarRocks/starrocks/pull/79299) * Del-file tombstone rows now count toward `cloud_native_pk_index_rebuild_rows_threshold`, so a delete-heavy shared-data workload triggers the primary-key index early flush it was supposed to. [#76222](https://github.com/StarRocks/starrocks/pull/76222) * A shared-data column-upsert publish that supersedes its own segments now merges the delete vector, which previously could leave two live rows for one primary key. The damage only surfaced much later, when a CN restart rebuilt the persistent index and reported `insert found duplicate key`. [#78797](https://github.com/StarRocks/starrocks/pull/78797) * The shared-data primary-key index compactor no longer stops silently while the flush side keeps producing sstables, which left a tablet whose persistent index grew without bound. [#78812](https://github.com/StarRocks/starrocks/pull/78812) * HDFS files are no longer `hsync`ed before they are closed. The `hsync` was a workaround for a crash that a separate change had already fixed, and it forced an `fsync()` on every DataNode replica on each close. [#78864](https://github.com/StarRocks/starrocks/pull/78864) * A Parquet `null_count` that under-reports NULLs is no longer taken at face value during decoding and pruning. Files written by older parquet-mr versions previously returned corrupted values or dropped rows. [#78329](https://github.com/StarRocks/starrocks/pull/78329) * List partition pruning of `!=`, `NOT IN`, and `IS NOT NULL` no longer discards a multi-value partition just because its value list also contains `NULL`. Queries on such a table, including joins that derive `IS NOT NULL` on the join key, return the non-null rows that were previously lost. [#78946](https://github.com/StarRocks/starrocks/pull/78946) * Batch DDL can now build date range partitions that start at year `0000`, which is a legal `DATE` value. Such a statement previously failed with `the range of the partition is invalid`. [#78746](https://github.com/StarRocks/starrocks/pull/78746) * `max_partitions_in_one_batch` and the partition-number limit now count the partitions the request in hand is about to create. An oversized batch is rejected up front instead of being admitted and only failing the following batch. [#79379](https://github.com/StarRocks/starrocks/pull/79379) * `TRUNCATE TABLE ..
` now checks the `DELETE` privilege against the catalog named in the statement instead of the session's current catalog. A user who was previously allowed through may now be denied, and vice versa. [#79047](https://github.com/StarRocks/starrocks/pull/79047) * Streaming aggregation tuning is now driven by the group-by cardinality instead of the rows left after `HAVING`, so a query whose `HAVING` clause filters most groups away no longer loses its local pre-aggregation. [#79064](https://github.com/StarRocks/starrocks/pull/79064) * Audit log rows now carry `scanBytes`, `scanRows`, `returnRows`, `cpuCostNs`, and `memCostBytes` for DML that targets an external Iceberg or Hive table or the `FILES()` table function, and for asynchronous materialized-view refreshes. These fields previously reported `0`. [#74072](https://github.com/StarRocks/starrocks/pull/74072) [#74954](https://github.com/StarRocks/starrocks/pull/74954) * The query profile and `EXPLAIN ANALYZE` now report the plan a retried query actually ran, rather than the plan from the first attempt. [#79225](https://github.com/StarRocks/starrocks/pull/79225) * A Broker Load no longer writes its submit-time snapshot of `enable_tablet_pre_split` back onto the submitting client's live session variable. [#79177](https://github.com/StarRocks/starrocks/pull/79177) * `ALTER MATERIALIZED VIEW ... ACTIVE` works again for non-aggregate incremental materialized views; the storage-filled `__ROW_ID__` column is kept out of the materialized-view DDL column list. [#77017](https://github.com/StarRocks/starrocks/pull/77017) * The `Operation` column of `SHOW ALTER TABLE OPTIMIZE` now shows a readable description of the optimize operation instead of an internal object address such as `com.starrocks.sql.ast.OptimizeClause@b5dc069`. [#75948](https://github.com/StarRocks/starrocks/pull/75948) * A column that is missing from the tablet schema no longer aborts the BE during predicate push-down; the predicate is simply not pushed down. [#79234](https://github.com/StarRocks/starrocks/pull/79234) * When `OlapTableSink` initialization fails, the partially initialized sink is no longer taken through the full close path, which could dereference state that was never created and abort the BE. The original initialization error is returned instead. [#79370](https://github.com/StarRocks/starrocks/pull/79370) * Short key index page geometry is now validated without 32-bit overflow, so a corrupted page footer is rejected instead of wrapping and causing a heap out-of-bounds write. [#79048](https://github.com/StarRocks/starrocks/pull/79048) * An HTTPS request whose TLS handshake times out is no longer treated as connected and retried on the same socket. A stalled request now fails after approximately one timeout instead of `connect_timeout` plus two receive timeouts. [#79369](https://github.com/StarRocks/starrocks/pull/79369) ##### Improvements[​](#improvements "Direct link to Improvements") * LDAP and Active Directory groups can now be resolved from the user's `memberOf` attribute, so a new directory group no longer requires a matching group-provider configuration on the cluster. [#79254](https://github.com/StarRocks/starrocks/pull/79254) * The audit log now records the query's error message, not just the coarse `ErrorCode`, so a syntax error, an unknown table, and a privilege failure are distinguishable without correlating back to `fe.log`. [#78798](https://github.com/StarRocks/starrocks/pull/78798) * Added server-level spill metrics that aggregate spill activity across all spillable operators, broken down by local and remote storage backend. [#71827](https://github.com/StarRocks/starrocks/pull/71827) * Hudi metadata readers are now reused across the partitions of one scan instead of being rebuilt per partition, which shortens planning for tables with many partitions when the Hudi metadata table is enabled. [#79367](https://github.com/StarRocks/starrocks/pull/79367) * Tablet split points are now computed from a segment's existing indexes instead of a dedicated full sort key index page, so tables that never split no longer pay for writing and storing that page. [#79050](https://github.com/StarRocks/starrocks/pull/79050) * `/memz` now accepts jemalloc's own statistics options, which makes the per-arena dirty and muzzy page counts and the extent size-class breakdown reachable from a running BE. [#79090](https://github.com/StarRocks/starrocks/pull/79090) ##### Security[​](#security "Direct link to Security") * \[CVE-2026-55971] Bumped Apache Thrift to 0.24.0 to fix a heap-based buffer overflow in `THeaderTransport::untransform()` in the C++ bindings. [#76922](https://github.com/StarRocks/starrocks/pull/76922) * \[CVE-2026-42537, CVE-2026-59646] Bumped Apache Ranger to 2.9.0 to fix an RCE via JDBC URL injection, and Bouncy Castle to 1.85 to fix a DTLS reassembler buffer overrun. [#79331](https://github.com/StarRocks/starrocks/pull/79331) * \[CVE-2026-71290] Bumped Apache HttpClient5 to 5.6.4. [#79538](https://github.com/StarRocks/starrocks/pull/79538) ##### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") The following issues have been fixed: * `SHOW PROFILELIST`, `ANALYZE PROFILE`, `get_query_profile()`, and `/api/profile` returned every cached profile, including other users' SQL text and plans, to any authenticated user. The new FE configuration `authorization_enable_query_profile_access_check` restricts a cached profile to the user who ran the query and to holders of `SYSTEM OPERATE`. [#79375](https://github.com/StarRocks/starrocks/pull/79375) * The authorization check, `INSERT` planning, view expansion, the privilege check, and `inspect_mv_refresh_info` made external-catalog round trips while holding an FE metadata lock, so a slow metastore stalled every DDL waiting on the same lock. Those calls now happen outside the lock. [#79319](https://github.com/StarRocks/starrocks/pull/79319) [#79226](https://github.com/StarRocks/starrocks/pull/79226) * A gtid and a partition version epoch could be generated on a non-leader FE, so two nodes could produce the same number at the same moment. [#78799](https://github.com/StarRocks/starrocks/pull/78799) * `FULL OUTER JOIN ... USING (c1, c2, ...)` returned the wrong values: the first `USING` column was correct and every later one carried the previous column's value. Adding `ORDER BY ` to the same query failed with an `IndexOutOfBoundsException`. [#79298](https://github.com/StarRocks/starrocks/pull/79298) * A `HAVING` predicate was lost when `GROUPING SETS`, `ROLLUP`, or `CUBE` was rewritten into a union, so the query returned rows it was supposed to filter out. [#79415](https://github.com/StarRocks/starrocks/pull/79415) * `GROUP BY ROLLUP(a, b, a)`, and `CUBE` or `GROUPING SETS` naming a column twice within one set, produced a grouping set that carried the same key twice. [#78261](https://github.com/StarRocks/starrocks/pull/78261) * Removing the aggregation above an aggregate-key table dropped the query's `LIMIT` before it reached the storage layer, so the BE read the whole table. [#79414](https://github.com/StarRocks/starrocks/pull/79414) * Splitting an `OR` join into a union lined up the branches inconsistently, because the union's output columns and the first branch were ordered by column id while the other branches followed the projection map's iteration order. [#79473](https://github.com/StarRocks/starrocks/pull/79473) * The skew-join rewrite replicated the preserved side of an outer join, so one input row could be returned several times and the result moved between runs. Its salt table also received the `generate_series` arguments in the wrong order, so the series emitted nothing. [#79168](https://github.com/StarRocks/starrocks/pull/79168) [#79268](https://github.com/StarRocks/starrocks/pull/79268) * `SELECT DISTINCT` accepted an `ORDER BY` on a column that the `DISTINCT` drops, and the distinct push-down below a window left dropped columns in a projection. [#77217](https://github.com/StarRocks/starrocks/pull/77217) [#77218](https://github.com/StarRocks/starrocks/pull/77218) * A window over an always-non-null aggregate such as `bitmap_union_count()` lost its declared nullability, so a `DISTINCT` or `GROUP BY` over it failed with `error nullablel column`. [#79009](https://github.com/StarRocks/starrocks/pull/79009) * `percentile_cont` merge tracked exhaustion by value instead of by position, which could abort the BE. [#77261](https://github.com/StarRocks/starrocks/pull/77261) * `count(DISTINCT percentile_hash(...))`, and any `DISTINCT` aggregate over a `PERCENTILE` value, crashed the BE; such a call is now rejected. [#78795](https://github.com/StarRocks/starrocks/pull/78795) * The scalar agg-state combinators crashed on a null memory pool and raced on their nested function context. [#76910](https://github.com/StarRocks/starrocks/pull/76910) * A JSON subfield aggregate on an aggregate table failed with a raw `NullPointerException`, because the synthetic subfield column has no aggregation type. [#77952](https://github.com/StarRocks/starrocks/pull/77952) * Streaming pre-aggregation could run out of memory under `enable_spill = true`, because the limited-memory budget was latched once and never revised. [#76702](https://github.com/StarRocks/starrocks/pull/76702) * A spilling hash join cancelled while it was restoring build partitions crashed the BE, because the probe operator closed without waiting for the spill I/O tasks. [#79085](https://github.com/StarRocks/starrocks/pull/79085) * The local pass-through exchange kept appending chunks after the receiver was cancelled, and its channel could be used after free because the sink and the receiver did not share ownership. Sink readiness notifications also selected the sink observable but emitted a source event, so a driver blocked on `OUTPUT_FULL` could stay blocked. [#76603](https://github.com/StarRocks/starrocks/pull/76603) [#77088](https://github.com/StarRocks/starrocks/pull/77088) [#76782](https://github.com/StarRocks/starrocks/pull/76782) * The query cache could store an incomplete per-tablet result and serve it to later queries, and a fragment containing an ASOF join was cached even though the temporal condition never reached the digest. [#77066](https://github.com/StarRocks/starrocks/pull/77066) [#77563](https://github.com/StarRocks/starrocks/pull/77563) * An ASOF join whose temporal condition does not compare the two sides crashed the BE; such a condition is now rejected. [#79049](https://github.com/StarRocks/starrocks/pull/79049) * A nested-lambda slot with no parent tuple was treated as bound by the TopN node's tuple set when building runtime filters, and an order-by hint dereferenced a column-less slot pushed into the scan as a heavy expression. [#78050](https://github.com/StarRocks/starrocks/pull/78050) [#78527](https://github.com/StarRocks/starrocks/pull/78527) * Nested `array_map` and `array_sort` lambdas failed to plan or evaluate with `Input dependency cols check failed` or `Expr evaluate meet error: slot_id N not found`, because common sub expressions hoisted by an inner lambda escaped the scope that defined them. [#78645](https://github.com/StarRocks/starrocks/pull/78645) [#78769](https://github.com/StarRocks/starrocks/pull/78769) [#78915](https://github.com/StarRocks/starrocks/pull/78915) * `array_sort`'s comparator validation shared one error `Status` across drivers, which could double-free it. [#78314](https://github.com/StarRocks/starrocks/pull/78314) * `array_contains(NULL, [1, 2])` and `array_position(NULL, [1, 2])` failed with `class com.starrocks.type.NullType cannot be cast to class com.starrocks.type.ArrayType`. [#76970](https://github.com/StarRocks/starrocks/pull/76970) * Analyzing `__iceberg_transform_truncate` or `__iceberg_transform_bucket` mutated the shared builtin function object in place, stamping its wildcard decimal signature with the precision and scale of the first query analyzed. [#76777](https://github.com/StarRocks/starrocks/pull/76777) * `PARTITION BY RANGE(())` with an empty argument list raised a raw `IndexOutOfBoundsException` instead of a parse error. [#75080](https://github.com/StarRocks/starrocks/pull/75080) * A range predicate was mapped onto a partition expression regardless of whether the expression preserves order and in which direction, so range and list partition pruning could drop partitions that hold matching rows. A `NULL` partition key was also looked up at the column's type minimum rather than strictly below it. Retention conditions that cast a `DATE` or `DATETIME` to text are accepted again. [#78024](https://github.com/StarRocks/starrocks/pull/78024) [#79111](https://github.com/StarRocks/starrocks/pull/79111) [#79187](https://github.com/StarRocks/starrocks/pull/79187) [#79257](https://github.com/StarRocks/starrocks/pull/79257) [#79688](https://github.com/StarRocks/starrocks/pull/79688) * The Iceberg predicate converter stripped a `CastOperator` unconditionally, so `WHERE CAST(d AS BIGINT) = 10` on a `DECIMAL(5,2)` column could prune a file holding `10.50`. [#78351](https://github.com/StarRocks/starrocks/pull/78351) * An incremental materialized view on an Iceberg base table was permanently inactivated by routine `rewrite_data_files` compaction, because the delta boundary advanced onto a skipped `REPLACE` snapshot. [#76950](https://github.com/StarRocks/starrocks/pull/76950) * The column min/max statistics cache was not invalidated by partition or schema DDL, so a stale entry could be served indefinitely. [#77441](https://github.com/StarRocks/starrocks/pull/77441) * Hive-family catalogs failed outright when the metastore or AWS Glue denied permission to read column statistics; they now degrade to empty statistics. Glue column-statistics requests also carry `catalogId`, so a cross-account deployment reads statistics from the configured catalog. [#78455](https://github.com/StarRocks/starrocks/pull/78455) [#78461](https://github.com/StarRocks/starrocks/pull/78461) * Elasticsearch external tables and Elasticsearch catalogs rejected clusters running version 8 or 9 with `Unsupported/Unknown ES Cluster version`, and concurrent queries could overwrite each other's request URL parameters, which surfaced as unrelated analysis errors such as `Column 'id' cannot be resolved`. [#78267](https://github.com/StarRocks/starrocks/pull/78267) [#79391](https://github.com/StarRocks/starrocks/pull/79391) * Predicates on a `DECIMAL` column of a Paimon table were converted using the StarRocks type family's default precision and scale instead of the column's Paimon `DecimalType`, so the literal could not represent the value. [#78760](https://github.com/StarRocks/starrocks/pull/78760) * Azure Blob credential lookup failed when the endpoint was configured as a full URL, because the Hadoop configuration key embedded the scheme. ADLS2 storage volumes also dropped their Azure Workload Identity token-file property on serialization, so reloading the volume fell back to Managed Identity and failed with `Identity not found`. [#78241](https://github.com/StarRocks/starrocks/pull/78241) [#79444](https://github.com/StarRocks/starrocks/pull/79444) * A JSON path with an array index such as `'[0].value.url'` threw a C++ exception for every row whose array was shorter than the index, which turned a large scan into a lock convoy inside the stack unwinder. The index is now bounds-checked. [#78569](https://github.com/StarRocks/starrocks/pull/78569) * `timestampdiff(MONTH, ...)`, `timestampdiff(YEAR, ...)`, `months_diff`, and `years_diff` returned `-1` instead of `0` when both arguments fell in the same month or year and the second was the earlier one. [#79037](https://github.com/StarRocks/starrocks/pull/79037) * Constant folding of `CAST( AS VARCHAR)` returned the Java identity hash code of the backing byte array, such as `[B@233c25e5`, instead of the value. [#77718](https://github.com/StarRocks/starrocks/pull/77718) * `DICTIONARY_GET` rejected a non-NULL key as NULL when the input column's cached `has_null` flag stayed conservatively true. [#76881](https://github.com/StarRocks/starrocks/pull/76881) * A low-cardinality dictionary was kept on a nested-loop join's `ON` columns even though the rewriter never decodes them, dictionary expressions were folded through derived dictionaries that carry their own codes, and a "rolling low-cardinality" column repeatedly invalidated the global dictionary and forced a full-table re-collection. [#78229](https://github.com/StarRocks/starrocks/pull/78229) [#78633](https://github.com/StarRocks/starrocks/pull/78633) [#78486](https://github.com/StarRocks/starrocks/pull/78486) * A `DECIMAL256` literal was hashed from its string representation while the BE buckets rows from the raw 32-byte value, so point lookups on a table distributed by a `DECIMAL256` column were sent to the wrong bucket. [#78495](https://github.com/StarRocks/starrocks/pull/78495) * An OLAP scan named its sort key columns by column name rather than by column id, so a renamed column no longer matched the tablet schema. [#77967](https://github.com/StarRocks/starrocks/pull/77967) * `PruneComplexTypeUtil` disabled complex-type subfield pruning for every column in the statement when it met a single expression it could not describe. The bail-out is now scoped to the expression that caused it, and dead subfield slots pulled out of the scan by predicate expression reuse are dropped. [#79181](https://github.com/StarRocks/starrocks/pull/79181) [#77456](https://github.com/StarRocks/starrocks/pull/77456) * `AstToSQLBuilder` printed the columns of a `SELECT * EXCLUDE` list in double quotes, which the grammar accepts nowhere, so the regenerated SQL no longer parsed. [#78052](https://github.com/StarRocks/starrocks/pull/78052) * `ALTER TABLE ... ADD COLUMN` on an Aggregate table silently created a key column when neither an aggregate function nor the `KEY` keyword was given, changing the grain the table aggregates on. Such a statement is now rejected as ambiguous. [#78359](https://github.com/StarRocks/starrocks/pull/78359) * A column whose type has no BE key coder, such as `JSON`, could be used as a sort key and crashed the BE; such a definition is now rejected in `CREATE TABLE` and `ALTER TABLE` across all distribution types. [#76795](https://github.com/StarRocks/starrocks/pull/76795) * `USING GIN("IMP_LIB" = "builtin")` was accepted by `CREATE TABLE` and then broke every `INSERT` with `Can not get inverted imp type`, because the property key reached the BE with its original casing. [#77818](https://github.com/StarRocks/starrocks/pull/77818) * A legacy `NGRAMBF` index persisted without its `gram_num` property made the BE read past the UTF-8 offset vector and crash the CN while building a segment. [#76989](https://github.com/StarRocks/starrocks/pull/76989) * Null flags were indexed linearly rather than by ordinal when a pushed-down predicate read a sparse range, so every row after a gap took the null flag of a skipped row. The dictionary page decoder also failed to advance when a pushed-down filter rejected the whole dictionary, which aborted the BE. [#78389](https://github.com/StarRocks/starrocks/pull/78389) [#79007](https://github.com/StarRocks/starrocks/pull/79007) * A child expression error aborted the BE instead of being propagated. [#77442](https://github.com/StarRocks/starrocks/pull/77442) * `ThreadPool` swallowed task exceptions by default and counted the task as finished. [#76863](https://github.com/StarRocks/starrocks/pull/76863) * Cancelling a load through `OlapTableSink` crashed the BE or CN with a SIGSEGV when Jaeger tracing was enabled. [#76869](https://github.com/StarRocks/starrocks/pull/76869) * `AsyncFlushOutputStream::close()` satisfied its promise while still holding the stream's own mutex, so the I/O thread woke futexes on freed memory after the waiter had destroyed the stream. [#79182](https://github.com/StarRocks/starrocks/pull/79182) * A CN could crash with SIGSEGV during graceful shutdown when an in-flight load reached a StarOS-backed filesystem after `shutdown_staros_worker()` had released the global worker. [#79058](https://github.com/StarRocks/starrocks/pull/79058) * Parallel compaction ran filesystem I/O on the brpc bthread, so the CN aborted with `Resource deadlock avoided`. [#76925](https://github.com/StarRocks/starrocks/pull/76925) * Compaction aborted with SIGFPE when a vector-index rowset reported a negative `data_disk_size`. [#78959](https://github.com/StarRocks/starrocks/pull/78959) * A corrupted local cache copy was re-read on every retry when a lake txn log read hit corruption, and primary-key index sstable block reads never verified the crc32c block trailer that the writer had always produced. [#78733](https://github.com/StarRocks/starrocks/pull/78733) [#77479](https://github.com/StarRocks/starrocks/pull/77479) * A partial-update rewrite that degenerated into a copy dropped the source's encryption metadata, leaving a segment that no reader could open and that compaction could not rewrite. [#79053](https://github.com/StarRocks/starrocks/pull/79053) * A shared-data schema change could record a rowset that references no data, after which every `MERGE_TABLET` on the partition failed with `tablet merge source has an empty ordinary rowset`. [#79167](https://github.com/StarRocks/starrocks/pull/79167) * Range-colocate split bounds were compared as serialized protobuf bytes, which the FE and the BE do not encode identically. [#76968](https://github.com/StarRocks/starrocks/pull/76968) * A plain single-table query on a member of an unstable range-colocate group failed with `range colocate group ... is in an unaligned state`. [#76971](https://github.com/StarRocks/starrocks/pull/76971) * Shards of a new physical partition of a hash-colocate shared-data table were created without colocation placement information and had to be migrated afterwards. [#79223](https://github.com/StarRocks/starrocks/pull/79223) * The FE did not tell the BE that a dropped tablet's table is range-distributed, which leaked files after a reshard. [#76861](https://github.com/StarRocks/starrocks/pull/76861) * Replaying a reshard job update did not refresh the resharding tablets, so after an FE leader change a partition's publish could 404 forever on a tablet metadata object that was never written. Replaying a reshard job could also write to the edit log when the table had been dropped, and reshard publish retries spun instead of being paced. [#79426](https://github.com/StarRocks/starrocks/pull/79426) [#79485](https://github.com/StarRocks/starrocks/pull/79485) [#77691](https://github.com/StarRocks/starrocks/pull/77691) * Shared-data partition versions were mutated in memory before the mutating operation was journaled, so a replayed version could move backwards. They are now derived from the journal and only ever advance. [#79296](https://github.com/StarRocks/starrocks/pull/79296) * A replication job divided by zero when the target tablet reported no replica. [#79329](https://github.com/StarRocks/starrocks/pull/79329) * The DataCache key hash was taken from a runtime dispatch table, so any change to that dispatch orphaned every cached block across an upgrade or rollback. It is now pinned to MurmurHash3. [#79262](https://github.com/StarRocks/starrocks/pull/79262) * The FE scan-range heap-safety check ran once per physical partition instead of once per scan node, which made it quadratic in the number of physical partitions. [#76978](https://github.com/StarRocks/starrocks/pull/76978) * Rowset metadata cache entries kept their initial charge after lazy-loaded indexes had increased segment memory usage, so the cache under-reported its footprint. [#77893](https://github.com/StarRocks/starrocks/pull/77893) * `Tablet::get_average_row_size()` walked `_rs_version_map` without holding `_meta_lock`, against the contract that map is declared with. [#79250](https://github.com/StarRocks/starrocks/pull/79250) * The FE thrift accept loop stalled for minutes on worker-pool saturation, and workers kept serving connections that had been queued long after their callers were gone. [#78838](https://github.com/StarRocks/starrocks/pull/78838) [#79173](https://github.com/StarRocks/starrocks/pull/79173) * A non-leader FE whose journal applier was stuck kept serving stale metadata locally instead of forwarding, because the readiness verdict was only reached after a replay cycle returned. [#79041](https://github.com/StarRocks/starrocks/pull/79041) * The `getLoads` RPC returned every load record in a single message, so a busy cluster filled the leader log with `FE RPC failure ... MaxMessageSize reached` and the `loads_history` archive rolled back and permanently lost rows. The RPC is now paginated by job id. [#78898](https://github.com/StarRocks/starrocks/pull/78898) * A task run that migrated between the pending queue, the running map, the in-memory history, and the archived history while `task_runs` was being read could be reported twice. [#79438](https://github.com/StarRocks/starrocks/pull/79438) * `SHOW ALTER TABLE ROLLUP` and `SHOW ALTER MATERIALIZED VIEW` accepted `WHERE`, `ORDER BY`, and `LIMIT` and then ignored them; only `SHOW ALTER TABLE COLUMN` applied them. `FinishedTime` is also accepted in a `SHOW ALTER TABLE ROLLUP` predicate. [#79281](https://github.com/StarRocks/starrocks/pull/79281) [#79498](https://github.com/StarRocks/starrocks/pull/79498) * Loading the SQL blacklist rebuilt and sorted its immutable snapshot once per rule, which is quadratic in the number of rules and ran at every FE start and checkpoint. [#78752](https://github.com/StarRocks/starrocks/pull/78752) * `AsyncTaskQueue` busy-spun between `getOutputs()` and `tryGetOutputs()` after the consumer thread was interrupted, because the interrupt was swallowed without clearing `hasMoreOutput`. [#79297](https://github.com/StarRocks/starrocks/pull/79297) * A runtime change of `sys_log_level` was written to the configuration but never applied, even though the item is documented as mutable. [#79200](https://github.com/StarRocks/starrocks/pull/79200) * `has_enable_heap_profile()` read the boolean jemalloc node `prof.active` into an `int`, so it always reported heap profiling as disabled. [#78820](https://github.com/StarRocks/starrocks/pull/78820) * `System.print()` on a script's foreign objects printed `instance of ` instead of the object's text, because the `toString` bindings were registered as methods rather than getters. [#78877](https://github.com/StarRocks/starrocks/pull/78877) * Loading into an expression-partitioned table through the multi-table transaction stream load interface failed at commit whenever the BE had to create a partition, with `automatic create partition failed. error: txn ... not exist`. [#78790](https://github.com/StarRocks/starrocks/pull/78790) * In combined txn log mode, a coordinator whose own end-of-stream arrived early could wait for the txn log collector indefinitely when the load was cancelled. [#79389](https://github.com/StarRocks/starrocks/pull/79389) * A completed merge-commit load could report less than full merge-window progress in `information_schema.loads`. [#78723](https://github.com/StarRocks/starrocks/pull/78723) * The scan `SampleTime` counter reported a block count rather than elapsed time. [#78732](https://github.com/StarRocks/starrocks/pull/78732) * The LZ4 and ZSTD Arrow IPC codecs were missing from the FE classpath for the Arrow Flight SQL service. [#76921](https://github.com/StarRocks/starrocks/pull/76921) * Upgraded StarOS, starlet, and starcache to v4.1.4. [#79429](https://github.com/StarRocks/starrocks/pull/79429) #### 4.1.5[​](#415 "Direct link to 4.1.5") Release Date: September 8, 2026 ##### Improvements[​](#improvements-1 "Direct link to Improvements") * Supports `ANALYZE SAMPLE TABLE PREDICATE COLUMNS` for external tables. [#77243](https://github.com/StarRocks/starrocks/pull/77243) * Extended sample-based tablet pre-split: supports non-key expressions, dynamic-overwrite temporary partitions, and Iceberg and overwrite loads. [#77843](https://github.com/StarRocks/starrocks/pull/77843) [#77842](https://github.com/StarRocks/starrocks/pull/77842) [#77983](https://github.com/StarRocks/starrocks/pull/77983) * Streams source segments for lake column updates to reduce memory usage. [#77275](https://github.com/StarRocks/starrocks/pull/77275) * The background metadata refresh for Paimon catalogs now refreshes only recently accessed and changed tables instead of enumerating the whole warehouse each cycle. [#78057](https://github.com/StarRocks/starrocks/pull/78057) * Passes `limit` to the Paimon `getRemoteFiles` call during partition pruning. [#76943](https://github.com/StarRocks/starrocks/pull/76943) * Extended GROUPING SETS pushdown to support `AVG` and `COUNT`. [#77832](https://github.com/StarRocks/starrocks/pull/77832) * Allows including `EXPLAIN COSTS` output in the query profile. [#73005](https://github.com/StarRocks/starrocks/pull/73005) * Makes starlet object-store upload thresholds configurable at runtime. [#78448](https://github.com/StarRocks/starrocks/pull/78448) * Reduced lock contention: removed the global metacache lock for lake segments and made SQL blacklist verification lock-free. [#78260](https://github.com/StarRocks/starrocks/pull/78260) [#78476](https://github.com/StarRocks/starrocks/pull/78476) * Reclaims tablet metadata by dropping `delete_predicate` on compaction and replication archival. [#76870](https://github.com/StarRocks/starrocks/pull/76870) * Checksums shared-data delete files with CRC32C. [#77798](https://github.com/StarRocks/starrocks/pull/77798) * Bounds how long a cancelled query waits for unreported instances. [#76909](https://github.com/StarRocks/starrocks/pull/76909) * Various jemalloc improvements for better memory efficiency and observability: builds a 4K-page variant alongside the 64K one on arm64, exposes dirty/muzzy page and mapped/retained metrics, and applies mutable `jemalloc_conf` options at runtime. [#77886](https://github.com/StarRocks/starrocks/pull/77886) [#78159](https://github.com/StarRocks/starrocks/pull/78159) [#78225](https://github.com/StarRocks/starrocks/pull/78225) [#78320](https://github.com/StarRocks/starrocks/pull/78320) [#78333](https://github.com/StarRocks/starrocks/pull/78333) * Added observability: a `max_journal_replay_lag` gauge on the leader FE, and additional lake publish-version tracing. [#78382](https://github.com/StarRocks/starrocks/pull/78382) [#76658](https://github.com/StarRocks/starrocks/pull/76658) [#77048](https://github.com/StarRocks/starrocks/pull/77048) * Releases the reshard output shards' placement pin when the job finishes. [#76945](https://github.com/StarRocks/starrocks/pull/76945) * Upgraded StarOS to v4.1.2. [#77657](https://github.com/StarRocks/starrocks/pull/77657) ##### Security[​](#security-1 "Direct link to Security") * \[CVE-2025-55163] Aligned all `io.grpc` artifacts via `grpc-bom` so `grpc-netty` matches `grpc-core`. [#77171](https://github.com/StarRocks/starrocks/pull/77171) * \[CVE-2026-59902] Bumped Netty to 4.1.137.Final. [#78287](https://github.com/StarRocks/starrocks/pull/78287) * \[CVE-2026-54399] Bumped Apache HttpCore to 5.4.3 to fix an uncontrolled-resource-consumption (DoS) vulnerability in the HTTP/1.1 message parser. [#77753](https://github.com/StarRocks/starrocks/pull/77753) * \[CVE-2025-48924] Upgraded the Broker `commons-lang3` to 3.20.0 and removed unused `commons-lang` 2.6 and `commons-configuration` 1.6. [#77748](https://github.com/StarRocks/starrocks/pull/77748) * \[CVE] Rebuilt the pprof binary on a newer release and removed unused flamegraph tooling from the BE container image. [#77846](https://github.com/StarRocks/starrocks/pull/77846) ##### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") The following issues have been fixed: **Crashes and stability** * Force BE exit when the fatal-signal handler hangs after marking the process as crashing. [#76491](https://github.com/StarRocks/starrocks/pull/76491) * Fixed BE crashes from constant/invalid arguments in `tokenize`, `ngram_search`, `generate_series`, `corr`/`covar_pop`/`covar_samp`, `strpos` (INT32\_MIN), and `count()` over an array of decimal. [#77102](https://github.com/StarRocks/starrocks/pull/77102) [#77103](https://github.com/StarRocks/starrocks/pull/77103) [#77216](https://github.com/StarRocks/starrocks/pull/77216) [#77260](https://github.com/StarRocks/starrocks/pull/77260) [#77092](https://github.com/StarRocks/starrocks/pull/77092) * Fixed a BE crash when one `ALTER` adds an ordinary column and a generated column, and a cache-select crash after a read-schema reorder. [#77436](https://github.com/StarRocks/starrocks/pull/77436) [#77245](https://github.com/StarRocks/starrocks/pull/77245) * Fixed a BE crash when the query cache reuses a lane for a partitioned hash join, and a hang when a lane's join has an empty build side. [#77564](https://github.com/StarRocks/starrocks/pull/77564) [#77571](https://github.com/StarRocks/starrocks/pull/77571) * Fixed an out-of-range read and silent row loss in `JsonColumn` append, an integer overflow in `BinaryColumnBase::replicate`, and an `array_map` DCHECK failure when one array column is passed as several lambda arguments. [#77090](https://github.com/StarRocks/starrocks/pull/77090) [#77545](https://github.com/StarRocks/starrocks/pull/77545) [#77433](https://github.com/StarRocks/starrocks/pull/77433) * Fixed `substr` returning bytes of neighboring rows on invalid UTF-8, and `array_sort`'s lambda comparator driving the sort out of bounds. [#78722](https://github.com/StarRocks/starrocks/pull/78722) [#77687](https://github.com/StarRocks/starrocks/pull/77687) * Keep parquet decoder scratch buffers off the stack, and bound the raw-JSON view in the `SimdJsonConverter` error path. [#77615](https://github.com/StarRocks/starrocks/pull/77615) * Fixed a tablet-scheduler NPE on a disk with no reported storage medium, and NPEs when deparsing `INSERT ... VALUES (DEFAULT, ...)` and in `PrepareCollectMetaTask` when the optimizer has no query id. [#76966](https://github.com/StarRocks/starrocks/pull/76966) [#76963](https://github.com/StarRocks/starrocks/pull/76963) [#77051](https://github.com/StarRocks/starrocks/pull/77051) **Query engine and optimizer** * Fixed wrong or missing results in several optimizer paths: streaming distinct global limit, ranking window pre-agg across sort groups, constant aggregated value re-appended in the merge phase, GROUP BY rules not applied to every child, and analytic frames starting at a FOLLOWING offset. [#77086](https://github.com/StarRocks/starrocks/pull/77086) [#77068](https://github.com/StarRocks/starrocks/pull/77068) [#77069](https://github.com/StarRocks/starrocks/pull/77069) [#77099](https://github.com/StarRocks/starrocks/pull/77099) [#77342](https://github.com/StarRocks/starrocks/pull/77342) * Fixed an NL Join null-aware anti join losing probe rows, conjunct evaluation on an empty chunk in NL Join probe, and a `GROUPING SETS` aggregation joined on its group-by keys failing with no executable plan. [#77970](https://github.com/StarRocks/starrocks/pull/77970) [#77344](https://github.com/StarRocks/starrocks/pull/77344) [#77296](https://github.com/StarRocks/starrocks/pull/77296) * Fixed grouping-sets-to-union-all rewrite ordering, duplicating a limit as a set operator, and keeping Set operators out of colocate exec groups to avoid a hang. [#77380](https://github.com/StarRocks/starrocks/pull/77380) [#77692](https://github.com/StarRocks/starrocks/pull/77692) [#77025](https://github.com/StarRocks/starrocks/pull/77025) * Fixed subquery planning in a window's `PARTITION BY`/`ORDER BY`, keeping the decimal type of the `if()` built for multi-column `count(distinct)`, and normalizing scan heavy-expr result columns to the slot's nullability. [#77087](https://github.com/StarRocks/starrocks/pull/77087) [#77346](https://github.com/StarRocks/starrocks/pull/77346) [#77242](https://github.com/StarRocks/starrocks/pull/77242) * Fixed an untyped array literal gaining a bogus type (breaking views), a CTE anchoring issue when the force-inline limit is hit, and reuse of inferred CTAS columns for a recursive CTE. [#76986](https://github.com/StarRocks/starrocks/pull/76986) [#77018](https://github.com/StarRocks/starrocks/pull/77018) [#76453](https://github.com/StarRocks/starrocks/pull/76453) * Fixed several query cache correctness issues by adding the scan schema, join predicate common sub-expressions, and the ANN search spec to the query cache digest, and disabling the cache when a `SortNode` builds the TOPN filter. [#77559](https://github.com/StarRocks/starrocks/pull/77559) [#77565](https://github.com/StarRocks/starrocks/pull/77565) [#77586](https://github.com/StarRocks/starrocks/pull/77586) [#77404](https://github.com/StarRocks/starrocks/pull/77404) * Fixed folding `COUNT(*)` from row counts that never covered the visible version, unconditional UKFK constraint collection in join reorder, and keeping columns read by OR-resident predicates after inverted-index filtering. [#77190](https://github.com/StarRocks/starrocks/pull/77190) [#77197](https://github.com/StarRocks/starrocks/pull/77197) [#77659](https://github.com/StarRocks/starrocks/pull/77659) * Fixed datetime parsing correctness: bounded the Teradata and Joda parsers to the input string, and kept `DateValue::weekday()` in range for out-of-range Julian values. [#77098](https://github.com/StarRocks/starrocks/pull/77098) [#77310](https://github.com/StarRocks/starrocks/pull/77310) * Serialize `EXCEPT`/`INTERSECT` keys by row when the batch buffer would be huge, and rebuild the JSON key column when a set operation deserializes its keys. [#77821](https://github.com/StarRocks/starrocks/pull/77821) [#77215](https://github.com/StarRocks/starrocks/pull/77215) * Additional optimizer/expression fixes: check the status before unwrapping a `StatusOr`, decide property satisfaction once when enforcing it, read constant arguments at index 0 in `encode_fingerprint_sha256`, skip `CHAR` predicate zero-padding for non-positive column length, do not push a non-positive ranking predicate into a TopN, keep the NULL partition when pruning with a deduced partition predicate, stop reading past the end of the output schema in `_switch_context`, sort a run and merge runs by the same NULL convention, fill `Analytor::_fns` before prepare can fail, and stop forcing external CTAS list partition columns to `NOT NULL`. [#77052](https://github.com/StarRocks/starrocks/pull/77052) [#77091](https://github.com/StarRocks/starrocks/pull/77091) [#77104](https://github.com/StarRocks/starrocks/pull/77104) [#77444](https://github.com/StarRocks/starrocks/pull/77444) [#77445](https://github.com/StarRocks/starrocks/pull/77445) [#77596](https://github.com/StarRocks/starrocks/pull/77596) [#77660](https://github.com/StarRocks/starrocks/pull/77660) [#77679](https://github.com/StarRocks/starrocks/pull/77679) [#77758](https://github.com/StarRocks/starrocks/pull/77758) [#78468](https://github.com/StarRocks/starrocks/pull/78468) * Fixed statement/deparse issues: validate the static partition clause for Iceberg/Hive `INSERT` with a column list, keep the `TEMPORARY` qualifier when deparsing a partition reference, support CTAS deparse to fix an empty SQL statement in the profile, and set `isSingleStmt` in `proxyExecute` so forwarded statements keep the origin SQL. [#76659](https://github.com/StarRocks/starrocks/pull/76659) [#76987](https://github.com/StarRocks/starrocks/pull/76987) [#77978](https://github.com/StarRocks/starrocks/pull/77978) [#77982](https://github.com/StarRocks/starrocks/pull/77982) **JSON and data types** * Fixed a large `BinaryColumn` SerDe issue, statistics collection for columns with dots in names, and reading `multi_unnest`/`unnest` array offsets as `uint32` instead of `int32`. [#75504](https://github.com/StarRocks/starrocks/pull/75504) [#71301](https://github.com/StarRocks/starrocks/pull/71301) [#78479](https://github.com/StarRocks/starrocks/pull/78479) [#78480](https://github.com/StarRocks/starrocks/pull/78480) * Fixed reduced Flat JSON vertical-compaction memory usage and reading the values themselves when a struct-subfield predicate is evaluated by rowid. [#77682](https://github.com/StarRocks/starrocks/pull/77682) [#77891](https://github.com/StarRocks/starrocks/pull/77891) **Materialized views** * Fixed IVM refresh with external authorization, the MV rewrite staleness baseline (now uses the last-freshness-confirmed time), and skipping MV rewrite for time-travel queries. [#77210](https://github.com/StarRocks/starrocks/pull/77210) [#76758](https://github.com/StarRocks/starrocks/pull/76758) [#77683](https://github.com/StarRocks/starrocks/pull/77683) * Fixed assigning index ids to GIN indexes created on a materialized view, and keeping view scan projection when replacing it back in view-based MV rewrite. [#77340](https://github.com/StarRocks/starrocks/pull/77340) [#77613](https://github.com/StarRocks/starrocks/pull/77613) * Additional materialized view fixes: validate `default_mv_refresh_mode`, fix `ORDER BY` on an incremental MV widening its primary key instead of sorting, and bound `information_schema.materialized_views` internal queries by `query_timeout`. [#77231](https://github.com/StarRocks/starrocks/pull/77231) [#77680](https://github.com/StarRocks/starrocks/pull/77680) [#77585](https://github.com/StarRocks/starrocks/pull/77585) **Data Lake connectors (Iceberg, Paimon, Delta, Hive)** * Fixed Iceberg correctness and stability: enforce NOT NULL columns on the write path, validate rewrite commits from the plan-time snapshot, invalidate the metadata cache after `TRUNCATE`, drop a table whose metadata file is missing, refresh the native table before cast-on-string-partition metadata delete, silent data loss in `rewrite_data_files` with a `WHERE` clause, and a too-small retention window in `remove_orphan_files`. [#77589](https://github.com/StarRocks/starrocks/pull/77589) [#77709](https://github.com/StarRocks/starrocks/pull/77709) [#77385](https://github.com/StarRocks/starrocks/pull/77385) [#77786](https://github.com/StarRocks/starrocks/pull/77786) [#77335](https://github.com/StarRocks/starrocks/pull/77335) [#77797](https://github.com/StarRocks/starrocks/pull/77797) [#77394](https://github.com/StarRocks/starrocks/pull/77394) * Isolated Delta Lake per-table cloud credentials. [#77424](https://github.com/StarRocks/starrocks/pull/77424) * Supports OpenX JsonSerDe column-name mapping for Hive external tables, one JSON field mapping to multiple Hive columns, and fixed table qualification in connector views for absent default-catalog and CTE references. [#65515](https://github.com/StarRocks/starrocks/pull/65515) [#77716](https://github.com/StarRocks/starrocks/pull/77716) [#77887](https://github.com/StarRocks/starrocks/pull/77887) * Fixed Ranger view/MV authorization for external tables and restored the Ranger Solr audit destination on the FE classpath. [#56316](https://github.com/StarRocks/starrocks/pull/56316) [#77294](https://github.com/StarRocks/starrocks/pull/77294) * Fixed the corrupt `DELTA_BINARY_PACKED` header being returned instead of thrown, and marking filter-excluded FLBA dictionary rows NULL in the Parquet reader. [#77446](https://github.com/StarRocks/starrocks/pull/77446) [#78685](https://github.com/StarRocks/starrocks/pull/78685) * Fixed an NPE on `ALTER TABLE RENAME` for Iceberg REST catalogs, an NPE reading Iceberg `$partitions` with decimal or null `timestamptz` partition values, sharing one Hadoop `FileSystem` across Iceberg catalogs, and honoring `paimon.option.cache-enabled=false` again. [#76498](https://github.com/StarRocks/starrocks/pull/76498) [#77742](https://github.com/StarRocks/starrocks/pull/77742) [#77297](https://github.com/StarRocks/starrocks/pull/77297) [#78271](https://github.com/StarRocks/starrocks/pull/78271) **Shared-data: compaction, vacuum, reshard, and replication** * Compaction fixes: cache the compaction txn log that file bundling skips, sort candidates by max tablet score, fix read-chunk-size estimation, fix the Limiter resize losing tokens on shrink, and skip the publish-time segment rewrite for a delete-only write. [#77578](https://github.com/StarRocks/starrocks/pull/77578) [#77482](https://github.com/StarRocks/starrocks/pull/77482) [#78192](https://github.com/StarRocks/starrocks/pull/78192) [#78087](https://github.com/StarRocks/starrocks/pull/78087) [#78071](https://github.com/StarRocks/starrocks/pull/78071) * Vacuum fixes: fixed wasted reads and stray v1 deletes in grace-blocked vacuum, kept the autovacuum retain floor at or above a reshard takeover, and avoided redundant lake metadata 404 probes. [#78460](https://github.com/StarRocks/starrocks/pull/78460) [#78545](https://github.com/StarRocks/starrocks/pull/78545) [#78466](https://github.com/StarRocks/starrocks/pull/78466) * Reshard/split fixes: reject a reshard whose source tablet lives in a superseded materialized index, fix tablet split fallback cross-publish, silence expected reshard history lookup warnings, and prevent lake alter jobs from getting stuck after failover. [#77512](https://github.com/StarRocks/starrocks/pull/77512) [#78270](https://github.com/StarRocks/starrocks/pull/78270) [#78059](https://github.com/StarRocks/starrocks/pull/78059) [#78517](https://github.com/StarRocks/starrocks/pull/78517) * Replication fixes: preserve shared lake files, read remote bundled lake metadata, and re-encrypt source lake files during replication. [#77361](https://github.com/StarRocks/starrocks/pull/77361) [#77362](https://github.com/StarRocks/starrocks/pull/77362) [#78080](https://github.com/StarRocks/starrocks/pull/78080) * Recover corrupted local caches: drop and retry on checksum failure for delvec pages, and rebuild on corruption for PK-index SST compaction and tablet-merge SSTable rebuild. [#78086](https://github.com/StarRocks/starrocks/pull/78086) [#77481](https://github.com/StarRocks/starrocks/pull/77481) [#78003](https://github.com/StarRocks/starrocks/pull/78003) * Fixed reading lake version-1 metadata from the partition-shared object, protecting snapshot-owned shared shard groups, and preserving dependency ordering in `TransactionGraph.remove`. [#78731](https://github.com/StarRocks/starrocks/pull/78731) [#78800](https://github.com/StarRocks/starrocks/pull/78800) [#78077](https://github.com/StarRocks/starrocks/pull/78077) * Additional shared-data fixes: throttle transactions at final commit, do not pick a `DECOMMISSION` replica as the load primary, fix DML with shadow generated columns during online optimize, make `spill_enable_direct_io` actually engage `O_DIRECT`, fix a label-uniqueness race between concurrent `BEGIN WITH LABEL`, derive the query tracker reserve limit from the effective query memory limit, safely remap rowset IDs and references during tablet merge, apply a cross-published del file only to the keys the child owns, make tablet merge honor range-distribution colocate, refuse an incomplete combined txn log and stop `replication_num` from weakening the shared-data write quorum, keep explicit transactions on one tablet layout during reshard, drop a del file's local cache and retry on checksum failure, and report the abort reason when a closed lake delta writer rejects a task. [#77015](https://github.com/StarRocks/starrocks/pull/77015) [#77035](https://github.com/StarRocks/starrocks/pull/77035) [#77252](https://github.com/StarRocks/starrocks/pull/77252) [#77255](https://github.com/StarRocks/starrocks/pull/77255) [#77401](https://github.com/StarRocks/starrocks/pull/77401) [#77408](https://github.com/StarRocks/starrocks/pull/77408) [#77690](https://github.com/StarRocks/starrocks/pull/77690) [#77744](https://github.com/StarRocks/starrocks/pull/77744) [#78113](https://github.com/StarRocks/starrocks/pull/78113) [#78162](https://github.com/StarRocks/starrocks/pull/78162) [#78625](https://github.com/StarRocks/starrocks/pull/78625) [#78022](https://github.com/StarRocks/starrocks/pull/78022) [#78632](https://github.com/StarRocks/starrocks/pull/78632) * Fixed shared-data sort-key and rowset-identity edge cases: project split boundaries onto the current sort key after a trailing key-column add, surface internal `INSERT` failures in recursive CTE execution, do not let an apportioned row count decide whether a cross-published rowset exists, make `encode_sort_key` reachable for narrow fixed-length IVM row ids and expand its constant argument, resolve the delta column group of an extended column through its root column, conserve range-split cross-publish statistics, fix FE max tablet compaction score collection, skip dead nodes when dropping the auto-increment map, measure the unsorted PK SST writer's map instead of counting allocations, propagate `compression_level` to synthetic sub-column metas, and skip standalone GIN index directory handling for the builtin inverted-index backend. [#77504](https://github.com/StarRocks/starrocks/pull/77504) [#77510](https://github.com/StarRocks/starrocks/pull/77510) [#77511](https://github.com/StarRocks/starrocks/pull/77511) [#77751](https://github.com/StarRocks/starrocks/pull/77751) [#77847](https://github.com/StarRocks/starrocks/pull/77847) [#77759](https://github.com/StarRocks/starrocks/pull/77759) [#78269](https://github.com/StarRocks/starrocks/pull/78269) [#78496](https://github.com/StarRocks/starrocks/pull/78496) [#78561](https://github.com/StarRocks/starrocks/pull/78561) [#77498](https://github.com/StarRocks/starrocks/pull/77498) [#76949](https://github.com/StarRocks/starrocks/pull/76949) [#77101](https://github.com/StarRocks/starrocks/pull/77101) **Loading, transactions, and DML** * Fixed bounding and checking chunk capacity on the data-load and column-mode partial-update paths, deciding condition update and partial update per owning child, and recording a merged rowset's del `op_offset` per statement. [#77163](https://github.com/StarRocks/starrocks/pull/77163) [#78099](https://github.com/StarRocks/starrocks/pull/78099) [#78313](https://github.com/StarRocks/starrocks/pull/78313) * Fixed not journaling an insert-overwrite failure for a dropped table, and keeping the current row lifecycle correct across several partial-update paths. [#77212](https://github.com/StarRocks/starrocks/pull/77212) * Fixed handing the disconnect kill to a worker instead of the I/O thread, and propagating `accept()` status in `GroupedExecutionSinkOperator::push_chunk`. [#77590](https://github.com/StarRocks/starrocks/pull/77590) [#77737](https://github.com/StarRocks/starrocks/pull/77737) * Fail an internal DML/CONTROL statement that only records an error state, so the failure surfaces instead of being silently swallowed. [#77388](https://github.com/StarRocks/starrocks/pull/77388) **Statistics, metrics, and FE/metadata** * Fixed statistics correctness: reject null buckets/MCV in histograms, and make `CancelableAnalyzeTask.cancel` pick a single winner. [#77024](https://github.com/StarRocks/starrocks/pull/77024) [#78593](https://github.com/StarRocks/starrocks/pull/78593) * Registered the `pipe_driver_queue_len` metric, added a BE metric for NotFound lake tablet metadata reads, skipped table metrics for non-table StarOS shards and checkpoint thread-pool metrics, and fixed the misspelled `ZoneMapIndexFilter` key in the profile dictionary. [#77890](https://github.com/StarRocks/starrocks/pull/77890) [#78451](https://github.com/StarRocks/starrocks/pull/78451) [#78116](https://github.com/StarRocks/starrocks/pull/78116) [#77367](https://github.com/StarRocks/starrocks/pull/77367) [#78728](https://github.com/StarRocks/starrocks/pull/78728) * FE/metadata fixes: ignore recycled missing shard groups in meta sync, restore multiple connection groups per endpoint in bRPC, resolve compute-node hostnames in FE before shipping them to BE, make dropped FE exit itself on `DROP FOLLOWER`/`OBSERVER`, and expose colocate group scheduling sample knobs. [#76604](https://github.com/StarRocks/starrocks/pull/76604) [#76765](https://github.com/StarRocks/starrocks/pull/76765) [#78390](https://github.com/StarRocks/starrocks/pull/78390) [#78215](https://github.com/StarRocks/starrocks/pull/78215) [#78450](https://github.com/StarRocks/starrocks/pull/78450) * Fixed skipping non-extended access paths when extending meta-scan tablets, matching a scan's access paths by column id, narrowing an OLAP scan's complex type only when the BE prunes it, and not pruning subfields of aggregate-state columns. [#76593](https://github.com/StarRocks/starrocks/pull/76593) [#77987](https://github.com/StarRocks/starrocks/pull/77987) [#77894](https://github.com/StarRocks/starrocks/pull/77894) [#77901](https://github.com/StarRocks/starrocks/pull/77901) * Do not use a global dictionary for aggregate-state columns or dictionarize expressions returning `ARRAY`/`STRUCT`, and index page null flags by ordinal when a pushed-down predicate reads a sparse range. [#77096](https://github.com/StarRocks/starrocks/pull/77096) [#78488](https://github.com/StarRocks/starrocks/pull/78488) * Additional metrics and FE fixes: report bRPC method error bvars as counters, report the real tablet count as Buckets for range-distribution partitions, fix an NPE in `show proc '/global_current_queries'`, report data cache read stats in intermediate query statistics, use a minimal TSD fetch in jemalloc `malloc_usable_size`, stop the heap-profile toggle from writing `prof.thread_active_init`, prevent FE connection ID collisions after wrap, fix table stats health pinned at ~1.0 when many partitions are stale, use `feature_log_dir` for FE features-log paths, preserve `inputIsBinary` when copying logical window operators, and fix a Thrift client leak after cache invalidation. [#76956](https://github.com/StarRocks/starrocks/pull/76956) [#77465](https://github.com/StarRocks/starrocks/pull/77465) [#77407](https://github.com/StarRocks/starrocks/pull/77407) [#77696](https://github.com/StarRocks/starrocks/pull/77696) [#77625](https://github.com/StarRocks/starrocks/pull/77625) [#78384](https://github.com/StarRocks/starrocks/pull/78384) [#78217](https://github.com/StarRocks/starrocks/pull/78217) [#76927](https://github.com/StarRocks/starrocks/pull/76927) [#77221](https://github.com/StarRocks/starrocks/pull/77221) [#77058](https://github.com/StarRocks/starrocks/pull/77058) [#77072](https://github.com/StarRocks/starrocks/pull/77072) **Other** * Applied request timeouts to the pooled Poco session, upgraded Hyperscan to Vectorscan 5.4.12 on Linux AArch64, escalated the `JAVA_OPTS_FOR_JDK_*` warning, and stopped shipping test-scope jars in the java-extensions reader libs. [#78361](https://github.com/StarRocks/starrocks/pull/78361) [#78326](https://github.com/StarRocks/starrocks/pull/78326) [#78328](https://github.com/StarRocks/starrocks/pull/78328) [#77752](https://github.com/StarRocks/starrocks/pull/77752) * Serialize `dictionary_get` keys by key size, catch exceptions in the thrift processor, and keep `files()` schema inference off the planner meta lock for CTAS. [#78706](https://github.com/StarRocks/starrocks/pull/78706) [#78805](https://github.com/StarRocks/starrocks/pull/78805) [#78772](https://github.com/StarRocks/starrocks/pull/78772) * Read the last conf line when the file has no trailing newline, resolve the config YAML path against the cwd, name the GCS upload-threshold BE config `gcs` (not `gs`), upgrade StarOS and Starlet to 4.1.3, and removed an obsolete config forward-header split and a stray U+200E in doc filenames. [#78065](https://github.com/StarRocks/starrocks/pull/78065) [#77132](https://github.com/StarRocks/starrocks/pull/77132) [#78452](https://github.com/StarRocks/starrocks/pull/78452) [#78038](https://github.com/StarRocks/starrocks/pull/78038) [#78780](https://github.com/StarRocks/starrocks/pull/78780) [#78612](https://github.com/StarRocks/starrocks/pull/78612) #### 4.1.4[​](#414 "Direct link to 4.1.4") Release Date: August 5, 2026 ##### Behavior Changes[​](#behavior-changes-1 "Direct link to Behavior Changes") * In shared-data mode, the `TABLESAMPLE` / `SAMPLE` clause and `ANALYZE SAMPLE TABLE` now take effect on lake tables. Previously the sample options never reached the BE, so a full scan was performed. [#71874](https://github.com/StarRocks/starrocks/pull/71874) * `ALTER TABLE` changes to the `flat_json` configuration are now propagated to BE nodes through a versioned task so they take effect reliably. [#74747](https://github.com/StarRocks/starrocks/pull/74747) * `NOT MATCH` predicates answered by a GIN inverted index no longer return NULL rows; NULL rows are now correctly excluded from the result. [#75578](https://github.com/StarRocks/starrocks/pull/75578) * `ANALYZE ... UPDATE HISTOGRAM ON` no longer computes histograms for char-family columns (which the optimizer does not use) and computes only MCVs, reducing analyze cost. [#75968](https://github.com/StarRocks/starrocks/pull/75968) * `rewrite_manifests` now clusters output manifests by order-preserving partition ranges, producing tighter per-manifest partition bounds and better manifest pruning on tables with many partitions. [#76193](https://github.com/StarRocks/starrocks/pull/76193) * Multi-table stream load now honors each table's `file_bundling` property when deciding whether to use a combined transaction log, instead of following only the global `lake_use_combined_txn_log` config. [#76806](https://github.com/StarRocks/starrocks/pull/76806) * A GIN inverted index combined with column-mode partial update on a Primary Key table no longer returns corrupted results; the index is now served from the delta column group segment. [#76271](https://github.com/StarRocks/starrocks/pull/76271) * `array_difference` on integer arrays now computes adjacent differences in 64-bit before widening to `BIGINT`, fixing int32 overflow. [#76569](https://github.com/StarRocks/starrocks/pull/76569) * Division expressions with a non-constant divisor are no longer treated as monotonic, fixing incorrect ZoneMap pruning that could produce wrong results. [#76744](https://github.com/StarRocks/starrocks/pull/76744) * Array and map constructors now raise an error instead of silently corrupting data when the flattened result of a chunk exceeds 4 GB. [#76419](https://github.com/StarRocks/starrocks/pull/76419) * Primary Key inserts now correctly reject keys that already exist in the L1/L2 persistent index; the existence check was previously skipped. [#76591](https://github.com/StarRocks/starrocks/pull/76591) * Sample-based tablet pre-split on range-colocate tables now spreads new shards across compute nodes instead of packing them onto the source tablet's worker, fixing range batch imports that were roughly 3x slower than hash. [#76608](https://github.com/StarRocks/starrocks/pull/76608) * Incremental-or-AUTO materialized views now rebuild the maintenance query for the PCT fallback in `AUTO` refresh mode, fixing refresh failures for shapes that IVM does not support. [#75961](https://github.com/StarRocks/starrocks/pull/75961) * `ALTER MATERIALIZED VIEW ... ACTIVE` now works for non-aggregate INCREMENTAL materialized views; the storage-filled `__ROW_ID__` column is kept out of the MV DDL column list so an inactive IVM can be re-activated. [#77017](https://github.com/StarRocks/starrocks/pull/77017) * The Iceberg partition cache is now bounded by memory (not just entry count) and its usage is exposed, preventing unbounded growth on tables with many partitions. [#76165](https://github.com/StarRocks/starrocks/pull/76165) * Iceberg REST catalogs using OAuth2 client credentials now self-heal after the background token-refresh task dies, instead of failing all subsequent requests. [#76457](https://github.com/StarRocks/starrocks/pull/76457) * Lake full vacuum now reclaims orphaned `.lcrm` (Lake Compaction Rows Mapper) files that were previously skipped. [#76522](https://github.com/StarRocks/starrocks/pull/76522) * Shared-data publish now refuses to write a bundle tablet metadata file that is missing tablets, preventing a partition from getting permanently stuck on publish. [#76850](https://github.com/StarRocks/starrocks/pull/76850) * Abandoned external (Spark/Flink connector) scan contexts now cancel their pipeline fragments when reaped, and external scan plans set `query_delivery_timeout` to bound the QueryContext lifetime. [#76535](https://github.com/StarRocks/starrocks/pull/76535) [#76536](https://github.com/StarRocks/starrocks/pull/76536) * Materialized view pinned-range maps are now keyed correctly (by table UUID), fixing MV bootstrap pinning being disabled for OLAP tables and mishandling of same-named tables in different databases. [#76320](https://github.com/StarRocks/starrocks/pull/76320) [#76351](https://github.com/StarRocks/starrocks/pull/76351) * Audit logs for statements forwarded to the leader FE now record fully-qualified queried relations (matching the leader) instead of CTE aliases and unqualified table names. [#76387](https://github.com/StarRocks/starrocks/pull/76387) * Aggregating a full `STRUCT` column together with `ROLLUP`, `CUBE`, or `GROUPING SETS` no longer fails at plan time with a `usedStructFiledPos` error. [#76804](https://github.com/StarRocks/starrocks/pull/76804) * The FE memory `Estimator` now counts container overhead, and the Parquet scanner now pads missing columns using the bounded chunk size instead of the full batch size, improving memory accounting accuracy. [#75971](https://github.com/StarRocks/starrocks/pull/75971) [#75981](https://github.com/StarRocks/starrocks/pull/75981) ##### Improvements[​](#improvements-2 "Direct link to Improvements") * Added support for an `INCLUDE METADATA` clause in Routine Load to expose Kafka/Pulsar message metadata (such as partition, offset, and timestamp). [#73840](https://github.com/StarRocks/starrocks/pull/73840) * Made the Routine Load metadata alias optional. [#76294](https://github.com/StarRocks/starrocks/pull/76294) * Added a `LAST_FRESHNESS_CONFIRMED_AT` column to `information_schema.materialized_views`. [#74585](https://github.com/StarRocks/starrocks/pull/74585) * Exposed data cache metrics through the `/metrics` endpoint, and added FE compaction metrics, `ALTER TABLE` column-operation metrics and durations, and a `CompressedInputBytes` metric on the exchange sink. [#58204](https://github.com/StarRocks/starrocks/pull/58204) [#72941](https://github.com/StarRocks/starrocks/pull/72941) [#76247](https://github.com/StarRocks/starrocks/pull/76247) [#76309](https://github.com/StarRocks/starrocks/pull/76309) * Exposed authentication and connectivity error details from connector metadata operations to end users. [#75490](https://github.com/StarRocks/starrocks/pull/75490) * Improved error messages for large-column capacity limit checks. [#76303](https://github.com/StarRocks/starrocks/pull/76303) * Added the `hdfs_backend_selector_cache_replica_num` variable, and made the object storage client cache size runtime mutable. [#75023](https://github.com/StarRocks/starrocks/pull/75023) [#75851](https://github.com/StarRocks/starrocks/pull/75851) * Extended sample-based tablet pre-split meta-tier readers to more sort-key types, including `CHAR`, composite sort keys, and UTC-adjusted Parquet `TIMESTAMP` / ORC `TIMESTAMP_INSTANT`. [#75937](https://github.com/StarRocks/starrocks/pull/75937) [#76011](https://github.com/StarRocks/starrocks/pull/76011) [#76114](https://github.com/StarRocks/starrocks/pull/76114) * Improved external-table statistics collection: predicate column usage is now tracked, multiple columns are collected in a single scan, and a bounded-cost scan budget is applied for Iceberg tables. [#75938](https://github.com/StarRocks/starrocks/pull/75938) [#76638](https://github.com/StarRocks/starrocks/pull/76638) [#76549](https://github.com/StarRocks/starrocks/pull/76549) * Supports right-outer, semi, anti, and full-outer range-colocate joins. [#76040](https://github.com/StarRocks/starrocks/pull/76040) * Optimized Query Queue V2 cost estimators. [#76609](https://github.com/StarRocks/starrocks/pull/76609) * Pushed the wildcard literal prefix down to the S3 `ListObjectsV2` call for `FILES()` globs, reducing objects listed. [#76210](https://github.com/StarRocks/starrocks/pull/76210) * When scanning Iceberg manifest entries in `remove_orphan_files`, only the `file_path` column is now projected. [#76020](https://github.com/StarRocks/starrocks/pull/76020) * Supports base compaction for lake Primary Key tablets, and load spill plus eager PK-index SST for separate-sort-key cloud-native PK tables. [#76794](https://github.com/StarRocks/starrocks/pull/76794) [#76094](https://github.com/StarRocks/starrocks/pull/76094) * Supports a metadata-only add of a trailing sort-key column on shared-data range-distribution tables. [#76341](https://github.com/StarRocks/starrocks/pull/76341) * Cancels in-flight exchange sink RPCs on fragment cancellation, and cancels in-flight compactions on resharded partitions during reshard cleaning. [#75613](https://github.com/StarRocks/starrocks/pull/75613) [#76759](https://github.com/StarRocks/starrocks/pull/76759) * Fires multi-statement stream load channels before waiting, reducing load latency. [#76715](https://github.com/StarRocks/starrocks/pull/76715) * Added a memory-limit check for column upgrades in `Analytor::process`. [#75821](https://github.com/StarRocks/starrocks/pull/75821) * Preserves the real error status when an async delta writer is stopped. [#76216](https://github.com/StarRocks/starrocks/pull/76216) * Records generation version on lake PK-index SSTables, applies `op_write.seg_delvecs` at lake Primary Key publish, and skips opening output segment footers during compaction publish conflict resolution. [#76208](https://github.com/StarRocks/starrocks/pull/76208) [#76474](https://github.com/StarRocks/starrocks/pull/76474) [#76657](https://github.com/StarRocks/starrocks/pull/76657) * Added stack-trace and fine-grained trace counters for lake load and publish-version paths to aid diagnosis. [#75901](https://github.com/StarRocks/starrocks/pull/75901) [#76810](https://github.com/StarRocks/starrocks/pull/76810) ##### Security[​](#security-2 "Direct link to Security") * \[CVE-2026-44891] Bumped Netty to 4.1.136.Final to fix a memory-exhaustion (DoS) vulnerability in the STOMP subframe decoder. [#76555](https://github.com/StarRocks/starrocks/pull/76555) * \[CVE-2026-55971] \[CVE-2026-43871] Bumped Apache Thrift to 0.24.0 to fix a heap-based buffer overflow in the C++ bindings and an infinite-loop vulnerability. [#76922](https://github.com/StarRocks/starrocks/pull/76922) * \[CVE-2026-10050] Excluded vulnerable Jetty jars (client-side Digest auth bypass, pulled in transitively via Hadoop) and bumped pgjdbc to 42.7.12. [#76783](https://github.com/StarRocks/starrocks/pull/76783) * \[CVE-2011-4969] \[CVE-2014-6071] Excluded the unused `avro-ipc` jar that bundled a vulnerable jQuery 1.4.2 (and other jQuery XSS CVEs). [#76270](https://github.com/StarRocks/starrocks/pull/76270) * \[CVE-2024-29857] Removed vulnerable stale transitive dependencies that shipped alongside their fixed counterparts (for example `bcprov-jdk15on` 1.70 and EOL `okhttp` 2.x) and added dependency bans to prevent regressions. [#76097](https://github.com/StarRocks/starrocks/pull/76097) ##### Bug Fixes[​](#bug-fixes-2 "Direct link to Bug Fixes") The following issues have been fixed: * Planner crash on an empty range in join predicate derivation. [#75011](https://github.com/StarRocks/starrocks/pull/75011) * Aggregate incorrectly pushed down over a `CASE` with a non-null constant `ELSE`. [#75037](https://github.com/StarRocks/starrocks/pull/75037) * `PARTITION-TOP-N` partition-by rewritten to a pruned dictionary slot. [#75956](https://github.com/StarRocks/starrocks/pull/75956) * Wrong operator precedence in the join shuffle-join output-property branch, and `predicateCommonOperators` not carried through join operator construction. [#76203](https://github.com/StarRocks/starrocks/pull/76203) [#76330](https://github.com/StarRocks/starrocks/pull/76330) [#76388](https://github.com/StarRocks/starrocks/pull/76388) * Incorrect nullability for view columns and for `ROLLUP` keys at analysis time. [#75684](https://github.com/StarRocks/starrocks/pull/75684) [#76149](https://github.com/StarRocks/starrocks/pull/76149) * `regexp_extract_all` infinite loop on a zero-length capture group. [#75798](https://github.com/StarRocks/starrocks/pull/75798) * Wrong `nullsFraction` clamp in `LargeOrCalculatingVisitor`. [#75864](https://github.com/StarRocks/starrocks/pull/75864) * Incorrect `CAST` semantics for string-to-number schema-change conversion. [#75538](https://github.com/StarRocks/starrocks/pull/75538) * Sync MV rewrite losing a rollup column for same-column `min`/`max`. [#75528](https://github.com/StarRocks/starrocks/pull/75528) * Rewrite incorrectly applied inside an `array_map` lambda when pulling scan predicates up. [#76380](https://github.com/StarRocks/starrocks/pull/76380) * `COUNT(DISTINCT)` over-count on Iceberg bucket tables under bucket-aware execution. [#76601](https://github.com/StarRocks/starrocks/pull/76601) * Error on nullable output for `UNNEST` + `GROUP BY` of a required Iceberg column. [#76730](https://github.com/StarRocks/starrocks/pull/76730) * Reduced lock contention across the `INSERT OVERWRITE` code path, and restored `Operator` salt to avoid a Memo self-reference during CBO table prune. [#75828](https://github.com/StarRocks/starrocks/pull/75828) [#76542](https://github.com/StarRocks/starrocks/pull/76542) * Iceberg partition pruning, manifest row-count estimation, metadata delete, equality-delete application, and Delta Lake partition pruning were incorrect when a string date partition column is compared with a temporal value. [#76068](https://github.com/StarRocks/starrocks/pull/76068) [#76107](https://github.com/StarRocks/starrocks/pull/76107) [#76197](https://github.com/StarRocks/starrocks/pull/76197) [#76280](https://github.com/StarRocks/starrocks/pull/76280) [#76348](https://github.com/StarRocks/starrocks/pull/76348) * Iceberg reads did not honor the targeted snapshot's schema and partition spec; query failure on an Iceberg V1 table after dropping a partition field; and MV rewrite serving stale results after `rollback_to_snapshot`. [#74711](https://github.com/StarRocks/starrocks/pull/74711) [#75149](https://github.com/StarRocks/starrocks/pull/75149) [#75924](https://github.com/StarRocks/starrocks/pull/75924) * Iceberg manifest data-file cache served an incomplete file set; the incremental scan-range iterator was unsafe to close concurrently; and analysis-time time-travel snapshot binding is now best-effort. [#76215](https://github.com/StarRocks/starrocks/pull/76215) [#75953](https://github.com/StarRocks/starrocks/pull/75953) [#76448](https://github.com/StarRocks/starrocks/pull/76448) * Iceberg/Delta metadata-derived statistics are now tagged with `StatsSource=TABLE_METADATA`. [#76560](https://github.com/StarRocks/starrocks/pull/76560) * Delta Lake and Kudu un-partitioned materialized view query rewrite. [#76359](https://github.com/StarRocks/starrocks/pull/76359) * GCS vended credentials ignored due to a `gcs-connector` 3.x config rename. [#75979](https://github.com/StarRocks/starrocks/pull/75979) * Hive `getTable()` now reconnects before the `get_table_req` fallback and detects table-not-found by exception type instead of message text. [#76456](https://github.com/StarRocks/starrocks/pull/76456) [#76459](https://github.com/StarRocks/starrocks/pull/76459) * `BOOLEAN` min/max not supported in Parquet column-index statistics. [#74752](https://github.com/StarRocks/starrocks/pull/74752) * Flat-JSON subfield reads for keys whose name contains `.`, and reconstruction of an intermediate flat-JSON object instead of returning NULL; JSON subfield pushdown is now skipped when subfield keys collide case-insensitively. [#75583](https://github.com/StarRocks/starrocks/pull/75583) [#75764](https://github.com/StarRocks/starrocks/pull/75764) [#76594](https://github.com/StarRocks/starrocks/pull/76594) * Kept OR-nested predicates when only one branch is bitmap-indexed. [#76275](https://github.com/StarRocks/starrocks/pull/76275) * BE crash in Primary Key auto-increment partial-update apply. [#76119](https://github.com/StarRocks/starrocks/pull/76119) * BE crash from a `LoadChunkSpiller` init race during load spill. [#76098](https://github.com/StarRocks/starrocks/pull/76098) * Native Parquet reader crash on an incomplete nested lake schema, and an ASAN crash for `BOOLEAN` nested in Avro complex-type columns. [#76455](https://github.com/StarRocks/starrocks/pull/76455) [#76041](https://github.com/StarRocks/starrocks/pull/76041) * `bad_variant_access` during schema change when converting an invalid string to a `NOT NULL` numeric column. [#76707](https://github.com/StarRocks/starrocks/pull/76707) * Heap-buffer-overflow in the `SimdJsonConverter` error path, and a use-after-free on a multi-character CSV delimiter straddling a buffer expansion. [#76752](https://github.com/StarRocks/starrocks/pull/76752) [#76718](https://github.com/StarRocks/starrocks/pull/76718) * Use-after-free on `MorselQueueFactory` during scan teardown, in the spillable join build `set_finishing` at cancel time, and a global runtime-filter timer leak / unscheduled timer in the `PipelineDriver` destructor. [#76259](https://github.com/StarRocks/starrocks/pull/76259) [#76633](https://github.com/StarRocks/starrocks/pull/76633) [#76252](https://github.com/StarRocks/starrocks/pull/76252) * Null-safe `SparseRangeIterator::has_more()` to fix a physical-split empty-tablet CN crash, and a bRPC stub cache clean timer leak. [#75985](https://github.com/StarRocks/starrocks/pull/75985) [#75973](https://github.com/StarRocks/starrocks/pull/75973) * `ConnectContext` not restored on query-deploy workers. [#76366](https://github.com/StarRocks/starrocks/pull/76366) * Preserve upsert/delete order within a transaction for shared-data Primary Key tables, including making the spill merge op-aware. [#75338](https://github.com/StarRocks/starrocks/pull/75338) [#75366](https://github.com/StarRocks/starrocks/pull/75366) * Read durable metadata in `cal_new_base_version` to avoid a dangling `prev_garbage_version`; keep `base_version` in sync with `base_metadata` in `publish_version`; and persist the merged parallel-compaction transaction log on the regular path. [#75904](https://github.com/StarRocks/starrocks/pull/75904) [#76313](https://github.com/StarRocks/starrocks/pull/76313) [#76460](https://github.com/StarRocks/starrocks/pull/76460) * Range-colocate fixes: stopped an alignment job storm and fail-close unaligned colocate joins, fixed null-safe joins dropping matches under bucket-shuffle, and improved range-colocate join detection in plan feedback. [#75930](https://github.com/StarRocks/starrocks/pull/75930) [#76104](https://github.com/StarRocks/starrocks/pull/76104) [#76121](https://github.com/StarRocks/starrocks/pull/76121) * Tablet split/reshard fixes: bump the table optimistic version on split so concurrent queries re-plan, flush the PK-index memtable before identical-tablet reshard, retain lake vacuum files by version interval across reshard, stop the auto-split job loop on un-splittable range-distribution tablets, and fix `IS NULL` pruning on range-distributed sort keys after a split. [#76123](https://github.com/StarRocks/starrocks/pull/76123) [#76367](https://github.com/StarRocks/starrocks/pull/76367) [#76209](https://github.com/StarRocks/starrocks/pull/76209) [#76663](https://github.com/StarRocks/starrocks/pull/76663) [#76797](https://github.com/StarRocks/starrocks/pull/76797) * Reject a full-column `ORDER BY` that reorders the key columns of a key-derived range-distribution table. [#76256](https://github.com/StarRocks/starrocks/pull/76256) * Consolidate load-spill parallel merge results in flush order, and exclude touched indexes by `metaId` in file-bundling carry-forward. [#75951](https://github.com/StarRocks/starrocks/pull/75951) [#76368](https://github.com/StarRocks/starrocks/pull/76368) * Reserve dictionary size + 1 for NULL-absorbing dict-mapping group-by keys, and check driver readiness before a profile-report reschedule. [#75357](https://github.com/StarRocks/starrocks/pull/75357) [#75725](https://github.com/StarRocks/starrocks/pull/75725) * `PlannerMetaLocker` skipped unlock when the lock never succeeded. [#74041](https://github.com/StarRocks/starrocks/pull/74041) * Filter resource groups by warehouse. [#73209](https://github.com/StarRocks/starrocks/pull/73209) * Escape the `jsonpaths` value in `SHOW CREATE ROUTINE LOAD` output. [#75755](https://github.com/StarRocks/starrocks/pull/75755) * Fixed Arrow Flight prepared-statement forwarding. [#76310](https://github.com/StarRocks/starrocks/pull/76310) * Fixed dictionary refresh-interval overflow and unintended auto refresh. [#76634](https://github.com/StarRocks/starrocks/pull/76634) * Fixed a statistics cache load failure on empty min/max under `ERROR_IF_OVERFLOW`. [#76684](https://github.com/StarRocks/starrocks/pull/76684) * Made the catalog-drop existence check atomic under a write lock. [#76778](https://github.com/StarRocks/starrocks/pull/76778) #### 4.1.3[​](#413 "Direct link to 4.1.3") Release Date: July 14, 2026 ##### Behavior Changes[​](#behavior-changes-2 "Direct link to Behavior Changes") * `CTAS` now preserves explicitly declared `VARCHAR(N)` column lengths instead of widening them to `VARCHAR(MAX)`. Existing tables are unaffected; new tables created with `CTAS` will enforce the declared length on subsequent writes. [#73498](https://github.com/StarRocks/starrocks/pull/73498) * Querying `sys.fe_memory_usage` or `sys.fe_locks` without the `OPERATE ON SYSTEM` privilege now returns a clear access-denied error instead of a misleading node-lookup failure. [#73567](https://github.com/StarRocks/starrocks/pull/73567) * `FILES()` and broker/stream load no longer apply a session timezone shift to `INT64` Parquet timestamps written with `isAdjustedToUTC=false`; those timestamps are now treated as wall-clock values and loaded as-is. Data loaded from such files before v4.1.3 may differ from data loaded after upgrading; reload if consistency is required. [#73674](https://github.com/StarRocks/starrocks/pull/73674) * Successfully committed multi-table transaction stream load jobs are now correctly shown as `VISIBLE` in `information_schema.loads` and `SHOW STREAM LOAD` instead of remaining stuck at `PREPARING`. [#74386](https://github.com/StarRocks/starrocks/pull/74386) * Connector incremental scan range scheduling now consistently reuses the deployed fragment's driver layout, preventing scan ranges from being incorrectly assigned to non-existent drivers. [#74674](https://github.com/StarRocks/starrocks/pull/74674) * `LIKE` constant-folding now matches MySQL 8 backslash-escape semantics, correcting cases where patterns such as `'a\\\\b'` previously returned opposite results. [#74814](https://github.com/StarRocks/starrocks/pull/74814) * Routine Load now supports the `property.kafka_partition_discovery` property, which allows partition auto-discovery to continue even when `kafka_partitions` and `kafka_offsets` are specified to seed exact starting offsets. When `property.kafka_default_offsets` is not set, the default starting offset for partitions discovered after the job already has consuming progress changes from `OFFSET_END` to `OFFSET_BEGINNING` — and this applies to **all** auto-discovery jobs, not only those using the new property. [#74729](https://github.com/StarRocks/starrocks/pull/74729) * Non-group-by aggregates are now pushed down through `UNION ALL` branches before merging, reducing network transfer and memory usage for queries that aggregate over a union. [#73930](https://github.com/StarRocks/starrocks/pull/73930) * IVM maintenance queries are now re-derived from the current view definition at each refresh instead of using the frozen query text stored at `CREATE` time; existing MVs automatically benefit from rewriter bug fixes without needing to be recreated. [#74881](https://github.com/StarRocks/starrocks/pull/74881) * Sample-based tablet pre-split now spreads pre-split shards across all compute nodes (`SPREAD` placement) instead of packing them onto the source tablet's worker (`PACK` placement), improving load parallelism. [#75514](https://github.com/StarRocks/starrocks/pull/75514) * `ALTER TABLE ... MODIFY COLUMN` that changes only the column comment now takes the lightweight metadata-only path instead of spawning a full schema-change job, and this now works on Primary Key columns as well. [#75325](https://github.com/StarRocks/starrocks/pull/75325) * `FLOOR` and `CEIL` are now treated as non-reserved keywords and can be used as column names without quoting. [#75241](https://github.com/StarRocks/starrocks/pull/75241) * `SHOW FUNCTIONS` output now always includes the `isolation` property (`shared` or `isolated`) in the Properties column for UDFs and UDAFs. [#75255](https://github.com/StarRocks/starrocks/pull/75255) * The default value of `lake_vacuum_min_batch_delete_size` is raised from 100 to 200, improving vacuum throughput on object storage by batching more stale-file deletions per `DeleteObjects` request. [#74304](https://github.com/StarRocks/starrocks/pull/74304) * Iceberg REST catalog tables with vended credentials are now cached and their credentials are refreshed in the background, eliminating per-`getTable()` `GetDataAccess` calls that caused AWS Lake Formation rate limiting. [#75431](https://github.com/StarRocks/starrocks/pull/75431) * IVM `bitmap_union`, `hll_union`, and `percentile_union` aggregate states are now stored once in the materialized view instead of twice (visible column + hidden `__AGG_STATE_` column), halving storage for those sketch types. [#75760](https://github.com/StarRocks/starrocks/pull/75760) * Incremental materialized views now support `bitmap_agg`, `hll_union`, `percentile_union`, and `bitmap_union` aggregate functions, enabling exact distinct-count and sketch-based aggregations to be maintained incrementally. [#75587](https://github.com/StarRocks/starrocks/pull/75587) [#75610](https://github.com/StarRocks/starrocks/pull/75610) * Sample-based tablet pre-split tablet count is now rounded up to the nearest multiple of the active compute-node count for even distribution, and bounded below by a minimum tablet size to avoid excessive fragmentation on small loads. [#75360](https://github.com/StarRocks/starrocks/pull/75360) [#75584](https://github.com/StarRocks/starrocks/pull/75584) ##### Improvements[​](#improvements-3 "Direct link to Improvements") * The `ngram_search` function now accepts a non-constant needle argument. [#74675](https://github.com/StarRocks/starrocks/pull/74675) * Added an HTTP authentication framework controlled by the `enable_http_auth` FE configuration, gating authentication and RBAC enforcement on all external HTTP endpoints. [#73822](https://github.com/StarRocks/starrocks/pull/73822) * Added refresh and placement observability columns (`refresh_warehouse`, `refresh_resource_group`, `refresh_mode`, `refresh_type`, `last_refresh_details`) to `information_schema.materialized_views`. [#74342](https://github.com/StarRocks/starrocks/pull/74342) * Added opt-in lazy refresh of the external statistics cache on journal replay, controlled by a new FE config, to prevent a slow or stuck external metastore from stalling FE journal replay or startup. [#74371](https://github.com/StarRocks/starrocks/pull/74371) * `VARCHAR` length increase is now allowed on range-distribution (shared-data) sort-key columns via fast schema evolution without a data rewrite. [#74698](https://github.com/StarRocks/starrocks/pull/74698) * Added a stack-trace dump when a shared-data transaction log write exceeds a configurable threshold, making slow `put_txn_log` / `put_combined_txn_log` calls easier to diagnose. [#74704](https://github.com/StarRocks/starrocks/pull/74704) * Tablet pre-split meta-tier footer readers now support `DATE`, `DATETIME`, `DECIMAL`, `VARCHAR`, and ORC `TIMESTAMP` sort keys, reducing the number of loads that must fall back to data-tier sampling. [#74710](https://github.com/StarRocks/starrocks/pull/74710) [#74739](https://github.com/StarRocks/starrocks/pull/74739) [#74792](https://github.com/StarRocks/starrocks/pull/74792) [#74902](https://github.com/StarRocks/starrocks/pull/74902) [#74955](https://github.com/StarRocks/starrocks/pull/74955) [#75186](https://github.com/StarRocks/starrocks/pull/75186) [#75209](https://github.com/StarRocks/starrocks/pull/75209) [#75427](https://github.com/StarRocks/starrocks/pull/75427) [#75697](https://github.com/StarRocks/starrocks/pull/75697) * Sample-based tablet pre-split now applies to `INSERT INTO ... SELECT ... FROM ` loads, and also to column-list `INSERT` statements that include all sort-key columns. [#74828](https://github.com/StarRocks/starrocks/pull/74828) [#75345](https://github.com/StarRocks/starrocks/pull/75345) * Added Adler-32 checksum protection for shared-data tablet metadata and transaction log files, enabling silent corruption to be detected on read. [#74924](https://github.com/StarRocks/starrocks/pull/74924) * Added the `txn_max_committed_pending_publish_ms` FE metric per database, reporting the age of the oldest committed-but-not-yet-published transaction to help detect stalled version publishing. [#75025](https://github.com/StarRocks/starrocks/pull/75025) * Tablet split/merge is now triggered in real time from the publish-version response, reducing the lag between a load completing and automatic split/merge being initiated. [#75010](https://github.com/StarRocks/starrocks/pull/75010) * Optimized condition-update compare phase for lake primary-key tables by routing no-SST condition-merge tasks to the `pk_index_execution` thread pool. [#74572](https://github.com/StarRocks/starrocks/pull/74572) * Scoped lake schema-change and rollup job locks to the table level instead of the whole database, reducing lock contention on concurrent operations for other tables in the same database. [#75087](https://github.com/StarRocks/starrocks/pull/75087) * Narrowed several database-level write locks to table-scoped intensive write locks in shared-nothing mode, reducing lock contention during BE report callbacks and cooldown operations. [#74521](https://github.com/StarRocks/starrocks/pull/74521) [#74523](https://github.com/StarRocks/starrocks/pull/74523) * Avro Routine Load now supports native `MAP` and `STRUCT` target columns. [#74901](https://github.com/StarRocks/starrocks/pull/74901) * Range-colocate tablet stability gating now waits for StarOS placement convergence before marking a group stable, ensuring colocate joins achieve host-local execution. [#75290](https://github.com/StarRocks/starrocks/pull/75290) [#75656](https://github.com/StarRocks/starrocks/pull/75656) [#75883](https://github.com/StarRocks/starrocks/pull/75883) * Improved CBO statistics for external tables: the optimizer now estimates row counts from Iceberg manifests without full file enumeration, corrects Hive/Hudi row-count underestimation for Parquet/ORC compression, adds async row-count statistics for JDBC connectors, and provides NDV estimation fallback for Iceberg and external connectors when Puffin stats are unavailable. [#75280](https://github.com/StarRocks/starrocks/pull/75280) [#75082](https://github.com/StarRocks/starrocks/pull/75082) [#75083](https://github.com/StarRocks/starrocks/pull/75083) [#75092](https://github.com/StarRocks/starrocks/pull/75092) [#75097](https://github.com/StarRocks/starrocks/pull/75097) [#75382](https://github.com/StarRocks/starrocks/pull/75382) [#75474](https://github.com/StarRocks/starrocks/pull/75474) * Iceberg manifest column statistics are now cached selectively for clustered columns only, reducing FE heap consumption for wide tables with many data files. [#75395](https://github.com/StarRocks/starrocks/pull/75395) * External table statistics collection now supports persistent predicate-column tracking across FE restarts and HA failovers, enabling auto-ANALYZE to target the correct columns. [#75653](https://github.com/StarRocks/starrocks/pull/75653) * Added structured `[ExternalStats]` log lines covering the full lifecycle of external-table statistics collection from scheduling through execution. [#75335](https://github.com/StarRocks/starrocks/pull/75335) [#75529](https://github.com/StarRocks/starrocks/pull/75529) * `SHOW ANALYZE STATUS` now includes partition, column, and snapshot metadata in the Properties column for external table statistics jobs. [#75630](https://github.com/StarRocks/starrocks/pull/75630) * The statistics source (`TABLE_METADATA`, `ANALYZE`, or `NONE`) for each external table is now exposed in the query runtime profile. [#75253](https://github.com/StarRocks/starrocks/pull/75253) * Added support for partition filter requirement and partition count limit for Iceberg and Delta Lake external tables (previously only available for Hive, Hudi, and Paimon). [#75790](https://github.com/StarRocks/starrocks/pull/75790) * Supports sub-1% sampling ratios in `TABLE SAMPLE` and histogram `ANALYZE`, fixing failures on large tables where the computed ratio would truncate to zero. [#74551](https://github.com/StarRocks/starrocks/pull/74551) * Added the `jemalloc_conf` BE configuration item, making jemalloc runtime options visible via `information_schema.be_configs`. [#75344](https://github.com/StarRocks/starrocks/pull/75344) * Added the `compaction_chunk_reset_memory_tracker_threshold_percent` BE configuration to reduce memory usage during Primary Key compaction in shared-nothing mode by releasing retained chunk capacity. [#75091](https://github.com/StarRocks/starrocks/pull/75091) * Upgraded staros to v4.1.1, including persistent `datacache.enable` across restarts, per-worker-group shard warmup timeout override, and improved S3 retry jitter. [#75204](https://github.com/StarRocks/starrocks/pull/75204) * Optimized SQL credential redaction on the audit hot path by skipping the regex scan when no credential marker is present in the SQL string. [#74812](https://github.com/StarRocks/starrocks/pull/74812) * Expression-driven on-demand lazy column loading for the Parquet scanner reduces unnecessary I/O on multi-branch `OR` queries. [#74886](https://github.com/StarRocks/starrocks/pull/74886) * `ds_hll_count_distinct` / `DataSketchesHll` now produces stable cardinality estimates by using the composite estimator instead of the order-dependent HIP estimator. [#75053](https://github.com/StarRocks/starrocks/pull/75053) ##### Security[​](#security-3 "Direct link to Security") * \[CVE-2026-45416] \[CVE-2026-44249] \[CVE-2026-45673] Upgraded Netty to 4.1.135.Final to fix SNI handler heap exhaustion (DoS), IPv6 subnet filter bypass, and DNS cache poisoning. [#74668](https://github.com/StarRocks/starrocks/pull/74668) * \[CVE-2026-54512] \[CVE-2026-54513] Upgraded `jackson-databind` to 2.21.4 to fix two deserialization vulnerabilities. [#75373](https://github.com/StarRocks/starrocks/pull/75373) * \[GHSA-2r2c-cx56-8933] \[GHSA-47qp-hqvx-6r3f] Excluded `org.jline:jline-remote-telnet` from Hadoop transitive dependencies to remediate unauthenticated Telnet-server DoS vulnerabilities. [#75066](https://github.com/StarRocks/starrocks/pull/75066) * \[CVE-2026-39822] Updated pprof prebuilt to fix a vulnerability in the pprof binary. [#76248](https://github.com/StarRocks/starrocks/pull/76248) [#74669](https://github.com/StarRocks/starrocks/pull/74669) * Fixed SQL injection in `information_schema.task_runs` where a single quote in a predicate value could escape the literal boundary. [#75520](https://github.com/StarRocks/starrocks/pull/75520) * `tencent.cos.access_key`, `tencent.cos.secret_key`, and `iceberg.catalog.jdbc.password` are now masked in `SHOW CREATE CATALOG` output. [#74696](https://github.com/StarRocks/starrocks/pull/74696) * Fixed an out-of-bounds read in `url_decode` when the input ends with a truncated percent-escape sequence. [#75139](https://github.com/StarRocks/starrocks/pull/75139) * Fixed `HyperLogLog::deserialize` accepting out-of-range `SPARSE` register indices, which could corrupt heap memory and crash the BE on malformed input. [#75521](https://github.com/StarRocks/starrocks/pull/75521) * Fixed `bar()` rejecting negative width values, which previously allowed unbounded string growth and BE memory exhaustion. [#75143](https://github.com/StarRocks/starrocks/pull/75143) ##### Bug Fixes[​](#bug-fixes-3 "Direct link to Bug Fixes") The following issues have been fixed: * `add_files` populated Iceberg file bounds with Parquet physical encoding bytes instead of logical typed values, causing incorrect file-level min/max pruning (for example, on `DECIMAL` columns). [#69207](https://github.com/StarRocks/starrocks/pull/69207) * `ApplyTuningGuideRule` threw `UnsupportedOperationException` when traversing plan nodes whose input lists were built as immutable `List.of(...)`. [#70785](https://github.com/StarRocks/starrocks/pull/70785) * `INSERT OVERWRITE` two-phase re-plan could produce stale lambda-argument column reference IDs from the first planning session, causing `expr_type does not match slot_type` errors. [#73273](https://github.com/StarRocks/starrocks/pull/73273) * Partial updates on tables with GIN (inverted) indexes caused queries to hang indefinitely or fail when the GIN-indexed column was omitted from the update. [#73773](https://github.com/StarRocks/starrocks/pull/73773) * Lake PCU (partial column update) crashed or silently corrupted data when a schema drift occurred between the rowset schema and the tablet schema. [#74005](https://github.com/StarRocks/starrocks/pull/74005) * A combined `ALTER TABLE` on an external Iceberg table with multiple schema clauses incorrectly re-executed all previously queued actions on every clause dispatch. [#74036](https://github.com/StarRocks/starrocks/pull/74036) * `PartitionedSpillerWriter` crashed with `SIGSEGV` when the `num_rows` snapshot exceeded the actual chunk row count during partition-flush and resource-group-cancellation interleaving. [#74081](https://github.com/StarRocks/starrocks/pull/74081) * BE process could exit unexpectedly during startup (typically right after deployment) because SIGPIPE was not ignored in BE signal initialization. [#74424](https://github.com/StarRocks/starrocks/pull/74424) * Parquet temporary dict-code columns leaked to upper layers when a struct VARCHAR subfield fill was skipped due to row-range filtering, causing type mismatches. [#74452](https://github.com/StarRocks/starrocks/pull/74452) * `SELECT ... INTO OUTFILE` recorded `ReturnRows=0` in the audit log instead of the actual exported row count. [#74467](https://github.com/StarRocks/starrocks/pull/74467) * `TabletChecker.doCheck()` threw `IllegalMonitorStateException` in `blockingAddTabletCtxToScheduler` due to a lock type mismatch, causing entire checker rounds to abort silently. [#74596](https://github.com/StarRocks/starrocks/pull/74596) * `information_schema.COLUMNS` always returned `NULL` for `DATETIME_PRECISION`, breaking MySQL-protocol clients that derive column size from that field. [#74623](https://github.com/StarRocks/starrocks/pull/74623) * MV refresh failed with `Duplicate key` when the query joined two tables with the same unqualified name across different databases or catalogs. [#74730](https://github.com/StarRocks/starrocks/pull/74730) * Spillable hash join probe crashed under certain conditions. [#74978](https://github.com/StarRocks/starrocks/pull/74978) [#75140](https://github.com/StarRocks/starrocks/pull/75140) * Iceberg `truncate` and `bucket` transform functions crashed the BE with `SIGFPE` when the width or bucket count argument was zero. [#74998](https://github.com/StarRocks/starrocks/pull/74998) * `mod()` and `pmod()` crashed the BE with `SIGFPE` when the dividend was `TYPE_MIN` and the divisor was `-1`. [#74980](https://github.com/StarRocks/starrocks/pull/74980) * `histogram()` crashed the BE with `SIGFPE` when `bucket_num` was zero or negative. [#75041](https://github.com/StarRocks/starrocks/pull/75041) * `encode_fingerprint_sha256` crashed with `SIGSEGV` when all input rows were `NULL`. [#75042](https://github.com/StarRocks/starrocks/pull/75042) * `LIKE` patterns containing the single-char wildcard `_` returned incorrect results when evaluated via a GIN inverted index. [#75551](https://github.com/StarRocks/starrocks/pull/75551) * AND-only `MATCH` queries against a GIN inverted index returned a spurious error when the target segment was empty. [#75161](https://github.com/StarRocks/starrocks/pull/75161) * CLucene `match_all` queries returned incorrect results; resolved by upgrading the CLucene dependency. [#75180](https://github.com/StarRocks/starrocks/pull/75180) * Vector index rewrite registered a synthetic distance column directly on the shared table schema, causing `Multiple entries with same key` errors for unrelated concurrent queries on the same table. [#74785](https://github.com/StarRocks/starrocks/pull/74785) * Join reorder pruning could prune columns still referenced by scan predicates, causing statistics estimation to throw `missing statistic of col`. [#74791](https://github.com/StarRocks/starrocks/pull/74791) * `avg(DISTINCT x)` was incorrectly rewritten via a sum/count materialized view, silently dropping the `DISTINCT` and returning wrong results when duplicates existed. [#75071](https://github.com/StarRocks/starrocks/pull/75071) * `ALTER TABLE ... MODIFY COLUMN ... AFTER ` threw an internal `NullPointerException` instead of a clean semantic error. [#75073](https://github.com/StarRocks/starrocks/pull/75073) * `SHOW CREATE ROUTINE LOAD` emitted a spurious leading comma before the first load-description clause for jobs without a `COLUMNS TERMINATED BY` clause. [#75522](https://github.com/StarRocks/starrocks/pull/75522) * `SECURITY INVOKER` views with CTEs could fail privilege checks with an NPE when the CTE name was mistaken for a real table reference. [#74813](https://github.com/StarRocks/starrocks/pull/74813) * `ReduceCastRule` aborted query planning with a `SemanticException` when a date/datetime boundary literal shifting would overflow the representable range (for example, `<= '9999-12-31'`). [#75036](https://github.com/StarRocks/starrocks/pull/75036) * `SplitJoinORToUnionRule` emitted duplicate rows when a join condition used null-safe-equal (`<=>`) disjuncts. [#75038](https://github.com/StarRocks/starrocks/pull/75038) * `Tracers` shared across parallel metadata-preparation threads for multi-table external queries caused `IllegalStateException` under `enable_profile=true`. [#74746](https://github.com/StarRocks/starrocks/pull/74746) * Partition consumer errors in `ChunksPartitioner` were silently discarded, allowing a partitioned TopN to return partial or wrong results without surfacing an error. [#74693](https://github.com/StarRocks/starrocks/pull/74693) * BE vacuum tasks continued running as zombies after the FE caller's timeout elapsed, exhausting the `RELEASE_SNAPSHOT` thread pool and collapsing vacuum throughput. [#74694](https://github.com/StarRocks/starrocks/pull/74694) * An autovacuum race could momentarily compute a `minActiveTxnId` one greater than an in-flight transaction, causing the BE to delete still-needed combined transaction logs and permanently wedging publish. [#74906](https://github.com/StarRocks/starrocks/pull/74906) * Queries were incorrectly marked as canceled after finishing successfully due to a race between FE EOS-cancel and BE stage-2 deploy. [#75009](https://github.com/StarRocks/starrocks/pull/75009) * BE crashed with `SIGSEGV` in `AggTopNRuntimeFilterUpdaterImpl` when the aggregate TopN runtime filter build key was a `ConstColumn`. [#74809](https://github.com/StarRocks/starrocks/pull/74809) [#74941](https://github.com/StarRocks/starrocks/pull/74941) * `array_map` / `transform` silently dropped `NULL` rows and returned wrong row counts when all non-null arrays were empty. [#75141](https://github.com/StarRocks/starrocks/pull/75141) * `LARGEINT` / `DECIMAL128` literals above 2^64 were silently truncated to 64 bits in JIT-compiled expressions. [#75137](https://github.com/StarRocks/starrocks/pull/75137) * UTF-8 string functions (`split`, `split_part`, `str_to_map`) read beyond the end of the string when the last character had a truncated or invalid multi-byte lead byte with an empty delimiter. [#75068](https://github.com/StarRocks/starrocks/pull/75068) * `parse_json()` silently returned `NULL` for malformed JSON even in `ALLOW_THROW_EXCEPTION` SQL mode instead of failing the query. [#74976](https://github.com/StarRocks/starrocks/pull/74976) * Strict-mode numeric narrowing casts incorrectly raised overflow errors on `NULL` rows whose slot data was undefined. [#74903](https://github.com/StarRocks/starrocks/pull/74903) * Pre-1970 Parquet `INT64` timestamps with a non-zero sub-second part were decoded to garbage values due to negative truncating-division remainder. [#75207](https://github.com/StarRocks/starrocks/pull/75207) * Pre-1970 ORC `TIMESTAMP` values had their sub-second component dropped on load. [#75432](https://github.com/StarRocks/starrocks/pull/75432) * ORC stripe min/max timestamp statistics were decoded incorrectly for pre-1970 and sub-second bounds, causing data files to be incorrectly pruned. [#75543](https://github.com/StarRocks/starrocks/pull/75543) * Nested `INT96` Parquet timestamps inside `ARRAY`, `MAP`, or `STRUCT` columns lost one session-timezone offset on load. [#74868](https://github.com/StarRocks/starrocks/pull/74868) * Parquet `UINT_32` values were sign-extended instead of zero-extended when loaded into a `BIGINT` column, silently storing negative values for high-bit unsigned integers. [#75002](https://github.com/StarRocks/starrocks/pull/75002) * `HiveDataSource` destructor caused a heap-use-after-free by destroying `_pool` (and its `Expr` nodes) before `_scanner_ctx` (which holds predicates referencing those nodes). [#74818](https://github.com/StarRocks/starrocks/pull/74818) * Reading gzip-compressed JSON Hive external tables with OpenX SerDe failed with `UTF8_ERROR` when a multi-byte UTF-8 character straddled an 8 MB decompression buffer boundary. [#74827](https://github.com/StarRocks/starrocks/pull/74827) * ADLS2 `ListPaths` crashed with `SIGSEGV` on non-HNS accounts due to missing JSON fields that the client accessed unconditionally. [#75166](https://github.com/StarRocks/starrocks/pull/75166) * `unnest` crashed or returned wrong results when multiple `UNNEST` operators shared the same input array column and consumed different subfields. [#75012](https://github.com/StarRocks/starrocks/pull/75012) [#75445](https://github.com/StarRocks/starrocks/pull/75445) [#76002](https://github.com/StarRocks/starrocks/pull/76002) * `query_mem_limit` was not enforced during `unnest` execution, allowing `unnest` over large arrays to OOM-kill the BE instead of failing the query. [#75179](https://github.com/StarRocks/starrocks/pull/75179) * `TopN` with `RANK` boundary dropped one row when the rank limit fell exactly on a chunk boundary. [#75045](https://github.com/StarRocks/starrocks/pull/75045) * Column pruning after `PushDownDistinctAggregateRule` could generate an empty analytic (window) operator, causing planning or execution errors. [#74810](https://github.com/StarRocks/starrocks/pull/74810) * `EliminateSortColumnWithEqualityPredicateRule` set the row limit only on the scan operator without setting a global limit, causing `COUNT(*)` over a limited sub-query to return more rows than expected under concurrency. [#74983](https://github.com/StarRocks/starrocks/pull/74983) * Lake primary-key persistent index rebuild used wrong segment iterator positions in segment-range mode, causing incorrect key-range filter application. [#74887](https://github.com/StarRocks/starrocks/pull/74887) [#75206](https://github.com/StarRocks/starrocks/pull/75206) * `DROP PERSISTENT INDEX` modified `rebuildPindexVersion` without a table lock; `RestoreJob` post-restore mutated MV base-table info under only a DB READ lock; `FinalizeCreateTableAction` passed a DB-level lock across iterator creation. [#74968](https://github.com/StarRocks/starrocks/pull/74968) * `dumpImage` could strand the global meta lock indefinitely if acquiring a per-database lock threw mid-loop. [#75488](https://github.com/StarRocks/starrocks/pull/75488) * Multi-statement stream load leaked one `TxnStateCallbackFactory` entry per transaction, growing without bound and eventually exhausting FE heap. [#75188](https://github.com/StarRocks/starrocks/pull/75188) * `information_schema.task_runs` row count for histogram statistics could overflow `primary_key_limit_size` (128 bytes) when catalog, database, table, or partition names were long. [#75735](https://github.com/StarRocks/starrocks/pull/75735) * BE JVM metrics emitted invalid Prometheus `# TYPE` lines (label sets inside the metric name), causing Prometheus to abort the entire scrape. [#75240](https://github.com/StarRocks/starrocks/pull/75240) * `SHOW PARTITIONS` and `information_schema.partitions_meta` reported all physical partitions' bucket counts as the table-level default instead of each partition's actual bucket count on shared-data tables. [#75734](https://github.com/StarRocks/starrocks/pull/75734) * `SHOW PROC '.../index_schema/'` returned the base table schema for all rollup indexes on shared-data (`CLOUD_NATIVE`) tables. [#76069](https://github.com/StarRocks/starrocks/pull/76069) * `ALTER TABLE ... MODIFY COLUMN` no-op clauses were incorrectly routed to the lightweight comment path, causing `MODIFY COLUMN COMMENT can not be combined with other alter operations` errors in batch `ALTER TABLE` statements. [#75736](https://github.com/StarRocks/starrocks/pull/75736) * `isCommentOnlyModification` could misidentify key/aggregation columns as comment-only changes due to incorrect `isKey` / `aggregationType` normalization. [#75545](https://github.com/StarRocks/starrocks/pull/75545) * `ALTER VIEW` could commit a cyclic view definition that caused subsequent `SELECT` to throw `StackOverflowError`. [#75033](https://github.com/StarRocks/starrocks/pull/75033) * `OrderedPartitionExchanger` caused a heap-use-after-free when a downstream consumer mutated the previous chunk while `accept()` still held a pointer to it. [#75279](https://github.com/StarRocks/starrocks/pull/75279) * NLJoin crashed when the build-side's slot descriptor was non-nullable but the runtime state was nullable. [#75343](https://github.com/StarRocks/starrocks/pull/75343) [#75788](https://github.com/StarRocks/starrocks/pull/75788) * `CAST(json/variant AS struct)` crashed the BE at fragment prepare when a struct field name was not a parseable JSON path. [#75355](https://github.com/StarRocks/starrocks/pull/75355) * Dict-decode for nested dictionary expressions could produce incompatible dictionary translations between producer and consumer fragments, causing `Dict Decode failed` errors at runtime. [#75246](https://github.com/StarRocks/starrocks/pull/75246) * Schema change crashed with a null-pointer dereference when `get_rowset_by_version` returned `nullptr` and the `gtid` comparison was placed before the null check. [#74855](https://github.com/StarRocks/starrocks/pull/74855) * Shared-data cluster snapshots became unrestorable after a tablet split/merge because the snapshot manager did not consider reshard jobs when deciding whether to reap parent tablet metadata. [#75638](https://github.com/StarRocks/starrocks/pull/75638) * File-bundling vacuum incorrectly flagged zero-row bundled segments of sibling tablets as non-shared, causing their bundle file to be deleted while other tablets still referenced it. [#75689](https://github.com/StarRocks/starrocks/pull/75689) * Shared-data table compaction publish dropped rollup/synchronous-MV indexes that became visible after the compaction transaction began, leaving the bundle file without those indexes. [#76105](https://github.com/StarRocks/starrocks/pull/76105) * Shared-data persistent-index compaction incorrectly deleted pass-through-reused SSTable files when a compaction was dropped during tablet reshard. [#75726](https://github.com/StarRocks/starrocks/pull/75726) * `NOT NULL`-to-nullable flat-JSON column schema evolution caused a `CHECK` crash in the compaction read path. [#75680](https://github.com/StarRocks/starrocks/pull/75680) * `count_combine` over a nullable column crashed the BE with `SIGSEGV` in the streaming pre-aggregation pass-through path. [#75298](https://github.com/StarRocks/starrocks/pull/75298) * Java UDFs failed to load on JDK 21+ due to a reflective `DirectByteBuffer` constructor lookup that was removed in JDK 21. [#75666](https://github.com/StarRocks/starrocks/pull/75666) * `CTAS` into a Unified catalog (Hive metastore) always failed because the parser did not support the `ENGINE` clause in `CREATE TABLE AS SELECT`. [#75771](https://github.com/StarRocks/starrocks/pull/75771) * `JoinTuningGuide` feedback-driven join rebuild lost `predicateCommonOperators`, causing `InputDependenciesChecker` validation failures on plans with common sub-expression reuse. [#75773](https://github.com/StarRocks/starrocks/pull/75773) * Query cache normalization crashed with `Preconditions.checkState` on tables with sub-partitions where empty sub-partitions were pruned before the version list was built. [#75789](https://github.com/StarRocks/starrocks/pull/75789) * `replayFromJson` silently skipped session variables stored under a legacy alias, causing query dump replay to fall back to default values. [#75813](https://github.com/StarRocks/starrocks/pull/75813) * Iceberg `_row_id` virtual column returned incorrect values for data files with more than one Parquet row group due to double-counting of the row-group start offset. [#75758](https://github.com/StarRocks/starrocks/pull/75758) * Iceberg DELETE/UPDATE planner could not locate the target scan node because it matched by synthetic table ID instead of physical table identity, losing the base snapshot ID and conflict-detection filter. [#76013](https://github.com/StarRocks/starrocks/pull/76013) * `FragmentContext::set_final_status` crashed with `SIGSEGV` when a `cancel_plan_fragment` RPC arrived before `PipelineExecutorSet::start()` was called. [#75030](https://github.com/StarRocks/starrocks/pull/75030) * `QueryContext` could be reclaimed while `FragmentExecutor` was still tearing down a fragment, causing heap-use-after-free in `ResGuard::reset()`. [#74978](https://github.com/StarRocks/starrocks/pull/74978) * `StringSearch::_pattern` was uninitialized, allowing a default-constructed `search()` to dereference an uninitialized pointer. [#75614](https://github.com/StarRocks/starrocks/pull/75614) * `DATETIME` microseconds were rendered using the JVM default locale's digit set, causing non-ASCII digits on locales such as Arabic or Persian that broke boundary value parsing in tablet pre-split. [#75001](https://github.com/StarRocks/starrocks/pull/75001) * Partition row counts could be written as zero into `_statistics_.column_statistics` after `INSERT OVERWRITE` before tablet stats were refreshed, causing the optimizer to collapse partition cardinality estimates. [#74801](https://github.com/StarRocks/starrocks/pull/74801) * `enable_statistic_collect_on_first_load` table-level override could not enable first-load statistics collection when the global config disabled it. [#74794](https://github.com/StarRocks/starrocks/pull/74794) * `PushDownNonGroupedAggregateBelowUnion` produced nullable outputs with non-nullable declared types when a union branch had no input rows, causing BE `CHECK` failures. [#76101](https://github.com/StarRocks/starrocks/pull/76101) #### 4.1.2[​](#412 "Direct link to 4.1.2") Release Date: June 18, 2026 ##### Behavior Changes[​](#behavior-changes-3 "Direct link to Behavior Changes") * Connecting to a database that the user has no privilege on now returns a proper MySQL error packet instead of closing the connection with ERROR 2013. [#70072](https://github.com/StarRocks/starrocks/pull/70072) * `SHOW FUNCTIONS` now masks UDF file and object-file paths as `***` for users who can see a function through function-level privileges but do not hold the create-function scope privilege. [#73425](https://github.com/StarRocks/starrocks/pull/73425) * Ranger row filter and column masking policies are now correctly applied when querying Hive views from external catalogs. [#73265](https://github.com/StarRocks/starrocks/pull/73265) * `ALTER TABLE ... ADD COLUMN ... DEFAULT current_timestamp` now correctly preserves the `current_timestamp` generator expression. `DESCRIBE` and `information_schema` now reflect the expression rather than the backfill-time literal. [#73455](https://github.com/StarRocks/starrocks/pull/73455) * `information_schema.loads` load-time filtering no longer shifts filter bounds on clusters whose session timezone differs from UTC+8. Load times are now exchanged as UTC epoch milliseconds across the FE–BE boundary. [#73365](https://github.com/StarRocks/starrocks/pull/73365) * The `connector_max_split_size` session variable now correctly applies to Paimon scan split calculation instead of always using the default value. [#71756](https://github.com/StarRocks/starrocks/pull/71756) * `pipeline_enable_large_column_checker` is now enabled by default. [#72798](https://github.com/StarRocks/starrocks/pull/72798) * Hive partition statistics are no longer refreshed automatically per key on a timer. Partition stats are now only refreshed during explicit `refreshTable()` calls, reducing HMS load on large partitioned tables. [#73563](https://github.com/StarRocks/starrocks/pull/73563) * When an Iceberg or external-catalog base table undergoes schema drift (column type change, column drop, or table drop), the dependent materialized view is now marked inactive at the next refresh instead of silently producing NULL rows or opaque errors. [#73770](https://github.com/StarRocks/starrocks/pull/73770) * The Iceberg connector now partially pushes down the convertible side of `AND` compound predicates instead of discarding the entire predicate when only one side is convertible, improving partition pruning and data skipping. [#70293](https://github.com/StarRocks/starrocks/pull/70293) * Explicit-transaction `COMMIT` now correctly waits up to `query_timeout` seconds (not milliseconds) for the database write lock, preventing spurious lock-timeout failures under brief concurrent write activity. [#73549](https://github.com/StarRocks/starrocks/pull/73549) * IVM refresh now surfaces strict-load filter errors to the caller instead of silently dropping filtered rows. [#73938](https://github.com/StarRocks/starrocks/pull/73938) * `count_combine(nullable_col)` now correctly excludes NULL rows, matching `COUNT(col)` semantics. Incremental MVs backed by `COUNT()` were previously materializing inflated counts. [#74029](https://github.com/StarRocks/starrocks/pull/74029) * `SHOW ALTER TABLE COLUMN` now also displays asynchronous metadata-only alter jobs triggered by `ALTER TABLE ... SET (...)` for properties such as `file_bundling` and `enable_persistent_index` on cloud-native (shared-data) tables. [#74198](https://github.com/StarRocks/starrocks/pull/74198) * Creating an incremental MV with a `HAVING` clause that references an aggregate function now fails at `CREATE` time with a clear error instead of producing an internal plan error on first refresh. [#74054](https://github.com/StarRocks/starrocks/pull/74054) * IVM now supports `MIN`/`MAX(DECIMAL)` aggregate functions in incremental materialized views. [#73969](https://github.com/StarRocks/starrocks/pull/73969) * IVM adaptive refresh now correctly bounds the delta window when the first delta trait already exceeds `mv_max_rows_per_refresh`, preventing the entire backlog from being refreshed in one task run. [#74464](https://github.com/StarRocks/starrocks/pull/74464) * GROUP-BY-only incremental MVs (for example, `SELECT k FROM t GROUP BY k`) now correctly encode `__ROW_ID__` as VARCHAR, fixing a crash on the second refresh. [#74030](https://github.com/StarRocks/starrocks/pull/74030) ##### Improvements[​](#improvements-4 "Direct link to Improvements") * Supports Paimon views, including `CREATE`/`REPLACE`/`DROP`, `SHOW`/`DESC`, and querying Paimon views from external catalogs. Table references inside Paimon views are now resolved against the Paimon catalog instead of `default_catalog`. [#56058](https://github.com/StarRocks/starrocks/pull/56058) [#70217](https://github.com/StarRocks/starrocks/pull/70217) * Supports an explicit `schema` parameter in `FILES()` for stable schema control when reading files with schema drift or complex nested types. [#72033](https://github.com/StarRocks/starrocks/pull/72033) * `get_query_profile()` now retrieves query profile information across all FE nodes, not only the connected FE. [#71123](https://github.com/StarRocks/starrocks/pull/71123) * Added the `query_id()` built-in function, which returns the UUID of the currently executing query. [#73621](https://github.com/StarRocks/starrocks/pull/73621) * `CREATE`/`ALTER STORAGE VOLUME` in shared-data mode now validates storage location accessibility (credentials and endpoints) before persisting metadata, failing early on misconfiguration. [#70053](https://github.com/StarRocks/starrocks/pull/70053) * Added `WebIdentity` token provider support for AWS S3 credentials in BE, matching the existing FE support for `AWS_S3_USE_WEB_IDENTITY_TOKEN_FILE`. [#69966](https://github.com/StarRocks/starrocks/pull/69966) * Added the `ADMIN SKIP COMMITTED TRANSACTION` command to unblock stuck `COMMITTED` transactions on shared-data tables when publish is permanently blocked by missing `txnlog`, lost segments, or slow remote I/O. [#73553](https://github.com/StarRocks/starrocks/pull/73553) * `information_schema.tables_config` now pushes down `table_name` predicates to the FE, greatly reducing overhead for single-table lookups. [#73210](https://github.com/StarRocks/starrocks/pull/73210) * Added missing MySQL 8 columns to `information_schema` tables to improve compatibility with BI tools and JDBC drivers that inspect MySQL 8 schemas during connection introspection. [#73370](https://github.com/StarRocks/starrocks/pull/73370) * Added the `enable_pipeline_event_scheduler` BE configuration as a cluster-wide kill switch that overrides the per-session variable when set to `false`. [#73264](https://github.com/StarRocks/starrocks/pull/73264) * Added opt-in wide-string column isolation for statistics collection to reduce per-query memory peaks when collecting statistics on tables with multiple wide string columns. [#73258](https://github.com/StarRocks/starrocks/pull/73258) * Slow-lock logging now supports per-event rate limiting and configurable stack-capture controls to prevent JVM safepoint stalls under high lock contention. [#73647](https://github.com/StarRocks/starrocks/pull/73647) * MV refresh log entries now include the database name in the prefix, making log lines distinguishable in multi-tenant deployments where the same MV name exists in multiple schemas. [#73521](https://github.com/StarRocks/starrocks/pull/73521) * `enable_profile_log` FE configuration is now mutable and can be toggled at runtime via `ADMIN SET FRONTEND CONFIG` without an FE restart. [#73894](https://github.com/StarRocks/starrocks/pull/73894) * Added the `enable_print_load_profile_to_log` FE configuration (default `false`) to write load profiles (stream load, routine load, broker load, and merge-commit load) to `fe.profile.log`, preserving them even when the in-memory store is evicted by query-profile bursts. [#74150](https://github.com/StarRocks/starrocks/pull/74150) * `SHOW ROUTINE LOAD` now correctly renders column mappings in `JobProperties` instead of Java object references. [#74199](https://github.com/StarRocks/starrocks/pull/74199) * `CachingIcebergCatalog` now uses table-level locking instead of catalog-level locking, reducing refresh serialization lag on catalogs with many concurrently active tables. [#73079](https://github.com/StarRocks/starrocks/pull/73079) * Meta scan (background statistics collection) now handles `ADD COLUMN`, `DROP COLUMN`, `RENAME COLUMN`, and `REORDER COLUMN` schema changes gracefully instead of failing with a not-found error on post-change segment files. [#72901](https://github.com/StarRocks/starrocks/pull/72901) * Sample-based tablet pre-split now covers multi-partition range-distribution tables and Broker Load, enabling first-load parallelism without an existing data-tier baseline. [#73101](https://github.com/StarRocks/starrocks/pull/73101) [#73912](https://github.com/StarRocks/starrocks/pull/73912) [#74048](https://github.com/StarRocks/starrocks/pull/74048) * MySQL result serialization no longer uses per-row virtual dispatch; a typed column writer is built once per chunk, reducing serialization overhead for wide or large result sets. [#66316](https://github.com/StarRocks/starrocks/pull/66316) * `DATETIME`/`DATE`-to-string casts now write directly to the output buffer, eliminating per-row heap allocations. [#73801](https://github.com/StarRocks/starrocks/pull/73801) * Query statistics merge path replaced a `SpinLock` with a lock-free parallel map, reducing CPU usage on large clusters when workers send intermediate or final statistics. [#73796](https://github.com/StarRocks/starrocks/pull/73796) * Aggregation hash-map and hash-set prefetch is now gated on L2-cache residency, avoiding a 4–9% regression when the bucket array fits in L2. The prefetch distance is now configurable. [#73943](https://github.com/StarRocks/starrocks/pull/73943) * Pipelined per-segment `.lcrm` reads for light-compaction publish on shared-data Primary Key tables reduce sequential object-storage round-trips. [#73992](https://github.com/StarRocks/starrocks/pull/73992) * Cold PK-index rebuild scan in shared-data mode is now parallelized across segments, reducing rebuild time when segment reads are remote-I/O bound. [#74249](https://github.com/StarRocks/starrocks/pull/74249) * Internal queries (statistics collection, task runs, MV refreshes) are now visible in `SHOW PROC '/current_queries'` and can be killed with `KILL QUERY`. [#74488](https://github.com/StarRocks/starrocks/pull/74488) * Added lake vacuum batch-size and retry-count bvar metrics for monitoring S3 throttling and tuning `lake_vacuum_min_batch_delete_size`. [#74112](https://github.com/StarRocks/starrocks/pull/74112) * Added `CatalogRecycleBin` size gauge metrics to surface recycle-bin growth before it pressures FE heap. [#74440](https://github.com/StarRocks/starrocks/pull/74440) * `LIST`-partitioned tables now open all partitions in `OlapTableSink` instead of applying the latest-N heuristic designed for range-partition tables, reducing incremental-open RPC overhead. [#74099](https://github.com/StarRocks/starrocks/pull/74099) * Supports loading `LARGE_LIST` and `FIXED_SIZE_LIST` Arrow types into JSON columns via `FILES()` or Broker Load. [#73714](https://github.com/StarRocks/starrocks/pull/73714) [#73718](https://github.com/StarRocks/starrocks/pull/73718) * Supports combined transaction log and file bundling for merge-commit (`FRONTEND_STREAMING`) loads on shared-data tables, aligning them with other load types. [#74460](https://github.com/StarRocks/starrocks/pull/74460) * Added the mutable FE configuration `slow_publish_partition_log_threshold_ms` (default 3000 ms) to control the lake publish phase-breakdown warning threshold without an FE restart. [#74043](https://github.com/StarRocks/starrocks/pull/74043) ##### Security[​](#security-4 "Direct link to Security") * \[CVE-2026-43869] Bumped `libthrift` to 0.23.0 to address improper certificate host validation. [#73243](https://github.com/StarRocks/starrocks/pull/73243) * \[CVE-2026-41293] Bumped Apache Tomcat to 9.0.118 to address HTTP/2 request header validation. [#73797](https://github.com/StarRocks/starrocks/pull/73797) * \[CVE-2026-45416] \[CVE-2026-44249] \[CVE-2026-45673] Bumped Netty to 4.1.135.Final to address SNI handler heap exhaustion (DoS), IPv6 subnet filter bypass, and DNS cache poisoning. [#74668](https://github.com/StarRocks/starrocks/pull/74668) * Upgraded pprof prebuilt binaries to Go 1.25.11 to include Go standard library security fixes. [#73545](https://github.com/StarRocks/starrocks/pull/73545) [#74669](https://github.com/StarRocks/starrocks/pull/74669) ##### Bug Fixes[​](#bug-fixes-4 "Direct link to Bug Fixes") The following issues have been fixed: * `parse_url()` returned a wrong host when the URL contained `:` outside a `host:port` pattern. [#63542](https://github.com/StarRocks/starrocks/pull/63542) * Dictionary-translated expressions incorrectly assumed `f(null) = null` for expressions where this does not hold (for example, `IF(col = '1', NULL, 'ok')`). [#69376](https://github.com/StarRocks/starrocks/pull/69376) * Transaction stream load used the default RPC timeout instead of the user-specified timeout, causing premature timeouts. [#67584](https://github.com/StarRocks/starrocks/pull/67584) * Iceberg equality delete files with NULL values in identity columns failed to delete matching rows because `NULL = NULL` evaluates to UNKNOWN in the join predicate. [#67321](https://github.com/StarRocks/starrocks/pull/67321) * Error messages for tables with `INJECTED` partition-projection columns are now more descriptive, showing which columns are causing the issue. [#68052](https://github.com/StarRocks/starrocks/pull/68052) * Queries on insert-only ACID Hive tables returned more rows than expected because insert-overwrite operations were not recognized. [#71460](https://github.com/StarRocks/starrocks/pull/71460) * Disk cache overflowed its configured capacity when Iceberg metadata entries were pinned during concurrent reads. [#71651](https://github.com/StarRocks/starrocks/pull/71651) * Paimon primary key columns were incorrectly marked as non-nullable when querying external catalogs. [#71660](https://github.com/StarRocks/starrocks/pull/71660) * Optimizer timed out when `MultiDistinctByMultiFuncRewriter` applied the same rule repeatedly on queries with multiple `ARRAY_AGG(DISTINCT )` inputs. [#70605](https://github.com/StarRocks/starrocks/pull/70605) * Oracle JDBC date predicates pushed down without the `DATE`/`TIMESTAMP` keyword caused NLS format errors. [#71412](https://github.com/StarRocks/starrocks/pull/71412) * Partition TopN could lose required output columns from its child operator. [#72848](https://github.com/StarRocks/starrocks/pull/72848) * Unpartitioned MVs could not be created on Iceberg tables with partition evolution. [#72285](https://github.com/StarRocks/starrocks/pull/72285) * Compaction task stats were overwritten and lost for parallel subtasks in `information_schema.be_cloud_native_compactions`. [#72331](https://github.com/StarRocks/starrocks/pull/72331) * `SHOW CREATE MATERIALIZED VIEW` for sync MVs failed with "Table is not found". [#73396](https://github.com/StarRocks/starrocks/pull/73396) * Lake publish multi-statement transactions deadlocked during schema change when statement `.log` files landed at the 4-segment path. [#73423](https://github.com/StarRocks/starrocks/pull/73423) * Sort merge provider errors were not propagated to the fragment context, causing silent query failures. [#73337](https://github.com/StarRocks/starrocks/pull/73337) * `ConnectorTableId` overflowed from `int` to negative values on long-running follower FEs, causing Iceberg and Hive queries to fail with misleading "Invalid table type" errors. [#73344](https://github.com/StarRocks/starrocks/pull/73344) * `ALTER TABLE` with an empty optimize clause (no distribution or partition spec) was incorrectly parsed and could corrupt the table's default distribution on FE replay. [#73352](https://github.com/StarRocks/starrocks/pull/73352) * FE startup failed during ADLS2 shared-data disaster recovery because `AZURE_PATH_KEY` was not recognized as a valid `StorageVolumeMgr` parameter. [#73509](https://github.com/StarRocks/starrocks/pull/73509) * Avro complex-type decoding failed when the optimizer pruned part of a nested type to `UNKNOWN_TYPE`, or when nullable array, map, or struct schemas were used. [#73474](https://github.com/StarRocks/starrocks/pull/73474) * COW column mutation optimization caused a crash in `map_apply` and similar functions due to two `NullableColumn`s sharing the same `NullColumn` object. [#73480](https://github.com/StarRocks/starrocks/pull/73480) * Iceberg tables with a custom `LocationProvider` failed on `SELECT` queries with `ClassNotFoundException` because the provider was eagerly instantiated on FE. [#73482](https://github.com/StarRocks/starrocks/pull/73482) * JDBC `getTable()` performed an extra `getTableComment()` round-trip on every cache miss, extending the intensive planning-phase lock hold and blocking concurrent DDL. [#73488](https://github.com/StarRocks/starrocks/pull/73488) * Nested MV refresh threw `NullPointerException` when a nested MV returned `FULL` or `UNKNOWN` timeliness. [#73644](https://github.com/StarRocks/starrocks/pull/73644) * FE worker blocked indefinitely sending query results to a slow MySQL client. The result-send path now enforces a write timeout. [#73646](https://github.com/StarRocks/starrocks/pull/73646) * PK `.del` files were not transcoded when replicating Primary Key tables from a V1-encoded (shared-nothing) to a V2-encoded (shared-data) cluster, or between two shared-data clusters. [#73649](https://github.com/StarRocks/starrocks/pull/73649) [#73958](https://github.com/StarRocks/starrocks/pull/73958) * Duplicate replicas accumulated in `TabletInvertedIndex` during `VERSION_INCOMPLETE` recovery because the stale replica reference was not removed before adding the live one. [#73661](https://github.com/StarRocks/starrocks/pull/73661) * Shared-data lake replication file-copy crashed the CN because `REPLICATE_SNAPSHOT` tasks and per-file copy sub-tasks shared the same thread pool. [#73666](https://github.com/StarRocks/starrocks/pull/73666) * `RuntimeProfileParser` threw `NumberFormatException` when unit counters were formatted with a `.000` decimal suffix by the BE. [#73683](https://github.com/StarRocks/starrocks/pull/73683) * Physical rowid encoding for shared segments in shared-data PK tablet split was incorrect, causing wrong `rss_rowid` entries. [#73686](https://github.com/StarRocks/starrocks/pull/73686) * Mixed range-colocate × hash-distributed `JOIN` queries returned `Unknown error` instead of valid results. [#73702](https://github.com/StarRocks/starrocks/pull/73702) * `TimeUtils.longToTimeString` used a fixed UTC+8 formatter; output now respects the session `time_zone`. [#73619](https://github.com/StarRocks/starrocks/pull/73619) * Decimal-typed columns lost scale when all values were `NULL` and the column passed through the nullable unary function path, corrupting downstream result types. [#73789](https://github.com/StarRocks/starrocks/pull/73789) * JSON partial-append for nested types caused an ASAN crash. [#73715](https://github.com/StarRocks/starrocks/pull/73715) * Privilege cache for the `public` role was not invalidated on `GRANT`/`REVOKE`, leaving stale privileges in effect until expiration. [#73717](https://github.com/StarRocks/starrocks/pull/73717) * `FlatJson` crashed when sub-writers had no appends. [#73730](https://github.com/StarRocks/starrocks/pull/73730) * Aggregate MV rewrite was incorrectly applied when the MV itself had a `HAVING` predicate, potentially returning incomplete results. [#73610](https://github.com/StarRocks/starrocks/pull/73610) * Data race on the spill writer's `auto_flush` flag when entering parallel merge mode caused unintended segment flushes on ARM. [#73616](https://github.com/StarRocks/starrocks/pull/73616) * Routine load scheduler held a per-job write lock while making blocking BE RPCs to fetch Kafka or Pulsar partition metadata, causing lock-hold times of up to 33.6 s. [#73591](https://github.com/StarRocks/starrocks/pull/73591) * Colocate tablets on dead BEs were incorrectly reported as `HEALTHY` when `tablet_sched_disable_colocate_balance` was enabled. [#73550](https://github.com/StarRocks/starrocks/pull/73550) * `ADMIN SHOW REPLICA STATUS` desynchronized the MySQL result stream when a MISSING (phantom) replica row was present, causing client hangs or disconnects. [#74393](https://github.com/StarRocks/starrocks/pull/74393) * Per-partition coordinator claim was not re-recorded on every sender's `open` RPC in shared-data mode, causing some senders to be missed in the coordinator election and leaving `combined_txn_log` files unwritten. [#73962](https://github.com/StarRocks/starrocks/pull/73962) * `_statistics_.pipe_file_list` internal table was not re-created after the `_statistics_` database or table was dropped. [#73970](https://github.com/StarRocks/starrocks/pull/73970) * Task runs force-killed by `TaskCleaner` were not archived, causing them to disappear from `information_schema.task_runs` with no trace. [#74146](https://github.com/StarRocks/starrocks/pull/74146) * `RENAME TABLE` and `SWAP TABLE`/`SWAP MATERIALIZED VIEW` held only an intensive table lock instead of a database write lock, allowing concurrent readers to observe a torn intermediate name-to-table mapping. [#74100](https://github.com/StarRocks/starrocks/pull/74100) * PK index compaction output sstables were opened without tablet metadata, causing permanent `metadata is null when loading delvec` failures. [#74037](https://github.com/StarRocks/starrocks/pull/74037) * A partial-update `INSERT` inside an explicit transaction targeting a table already modified in the same transaction silently corrupted data at `COMMIT`. [#74344](https://github.com/StarRocks/starrocks/pull/74344) * `ALTER TABLE` operations incompatible with range distribution (schema change, sort-key change) are now rejected with actionable errors instead of silently corrupting metadata. [#74020](https://github.com/StarRocks/starrocks/pull/74020) * Aggregate functions with type-mismatched children in the optimizer caused incorrect query results. [#74159](https://github.com/StarRocks/starrocks/pull/74159) * `ALTER ROUTINE LOAD` with a reserved-keyword table name wrote an unparseable `origStmt`, causing column mappings to be lost after FE restart. [#74188](https://github.com/StarRocks/starrocks/pull/74188) * IVM `state_union` compatibility check did not recurse into nested types (for example, `ARRAY`), causing `CREATE MATERIALIZED VIEW` to fail for `ARRAY_AGG` IMVs. [#73627](https://github.com/StarRocks/starrocks/pull/73627) * Parquet temporary dictionary-code columns leaked to upper layers when a scan range was fully filtered out, causing downstream type mismatches. [#74452](https://github.com/StarRocks/starrocks/pull/74452) * `CASE WHEN` with mixed float and integer `WHEN` and result types produced invalid JIT IR, causing incorrect results or crashes. [#74382](https://github.com/StarRocks/starrocks/pull/74382) * JIT compilation failure caused use-after-free of `LLVMContext`, resulting in SIGSEGV. [#74396](https://github.com/StarRocks/starrocks/pull/74396) * Background statistics tasks overwrote the session `WAREHOUSE` setting, affecting subsequent user queries on the same connection context. [#74385](https://github.com/StarRocks/starrocks/pull/74385) * `CatalogRecycleBin` stopped erasing entries when no cluster snapshot had ever completed successfully, causing unbounded FE memory growth under heavy `INSERT OVERWRITE` workloads. [#74379](https://github.com/StarRocks/starrocks/pull/74379) * Non-PK replica version holes were not detected by the FE; queries were permanently routed to holed replicas with a frozen `max_version`. [#74408](https://github.com/StarRocks/starrocks/pull/74408) * Data race on `MaterializedIndexMeta.updateSchemaBackendId` (a `HashSet` mutated under a shared read lock) could cause lost entries or set corruption. [#74412](https://github.com/StarRocks/starrocks/pull/74412) * Vacuum watermark was not reported correctly when retain-boundary metadata had already been vacuumed, stalling the `file_bundling` switch-version cleanup. [#74429](https://github.com/StarRocks/starrocks/pull/74429) * Lake vacuum retry used deterministic exponential backoff; decorrelated jitter is now added to spread retries across CNs under S3 throttling. [#74108](https://github.com/StarRocks/starrocks/pull/74108) * Memory accounting in `OlapTableSink` was inflated because RPC requests allocated from the query memory pool were released in the process context. [#73807](https://github.com/StarRocks/starrocks/pull/73807) * Race condition in `TabletSinkSender::_send_chunk_by_node` when automatic partition creation triggered `_incremental_open_node_channel` concurrently. [#73820](https://github.com/StarRocks/starrocks/pull/73820) * UDAF context created by a backport of upstream changes caused a memory leak via `unique_ptr::release`. [#74025](https://github.com/StarRocks/starrocks/pull/74025) * Potential out-of-bounds in partitioned join probe caused by inaccurate memory accounting in `append_selective`. [#74315](https://github.com/StarRocks/starrocks/pull/74315) * `azure_adls2_oauth2_client_endpoint` config field had a typo in its name. [#74581](https://github.com/StarRocks/starrocks/pull/74581) * `StarMgrMetaSyncer` incorrectly reaped range-colocate PACK shard groups as orphans, permanently deleting active shards in shared-data mode. [#74117](https://github.com/StarRocks/starrocks/pull/74117) * Sort-key arity for colocate tablet split was resolved from the base schema instead of the materialized schema for PRIMARY KEY tables and tables without an explicit `ORDER BY`, causing split jobs to complete without reducing tablet size. [#74409](https://github.com/StarRocks/starrocks/pull/74409) * Auto-merge daemon merged pre-split tablets back together when partition data was below the merge threshold, undoing the parallelism benefit of sample-based pre-split. [#74583](https://github.com/StarRocks/starrocks/pull/74583) * db-level UDFs were missing on follower FEs after `RESTORE ... AS ` because the function's `FunctionName.db` still pointed at the source database. [#74313](https://github.com/StarRocks/starrocks/pull/74313) * Wrong CN group was assigned for immutable-partition tablet locations in shared-data `DISTRIBUTED BY RANDOM` CTAS/INSERT when the warehouse had multiple CN groups. [#74316](https://github.com/StarRocks/starrocks/pull/74316) * `NullPointerException` in `StatisticsCalcUtils` when a partition was dropped concurrently during statistics estimation. [#73711](https://github.com/StarRocks/starrocks/pull/73711) * `InformationSchemaDataSource` and `FrontendServiceImpl` metadata RPC handlers held full database READ locks, blocking DDL on unrelated tables. [#73936](https://github.com/StarRocks/starrocks/pull/73936) [#73913](https://github.com/StarRocks/starrocks/pull/73913) * Pipeline operators that flipped their finished state without notifying shared-context observers could stall peer drivers under the event scheduler. [#74055](https://github.com/StarRocks/starrocks/pull/74055) [#74056](https://github.com/StarRocks/starrocks/pull/74056) * Non-root compound predicates in predicate pushdown yielded `NotPushDown` instead of a scan-level EOF, causing `OlapScanNode` to emit no rows when an impossible nested AND branch was present under UNION. [#74218](https://github.com/StarRocks/starrocks/pull/74218) * `BackendLoadStatistic.init` performed an expensive per-replica scan on BEs with a single storage medium; the check is now O(1) for homogeneous-disk BEs. [#73555](https://github.com/StarRocks/starrocks/pull/73555) * Thread-name setting race on data-dir load threads caused noisy `failed to set thread name` warnings on every BE startup. [#73862](https://github.com/StarRocks/starrocks/pull/73862) * Task manager wrote an illegal `RUNNING→RUNNING` edit log, causing task runs to appear stuck in the running map indefinitely. [#73882](https://github.com/StarRocks/starrocks/pull/73882) * PK multi-statement batch transactions did not accumulate `num_rows`, `data_size`, and `num_dels` across composite rowsets, causing incorrect row-count statistics on shared-data Primary Key tables. [#74059](https://github.com/StarRocks/starrocks/pull/74059) * Lake load spill cleanup now uses txn-id-based vacuum-driven reclaim, preventing orphaned spill files after BE crash or OOM. [#73064](https://github.com/StarRocks/starrocks/pull/73064) * PostgreSQL JDBC time values starting from year `0000` caused incorrect type mapping results. [#70842](https://github.com/StarRocks/starrocks/pull/70842) * Null-check was missing before reading `gtid` from a rowset during schema change, causing an NPE crash. [#74855](https://github.com/StarRocks/starrocks/pull/74855) #### 4.1.1[​](#411 "Direct link to 4.1.1") Release Date: May 29, 2026 ##### Behavior Changes[​](#behavior-changes-4 "Direct link to Behavior Changes") * The Hive connector now uses a native C++ Avro scanner instead of the JNI Avro scanner by default. [#73237](https://github.com/StarRocks/starrocks/pull/73237) [#73569](https://github.com/StarRocks/starrocks/pull/73569) * Query rewrite over INCREMENTAL/AUTO materialized views is now disabled, and FORCE refresh and partition refresh are rejected for INCREMENTAL/AUTO materialized views. [#72890](https://github.com/StarRocks/starrocks/pull/72890) [#72336](https://github.com/StarRocks/starrocks/pull/72336) [#71355](https://github.com/StarRocks/starrocks/pull/71355) ##### Improvements[​](#improvements-5 "Direct link to Improvements") * Java UDF/UDAF/UDTF now support more types: STRUCT arguments and return values for UDAF/UDTF, nested ARRAY/MAP types, DATE/DATETIME, DECIMAL, and varargs. [#72911](https://github.com/StarRocks/starrocks/pull/72911) [#72283](https://github.com/StarRocks/starrocks/pull/72283) [#72337](https://github.com/StarRocks/starrocks/pull/72337) [#72208](https://github.com/StarRocks/starrocks/pull/72208) [#68596](https://github.com/StarRocks/starrocks/pull/68596) * Scalar UDFs now support STRUCT arguments. [#72620](https://github.com/StarRocks/starrocks/pull/72620) * Python UDFs now support nested ARRAY/MAP types. [#72210](https://github.com/StarRocks/starrocks/pull/72210) * UDAFs are now loaded and initialized once and reused across queries, reducing per-query overhead. [#72038](https://github.com/StarRocks/starrocks/pull/72038) * Replaced the JNI Avro scanner with a native C++ scanner for the Hive connector, with direct binary decoding and support for `avro.schema.literal` and `avro.schema.url`. [#73237](https://github.com/StarRocks/starrocks/pull/73237) [#73283](https://github.com/StarRocks/starrocks/pull/73283) [#73257](https://github.com/StarRocks/starrocks/pull/73257) [#73569](https://github.com/StarRocks/starrocks/pull/73569) * Supports the Trino `WITH` clause in CTAS statements. [#71960](https://github.com/StarRocks/starrocks/pull/71960) * Completed Iceberg `timestamptz` partition transform support on the sink path. [#73397](https://github.com/StarRocks/starrocks/pull/73397) * Enabled TopN runtime filter pushdown for Iceberg table aggregation. [#72332](https://github.com/StarRocks/starrocks/pull/72332) * Supports Iceberg datetime min/max optimization. [#71870](https://github.com/StarRocks/starrocks/pull/71870) * Allows HDFS HA configuration passthrough in Catalog and BE to support accessing multiple HDFS clusters. [#71521](https://github.com/StarRocks/starrocks/pull/71521) * Added a partition scan number limit for external table queries. [#68480](https://github.com/StarRocks/starrocks/pull/68480) * Fails fast for unsupported Iceberg V3 features. [#70242](https://github.com/StarRocks/starrocks/pull/70242) * Supports `csv.enclose` and `csv.escape` for CSV exports via INSERT INTO FILES. [#71589](https://github.com/StarRocks/starrocks/pull/71589) * Added the `enable_push_down_schema` INSERT property for full schema push-down to `files()`. [#70978](https://github.com/StarRocks/starrocks/pull/70978) * Routine Load jobs are now paused on non-retryable errors (for example, primary key size exceeded). [#71161](https://github.com/StarRocks/starrocks/pull/71161) * Supports join reorder for complex expressions from two children. [#71615](https://github.com/StarRocks/starrocks/pull/71615) * Improved CBO statistics estimation, including MCV/null-fraction propagation for `date_trunc`, `array_map`, CASE WHEN, IS NULL, UNION, and constants. [#72233](https://github.com/StarRocks/starrocks/pull/72233) [#70372](https://github.com/StarRocks/starrocks/pull/70372) [#70221](https://github.com/StarRocks/starrocks/pull/70221) [#70865](https://github.com/StarRocks/starrocks/pull/70865) [#70989](https://github.com/StarRocks/starrocks/pull/70989) [#71000](https://github.com/StarRocks/starrocks/pull/71000) * Improved skew join detection: skew is only detected when all join keys are skewed, and a `force_group_by_skew_eliminate_when_skewed` switch was added to force the skew rule. [#72753](https://github.com/StarRocks/starrocks/pull/72753) [#71382](https://github.com/StarRocks/starrocks/pull/71382) * Supports constant folding for `regexp_replace` in the FE. [#70804](https://github.com/StarRocks/starrocks/pull/70804) * Optimized MIN/MAX on date partition columns with constant partition values. [#69880](https://github.com/StarRocks/starrocks/pull/69880) * Introduced the `SCHEDULE` keyword as a synonym for `ASYNC` in materialized view refresh. [#72329](https://github.com/StarRocks/starrocks/pull/72329) * Supports tablet creation retry for Lake tables in shared-data mode. [#71068](https://github.com/StarRocks/starrocks/pull/71068) * Supports conditional update for Lake column-mode partial update. [#71961](https://github.com/StarRocks/starrocks/pull/71961) * Parallelized partial-update publish, persistent index initialization, and SSTable opening to improve ingestion throughput. [#71652](https://github.com/StarRocks/starrocks/pull/71652) [#71217](https://github.com/StarRocks/starrocks/pull/71217) [#72112](https://github.com/StarRocks/starrocks/pull/72112) [#71145](https://github.com/StarRocks/starrocks/pull/71145) [#72986](https://github.com/StarRocks/starrocks/pull/72986) * Supports DCG file synchronization during shared-nothing to shared-data replication. [#69339](https://github.com/StarRocks/starrocks/pull/69339) * Supports schema evolution for widening VARCHAR length on both key and non-key columns. [#70747](https://github.com/StarRocks/starrocks/pull/70747) * Added the `snapshot_meta.json` marker for cluster snapshot integrity checks. [#71209](https://github.com/StarRocks/starrocks/pull/71209) * Supports LDAP direct bind authentication via a DN pattern. [#71559](https://github.com/StarRocks/starrocks/pull/71559) * Added the `get_query_dump_from_query_id` meta function for easier query troubleshooting. [#72875](https://github.com/StarRocks/starrocks/pull/72875) * Supports auditing queried relations in the audit log. [#71596](https://github.com/StarRocks/starrocks/pull/71596) * Added session variables for MySQL binary result encoding. [#71415](https://github.com/StarRocks/starrocks/pull/71415) * Added metrics for better observability, including `tablet_num` for shared-data clusters, `MemtableIOSpeed`, `staros_shard_count`, and Iceberg metadata-table query metrics. [#71444](https://github.com/StarRocks/starrocks/pull/71444) [#69842](https://github.com/StarRocks/starrocks/pull/69842) [#73096](https://github.com/StarRocks/starrocks/pull/73096) [#70825](https://github.com/StarRocks/starrocks/pull/70825) * Added the FE configuration `deploy_serialization_min_thread_pool_size`. [#72274](https://github.com/StarRocks/starrocks/pull/72274) * Added the `tablet_reshard_enable_tablet_merge` configuration to disable MergeTabletJob creation. [#70906](https://github.com/StarRocks/starrocks/pull/70906) * Eliminated HTTP-server accept thundering-herd via `SO_REUSEPORT`. [#72956](https://github.com/StarRocks/starrocks/pull/72956) * Supports SQL UDF creation via `CREATE FUNCTION ... AS `. [#67558](https://github.com/StarRocks/starrocks/pull/67558) * Supports loading UDFs from S3. [#64541](https://github.com/StarRocks/starrocks/pull/64541) * Added the `uuid_v7` function, which generates time-ordered UUID v7 values. [#67694](https://github.com/StarRocks/starrocks/pull/67694) * Added per-catalog-type query metrics for external catalog observability. [#70533](https://github.com/StarRocks/starrocks/pull/70533) * Supports an explicit skew hint for window functions, automatically optimizing window functions with skewed partition keys by splitting into UNION. [#68739](https://github.com/StarRocks/starrocks/pull/68739) ##### Security[​](#security-5 "Direct link to Security") * \[CVE] Upgraded Netty to 4.1.133.Final. [#72905](https://github.com/StarRocks/starrocks/pull/72905) * \[CVE-2026-42198] \[CVE-2026-5598] Bumped pgjdbc to 42.7.11 (client-side DoS via unbounded SCRAM PBKDF2 iteration count) and BouncyCastle to 1.84 (FrodoKEM private-key leakage). [#72797](https://github.com/StarRocks/starrocks/pull/72797) * \[CVE-2026-32280] \[CVE-2026-32282] Built pprof with go1.25.9 to eliminate Golang CVEs. [#71944](https://github.com/StarRocks/starrocks/pull/71944) [#73545](https://github.com/StarRocks/starrocks/pull/73545) * Upgraded jetty-http to 9.4.58.v20250814. [#71762](https://github.com/StarRocks/starrocks/pull/71762) * Cleaned up Broker dependency CVEs and removed `wildfly-openssl`. [#72184](https://github.com/StarRocks/starrocks/pull/72184) [#71908](https://github.com/StarRocks/starrocks/pull/71908) * Redacted credentials in INSERT INTO FILES error messages. [#71245](https://github.com/StarRocks/starrocks/pull/71245) ##### Bug Fixes[​](#bug-fixes-5 "Direct link to Bug Fixes") The following issues have been fixed: * CN segfault on startup caused by `hash_util` static initialization order. [#71825](https://github.com/StarRocks/starrocks/pull/71825) * CN crash when scanning an empty tablet with physical split enabled. [#70281](https://github.com/StarRocks/starrocks/pull/70281) * BE crash when querying `information_schema.warehouse_queries`. [#72019](https://github.com/StarRocks/starrocks/pull/72019) * SIGFPE in Lake compaction when rowset `num_rows` is zero. [#71742](https://github.com/StarRocks/starrocks/pull/71742) * Division-by-zero in ExecutionDAG fragment connection. [#67918](https://github.com/StarRocks/starrocks/pull/67918) * Graceful-exit crash in SinkBuffer. [#73202](https://github.com/StarRocks/starrocks/pull/73202) * Spillable hash join probe crash. [#72397](https://github.com/StarRocks/starrocks/pull/72397) * Stack-buffer-overflow when formatting into a temporary `std::string`. [#72728](https://github.com/StarRocks/starrocks/pull/72728) * Crash in `reverse(DecimalV3)`. [#71834](https://github.com/StarRocks/starrocks/pull/71834) * Use-after-free in `LoadChannel::get_load_replica_status` caused by temporary `shared_ptr` destruction. [#71843](https://github.com/StarRocks/starrocks/pull/71843) * Use-after-free in `ThreadPool::do_submit` when thread creation fails. [#71276](https://github.com/StarRocks/starrocks/pull/71276) * Hive partition descriptor use-after-free across fragment teardown. [#73176](https://github.com/StarRocks/starrocks/pull/73176) * An information schema sink use-after-free. [#71513](https://github.com/StarRocks/starrocks/pull/71513) * An FE file descriptor leak by reusing HttpClient instances. [#73239](https://github.com/StarRocks/starrocks/pull/73239) * JNI local-reference leak in `JDBCScanner::_init_jdbc_scanner`. [#72913](https://github.com/StarRocks/starrocks/pull/72913) * Memory leak when caching the MV plan context. [#72300](https://github.com/StarRocks/starrocks/pull/72300) * Unexpected memory overuse in local exchange. [#72262](https://github.com/StarRocks/starrocks/pull/72262) * Race on `response->tablet_metas` in Lake `publish_version`. [#73274](https://github.com/StarRocks/starrocks/pull/73274) * Concurrent `SegmentFlushTask` race in `DeltaWriter::commit()`. [#73371](https://github.com/StarRocks/starrocks/pull/73371) * `RuntimeProfile` min/max race during serialization. [#72904](https://github.com/StarRocks/starrocks/pull/72904) * Race condition in `PipelineTimerTask` during query context destruction. [#73082](https://github.com/StarRocks/starrocks/pull/73082) * Race condition in `_all_global_rf_ready_or_timeout`. [#70920](https://github.com/StarRocks/starrocks/pull/70920) * Shared `NullColumn` issue in `map_apply` and `array_length`. [#71258](https://github.com/StarRocks/starrocks/pull/71258) * Batch-publish deadlock caused by a partition version gap. [#71483](https://github.com/StarRocks/starrocks/pull/71483) * Deadlock when warming up the LRU cache for rowset metadata in shared-nothing mode. [#71459](https://github.com/StarRocks/starrocks/pull/71459) * `Locker` rollback is not exception-safe and unlock order is incorrect. [#72789](https://github.com/StarRocks/starrocks/pull/72789) * Lock contention with DDL and StarOS RPCs caused by several DB locks on read-only and metadata paths. [#73067](https://github.com/StarRocks/starrocks/pull/73067) [#72475](https://github.com/StarRocks/starrocks/pull/72475) [#72108](https://github.com/StarRocks/starrocks/pull/72108) [#72218](https://github.com/StarRocks/starrocks/pull/72218) [#72178](https://github.com/StarRocks/starrocks/pull/72178) * Incorrect shuffle distribution due to a missing project node. [#71075](https://github.com/StarRocks/starrocks/pull/71075) * An AGG TopN runtime filter `exprOrder` mismatch causing crashes and wrong results. [#71479](https://github.com/StarRocks/starrocks/pull/71479) * Wrong results from dict-merge GROUP BY. [#70866](https://github.com/StarRocks/starrocks/pull/70866) * Query cache conflicts with local shuffle aggregation. [#73194](https://github.com/StarRocks/starrocks/pull/73194) * Inconsistent global dictionary generation in flat JSON. [#72953](https://github.com/StarRocks/starrocks/pull/72953) * Flat JSON merge empty inconsistency. [#72973](https://github.com/StarRocks/starrocks/pull/72973) * A type mismatch in map literals when explicit key/value types are declared. [#71316](https://github.com/StarRocks/starrocks/pull/71316) * COALESCE children is not casted to a common type in the JOIN USING transformer. [#72338](https://github.com/StarRocks/starrocks/pull/72338) * VARCHAR length is not preserved after reduce-cast with global variables. [#70269](https://github.com/StarRocks/starrocks/pull/70269) * VARBINARY is incorrectly encoded inside nested types in MySQL result sets. [#71346](https://github.com/StarRocks/starrocks/pull/71346) * Check-having-clause issue when disabling aggregation spill on small LIMIT. [#72705](https://github.com/StarRocks/starrocks/pull/72705) * Quotes are not stripped before date parsing, and PostgreSQL date/time bug. [#48517](https://github.com/StarRocks/starrocks/pull/48517) [#71016](https://github.com/StarRocks/starrocks/pull/71016) * Data file shared-flag loss that caused vacuum to delete files still referenced by sibling split tablets. [#71585](https://github.com/StarRocks/starrocks/pull/71585) * Tablet merge correctness for the split→compaction→merge sequence. [#72350](https://github.com/StarRocks/starrocks/pull/72350) * Cross-published txn log num\_rows/data\_size inflation during tablet splits. [#71144](https://github.com/StarRocks/starrocks/pull/71144) * Delvec orphan entries caused by write-before-compaction in the same publish batch. [#71001](https://github.com/StarRocks/starrocks/pull/71001) * "no queryable replica" on follower FE by syncing the StarMgr journal replay. [#71263](https://github.com/StarRocks/starrocks/pull/71263) * `merge_condition` is not preserved when applying a normal rowset commit. [#72542](https://github.com/StarRocks/starrocks/pull/72542) * Iceberg DELETE conflict detection using an incorrect snapshot ID and filter. [#73354](https://github.com/StarRocks/starrocks/pull/73354) * An NPE on invalid Iceberg transform arguments. [#71917](https://github.com/StarRocks/starrocks/pull/71917) * Iceberg min/max optimization being skipped due to extra columns injected by the planner. [#71863](https://github.com/StarRocks/starrocks/pull/71863) * Aggregate-join-pushdown MV rewrite on Iceberg base tables. [#71856](https://github.com/StarRocks/starrocks/pull/71856) * Missing Hive partition directory before INSERT OVERWRITE commit. [#71810](https://github.com/StarRocks/starrocks/pull/71810) * AWS assume-role not being applied for the JNI scanner. [#71422](https://github.com/StarRocks/starrocks/pull/71422) * Avro complex-type decoding for pruned children and nested nullable schemas. [#73474](https://github.com/StarRocks/starrocks/pull/73474) * File/column/row context is missing in Parquet Broker Load errors. [#73236](https://github.com/StarRocks/starrocks/pull/73236) * Lacking support for Arrow dictionary values in the Parquet scanner. [#71855](https://github.com/StarRocks/starrocks/pull/71855) * The Primary Key for Paimon tables is not shown in SHOW CREATE and DESC returns. [#70535](https://github.com/StarRocks/starrocks/pull/70535) * PostgreSQL/Oracle JDBC type compatibility and JDBC URL construction with trailing slashes. [#70626](https://github.com/StarRocks/starrocks/pull/70626) [#70992](https://github.com/StarRocks/starrocks/pull/70992) * Materialized view refresh issue with SQL Server tables in a JDBC catalog. [#72962](https://github.com/StarRocks/starrocks/pull/72962) * Lazy-materialization slot nullability issue for materialized views over outer joins. [#72621](https://github.com/StarRocks/starrocks/pull/72621) * Rejected AUTO and INCREMENTAL materialized view partition refresh. [#71355](https://github.com/StarRocks/starrocks/pull/71355) * The materialized view scheduler is not stopped after a materialized view becomes inactive. [#71265](https://github.com/StarRocks/starrocks/pull/71265) * Lacking support for `SHOW GRANTS FOR CURRENT_USER()` for MySQL client compatibility. [#71959](https://github.com/StarRocks/starrocks/pull/71959) * SHOW statements are not allowed inside an explicit transaction. [#72954](https://github.com/StarRocks/starrocks/pull/72954) * Arrow Flight returning the column name `r` for an empty result set. [#71534](https://github.com/StarRocks/starrocks/pull/71534) * Lacking JNI exception-handling checks in Java UDF code. [#71734](https://github.com/StarRocks/starrocks/pull/71734) * `ai_query` function registration issue. [#72103](https://github.com/StarRocks/starrocks/pull/72103) * Stream Load profile collection issue when `enable_load_profile` is used. [#71952](https://github.com/StarRocks/starrocks/pull/71952) * Profile START\_TIME/END\_TIME is not displayed with the session timezone. [#71429](https://github.com/StarRocks/starrocks/pull/71429) * `star_mgr_meta_sync_interval_sec` is not runtime mutable. [#71675](https://github.com/StarRocks/starrocks/pull/71675) * `information_schema.tables` not escaping special characters in equality predicates. [#71273](https://github.com/StarRocks/starrocks/pull/71273) * Use-after-free in parallel segment/rowset loading on the error path. [#71083](https://github.com/StarRocks/starrocks/pull/71083) * Potential hash table data loss in aggregation spill `set_finishing`. [#70851](https://github.com/StarRocks/starrocks/pull/70851) * PK tablet rowset meta loss caused by a GC race during disk re-migration (A→B→A). [#70727](https://github.com/StarRocks/starrocks/pull/70727) * DB read lock leak in `SharedDataStorageVolumeMgr`. [#70987](https://github.com/StarRocks/starrocks/pull/70987) * IVM refresh recording incomplete PCT partition metadata. [#71092](https://github.com/StarRocks/starrocks/pull/71092) * NPE when analyzing generated columns in Stream Load/Broker Load if a referenced column is missing. [#71116](https://github.com/StarRocks/starrocks/pull/71116) * Missing partition predicate in short-circuit point lookup. [#71124](https://github.com/StarRocks/starrocks/pull/71124) #### 4.1.0[​](#410 "Direct link to 4.1.0") Release Date: April 13, 2026 ##### Shared-data Architecture[​](#shared-data-architecture "Direct link to Shared-data Architecture") * **New Multi-Tenant Data Management** Shared-data clusters now support range-based data distribution and automatic splitting and merging of tablets. Tablets can be automatically split when they become oversized or hotspots, without requiring schema changes, SQL modifications, or data re-ingestion. This feature can significantly improve usability, directly addressing data skew and hotspot issues in multi-tenant workloads. [#65199](https://github.com/StarRocks/starrocks/pull/65199) [#66342](https://github.com/StarRocks/starrocks/pull/66342) [#67056](https://github.com/StarRocks/starrocks/pull/67056) [#67386](https://github.com/StarRocks/starrocks/pull/67386) [#68342](https://github.com/StarRocks/starrocks/pull/68342) [#68569](https://github.com/StarRocks/starrocks/pull/68569) [#66743](https://github.com/StarRocks/starrocks/pull/66743) [#67441](https://github.com/StarRocks/starrocks/pull/67441) [#68497](https://github.com/StarRocks/starrocks/pull/68497) [#68591](https://github.com/StarRocks/starrocks/pull/68591) [#66672](https://github.com/StarRocks/starrocks/pull/66672) [#69155](https://github.com/StarRocks/starrocks/pull/69155) * **Large-Capacity Tablet Support (Phase 1)** Enables shared-data clusters to host significantly more data per tablet, with a long-term target of 100 GB per tablet. Phase 1 introduces intra-tablet parallelism across the entire ingestion, Primary Key update, and Compaction pipeline, so a single Lake tablet no longer becomes a single-threaded bottleneck as it grows. Improvements include parallel Compaction within a single tablet (with segment-level splitting), parallel MemTable finalization, flush, and merge for Lake loading (including the load-spill path), tablet-internal parallel publish and parallel condition updates for Primary Key tables, and range-split / parallel / size-tiered Compaction for the cloud-native Primary Key index with remote-storage mapper file support. Together, these changes substantially reduce ingestion memory overhead, compaction amplification, and FE metadata pressure for large-tablet workloads. [#66424](https://github.com/StarRocks/starrocks/pull/66424) [#66522](https://github.com/StarRocks/starrocks/pull/66522) [#66778](https://github.com/StarRocks/starrocks/pull/66778) [#66586](https://github.com/StarRocks/starrocks/pull/66586) [#67432](https://github.com/StarRocks/starrocks/pull/67432) [#67478](https://github.com/StarRocks/starrocks/pull/67478) [#67554](https://github.com/StarRocks/starrocks/pull/67554) [#66796](https://github.com/StarRocks/starrocks/pull/66796) [#67392](https://github.com/StarRocks/starrocks/pull/67392) [#67878](https://github.com/StarRocks/starrocks/pull/67878) [#65908](https://github.com/StarRocks/starrocks/pull/65908) [#68677](https://github.com/StarRocks/starrocks/pull/68677) [#68123](https://github.com/StarRocks/starrocks/pull/68123) [#69865](https://github.com/StarRocks/starrocks/pull/69865) * **Fast Schema Evolution V2** Shared-data clusters now support Fast Schema Evolution V2, which enables second-level DDL execution for schema operations, and further extends the support to materialized views. [#65726](https://github.com/StarRocks/starrocks/pull/65726) [#66774](https://github.com/StarRocks/starrocks/pull/66774) [#67915](https://github.com/StarRocks/starrocks/pull/67915) * **\[Beta] Inverted Index on shared-data** Enables built-in inverted indexes for shared-data clusters to accelerate text filtering and full-text search workloads. [#66541](https://github.com/StarRocks/starrocks/pull/66541) * **Cache Observability** Query-level cache hit ratio is now exposed in audit logs and the monitoring system for better cache transparency and latency diagnosis. Additional Data Cache metrics include memory and disk quota usage, and page cache statistics. [#63964](https://github.com/StarRocks/starrocks/pull/63964) * Added segment metadata filter for Lake tables to skip irrelevant segments based on sort key range during scans, reducing I/O for range-predicate queries. [#68124](https://github.com/StarRocks/starrocks/pull/68124) * Supports fast cancel for Lake DeltaWriter, reducing latency for cancelled ingestion jobs in shared-data clusters. [#68877](https://github.com/StarRocks/starrocks/pull/68877) * Added support for interval-based scheduling for automated cluster snapshots. [#67525](https://github.com/StarRocks/starrocks/pull/67525) * Supports pipeline execution for MemTable flush and merge, improving ingestion throughput for cloud-native tables in shared-data clusters. [#67878](https://github.com/StarRocks/starrocks/pull/67878) * Supports `dry_run` mode for repairing cloud-native tables, allowing users to preview repair actions before execution. [#68494](https://github.com/StarRocks/starrocks/pull/68494) * Added a thread pool for publish transactions in shared-nothing clusters, improving publish throughput. [#67797](https://github.com/StarRocks/starrocks/pull/67797) ##### Data Lake Analytics[​](#data-lake-analytics "Direct link to Data Lake Analytics") * **Iceberg DELETE Support** Supports writing position delete files for Iceberg tables, enabling DELETE operations on Iceberg tables directly from StarRocks. The support covers the full pipeline of Plan, Sink, Commit, and Audit. [#67259](https://github.com/StarRocks/starrocks/pull/67259) [#67277](https://github.com/StarRocks/starrocks/pull/67277) [#67421](https://github.com/StarRocks/starrocks/pull/67421) [#67567](https://github.com/StarRocks/starrocks/pull/67567) * **TRUNCATE for Hive and Iceberg Tables** Supports TRUNCATE TABLE on external Hive and Iceberg tables. [#64768](https://github.com/StarRocks/starrocks/pull/64768) [#65016](https://github.com/StarRocks/starrocks/pull/65016) * **Incremental materialized view on Iceberg** Extends the support for incremental materialized view refresh to Iceberg append-only tables, enabling query acceleration without full table refresh. [#65469](https://github.com/StarRocks/starrocks/pull/65469) [#62699](https://github.com/StarRocks/starrocks/pull/62699) * **VARIANT Type for Semi-Structured Data in Iceberg** Supports the VARIANT data type in Iceberg Catalog for flexible, schema-on-read storage and querying of semi-structured data. Supports read, write, type casting, and Parquet integration. [#63639](https://github.com/StarRocks/starrocks/pull/63639) [#66539](https://github.com/StarRocks/starrocks/pull/66539) * **Iceberg v3 Support** Added support for Iceberg v3 default value feature and row lineage. [#69525](https://github.com/StarRocks/starrocks/pull/69525) [#69633](https://github.com/StarRocks/starrocks/pull/69633) * **Iceberg Table Maintenance Procedures** Added support for `rewrite_manifests` procedure and extended `expire_snapshots` and `remove_orphan_files` procedures with additional arguments for finer-grained table maintenance. [#68817](https://github.com/StarRocks/starrocks/pull/68817) [#68898](https://github.com/StarRocks/starrocks/pull/68898) * Supports reading file path and row position metadata columns from Iceberg tables. [#67003](https://github.com/StarRocks/starrocks/pull/67003) * Supports reading `_row_id` from Iceberg v3 tables, and supports global late materialization for Iceberg v3. [#62318](https://github.com/StarRocks/starrocks/pull/62318) [#64133](https://github.com/StarRocks/starrocks/pull/64133) * Supports creating Iceberg views with custom properties, and displays properties in SHOW CREATE VIEW output. [#65938](https://github.com/StarRocks/starrocks/pull/65938) * Supports querying Paimon tables with a specific branch, tag, version, or timestamp. [#63316](https://github.com/StarRocks/starrocks/pull/63316) * Supports complex types (ARRAY, MAP, STRUCT) for Paimon tables. [#66784](https://github.com/StarRocks/starrocks/pull/66784) * Supports Partition Transforms with parentheses syntax when creating Iceberg tables. [#68945](https://github.com/StarRocks/starrocks/pull/68945) * Supports Iceberg global shuffle based on Transform Partition for improved data organization. [#70009](https://github.com/StarRocks/starrocks/pull/70009) * Supports dynamically enabling global shuffle for Iceberg table sink. [#67442](https://github.com/StarRocks/starrocks/pull/67442) * Introduced a Commit queue for Iceberg table sink to avoid concurrent Commit conflicts. [#68084](https://github.com/StarRocks/starrocks/pull/68084) * Added host-level sorting for Iceberg table sink to improve data organization and reading performance. [#68121](https://github.com/StarRocks/starrocks/pull/68121) * Enabled additional optimizations in ETL execution mode by default, improving performance for INSERT INTO SELECT, CREATE TABLE AS SELECT, and similar batch operations without explicit configuration. [#66841](https://github.com/StarRocks/starrocks/pull/66841) * Added commit audit information for INSERT and DELETE operations on Iceberg tables. [#69198](https://github.com/StarRocks/starrocks/pull/69198) * Supports enabling or disabling view endpoint operations in Iceberg REST Catalog. [#66083](https://github.com/StarRocks/starrocks/pull/66083) * Optimized cache lookup efficiency in CachingIcebergCatalog. [#66388](https://github.com/StarRocks/starrocks/pull/66388) * Supports EXPLAIN on various Iceberg catalog types. [#66563](https://github.com/StarRocks/starrocks/pull/66563) * Supports partition projection for tables in AWS Glue Catalog tables. [#67601](https://github.com/StarRocks/starrocks/pull/67601) * Added resource share type support for AWS Glue `GetDatabases` API. [#69056](https://github.com/StarRocks/starrocks/pull/69056) * Supports Azure ABFS/WASB path mapping with endpoint injection (`azblob`/`adls2`). [#67847](https://github.com/StarRocks/starrocks/pull/67847) * Added a database metadata cache for JDBC catalog to reduce remote RPC overhead and impact of external system failures. [#68256](https://github.com/StarRocks/starrocks/pull/68256) * Supports column comments for PostgreSQL tables in `information_schema`. [#70520](https://github.com/StarRocks/starrocks/pull/70520) * Improved Oracle and PostgreSQL JDBC type mapping. [#70315](https://github.com/StarRocks/starrocks/pull/70315) [#70566](https://github.com/StarRocks/starrocks/pull/70566) ##### Query Engine[​](#query-engine "Direct link to Query Engine") * **Recursive CTE** Supports Recursive Common Table Expressions (CTEs) for hierarchical traversals, graph queries, and iterative SQL computations. [#65932](https://github.com/StarRocks/starrocks/pull/65932) * Improved Skew Join v2 rewrite with statistics-based skew detection, histogram support, and NULL-skew awareness. [#68680](https://github.com/StarRocks/starrocks/pull/68680) [#68886](https://github.com/StarRocks/starrocks/pull/68886) * Improved COUNT DISTINCT over windows and added support for fused multi-distinct aggregations. [#67453](https://github.com/StarRocks/starrocks/pull/67453) * Supports explicit skew hint for window functions, with automatic optimization of window functions with skewed partition keys by splitting into UNION. [#67944](https://github.com/StarRocks/starrocks/pull/67944) * Supports EXPLAIN and EXPLAIN ANALYZE for INSERT statements in Trino Parser. [#70174](https://github.com/StarRocks/starrocks/pull/70174) * Supports EXPLAIN for query queue visibility. [#69933](https://github.com/StarRocks/starrocks/pull/69933) ##### Functions and SQL Syntax[​](#functions-and-sql-syntax "Direct link to Functions and SQL Syntax") * Added the following functions: * `array_top_n`: Returns the top N elements from an array ranked by value. [#63376](https://github.com/StarRocks/starrocks/pull/63376) * `arrays_zip`: Combines multiple arrays element-wise into an array of structs. [#65556](https://github.com/StarRocks/starrocks/pull/65556) * `json_pretty`: Formats a JSON string with indentation. [#66695](https://github.com/StarRocks/starrocks/pull/66695) * `json_set`: Sets a value at a specified path within a JSON string. [#66193](https://github.com/StarRocks/starrocks/pull/66193) * `initcap`: Converts the first letter of each word to uppercase. [#66837](https://github.com/StarRocks/starrocks/pull/66837) * `sum_map`: Sums MAP values across rows with the same key. [#67482](https://github.com/StarRocks/starrocks/pull/67482) * `current_timezone`: Returns the current session timezone. [#63653](https://github.com/StarRocks/starrocks/pull/63653) * `current_warehouse`: Returns the name of the current warehouse. [#66401](https://github.com/StarRocks/starrocks/pull/66401) * `sec_to_time`: Converts the number of seconds to a TIME value. [#62797](https://github.com/StarRocks/starrocks/pull/62797) * `ai_query`: Calls an external AI model from SQL for inference workloads. [#61583](https://github.com/StarRocks/starrocks/pull/61583) * `raise_error`: Raises a user-defined error in SQL expressions. [#69661](https://github.com/StarRocks/starrocks/pull/69661) * Provides the following function or syntactic extensions: * Supports a lambda comparator in `array_sort` for custom sort ordering. [#66607](https://github.com/StarRocks/starrocks/pull/66607) * Supports USING clause for FULL OUTER JOIN with SQL-standard semantics. [#65122](https://github.com/StarRocks/starrocks/pull/65122) * Supports DISTINCT aggregation over framed window functions with ORDER BY/PARTITION BY. [#65815](https://github.com/StarRocks/starrocks/pull/65815) [#65030](https://github.com/StarRocks/starrocks/pull/65030) [#67453](https://github.com/StarRocks/starrocks/pull/67453) * Supports ARRAY type in `lead`/`lag`/`first_value`/`last_value` window functions. [#63547](https://github.com/StarRocks/starrocks/pull/63547) * Supports VARBINARY for count distinct-like aggregate functions. [#68442](https://github.com/StarRocks/starrocks/pull/68442) * Supports date and string type casting in IN expressions. [#61746](https://github.com/StarRocks/starrocks/pull/61746) * Supports WITH LABEL syntax for BEGIN/START TRANSACTION. [#68320](https://github.com/StarRocks/starrocks/pull/68320) * Supports WHERE/ORDER/LIMIT clauses in SHOW statements. [#68834](https://github.com/StarRocks/starrocks/pull/68834) * Supports `ALTER TASK` statements for task management. [#68675](https://github.com/StarRocks/starrocks/pull/68675) * Supports multiple compression formats (GZIP/SNAPPY/ZSTD/LZ4/DEFLATE/ZLIB/BZIP2) for CSV file exports. [#68054](https://github.com/StarRocks/starrocks/pull/68054) * Supports `STRUCT_CAST_BY_NAME` SQL mode for name-based struct field matching. [#69845](https://github.com/StarRocks/starrocks/pull/69845) ##### Management & Observability[​](#management--observability "Direct link to Management & Observability") * Supports `warehouses`, `cpu_weight_percent`, and `exclusive_cpu_weight` attributes for resource groups to improve multi-warehouse CPU resource isolation. [#66947](https://github.com/StarRocks/starrocks/pull/66947) * Introduces the `information_schema.fe_threads` system view to inspect the FE thread state. [#65431](https://github.com/StarRocks/starrocks/pull/65431) * Supports SQL Digest Blacklist to block specific query patterns at the cluster level. [#66499](https://github.com/StarRocks/starrocks/pull/66499) * Supports Arrow Flight Data Retrieval from nodes that are otherwise inaccessible due to network topology constraints. [#66348](https://github.com/StarRocks/starrocks/pull/66348) * Introduces the REFRESH CONNECTIONS command to propagate global variable changes to existing connections without reconnecting. [#64964](https://github.com/StarRocks/starrocks/pull/64964) * Added built-in UI functions to analyze query profiles and view formatted SQL, making query tuning more accessible. [#63867](https://github.com/StarRocks/starrocks/pull/63867) * Implements `ClusterSummaryActionV2` API endpoint to provide a structured cluster overview. [#68836](https://github.com/StarRocks/starrocks/pull/68836) * Added a global read-only system variable `@@run_mode` to query the current cluster run mode (shared-data or shared-nothing). [#69247](https://github.com/StarRocks/starrocks/pull/69247) * Enabled `query_queue_v2` by default for improved query queue management. [#67462](https://github.com/StarRocks/starrocks/pull/67462) * Supports user-level default warehouse for Stream Load and Merge Commit operations. [#68106](https://github.com/StarRocks/starrocks/pull/68106) [#68616](https://github.com/StarRocks/starrocks/pull/68616) * Added `skip_black_list` session variable to bypass backend blacklist verification when needed. [#67467](https://github.com/StarRocks/starrocks/pull/67467) * Added `enable_table_metrics_collect` option for the metrics API. [#68691](https://github.com/StarRocks/starrocks/pull/68691) * Added impersonate user support for query detail HTTP API. [#68674](https://github.com/StarRocks/starrocks/pull/68674) * Added `table_query_timeout` as a table-level property. [#67547](https://github.com/StarRocks/starrocks/pull/67547) * Supports adding FE observer nodes. [#67778](https://github.com/StarRocks/starrocks/pull/67778) * Supports Merge Commit information in `information_schema.loads` for better load job visibility. [#67879](https://github.com/StarRocks/starrocks/pull/67879) * Supports showing tablet status in cloud-native tables for better troubleshooting. [#69616](https://github.com/StarRocks/starrocks/pull/69616) ##### Security[​](#security-6 "Direct link to Security") * \[CVE-2026-33870] \[CVE-2026-33871] Replaced AWS bundle and bumped Netty to 4.1.132.Final. [#71017](https://github.com/StarRocks/starrocks/pull/71017) * \[CVE-2025-27821] Upgraded Hadoop to v3.4.2. [#68529](https://github.com/StarRocks/starrocks/pull/68529) * \[CVE-2025-54920] Upgraded `spark-core_2.12` to 3.5.7. [#70862](https://github.com/StarRocks/starrocks/pull/70862) ##### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") The following issues have been fixed: * Fixed data loss after tablet split by skipping data file deletion for range distribution tablets. [#71135](https://github.com/StarRocks/starrocks/pull/71135) * Fixed a memory leak in `DefaultValueColumnIterator` for complex types. [#71142](https://github.com/StarRocks/starrocks/pull/71142) * Fixed a memory leak caused by `shared_ptr` cycle between `BatchUnit` and `FetchTaskContext`. [#71126](https://github.com/StarRocks/starrocks/pull/71126) * Fixed double-free crash in SystemMetrics due to concurrent getline access. [#71040](https://github.com/StarRocks/starrocks/pull/71040) * Fixed crash in SpillMemTableSink when eager merge consumes all blocks. [#69046](https://github.com/StarRocks/starrocks/pull/69046) * Fixed NPE when auto-created partition is dropped by TTL cleaner. [#68257](https://github.com/StarRocks/starrocks/pull/68257) * Fixed NPE in `IcebergCatalog.getPartitionLastUpdatedTime` when snapshot is expired. [#68925](https://github.com/StarRocks/starrocks/pull/68925) * Fixed incorrect predicate rewrite for outer join with constant-side column reference. [#67072](https://github.com/StarRocks/starrocks/pull/67072) * Fixed error query results after modify CHAR column length in shared-data. [#68808](https://github.com/StarRocks/starrocks/pull/68808) * Fixed MV refresh bug in the case of multiple tables. [#61763](https://github.com/StarRocks/starrocks/pull/61763) * Fixed incorrect MV recycle time if force refreshed. [#68673](https://github.com/StarRocks/starrocks/pull/68673) * Fixed all-null value handling bug in sync MV. [#69136](https://github.com/StarRocks/starrocks/pull/69136) * Fixed duplicate column id error when querying MV after fast schema change ADD COLUMN. [#71072](https://github.com/StarRocks/starrocks/pull/71072) * Fixed low-cardinality rewrite NPE caused by shared DecodeInfo. [#68799](https://github.com/StarRocks/starrocks/pull/68799) * Fixed low-cardinality join predicate type mismatch. [#68568](https://github.com/StarRocks/starrocks/pull/68568) * Fixed Segfault in Parquet Page Index Filter when `null_counts` empty. [#68463](https://github.com/StarRocks/starrocks/pull/68463) * Fixed JSON flatten array and object conflict on identical paths. [#68804](https://github.com/StarRocks/starrocks/pull/68804) * Fixed Iceberg cache weigher inaccuracies. [#69058](https://github.com/StarRocks/starrocks/pull/69058) * Fixed Iceberg table cache memory limit. [#67769](https://github.com/StarRocks/starrocks/pull/67769) * Fixed Iceberg delete column nullability issue. [#68649](https://github.com/StarRocks/starrocks/pull/68649) * Fixed Azure ABFS/WASB FileSystem cache key to include container. [#68901](https://github.com/StarRocks/starrocks/pull/68901) * Fixed deadlock when the HMS connection pool is full. [#68033](https://github.com/StarRocks/starrocks/pull/68033) * Fixed incorrect length for VARCHAR field type in Paimon Catalog. [#68383](https://github.com/StarRocks/starrocks/pull/68383) * Fixed Paimon catalog refresh crash with ClassCastException on ObjectTable. [#70224](https://github.com/StarRocks/starrocks/pull/70224) * Fixed FULL OUTER JOIN USING with constant subqueries. [#69028](https://github.com/StarRocks/starrocks/pull/69028) * Fixed join on clause bug with CTE scope. [#68809](https://github.com/StarRocks/starrocks/pull/68809) * Fixed ConnectContext memory leaks by using bindScope() pattern. [#68215](https://github.com/StarRocks/starrocks/pull/68215) * Fixed memory leak in `CatalogRecycleBin.asyncDeleteForTables` for shared-nothing clusters. [#68275](https://github.com/StarRocks/starrocks/pull/68275) * Fixed Thrift accept thread from exiting when it encounters any exception. [#68644](https://github.com/StarRocks/starrocks/pull/68644) * Fixed UDF resolution in routine load column mappings. [#68201](https://github.com/StarRocks/starrocks/pull/68201) * Fixed `DROP FUNCTION IF EXISTS` ignoring `ifExists` flag. [#69216](https://github.com/StarRocks/starrocks/pull/69216) * Fixed scan result error when dict page is too large. [#68258](https://github.com/StarRocks/starrocks/pull/68258) * Fixed range partition overlap. [#68255](https://github.com/StarRocks/starrocks/pull/68255) * Fixed query queue allocation time and pending timeout. [#65802](https://github.com/StarRocks/starrocks/pull/65802) * Fixed `array_map` crash when processing null literal array. [#70629](https://github.com/StarRocks/starrocks/pull/70629) * Fixed stack overflow for `to_base64`. [#70623](https://github.com/StarRocks/starrocks/pull/70623) * Fixed case-insensitive username normalization for LDAP authentication. [#67966](https://github.com/StarRocks/starrocks/pull/67966) * Mitigated SSRF risk for API `proc_file`. [#68997](https://github.com/StarRocks/starrocks/pull/68997) * Masked user auth strings in audit and SQL redaction. [#70360](https://github.com/StarRocks/starrocks/pull/70360) ##### Behavior Changes[​](#behavior-changes-5 "Direct link to Behavior Changes") * ETL execution mode optimizations are now enabled by default. This benefits INSERT INTO SELECT, CREATE TABLE AS SELECT, and similar batch workloads without explicit configuration changes. [#66841](https://github.com/StarRocks/starrocks/pull/66841) * The third argument of `lag`/`lead` window functions now supports column references in addition to constant values. [#60209](https://github.com/StarRocks/starrocks/pull/60209) * FULL OUTER JOIN USING now follows SQL-standard semantics: the USING column appears once in the output instead of twice. [#65122](https://github.com/StarRocks/starrocks/pull/65122) * `query_queue_v2` is now enabled by default. [#67462](https://github.com/StarRocks/starrocks/pull/67462) * SQL transactions are gated behind the session variable `enable_sql_transaction` by default. [#63535](https://github.com/StarRocks/starrocks/pull/63535) * Previously, the FE configuration item `max_scalar_operator_flat_children` (default `10000`) limited the node count only of expressions built by `CASE WHEN`. It now limits every scalar expression that the optimizer rewrites, and the node count is recalculated after each rewrite instead of reusing a cached value. As a result, after an upgrade a query that ran before can be rejected in the analysis phase with `Expression too complex. Current nodes: N, Limit: M.`, typically when inlining a view, a CTE, or a subquery expands its expression tree. Simplify the expression, or increase `max_scalar_operator_flat_children`, which can be modified dynamically. [#66324](https://github.com/StarRocks/starrocks/pull/66324) --- ## Search ### Search the documentation [Skip to main content](#__docusaurus_skipToContent_fallback) 🎉️ **[NEXT QUERY is coming November 4th. Join us online!](https://nextquery.phoenixdata.ai/?utm_source=website\&utm_medium=website\&utm_campaign=FY26_PHX_OEV_NEXT_QUERY_2026\&utm_content=starrocks-doc)** 🎉️ [![StarRocks Logo](/img/logo.svg)![StarRocks Logo](/img/logo.svg)](https://www.starrocks.io/) [**StarRocks**](https://www.starrocks.io/)[Docs](https://docs.starrocks.io/docs/introduction) [NEXT QUERY 2026](https://nextquery.phoenixdata.ai/?utm_source=website\&utm_medium=website\&utm_campaign=FY26_PHX_OEV_NEXT_QUERY_2026\&utm_content=starrocks-doc)[Release Notes](https://docs.starrocks.io/releasenotes/release-4.1.md)[](https://github.com/StarRocks/starrocks) [Community](#) * [Slack](https://docs.starrocks.io/join/) [Privacy policy](https://www.starrocks.io/product/privacy-policy) Search Type your search here Latest-4.1 (4.1) Powered by[](https://www.algolia.com/) * [StarRocks.io](https://www.starrocks.io/) * [Privacy policy](https://www.starrocks.io/product/privacy-policy) Docs built with Docusaurus. ![Rocky the happy otter](/img/rocky.svg)StarRocks Assistant × AI generated answers are based on docs and other sources. Please test answers in non-production environments. SendReset --- ## Spark_connector ### Releases of StarRocks Connector for Spark #### Notifications[​](#notifications "Direct link to Notifications") **User guide:** * [Load data into StarRocks using Spark connector](https://docs.starrocks.io/docs/loading/Spark-connector-starrocks/) * [Read data from StarRocks using Spark connector](https://docs.starrocks.io/docs/unloading/Spark_connector/) **Source codes**: [starrocks-connector-for-apache-spark](https://github.com/StarRocks/starrocks-connector-for-apache-spark) **Naming format of the JAR file**: `starrocks-spark-connector-${spark_version}_${scala_version}-${connector_version}.jar` **Methods to obtain the JAR file:** * Directly download the Spark connector JAR file from the [Maven Central Repository](https://repo1.maven.org/maven2/com/starrocks). * Add the Spark connector as a dependency in your Maven project's `pom.xml` file and download it. For specific instructions, see [user guide](https://docs.starrocks.io/docs/loading/Spark-connector-starrocks/#obtain-spark-connector). * Compile the source codes into Spark connector JAR file. For specific instructions, see [user guide](https://docs.starrocks.io/docs/loading/Spark-connector-starrocks/#obtain-spark-connector). **Version requirements:** | Spark connector | Spark | StarRocks | Java | Scala | | --------------- | ------------------ | -------------- | ---- | ----- | | 1.1.4 | 4.0, 4.1 | 2.5 and later | 17 | 2.13 | | 1.1.4 | 3.3, 3.4, 3.5 | 2.5 and later | 8 | 2.12 | | 1.1.3 | 3.2, 3.3, 3.4, 3.5 | 2.5 and later | 8 | 2.12 | | 1.1.2 | 3.2, 3.3, 3.4, 3.5 | 2.5 and later | 8 | 2.12 | | 1.1.1 | 3.2, 3.3, or 3.4 | 2.5 and later | 8 | 2.12 | | 1.1.0 | 3.2, 3.3, or 3.4 | 2.5 and later | 8 | 2.12 | | 1.0.0 | 3.x | 1.18 and later | 8 | 2.12 | | 1.0.0 | 2.x | 1.18 and later | 8 | 2.11 | #### Release notes[​](#release-notes "Direct link to Release notes") ##### 1.1[​](#11 "Direct link to 1.1") ###### 1.1.4[​](#114 "Direct link to 1.1.4") This release mainly includes some features and improvements. **Features** * Added support for StarRocks catalog. [#109](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/109) * Added support for `bitmap_hash64`. [#130](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/130) * Supports nested fields (Structs, Arrays, and Maps). [#152](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/152) * Supports reading and writing through Spark catalog. [#140](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/140) * Supports Spark 4.0 and Scala 2.13. [#154](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/154) **Improvements** * Added the security policy. [#139](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/139) **BugFix** * Incorrect filter pushdown in DataSource V2. [#141](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/141) ###### 1.1.3[​](#113 "Direct link to 1.1.3") This release mainly includes some features and improvements. **Features** * Sink supports LZ4 compression. [#110](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/110) **Improvements** * Supports reading and writing StarRocks DATETIME data type with microseconds. [#123](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/123) * Supports configuring the socket timeout for write. [#122](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/122) * Enlarged the maximum string length of `jackson-core`. [#129](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/129) **BugFix** * Parse error caused by keywords used as column names. (After the bug fix, quotes are added to column names in BuildScan.) [#103](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/103) * Exception caused by that `java.lang.String` is not a valid external type for DATE/TIMESTAMP. [#111](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/111) * The JSON type was written incorrectly. [#115](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/115) ###### 1.1.2[​](#112 "Direct link to 1.1.2") **Features** * Supports Spark v3.5. [#89](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/89) * Supports the `starrocks.filter.query` parameter when Spark SQL is used to read data from StarRocks. [#92](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/92) * Supports reading columns of JSON type from StarRocks. [#100](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/100) **Improvements** * Optimized error messages. When Spark connector reads data from StarRocks, and columns specified in the `starrocks.columns` parameter do not exist in the StarRocks table, the returned error message explicitly shows the names of columns that do not exist. [#97](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/97) * If an exception occurs when Spark connector requests a query plan from StarRocks FE via HTTP, the FE will return the exception information to the Spark connector through the HTTP status and entity. [#98](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/98) ###### 1.1.1[​](#111 "Direct link to 1.1.1") This release mainly includes some features and improvements for loading data to StarRocks. > **NOTICE** > > Take note of the some changes when you upgrade the Spark connector to this version. For details, see [Upgrade Spark connector](https://docs.starrocks.io/docs/loading/Spark-connector-starrocks/#upgrade-from-version-110-to-111). **Features** * The sink supports retrying. [#61](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/61) * Support to load data to BITMAP and HLL columns. [#67](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/67) * Support to load ARRAY-type data. [#74](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/74) * Support to flush according to the number of buffered rows. [#78](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/78) **Improvements** * Remove useless dependency, and make the Spark connector JAR file lightweight. [#55](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/55) [#57](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/57) * Replace fastjson with jackson. [#58](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/58) * Add the missing Apache license header. [#60](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/60) * Do not package the MySQL JDBC driver in the Spark connector JAR file. [#63](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/63) * Support to configure timezone parameter and become compatible with Spark Java8 API datetime. [#64](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/64) * Optimize row-string converter to reduce CPU costs. [#68](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/68) * The `starrocks.fe.http.url` parameter supports to add a http scheme. [#71](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/71) * The interface BatchWrite#useCommitCoordinator is implemented to run on DataBricks 13.1 [#79](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/79) * Add the hint of checking the privileges and parameters in the error log. [#81](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/81) **Bug fixes** * Parse escape characters in the CSV related parameters `column_separator` and `row_delimiter`. [#85](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/85) **Doc** * Refactor the docs. [#66](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/66) * Add examples of load data to BITMAP and HLL columns. [#70](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/70) * Add examples of Spark applications written in Python. [#72](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/72) * Add examples of loading ARRAY-type data. [#75](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/75) * Add examples for performing partial updates and conditional updates on Primary Key tables. [#80](https://github.com/StarRocks/starrocks-connector-for-apache-spark/pull/80) **1.1.0** **Features** * Support to load data into StarRocks. ##### 1.0[​](#10 "Direct link to 1.0") **Features** * Support to unload data from StarRocks. --- ## Sql Reference ### Reference #### [🗃️ Data types](https://docs.starrocks.io/docs/sql-reference/data-types) [5 items](https://docs.starrocks.io/docs/sql-reference/data-types) #### [🗃️ SQL Functions](https://docs.starrocks.io/docs/sql-reference/sql-functions) [30 items](https://docs.starrocks.io/docs/sql-reference/sql-functions) #### [🗃️ SQL statements](https://docs.starrocks.io/docs/sql-reference/sql-statements.md) [17 items](https://docs.starrocks.io/docs/sql-reference/sql-statements.md) #### [📄️ Error Codes](https://docs.starrocks.io/docs/sql-reference/Error_code.md) [Reference for common StarRocks error codes returned for query requests.](https://docs.starrocks.io/docs/sql-reference/Error_code.md) #### [📄️ System limits](https://docs.starrocks.io/docs/sql-reference/System_limit.md) [System limits and naming rules for StarRocks objects including databases, tables, partitions, columns, and identifiers.](https://docs.starrocks.io/docs/sql-reference/System_limit.md) #### [📄️ System variables](https://docs.starrocks.io/docs/sql-reference/System_variable.md) [StarRocks provides many system variables that can be set and modified to suit your requirements.](https://docs.starrocks.io/docs/sql-reference/System_variable.md) #### [📄️ AWS IAM policies](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md) [A policy in AWS IAM declares a set of permissions on a specific AWS resource.](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md) #### [📄️ HTTP SQL API](https://docs.starrocks.io/docs/sql-reference/http_sql_api.md) [StarRocks v3.2.0 introduces the HTTP SQL API for users to perform various types of queries using HTTP.](https://docs.starrocks.io/docs/sql-reference/http_sql_api.md) #### [🗃️ Information Schema](https://docs.starrocks.io/docs/sql-reference/information_schema.md) [52 items](https://docs.starrocks.io/docs/sql-reference/information_schema.md) #### [🗃️ System metadatabase](https://docs.starrocks.io/docs/sql-reference/sys.md) [6 items](https://docs.starrocks.io/docs/sql-reference/sys.md) #### [📄️ User-defined variables](https://docs.starrocks.io/docs/sql-reference/user_defined_variables.md) [StarRocks 2.5 and later versions support user-defined variables.](https://docs.starrocks.io/docs/sql-reference/user_defined_variables.md) --- ### AWS IAM policies A policy in AWS IAM declares a set of permissions on a specific AWS resource. After creating a policy, you need to attach it to an IAM role or user. As such, the IAM role or user is assigned the permissions declared in the policy to access the specified AWS resource. Different operations in StarRocks require you to have access permissions on different AWS resources, and therefore you need to configure different policies. This topic provides the IAM policies that you need to configure for StarRocks to integrate with different AWS resources in various business scenarios when you choose [the instance profile, assumed role, or IAM user-based authentication method](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#preparations). #### Batch load data from AWS S3[​](#batch-load-data-from-aws-s3 "Direct link to Batch load data from AWS S3") If you want to load data from your S3 bucket, configure the following IAM policy: > **NOTICE** > > Replace `` in the following JSON policy template with the name of your S3 bucket that stores your data files. ```sql { "Version": "2012-10-17", "Statement": [ { "Sid": "s3", "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::/*" ] }, { "Sid": "s3list", "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::" ] } ] } ``` #### Read/write AWS S3[​](#readwrite-aws-s3 "Direct link to Read/write AWS S3") If you want to query data from your S3 bucket, configure the following IAM policy: > **NOTICE** > > Replace `` in the following JSON policy template with the name of your S3 bucket that stores your data files. ```sql { "Version": "2012-10-17", "Statement": [ { "Sid": "s3", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::/*" ] }, { "Sid": "s3list", "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::" ] } ] } ``` #### Integrate with AWS Glue[​](#integrate-with-aws-glue "Direct link to Integrate with AWS Glue") If you want to integrate with your AWS Glue Data Catalog, configure the following IAM policy: ```sql { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:BatchCreatePartition", "glue:UpdateDatabase", "glue:GetConnections", "glue:CreateTable", "glue:DeleteDatabase", "glue:BatchUpdatePartition", "glue:GetTables", "glue:GetTableVersions", "glue:GetPartitions", "glue:UpdateTable", "glue:BatchGetPartition", "glue:DeleteTable", "glue:GetDatabases", "glue:GetDevEndpoint", "glue:GetTable", "glue:GetDatabase", "glue:GetPartition", "glue:GetDevEndpoints", "glue:GetConnection", "glue:CreateDatabase", "glue:CreatePartition", "glue:DeletePartition", "glue:UpdatePartition" ], "Resource": [ "*" ] } ] } ``` --- ### DATE DATE type. The value range is \['0000-01-01', '9999-12-31'] and the default format is `YYYY-MM-DD`. tip * [Date and time SQL functions](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions) * [Date and time data types](https://docs.starrocks.io/docs/sql-reference/data-types/date-types) #### Examples[​](#examples "Direct link to Examples") Example 1: Specify a column as the DATE type when you create a table. ```sql CREATE TABLE dateDemo ( pk INT COMMENT "range [-2147483648, 2147483647]", make_time DATE NOT NULL COMMENT "YYYY-MM-DD" ) ENGINE=OLAP DUPLICATE KEY(pk) COMMENT "OLAP" DISTRIBUTED BY HASH(pk) ``` Example 2: Convert a DATETIME value into a DATE value. ```sql mysql> SELECT DATE('2003-12-31 01:02:03'); -> '2003-12-31' ``` For more information, see the [date](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date.md) function. --- ### DATETIME Date and time type. The value range is \['0000-01-01 00:00:00', '9999-12-31 23:59:59']. tip * [Date and time SQL functions](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions) * [Date and time data types](https://docs.starrocks.io/docs/sql-reference/data-types/date-types) The form of printing is `YYYY-MM-DD HH:MM:SS`. From v3.3.5, DATETIME supports millisecond and microsecond precision. The form of printing is `YYYY-MM-DD HH:MM:SS.ffffff`. --- ### Numeric #### [📄️ BIGINT](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/BIGINT.md) [BIGINT is an 8-byte signed integer.](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/BIGINT.md) #### [📄️ BOOLEAN](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/BOOLEAN.md) [BOOL, BOOLEAN](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/BOOLEAN.md) #### [📄️ DECIMAL](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/DECIMAL.md) [DECIMAL(P\[,S\]) is a high-precision fixed-point value.](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/DECIMAL.md) #### [📄️ DOUBLE](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/DOUBLE.md) [DOUBLE is an 8-byte floating point number](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/DOUBLE.md) #### [📄️ FLOAT](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/FLOAT.md) [FLOAT](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/FLOAT.md) #### [📄️ INT](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/INT.md) [INT is a 4-byte signed integer.](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/INT.md) #### [📄️ LARGEINT](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/LARGEINT.md) [LARGEINT is a 16-byte signed integer.](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/LARGEINT.md) #### [📄️ SMALLINT](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/SMALLINT.md) [SMALLINT is a 2-byte signed integer.](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/SMALLINT.md) #### [📄️ TINYINT](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/TINYINT.md) [TINYINT is a 1-byte signed integer.](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/TINYINT.md) --- ### BIGINT BIGINT is an 8-byte signed integer. The value range is \[-9223372036854775808, 9223372036854775807]. --- ### BOOLEAN BOOL, BOOLEAN Like TINYINT, 0 stands for false and 1 for true. --- ### DECIMAL DECIMAL(P\[,S]) is a high-precision fixed-point value. `P` stands for the total number of significant numbers (precision). `S` stands for the maximum number of decimal points (scale). If `P` is omitted, the default value is 10. If `S` is omitted, the default value is 0. * For versions earlier than v4.0, StarRocks supports DECIMAL128 based on [Fast DECIMAL](#fast-decimal). * For v4.0 and later versions, StarRocks supports [DECIMAL256](#decimal256). #### Features[​](#features "Direct link to Features") ##### Fast DECIMAL[​](#fast-decimal "Direct link to Fast DECIMAL") Controlled by the FE dynamic parameter `enable_decimal_v3`, Fast DECIMAL is enabled by default. For Fast DECIMAL, the range of `P` is \[1,38] and the range of `S` is \[0, P]. The default value of `S` is 0. Fast DECIMAL uses variable-width integers to express decimals. * Decimal(P ≤ 18, S) * LogicalType: Decimal64 * Stored as: int64 * Delegate LogicalType: BIGINT * Decimal(P > 18 & P ≤ 38, S) * LogicalType: Decimal128 * Stored as: int128 * Delegate LogicalType: LARGEINT From v3.1 onwards, StarRocks supports Fast Decimal entries in [ARRAY](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/Array.md), [MAP](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/Map.md), and [STRUCT](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/STRUCT.md). #### DECIMAL256[​](#decimal256 "Direct link to DECIMAL256") StarRocks introduces DECIMAL256 from v4.0 onwards. For DECIMAL256, the range of `P` is (38,76] and the range of `S` is \[0, P]. The default value of `S` is 0. DECIMAL256 expands the upper limit of precision to 76 bits. Its numerical capacity is 10³⁸ times higher than that of DECIMAL128, reducing the probability of overflow to a negligible level. DECIMAL256 uses the same strategy to express decimals as that of Fast DECIMAL. In addition to the two decimal types mentioned above, it supports: * Decimal(P > 38 & P ≤ 76, S) * LogicalType: Decimal256 * Stored as: int256 * Delegate LogicalType: INT\_256 The actual representable range of DECIMAL256 is `-57896044618658097711785492504343953926634992332820282019728792003956564819968` to `57896044618658097711785492504343953926634992332820282019728792003956564819967`. ###### Type casting operations[​](#type-casting-operations "Direct link to Type casting operations") DECIMAL256 supports bidirectional type conversion between all types listed below: | Source Type | Target Types | | ---------------- | ---------------------------------------------------------------------- | | TYPE\_DECIMAL256 | BOOL | | TYPE\_DECIMAL256 | TYPE\_TINYINT, TYPE\_SMALLINT, TYPE\_INT, TYPE\_BIGINT, TYPE\_LARGEINT | | TYPE\_DECIMAL256 | TYPE\_FLOAT, TYPE\_DOUBLE | | TYPE\_DECIMAL256 | TYPE\_VARCHAR | ###### Aggregate functions[​](#aggregate-functions "Direct link to Aggregate functions") Currently, DECIMAL256 supports these aggregation functions: `COUNT`, `COUNT DISTINCT`, `SUM`, `SUM DISTINCT`, `AVG`, `AVG DISTINCT`, `MAX`, `MIN`, and `ABS`. ###### Limitations[​](#limitations "Direct link to Limitations") DECIMAL256 has the following limitations: * Currently, DECIMAL256 does not support automatic precision scale-up for decimal operations. For example, `DECIMAL128 * DECIMAL128` will not be automatically scaled up to DECIMAL256. DECIMAL256 features are only applied when `DECIMAL256` is explicitly specified as the operand either in the CREATE TABLE statement or using CAST (`SELECT CAST(p38s10 as DECIMAL(70, 30))`). * DECIMAL256 does not support window functions. * Aggregate tables do not support DECIMAL256 type. #### Examples[​](#examples "Direct link to Examples") ##### Fast DECIMAL example[​](#fast-decimal-example "Direct link to Fast DECIMAL example") Define DECIMAL columns when creating a table. ```sql CREATE TABLE decimalDemo ( pk BIGINT(20) NOT NULL COMMENT "", account DECIMAL(20,10) COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(pk) COMMENT "OLAP" DISTRIBUTED BY HASH(pk); INSERT INTO decimalDemo VALUES (1,3.141592656), (2,21.638378), (3,4873.6293048479); SELECT * FROM decimalDemo; +------+-----------------+ | pk | account | +------+-----------------+ | 1 | 3.1415926560 | | 3 | 4873.6293048479 | | 2 | 21.6383780000 | +------+-----------------+ ``` ##### DECIMAL256 example[​](#decimal256-example "Direct link to DECIMAL256 example") Create a table with a DECIMAL256 type column. ```sql CREATE TABLE test_decimal256( p50s48 DECIMAL(50, 48) COMMENT "" ); ``` INSERT different values: ```sql INSERT INTO test_decimal256(p50s48) SELECT 1.222222; ``` This statement succeeds with the values written correctly. ```sql INSERT INTO test_decimal256(p50s48) SELECT 11111111111111111111111111111111111111111111.222222; ``` This statement succeeds with the values written as NULL. Because the integer part and decimal part (which has 44 digits) with the specified scale (which is `48`) together exceed the maximum integer that can be represented by 256 bits, FE directly casts the value to NULL. ```sql INSERT INTO test_decimal256(p50s48) SELECT 333; ``` This statement will be returned with an error `ERROR 1064 (HY000): Insert has filtered data`. Because the integer part and decimal part with the specified scale (which is `48`) together exceed the maximum value represented by a 50-digit integer, BE returns the error. #### Usage notes[​](#usage-notes "Direct link to Usage notes") You can set the system variable `sql_mode` to `ERROR_IF_OVERFLOW` to allow the system to return an error instead of NULL in the case of arithmetic overflow. #### Keywords[​](#keywords "Direct link to Keywords") decimal, decimalv3, fast decimal, decimal128, decimal256 --- ### DOUBLE DOUBLE is an 8-byte floating point number --- ### FLOAT FLOAT 4-byte floating point number --- ### INT INT is a 4-byte signed integer. The value range is \[-2147483648, 2147483647]. --- ### LARGEINT LARGEINT is a 16-byte signed integer. The value range is \[-2^127 + 1, 2^127 - 1]. --- ### SMALLINT SMALLINT is a 2-byte signed integer. The value range is \[-32768, 32767]. --- ### TINYINT TINYINT is a 1-byte signed integer. The value range is \[-128, 127]. --- ### Other #### [📄️ BITMAP](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/BITMAP.md) [BITMAP is often used to accelerate count distinct.](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/BITMAP.md) #### [📄️ HLL (HyperLogLog)](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/HLL.md) [HLL is used for approximate count distinct.](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/HLL.md) --- ### BITMAP BITMAP is often used to accelerate count distinct. It is similar but more accurate in count distinct than HyperLogLog (HLL). BITMAP consumes more memory and disk resources. It supports only aggregation of INT data. If you want to apply bitmap to string data, you must map the data using the low-cardinality dictionary. This topic provides a simple example on how to create a BITMAP column and use bitmap functions to aggregate data for that column. For detailed function definitions or more Bitmap functions, see "Bitmap functions". #### Create a table[​](#create-a-table "Direct link to Create a table") * Create an Aggregate table, in which the data type of the `user_id` column is BITMAP and the bitmap\_union() function is used to aggregate data. ```sql CREATE TABLE `pv_bitmap` ( `dt` int(11) NULL COMMENT "", `page` varchar(10) NULL COMMENT "", `user_id` bitmap BITMAP_UNION NULL COMMENT "" ) ENGINE=OLAP AGGREGATE KEY(`dt`, `page`) COMMENT "OLAP" DISTRIBUTED BY HASH(`dt`); ``` * Create a Primary Key table, in which the data type of the `userid` column is BITMAP. ```sql CREATE TABLE primary_bitmap ( `tagname` varchar(65533) NOT NULL COMMENT "Tag name", `tagvalue` varchar(65533) NOT NULL COMMENT "Tag value", `userid` bitmap NOT NULL COMMENT "User ID") ENGINE=OLAP PRIMARY KEY(`tagname`, `tagvalue`) COMMENT "OLAP" DISTRIBUTED BY HASH(`tagname`); ``` Before you insert data into BITMAP columns, you must first use the to\_bitmap() function to convert data. For details about how to use BITMAP, for example, load BITMAP data into a table, see [bitmap](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/bitmap.md). --- ### HLL (HyperLogLog) HLL is used for [approximate count distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_HLL.md). HLL enables the development of programmes based on the HyperLogLog algorithm. It is used to store intermediate results of the HyperLogLog calculation process. It can only be used as the value column type of the table. HLL reduces the amount of data through aggregation to speed up the query process. There may be a 1% deviation in the estimated results. HLL column is generated based on the imported data or data from other columns. When data is imported, the [hll\_hash](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_hash.md) function specifies which column will be used to generated the HLL column. HLL is often used to replace COUNT DISTINCT and quickly calculate unique views (UVs) with rollup. The storage space used by HLL is determined by the distinct values in the hash value. The storage space varies depending on three conditions: * HLL is empty. No value is inserted into HLL and the storage cost is the lowest, which is 80 bytes. * The number of distinct hash values in HLL is less than or equal to 160. The highest storage cost is 1360 bytes (80 + 160 \* 8 = 1360). * The number of distinct hash values in HLL is greater than 160. The storage cost is fixed at 16,464 bytes (80 + 16 \* 1024 = 16464). In actual business scenarios, data volume and data distribution affect the memory usage of queries and the accuracy of the approximate result. You need to consider these two factors: * Data volume: HLL returns an approximate value. A larger data volume results in a more accurate result. A smaller data volume results in larger deviation. * Data distribution: In the case of large data volume and high-cardinality dimension column for GROUP BY, data computation will use more memory. HLL is not recommended in this situation. It is recommended when you perform no-group-by count distinct or GROUP BY on low-cardinality dimension columns. * Query granularity: If you query data at a large query granularity, we recommend you use the Aggregate table or materialized view to pre-aggregate data to reduce data volume. #### Related functions[​](#related-functions "Direct link to Related functions") * [HLL\_UNION\_AGG(hll)](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/hll_union_agg.md): This function is an aggregate function used to estimate the cardinality of all data that meet the conditions. This can also be used to analyze functions. It only supports default window and does not support window clause. * [HLL\_RAW\_AGG(hll)](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/hll_raw_agg.md): This function is an aggregate function used to aggregate fields of hll type and returns with hll type. * HLL\_CARDINALITY(hll): This function is used to estimate the cardinality of a single hll column. * [HLL\_HASH(column\_name)](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_hash.md): This generates HLL column type and is used for inserts or imports. See the instructions for the use of imports. * [HLL\_EMPTY](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_empty.md): This generates empty HLL column and is used to fill in default values during inserts or imports. See the instructions for the use of imports. #### Examples[​](#examples "Direct link to Examples") 1. Create a table with HLL columns `set1` and `set2`. ```sql create table test( dt date, id int, name char(10), province char(10), os char(1), set1 hll hll_union, set2 hll hll_union) distributed by hash(id); ``` 2. Load data by using [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md). ```plain a. Use table columns to generate an HLL column. curl --location-trusted -uname:password -T data -H "label:load_1" \ -H "columns:dt, id, name, province, os, set1=hll_hash(id), set2=hll_hash(name)" http://host/api/test_db/test/_stream_load b. Use data columns to generate an HLL column. curl --location-trusted -uname:password -T data -H "label:load_1" \ -H "columns:dt, id, name, province, sex, cuid, os, set1=hll_hash(cuid), set2=hll_hash(os)" http://host/api/test_db/test/_stream_load ``` 3. Aggregate data in the following three ways: (Without aggregation, direct query on base table may be as slow as using approx\_count\_distinct) ```sql -- a. Create a rollup to aggregate HLL column. alter table test add rollup test_rollup(dt, set1); -- b. Create another table to calculate uv and insert data into it create table test_uv( dt date, id int, uv_set hll hll_union) distributed by hash(id); insert into test_uv select dt, id, set1 from test; -- c. Create another table to calculate UV. Insert data and generate HLL column by testing other columns through hll_hash. create table test_uv( dt date, id int, id_set hll hll_union) distributed by hash(id); insert into test_uv select dt, id, hll_hash(id) from test; ``` 4. Query data. HLL columns do not support direct query to its original values. It can be queried by matching functions. ```plain a. Calculate the total UV. select HLL_UNION_AGG(uv_set) from test_uv; b. Calculate the UV for each day. select dt, HLL_CARDINALITY(uv_set) from test_uv; c. Calculate the aggregation value of set1 in the test table. select dt, HLL_CARDINALITY(uv) from (select dt, HLL_RAW_AGG(set1) as uv from test group by dt) tmp; select dt, HLL_UNION_AGG(set1) as uv from test group by dt; ``` --- ### Semi-structured #### [📄️ ARRAY](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/Array.md) [ARRAY, as an extended type of database, is supported in various database systems such as PostgreSQL, ClickHouse, and Snowflake.](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/Array.md) #### [📄️ JSON](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/JSON.md) [Since version 2.2.0, StarRocks supports JSON.](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/JSON.md) #### [📄️ MAP](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/Map.md) [MAP is a complex data type that stores a set of key-value pairs, for example, {a:1, b:2, c:3}.](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/Map.md) #### [📄️ STRUCT](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/STRUCT.md) [STRUCT is widely used to express complex data types.](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/STRUCT.md) #### [📄️ VARIANT](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/VARIANT.md) [The VARIANT type is supported only for tables in Iceberg Catalog.](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/VARIANT.md) --- ### ARRAY ARRAY, as an extended type of database, is supported in various database systems such as PostgreSQL, ClickHouse, and Snowflake. ARRAY is widely used in scenarios such as A/B tests, user tag analysis, and user profiling. StarRocks supports multidimensional array nesting, array slicing, comparison, and filtering. #### Define ARRAY columns[​](#define-array-columns "Direct link to Define ARRAY columns") You can define an ARRAY column when you create a table. ```sql -- Define a one-dimensional array. ARRAY -- Define a nested array. ARRAY> -- Define an array column as NOT NULL. ARRAY NOT NULL ``` `type` specifies the data types of elements in an array. StarRocks supports the following element types: BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, VARCHAR, CHAR, DATETIME, DATE, JSON, ARRAY (since v3.1), MAP (since v3.1), and STRUCT (since v3.1). Elements in an array are nullable by default, for example, `[null, 1 ,2]`. You cannot specify elements in an array as NOT NULL. However, you can specify an ARRAY column as NOT NULL when you create a table, such as the third example in the following code snippet. Examples: ```sql -- Define c1 as a one-dimensional array whose element type is INT. create table t0( c0 INT, c1 ARRAY ) duplicate key(c0) distributed by hash(c0); -- Define c1 as an nested array whose element type is VARCHAR. create table t1( c0 INT, c1 ARRAY> ) duplicate key(c0) distributed by hash(c0); -- Define c1 as a NOT NULL array column. create table t2( c0 INT, c1 ARRAY NOT NULL ) duplicate key(c0) distributed by hash(c0); ``` #### Limits[​](#limits "Direct link to Limits") The following limits apply when you create ARRAY columns in StarRocks tables: * In versions earlier than v2.1, you can create ARRAY columns only in Duplicate Key tables. From v2.1 onwards, you can also create ARRAY columns in other types of tables (Primary Key, Unique Key, Aggregate). Note that in an Aggregate table, you can create an ARRAY column only when the function used to aggregate data in that column is replace() or replace\_if\_not\_null(). For more information, see [Aggregate table](https://docs.starrocks.io/docs/table_design/table_types/aggregate_table.md). * ARRAY columns cannot be used as key columns. * ARRAY columns cannot be used as partition keys (included in PARTITION BY) or bucketing keys (included in DISTRIBUTED BY). * An array can have a maximum of 14-level nesting. #### Construct arrays in SQL[​](#construct-arrays-in-sql "Direct link to Construct arrays in SQL") Arrays can be constructed in SQL using brackets `[]`, with each array element separated by a comma (`,`). ```plain mysql> select [1, 2, 3] as numbers; +---------+ | numbers | +---------+ | [1,2,3] | +---------+ mysql> select ["apple", "orange", "pear"] as fruit; +---------------------------+ | fruit | +---------------------------+ | ["apple","orange","pear"] | +---------------------------+ mysql> select [true, false] as booleans; +----------+ | booleans | +----------+ | [1,0] | +----------+ ``` StarRocks automatically infers data types if an array consists of elements of multiple types: ```plain mysql> select [1, 1.2] as floats; +---------+ | floats | +---------+ | [1.0,1.2] | +---------+ mysql> select [12, "100"]; +--------------+ | [12,'100'] | +--------------+ | ["12","100"] | +--------------+ ``` You can use pointed brackets (`<>`) to show the declared array type. ```plain mysql> select ARRAY[1, 2]; +-----------------------+ | ARRAY[1.0,2.0] | +-----------------------+ | [1,2] | +-----------------------+ mysql> select ARRAY["12", "100"]; +------------------------+ | ARRAY[12,100] | +------------------------+ | [12,100] | +------------------------+ ``` NULLs can be included in the element. ```plain mysql> select [1, NULL]; +----------+ | [1,NULL] | +----------+ | [1,null] | +----------+ ``` For an empty array, you can use pointed brackets to show the declared type, or you can write \[] directly for StarRocks to infer the type based on the context. If StarRocks cannot infer the type, it will report an error. ```plain mysql> select []; +------+ | [] | +------+ | [] | +------+ mysql> select ARRAY[]; +----------------------------------+ | ARRAY[] | +----------------------------------+ | [] | +----------------------------------+ mysql> select array_append([], 10); +----------------------+ | array_append([], 10) | +----------------------+ | [10] | +----------------------+ ``` #### Load Array data[​](#load-array-data "Direct link to Load Array data") StarRocks supports loading Array data in three ways: * INSERT INTO is suitable for loading small-scale data for testing. * Broker Load is suitable for loading ORC or Parquet files with large-scale data. * Stream Load and Routine Load are suitable for loading CSV files with large-scale data. ##### Use INSERT INTO to load arrays[​](#use-insert-into-to-load-arrays "Direct link to Use INSERT INTO to load arrays") You can use INSERT INTO to load small-scale data column by column, or perform ETL on data before loading the data. ```sql create table t0( c0 INT, c1 ARRAY ) duplicate key(c0) distributed by hash(c0); INSERT INTO t0 VALUES(1, [1,2,3]); ``` ##### Use Broker Load to load arrays from ORC or Parquet files[​](#use-broker-load-to-load-arrays-from-orc-or-parquet-files "Direct link to Use Broker Load to load arrays from ORC or Parquet files") The array type in StarRocks corresponds to the list structure in ORC and Parquet files, which eliminates the need for you to specify different data types in StarRocks. For more information about data loading, see [Broker load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ##### Use Stream Load or Routine Load to load CSV-formatted arrays[​](#use-stream-load-or-routine-load-to-load-csv-formatted-arrays "Direct link to Use Stream Load or Routine Load to load CSV-formatted arrays") Arrays in CSV files are separated with comma by default. You can use [Stream Load or Routine Load](https://docs.starrocks.io/docs/loading/loading_introduction.md) to load CSV text files or CSV data in Kafka. #### Query ARRAY data[​](#query-array-data "Direct link to Query ARRAY data") You can access elements in an array using `[]` and subscripts, starting from `1`. ```plain mysql> select [1,2,3][1]; +------------+ | [1,2,3][1] | +------------+ | 1 | +------------+ 1 row in set (0.00 sec) ``` If the subscript is 0 or a negative number, **no error is reported and NULL is returned**. ```plain mysql> select [1,2,3][0]; +------------+ | [1,2,3][0] | +------------+ | NULL | +------------+ 1 row in set (0.01 sec) ``` If the subscript exceeds the length of the array (the number of elements in the array), **NULL will be returned**. ```plain mysql> select [1,2,3][4]; +------------+ | [1,2,3][4] | +------------+ | NULL | +------------+ 1 row in set (0.01 sec) ``` For multidimensional arrays, the elements can be accessed **recursively**. ```plain mysql(ARRAY)> select [[1,2],[3,4]][2]; +------------------+ | [[1,2],[3,4]][2] | +------------------+ | [3,4] | +------------------+ 1 row in set (0.00 sec) mysql> select [[1,2],[3,4]][2][1]; +---------------------+ | [[1,2],[3,4]][2][1] | +---------------------+ | 3 | +---------------------+ 1 row in set (0.01 sec) ``` --- ### JSON Since version 2.2.0, StarRocks supports JSON. This article introduces the basic concepts of JSON, and how StarRocks creates JSON-type columns, loads, and queries JSON data, and constructs and processes JSON data through JSON functions and operators. #### What is JSON[​](#what-is-json "Direct link to What is JSON") JSON is a lightweight data interchange format. JSON-type data is semi-structured and supports a tree structure. JSON data is hierarchical, flexible, easy to read and process, and is widely used in data storage and analysis scenarios. JSON supports data types such as NUMBER, STRING, BOOLEAN, ARRAY, OBJECT, and NULL values. For more information on JSON, please refer to the [JSON official website](https://www.json.org/json-en.html). For JSON data input and output syntax, please refer to the JSON specification [RFC 7159](https://tools.ietf.org/html/rfc7159?spm=a2c63.p38356.0.0.14d26b9fcp7fcf#page-4). StarRocks supports storing and efficiently querying and analyzing JSON data. StarRocks uses binary format encoding to store JSON data instead of storing the input text directly, which reduces parsing costs during data computation and queries, thereby improving query efficiency. #### Using JSON Data[​](#using-json-data "Direct link to Using JSON Data") ##### Creating JSON-Type Columns[​](#creating-json-type-columns "Direct link to Creating JSON-Type Columns") When creating a table, specify the column `j` as JSON type using the keyword `JSON`. ```sql CREATE TABLE `tj` ( `id` INT(11) NOT NULL COMMENT "", `j` JSON NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`) PROPERTIES ( "replication_num" = "3", "storage_format" = "DEFAULT" ); ``` ##### Loading Data and Storing as JSON Type[​](#loading-data-and-storing-as-json-type "Direct link to Loading Data and Storing as JSON Type") StarRocks supports the following methods to load data and store it as JSON type. * Method 1: Use `INSERT INTO` to write data into a JSON-type column (e.g., column `j`). ```sql INSERT INTO tj (id, j) VALUES (1, parse_json('{"a": 1, "b": true}')); INSERT INTO tj (id, j) VALUES (2, parse_json('{"a": 2, "b": false}')); INSERT INTO tj (id, j) VALUES (3, parse_json('{"a": 3, "b": true}')); INSERT INTO tj (id, j) VALUES (4, json_object('a', 4, 'b', false)); ``` > The PARSE\_JSON function can construct JSON-type data based on string-type data. The JSON\_OBJECT function can construct JSON object-type data, allowing existing tables to be converted to JSON type. For more information, please refer to [PARSE\_JSON](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-constructor-functions/parse_json.md) and [JSON\_OBJECT](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-constructor-functions/json_object.md). * Method 2: Use Stream Load to import JSON files and store them as JSON type. For import methods, please refer to [Import JSON Data](https://docs.starrocks.io/docs/loading/StreamLoad.md). * To import and store the JSON object at the root node of a JSON file as JSON type, set `jsonpaths` to `$`. * To import and store the value of a JSON object in a JSON file as JSON type, set `jsonpaths` to `$.a` (where `a` represents the key). For more JSON path expressions, refer to [JSON path](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md#json-path). * Method 3: Use Broker Load to import Parquet files and store them as JSON type. For import methods, please refer to [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). Data type conversion is supported during import as follows: | Data Type in Parquet File | Converted JSON Data Type | | ------------------------------------------------------------------------ | ------------------------ | | Integer types (INT8, INT16, INT32, INT64, UINT8, UINT16, UINT32, UINT64) | JSON Number | | Floating-point types (FLOAT, DOUBLE) | JSON Number | | BOOLEAN | JSON Boolean | | STRING | JSON String | | MAP | JSON Object | | STRUCT | JSON Object | | LIST | JSON Array | | UNION, TIMESTAMP, and other types | Not supported | * Method 4: Use [Routine Load](https://docs.starrocks.io/docs/loading/loading_introduction.md) to continuously consume JSON format data from Kafka and import it into StarRocks. ##### Querying and Processing JSON-Type Data[​](#querying-and-processing-json-type-data "Direct link to Querying and Processing JSON-Type Data") StarRocks supports querying and processing JSON-type data and supports using JSON functions and operators. This example uses the table `tj` for illustration. ```sql mysql> select * from tj; +------+----------------------+ | id | j | +------+----------------------+ | 1 | {"a": 1, "b": true} | | 2 | {"a": 2, "b": false} | | 3 | {"a": 3, "b": true} | | 4 | {"a": 4, "b": false} | +------+----------------------+ ``` Example 1: Filter data in the JSON-type column that meets the condition `id=1`. ```sql mysql> select * from tj where id = 1; +------+---------------------+ | id | j | +------+---------------------+ | 1 | {"a": 1, "b": true} | +------+---------------------+ ``` Example 2: Filter data in the table based on the JSON-type column. > In the following example, `j->'a'` returns JSON-type data. You can compare it with the first example, which performs implicit conversion on the data; or use the CAST function to construct JSON-type data as INT for comparison. ```sql mysql> select * from tj where j->'a' = 1; +------+---------------------+ | id | j | +------+---------------------+ | 1 | {"a": 1, "b": true} | +------+---------------------+ mysql> select * from tj where cast(j->'a' as INT) = 1; +------+---------------------+ | id | j | +------+---------------------+ | 1 | {"a": 1, "b": true} | +------+---------------------+ ``` Example 3: Filter data in the table based on the JSON-type column (you can use the CAST function to construct the JSON-type column as BOOLEAN type). ```sql mysql> select * from tj where cast(j->'b' as boolean); +------+---------------------+ | id | j | +------+---------------------+ | 1 | {"a": 1, "b": true} | | 3 | {"a": 3, "b": true} | +------+---------------------+ ``` Example 4: Filter data in the JSON-type column that meets the condition and perform numerical operations. ```sql mysql> select cast(j->'a' as int) from tj where cast(j->'b' as boolean); +-----------------------+ | CAST(j->'a' AS INT) | +-----------------------+ | 3 | | 1 | +-----------------------+ mysql> select sum(cast(j->'a' as int)) from tj where cast(j->'b' as boolean); +----------------------------+ | sum(CAST(j->'a' AS INT)) | +----------------------------+ | 4 | +----------------------------+ ``` Example 5: Sort based on the JSON-type column. ```sql mysql> select * from tj where j->'a' <= 3 order by cast(j->'a' as int); +------+----------------------+ | id | j | +------+----------------------+ | 1 | {"a": 1, "b": true} | | 2 | {"a": 2, "b": false} | | 3 | {"a": 3, "b": true} | | 4 | {"a": 4, "b": false} | +------+----------------------+ ``` #### JSON Functions and Operators[​](#json-functions-and-operators "Direct link to JSON Functions and Operators") JSON functions and operators can be used to construct and process JSON data. For detailed information, please refer to [JSON Functions and Operators](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md). #### JSON Array[​](#json-array "Direct link to JSON Array") JSON can contain nested data, such as Objects, Arrays, or other JSON data types nested within an Array. StarRocks provides a rich set of functions and operators to handle these complex nested JSON data structures. The following will introduce how to handle JSON data containing arrays. Suppose there is a JSON field `event_data` in the `events` table, with the following content: ```text { "user": "Alice", "actions": [ {"type": "click", "timestamp": "2024-03-17T10:00:00Z", "quantity": 1}, {"type": "view", "timestamp": "2024-03-17T10:05:00Z", "quantity": 2}, {"type": "purchase", "timestamp": "2024-03-17T10:10:00Z", "quantity": 3} ] } ``` The following examples demonstrate several common JSON array analysis scenarios: 1. Extract array elements: Extract specific fields such as type, timestamp, etc., from the actions array and perform projection operations. 2. Array expansion: Use the `json_each` function to expand the nested JSON array into a multi-row, multi-column table structure for subsequent analysis. 3. Array computation: Use Array Functions to filter, transform, and aggregate array elements, such as counting the number of specific types of operations. ##### 1. Extract Elements from JSON Array[​](#1-extract-elements-from-json-array "Direct link to 1. Extract Elements from JSON Array") To extract a nested element from a JSON Array, you can use the following syntax: * The return type is still a JSON Array, and you can use the CAST expression for type conversion. ```text MySQL > SELECT json_query(event_data, '$.actions[*].type') as json_array FROM events; +-------------------------------+ | json_array | +-------------------------------+ | ["click", "view", "purchase"] | +-------------------------------+ MySQL > SELECT cast(json_query(event_data, '$.actions[*].type') as array) array_string FROM events; +-----------------------------+ | array_string | +-----------------------------+ | ["click","view","purchase"] | +-----------------------------+ ``` ##### 2. Expand Using json\_each[​](#2-expand-using-json_each "Direct link to 2. Expand Using json_each") StarRocks provides the `json_each` function to expand JSON arrays, converting them into multiple rows of data. For example: ```text MySQL > select value from events, json_each(event_data->'actions'); +--------------------------------------------------------------------------+ | value | +--------------------------------------------------------------------------+ | {"quantity": 1, "timestamp": "2024-03-17T10:00:00Z", "type": "click"} | | {"quantity": 2, "timestamp": "2024-03-17T10:05:00Z", "type": "view"} | | {"quantity": 3, "timestamp": "2024-03-17T10:10:00Z", "type": "purchase"} | +--------------------------------------------------------------------------+ ``` To extract the type and timestamp fields separately: ```text MySQL > select value->'timestamp', value->'type' from events, json_each(event_data->'actions'); +------------------------+---------------+ | value->'timestamp' | value->'type' | +------------------------+---------------+ | "2024-03-17T10:00:00Z" | "click" | | "2024-03-17T10:05:00Z" | "view" | | "2024-03-17T10:10:00Z" | "purchase" | +------------------------+---------------+ ``` After this, the JSON Array data becomes a familiar relational model, allowing the use of common functions for analysis. ##### 3. Use Array Functions for Filtering and Calculation[​](#3-use-array-functions-for-filtering-and-calculation "Direct link to 3. Use Array Functions for Filtering and Calculation") StarRocks also supports ARRAY-related functions, which can be used in conjunction with JSON functions for more efficient queries. By combining these functions, you can filter, transform, and aggregate JSON array data. The following example demonstrates how to use these functions: ```text MySQL > WITH step1 AS ( SELECT cast(event_data->'actions' as ARRAY) as docs FROM events ) SELECT array_filter(doc -> get_json_string(doc, 'type') = 'click', docs) as clicks FROM step1 +---------------------------------------------------------------------------+ | clicks | +---------------------------------------------------------------------------+ | ['{"quantity": 1, "timestamp": "2024-03-17T10:00:00Z", "type": "click"}'] | +---------------------------------------------------------------------------+ ``` Furthermore, you can combine other ARRAY Functions to perform aggregation calculations on array elements: ```text MySQL > WITH step1 AS ( SELECT cast(event_data->'actions' as ARRAY) as docs FROM events ), step2 AS ( SELECT array_filter(doc -> get_json_string(doc, 'type') = 'click', docs) as clicks FROM step1 ) SELECT array_sum( array_map(doc -> get_json_double(doc, 'quantity'), clicks) ) as click_amount FROM step2 +--------------+ | click_amount | +--------------+ | 1.0 | +--------------+ ``` #### Limitations and Considerations[​](#limitations-and-considerations "Direct link to Limitations and Considerations") * The maximum supported length for JSON-type data is currently 16 MB. * ORDER BY, GROUP BY, and JOIN clauses do not support referencing JSON-type columns. If you need to reference them, you can use the CAST function in advance to convert JSON-type columns to other SQL types. For specific conversion methods, please refer to [JSON Type Conversion](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/cast.md). * JSON-type columns can exist in Duplicate Key tables, Primary Key tables, and Unique Key tables, but not in Aggregate tables. * JSON-type columns are not supported as partition keys, bucketing keys, or dimension columns in Duplicate Key tables, Primary Key tables, and Unique Key tables, and cannot be used in JOIN, GROUP BY, or ORDER BY clauses. * StarRocks supports using `<`, `<=`, `>`, `>=`, `=`, `!=` operators to query JSON data but does not support the IN operator. --- ### MAP MAP is a complex data type that stores a set of key-value pairs, for example, `{a:1, b:2, c:3}`. Keys in a map must be unique. A nested map can contain up to 14 levels of nesting. The MAP data type is supported from v3.1 onwards. In v3.1, you can define MAP columns when you create a StarRocks table, load MAP data into that table, and query MAP data. From v2.5 onwards, StarRocks supports querying complex data types MAP and STRUCT from data lakes. You can use external catalogs provided by StarRocks to query MAP and STRUCT data from Apache Hive™, Apache Hudi, and Apache Iceberg. You can only query data from ORC and Parquet files. For more information about how to use external catalogs to query external data sources, see [Overview of catalogs](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md) and topics related to the required catalog type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext MAP ``` * `key_type`: the data type of the key. The key must be of a primitive type supported by StarRocks, such as numeric, string, or date. It cannot be of the HLL, JSON, ARRAY, MAP, BITMAP, or STRUCT type. * `value_type`: the data type of the value. The value can be of any supported type. Keys and values are **natively nullable**. #### Define a MAP column in StarRocks[​](#define-a-map-column-in-starrocks "Direct link to Define a MAP column in StarRocks") You can define a MAP column when you create a table and load MAP data into this column. ```sql -- Define a one-dimensional map. CREATE TABLE t0( c0 INT, c1 MAP ) DUPLICATE KEY(c0); -- Define a nested map. CREATE TABLE t1( c0 INT, c1 MAP> ) DUPLICATE KEY(c0); -- Define a NOT NULL map. CREATE TABLE t2( c0 INT, c1 MAP NOT NULL ) DUPLICATE KEY(c0); ``` Columns with the MAP type have the following restrictions: * Cannot be used as key columns in a table. They can only be used as value columns. * Cannot be used as partition key columns (the columns following PARTITION BY) in a table. * Cannot be used as bucketing columns (the columns following DISTRIBUTED BY) in a table. #### Construct maps in SQL[​](#construct-maps-in-sql "Direct link to Construct maps in SQL") Map can be constructed in SQL using the following two syntaxes: * `map{key_expr:value_expr, ...}`: Map elements are separated by a comma (`,`), and keys and values are separated by a colon (`:`), for example, `map{a:1, b:2, c:3}`. * `map(key_expr, value_expr ...)`: The expressions of keys and values must be in pairs, for example, `map(a,1,b,2,c,3)`. StarRocks can derive the data types of keys and values from all the input keys and values. ```sql select map{1:1, 2:2, 3:3} as numbers; select map(1,1,2,2,3,3) as numbers; -- Return {1:1,2:2,3:3}. select map{1:"apple", 2:"orange", 3:"pear"} as fruit; select map(1, "apple", 2, "orange", 3, "pear") as fruit; -- Return {1:"apple",2:"orange",3:"pear"}. select map{true:map{3.13:"abc"}, false:map{}} as nest; select map(true, map(3.13, "abc"), false, map{}) as nest; -- Return {1:{3.13:"abc"},0:{}}. ``` If the keys or values have different types, StarRocks automatically derives the appropriate type (supertype). ```sql select map{1:2.2, 1.2:21} as floats_floats; -- Return {1.0:2.2,1.2:21.0}. select map{12:"a", "100":1, NULL:NULL} as string_string; -- Return {"12":"a","100":"1",null:null}. ``` You can also define the data type using `<>` when you construct a map. The input keys or values must be able to cast into the specified types. ```sql select map{1:2}; -- Return {1:2}. select map{"12": "100"}; -- Return {12:100}. ``` Keys and values are nullable. ```sql select map{1:NULL}; ``` Construct empty maps. ```sql select map{} as empty_map; select map() as empty_map; -- Return {}. ``` #### Load MAP data into StarRocks[​](#load-map-data-into-starrocks "Direct link to Load MAP data into StarRocks") You can load map data into StarRocks using two methods: [INSERT INTO](https://docs.starrocks.io/docs/loading/InsertInto.md), and [ORC/Parquet loading](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). Note that StarRocks will remove duplicate keys of each map when loading MAP data. ##### INSERT INTO[​](#insert-into "Direct link to INSERT INTO") ```sql CREATE TABLE t0( c0 INT, c1 MAP ) DUPLICATE KEY(c0); INSERT INTO t0 VALUES(1, map{1:2,3:NULL}); ``` ##### Load MAP data from ORC and Parquet files[​](#load-map-data-from-orc-and-parquet-files "Direct link to Load MAP data from ORC and Parquet files") The MAP data type in StarRocks corresponds to the map structure in ORC or Parquet format. No additional specification is needed. You can load MAP data from ORC or Parquet files by following the instructions in [ORC/Parquet loading](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). #### Access MAP data[​](#access-map-data "Direct link to Access MAP data") Example 1: Query MAP column `c1` from table `t0`. ```plain mysql> select c1 from t0; +--------------+ | c1 | +--------------+ | {1:2,3:null} | +--------------+ ``` Example 2: Use the `[ ]` operator to retrieve values from a map by key, or use the `element_at(any_map, any_key)` function. The following example queries the value corresponding to key `1`. ```plain mysql> select map{1:2,3:NULL}[1]; +-----------------------+ | map(1, 2, 3, NULL)[1] | +-----------------------+ | 2 | +-----------------------+ mysql> select element_at(map{1:2,3:NULL},1); +--------------------+ | map{1:2,3:NULL}[1] | +--------------------+ | 2 | +--------------------+ ``` If the key does not exist in the map, `NULL` is returned. The following example queries the value corresponding to key 2, which does not exist. ```plain mysql> select map{1:2,3:NULL}[2]; +-----------------------+ | map(1, 2, 3, NULL)[2] | +-----------------------+ | NULL | +-----------------------+ ``` Example 3: Query multidimensional maps **recursively**. The following example first queries the value corresponding to key `1`, which is `map{2:1}` and then recursively queries the value corresponding to key `2` in `map{2:1}`. ```plain mysql> select map{1:map{2:1},3:NULL}[1][2]; +----------------------------------+ | map(1, map(2, 1), 3, NULL)[1][2] | +----------------------------------+ | 1 | +----------------------------------+ ``` #### References[​](#references "Direct link to References") * [Map functions](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_values.md) * [element\_at](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/element_at.md) --- ### STRUCT STRUCT is widely used to express complex data types. It represents a collection of elements (also called fields) with different data types, for example, ``. Field names in a struct must be unique. Fields can be of primitive data types (such as numeric, string, or date) or complex data types (such as ARRAY or MAP). A field within a struct can also be another STRUCT, ARRAY, or MAP, which allows you to create nested data structures, for example, `STRUCT, c MAP, d ARRAY>`. The STRUCT data type is supported from v3.1 onwards. In v3.1, you can define STRUCT columns when you create a StarRocks table, load STRUCT data into that table, and query MAP data. From v2.5 onwards, StarRocks supports querying complex data types MAP and STRUCT from data lakes. You can use external catalogs provided by StarRocks to query MAP and STRUCT data from Apache Hive™, Apache Hudi, and Apache Iceberg. You can only query data from ORC and Parquet files. For more information about how to use external catalogs to query external data sources, see [Overview of catalogs](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md) and topics related to the required catalog type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext STRUCT ``` * `name`: the field name, same as the column name defined in the CREATE TABLE statement. * `type`: the field type. It can be of any supported type. #### Define a STRUCT column in StarRocks[​](#define-a-struct-column-in-starrocks "Direct link to Define a STRUCT column in StarRocks") You can define a STRUCT column when you create a table and load STRUCT data into this column. ```sql -- Define a one-dimensional struct. CREATE TABLE t0( c0 INT, c1 STRUCT ) DUPLICATE KEY(c0); -- Define a complex struct. CREATE TABLE t1( c0 INT, c1 STRUCT, c MAP, d ARRAY> ) DUPLICATE KEY(c0); -- Define a NOT NULL struct. CREATE TABLE t2( c0 INT, c1 STRUCT NOT NULL ) DUPLICATE KEY(c0); ``` Columns with the STRUCT type have the following restrictions: * Cannot be used as key columns in a table. They can only be used as value columns. * Cannot be used as partition key columns (following PARTITION BY) in a table. * Cannot be used as bucketing columns (following DISTRIBUTED BY) in a table. * Only supports the replace() function when used as a value column in an [Aggregate table](https://docs.starrocks.io/docs/table_design/table_types/aggregate_table.md). #### Construct structs in SQL[​](#construct-structs-in-sql "Direct link to Construct structs in SQL") STRUCT can be constructed in SQL using the following functions: [row, struct](https://docs.starrocks.io/docs/sql-reference/sql-functions/struct-functions/row.md), and [named\_struct](https://docs.starrocks.io/docs/sql-reference/sql-functions/struct-functions/named_struct.md). struct() is the alias of row(). * `row` and `struct` support unnamed struct. You do not need to specify the field names. StarRocks automatically generates column names, like `col1`, `col2`... * `named_struct` supports named struct. The expressions of names and values must be in pairs. StarRocks automatically determines the type of the struct based on the input values. ```sql select row(1, 2, 3, 4) as numbers; -- Return {"col1":1,"col2":2,"col3":3,"col4":4}. select row(1, 2, null, 4) as numbers; -- Return {"col1":1,"col2":2,"col3":null,"col4":4}. select row(null) as nulls; -- Return {"col1":null}. select struct(1, 2, 3, 4) as numbers; -- Return {"col1":1,"col2":2,"col3":3,"col4":4}. select named_struct('a', 1, 'b', 2, 'c', 3, 'd', 4) as numbers; -- Return {"a":1,"b":2,"c":3,"d":4}. ``` #### Load STRUCT data[​](#load-struct-data "Direct link to Load STRUCT data") You can load STRUCT data into StarRocks using two methods: [INSERT INTO](https://docs.starrocks.io/docs/loading/InsertInto.md), and [ORC/Parquet loading](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). Note that StarRocks automatically casts the data type into the corresponding STRUCT type. When loading data to a named struct column, please ensure that the field names within the struct are in the same order as specified when creating the table. ##### INSERT INTO[​](#insert-into "Direct link to INSERT INTO") ```sql CREATE TABLE t0( c0 INT, c1 STRUCT ) DUPLICATE KEY(c0); INSERT INTO t0 VALUES(1, row(1, 1)); SELECT * FROM t0; +------+---------------+ | c0 | c1 | +------+---------------+ | 1 | {"a":1,"b":1} | +------+---------------+ ``` ##### Load STRUCT data from ORC/Parquet files[​](#load-struct-data-from-orcparquet-files "Direct link to Load STRUCT data from ORC/Parquet files") The STRUCT data type in StarRocks corresponds to the nested columns structure in ORC or Parquet format. No additional specification is needed. You can load STRUCT data from ORC or Parquet files by following the instructions in [ORC/Parquet loading](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). #### Access STRUCT fields[​](#access-struct-fields "Direct link to Access STRUCT fields") To query a subfield of a struct, you can use the dot (`.`) operator to query a value by its field name, or use `[]` to call a value by its index. ```plain mysql> select named_struct('a', 1, 'b', 2, 'c', 3, 'd', 4).a; +------------------------------------------------+ | named_struct('a', 1, 'b', 2, 'c', 3, 'd', 4).a | +------------------------------------------------+ | 1 | +------------------------------------------------+ mysql> select row(1, 2, 3, 4).col1; +-----------------------+ | row(1, 2, 3, 4).col1 | +-----------------------+ | 1 | +-----------------------+ mysql> select row(2, 4, 6, 8)[2]; +--------------------+ | row(2, 4, 6, 8)[2] | +--------------------+ | 4 | +--------------------+ mysql> select row(map{'a':1}, 2, 3, 4)[1]; +-----------------------------+ | row(map{'a':1}, 2, 3, 4)[1] | +-----------------------------+ | {"a":1} | +-----------------------------+ ``` --- ### VARIANT important The VARIANT type is supported only for tables in Iceberg Catalog. It is not supported by StarRocks native tables. From v4.1 onwards, StarRocks supports the VARIANT data type for querying semi-structured data from Iceberg tables in Parquet format. This article introduces the basic concepts of VARIANT, and how StarRocks queries VARIANT-type data and processes it through VARIANT functions. #### What is VARIANT[​](#what-is-variant "Direct link to What is VARIANT") VARIANT is a semi-structured data type that can store values of different data types, including primitive types (integers, floating-point numbers, strings, booleans, dates, timestamps), and complex types (structs, maps, arrays). VARIANT data is encoded in binary format for efficient storage and querying. The VARIANT type is particularly useful when working with data from Apache Iceberg tables that use the Parquet format with variant encoding, which allows for flexible schema evolution and efficient storage of heterogeneous data. For more information on the Parquet variant encoding format, see the [Parquet Variant Encoding specification](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md). #### Using VARIANT Data[​](#using-variant-data "Direct link to Using VARIANT Data") ##### Querying VARIANT-Type Data from Iceberg Tables[​](#querying-variant-type-data-from-iceberg-tables "Direct link to Querying VARIANT-Type Data from Iceberg Tables") StarRocks supports querying VARIANT-type data from Iceberg tables stored in Parquet format. VARIANT columns are automatically recognized when querying Iceberg tables that use Parquet's variant encoding. Example of querying an Iceberg table with VARIANT columns: ```sql -- Query a table with VARIANT columns SELECT id, variant_col FROM iceberg_catalog.db.table_with_variants; ``` ##### Extracting Values from VARIANT Data[​](#extracting-values-from-variant-data "Direct link to Extracting Values from VARIANT Data") StarRocks provides several functions to extract typed values from VARIANT data. Example 1: Extract primitive values using typed getter functions. ```sql SELECT get_variant_int(variant_col, '$') AS int_value, get_variant_string(variant_col, '$') AS string_value, get_variant_bool(variant_col, '$') AS bool_value, get_variant_double(variant_col, '$') AS double_value FROM iceberg_catalog.db.table_with_variants; ``` Example 2: Navigate nested structures using JSON path expressions. ```sql SELECT get_variant_string(variant_col, '$.user.name') AS user_name, get_variant_int(variant_col, '$.user.age') AS user_age, get_variant_string(variant_col, '$.address.city') AS city FROM iceberg_catalog.db.table_with_variants; ``` Example 3: Access array elements. ```sql SELECT get_variant_int(variant_col, '$.scores[0]') AS first_score, get_variant_int(variant_col, '$.scores[1]') AS second_score FROM iceberg_catalog.db.table_with_variants; ``` Example 4: Query nested variant data and return VARIANT type. ```sql SELECT variant_query(variant_col, '$.metadata') AS metadata, variant_query(variant_col, '$.items[0]') AS first_item FROM iceberg_catalog.db.table_with_variants; ``` Example 5: Check the type of a VARIANT value. ```sql SELECT variant_typeof(variant_col) AS root_type, variant_typeof(variant_query(variant_col, '$.data')) AS data_type FROM iceberg_catalog.db.table_with_variants; ``` ##### Casting JSON to VARIANT[​](#casting-json-to-variant "Direct link to Casting JSON to VARIANT") StarRocks supports casting JSON values to VARIANT. If your input is a STRING, convert it to JSON first. ```sql SELECT CAST(parse_json('{"id": 1, "flags": {"active": true}, "scores": [1.5, null]}') AS VARIANT) AS variant_value; ``` ```sql SELECT CAST(json_col AS VARIANT) AS variant_value FROM db.table_with_json; ``` ##### Casting VARIANT Data to SQL Types[​](#casting-variant-data-to-sql-types "Direct link to Casting VARIANT Data to SQL Types") You can use the CAST function to convert VARIANT data to standard SQL types: ```sql SELECT CAST(variant_query(variant_col, '$.count') AS INT) AS count, CAST(variant_query(variant_col, '$.price') AS DECIMAL(10, 2)) AS price, CAST(variant_query(variant_col, '$.active') AS BOOLEAN) AS is_active, CAST(variant_query(variant_col, '$.name') AS STRING) AS name FROM iceberg_catalog.db.table_with_variants; ``` Complex types can also be cast from VARIANT: ```sql SELECT CAST(variant_col AS STRUCT) AS user_struct, CAST(variant_col AS MAP) AS config_map, CAST(variant_col AS ARRAY) AS values_array FROM iceberg_catalog.db.table_with_variants; ``` ##### Casting SQL Types to VARIANT[​](#casting-sql-types-to-variant "Direct link to Casting SQL Types to VARIANT") You can cast SQL values to VARIANT. Supported input types include BOOLEAN, integer types, FLOAT/DOUBLE, DECIMAL, STRING/CHAR/VARCHAR, JSON, DATE/DATETIME/TIME, and complex types (ARRAY, MAP, STRUCT). For MAP, keys are cast to strings during encoding. Types such as HLL, BITMAP, PERCENTILE, and VARBINARY are not supported. ```sql SELECT CAST(123 AS VARIANT) AS v_int, CAST(3.14 AS VARIANT) AS v_double, CAST(CAST('12.34' AS DECIMAL(10, 2)) AS VARIANT) AS v_decimal, CAST('hello' AS VARIANT) AS v_string, CAST(PARSE_JSON('{"k":1}') AS VARIANT) AS v_json; ``` #### VARIANT Functions[​](#variant-functions "Direct link to VARIANT Functions") VARIANT functions can be used to query and extract data from VARIANT columns. For detailed information, please refer to the individual function documentation: * [variant\_query](https://docs.starrocks.io/docs/sql-reference/sql-functions/variant-functions/variant_query.md): Queries a path in a VARIANT value and returns a VARIANT * [get\_variant](https://docs.starrocks.io/docs/sql-reference/sql-functions/variant-functions/get_variant.md): Extracts typed values (int, bool, double, string) from a VARIANT * [variant\_typeof](https://docs.starrocks.io/docs/sql-reference/sql-functions/variant-functions/variant_typeof.md): Returns the type name of a VARIANT value #### VARIANT Path Expressions[​](#variant-path-expressions "Direct link to VARIANT Path Expressions") VARIANT functions use JSON path expressions to navigate through the data structure. The path syntax is similar to JSON path: * `$` represents the root of the VARIANT value * `.` is used to access object fields * `[index]` is used to access array elements (0-based indexing) * Field names with special characters (like dots) can be quoted: `$."field.name"` Examples of path expressions: ```plaintext $ -- Root element $.field -- Object field access $.nested.field -- Nested field access $."field.with.dots" -- Quoted field name $[0] -- First array element $.array[1] -- Second element of array field $.users[0].name -- Nested array access $.config["key"] -- Map-style access ``` #### Data Type Conversion[​](#data-type-conversion "Direct link to Data Type Conversion") When data is read from Parquet files with variant encoding, the following type conversions are supported: | Parquet Variant Type | StarRocks VARIANT Type | | ------------------------- | ------------------------- | | INT8, INT16, INT32, INT64 | int8, int16, int32, int64 | | FLOAT, DOUBLE | float, double | | BOOLEAN | boolean | | STRING | string | | DATE | Date | | TIMESTAMP, TIMESTAMP\_NTZ | Timestamp | | DECIMAL | Decimal, float, double | | STRUCT | Object | | MAP | Object | | ARRAY | Array | #### Limitations and Considerations[​](#limitations-and-considerations "Direct link to Limitations and Considerations") * VARIANT is supported for reading data from Iceberg tables in Parquet format with variant encoding, and for writing Parquet files using StarRocks file writers (unshredded variant encoding). * The size of a VARIANT value is limited to 16 MB. * Currently only unshredded variant values are supported for both read and write. * VARIANT can be created by casting from JSON values or supported SQL types (including ARRAY, MAP, and STRUCT). * The maximum depth of nested structures depends on the underlying Parquet file structure. --- ### String #### [📄️ BINARY/VARBINARY](https://docs.starrocks.io/docs/sql-reference/data-types/string-type/BINARY.md) [BINARY(M)](https://docs.starrocks.io/docs/sql-reference/data-types/string-type/BINARY.md) #### [📄️ CHAR](https://docs.starrocks.io/docs/sql-reference/data-types/string-type/CHAR.md) [CHAR(M)](https://docs.starrocks.io/docs/sql-reference/data-types/string-type/CHAR.md) #### [📄️ STRING](https://docs.starrocks.io/docs/sql-reference/data-types/string-type/STRING.md) [STRING](https://docs.starrocks.io/docs/sql-reference/data-types/string-type/STRING.md) #### [📄️ VARCHAR](https://docs.starrocks.io/docs/sql-reference/data-types/string-type/VARCHAR.md) [VARCHAR(M)](https://docs.starrocks.io/docs/sql-reference/data-types/string-type/VARCHAR.md) --- ### BINARY/VARBINARY BINARY(M) VARBINARY(M) Since v3.0, StarRocks supports the BINARY/VARBINARY data type, which is used to store binary data. The maximum supported length is the same as VARCHAR \[1, 1048576]. Unit: bytes. If `M` is not specified, 1048576 is used by default. Binary data types contain byte strings while character data types contain character strings. BINARY is an alias of VARBINARY. The usage is the same as VARBINARY. #### Limits and usage notes[​](#limits-and-usage-notes "Direct link to Limits and usage notes") * VARBINARY columns are supported in Duplicate Key, Primary Key, and Unique Key tables. They are not supported in Aggregate tables. * VARBINARY columns cannot be used as partition keys, bucketing keys, or dimension columns of Duplicate Key, Primary Key, and Unique Key tables. They cannot be used in ORDER BY, GROUP BY, and JOIN clauses. * BINARY(M)/VARBINARY(M) are not right-padded in the case of unaligned length. #### Examples[​](#examples "Direct link to Examples") ##### Create a column of VARBINARY type[​](#create-a-column-of-varbinary-type "Direct link to Create a column of VARBINARY type") When creating a table, use the keyword `VARBINARY` to specify column `j` as a VARBINARY column. ```sql CREATE TABLE `test_binary` ( `id` INT(11) NOT NULL COMMENT "", `j` VARBINARY NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`) PROPERTIES ( "replication_num" = "3", "storage_format" = "DEFAULT" ); mysql> DESC test_binary; +-------+-----------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+-----------+------+-------+---------+-------+ | id | int | NO | true | NULL | | | j | varbinary | YES | false | NULL | | +-------+-----------+------+-------+---------+-------+ 2 rows in set (0.01 sec) ``` ##### Load data and store it as BINARY type[​](#load-data-and-store-it-as-binary-type "Direct link to Load data and store it as BINARY type") StarRocks supports the following ways to load data and store it as BINARY type. * Method 1: Use INSERT INTO to write data to a constant column of BINARY type (such as column `j`), where the constant column is prefixed with `x''`. ```sql INSERT INTO test_binary (id, j) VALUES (1, x'abab'); INSERT INTO test_binary (id, j) VALUES (2, x'baba'); INSERT INTO test_binary (id, j) VALUES (3, x'010102'); INSERT INTO test_binary (id, j) VALUES (4, x'0000'); ``` * Method 2: Use the [to\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/binary-functions/to_binary.md) function to convert VARCHAR data to binary data. ```sql INSERT INTO test_binary select 5, to_binary('abab', 'hex'); INSERT INTO test_binary select 6, to_binary('abab', 'base64'); INSERT INTO test_binary select 7, to_binary('abab', 'utf8'); ``` * Method 3: Use Broker Load to load a Parquet or ORC file and store the file as BINARY data. For more information, see [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). * For Parquet files, convert `parquet::Type::type::BYTE_ARRAY` to `TYPE_VARBINARY` directly. * For ORC files, convert `orc::BINARY` to `TYPE_VARBINARY` directly. * Method 4: Use Stream Load to load a CSV file and store the file as `BINARY` data. For more information, see [Load CSV data](https://docs.starrocks.io/docs/loading/StreamLoad.md#load-csv-data). * CSV file uses the hex format for binary data. Please ensure the input binary value is a valid hex value. * `BINARY` type is only supported in CSV file. JSON file does not support `BINARY` type. For example, `t1` is a table with a VARBINARY column `b`. ```sql CREATE TABLE `t1` ( `k` int(11) NOT NULL COMMENT "", `v` int(11) NOT NULL COMMENT "", `b` varbinary ) ENGINE = OLAP DUPLICATE KEY(`k`) PARTITION BY RANGE(`v`) ( PARTITION p1 VALUES [("-2147483648"), ("0")), PARTITION p2 VALUES [("0"), ("10")), PARTITION p3 VALUES [("10"), ("20"))) DISTRIBUTED BY HASH(`k`) PROPERTIES ("replication_num" = "1"); -- csv file -- cat temp_data 0,0,ab -- Load CSV file using Stream Load. curl --location-trusted -u : -T temp_data -XPUT -H column_separator:, -H label:xx http://172.17.0.1:8131/api/test_mv/t1/_stream_load -- Query the loaded data. mysql> select * from t1; +------+------+------------+ | k | v | xx | +------+------+------------+ | 0 | 0 | 0xAB | +------+------+------------+ 1 rows in set (0.11 sec) ``` ##### Query and process BINARY data[​](#query-and-process-binary-data "Direct link to Query and process BINARY data") StarRocks supports querying and processing BINARY data, and supports the use of BINARY functions and operators. This example uses table `test_binary`. Note: If you add the `--binary-as-hex` option When you access StarRocks from your MySQL client, binary data will be displayed using hex notation. ```plain mysql> select * from test_binary; +------+------------+ | id | j | +------+------------+ | 1 | 0xABAB | | 2 | 0xBABA | | 3 | 0x010102 | | 4 | 0x0000 | | 5 | 0xABAB | | 6 | 0xABAB | | 7 | 0x61626162 | +------+------------+ 7 rows in set (0.08 sec) ``` Example 1: View binary data using the [hex](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/hex.md) function. ```plain mysql> select id, hex(j) from test_binary; +------+----------+ | id | hex(j) | +------+----------+ | 1 | ABAB | | 2 | BABA | | 3 | 010102 | | 4 | 0000 | | 5 | ABAB | | 6 | ABAB | | 7 | 61626162 | +------+----------+ 7 rows in set (0.02 sec) ``` Example 2: View binary data using the [to\_base64](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/to_base64.md) function. ```plain mysql> select id, to_base64(j) from test_binary; +------+--------------+ | id | to_base64(j) | +------+--------------+ | 1 | q6s= | | 2 | uro= | | 3 | AQEC | | 4 | AAA= | | 5 | q6s= | | 6 | q6s= | | 7 | YWJhYg== | +------+--------------+ 7 rows in set (0.01 sec) ``` Example 3: View binary data using the [from\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/binary-functions/from_binary.md) function. ```plain mysql> select id, from_binary(j, 'hex') from test_binary; +------+-----------------------+ | id | from_binary(j, 'hex') | +------+-----------------------+ | 1 | ABAB | | 2 | BABA | | 3 | 010102 | | 4 | 0000 | | 5 | ABAB | | 6 | ABAB | | 7 | 61626162 | +------+-----------------------+ 7 rows in set (0.01 sec) ``` --- ### CHAR CHAR(M) A fixed-length string, `M` represents the length of a fixed-length string. Unit: bytes. The range of `M` is \[1, 255]. --- ### STRING STRING A variable length string. The maximum length is 65533 bytes. --- ### VARCHAR VARCHAR(M) A variable-length string. `M` indicates the length of the string. The default value is `1`. Unit: bytes. * In versions earlier than StarRocks 2.1, the value range of `M` is \[1, 65533]. * \[Preview] In StarRocks 2.1 and later versions, the value range of `M` is \[1, 1048576]. #### Examples[​](#examples "Direct link to Examples") Create a table and specify the column type as VARCHAR. ```sql CREATE TABLE varcharDemo ( pk INT COMMENT "range [-2147483648, 2147483647]", pd_type VARCHAR(20) COMMENT "range char(m),m in (1-65533) " ) ENGINE=OLAP DUPLICATE KEY(pk) COMMENT "OLAP" DISTRIBUTED BY HASH(pk) ``` --- ### Error Codes This section describes the common error codes for query requests. | Serial number | Error message | | ------------- | --------------------------------------------------------------------------------------------------------------- | | 1005 | Table creation failed. Returns a specific reason. | | 1007 | Cannot create a database with the same name. | | 1008 | Cannot delete non-existent database. | | 1044 | Cannot access unauthorized database. | | 1045 | Username and password do not match and therefore cannot access the system. | | 1046 | The target database is not specified. | | 1047 | An invalid command is specified. | | 1049 | An invalid database is specified. | | 1050 | A table with the same name already exists. | | 1051 | An invalid table is specified. | | 1052 | The specified column name is ambiguous and therefore the corresponding column cannot be uniquely identified. | | 1053 | An illegal data column was specified for the Semi-Join/Anti-Join query. | | 1054 | The specified column does not exist in the table. | | 1058 | The number of columns selected in the query statement does not match the number of columns in the query result. | | 1060 | There are duplicate column names. | | 1064 | There is no surviving BE node. | | 1066 | Duplicate table alias appears in the query statement. | | 1094 | Thread ID is invalid. | | 1095 | The non-owner of a thread cannot terminate the thread. | | 1096 | The query statement does not specify the table to be queried. | | 1102 | The database name is incorrect. | | 1104 | The table name is incorrect. | | 1105 | Other errors. | | 1110 | Duplicate columns were specified in the subquery. | | 1111 | Illegal use of aggregation function in `WHERE` clause | | 1113 | The set of columns in the new table cannot be empty. | | 1115 | Unsupported character sets are used. | | 1130 | An unauthorized IP address is used by the client. | | 1132 | No permission to change user password. | | 1141 | Specified entries don’t have privileges to revoke. | | 1142 | An unauthorized action was performed. | | 1166 | The data column name is incorrect. | | 1193 | System variable has invalid name | | 1203 | The number of active connections used exceeded the limit. | | 1211 | Not allowed to create new users. | | 1227 | The user has performed an out-of-authority operation. | | 1228 | Session variables cannot be modified by the `SET GLOBAL` command. | | 1229 | Global variables should be modified by the `SET GLOBAL` command. | | 1230 | Related system variables do not have default values. | | 1231 | An invalid value was set for a system variable. | | 1232 | A value of the wrong data type was set for a system variable. | | 1248 | No alias was set for an inline view. | | 1251 | The client does not support the user authentication protocol required by the server. | | 1286 | The storage engine is incorrectly configured. | | 1298 | The time zone is incorrectly configured. | | 1347 | The object does not match the expected type. | | 1353 | The specified column number in the `Select` clause of the view is not equal to the defined column number. | | 1364 | No default value is set for columns that do not allow NULL values. | | 1372 | The password is not long enough. | | 1396 | The performed operation failed. | | 1471 | The specified table is not allowed to insert data. | | 1507 | Delete nonexistent partition, and no condition is specified to only delete existing partitions. | | 1508 | All partitions should be deleted by a delete table operation. | | 1517 | There are duplicate partition names. | | 1524 | The specified plugin has not been loaded. | | 1567 | The name of the partition is incorrect. | | 1621 | The specified system variable is read-only. | | 1735 | The specified partition name does not exist in the table. | | 1748 | Data cannot be inserted into a table that does not have a partition. | | 1749 | The specified partition does not exist. | | 5000 | The specified table is not an OLAP table. | | 5001 | The specified storage path is invalid. | | 5002 | The name of the specified column should be displayed. | | 5003 | The dimension column should be preceded by the index column. | | 5004 | The table should contain at least 1 dimension column. | | 5005 | The cluster ID is invalid. | | 5006 | Invalid query plan. | | 5007 | Conflicting query plans. | | 5008 | The data insert is only available for data tables with partitions. | | 5009 | The `PARTITION` clause cannot be used to insert data into tables without partitions. | | 5010 | The number of columns in the table to be created is not equal to the number of columns in the `SELECT` clause. | | 5011 | Table reference could not be accessed. | | 5012 | The specified value is not a valid number. | | 5013 | The time unit is not supported. | | 5014 | The table status is not normal. | | 5015 | The partition status is not normal. | | 5016 | A data import job exists in the partition. | | 5017 | The specified column is not a dimension column. | | 5018 | The format of the value is invalid. | | 5019 | The data replica does not match the version. | | 5021 | The BE node is offline. | | 5022 | The number of partitions in a non-partitioned table is not 1 | | 5023 | No action was specified in the statement used to modify the table or data. | | 5024 | Job execution timed out. | | 5025 | Data insertion failed. | | 5026 | An unsupported data type was used when creating a table via the `SELECT` statement. | | 5027 | The specified parameter was not set. | | 5028 | The specified cluster was not found. | | 5030 | The user does not have permission to access the cluster. | | 5031 | No parameter specified or invalid parameter. | | 5032 | The number of cluster instances was not specified. | | 5034 | A cluster with the same name already exists. | | 5035 | The number of cluster instances is incorrectly configured. | | 5036 | Insufficient BE nodes in the cluster. | | 5037 | All databases should be deleted before deleting the cluster. | | 5038 | The BE node with the specified ID does not exist in the cluster. | | 5040 | No cluster with the same name exists. | | 5042 | No permissions. | | 5043 | The number of instances should be greater than 0. | | 5046 | Source cluster does not exist. | | 5047 | Destination cluster does not exist. | | 5048 | Source database does not exist. | | 5049 | Destination database does not exist. | | 5050 | No cluster selected. | | 5051 | The source database should be associated with the destination database first. | | 5052 | Intra-cluster error: BE node information is incorrect. | | 5053 | There is no migration task from the source database to the destination database. | | 5054 | The specified database has been associated with the destination database, or data is being migrated. | | 5055 | Database associations or data migrations cannot be performed within the same cluster. | | 5056 | Database cannot be deleted: it is associated with another database or data is being migrated. | | 5056 | Database cannot be renamed: it is associated with another database or data is being migrated. | | 5056 | Insufficient BE nodes in the cluster. | | 5056 | The specified number of BE nodes already exists in the cluster. | | 5059 | There are BE nodes in the cluster that are in the offline state. | | 5063 | The type name is incorrect. | | 5064 | Generic error message. | | 5063 | The Colocate feature has been disabled by the administrator. | | 5063 | A colocate data table with the same name does not exist. | | 5063 | The Colocate table must be an OLAP table. | | 5063 | Colocate tables should have the same number of replicas. | | 5063 | Colocate tables should have the same number of split buckets. | | 5063 | Colocate tables should have the same number of partition columns. | | 5063 | Colocate tables should have the same data type of partitioned columns. | | 5064 | The specified table is not a colocate table. | | 5065 | The specified operation is invalid. | | 5065 | The specified time unit is illegal. The correct units are `DAY`, `WEEK`, and `MONTH`. | | 5066 | The start value of the dynamic partition should be less than 0. | | 5066 | The start value of the dynamic partition is not a valid number. | | 5066 | The end value of the dynamic partition should be greater than 0. | | 5066 | The end value of the dynamic partition is not a valid number. | | 5066 | The end value of the dynamic partition is null. | | 5067 | The bucket number of the dynamic partition should be greater than 0. | | 5067 | The bucket number of the dynamic partition is not a valid number. | | 5066 | The bucket number of the dynamic partition is empty. | | 5068 | Whether to allow dynamic partition where the value is not a valid boolean: true or false. | | 5069 | The name of the specified dynamic partition has an illegal prefix. | | 5070 | The specified operation is disabled. | | 5071 | The number of replicas of the dynamic partition should be greater than 0. | | 5072 | The number of replicas of the dynamic partition is not a valid number. | --- ### HTTP SQL API StarRocks v3.2.0 introduces the HTTP SQL API for users to perform various types of queries using HTTP. Currently, this API supports SELECT, SHOW, EXPLAIN, and KILL statements. Syntax using the curl command: ```shell curl -X POST 'http://:/api/v1/catalogs//databases//sql' \ -u ':' -d '{"query": ";", "sessionVariables":{"":}}' \ --header "Content-Type: application/json" ``` #### Request message[​](#request-message "Direct link to Request message") ##### Request line[​](#request-line "Direct link to Request line") ```shell POST 'http://:/api/v1/catalogs//databases//sql' ``` | Field | Description | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | fe\_ip | FE node IP address. | | fe\_http\_port | FE HTTP port. | | catalog\_name | The catalog name. In v3.2.0, you can use this API to query only StarRocks internal tables, which means `` can only be set to `default_catalog`. Since v3.2.1, you can use this API to query tables in [external catalogs](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md). | | database\_name | The database name. If no database name is specified in the request line and a table name is used in the SQL query, you must prefix the table name with its database name, for example, `database_name.table_name`. | * Query data across databases in a specified catalog. If a table is used in the SQL query, you must prefix the table name with its database name. ```shell POST /api/v1/catalogs//sql ``` * Query data from a specified catalog and database. ```shell POST /api/v1/catalogs//databases//sql ``` ##### Authentication method[​](#authentication-method "Direct link to Authentication method") ```shell Authorization: Basic ``` Basic authentication is used, that is, enter the username and password for `credentials` (`-u ':'`). If no password is set for the username, you can pass in only `:` and leave the password empty. For example, if the root account is used, you can enter `-u 'root:'`. ##### Request body[​](#request-body "Direct link to Request body") ```shell -d '{"query": ";", "sessionVariables":{"":}}' ``` | Field | Description | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | query | The SQL query, in STRING format. Only SELECT, SHOW, EXPLAIN, and KILL statements are supported. You can run only one SQL query for an HTTP request. | | sessionVariables | The [session variable](https://docs.starrocks.io/docs/sql-reference/System_variable.md) you want to set for the query, in JSON format. This field is optional. Default is empty. The session variable you set takes effect for the same connection and becomes ineffective when the connection is closed. | ##### Request header[​](#request-header "Direct link to Request header") ```shell --header "Content-Type: application/json" ``` This header indicates that the request body is a JSON string. #### Response message[​](#response-message "Direct link to Response message") ##### Status code[​](#status-code "Direct link to Status code") * 200: HTTP request succeeded and the server is normal before data is sent to the client. * 4xx: HTTP request error, which indicates a client error. * `500 Internal Server Error`: HTTP request succeeded but the server encounters an error before data is sent to the client. * 503: HTTP request succeeded but the FE cannot provide service. ##### Response header[​](#response-header "Direct link to Response header") `content-type` indicates the format of the response body. Newline delimited JSON is used, which means the response body consists of multiple JSON objects that are separated by `\n`. | | Description | | -------------------- | --------------------------------------------------------------------------------------- | | content-type | The format is Newline delimited JSON, defaults to "application/x-ndjson charset=UTF-8". | | X-StarRocks-Query-Id | Query ID. | ##### Response body[​](#response-body "Direct link to Response body") ###### Failed before the request is sent[​](#failed-before-the-request-is-sent "Direct link to Failed before the request is sent") Request fails on the client side or the server encounters an error before it returns data to the client. The response body is in the following format, where `msg` is the error information. ```json { "status":"FAILED", "msg":"xxx" } ``` ###### Failed after the request is sent[​](#failed-after-the-request-is-sent "Direct link to Failed after the request is sent") Part of the result is returned and the HTTP status code is 200. Data sending is suspended, the connection is closed, and the error is logged. ###### Succeeded[​](#succeeded "Direct link to Succeeded") Each row in the response message is a JSON object. JSON objects separated by `\n`. * For a SELECT statement, the following JSON objects are returned. | Object | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------- | | `connectionId` | Connection ID. You can cancel a query that is pending for a long time by calling KILL ``. | | `meta` | Represents a column. The key is `meta` and the value is a JSON array, where each object in the array represents a column. | | `data` | The data row, where the key is `data` and the value is a JSON array which contains a row of data. | | `statistics` | Statistical information of the query. | * For a SHOW statement, `meta`, `data`, and `statistics` are returned. * For the EXPLAIN statement, an `explain` object is returned to show detailed execution plan of the query. The following example uses `\n` as the separator. StarRocks transmits data using HTTP chunked mode. Each time the FE obtains a data chunk, it streams the data chunk to the client. The client can parse data by row, which eliminates the need for data caching and the need to wait for the entire data, reducing memory consumption for the client. ```json {"connectionId": 7}\n {"meta": [ { "name": "stock_symbol", "type": "varchar" }, { "name": "closing_price", "type": "decimal64(8, 2)" }, { "name": "closing_date", "type": "datetime" } ]}\n {"data": ["JDR", 12.86, "2014-10-02 00:00:00"]}\n {"data": ["JDR",14.8, "2014-10-10 00:00:00"]}\n ... {"statistics": {"scanRows": 0,"scanBytes": 0,"returnRows": 9}} ``` #### Examples[​](#examples "Direct link to Examples") ##### Run a SELECT query[​](#run-a-select-query "Direct link to Run a SELECT query") * Query data from a StarRocks internal table (`catalog_name` is `default_catalog`). ```shell curl -X POST 'http://127.0.0.1:8030/api/v1/catalogs/default_catalog/databases/test/sql' -u 'root:' -d '{"query": "select * from agg;"}' --header "Content-Type: application/json" ``` Result: ```json {"connectionId":49} {"meta":[{"name":"no","type":"int(11)"},{"name":"k","type":"decimal64(10, 2)"},{"name":"v","type":"decimal64(10, 2)"}]} {"data":[1,"10.00",null]} {"data":[2,"10.00","11.00"]} {"data":[2,"20.00","22.00"]} {"data":[2,"25.00",null]} {"data":[2,"30.00","35.00"]} {"statistics":{"scanRows":0,"scanBytes":0,"returnRows":5}} ``` * Query data from an Iceberg table. ```shell curl -X POST 'http://172.26.93.145:8030/api/v1/catalogs/iceberg_catalog/databases/ywb/sql' -u 'root:' -d '{"query": "select * from iceberg_analyze;"}' --header "Content-Type: application/json" ``` Result: ```json {"connectionId":13} {"meta":[{"name":"k1","type":"int(11)"},{"name":"k2","type":"int(11)"}]} {"data":[1,2]} {"data":[1,1]} {"statistics":{"scanRows":0,"scanBytes":0,"returnRows":2}} ``` ##### Cancel a query[​](#cancel-a-query "Direct link to Cancel a query") To cancel a query that run an unexpected long time, you can close the connection. StarRocks will cancel this query when it detects the connection is closed. You can also call KILL `connectionId` to cancel this query. For example: ```shell curl -X POST 'http://127.0.0.1:8030/api/v1/catalogs/default_catalog/databases/test/sql' -u 'root:' -d '{"query": "kill 17;"}' --header "Content-Type: application/json" ``` You can obtain the `connectionId` from the response body or by calling SHOW PROCESSLIST. For example: ```shell curl -X POST 'http://127.0.0.1:8030/api/v1/catalogs/default_catalog/databases/test/sql' -u 'root:' -d '{"query": "show processlist;"}' --header "Content-Type: application/json" ``` ##### Run a query with a session variable set for this query[​](#run-a-query-with-a-session-variable-set-for-this-query "Direct link to Run a query with a session variable set for this query") ```shell curl -X POST 'http://127.0.0.1:8030/api/v1/catalogs/default_catalog/databases/test/sql' -u 'root:' -d '{"query": "SHOW VARIABLES;", "sessionVariables":{"broadcast_row_limit":14000000}}' --header "Content-Type: application/json" ``` --- ### Information Schema The StarRocks Information Schema is a database within each StarRocks instance. Information Schema contains several read-only, system-defined views that store extensive metadata information of all objects that the StarRocks instance maintains. The StarRocks Information Schema is based on the SQL-92 ANSI Information Schema, but with the addition of views and functions that are specific to StarRocks. From v3.2.0, The StarRocks Information Schema supports manage metadata for external catalogs. #### View metadata via Information Schema[​](#view-metadata-via-information-schema "Direct link to View metadata via Information Schema") You can view the metadata information within a StarRocks instance by querying the content of views in Information Schema. The following example checks metadata information about a table named `table1` in StarRocks by querying the view `tables`. ```plain MySQL > SELECT * FROM information_schema.tables WHERE TABLE_NAME like 'table1'\G *************************** 1. row *************************** TABLE_CATALOG: def TABLE_SCHEMA: test_db TABLE_NAME: table1 TABLE_TYPE: BASE TABLE ENGINE: StarRocks VERSION: NULL ROW_FORMAT: TABLE_ROWS: 4 AVG_ROW_LENGTH: 1657 DATA_LENGTH: 6630 MAX_DATA_LENGTH: NULL INDEX_LENGTH: NULL DATA_FREE: NULL AUTO_INCREMENT: NULL CREATE_TIME: 2023-06-13 11:37:00 UPDATE_TIME: 2023-06-13 11:38:06 CHECK_TIME: NULL TABLE_COLLATION: utf8_general_ci CHECKSUM: NULL CREATE_OPTIONS: TABLE_COMMENT: 1 row in set (0.01 sec) ``` #### Views in Information Schema[​](#views-in-information-schema "Direct link to Views in Information Schema") The StarRocks Information Schema contains the following metadata views: #### [📄️ analyze\_status](https://docs.starrocks.io/docs/sql-reference/information_schema/analyze_status.md) [analyze\_status provides information about the status of analyze jobs.](https://docs.starrocks.io/docs/sql-reference/information_schema/analyze_status.md) #### [📄️ applicable\_roles](https://docs.starrocks.io/docs/sql-reference/information_schema/applicable_roles.md) [applicable\_roles provides information about roles that are applicable to the current user.](https://docs.starrocks.io/docs/sql-reference/information_schema/applicable_roles.md) #### [📄️ be\_bvars](https://docs.starrocks.io/docs/sql-reference/information_schema/be_bvars.md) [be\_bvars provides statistical information regarding bRPC.](https://docs.starrocks.io/docs/sql-reference/information_schema/be_bvars.md) #### [📄️ be\_cloud\_native\_compactions](https://docs.starrocks.io/docs/sql-reference/information_schema/be_cloud_native_compactions.md) [be\_cloud\_native\_compactions provides information on compaction transactions running on CNs (or BEs for v3.0) of a shared-data cluster.](https://docs.starrocks.io/docs/sql-reference/information_schema/be_cloud_native_compactions.md) #### [📄️ be\_compactions](https://docs.starrocks.io/docs/sql-reference/information_schema/be_compactions.md) [be\_compactions provides statistical information on compaction tasks.](https://docs.starrocks.io/docs/sql-reference/information_schema/be_compactions.md) #### [📄️ be\_configs](https://docs.starrocks.io/docs/sql-reference/information_schema/be_configs.md) [be\_configs provides information about the configuration parameters of each BE node.](https://docs.starrocks.io/docs/sql-reference/information_schema/be_configs.md) #### [📄️ be\_logs](https://docs.starrocks.io/docs/sql-reference/information_schema/be_logs.md) [be\_logs provides information about the logs of each BE node.](https://docs.starrocks.io/docs/sql-reference/information_schema/be_logs.md) #### [📄️ be\_metrics](https://docs.starrocks.io/docs/sql-reference/information_schema/be_metrics.md) [be\_metrics provides information about the metrics of each BE node.](https://docs.starrocks.io/docs/sql-reference/information_schema/be_metrics.md) #### [📄️ be\_tablets](https://docs.starrocks.io/docs/sql-reference/information_schema/be_tablets.md) [be\_tablets provides information about tablets on each BE node.](https://docs.starrocks.io/docs/sql-reference/information_schema/be_tablets.md) #### [📄️ be\_threads](https://docs.starrocks.io/docs/sql-reference/information_schema/be_threads.md) [be\_threads provides information about the threads running on each BE node.](https://docs.starrocks.io/docs/sql-reference/information_schema/be_threads.md) #### [📄️ be\_txns](https://docs.starrocks.io/docs/sql-reference/information_schema/be_txns.md) [be\_txns provides information about transactions on each BE node.](https://docs.starrocks.io/docs/sql-reference/information_schema/be_txns.md) #### [📄️ character\_sets](https://docs.starrocks.io/docs/sql-reference/information_schema/character_sets.md) [character\_sets identifies the character sets available.](https://docs.starrocks.io/docs/sql-reference/information_schema/character_sets.md) #### [📄️ collations](https://docs.starrocks.io/docs/sql-reference/information_schema/collations.md) [collations contains the available collations.](https://docs.starrocks.io/docs/sql-reference/information_schema/collations.md) #### [📄️ column\_privileges](https://docs.starrocks.io/docs/sql-reference/information_schema/column_privileges.md) [Information schema placeholder view not implemented in StarRocks.](https://docs.starrocks.io/docs/sql-reference/information_schema/column_privileges.md) #### [📄️ column\_stats\_usage](https://docs.starrocks.io/docs/sql-reference/information_schema/column_stats_usage.md) [column\_stats\_usage provides information about the usage of column statistics.](https://docs.starrocks.io/docs/sql-reference/information_schema/column_stats_usage.md) #### [📄️ columns](https://docs.starrocks.io/docs/sql-reference/information_schema/columns.md) [columns contains information about all table columns (or view columns).](https://docs.starrocks.io/docs/sql-reference/information_schema/columns.md) #### [📄️ engines](https://docs.starrocks.io/docs/sql-reference/information_schema/engines.md) [Information schema placeholder view not implemented in StarRocks.](https://docs.starrocks.io/docs/sql-reference/information_schema/engines.md) #### [📄️ events](https://docs.starrocks.io/docs/sql-reference/information_schema/events.md) [Information schema placeholder view not implemented in StarRocks.](https://docs.starrocks.io/docs/sql-reference/information_schema/events.md) #### [📄️ fe\_metrics](https://docs.starrocks.io/docs/sql-reference/information_schema/fe_metrics.md) [fe\_metrics provides information about the metrics of each FE node.](https://docs.starrocks.io/docs/sql-reference/information_schema/fe_metrics.md) #### [📄️ fe\_tablet\_schedules](https://docs.starrocks.io/docs/sql-reference/information_schema/fe_tablet_schedules.md) [fe\_tablet\_schedules provides information about tablet scheduling tasks on FE nodes.](https://docs.starrocks.io/docs/sql-reference/information_schema/fe_tablet_schedules.md) #### [📄️ fe\_threads](https://docs.starrocks.io/docs/sql-reference/information_schema/fe_threads.md) [fe\_threads provides information about the threads running on each FE node.](https://docs.starrocks.io/docs/sql-reference/information_schema/fe_threads.md) #### [📄️ global\_variables](https://docs.starrocks.io/docs/sql-reference/information_schema/global_variables.md) [global\_variables provides information about global variables.](https://docs.starrocks.io/docs/sql-reference/information_schema/global_variables.md) #### [📄️ key\_column\_usage](https://docs.starrocks.io/docs/sql-reference/information_schema/key_column_usage.md) [Information schema placeholder view not implemented in StarRocks.](https://docs.starrocks.io/docs/sql-reference/information_schema/key_column_usage.md) #### [📄️ load\_tracking\_logs](https://docs.starrocks.io/docs/sql-reference/information_schema/load_tracking_logs.md) [load\_tracking\_logs provides error logs of load jobs.](https://docs.starrocks.io/docs/sql-reference/information_schema/load_tracking_logs.md) #### [📄️ loads](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) [loads provides the results of load jobs.](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) #### [📄️ materialized\_view\_refresh\_jobs](https://docs.starrocks.io/docs/sql-reference/information_schema/materialized_view_refresh_jobs.md) [materialized\_view\_refresh\_jobs provides job-level information about materialized view refreshes.](https://docs.starrocks.io/docs/sql-reference/information_schema/materialized_view_refresh_jobs.md) #### [📄️ materialized\_views](https://docs.starrocks.io/docs/sql-reference/information_schema/materialized_views.md) [materialized\_views provides information about all materialized views.](https://docs.starrocks.io/docs/sql-reference/information_schema/materialized_views.md) #### [📄️ partitions](https://docs.starrocks.io/docs/sql-reference/information_schema/partitions.md) [Information schema placeholder view not implemented in StarRocks.](https://docs.starrocks.io/docs/sql-reference/information_schema/partitions.md) #### [📄️ partitions\_meta](https://docs.starrocks.io/docs/sql-reference/information_schema/partitions_meta.md) [partitions\_meta provides information about partitions of tables.](https://docs.starrocks.io/docs/sql-reference/information_schema/partitions_meta.md) #### [📄️ pipe\_files](https://docs.starrocks.io/docs/sql-reference/information_schema/pipe_files.md) [pipe\_files provides the status of the data files to be loaded via a specified pipe.](https://docs.starrocks.io/docs/sql-reference/information_schema/pipe_files.md) #### [📄️ pipes](https://docs.starrocks.io/docs/sql-reference/information_schema/pipes.md) [pipes provides information about all pipes stored in the current or specified database.](https://docs.starrocks.io/docs/sql-reference/information_schema/pipes.md) #### [📄️ recyclebin\_catalogs](https://docs.starrocks.io/docs/sql-reference/information_schema/recyclebin_catalogs.md) [recyclebin\_catalogs provides metadata information of deleted databases, tables, and partitions temporarily stored in the FE recycle bin.](https://docs.starrocks.io/docs/sql-reference/information_schema/recyclebin_catalogs.md) #### [📄️ referential\_constraints](https://docs.starrocks.io/docs/sql-reference/information_schema/referential_constraints.md) [Information schema placeholder view not implemented in StarRocks.](https://docs.starrocks.io/docs/sql-reference/information_schema/referential_constraints.md) #### [📄️ routine\_load\_jobs](https://docs.starrocks.io/docs/sql-reference/information_schema/routine_load_jobs.md) [routine\_load\_jobs provides information about routine load jobs.](https://docs.starrocks.io/docs/sql-reference/information_schema/routine_load_jobs.md) #### [📄️ routines](https://docs.starrocks.io/docs/sql-reference/information_schema/routines.md) [Information schema placeholder view not implemented in StarRocks.](https://docs.starrocks.io/docs/sql-reference/information_schema/routines.md) #### [📄️ schema\_privileges](https://docs.starrocks.io/docs/sql-reference/information_schema/schema_privileges.md) [Information schema placeholder view not implemented in StarRocks.](https://docs.starrocks.io/docs/sql-reference/information_schema/schema_privileges.md) #### [📄️ schemata](https://docs.starrocks.io/docs/sql-reference/information_schema/schemata.md) [schemata provides information about databases.](https://docs.starrocks.io/docs/sql-reference/information_schema/schemata.md) #### [📄️ session\_variables](https://docs.starrocks.io/docs/sql-reference/information_schema/session_variables.md) [session\_variables provides information about session variables.](https://docs.starrocks.io/docs/sql-reference/information_schema/session_variables.md) #### [📄️ statistics](https://docs.starrocks.io/docs/sql-reference/information_schema/statistics.md) [Information schema placeholder view not implemented in StarRocks.](https://docs.starrocks.io/docs/sql-reference/information_schema/statistics.md) #### [📄️ stream\_loads](https://docs.starrocks.io/docs/sql-reference/information_schema/stream_loads.md) [stream\_loads provides information about stream load jobs.](https://docs.starrocks.io/docs/sql-reference/information_schema/stream_loads.md) #### [📄️ table\_constraints](https://docs.starrocks.io/docs/sql-reference/information_schema/table_constraints.md) [Information schema placeholder view not implemented in StarRocks.](https://docs.starrocks.io/docs/sql-reference/information_schema/table_constraints.md) #### [📄️ table\_privileges](https://docs.starrocks.io/docs/sql-reference/information_schema/table_privileges.md) [Information schema placeholder view not implemented in StarRocks.](https://docs.starrocks.io/docs/sql-reference/information_schema/table_privileges.md) #### [📄️ tables](https://docs.starrocks.io/docs/sql-reference/information_schema/tables.md) [tables provides information about tables.](https://docs.starrocks.io/docs/sql-reference/information_schema/tables.md) #### [📄️ tables\_config](https://docs.starrocks.io/docs/sql-reference/information_schema/tables_config.md) [tables\_config provides information about the configuration of tables.](https://docs.starrocks.io/docs/sql-reference/information_schema/tables_config.md) #### [📄️ task\_runs](https://docs.starrocks.io/docs/sql-reference/information_schema/task_runs.md) [task\_runs provides information about the execution of asynchronous tasks.](https://docs.starrocks.io/docs/sql-reference/information_schema/task_runs.md) #### [📄️ tasks](https://docs.starrocks.io/docs/sql-reference/information_schema/tasks.md) [tasks provides information about asynchronous tasks.](https://docs.starrocks.io/docs/sql-reference/information_schema/tasks.md) #### [📄️ triggers](https://docs.starrocks.io/docs/sql-reference/information_schema/triggers.md) [Information schema placeholder view not implemented in StarRocks.](https://docs.starrocks.io/docs/sql-reference/information_schema/triggers.md) #### [📄️ user\_privileges](https://docs.starrocks.io/docs/sql-reference/information_schema/user_privileges.md) [Information schema placeholder view not implemented in StarRocks.](https://docs.starrocks.io/docs/sql-reference/information_schema/user_privileges.md) #### [📄️ verbose\_session\_variables](https://docs.starrocks.io/docs/sql-reference/information_schema/verbose_session_variables.md) [verbose\_session\_variables provides session variable details including default values and whether they have been modified.](https://docs.starrocks.io/docs/sql-reference/information_schema/verbose_session_variables.md) #### [📄️ views](https://docs.starrocks.io/docs/sql-reference/information_schema/views.md) [views provides information about all user-defined views.](https://docs.starrocks.io/docs/sql-reference/information_schema/views.md) #### [📄️ warehouse\_metrics](https://docs.starrocks.io/docs/sql-reference/information_schema/warehouse_metrics.md) [warehouse\_metrics provides information about the metrics of each warehouse.](https://docs.starrocks.io/docs/sql-reference/information_schema/warehouse_metrics.md) #### [📄️ warehouse\_queries](https://docs.starrocks.io/docs/sql-reference/information_schema/warehouse_queries.md) [warehouse\_queries provides information about queries running on each warehouse.](https://docs.starrocks.io/docs/sql-reference/information_schema/warehouse_queries.md) --- ### analyze_status `analyze_status` provides information about the status of analyze jobs. The following fields are provided in `analyze_status`: | **Field** | **Description** | | ---------- | ---------------------------------------------------------------------------------- | | Id | ID of the analyze job. | | Catalog | Catalog where the table belongs. | | Database | Database where the table belongs. | | Table | Name of the table being analyzed. | | Columns | Columns being analyzed. | | Type | Type of analyze job. Valid values: `FULL`, `SAMPLE`. | | Schedule | Schedule type of the analyze job. Valid values: `ONCE`, `AUTOMATIC`. | | Status | Status of the analyze job. Valid values: `PENDING`, `RUNNING`, `FINISH`, `FAILED`. | | StartTime | Start time of the analyze job. | | EndTime | End time of the analyze job. | | Properties | Properties of the analyze job. | | Reason | Reason for the analyze job status. | --- ### applicable_roles `applicable_roles` provides information about roles that are applicable to the current user. The following fields are provided in `applicable_roles`: | **Field** | **Description** | | ------------- | ---------------------------------------------------------- | | USER | The user to whom the role is applicable. | | HOST | The host from which the user connects. | | GRANTEE | The user or role that granted the role. | | GRANTEE\_HOST | The host from which the grantor connects. | | ROLE\_NAME | The name of the role. | | ROLE\_HOST | The host associated with the role. | | IS\_GRANTABLE | Whether the role can be granted to others (`YES` or `NO`). | | IS\_DEFAULT | Whether the role is a default role (`YES` or `NO`). | | IS\_MANDATORY | Whether the role is mandatory (`YES` or `NO`). | --- ### be_bvars `be_bvars` provides statistical information regarding bRPC. You can view RPC latency, QPS, and other statistics for some components of StarRocks. The following fields are provided in `be_bvars`: | **Field** | **Description** | | --------- | --------------------------------------- | | BE\_ID | ID of the BE where the bvar is located. | | NAME | Name of the bvar. | | VALUE | Value of the bvar. | --- ### be_cloud_native_compactions `be_cloud_native_compactions` provides information on compaction transactions running on CNs (or BEs for v3.0) of a shared-data cluster. A compaction transaction is divided into multiple tasks on the partition level, and each row in `be_cloud_native_compactions` corresponds to a task in the compaction transaction. The following fields are provided in `be_cloud_native_compactions`: | **Field** | **Description** | | ------------ | ------------------------------------------------------------------------------------------------------------------- | | BE\_ID | ID of the CN (BE). | | TXN\_ID | ID of the compaction transaction. It can be duplicated because each compaction transaction may have multiple tasks. | | TABLET\_ID | ID of the tablet the task corresponds to. | | VERSION | Version of the data that is input to the task. | | SKIPPED | Whether the task is skipped. | | RUNS | Number of task executions. A value greater than `1` indicates that retries have occurred. | | START\_TIME | Task start time. | | FINISH\_TIME | Task finish time. `NULL` is returned if the task is still in progress. | | PROGRESS | Percentage of progress, ranging from 0 to 100. | | STATUS | Task status. | | PROFILE | Real time profile, including data read time、data write time、in-queue time, etc. | --- ### be_compactions `be_compactions` provides statistical information on compaction tasks. The following fields are provided in `be_compactions`: | **Field** | **Description** | | ----------------------------------- | ------------------------------------------------------- | | BE\_ID | ID of the BE. | | CANDIDATES\_NUM | Number of candidates for compaction tasks. | | BASE\_COMPACTION\_CONCURRENCY | Number of base compaction tasks that are running. | | CUMULATIVE\_COMPACTION\_CONCURRENCY | Number of cumulative compaction tasks that are running. | | LATEST\_COMPACTION\_SCORE | Compaction score of the last compaction task. | | CANDIDATE\_MAX\_SCORE | The maximum compaction score of the task candidate. | | MANUAL\_COMPACTION\_CONCURRENCY | Number of manual compaction tasks that are running. | | MANUAL\_COMPACTION\_CANDIDATES\_NUM | Number of candidates for manual compaction tasks. | --- ### be_configs `be_configs` provides information about the configuration parameters of each BE node. The following fields are provided in `be_configs`: | **Field** | **Description** | | --------- | ---------------------------------------------------------------------------------------------------- | | BE\_ID | ID of the BE node. | | NAME | Name of the configuration parameter. | | VALUE | Current value of the configuration parameter. | | TYPE | Data type of the configuration parameter. | | DEFAULT | Default value of the configuration parameter. | | MUTABLE | Indicates whether the configuration parameter can be modified dynamically (`true`) or not (`false`). | --- ### be_logs `be_logs` provides information about the logs of each BE node. The following fields are provided in `be_logs`: | **Field** | **Description** | | --------- | --------------------------------------------- | | BE\_ID | ID of the BE node. | | LEVEL | Log level (e.g., `INFO`, `WARNING`, `ERROR`). | | TIMESTAMP | Timestamp of the log entry. | | TID | Thread ID that generated the log entry. | | LOG | Log message content. | --- ### be_metrics `be_metrics` provides information about the metrics of each BE node. The following fields are provided in `be_metrics`: | **Field** | **Description** | | --------- | ---------------------------------- | | BE\_ID | ID of the BE node. | | NAME | Name of the metric. | | LABELS | Labels associated with the metric. | | VALUE | Current value of the metric. | --- ### be_tablets `be_tablets` provides information about tablets on each BE node. The following fields are provided in `be_tablets`: | **Field** | **Description** | | ------------- | -------------------------------------------------------- | | BE\_ID | ID of the BE node. | | TABLE\_ID | ID of the table to which the tablet belongs. | | PARTITION\_ID | ID of the partition to which the tablet belongs. | | TABLET\_ID | ID of the tablet. | | NUM\_VERSION | Number of versions in the tablet. | | MAX\_VERSION | Maximum version of the tablet. | | MIN\_VERSION | Minimum version of the tablet. | | NUM\_ROWSET | Number of rowsets in the tablet. | | NUM\_ROW | Number of rows in the tablet. | | DATA\_SIZE | Data size of the tablet (in bytes). | | INDEX\_MEM | Index memory usage of the tablet (in bytes). | | CREATE\_TIME | Creation time of the tablet (Unix timestamp in seconds). | | STATE | State of the tablet (e.g., `NORMAL`, `REPLICA_MISSING`). | | TYPE | Type of the tablet. | | DATA\_DIR | Data directory where the tablet is stored. | | SHARD\_ID | Shard ID of the tablet. | | SCHEMA\_HASH | Schema hash of the tablet. | | INDEX\_DISK | Index disk usage of the tablet (in bytes). | | MEDIUM\_TYPE | Medium type of the tablet (e.g., `HDD`, `SSD`). | | NUM\_SEGMENT | Number of segments in the tablet. | --- ### be_threads `be_threads` provides information about the threads running on each BE node. The following fields are provided in `be_threads`: | **Field** | **Description** | | --------------- | --------------------------------------------------------------- | | BE\_ID | ID of the BE node. | | GROUP | Thread group name. | | NAME | Thread name. | | PTHREAD\_ID | Pthread ID. | | TID | Thread ID. | | IDLE | Indicates whether the thread is idle (`true`) or not (`false`). | | FINISHED\_TASKS | Number of tasks finished by the thread. | | BOUND\_CPUS | CPUs to which the thread is bound. | --- ### be_txns `be_txns` provides information about transactions on each BE node. The following fields are provided in `be_txns`: | **Field** | **Description** | | ------------- | ------------------------------------------------------------- | | BE\_ID | ID of the BE node. | | LOAD\_ID | ID of the load job. | | TXN\_ID | ID of the transaction. | | PARTITION\_ID | ID of the partition involved in the transaction. | | TABLET\_ID | ID of the tablet involved in the transaction. | | CREATE\_TIME | Creation time of the transaction (Unix timestamp in seconds). | | COMMIT\_TIME | Commit time of the transaction (Unix timestamp in seconds). | | PUBLISH\_TIME | Publish time of the transaction (Unix timestamp in seconds). | | ROWSET\_ID | ID of the rowset involved in the transaction. | | NUM\_SEGMENT | Number of segments in the rowset. | | NUM\_DELFILE | Number of delete files in the rowset. | | NUM\_ROW | Number of rows in the rowset. | | DATA\_SIZE | Data size of the rowset (in bytes). | | VERSION | Version of the rowset. | --- ### character_sets `character_sets` identifies the character sets available. The following fields are provided in `character_sets`: | **Field** | **Description** | | ---------------------- | ------------------------------------------------------------ | | CHARACTER\_SET\_NAME | The character set name. | | DEFAULT\_COLLATE\_NAME | The default collation for the character set. | | DESCRIPTION | A description of the character set. | | MAXLEN | The maximum number of bytes required to store one character. | --- ### collations `collations` contains the available collations. The following fields are provided in `collations`: | **Field** | **Description** | | -------------------- | ------------------------------------------------------------------------------------------------ | | COLLATION\_NAME | The collation name. | | CHARACTER\_SET\_NAME | The name of the character set with which the collation is associated. | | ID | The collation ID. | | IS\_DEFAULT | Whether the collation is the default for its character set. | | IS\_COMPILED | Whether the character set is compiled into the server. | | SORTLEN | This is related to the amount of memory required to sort strings expressed in the character set. | --- ### column_privileges note This view does not apply to the available features in StarRocks. `column_privileges` identifies all privileges granted on columns to a currently enabled role or by a currently enabled role. The following fields are provided in `column_privileges`: | **Field** | **Description** | | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GRANTEE | The name of the user to which the privilege is granted. | | TABLE\_CATALOG | The name of the catalog to which the table containing the column belongs. This value is always `def`. | | TABLE\_SCHEMA | The name of the database to which the table containing the column belongs. | | TABLE\_NAME | The name of the table containing the column. | | COLUMN\_NAME | The name of the column. | | PRIVILEGE\_TYPE | The privilege granted. The value can be any privilege that can be granted at the column level. Each row lists a single privilege, so there is one row per column privilege held by the grantee. | | IS\_GRANTABLE | `YES` if the user has the `GRANT OPTION` privilege, `NO` otherwise. The output does not list `GRANT OPTION` as a separate row with `PRIVILEGE_TYPE='GRANT OPTION'`. | --- ### column_stats_usage `column_stats_usage` provides information about the usage of column statistics. The following fields are provided in `column_stats_usage`: | **Field** | **Description** | | --------------- | ----------------------------------------------- | | TABLE\_CATALOG | The catalog name of the table. | | TABLE\_DATABASE | The database name of the table. | | TABLE\_NAME | The table name. | | COLUMN\_NAME | The column name. | | USAGE | Use case of this column statistics. | | LAST\_USED | Last time when this column statistics was used. | | CREATED | Creation time of this column statistics. | --- ### columns `columns` contains information about all table columns (or view columns). note The metadata of [synchronous materialized views](https://docs.starrocks.io/docs/using_starrocks/Materialized_view-single_table.md) is not recorded in `columns`. You can access it by executing `SHOW PROC '/dbs/db/table/index_schema'`. The following fields are provided in `columns`: | **Field** | **Description** | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | TABLE\_CATALOG | The name of the catalog to which the table containing the column belongs. This value is always `NULL`. | | TABLE\_SCHEMA | The name of the database to which the table containing the column belongs. | | TABLE\_NAME | The name of the table containing the column. | | COLUMN\_NAME | The name of the column. | | ORDINAL\_POSITION | The ordinal position of the column within the table. | | COLUMN\_DEFAULT | The default value for the column. This is `NULL` if the column has an explicit default of `NULL`, or if the column definition includes no `DEFAULT` clause. | | IS\_NULLABLE | The column nullability. The value is `YES` if `NULL` values can be stored in the column, `NO` if not. | | DATA\_TYPE | The column data type. The `DATA_TYPE` value is the type name only with no other information. The `COLUMN_TYPE` value contains the type name and possibly other information such as the precision or length. | | CHARACTER\_MAXIMUM\_LENGTH | For string columns, the maximum length in characters. | | CHARACTER\_OCTET\_LENGTH | For string columns, the maximum length in bytes. | | NUMERIC\_PRECISION | For numeric columns, the numeric precision. | | NUMERIC\_SCALE | For numeric columns, the numeric scale. | | DATETIME\_PRECISION | For temporal columns, the fractional seconds precision. | | CHARACTER\_SET\_NAME | For character string columns, the character set name. | | COLLATION\_NAME | For character string columns, the collation name. | | COLUMN\_TYPE | The column data type.
The `DATA_TYPE` value is the type name only with no other information. The `COLUMN_TYPE` value contains the type name and possibly other information such as the precision or length. | | COLUMN\_KEY | Whether the column is indexed:- If `COLUMN_KEY` is empty, the column either is not indexed or is indexed only as a secondary column in a multiple-column, nonunique index.
- If `COLUMN_KEY` is `PRI`, the column is a `PRIMARY KEY` or is one of the columns in a multiple-column `PRIMARY KEY`.
- If `COLUMN_KEY` is `UNI`, the column is the first column of a `UNIQUE` index. (A `UNIQUE` index permits multiple `NULL` values, but you can tell whether the column permits `NULL` by checking the `Null` column.)
- If `COLUMN_KEY` is `DUP`, the column is the first column of a nonunique index in which multiple occurrences of a given value are permitted within the column.If more than one of the `COLUMN_KEY` values applies to a given column of a table, `COLUMN_KEY` displays the one with the highest priority, in the order `PRI`, `UNI`, `DUP`.
A `UNIQUE` index may be displayed as `PRI` if it cannot contain `NULL` values and there is no `PRIMARY KEY` in the table. A `UNIQUE` index may display as `MUL` if several columns form a composite `UNIQUE` index; although the combination of the columns is unique, each column can still hold multiple occurrences of a given value. | | EXTRA | Any additional information that is available about a given column. | | PRIVILEGES | The privileges you have for the column. | | COLUMN\_COMMENT | The comments of the column. | | COLUMN\_SIZE | The maximum length of the column in bytes. | | DECIMAL\_DIGITS | The number of fractional digits for numeric types. | | GENERATION\_EXPRESSION | The expression used to generate the column's value. | | SRS\_ID | The spatial reference system identifier for spatial columns. | | COLUMN\_SIZE | | | DECIMAL\_DIGITS | | | GENERATION\_EXPRESSION | For generated columns, displays the expression used to compute column values. Empty for nongenerated columns. | | SRS\_ID | This value applies to spatial columns. It contains the column `SRID` value that indicates the spatial reference system for values stored in the column. | --- ### engines note This view does not apply to the available features in StarRocks. `engines` provides information about storage engines. The following fields are provided in `engines`: | **Field** | **Description** | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ENGINE | The name of the storage engine. | | SUPPORT | The server's level of support for the storage engine. Valid values:- `YES`: The engine is supported and active.
- `DEFAULT`: Like `YES`, plus this is the default engine.
- `NO`: The engine is not supported.
- `DISABLED`: The engine is supported but has been disabled. | | COMMENT | A brief description of the storage engine. | | TRANSACTIONS | Whether the storage engine supports transactions. | | XA | Whether the storage engine supports XA transactions. | | SAVEPOINTS | Whether the storage engine supports savepoints. | --- ### events note This view does not apply to the available features in StarRocks. `events` provides information about Event Manager events. The following fields are provided in `events`: | **Field** | **Description** | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | EVENT\_CATALOG | The name of the catalog to which the event belongs. This value is always `def`. | | EVENT\_SCHEMA | The name of the database to which the event belongs. | | EVENT\_NAME | The name of the event. | | DEFINER | The user named in the `DEFINER` clause (often the user who created the event). | | TIME\_ZONE | The event time zone, which is the time zone used for scheduling the event and that is in effect within the event as it executes. The default value is `SYSTEM`. | | EVENT\_BODY | The language used for the statements in the event's `DO` clause. The value is always `SQL`. | | EVENT\_DEFINITION | The text of the SQL statement making up the event's `DO` clause; in other words, the statement executed by this event. | | EVENT\_TYPE | The event repetition type, either `ONE TIME` (transient) or `RECURRING` (repeating). | | EXECUTE\_AT | For a one-time event, this is the `DATETIME` value specified in the `AT` clause of the `CREATE EVENT` statement used to create the event, or of the last `ALTER EVENT` statement that modified the event. The value shown in this column reflects the addition or subtraction of any `INTERVAL` value included in the event's `AT` clause. For example, if an event is created using `ON SCHEDULE AT CURRENT_DATETIME + '1:6' DAY_HOUR`, and the event was created at 2018-02-09 14:05:30, the value shown in this column would be `'2018-02-10 20:05:30'`. If the event's timing is determined by an `EVERY` clause instead of an `AT` clause (that is, if the event is recurring), the value of this column is `NULL`. | | INTERVAL\_VALUE | For a recurring event, the number of intervals to wait between event executions. For a transient event, the value is always `NULL`. | | INTERVAL\_FIELD | The time units used for the interval which a recurring event waits before repeating. For a transient event, the value is always `NULL`. | | SQL\_MODE | The SQL mode in effect when the event was created or altered, and under which the event executes. | | STARTS | The start date and time for a recurring event. This is displayed as a `DATETIME` value, and is `NULL` if no start date and time are defined for the event. For a transient event, this column is always `NULL`. For a recurring event whose definition includes a `STARTS` clause, this column contains the corresponding `DATETIME` value. As with the `EXECUTE_AT` column, this value resolves any expressions used. If there is no `STARTS` clause affecting the timing of the event, this column is `NULL`. | | ENDS | For a recurring event whose definition includes a `ENDS` clause, this column contains the corresponding `DATETIME` value. As with the `EXECUTE_AT` column, this value resolves any expressions used. If there is no `ENDS` clause affecting the timing of the event, this column is `NULL`. | | STATUS | The event status. One of `ENABLED`, `DISABLED`, or `SLAVESIDE_DISABLED`. `SLAVESIDE_DISABLED` indicates that the creation of the event occurred on another MySQL server acting as a replication source and replicated to the current MySQL server which is acting as a replica, but the event is not presently being executed on the replica. | | ON\_COMPLETION | Valid values: `PRESERVE` and `NOT PRESERVE`. | | CREATED | The date and time when the event was created. This is a `DATETIME` value. | | LAST\_ALTERED | The date and time when the event was last modified. This is a `DATETIME` value. If the event has not been modified since its creation, this value is the same as the `CREATED` value. | | LAST\_EXECUTED | The date and time when the event last executed. This is a `DATETIME` value. If the event has never executed, this column is `NULL`.`LAST_EXECUTED` indicates when the event started. As a result, the `ENDS` column is never less than `LAST_EXECUTED`. | | EVENT\_COMMENT | The text of the comment, if the event has one. If not, this value is empty. | | ORIGINATOR | The server ID of the MySQL server on which the event was created; used in replication. This value may be updated by `ALTER EVENT` to the server ID of the server on which that statement occurs, if executed on a replication source. The default value is 0. | | CHARACTER\_SET\_CLIENT | The session value of the `character_set_client` system variable when the event was created. | | COLLATION\_CONNECTION | The session value of the `collation_connection` system variable when the event was created. | | DATABASE\_COLLATION | The collation of the database with which the event is associated. | --- ### fe_metrics `fe_metrics` provides information about the metrics of each FE node. The following fields are provided in `fe_metrics`: | **Field** | **Description** | | --------- | ---------------------------------- | | FE\_ID | ID of the FE node. | | NAME | Name of the metric. | | LABELS | Labels associated with the metric. | | VALUE | Current value of the metric. | --- ### fe_tablet_schedules `fe_tablet_schedules` provides information about tablet scheduling tasks on FE nodes. The following fields are provided in `fe_tablet_schedules`: | **Field** | **Description** | | ---------------------------- | ----------------------------------------------------------- | | TABLET\_ID | The ID of the Tablet. | | TABLE\_ID | The ID of the table to which the Tablet belongs. | | PARTITION\_ID | The ID of the partition to which the Tablet belongs. | | TYPE | The type of the task. Valid values: `REPAIR` and `BALANCE`. | | STATE | The status of the task. | | SCHEDULE\_REASON | The reason for the task scheduling. | | MEDIUM | The storage medium where the Tablet is located. | | PRIORITY | The current priority of the task. | | ORIG\_PRIORITY | The original priority of the task. | | LAST\_PRIORITY\_ADJUST\_TIME | The time when the task's priority was last adjusted. | | VISIBLE\_VERSION | The visible data version of the Tablet. | | COMMITTED\_VERSION | The committed data version of the Tablet. | | SRC\_BE\_ID | The ID of the BE where the source replica resides. | | SRC\_PATH | The path where the source replica resides. | | DEST\_BE\_ID | ID of the BE where the target replica is located. | | DEST\_PATH | Path where the target replica is located. | | CREATE\_TIME | The time when the task was created. | | SCHEDULE\_TIME | The time when the task starts scheduling execution. | | FINISH\_TIME | The time when the task was finished. | | CLONE\_BYTES | Size of the file cloned in bytes. | | CLONE\_DURATION | Time elapsed for Clone in seconds. | | CLONE\_RATE | Clone rate in MB/s. | | FAILED\_SCHEDULE\_COUNT | The number of task scheduling failures. | | FAILED\_RUNNING\_COUNT | Number of task execution failures. | | MSG | Information on task scheduling and exection. | --- ### fe_threads `fe_threads` provides information about the threads running on each FE node. The following fields are provided in `fe_threads`: | **Field** | **Description** | | -------------- | ------------------------------------------------------------------------------------------------------------- | | FE\_ADDRESS | Address of the FE node (format: `host:port`). | | THREAD\_ID | Java thread ID. | | THREAD\_NAME | Name of the thread. | | THREAD\_STATE | State of the thread. Possible values: `NEW`, `RUNNABLE`, `BLOCKED`, `WAITING`, `TIMED_WAITING`, `TERMINATED`. | | IS\_DAEMON | Indicates whether the thread is a daemon thread (`true`) or not (`false`). | | PRIORITY | Thread priority (typically 1-10). | | CPU\_TIME\_MS | CPU time consumed by the thread in milliseconds. Returns `-1` if CPU time measurement is not supported. | | USER\_TIME\_MS | User time consumed by the thread in milliseconds. Returns `-1` if user time measurement is not supported. | #### Usage Examples[​](#usage-examples "Direct link to Usage Examples") ##### Query all threads[​](#query-all-threads "Direct link to Query all threads") ```sql SELECT * FROM information_schema.fe_threads; ``` ##### Query threads by state[​](#query-threads-by-state "Direct link to Query threads by state") ```sql SELECT thread_id, thread_name, thread_state FROM information_schema.fe_threads WHERE thread_state = 'RUNNABLE'; ``` ##### Count threads by state[​](#count-threads-by-state "Direct link to Count threads by state") ```sql SELECT thread_state, COUNT(*) AS count FROM information_schema.fe_threads GROUP BY thread_state ORDER BY count DESC; ``` ##### Find daemon threads[​](#find-daemon-threads "Direct link to Find daemon threads") ```sql SELECT thread_id, thread_name, thread_state FROM information_schema.fe_threads WHERE is_daemon = true; ``` ##### Find threads with highest CPU usage[​](#find-threads-with-highest-cpu-usage "Direct link to Find threads with highest CPU usage") ```sql SELECT thread_id, thread_name, cpu_time_ms FROM information_schema.fe_threads WHERE cpu_time_ms >= 0 ORDER BY cpu_time_ms DESC LIMIT 10; ``` ##### Search for specific thread names[​](#search-for-specific-thread-names "Direct link to Search for specific thread names") ```sql SELECT thread_id, thread_name, thread_state FROM information_schema.fe_threads WHERE thread_name LIKE '%main%' OR thread_name LIKE '%GC%'; ``` ##### Count threads by daemon status[​](#count-threads-by-daemon-status "Direct link to Count threads by daemon status") ```sql SELECT is_daemon, COUNT(*) AS count FROM information_schema.fe_threads GROUP BY is_daemon; ``` ##### Get thread statistics[​](#get-thread-statistics "Direct link to Get thread statistics") ```sql SELECT COUNT(*) AS total_threads, COUNT(DISTINCT thread_state) AS distinct_states, COUNT(DISTINCT thread_name) AS distinct_names, SUM(CASE WHEN is_daemon THEN 1 ELSE 0 END) AS daemon_count, SUM(CASE WHEN NOT is_daemon THEN 1 ELSE 0 END) AS non_daemon_count FROM information_schema.fe_threads; ``` --- ### global_variables `global_variables` provides information about global variables. The following fields are provided in `global_variables`: | **Field** | **Description** | | --------------- | ---------------------- | | VARIABLE\_NAME | Name of the variable. | | VARIABLE\_VALUE | Value of the variable. | --- ### key_column_usage note This view does not apply to the available features in StarRocks. `key_column_usage` identifies all columns that are restricted by some unique, primary key, or foreign key constraint. The following fields are provided in `key_column_usage`: | **Field** | **Description** | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | CONSTRAINT\_CATALOG | The name of the catalog to which the constraint belongs. This value is always `def`. | | CONSTRAINT\_SCHEMA | The name of the database to which the constraint belongs. | | CONSTRAINT\_NAME | The name of the constraint. | | TABLE\_CATALOG | The name of the catalog to which the table belongs. This value is always `def`. | | TABLE\_SCHEMA | The name of the database to which the table belongs. | | TABLE\_NAME | The name of the table that has the constraint. | | COLUMN\_NAME | The name of the column that has the constraint.If the constraint is a foreign key, then this is the column of the foreign key, not the column that the foreign key references. | | ORDINAL\_POSITION | The column's position within the constraint, not the column's position within the table. Column positions are numbered beginning with 1. | | POSITION\_IN\_UNIQUE\_CONSTRAINT | `NULL` for unique and primary-key constraints. For foreign-key constraints, this column is the ordinal position in key of the table that is being referenced. | | REFERENCED\_TABLE\_SCHEMA | The name of the schema referenced by the constraint. | | REFERENCED\_TABLE\_NAME | The name of the table referenced by the constraint. | | REFERENCED\_COLUMN\_NAME | The name of the column referenced by the constraint. | --- ### load_tracking_logs `load_tracking_logs` provides error logs of load jobs. This view is supported from StarRocks v3.0 onwards. The following fields are provided in `load_tracking_logs`: | **Field** | **Description** | | -------------- | ------------------------------------------ | | ID | The ID of the load. | | JOB\_ID | The ID of the load job(deprecated). | | LABEL | The label of the load job. | | DATABASE\_NAME | The database that the load job belongs to. | | TRACKING\_LOG | Errors (if any) of the load job. | tip To query the view `load_tracking_logs` you will need to filter on either a `JOB_ID` or a `LABEL`. You can retrieve the `JOB_ID` or `LABEL` of a load job from the view `information_schema.loads`. Example: ```sql SELECT * from information_schema.load_tracking_logs WHERE label ='user_behavior'\G *************************** 1. row *************************** JOB_ID: 10141 LABEL: user_behavior DATABASE_NAME: mydatabase TRACKING_LOG: NULL TYPE: BROKER 1 row in set (0.02 sec) ``` --- ### loads `loads` provides the results of load jobs. This view is supported from StarRocks v3.1 onwards. The following fields are provided in `loads`: | Field | Description | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ID | Globally unique identifier. | | LABEL | Label of the load job. | | PROFILE\_ID | The ID of the Profile, which can be analyzed via `ANALYZE PROFILE`. | | DB\_NAME | The database to which the target table belongs. | | TABLE\_NAME | The target table. | | USER | The user who initiates the load job. | | WAREHOUSE | The warehouse to which the load job belongs. | | STATE | The state of the load job. Valid values:- `PENDING`/`BEGIN`: The load job is created.
- `QUEUEING`/`BEFORE_LOAD`: The load job is in the queue waiting to be scheduled.
- `LOADING`: The load job is running.
- `PREPARING`: The transaction is being pre-committed.
- `PREPARED`: The transaction has been pre-committed.
- `COMMITED`: The transaction has been committed.
- `FINISHED`: The load job succeeded.
- `CANCELLED`: The load job failed. | | PROGRESS | The progress of the ETL stage and LOADING stage of the load job. | | TYPE | The type of the load job. For Broker Load, the return value is `BROKER`. For INSERT, the return value is `INSERT`. For Stream Load, the return value is `STREAM`. For Routine Load Load, the return value is `ROUTINE`. | | PRIORITY | The priority of the load job. Valid values: `HIGHEST`, `HIGH`, `NORMAL`, `LOW`, and `LOWEST`. | | SCAN\_ROWS | The number of data rows that are scanned. | | SCAN\_BYTES | The number of bytes that are scanned. | | FILTERED\_ROWS | The number of data rows that are filtered out due to inadequate data quality. | | UNSELECTED\_ROWS | The number of data rows that are filtered out due to the conditions specified in the WHERE clause. | | SINK\_ROWS | The number of data rows that are loaded. | | RUNTIME\_DETAILS | Load runtime metadata. For details, see [RUNTIME\_DETAILS](#runtime_details). | | CREATE\_TIME | The time at which the load job was created. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | LOAD\_START\_TIME | The start time of the LOADING stage of the load job. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | LOAD\_COMMIT\_TIME | The time at which the loading transaction was committed. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | LOAD\_FINISH\_TIME | The end time of the LOADING stage of the load job. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | PROPERTIES | The static properties of the load job. For details, see [PROPERTIES](#properties). | | ERROR\_MSG | The error message of the load job. If the load job did not encounter any error, `NULL` is returned. | | TRACKING\_SQL | The SQL statement that can be used to query the tracking log of the load job. A SQL statement is returned only when the load job involves unqualified data rows. If the load job does not involve any unqualified data rows, `NULL` is returned. | | REJECTED\_RECORD\_PATH | The path from which you can access all the unqualified data rows that are filtered out in the load job. The number of unqualified data rows logged is determined by the `log_rejected_record_num` parameter configured in the load job. You can use the `wget` command to access the path. If the load job does not involve any unqualified data rows, `NULL` is returned. | #### RUNTIME\_DETAILS[​](#runtime_details "Direct link to RUNTIME_DETAILS") * Universal metrics: | Metric | Description | | -------- | ---------------------------------------------- | | load\_id | Globally unique ID of the load execution plan. | | txn\_id | Load transaction ID. | * Specific metrics for Broker Load, INSERT INTO, and Spark Load: | Metric | Description | | -------------------- | ----------------------------------------------------------------------------------------------------------------- | | etl\_info | ETL Details. This field is only valid for Spark Load jobs. For other types of load jobs, the value will be empty. | | etl\_start\_time | The start time of the ETL stage of the load job. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | etl\_start\_time | The end time of the ETL stage of the load job. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | unfinished\_backends | List of BEs with unfinished executions. | | backends | List of BEs participating in execution. | | file\_num | Number of files read. | | file\_size | Total size of files read. | | task\_num | Number of subtasks. | * Specific metrics for Routine Load: | Metric | Description | | --------------------- | ----------------------------------------------------------------------------------- | | schedule\_interval | The interval for Routine Load to be scheduled. | | wait\_slot\_time | Time elapsed while the Routine Load task waits for execution slots. | | check\_offset\_time | Time consumed when checking offset information during Routine Load task scheduling. | | consume\_time | Time consumed by the Routine Load task to read upstream data. | | plan\_time | Time for generating the execution plan. | | commit\_publish\_time | Time consumed to execute the COMMIT RPC. | * Specific metrics for Stream Load: | Metric | Description | | ------------------------- | ---------------------------------------- | | timeout | Timeout for load tasks. | | begin\_txn\_ms | Time consumed to begin the transaction. | | plan\_time\_ms | Time for generating the execution plan. | | receive\_data\_time\_ms | Time for receiving data. | | commit\_publish\_time\_ms | Time consumed to execute the COMMIT RPC. | | client\_ip | Client IP address. | #### PROPERTIES[​](#properties "Direct link to PROPERTIES") * Specific properties for Broker Load, INSERT INTO, and Spark Load: | Property | Description | | ------------------ | -------------------------------------------------------------------------------- | | timeout | Timeout for load tasks. | | max\_filter\_ratio | Maximum ratio of data rows that are filtered out due to inadequate data quality. | * Specific properties for Routine Load: | Property | Description | | --------- | ------------------------------------------------- | | job\_name | Routine Load job name. | | task\_num | Number of subtasks actually executed in parallel. | | timeout | Timeout for load tasks. | --- ### materialized_view_refresh_jobs `materialized_view_refresh_jobs` provides job-level information about materialized view refreshes. A single refresh job may comprise multiple task runs (for example, partition-by-partition refresh batches); this view rolls those task runs up into one row per job. It shares its data source with [`task_runs`](https://docs.starrocks.io/docs/sql-reference/information_schema/task_runs.md), so you can use `JOB_ID` to drill down into the individual task runs of a job (`SELECT * FROM information_schema.task_runs WHERE JOB_ID = ''`), and job records are retained for the same window as `task_runs` history. The following fields are provided in `materialized_view_refresh_jobs`: | **Field** | **Description** | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | JOB\_ID | ID of the refresh job. All task runs of one refresh share this ID; use it to drill down into `task_runs.JOB_ID`. | | MATERIALIZED\_VIEW\_ID | ID of the materialized view. | | TABLE\_SCHEMA | Database the materialized view belongs to. | | TABLE\_NAME | Name of the materialized view. `NULL` if the materialized view has been dropped. | | TASK\_ID | ID of the refresh task. | | WAREHOUSE | Warehouse used by the refresh job. | | RESOURCE\_GROUP | Resource group used by the refresh job. This is the materialized view's configured `resource_group` property; returns `default_mv_wg` when not configured. | | CREATOR | User that created the materialized view (its create-user; the run identity is in RUN\_AS\_USER). | | SUBMIT\_USER | User who submitted the refresh job. For a manual refresh this is the user who issued it; for scheduled or base-table-change refreshes it is submitted by the system. | | RUN\_AS\_USER | User identity the refresh runs as. With creator-based authorization (the default, `mv_use_creator_based_authorization=true`) this is the materialized view's creator; with root-based authorization it is `'root'@'%'`. | | SUBMIT\_TIME | Time when the job was submitted (the first task run's creation time). | | REFRESH\_STATE | State of the job, rolled up from the last task run. Valid values: `PENDING`, `RUNNING`, `FAILED`, `SUCCESS`, and `SKIPPED`. | | FINISH\_TIME | Time when the job finished. `NULL` if the job has not finished. | | DURATION\_TIME | Wall-clock duration of the job, in seconds (the last task run's finish time minus the first task run's process-start time). `NULL` if the job has not finished. | | REFRESH\_TRIGGER | How this job was triggered. `MANUAL` for a manually-issued `REFRESH MATERIALIZED VIEW` (even when the materialized view's scheme is scheduled or automatic); otherwise the materialized view's configured scheme. Valid values: `MANUAL`, `SCHEDULED`, `ON_BASE_TABLE_CHANGE`, and `NONE`. `UNKNOWN` if the materialized view has been dropped and the job was not manual. | | REFRESH\_MODE | The materialized view's configured refresh mode. Valid values: `PCT` and `INCREMENTAL`. `NULL` if the materialized view has been dropped. Note that this is the *configured* mode, not the mode a given refresh actually ran in: they differ for the bootstrap refresh and whenever INCREMENTAL planning fails and falls back to PCT. For the mode a refresh actually ran in, query `get_json_string(EXTRA_MESSAGE, '$.refreshMode')` on `information_schema.task_runs`. | | IMV\_SOURCE\_VERSION\_RANGE | JSON of the source version ranges consumed by an incremental refresh. Returns `NULL` for a non-incremental (PCT) refresh or when no source ranges were consumed. | | IMV\_SOURCE\_TIMESTAMP\_RANGE | JSON of the source timestamp ranges consumed by an incremental refresh. Returns `NULL` for a non-incremental (PCT) refresh or when no source ranges were consumed. | | IMV\_SOURCE\_PINNED\_SNAPSHOT\_ID\_MAP | JSON of pinned source snapshot IDs. Its JSON key is the connector table identifier (for Iceberg, `
:`), which differs from the `..
` key used by IMV\_SOURCE\_VERSION\_RANGE and IMV\_SOURCE\_TIMESTAMP\_RANGE. Populated on the baseline/PCT-path refresh; returns `NULL` on a pure incremental run or when no snapshot was pinned. | | FAILED\_TASK\_RUN\_ID | Task-run ID of the failed run within the job. `NULL` if no run failed. To drill down into `task_runs`, join on `FAILED_QUERY_ID = task_runs.QUERY_ID` (or on `JOB_ID`); `task_runs` does not expose a task-run-id column. | | FAILED\_QUERY\_ID | Query ID of the failed run. `NULL` if no run failed. | | ERROR\_CODE | Error code of the failed run. `NULL` if no run failed. | | ERROR\_MESSAGE | Error message of the failed run. `NULL` if no run failed. | note This view has no persistent storage. Its rows are derived from `task_runs` at query time, so record retention follows the `task_runs` history settings. Because each job is aggregated from its `task_runs` rows at query time, a job is only fully represented while all of its task runs are still within the `task_runs` history window; older jobs are not shown, and a job straddling the retention boundary may be partially aggregated (for example, its `SUBMIT_TIME` or `IMV_SOURCE_*` range may reflect only the retained runs). --- ### materialized_views `materialized_views` provides information about all materialized views. The following fields are provided in `materialized_views`: | **Field** | **Description** | | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | MATERIALIZED\_VIEW\_ID | ID of the materialized view. | | TABLE\_SCHEMA | Database in which the materialized view resides. | | TABLE\_NAME | Name of the materialized view. | | REFRESH\_TYPE | Refresh type of the materialized view. Valid values: `SYNC` (synchronous materialized view) and `ASYNC` (asynchronous materialized view, regardless of how the refresh is triggered). When the value is `SYNC`, all fields related to activation status and refresh are empty. See `REFRESH_TRIGGER` and `REFRESH_POLICY` for how an asynchronous materialized view is refreshed. | | IS\_ACTIVE | Indicates whether the materialized view is active. Inactive materialized views cannot be refreshed or queried. | | INACTIVE\_REASON | The reason that the materialized view is inactive. | | PARTITION\_TYPE | Type of partitioning strategy for the materialized view. | | TASK\_ID | ID of the task responsible for refreshing the materialized view. | | TASK\_NAME | Name of the task responsible for refreshing the materialized view. | | LAST\_REFRESH\_START\_TIME | Start time of the most recent refresh task. | | LAST\_REFRESH\_FINISHED\_TIME | End time of the most recent refresh task. | | LAST\_REFRESH\_DURATION | Wall-clock duration of the most recent refresh, in seconds (the last task run's finish time minus the first task run's process-start time). Matches `materialized_view_refresh_jobs.DURATION_TIME` for that job. | | LAST\_REFRESH\_STATE | State of the most recent refresh task. | | LAST\_REFRESH\_FORCE\_REFRESH | Indicates whether the most recent refresh task was a force refresh. | | LAST\_REFRESH\_START\_PARTITION | Starting partition for the most recent refresh task. | | LAST\_REFRESH\_END\_PARTITION | Ending partition for the most recent refresh task. | | LAST\_REFRESH\_BASE\_REFRESH\_PARTITIONS | Base table partitions involved in the most recent refresh task. | | LAST\_REFRESH\_MV\_REFRESH\_PARTITIONS | Materialized view partitions refreshed in the most recent refresh task. | | LAST\_REFRESH\_ERROR\_CODE | Error code of the most recent refresh task. | | LAST\_REFRESH\_ERROR\_MESSAGE | Error message of the most recent refresh task. | | TABLE\_ROWS | Number of data rows in the materialized view, based on approximate background statistics. | | MATERIALIZED\_VIEW\_DEFINITION | SQL definition of the materialized view. | | EXTRA\_MESSAGE | Extra message of the materialized view. | | QUERY\_REWRITE\_STATUS | Query rewrite status of the materialized view. | | CREATOR | Creator of the materialized view. | | LAST\_REFRESH\_PROCESS\_TIME | Process time of the most recent refresh task. | | LAST\_REFRESH\_JOB\_ID | Job ID of the most recent refresh task. | | LAST\_REFRESH\_TIME | Time up to which base table updates are reflected in the materialized view. | | WAREHOUSE | Name of the warehouse that the asynchronous materialized view uses for its refresh tasks. Empty in shared-nothing mode, or for synchronous (rollup) materialized views. | | REFRESH\_MODE | Configured refresh mode of the asynchronous materialized view. Valid values: `PCT` (partition change tracking, where only changed partitions are refreshed) and `INCREMENTAL` (incremental view maintenance). Empty for synchronous materialized views. | | REFRESH\_TRIGGER | How a refresh is triggered. Valid values: `NONE` (synchronous materialized view), `MANUAL` (only via REFRESH MATERIALIZED VIEW), `SCHEDULED` (periodic, via an EVERY interval), and `ON_BASE_TABLE_CHANGE` (automatically when a base table loads or changes). | | REFRESH\_POLICY | Human-readable refresh policy. Valid values: `NONE`, `MANUAL`, `ON_BASE_TABLE_CHANGE`, or a schedule such as `START("yyyy-MM-dd HH:mm:ss") EVERY(INTERVAL n unit)` (the `START` clause is present only if a start time was defined). | | RESOURCE\_GROUP | Resource group used for the materialized view's refresh tasks (from the materialized view's `resource_group` property). Defaults to `default_mv_wg` when not set. | | QUERY\_REWRITE\_STATUS\_REASON | The reason behind `QUERY_REWRITE_STATUS`. Valid values: `OK`, `MV_INACTIVE`, `QUERY_REWRITE_DISABLED`, `UNSUPPORTED_DEFINITION`, and `UNKNOWN`. | | BASE\_TABLE\_REFRESH\_VERSION\_TIMES | Per-base-table data version time, as a JSON object mapping each base table's `catalog.database.table` name to the latest data version time observed for it. This is the per-table detail behind `LAST_REFRESH_TIME` (their single maximum): external/data lake base tables report the partition source modified time, and OLAP (internal) base tables report the visible-version commit time. `{}` when no base table has a recorded time. This column only advances on a successful refresh (a failed or skipped refresh leaves it unchanged). Its resolution is one second, so a write and a refresh landing in the same second are indistinguishable. | | LAST\_FRESHNESS\_CONFIRMED\_AT | Start time of the most recent successful refresh, recorded once the entire refresh (all of its task runs) completes; a refresh that found no base-table changes to apply also confirms freshness. The materialized view reflects base-table data as of this moment. Unlike `LAST_REFRESH_TIME` (the base-table data version), this is wall-clock time. `NULL` until the first successful refresh, and for synchronous materialized views. A partition-scoped (partial) REFRESH does not advance it. | --- ### partitions note This view does not apply to the available features in StarRocks. `partitions` provides information about table partitions. The following fields are provided in `partitions`: | **Field** | **Description** | | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | TABLE\_CATALOG | The name of the catalog to which the table belongs. This value is always `def`. | | TABLE\_SCHEMA | The name of the database to which the table belongs. | | TABLE\_NAME | The name of the table containing the partition. | | PARTITION\_NAME | The name of the partition. | | SUBPARTITION\_NAME | If the `PARTITIONS` table row represents a subpartition, the name of subpartition; otherwise `NULL`.For `NDB`: This value is always `NULL`. | | PARTITION\_ORDINAL\_POSITION | All partitions are indexed in the same order as they are defined, with `1` being the number assigned to the first partition. The indexing can change as partitions are added, dropped, and reorganized; the number shown is this column reflects the current order, taking into account any indexing changes. | | PARTITION\_METHOD | Valid values: `RANGE`, `LIST`, `HASH`, `LINEAR HASH`, `KEY`, or `LINEAR KEY`. | | SUBPARTITION\_METHOD | Valid values: `HASH`, `LINEAR HASH`, `KEY`, or `LINEAR KEY` | | PARTITION\_EXPRESSION | The expression for the partitioning function used in the `CREATE TABLE` or `ALTER TABLE` statement that created the table's current partitioning scheme. | | SUBPARTITION\_EXPRESSION | This works in the same fashion for the subpartitioning expression that defines the subpartitioning for a table as `PARTITION_EXPRESSION` does for the partitioning expression used to define a table's partitioning. If the table has no subpartitions, this column is `NULL`. | | PARTITION\_DESCRIPTION | This column is used for RANGE and LIST partitions. For a `RANGE` partition, it contains the value set in the partition's `VALUES LESS THAN` clause, which can be either an integer or `MAXVALUE`. For a `LIST` partition, this column contains the values defined in the partition's `VALUES IN` clause, which is a list of comma-separated integer values.For partitions whose `PARTITION_METHOD` is other than `RANGE` or `LIST`, this column is always `NULL`. | | TABLE\_ROWS | The number of table rows in the partition. | | AVG\_ROW\_LENGTH | The average length of the rows stored in this partition or subpartition, in bytes. This is the same as `DATA_LENGTH` divided by `TABLE_ROWS`. | | DATA\_LENGTH | The total length of all rows stored in this partition or subpartition, in bytes; that is, the total number of bytes stored in the partition or subpartition. | | MAX\_DATA\_LENGTH | The maximum number of bytes that can be stored in this partition or subpartition. | | INDEX\_LENGTH | The length of the index file for this partition or subpartition, in bytes. | | DATA\_FREE | The number of bytes allocated to the partition or subpartition but not used. | | CREATE\_TIME | The time that the partition or subpartition was created. | | UPDATE\_TIME | The time that the partition or subpartition was last modified. | | CHECK\_TIME | The last time that the table to which this partition or subpartition belongs was checked. | | CHECKSUM | The checksum value, if any; otherwise `NULL`. | | PARTITION\_COMMENT | The text of the comment, if the partition has one. If not, this value is empty. | | NODEGROUP | This is the nodegroup to which the partition belongs. | | TABLESPACE\_NAME | The name of the tablespace to which the partition belongs. | --- ### partitions_meta `partitions_meta` provides information about partitions of tables. The following fields are provided in `partitions_meta`: | **Field** | **Description** | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | DB\_NAME | Name of the database to which the partition belongs. | | TABLE\_NAME | Name of the table to which the partition belongs. | | PARTITION\_NAME | Name of the partition. | | PARTITION\_ID | ID of the partition. | | COMPACT\_VERSION | Compact version of the partition. | | VISIBLE\_VERSION | Visible version of the partition. | | VISIBLE\_VERSION\_TIME | Visible version time of the partition. | | NEXT\_VERSION | Next version of the partition. | | DATA\_VERSION | Data version of the partition. | | VERSION\_EPOCH | Version epoch of the partition. | | VERSION\_TXN\_TYPE | Version transaction type of the partition. | | PARTITION\_KEY | Partition key of the partition. | | PARTITION\_VALUE | Partition value of the partition (e.g., `Range` or `List`). | | DISTRIBUTION\_KEY | Distribution key of the partition. | | BUCKETS | Number of buckets in the partition. For a table with range distribution, this is the actual number of tablets in the partition's base index, which changes as tablets are split and merged. | | REPLICATION\_NUM | Replication number of the partition. | | STORAGE\_MEDIUM | Storage medium of the partition. | | COOLDOWN\_TIME | Cooldown time of the partition. | | LAST\_CONSISTENCY\_CHECK\_TIME | Last consistency check time of the partition. | | IS\_IN\_MEMORY | Indicates whether the partition is in memory (`true`) or not (`false`). | | IS\_TEMP | Indicates whether the partition is temporary (`true`) or not (`false`). | | DATA\_SIZE | Data size of the partition. | | ROW\_COUNT | Number of rows in the partition. | | ENABLE\_DATACACHE | Indicates whether data cache is enabled for the partition (`true`) or not (`false`). | | AVG\_CS | Average compaction score of the partition. | | P50\_CS | 50th percentile compaction score of the partition. | | MAX\_CS | Maximum compaction score of the partition. | | STORAGE\_PATH | Storage path of the partition. | | STORAGE\_SIZE | Storage size of the partition. | | METADATA\_SWITCH\_VERSION | Metadata switch version of the partition. | | TABLET\_BALANCED | Whether the tablet distribution is balanced in the partition. | --- ### pipe_files `pipe_files` provides the status of the data files to be loaded via a specified pipe. This view is supported from StarRocks v3.2 onwards. The following fields are provided in `pipe_files`: | **Field** | **Description** | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------- | | DATABASE\_NAME | The name of the database in which the pipe is stored. | | PIPE\_ID | The unique ID of the pipe. | | PIPE\_NAME | The name of the pipe. | | FILE\_NAME | The name of the data file. | | FILE\_VERSION | The digest of the data file. | | FILE\_SIZE | The size of the data file. Unit: bytes. | | LAST\_MODIFIED | The last time when the data file was modified. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | LOAD\_STATE | The load status of the data file. Valid values: `UNLOADED`, `LOADING`, `FINISHED`, and `ERROR`. | | STAGED\_TIME | The date and time when the data file was first recorded by the pipe. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | START\_LOAD\_TIME | The date and time when the loading of the data file began. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | FINISH\_LOAD\_TIME | The date and time when the loading of the data file finished. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | ERROR\_MSG | The details about the load error for the data file. | --- ### pipes `pipes` provides information about all pipes stored in the current or specified database. This view is supported from StarRocks v3.2 onwards. note You can also use [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md) to view the pipes stored in a specified database or in the current database in use. This command is also supported from v3.2 onwards. The following fields are provided in `pipes`: | **Field** | **Description** | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | DATABASE\_NAME | The name of the database in which the pipe is stored. | | PIPE\_ID | The unique ID of the pipe. | | PIPE\_NAME | The name of the pipe. | | PROPERTIES | The properties of the pipe. | | TABLE\_NAME | The name of the destination table. Format: `.`. | | STATE | The status of the pipe. Valid values: `RUNNING`, `FINISHED`, `SUSPENDED`, and `ERROR`. | | LOAD\_STATUS | The overall status of the data files to be loaded via the pipe, including the following sub-fields:- `loadedFiles`: the number of data files that have been loaded.
- `loadedBytes`: the volume of data that has been loaded, measured in bytes.
- `loadingFiles`: the number of data files that are being loaded. | | LAST\_ERROR | The details about the last error that occurred during the pipe execution. Default value: `NULL`. | | CREATED\_TIME | The date and time when the pipe was created. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | --- ### recyclebin_catalogs `recyclebin_catalogs` provides metadata information of deleted databases, tables, and partitions temporarily stored in the FE recycle bin. `recyclebin_catalogs` provides the following fields: | **Field** | **Description** | | ------------- | -------------------------------------------------------------------------------- | | TYPE | The type of metadata that was deleted, including Database, Table, and Partition. | | NAME | The name of the deleted object. | | DB\_ID | The DB ID of the deleted object. | | TABLE\_ID | The Table ID of the deleted object. | | PARTITION\_ID | The Partition ID of the deleted object. | | DROP\_TIME | The time when the object is deleted. | --- ### referential_constraints note This view does not apply to the available features in StarRocks. `referential_constraints` contains all referential (foreign key) constraints. The following fields are provided in `referential_constraints`: | **Field** | **Description** | | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | CONSTRAINT\_CATALOG | The name of the catalog to which the constraint belongs. This value is always `def`. | | CONSTRAINT\_SCHEMA | The name of the database to which the constraint belongs. | | CONSTRAINT\_NAME | The name of the constraint. | | UNIQUE\_CONSTRAINT\_CATALOG | The name of the catalog containing the unique constraint that the constraint references. This value is always `def`. | | UNIQUE\_CONSTRAINT\_SCHEMA | The name of the schema containing the unique constraint that the constraint references. | | UNIQUE\_CONSTRAINT\_NAME | The name of the unique constraint that the constraint references. | | MATCH\_OPTION | The value of the constraint `MATCH` attribute. The only valid value at this time is `NONE`. | | UPDATE\_RULE | The value of the constraint `ON UPDATE` attribute. Valid values: `CASCADE`, `SET NULL`, `SET DEFAULT`, `RESTRICT`, and `NO ACTION`. | | DELETE\_RULE | The value of the constraint `ON DELETE` attribute. Valid values: `CASCADE`, `SET NULL`, `SET DEFAULT`, `RESTRICT`, and `NO ACTION`. | | TABLE\_NAME | The name of the table. This value is the same as in the `TABLE_CONSTRAINTS` table. | | REFERENCED\_TABLE\_NAME | The name of the table referenced by the constraint. | --- ### routine_load_jobs `routine_load_jobs` provides information about routine load jobs. The following fields are provided in `routine_load_jobs`: | **Field** | **Description** | | --------------------------- | ----------------------------------------------------------- | | ID | ID of the routine load job. | | NAME | Name of the routine load job. | | CREATE\_TIME | Creation time of the routine load job. | | PAUSE\_TIME | Pause time of the routine load job. | | END\_TIME | End time of the routine load job. | | DB\_NAME | Name of the database to which the routine load job belongs. | | TABLE\_NAME | Name of the table to which the data is loaded. | | STATE | State of the routine load job. | | DATA\_SOURCE\_TYPE | Type of the data source. | | CURRENT\_TASK\_NUM | Number of current tasks. | | JOB\_PROPERTIES | Properties of the routine load job. | | DATA\_SOURCE\_PROPERTIES | Properties of the data source. | | CUSTOM\_PROPERTIES | Custom properties of the routine load job. | | STATISTICS | Statistics of the routine load job. | | PROGRESS | Progress of the routine load job. | | REASONS\_OF\_STATE\_CHANGED | Reasons for state changes. | | ERROR\_LOG\_URLS | URLs of error logs. | | TRACKING\_SQL | SQL statement for tracking. | | OTHER\_MSG | Other messages. | | LATEST\_SOURCE\_POSITION | Latest source position in JSON format. | | OFFSET\_LAG | Offset lag in JSON format. | | TIMESTAMP\_PROGRESS | Timestamp progress in JSON format. | --- ### routines note This view does not apply to the available features in StarRocks. `routines` contains all stored routines (stored procedures and stored functions). The following fields are provided in `routine`: | **Field** | **Description** | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | SPECIFIC\_NAME | The name of the routine. | | ROUTINE\_CATALOG | The name of the catalog to which the routine belongs. This value is always `def`. | | ROUTINE\_SCHEMA | The name of the database to which the routine belongs. | | ROUTINE\_NAME | The name of the routine. | | ROUTINE\_TYPE | `PROCEDURE` for stored procedures, `FUNCTION` for stored functions. | | DTD\_IDENTIFIER | If the routine is a stored function, the return value data type. If the routine is a stored procedure, this value is empty. | | ROUTINE\_BODY | The language used for the routine definition. This value is always `SQL`. | | ROUTINE\_DEFINITION | The text of the SQL statement executed by the routine. | | EXTERNAL\_NAME | This value is always `NULL`. | | EXTERNAL\_LANGUAGE | The language of the stored routine. | | PARAMETER\_STYLE | This value is always `SQL`. | | IS\_DETERMINISTIC | `YES` or `NO`, depending on whether the routine is defined with the `DETERMINISTIC` characteristic. | | SQL\_DATA\_ACCESS | The data access characteristic for the routine. The value is one of `CONTAINS SQL`, `NO SQL`, `READS SQL DATA`, or `MODIFIES SQL DATA`. | | SQL\_PATH | This value is always `NULL`. | | SECURITY\_TYPE | The routine `SQL SECURITY` characteristic. The value is one of `DEFINER` or `INVOKER`. | | CREATED | The date and time when the routine was created. This is a `DATETIME` value. | | LAST\_ALTERED | The date and time when the routine was last modified. This is a `DATETIME` value. If the routine has not been modified since its creation, this value is the same as the `CREATED` value. | | SQL\_MODE | The SQL mode in effect when the routine was created or altered, and under which the routine executes. | | ROUTINE\_COMMENT | The text of the comment, if the routine has one. If not, this value is empty. | | DEFINER | The user named in the `DEFINER` clause (often the user who created the routine). | | CHARACTER\_SET\_CLIENT | | | COLLATION\_CONNECTION | | | DATABASE\_COLLATION | The collation of the database with which the routine is associated. | --- ### schema_privileges note This view does not apply to the available features in StarRocks. `schema_privileges` provides information about database privileges. The following fields are provided in `schema_privileges`: | **Field** | **Description** | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GRANTEE | The name of the user to which the privilege is granted. | | TABLE\_CATALOG | The name of the catalog to which the schema belongs. This value is always `def`. | | TABLE\_SCHEMA | The name of the schema. | | PRIVILEGE\_TYPE | The privilege granted. Each row lists a single privilege, so there is one row per schema privilege held by the grantee. | | IS\_GRANTABLE | `YES` if the user has the `GRANT OPTION` privilege, `NO` otherwise. The output does not list `GRANT OPTION` as a separate row with `PRIVILEGE_TYPE='GRANT OPTION'`. | --- ### schemata `schemata` provides information about databases. The following fields are provided in `schemata`: | **Field** | **Description** | | ----------------------------- | --------------------------------------------------------------------------------- | | CATALOG\_NAME | The name of the catalog to which the schema belongs. This value is always `NULL`. | | SCHEMA\_NAME | The name of the schema. | | DEFAULT\_CHARACTER\_SET\_NAME | The schema default character set. | | DEFAULT\_COLLATION\_NAME | The schema default collation. | | SQL\_PATH | This value is always `NULL`. | --- ### session_variables `session_variables` provides information about session variables. The following fields are provided in `session_variables`: | **Field** | **Description** | | --------------- | ---------------------- | | VARIABLE\_NAME | Name of the variable. | | VARIABLE\_VALUE | Value of the variable. | --- ### statistics note This view does not apply to the available features in StarRocks. `statistics` provides information about table indexes. The following fields are provided in `statistics`: | **Field** | **Description** | | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | TABLE\_CATALOG | The name of the catalog to which the table containing the index belongs. This value is always `def`. | | TABLE\_SCHEMA | The name of the database to which the table containing the index belongs. | | TABLE\_NAME | The name of the table containing the index. | | NON\_UNIQUE | 0 if the index cannot contain duplicates, 1 if it can. | | INDEX\_SCHEMA | The name of the database to which the index belongs. | | INDEX\_NAME | The name of the index. If the index is the primary key, the name is always `PRIMARY`. | | SEQ\_IN\_INDEX | The column sequence number in the index, starting with 1. | | COLUMN\_NAME | The column name. See also the description for the `EXPRESSION` column. | | COLLATION | How the column is sorted in the index. This can have values `A` (ascending), `D` (descending), or `NULL` (not sorted). | | CARDINALITY | An estimate of the number of unique values in the index. | | SUB\_PART | The index prefix. That is, the number of indexed characters if the column is only partly indexed, `NULL` if the entire column is indexed. | | PACKED | Indicates how the key is packed. `NULL` if it is not. | | NULLABLE | Contains `YES` if the column may contain `NULL` values and `''` if not. | | INDEX\_TYPE | The index method used (`BTREE`, `FULLTEXT`, `HASH`, `RTREE`). | | COMMENT | Information about the index not described in its own column, such as `disabled` if the index is disabled. | | INDEX\_COMMENT | Any comment provided for the index with a `COMMENT` attribute when the index was created. | --- ### stream_loads `stream_loads` provides information about stream load jobs. The following fields are provided in `stream_loads`: | **Field** | **Description** | | --------------------------- | ---------------------------------------------------------- | | LABEL | Label of the stream load job. | | ID | ID of the stream load job. | | LOAD\_ID | Load ID of the stream load job. | | TXN\_ID | Transaction ID of the stream load job. | | DB\_NAME | Name of the database to which the stream load job belongs. | | TABLE\_NAME | Name of the table to which the data is loaded. | | STATE | State of the stream load job. | | ERROR\_MSG | Error message if the stream load job failed. | | TRACKING\_URL | URL for tracking the stream load job. | | CHANNEL\_NUM | Number of channels used in the stream load job. | | PREPARED\_CHANNEL\_NUM | Number of prepared channels in the stream load job. | | NUM\_ROWS\_NORMAL | Number of normal rows loaded. | | NUM\_ROWS\_AB\_NORMAL | Number of abnormal rows loaded. | | NUM\_ROWS\_UNSELECTED | Number of unselected rows. | | NUM\_LOAD\_BYTES | Number of bytes loaded. | | TIMEOUT\_SECOND | Timeout in seconds for the stream load job. | | CREATE\_TIME\_MS | Creation time of the stream load job (in milliseconds). | | BEFORE\_LOAD\_TIME\_MS | Time before loading started (in milliseconds). | | START\_LOADING\_TIME\_MS | Time when loading started (in milliseconds). | | START\_PREPARING\_TIME\_MS | Time when preparing started (in milliseconds). | | FINISH\_PREPARING\_TIME\_MS | Time when preparing finished (in milliseconds). | | END\_TIME\_MS | End time of the stream load job (in milliseconds). | | CHANNEL\_STATE | State of the channels. | | TYPE | Type of the stream load job. | | TRACKING\_SQL | SQL statement for tracking the stream load job. | --- ### table_constraints note This view does not apply to the available features in StarRocks. `table_constraints` describes which tables have constraints. The following fields are provided in `table_constraints`: | **Field** | **Description** | | ------------------- | ------------------------------------------------------------------------------------ | | CONSTRAINT\_CATALOG | The name of the catalog to which the constraint belongs. This value is always `def`. | | CONSTRAINT\_SCHEMA | The name of the database to which the constraint belongs. | | CONSTRAINT\_NAME | The name of the constraint. | | TABLE\_SCHEMA | The name of the database to which the table belongs. | | TABLE\_NAME | The name of the table. | | CONSTRAINT\_TYPE | The type of constraint. | --- ### table_privileges note This view does not apply to the available features in StarRocks. `table_privileges` provides information about table privileges. The following fields are provided in `table_privileges`: | **Field** | **Description** | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GRANTEE | The name of the user to which the privilege is granted. | | TABLE\_CATALOG | The name of the catalog to which the table belongs. This value is always `def`. | | TABLE\_SCHEMA | The name of the database to which the table belongs. | | TABLE\_NAME | The name of the table. | | PRIVILEGE\_TYPE | The privilege granted. The value can be any privilege that can be granted at the table level. | | IS\_GRANTABLE | `YES` if the user has the `GRANT OPTION` privilege, `NO` otherwise. The output does not list `GRANT OPTION` as a separate row with `PRIVILEGE_TYPE='GRANT OPTION'`. | --- ### tables `tables` provides information about tables. The following fields are provided in `tables`: | **Field** | **Description** | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | TABLE\_CATALOG | Name of the catalog that stores the table. | | TABLE\_SCHEMA | Name of the database that stores the table. | | TABLE\_NAME | Name of the table. | | TABLE\_TYPE | Type of the table. Valid values: `TABLE`(`BASE TABLE`), `VIEW` and `SYSTEM VIEW` (supported only when `mysql_server_version` is `8.0.33`). | | ENGINE | Engine type of the table. Valid values: `StarRocks`, `MySQL`, `MEMORY` or an empty string. | | VERSION | Applies to a feature not available in StarRocks. | | ROW\_FORMAT | Applies to a feature not available in StarRocks. | | TABLE\_ROWS | Row count of the table. | | AVG\_ROW\_LENGTH | Average row length (size) of the table. It is equivalent to `DATA_LENGTH`/`TABLE_ROWS`. Unit: Byte. | | DATA\_LENGTH | The data length of the table is determined by summing the data length of the table across all replicas. Unit: Byte. | | MAX\_DATA\_LENGTH | Applies to a feature not available in StarRocks. | | INDEX\_LENGTH | Applies to a feature not available in StarRocks. | | DATA\_FREE | Applies to a feature not available in StarRocks. | | AUTO\_INCREMENT | Applies to a feature not available in StarRocks. | | CREATE\_TIME | The time when the table was created. | | UPDATE\_TIME | The last time when the table was updated. | | CHECK\_TIME | The last time when a consistency check was performed on the table. | | TABLE\_COLLATION | The default collation of the table. | | CHECKSUM | Applies to a feature not available in StarRocks. | | CREATE\_OPTIONS | Applies to a feature not available in StarRocks. | | TABLE\_COMMENT | Comment on the table. | --- ### tables_config `tables_config` provides information about the configuration of tables. The following fields are provided in `tables_config`: | **Field** | **Description** | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | TABLE\_SCHEMA | Name of the database that stores the table. | | TABLE\_NAME | Name of the table. | | TABLE\_ENGINE | Engine type of the table. | | TABLE\_MODEL | Data model of the table. Valid values: `DUP_KEYS`, `AGG_KEYS`, `UNQ_KEYS` or `PRI_KEYS`. | | PRIMARY\_KEY | Primary key of a Primary Key table or a Unique Key table. An empty string is returned if the table is not a Primary Key table or a Unique Key table. | | PARTITION\_KEY | Partitioning columns of the table. | | DISTRIBUTE\_KEY | Bucketing columns of the table. | | DISTRIBUTE\_TYPE | Data distribution method of the table. | | DISTRIBUTE\_BUCKET | Number of buckets in the table. | | SORT\_KEY | Sort keys of the table. | | PROPERTIES | Properties of the table. | | TABLE\_ID | ID of the table. | --- ### task_runs `task_runs` provides information about the execution of asynchronous tasks. The following fields are provided in `task_runs`: | **Field** | **Description** | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | QUERY\_ID | ID of the query. | | TASK\_NAME | Name of the task. | | CREATE\_TIME | Time when the task was created. | | FINISH\_TIME | Time when the task finished. | | STATE | State of the task. Valid values: `PENDING`, `RUNNING`, `FAILED`, `SUCCESS`, `MERGED`, and `SKIPPED`. `MERGED` is for materialized view refresh tasks. When a new refresh task is submitted and the old task is still in the pending queue, these tasks will be merged and their priority level will be maintained. `SKIPPED` is for materialized view refresh tasks. When no data changes are detected on the base table partition, the refresh for the corresponding materialized view partition is skipped. | | CATALOG | Catalog where the task belongs. | | DATABASE | Database where the task belongs. | | DEFINITION | SQL definition of the task. | | EXPIRE\_TIME | Time when the task expires. | | ERROR\_CODE | Error code of the task. | | ERROR\_MESSAGE | Error message of the task. | | PROGRESS | The progress of the task. | | EXTRA\_MESSAGE | Extra message for the task, for example, the partition information in an asynchronous materialized view creation task. | | PROPERTIES | Properties of the task. | | JOB\_ID | Job ID of the task. | | PROCESS\_TIME | Process time of the task. | | TASK\_SOURCE | Source that submitted the task. Valid values: `CTAS`, `MV`, `INSERT`, `PIPE`, and `DATACACHE_SELECT`. `UNKNOWN` is returned for legacy records whose source was not recorded. | A task run record is produced by either [SUBMIT TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md) or [CREATE MATRIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md). note A materialized view operation may generate multi task runs, and each task run represents a refresh sub-task which is split in accordance with the `partition_refresh_number` configuration. #### `EXTRA_MESSAGE`[​](#extra_message "Direct link to extra_message") For a materialized view task run, `EXTRA_MESSAGE` field will contain the materialized view task run's detail messages, you can find more details in [materialized\_view\_task\_run\_details](https://docs.starrocks.io/docs/using_starrocks/async_mv/materialized_view_task_run_details.md). --- ### tasks `tasks` provides information about asynchronous tasks. The following fields are provided in `tasks`: | **Field** | **Description** | | ------------ | --------------------------------------------------------------------------------------------- | | TASK\_NAME | Name of the task. | | CREATE\_TIME | Time when the task was created. | | SCHEDULE | Task schedule. If the task is regularly triggered, this field displays `START xxx EVERY xxx`. | | DATABASE | Database where the task belongs. | | DEFINITION | SQL definition of the task. | | EXPIRE\_TIME | Time when the task expires. | #### Reference[​](#reference "Direct link to Reference") [`SUBMIT_TASK` documentation](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md) --- ### triggers note This view does not apply to the available features in StarRocks. `triggers` provides information about triggers. The following fields are provided in `triggers`: | **Field** | **Description** | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | TRIGGER\_CATALOG | The name of the catalog to which the trigger belongs. This value is always `def`. | | TRIGGER\_SCHEMA | The name of the database to which the trigger belongs. | | TRIGGER\_NAME | The name of the trigger. | | EVENT\_MANIPULATION | The trigger event. This is the type of operation on the associated table for which the trigger activates. The value is `INSERT` (a row was inserted), `DELETE` (a row was deleted), or `UPDATE` (a row was modified). | | EVENT\_OBJECT\_CATALOG | Every trigger is associated with exactly one table. The catalog in which this table occurs. | | EVENT\_OBJECT\_SCHEMA | Every trigger is associated with exactly one table. The database in which this table occurs. | | EVENT\_OBJECT\_TABLE | The name of the table that the trigger is associated with. | | ACTION\_ORDER | The ordinal position of the trigger's action within the list of triggers on the same table with the same `EVENT_MANIPULATION` and `ACTION_TIMING` values. | | ACTION\_CONDITION | This value is always `NULL`. | | ACTION\_STATEMENT | The trigger body; that is, the statement executed when the trigger activates. This text uses UTF-8 encoding. | | ACTION\_ORIENTATION | This value is always `ROW`. | | ACTION\_TIMING | Whether the trigger activates before or after the triggering event. The value is `BEFORE` or `AFTER`. | | ACTION\_REFERENCE\_OLD\_TABLE | This value is always `NULL`. | | ACTION\_REFERENCE\_NEW\_TABLE | This value is always `NULL`. | | ACTION\_REFERENCE\_OLD\_ROW | The old column identifiers. The value is always `OLD`. | | ACTION\_REFERENCE\_NEW\_ROW | The new column identifiers. The value is always `NEW`. | | CREATED | The date and time when the trigger was created. This is a `DATETIME(2)` value (with a fractional part in hundredths of seconds) for triggers. | | SQL\_MODE | The SQL mode in effect when the trigger was created, and under which the trigger executes. | | DEFINER | The user named in the `DEFINER` clause (often the user who created the trigger). | | CHARACTER\_SET\_CLIENT | | | COLLATION\_CONNECTION | | | DATABASE\_COLLATION | The collation of the database with which the trigger is associated. | --- ### user_privileges note This view does not apply to the available features in StarRocks. `user_privileges` provides information about user privileges. The following fields are provided in `user_privileges`: | **Field** | **Description** | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GRANTEE | The name of the user to which the privilege is granted. | | TABLE\_CATALOG | The name of the catalog. This value is always `def`. | | PRIVILEGE\_TYPE | The privilege granted. The value can be any privilege that can be granted at the global level. | | IS\_GRANTABLE | `YES` if the user has the `GRANT OPTION` privilege, `NO` otherwise. The output does not list `GRANT OPTION` as a separate row with `PRIVILEGE_TYPE='GRANT OPTION'`. | --- ### verbose_session_variables `verbose_session_variables` provides detailed information about session variables, including their default values and whether they have been changed from the default. The following fields are provided in `verbose_session_variables`: | **Field** | **Description** | | --------------- | --------------------------------------------------------------------------------------------------- | | VARIABLE\_NAME | The name of the session variable. | | VARIABLE\_VALUE | The current value of the session variable. | | DEFAULT\_VALUE | The default value of the session variable. | | IS\_CHANGED | Indicates whether the variable's value has been changed from its default (`true`) or not (`false`). | --- ### views `views` provides information about all user-defined views. The following fields are provided in `views`: | **Field** | **Description** | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | TABLE\_CATALOG | The name of the catalog to which the view belongs. This value is always `def`. | | TABLE\_SCHEMA | The name of the database to which the view belongs. | | TABLE\_NAME | The name of the view. | | VIEW\_DEFINITION | The `SELECT` statement that provides the definition of the view. | | CHECK\_OPTION | The value of the `CHECK_OPTION` attribute. The value is one of `NONE`, `CASCADE`, or `LOCAL`. | | IS\_UPDATABLE | Whether the view is updatable. The flag is set to `YES` (true) if `UPDATE` and `DELETE` (and similar operations) are legal for the view. Otherwise, the flag is set to `NO` (false). If a view is not updatable, statements such `UPDATE`, `DELETE`, and `INSERT` are illegal and are rejected. | | DEFINER | The user of the user who created the view. | | SECURITY\_TYPE | | | CHARACTER\_SET\_CLIENT | | | COLLATION\_CONNECTION | | --- ### warehouse_metrics `warehouse_metrics` provides information about the metrics of each warehouse. The following fields are provided in `warehouse_metrics`: | **Field** | **Description** | | --------------------------- | --------------------------------------------------------- | | WAREHOUSE\_ID | ID of the warehouse. | | WAREHOUSE\_NAME | Name of the warehouse. | | QUEUE\_PENDING\_LENGTH | Number of pending queries in the queue. | | QUEUE\_RUNNING\_LENGTH | Number of running queries in the queue. | | MAX\_PENDING\_LENGTH | Maximum number of pending queries allowed in the queue. | | MAX\_PENDING\_TIME\_SECOND | Maximum pending time in seconds for queries in the queue. | | EARLIEST\_QUERY\_WAIT\_TIME | Earliest query wait time. | | MAX\_REQUIRED\_SLOTS | Maximum required slots. | | SUM\_REQUIRED\_SLOTS | Sum of required slots. | | REMAIN\_SLOTS | Remaining slots. | | MAX\_SLOTS | Maximum slots. | | EXTRA\_MESSAGE | Extra message. | --- ### warehouse_queries `warehouse_queries` provides information about queries running on each warehouse. The following fields are provided in `warehouse_queries`: | **Field** | **Description** | | --------------------- | ------------------------------------------------------------ | | WAREHOUSE\_ID | ID of the warehouse. | | WAREHOUSE\_NAME | Name of the warehouse. | | QUERY\_ID | ID of the query. | | STATE | State of the query (e.g., `PENDING`, `RUNNING`, `FINISHED`). | | EST\_COSTS\_SLOTS | Estimated cost slots for the query. | | ALLOCATE\_SLOTS | Allocated slots for the query. | | QUEUED\_WAIT\_SECONDS | Time in seconds the query waited in the queue. | | QUERY | The SQL query string. | | QUERY\_START\_TIME | Start time of the query. | | QUERY\_END\_TIME | End time of the query. | | QUERY\_DURATION | Duration of the query. | | EXTRA\_MESSAGE | Extra message. | --- ### Aggregate #### [📄️ any\_value](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/any_value.md) [Obtains an arbitrary row from each aggregated group.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/any_value.md) #### [📄️ approx\_count\_distinct](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/approx_count_distinct.md) [Returns the approximate value of aggregate function similar to the result of COUNT(DISTINCT col).](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/approx_count_distinct.md) #### [📄️ approx\_top\_k](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/approx_top_k.md) [Returns the top k most frequently occurring item values in an expr along with their approximate counts.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/approx_top_k.md) #### [📄️ avg](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/avg.md) [Returns the average value of selected fields.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/avg.md) #### [📄️ bitmap](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/bitmap.md) [Here is a simple example to illustrate the usage of several aggregate functions in Bitmap.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/bitmap.md) #### [📄️ bool\_or](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/bool_or.md) [Returns true if at least one row for expr is true.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/bool_or.md) #### [📄️ corr](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/corr.md) [Returns the Pearson correlation coefficient between two expressions.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/corr.md) #### [📄️ count](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/count.md) [Returns the total number of rows specified by an expression.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/count.md) #### [📄️ count\_if](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/count_if.md) [Returns the number of records that meet the specified condition or 0 if no records satisfy the condition.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/count_if.md) #### [📄️ covar\_pop](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/covar_pop.md) [Returns the population covariance of two expressions.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/covar_pop.md) #### [📄️ covar\_samp](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/covar_samp.md) [Returns the sample covariance of two expressions.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/covar_samp.md) #### [📄️ ds\_hll\_accumulate](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/ds_hll_accumulate.md) [Accumulates values into a DataSketches HyperLogLog (HLL) sketch and returns the serialized sketch as VARBINARY for approximate distinct counting.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/ds_hll_accumulate.md) #### [📄️ ds\_hll\_combine](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/ds_hll_combine.md) [Combines multiple serialized DataSketches HyperLogLog (HLL) sketches into a single serialized sketch for approximate distinct counting.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/ds_hll_combine.md) #### [📄️ ds\_hll\_count\_distinct](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/ds_hll_count_distinct.md) [Returns the approximate distinct count using DataSketches HyperLogLog (HLL). Similar to APPROX\_COUNT\_DISTINCT but with higher precision.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/ds_hll_count_distinct.md) #### [📄️ ds\_hll\_estimate](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/ds_hll_estimate.md) [Estimates the approximate distinct count from a serialized DataSketches HyperLogLog (HLL) sketch.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/ds_hll_estimate.md) #### [📄️ ds\_theta\_count\_distinct](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/ds_theta_count_distinct.md) [Returns approximate distinct count using DataSketches Theta sketch. Faster than COUNT(DISTINCT) with lower memory usage for high-cardinality columns.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/ds_theta_count_distinct.md) #### [📄️ group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/group_concat.md) [Concatenates non-null values from a group into a single string, with a sep argument, which is , by default if not specified.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/group_concat.md) #### [📄️ grouping](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/grouping.md) [Indicates whether a column is an aggregate column.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/grouping.md) #### [📄️ grouping\_id](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/grouping_id.md) [grouping\_id is used to distinguish the grouping statistics results of the same grouping standard.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/grouping_id.md) #### [📄️ hll\_raw\_agg](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/hll_raw_agg.md) [Aggregate function that combines multiple HLL values into a single HLL for approximate COUNT(DISTINCT).](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/hll_raw_agg.md) #### [📄️ hll\_union](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/hll_union.md) [Returns the concatenation of a set of HLL values.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/hll_union.md) #### [📄️ hll\_union\_agg](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/hll_union_agg.md) [HLL is an engineering implementation based on the HyperLogLog algorithm, which is used to save the intermediate results of HyperLogGog calculation process.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/hll_union_agg.md) #### [📄️ mann\_whitney\_u\_test](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/mann_whitney_u_test.md) [mann\_whitney\_u\_test performs the Mann-Whitney rank test on samples derived from two populations.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/mann_whitney_u_test.md) #### [📄️ max](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/max.md) [Returns the maximum value of the expr expression.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/max.md) #### [📄️ max\_by](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/max_by.md) [Returns the value of x associated with the maximum value of y.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/max_by.md) #### [📄️ min](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/min.md) [Returns the minimum value of the expr expression.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/min.md) #### [📄️ min\_by](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/min_by.md) [Returns the value of x associated with the minimum value of y.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/min_by.md) #### [📄️ MIN\_N, MAX\_N](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/min_n_max_n.md) [Returns the n smallest or largest values from an expression as an array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/min_n_max_n.md) #### [📄️ multi\_distinct\_count](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/multi_distinct_count.md) [Returns the total number of rows of the expr, equivalent to count(distinct expr).](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/multi_distinct_count.md) #### [📄️ multi\_distinct\_sum](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/multi_distinct_sum.md) [Returns the sum of distinct values in expr, equivalent to sum(distinct expr).](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/multi_distinct_sum.md) #### [📄️ percentile\_approx](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/percentile_approx.md) [Returns the approximate value for a given percentile p, or an array of values for corresponding percentiles if p is an array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/percentile_approx.md) #### [📄️ percentile\_approx\_weight](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/percentile_approx_weight.md) [Returns the approximation of the p-th percentile with weight. A weighted version of PERCENTILE\_APPROX that accepts a weight column or constant for each...](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/percentile_approx_weight.md) #### [📄️ percentile\_cont](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/percentile_cont.md) [Computes the percentile value of expr with linear interpolation.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/percentile_cont.md) #### [📄️ percentile\_disc](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/percentile_disc.md) [Returns a percentile value based on a discrete distribution of the input column expr.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/percentile_disc.md) #### [📄️ percentile\_disc\_lc](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/percentile_disc_lc.md) [Returns a percentile value based on a discrete distribution of the input column expr.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/percentile_disc_lc.md) #### [📄️ retention](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/retention.md) [Calculates the user retention rate within a specified period of time.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/retention.md) #### [📄️ std](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/std.md) [Returns the standard deviation of an expression.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/std.md) #### [📄️ stddev,stddev\_pop,std](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/stddev.md) [Returns the population standard deviation of the expr expression.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/stddev.md) #### [📄️ stddev\_samp](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/stddev_samp.md) [Returns the sample standard deviation of an expression.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/stddev_samp.md) #### [📄️ sum](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/sum.md) [Returns the sum of non-null values for expr.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/sum.md) #### [📄️ sum\_map](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/sum_map.md) [Aggregates MAP values by summing numeric values for matching keys across multiple rows.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/sum_map.md) #### [📄️ var\_samp,variance\_samp](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/var_samp.md) [Returns the sample variance of an expression.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/var_samp.md) #### [📄️ variance,var\_pop,variance\_pop](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/variance.md) [Returns the population variance of an expression.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/variance.md) #### [📄️ window\_funnel](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/window_funnel.md) [Searches for an event chain in a sliding window and calculates the maximum number of consecutive events in the event chain.](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/window_funnel.md) --- ### any_value Obtains an arbitrary row from each aggregated group. You can use this function to optimize a query that has a `GROUP BY` clause. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext ANY_VALUE(expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: the expression that gets aggregated. Since v3.2, `expr` can evaluate to ARRAY, MAP, and STRUCT. #### Return value[​](#return-value "Direct link to Return value") Returns an arbitrary row from each aggregated group. The return value is non-deterministic. #### Examples[​](#examples "Direct link to Examples") Create a table `t0` and insert data into this table. ```sql CREATE TABLE t0( a INT, b BIGINT, c SMALLINT, d ARRAY, e JSON, f MAP, g STRUCT ) DUPLICATE KEY(a) DISTRIBUTED BY HASH(a); INSERT INTO t0 VALUES (1, 1, 1, [2,3,4],parse_json('{"a":1, "b":true}'), map{1:1,3:4}, row(1, 2)), (1, 2, 1, [2,3,5],parse_json('{"a":2, "b":true}'), map{1:2,3:3},row(2, 2)), (2, 1, 1, [2,3,6],parse_json('{"a":3, "b":true}'), map{2:1,3:2},row(3, 2)), (2, 2, 2, [2,4,5],parse_json('{"a":4, "b":false}'),map{1:3,3:1},row(4, 2)), (3, 1, 1, [3,3,5],parse_json('{"a":5, "b":false}'),map{2:1,3:3},row(1, 2)); ``` ```plain mysql> select * from t0 order by a; +------+------+------+---------+----------------------+-----------+-----------------+ | a | b | c | d | e | f | g | +------+------+------+---------+----------------------+-----------+-----------------+ | 1 | 1 | 1 | [2,3,4] | {"a": 1, "b": true} | {1:1,3:4} | {"a":"1","b":2} | | 1 | 2 | 1 | [2,3,5] | {"a": 2, "b": true} | {1:2,3:3} | {"a":"2","b":2} | | 2 | 1 | 1 | [2,3,6] | {"a": 3, "b": true} | {2:1,3:2} | {"a":"3","b":2} | | 2 | 2 | 2 | [2,4,5] | {"a": 4, "b": false} | {1:3,3:1} | {"a":"4","b":2} | | 3 | 1 | 1 | [3,3,5] | {"a": 5, "b": false} | {2:1,3:3} | {"a":"1","b":2} | +------+------+------+---------+----------------------+-----------+-----------------+ 5 rows in set (0.01 sec) ``` Use `any_value` for data query. For `a = 1` and `a = 2`, an arbitrary row is returned for `b`. ```plain mysql> select a,any_value(b),sum(c) from t0 group by a; +------+----------------+----------+ | a | any_value(`b`) | sum(`c`) | +------+----------------+----------+ | 1 | 1 | 2 | | 2 | 1 | 3 | | 3 | 1 | 1 | +------+----------------+----------+ 3 rows in set (0.01 sec) mysql> select a,any_value(d),sum(b) from t0 group by a; +------+--------------+--------+ | a | any_value(d) | sum(b) | +------+--------------+--------+ | 3 | [3,3,5] | 1 | | 1 | [2,3,4] | 3 | | 2 | [2,3,6] | 3 | +------+--------------+--------+ 2 rows in set (0.01 sec) ``` --- ### approx_count_distinct Returns the approximate value of aggregate function similar to the result of COUNT(DISTINCT col). It is faster than the COUNT and DISTINCT combination and uses a fixed-size memory, so less memory is used for columns of high cardinality. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext APPROX_COUNT_DISTINCT(expr) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select approx_count_distinct(query_id) from log_statis group by datetime; +-----------------------------------+ | approx_count_distinct(`query_id`) | +-----------------------------------+ | 17721 | +-----------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") APPROX\_COUNT\_DISTINCT --- ### approx_top_k Returns the top `k` most frequently occurring item values in an `expr` along with their approximate counts. This function is supported from v3.0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext APPROX_TOP_K( [ , [ , ] ] ) ``` #### Arguments[​](#arguments "Direct link to Arguments") * `expr`: An expression of STRING, BOOLEAN, DATE, DATETIME, or numeric type. * `k`: An optional INTEGER literal greater than 0. If `k` is not specified, it defaults to `5`. The maximum value is `100000`. * `counter_num`: An optional INTEGER literal greater than or equal to `k`, The larger the `counter_num` is, the more accurate the result will be. However, this also comes with increased CPU and memory costs. * The maximum value is `100000`. * If `counter_num` is not specified, it defaults to `max(min(2 * k, 100), 100000)`. #### Returns[​](#returns "Direct link to Returns") Results are returned as an ARRAY of type STRUCT, where each STRUCT contains an `item` field for the value (with its original input type) and a `count` field (of type BIGINT) with the approximate number of occurrences. The array is sorted by `count` descending. The aggregate function returns the top `k` most frequently occurring item values in an expression expr along with their approximate counts. The error in each count may be up to `2.0 * numRows / counter_num` where `numRows` is the total number of rows. Higher values of `counter_num` provide better accuracy at the cost of increased memory usage. Expressions that have fewer than `counter_num` distinct items will yield exact item counts. Results include `NULL` values as their own item in the results. #### Examples[​](#examples "Direct link to Examples") Use data in the [scores](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function.md#window-function-sample-table) table as an example. ```plaintext -- Calculate the score distribution of each subject. MySQL > SELECT subject, APPROX_TOP_K(score) AS top_k FROM scores GROUP BY subject; +---------+--------------------------------------------------------------------------------------------------------------------+ | subject | top_k | +---------+--------------------------------------------------------------------------------------------------------------------+ | physics | [{"item":99,"count":2},{"item":null,"count":1},{"item":100,"count":1},{"item":85,"count":1},{"item":60,"count":1}] | | english | [{"item":null,"count":1},{"item":92,"count":1},{"item":98,"count":1},{"item":100,"count":1},{"item":85,"count":1}] | | NULL | [{"item":90,"count":1}] | | math | [{"item":80,"count":2},{"item":null,"count":1},{"item":92,"count":1},{"item":95,"count":1},{"item":70,"count":1}] | +---------+--------------------------------------------------------------------------------------------------------------------+ -- Calculate the score distribution of the math subject. MySQL > SELECT subject, APPROX_TOP_K(score) AS top_k FROM scores WHERE subject IN ('math') GROUP BY subject; +---------+-------------------------------------------------------------------------------------------------------------------+ | subject | top_k | +---------+-------------------------------------------------------------------------------------------------------------------+ | math | [{"item":80,"count":2},{"item":null,"count":1},{"item":95,"count":1},{"item":92,"count":1},{"item":70,"count":1}] | +---------+-------------------------------------------------------------------------------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") APPROX\_TOP\_K --- ### avg Returns the average value of selected fields. Optional field DISTINCT parameters can be used to return the weighted average. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext AVG([DISTINCT] expr) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT datetime, AVG(cost_time) FROM log_statis group by datetime; +---------------------+--------------------+ | datetime | avg(`cost_time`) | +---------------------+--------------------+ | 2019-07-03 21:01:20 | 25.827794561933533 | +---------------------+--------------------+ MySQL > SELECT datetime, AVG(distinct cost_time) FROM log_statis group by datetime; +---------------------+---------------------------+ | datetime | avg(DISTINCT `cost_time`) | +---------------------+---------------------------+ | 2019-07-04 02:23:24 | 20.666666666666668 | +---------------------+---------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") AVG --- ### bitmap Here is a simple example to illustrate the usage of several aggregate functions in Bitmap. For detailed function definitions or more Bitmap functions, see bitmap-functions. #### Create table[​](#create-table "Direct link to Create table") The aggregation model is needed when creating table. The data type is bitmap and the aggregation function is bitmap\_union. ```sql CREATE TABLE `pv_bitmap` ( `dt` int(11) NULL COMMENT "", `page` varchar(10) NULL COMMENT "", `user_id` bitmap BITMAP_UNION NULL COMMENT "" ) ENGINE=OLAP AGGREGATE KEY(`dt`, `page`) COMMENT "OLAP" DISTRIBUTED BY HASH(`dt`); ``` Note: With an amount of data, you'd better create a rollup table corresponding to high-frequent bitmap\_union. ```sql ALTER TABLE pv_bitmap ADD ROLLUP pv (page, user_id); ``` #### Data Load[​](#data-load "Direct link to Data Load") `TO_BITMAP (expr)`: Convert 0 ~ 18446744073709551615 unsigned bigint to bitmap `BITMAP_EMPTY ()`: Generate empty bitmap columns, used for the default value to be filled in when inserting or inputting `BITMAP_HASH (expr)`: Convert columns of any type to a bitmap by 32-bit hashing `BITMAP_HASH64 (expr)`: Convert columns of any type to a bitmap by 64-bit hashing ##### Stream Load[​](#stream-load "Direct link to Stream Load") When inputting data using Stream Load, you can convert the data to a BItmap field as follows: ```bash cat data | curl --location-trusted -u user:passwd -T - \ -H "columns: dt,page,user_id, user_id=to_bitmap(user_id)" \ http://host:8410/api/test/testDb/_stream_load ``` ```bash cat data | curl --location-trusted -u user:passwd -T - \ -H "columns: dt,page,user_id, user_id=bitmap_hash(user_id)" \ http://host:8410/api/test/testDb/_stream_load ``` ```bash cat data | curl --location-trusted -u user:passwd -T - \ -H "columns: dt,page,user_id, user_id=bitmap_hash64(user_id)" \ http://host:8410/api/test/testDb/_stream_load ``` ```bash cat data | curl --location-trusted -u user:passwd -T - \ -H "columns: dt,page,user_id, user_id=bitmap_empty()" \ http://host:8410/api/test/testDb/_stream_load ``` ##### Insert Into[​](#insert-into "Direct link to Insert Into") When inputting data using Insert Into, you need to select the corresponding mode based on the type of columns in the source table. * id2's column type in source table is bitmap ```sql insert into bitmap_table1 select id, id2 from bitmap_table2; ``` * id2's column type in target table is bitmap ```sql insert into bitmap_table1 (id, id2) values (1001, to_bitmap(1000)) , (1001, to_bitmap(2000)); ``` * id2's column type in source table is bitmap, and is the result of aggregation using bit\_map\_union(). ```sql insert into bitmap_table1 select id, bitmap_union(id2) from bitmap_table2 group by id; ``` * id2's column type in source table is INT, and the bitmap type is generated by to\_bitmap(). ```sql insert into bitmap_table1 select id, to_bitmap(id2) from table; ``` * id2's column type in source table is STRING, and the bitmap type is generated by bitmap\_hash(). ```sql insert into bitmap_table1 select id, bitmap_hash(id2) from table; ``` * id2's column type in source table is STRING, and the bitmap type is generated by bitmap\_hash64(). ```sql insert into bitmap_table1 select id, bitmap_hash64(id2) from table; ``` #### Data Query[​](#data-query "Direct link to Data Query") ##### Syntax[​](#syntax "Direct link to Syntax") \`\`BITMAP\_UNION (expr)\`: Calculate the union of the input Bitmaps, and returns the new Bitmap. `BITMAP_UNION_COUNT (expr)`: Calculate the union of the input Bitmaps, and returns its cardinality, equivalent to BITMAP\_COUNT (BITMAP\_UNION (expr)). It is recommended to use the BITMAP\_UNION\_COUNT function first, for its performance is better than BITMAP\_COUNT (BITMAP\_UNION (expr)). `BITMAP_UNION_INT (expr)`: Calculate the number of different values in columns of type TINYINT, SMALLINT and INT, return the value same as COUNT (DISTINCT expr). `INTERSECT_COUNT (bitmap_column_to_count, filter_column, filter_values ...)`: Calculate the cardinality of the intersection of multiple bitmaps that satisfy filter\_column condition. bitmap\_column\_to\_count is a column of type bitmap, filter\_column is a column of varying dimensions, and filter\_values is a list of dimension values. `BITMAP_INTERSECT(expr)`: Calculate the intersection of this group of bitmap values and returns a new bitmap. ##### Example[​](#example "Direct link to Example") The following SQL uses the `pv_bitmap` table above as an example: Calculate the deduplicated value for `user_id`: ```sql select bitmap_union_count(user_id) from pv_bitmap; select bitmap_count(bitmap_union(user_id)) from pv_bitmap; ``` Calculate the deduplicated value of `id`: ```sql select bitmap_union_int(id) from pv_bitmap; ``` Calculate the retention of `user_id`: ```sql select intersect_count(user_id, page, 'game') as game_uv, intersect_count(user_id, page, 'shopping') as shopping_uv, intersect_count(user_id, page, 'game', 'shopping') as retention -- Number of users that access both the 'game' and 'shopping' pages from pv_bitmap where page in ('game', 'shopping'); ``` #### keyword[​](#keyword "Direct link to keyword") BITMAP,BITMAP\_COUNT,BITMAP\_EMPTY,BITMAP\_UNION,BITMAP\_UNION\_INT,TO\_BITMAP,BITMAP\_UNION\_COUNT,INTERSECT\_COUNT,BITMAP\_INTERSECT --- ### bool_or Returns true if at least one row for `expr` is true. Otherwise, returns false. `boolor_agg` is an alias for this function. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext bool_or(expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: The expression must evaluate to a boolean value, that is, true or false. #### Return value[​](#return-value "Direct link to Return value") Returns a boolean value. An error is returned if `expr` does not exist. #### Usage notes[​](#usage-notes "Direct link to Usage notes") This function ignores NULL values. #### Examples[​](#examples "Direct link to Examples") 1. Create a table named `employees`. ```sql CREATE TABLE IF NOT EXISTS employees ( region_num TINYINT, id BIGINT, is_manager BOOLEAN ) DISTRIBUTED BY HASH(region_num); ``` 2. Insert data into `employees`. ```sql INSERT INTO employees VALUES (3,432175, TRUE), (4,567832, FALSE), (3,777326, FALSE), (5,342611, TRUE), (2,403882, FALSE); ``` 3. Query data from `employees`. ```plain MySQL > select * from employees; +------------+--------+------------+ | region_num | id | is_manager | +------------+--------+------------+ | 3 | 432175 | 1 | | 4 | 567832 | 0 | | 3 | 777326 | 0 | | 5 | 342611 | 1 | | 2 | 403882 | 0 | +------------+--------+------------+ 5 rows in set (0.01 sec) ``` 4. Use this function to check for managers in each region. Example 1: Check if there is at least one manager in each region. ```plain MySQL > SELECT region_num, bool_or(is_manager) from employees group by region_num; +------------+---------------------+ | region_num | bool_or(is_manager) | +------------+---------------------+ | 3 | 1 | | 4 | 0 | | 5 | 1 | | 2 | 0 | +------------+---------------------+ 4 rows in set (0.01 sec) ``` #### keyword[​](#keyword "Direct link to keyword") BOOL\_OR, bool\_or, boolor\_agg --- ### corr Returns the Pearson correlation coefficient between two expressions. This function is supported from v2.5.10. It can also be used as a window function. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext CORR(expr1, expr2) ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr1` and `expr2` must evaluate to TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, or DECIMAL. If `expr1` and `expr2` are table columns, this function calculates the correlation coefficient of these two columns. #### Return value[​](#return-value "Direct link to Return value") Returns a DOUBLE value. The formula is as follows, where `n` represents the row count of the table: ![corr formula](/assets/images/corr_formula-8cb13bade3e73013519faa251ce79413.png) #### Usage notes[​](#usage-notes "Direct link to Usage notes") * A data row is counted only when the two columns in this row are non-null values. Otherwise, this data row is eliminated from the result. * 0 is returned if `n` is 1. * NULL is returned in any input is NULL. #### Examples[​](#examples "Direct link to Examples") Suppose table `agg` has the following data: ```plaintext mysql> select * from agg; +------+-------+-------+ | no | k | v | +------+-------+-------+ | 1 | 10.00 | NULL | | 2 | 10.00 | 11.00 | | 2 | 20.00 | 22.00 | | 2 | 25.00 | NULL | | 2 | 30.00 | 35.00 | +------+-------+-------+ ``` Calculate the correlation coefficient of the `k` and `v` columns: ```plaintext mysql> select no,CORR(k,v) from agg group by no; +------+--------------------+ | no | corr(k, v) | +------+--------------------+ | 1 | NULL | | 2 | 0.9988445981121532 | +------+--------------------+ ``` --- ### count Returns the total number of rows specified by an expression. This function has three variations: * `COUNT(*)` counts all rows in a table, no matter whether they contain NULL values. * `COUNT(expr)` counts the number of rows with non-NULL values in a specific column. * `COUNT(DISTINCT expr)` counts the number of distinct non-NULL values in a column. `COUNT(DISTINCT expr)` is used for exact count distinct. If you require higher count distinct performance, see [Use bitmap for exact count discount](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_bitmap.md). From StarRocks 2.4 onwards, you can use multiple COUNT(DISTINCT) in one statement. #### Syntax[​](#syntax "Direct link to Syntax") ```haskell COUNT(expr) COUNT(DISTINCT expr [,expr,...])` ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: the column or expression based on which `count()` is performed. If `expr` is a column name, the column can be of any data type. #### Return value[​](#return-value "Direct link to Return value") Returns a numeric value. If no rows can be found, 0 is returned. This function ignores NULL values. #### Examples[​](#examples "Direct link to Examples") Suppose there is a table named `test`. Query the country, category, and supplier of each order by `id`. ```plain select * from test order by id; +------+----------+----------+------------+ | id | country | category | supplier | +------+----------+----------+------------+ | 1001 | US | A | supplier_1 | | 1002 | Thailand | A | supplier_2 | | 1003 | Turkey | B | supplier_3 | | 1004 | US | A | supplier_2 | | 1005 | China | C | supplier_4 | | 1006 | Japan | D | supplier_3 | | 1007 | Japan | NULL | supplier_5 | +------+----------+----------+------------+ ``` Example 1: Count the number of rows in table `test`. ```plain select count(*) from test; +----------+ | count(*) | +----------+ | 7 | +----------+ ``` Example 2: Count the number of values in the `id` column. ```plain select count(id) from test; +-----------+ | count(id) | +-----------+ | 7 | +-----------+ ``` Example 3: Count the number of values in the `category` column while ignoring NULL values. ```plain select count(category) from test; +-----------------+ | count(category) | +-----------------+ | 6 | +-----------------+ ``` Example 4: Count the number of distinct values in the `category` column. ```plain select count(distinct category) from test; +-------------------------+ | count(DISTINCT category) | +-------------------------+ | 4 | +-------------------------+ ``` Example 5: Count the number of combinations that can be formed by `category` and `supplier`. ```plain select count(distinct category, supplier) from test; +------------------------------------+ | count(DISTINCT category, supplier) | +------------------------------------+ | 5 | +------------------------------------+ ``` In the output, the combination with `id` 1004 duplicates with the combination with `id` 1002. They are counted only once. The combination with `id` 1007 has a NULL value and is not counted. Example 6: Use multiple COUNT(DISTINCT) in one statement. ```plain select count(distinct country, category), count(distinct country,supplier) from test; +-----------------------------------+-----------------------------------+ | count(DISTINCT country, category) | count(DISTINCT country, supplier) | +-----------------------------------+-----------------------------------+ | 6 | 7 | +-----------------------------------+-----------------------------------+ ``` --- ### count_if Returns the number of records that meet the specified condition or `0` if no records satisfy the condition. This function does not support `DISTINCT`, for example `count_if(DISTINCT x)`. This function is internally transformed to `COUNT` + `IF`: * Before: `COUNT_IF(x)` * After: `COUNT(IF(x, 1, NULL))` #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext COUNT_IF(condition) ``` #### Parameters[​](#parameters "Direct link to Parameters") `condition`: the condition that should evaluate to a BOOLEAN value. You can specify multiple count\_if conditions in one SELECT statement. #### Return value[​](#return-value "Direct link to Return value") Returns a numeric value. If no records can be found, `0` is returned. This function ignores NULL values. #### Examples[​](#examples "Direct link to Examples") Suppose there is a table named `test_count_if`. ```plain CREATE TABLE `test_count_if` ( `v1` varchar(65533) NULL COMMENT "", `v2` varchar(65533) NULL COMMENT "", `v3` datetime NULL COMMENT "", `v4` int null ) ENGINE=OLAP DUPLICATE KEY(v1, v2, v3) PARTITION BY RANGE(`v3`) (PARTITION p20220418 VALUES [("2022-04-18 00:00:00"), ("2022-04-19 00:00:00")), PARTITION p20220419 VALUES [("2022-04-19 00:00:00"), ("2022-04-20 00:00:00")), PARTITION p20220420 VALUES [("2022-04-20 00:00:00"), ("2022-04-21 00:00:00")), PARTITION p20220421 VALUES [("2022-04-21 00:00:00"), ("2022-04-22 00:00:00"))) DISTRIBUTED BY HASH(`v1`) BUCKETS 4; INSERT INTO test_count_if VALUES ('a','a', '2022-04-18 01:01:00', 1), ('a','b', '2022-04-18 02:01:00', NULL), ('a',NULL, '2022-04-18 02:05:00', 1), ('a','b', '2022-04-18 02:15:00', 3), ('a','b', '2022-04-18 03:15:00', 7), ('c',NULL, '2022-04-18 03:45:00', NULL), ('c',NULL, '2022-04-18 03:25:00', 2), ('c','a', '2022-04-18 03:27:00', 3); SELECT * FROM test_count_if; +------+------+---------------------+------+ | v1 | v2 | v3 | v4 | +------+------+---------------------+------+ | a | NULL | 2022-04-18 02:05:00 | 1 | | a | a | 2022-04-18 01:01:00 | 1 | | a | b | 2022-04-18 02:01:00 | NULL | | a | b | 2022-04-18 02:15:00 | 3 | | a | b | 2022-04-18 03:15:00 | 7 | | c | NULL | 2022-04-18 03:25:00 | 2 | | c | NULL | 2022-04-18 03:45:00 | NULL | | c | a | 2022-04-18 03:27:00 | 3 | +------+------+---------------------+------+ ``` Example 1: Count the number of rows in table `test_count_if` where `v2` is null. ```plain select count_if(v2 is null) from test_count_if; +----------------------+ | count_if(v2 IS NULL) | +----------------------+ | 3 | +----------------------+ ``` Example 2: Count the number of rows where `v1 >= v2 or v4 = 1` ```plain select count_if(v1 >= v2 or v4 = 1)from test_count_if; +----------------------------------+ | count_if((v1 >= v2) OR (v4 = 1)) | +----------------------------------+ | 3 | +----------------------------------+ ``` Example 3: Specify multiple conditions. ```plain select count_if(v1 >= v2), count_if(v1 >= v2 or v4 = 1), count_if(v1 >= v2 and v4 = 1) from test_count_if; +--------------------+----------------------------------+-----------------------------------+ | count_if(v1 >= v2) | count_if((v1 >= v2) OR (v4 = 1)) | count_if((v1 >= v2) AND (v4 = 1)) | +--------------------+----------------------------------+-----------------------------------+ | 2 | 3 | 1 | ``` --- ### covar_pop Returns the population covariance of two expressions. This function is supported from v2.5.10. It can also be used as a window function. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext COVAR_POP(expr1, expr2) ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr1` and `expr2` must evaluate to TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, or DECIMAL. If `expr1` and `expr2` are table columns, this function calculates the population covariance of these two columns. #### Return value[​](#return-value "Direct link to Return value") Returns a DOUBLE value. The formula is as follows, where `n` represents the row count of the table: ![covar\_pop formula](/assets/images/covar_pop_formula-e72ba738bcac2b27294ab3c024c46954.png) #### Usage notes[​](#usage-notes "Direct link to Usage notes") * A data row is counted only when the two columns in this row are non-null values. Otherwise, this data row is eliminated from the result. * NULL is returned in any input is NULL. #### Examples[​](#examples "Direct link to Examples") Suppose table `agg` has the following data: ```plaintext mysql> select * from agg; +------+-------+-------+ | no | k | v | +------+-------+-------+ | 1 | 10.00 | NULL | | 2 | 10.00 | 11.00 | | 2 | 20.00 | 22.00 | | 2 | 25.00 | NULL | | 2 | 30.00 | 35.00 | +------+-------+-------+ ``` Calculate the population covariance of the `k` and `v` columns: ```plaintext mysql> select no,COVAR_POP(k,v) from agg group by no; +------+-------------------+ | no | covar_pop(k, v) | +------+-------------------+ | 1 | NULL | | 2 | 79.99999999999999 | +------+-------------------+ ``` --- ### covar_samp Returns the sample covariance of two expressions. This function is supported from v2.5.10. It can also be used as a window function. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext COVAR_SAMP(expr1, expr2) ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr1` and `expr2` must evaluate to TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, or DECIMAL. If `expr1` and `expr2` are table columns, this function calculates the sample covariance of these two columns. #### Return value[​](#return-value "Direct link to Return value") Returns a DOUBLE value. The formula is as follows, where `n` represents the row count of the table: ![covar\_samp formula](/assets/images/covar_samp_formula-6445511c7b1e75dfa693076fedb9adc0.png) #### Usage notes[​](#usage-notes "Direct link to Usage notes") * A data row is counted only when the two columns in this row are non-null values. Otherwise, this data row is eliminated from the result. * 0 is returned if `n` is 1. * NULL is returned in any input is NULL. #### Examples[​](#examples "Direct link to Examples") Suppose table `agg` has the following data: ```plaintext mysql> select * from agg; +------+-------+-------+ | no | k | v | +------+-------+-------+ | 1 | 10.00 | NULL | | 2 | 10.00 | 11.00 | | 2 | 20.00 | 22.00 | | 2 | 25.00 | NULL | | 2 | 30.00 | 35.00 | +------+-------+-------+ ``` Calculate the sample covariance of the `k` and \`v\`\` columns: ```plaintext mysql> select no,COVAR_SAMP(k,v) from agg group by no; +------+--------------------+ | no | covar_samp(k, v) | +------+--------------------+ | 1 | NULL | | 2 | 119.99999999999999 | +------+--------------------+ ``` --- ### ds_hll_accumulate Accumulates values into a HyperLogLog sketch and returns the serialized sketch as a VARBINARY. This function is part of the DataSketches HLL family of functions for approximate distinct counting. `ds_hll_accumulate` creates a serialized HyperLogLog sketch that can be combined with other sketches using `ds_hll_combine` and estimated using `ds_hll_estimate`. It is based on Apache DataSketches and provides high precision for approximate distinct counting. For more information, see [HyperLogLog Sketches](https://datasketches.apache.org/docs/HLL/HllSketches.html). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext sketch ds_hll_accumulate(expr) sketch ds_hll_accumulate(expr, log_k) sketch ds_hll_accumulate(expr, log_k, tgt_type) ``` ##### Parameters[​](#parameters "Direct link to Parameters") * `expr`: The expression to accumulate into the sketch. Can be any data type. * `log_k`: Integer. Range \[4, 21]. Default: 17. Controls the precision and memory usage of the sketch. * `tgt_type`: Valid values are `HLL_4`, `HLL_6` (default) and `HLL_8`. Controls the target type of the HyperLogLog sketch. #### Return Type[​](#return-type "Direct link to Return Type") Returns a VARBINARY containing the serialized HyperLogLog sketch. #### Examples[​](#examples "Direct link to Examples") ```sql -- Create test table CREATE TABLE t1 ( id BIGINT, province VARCHAR(64), age SMALLINT, dt VARCHAR(10) ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 3; -- Insert test data INSERT INTO t1 SELECT generate_series, generate_series, generate_series % 100, "2024-07-24" FROM table(generate_series(1, 1000)); -- Basic usage with single argument SELECT ds_hll_accumulate(id) FROM t1; -- With custom log_k parameter SELECT ds_hll_accumulate(province, 20) FROM t1; -- With both log_k and tgt_type parameters SELECT ds_hll_accumulate(age, 12, "HLL_6") FROM t1; -- Group by usage SELECT dt, ds_hll_accumulate(id), ds_hll_accumulate(province, 20), ds_hll_accumulate(age, 12, "HLL_6"), ds_hll_accumulate(dt) FROM t1 GROUP BY dt ORDER BY 1 LIMIT 3; ``` #### Related Functions[​](#related-functions "Direct link to Related Functions") * `ds_hll_combine`: Combines multiple serialized sketches into a single sketch * `ds_hll_estimate`: Estimates the distinct count from a serialized sketch * `ds_hll_count_distinct`: Direct approximate distinct counting function #### Keywords[​](#keywords "Direct link to Keywords") ds\_HLL\_ACCUMULATE, HLL, HYPERLOGLOG, APPROXIMATE, DISTINCT, COUNT --- ### ds_hll_combine Combines multiple serialized HyperLogLog sketches into a single serialized sketch. This function is part of the DataSketches HLL family of functions for approximate distinct counting. `ds_hll_combine` takes multiple VARBINARY serialized sketches created by `ds_hll_accumulate` and merges them into a single sketch that represents the union of all distinct values. It is based on Apache DataSketches and provides high precision for approximate distinct counting. For more information, see [HyperLogLog Sketches](https://datasketches.apache.org/docs/HLL/HllSketches.html). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext sketch ds_hll_combine(sketch) ``` ##### Parameters[​](#parameters "Direct link to Parameters") * `sketch`: A VARBINARY column containing serialized HyperLogLog sketches created by `ds_hll_accumulate`. #### Return Type[​](#return-type "Direct link to Return Type") Returns a VARBINARY containing the combined serialized HyperLogLog sketch. #### Examples[​](#examples "Direct link to Examples") ```sql -- Create test table CREATE TABLE t1 ( id BIGINT, province VARCHAR(64), age SMALLINT, dt VARCHAR(10) ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 3; -- Insert test data INSERT INTO t1 SELECT generate_series, generate_series, generate_series % 100, "2024-07-24" FROM table(generate_series(1, 1000)); -- Create table to store sketches CREATE TABLE t2 ( `id` bigint, `dt` varchar(10), `ds_id` binary, `ds_province` binary, `ds_age` binary, `ds_dt` binary ) ENGINE=OLAP DISTRIBUTED BY HASH(id) BUCKETS 3; -- Store sketches created by ds_hll_accumulate INSERT INTO t2 SELECT id, dt, ds_hll_accumulate(id), ds_hll_accumulate(province, 10), ds_hll_accumulate(age, 20, "HLL_6"), ds_hll_accumulate(dt, 10, "HLL_8") FROM t1 GROUP BY id, dt; -- Combine sketches grouped by date SELECT dt, ds_hll_combine(ds_id), ds_hll_combine(ds_province), ds_hll_combine(ds_age), ds_hll_combine(ds_dt) FROM t2 GROUP BY dt ORDER BY 1 LIMIT 3; ``` #### Related Functions[​](#related-functions "Direct link to Related Functions") * `ds_hll_accumulate`: Creates serialized HyperLogLog sketches from data * `ds_hll_estimate`: Estimates the distinct count from a serialized sketch * `ds_hll_count_distinct`: Direct approximate distinct counting function #### Keywords[​](#keywords "Direct link to Keywords") ds\_HLL\_COMBINE, HLL, HYPERLOGLOG, APPROXIMATE, DISTINCT, COUNT, MERGE, UNION --- ### ds_hll_count_distinct Returns the approximate value of aggregate function similar to the result of COUNT(DISTINCT col). APPROX\_COUNT\_DISTINCT(expr) is similar function. ds\_hll\_count\_distinct is faster than the COUNT and DISTINCT combination and uses a fixed-size memory, so less memory is required for columns of high cardinality. It is slower than APPROX\_COUNT\_DISTINCT(expr) but with higher precision because it adopts of Apache Datasketches. For more information, see [HyperLogLog Sketches](https://datasketches.apache.org/docs/HLL/HllSketches.html). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext ds_hll_count_distinct(expr, [log_k], [tgt_type]) ``` * `log_k`: Integer. Range \[4, 21]. Default: 17. * `tgt_type`: Valid values are `HLL_4`, `HLL_6` (default) and `HLL_8`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> CREATE TABLE t1 ( -> id BIGINT NOT NULL, -> province VARCHAR(64), -> age SMALLINT, -> dt VARCHAR(10) NOT NULL -> ) -> DUPLICATE KEY(id) -> DISTRIBUTED BY HASH(id) BUCKETS 4; Query OK, 0 rows affected (0.02 sec) mysql> insert into t1 select generate_series, generate_series, generate_series % 100, "2024-07-24" from table(generate_series(1, 100000)); Query OK, 100000 rows affected (0.29 sec) mysql> select ds_hll_count_distinct(id), ds_hll_count_distinct(province), ds_hll_count_distinct(age), ds_hll_count_distinct(dt) from t1 order by 1, 2; +---------------------------+---------------------------------+----------------------------+---------------------------+ | ds_hll_count_distinct(id) | ds_hll_count_distinct(province) | ds_hll_count_distinct(age) | ds_hll_count_distinct(dt) | +---------------------------+---------------------------------+----------------------------+---------------------------+ | 100090 | 100140 | 100 | 1 | +---------------------------+---------------------------------+----------------------------+---------------------------+ 1 row in set (0.07 sec) mysql> select ds_hll_count_distinct(id, 21), ds_hll_count_distinct(province, 21), ds_hll_count_distinct(age, 21), ds_hll_count_distinct(dt, 21) from t1 order by 1, 2; +-------------------------------+-------------------------------------+--------------------------------+-------------------------------+ | ds_hll_count_distinct(id, 21) | ds_hll_count_distinct(province, 21) | ds_hll_count_distinct(age, 21) | ds_hll_count_distinct(dt, 21) | +-------------------------------+-------------------------------------+--------------------------------+-------------------------------+ | 99995 | 100001 | 100 | 1 | +-------------------------------+-------------------------------------+--------------------------------+-------------------------------+ 1 row in set (0.07 sec) mysql> select ds_hll_count_distinct(id, 10, "HLL_8"), ds_hll_count_distinct(province, 10, "HLL_8"), ds_hll_count_distinct(age, 10, "HLL_8"), ds_hll_count_distinct(dt, 10, "HLL_8") from t1 order by 1, 2; +----------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------------+ | ds_hll_count_distinct(id, 10, 'HLL_8') | ds_hll_count_distinct(province, 10, 'HLL_8') | ds_hll_count_distinct(age, 10, 'HLL_8') | ds_hll_count_distinct(dt, 10, 'HLL_8') | +----------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------------+ | 99844 | 101905 | 96 | 1 | +----------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------------+ 1 row in set (0.09 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") DS\_HLL\_COUNT\_DISTINCT,APPROX\_COUNT\_DISTINCT --- ### ds_hll_estimate Estimates the approximate distinct count from a serialized HyperLogLog sketch. This function is part of the DataSketches HLL family of functions for approximate distinct counting. `ds_hll_estimate` takes a VARBINARY serialized sketch created by `ds_hll_accumulate` or `ds_hll_combine` and returns the estimated number of distinct values. It is based on Apache DataSketches and provides high precision for approximate distinct counting. For more information, see [HyperLogLog Sketches](https://datasketches.apache.org/docs/HLL/HllSketches.html). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext bigint ds_hll_estimate(sketch) ``` ##### Parameters[​](#parameters "Direct link to Parameters") * `sketch`: A VARBINARY column containing serialized HyperLogLog sketches created by `ds_hll_accumulate` or `ds_hll_combine`. #### Return Type[​](#return-type "Direct link to Return Type") Returns a BIGINT representing the estimated distinct count. #### Examples[​](#examples "Direct link to Examples") ```sql -- Create test table CREATE TABLE t1 ( id BIGINT, province VARCHAR(64), age SMALLINT, dt VARCHAR(10) ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 3; -- Insert test data INSERT INTO t1 SELECT generate_series, generate_series, generate_series % 100, "2024-07-24" FROM table(generate_series(1, 1000)); -- Create table to store sketches CREATE TABLE t2 ( `id` bigint, `dt` varchar(10), `ds_id` binary, `ds_province` binary, `ds_age` binary, `ds_dt` binary ) ENGINE=OLAP DISTRIBUTED BY HASH(id) BUCKETS 3; -- Store sketches created by ds_hll_accumulate INSERT INTO t2 SELECT id, dt, ds_hll_accumulate(id), ds_hll_accumulate(province, 10), ds_hll_accumulate(age, 20, "HLL_6"), ds_hll_accumulate(dt, 10, "HLL_8") FROM t1 GROUP BY id, dt; -- Estimate distinct counts grouped by date SELECT dt, ds_hll_estimate(ds_id), ds_hll_estimate(ds_province), ds_hll_estimate(ds_age), ds_hll_estimate(ds_dt) FROM t2 GROUP BY dt ORDER BY 1 LIMIT 3; ``` #### Related Functions[​](#related-functions "Direct link to Related Functions") * `ds_hll_accumulate`: Creates serialized HyperLogLog sketches from data * `ds_hll_combine`: Combines multiple serialized sketches into a single sketch * `ds_hll_count_distinct`: Direct approximate distinct counting function #### Keywords[​](#keywords "Direct link to Keywords") ds\_HLL\_ESTIMATE, HLL, HYPERLOGLOG, APPROXIMATE, DISTINCT, COUNT, ESTIMATE --- ### ds_theta_count_distinct Returns the approximate value of aggregate function similar to the result of `COUNT(DISTINCT col)`. `ds_theta_count_distinct` is faster than `COUNT(DISTINCT col)` and uses less memory for columns of high cardinality. `ds_theta_count_distinct` is similar to `APPROX_COUNT_DISTINCT(expr)` and `DS_HLL_COUNT_DISTINCT(expr)` but with different precisions because it adopts Apache DataSketches. For more information, see [Theta Sketches](https://datasketches.apache.org/docs/Theta/InverseEstimate.html). The relative error is 3.125% (95% confidence). For more information, see the [relative error table](https://datasketches.apache.org/docs/Theta/ThetaErrorTable.html). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT ds_theta_count_distinct(expr) ``` * `expr`: The column in which to calculate the approximate count distinct values. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> CREATE TABLE t1 ( -> id BIGINT NOT NULL, -> province VARCHAR(64), -> age SMALLINT, -> dt VARCHAR(10) NOT NULL -> ) -> DUPLICATE KEY(id) -> DISTRIBUTED BY HASH(id) BUCKETS 4; Query OK, 0 rows affected (0.02 sec) mysql> insert into t1 select generate_series, generate_series, generate_series % 100, "2024-07-24" from table(generate_series(1, 100000)); Query OK, 100000 rows affected (0.29 sec) mysql> select ds_theta_count_distinct(id), ds_theta_count_distinct(province), ds_theta_count_distinct(age), ds_theta_count_distinct(dt) from t1 order by 1, 2; +-----------------------------+-----------------------------------+------------------------------+-----------------------------+ | ds_theta_count_distinct(id) | ds_theta_count_distinct(province) | ds_theta_count_distinct(age) | ds_theta_count_distinct(dt) | +-----------------------------+-----------------------------------+------------------------------+-----------------------------+ | 100215 | 100846 | 100 | 1 | +-----------------------------+-----------------------------------+------------------------------+-----------------------------+ 1 row in set (0.62 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") DS\_THETA\_COUNT\_DISTINCT,DS\_HLL\_COUNT\_DISTINCT,APPROX\_COUNT\_DISTINCT --- ### group_concat Concatenates non-null values from a group into a single string, with a `sep` argument, which is `,` by default if not specified. This function can be used to concatenate values from multiple rows of a column into one string. group\_concat supports DISTINCT and ORDER BY in 3.0 versions later than 3.0.6 and 3.1 versions later than 3.1.3. #### Syntax[​](#syntax "Direct link to Syntax") ```sql VARCHAR GROUP_CONCAT([DISTINCT] expr [,expr ...] [ORDER BY {unsigned_integer | col_name | expr} [ASC | DESC] [,col_name ...]] [SEPARATOR sep]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr`: the values to concatenate, with null values ignored. It must evaluate to VARCHAR. You can optionally specify `DISTINCT` to eliminate duplicate values from the output string. If you want to concatenate multiple `expr` directly, use [concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/concat.md) or [concat\_ws](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/concat_ws.md) to specify formats. * Items in ORDER BY can be unsigned integers (starting from 1), column names, or normal expressions. The results are sorted in ascending order by default. You can also explicitly specify the ASC keyword. If you want to sort results in descending order, add the DESC keyword to the name of the column you are sorting. * `sep`: the optional separator used to concatenate non-null values from different rows. If it is not specified, `,` (a comma) is used by default. To eliminate the separator, specify an empty string `''`. > **NOTE** > > From v3.0.6 and v3.1.3 onwards, there is a behavior change when you specify the separator. You must use `SEPARATOR` to declare the separator, for example, `select group_concat(name SEPARATOR '-') as res from ss;`. #### Return value[​](#return-value "Direct link to Return value") Returns a string value for each group and returns NULL if there are no non-NULL values. You can limit the length of the string returned by group\_concat by setting the [session variable](https://docs.starrocks.io/docs/sql-reference/System_variable.md) `group_concat_max_len`, which defaults to 1024. Minimum value: 4. Unit: characters. Example: ```sql SET [GLOBAL | SESSION] group_concat_max_len = ; ``` #### Examples[​](#examples "Direct link to Examples") 1. Create a table `ss` which contains subject scores. ```sql CREATE TABLE `ss` ( `id` int(11) NULL COMMENT "", `name` varchar(255) NULL COMMENT "", `subject` varchar(255) NULL COMMENT "", `score` int(11) NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`id`) DISTRIBUTED BY HASH(`id`) BUCKETS 4 PROPERTIES ( "replication_num" = "1" ); insert into ss values (1,"Tom","English",90); insert into ss values (1,"Tom","Math",80); insert into ss values (2,"Tom","English",NULL); insert into ss values (2,"Tom",NULL,NULL); insert into ss values (3,"May",NULL,NULL); insert into ss values (3,"Ti","English",98); insert into ss values (4,NULL,NULL,NULL); insert into ss values (NULL,"Ti","Phy",98); select * from ss order by id; +------+------+---------+-------+ | id | name | subject | score | +------+------+---------+-------+ | NULL | Ti | Phy | 98 | | 1 | Tom | English | 90 | | 1 | Tom | Math | 80 | | 2 | Tom | English | NULL | | 2 | Tom | NULL | NULL | | 3 | May | NULL | NULL | | 3 | Ti | English | 98 | | 4 | NULL | NULL | NULL | +------+------+---------+-------+ ``` 2. Use group\_concat. Example 1: Concatenate names into a string with the default separator and with null values ignored. Duplicate names are retained. ```sql select group_concat(name) as res from ss; +---------------------------+ | res | +---------------------------+ | Tom,Tom,Ti,Tom,Tom,May,Ti | +---------------------------+ ``` Example 2: Concatenate names into a string, connected by the separator `-` and with null values ignored. Duplicate names are retained. ```sql select group_concat(name SEPARATOR '-') as res from ss; +---------------------------+ | res | +---------------------------+ | Ti-May-Ti-Tom-Tom-Tom-Tom | +---------------------------+ ``` Example 3: Concatenate distinct names into a string with the default separator and with null values ignored. Duplicate names are removed. ```sql select group_concat(distinct name) as res from ss; +---------------------------+ | res | +---------------------------+ | Ti,May,Tom | +---------------------------+ ``` Example 4: Concatenate the name-subject strings of the same ID in ascending order of `score`. For example, `TomMath` and `TomEnglish` share ID 1 and they are concatenated with a comma in ascending order of `score`. ```sql select id, group_concat(distinct name,subject order by score) as res from ss group by id order by id; +------+--------------------+ | id | res | +------+--------------------+ | NULL | TiPhy | | 1 | TomMath,TomEnglish | | 2 | TomEnglish | | 3 | TiEnglish | | 4 | NULL | +------+--------------------+ ``` Example 5: group\_concat is nested with concat(), which is used to combine `name`, `-`, and `subject` as a string. The strings in the same row are sorted in ascending order of `score`. ```sql select id, group_concat(distinct concat(name, '-',subject) order by score) as res from ss group by id order by id; +------+----------------------+ | id | res | +------+----------------------+ | NULL | Ti-Phy | | 1 | Tom-Math,Tom-English | | 2 | Tom-English | | 3 | Ti-English | | 4 | NULL | +------+----------------------+ ``` Example 6: No matching result is found and NULL is returned. ```sql select group_concat(distinct name) as res from ss where id < 0; +------+ | res | +------+ | NULL | +------+ ``` Example 7: Limit the length of the returned string to six characters. ```sql set group_concat_max_len = 6; select id, group_concat(distinct name,subject order by score) as res from ss group by id order by id; +------+--------+ | id | res | +------+--------+ | NULL | TiPhy | | 1 | TomMat | | 2 | NULL | | 3 | TiEngl | | 4 | NULL | +------+--------+ ``` #### keyword[​](#keyword "Direct link to keyword") GROUP\_CONCAT,CONCAT,ARRAY\_AGG --- ### grouping Indicates whether a column is an aggregate column. If it is an aggregate column, 0 is returned. Otherwise, 1 is returned. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext GROUPING(col_expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") `col_expr`: An expression for a dimension column specified in the expression-list of the ROLLUP, CUBE, or GROUPING SETS expansion of the GROUP BY clause. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select * from t; +------+------+ | k1 | k2 | +------+------+ | NULL | B | | NULL | b | | A | NULL | | A | B | | A | b | | a | NULL | | a | B | | a | b | +------+------+ 8 rows in set (0.12 sec) MySQL > SELECT k1, k2, GROUPING(k1), GROUPING(k2), COUNT(*) FROM t GROUP BY ROLLUP(k1, k2); +------+------+--------------+--------------+----------+ | k1 | k2 | grouping(k1) | grouping(k2) | count(*) | +------+------+--------------+--------------+----------+ | NULL | NULL | 1 | 1 | 8 | | NULL | NULL | 0 | 1 | 2 | | NULL | B | 0 | 0 | 1 | | NULL | b | 0 | 0 | 1 | | A | NULL | 0 | 1 | 3 | | a | NULL | 0 | 1 | 3 | | A | NULL | 0 | 0 | 1 | | A | B | 0 | 0 | 1 | | A | b | 0 | 0 | 1 | | a | NULL | 0 | 0 | 1 | | a | B | 0 | 0 | 1 | | a | b | 0 | 0 | 1 | +------+------+--------------+--------------+----------+ 12 rows in set (0.12 sec) ``` --- ### grouping_id grouping\_id is used to distinguish the grouping statistics results of the same grouping standard. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext GROUPING_ID(expr) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT COL1,GROUPING_ID(COL2) AS 'GroupingID' FROM tbl GROUP BY ROLLUP (COL1, COL2); +------+------------+ | COL1 | GroupingID | +------+------------+ | NULL | 1 | | 2.20 | 1 | | 2.20 | 0 | | 1.10 | 1 | | 1.10 | 0 | +------+------------+ ``` --- ### hll_raw_agg This function is an aggregate function that is used to aggregate HLL fields. It returns an HLL value. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext hll_raw_agg(hll) ``` #### Parameters[​](#parameters "Direct link to Parameters") `hll`: the HLL column that is generated by other columns or based on the loaded data. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the HLL type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select k1, hll_cardinality(hll_raw_agg(v1)) from tbl group by k1; +------+----------------------------------+ | k1 | hll_cardinality(hll_raw_agg(`v1`)) | +------+----------------------------------+ | 2 | 4 | | 1 | 3 | +------+----------------------------------+ ``` --- ### hll_union Returns the concatenation of a set of HLL values. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext hll_union(hll) ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select k1, hll_cardinality(hll_union(v1)) from tbl group by k1; +------+----------------------------------+ | k1 | hll_cardinality(hll_union(`v1`)) | +------+----------------------------------+ | 2 | 4 | | 1 | 3 | +------+----------------------------------+ ``` --- ### hll_union_agg HLL is an engineering implementation based on the HyperLogLog algorithm, which is used to save the intermediate results of HyperLogGog calculation process. It can only be used as the value column of a table and reduce the amount of data through aggregation to achieve the purpose of speeding up the query. An estimated result with an error of about 1% based on HLL. The HLL column is generated by other columns or based on data loaded into the table. During loading, the [hll\_hash](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_hash.md) function is used to specify which column is used to generate the HLL column. It is often used to replace Count Distinct, and to calculate UVs quickly in business by combining rollup. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext HLL_UNION_AGG(hll) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select HLL_UNION_AGG(uv_set) from test_uv; +-------------------------+ | HLL_UNION_AGG(`uv_set`) | +-------------------------+ | 17721 | +-------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") HLL\_UNION\_AGG,HLL,UNION,AGG --- ### mann_whitney_u_test `mann_whitney_u_test` performs the Mann-Whitney rank test on samples derived from two populations. The Mann-Whitney U test is a non-parametric test that can be used to determine if two populations were selected from the same distribution. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext MANN_WHITNEY_U_TEST (sample_data, sample_treatment[, alternative[, continuity_correction]]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `sample_data`: the value of sample data. It must be of numeric data types. * `sample_treatment`: the index of sample data. where each element indicates the treatment group that the corresponding sample belongs to. The values should be of type boolean, with `false` representing the first group and `true` representing the second group. * `alternative` (optional): A const string specifying the alternative hypothesis. It can be one of the following: * * 'two-sided': The default value. Tests whether the means of the two populations are different. * * 'less': Tests whether the mean of the first population is less than the mean of the second population. * * 'greater': Tests whether the mean of the first population is greater than the mean of the second population. * `continuity_correction` (optional): A const boolean value indicating whether to apply a continuity correction. The continuity correction adjusts the U statistic by `0.5` towards the mean of the U distribution, which can improve the accuracy of the test for small sample sizes. The default value is `true`. #### Return value[​](#return-value "Direct link to Return value") The function returns a json array containing the following two elements: The Mann-Whitney U statistic and the p-value associated with the test. #### Usage notes[​](#usage-notes "Direct link to Usage notes") This function ignores NULLs. #### Examples[​](#examples "Direct link to Examples") Suppose there is a table named `testing_data` with the following data. ```sql create table testing_data ( id int, score int, treatment boolean ) properties( "replication_num" = "1" ); insert into testing_data values (1, 80, false), (2, 100, false), (3, NULL, false), (4, 60, true), (5, 70, true), (6, 85, true); ``` ```plain select * from testing_data; +------+-------+-----------+ | id | score | treatment | +------+-------+-----------+ | 1 | 80 | 0 | | 2 | 100 | 0 | | 3 | NULL | 0 | | 4 | 60 | 1 | | 5 | 70 | 1 | | 6 | 85 | 1 | +------+-------+-----------+ ``` Query: ```sql SELECT MANN_WHITNEY_U_TEST(score, treatment) FROM testing_data; ``` Result: ```plain +---------------------------------------+ | mann_whitney_u_test(score, treatment) | +---------------------------------------+ | [5, 0.38647623077123283] | +---------------------------------------+ ``` Query: ```sql SELECT MANN_WHITNEY_U_TEST(score, treatment, 'less') FROM testing_data; ``` Result: ```plain +-----------------------------------------------+ | mann_whitney_u_test(score, treatment, 'less') | +-----------------------------------------------+ | [5, 0.9255426634106172] | +-----------------------------------------------+ ``` Query: ```sql SELECT MANN_WHITNEY_U_TEST(score, treatment, 'two-sided', 0) FROM testing_data; ``` Result: ```plain +-------------------------------------------------------+ | mann_whitney_u_test(score, treatment, 'two-sided', 0) | +-------------------------------------------------------+ | [5, 0.2482130789899235] | +-------------------------------------------------------+ ``` --- ### max Returns the maximum value of the expr expression. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext MAX(expr) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select max(scan_rows) from log_statis group by datetime; +------------------+ | max(`scan_rows`) | +------------------+ | 4671587 | +------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") MAX --- ### max_by Returns the value of `x` associated with the maximum value of `y`. For example, `SELECT max_by(subject, exam_result) FROM exam;` is to return the subject that has the highest exam score. This function is supported from v2.5. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext max_by(x,y) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `x`: an expression of any type. * `y`: an expression of a type that can be ordered. #### Return value[​](#return-value "Direct link to Return value") Returns a value that has the same type as `x`. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * `y` must be a sortable type. If you use an unsortable type of `y`, such as `bitmap` or `hll`, an error is returned. * If `y` contains a null value, the row that corresponds to the null value is ignored. * If more than one value of `x` has the same maximum value of `y`, this function returns the first value of `x` encountered. #### Examples[​](#examples "Direct link to Examples") 1. Create a table `exam`. ```sql CREATE TABLE exam ( subject_id INT, subject STRING, exam_result INT ) DISTRIBUTED BY HASH(`subject_id`); ``` 2. Insert values into this table and query data from this table. ```sql insert into exam values (1,'math',90), (2,'english',70), (3,'physics',95), (4,'chemistry',85), (5,'music',95), (6,'biology',null); select * from exam order by subject_id; +------------+-----------+-------------+ | subject_id | subject | exam_result | +------------+-----------+-------------+ | 1 | math | 90 | | 2 | english | 70 | | 3 | physics | 95 | | 4 | chemistry | 85 | | 5 | music | 95 | | 6 | biology | null | +------------+-----------+-------------+ 6 rows in set (0.03 sec) ``` 3. Obtain the subject that has the highest score. Two subjects `physics` and `music` have the same highest score `95` and the first subject encountered (`physics`) is returned. ```plain SELECT max_by(subject, exam_result) FROM exam; +------------------------------+ | max_by(subject, exam_result) | +------------------------------+ | physics | +------------------------------+ 1 row in set (0.01 sec) ``` --- ### min Returns the minimum value of the expr expression. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext MIN(expr) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select min(scan_rows) from log_statis group by datetime; +------------------+ | min(`scan_rows`) | +------------------+ | 0 | +------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") MIN --- ### min_by Returns the value of `x` associated with the minimum value of `y`. For example, `SELECT min_by(subject, exam_result) FROM exam;` is to return the subject that has the lowest exam score. This function is supported from v2.5. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext min_by(x,y) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `x`: an expression of any type. * `y`: an expression of a type that can be ordered. #### Return value[​](#return-value "Direct link to Return value") Returns a value that has the same type as `x`. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * `y` must be a sortable type. If you use an unsortable type of `y`, such as `bitmap` or `hll`, an error is returned. * If `y` contains a null value, the row that corresponds to the null value is ignored. * If more than one value of `x` has the same minimum value of `y`, this function returns the first value of `x` encountered. #### Examples[​](#examples "Direct link to Examples") 1. Create a table `exam`. ```sql CREATE TABLE exam ( subject_id INT, subject STRING, exam_result INT ) DISTRIBUTED BY HASH(`subject_id`); ``` 2. Insert values into this table and query data from this table. ```sql insert into exam values (1,'math',90), (2,'english',70), (3,'physics',95), (4,'chemistry',85), (5,'music',95), (6,'biology',null); select * from exam order by subject_id; +------------+-----------+-------------+ | subject_id | subject | exam_result | +------------+-----------+-------------+ | 1 | math | 90 | | 2 | english | 70 | | 3 | physics | 95 | | 4 | chemistry | 85 | | 5 | music | 95 | | 6 | biology | null | +------------+-----------+-------------+ 6 rows in set (0.03 sec) ``` 3. Obtain the subject that has the lowest score. The subject `english` that has the lowest score `70` is returned. ```plain SELECT min_by(subject, exam_result) FROM exam; +------------------------------+ | min_by(subject, exam_result) | +------------------------------+ | english | +------------------------------+ 1 row in set (0.01 sec) ``` --- ### MIN_N, MAX_N Returns the `n` smallest or largest values from an expression as an array. These functions are supported from v4.0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext MIN_N(, ) MAX_N(, ) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr`: An expression of any sortable type (STRING, BOOLEAN, DATE, DATETIME, or numeric type). * `n`: An INTEGER literal greater than 0. The maximum value is `10000`. #### Return value[​](#return-value "Direct link to Return value") * `MIN_N`: Returns an ARRAY containing the `n` smallest values from the expression, sorted in ascending order. * `MAX_N`: Returns an ARRAY containing the `n` largest values from the expression, sorted in ascending order. The array elements have the same type as the input expression. * If the number of distinct values is less than `n`, returns all values. * NULL values are ignored. * If all values are NULL, returns an empty array. #### Examples[​](#examples "Direct link to Examples") ##### Basic usage with MIN\_N and MAX\_N[​](#basic-usage-with-min_n-and-max_n "Direct link to Basic usage with MIN_N and MAX_N") 1. Create a table `scores`. ```sql CREATE TABLE scores ( student_id INT, subject STRING, score INT ) DISTRIBUTED BY HASH(`student_id`); ``` 2. Insert values into this table. ```sql INSERT INTO scores VALUES (1, 'math', 85), (2, 'math', 92), (3, 'math', 78), (4, 'math', 95), (5, 'math', 88), (6, 'math', 82), (7, 'math', 90), (8, 'math', 87), (9, 'math', 93), (10, 'math', 89); ``` 3. Get the 3 smallest and 3 largest scores. ```sql SELECT MIN_N(score, 3) AS min_3_scores, MAX_N(score, 3) AS max_3_scores FROM scores; +----------------+----------------+ | min_3_scores | max_3_scores | +----------------+----------------+ | [85,82,78] | [90,92,95] | +----------------+----------------+ ``` 4. Get the smallest and largest scores by subject. ```sql SELECT subject, MIN_N(score, 2) AS min_2_scores, MAX_N(score, 2) AS max_2_scores FROM scores GROUP BY subject; +---------+---------------+---------------+ | subject | min_2_scores | max_2_scores | +---------+---------------+---------------+ | math | [82,78] | [92,95] | +---------+---------------+---------------+ ``` ##### Using with string values[​](#using-with-string-values "Direct link to Using with string values") ```sql CREATE TABLE products ( id INT, name STRING, price DECIMAL(10,2) ) DISTRIBUTED BY HASH(`id`); INSERT INTO products VALUES (1, 'apple', 2.50), (2, 'banana', 1.80), (3, 'cherry', 4.20), (4, 'date', 3.10), (5, 'elderberry', 5.50); SELECT MIN_N(name, 3) AS min_3_names, MAX_N(name, 3) AS max_3_names FROM products; +---------------------------+--------------------------+ | min_3_names | max_3_names | +---------------------------+--------------------------+ | [cherry,banana,apple] | [cherry,date,elderberry] | +---------------------------+--------------------------+ ``` ##### Edge case: n larger than number of values[​](#edge-case-n-larger-than-number-of-values "Direct link to Edge case: n larger than number of values") ```sql SELECT MIN_N(score, 15) AS all_scores_min, MAX_N(score, 15) AS all_scores_max FROM scores; +----------------------------------+----------------------------------+ | all_scores_min | all_scores_max | +----------------------------------+----------------------------------+ | [95,93,92,90,89,88,87,85,82,78] | [78,82,85,87,88,89,90,92,93,95] | +----------------------------------+----------------------------------+ ``` ##### Using with date values[​](#using-with-date-values "Direct link to Using with date values") ```sql CREATE TABLE events ( id INT, event_name STRING, event_date DATE ) DISTRIBUTED BY HASH(`id`); INSERT INTO events VALUES (1, 'conference', '2024-03-15'), (2, 'workshop', '2024-01-20'), (3, 'seminar', '2024-05-10'), (4, 'meeting', '2024-02-28'), (5, 'training', '2024-04-05'); SELECT MIN_N(event_date, 3) AS earliest_3_dates, MAX_N(event_date, 3) AS latest_3_dates FROM events; +------------------------------------+------------------------------------+ | earliest_3_dates | latest_3_dates | +------------------------------------+------------------------------------+ | [2024-03-15,2024-02-28,2024-01-20] | [2024-03-15,2024-04-05,2024-05-10] | +------------------------------------+------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") MIN\_N, MAX\_N --- ### multi_distinct_count Returns the total number of rows of the `expr`, equivalent to count(distinct expr). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext multi_distinct_count(expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: the column or expression based on which `multi_distinct_count()` is performed. If `expr` is a column name, the column can be of any data type. #### Return value[​](#return-value "Direct link to Return value") Returns a numeric value. If no rows can be found, 0 is returned. This function ignores NULL values. #### Examples[​](#examples "Direct link to Examples") Suppose there is a table named `test`. Query the category and supplier of each order by `id`. ```plain select * from test order by id; +------+----------+----------+------------+ | id | country | category | supplier | +------+----------+----------+------------+ | 1001 | US | A | supplier_1 | | 1002 | Thailand | A | supplier_2 | | 1003 | Turkey | B | supplier_3 | | 1004 | US | A | supplier_2 | | 1005 | China | C | supplier_4 | | 1006 | Japan | D | supplier_3 | | 1007 | Japan | NULL | supplier_5 | +------+----------+----------+------------+ ``` Example 1: Count the number of distinct values in the `category` column. ```plain select multi_distinct_count(category) from test; +--------------------------------+ | multi_distinct_count(category) | +--------------------------------+ | 4 | +--------------------------------+ ``` Example 2: Count the number of distinct values in the `supplier` column. ```plain select multi_distinct_count(supplier) from test; +--------------------------------+ | multi_distinct_count(supplier) | +--------------------------------+ | 5 | +--------------------------------+ ``` --- ### multi_distinct_sum Returns the sum of distinct values in `expr`, equivalent to sum(distinct expr). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext multi_distinct_sum(expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr`: the columns involved in the calculation. Column values can be of the following types: TINYINT, SMALLINT, INT, LARGEINT, FLOAT, DOUBLE, or DECIMAL. #### Return value[​](#return-value "Direct link to Return value") The mapping between column values and return value types is as follows: * BOOLEAN -> BIGINT * TINYINT -> BIGINT * SMALLINT -> BIGINT * INT -> BIGINT * BIGINT -> BIGINT * FLOAT -> DOUBLE * DOUBLE -> DOUBLE * LARGEINT -> LARGEINT * DECIMALV2 -> DECIMALV2 * DECIMAL32 -> DECIMAL128 * DECIMAL64 -> DECIMAL128 * DECIMAL128 -> DECIMAL128 #### Examples[​](#examples "Direct link to Examples") 1. Create a table with `k0` as the INT field. ```sql CREATE TABLE tabl (k0 BIGINT NOT NULL) ENGINE=OLAP DUPLICATE KEY(`k0`) COMMENT "OLAP" DISTRIBUTED BY HASH(`k0`) PROPERTIES( "replication_num" = "3", "storage_format" = "DEFAULT" ); ``` 2. Insert values into the table. ```sql -- INSERT INTO tabl VALUES ('0'), ('1'), ('1'), ('1'), ('2'), ('2'); ``` 3. Use multi\_distinct\_sum() to calculate the sum of distinct values in the `k0` column. ```plain MySQL > select multi_distinct_sum(k0) from tabl; +------------------------+ | multi_distinct_sum(k0) | +------------------------+ | 3 | +------------------------+ 1 row in set (0.03 sec) ``` The distinct values of `k0` is 0, 1, 2 and we can get 3 after adding them together. --- ### percentile_approx Returns the approximate value for a given percentile p, or an array of values for corresponding percentiles if p is an array. All percentile values must be in the range \[0,1]. Compression parameter is optional and has a setting range of \[2048, 10000]. The larger the value, the higher the accuracy, the larger the memory consumption, and the longer the calculation time. If it is not specified or not beyond the range of \[2048, 10000], the function runs with a default compression parameter of 10000. This function uses fixed size memory, so less memory can be used for columns with high cardinality, and can be used to calculate statistics such as tp99. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DOUBLE PERCENTILE_APPROX(expr, DOUBLE|ARRAY p[, DOUBLE compression]) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select `table`, percentile_approx(cost_time,0.99) from log_statis group by `table`; +----------+--------------------------------------+ | table | percentile_approx(`cost_time`, 0.99) | +----------+--------------------------------------+ | test | 54.22 | +----------+--------------------------------------+ MySQL > select `table`, percentile_approx(cost_time,0.99, 4096) from log_statis group by `table`; +----------+----------------------------------------------+ | table | percentile_approx(`cost_time`, 0.99, 4096.0) | +----------+----------------------------------------------+ | test | 54.21 | +----------+----------------------------------------------+ MySQL > select percentile_approx(c2, [0.1, 0.5, 0.9], 10000) from t1; +-----------------------------------------------+ | percentile_approx(c2, [0.1, 0.5, 0.9], 10000) | +-----------------------------------------------+ | [4999.6005859375,25000,45000.3984375] | +-----------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") PERCENTILE\_APPROX,PERCENTILE,APPROX --- ### percentile_approx_weight Returns the approximation of the p-th percentile with weight. The value of p is between 0 and 1, and it can be a single value or an array. PERCENTILE\_APPROX\_WEIGHT is a weighted version of PERCENTILE\_APPROX that allows users to specify a weight (a constant value or numeric column) for each input value. This function uses fixed-size memory, so less memory can be used for columns with high cardinality. It also can be used to calculate statistics such as tp99. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DOUBLE PERCENTILE_APPROX_WEIGHT(expr, BIGINT weight, DOUBLE|ARRAY` where each element is in the range \[0, 1]. For example, 0.99 represents the 99th percentile. * `weight` : Weight column. It must be a positive constant number or column. * `compression` : (Optional) Compression ratio. Range: \[2048, 10000]. The larger the value, the higher the precision, the larger the memory consumption, and the longer the calculation time. If this parameter is not specified or the value is beyond the range of \[2048, 10000], the default value `10000` is used. #### Examples[​](#examples "Direct link to Examples") ```plain CREATE TABLE t1 ( c1 int, c2 double, c3 tinyint, c4 int, c5 bigint, c6 largeint, c7 string, c8 double, c9 date, c10 datetime, c11 array, c12 map, c13 struct ) DUPLICATE KEY(c1) DISTRIBUTED BY HASH(c1) BUCKETS 1 PROPERTIES ("replication_num" = "1"); insert into t1 select generate_series, generate_series, 11, 111, 1111, 11111, "111111", 1.1, "2024-09-01", "2024-09-01 18:00:00", [1, 2, 3], map(1, 5.5), row(100, 100) from table(generate_series(1, 50000, 3)); -- use constant value as weight mysql> select percentile_approx_weighted(c1, 1, 0.9) from t1; +----------------------------------------+ | percentile_approx_weighted(c1, 1, 0.9) | +----------------------------------------+ | 45000.3984375 | +----------------------------------------+ 1 row in set (0.07 sec) -- use a numeric column as weight mysql> select percentile_approx_weighted(c2, c1, 0.5) from t1; +-----------------------------------------+ | percentile_approx_weighted(c2, c1, 0.5) | +-----------------------------------------+ | 35355.97265625 | +-----------------------------------------+ 1 row in set (0.07 sec) -- use weight and compression to compute percentile mysql> select percentile_approx_weighted(c2, c1, 0.5, 10000) from t1; +------------------------------------------------+ | percentile_approx_weighted(c2, c1, 0.5, 10000) | +------------------------------------------------+ | 35355.97265625 | +------------------------------------------------+ 1 row in set (0.09 sec) mysql> select percentile_approx_weighted(c2, c1, [0.1, 0.5, 0.9], 10000) from t1; +------------------------------------------------------------+ | percentile_approx_weighted(c2, c1, [0.1, 0.5, 0.9], 10000) | +------------------------------------------------------------+ | [15811.6708984375,35355.97265625,47435.01171875] | +------------------------------------------------------------+ 1 row in set (0.03 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") PERCENTILE\_APPROX\_WEIGHT,PERCENTILE\_APPROX,PERCENTILE,APPROX --- ### percentile_cont Computes the percentile value of `expr` with linear interpolation. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext PERCENTILE_CONT (expr, percentile) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr`: the expression by which to order the values. It must be of numeric data types, DATE, or DATETIME. For example, if you want to find the median score of physics, specify the column that contains the physics scores. * `percentile`: the percentile of the value you want to find. It is a constant floating-point number from 0 to 1. For example, if you want to find the median value, set this parameter to `0.5`. #### Return value[​](#return-value "Direct link to Return value") Returns a value that is at the specified percentile. If no input value lies exactly at the desired percentile, the result is calculated using linear interpolation of the two nearest input values. The data type is the same as `expr`. #### Usage notes[​](#usage-notes "Direct link to Usage notes") This function ignores NULLs. #### Examples[​](#examples "Direct link to Examples") Suppose there is a table named `exam` with the following data. ```plain select * from exam order by Subject; +-----------+-------+ | Subject | Score | +-----------+-------+ | chemistry | 80 | | chemistry | 100 | | chemistry | NULL | | math | 60 | | math | 70 | | math | 85 | | physics | 75 | | physics | 80 | | physics | 85 | | physics | 99 | +-----------+-------+ ``` Calculate the median score of each subject while ignoring NULLs. Query: ```sql SELECT Subject, PERCENTILE_CONT (Score, 0.5) FROM exam group by Subject; ``` Result: ```plain +-----------+-----------------------------+ | Subject | percentile_cont(Score, 0.5) | +-----------+-----------------------------+ | chemistry | 90 | | math | 70 | | physics | 82.5 | +-----------+-----------------------------+ ``` --- ### percentile_disc Returns a percentile value based on a discrete distribution of the input column `expr`. If the exact percentile value cannot be found, this function returns the larger value between the two closest values. This function is supported from v2.5 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```sql PERCENTILE_DISC (expr, percentile) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr`: the column for which you want to calculate the percentile value. The column can be of any data type that is sortable. * `percentile`: the percentile of the value you want to find. It must be a constant floating-point number between 0 and 1. For example, if you want to find the median value, set this parameter to `0.5`. If you want to find the value at the 70th percentile, specify 0.7. #### Return value[​](#return-value "Direct link to Return value") The data type of the return value is the same as `expr`. #### Usage notes[​](#usage-notes "Direct link to Usage notes") NULL values are ignored in the calculation. #### Examples[​](#examples "Direct link to Examples") Create table `exam` and insert data into this table. ```sql CREATE TABLE exam ( subject STRING, score INT ) DISTRIBUTED BY HASH(`subject`); INSERT INTO exam VALUES ('chemistry',80), ('chemistry',100), ('chemistry',null), ('math',60), ('math',70), ('math',85), ('physics',75), ('physics',80), ('physics',85), ('physics',99); ``` ```plain select * from exam order by subject; +-----------+-------+ | subject | score | +-----------+-------+ | chemistry | 80 | | chemistry | 100 | | chemistry | NULL | | math | 60 | | math | 70 | | math | 85 | | physics | 75 | | physics | 80 | | physics | 85 | | physics | 99 | +-----------+-------+ ``` Calculate the median of each subject. ```sql select subject, percentile_disc(score, 0.5) from exam group by subject; ``` Output ```plain +-----------+-----------------------------+ | subject | percentile_disc(score, 0.5) | +-----------+-----------------------------+ | chemistry | 100 | | math | 70 | | physics | 85 | +-----------+-----------------------------+ ``` --- ### percentile_disc_lc Returns a percentile value based on a discrete distribution of the input column `expr`. Same behavior as percentile\_disc. However, the implementation algorithm is different. percentile\_disc needs to obtain all input data, and the memory consumed by merge sorting to obtain percentile values ​​is the memory of all input data. On the other hand, percentile\_disc\_lc builds a hash table of key->count, so when the input cardinality is low, there is no obvious memory increase even if the input data size is large. This function is supported from v3.4 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```sql PERCENTILE_DISC_LC (expr, percentile) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr`: the column for which you want to calculate the percentile value. The column can be of any data type that is sortable. * `percentile`: the percentile of the value you want to find. It must be a constant floating-point number between 0 and 1. For example, if you want to find the median value, set this parameter to `0.5`. If you want to find the value at the 70th percentile, specify 0.7. #### Return value[​](#return-value "Direct link to Return value") The data type of the return value is the same as `expr`. #### Usage notes[​](#usage-notes "Direct link to Usage notes") NULL values are ignored in the calculation. #### Examples[​](#examples "Direct link to Examples") Create table `exam` and insert data into this table. ```sql CREATE TABLE exam ( subject STRING, score INT ) DISTRIBUTED BY HASH(`subject`); INSERT INTO exam VALUES ('chemistry',80), ('chemistry',100), ('chemistry',null), ('math',60), ('math',70), ('math',85), ('physics',75), ('physics',80), ('physics',85), ('physics',99); ``` ```plain select * from exam order by subject; +-----------+-------+ | subject | score | +-----------+-------+ | chemistry | 80 | | chemistry | 100 | | chemistry | NULL | | math | 60 | | math | 70 | | math | 85 | | physics | 75 | | physics | 80 | | physics | 85 | | physics | 99 | +-----------+-------+ ``` Calculate the median of each subject. ```sql select subject, percentile_disc_lc (score, 0.5) from exam group by subject; ``` Output ```plain +-----------+--------------------------------+ | subject | percentile_disc_lc(score, 0.5) | +-----------+--------------------------------+ | physics | 85 | | chemistry | 100 | | math | 70 | +-----------+--------------------------------+ ``` --- ### retention Calculates the user retention rate within a specified period of time. This function accepts 1 to 31 conditions and evaluates whether each condition is true. If the condition evaluates to true, 1 is returned. Otherwise, 0 is returned. It eventually returns an array of 0 and 1. You can calculate the user retention rate based on this data. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext ARRAY retention(ARRAY input) ``` #### Parameters[​](#parameters "Direct link to Parameters") `input`: an array of conditions. A maximum of 31 conditions can be passed in. Separate multiple conditions with commas. #### Return value[​](#return-value "Direct link to Return value") Returns an array of 0 and 1. The number of 0 and 1 is the same as the number of input conditions. The evaluation starts from the first condition. * If the condition evaluates to true, 1 is returned. Otherwise, 0 is returned. * If the first condition is not true, the current position and its following positions are all set to 0. #### Examples[​](#examples "Direct link to Examples") 1. Create a table named `test` and insert data. ```sql CREATE TABLE test( id TINYINT, action STRING, time DATETIME ) ENGINE=olap DUPLICATE KEY(id) DISTRIBUTED BY HASH(id); INSERT INTO test VALUES (1,'pv','2022-01-01 08:00:05'), (2,'pv','2022-01-01 10:20:08'), (1,'buy','2022-01-02 15:30:10'), (2,'pv','2022-01-02 17:30:05'), (3,'buy','2022-01-01 05:30:09'), (3,'buy','22022-01-02 08:10:15'), (4,'pv','2022-01-02 21:09:15'), (5,'pv','2022-01-01 22:10:53'), (5,'pv','2022-01-02 19:10:52'), (5,'buy','2022-01-02 20:00:50'); ``` 2. Query data from `test`. ```plain MySQL > select * from test order by id; +------+--------+---------------------+ | id | action | time | +------+--------+---------------------+ | 1 | pv | 2022-01-01 08:00:05 | | 1 | buy | 2022-01-02 15:30:10 | | 2 | pv | 2022-01-01 10:20:08 | | 2 | pv | 2022-01-02 17:30:05 | | 3 | buy | 2022-01-01 05:30:09 | | 3 | buy | 2022-01-02 08:10:15 | | 4 | pv | 2022-01-02 21:09:15 | | 5 | pv | 2022-01-01 22:10:53 | | 5 | pv | 2022-01-02 19:10:52 | | 5 | buy | 2022-01-02 20:00:50 | +------+--------+---------------------+ 10 rows in set (0.01 sec) ``` 3. Use `retention` to calculate user retention rate. Example 1: Evaluate user behavior against the following conditions: view commodity page on 2022-01-01 (action='pv') and place an order on 2022-01-02 (action='buy'). ```plain MySQL > select id, retention([action='pv' and to_date(time)='2022-01-01', action='buy' and to_date(time)='2022-01-02']) as retention from test group by id order by id; +------+-----------+ | id | retention | +------+-----------+ | 1 | [1,1] | | 2 | [1,0] | | 3 | [0,0] | | 4 | [0,0] | | 5 | [1,1] | +------+-----------+ 5 rows in set (0.01 sec) ``` In the result: * Users 1 and 5 meets two conditions and \[1,1] is returned. * User 2 does not meet the second condition and \[1,0] is returned. * User 3 meets the second condition but does not meet the first condition. \[0,0] is returned. * User 4 meets no condition and \[0,0] is returned. Example 2: Calculate the percentage of users who have viewed commodity page on 2022-01-01 (action='pv') and placed an order on 2022-01-02 (action='buy'). ```plain MySQL > select sum(r[1]),sum(r[2])/sum(r[1]) from (select id, retention([action='pv' and to_date(time)='2022-01-01', action='buy' and to_date(time)='2022-01-02']) as r from test group by id order by id) t; +-----------+---------------------------+ | sum(r[1]) | (sum(r[2])) / (sum(r[1])) | +-----------+---------------------------+ | 3 | 0.6666666666666666 | +-----------+---------------------------+ 1 row in set (0.02 sec) ``` The return value is the user retention rate on 2022-01-02. #### keyword[​](#keyword "Direct link to keyword") retention, retention rate, RETENTION --- ### std Returns the standard deviation of an expression. Since v2.5.10, this function can also be used as a window function. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext STD(expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: the expression. If it is a table column, it must evaluate to TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, or DECIMAL. #### Return value[​](#return-value "Direct link to Return value") Returns a DOUBLE value. #### Examples[​](#examples "Direct link to Examples") ```plaintext MySQL > select * from std_test; +------+------+ | col0 | col1 | +------+------+ | 0 | 0 | | 1 | 2 | | 2 | 4 | | 3 | 6 | | 4 | 8 | +------+------+ ``` Calculate the standard deviation of `col0` and `col1`. ```plaintext MySQL > select std(col0) as std_of_col0, std(col1) as std_of_col1 from std_test; +--------------------+--------------------+ | std_of_col0 | std_of_col1 | +--------------------+--------------------+ | 1.4142135623730951 | 2.8284271247461903 | +--------------------+--------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") STD --- ### stddev,stddev_pop,std Returns the population standard deviation of the expr expression. Since v2.5.10, this function can also be used as a window function. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext STDDEV(expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: the expression. If it is a table column, it must evaluate to TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, or DECIMAL. #### Return value[​](#return-value "Direct link to Return value") Returns a DOUBLE value. The formula is as follows, where `n` represents the row count of the table: ![image](/assets/images/stddevpop_formula-e118dac264cc847c6e24c853d2eefd50.png) #### Examples[​](#examples "Direct link to Examples") ```plaintext mysql> SELECT stddev(lo_quantity), stddev_pop(lo_quantity) from lineorder; +---------------------+-------------------------+ | stddev(lo_quantity) | stddev_pop(lo_quantity) | +---------------------+-------------------------+ | 14.43100708360797 | 14.43100708360797 | +---------------------+-------------------------+ ``` #### See also[​](#see-also "Direct link to See also") [stddev\_samp](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/stddev_samp.md) #### keyword[​](#keyword "Direct link to keyword") STDDEV,STDDEV\_POP,POP --- ### stddev_samp Returns the sample standard deviation of an expression. Since v2.5.10, this function can also be used as a window function. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext STDDEV_SAMP(expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: the expression. If it is a table column, it must evaluate to TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, or DECIMAL. #### Return value[​](#return-value "Direct link to Return value") Returns a DOUBLE value. The formula is as follows, where `n` represents the row count of the table: ![image](/assets/images/stddevsamp_formula-4f1800ef28c407fb3148a3c2cae01e6f.png) #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select stddev_samp(scan_rows) from log_statis group by datetime; +--------------------------+ | stddev_samp(`scan_rows`) | +--------------------------+ | 2.372044195280762 | +--------------------------+ ``` #### See also[​](#see-also "Direct link to See also") [stddev](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/stddev.md) #### keyword[​](#keyword "Direct link to keyword") STDDEV\_SAMP,STDDEV,SAMP --- ### sum Returns the sum of non-null values for `expr`. You can use the DISTINCT keyword to compute the sum of distinct non-null values. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext SUM([DISTINCT] expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: the expression that evaluates to a numeric value. Supported data types are TINYINT, SMALLINT, INT, FLOAT, DOUBLE, or DECIMAL. #### Return value[​](#return-value "Direct link to Return value") Data type mapping between input value and return value: * TINYINT -> BIGINT * SMALLINT -> BIGINT * INT -> BIGINT * FLOAT -> DOUBLE * DOUBLE -> DOUBLE * DECIMAL -> DECIMAL #### Usage notes[​](#usage-notes "Direct link to Usage notes") * This function ignores nulls. * An error is returned if `expr` does not exist. * If a VARCHAR expression is passed, this function implicitly casts the input into DOUBLE values. If the cast fails, an error is returned. #### Examples[​](#examples "Direct link to Examples") 1. Create a table named `employees`. ```sql CREATE TABLE IF NOT EXISTS employees ( region_num TINYINT COMMENT "range [-128, 127]", id BIGINT COMMENT "range [-2^63 + 1 ~ 2^63 - 1]", hobby STRING NOT NULL COMMENT "upper limit value 65533 bytes", income DOUBLE COMMENT "8 bytes", sales DECIMAL(12,4) COMMENT "" ) DISTRIBUTED BY HASH(region_num); ``` 2. Insert data into `employees`. ```sql INSERT INTO employees VALUES (3,432175,'3',25600,1250.23), (4,567832,'3',37932,2564.33), (3,777326,'2',null,1932.99), (5,342611,'6',43727,45235.1), (2,403882,'4',36789,52872.4); ``` 3. Query data from `employees`. ```plain MySQL > select * from employees; +------------+--------+-------+--------+------------+ | region_num | id | hobby | income | sales | +------------+--------+-------+--------+------------+ | 5 | 342611 | 6 | 43727 | 45235.1000 | | 2 | 403882 | 4 | 36789 | 52872.4000 | | 4 | 567832 | 3 | 37932 | 2564.3300 | | 3 | 432175 | 3 | 25600 | 1250.2300 | | 3 | 777326 | 2 | NULL | 1932.9900 | +------------+--------+-------+--------+------------+ 5 rows in set (0.01 sec) ``` 4. Use this function to compute sum. Example 1: Calculate the total sales of each region. ```plain MySQL > SELECT region_num, sum(sales) from employees group by region_num; +------------+------------+ | region_num | sum(sales) | +------------+------------+ | 2 | 52872.4000 | | 5 | 45235.1000 | | 4 | 2564.3300 | | 3 | 3183.2200 | +------------+------------+ 4 rows in set (0.01 sec) ``` Example 2: Calculate the total employee income of each region. This function ignores nulls and the income of employee id `777326` is not counted. ```plain MySQL > select region_num, sum(income) from employees group by region_num; +------------+-------------+ | region_num | sum(income) | +------------+-------------+ | 2 | 36789 | | 5 | 43727 | | 4 | 37932 | | 3 | 25600 | +------------+-------------+ 4 rows in set (0.01 sec) ``` Example 3: Calculate the total number of hobbies. The `hobby` column is of the STRING type and will be implicitly converted to DOUBLE during computation. ```plain MySQL > select sum(DISTINCT hobby) from employees; +---------------------+ | sum(DISTINCT hobby) | +---------------------+ | 15 | +---------------------+ 1 row in set (0.01 sec) ``` Example 4: Use `sum` with the WHERE clause to calculate the total income of employees whose monthly income is higher than 30000. ```plain MySQL > select sum(income) from employees WHERE income > 30000; +-------------+ | sum(income) | +-------------+ | 118448 | +-------------+ 1 row in set (0.00 sec) ``` #### keyword[​](#keyword "Direct link to keyword") SUM, sum --- ### sum_map #### Description[​](#description "Direct link to Description") Aggregates MAP values by summing numeric values for matching keys across multiple rows. This function is inspired by ClickHouse's sumMap and is particularly useful for data aggregation scenarios involving maps with numeric values. Given multiple maps with the same key type and numeric value type, `sum_map` merges them by: 1. Collecting all unique keys from all input maps 2. Summing the values for each key across all maps 3. Returning a single map with the aggregated results #### Syntax[​](#syntax "Direct link to Syntax") ```sql sum_map(map_expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `map_expr`: A MAP expression where the value type must be numeric (TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, or DOUBLE). Both keys and values can be nullable. #### Return value[​](#return-value "Direct link to Return value") Returns a MAP with the same key and value types as the input, containing all unique keys and their summed values. #### NULL Handling[​](#null-handling "Direct link to NULL Handling") The `sum_map` function handles NULL keys and values according to the following rules: * **NULL values**: Entries with NULL values are **excluded** from the aggregation, regardless of whether the key is NULL or not. * **NULL keys**: Entries with NULL keys but **non-NULL values** are **included** in the result as `{NULL: aggregated_value}`. * **NULL maps**: NULL map inputs (i.e., the entire map is NULL) are skipped during aggregation. **Example behavior:** ```sql -- Map entries: {1:10, NULL:5, 2:NULL} -- After aggregation: -- - {1:10} is included -- - {NULL:5} is included (NULL key with non-NULL value) -- - {2:NULL} is excluded (NULL value) -- Result: {1:10, NULL:5} ``` #### Examples[​](#examples "Direct link to Examples") ##### Example 1: Basic usage with integer maps[​](#example-1-basic-usage-with-integer-maps "Direct link to Example 1: Basic usage with integer maps") ```sql CREATE TABLE metrics ( id INT, counters MAP ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 3; INSERT INTO metrics VALUES (1, map{"clicks":100, "views":200}), (2, map{"clicks":50, "impressions":150}), (3, map{"views":250, "impressions":350}); SELECT sum_map(counters) FROM metrics; -- Result: {"clicks":150, "views":450, "impressions":500} ``` ##### Example 2: Group by aggregation[​](#example-2-group-by-aggregation "Direct link to Example 2: Group by aggregation") ```sql CREATE TABLE daily_stats ( date DATE, category VARCHAR(20), metrics MAP ) DUPLICATE KEY(date) DISTRIBUTED BY HASH(date) BUCKETS 3; INSERT INTO daily_stats VALUES ('2024-01-01', 'A', map{"revenue":100.5, "cost":50.2}), ('2024-01-01', 'B', map{"revenue":200.3, "cost":80.1}), ('2024-01-02', 'A', map{"revenue":150.7, "cost":60.3}); SELECT date, sum_map(metrics) as total_metrics FROM daily_stats GROUP BY date ORDER BY date; -- Result: -- 2024-01-01 {"revenue":300.8, "cost":130.3} -- 2024-01-02 {"revenue":150.7, "cost":60.3} ``` ##### Example 3: NULL key and value handling[​](#example-3-null-key-and-value-handling "Direct link to Example 3: NULL key and value handling") ```sql -- Create a table with nullable map keys and values CREATE TABLE events ( id INT, tags MAP ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 3; -- Insert data with NULL keys and NULL values INSERT INTO events VALUES (1, map{"user":100, "system":50}), (2, map{"user":25, NULL:30}), -- NULL key with non-NULL value (3, map{"system":40, "admin":NULL}), -- NULL value (will be excluded) (4, map{NULL:15}); -- NULL key with non-NULL value SELECT sum_map(tags) FROM events; -- Result: {"user":125, "system":90, NULL:45} -- Explanation: -- - "user": 100 + 25 = 125 -- - "system": 50 + 40 = 90 -- - NULL key: 30 + 15 = 45 -- - "admin":NULL was excluded (NULL value) ``` ##### Example 4: NULL map handling[​](#example-4-null-map-handling "Direct link to Example 4: NULL map handling") ```sql SELECT sum_map(counters) FROM ( VALUES (map{1:10, 2:20}), (map{1:5, 3:15}), (NULL) -- Entire map is NULL (ignored) ) AS t(counters); -- Result: {1:15, 2:20, 3:15} -- NULL maps are ignored in aggregation ``` #### Usage notes[​](#usage-notes "Direct link to Usage notes") * **Empty maps**: Empty maps contribute nothing to the aggregation * **NULL maps**: NULL map inputs (entire map is NULL) are skipped during aggregation * **NULL keys**: Entries with NULL keys and non-NULL values are included and aggregated together under a single NULL key in the result * **NULL values**: Entries with NULL values are excluded from the result, regardless of whether the key is NULL or not * **Key merging**: Keys that appear in multiple maps have their values summed; keys that appear in only some maps are included with the sum of values from maps where they appear * **Supported types**: The function only supports maps with numeric value types (TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE) * **Use cases**: This function is particularly useful for aggregating counters, metrics, or any key-value data where values should be summed #### Keywords[​](#keywords "Direct link to Keywords") SUM\_MAP, MAP, AGGREGATE --- ### var_samp,variance_samp Returns the sample variance of an expression. Since v2.5.10, this function can also be used as a window function. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VAR_SAMP(expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: the expression. If it is a table column, it must evaluate to TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, or DECIMAL. #### Return value[​](#return-value "Direct link to Return value") Returns a DOUBLE value. #### Examples[​](#examples "Direct link to Examples") ```plaintext MySQL > select var_samp(scan_rows) from log_statis group by datetime; +-----------------------+ | var_samp(`scan_rows`) | +-----------------------+ | 5.6227132145741789 | +-----------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") VAR\_SAMP,VARIANCE\_SAMP,VAR,SAMP,VARIANCE --- ### variance,var_pop,variance_pop Returns the population variance of an expression. Since v2.5.10, this function can also be used as a window function. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARIANCE(expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: the expression. If it is a table column, it must evaluate to TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, or DECIMAL. #### Return value[​](#return-value "Direct link to Return value") Returns a DOUBLE value. #### Examples[​](#examples "Direct link to Examples") ```plaintext MySQL > select var_pop(i_current_price), i_rec_start_date from item group by i_rec_start_date; +--------------------------+------------------+ | var_pop(i_current_price) | i_rec_start_date | +--------------------------+------------------+ | 314.96177792808226 | 1997-10-27 | | 463.73633459357285 | NULL | | 302.02102643609123 | 1999-10-28 | | 337.9318386924913 | 2000-10-27 | | 333.80931439318346 | 2001-10-27 | +--------------------------+------------------+ MySQL > select variance(i_current_price), i_rec_start_date from item group by i_rec_start_date; +---------------------------+------------------+ | variance(i_current_price) | i_rec_start_date | +---------------------------+------------------+ | 314.96177792808226 | 1997-10-27 | | 463.7363345935729 | NULL | | 302.02102643609123 | 1999-10-28 | | 337.9318386924912 | 2000-10-27 | | 333.80931439318346 | 2001-10-27 | +---------------------------+------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") VARIANCE,VAR\_POP,VARIANCE\_POP --- ### window_funnel Searches for an event chain in a sliding window and calculates the maximum number of consecutive events in the event chain. This function is commonly used for analyzing conversion rate. It is supported from v2.3. This function works according to the following rules: * It starts the count from the first event in the event chain. If the first event is found, the event counter is set to 1 and the sliding window starts. If the first event is not found, 0 is returned. * In the sliding window, the counter is incremented if the events in the event chain occur in sequence. If the sliding window is exceeded, the event counter is no longer incremented. * If multiple event chains match the specified condition, the longest event chain is returned. #### Syntax[​](#syntax "Direct link to Syntax") ```plain BIGINT window_funnel(BIGINT window, DATE|DATETIME time, INT mode, array[cond1, cond2, ..., condN]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `window`: The length of the sliding window. The supported data type is BIGINT. The unit depends on the `time` parameter. If the data type of `time` is DATE, the unit is days. If the data type of `time` is DATETIME, the unit is seconds. * `time`: The column containing timestamps. DATE and DATETIME types are supported. * `mode`: The mode in which the event chain is filtered. The supported data type is INT. Value range: 0, 1, 2. * `0` is the default value, which indicates common funnel calculation. * `1` indicates the `DEDUPLICATION` mode, that is, the filtered event chain cannot have repeated events. Suppose the `array` parameter is `[event_type = 'A', event_type = 'B', event_type = 'C', event_type = 'D']` and the original event chain is "A-B-C-B-D". Event B is repeated and the filtered event chain is "A-B-C". * `2` indicates the `FIXED` mode, that is, the filtered event chain cannot have events that disrupt the specified sequence. Suppose the previous `array` parameter is used and the original event chain is "A-B-D-C". Event D interrupts the sequence and the filtered event chain is "A-B". * `4` indicates the `INCREASE` mode, which means the filtered events must have strictly increasing timestamps. Duplicate timestamp disrupts the event chain. This mode is supported since version 2.5. * `array`: The defined event chain. It must be an array. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BIGINT type. #### Examples[​](#examples "Direct link to Examples") **Example 1**: Calculate the maximum number of consecutive events based on `uid`. The sliding window is 1800s and the filtering mode is `0`. This example uses table `action`, in which data is sorted by `uid`. ```plaintext mysql> select * from action; +------+------------+---------------------+ | uid | event_type | time | +------+------------+---------------------+ | 1 | Browse | 2020-01-02 11:00:00 | | 1 | Click | 2020-01-02 11:10:00 | | 1 | Order | 2020-01-02 11:20:00 | | 1 | Pay | 2020-01-02 11:30:00 | | 1 | Browse | 2020-01-02 11:00:00 | | 2 | Order | 2020-01-02 11:00:00 | | 2 | Pay | 2020-01-02 11:10:00 | | 3 | Browse | 2020-01-02 11:20:00 | | 3 | Click | 2020-01-02 12:00:00 | | 4 | Browse | 2020-01-02 11:50:00 | | 4 | Click | 2020-01-02 12:00:00 | | 5 | Browse | 2020-01-02 11:50:00 | | 5 | Click | 2020-01-02 12:00:00 | | 5 | Order | 2020-01-02 11:10:00 | | 6 | Browse | 2020-01-02 11:50:00 | | 6 | Click | 2020-01-02 12:00:00 | | 6 | Order | 2020-01-02 12:10:00 | +------+------------+---------------------+ 17 rows in set (0.01 sec) ``` Execute the following statement: ```plaintext select uid, window_funnel(1800,time,0,[event_type='Browse', event_type='Click', event_type='Order', event_type='Pay']) AS level from action group by uid order by uid; +------+-------+ | uid | level | +------+-------+ | 1 | 4 | | 2 | 0 | | 3 | 1 | | 4 | 2 | | 5 | 2 | | 6 | 3 | +------+-------+ ``` Description of the results: * The matching event chain for `uid = 1` is "Browse-Click-Order-Pay", and `4` is returned. The time of the last "Browse" event (2020-01-02 11:00:00) does not meet the condition and is not counted. * The event chain of `uid = 2` does not start from the first event "Browse", and `0` is returned. * The matching event chain for `uid = 3` is "Browse", and `1` is returned. The "Click" event exceeds the 1800s time window and is not counted. * The matching event chain for `uid = 4` is "Browse-Click", and `2` is returned. * The matching event chain for `uid = 5` is "Browse-Click", and `2` is returned. The "Order" event (2020-01-02 11:10:00) does not belong to the event chain and is not counted. * The matching event chain for `uid = 6` is "Browse-Click-Order", and `3` is returned. **Example 2**: Calculate the maximum number of consecutive events based on `uid`. The sliding window is 1800s, and filtering modes `0` and `1` are used. This example uses table `action1`, in which data is sorted by `time`. ```plaintext mysql> select * from action1 order by time; +------+------------+---------------------+ | uid | event_type | time | +------+------------+---------------------+ | 1 | Browse | 2020-01-02 11:00:00 | | 2 | Browse | 2020-01-02 11:00:01 | | 1 | Click | 2020-01-02 11:10:00 | | 1 | Order | 2020-01-02 11:29:00 | | 1 | Click | 2020-01-02 11:29:50 | | 1 | Pay | 2020-01-02 11:30:00 | | 1 | Click | 2020-01-02 11:40:00 | +------+------------+---------------------+ 7 rows in set (0.03 sec) ``` Execute the following statement: ```plaintext select uid, window_funnel(1800,time,0,[event_type='Browse', event_type='Click', event_type='Order', event_type='Pay']) AS level from action1 group by uid order by uid; +------+-------+ | uid | level | +------+-------+ | 1 | 4 | | 2 | 1 | +------+-------+ 2 rows in set (0.02 sec) ``` For `uid = 1`, the "Click" event (2020-01-02 11:29:50) is a repeated event but it is still counted because mode `0` is used. Therefore, `4` is returned. Change `mode` to `1` and execute the statement again. ```plaintext +------+-------+ | uid | level | +------+-------+ | 1 | 3 | | 2 | 1 | +------+-------+ 2 rows in set (0.05 sec) ``` The longest event chain filtered after deduplication is "Browse-Click-Order", and `3` is returned. **Example 3**: Calculate the maximum number of consecutive events based on `uid`. The sliding window is 1900s, and filter modes `0` and `2` are used. This example uses table `action2`, in which data is sorted by `time`. ```plaintext mysql> select * from action2 order by time; +------+------------+---------------------+ | uid | event_type | time | +------+------------+---------------------+ | 1 | Browse | 2020-01-02 11:00:00 | | 2 | Browse | 2020-01-02 11:00:01 | | 1 | Click | 2020-01-02 11:10:00 | | 1 | Pay | 2020-01-02 11:30:00 | | 1 | Order | 2020-01-02 11:31:00 | +------+------------+---------------------+ 5 rows in set (0.01 sec) ``` Execute the following statement: ```plaintext select uid, window_funnel(1900,time,0,[event_type='Browse', event_type='Click', event_type='Order', event_type='Pay']) AS level from action2 group by uid order by uid; +------+-------+ | uid | level | +------+-------+ | 1 | 3 | | 2 | 1 | +------+-------+ 2 rows in set (0.02 sec) ``` `3` is returned for `uid = 1` because mode `0` is used and the "Pay" event (2020-01-02 11:30:00) does not disrupt the event chain. Change `mode` to `2` and execute the statement again. ```plaintext select uid, window_funnel(1900,time,2,[event_type='Browse', event_type='Click', event_type='Order', event_type='Pay']) AS level from action2 group by uid order by uid; +------+-------+ | uid | level | +------+-------+ | 1 | 2 | | 2 | 1 | +------+-------+ 2 rows in set (0.06 sec) ``` `2` is returned because the "Pay" event disrupts the event chain and the event counter stops. The filtered event chain is "Browse-Click". **Example 4**: Calculate the maximum number of consecutive events based on `uid`. The sliding window is 1900s, and filter modes `0` and `4` are used. This example uses table `action3`, in which data is sorted by `time`. ```plaintext select * from action3 order by time; +------+------------+---------------------+ | uid | event_type | time | +------+------------+---------------------+ | 1 | Browse | 2020-01-02 11:00:00 | | 1 | Click | 2020-01-02 11:00:01 | | 2 | Browse | 2020-01-02 11:00:03 | | 1 | Order | 2020-01-02 11:00:31 | | 2 | Click | 2020-01-02 11:00:03 | | 2 | Order | 2020-01-02 11:01:03 | +------+------------+---------------------+ 3 rows in set (0.02 sec) ``` Execute the following statement: ```plaintext select uid, window_funnel(1900,time,0,[event_type='Browse', event_type='Click', event_type='Order']) AS level from action3 group by uid order by uid; +------+-------+ | uid | level | +------+-------+ | 1 | 3 | | 2 | 3 | +------+-------+ ``` `3` is returned for `uid = 1` and `uid = 2`. Change `mode` to `4` and execute the statement again. ```plaintext select uid, window_funnel(1900,time,4,[event_type='Browse', event_type='Click', event_type='Order']) AS level from action3 group by uid order by uid; +------+-------+ | uid | level | +------+-------+ | 1 | 3 | | 2 | 1 | +------+-------+ 1 row in set (0.02 sec) ``` `1` is returned for `uid = 2` because mode `4` (strictly increasing) is used. "Click" happens at the same second as "BROWSE". Therefore, "Click" and "Order" are not counted. #### Keywords[​](#keywords "Direct link to Keywords") window funnel, funnel, window\_funnel --- ### Array #### [📄️ all\_match](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/all_match.md) [Returns whether all elements of an array match the given predicate.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/all_match.md) #### [📄️ any\_match](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/any_match.md) [Returns whether any elements of an array match the given predicate.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/any_match.md) #### [📄️ array\_agg](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_agg.md) [Aggregates values (including NULL) in a column into an array (multiple rows to one row), and optionally order the elements by specific columns.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_agg.md) #### [📄️ array\_append](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_append.md) [Adds a new element to the end of the array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_append.md) #### [📄️ array\_avg](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_avg.md) [Returns the average of all elements in an ARRAY, ignoring NULL values.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_avg.md) #### [📄️ array\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_concat.md) [Concatenates multiple arrays into one array that contains all the elements in the arrays.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_concat.md) #### [📄️ array\_contains](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_contains.md) [Checks whether the array contains a certain element.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_contains.md) #### [📄️ array\_contains\_all](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_contains_all.md) [Checks whether arr1 contains all the elements of arr2, that is, whether arr2 is a subset of arr1.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_contains_all.md) #### [📄️ array\_contains\_seq](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_contains_seq.md) [Checks whether all the elements of array2 appear in array1 in the same exact order.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_contains_seq.md) #### [📄️ array\_cum\_sum](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_cum_sum.md) [Returns the cumulative sums of elements in an array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_cum_sum.md) #### [📄️ array\_difference](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_difference.md) [Calculates the difference between every two neighboring elements of an array by subtracting each element from its following element and returns an array...](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_difference.md) #### [📄️ array\_distinct](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_distinct.md) [Removes duplicate elements from an array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_distinct.md) #### [📄️ array\_filter](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_filter.md) [Returns values from an array that matches the given filter.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_filter.md) #### [📄️ array\_flatten](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_flatten.md) [array\_flatten flattens one layer of nested arrays.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_flatten.md) #### [📄️ array\_generate](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_generate.md) [Returns an array of distinct values within the range specified by start and end, with an increment of step.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_generate.md) #### [📄️ array\_intersect](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_intersect.md) [Returns an array of the elements in the intersection of one or more arrays.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_intersect.md) #### [📄️ array\_join](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_join.md) [Concatenates the elements of an array into a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_join.md) #### [📄️ array\_length](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_length.md) [Returns the number of elements in an array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_length.md) #### [📄️ array\_map](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_map.md) [array\_map() is a higher-order SQL function that can take lambda functions as arguments.It applies the lambda function to the input arrays and returns a new...](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_map.md) #### [📄️ array\_max](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_max.md) [Returns the maximum value among all elements in an ARRAY.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_max.md) #### [📄️ array\_min](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_min.md) [Returns the minimum value among all elements in an ARRAY.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_min.md) #### [📄️ array\_position](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_position.md) [Obtains the position of an element in an array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_position.md) #### [📄️ array\_remove](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_remove.md) [Removes an element from an array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_remove.md) #### [📄️ array\_repeat](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_repeat.md) [array\_repeat returns an array containing a given element repeated a specified number of times.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_repeat.md) #### [📄️ array\_slice](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_slice.md) [Returns a slice of an array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_slice.md) #### [📄️ array\_sort](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_sort.md) [Sorts the elements of an array in ascending order.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_sort.md) #### [📄️ array\_sortby](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_sortby.md) [Sorts elements in an array according to the ascending order of elements in another array or array converted from a lambda expression.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_sortby.md) #### [📄️ array\_sum](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_sum.md) [Sums all the elements in an array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_sum.md) #### [📄️ array\_to\_bitmap](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_to_bitmap.md) [Converts an array into BITMAP values.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_to_bitmap.md) #### [📄️ array\_top\_n](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_top_n.md) [Sorts the elements of a given array in their natural descending order, and returns the top-n elements in an array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_top_n.md) #### [📄️ array\_unique\_agg](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_unique_agg.md) [Aggregates distinct values (including NULL) in an ARRAY column into an array (from multiple rows to one row).](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_unique_agg.md) #### [📄️ arrays\_overlap](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/arrays_overlap.md) [Checks whether the intersection of two arrays is empty.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/arrays_overlap.md) #### [📄️ arrays\_zip](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/arrays_zip.md) [Merges the given arrays by index into an array of structs in which the n-th struct contains n-th values of all input arrays.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/arrays_zip.md) #### [📄️ cardinality](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/cardinality.md) [Returns the number of elements in an array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/cardinality.md) #### [📄️ element\_at](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/element_at.md) [Returns the element at the specified position (index) from a given array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/element_at.md) #### [📄️ reverse](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/reverse.md) [Reverses a string or array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/reverse.md) #### [📄️ unnest](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/unnest.md) [UNNEST is a table function that takes an array and converts elements in that array into multiple rows of a table.](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/unnest.md) --- ### all_match Returns whether all elements of an array match the given predicate. * Returns `true` (1) if all the elements match the predicate (a special case is when the array is empty). * Returns `false` (0) if one or more elements do not match. * Returns NULL if the predicate returns NULL for one or more elements and `true` for all other elements. This function is supported from v3.0.6 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext all_match(lambda_function, arr1, arr2...) ``` Returns whether all elements of `arr1` match the given predicate in the lambda function. #### Parameters[​](#parameters "Direct link to Parameters") * `arr1`: the array to match. * `arrN`: optional arrays used in the lambda function. * `lambda_function`: the lambda function used to match values. #### Return value[​](#return-value "Direct link to Return value") Returns a BOOLEAN value. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The lambda function follows the usage notes in [array\_map()](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_map.md). * If the input array is null or the lambda function results in null, null is returned. * If `arr1` is empty, `true` is returned. * To apply this function to MAP, rewrite `all_match((k,v)->k>v,map)` to `all_match(map_values(transform_values((k,v)->k>v, map)))`. For example, `select all_match(map_values(transform_values((k,v)->k>v, map{2:1})));` returns 1. #### Examples[​](#examples "Direct link to Examples") Check whether all elements in `x` are less than the elements in `y`. ```plain select all_match((x,y) -> x < y, [1,2,-8], [4,5,6]); +---------------------------------------------------+ | all_match((x, y) -> x < y, [1, 2, -8], [4, 5, 6]) | +---------------------------------------------------+ | 1 | +---------------------------------------------------+ select all_match((x,y) -> x < y, [1,2,null], [4,5,6]); +-----------------------------------------------------+ | all_match((x, y) -> x < y, [1, 2, NULL], [4, 5, 6]) | +-----------------------------------------------------+ | NULL | +-----------------------------------------------------+ select all_match((x,y) -> x < y, [1,2,8], [4,5,6]); +--------------------------------------------------+ | all_match((x, y) -> x < y, [1, 2, 8], [4, 5, 6]) | +--------------------------------------------------+ | 0 | +--------------------------------------------------+ select all_match((x,y) -> x < y, [], []); +------------------------------------+ | all_match((x, y) -> x < y, [], []) | +------------------------------------+ | 1 | +------------------------------------+ select all_match((x,y) -> x < y, null, [4,5,6]); +---------------------------------------------+ | all_match((x, y) -> x < y, NULL, [4, 5, 6]) | +---------------------------------------------+ | NULL | +---------------------------------------------+ ``` --- ### any_match Returns whether any elements of an array match the given predicate. * Returns `true` (1) if one or more elements match the predicate. * Returns `false` (0) if none of the elements matches (a special case is when the array is empty). * Returns NULL if the predicate returns NULL for one or more elements and `false` for all other elements. This function is supported from v3.0.6 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext any_match(lambda_function, arr1, arr2...) ``` Returns whether any elements of `arr1` match the given predicate in the lambda function. #### Parameters[​](#parameters "Direct link to Parameters") * `arr1`: the array to match. * `arrN`: optional arrays used in the lambda function. * `lambda_function`: the lambda function used to match values. #### Return value[​](#return-value "Direct link to Return value") Returns a BOOLEAN value. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The lambda function follows the usage notes in [array\_map()](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_map.md). * If the input array is null or the lambda function results in null, null is returned. * If `arr1` is empty, `false` is returned. * To apply this function to MAP, rewrite `any_match((k,v)->k>v,map)` to `any_match(map_values(transform_values((k,v)->k>v, map)))`. For example, `select any_match(map_values(transform_values((k,v)->k>v, map{2:1})));` returns 1. #### Examples[​](#examples "Direct link to Examples") Check whether any element in `x` is less than the elements in `y`. ```plain select any_match((x,y) -> x < y, [1,2,8], [4,5,6]); +--------------------------------------------------+ | any_match((x, y) -> x < y, [1, 2, 8], [4, 5, 6]) | +--------------------------------------------------+ | 1 | +--------------------------------------------------+ select any_match((x,y) -> x < y, [11,12,8], [4,5,6]); +----------------------------------------------------+ | any_match((x, y) -> x < y, [11, 12, 8], [4, 5, 6]) | +----------------------------------------------------+ | 0 | +----------------------------------------------------+ select any_match((x,y) -> x < y, [11,12,null], [4,5,6]); +-------------------------------------------------------+ | any_match((x, y) -> x < y, [11, 12, NULL], [4, 5, 6]) | +-------------------------------------------------------+ | NULL | +-------------------------------------------------------+ select any_match((x,y) -> x < y, [], []); +------------------------------------+ | any_match((x, y) -> x < y, [], []) | +------------------------------------+ | 0 | +------------------------------------+ select any_match((x,y) -> x < y, null, [4,5,6]); +---------------------------------------------+ | any_match((x, y) -> x < y, NULL, [4, 5, 6]) | +---------------------------------------------+ | NULL | +---------------------------------------------+ ``` --- ### array_agg Aggregates values (including `NULL`) in a column into an array (multiple rows to one row), and optionally order the elements by specific columns. From v3.0, array\_agg() supports using ORDER BY to sort elements. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext ARRAY_AGG([distinct] col [order by col0 [desc | asc] [nulls first | nulls last] ...]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `col`: the column whose values you want to aggregate. Supported data types are BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, VARCHAR, CHAR, DATETIME, DATE, ARRAY (since v3.1), MAP (since v3.1), and STRUCT (since v3.1). * `col0`: the column which decides the order of `col`. There may be more than one ORDER BY column. * `[desc | asc]`: specifies whether to sort the elements in ascending order (default) or descending order of `col0`. * `[nulls first | nulls last]`: specifies whether null values are placed at the first or last place. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the ARRAY type, optionally sorted by `col0`. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The order of the elements in an array is random, which means it may be different from the order of the values in the column if no ORDER BY columns or no sorted by order by columns are specified. * The data type of the elements in the returned array is the same as the data type of the values in the column. * Returns `NULL` if the input is empty and without group-by columns. * ARRAY\_AGG\_DISTINCT() is an alias of ARRAY\_AGG(DISTINCT). #### Examples[​](#examples "Direct link to Examples") Take the following data table as an example: ```plaintext mysql> select * from t; +------+------+------+ | a | name | pv | +------+------+------+ | 11 | | 33 | | 2 | NULL | 334 | | 1 | fzh | 3 | | 1 | fff | 4 | | 1 | fff | 5 | +------+------+------+ ``` Example 1: Group the values in column `a` and aggregate values in column `pv` into an array by ordering `a` by `name`. ```plaintext mysql> select a, array_agg(pv order by name nulls first) from t group by a; +------+---------------------------------+ | a | array_agg(pv ORDER BY name ASC) | +------+---------------------------------+ | 2 | [334] | | 11 | [33] | | 1 | [4,5,3] | +------+---------------------------------+ -- Aggregate values with no order. mysql> select a, array_agg(pv) from t group by a; +------+---------------+ | a | array_agg(pv) | +------+---------------+ | 11 | [33] | | 2 | [334] | | 1 | [3,4,5] | +------+---------------+ 3 rows in set (0.03 sec) ``` Example 2: Aggregate values in column `pv` into an array with ordering by `name`. ```plaintext mysql> select array_agg(pv order by name desc nulls last) from t; +----------------------------------+ | array_agg(pv ORDER BY name DESC) | +----------------------------------+ | [3,4,5,33,334] | +----------------------------------+ 1 row in set (0.02 sec) -- Aggregate values with no order. mysql> select array_agg(pv) from t; +----------------+ | array_agg(pv) | +----------------+ | [3,4,5,33,334] | +----------------+ 1 row in set (0.03 sec) ``` Example 3: Aggregate values in column `pv` using the WHERE clause. If no data in `pv` meets the filter condition, a `NULL` value is returned. ```plaintext mysql> select array_agg(pv order by name desc nulls last) from t where a < 0; +----------------------------------+ | array_agg(pv ORDER BY name DESC) | +----------------------------------+ | NULL | +----------------------------------+ 1 row in set (0.02 sec) -- Aggregate values with no order. mysql> select array_agg(pv) from t where a < 0; +---------------+ | array_agg(pv) | +---------------+ | NULL | +---------------+ 1 row in set (0.03 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") ARRAY\_AGG, ARRAY --- ### array_append Adds a new element to the end of the array. Returns an array. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_append(any_array, any_element) ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select array_append([1, 2], 3); +------------------------+ | array_append([1,2], 3) | +------------------------+ | [1,2,3] | +------------------------+ 1 row in set (0.00 sec) ``` You can add NULL to the array. ```plain mysql> select array_append([1, 2], NULL); +---------------------------+ | array_append([1,2], NULL) | +---------------------------+ | [1,2,NULL] | +---------------------------+ 1 row in set (0.01 sec) ``` #### keyword[​](#keyword "Direct link to keyword") ARRAY\_APPEND,ARRAY --- ### array_avg Calculates the average value of all data in an ARRAY and return this result. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_avg(array(type)) ``` `array(type)` supports the following types of elements: BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, DECIMALV2. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select array_avg([11, 11, 12]); +-----------------------+ | array_avg([11,11,12]) | +-----------------------+ | 11.333333333333334 | +-----------------------+ mysql> select array_avg([11.33, 11.11, 12.324]); +---------------------------------+ | array_avg([11.33,11.11,12.324]) | +---------------------------------+ | 11.588 | +---------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ARRAY\_AVG,ARRAY --- ### array_concat Concatenates multiple arrays into one array that contains all the elements in the arrays. Elements in the arrays to concatenate can be of the same type or different types. However, we recommend that the elements be of the same type. Nulls are processed as normal values. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_concat(input0, input1, ...) ``` #### Parameters[​](#parameters "Direct link to Parameters") `input`: one or more arrays that you want to concatenate. Specify arrays in the `(input0, input1, ...)` format. This function supports the following types of array elements: BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, VARCHAR, DECIMALV2, DATETIME, DATE, and JSON. **JSON is supported from 2.5.** #### Return value[​](#return-value "Direct link to Return value") Returns an array that contains all the elements held in the arrays that are specified by the `input` parameter. The elements of the returned array are of the same data type as the elements of the input arrays. Additionally, the elements of the returned array follow the order of the input arrays and their elements. #### Examples[​](#examples "Direct link to Examples") Example 1: Concatenate arrays that contain numeric elements. ```plaintext select array_concat([57.73,97.32,128.55,null,324.2], [3], [5]) as res; +-------------------------------------+ | res | +-------------------------------------+ | [57.73,97.32,128.55,null,324.2,3,5] | +-------------------------------------+ ``` Example 2: Concatenate arrays that contain string elements. ```plaintext select array_concat(["sql","storage","execute"], ["Query"], ["Vectorized", "cbo"]); +----------------------------------------------------------------------------+ | array_concat(['sql','storage','execute'], ['Query'], ['Vectorized','cbo']) | +----------------------------------------------------------------------------+ | ["sql","storage","execute","Query","Vectorized","cbo"] | +----------------------------------------------------------------------------+ ``` Example 3: Concatenate two arrays of different types. ```plaintext select array_concat([57,65], ["pear","apple"]); +-------------------------------------------+ | array_concat([57, 65], ['pear', 'apple']) | +-------------------------------------------+ | ["57","65","pear","apple"] | +-------------------------------------------+ ``` Example 4: Process nulls as normal values. ```plaintext select array_concat(["sql",null], [null], ["Vectorized", null]); +---------------------------------------------------------+ | array_concat(['sql',NULL], [NULL], ['Vectorized',NULL]) | +---------------------------------------------------------+ | ["sql",null,null,"Vectorized",null] | +---------------------------------------------------------+ ``` --- ### array_contains Checks whether the array contains a certain element. If yes, it returns 1; otherwise, it returns 0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_contains(any_array, any_element) ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select array_contains(["apple","orange","pear"], "orange"); +-----------------------------------------------------+ | array_contains(['apple','orange','pear'], 'orange') | +-----------------------------------------------------+ | 1 | +-----------------------------------------------------+ 1 row in set (0.01 sec) ``` You can also check whether the array contains NULL. ```plain mysql> select array_contains([1, NULL], NULL); +--------------------------------+ | array_contains([1,NULL], NULL) | +--------------------------------+ | 1 | +--------------------------------+ 1 row in set (0.00 sec) ``` You can check whether the multi-dimensional array contains a certain subarray. At this time, you need to ensure that the subarray elements match exactly, including the element arrangement order. ```plain mysql> select array_contains([[1,2,3], [4,5,6]], [4,5,6]); +--------------------------------------------+ | array_contains([[1,2,3],[4,5,6]], [4,5,6]) | +--------------------------------------------+ | 1 | +--------------------------------------------+ 1 row in set (0.00 sec) mysql> select array_contains([[1,2,3], [4,5,6]], [4,6,5]); +--------------------------------------------+ | array_contains([[1,2,3],[4,5,6]], [4,6,5]) | +--------------------------------------------+ | 0 | +--------------------------------------------+ 1 row in set (0.00 sec) ``` #### keyword[​](#keyword "Direct link to keyword") ARRAY\_CONTAINS,ARRAY --- ### array_contains_all Checks whether `arr1` contains all the elements of `arr2`, that is, whether `arr2` is a subset of `arr1`. If yes, 1 is returned. If not, 0 is returned. #### Syntax[​](#syntax "Direct link to Syntax") ```haskell BOOLEAN array_contains_all(arr1, arr2) ``` #### Parameters[​](#parameters "Direct link to Parameters") `arr`: the two arrays to compare. This syntax checks whether `arr2` is a subset of `arr1`. The data types of elements in the two arrays must be the same. For the data types of array elements supported by StarRocks, see [ARRAY](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/Array.md). #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BOOLEAN type. 1 is returned if `arr2` is a subset of `arr1`. Otherwise, 0 is returned. If any of the two arrays is NULL, NULL is returned. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * If an array contains `null` elements, `null` is processed as a value. * An empty array is a subset of any array. * Elements in the two arrays can have different order. #### Examples[​](#examples "Direct link to Examples") 1. Create a table named `t1` and insert data into this table. ```sql CREATE TABLE t1 ( c0 INT, c1 ARRAY, c2 ARRAY ) ENGINE=OLAP DUPLICATE KEY(c0) DISTRIBUTED BY HASH(c0); INSERT INTO t1 VALUES (1,[1,2,3],[1,2]), (2,[1,2,3],[1,4]), (3,NULL,[1]), (4,[1,2,null],NULL), (5,[1,2,null],[null]), (6,[2,3],[]); ``` 2. Query data from this table. ```plain SELECT * FROM t1 ORDER BY c0; +------+------------+----------+ | c0 | c1 | c2 | +------+------------+----------+ | 1 | [1,2,3] | [1,2] | | 2 | [1,2,3] | [1,4] | | 3 | NULL | [1] | | 4 | [1,2,null] | NULL | | 5 | [1,2,null] | [null] | | 6 | [2,3] | [] | +------+------------+----------+ ``` 3. Check whether each row of `c2` is a subset of the corresponding row of `c1`. ```plaintext SELECT c0, c1, c2, array_contains_all(c1, c2) FROM t1 ORDER BY c0; +------+------------+----------+----------------------------+ | c0 | c1 | c2 | array_contains_all(c1, c2) | +------+------------+----------+----------------------------+ | 1 | [1,2,3] | [1,2] | 1 | | 2 | [1,2,3] | [1,4] | 0 | | 3 | NULL | [1] | NULL | | 4 | [1,2,null] | NULL | NULL | | 5 | [1,2,null] | [null] | 1 | | 6 | [2,3] | [] | 1 | +------+------------+----------+----------------------------+ ``` In the output: For row 1, `c2` is a subset of `c1` and 1 is returned. For row 2, `c2` is not a subset of `c1` and 0 is returned. For row 3, `c1` is NULL and NULL is returned. For row 4, `c2` is NULL and NULL is returned. For row 5, the two arrays contain `null` and `null` is processed as a normal value, `1` is returned. For row 6, `c2` is an empty array and considered a subset of `c1`. Therefore, `1` is returned. --- ### array_contains_seq Checks whether all the elements of array2 appear in array1 in the same exact order. Therefore, the function will return 1, if and only if `array1 = prefix + array2 + suffix`. For example: * `select array_contains_seq([1,2,3,4], [1,2,3]);` returns 1. * `select array_contains_seq([1,2,3,4], [4,3]);` returns 0. This function is supported from v3.3 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```haskell BOOLEAN array_contains_seq(arr1, arr2) ``` #### Parameters[​](#parameters "Direct link to Parameters") `arr`: the two arrays to compare. This syntax checks whether `arr2` is a subset of `arr1` and in the same exact order. The data types of elements in the two arrays must be the same. For the data types of array elements supported by StarRocks, see [ARRAY](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/Array.md). #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BOOLEAN type. * `1` is returned if `arr2` is a subset of `arr1` and the elements in `arr2` observe the same order as those in `arr1`. Otherwise, `0` is returned. * An empty array is a subset of any array. Therefore, `1` is returned if `arr2` is empty but `arr1` is a valid array. * NULL is returned if any input array is NULL. * Nulls in arrays are processed as normal values. For example, `SELECT array_contains_seq([1, 2, NULL, 3, 4], [2,3])` will return 0. However, `SELECT array_contains_seq([1, 2, NULL, 3, 4], [2,NULL,3])` will return 1. #### Examples[​](#examples "Direct link to Examples") ```plaintext MySQL > select array_contains_seq([1,2,3,4], [1,2,3]); +---------------------------------------------+ | array_contains_seq([1, 2, 3, 4], [1, 2, 3]) | +---------------------------------------------+ | 1 | +---------------------------------------------+ MySQL > select array_contains_seq([1,2,3,4], [3,2]); +------------------------------------------+ | array_contains_seq([1, 2, 3, 4], [3, 2]) | +------------------------------------------+ | 0 | +------------------------------------------+ MySQL > select array_contains_seq([1, 2, NULL, 3, 4], ['a']); +-----------------------------------------------+ | array_contains_all([1, 2, NULL, 3, 4], ['a']) | +-----------------------------------------------+ | 0 | +-----------------------------------------------+ MySQL > select array_contains_seq([1,2,3,4,null], null); +------------------------------------------+ | array_contains([1, 2, 3, 4, NULL], NULL) | +------------------------------------------+ | NULL | +------------------------------------------+ MySQL > select array_contains_seq([1,2,3,4], []); +--------------------------------------+ | array_contains_seq([1, 2, 3, 4], []) | +--------------------------------------+ | 1 | +--------------------------------------+ ``` --- ### array_cum_sum Returns the cumulative sums of elements in an array. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_cum_sum(array(bigint)) array_cum_sum(array(double)) ``` #### Parameters[​](#parameters "Direct link to Parameters") `array`: The elements in the array must be of the BIGINT (8-byte signed integer) or DOUBLE (8-byte floating-point number) type. #### Return values[​](#return-values "Direct link to Return values") Returns an array. NULL is returned if the input array is NULL. #### Examples[​](#examples "Direct link to Examples") Example 1: Return the cumulative sums of an array. ```plain select array_cum_sum([11, 11, 12]); +---------------------------+ | array_cum_sum([11,11,12]) | +---------------------------+ | [11,22,34] | +---------------------------+ ``` Example 2: Use this function with CAST. ```plain select array_cum_sum([cast(11.33 as double),cast(11.11 as double),cast(12.324 as double)]); +---------------------------------------------------------------------------------------+ | array_cum_sum([CAST(11.33 AS DOUBLE), CAST(11.11 AS DOUBLE), CAST(12.324 AS DOUBLE)]) | +---------------------------------------------------------------------------------------+ | [11.33,22.439999999999998,34.763999999999996] | +---------------------------------------------------------------------------------------+ ``` Example 3: The input array contains a null. ```plain select array_cum_sum([null,1,2]); +---------------------------------+ | array_cum_sum([null,1,2]) | +---------------------------------+ | [null,1,3] | +---------------------------------+ ``` Example 4: The input array is null. ```plain select array_cum_sum(null); +---------------------------------+ | array_cum_sum(NULL) | +---------------------------------+ | NULL | +---------------------------------+ ``` --- ### array_difference Calculates the difference between every two neighboring elements of an array by subtracting each element from its following element and returns an array that consists of the differences. #### Syntax[​](#syntax "Direct link to Syntax") ```sql array_difference(input) ``` #### Parameters[​](#parameters "Direct link to Parameters") `input`: the array for which you want to calculate the differences between every two neighboring elements. #### Return value[​](#return-value "Direct link to Return value") Returns an array with the same length as `input`. The return element type is determined by the input element type as follows: | Input element type | Return element type | | ---------------------------------------------------- | ---------------------------- | | `BOOLEAN`, `TINYINT`, `SMALLINT`, `INT`, or `BIGINT` | `BIGINT` | | `LARGEINT` | `LARGEINT` | | `FLOAT` or `DOUBLE` | `DOUBLE` | | `DECIMAL(P, S)`, where `P <= 38` | `DECIMAL(min(P + 1, 38), S)` | `ARRAY` (`P > 38`) is not supported. #### Examples[​](#examples "Direct link to Examples") Example 1: ```plain mysql> SELECT array_difference([342, 32423, 213, 23432]); +-----------------------------------------+ | array_difference([342,32423,213,23432]) | +-----------------------------------------+ | [0,32081,-32210,23219] | +-----------------------------------------+ ``` Example 2: ```plain mysql> SELECT array_difference([342, 32423, 213, null, 23432]); +----------------------------------------------+ | array_difference([342,32423,213,NULL,23432]) | +----------------------------------------------+ | [0,32081,-32210,null,null] | +----------------------------------------------+ ``` Example 3: ```plain mysql> SELECT array_difference([1.2, 2.3, 3.2, 4324242.55]); +--------------------------------------------+ | array_difference([1.2,2.3,3.2,4324242.55]) | +--------------------------------------------+ | [0,1.1,0.9,4324239.35] | +--------------------------------------------+ ``` Example 4: ```plain mysql> SELECT array_difference([false, true, false]); +----------------------------------------+ | array_difference([FALSE, TRUE, FALSE]) | +----------------------------------------+ | [0,1,-1] | +----------------------------------------+ ``` --- ### array_distinct Removes duplicate elements from an array. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_distinct(array) ``` #### Parameters[​](#parameters "Direct link to Parameters") `array`: the array from which you want to remove duplicate elements. Only the ARRAY data type is supported. #### Return value[​](#return-value "Direct link to Return value") Returns an array. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The elements of the array that is returned may be sorted in a different order than the elements of the array that you specify. * The elements of the array that is returned are of the same data type as the elements of the array that you specify. #### Examples[​](#examples "Direct link to Examples") In this section, the following table is used as an example: ```plaintext mysql> select * from test; +------+---------------+ | c1 | c2 | +------+---------------+ | 1 | [1,1,2] | | 2 | [1,null,null] | | 3 | NULL | | 4 | [null] | +------+---------------+ ``` Remove duplicate values from column `c2`. ```plaintext mysql> select c1, array_distinct(c2) from test; +------+----------------------+ | c1 | array_distinct(`c2`) | +------+----------------------+ | 1 | [2,1] | | 2 | [null,1] | | 3 | NULL | | 4 | [null] | +------+----------------------+ ``` --- ### array_filter Returns values from an array that matches the given filter. This function comes in the following two forms. The lambda adoption allows for more flexible array filtering. For more information about lambda functions, see [Lambda expression](https://docs.starrocks.io/docs/sql-reference/sql-functions/Lambda_expression.md). This function is supported from v2.5. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_filter(array, array) array_filter(lambda_function, arr1,arr2...) ``` * `array_filter(array, array)` Returns values from an array that matches `array`. * `array_filter(lambda_function, arr1,arr2...)` Returns values from an array that matches the lambda function. #### Parameters[​](#parameters "Direct link to Parameters") `array`: the array to filter values from. `array`: the expression used to filter values. `lambda_function`: the lambda function used to filter values. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The two input parameters of `array_filter(array, array)` must be ARRAY and the filter expression can evaluate to `array`. * The lambda function in `array_filter(lambda_function, arr1,arr2...)` follows the usage notes in [array\_map()](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_map.md). * If the input array is null, null is returned. If the filter array is null, an empty array is returned. #### Examples[​](#examples "Direct link to Examples") * Examples without using lambda functions ```plain -- All the elements in the array match the filter. select array_filter([1,2,3],[1,1,1]); +------------------------------------+ | array_filter([1, 2, 3], [1, 1, 1]) | +------------------------------------+ | [1,2,3] | +------------------------------------+ 1 row in set (0.01 sec) -- The filter is null and an empty array is returned. select array_filter([1,2,3],null); +-------------------------------+ | array_filter([1, 2, 3], NULL) | +-------------------------------+ | [] | +-------------------------------+ 1 row in set (0.01 sec) -- The input array is null, null is returned. select array_filter(null,[1]); +-------------------------+ | array_filter(NULL, [1]) | +-------------------------+ | NULL | +-------------------------+ -- Both the input array and filter are null. Null is returned. select array_filter(null,null); +--------------------------+ | array_filter(NULL, NULL) | +--------------------------+ | NULL | +--------------------------+ 1 row in set (0.01 sec) -- The filter contains a null element and an empty array is returned. select array_filter([1,2,3],[null]); +---------------------------------+ | array_filter([1, 2, 3], [NULL]) | +---------------------------------+ | [] | +---------------------------------+ 1 row in set (0.01 sec) -- The filter contains two null elements and an empty array is returned. select array_filter([1,2,3],[null,null]); +---------------------------------------+ | array_filter([1, 2, 3], [NULL, NULL]) | +---------------------------------------+ | [] | +---------------------------------------+ 1 row in set (0.00 sec) -- Only one element matches the filter. select array_filter([1,2,3],[null,1,0]); +---------------------------------------+ | array_filter([1, 2, 3], [NULL, 1, 0]) | +---------------------------------------+ | [2] | +---------------------------------------+ 1 row in set (0.00 sec) ``` * Examples using lambda functions ```sql -- Return the elements in x that are less than the elements in y. select array_filter((x,y) -> x < y, [1,2,null], [4,5,6]); +--------------------------------------------------------+ | array_filter((x, y) -> x < y, [1, 2, NULL], [4, 5, 6]) | +--------------------------------------------------------+ | [1,2] | +--------------------------------------------------------+ ``` ```sql -- Return the elements in x that are greater than or equal to 3. SELECT array_filter(x->x>=3, [2,3,4]); +--------------------------------------+ | array_filter(x -> x >= 3, [2, 3, 4]) | +--------------------------------------+ | [3,4] | +--------------------------------------+ ``` --- ### array_flatten `array_flatten` flattens one layer of nested arrays. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_flatten(param) ``` #### Parameters[​](#parameters "Direct link to Parameters") `param`: a nested array that needs to be flattened. Only nested arrays are supported, and it can be a multi-level nested array, the elements of the array can be any data type supported by StarRocks. #### Return value[​](#return-value "Direct link to Return value") The data type of the return value is an array type after flattening one layer. #### Examples[​](#examples "Direct link to Examples") Example 1: Flatten a 2-level nested array. ```plaintext mysql> SELECT array_flatten([[1, 2], [1, 4]]) as res; +-----------+ | res | +-----------+ | [1,2,1,4] | +-----------+ ``` Example 2: Flatten a 3-level nested array. ```plaintext mysql> SELECT array_flatten([[[1],[2]], [[3],[4]]]) as res; +-------------------+ | res | +-------------------+ | [[1],[2],[3],[4]] | +-------------------+ ``` --- ### array_generate Returns an array of distinct values within the range specified by `start` and `end`, with an increment of `step`. This function is supported from v3.1. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext ARRAY array_generate([start,] end [, step]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `start`: optional, the start value. It must be a constant or a column that evaluates to TINYINT, SMALLINT, INT, BIGINT, or LARGEINT. The default value is 1. * `end`: required, the end value. It must be a constant or a column that evaluates to TINYINT, SMALLINT, INT, BIGINT, or LARGEINT. * `step`: optional, the increment. It must be a constant or a column that evaluates to TINYINT, SMALLINT, INT, BIGINT, or LARGEINT. When `start` is less than `end`, the default value is 1. When `start` is greater than `end`, the default value is -1. #### Return value[​](#return-value "Direct link to Return value") Returns an array whose elements have the same data type as the input parameters. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * If any input parameter is a column, you must specify the table to which the column belongs. * If any input parameter is a column, you must specify other parameters. Default values are not supported. * If any input parameter is NULL, NULL is returned. * If `step` is 0, an empty array is returned. * If `start` equals to `end`, the value is returned. #### Examples[​](#examples "Direct link to Examples") ##### The input parameters are constants[​](#the-input-parameters-are-constants "Direct link to The input parameters are constants") ```plain mysql> select array_generate(9); +---------------------+ | array_generate(9) | +---------------------+ | [1,2,3,4,5,6,7,8,9] | +---------------------+ select array_generate(9,12); +-----------------------+ | array_generate(9, 12) | +-----------------------+ | [9,10,11,12] | +-----------------------+ select array_generate(9,6); +----------------------+ | array_generate(9, 6) | +----------------------+ | [9,8,7,6] | +----------------------+ select array_generate(9,6,-1); +--------------------------+ | array_generate(9, 6, -1) | +--------------------------+ | [9,8,7,6] | +--------------------------+ select array_generate(3,3); +----------------------+ | array_generate(3, 3) | +----------------------+ | [3] | +----------------------+ ``` ##### One of the input parameter is a column[​](#one-of-the-input-parameter-is-a-column "Direct link to One of the input parameter is a column") ```sql CREATE TABLE `array_generate` ( `c1` TINYINT, `c2` SMALLINT, `c3` INT ) ENGINE = OLAP DUPLICATE KEY(`c1`) DISTRIBUTED BY HASH(`c1`); INSERT INTO `array_generate` VALUES (1, 6, 3), (2, 9, 4); ``` ```plain mysql> select array_generate(1,c2,2) from `array_generate`; +--------------------------+ | array_generate(1, c2, 2) | +--------------------------+ | [1,3,5] | | [1,3,5,7,9] | +--------------------------+ ``` --- ### array_intersect Returns an array of the elements in the intersection of one or more arrays. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_intersect(input0, input1, ...) ``` #### Parameters[​](#parameters "Direct link to Parameters") `input`: one or more arrays whose intersection you want to obtain. Specify arrays in the `(input0, input1, ...)` format and make sure that the arrays that you specify are of the same data type. #### Return value[​](#return-value "Direct link to Return value") Returns an array of the same data type as the arrays that you specify. #### Examples[​](#examples "Direct link to Examples") Example 1: ```plain mysql> SELECT array_intersect(["SQL", "storage"], ["mysql", "query", "SQL"], ["SQL"]) AS no_intersect ; +--------------+ | no_intersect | +--------------+ | ["SQL"] | +--------------+ ``` Example 2: ```plain mysql> SELECT array_intersect(["SQL", "storage"], ["mysql", null], [null]) AS no_intersect ; +--------------+ | no_intersect | +--------------+ | [] | +--------------+ ``` Example 3: ```plain mysql> SELECT array_intersect(["SQL", null, "storage"], ["mysql", null], [null]) AS no_intersect ; +--------------+ | no_intersect | +--------------+ | [null] | +--------------+ ``` --- ### array_join Concatenates the elements of an array into a string. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_join(array, sep[, null_replace_str]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `array`: the array whose elements you want to concatenate. Only the ARRAY data type is supported. * `sep`: the delimiter that is used to separate the concatenated array elements. Only the VARCHAR data type is supported. * `null_replace_str`: the string that is used to substitute `NULL` values. Only the VARCHAR data type is supported. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR data type. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The value of the `array` parameter must be a one-dimensional array. * The `array` parameter does not support DECIMAL values. * If you set the `sep` parameter to `NULL`, the return value is `NULL`. * If you do not specify the `null_replace_str` parameter, `NULL` values are discarded. * If you set the `null_replace_str` parameter to `NULL`, the return value is `NULL`. #### Examples[​](#examples "Direct link to Examples") Example 1: Concatenate the elements of an array. In this example, the `NULL` value in the array is discarded, and the concatenated array elements are separated by underscores (`_`). ```plaintext mysql> select array_join([1, 3, 5, null], '_'); +-------------------------------+ | array_join([1,3,5,NULL], '_') | +-------------------------------+ | 1_3_5 | +-------------------------------+ ``` Example 2: Concatenate the elements of an array. In this example, the `NULL` value in the array is substituted with `NULL` strings, and the concatenated array elements are separated by underscores (`_`). ```plaintext mysql> select array_join([1, 3, 5, null], '_', 'NULL'); +---------------------------------------+ | array_join([1,3,5,NULL], '_', 'NULL') | +---------------------------------------+ | 1_3_5_NULL | +---------------------------------------+ ``` --- ### array_length Returns the number of elements in an array. The result type is INT. If the input parameter is NULL, the result is also NULL. Null elements are counted in the length. It has an alias [cardinality()](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/cardinality.md). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT array_length(any_array) ``` #### Parameters[​](#parameters "Direct link to Parameters") `any_array`: the ARRAY value from which you want to retrieve the number of elements. #### Return value[​](#return-value "Direct link to Return value") Returns an INT value. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select array_length([1,2,3]); +-----------------------+ | array_length([1,2,3]) | +-----------------------+ | 3 | +-----------------------+ 1 row in set (0.00 sec) mysql> select array_length([1,2,3,null]); +-------------------------------+ | array_length([1, 2, 3, NULL]) | +-------------------------------+ | 4 | +-------------------------------+ mysql> select array_length([[1,2], [3,4]]); +-----------------------------+ | array_length([[1,2],[3,4]]) | +-----------------------------+ | 2 | +-----------------------------+ 1 row in set (0.01 sec) ``` #### keywords[​](#keywords "Direct link to keywords") ARRAY\_LENGTH, ARRAY, CARDINALITY --- ### array_map array\_map() is a higher-order SQL function that can take lambda functions as arguments.It applies the lambda function to the input arrays and returns a new array. array\_map() can accept multiple arrays and can be nested with lambda functions to capture a variable. For more information about lambda functions, see [Lambda expression](https://docs.starrocks.io/docs/sql-reference/sql-functions/Lambda_expression.md). This function is supported from v2.5. The alias is transform(). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_map(lambda_function, arr1, arr2...) array_map(arr1, arr2..., lambda_function) ``` #### Usage notes[​](#usage-notes "Direct link to Usage notes") * Lambda functions can only be used as the **first** or **last** parameter of a higher-order function. Either position does not affect the calculation. * The number of arrays must be the same as the input parameters in the lambda function. Otherwise, an error is returned. * All the arrays must have the same number of elements. Otherwise, an error is returned. #### Examples[​](#examples "Direct link to Examples") Example 1: Increment every element of the array by 100. ```plain select array_map(x -> x+100,[1,2,3]); +------------------------------------+ | array_map(x -> x + 100, [1, 2, 3]) | +------------------------------------+ | [101,102,103] | +------------------------------------+ ``` Example 2: Sum array1 \[1,2,3] and array2 \[11,12,13] element by element. ```plain select array_map((x,y) -> x + y, [1,2,3], [11,12,13]); +-----------------------------------------------------+ | array_map((x, y) -> x + y, [1, 2, 3], [11, 12, 13]) | +-----------------------------------------------------+ | [12,14,16] | +-----------------------------------------------------+ ``` Example 3: Determine whether the element in `x` is greater than 1.5. If yes, log(x) is returned for the element. If not, (x+y) is returned for the corresponding element. ```plain select array_map((x,y) -> if(x>1.5,log(x),x+y), [1,2,3], [11,12,13]); +--------------------------------------------------------------------------+ | array_map((x, y) -> if(x > 1.5, log(x), x + y), [1, 2, 3], [11, 12, 13]) | +--------------------------------------------------------------------------+ | [12,0.6931471805599453,1.0986122886681098] | +--------------------------------------------------------------------------+ ``` Example 4: Use array\_map() to capture variables. The returned element whose value is 1 indicates the condition is met. ```plain -- In the example table, last_avg is the average score in the last exam. score represents the scores of three subjects in the current exam. +------+----------+------------+ | id | last_avg | score | +------+----------+------------+ | 1 | 55 | [50,60,70] | | 2 | 73 | [70,65,75] | | 3 | 89 | [88,92,90] | +------+----------+------------+ -- Find the score that is higher than the average score of the last exam. select array_map(x -> x > last_avg, score) from test_tbl; +--------------------------------------+ | array_map(x -> x > last_avg, score) | +--------------------------------------+ | [0,1,1] | | [0,0,1] | | [0,1,1] | +--------------------------------------+ ``` Example 5: Use nested lambda functions. ```plain select array_map(x -> array_map(x->x+100, x),[[1,2.3],[4,3,2]]); +-------------------------------------------------------------------+ | array_map(x -> array_map(x -> x + 100, x), [[1, 2.3], [4, 3, 2]]) | +-------------------------------------------------------------------+ | [[101,102.3],[104,103,102]] | +-------------------------------------------------------------------+ ``` Example 6: An error is returned because the numbers of arrays and lambda parameters are inconsistent. The lambda function requires only one array but two arrays are passed in. ```plain select array_map(x -> x,[1],[2,4]); ERROR 1064 (HY000): Lambda arguments should equal to lambda input arrays. ``` --- ### array_max Obtains the maximum value of all data in an ARRAY and return this result. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_max(array(type)) ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select array_max([113, 11, 12]); +------------------------+ | array_max([113,11,12]) | +------------------------+ | 113 | +------------------------+ mysql> select array_max([11.33, 11.11, 12.324]); +---------------------------------+ | array_max([11.33,11.11,12.324]) | +---------------------------------+ | 12.324 | +---------------------------------+ mysql> select array_max([cast('2020-02-25 11:35:17' as datetime), cast('2019-08-25 17:07:10' as datetime), cast('2025-08-25 17:07:10' as datetime)]); +--------------------------------------------------------------------------------------------------------------------------------------+ | array_max([CAST('2020-02-25 11:35:17' AS DATETIME),CAST('2019-08-25 17:07:10' AS DATETIME),CAST('2025-08-25 17:07:10' AS DATETIME)]) | +--------------------------------------------------------------------------------------------------------------------------------------+ | 2025-08-25 17:07:10 | +--------------------------------------------------------------------------------------------------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ARRAY\_MAX,ARRAY --- ### array_min Obtains the minimum value of all data in an ARRAY and return this result. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_min(array(type)) ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select array_min([113, 11, 12]); +------------------------+ | array_min([113,11,12]) | +------------------------+ | 11 | +------------------------+ mysql> select array_min([11.33, 11.11, 12.324]); +---------------------------------+ | array_min([11.33,11.11,12.324]) | +---------------------------------+ | 11.11 | +---------------------------------+ mysql> select array_min([cast('2020-02-25 11:35:17' as datetime), cast('2019-08-25 17:07:10' as datetime), cast('2025-08-25 17:07:10' as datetime)]); +--------------------------------------------------------------------------------------------------------------------------------------+ | array_min([CAST('2020-02-25 11:35:17' AS DATETIME),CAST('2019-08-25 17:07:10' AS DATETIME),CAST('2025-08-25 17:07:10' AS DATETIME)]) | +--------------------------------------------------------------------------------------------------------------------------------------+ | 2019-08-25 17:07:10 | +--------------------------------------------------------------------------------------------------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ARRAY\_MIN,ARRAY --- ### array_position Obtains the position of an element in an array. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_position(any_array, any_element) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `any_array`: the array to be searched. * `any_element`: an expression that matches the element in an array. #### Return value[​](#return-value "Direct link to Return value") Returns the position of the specified element if the element can be found in the specified array and returns 0 if the element cannot be found in the array. #### Examples[​](#examples "Direct link to Examples") * Example 1: Obtain the position of an element in an array. ```plaintext mysql> select array_position(["apple","orange","pear"], "orange"); +-----------------------------------------------------+ | array_position(['apple','orange','pear'], 'orange') | +-----------------------------------------------------+ | 2 | +-----------------------------------------------------+ 1 row in set (0.01 sec) ``` * Example 2: Obtain the position of `NULL` in an array. ```sql mysql> select array_position([1, NULL], NULL); ``` ```plaintext +--------------------------------+ | array_position([1,NULL], NULL) | +--------------------------------+ | 2 | +--------------------------------+ 1 row in set (0.00 sec) ``` * Example 3: Obtain the position of a sub-array in a multi-dimensional array. A position can be returned only when a sub-array that consists of the same elements in the same order as the specified sub-array can be found in the multi-dimensional array. ```lua mysql> select array_position([[1,2,3], [4,5,6]], [4,5,6]); +--------------------------------------------+ | array_position([[1,2,3],[4,5,6]], [4,5,6]) | +--------------------------------------------+ | 2 | +--------------------------------------------+ 1 row in set (0.00 sec) mysql> select array_position([[1,2,3], [4,5,6]], [4,6,5]); +--------------------------------------------+ | array_position([[1,2,3],[4,5,6]], [4,6,5]) | +--------------------------------------------+ | 0 | +--------------------------------------------+ 1 row in set (0.00 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") ARRAY\_POSITION, ARRAY --- ### array_remove Removes an element from an array. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_remove(any_array, any_element) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `any_array`: the array to be searched. * `any_element`: an expression that matches the element in an array. #### Return value[​](#return-value "Direct link to Return value") Returns the array from which specified element has been removed. #### Examples[​](#examples "Direct link to Examples") ```plaintext mysql> select array_remove([1,2,3,null,3], 3); +---------------------------------+ | array_remove([1,2,3,NULL,3], 3) | +---------------------------------+ | [1,2,null] | +---------------------------------+ 1 row in set (0.01 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") ARRAY\_REMOVE, ARRAY --- ### array_repeat `array_repeat` returns an array containing a given element repeated a specified number of times. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_repeat(element, count) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `element`: The element to be repeated can be any data type supported by StarRocks. * `count`: The number of repetitions, of type INT. #### Return value[​](#return-value "Direct link to Return value") The data type of the return value is the ARRAY type of element. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * When count is less than 1, an empty array is returned. * When the element parameter is NULL, the result is an array consisting of count NULLs. * When the count parameter is NULL, the result is NULL. #### Examples[​](#examples "Direct link to Examples") Example 1: ```sql select array_repeat(1,5) as res; ``` ```sql +-------------+ | res | +-------------+ | [1,1,1,1,1] | +-------------+ ``` Example 2: ```sql select array_repeat([1,2],3) as res; ``` ```sql +---------------------+ | res | +---------------------+ | [[1,2],[1,2],[1,2]] | +---------------------+ ``` Example 3: ```sql select array_repeat(1,-1) as res; ``` ```sql +------+ | res | +------+ | [] | +------+ ``` Example 4: ```sql select array_repeat(null,3) as res; ``` ```sql +------+ | res | +------+ | NULL | +------+ ``` Example 5: ```sql CREATE TABLE IF NOT EXISTS test (COLA INT, COLB INT) PROPERTIES ("replication_num"="1"); INSERT INTO test (COLA, COLB) VALUES (1, 3), (NULL, 3), (2, NULL); select array_repeat(COLA,COLB) from test; ``` ```sql +--------------------------+ | array_repeat(COLA, COLB) | +--------------------------+ | [1,1,1] | | [null,null,null] | | NULL | +--------------------------+ ``` --- ### array_slice Returns a slice of an array. This function intercepts `length` elements from `input` from the position specified by `offset`. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_slice(input, offset, length) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `input`: the array whose slice you want to intercept. This function supports the following types of array elements: BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, VARCHAR, DECIMALV2, DATETIME, DATE, and JSON. **JSON is supported from 2.5.** * `offset`: the position from which to intercept elements. Valid values start from `1`. It must be a BIGINT value. * `length`: the length of the slice that you want to intercept. It must be a BIGINT value. #### Return value[​](#return-value "Direct link to Return value") Returns an array that have the same data type as the array specified by the `input` parameter. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The offset starts from 1. * If the specified length exceeds the actual number of elements that can be intercepted, all the matching elements are returned. See Example 4. #### Examples[​](#examples "Direct link to Examples") Example 1: Intercept 2 elements starting from the third element. ```plain mysql> select array_slice([1,2,4,5,6], 3, 2) as res; +-------+ | res | +-------+ | [4,5] | +-------+ ``` Example 2: Intercept 2 elements starting from the first element. ```plain mysql> select array_slice(["sql","storage","query","execute"], 1, 2) as res; +-------------------+ | res | +-------------------+ | ["sql","storage"] | +-------------------+ ``` Example 3: Null elements are treated as normal values. ```plain mysql> select array_slice([57.73,97.32,128.55,null,324.2], 3, 3) as res; +---------------------+ | res | +---------------------+ | [128.55,null,324.2] | +---------------------+ ``` Example 4: Intercept 5 elements starting from the third element. This function intends to intercept 5 elements but there are only 3 elements from the third element. As a result, all of these 3 elements are returned. ```plain mysql> select array_slice([57.73,97.32,128.55,null,324.2], 3, 5) as res; +---------------------+ | res | +---------------------+ | [128.55,null,324.2] | +---------------------+ ``` --- ### array_sort Sorts the elements of an array in ascending order. From v4.0, `array_sort` supports Lambda Comparator. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext -- Without Lambda Comparator array_sort(array) -- With Lambda Comparator array_sort(array, (x,y)->expr(x,y)) ``` #### Parameters[​](#parameters "Direct link to Parameters") `array`: the array whose elements you want to sort. Only the ARRAY data type is supported. `lambda_comparator`: an optional lambda function comparator with format `(x,y)->expr(x,y)`. The lambda comparator: * Must only depend on arguments x and y * Must return boolean or numeric type * Result should not be NULL * Must satisfy strict weak ordering requirements #### Return value[​](#return-value "Direct link to Return value") Returns an array. #### Usage notes[​](#usage-notes "Direct link to Usage notes") ##### Basic Sorting[​](#basic-sorting "Direct link to Basic Sorting") * This function sorts the elements of an array in ascending order when used without a lambda comparator. * `NULL` values are placed at the beginning of the array that is returned. * The elements of the returned array have the same data type as the elements of the input array. ##### Lambda Comparator Sorting[​](#lambda-comparator-sorting "Direct link to Lambda Comparator Sorting") * When using a lambda comparator, the sorting order is determined by the expression result: * For boolean return: `true` means x should come before y * For numeric return: negative means x comes before y * The lambda comparator must satisfy strict weak ordering requirements: * **Irreflexivity**: For all `x`, `expr(x, x)` must return false (boolean) or non-negative (numeric) * **Asymmetry**: If `expr(x, y)` holds, then `expr(y, x)` must not hold * **Transitivity**: If `expr(x, y)` and `expr(y, z)` hold, then `expr(x, z)` must hold * **Connectedness**: For all `x` and `y`, one of the following must be true: * `expr(x, y)` * `expr(y, x)` * `x = y` * The lambda comparator expression must: * Only depend on parameters `x` and `y` * Return a boolean or numeric type * Never return NULL * Always produce consistent results for the same input values ##### Descending Order[​](#descending-order "Direct link to Descending Order") If you want to sort the elements of an array in descending order, use the [reverse](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/reverse.md) function. #### Examples[​](#examples "Direct link to Examples") ##### Basic Examples[​](#basic-examples "Direct link to Basic Examples") The following table is used as an example: ```plaintext mysql> select * from test; +------+--------------+ | c1 | c2 | +------+--------------+ | 1 | [4,3,null,1] | | 2 | NULL | | 3 | [null] | | 4 | [8,5,1,4] | +------+--------------+ ``` Sort the values of column `c2` in ascending order. ```plaintext mysql> select c1, array_sort(c2) from test; +------+------------------+ | c1 | array_sort(`c2`) | +------+------------------+ | 1 | [null,1,3,4] | | 2 | NULL | | 3 | [null] | | 4 | [1,4,5,8] | +------+------------------+ ``` ##### Lambda Comparator Examples[​](#lambda-comparator-examples "Direct link to Lambda Comparator Examples") Sort an array of integers in descending order using a lambda comparator: ```plaintext mysql> select array_sort([3,1,4,2], (x,y) -> x > y); +-------------------------------------------+ | array_sort([3, 1, 4, 2], (x, y) -> x > y) | +-------------------------------------------+ | [4, 3, 2, 1] | +-------------------------------------------+ ``` Sort an array of integers using numeric return values: ```plaintext mysql> select array_sort([3,1,4,2], (x,y) -> x - y); +-------------------------------------------+ | array_sort([3, 1, 4, 2], (x, y) -> x - y) | +-------------------------------------------+ | [1, 2, 3, 4] | +-------------------------------------------+ ``` Sort an array of strings by length: ```plaintext mysql> select array_sort(['apple', 'banana', 'cherry', 'date'], (x,y) -> length(x) - length(y)); +------------------------------------------------------------------------------------+ | array_sort(['apple', 'banana', 'cherry', 'date'], (x, y) -> length(x) - length(y)) | +------------------------------------------------------------------------------------+ | ['date', 'apple', 'banana', 'cherry'] | +------------------------------------------------------------------------------------+ ``` Sort an array of strings alphabetically using boolean return: ```plaintext mysql> select array_sort(['banana', 'apple', 'cherry'], (x,y) -> x > y); +------------------------------------------------------------+ | array_sort(['banana', 'apple', 'cherry'], (x, y) -> x > y) | +------------------------------------------------------------+ | ['apple', 'banana', 'cherry'] | +------------------------------------------------------------+ ``` Sort an array with custom business logic (sort even numbers first, then odd numbers): ```plaintext mysql> select array_sort([5,2,8,1,9,4], (x,y) -> case when x % 2 = 0 and y % 2 = 0 then x - y when x % 2 = 0 and y % 2 = 1 then -1 when x % 2 = 1 and y % 2 = 0 then 1 else x - y end); +--------------------------------------------------------------------------------------------------------------------+ | array_sort([5, 2, 8, 1, 9, 4], (x, y) -> case when x % 2 = 0 and y % 2 = 0 then x - y when x % 2 = 0 and y % 2 = 1 then -1 when x % 2 = 1 and y % 2 = 0 then 1 else x - y end) | +--------------------------------------------------------------------------------------------------------------------+ | [2, 4, 8, 1, 5, 9] | +--------------------------------------------------------------------------------------------------------------------+ ``` Sort an array of decimals with custom precision comparison: ```plaintext mysql> select array_sort([3.141, 2.718, 1.414, 2.236], (x,y) -> round(x, 2) - round(y, 2)); +----------------------------------------------------------------------------------------+ | array_sort([3.141, 2.718, 1.414, 2.236], (x, y) -> round(x, 2) - round(y, 2)) | +----------------------------------------------------------------------------------------+ | [1.414, 2.236, 2.718, 3.141] | +----------------------------------------------------------------------------------------+ ``` --- ### array_sortby Sorts elements in an array according to the ascending order of elements in another array or array converted from a lambda expression. For more information, see [Lambda expression](https://docs.starrocks.io/docs/sql-reference/sql-functions/Lambda_expression.md). This function is supported from v2.5. Elements in the two arrays are like key-value pairs. For example, b = \[7,5,6] is the sorting key of a = \[3,1,4]. According to the key-value pair relationship, elements in the two arrays have the following one-to-one mapping. | **Array** | **Element 1** | **Element 2** | **Element 3** | | --------- | ------------- | ------------- | ------------- | | a | 3 | 1 | 4 | | b | 7 | 5 | 6 | After array `b` is sorted in ascending order, it becomes \[5,6,7]. Array `a` becomes \[1,4,3] accordingly. | **Array** | **Element 1** | **Element 2** | **Element 3** | | --------- | ------------- | ------------- | ------------- | | a | 1 | 4 | 3 | | b | 5 | 6 | 7 | when the number of arrays involved in sorting is greater than two，The purpose of this function is to sort array0 based on the values in multiple array columns (such as array1, array2, array3, etc.). The sorting rules are as follows: First, compare the corresponding elements in array1; If they are the same, compare the corresponding elements in array2; And so on, until the last array column. **Note:** The elements in the arrays participating in the sort must be of a sortable type or JSON type. The size of the arrays participating in the sort must be consistent with the original array (except for NULL values). **Example Explanation** Given the following four arrays: ```text array0 = [1, 2, 3, 4, 5] array1 = [6, 5, 5, 5, 4] array2 = ["d", "b", "a", "b", "4"] array3 = ["2023-01-01", "2023-01-04", "2023-01-03", "2023-01-05", "2023-01-02"] ``` Sorting Steps: 1. Compare array1: * The sorted index order of array1 is \[4, 2, 3, 1, 0] because 4 < 5 < 5 < 5 < 6. 2. For the elements \[5, 5, 5] in array1, compare array2: * The sorted order of \[a, b, b] in array2 is \[2, 3, 4] because a < b = b. 3. For the elements \[b, b] in array1 and array2, compare array3: * The sorted order of \[2023-01-04, 2023-01-05] in array3 is \[2, 4] because 2023-01-04 < 2023-01-05. Final Sorting Result for array0: ```text array0 = [5, 3, 2, 4, 1] ``` #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_sortby(array0, array1) array_sortby(, array0 [, array1...]) array_sortby(array0, array1, [array2, array3...]) ``` * `array_sortby(array0, array1)` Sorts `array0` according to the ascending order of `array1`. * `array_sortby(, array0 [, array1...])` Sorts `array0` according to the array returned from the lambda function. * `array_sortby(array0, array1, [array2, array3...])` Sort array0 in ascending order based on the values of multiple array columns (such as array1, array2, array3, etc.). The sorting rule is: first, compare the corresponding elements of array1; if they are the same, compare the corresponding elements of array2; and so on, until the last array column. #### Parameters[​](#parameters "Direct link to Parameters") * `array0`: the array you want to sort. It must be an array, array expression, or `null`. Elements in the array must be sortable. * `array1`: the sorting array used to sort `array0`. It must be an array, array expression, or `null`. * `lambda function`: the lambda expression used to generate the sorting array. * `array1, [array2, array3...]`：the sorting array used to sort `array0`. It must be an array, array expression, or `null`. #### Return value[​](#return-value "Direct link to Return value") Returns an array. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * This function can sort elements of an array only in ascending order. * `NULL` values are placed at the beginning of the array that is returned. * If you want to sort elements of an array in descending order, use the [reverse](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/reverse.md) function. * If the sorting array (`array1`) is null, data in `array0` remains unchanged. * The elements of the returned array have the same data type as the elements of `array0`. The attribute of null values are also the same. * All arrays must have the same number of elements. Otherwise, an error is returned. #### Examples[​](#examples "Direct link to Examples") The following table is used to demonstrate how to use this function. ```sql CREATE TABLE `test_array` ( `c1` int(11) NULL COMMENT "", `c2` ARRAY NULL COMMENT "", `c3` ARRAY NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`c1`) COMMENT "OLAP" DISTRIBUTED BY HASH(`c1`) PROPERTIES ( "replication_num" = "3", "storage_format" = "DEFAULT", "enable_persistent_index" = "true", "compression" = "LZ4" ); insert into test_array values (1,[4,3,5],[82,1,4]), (2,null,[23]), (3,[4,2],[6,5]), (4,null,null), (5,[],[]), (6,NULL,[]), (7,[],null), (8,[null,null],[3,6]), (9,[432,21,23],[5,4,null]); select * from test_array order by c1; +------+-------------+------------+ | c1 | c2 | c3 | +------+-------------+------------+ | 1 | [4,3,5] | [82,1,4] | | 2 | NULL | [23] | | 3 | [4,2] | [6,5] | | 4 | NULL | NULL | | 5 | [] | [] | | 6 | NULL | [] | | 7 | [] | NULL | | 8 | [null,null] | [3,6] | | 9 | [432,21,23] | [5,4,null] | +------+-------------+------------+ 9 rows in set (0.00 sec) ``` Example 1: Sort `c3` according to `c2`. This example also provides the result of array\_sort() for comparison. ```plaintext select c1, c3, c2, array_sort(c2), array_sortby(c3,c2) from test_array order by c1; +------+------------+-------------+----------------+----------------------+ | c1 | c3 | c2 | array_sort(c2) | array_sortby(c3, c2) | +------+------------+-------------+----------------+----------------------+ | 1 | [82,1,4] | [4,3,5] | [3,4,5] | [1,82,4] | | 2 | [23] | NULL | NULL | [23] | | 3 | [6,5] | [4,2] | [2,4] | [5,6] | | 4 | NULL | NULL | NULL | NULL | | 5 | [] | [] | [] | [] | | 6 | [] | NULL | NULL | [] | | 7 | NULL | [] | [] | NULL | | 8 | [3,6] | [null,null] | [null,null] | [3,6] | | 9 | [5,4,null] | [432,21,23] | [21,23,432] | [4,null,5] | +------+------------+-------------+----------------+----------------------+ ``` Example 2: Sort array `c3` based on `c2` generated from a lambda expression. This example is equivalent to Example 1. It also provides the result of array\_sort() for comparison. ```plaintext select c1, c3, c2, array_sort(c2) as sorted_c2_asc, array_sortby((x,y) -> y, c3, c2) as sorted_c3_by_c2 from test_array order by c1; +------+------------+-------------+---------------+-----------------+ | c1 | c3 | c2 | sorted_c2_asc | sorted_c3_by_c2 | +------+------------+-------------+---------------+-----------------+ | 1 | [82,1,4] | [4,3,5] | [3,4,5] | [82,1,4] | | 2 | [23] | NULL | NULL | [23] | | 3 | [6,5] | [4,2] | [2,4] | [5,6] | | 4 | NULL | NULL | NULL | NULL | | 5 | [] | [] | [] | [] | | 6 | [] | NULL | NULL | [] | | 7 | NULL | [] | [] | NULL | | 8 | [3,6] | [null,null] | [null,null] | [3,6] | | 9 | [5,4,null] | [432,21,23] | [21,23,432] | [4,null,5] | +------+------------+-------------+---------------+-----------------+ ``` Example 3: Sort array `c3` based on the ascending order of `c2+c3`. ```plain select c3, c2, array_map((x,y)-> x+y,c3,c2) as sum, array_sort(array_map((x,y)-> x+y, c3, c2)) as sorted_sum, array_sortby((x,y) -> x+y, c3, c2) as sorted_c3_by_sum from test_array where c1=1; +----------+---------+----------+------------+------------------+ | c3 | c2 | sum | sorted_sum | sorted_c3_by_sum | +----------+---------+----------+------------+------------------+ | [82,1,4] | [4,3,5] | [86,4,9] | [4,9,86] | [1,4,82] | +----------+---------+----------+------------+------------------+ ``` ```sql CREATE TABLE test_array_sortby_muliti ( id INT(11) not null, array_col1 ARRAY, array_col2 ARRAY, array_col3 ARRAY, array_col4 ARRAY ) ENGINE=OLAP DUPLICATE KEY(id) COMMENT "OLAP" DISTRIBUTED BY HASH(id) PROPERTIES ( "replication_num" = "1", "storage_format" = "DEFAULT", "enable_persistent_index" = "true", "compression" = "LZ4" ); INSERT INTO test_array_sortby_multi VALUES (1, [4, 3, 5], [1.1, 2.2, 2.2], ['a', 'b', 'c'], ['2023-01-01', '2023-01-02', '2023-01-03']), (2, [6, 7, 8], [6.6, 5.5, 6.6], ['d', 'e', 'd'], ['2023-01-04', '2023-01-05', '2023-01-06']), (3, NULL, [7.7, 8.8, 8.8], ['g', 'h', 'h'], ['2023-01-07', '2023-01-08', '2023-01-09']), (4, [9, 10, 11], NULL, ['k', 'k', 'j'], ['2023-01-10', '2023-01-12', '2023-01-11']), (5, [12, 13, 14], [10.10, 11.11, 11.11], NULL, ['2023-01-13', '2023-01-14', '2023-01-15']), (6, [15, 16, 17], [14.14, 13.13, 14.14], ['m', 'o', 'o'], NULL), (7, [18, 19, 20], [16.16, 16.16, 18.18], ['p', 'p', 'r'], ['2023-01-16', NULL, '2023-01-18']), (8, [21, 22, 23], [19.19, 20.20, 19.19], ['a', 't', 'a'], ['2023-01-19', '2023-01-20', '2023-01-21']), (9, [24, 25, 26], NULL, ['y', 'y', 'z'], ['2023-01-25', '2023-01-24', '2023-01-26']), (10, [24, 25, 26], NULL, ['y', 'y', 'z'], ['2023-01-25', NULL, '2023-01-26']); select * from test_array_sortby_multi order by id asc; +------+------------+---------------------+---------------+------------------------------------------+ | id | array_col1 | array_col2 | array_col3 | array_col4 | +------+------------+---------------------+---------------+------------------------------------------+ | 1 | [4,3,5] | [1.1,2.2,2.2] | ["a","b","c"] | ["2023-01-01","2023-01-02","2023-01-03"] | | 2 | [6,7,8] | [6.6,5.5,6.6] | ["d","e","d"] | ["2023-01-04","2023-01-05","2023-01-06"] | | 3 | NULL | [7.7,8.8,8.8] | ["g","h","h"] | ["2023-01-07","2023-01-08","2023-01-09"] | | 4 | [9,10,11] | NULL | ["k","k","j"] | ["2023-01-10","2023-01-12","2023-01-11"] | | 5 | [12,13,14] | [10.1,11.11,11.11] | NULL | ["2023-01-13","2023-01-14","2023-01-15"] | | 6 | [15,16,17] | [14.14,13.13,14.14] | ["m","o","o"] | NULL | | 7 | [18,19,20] | [16.16,16.16,18.18] | ["p","p","r"] | ["2023-01-16",null,"2023-01-18"] | | 8 | [21,22,23] | [19.19,20.2,19.19] | ["a","t","a"] | ["2023-01-19","2023-01-20","2023-01-21"] | | 9 | [24,25,26] | NULL | ["y","y","z"] | ["2023-01-25","2023-01-24","2023-01-26"] | | 10 | [24,25,26] | NULL | ["y","y","z"] | ["2023-01-25",null,"2023-01-26"] | +------+------------+---------------------+---------------+------------------------------------------+ ``` Example 1: Sort `array_col1` according to `array_col2`, `array_col3`. ```plaintext select id, array_col1, array_col2, array_col3, array_sortby(array_col1, array_col2, array_col3) from test_array_sortby_multi order by id asc; +------+------------+---------------------+---------------+--------------------------------------------------------+ | id | array_col1 | array_col2 | array_col3 | array_sortby(array_col1, array_col2, array_col3) | +------+------------+---------------------+---------------+--------------------------------------------------------+ | 1 | [4,3,5] | [1.1,2.2,2.2] | ["a","b","c"] | [4,3,5] | | 2 | [6,7,8] | [6.6,5.5,6.6] | ["d","e","d"] | [7,6,8] | | 3 | NULL | [7.7,8.8,8.8] | ["g","h","h"] | NULL | | 4 | [9,10,11] | NULL | ["k","k","j"] | [11,9,10] | | 5 | [12,13,14] | [10.1,11.11,11.11] | NULL | [12,13,14] | | 6 | [15,16,17] | [14.14,13.13,14.14] | ["m","o","o"] | [16,15,17] | | 7 | [18,19,20] | [16.16,16.16,18.18] | ["p","p","r"] | [18,19,20] | | 8 | [21,22,23] | [19.19,20.2,19.19] | ["a","t","a"] | [21,23,22] | | 9 | [24,25,26] | NULL | ["y","y","z"] | [24,25,26] | | 10 | [24,25,26] | NULL | ["y","y","z"] | [24,25,26] | +------+------------+---------------------+---------------+--------------------------------------------------------+ ``` Example 2: Sort `array_col1` according to `array_col2`, `array_col3`, `array_col4`. ```plaintext select id, array_col1, array_col2, array_col3, array_col4, array_sortby(array_col1, array_col2, array_col3, array_col4) from test_array_sortby_multi order by id asc; +------+------------+---------------------+---------------+------------------------------------------+--------------------------------------------------------------------+ | id | array_col1 | array_col2 | array_col3 | array_col4 | array_sortby(array_col1, array_col2, array_col3, array_col4) | +------+------------+---------------------+---------------+------------------------------------------+--------------------------------------------------------------------+ | 1 | [4,3,5] | [1.1,2.2,2.2] | ["a","b","c"] | ["2023-01-01","2023-01-02","2023-01-03"] | [4,3,5] | | 2 | [6,7,8] | [6.6,5.5,6.6] | ["d","e","d"] | ["2023-01-04","2023-01-05","2023-01-06"] | [7,6,8] | | 3 | NULL | [7.7,8.8,8.8] | ["g","h","h"] | ["2023-01-07","2023-01-08","2023-01-09"] | NULL | | 4 | [9,10,11] | NULL | ["k","k","j"] | ["2023-01-10","2023-01-12","2023-01-11"] | [11,9,10] | | 5 | [12,13,14] | [10.1,11.11,11.11] | NULL | ["2023-01-13","2023-01-14","2023-01-15"] | [12,13,14] | | 6 | [15,16,17] | [14.14,13.13,14.14] | ["m","o","o"] | NULL | [16,15,17] | | 7 | [18,19,20] | [16.16,16.16,18.18] | ["p","p","r"] | ["2023-01-16",null,"2023-01-18"] | [19,18,20] | | 8 | [21,22,23] | [19.19,20.2,19.19] | ["a","t","a"] | ["2023-01-19","2023-01-20","2023-01-21"] | [21,23,22] | | 9 | [24,25,26] | NULL | ["y","y","z"] | ["2023-01-25","2023-01-24","2023-01-26"] | [25,24,26] | | 10 | [24,25,26] | NULL | ["y","y","z"] | ["2023-01-25",null,"2023-01-26"] | [25,24,26] | +------+------------+---------------------+---------------+------------------------------------------+--------------------------------------------------------------------+ ``` #### References[​](#references "Direct link to References") [array\_sort](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_sort.md) --- ### array_sum Sums all the elements in an array. From StarRocks 2.5, array\_sum() can take a lambda expression as an argument. However, it cannot work directly with the Lambda expression. It must work on the result converted from [array\_map()](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_map.md). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext array_sum(array(type)) array_sum(lambda_function, arr1,arr2...) = array_sum(array_map(lambda_function, arr1,arr2...)) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `array(type)`: the array you want to calculate the sum. Array elements support the following data types: BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, and DECIMALV2. * `lambda_function`: the lambda expression used to calculate the target array for array\_sum(). #### Return value[​](#return-value "Direct link to Return value") Returns a numeric value. #### Examples[​](#examples "Direct link to Examples") ##### Use array\_sum without lambda function[​](#use-array_sum-without-lambda-function "Direct link to Use array_sum without lambda function") ```plain mysql> select array_sum([11, 11, 12]); +-----------------------+ | array_sum([11,11,12]) | +-----------------------+ | 34 | +-----------------------+ mysql> select array_sum([11.33, 11.11, 12.324]); +---------------------------------+ | array_sum([11.33,11.11,12.324]) | +---------------------------------+ | 34.764 | +---------------------------------+ ``` ##### Use array\_sum with lambda function[​](#use-array_sum-with-lambda-function "Direct link to Use array_sum with lambda function") ```plain -- Multiply [1,2,3] by [1,2,3] and sum the elements. select array_sum(array_map(x->x*x,[1,2,3])); +---------------------------------------------+ | array_sum(array_map(x -> x * x, [1, 2, 3])) | +---------------------------------------------+ | 14 | +---------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ARRAY\_SUM,ARRAY --- ### array_to_bitmap Converts an array into BITMAP values. This function is supported from v2.3. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP array_to_bitmap(array) ``` #### Parameters[​](#parameters "Direct link to Parameters") `array`: Elements in an array can be of the BIGINT, INT, TINYINT, or SMALLINT type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BITMAP type. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * If the data type of elements in the input array is invalid, such as STRING or DECIMAL, an error is returned. * If an empty array is entered, an empty BITMAP value is returned. * If `NULL` is entered, `NULL` is returned. #### Examples[​](#examples "Direct link to Examples") Example 1: Convert an array into BITMAP values. This function must be nested in `bitmap_to_array` because BITMAP values cannot be displayed. ```plain MySQL > select bitmap_to_array(array_to_bitmap([1,2,3])); +-------------------------------------------+ | bitmap_to_array(array_to_bitmap([1,2,3])) | +-------------------------------------------+ | [1,2,3] | +-------------------------------------------+ ``` Example 2: Enter an empty array and an empty array is returned. ```plain MySQL > select bitmap_to_array(array_to_bitmap([])); +--------------------------------------+ | bitmap_to_array(array_to_bitmap([])) | +--------------------------------------+ | [] | +--------------------------------------+ ``` Example 3: Enter `NULL` and `NULL` is returned. ```plain MySQL > select array_to_bitmap(NULL); +-----------------------+ | array_to_bitmap(NULL) | +-----------------------+ | NULL | +-----------------------+ ``` --- ### array_top_n Sorts the elements of a given array in their natural descending order, and returns the top-n elements in an array. If n is larger than the length of the input array, the length of the returned array will be the same as the input instead of n. #### Syntax[​](#syntax "Direct link to Syntax") ```sql array_top_n(array, count) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `array`: The array from which to extract the top-n elements. The array can contain elements of any comparable data type supported by StarRocks. * `count`: The number of top elements to return. It must be a non-negative integer. #### Return value[​](#return-value "Direct link to Return value") Returns an array containing the top-n elements from the input array, sorted in descending order. The data type of the return value is the same ARRAY type as the input array. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * Elements are sorted in descending order according to their natural ordering. * If `count` is 0 or negative, an empty array is returned. * If `count` is larger than the array size, the entire array, sorted in descending order, is returned. * If the input `array` or `count` is NULL, the result is NULL. * NULL elements within the array are considered smaller than any non-NULL value and will appear at the end if included in the result. #### Examples[​](#examples "Direct link to Examples") Example 1: Get the top 3 elements from an integer array. ```sql mysql> SELECT array_top_n([1, 100, 2, 5, 3], 3); +-----------------------------------+ | array_top_n([1,100,2,5,3], 3) | +-----------------------------------+ | [100,5,3] | +-----------------------------------+ ``` Example 2: Get the top 5 elements from an array of two integers (`count` is larger than the array length). ```sql mysql> SELECT array_top_n([1, 100], 5); +--------------------------+ | array_top_n([1,100], 5) | +--------------------------+ | [100,1] | +--------------------------+ ``` Example 3: Get the top 3 elements from a string array. ```sql mysql> SELECT array_top_n(['a', 'zzz', 'zz', 'b', 'g', 'f'], 3); +----------------------------------------------------+ | array_top_n(['a','zzz','zz','b','g','f'], 3) | +----------------------------------------------------+ | ['zzz','zz','g'] | +----------------------------------------------------+ ``` Example 4: Call the function when `count` is 0. ```sql mysql> SELECT array_top_n([1, 2, 3], 0); +--------------------------+ | array_top_n([1,2,3], 0) | +--------------------------+ | [] | +--------------------------+ ``` Example 5: Get the top 3 elements from an integer array where NULL is one of the elements. ```sql mysql> SELECT array_top_n([1, NULL, 3, 2], 3); +---------------------------------+ | array_top_n([1,null,3,2], 3) | +---------------------------------+ | [3,2,1] | +---------------------------------+ ``` Example 6: Call the function with table data. ```sql mysql> CREATE TABLE IF NOT EXISTS test (id INT, arr ARRAY) PROPERTIES ("replication_num"="1"); mysql> INSERT INTO test VALUES (1, [5, 2, 8, 1, 9]), (2, [10, 3, 7]); mysql> SELECT id, array_top_n(arr, 2) FROM test; +------+----------------------+ | id | array_top_n(arr, 2) | +------+----------------------+ | 1 | [9,8] | | 2 | [10,7] | +------+----------------------+ ``` --- ### array_unique_agg Aggregates distinct values (including `NULL`) in an ARRAY column into an array (from multiple rows to one row). This function is supported from v3.2. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext ARRAY_UNIQUE_AGG(col) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `col`: the column whose values you want to aggregate. Supported data type is ARRAY. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the ARRAY type. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The order of the elements in an array is random. * The data type of the elements in the returned array is the same as the data type of the elements in the input column. * Returns `NULL` if there is no matched value. #### Examples[​](#examples "Direct link to Examples") Take the following data table as an example: ```plaintext mysql > select * from array_unique_agg_example; +------+--------------+ | a | b | +------+--------------+ | 2 | [1,null,2,4] | | 2 | [1,null,3] | | 1 | [1,1,2,3] | | 1 | [2,3,4] | +------+--------------+ ``` Example 1: Group the values in column `a` and aggregate distinct values in column `b` into an array. ```plaintext mysql > select a, array_unique_agg(b) from array_unique_agg_example group by a; +------+---------------------+ | a | array_unique_agg(b) | +------+---------------------+ | 1 | [4,1,2,3] | | 2 | [4,1,2,3,null] | +------+---------------------+ ``` Example 2: Aggregate values in column `b` using the WHERE clause. If no data meets the filter condition, a `NULL` value is returned. ```plaintext mysql > select array_unique_agg(b) from array_unique_agg_example where a < 0; +---------------------+ | array_unique_agg(b) | +---------------------+ | NULL | +---------------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") ARRAY\_UNIQUE\_AGG, ARRAY --- ### arrays_overlap Checks whether the intersection of two arrays is empty. The arrays that you specify must be of the same data type. If the intersection is empty, this function returns `0`. Otherwise, this function returns `1`. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext arrays_overlap(input0, input1) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `input0`: one of the two arrays that you want to compare. * `input1`: the other of the two arrays that you want to compare. The value of this parameter must be of the same data type as the value of the `input0` parameter. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BOOLEAN data type. #### Examples[​](#examples "Direct link to Examples") Example 1: ```plain mysql> select arrays_overlap([11, 9, 3, 2], [null, 11]); +--------------------------------------+ | arrays_overlap([11,9,3,2], [NULL,11]) | +--------------------------------------+ | 1 | +--------------------------------------+ ``` Example 2: ```plain mysql> select arrays_overlap([9, 3, 2], [null, 11]); +-----------------------------------+ | arrays_overlap([9,3,2], [NULL,11]) | +-----------------------------------+ | 0 | +-----------------------------------+ ``` Example 3: ```plain mysql> select arrays_overlap([9, 3, null, 2], [null, 11]); +----------------------------------------+ | arrays_overlap([9,3,NULL,2], [NULL,11]) | +----------------------------------------+ | 1 | +----------------------------------------+ ``` Example 4: ```plain mysql> select arrays_overlap([9, 3, "SQL", 2], [null, "SQL"]); +--------------------------------------------+ | arrays_overlap([9,3,'SQL',2], [NULL,'SQL']) | +--------------------------------------------+ | 1 | +--------------------------------------------+ ``` --- ### arrays_zip Merges the given arrays by index into an array of structs in which the n-th struct contains n-th values of all input arrays. When array lengths differ, missing elements are filled with NULL. #### Syntax[​](#syntax "Direct link to Syntax") ```sql arrays_zip(array_1, array_2[, ...]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `array_n`: Arrays to be merged. Each array can contain elements of any supported type. #### Return value[​](#return-value "Direct link to Return value") Returns an ARRAY of STRUCT, where the n-th struct contains the n-th field of each input array. If input arrays have different lengths, shorter arrays are padded with NULL values. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * If the input arrays have different lengths, the result array length is the maximum length among input arrays. * Missing values in shorter arrays are filled with NULL. * If any input array is NULL, the result is NULL. * If all input arrays are empty, the result is an empty array. #### Examples[​](#examples "Direct link to Examples") Example 1: Merge two arrays with equal length. ```sql mysql> SELECT arrays_zip([1, 2], ['a', 'b']) AS result; +---------------------------------------------------+ | result | +---------------------------------------------------+ | [{"col1":"1","col2":"a"},{"col1":"2","col2":"b"}] | +---------------------------------------------------+ ``` Example 2: Merge two arrays with different lengths. ```sql mysql> SELECT arrays_zip([1, 2], ['a', null, 'c']) AS result; +-----------------------------------------------------------------------------+ | result | +-----------------------------------------------------------------------------+ | [{"col1":"1","col2":"a"},{"col1":"2","col2":null},{"col1":null,"col2":"c"}] | +-----------------------------------------------------------------------------+ ``` Example 3: Merge three arrays. ```sql mysql> SELECT arrays_zip([1, 2, 3], ['a', 'b', 'c'], [10, 20, 30]) AS result; +---------------------------------------------------------------------------------------------------------------+ | result | +---------------------------------------------------------------------------------------------------------------+ | [{"col1":"1","col2":"a","col3":"10"},{"col1":"2","col2":"b","col3":"20"},{"col1":"3","col2":"c","col3":"30"}] | +---------------------------------------------------------------------------------------------------------------+ ``` Example 4: Merge two empty arrays. ```sql mysql> SELECT arrays_zip([], []) AS result; +--------+ | result | +--------+ | [] | +--------+ ``` Example 5: Merge a single array. ```sql mysql> SELECT arrays_zip([1, 2, 3]) AS result; +------------------------------------------+ | result | +------------------------------------------+ | [{"col1":"1"},{"col1":"2"},{"col1":"3"}] | +------------------------------------------+ ``` Example 6: Merge arrays when one of them is NULL. ```sql mysql> SELECT arrays_zip(null, [1, 2]) AS result; +--------+ | result | +--------+ | NULL | +--------+ ``` Example 7: Merge arrays of mixed types with NULL values. ```sql mysql> SELECT arrays_zip([1, null, 3], ['a', 'b', null]) AS result; +-----------------------------------------------------------------------------+ | result | +-----------------------------------------------------------------------------+ | [{"col1":"1","col2":"a"},{"col1":null,"col2":"b"},{"col1":"3","col2":null}] | +-----------------------------------------------------------------------------+ ``` Example 8: Merge arrays from table data. ```sql mysql> CREATE TABLE IF NOT EXISTS test (id INT, arr1 ARRAY, arr2 ARRAY) PROPERTIES ("replication_num"="1"); mysql> INSERT INTO test VALUES (1, [1, 2], ['a', 'b']), (2, [3, 4, 5], ['x', 'y']); mysql> SELECT id, arrays_zip(arr1, arr2) AS result FROM test; +------+----------------------------------------------------------------------------+ | id | result | +------+----------------------------------------------------------------------------+ | 1 | [{"col1":"1","col2":"a"},{"col1":"2","col2":"b"}] | | 2 | [{"col1":"3","col2":"x"},{"col1":"4","col2":"y"},{"col1":"5","col2":null}] | +------+----------------------------------------------------------------------------+ ``` --- ### cardinality Returns the number of elements in an array. The result type is INT. If the input parameter is NULL, the result is also NULL. Null elements are counted in the length. It is the alias of [array\_length()](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_length.md). This function is supported from v3.0 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT cardinality(any_array) ``` #### Parameters[​](#parameters "Direct link to Parameters") `any_array`: the ARRAY value from which you want to retrieve the number of elements. #### Return value[​](#return-value "Direct link to Return value") Returns an INT value. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select cardinality([1,2,3]); +-----------------------+ | cardinality([1,2,3]) | +-----------------------+ | 3 | +-----------------------+ 1 row in set (0.00 sec) mysql> select cardinality([1,2,3,null]); +------------------------------+ | cardinality([1, 2, 3, NULL]) | +------------------------------+ | 4 | +------------------------------+ mysql> select cardinality([[1,2], [3,4]]); +-----------------------------+ | cardinality([[1,2],[3,4]]) | +-----------------------------+ | 2 | +-----------------------------+ 1 row in set (0.01 sec) ``` #### keyword[​](#keyword "Direct link to keyword") CARDINALITY, ARRAY\_LENGTH, ARRAY --- ### element_at Returns the element at the specified position (index) from a given array. If any parameter is NULL or if the position does not exist, the result is NULL. This function is the alias of the subscript operator `[]`. It is supported from v3.0 onwards. If you want to retrieve a value from a key-value pair in a map, see [element\_at](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/element_at.md). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext element_at(any_array, position) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `any_array`: an array expression from which to retrieve elements. * `position`: the position of the element in the array. It must be a positive integer. Value range: \[1, array length]. If `position` does not exist, NULL is returned. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select element_at([2,3,11],3); +---------------+ | [2, 3, 11][3] | +---------------+ | 11 | +---------------+ mysql> select element_at(["a","b","c"],1); +--------------------+ | ['a', 'b', 'c'][1] | +--------------------+ | a | +--------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ELEMENT\_AT, ARRAY --- ### reverse Reverses a string or array. Returns a string or array with the characters in the string or array elements in reverse order. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext reverse(param) ``` #### Parameters[​](#parameters "Direct link to Parameters") `param`: the string or array to reverse. It can be of the VARCHAR, CHAR, or ARRAY type. Currently, this function supports only one-dimensional arrays and the array elements cannot be of the DECIMAL type. This function supports the following types of array elements: BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, VARCHAR, DECIMALV2, DATETIME, DATE, and JSON. **JSON is supported from 2.5.** #### Return value[​](#return-value "Direct link to Return value") The return type is the same as `param`. #### Examples[​](#examples "Direct link to Examples") Example 1: Reverse a string. ```plain MySQL > SELECT REVERSE('hello'); +------------------+ | REVERSE('hello') | +------------------+ | olleh | +------------------+ 1 row in set (0.00 sec) ``` Example 2: Reverse an array. ```plain MYSQL> SELECT REVERSE([4,1,5,8]); +--------------------+ | REVERSE([4,1,5,8]) | +--------------------+ | [8,5,1,4] | +--------------------+ ``` --- ### unnest UNNEST is a table function that takes an array and converts elements in that array into multiple rows of a table. The conversion is also known as "flattening". You can use Lateral Join with UNNEST to implement common conversions, for example, from STRING, ARRAY, or BITMAP to multiple rows. For more information, see [Lateral join](https://docs.starrocks.io/docs/using_starrocks/Lateral_join.md). From v2.5, UNNEST can take a variable number of array parameters. The arrays can vary in type and length (number of elements). If the arrays have different lengths, the largest length prevails, which means nulls will be added to arrays that are less than this length. See [Example 2](#example-2-unnest-takes-multiple-parameters) for more information. From v3.2.7, UNNEST can be used with LEFT JOIN ON TRUE, which is to retain all rows in the left table even if the corresponding rows in the right table are empty or have null values. NULLs are returned for such empty or NULL rows. See [Example 3](#example-3-unnest-with-left-join-on-true) for more information. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext unnest(array0[, array1 ...]) ``` #### Parameters[​](#parameters "Direct link to Parameters") `array`: the array you want to convert. It must be an array or expression that can evaluate to an ARRAY data type. You can specify one or more arrays or array expressions. #### Return value[​](#return-value "Direct link to Return value") Returns the multiple rows converted from the array. The type of return value depends on the types of elements in the array. For the element types supported in an array, see [ARRAY](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/Array.md). #### Usage notes[​](#usage-notes "Direct link to Usage notes") * UNNEST is a table function. It must be used with Lateral Join but the keyword Lateral Join does not need to be explicitly specified. * If the array expression evaluates to NULL or it is empty, no rows will be returned (except for LEFT JOIN ON TRUE). * If an element in the array is NULL, NULL is returned for that element. #### Examples[​](#examples "Direct link to Examples") ##### Example 1: UNNEST takes one parameter[​](#example-1-unnest-takes-one-parameter "Direct link to Example 1: UNNEST takes one parameter") ```sql -- Create table student_score where scores is an ARRAY column. CREATE TABLE student_score ( `id` bigint(20) NULL COMMENT "", `scores` ARRAY NULL COMMENT "" ) DUPLICATE KEY (id) DISTRIBUTED BY HASH(`id`); -- Insert data into this table. INSERT INTO student_score VALUES (1, [80,85,87]), (2, [77, null, 89]), (3, null), (4, []), (5, [90,92]); -- Query data from this table. SELECT * FROM student_score ORDER BY id; +------+--------------+ | id | scores | +------+--------------+ | 1 | [80,85,87] | | 2 | [77,null,89] | | 3 | NULL | | 4 | [] | | 5 | [90,92] | +------+--------------+ -- Use UNNEST to flatten the scores column into multiple rows. SELECT id, scores, unnest FROM student_score, unnest(scores) AS unnest; +------+--------------+--------+ | id | scores | unnest | +------+--------------+--------+ | 1 | [80,85,87] | 80 | | 1 | [80,85,87] | 85 | | 1 | [80,85,87] | 87 | | 2 | [77,null,89] | 77 | | 2 | [77,null,89] | NULL | | 2 | [77,null,89] | 89 | | 5 | [90,92] | 90 | | 5 | [90,92] | 92 | +------+--------------+--------+ ``` \[80,85,87] corresponding to `id = 1` is converted into three rows. \[77,null,89] corresponding to `id = 2` retains the null value. `scores` corresponding to `id = 3` and `id = 4` are NULL and empty, which are skipped. ##### Example 2: UNNEST takes multiple parameters[​](#example-2-unnest-takes-multiple-parameters "Direct link to Example 2: UNNEST takes multiple parameters") ```sql -- Create table example_table where the type and scores columns vary in type. CREATE TABLE example_table ( id varchar(65533) NULL COMMENT "", type varchar(65533) NULL COMMENT "", scores ARRAY NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(id) COMMENT "OLAP" DISTRIBUTED BY HASH(id) PROPERTIES ( "replication_num" = "3"); -- Insert data into the table. INSERT INTO example_table VALUES ("1", "typeA;typeB", [80,85,88]), ("2", "typeA;typeB;typeC", [87,90,95]); -- Query data from the table. SELECT * FROM example_table; +------+-------------------+------------+ | id | type | scores | +------+-------------------+------------+ | 1 | typeA;typeB | [80,85,88] | | 2 | typeA;typeB;typeC | [87,90,95] | +------+-------------------+------------+ -- Use UNNEST to convert type and scores into multiple rows. SELECT id, unnest.type, unnest.scores FROM example_table, unnest(split(type, ";"), scores) AS unnest(type,scores); +------+-------+--------+ | id | type | scores | +------+-------+--------+ | 1 | typeA | 80 | | 1 | typeB | 85 | | 1 | NULL | 88 | | 2 | typeA | 87 | | 2 | typeB | 90 | | 2 | typeC | 95 | +------+-------+--------+ ``` `type` and `scores` in `UNNEST` vary in type and length. `type` is a VARCHAR column while `scores` is an ARRAY column. The split() function is used to convert `type` into ARRAY. For `id = 1`, `type` is converted into \["typeA","typeB"], which has two elements. For `id = 2`, `type` is converted into \["typeA","typeB","typeC"], which has three elements. To ensure consistent numbers of rows for each `id`, a null element is added to \["typeA","typeB"]. ##### Example 3: UNNEST with LEFT JOIN ON TRUE[​](#example-3-unnest-with-left-join-on-true "Direct link to Example 3: UNNEST with LEFT JOIN ON TRUE") ```sql -- Create table student_score where scores is an ARRAY column. CREATE TABLE student_score ( `id` bigint(20) NULL COMMENT "", `scores` ARRAY NULL COMMENT "" ) DUPLICATE KEY (id) DISTRIBUTED BY HASH(`id`) PROPERTIES ( "replication_num" = "1" ); -- Insert data into this table. INSERT INTO student_score VALUES (1, [80,85,87]), (2, [77, null, 89]), (3, null), (4, []), (5, [90,92]); -- Query data from this table. SELECT * FROM student_score ORDER BY id; +------+--------------+ | id | scores | +------+--------------+ | 1 | [80,85,87] | | 2 | [77,null,89] | | 3 | NULL | | 4 | [] | | 5 | [90,92] | +------+--------------+ -- Use LEFT JOIN ON TRUE. SELECT id, scores, unnest FROM student_score LEFT JOIN unnest(scores) AS unnest ON TRUE ORDER BY 1, 3; +------+--------------+--------+ | id | scores | unnest | +------+--------------+--------+ | 1 | [80,85,87] | 80 | | 1 | [80,85,87] | 85 | | 1 | [80,85,87] | 87 | | 2 | [77,null,89] | NULL | | 2 | [77,null,89] | 77 | | 2 | [77,null,89] | 89 | | 3 | NULL | NULL | | 4 | [] | NULL | | 5 | [90,92] | 90 | | 5 | [90,92] | 92 | +------+--------------+--------+ ``` \[80,85,87] corresponding to `id = 1` is converted into three rows. The null value in \[77,null,89] corresponding to `id = 2` is retained. `scores` corresponding to `id = 3` and `id = 4` are NULL and empty. Left Join reserves these two rows and returns NULLs for them. --- ### Binary #### [📄️ from\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/binary-functions/from_binary.md) [Converts a binary value to a VARCHAR string based on the specified binary format (binary\_type).](https://docs.starrocks.io/docs/sql-reference/sql-functions/binary-functions/from_binary.md) #### [📄️ to\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/binary-functions/to_binary.md) [Converts a VARCHAR string to a binary value based on the specified binary format (binary\_type) of the input string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/binary-functions/to_binary.md) --- ### from_binary Converts a binary value to a VARCHAR string based on the specified binary format (`binary_type`). The following binary formats are supported: `hex`, `encode64`, and `utf8`. If no `binary_type` is specified, `hex` is the default. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext from_binary(binary[, binary_type]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `binary`: the input binary to convert, required. * `binary_type`: the binary format for conversion, optional. * `hex` (default): `from_binary` uses the `hex` method to encode the input binary to a VARCHAR string. * `encode64`: `from_binary` uses the `base64` method to encode the input binary to a VARCHAR string. * `utf8`: `from_binary` converts the input binary to a VARCHAR string without any transformation. #### Return value[​](#return-value "Direct link to Return value") Returns a VARCHAR string. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select from_binary(to_binary('ABAB', 'hex'), 'hex'); +----------------------------------------------+ | from_binary(to_binary('ABAB', 'hex'), 'hex') | +----------------------------------------------+ | ABAB | +----------------------------------------------+ 1 row in set (0.02 sec) mysql> select from_base64(from_binary(to_binary('U1RBUlJPQ0tT', 'encode64'), 'encode64')); +-----------------------------------------------------------------------------+ | from_base64(from_binary(to_binary('U1RBUlJPQ0tT', 'encode64'), 'encode64')) | +-----------------------------------------------------------------------------+ | STARROCKS | +-----------------------------------------------------------------------------+ 1 row in set (0.01 sec) mysql> select from_binary(to_binary('STARROCKS', 'utf8'), 'utf8'); +-----------------------------------------------------+ | from_binary(to_binary('STARROCKS', 'utf8'), 'utf8') | +-----------------------------------------------------+ | STARROCKS | +-----------------------------------------------------+ 1 row in set (0.01 sec) ``` #### References[​](#references "Direct link to References") * [to\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/binary-functions/to_binary.md) * [BINARY/VARBINARY data type](https://docs.starrocks.io/docs/sql-reference/data-types/string-type/BINARY.md) --- ### to_binary Converts a VARCHAR string to a binary value based on the specified binary format (`binary_type`) of the input string. The following binary formats are supported: `hex`, `encode64`, and `utf8`. If no `binary_type` is specified, `hex` is the default. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext to_binary(str[, binary_type]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str`: the string to convert, required. It must be a VARCHAR string. * `binary_type`: the binary format for conversion, optional. * `hex`(default): `to_binary` assumes the input string is a hex string in which all characters are in '0123456789abcdef'. If the input string is not valid, an empty binary is returned (exceptions will not be thrown). `to_binary` will convert the input string into binary directly. For example, `"abab"` will be converted to `x'abab'`. The input string is not case-sensitive. * `encode64`: `to_binary` assumes the input string is a base64-encoded string . If the input string is not valid, an empty binary is returned (exceptions will not be thrown). `to_binary` will decode the base64-encoded string as the binary result. For example, `"YWJhYg=="` will be converted to `x'abab'`. * `utf8`: `to_binary` converts the input string as a binary value without any transformation. #### Return value[​](#return-value "Direct link to Return value") Returns a VARCHAR value. #### Examples[​](#examples "Direct link to Examples") The following examples assume that the `--binary-as-hex` option is enabled when you access StarRocks from your MySQL client. This way, binary data is displayed using hexadecimal notation. ```plain mysql> select to_binary('ABAB', 'hex'); +----------------------------------------------------+ | to_binary('ABAB', 'hex') | +----------------------------------------------------+ | 0xABAB | +----------------------------------------------------+ 1 row in set (0.01 sec) mysql> select to_binary('U1RBUlJPQ0tT', 'encode64'); +------------------------------------------------------------------------------+ | to_binary('U1RBUlJPQ0tT', 'encode64') | +------------------------------------------------------------------------------+ | 0x53544152524F434B53 | +------------------------------------------------------------------------------+ 1 row in set (0.00 sec) mysql> select to_binary('STARROCKS', 'utf8'); +----------------------------------------------------------------+ | to_binary('STARROCKS', 'utf8') | +----------------------------------------------------------------+ | 0x53544152524F434B53 | +----------------------------------------------------------------+ 1 row in set (0.00 sec) -- The input string does not match the binary format and an empty binary is returned. mysql> select to_binary('U1RBUlJPQ0tT', 'hex'); +--------------------------------------------------------------------+ | to_binary('U1RBUlJPQ0tT', 'hex') | +--------------------------------------------------------------------+ | 0x | +--------------------------------------------------------------------+ 1 row in set (0.00 sec) ``` #### References[​](#references "Direct link to References") * [from\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/binary-functions/from_binary.md) * [BINARY/VARBINARY data type](https://docs.starrocks.io/docs/sql-reference/data-types/string-type/BINARY.md) --- ### Bit #### [📄️ bit\_shift\_left](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bit_shift_left.md) [Shifts the binary representation of a numeric expression to the left by a specified number of bits.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bit_shift_left.md) #### [📄️ bit\_shift\_right](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bit_shift_right.md) [Shifts the binary representation of a numeric expression to the right by a specified number of bits, preserving the sign.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bit_shift_right.md) #### [📄️ bit\_shift\_right\_logical](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bit_shift_right_logical.md) [Shifts the binary representation of a numeric expression to the right by a specified number of bits, without preserving the sign.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bit_shift_right_logical.md) #### [📄️ bitand](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bitand.md) [Returns the bitwise AND of two numeric expressions.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bitand.md) #### [📄️ bitnot](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bitnot.md) [Returns the bitwise negation of a numeric expression.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bitnot.md) #### [📄️ bitor](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bitor.md) [Returns the bitwise OR of two numeric expressions.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bitor.md) #### [📄️ bitxor](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bitxor.md) [Returns the bitwise XOR of two numeric expressions.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bitxor.md) --- ### bit_shift_left Shifts the binary representation of a numeric expression to the left by a specified number of bits. This function performs an **arithmetic left shift**, during which the bit length does not change, 0 is appended to the end, and the high bit remains unchanged. For example, shifting `10101` to the left by one bit results in `11010`. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext bit_shift_left(value, shift) ``` #### Parameters[​](#parameters "Direct link to Parameters") `value`: the value or numeric expression to shift. Supported data types are TINYINT, SMALLINT, INT, BIGINT, and LARGEINT. `shift`: the number of bits to shift. The supported data type is BIGINT. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the same type as `value`. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * If any input parameter is NULL, NULL is returned. * If `shift` is less than 0, 0 is returned. * Shifting a `value` by `0` always results in the original `value`. * Shifting `0` by a `shift` always results in `0`. * If the data type of `value` is numeric but not an integer, that value will be cast to an integer. See [Examples](#examples). * If the data type of `value` is a string, the value will be cast to an integer if possible. If the value cannot be cast to an integer, it will be processed as NULL. See [Examples](#examples). #### Examples[​](#examples "Direct link to Examples") Use this function to shift numeric values. ```plain SELECT bit_shift_left(2, 1); +----------------------+ | bit_shift_left(2, 1) | +----------------------+ | 4 | +----------------------+ SELECT bit_shift_left(2.2, 1); +------------------------+ | bit_shift_left(2.2, 1) | +------------------------+ | 4 | +------------------------+ SELECT bit_shift_left("2", 1); +------------------------+ | bit_shift_left('2', 1) | +------------------------+ | 4 | +------------------------+ SELECT bit_shift_left(-2, 1); +-----------------------+ | bit_shift_left(-2, 1) | +-----------------------+ | -4 | +-----------------------+ ``` #### References[​](#references "Direct link to References") * [bit\_shift\_right](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bit_shift_right.md) * [bit\_shift\_right\_logical](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bit_shift_right_logical.md) --- ### bit_shift_right Shifts the binary representation of a numeric expression to the right by a specified number of bits. This function performs an **arithmetic right shift**, during which the bit length does not change, the low bit is dropped, and the sign bit is used as the high bit. For example, shifting `10101` to the right by one bit results in `11010`. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext bit_shift_right(value, shift) ``` #### Parameters[​](#parameters "Direct link to Parameters") `value`: the value or numeric expression to shift. Supported data types are TINYINT, SMALLINT, INT, BIGINT, and LARGEINT. `shift`: the number of bits to shift. The supported data type is BIGINT. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the same type as `value`. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * If any input parameter is NULL, NULL is returned. * If `shift` is less than 0, 0 is returned. * Shifting a `value` by `0` always results in the original `value`. * Shifting `0` by a `shift` always results in `0`. * If the data type of `value` is numeric but not an integer, that value will be cast to an integer. See [Examples](#examples). * If the data type of `value` is a string, the value will be cast to an integer if possible. For example, the string "2.3" will be cast to 2. If the value cannot be cast to an integer, the value will be treated as NULL. See [Examples](#examples). #### Examples[​](#examples "Direct link to Examples") Use this function to shift numeric values. ```plain SELECT bit_shift_right(2, 1); +-----------------------+ | bit_shift_right(2, 1) | +-----------------------+ | 1 | +-----------------------+ SELECT bit_shift_right(2.2, 1); +-------------------------+ | bit_shift_right(2.2, 1) | +-------------------------+ | 1 | +-------------------------+ SELECT bit_shift_right("2", 1); +-------------------------+ | bit_shift_right('2', 1) | +-------------------------+ | 1 | +-------------------------+ SELECT bit_shift_right(-2, 1); +------------------------+ | bit_shift_right(-2, 1) | +------------------------+ | -1 | +------------------------+ ``` #### References[​](#references "Direct link to References") * [bit\_shift\_left](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bit_shift_left.md) * [bit\_shift\_right\_logical](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bit_shift_right_logical.md) --- ### bit_shift_right_logical Shifts the binary representation of a numeric expression to the right by a specified number of bits. This function performs a **logical right shift,** during which the bit length does not change, the low bit is dropped, and 0 is appended to the high bit despite whether the original bit is positive or negative. **Logical** shifts are unsigned shifts. For example, shifting `10101` by one bit results in `00101`. bit\_shift\_right() and bit\_shift\_right\_logical() return the same result for a positive value. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext bit_shift_right_logical(value, shift) ``` #### Parameters[​](#parameters "Direct link to Parameters") `value`: the value or numeric expression to shift. Supported data types are TINYINT, SMALLINT, INT, BIGINT, and LARGEINT. `shift`: the number of bits to shift. The supported data type is BIGINT. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the same type as `value`. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * If any input parameter is NULL, NULL is returned. * If `shift` is less than 0, 0 is returned. * Shifting a `value` by `0` always results in the original `value`. * Shifting `0` by a `shift` always results in `0`. * If the data type of `value` is numeric but not an integer, that value will be cast to an integer. See [Examples](#examples). * If the data type of `value` is a string, the value will be cast to an integer if possible. For example, the string "2.3" will be cast to 2. If the value cannot be cast to an integer, the value will be treated as NULL. See [Examples](#examples). #### Examples[​](#examples "Direct link to Examples") Use this function to shift numeric values. ```plain SELECT bit_shift_right_logical(2, 1); +-------------------------------+ | bit_shift_right_logical(2, 1) | +-------------------------------+ | 1 | +-------------------------------+ SELECT bit_shift_right_logical(2.2, 1); +---------------------------------+ | bit_shift_right_logical(2.2, 1) | +---------------------------------+ | 1 | +---------------------------------+ SELECT bit_shift_right_logical("2", 1); +---------------------------------+ | bit_shift_right_logical('2', 1) | +---------------------------------+ | 1 | +---------------------------------+ SELECT bit_shift_right_logical(cast('-2' AS INTEGER(32)), 1); +-----------------------------------------------+ | bit_shift_right_logical(CAST('-2' AS INT), 1) | +-----------------------------------------------+ | 2147483647 | +-----------------------------------------------+ ``` #### References[​](#references "Direct link to References") * [bit\_shift\_left](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bit_shift_left.md) * [bit\_shift\_right](https://docs.starrocks.io/docs/sql-reference/sql-functions/bit-functions/bit_shift_right.md) --- ### bitand Returns the bitwise AND of two numeric expressions. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITAND(x,y); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `x`: This expression must evaluate to any of the following data types: TINYINT, SMALLINT, INT, BIGINT, LARGEINT. * `y`: This expression must evaluate to any of the following data types: TINYINT, SMALLINT, INT, BIGINT, LARGEINT. > `x` and `y` must agree in data type. #### Return value[​](#return-value "Direct link to Return value") The return value has the same type as `x`. If any value is NULL, the result is NULL. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select bitand(3,0); +--------------+ | bitand(3, 0) | +--------------+ | 0 | +--------------+ 1 row in set (0.01 sec) ``` --- ### bitnot Returns the bitwise negation of a numeric expression. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITNOT(x); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: This expression must evaluate to any of the following data types: TINYINT, SMALLINT, INT, BIGINT, LARGEINT. #### Return value[​](#return-value "Direct link to Return value") The return value has the same type as `x`. If any value is NULL, the result is NULL. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select bitnot(3); +-----------+ | bitnot(3) | +-----------+ | -4 | +-----------+ ``` --- ### bitor Returns the bitwise OR of two numeric expressions. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITOR(x,y); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `x`: This expression must evaluate to any of the following data types: TINYINT, SMALLINT, INT, BIGINT, LARGEINT. * `y`: This expression must evaluate to any of the following data types: TINYINT, SMALLINT, INT, BIGINT, LARGEINT. > `x` and `y` must agree in data type. #### Return value[​](#return-value "Direct link to Return value") The return value has the same type as `x`. If any value is NULL, the result is NULL. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select bitor(3,0); +-------------+ | bitor(3, 0) | +-------------+ | 3 | +-------------+ 1 row in set (0.00 sec) ``` --- ### bitxor Returns the bitwise XOR of two numeric expressions. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITXOR(x,y); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `x`: This expression must evaluate to any of the following data types: TINYINT, SMALLINT, INT, BIGINT, LARGEINT. * `y`: This expression must evaluate to any of the following data types: TINYINT, SMALLINT, INT, BIGINT, LARGEINT. > `x` and `y` must agree in data type. #### Return value[​](#return-value "Direct link to Return value") The return value has the same type as `x`. If any value is NULL, the result is NULL. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select bitxor(3,0); +--------------+ | bitxor(3, 0) | +--------------+ | 3 | +--------------+ 1 row in set (0.00 sec) ``` --- ### Bitmap #### [📄️ base64\_to\_bitmap](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/base64_to_bitmap.md) [Before you import bitmap data into StarRocks, you need to serialize the data and encode the data as a Base64 string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/base64_to_bitmap.md) #### [📄️ bitmap\_agg](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_agg.md) [Aggregates values (excluding NULLs) in a column into a bitmap (multiple rows into one row).](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_agg.md) #### [📄️ bitmap\_and](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_and.md) [Calculates the intersection of two input bitmaps and returns the new bitmap.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_and.md) #### [📄️ bitmap\_andnot](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_andnot.md) [Returns bitmap values that exist in lhs but do not exist in rhs, and returns the new bitmap.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_andnot.md) #### [📄️ bitmap\_contains](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_contains.md) [Calculates whether the input value is in the bitmap column, and returns a Boolean value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_contains.md) #### [📄️ bitmap\_count](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_count.md) [Returns the 1-bit count for the input bitmap.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_count.md) #### [📄️ bitmap\_empty](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_empty.md) [Returns an empty bitmap.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_empty.md) #### [📄️ bitmap\_from\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_from_binary.md) [Converts a binary string with a specific format to a bitmap.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_from_binary.md) #### [📄️ bitmap\_from\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_from_string.md) [Converts a string into a BITMAP.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_from_string.md) #### [📄️ bitmap\_has\_any](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_has_any.md) [Calculates whether there are intersecting elements between two Bitmap columns, and the return value is Boolean value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_has_any.md) #### [📄️ bitmap\_hash](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_hash.md) [Calculates a 32-bit hash value for any type of input and return the bitmap containing the hash value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_hash.md) #### [📄️ bitmap\_hash64](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_hash64.md) [Calculates a 64-bit hash value for any type of input and returns a bitmap containing the hash value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_hash64.md) #### [📄️ bitmap\_intersect](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_intersect.md) [Aggregation function, used to calculate the bitmap intersection after grouping.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_intersect.md) #### [📄️ bitmap\_max](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_max.md) [Obtains the maximum value of a bitmap.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_max.md) #### [📄️ bitmap\_min](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_min.md) [Obtains the minimum value of a bitmap.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_min.md) #### [📄️ bitmap\_or](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_or.md) [Calculates the union of two input bitmaps and return a new bitmap.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_or.md) #### [📄️ bitmap\_remove](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_remove.md) [Removes input from the Bitmap value lhs and returns a result set.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_remove.md) #### [📄️ bitmap\_subset\_in\_range](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_subset_in_range.md) [Intercepts elements from a Bitmap value within the range of start\_range and end\_range (exclusive).](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_subset_in_range.md) #### [📄️ bitmap\_subset\_limit](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_subset_limit.md) [Intercepts a specified number of elements from a BITMAP value with element value starting from start range.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_subset_limit.md) #### [📄️ bitmap\_to\_array](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_to_array.md) [Converts a BITMAP into a BIGINT array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_to_array.md) #### [📄️ bitmap\_to\_base64](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_to_base64.md) [Converts a bitmap to a Base64-encoded string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_to_base64.md) #### [📄️ bitmap\_to\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_to_binary.md) [Converts Bitmap values to a Binary string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_to_binary.md) #### [📄️ bitmap\_to\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_to_string.md) [Converts an input bitmap into a string that is separated by commas (,).](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_to_string.md) #### [📄️ bitmap\_union](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_union.md) [Calculates the bitmap union of a set of values after grouping.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_union.md) #### [📄️ bitmap\_union\_count](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_union_count.md) [Returns the union of a set of bitmap values and returns the cardinality of the union.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_union_count.md) #### [📄️ bitmap\_union\_int](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_union_int.md) [Count the number of different values ​​in columns of type TINYINT, SMALLINT and INT, return the sum of COUNT (DISTINCT expr) same.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_union_int.md) #### [📄️ bitmap\_xor](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_xor.md) [Calculates the set consisting elements unique to lhs and rhs.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_xor.md) #### [📄️ intersect\_count](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/intersect_count.md) [A function to find the intersection size (number of same elements) of two Bitmap values, without requiring the data distribution to be orthogonal.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/intersect_count.md) #### [📄️ sub\_bitmap](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/sub_bitmap.md) [Intercepts len elements from a BITMAP value src starting from the position specified by offset.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/sub_bitmap.md) #### [📄️ subdivide\_bitmap](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/subdivide_bitmap.md) [Splits a large bitmap into multiple sub-bitmaps.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/subdivide_bitmap.md) #### [📄️ to\_bitmap](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/to_bitmap.md) [Converts an unsigned BIGINT to a BITMAP containing that single element.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/to_bitmap.md) #### [📄️ unnest\_bitmap](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/unnest_bitmap.md) [unnest\_bitmap is a table function that takes a bitmap and converts elements in that bitmap into multiple rows of a table.](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/unnest_bitmap.md) --- ### base64_to_bitmap Before you import bitmap data into StarRocks, you need to serialize the data and encode the data as a Base64 string. When you import the Base64 string into StarRocks, you need to convert the string into bitmap data. This function is used to convert Base64 strings into bitmap data. This function is supported from v2.3. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP base64_to_bitmap(VARCHAR bitmap) ``` #### Parameters[​](#parameters "Direct link to Parameters") `bitmap`: The supported data type is VARCHAR. Before you load Bitmap data into StarRocks, you can use Java or C++ to [create a BitmapValue object](https://github.com/StarRocks/starrocks/blob/main/fe/plugin-common/src/test/java/com/starrocks/types/BitmapValueTest.java), add an element, serialize the data, and encode the data as a Base64 string. Then, pass the Base64 string as an input parameter into this function. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BITMAP type. #### Examples[​](#examples "Direct link to Examples") Create a database named `bitmapdb` and a table named `bitmap`. Use Stream Load to import JSON data into `bitmap_table`. During this process, use base64\_to\_bitmap to convert the Base64 string in the JSON file into bitmap data. 1. Create a database and a table in StarRocks. In this example, a Primary Key table is created. ```sql CREATE database bitmapdb; USE bitmapdb; CREATE TABLE `bitmap_table` ( `tagname` varchar(65533) NOT NULL COMMENT "Tag name", `tagvalue` varchar(65533) NOT NULL COMMENT "Tag value", `userid` bitmap NOT NULL COMMENT "User ID" ) ENGINE=OLAP PRIMARY KEY(`tagname`, `tagvalue`) COMMENT "OLAP" DISTRIBUTED BY HASH(`tagname`) PROPERTIES ( "replication_num" = "3", "storage_format" = "DEFAULT" ); ``` 2. Use [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) to import JSON data into `bitmap_table`. Suppose there is a JSON file named **simpledata**. This file has the following content and `userid` is a Base64-encoded string. ```json { "tagname": "Product", "tagvalue": "Insurance", "userid":"AjowAAABAAAAAAACABAAAAABAAIAAwA=" } ``` Use base64\_to\_bitmap to convert `userid` into a bitmap value. ```plain curl --location-trusted -u :\ -H "columns: c1,c2,c3,tagname=c1,tagvalue=c2,userid=base64_to_bitmap(c3)"\ -H "label:bitmap123"\ -H "format: json"\ -H "jsonpaths: [\"$.tagname\",\"$.tagvalue\",\"$.userid\"]"\ -T simpleData http://host:port/api/bitmapdb/bitmap_table/_stream_load ``` 3. Query data from `bitmap_table`. ```plaintext mysql> select tagname,tagvalue,bitmap_to_string(userid) from bitmap_table; +--------------+----------+----------------------------+ | tagname | tagvalue | bitmap_to_string(`userid`) | +--------------+----------+----------------------------+ | Product | Insurance | 1,2,3 | +--------------+----------+----------------------------+ 1 rows in set (0.01 sec) ``` --- ### bitmap_agg Aggregates values (excluding NULLs) in a column into a bitmap (multiple rows into one row). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP_AGG(col) ``` #### Parameters[​](#parameters "Direct link to Parameters") `col`: the column whose values you want to aggregate. It must evaluate to BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, and LARGEINT. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BITMAP type. #### Usage notes[​](#usage-notes "Direct link to Usage notes") If the value in a row is less than 0 or greater than 18446744073709551615, the value will be ignored and not added to the Bitmap (see Example 3). #### Examples[​](#examples "Direct link to Examples") Take the following data table as an example: ```plaintext mysql> CREATE TABLE t1_test ( c1 int, c2 boolean, c3 tinyint, c4 int, c5 bigint, c6 largeint ) DUPLICATE KEY(c1) DISTRIBUTED BY HASH(c1) BUCKETS 1 PROPERTIES ("replication_num" = "3"); INSERT INTO t1_test VALUES (1, true, 11, 111, 1111, 11111), (2, false, 22, 222, 2222, 22222), (3, true, 33, 333, 3333, 33333), (4, null, null, null, null, null), (5, -1, -11, -111, -1111, -11111), (6, null, null, null, null, "36893488147419103232"); select * from t1_test order by c1; +------+------+------+------+-------+----------------------+ | c1 | c2 | c3 | c4 | c5 | c6 | +------+------+------+------+-------+----------------------+ | 1 | 1 | 11 | 111 | 1111 | 11111 | | 2 | 0 | 22 | 222 | 2222 | 22222 | | 3 | 1 | 33 | 333 | 3333 | 33333 | | 4 | NULL | NULL | NULL | NULL | NULL | | 5 | 1 | -11 | -111 | -1111 | -11111 | | 6 | NULL | NULL | NULL | NULL | 36893488147419103232 | +------+------+------+------+-------+----------------------+ ``` Example 1: Aggregate values in column `c1` into one bitmap. ```plaintext mysql> select bitmap_to_string(bitmap_agg(c1)) from t1_test; +----------------------------------+ | bitmap_to_string(bitmap_agg(c1)) | +----------------------------------+ | 1,2,3,4,5,6 | +----------------------------------+ ``` Example 2: Aggregate values in column `c2` into one bitmap (NULLs are ignored). ```plaintext mysql> SELECT BITMAP_TO_STRING(BITMAP_AGG(c2)) FROM t1_test; +----------------------------------+ | bitmap_to_string(bitmap_agg(c2)) | +----------------------------------+ | 0,1 | +----------------------------------+ ``` Example 3: Aggregate values in column `c6` into one bitmap (the last two values that exceed the value range are ignored). ```plaintext mysql> select bitmap_to_string(bitmap_agg(c6)) from t1_test; +----------------------------------+ | bitmap_to_string(bitmap_agg(c6)) | +----------------------------------+ | 11111,22222,33333 | +----------------------------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") BITMAP\_AGG, BITMAP --- ### bitmap_and Calculates the intersection of two input bitmaps and returns the new bitmap. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP BITMAP_AND(BITMAP lhs, BITMAP rhs) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select bitmap_count(bitmap_and(to_bitmap(1), to_bitmap(2))) cnt; +------+ | cnt | +------+ | 0 | +------+ MySQL > select bitmap_count(bitmap_and(to_bitmap(1), to_bitmap(1))) cnt; +------+ | cnt | +------+ | 1 | +------+ ``` #### keyword[​](#keyword "Direct link to keyword") BITMAP\_AND,BITMAP --- ### bitmap_andnot Returns bitmap values that exist in `lhs` but do not exist in `rhs`, and returns the new bitmap. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext bitmap_andnot(BITMAP lhs, BITMAP rhs) ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select bitmap_to_string(bitmap_andnot(bitmap_from_string('1, 3'), bitmap_from_string('2'))) cnt; +------+ |cnt | +------+ |1,3 | +------+ mysql> select bitmap_to_string(bitmap_andnot(bitmap_from_string('1,3,5'), bitmap_from_string('1'))) cnt; +------+ |cnt | +------+ |3,5 | +------+ ``` #### keyword[​](#keyword "Direct link to keyword") BITMAP\_ANDNOT, BITMAP --- ### bitmap_contains Calculates whether the input value is in the bitmap column, and returns a Boolean value. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext B00LEAN BITMAP_CONTAINS(BITMAP bitmap, BIGINT input) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select bitmap_contains(to_bitmap(1),2) cnt; +------+ | cnt | +------+ | 0 | +------+ MySQL > select bitmap_contains(to_bitmap(1),1) cnt; +------+ | cnt | +------+ | 1 | +------+ ``` #### keyword[​](#keyword "Direct link to keyword") BITMAP\_CONTAINS,BITMAP --- ### bitmap_count Returns the 1-bit count for the input bitmap. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT BITMAP_COUNT(any_bitmap) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select bitmap_count(bitmap_from_string("1,2,4")); +-------------------------------------------+ | bitmap_count(bitmap_from_string('1,2,4')) | +-------------------------------------------+ | 3 | +-------------------------------------------+ MySQL > select bitmap_count(NULL); +--------------------+ | bitmap_count(NULL) | +--------------------+ | 0 | +--------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") BITMAP,BITMAP\_COUNT --- ### bitmap_empty Returns an empty bitmap. It is mainly used to fill in the default value during insert or stream load. For example: ```bash cat data | curl --location-trusted -u user:passwd -T - \ -H "columns: dt,page,v1,v2=bitmap_empty()" \ http://host:8410/api/test/testDb/_stream_load ``` #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP BITMAP_EMPTY() ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select bitmap_count(bitmap_empty()); +------------------------------+ | bitmap_count(bitmap_empty()) | +------------------------------+ | 0 | +------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") BITMAP\_EMPTY,BITMAP --- ### bitmap_from_binary Converts a binary string with a specific format to a bitmap. This function can be used to load bitmap data to StarRocks. This function is supported from v3.0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP bitmap_from_binary(VARBINARY str) ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: The supported data type is VARBINARY. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BITMAP type. #### Examples[​](#examples "Direct link to Examples") Example 1: Use this function with other Bitmap functions. ```plain mysql> select bitmap_to_string(bitmap_from_binary(bitmap_to_binary(bitmap_from_string("0,1,2,3")))); +---------------------------------------------------------------------------------------+ | bitmap_to_string(bitmap_from_binary(bitmap_to_binary(bitmap_from_string('0,1,2,3')))) | +---------------------------------------------------------------------------------------+ | 0,1,2,3 | +---------------------------------------------------------------------------------------+ 1 row in set (0.01 sec) ``` --- ### bitmap_from_string Converts a string into a BITMAP. The string is composed of a set of UINT64 numbers separated by commas. For example, the "0, 1, 2" string will be converted into a Bitmap, in which bits 0, 1 and 2 are set. If the input field is invalid, NULL will be returned. This function deduplicates the input string during the conversion. It must be used with other functions, such as [bitmap\_to\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_to_string.md) to return results on terminals. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP BITMAP_FROM_STRING(VARCHAR input) ``` #### Return value[​](#return-value "Direct link to Return value") Returns a BITMAP value. If the input string is invalid, NULL is returned. If the input string is empty, an empty value is returned. #### Examples[​](#examples "Direct link to Examples") ```plain -- The input is empty and an empty value is returned. MySQL > select bitmap_to_string(bitmap_empty()); +----------------------------------+ | bitmap_to_string(bitmap_empty()) | +----------------------------------+ | | +----------------------------------+ -- `0,1,2` is returned. MySQL > select bitmap_to_string(bitmap_from_string("0, 1, 2")); +-------------------------------------------------+ | bitmap_to_string(bitmap_from_string('0, 1, 2')) | +-------------------------------------------------+ | 0,1,2 | +-------------------------------------------------+ -- `-1` is an invalid input and NULL is returned. MySQL > select bitmap_to_string(bitmap_from_string("-1, 0, 1, 2")); +-----------------------------------+ | bitmap_from_string('-1, 0, 1, 2') | +-----------------------------------+ | NULL | +-----------------------------------+ -- 2^64 is an invalid input and NULL is returned. MySQL > select bitmap_to_string(bitmap_from_string("0, 18446744073709551616")); +-----------------------------------------------------------------+ | bitmap_to_string(bitmap_from_string('0, 18446744073709551616')) | +-----------------------------------------------------------------+ | NULL | +-----------------------------------------------------------------+ -- The input string is deduplicated. MySQL > select bitmap_to_string(bitmap_from_string("0, 1, 1")); +-------------------------------------------------+ | bitmap_to_string(bitmap_from_string('0, 1, 1')) | +-------------------------------------------------+ | 0,1 | +-------------------------------------------------+ ``` #### keywords[​](#keywords "Direct link to keywords") BITMAP\_FROM\_STRING,BITMAP --- ### bitmap_has_any Calculates whether there are intersecting elements between two Bitmap columns, and the return value is Boolean value. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext B00LEAN BITMAP_HAS_ANY(BITMAP lhs, BITMAP rhs) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select bitmap_has_any(to_bitmap(1),to_bitmap(2)) cnt; +------+ | cnt | +------+ | 0 | +------+ MySQL > select bitmap_has_any(to_bitmap(1),to_bitmap(1)) cnt; +------+ | cnt | +------+ | 1 | +------+ ``` #### keyword[​](#keyword "Direct link to keyword") BITMAP\_HAS\_ANY,BITMAP --- ### bitmap_hash Calculates a 32-bit hash value for any type of input and return the bitmap containing the hash value. It is mainly used for the stream load task to import non integer fields into the bitmap field of the StarRocks table. For example: ```bash cat data | curl --location-trusted -u user:passwd -T - \ -H "columns: dt,page,device_id, device_id=bitmap_hash(device_id)" \ http://host:8410/api/test/testDb/_stream_load ``` #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP BITMAP_HASH(expr) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select bitmap_count(bitmap_hash('hello')); +------------------------------------+ | bitmap_count(bitmap_hash('hello')) | +------------------------------------+ | 1 | +------------------------------------+ select bitmap_to_string(bitmap_hash('hello')); +----------------------------------------+ | bitmap_to_string(bitmap_hash('hello')) | +----------------------------------------+ | 1321743225 | +----------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") BITMAP\_HASH,BITMAP --- ### bitmap_hash64 Calculates a 64-bit hash value for any type of input and returns a bitmap containing the hash value. It is mainly used to load non-integer data into the BITMAP field of the StarRocks table via Stream Load. For example: ```bash cat data | curl --location-trusted -u user:passwd -T - \ -H "columns: dt,page,device_id, device_id=bitmap_hash64(device_id)" \ http://:8030/api/test_db/tbl1/_stream_load ``` #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP BITMAP_HASH64(expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: The input can be of any data type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BITMAP data type. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select bitmap_count(bitmap_hash64('hello')); +--------------------------------------+ | bitmap_count(bitmap_hash64('hello')) | +--------------------------------------+ | 1 | +--------------------------------------+ select bitmap_to_string(bitmap_hash64('hello')); +------------------------------------------+ | bitmap_to_string(bitmap_hash64('hello')) | +------------------------------------------+ | 10760762337991515389 | +------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") BITMAP\_HASH64,BITMAP --- ### bitmap_intersect Aggregation function, used to calculate the bitmap intersection after grouping. Common usage scenarios, such as calculating user retention rate. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP BITMAP_INTERSECT(BITMAP value) ``` Enter a set of bitmap values, find the intersection of this set of bitmap values, and return the result. #### Example[​](#example "Direct link to Example") Table structure ```yml KeysType: AGG_KEY Columns: tag varchar, date datetime, user_id bitmap bitmap_union ``` ```sql -- Calculate users retention under different tags today and yesterday. select tag, bitmap_intersect(user_id) from ( select tag, date, bitmap_union(user_id) user_id from table where date in ('2020-05-18', '2020-05-19') group by tag, date) a group by tag; ``` Use with bitmap\_to\_string function to obtain the specific data of the intersection. ```sql --Find out users retained under different tags today and yesterday. select tag, bitmap_to_string(bitmap_intersect(user_id)) from ( select tag, date, bitmap_union(user_id) user_id from table where date in ('2020-05-18', '2020-05-19') group by tag, date) a group by tag; ``` #### keyword[​](#keyword "Direct link to keyword") BITMAP\_INTERSECT, BITMAP --- ### bitmap_max Obtains the maximum value of a bitmap. If the bitmap is `NULL`, this function returns `NULL`. If the bitmap is empty, this function returns `NULL` by default. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext bitmap_max(bitmap) ``` #### Parameters[​](#parameters "Direct link to Parameters") `bitmap`: the bitmap whose maximum value you want to obtain. Only the BITMAP data type is supported. You can specify a bitmap that you construct by using functions such as [bitmap\_from\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_from_string.md). #### Return value[​](#return-value "Direct link to Return value") Returns a value of the LARGEINT data type. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select bitmap_max(bitmap_from_string("0, 1, 2, 3")); +-------------------------------------------------+ | bitmap_max(bitmap_from_string("0, 1, 2, 3")) | +-------------------------------------------------+ | 3 | +-------------------------------------------------+ MySQL > select bitmap_max(bitmap_from_string("-1, 0, 1, 2")); +-------------------------------------------------+ | bitmap_max(bitmap_from_string("-1, 0, 1, 2")) | +-------------------------------------------------+ | NULL | +-------------------------------------------------+ MySQL > select bitmap_max(bitmap_empty()); +----------------------------------+ | bitmap_max(bitmap_empty()) | +----------------------------------+ | NULL | +----------------------------------+ mysql> select bitmap_max(bitmap_from_string("1, 16501189037412846863")); +--------------------------------------------------------------+ | bitmap_max(bitmap_from_string('1, 16501189037412846863')) | +--------------------------------------------------------------+ | 16501189037412846863 | +--------------------------------------------------------------+ 1 row in set (0.02 sec) ``` --- ### bitmap_min Obtains the minimum value of a bitmap. If the bitmap is `NULL`, this function returns `NULL`. If the bitmap is empty, this function returns `NULL` by default. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext bitmap_min(bitmap) ``` #### Parameters[​](#parameters "Direct link to Parameters") `bitmap`: the bitmap whose minimum value you want to obtain. Only the BITMAP data type is supported. You can specify a bitmap that you construct by using functions such as [bitmap\_from\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_from_string.md). #### Return value[​](#return-value "Direct link to Return value") Returns a value of the LARGEINT data type. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select bitmap_min(bitmap_from_string("0, 1, 2, 3")); +-------------------------------------------------+ | bitmap_min(bitmap_from_string('0, 1, 2, 3')) | +-------------------------------------------------+ | 0 | +-------------------------------------------------+ MySQL > select bitmap_min(bitmap_from_string("-1, 0, 1, 2")); +-------------------------------------------------+ | bitmap_min(bitmap_from_string('-1, 0, 1, 2')) | +-------------------------------------------------+ | NULL | +-------------------------------------------------+ MySQL > select bitmap_min(bitmap_empty()); +----------------------------------+ | bitmap_min(bitmap_empty()) | +----------------------------------+ | NULL | +----------------------------------+ mysql> select bitmap_min(bitmap_from_string("16501189037412846863")); +--------------------------------------------------------+ | bitmap_min(bitmap_from_string('16501189037412846863')) | +--------------------------------------------------------+ | 16501189037412846863 | +--------------------------------------------------------+ 1 row in set (0.03 sec) ``` --- ### bitmap_or Calculates the union of two input bitmaps and return a new bitmap. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP BITMAP_OR(BITMAP lhs, BITMAP rhs) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select bitmap_count(bitmap_or(to_bitmap(1), to_bitmap(2))) cnt; +------+ | cnt | +------+ | 2 | +------+ MySQL > select bitmap_count(bitmap_or(to_bitmap(1), to_bitmap(1))) cnt; +------+ | cnt | +------+ | 1 | +------+ ``` #### keyword[​](#keyword "Direct link to keyword") BITMAP\_OR,BITMAP --- ### bitmap_remove Removes `input` from the Bitmap value `lhs` and returns a result set. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext bitmap_remove(BITMAP lhs, BIGINT input) ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select bitmap_to_string(**bitmap_remove**(bitmap_from_string('1, 3'), 3)) cnt; +------+ |cnt | +------+ |1 | +------+ mysql> select bitmap_to_string(**bitmap_remove**(bitmap_from_string('1,3,5'), 6)) cnt; +------+ |cnt | +------+ |1,3,5 | +------+ ``` #### keyword[​](#keyword "Direct link to keyword") BITMAP\_REMOVE, BITMAP --- ### bitmap_subset_in_range Intercepts elements from a Bitmap value within the range of `start_range` and `end_range` (exclusive). The output elements are a subset of the Bitmap value. This function is mainly used for scenarios such as paginated queries. It is supported from v3.1. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP bitmap_subset_in_range(BITMAP src, BIGINT start_range, BIGINT end_range) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `src`: the Bitmap value from which to obtain elements. * `start_range`: the start range to intercept elements. It must be a BIGINT value. If the specified start range exceeds the maximum length of the BITMAP value, NULL is returned. See Example 4. * `end_range`: the end range to intercept elements. It must be a BIGINT value. If `end_range` equals or is less than `start range`, NULL is returned. See Example 3. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BITMAP type. NULL is returned if any of the input parameters is invalid. #### Usage notes[​](#usage-notes "Direct link to Usage notes") The subset elements include `start_range` but exclude `end_range`. See Example 5. #### Examples[​](#examples "Direct link to Examples") In the following examples, the input of bitmap\_subset\_in\_range() is the output of [bitmap\_from\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_from_string.md). For example, `bitmap_from_string('1,1,3,1,5,3,5,7,7,9')` returns `1, 3, 5, 7, 9`. bitmap\_subset\_in\_range() takes this BITMAP value as the input. Example 1: Obtain subset elements from the BITMAP value with elements values in the range of 1 to 4. The values within this range are 1 and 3. ```plaintext select bitmap_to_string(bitmap_subset_in_range(bitmap_from_string('1,1,3,1,5,3,5,7,7,9'), 1, 4)) value; +-------+ | value | +-------+ | 1,3 | +-------+ ``` Example 2: Obtain subset elements from the BITMAP value with elements value in the range 1 to 100. The end value exceeds the maximum length of the BITMAP value and all the matching elements are returned. ```plaintext select bitmap_to_string(bitmap_subset_in_range(bitmap_from_string('1,1,3,1,5,3,5,7,7,9'), 0, 100)) value; +-----------+ | value | +-----------+ | 1,3,5,7,9 | +-----------+ ``` Example 3: NULL is returned because the end range `3` is less than the start range `4`. ```plaintext select bitmap_to_string(bitmap_subset_in_range(bitmap_from_string('1,1,3,1,5,3,5,7,7,9'), 4, 3)) value; +-------+ | value | +-------+ | NULL | +-------+ ``` Example 4: The start range 10 exceeds the maximum length (5) of the BITMAP value `1,3,5,7,9`. NULL is returned. ```plain select bitmap_to_string(bitmap_subset_in_range(bitmap_from_string('1,1,3,1,5,3,5,7,7,9'), 10, 15)) value; +-------+ | value | +-------+ | NULL | +-------+ ``` Example 5: The returned subset includes the start value `1` but excludes the end value `3`. ```plaintext select bitmap_to_string(bitmap_subset_in_range(bitmap_from_string('1,1,3,1,5,4,5,6,7,9'), 1, 3)) value; +-------+ | value | +-------+ | 1 | +-------+ ``` #### References[​](#references "Direct link to References") [bitmap\_subset\_limit](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_subset_limit.md) --- ### bitmap_subset_limit Intercepts a specified number of elements from a BITMAP value with element value starting from `start range`. The output elements are a subset of `src`. This function is mainly used for scenarios such as paginated queries. It is supported from v3.1. This function is similar to [sub\_bitmap](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/sub_bitmap.md). The difference is that this function intercepts elements starting from an element value (`start_range`) whereas sub\_bitmap intercepts elements starting from an offset. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP bitmap_subset_limit(BITMAP src, BIGINT start_range, BIGINT limit) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `src`: the BITMAP value from which to obtain elements. * `start_range`: the start range to intercept elements. It must be a BIGINT value. If the specified start range exceeds the maximum element of the BITMAP value and `limit` is positive, NULL is returned. See Example 4. * `limit`: the number of elements to obtain starting from `start_range`. Negative limits are counted from right to left. If the number of matching elements is less than the value of `limit`, all the matching elements are returned. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BITMAP type. NULL is returned if any of the input parameters is invalid. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The subset elements include `start range`. * Negative limits are counted from right to left. See example 3. #### Examples[​](#examples "Direct link to Examples") In the following examples, the input of bitmap\_subset\_limit() is the output of [bitmap\_from\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_from_string.md). For example, `bitmap_from_string('1,1,3,1,5,3,5,7,7,9')` returns `1, 3, 5, 7, 9`. bitmap\_subset\_limit() takes this BITMAP value as the input. Example 1: Obtain 4 elements from the BITMAP value with elements values starting from 1. ```plaintext select bitmap_to_string(bitmap_subset_limit(bitmap_from_string('1,1,3,1,5,3,5,7,7,9'), 1, 4)) value; +---------+ | value | +---------+ | 1,3,5,7 | +---------+ ``` Example 2: Obtain 100 elements from the BITMAP value with elements value starting from 1,. The limit exceeds the length of the BITMAP value and all the matching elements are returned. ```plaintext select bitmap_to_string(bitmap_subset_limit(bitmap_from_string('1,1,3,1,5,3,5,7,7,9'), 1, 100)) value; +-----------+ | value | +-----------+ | 1,3,5,7,9 | +-----------+ ``` Example 3: Obtain -2 elements from the BITMAP value with elements value starting from 5 (counting from right to left). ```plaintext select bitmap_to_string(bitmap_subset_limit(bitmap_from_string('1,1,3,1,5,3,5,7,7,9'), 5, -2)) value; +-----------+ | value | +-----------+ | 3,5 | +-----------+ ``` Example 4: The start range 10 exceeds the maximum element of the BITMAP value `1,3,5,7,9` and limit is positive. NULL is returned. ```plain select bitmap_to_string(bitmap_subset_in_range(bitmap_from_string('1,1,3,1,5,3,5,7,7,9'), 10, 15)) value; +-------+ | value | +-------+ | NULL | +-------+ ``` --- ### bitmap_to_array Converts a BITMAP into a BIGINT array. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext ARRAY BITMAP_TO_ARRAY (bitmap) ``` #### Parameters[​](#parameters "Direct link to Parameters") `bitmap`: the bitmap you want to convert. #### Return value[​](#return-value "Direct link to Return value") Returns a BIGINT array. #### Examples[​](#examples "Direct link to Examples") ```plain select bitmap_to_array(bitmap_from_string("1, 7")); +----------------------------------------------+ | bitmap_to_array(bitmap_from_string('1, 7')) | +----------------------------------------------+ | [1,7] | +----------------------------------------------+ select bitmap_to_array(NULL); +-----------------------+ | bitmap_to_array(NULL) | +-----------------------+ | NULL | +-----------------------+ ``` --- ### bitmap_to_base64 Converts a bitmap to a Base64-encoded string. This function is supported from v2.5. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR bitmap_to_base64(BITMAP bitmap) ``` #### Parameters[​](#parameters "Direct link to Parameters") `bitmap`: the bitmap to convert. This parameter is required. If the input value is invalid, an error is returned. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. #### Examples[​](#examples "Direct link to Examples") Example 1: Use this function with other bitmap functions. ```plain select bitmap_to_base64(bitmap_from_string("0, 1, 2, 3")); +----------------------------------------------------+ | bitmap_to_base64(bitmap_from_string('0, 1, 2, 3')) | +----------------------------------------------------+ | AjowAAABAAAAAAADABAAAAAAAAEAAgADAA== | +----------------------------------------------------+ 1 row in set (0.00 sec) select bitmap_to_base64(to_bitmap(1)); +--------------------------------+ | bitmap_to_base64(to_bitmap(1)) | +--------------------------------+ | AQEAAAA= | +--------------------------------+ 1 row in set (0.00 sec) select bitmap_to_base64(bitmap_empty()); +----------------------------------+ | bitmap_to_base64(bitmap_empty()) | +----------------------------------+ | AA== | +----------------------------------+ 1 row in set (0.00 sec) ``` Example 2: Convert each value in a BITMAP column into Base64-encoded strings. 1. Create an Aggregate table `page_uv` where the `AGGREGATE KEY` is (`page_id`, `visit_date`). This table contains a BITMAP column `visit_users` whose values are to be aggregated. ```sql CREATE TABLE `page_uv` (`page_id` INT NOT NULL, `visit_date` datetime NOT NULL, `visit_users` BITMAP BITMAP_UNION NOT NULL ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`) PROPERTIES ( "replication_num" = "3", "storage_format" = "DEFAULT" ); ``` 2. Insert data into this table. ```sql insert into page_uv values (1, '2020-06-23 01:30:30', to_bitmap(13)), (1, '2020-06-23 01:30:30', to_bitmap(23)), (1, '2020-06-23 01:30:30', to_bitmap(33)), (1, '2020-06-23 02:30:30', to_bitmap(13)), (2, '2020-06-23 01:30:30', to_bitmap(23)); select * from page_uv order by page_id; +---------+---------------------+-------------+ | page_id | visit_date | visit_users | +---------+---------------------+-------------+ | 1 | 2020-06-23 01:30:30 | NULL | | 1 | 2020-06-23 02:30:30 | NULL | | 2 | 2020-06-23 01:30:30 | NULL | +---------+---------------------+-------------+ ``` 3. Convert each value in the `visit_users` column into a Base64-encoded string. ```plain select page_id, bitmap_to_base64(visit_users) from page_uv; +---------+------------------------------------------+ | page_id | bitmap_to_base64(visit_users) | +---------+------------------------------------------+ | 1 | CgMAAAANAAAAAAAAABcAAAAAAAAAIQAAAAAAAAA= | | 1 | AQ0AAAA= | | 2 | ARcAAAA= | +---------+------------------------------------------+ ``` --- ### bitmap_to_binary Converts Bitmap values to a Binary string. bitmap\_to\_binary is mainly used for exporting Bitmap data. The compression effect is better than [bitmap\_to\_base64](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_to_base64.md). If you plan to export data directly to a binary file such as Parquet, this function is recommended. This function is supported from v3.0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARBINARY bitmap_to_binary(BITMAP bitmap) ``` #### Parameters[​](#parameters "Direct link to Parameters") `bitmap`: the Bitmap to convert, required. If the input value is invalid, an error is returned. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARBINARY type. #### Examples[​](#examples "Direct link to Examples") Example 1: Use this function with other bitmap functions. ```plain mysql> select hex(bitmap_to_binary(bitmap_from_string("0, 1, 2, 3"))); +---------------------------------------------------------+ | hex(bitmap_to_binary(bitmap_from_string('0, 1, 2, 3'))) | +---------------------------------------------------------+ | 023A3000000100000000000300100000000000010002000300 | +---------------------------------------------------------+ 1 row in set (0.01 sec) mysql> select hex(bitmap_to_binary(to_bitmap(1))); +-------------------------------------+ | hex(bitmap_to_binary(to_bitmap(1))) | +-------------------------------------+ | 0101000000 | +-------------------------------------+ 1 row in set (0.01 sec) mysql> select hex(bitmap_to_binary(bitmap_empty())); +---------------------------------------+ | hex(bitmap_to_binary(bitmap_empty())) | +---------------------------------------+ | 00 | +---------------------------------------+ 1 row in set (0.01 sec) ``` Example 2: Convert each value in a BITMAP column into a binary string. 1. Create an Aggregate table `page_uv` where the `AGGREGATE KEY` is (`page_id`, `visit_date`). This table contains a BITMAP column `visit_users` whose values are to be aggregated. ```sql CREATE TABLE `page_uv` (`page_id` INT NOT NULL, `visit_date` datetime NOT NULL, `visit_users` BITMAP BITMAP_UNION NOT NULL ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`) PROPERTIES ( "replication_num" = "3", "storage_format" = "DEFAULT" ); ``` 2. Insert data into this table. ```sql insert into page_uv values (1, '2020-06-23 01:30:30', to_bitmap(13)), (1, '2020-06-23 01:30:30', to_bitmap(23)), (1, '2020-06-23 01:30:30', to_bitmap(33)), (1, '2020-06-23 02:30:30', to_bitmap(13)), (2, '2020-06-23 01:30:30', to_bitmap(23)); select * from page_uv order by page_id; +---------+---------------------+-------------+ | page_id | visit_date | visit_users | +---------+---------------------+-------------+ | 1 | 2020-06-23 01:30:30 | NULL | | 1 | 2020-06-23 02:30:30 | NULL | | 2 | 2020-06-23 01:30:30 | NULL | +---------+---------------------+-------------+ ``` 3. Convert each value in the `visit_users` column into a binary-encoded string. ```plain mysql> select page_id, hex(bitmap_to_binary(visit_users)) from page_uv; +---------+------------------------------------------------------------+ | page_id | hex(bitmap_to_binary(visit_users)) | +---------+------------------------------------------------------------+ | 1 | 0A030000000D0000000000000017000000000000002100000000000000 | | 1 | 010D000000 | | 2 | 0117000000 | +---------+------------------------------------------------------------+ 3 rows in set (0.01 sec) ``` --- ### bitmap_to_string Converts an input bitmap into a string that is separated by commas (,). This string contains all the bits in the bitmap. If the input is null, null is returned. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR BITMAP_TO_STRING(BITMAP input) ``` #### Parameters[​](#parameters "Direct link to Parameters") `input`: the bitmap you want to convert. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. #### Examples[​](#examples "Direct link to Examples") Example 1: The input is null and null is returned. ```plain MySQL > select bitmap_to_string(null); +------------------------+ | bitmap_to_string(NULL) | +------------------------+ | NULL | +------------------------+ ``` Example 2: The input is empty and an empty string is returned. ```plain MySQL > select bitmap_to_string(bitmap_empty()); +----------------------------------+ | bitmap_to_string(bitmap_empty()) | +----------------------------------+ | | +----------------------------------+ ``` Example 3: Convert a bitmap that contains one bit into a string. ```plain MySQL > select bitmap_to_string(to_bitmap(1)); +--------------------------------+ | bitmap_to_string(to_bitmap(1)) | +--------------------------------+ | 1 | +--------------------------------+ ``` Example 4: Convert a bitmap that contains two bits into a string. ```plain MySQL > select bitmap_to_string(bitmap_or(to_bitmap(1), to_bitmap(2))); +---------------------------------------------------------+ | bitmap_to_string(bitmap_or(to_bitmap(1), to_bitmap(2))) | +---------------------------------------------------------+ | 1,2 | +---------------------------------------------------------+ ``` --- ### bitmap_union Calculates the bitmap union of a set of values after grouping. Common usage scenarios include calculating PV and UV. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP BITMAP_UNION(BITMAP value) ``` #### Examples[​](#examples "Direct link to Examples") ```sql select page_id, bitmap_union(user_id) from table group by page_id; ``` Use this function with bitmap\_count() to obtain the UV of a web page. ```sql select page_id, bitmap_count(bitmap_union(user_id)) from table group by page_id; ``` If `user_id` is an integer, the above query statement is equivalent to the following: ```sql select page_id, count(distinct user_id) from table group by page_id; ``` #### keyword[​](#keyword "Direct link to keyword") BITMAP\_UNION, BITMAP --- ### bitmap_union_count Returns the union of a set of bitmap values and returns the cardinality of the union. This function is supported from v2.3. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT bitmap_union_count(BITMAP value) ``` ##### Parameters[​](#parameters "Direct link to Parameters") `value`: a set of bitmap values. The supported data type is BITMAP. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BIGINT type. #### Examples[​](#examples "Direct link to Examples") Calculate the unique views (UVs) of a web page. If `user_id` is of the INT type, the latter two queries are equivalent. ```plaintext mysql> select * from test +---------+---------+ | page_id | user_id | +---------+---------+ | 1 | 1 | | 1 | 2 | | 2 | 1 | +---------+---------+ mysql> select page_id,count(distinct user_id) from test group by page_id; +---------+-------------------------+ | page_id | count(DISTINCT user_id) | +---------+-------------------------+ | 1 | 2 | | 2 | 1 | +---------+-------------------------+ mysql> select page_id,bitmap_union_count(to_bitmap(user_id)) from test group by page_id; +---------+----------------------------------------+ | page_id | bitmap_union_count(to_bitmap(user_id)) | +---------+----------------------------------------+ | 1 | 2 | | 2 | 1 | +---------+----------------------------------------+ ``` --- ### bitmap_union_int Count the number of different values ​​in columns of type TINYINT, SMALLINT and INT, return the sum of COUNT (DISTINCT expr) same. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT bitmap_union_int(expr) ``` ##### Parameters[​](#parameters "Direct link to Parameters") `expr`: column expression. The supported column type is TINYINT, SMALLINT and INT. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BIGINT type. #### Examples[​](#examples "Direct link to Examples") ```plaintext mysql> select bitmap_union_int(k1) from tbl1; +------------------------+ | bitmap_union_int(`k1`) | +------------------------+ | 2 | +------------------------+ ``` --- ### bitmap_xor Calculates the set consisting elements unique to `lhs` and `rhs`. It is logically equivalent to `bitmap_andnot(bitmap_or(lhs, rhs), bitmap_and(lhs, rhs))` (complementary set). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext bitmap_xor(BITMAP lhs, BITMAP rhs) ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select bitmap_to_string(bitmap_xor(bitmap_from_string('1, 3'), bitmap_from_string('2'))) cnt; +------+ |cnt | +------+ |1,2,3 | +------+ ``` #### keyword[​](#keyword "Direct link to keyword") BITMAP\_XOR, BITMAP --- ### intersect_count A function to find the intersection size (number of same elements) of two Bitmap values, without requiring the data distribution to be orthogonal. The first parameter is the Bitmap column, the second parameter is the dimension column used for filtering, and the third parameter is a variable-length parameter, meaning that different values of the filtering dimension column are taken. If there is no intersection, 0 is returned. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT INTERSECT_COUNT(bitmap_column, column_to_filter, filter_values) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `bitmap_column`: the bitmap column to be computed. * `column_to_filter`: the name of the column to be intersected. * `filter_values`: the different values of the filtered dimension columns. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BIGINT type. #### Example[​](#example "Direct link to Example") Create an Aggregate table with a Bitmap column `user_id` and insert data. ```sql CREATE TABLE `tbl` ( `dt` date NULL COMMENT "", `user_id` bitmap BITMAP_UNION NULL COMMENT "" ) ENGINE=OLAP AGGREGATE KEY(`dt`) COMMENT "OLAP" DISTRIBUTED BY HASH(`dt`); INSERT INTO `tbl` VALUES ('2020-10-01', to_bitmap(2)), ('2020-10-01', to_bitmap(3)), ('2020-10-01', to_bitmap(5)), ('2020-10-02', to_bitmap(2)), ('2020-10-02', to_bitmap(3)), ('2020-10-02', to_bitmap(4)), ('2020-10-03', to_bitmap(1)), ('2020-10-03', to_bitmap(5)), ('2020-10-03', to_bitmap(6)); SELECT dt, bitmap_to_string(user_id) from tbl; +------------+---------------------------+ | dt | bitmap_to_string(user_id) | +------------+---------------------------+ | 2020-10-02 | 2,3,4 | | 2020-10-03 | 1,5,6 | | 2020-10-01 | 2,3,5 | +------------+---------------------------+ ``` Calculate the number of same `user_id` corresponding to `'2020-10-01'` and `'2020-10-02'`. Two same elements (2 and 3) are found and the number `2` is returned. ```plaintext mysql> select intersect_count(user_id, dt, '2020-10-01', '2020-10-02') from tbl where dt in ('2020-10-01', '2020-10-02'); +----------------------------------------------------------+ | intersect_count(user_id, dt, '2020-10-01', '2020-10-02') | +----------------------------------------------------------+ | 2 | +----------------------------------------------------------+ ``` Calculate the number of same `user_id` corresponding to `'2020-10-01'` and `'2020-10-03'`. One element `5` is found and the number `1` is returned. ```plaintext mysql> select intersect_count(user_id, dt, '2020-10-01', '2020-10-03') from tbl where dt in ('2020-10-01', '2020-10-03'); +----------------------------------------------------------+ | intersect_count(user_id, dt, '2020-10-01', '2020-10-03') | +----------------------------------------------------------+ | 1 | +----------------------------------------------------------+ ``` --- ### sub_bitmap Intercepts `len` elements from a BITMAP value `src` starting from the position specified by `offset`. The output elements are a subset of `src`. This function is mainly used for scenarios such as paginated queries. It is supported from v2.5. This function is similar to [bitmap\_subset\_limit](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_subset_limit.md). The difference is that this function intercepts elements starting from an offset whereas bitmap\_subset\_limit intercepts elements starting from an element value (`start_range`). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP sub_bitmap(BITMAP src, BIGINT offset, BIGINT len) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `src`: the BITMAP value from which you want to obtain elements. * `offset`: the starting position. It must be a BIGINT value. Note the following points when you use `offset`: * Offsets start from 0. * Negative offsets are counted from right to left. See Examples 3 and 4. * If the starting position specified by `offset` exceeds the actual length of the BITMAP value, NULL is returned. See Example 6. * `len`: the number of elements to obtain. It must be a BIGINT value greater than or equal to 1. If the number of matching elements is less than the value of `len`, all the matching elements are returned. See Examples 2, 3, and 7. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BITMAP type. NULL is returned if any of the input parameters is invalid. #### Examples[​](#examples "Direct link to Examples") In the following examples, the input of sub\_bitmap() is the output of [bitmap\_from\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_from_string.md). For example, `bitmap_from_string('1,1,3,1,5,3,5,7,7,9')` returns `1, 3, 5, 7, 9`. sub\_bitmap() takes this BITMAP value as the input. Example 1: Obtain two elements from the BITMAP value with the offset set to 0. ```plaintext select bitmap_to_string(sub_bitmap(bitmap_from_string('1,1,3,1,5,3,5,7,7,9'), 0, 2)) value; +-------+ | value | +-------+ | 1,3 | +-------+ ``` Example 2: Obtain 100 elements from the BITMAP value with the offset set to 0. 100 exceeds the length of the BITMAP value and all the matching elements are returned. ```plaintext select bitmap_to_string(sub_bitmap(bitmap_from_string('1,1,3,1,5,3,5,7,7,9'), 0, 100)) value; +-----------+ | value | +-----------+ | 1,3,5,7,9 | +-----------+ ``` Example 3: Obtain 100 elements from the BITMAP value with the offset set to -3. 100 exceeds the length of the BITMAP value and all the matching elements are returned. ```plaintext select bitmap_to_string(sub_bitmap(bitmap_from_string('1,1,3,1,5,3,5,7,7,9'), -3, 100)) value; +-------+ | value | +-------+ | 5,7,9 | +-------+ ``` Example 4: Obtain two elements from the BITMAP value with the offset set to -3. ```plaintext select bitmap_to_string(sub_bitmap(bitmap_from_string('1,1,3,1,5,3,5,7,7,9'), -3, 2)) value; +-------+ | value | +-------+ | 5,7 | +-------+ ``` Example 5: NULL is returned because `-10` is an invalid input of `len`. ```plaintext select bitmap_to_string(sub_bitmap(bitmap_from_string('1,1,3,1,5,3,5,7,7,9'), 0, -10)) value; +-------+ | value | +-------+ | NULL | +-------+ ``` Example 6: The starting position specified by offset 5 exceeds the length of the BITMAP value `1,3,5,7,9`. NULL is returned. ```plain select bitmap_to_string(sub_bitmap(bitmap_from_string('1,1,3,1,5,3,5,7,7,9'), 5, 1)) value; +-------+ | value | +-------+ | NULL | +-------+ ``` Example 7: `len` is set to 5 but only two elements match the condition. All of these two elements are returned. ```plain select bitmap_to_string(sub_bitmap(bitmap_from_string('1,1,3,1,5,3,5,7,7,9'), -2, 5)) value; +-------+ | value | +-------+ | 7,9 | +-------+ ``` --- ### subdivide_bitmap Splits a large bitmap into multiple sub-bitmaps. This function is mainly used to export bitmaps. Bitmaps that are too large will exceed the maximum packet size allowed in the MySQL protocol. This function is supported from v2.5. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP subdivide_bitmap(bitmap, length) ``` #### Parameters[​](#parameters "Direct link to Parameters") `bitmap`: The bitmap that needs to be split, required. `length`: The maximum length of each sub-bitmap, required. Bitmaps larger than this value will be split into multiple small bitmaps. #### Return value[​](#return-value "Direct link to Return value") Returns multiple sub-bitmaps not larger than `length`. #### Examples[​](#examples "Direct link to Examples") Suppose there is a table `t1`, in which the `c2` column is a BITMAP column. ```plain -- Use bitmap_to_string() to convert values in `c2` into a string. mysql> select c1, bitmap_to_string(c2) from t1; +------+----------------------+ | c1 | bitmap_to_string(c2) | +------+----------------------+ | 1 | 1,2,3,4,5,6,7,8,9,10 | +------+----------------------+ -- Split `c2` into small bitmaps whose maximum length is 3. mysql> select c1, bitmap_to_string(subdivide_bitmap) from t1, subdivide_bitmap(c2, 3); +------+------------------------------------+ | c1 | bitmap_to_string(subdivide_bitmap) | +------+------------------------------------+ | 1 | 1,2,3 | | 1 | 4,5,6 | | 1 | 7,8,9 | | 1 | 10 | +------+------------------------------------+ ``` --- ### to_bitmap The input is unsigned bigint with the value ranging from 0 to 18446744073709551615, and the output is bitmap containing this element. This function is mainly used for the stream load task to import integer fields into the bitmap field of the StarRocks table. For example: ```bash cat data | curl --location-trusted -u user:passwd -T - \ -H "columns: dt,page,user_id, user_id=to_bitmap(user_id)" \ http://host:8410/api/test/testDb/_stream_load ``` #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BITMAP TO_BITMAP(expr) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select bitmap_count(to_bitmap(10)); +-----------------------------+ | bitmap_count(to_bitmap(10)) | +-----------------------------+ | 1 | +-----------------------------+ select bitmap_to_string(to_bitmap(10)); +---------------------------------+ | bitmap_to_string(to_bitmap(10)) | +---------------------------------+ | 10 | +---------------------------------+ select bitmap_to_string(to_bitmap(-5)); +---------------------------------+ | bitmap_to_string(to_bitmap(-5)) | +---------------------------------+ | NULL | +---------------------------------+ select bitmap_to_string(to_bitmap(null)); +-----------------------------------+ | bitmap_to_string(to_bitmap(NULL)) | +-----------------------------------+ | NULL | +-----------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") TO\_BITMAP,BITMAP --- ### unnest_bitmap unnest\_bitmap is a table function that takes a bitmap and converts elements in that bitmap into multiple rows of a table. StarRocks' [Lateral Join](https://docs.starrocks.io/docs/using_starrocks/Lateral_join.md) can be used in conjunction with the unnest\_bitmap function to implement common column-to-row logic. This function can be used to replace `unnest(bitmap_to_array(bitmap))`. It has better performance, takes up less memory resources, and is not limited by the length of the array. This function is supported from v3.1. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext unnest_bitmap(bitmap) ``` #### Parameters[​](#parameters "Direct link to Parameters") `bitmap`: the bitmap you want to convert. #### Return value[​](#return-value "Direct link to Return value") Returns multiple rows converted from the bitmap. The type of return value is BIGINT. #### Examples[​](#examples "Direct link to Examples") `c2` is a bitmap column in table `t1`. ```sql -- Use the bitmap_to_string function to convert values in the c2 column into a string. mysql> select c1, bitmap_to_string(c2) from t1; +------+----------------------+ | c1 | bitmap_to_string(c2) | +------+----------------------+ | 1 | 1,2,3,4,5,6,7,8,9,10 | +------+----------------------+ -- Use the unnest_bitmap function to expand the bitmap column into multiple rows. mysql> select c1, unnest_bitmap from t1, unnest_bitmap(c2); +------+---------------+ | c1 | unnest_bitmap | +------+---------------+ | 1 | 1 | | 1 | 2 | | 1 | 3 | | 1 | 4 | | 1 | 5 | | 1 | 6 | | 1 | 7 | | 1 | 8 | | 1 | 9 | | 1 | 10 | +------+---------------+ ``` --- ### CAST Converts an input into the specified type. For example, `cast (input as BIGINT)` converts the input into a BIGINT value. From v2.4, StarRocks supports conversion to the ARRAY type. note `CAST(input AS CHAR(N))` truncates the result to the first `N` characters, consistent with MySQL. For example, `CAST(1775580223839 AS CHAR(10))` returns `1775580223`. `CAST(input AS CHAR)` without a length and `CAST(input AS VARCHAR(N))` are not truncated. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext cast (input as type) ``` #### Parameters[​](#parameters "Direct link to Parameters") `input`: the data you want to convert. `type`: the destination data type. #### Return value[​](#return-value "Direct link to Return value") Returns a value whose data type is the same as `type`. #### Examples[​](#examples "Direct link to Examples") Example 1: common data conversions ```plain select cast('9.5' as DECIMAL(10,2)); +--------------------------------+ | CAST('9.5' AS DECIMAL64(10,2)) | +--------------------------------+ | 9.50 | +--------------------------------+ select cast(NULL AS INT); +-------------------+ | CAST(NULL AS INT) | +-------------------+ | NULL | +-------------------+ select cast(true AS BOOLEAN); +-----------------------+ | CAST(TRUE AS BOOLEAN) | +-----------------------+ | 1 | +-----------------------+ select cast (1 as BIGINT); +-------------------+ | CAST(1 AS BIGINT) | +-------------------+ | 1 | +-------------------+ ``` Example 2: Convert the input into ARRAY. ```plain -- Convert string to ARRAY. select cast('[1,2,3]' as array); +-------------------------------+ | CAST('[1,2,3]' AS ARRAY) | +-------------------------------+ | [1,2,3] | +-------------------------------+ select cast('[1,2,3]' as array); +----------------------------------+ | CAST('[1,2,3]' AS ARRAY) | +----------------------------------+ | [1,2,3] | +----------------------------------+ select cast('[1,2,3]' as array); +------------------------------------------+ | CAST('[1,2,3]' AS ARRAY) | +------------------------------------------+ | ["1","2","3"] | +------------------------------------------+ select cast('["a", "b", "c"]' as array); +--------------------------------------------------+ | CAST('["a", "b", "c"]' AS ARRAY) | +--------------------------------------------------+ | ["a","b","c"] | +--------------------------------------------------+ -- Convert JSON array to ARRAY. select cast(parse_json('[{"a":1}, {"a": 2}]') as array); +----------------------------------------------------------+ | CAST((parse_json('[{"a":1}, {"a": 2}]')) AS ARRAY) | +----------------------------------------------------------+ | ['{"a": 1}','{"a": 2}'] | +----------------------------------------------------------+ select cast(parse_json('[1, 2, 3]') as array); +-----------------------------------------------+ | CAST((parse_json('[1, 2, 3]')) AS ARRAY) | +-----------------------------------------------+ | [1,2,3] | +-----------------------------------------------+ select cast(parse_json('["1","2","3"]') as array); +--------------------------------------------------------------+ | CAST((parse_json('["1","2","3"]')) AS ARRAY) | +--------------------------------------------------------------+ | ["1","2","3"] | +--------------------------------------------------------------+ ``` Example 3: Convert data during loading. ```bash curl --location-trusted -u : -T ~/user_data/bigint \ -H "columns: tmp_k1, k1=cast(tmp_k1 as BIGINT)" \ http://host:port/api/test/bigint/_stream_load ``` > **Note** > > If the original value is a floating-point value (such as 12.0), it will be converted to NULL. If you want to forcibly convert this type into BIGINT, see the following example: ```bash curl --location-trusted -u : -T ~/user_data/bigint \ -H "columns: tmp_k1, k1=cast(cast(tmp_k1 as DOUBLE) as BIGINT)" \ http://host:port/api/test/bigint/_stream_load ``` ```plain MySQL > select cast(cast ("11.2" as double) as bigint); +----------------------------------------+ | CAST(CAST('11.2' AS DOUBLE) AS BIGINT) | +----------------------------------------+ | 11 | +----------------------------------------+ ``` --- ### Condition #### [📄️ case](https://docs.starrocks.io/docs/sql-reference/sql-functions/condition-functions/case_when.md) [CASE is a conditional expression.](https://docs.starrocks.io/docs/sql-reference/sql-functions/condition-functions/case_when.md) #### [📄️ coalesce](https://docs.starrocks.io/docs/sql-reference/sql-functions/condition-functions/coalesce.md) [Returns the first non-NULL expression among the input parameters.](https://docs.starrocks.io/docs/sql-reference/sql-functions/condition-functions/coalesce.md) #### [📄️ if](https://docs.starrocks.io/docs/sql-reference/sql-functions/condition-functions/if.md) [If expr1 evaluates to TRUE, returns expr2.](https://docs.starrocks.io/docs/sql-reference/sql-functions/condition-functions/if.md) #### [📄️ ifnull](https://docs.starrocks.io/docs/sql-reference/sql-functions/condition-functions/ifnull.md) [If expr1 is NULL, returns expr2.](https://docs.starrocks.io/docs/sql-reference/sql-functions/condition-functions/ifnull.md) #### [📄️ nullif](https://docs.starrocks.io/docs/sql-reference/sql-functions/condition-functions/nullif.md) [Returns NULL if expr1 is equal to expr2.](https://docs.starrocks.io/docs/sql-reference/sql-functions/condition-functions/nullif.md) --- ### case CASE is a conditional expression. It returns the result in the THEN clause if a condition in the WHEN clause evaluates to true. If none of the conditions evaluate to true, it returns the result in the optional ELSE clause. If ELSE is not present, NULL is returned. #### Syntax[​](#syntax "Direct link to Syntax") The CASE expression comes in two forms: * Simple CASE ```sql CASE expression WHEN expression1 THEN result1 [WHEN expression2 THEN result2] ... [WHEN expressionN THEN resultN] [ELSE result] END ``` For this syntax, `expression` is compared to each expression in the WHEN clause. If an equal expression is found, the result in the THEN clause is returned. If no equal expression is found, the result in the ELSE clause is returned if ELSE is present. * Searched CASE ```sql CASE WHEN condition1 THEN result1 [WHEN condition2 THEN result2] ... [WHEN conditionN THEN resultN] [ELSE result] END ``` For this syntax, each condition in the WHEN clause is evaluated until one is true and the corresponding result in the THEN clause is returned. If no condition evaluates to true, the result in the ELSE clause is returned, if ELSE is present. The first CASE equals the second one as follows: ```sql CASE WHEN expression = expression1 THEN result1 [WHEN expression = expression2 THEN result2] ... [WHEN expression = expressionN THEN resultN] [ELSE result] END ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expressionN`: the expression to compare. Multiple expressions must be compatible in data types. * `conditionN`: the condition that can evaluate to a BOOLEAN value. * `resultN` must be compatible in data types. #### Return value[​](#return-value "Direct link to Return value") The return value is of the common type of all types in the THEN clause. #### Examples[​](#examples "Direct link to Examples") Suppose table `test_case` has the following data: ```sql CREATE TABLE test_case( name STRING, gender INT, ) DISTRIBUTED BY HASH(name); INSERT INTO test_case VALUES ("Andy",1), ("Jules",0), ("Angel",-1), ("Sam",null); SELECT * FROM test_case; +-------+--------+ | name | gender | +-------+--------+ | Angel | -1 | | Andy | 1 | | Sam | NULL | | Jules | 0 | +-------+--------+-------+ ``` ##### Use simple CASE[​](#use-simple-case "Direct link to Use simple CASE") * ELSE is specified and the result in ELSE is returned if no equal expression is found. ```plain mysql> select gender, case gender when 1 then 'male' when 0 then 'female' else 'error' end gender_str from test_case; +--------+------------+ | gender | gender_str | +--------+------------+ | NULL | error | | 0 | female | | 1 | male | | -1 | error | +--------+------------+ ``` * ELSE is not specified and NULL is returned if no condition evaluates to true. ```plain select gender, case gender when 1 then 'male' when 0 then 'female' end gender_str from test_case; +--------+------------+ | gender | gender_str | +--------+------------+ | 1 | male | | -1 | NULL | | NULL | NULL | | 0 | female | +--------+------------+ ``` ##### Use searched CASE with no ELSE specified[​](#use-searched-case-with-no-else-specified "Direct link to Use searched CASE with no ELSE specified") ```plain mysql> select gender, case when gender = 1 then 'male' when gender = 0 then 'female' end gender_str from test_case; +--------+------------+ | gender | gender_str | +--------+------------+ | NULL | NULL | | -1 | NULL | | 1 | male | | 0 | female | +--------+------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") case when, case, case\_when, case...when --- ### coalesce Returns the first non-NULL expression among the input parameters. Returns NULL if non-NULL expressions cannot be found. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext coalesce(expr1,...); ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr1`: the input expressions, which must evaluate to compatible data types. #### Return value[​](#return-value "Direct link to Return value") The return value has the same type as `expr1`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select coalesce(3,NULL,1,1); +-------------------------+ | coalesce(3, NULL, 1, 1) | +-------------------------+ | 3 | +-------------------------+ 1 row in set (0.00 sec) ``` --- ### if If `expr1` evaluates to TRUE, returns `expr2`. Otherwise, returns `expr3`. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext if(expr1,expr2,expr3); ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr1`: the condition. It must be a BOOLEAN value. `expr2` and `expr3` must be compatible in data type. #### Return value[​](#return-value "Direct link to Return value") The return value has the same type as `expr2`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select if(true,1,2); +----------------+ | if(TRUE, 1, 2) | +----------------+ | 1 | +----------------+ mysql> select if(false,2.14,2); +--------------------+ | if(FALSE, 2.14, 2) | +--------------------+ | 2.00 | +--------------------+ ``` --- ### ifnull If `expr1` is NULL, returns expr2. If `expr1` is not NULL, returns `expr1`. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext ifnull(expr1,expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr1` and `expr2` must be compatible in data type. #### Return value[​](#return-value "Direct link to Return value") The return value has the same type as `expr1`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select ifnull(2,4); +--------------+ | ifnull(2, 4) | +--------------+ | 2 | +--------------+ mysql> select ifnull(NULL,2); +-----------------+ | ifnull(NULL, 2) | +-----------------+ | 2 | +-----------------+ 1 row in set (0.01 sec) ``` --- ### nullif Returns NULL if `expr1` is equal to `expr2`. Otherwise, returns `expr1`. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext nullif(expr1,expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr1` and `expr2` must be compatible in data type. #### Return value[​](#return-value "Direct link to Return value") The return value has the same type as `expr1`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select nullif(1,2); +--------------+ | nullif(1, 2) | +--------------+ | 1 | +--------------+ mysql> select nullif(1,1); +--------------+ | nullif(1, 1) | +--------------+ | NULL | +--------------+ ``` --- ### Cryptographic #### [📄️ aes\_decrypt](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/aes_decrypt.md) [Uses the specified Advanced Encryption Standard (AES) algorithm and the encryption key to decrypt a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/aes_decrypt.md) #### [📄️ aes\_encrypt](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/aes_encrypt.md) [Uses the specified Advanced Encryption Standard (AES) algorithm and an encryption key to encrypt a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/aes_encrypt.md) #### [📄️ base64\_decode\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/base64_decode_binary.md) [Decodes a Base64-encoded string and return a BINARY.](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/base64_decode_binary.md) #### [📄️ base64\_decode\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/base64_decode_string.md) [Alias of from\_base64; decodes a Base64-encoded string to its original string value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/base64_decode_string.md) #### [📄️ from\_base64](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/from_base64.md) [Decodes a Base64-encoded string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/from_base64.md) #### [📄️ md5](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/md5.md) [Uses the MD5 Message-Digest Algorithm to calculate the 128-bit checksum of a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/md5.md) #### [📄️ md5sum](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/md5sum.md) [Calculates the 128-bit checksum of multiple strings.](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/md5sum.md) #### [📄️ md5sum\_numeric](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/md5sum_numeric.md) [Calculates the 128-bit checksum of multiple strings.](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/md5sum_numeric.md) #### [📄️ sha2](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/sha2.md) [Calculates the SHA-224 hash value, SHA-256 hash value, SHA-384 hash value, or SHA-512 hash value of a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/sha2.md) #### [📄️ sm3](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/sm3.md) [Uses the SM3 digest algorithm to encrypt a string into a 256-bit hexadecimal string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/sm3.md) #### [📄️ to\_base64](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/to_base64.md) [Converts a string into a Base64-encoded string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/to_base64.md) --- ### aes_decrypt Uses the specified Advanced Encryption Standard (AES) algorithm and the encryption key to decrypt a string. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext aes_decrypt(str, key_str[, init_vector][, encryption_mode][, aad_str]); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str` (Required): the string to decrypt. It must be of the VARCHAR type. * `key_str` (Required): the key used to encrypt `str`. It must be of the VARCHAR type. * `init_vector` (Optional): Initialization Vector (IV): A crucial security parameter in AES encryption that ensures identical plaintexts yield distinct ciphertexts. It is only utilized in CBC, CFB, OFB, CTR, and GCM modes. It must be of the VARCHAR type. * `encryption_mode` (Optional): The encryption algorithm. It must be of the VARCHAR type. Default value: `AES_128_ECB`. Supported algorithms: | ECB | CBC | CFB | CFB1 | CFB8 | CFB128 | OFB | CTR | GCM | | ------------- | ------------- | ------------- | -------------- | -------------- | ---------------- | ------------- | ------------- | ------------- | | AES\_128\_ECB | AES\_128\_CBC | AES\_128\_CFB | AES\_128\_CFB1 | AES\_128\_CFB8 | AES\_128\_CFB128 | AES\_128\_OFB | AES\_128\_CTR | AES\_128\_GCM | | AES\_192\_ECB | AES\_192\_CBC | AES\_192\_CFB | AES\_192\_CFB1 | AES\_192\_CFB8 | AES\_192\_CFB128 | AES\_192\_OFB | AES\_192\_CTR | AES\_192\_GCM | | AES\_256\_ECB | AES\_256\_CBC | AES\_256\_CFB | AES\_256\_CFB1 | AES\_256\_CFB8 | AES\_256\_CFB128 | AES\_256\_OFB | AES\_256\_CTR | AES\_256\_GCM | * `aad_str` (Optional): Denotes Additional Authenticated Data (AAD). This is a parameter unique to authenticated encryption modes (e.g., GCM). It allows for including data that must be authenticated for integrity (preventing tampering) but does not require confidentiality (it remains unencrypted). It must be of the VARCHAR type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. If the input is invalid, NULL is returned. #### Examples[​](#examples "Direct link to Examples") Decode a Base64 string and use this function to decrypt the decoded string into the original string. ```plain mysql> select AES_DECRYPT(from_base64('uv/Lhzm74syo8JlfWarwKA== '),'F3229A0B371ED2D9441B830D21A390C3'); +--------------------------------------------------------------------------------------------+ | aes_decrypt(from_base64('uv/Lhzm74syo8JlfWarwKA== '), 'F3229A0B371ED2D9441B830D21A390C3') | +--------------------------------------------------------------------------------------------+ | starrocks | +--------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) ``` ```text mysql> select AES_DECRYPT(from_base64('uv/Lhzm74syo8JlfWarwKA=='),'F3229A0B371ED2D9441B830D21A390C3', NULL, "AES_128_ECB"); +---------------------------------------------------------------------------------------------------------------+ | aes_decrypt(from_base64('uv/Lhzm74syo8JlfWarwKA=='), 'F3229A0B371ED2D9441B830D21A390C3', NULL, 'AES_128_ECB') | +---------------------------------------------------------------------------------------------------------------+ | starrocks | +---------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) ``` ```text mysql> select AES_DECRYPT(from_base64('taXlwIvir9yff94F5Uv/KA=='),'F3229A0B371ED2D9441B830D21A390C3', "abcdefg", "AES_128_CBC"); +--------------------------------------------------------------------------------------------------------------------+ | aes_decrypt(from_base64('taXlwIvir9yff94F5Uv/KA=='), 'F3229A0B371ED2D9441B830D21A390C3', 'abcdefg', 'AES_128_CBC') | +--------------------------------------------------------------------------------------------------------------------+ | starrocks | +--------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) ``` ```text mysql> select AES_DECRYPT(from_base64('YWJjZGVmZ2hpamtsdpJC2rnrGmvqKQv/WcoO6NuOCXvUnC8pCw=='),'F3229A0B371ED2D9441B830D21A390C3', "abcdefghijklmnop", "AES_128_GCM", "abcdefg"); +--------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | aes_decrypt(from_base64('YWJjZGVmZ2hpamtsdpJC2rnrGmvqKQv/WcoO6NuOCXvUnC8pCw=='), 'F3229A0B371ED2D9441B830D21A390C3', 'abcdefghijklmnop', 'AES_128_GCM', 'abcdefg') | +--------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | starrocks | +--------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) ``` --- ### aes_encrypt Uses the specified Advanced Encryption Standard (AES) algorithm and an encryption key to encrypt a string. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext aes_encrypt(str, key_str[, init_vector][, encryption_mode][, aad_str]); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str` (Required): the string to encrypt. It must be of the VARCHAR type. * `key_str` (Required): the key used to encrypt `str`. It must be of the VARCHAR type. * `init_vector` (Optional): Initialization Vector (IV): A crucial security parameter in AES encryption that ensures identical plaintexts yield distinct ciphertexts. It is only utilized in CBC, CFB, OFB, CTR, and GCM modes. It must be of the VARCHAR type. * `encryption_mode` (Optional): The encryption algorithm. It must be of the VARCHAR type. Default value: `AES_128_ECB`. Supported algorithms: | ECB | CBC | CFB | CFB1 | CFB8 | CFB128 | OFB | CTR | GCM | | ------------- | ------------- | ------------- | -------------- | -------------- | ---------------- | ------------- | ------------- | ------------- | | AES\_128\_ECB | AES\_128\_CBC | AES\_128\_CFB | AES\_128\_CFB1 | AES\_128\_CFB8 | AES\_128\_CFB128 | AES\_128\_OFB | AES\_128\_CTR | AES\_128\_GCM | | AES\_192\_ECB | AES\_192\_CBC | AES\_192\_CFB | AES\_192\_CFB1 | AES\_192\_CFB8 | AES\_192\_CFB128 | AES\_192\_OFB | AES\_192\_CTR | AES\_192\_GCM | | AES\_256\_ECB | AES\_256\_CBC | AES\_256\_CFB | AES\_256\_CFB1 | AES\_256\_CFB8 | AES\_256\_CFB128 | AES\_256\_OFB | AES\_256\_CTR | AES\_256\_GCM | * `aad_str` (Optional): Denotes Additional Authenticated Data (AAD). This is a parameter unique to authenticated encryption modes (e.g., GCM). It allows for including data that must be authenticated for integrity (preventing tampering) but does not require confidentiality (it remains unencrypted). It must be of the VARCHAR type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. If the input is NULL, NULL is returned. #### Examples[​](#examples "Direct link to Examples") Use this function to AES encrypt `starrocks` and convert the encrypted string into a Base64-encoded string. ```plain mysql> select to_base64(AES_ENCRYPT('starrocks','F3229A0B371ED2D9441B830D21A390C3')); +-------------------------------------------------------------------------+ | to_base64(aes_encrypt('starrocks', 'F3229A0B371ED2D9441B830D21A390C3')) | +-------------------------------------------------------------------------+ | uv/Lhzm74syo8JlfWarwKA== | +-------------------------------------------------------------------------+ 1 row in set (0.01 sec) ``` ```text mysql> select to_base64(AES_ENCRYPT('starrocks','F3229A0B371ED2D9441B830D21A390C3', NULL, "AES_128_ECB")); +----------------------------------------------------------------------------------------------+ | to_base64(aes_encrypt('starrocks', 'F3229A0B371ED2D9441B830D21A390C3', NULL, 'AES_128_ECB')) | +----------------------------------------------------------------------------------------------+ | uv/Lhzm74syo8JlfWarwKA== | +----------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) ``` ```text mysql> select to_base64(AES_ENCRYPT('starrocks','F3229A0B371ED2D9441B830D21A390C3', "abcdefg", "AES_128_CBC")); +---------------------------------------------------------------------------------------------------+ | to_base64(aes_encrypt('starrocks', 'F3229A0B371ED2D9441B830D21A390C3', 'abcdefg', 'AES_128_CBC')) | +---------------------------------------------------------------------------------------------------+ | taXlwIvir9yff94F5Uv/KA== | +---------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) ``` ```text mysql> select to_base64(AES_ENCRYPT('starrocks','F3229A0B371ED2D9441B830D21A390C3', "abcdefghijklmnop", "AES_128_GCM", "abcdefg")); +-----------------------------------------------------------------------------------------------------------------------+ | to_base64(aes_encrypt('starrocks', 'F3229A0B371ED2D9441B830D21A390C3', 'abcdefghijklmnop', 'AES_128_GCM', 'abcdefg')) | +-----------------------------------------------------------------------------------------------------------------------+ | YWJjZGVmZ2hpamtsdpJC2rnrGmvqKQv/WcoO6NuOCXvUnC8pCw== | +-----------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.01 sec) ``` --- ### base64_decode_binary Decodes a Base64-encoded string and return a BINARY. This function is supported from v3.0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext base64_decode_binary(str); ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: the string to decode. It must be of the VARCHAR type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARBINARY type. If the input is NULL or an invalid Base64 string, NULL is returned. If the input is empty, an error is returned. This function accepts only one string. More than one input string causes an error. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select hex(base64_decode_binary(to_base64("Hello StarRocks"))); +---------------------------------------------------------+ | hex(base64_decode_binary(to_base64('Hello StarRocks'))) | +---------------------------------------------------------+ | 48656C6C6F2053746172526F636B73 | +---------------------------------------------------------+ mysql> select base64_decode_binary(NULL); +--------------------------------------------------------+ | base64_decode_binary(NULL) | +--------------------------------------------------------+ | NULL | +--------------------------------------------------------+ ``` --- ### base64_decode_string This function is the same as [from\_base64](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/from_base64.md). Decodes a Base64-encoded string. and an inverse of [to\_base64](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/to_base64.md). This function is supported from v3.0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext base64_decode_string(str); ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: the string to decode. It must be of the VARCHAR type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. If the input is NULL or an invalid Base64 string, NULL is returned. If the input is empty, an error is returned. This function accepts only one string. More than one input string causes an error. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select base64_decode_string(to_base64("Hello StarRocks")); +----------------------------------------------------+ | base64_decode_string(to_base64('Hello StarRocks')) | +----------------------------------------------------+ | Hello StarRocks | +----------------------------------------------------+ ``` --- ### from_base64 Decodes a Base64-encoded string. This function is an inverse of [to\_base64](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/to_base64.md). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext from_base64(str); ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: the string to decode. It must be of the VARCHAR type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. If the input is NULL or an invalid Base64 string, NULL is returned. If the input is empty, an error is returned. This function accepts only one string. More than one input string causes an error. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select from_base64("starrocks"); +--------------------------+ | from_base64('starrocks') | +--------------------------+ | ²֫®$ | +--------------------------+ 1 row in set (0.00 sec) mysql> select from_base64('c3RhcnJvY2tz'); +-----------------------------+ | from_base64('c3RhcnJvY2tz') | +-----------------------------+ | starrocks | +-----------------------------+ ``` --- ### md5 Uses the MD5 Message-Digest Algorithm to calculate the 128-bit checksum of a string. The checksum is represented by a 32-character hexadecimal string. #### Syntax[​](#syntax "Direct link to Syntax") ```sql md5(expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: the string to calculate. It must be of the VARCHAR type. #### Return value[​](#return-value "Direct link to Return value") Returns a checksum of the VARCHAR type, which is a 32-character hexadecimal string. If the input is NULL, NULL is returned. #### Examples[​](#examples "Direct link to Examples") ```sql select md5('abc'); ``` ```plaintext +----------------------------------+ | md5('abc') | +----------------------------------+ | 900150983cd24fb0d6963f7d28e17f72 | +----------------------------------+ 1 row in set (0.01 sec) ``` ```sql select md5(null); ``` ```plaintext +-----------+ | md5(NULL) | +-----------+ | NULL | +-----------+ 1 row in set (0.00 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") MD5, ENCRYPTION --- ### md5sum Calculates the 128-bit checksum of multiple strings. The checksum is represented by a 32-character hexadecimal string. If only one string is passed in, the result is the same as that of [md5](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/md5.md). Compared to md5(), md5sum() accepts more parameters and therefore, is more efficient in calculating the checksum of multiple files. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext md5sum(expr,...); ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: the string to calculate. It must be of the VARCHAR type. #### Return value[​](#return-value "Direct link to Return value") Returns a checksum of the VARCHAR type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select md5sum("starrocks"); +----------------------------------+ | md5sum('starrocks') | +----------------------------------+ | f75523a916caf65f1ad487a9f8017f75 | +----------------------------------+ 1 row in set (0.01 sec) mysql> select md5sum("starrocks","star"); +----------------------------------+ | md5sum('starrocks', 'star') | +----------------------------------+ | 7af4bfe35b8df2786ad133c57cb2aed8 | +----------------------------------+ 1 row in set (0.01 sec) mysql> select md5sum(null); +----------------------------------+ | md5sum(NULL) | +----------------------------------+ | d41d8cd98f00b204e9800998ecf8427e | +----------------------------------+ 1 row in set (0.01 sec) ``` --- ### md5sum_numeric Calculates the 128-bit checksum of multiple strings. The checksum is represented by a decimal string of digits. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext LARGEINT md5sum_numeric(VARCHAR expr,...); ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: the string to calculate. It must be of the VARCHAR type. #### Return value[​](#return-value "Direct link to Return value") Returns a checksum of the LARGEINT type. If the input is empty, an error is returned. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select md5sum_numeric("starrocks"); +-----------------------------------------+ | md5sum_numeric('starrocks') | +-----------------------------------------+ | 328761383472799310362963866384446095221 | +-----------------------------------------+ 1 row in set (0.01 sec) mysql> select md5sum_numeric("starrocks","star"); +-----------------------------------------+ | md5sum_numeric('starrocks', 'star') | +-----------------------------------------+ | 163436627872604162110984707546327199448 | +-----------------------------------------+ 1 row in set (0.01 sec) mysql> select md5sum_numeric(null); +-----------------------------------------+ | md5sum_numeric(NULL) | +-----------------------------------------+ | 281949768489412648962353822266799178366 | +-----------------------------------------+ 1 row in set (0.01 sec) ``` --- ### sha2 Calculates the SHA-224 hash value, SHA-256 hash value, SHA-384 hash value, or SHA-512 hash value of a string. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR sha2(VARCHAR expr, INT hash_length) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `Expr`: the string whose value you want to calculate. * `hash_length`: the length of a hash value. The value of this parameter can be 224, 256, 384, 512, or 0. The value 0 is equivalent to 256. If you set this parameter to any other value, this function returns `NULL`. #### Return value[​](#return-value "Direct link to Return value") Returns a hash value of the VARCHAR type. If any of the two input parameters is `Null`, `Null` is returned. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select sha2('abc',224); +----------------------------------------------------------+ | sha2('abc', 224) | +----------------------------------------------------------+ | 23097d223405d8228642a477bda255b32aadbce4bda0b3f7e36c9da7 | +----------------------------------------------------------+ 1 row in set (0.01 sec) mysql> select sha2('abc', 384); +--------------------------------------------------------------------------------------------------+ | sha2('abc', 384) | +--------------------------------------------------------------------------------------------------+ | cb00753f45a35e8bb5a03d699ac65007272c32ab0eded1631a8b605a43ff5bed8086072ba1e7cc2358baeca134c825a7 | +--------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) mysql> select sha2('abc', 1),sha2('abc', null), sha2(null, 384); +----------------+-------------------+-----------------+ | sha2('abc', 1) | sha2('abc', NULL) | sha2(NULL, 384) | +----------------+-------------------+-----------------+ | NULL | NULL | NULL | +----------------+-------------------+-----------------+ 1 row in set (0.01 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") SHA2, ENCRYPTION --- ### sm3 Uses the SM3 digest algorithm to encrypt a string into a 256-bit hexadecimal string. Every 32 bits are separated by a space. Digest algorithms have a broad range of use cases in digital signature, message authentication, and data integrity check. The SM3 algorithm is an enhancement of SHA-256. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext SM3(str); ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: the string to encrypt. It must be of the VARCHAR type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. If the input is NULL, NULL is returned. If the input is empty, an error is returned. This function accepts only one string. More than one input string causes an error. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select sm3('l'); +-------------------------------------------------------------------------+ | sm3('l') | +-------------------------------------------------------------------------+ | 1dec1a7a 94236240 49db411e 2c32c62d c0c93856 8208ac3a 09d395eb 2468b445 | +-------------------------------------------------------------------------+ 1 row in set (0.01 sec) ``` --- ### to_base64 Converts a string into a Base64-encoded string. This function is an inverse of [from\_base64](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/from_base64.md). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext to_base64(str); ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: the string to encode. It must be of the VARCHAR type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. If the input is NULL, NULL is returned. If the input is empty, an error is returned. This function accepts only one string. More than one input string causes an error. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select to_base64("starrocks"); +------------------------+ | to_base64('starrocks') | +------------------------+ | c3RhcnJvY2tz | +------------------------+ 1 row in set (0.00 sec) mysql> select to_base64(123); +----------------+ | to_base64(123) | +----------------+ | MTIz | +----------------+ ``` --- ### add_months Adds a specified number of months to a given date (DATE or DATETIME). The [months\_add](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/months_add.md) function provides similar functionalities. The day component in the resulting month remains the same as that specified in `date`, unless the resulting month has fewer days than the day component of the given date, in which case the day will be the last day of the resulting month. For example, `select add_months('2022-01-31', 1);` returns `2022-02-28 00:00:00`. If the resulting month has more days than the day component of the given date, the result has the same day component as `date`. For example, `select add_months('2022-02-28', 1)` returns `2022-03-28 00:00:00`. Difference with Oracle: In Oracle, if `date` is the last day of the month, then the result is the last day of the resulting month. Returns NULL if an invalid date or a NULL argument is passed in. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext ADD_MONTH(date, months) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `date`: It must be a valid date or datetime expression. * `months`: the months you want to add. It must be an integer. A positive integer adds months to `date`. A negative integer subtracts months from `date`. #### Return value[​](#return-value "Direct link to Return value") Returns a DATETIME value. If the date does not exist, for example, `2020-02-30`, NULL is returned. If the date is a DATE value, it will be converted into a DATETIME value. #### Examples[​](#examples "Direct link to Examples") ```plain > select add_months('2022-01-01', 2); +-----------------------------+ | add_months('2022-01-01', 2) | +-----------------------------+ | 2022-03-01 00:00:00 | +-----------------------------+ > select add_months('2022-01-01', -5); +------------------------------+ | add_months('2022-01-01', -5) | +------------------------------+ | 2021-08-01 00:00:00 | +------------------------------+ > select add_months('2022-01-31', 1); +-----------------------------+ | add_months('2022-01-31', 1) | +-----------------------------+ | 2022-02-28 00:00:00 | +-----------------------------+ > select add_months('2022-01-31 17:01:02', -2); +---------------------------------------+ | add_months('2022-01-31 17:01:02', -2) | +---------------------------------------+ | 2021-11-30 17:01:02 | +---------------------------------------+ > select add_months('2022-02-28', 1); +-----------------------------+ | add_months('2022-02-28', 1) | +-----------------------------+ | 2022-03-28 00:00:00 | +-----------------------------+ ``` --- ### adddate,date_add This function is a synonym of [date\_add](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_add.md). --- ### convert_tz Converts a DATE or DATETIME value from one time zone to another. This function may return different results for different time zones. For more information, see [Configure a time zone](https://docs.starrocks.io/docs/administration/management/timezone.md). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME CONVERT_TZ(DATETIME|DATE dt, VARCHAR from_tz, VARCHAR to_tz) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `dt`: the DATE or DATETIME value to convert. * `from_tz`: the source time zone. VARCHAR is supported. The time zone can be represented in two formats: one is Time Zone Database (for example, Asia/Shanghai) and the other is UTC offset (for example, +08:00). * `to_tz`: the destination time zone. VARCHAR is supported. Its format is the same as `from_tz`. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DATETIME data type. If the input is a DATE value, it will be converted into a DATETIME value. This function returns NULL if any of the input parameters is invalid or NULL. #### Usage notes[​](#usage-notes "Direct link to Usage notes") For Time Zone Database, see [List of tz database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) (in Wikipedia). #### Examples[​](#examples "Direct link to Examples") Example 1: Convert a datetime in Shanghai to Los\_Angeles. ```plaintext select convert_tz('2019-08-01 13:21:03', 'Asia/Shanghai', 'America/Los_Angeles'); +---------------------------------------------------------------------------+ | convert_tz('2019-08-01 13:21:03', 'Asia/Shanghai', 'America/Los_Angeles') | +---------------------------------------------------------------------------+ | 2019-07-31 22:21:03 | +---------------------------------------------------------------------------+ 1 row in set (0.00 sec) | ``` Example 2: Convert a date in Shanghai to Los\_Angeles. ```plaintext select convert_tz('2019-08-01', 'Asia/Shanghai', 'America/Los_Angeles'); +------------------------------------------------------------------+ | convert_tz('2019-08-01', 'Asia/Shanghai', 'America/Los_Angeles') | +------------------------------------------------------------------+ | 2019-07-31 09:00:00 | +------------------------------------------------------------------+ 1 row in set (0.00 sec) ``` Example 3: Convert a datetime in UTC+08:00 to Los\_Angeles. ```plaintext select convert_tz('2019-08-01 13:21:03', '+08:00', 'America/Los_Angeles'); +--------------------------------------------------------------------+ | convert_tz('2019-08-01 13:21:03', '+08:00', 'America/Los_Angeles') | +--------------------------------------------------------------------+ | 2019-07-31 22:21:03 | +--------------------------------------------------------------------+ 1 row in set (0.00 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") CONVERT\_TZ, timezone, time zone --- ### curdate,current_date Obtains the current date and returns a value of the DATE type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATE CURDATE() ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> SELECT CURDATE(); +------------+ | curdate() | +------------+ | 2024-03-27 | +------------+ mysql> SELECT CURRENT_DATE(); +----------------+ | current_date() | +----------------+ | 2024-03-27 | +----------------+ mysql> SELECT CURDATE() + 0; +-----------------+ | (curdate()) + 0 | +-----------------+ | 20240327 | +-----------------+ mysql> SELECT CURDATE() +2; +-----------------+ | (curdate()) + 2 | +-----------------+ | 20240329 | +-----------------+ mysql> SELECT CURDATE() - INTERVAL 5 DAY; +----------------------------+ | curdate() - INTERVAL 5 DAY | +----------------------------+ | 2024-03-22 00:00:00 | +----------------------------+ mysql> SELECT QUARTER(CURDATE()); +--------------------+ | quarter(curdate()) | +--------------------+ | 1 | +--------------------+ ``` --- ### current_timestamp Obtains the current date and returns a value if the DATETIME type. This function is a synonym of the [now()](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/now.md) function. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME CURRENT_TIMESTAMP() DATETIME CURRENT_TIMESTAMP(INT p) ``` #### Parameters[​](#parameters "Direct link to Parameters") `p`: optional, the specified precision, that is, the number of digits to retain after seconds. It must be an INT value within the range of \[1,6]. `select current_timestamp(0)` is equivalent to `select current_timestamp()`. #### Return value[​](#return-value "Direct link to Return value") * If `p` is not specified, this function returns a DATETIME value accurate to the second. * If `p` is specified, this function returns a date and time value of the specified precision. #### Examples[​](#examples "Direct link to Examples") Example 1: Return the current time. ```plain MySQL > select current_timestamp(); +---------------------+ | current_timestamp() | +---------------------+ | 2019-05-27 15:59:33 | +---------------------+ ``` Example 2: When you create a table, you can use this function for a column so that current time is the default value of the column. ```sql CREATE TABLE IF NOT EXISTS sr_member ( sr_id INT, name STRING, city_code INT, reg_date DATETIME DEFAULT current_timestamp, verified BOOLEAN ); ``` #### keyword[​](#keyword "Direct link to keyword") CURRENT\_TIMESTAMP,CURRENT,TIMESTAMP --- ### current_timezone Obtains the current time zone and returns a value of the VARCHAR type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR CURRENT_TIMETIME() ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select current_timezone(); +---------------------+ | current_timezone() | +---------------------+ | America/Los_Angeles | +---------------------+ ``` --- ### curtime,current_time Obtains the current time and returns a value of the TIME type. This function may return different results for different time zones. For more information, see [Configure a time zone](https://docs.starrocks.io/docs/administration/management/timezone.md). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext TIME CURTIME() ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select current_time(); +----------------+ | current_time() | +----------------+ | 15:25:47 | +----------------+ ``` #### keyword[​](#keyword "Direct link to keyword") CURTIME,CURRENT\_TIME --- ### date Extracts the date part of a date or datetime expression. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATE date(DATETIME|DATE expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: the date or datetime expression. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DATE type. NULL is returned if the input is NULL or invalid. #### Examples[​](#examples "Direct link to Examples") Example 1: Extract the date part of a datetime value. ```plaintext SELECT DATE("2017-12-31 11:20:59"); +-----------------------------+ | date('2017-12-31 11:20:59') | +-----------------------------+ | 2017-12-31 | +-----------------------------+ 1 row in set (0.05 sec) ``` Example 2: Extract the date part of a date value. ```plaintext SELECT DATE('2017-12-31'); +--------------------+ | date('2017-12-31') | +--------------------+ | 2017-12-31 | +--------------------+ 1 row in set (0.08 sec) ``` Example 3: Extract the date part of the current timestamp. ```plaintext SELECT DATE(current_timestamp()); +---------------------------+ | date(current_timestamp()) | +---------------------------+ | 2022-11-08 | +---------------------------+ 1 row in set (0.05 sec) ``` --- ### date_add,adddate Adds a specified time interval to a date. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME DATE_ADD(DATETIME|DATE date,INTERVAL expr type) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `date`: It must be a valid date or datetime expression. * `expr`: the time interval you want to add. It must be of the INT type. * `type`: the unit of the time interval. It can only be set to any of the following values: YEAR, QUARTER, MONTH, DAY, HOUR, MINUTE, SECOND, MILLISECOND (since 3.1.7), and MICROSECOND (since 3.1.7). #### Return value[​](#return-value "Direct link to Return value") Returns a DATETIME value. If the date does not exist, for example, `2020-02-30`, NULL is returned. If the date is a DATE value, it will be converted into a DATETIME value. #### Examples[​](#examples "Direct link to Examples") ```plain select date_add('2010-11-30 23:59:59', INTERVAL 2 DAY); +-------------------------------------------------+ | date_add('2010-11-30 23:59:59', INTERVAL 2 DAY) | +-------------------------------------------------+ | 2010-12-02 23:59:59 | +-------------------------------------------------+ select date_add('2010-12-03', INTERVAL 2 DAY); +----------------------------------------+ | date_add('2010-12-03', INTERVAL 2 DAY) | +----------------------------------------+ | 2010-12-05 00:00:00 | +----------------------------------------+ select date_add('2010-11-30 23:59:59', INTERVAL 2 QUARTER); +-----------------------------------------------------+ | date_add('2010-11-30 23:59:59', INTERVAL 2 QUARTER) | +-----------------------------------------------------+ | 2011-05-30 23:59:59 | +-----------------------------------------------------+ select adddate('2023-10-31 23:59:59', INTERVAL 1 MILLISECOND); +--------------------------------------------------------+ | adddate('2023-10-31 23:59:59', INTERVAL 1 MILLISECOND) | +--------------------------------------------------------+ | 2023-10-31 23:59:59.001000 | +--------------------------------------------------------+ select adddate('2023-10-31 23:59:59', INTERVAL 1 MICROSECOND); +--------------------------------------------------------+ | adddate('2023-10-31 23:59:59', INTERVAL 1 MICROSECOND) | +--------------------------------------------------------+ | 2023-10-31 23:59:59.000001 | +--------------------------------------------------------+ ``` --- ### date_diff Returns the difference between two date values in the specified unit. This function returns the value of `expr1 - expr2` expressed in terms of `unit`. For example, `date_diff('second', '2010-11-30 23:59:59', '2010-11-30 20:58:59')` returns the difference between the two DATETIME values in unit of seconds. This function is supported from v3.1. The difference between date\_diff and [datediff](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/datediff.md) lies in that datediff() does not support the `unit` parameter. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT DATE_DIFF(VARCHAR unit, DATETIME expr1, DATETIME expr2) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr1` and `expr2`: the two datetime values you want to compare, required. Supported data types are DATETIME and DATE. * `unit`: the unit used to express the time difference, required. The following `unit` values are supported: year, quarter, month, week, day, hour, minute, second, millisecond. #### Return value[​](#return-value "Direct link to Return value") Returns a BIGINT value. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * If `expr1` is earlier than `expr2`, a negative value is returned. * If `unit` is invalid, an error is returned. * If any input value is NULL, NULL is returned. * If the specified date does not exist, for example, `2022-11-31`, NULL is returned. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select date_diff('second', '2010-11-30 23:59:59', '2010-11-30 20:58:59'); +-------------------------------------------------------------------+ | date_diff('second', '2010-11-30 23:59:59', '2010-11-30 20:58:59') | +-------------------------------------------------------------------+ | 10860 | +-------------------------------------------------------------------+ mysql> select date_diff('minute', '2010-11-30 23:59:59', '2010-11-30 20:58:59'); +-------------------------------------------------------------------+ | date_diff('minute', '2010-11-30 23:59:59', '2010-11-30 20:58:59') | +-------------------------------------------------------------------+ | 181 | +-------------------------------------------------------------------+ mysql> select date_diff('hour', '2010-11-30 23:59:59', '2010-11-30 20:58:59'); +-----------------------------------------------------------------+ | date_diff('hour', '2010-11-30 23:59:59', '2010-11-30 20:58:59') | +-----------------------------------------------------------------+ | 3 | +-----------------------------------------------------------------+ mysql> select date_diff('millisecond', '2010-11-30 23:59:59', '2010-11-30 20:58:59'); +------------------------------------------------------------------------+ | date_diff('millisecond', '2010-11-30 23:59:59', '2010-11-30 20:58:59') | +------------------------------------------------------------------------+ | 10860000 | +------------------------------------------------------------------------+ ``` #### References[​](#references "Direct link to References") [datediff](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/datediff.md) --- ### date_format Converts a date into a string according to the specified format. Currently it supports strings with a maximum of 128 bytes. If the length of the returned value exceeds 128, NULL is returned. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR DATE_FORMAT(DATETIME date, VARCHAR format) ``` #### Parameters[​](#parameters "Direct link to Parameters") * The `date` parameter must be a valid date or date expression. * `format` specifies the output format of the date or time. Here are the formats available: ```plain %a | Abbreviated weekday name (Sun to Sat) %b | Abbreviated month name (Jan to Dec) %c | Numeric month name (0-12) %D | Day of the month as a numeric value, followed by suffix in English %d | Day of the month as a numeric value (00-31) %e | Day of the month as a numeric value (0-31) %f | Microseconds %H | Hour (00-23) %h | Hour (01-12) %I | Hour (01-12) %i | Minutes (00-59) %j | Day of the year (001-366) %k | Hour (0-23) %l | Hour (1-12) %M | Month name in full %m | Month name as a numeric value (00-12) %p | AM or PM %r | Time in 12 hour (hh:mm:ss AM or PM) %S | Seconds (00-59) %s | Seconds (00-59) %T | Time in 24 hour format (hh:mm:ss) %U | Week (00-53) where Sunday is the first day of the week %u | Week (00-53) where Monday is the first day of the week %V | Week (01-53) where Sunday is the first day of the week. Used with %X. %v | Week (01-53) where Monday is the first day of the week. Used with %x. %W | Weekday name in full %w | Day of the week where Sunday=0 and Saturday=6 %X | Year for the week where Sunday is the first day of the week. 4-digital value. Used with %V. %x | Year for the week where Monday is the first day of the week. 4-digital value. Used with %v. %Y | Year. 4-digital value. %y | Year. 2-digital value. %% | Represent %. ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select date_format('2009-10-04 22:23:00', '%W %M %Y'); +------------------------------------------------+ | date_format('2009-10-04 22:23:00', '%W %M %Y') | +------------------------------------------------+ | Sunday October 2009 | +------------------------------------------------+ MySQL > select date_format('2007-10-04 22:23:00', '%H:%i:%s'); +------------------------------------------------+ | date_format('2007-10-04 22:23:00', '%H:%i:%s') | +------------------------------------------------+ | 22:23:00 | +------------------------------------------------+ MySQL > select date_format('1900-10-04 22:23:00', '%D %y %a %d %m %b %j'); +------------------------------------------------------------+ | date_format('1900-10-04 22:23:00', '%D %y %a %d %m %b %j') | +------------------------------------------------------------+ | 4th 00 Thu 04 10 Oct 277 | +------------------------------------------------------------+ MySQL > select date_format('1997-10-04 22:23:00', '%H %k %I %r %T %S %w'); +------------------------------------------------------------+ | date_format('1997-10-04 22:23:00', '%H %k %I %r %T %S %w') | +------------------------------------------------------------+ | 22 22 10 10:23:00 PM 22:23:00 00 6 | +------------------------------------------------------------+ MySQL > select date_format('1999-01-01 00:00:00', '%X %V'); +---------------------------------------------+ | date_format('1999-01-01 00:00:00', '%X %V') | +---------------------------------------------+ | 1998 52 | +---------------------------------------------+ MySQL > select date_format('2006-06-01', '%d'); +------------------------------------------+ | date_format('2006-06-01 00:00:00', '%d') | +------------------------------------------+ | 01 | +------------------------------------------+ MySQL > select date_format('2006-06-01', '%%%d'); +--------------------------------------------+ | date_format('2006-06-01 00:00:00', '%%%d') | +--------------------------------------------+ | %01 | +--------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") DATE\_FORMAT,DATE,FORMAT --- ### date_slice Converts a given time into the beginning or end of a time interval based on the specified time granularity. This function is supported from v2.5. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATE date_slice(DATE dt, INTERVAL N type[, boundary]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `dt`: the time to convert, DATE. * `INTERVAL N type`: the time granularity, for example, `interval 5 day`. * `N` is the length of time interval. It must be an INT value. * `type` is the unit, which can be YEAR, QUARTER, MONTH, WEEK, DAY, HOUR, MINUTE, SECOND. If `type` is set to HOUR, MINUTE, or SECOND for a DATE value, an error is returned. * `boundary`: optional. It is used to specify whether to return the beginning (`FLOOR`) or end (`CEIL`) of the time interval. Valid values: FLOOR, CEIL. If this parameter is not specified, `FLOOR` is the default. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DATE type. #### Usage notes[​](#usage-notes "Direct link to Usage notes") The time interval starts from A.D. `0001-01-01 00:00:00`. #### Examples[​](#examples "Direct link to Examples") Example 1: Convert a given time to the beginning of a 5-year time interval without specifying the `boundary` parameter. ```plaintext select date_slice('2022-04-26', interval 5 year); +--------------------------------------------------+ | date_slice('2022-04-26', INTERVAL 5 year, floor) | +--------------------------------------------------+ | 2021-01-01 | +--------------------------------------------------+ ``` Example 2: Convert a given time to the end of a 5-day time interval. ```plaintext select date_slice('0001-01-07', interval 5 day, CEIL); +------------------------------------------------+ | date_slice('0001-01-07', INTERVAL 5 day, ceil) | +------------------------------------------------+ | 0001-01-11 | +------------------------------------------------+ ``` The following examples are provided based on the `test_all_type_select` table. ```plaintext select * from test_all_type_select order by id_int; +------------+---------------------+--------+ | id_date | id_datetime | id_int | +------------+---------------------+--------+ | 2052-12-26 | 1691-12-23 04:01:09 | 0 | | 2168-08-05 | 2169-12-18 15:44:31 | 1 | | 1737-02-06 | 1840-11-23 13:09:50 | 2 | | 2245-10-01 | 1751-03-21 00:19:04 | 3 | | 1889-10-27 | 1861-09-12 13:28:18 | 4 | +------------+---------------------+--------+ 5 rows in set (0.06 sec) ``` Example 3: Convert a given DATE value to the beginning of a 5-second time interval. ```plaintext select date_slice(id_date, interval 5 second, FLOOR) from test_all_type_select order by id_int; ERROR 1064 (HY000): can't use date_slice for date with time(hour/minute/second) ``` An error is returned because the system cannot find the second part of a DATE value. Example 4: Convert a given DATE value to the beginning of a 5-day time interval. ```plaintext select date_slice(id_date, interval 5 day, FLOOR) from test_all_type_select order by id_int; +--------------------------------------------+ | date_slice(id_date, INTERVAL 5 day, floor) | +--------------------------------------------+ | 2052-12-24 | | 2168-08-03 | | 1737-02-04 | | 2245-09-29 | | 1889-10-25 | +--------------------------------------------+ 5 rows in set (0.14 sec) ``` Example 5: Convert a given DATE value to the end of a 5-day time interval. ```plaintext select date_slice(id_date, interval 5 day, CEIL) from test_all_type_select order by id_int; +-------------------------------------------+ | date_slice(id_date, INTERVAL 5 day, ceil) | +-------------------------------------------+ | 2052-12-29 | | 2168-08-08 | | 1737-02-09 | | 2245-10-04 | | 1889-10-30 | +-------------------------------------------+ 5 rows in set (0.17 sec) ``` --- ### date_sub,subdate Subtracts the specified time interval from a date. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME DATE_SUB(DATETIME|DATE date,INTERVAL expr type) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `date`: It must be a valid DATE or DATETIME expression. * `expr`: the time interval you want to subtract. It must be of the INT type. * `type`: the unit of the time interval. It can only be set to any of the following values: YEAR, QUARTER, MONTH, DAY, HOUR, MINUTE, SECOND, MILLISECOND (since 3.1.7), and MICROSECOND (since 3.1.7). #### Return value[​](#return-value "Direct link to Return value") Returns a DATETIME value. If the date does not exist, for example, `2020-02-30`, or if the date is not a DATE or DATETIME value, NULL is returned. #### Examples[​](#examples "Direct link to Examples") ```plain select date_sub('2010-11-30 23:59:59', INTERVAL 2 DAY); +-------------------------------------------------+ | date_sub('2010-11-30 23:59:59', INTERVAL 2 DAY) | +-------------------------------------------------+ | 2010-11-28 23:59:59 | +-------------------------------------------------+ select date_sub('2010-11-30', INTERVAL 2 hour); +-----------------------------------------+ | date_sub('2010-11-30', INTERVAL 2 HOUR) | +-----------------------------------------+ | 2010-11-29 22:00:00 | +-----------------------------------------+ select date_sub('2010-02-30', INTERVAL 2 DAY); +----------------------------------------+ | date_sub('2010-02-30', INTERVAL 2 DAY) | +----------------------------------------+ | NULL | +----------------------------------------+ select date_sub('2010-11-30 23:59:59', INTERVAL 2 QUARTER); +-----------------------------------------------------+ | date_sub('2010-11-30 23:59:59', INTERVAL 2 QUARTER) | +-----------------------------------------------------+ | 2010-05-30 23:59:59 | +-----------------------------------------------------+ select subdate('2010-11-30 23:59:59', INTERVAL 2 millisecond); +--------------------------------------------------------+ | subdate('2010-11-30 23:59:59', INTERVAL 2 MILLISECOND) | +--------------------------------------------------------+ | 2010-11-30 23:59:58.998000 | +--------------------------------------------------------+ select date_sub('2010-11-30 23:59:59', INTERVAL 2 microsecond); +---------------------------------------------------------+ | date_sub('2010-11-30 23:59:59', INTERVAL 2 MICROSECOND) | +---------------------------------------------------------+ | 2010-11-30 23:59:58.999998 | +---------------------------------------------------------+ ``` --- ### date_trunc Truncates a time value based on the specified date part, such as year, day, hour, or minute. StarRocks also provides the year, quarter, month, week, day, and hour functions for you to extract the specified date part. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME|DATE date_trunc(VARCHAR fmt, DATETIME|DATE datetime) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `datetime`: the time to truncate, which can be of the DATETIME or DATE type. The date and time must exist. Otherwise, NULL will be returned. For example, `2021-02-29 11:12:13` does not exist as a date and NULL will be returned. * `fmt`: the date part, that is, to which precision `datetime` will be truncated. The value must be a VARCHAR constant. `fmt` must be set to a value listed in the following table. If the value is incorrect, an error will be returned. If `datetime` is a DATE value, `fmt` can only be `year`, `quarter`, `month`, `week`, or `day`. If you set `fmt` to other date units, for example `hour`, an error is reported. See Example 8. | Value | Description | | ----------- | ---------------------------------------------------------------------------------------------------- | | microsecond | Truncates to the microsecond (since 3.1.7). | | millisecond | Truncates to the millisecond (since 3.1.7). | | second | Truncates to the second. | | minute | Truncates to the minute. The second part will be zero out. | | hour | Truncates to the hour. The minute and second parts will be zero out. | | day | Truncates to the day. The time part will be zero out. | | week | Truncates to the first date of the week that `datetime` falls in. The time part will be zero out. | | month | Truncates to the first date of the month that `datetime` falls in. The time part will be zero out. | | quarter | Truncates to the first date of the quarter that `datetime` falls in. The time part will be zero out. | | year | Truncates to the first date of the year that `datetime` falls in. The time part will be zero out. | #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DATETIME or DATE type. If `datetime` is of the DATE type and `fmt` is set to `hour`, `minute`, or `second`, the time part of the returned value defaults to `00:00:00`. #### Examples[​](#examples "Direct link to Examples") Example 1: Truncate the input time to the microsecond. ```plain select date_trunc('microsecond', '2023-10-31 23:59:59.001002'); +---------------------------------------------------------+ | date_trunc('microsecond', '2023-10-31 23:59:59.001002') | +---------------------------------------------------------+ | 2023-10-31 23:59:59.001002 | +---------------------------------------------------------+ ``` Example 2: Truncate the input time to the millisecond. ```plain select date_trunc('millisecond', '2023-10-31 23:59:59.001002'); +---------------------------------------------------------+ | date_trunc('millisecond', '2023-10-31 23:59:59.001002') | +---------------------------------------------------------+ | 2023-10-31 23:59:59.001000 | +---------------------------------------------------------+ select date_trunc('millisecond', '2023-10-31'); +-----------------------------------------+ | date_trunc('millisecond', '2023-10-31') | +-----------------------------------------+ | 2023-10-31 00:00:00 | +-----------------------------------------+ ``` Example 3: Truncate the input time to the minute. ```plain select date_trunc('minute', '2023-11-04 23:59:59.001002'); +----------------------------------------------------+ | date_trunc('minute', '2023-11-04 23:59:59.001002') | +----------------------------------------------------+ | 2023-11-04 23:59:00 | +----------------------------------------------------+ ``` Example 4: Truncate the input time to the hour. ```plain select date_trunc("hour", "2020-11-04 11:12:13"); +-------------------------------------------+ | date_trunc('hour', '2020-11-04 11:12:13') | +-------------------------------------------+ | 2020-11-04 11:00:00 | +-------------------------------------------+ ``` Example 5: Truncate the input time to the first day of a week. ```plain select date_trunc("week", "2020-11-04 11:12:13"); +-------------------------------------------+ | date_trunc('week', '2020-11-04 11:12:13') | +-------------------------------------------+ | 2020-11-02 00:00:00 | +-------------------------------------------+ ``` Example 6: Truncate the input time to the first day of a quarter. ```plain select date_trunc("quarter", "2020-11-04 11:12:13"); +----------------------------------------------+ | date_trunc('quarter', '2020-11-04 11:12:13') | +----------------------------------------------+ | 2020-10-01 00:00:00 | +----------------------------------------------+ ``` Example 7: Truncate the input time to the first day of a year. ```plain select date_trunc('year', '2023-11-04 23:59:59.001002'); +--------------------------------------------------+ | date_trunc('year', '2023-11-04 23:59:59.001002') | +--------------------------------------------------+ | 2023-01-01 00:00:00 | +--------------------------------------------------+ ``` Example 8: Truncate a DATE value to the hour. An error is returned. ```plain select date_trunc("hour", cast("2020-11-04" as date)); ERROR 1064 (HY000): Getting analyzing error from line 1, column 26 to line 1, column 51. Detail message: date_trunc function can't support argument other than year|quarter|month|week|day. ``` Example 9: Truncate the input date to the first day of a year. ```plain select date_trunc("month", cast("2020-11-04" as date)); +-------------------------------------------------+ | date_trunc('month', CAST('2020-11-04' AS DATE)) | +-------------------------------------------------+ | 2020-11-01 | +-------------------------------------------------+ ``` --- ### datediff Calculates the difference between two date values (`expr1 - expr2`) and returns a result in days. `expr1` and `expr2` are valid DATE or DATETIME expressions. Note: Only the date parts of the values are used in the calculation. Difference between datediff and [days\_diff](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/days_diff.md): | Function | Behavior | Example | | ---------- | -------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | datediff | Accurate to the day | The difference between '2020-12-25 23:00:00' and '2020-12-24 23:00:01' is 1. | | days\_diff | Accurate to the second and rounded down to the nearest lower integer | The difference between '2020-12-25 23:00:00' and '2020-12-24 23:00:01' is 0. | The difference between datediff and [date\_diff](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_diff.md) lies in that date\_diff supports the `unit` parameter, which can return the difference between two date values in the specified unit. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT DATEDIFF(DATETIME expr1,DATETIME expr2) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select datediff(CAST('2007-12-31 23:59:59' AS DATETIME), CAST('2007-12-30' AS DATETIME)); +-----------------------------------------------------------------------------------+ | datediff(CAST('2007-12-31 23:59:59' AS DATETIME), CAST('2007-12-30' AS DATETIME)) | +-----------------------------------------------------------------------------------+ | 1 | +-----------------------------------------------------------------------------------+ MySQL > select datediff(CAST('2010-11-30 23:59:59' AS DATETIME), CAST('2010-12-31' AS DATETIME)); +-----------------------------------------------------------------------------------+ | datediff(CAST('2010-11-30 23:59:59' AS DATETIME), CAST('2010-12-31' AS DATETIME)) | +-----------------------------------------------------------------------------------+ | -31 | +-----------------------------------------------------------------------------------+ ``` #### References[​](#references "Direct link to References") [date\_diff](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_diff.md) --- ### day Extracts the day part of a date or datetime expression and returns a value that ranges from 1 to 31. The `date` parameter must be of the DATE or DATETIME type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT DAY(DATETIME|DATE date) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select day('1987-01-31'); +----------------------------+ | day('1987-01-31 00:00:00') | +----------------------------+ | 31 | +----------------------------+ MySQL > select day('1987-01-31 20:10:59'); +----------------------------+ | day('1987-01-31 20:10:59') | +----------------------------+ | 31 | +----------------------------+ ``` #### keywords[​](#keywords "Direct link to keywords") DAY, day --- ### dayofweek_iso Returns the ISO standard day of the week for the specified date as an integer within the range of `1` to `7`. In this standard, `1` represents Monday, and `7` represents Sunday. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT DAY_OF_WEEK_ISO(DATETIME date) ``` #### Parameters[​](#parameters "Direct link to Parameters") `date`: the date you want to convert. It must be of the DATE or DATETIME type. #### Examples[​](#examples "Direct link to Examples") The following example returns the ISO standard day of the week for the date `2023-01-01`: ```sql MySQL > select dayofweek_iso('2023-01-01'); +-----------------------------+ | dayofweek_iso('2023-01-01') | +-----------------------------+ | 7 | +-----------------------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") DAY\_OF\_WEEK\_ISO --- ### dayname Returns the day corresponding to a date. The `date` parameter must be of the DATE or DATETIME type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR DAYNAME(date) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select dayname('2007-02-03 00:00:00'); +--------------------------------+ | dayname('2007-02-03 00:00:00') | +--------------------------------+ | Saturday | +--------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") DAYNAME --- ### dayofmonth Obtains the day part in a date and returns a value that ranges from 1 to 31. The `date` parameter must be of the DATE or DATETIME type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT DAYOFMONTH(DATETIME date) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select dayofmonth('1987-01-31'); +-----------------------------------+ | dayofmonth('1987-01-31 00:00:00') | +-----------------------------------+ | 31 | +-----------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") DAYOFMONTH --- ### dayofweek Returns the weekday index for a given date. For example, the index for Sunday is 1, for Monday is 2, for Saturday is 7. The `date` parameter must be of the DATE or DATETIME type, or a valid expression that can be cast into a DATE or DATETIME value. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT dayofweek(DATETIME date) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select dayofweek('2019-06-25'); +----------------------------------+ | dayofweek('2019-06-25 00:00:00') | +----------------------------------+ | 3 | +----------------------------------+ MySQL > select dayofweek(cast(20190625 as date)); +-----------------------------------+ | dayofweek(CAST(20190625 AS DATE)) | +-----------------------------------+ | 3 | +-----------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") DAYOFWEEK --- ### dayofyear Returns the day of the year for a given date. The `date` parameter must be of the DATE or DATETIME type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT DAYOFYEAR(DATETIME|DATE date) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select dayofyear('2007-02-03 00:00:00'); +----------------------------------+ | dayofyear('2007-02-03 00:00:00') | +----------------------------------+ | 34 | +----------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") DAYOFYEAR --- ### days_add Adds a specified number of days to a given date or date time. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME days_add(DATETIME|DATE date, INT n); ``` #### Parameters[​](#parameters "Direct link to Parameters") `date`: a DATE or DATETIME expression. `n`: the number of days to add. #### Return value[​](#return-value "Direct link to Return value") Returns a DATETIME value. If any input parameter is NULL or invalid, NULL is returned. If the output date exceeds the range of \[0000-01-01 00:00:00, 9999-12-31 00:00:00], NULL is returned. #### Examples[​](#examples "Direct link to Examples") ```plain select days_add('2023-10-31 23:59:59', 1); +------------------------------------+ | days_add('2023-10-31 23:59:59', 1) | +------------------------------------+ | 2023-11-01 23:59:59 | +------------------------------------+ select days_add('2023-10-31 23:59:59', 1000); +---------------------------------------+ | days_add('2023-10-31 23:59:59', 1000) | +---------------------------------------+ | 2026-07-27 23:59:59 | +---------------------------------------+ select days_add('2023-10-31', 1); +---------------------------+ | days_add('2023-10-31', 1) | +---------------------------+ | 2023-11-01 00:00:00 | +---------------------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") DAY,day --- ### days_diff Returns the day difference between two date expressions (`expr1` − `expr2`), accurate to the day. Difference between days\_diff and [datediff](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/datediff.md): | Function | Behavior | Example | | ---------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | | days\_diff | The calculation is accurate to the second and rounded down to the nearest integer. | days\_diff between '2020-12-25 23:00:00' and '2020-12-24 23:00:01' is 0. | | datediff | The calculation is accurate to the day. | datediff between '2020-12-25 23:00:00' and '2020-12-24 23:00:01' is 1. | #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT days_diff(DATETIME expr1,DATETIME expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr1`: the end time. It must be of the DATETIME or DATE type. * `expr2`: the start time. It must be of the DATETIME or DATE type. #### Return value[​](#return-value "Direct link to Return value") Returns a BIGINT value. NULL is returned if the date does not exist, for example, 2022-02-29. days\_diff() takes the hour, minute, and second parts into calculation. If the difference is less than 1 day, 0 is returned. #### Examples[​](#examples "Direct link to Examples") ```plain select days_diff('2010-11-30 23:00:00', '2010-11-29 23:00:00') +---------------------------------------------------------+ | days_diff('2010-11-30 23:00:00', '2010-11-29 23:00:00') | +---------------------------------------------------------+ | 1 | +---------------------------------------------------------+ select days_diff('2010-11-30 23:00:00', '2010-11-29 23:10:00') +---------------------------------------------------------+ | days_diff('2010-11-30 23:00:00', '2010-11-29 23:10:00') | +---------------------------------------------------------+ | 0 | +---------------------------------------------------------+ ``` --- ### days_sub Subtracts specified days from a date or datetime to get a new datetime. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME days_sub(DATETIME|DATE d, INT n); ``` #### Parameters[​](#parameters "Direct link to Parameters") `d`: the date or datetime expression. `n`: the number of days to subtract from the `d`. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the datetime type, NULL is returned if any parameter is NULL or invalid. NULL is returned if the result exceeds the range \[0000-01-01 00:00:00, 9999-12-31 00:00:00]. #### Examples[​](#examples "Direct link to Examples") ```plain SELECT DAYS_SUB('2022-12-20 12:00:00', 10); +--------------------------------------------------+ | days_sub('2022-12-20 12:00:00', INTERVAL 10 DAY) | +--------------------------------------------------+ | 2022-12-10 12:00:00 | +--------------------------------------------------+ SELECT DAYS_SUB('2022-12-20 12:00:00', -10); +---------------------------------------------------+ | days_sub('2022-12-20 12:00:00', INTERVAL -10 DAY) | +---------------------------------------------------+ | 2022-12-30 12:00:00 | +---------------------------------------------------+ SELECT DAYS_SUB('2022-12-20 12:00:00', 738874); +------------------------------------------------------+ | days_sub('2022-12-20 12:00:00', INTERVAL 738874 DAY) | +------------------------------------------------------+ | 0000-01-01 12:00:00 | +------------------------------------------------------+ SELECT DAYS_SUB('2022-12-20 12:00:00', 738875); +------------------------------------------------------+ | days_sub('2022-12-20 12:00:00', INTERVAL 738875 DAY) | +------------------------------------------------------+ | NULL | +------------------------------------------------------+ SELECT DAYS_SUB('2022-12-20 12:00:00', -2913550); +--------------------------------------------------------+ | days_sub('2022-12-20 12:00:00', INTERVAL -2913550 DAY) | +--------------------------------------------------------+ | 9999-12-31 12:00:00 | +--------------------------------------------------------+ SELECT DAYS_SUB('2022-12-20 12:00:00', -2913551); +--------------------------------------------------------+ | days_sub('2022-12-20 12:00:00', INTERVAL -2913551 DAY) | +--------------------------------------------------------+ | NULL | +--------------------------------------------------------+ ``` #### keywords[​](#keywords "Direct link to keywords") DAY,day --- ### from_days Returns a date from 0000-01-01. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATE FROM_DAYS(INT N) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select from_days(730669); +-------------------+ | from_days(730669) | +-------------------+ | 2000-07-03 | +-------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") FROM\_DAYS,FROM,DAYS --- ### from_unixtime Converts a UNIX timestamp into the required time format. The default format is `yyyy-MM-dd HH:mm:ss`. It also supports the formats in [date\_format](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_format.md). Currently, `string_format` supports the following formats: ```plain %Y: Year e.g.: 2014, 1900 %m: Month e.g.: 12, 09 %d: Day e.g.: 11, 01 %H: Hour e.g.: 23, 01, 12 %i: Minute e.g.: 05, 11 %s: Second e.g.: 59, 01 ``` Other formats are invalid and NULL will be returned. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR from_unixtime(BIGINT unix_timestamp[, VARCHAR string_format]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `unix_timestamp`: the UNIX timestamp you want to convert. It must be of the BIGINT type. If the specified timestamp is less than 0 or larger than 253402243199, NULL will be returned. That is, the range for timestamp is `1970-01-01 00:00:00` to `9999-12-30 11:59:59`(varies because of timezone). * `string_format`: the required time format. #### Return value[​](#return-value "Direct link to Return value") Returns a DATETIME or DATE value of the VARCHAR type. If `string_format` specifies the DATE format, a DATE value of the VARCHAR type is returned. If the timestamp exceeds the value range or if `string_format` is invalid, NULL will be returned. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select from_unixtime(1196440219); +---------------------------+ | from_unixtime(1196440219) | +---------------------------+ | 2007-12-01 00:30:19 | +---------------------------+ MySQL > select from_unixtime(1196440219, 'yyyy-MM-dd HH:mm:ss'); +--------------------------------------------------+ | from_unixtime(1196440219, 'yyyy-MM-dd HH:mm:ss') | +--------------------------------------------------+ | 2007-12-01 00:30:19 | +--------------------------------------------------+ MySQL > select from_unixtime(1196440219, '%Y-%m-%d'); +-----------------------------------------+ | from_unixtime(1196440219, '%Y-%m-%d') | +-----------------------------------------+ | 2007-12-01 | +-----------------------------------------+ MySQL > select from_unixtime(1196440219, '%Y-%m-%d %H:%i:%s'); +--------------------------------------------------+ | from_unixtime(1196440219, '%Y-%m-%d %H:%i:%s') | +--------------------------------------------------+ | 2007-12-01 00:30:19 | +--------------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") FROM\_UNIXTIME,FROM,UNIXTIME --- ### hour Returns the hour for a given date. The return value ranges from 0 to 23. The `date` parameter must be of the DATE or DATETIME type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT HOUR(DATETIME|DATE date) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select hour('2018-12-31 23:59:59'); +-----------------------------+ | hour('2018-12-31 23:59:59') | +-----------------------------+ | 23 | +-----------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") HOUR --- ### hours_add Adds hours to a date or datetime. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME hours_add(DATETIME|DATE date, INT hours); ``` #### Parameters[​](#parameters "Direct link to Parameters") `date`: Base date or datetime. `hours`: Hours to be added. #### Return value[​](#return-value "Direct link to Return value") Returns a DATETIME value. NULL is returned when any of the input parameters is NULL. #### Examples[​](#examples "Direct link to Examples") ```plain select hours_add('2022-01-01 01:01:01', 2); +-------------------------------------+ | hours_add('2022-01-01 01:01:01', 2) | +-------------------------------------+ | 2022-01-01 03:01:01 | +-------------------------------------+ select hours_add('2022-01-01 01:01:01', -1); +--------------------------------------+ | hours_add('2022-01-01 01:01:01', -1) | +--------------------------------------+ | 2022-01-01 00:01:01 | +--------------------------------------+ select hours_add('2022-01-01', 1); +----------------------------+ | hours_add('2022-01-01', 1) | +----------------------------+ | 2022-01-01 01:00:00 | +----------------------------+ ``` --- ### hours_diff Returns the hour difference between two date expressions (`expr1` − `expr2`), accurate to the hour. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT hours_diff(DATETIME expr1, DATETIME expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr1`: the end time. It must be of the DATETIME type. * `expr2`: the start time. It must be of the DATETIME type. #### Return value[​](#return-value "Direct link to Return value") Returns a BIGINT value. NULL is returned if the date does not exist, for example, 2022-02-29. #### Examples[​](#examples "Direct link to Examples") ```plain select hours_diff('2010-11-30 23:59:59', '2010-11-30 20:58:59'); +----------------------------------------------------------+ | hours_diff('2010-11-30 23:59:59', '2010-11-30 20:58:59') | +----------------------------------------------------------+ | 3 | +----------------------------------------------------------+ ``` --- ### hours_sub Reduces the specified date and time by a specified number of hours. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME hours_sub(DATETIME|DATE date, INT hours); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `date`: It must be a valid DATE or DATETIME expression. * `hours`: The number of hours reduced. The supported data type is INT. #### Return value[​](#return-value "Direct link to Return value") Returns a DATETIME value. If the date does not exist, for example, `2022-21-01`, or if the date is not a DATE or DATETIME value, NULL is returned. #### Examples[​](#examples "Direct link to Examples") ```plain select hours_sub('2022-01-01 01:01:01', 1); +-------------------------------------+ | hours_sub('2022-01-01 01:01:01', 1) | +-------------------------------------+ | 2022-01-01 00:01:01 | +-------------------------------------+ select hours_sub('2022-01-01 01:01:01', -1); +--------------------------------------+ | hours_sub('2022-01-01 01:01:01', -1) | +--------------------------------------+ | 2022-01-01 02:01:01 | +--------------------------------------+ select hours_sub('2022-01-01', 1); +----------------------------+ | hours_sub('2022-01-01', 1) | +----------------------------+ | 2021-12-31 23:00:00 | +----------------------------+ select hours_sub('2022-01-01', -1); +-----------------------------+ | hours_sub('2022-01-01', -1) | +-----------------------------+ | 2022-01-01 01:00:00 | +-----------------------------+ Error case: select hours_sub('2022-21-01', -1); +--------------------------------------+ | hours_sub('2022-21-01', -1) | +--------------------------------------+ | NULL | +--------------------------------------+ ``` --- ### jodatime_format Converts the specified date into a string in the specified Joda DateTimeFormat pattern format. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR JODATIME_FORMAT(DATETIME | DATE date, VARCHAR format) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `date`: the date you want to convert. It must be a valid date or date expression. * `format`: the format of the date and time to be returned. For information about the available formats, see [Joda-Time format](https://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html). #### Return value[​](#return-value "Direct link to Return value") Currently, the return value is a string that contains a maximum of 128 bytes in length. If the length of the string converted from the specified date exceeds 128 bytes, `NULL` is returned. #### Examples[​](#examples "Direct link to Examples") Example 1: Convert the date `2020-06-25 15:58:51` into a string in `yyyy-MM-dd` format. ```sql MySQL > select jodatime_format('2020-06-25 15:58:51', 'yyyy-MM-dd'); +------------------------------------------------------+ | jodatime_format('2020-06-25 15:58:51', 'yyyy-MM-dd') | +------------------------------------------------------+ | 2020-06-25 | +------------------------------------------------------+ ``` Example 2: Convert the date `2020-06-25 15:58:51` into a string in `yyyy-MM-dd HH:mm:ss` format. ```sql MySQL > select jodatime_format('2020-06-25 15:58:51', 'yyyy-MM-dd HH:mm:ss'); +---------------------------------------------------------------+ | jodatime_format('2020-06-25 15:58:51', 'yyyy-MM-dd HH:mm:ss') | +---------------------------------------------------------------+ | 2020-06-25 15:58:51 | +---------------------------------------------------------------+ ``` Example 3: Convert the date `2020-06-25 15:58:51` into a string in `MM dd ee EE` format. ```sql MySQL > select jodatime_format('2020-06-25 15:58:51', 'MM dd ee EE'); +-------------------------------------------------------+ | jodatime_format('2020-06-25 15:58:51', 'MM dd ee EE') | +-------------------------------------------------------+ | 06 25 04 Thu | +-------------------------------------------------------+ ``` Example 4: Convert the date `2020-06-25 15:58:51` into a string in `MMM dd ee EEE` format. ```sql MySQL > select jodatime_format('2020-06-25 15:58:51', 'MMM dd ee EEE'); +---------------------------------------------------------+ | jodatime_format('2020-06-25 15:58:51', 'MMM dd ee EEE') | +---------------------------------------------------------+ | Jun 25 04 Thu | +---------------------------------------------------------+ ``` Example 5: Convert the date `2020-06-25 15:58:51` into a string in `MMMM dd ee EEEE` format. ```sql MySQL > select jodatime_format('2020-06-25 15:58:51', 'MMMM dd ee EEEE'); +-----------------------------------------------------------+ | jodatime_format('2020-06-25 15:58:51', 'MMMM dd ee EEEE') | +-----------------------------------------------------------+ | June 25 04 Thursday | +-----------------------------------------------------------+ ``` Example 6: Convert the date `2023-06-25 12:00:00` into a string in `KK:mm:ss a` format. ```sql MySQL > select jodatime_format('2023-06-25 12:00:00', 'KK:mm:ss a'); +------------------------------------------------------+ | jodatime_format('2023-06-25 12:00:00', 'KK:mm:ss a') | +------------------------------------------------------+ | 00:00:00 PM | +------------------------------------------------------+ ``` Example 7: Convert the date `2023-06-25 12:00:00` into a string in `hh:mm:ss a` format. ```sql MySQL > select jodatime_format('2023-06-25 12:00:00', 'hh:mm:ss a'); +------------------------------------------------------+ | jodatime_format('2023-06-25 12:00:00', 'hh:mm:ss a') | +------------------------------------------------------+ | 12:00:00 PM | +------------------------------------------------------+ ``` Example 8: Convert the date `2023-06-25 00:00:00` into a string in `yyyyMMdd ''starrocks''` format. ```sql MySQL > select jodatime_format('2023-06-25 00:00:00', 'yyyyMMdd ''starrocks'''); +------------------------------------------------------------------+ | jodatime_format('2023-06-25 00:00:00', 'yyyyMMdd \'starrocks\'') | +------------------------------------------------------------------+ | 20230625 starrocks | +------------------------------------------------------------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") JODATIME\_FORMAT, JODA, FORMAT --- ### last_day Returns the last day of an input DATE or DATETIME expression based on the specified date part. For example, `last_day('2023-05-10', 'month')` returns the last day of the month in which '2023-05-10' falls. If the date part is not specified, this function returns the last day of the month for a given date. This function is supported from v3.1. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DATE last_day(DATETIME|DATE date_expr[, VARCHAR unit]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `date_expr`: a DATE or DATETIME expression, required. * `unit`: the date part, optional. Valid values include `month`, `quarter`, and `year`, default to `month`. If `unit` is invalid, an error is returned. #### Return value[​](#return-value "Direct link to Return value") Returns a DATE value. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select last_day('2023-05-10', 'month'); +----------------------------------+ | last_day('2023-05-10', 'month') | +----------------------------------+ | 2023-05-31 | +----------------------------------+ MySQL > select last_day('2023-05-10'); +------------------------+ | last_day('2023-05-10') | +------------------------+ | 2023-05-31 | +------------------------+ MySQL > select last_day('2023-05-10', 'quarter'); +-----------------------------------+ | last_day('2023-05-10', 'quarter') | +-----------------------------------+ | 2023-06-30 | +-----------------------------------+ MySQL > select last_day('2023-05-10', 'year'); +---------------------------------------+ | last_day('2023-05-10', 'year') | +---------------------------------------+ | 2023-12-31 | +---------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") LAST\_DAY, LAST --- ### makedate Creates and returns a date based on the given year and day of year values. This function is supported from v3.1. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATE makedate(INT year, INT dayOfYear); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `year`: It ranges from 0 to 9999. NULL is returned if this range is exceeded. The supported data type is INT. * `dayOfYear`: the day of year. The supported data type is INT. In order to maintain the same semantics as the function [dayofyear](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/dayofyear.md), if this number is greater than 366 or is 366 in a non-leap year, it is not a day of year. #### Return value[​](#return-value "Direct link to Return value") Returns the date on the dayOfYear-th day of the given year. * `year` must be in the range of \[0,9999]. Otherwise, NULL is returned. * `dayOfYear` must be between 1 and the number of days in the current year (365 in normal years, 366 in leap years). Otherwise, NULL is returned. * The result is also NULL if either input parameter is NULL. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select makedate(2023,0); +-------------------+ | makedate(2023, 0) | +-------------------+ | NULL | +-------------------+ mysql> select makedate(2023,32); +--------------------+ | makedate(2023, 32) | +--------------------+ | 2023-02-01 | +--------------------+ mysql> select makedate(2023,365); +---------------------+ | makedate(2023, 365) | +---------------------+ | 2023-12-31 | +---------------------+ mysql> select makedate(2023,366); +---------------------+ | makedate(2023, 366) | +---------------------+ | NULL | +---------------------+ mysql> select makedate(9999,365); +---------------------+ | makedate(9999, 365) | +---------------------+ | 9999-12-31 | +---------------------+ mysql> select makedate(10000,1); +--------------------+ | makedate(10000, 1) | +--------------------+ | NULL | +--------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") MAKEDATE,MAKE --- ### microseconds_add Adds a time interval to a date value. The time interval is in microseconds. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME microseconds_add(DATETIME expr1,INT expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr1`: the time expression. It must be of the DATETIME type. `expr2`: the time interval you want to add, in microseconds. It must be of the INT type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DATETIME type. If the input value is of the DATE type, the hour, minute, and seconds parts are processed as `00:00:00`. #### Examples[​](#examples "Direct link to Examples") ```plain select microseconds_add('2010-11-30 23:50:50', 2); +--------------------------------------------+ | microseconds_add('2010-11-30 23:50:50', 2) | +--------------------------------------------+ | 2010-11-30 23:50:50.000002 | +--------------------------------------------+ 1 row in set (0.00 sec) select microseconds_add('2010-11-30', 2); +-----------------------------------+ | microseconds_add('2010-11-30', 2) | +-----------------------------------+ | 2010-11-30 00:00:00.000002 | +-----------------------------------+ ``` --- ### microseconds_sub Subtracts a time interval from a date value. The time interval is in microseconds. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME microseconds_sub(DATETIME expr1,INT expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr1`: the time expression. It must be of the DATETIME type. `expr2`: the time interval you want to substract, in microseconds. It must be of the INT type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DATETIME type. If the input value is of the DATE type, the hour, minute, and seconds parts are processed as `00:00:00`. #### Examples[​](#examples "Direct link to Examples") ```plain select microseconds_sub('2010-11-30 23:50:50', 2); +--------------------------------------------+ | microseconds_sub('2010-11-30 23:50:50', 2) | +--------------------------------------------+ | 2010-11-30 23:50:49.999998 | +--------------------------------------------+ select microseconds_sub('2010-11-30', 2); +-----------------------------------+ | microseconds_sub('2010-11-30', 2) | +-----------------------------------+ | 2010-11-29 23:59:59.999998 | +-----------------------------------+ ``` --- ### milliseconds_diff Returns the time difference between the start date and end date in milliseconds. This function is supported from v3.2.4. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT milliseconds_diff(DATETIME expr1, DATETIME expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr1`: the end time. It must be of the DATETIME type. If `expr1` is earlier than `expr2`, a negative value is returned. * `expr2`: the start time. It must be of the DATETIME type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BIGINT type. NULL is returned if the input date does not exist. #### Examples[​](#examples "Direct link to Examples") ```plain select milliseconds_diff('2024-01-25 21:00:00.423','2024-01-25 21:00:00.123'); +-------------------------------------------------------------------------+ | milliseconds_diff('2024-01-25 21:00:00.423', '2024-01-25 21:00:00.123') | +-------------------------------------------------------------------------+ | 300 | +-------------------------------------------------------------------------+ select milliseconds_diff('2024-01-25 21:00:01', '2024-01-25 21:00:00'); +-----------------------------------------------------------------+ | milliseconds_diff('2024-01-25 21:00:01', '2024-01-25 21:00:00') | +-----------------------------------------------------------------+ | 1000 | +-----------------------------------------------------------------+ select milliseconds_diff('2024-01-25 00:00:01', '2024-01-25'); +--------------------------------------------------------+ | milliseconds_diff('2024-01-25 00:00:01', '2024-01-25') | +--------------------------------------------------------+ | 1000 | +--------------------------------------------------------+ ``` --- ### minute Returns the minute for a given date. The return value ranges from 0 to 59. The `date` parameter must be of the DATE or DATETIME type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT MINUTE(DATETIME|DATE date) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select minute('2018-12-31 23:59:59'); +-----------------------------+ |minute('2018-12-31 23:59:59')| +-----------------------------+ | 59 | +-----------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") MINUTE --- ### minutes_add Adds the specified minutes to the date, accurate to the minute. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME minutes_add(DATETIME|DATE date, INT minutes); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `date`: the start time. It must be of the DATETIME or DATE type. * `minutes`: the minutes to add. It must be of the INT type, it could be greater, equal or less than zero. #### Return value[​](#return-value "Direct link to Return value") Returns a DATETIME value. Returns NULL if either `date` or `minutes` is NULL. #### Examples[​](#examples "Direct link to Examples") ```plain select minutes_add('2022-01-01 01:01:01', 2); +---------------------------------------+ | minutes_add('2022-01-01 01:01:01', 2) | +---------------------------------------+ | 2022-01-01 01:03:01 | +---------------------------------------+ select minutes_add('2022-01-01 01:01:01', -1); +----------------------------------------+ | minutes_add('2022-01-01 01:01:01', -1) | +----------------------------------------+ | 2022-01-01 01:00:01 | +----------------------------------------+ select minutes_add('2022-01-01', 1); +------------------------------+ | minutes_add('2022-01-01', 1) | +------------------------------+ | 2022-01-01 00:01:00 | +------------------------------+ ``` --- ### minutes_diff Returns the minute difference between two date expressions (`expr1` − `expr2`), accurate to the minute. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT minutes_diff(DATETIME expr1,DATETIME expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr1`: the end time. It must be of the DATETIME type. * `expr2`: the start time. It must be of the DATETIME type. #### Return value[​](#return-value "Direct link to Return value") Returns a BIGINT value. NULL is returned if the date does not exist, for example, 2022-02-29. #### Examples[​](#examples "Direct link to Examples") ```plain select minutes_diff('2010-11-30 23:59:59', '2010-11-30 20:58:59'); +------------------------------------------------------------+ | minutes_diff('2010-11-30 23:59:59', '2010-11-30 20:58:59') | +------------------------------------------------------------+ | 181 | +------------------------------------------------------------+ ``` --- ### minutes_sub Subtracts specified minutes from the date, accurate to the minute. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME minutes_sub(DATETIME|DATE date, INT minutes); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `date`: the base date. It must be of the DATETIME or DATE type. * `minutes`: the minutes to reduce. It must be of the INT type, it could be greater, equal or less than zero. #### Return value[​](#return-value "Direct link to Return value") Returns a DATETIME value. Returns NULL if either `date` or `minutes` is NULL. #### Examples[​](#examples "Direct link to Examples") ```plain select minutes_sub('2022-01-01 01:03:01', 2); +---------------------------------------+ | minutes_sub('2022-01-01 01:03:01', 2) | +---------------------------------------+ | 2022-01-01 01:01:01 | +---------------------------------------+ select minutes_sub('2022-01-01 01:01:01', -1); +----------------------------------------+ | minutes_sub('2022-01-01 01:01:01', -1) | +----------------------------------------+ | 2022-01-01 01:02:01 | +----------------------------------------+ select minutes_sub('2022-01-01', 1); +------------------------------+ | minutes_sub('2022-01-01', 1) | +------------------------------+ | 2021-12-31 23:59:00 | +------------------------------+ ``` --- ### month Returns the month for a given date. The return value ranges from 1 to 12. The `date` parameter must be of the DATE or DATETIME type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT MONTH(DATETIME date) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select month('1987-01-01'); +-----------------------------+ |month('1987-01-01 00:00:00') | +-----------------------------+ | 1 | +-----------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") MONTH --- ### monthname Returns the name of the month for a given date. The `date` parameter must be of the DATE or DATETIME type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR MONTHNAME(date) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select monthname('2008-02-03 00:00:00'); +----------------------------------+ | monthname('2008-02-03 00:00:00') | +----------------------------------+ | February | +----------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") MONTHNAME, monthname --- ### months_add Adds a specified number of months to the date, accurate to the month. The [add\_months](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/add_months.md) function provides similar functionalities. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME months_add(DATETIME expr1, INT expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr1`: the start time. It must be of the DATETIME or DATE type. * `expr2`: the months to add. It must be of the INT type. It can be greater, equal, or less than zero. A negative value subtracts months from `date`. #### Return value[​](#return-value "Direct link to Return value") Returns a DATETIME value. #### Examples[​](#examples "Direct link to Examples") ```plain select months_add('2019-08-01 13:21:03', 8); +--------------------------------------+ | months_add('2019-08-01 13:21:03', 8) | +--------------------------------------+ | 2020-04-01 13:21:03 | +--------------------------------------+ select months_add('2019-08-01', 8); +-----------------------------+ | months_add('2019-08-01', 8) | +-----------------------------+ | 2020-04-01 00:00:00 | +-----------------------------+ select months_add('2019-08-01 13:21:03', -8); +---------------------------------------+ | months_add('2019-08-01 13:21:03', -8) | +---------------------------------------+ | 2018-12-01 13:21:03 | +---------------------------------------+ select months_add('2019-02-28 13:21:03', 1); +--------------------------------------+ | months_add('2019-02-28 13:21:03', 1) | +--------------------------------------+ | 2019-03-28 13:21:03 | +--------------------------------------+ select months_add('2019-01-30 13:21:03', 1); +--------------------------------------+ | months_add('2019-01-30 13:21:03', 1) | +--------------------------------------+ | 2019-02-28 13:21:03 | +--------------------------------------+ ``` --- ### months_diff Returns the month difference between two date expressions (`expr1` − `expr2`), accurate to the month. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT months_diff(DATETIME expr1,DATETIME expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr1`: the end time. It must be of the DATETIME or DATE type. * `expr2`: the start time. It must be of the DATETIME or DATE type. #### Return value[​](#return-value "Direct link to Return value") Returns a BIGINT value. NULL is returned if the date does not exist, for example, 2022-02-29. #### Examples[​](#examples "Direct link to Examples") ```plain select months_diff('2010-11-30 23:59:59', '2010-1-1 23:59:59'); +---------------------------------------------------------+ | months_diff('2010-11-30 23:59:59', '2010-1-1 23:59:59') | +---------------------------------------------------------+ | 10 | +---------------------------------------------------------+ select months_diff('2010-11-30', '2010-1-1'); +---------------------------------------+ | months_diff('2010-11-30', '2010-1-1') | +---------------------------------------+ | 10 | +---------------------------------------+ ``` --- ### months_sub Subtracts specified months from the date. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME months_sub(DATETIME expr1, INT expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr1`: the specified date. It must be of the DATETIME or DATE type. * `expr2`: the month to subtract. It must be INT type. #### Return value[​](#return-value "Direct link to Return value") Returns a DATETIME value. #### Examples[​](#examples "Direct link to Examples") ```plain select months_sub('2022-02-28 15:04:05', 1); +--------------------------------------+ | months_sub('2022-02-28 15:04:05', 1) | +--------------------------------------+ | 2022-01-28 15:04:05 | +--------------------------------------+ select months_sub('2022-02-28', 1); +-----------------------------+ | months_sub('2022-02-28', 1) | +-----------------------------+ | 2022-01-28 00:00:00 | +-----------------------------+ ``` --- ### next_day Returns the date of the first specified day of week (DOW) that occurs after the input date (DATE or DATETIME). For example, `next_day('2023-04-06', 'Monday')` returns the date of the next Monday that occurred after '2023-04-06'. This function is supported from v3.1. It is the opposite of [previous\_day](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/previous_day.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql DATE next_day(DATETIME|DATE date_expr, VARCHAR dow) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `date_expr`: the input date. It must be a valid DATE or DATETIME expression. * `dow`: the day of week. Valid values include a number of abbreviations which are case-sensitive: | DOW\_FULL | DOW\_2 | DOW\_3 | | --------- | ------ | ------ | | Sunday | Su | Sun | | Monday | Mo | Mon | | Tuesday | Tu | Tue | | Wednesday | We | Wed | | Thursday | Th | Thu | | Friday | Fr | Fri | | Saturday | Sa | Sat | #### Return value[​](#return-value "Direct link to Return value") Returns a DATE value. Any invalid `dow` will cause an error. `dow` is case-sensitive. Returns NULL if an invalid date or a NULL argument is passed in. #### Examples[​](#examples "Direct link to Examples") ```plain -- Return the date of the next Monday that occurred after 2023-04-06. 2023-04-06 is Thursday and the date of the next Monday is 2023-04-10. MySQL > select next_day('2023-04-06', 'Monday'); +----------------------------------+ | next_day('2023-04-06', 'Monday') | +----------------------------------+ | 2023-04-10 | +----------------------------------+ MySQL > select next_day('2023-04-06', 'Tue'); +-------------------------------+ | next_day('2023-04-06', 'Tue') | +-------------------------------+ | 2023-04-11 | +-------------------------------+ MySQL > select next_day('2023-04-06 20:13:14', 'Fr'); +---------------------------------------+ | next_day('2023-04-06 20:13:14', 'Fr') | +---------------------------------------+ | 2023-04-07 | +---------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") NEXT\_DAY, NEXT --- ### now, current_timestamp, localtime, localtimestamp Returns the current date and time. From 3.1.6 onwards, this function can accept a precision input (`p`), which represents the number of digits to retain after seconds. A maximum of 6 digits can be retained (accurate to the microsecond). The digits after `p` are padded with 0. If `p` is not specified, a DATETIME value accurate to the second is returned. The following example returns date and time values when different precision is specified. ```plaintext mysql > select now(),now(1),now(2),now(3),now(4),now(5),now(6)\G *************************** 1. row *************************** now(): 2023-12-08 13:46:45 now(1): 2023-12-08 13:46:45.100000 now(2): 2023-12-08 13:46:45.110000 now(3): 2023-12-08 13:46:45.115000 now(4): 2023-12-08 13:46:45.115800 now(5): 2023-12-08 13:46:45.115840 now(6): 2023-12-08 13:46:45.115843 ``` This function may return different results for different time zones. For more information, see [Configure a time zone](https://docs.starrocks.io/docs/administration/management/timezone.md). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME NOW() DATETIME NOW(INT p) ``` #### Parameters[​](#parameters "Direct link to Parameters") `p`: optional, the specified precision, that is, the number of digits to retain after seconds. It must be an INT value within the range of \[1,6]. `select now(0)` is equivalent to `select now()`. #### Return value[​](#return-value "Direct link to Return value") * If `p` is not specified, this function returns a DATETIME value accurate to the second. * If `p` is specified, this function returns a date and time value of the specified precision. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select now(); +---------------------+ | now() | +---------------------+ | 2019-05-27 15:58:25 | +---------------------+ MySQL > select now(),now(1),now(2),now(3),now(4),now(5),now(6)\G *************************** 1. row *************************** now(): 2023-12-08 13:46:45 now(1): 2023-12-08 13:46:45.100000 now(2): 2023-12-08 13:46:45.110000 now(3): 2023-12-08 13:46:45.115000 now(4): 2023-12-08 13:46:45.115800 now(5): 2023-12-08 13:46:45.115840 now(6): 2023-12-08 13:46:45.115843 ``` #### keyword[​](#keyword "Direct link to keyword") NOW, now --- ### previous_day Returns the date of the first specified day of week (DOW) that occurs before the input date (DATE or DATETIME). For example, `previous_day('2023-04-06', 'Monday')` returns the date of the previous Monday that occurred before '2023-04-06'. This function is supported from v3.1. It is the opposite of [next\_day](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/next_day.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql DATE previous_day(DATETIME|DATE date_expr, VARCHAR dow) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `date_expr`: the input date. It must be a valid DATE or DATETIME expression. * `dow`: the day of week. Valid values include a number of abbreviations which are case-sensitive: | DOW\_FULL | DOW\_2 | DOW\_3 | | --------- | ------ | ------ | | Sunday | Su | Sun | | Monday | Mo | Mon | | Tuesday | Tu | Tue | | Wednesday | We | Wed | | Thursday | Th | Thu | | Friday | Fr | Fri | | Saturday | Sa | Sat | #### Return value[​](#return-value "Direct link to Return value") Returns a DATE value. Any invalid `dow` will cause an error. `dow` is case-sensitive. NULL is returned if an invalid date or a NULL argument is passed in. #### Examples[​](#examples "Direct link to Examples") ```plain -- Return the date of the previous Monday that occurred before 2023-04-06. 2023-04-06 is Thursday and the date of the previous Monday is 2023-04-03. MySQL > select previous_day('2023-04-06', 'Monday'); +--------------------------------------+ | previous_day('2023-04-06', 'Monday') | +--------------------------------------+ | 2023-04-03 | +--------------------------------------+ MySQL > select previous_day('2023-04-06', 'Tue'); +-----------------------------------+ | previous_day('2023-04-06', 'Tue') | +-----------------------------------+ | 2023-04-04 | +-----------------------------------+ MySQL > select previous_day('2023-04-06 20:13:14', 'Fr'); +-------------------------------------------+ | previous_day('2023-04-06 20:13:14', 'Fr') | +-------------------------------------------+ | 2023-03-31 | +-------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") PREVIOUS\_DAY, PREVIOUS, previousday --- ### quarter Returns the quarter part of a date, in the range of 1 to 4. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT quarter(DATETIME|DATE date); ``` #### Parameters[​](#parameters "Direct link to Parameters") `date`: It must be of the DATETIME or DATE type. #### Return value[​](#return-value "Direct link to Return value") Returns an INT value. NULL is returned in any of the following scenarios: * The date is not a valid DATETIME or DATE value. * The input is empty. * The date does not exist, for example, 2022-02-29. #### Examples[​](#examples "Direct link to Examples") Example 1: Return the quarter part of a DATETIME value. ```plain SELECT QUARTER("2022-10-09 15:59:33"); +--------------------------------+ | quarter('2022-10-09 15:59:33') | +--------------------------------+ | 4 | +--------------------------------+ ``` Example 2: Return the quarter part of a DATE value. ```plain SELECT QUARTER("2022-10-09"); +-----------------------+ | quarter('2022-10-09') | +-----------------------+ | 4 | +-----------------------+ ``` Example 3: Return the quarter part that corresponds to the current time or date. ```plain SELECT QUARTER(NOW()); +----------------+ | quarter(now()) | +----------------+ | 4 | +----------------+ SELECT QUARTER(CURDATE()); +--------------------+ | quarter(curdate()) | +--------------------+ | 4 | +--------------------+ ``` --- ### quarters_add Adds a specified number of quarters to the date. One quarter equals three months. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME quarters_add(DATETIME expr1, INT expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr1`: the start time. It must be of the DATETIME or DATE type. * `expr2`: the quarters to add. It must be of the INT type. It can be greater, equal, or less than zero. A negative value subtracts quarters from `expr1`. #### Return value[​](#return-value "Direct link to Return value") Returns a DATETIME value. #### Examples[​](#examples "Direct link to Examples") ```plain select quarters_add('2019-08-01 13:21:03', 1); +----------------------------------------+ | quarters_add('2019-08-01 13:21:03', 1) | +----------------------------------------+ | 2019-11-01 13:21:03 | +----------------------------------------+ select quarters_add('2019-08-01', 1); +-------------------------------+ | quarters_add('2019-08-01', 1) | +-------------------------------+ | 2019-11-01 00:00:00 | +-------------------------------+ select quarters_add('2019-08-01 13:21:03', -1); +-----------------------------------------+ | quarters_add('2019-08-01 13:21:03', -1) | +-----------------------------------------+ | 2019-05-01 13:21:03 | +-----------------------------------------+ ``` --- ### quarters_sub Subtracts a specified number of quarters from the date. One quarter equals three months. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME quarters_sub(DATETIME expr1, INT expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr1`: the specified date. It must be of the DATETIME or DATE type. * `expr2`: the quarters to subtract. It must be of the INT type. It can be greater, equal, or less than zero. A negative value adds quarters to `expr1`. #### Return value[​](#return-value "Direct link to Return value") Returns a DATETIME value. #### Examples[​](#examples "Direct link to Examples") ```plain select quarters_sub('2022-02-28 15:04:05', 1); +----------------------------------------+ | quarters_sub('2022-02-28 15:04:05', 1) | +----------------------------------------+ | 2021-11-28 15:04:05 | +----------------------------------------+ select quarters_sub('2022-02-28', 1); +-------------------------------+ | quarters_sub('2022-02-28', 1) | +-------------------------------+ | 2021-11-28 00:00:00 | +-------------------------------+ ``` --- ### sec_to_time #### Description[​](#description "Direct link to Description") The SEC\_TO\_TIME function converts a value in seconds into a TIME type, returning the result in the format HH:MM:SS. The input seconds represent the time elapsed since the start of a day (00:00:00). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext TIME sec_to_time(BIGINT sec) ``` #### Parameters[​](#parameters "Direct link to Parameters") `sec`: It must be of the INT type. #### Return value[​](#return-value "Direct link to Return value") Returns a TIME value in the format HH:MM:SS, representing the time calculated from the start of a day (00:00:00). If sec is NULL, the function returns NULL. #### Examples[​](#examples "Direct link to Examples") ```plain select sec_to_time(43994); +-----------------------------+ | sec_to_time(43994) | +-----------------------------+ | '12:13:14'| +-----------------------------+ ``` --- ### second Returns the second part for a given date. The return value ranges from 0 to 59. The `date` parameter must be of the DATE or DATETIME type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT SECOND(DATETIME date) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select second('2018-12-31 23:59:59'); +-----------------------------+ |second('2018-12-31 23:59:59')| +-----------------------------+ | 59 | +-----------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") SECOND --- ### seconds_add #### description[​](#description "Direct link to description") ##### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME SECONDS_ADD(DATETIME expr1,INT expr2) ``` Add a specified time interval to the date. Unit is second. The expr1 parameter is a valid datetime expression. The expr2 parameter is the seconds you want to add. #### example[​](#example "Direct link to example") ```plain select seconds_add('2010-11-30 23:50:50', 2); +---------------------------------------+ | seconds_add('2010-11-30 23:50:50', 2) | +---------------------------------------+ | 2010-11-30 23:50:52 | +---------------------------------------+ select seconds_add('2010-11-30', 2); +------------------------------+ | seconds_add('2010-11-30', 2) | +------------------------------+ | 2010-11-30 00:00:02 | +------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") SECONDS\_ADD,ADD --- ### seconds_diff Returns the second difference between two date expressions (`expr1` − `expr2`), accurate to the second. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT seconds_diff(DATETIME expr1,DATETIME expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr1`: the end time. It must be of the DATETIME type. * `expr2`: the start time. It must be of the DATETIME type. #### Return value[​](#return-value "Direct link to Return value") Returns a BIGINT value. NULL is returned if the date does not exist, for example, 2022-02-29. #### Examples[​](#examples "Direct link to Examples") ```plain select seconds_diff('2010-11-30 23:59:59', '2010-11-30 20:59:59'); +------------------------------------------------------------+ | seconds_diff('2010-11-30 23:59:59', '2010-11-30 20:59:59') | +------------------------------------------------------------+ | 10800 | +------------------------------------------------------------+ ``` --- ### seconds_sub Subtracts a time interval from a date value. The time interval is in seconds. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME seconds_sub(DATETIME|DATE date, INT seconds); ``` #### Parameters[​](#parameters "Direct link to Parameters") `date`: the time expression. It must be of the DATETIME type. `seconds`: the time interval you want to substract, in seconds. It must be of the INT type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DATETIME type. If the input value is of the DATE type, the hour, minute, and seconds parts are processed as `00:00:00`. #### Examples[​](#examples "Direct link to Examples") ```plain select seconds_sub('2022-01-01 01:01:01', 2); +---------------------------------------+ | seconds_sub('2022-01-01 01:01:01', 2) | +---------------------------------------+ | 2022-01-01 01:00:59 | +---------------------------------------+ select seconds_sub('2022-01-01 01:01:01', -1); +----------------------------------------+ | seconds_sub('2022-01-01 01:01:01', -1) | +----------------------------------------+ | 2022-01-01 01:01:02 | +----------------------------------------+ select seconds_sub('2022-01-01', 1); +------------------------------+ | seconds_sub('2022-01-01', 1) | +------------------------------+ | 2021-12-31 23:59:59 | +------------------------------+v ``` --- ### str_to_date Converts a string into a DATETIME value according to the specified format. If the conversion fails, NULL is returned. The format must be consistent with that described in [date\_format](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_format.md). This function is inverse to [date\_format](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_format.md). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME STR_TO_DATE(VARCHAR str, VARCHAR format) ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: the time expression you want to convert. It must be of the VARCHAR type. `format`: the format used to return the value. For the supported format, see [date\_format](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_format.md). #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DATETIME type. If `format` specifies a date, a DATE value is returned. NULL is returned if `str` or `format` is NULL. #### Examples[​](#examples "Direct link to Examples") Example 1: Convert the input into a DATETIME value. ```plain MySQL > select str_to_date('2014-12-21 12:34:56', '%Y-%m-%d %H:%i:%s'); +---------------------------------------------------------+ | str_to_date('2014-12-21 12:34:56', '%Y-%m-%d %H:%i:%s') | +---------------------------------------------------------+ | 2014-12-21 12:34:56 | +---------------------------------------------------------+ ``` Example 2: Convert the input into a DATE value. ```plain MySQL > select str_to_date('2014-12-21 12:34:56', '%Y-%m-%d'); +--------------------------------------------------------------+ | str_to_date('2014-12-21 12:34:56', '%Y-%m-%d') | +--------------------------------------------------------------+ | 2014-12-21 | +--------------------------------------------------------------+ ``` Example 3: Convert the input "200442 Monday" into a DATE value. ```plain MySQL > select str_to_date('200442 Monday', '%X%V %W'); +-----------------------------------------+ | str_to_date('200442 Monday', '%X%V %W') | +-----------------------------------------+ | 2004-10-18 | +-----------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") STR\_TO\_DATE,STR,TO,DATE --- ### str_to_jodatime Converts a Joda-formatted string into a DATETIME value in the specified Joda DateTime format like `yyyy-MM-dd HH:mm:ss`. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME str_to_jodatime(VARCHAR str, VARCHAR format) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str`: the time expression you want to convert. It must be of the VARCHAR type. * `format`: the Joda DateTime format of the DATETIME value to be returned. For information about the formats available, see [Joda DateTime](https://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html). #### Return value[​](#return-value "Direct link to Return value") * If parsing the input string succeeds, a DATETIME value is returned. * If parsing the input string fails, `NULL` is returned. #### Examples[​](#examples "Direct link to Examples") Example 1: Convert the string `2014-12-21 12:34:56` into a DATETIME value in `yyyy-MM-dd HH:mm:ss` format. ```sql MySQL > select str_to_jodatime('2014-12-21 12:34:56', 'yyyy-MM-dd HH:mm:ss'); +--------------------------------------------------------------+ | str_to_jodatime('2014-12-21 12:34:56', 'yyyy-MM-dd HH:mm:ss') | +--------------------------------------------------------------+ | 2014-12-21 12:34:56 | +--------------------------------------------------------------+ ``` Example 2: Convert the string `21/December/23 12:34:56` with a text-style month into a DATETIME value in `dd/MMMM/yy HH:mm:ss` format. ```sql MySQL > select str_to_jodatime('21/December/23 12:34:56', 'dd/MMMM/yy HH:mm:ss'); +------------------------------------------------------------------+ | str_to_jodatime('21/December/23 12:34:56', 'dd/MMMM/yy HH:mm:ss') | +------------------------------------------------------------------+ | 2023-12-21 12:34:56 | +------------------------------------------------------------------+ ``` Example 3: Convert the string `21/December/23 12:34:56.123` accurate to milliseconds into a DATETIME value in `dd/MMMM/yy HH:mm:ss.SSS` format. ```sql MySQL > select str_to_jodatime('21/December/23 12:34:56.123', 'dd/MMMM/yy HH:mm:ss.SSS'); +--------------------------------------------------------------------------+ | str_to_jodatime('21/December/23 12:34:56.123', 'dd/MMMM/yy HH:mm:ss.SSS') | +--------------------------------------------------------------------------+ | 2023-12-21 12:34:56.123000 | +--------------------------------------------------------------------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") STR\_TO\_JODATIME, DATETIME --- ### str2date Converts a string into a DATE value according to the specified format. If the conversion fails, NULL is returned. The format must be consistent with that described in [date\_format](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_format.md). This function is equivalent to [str\_to\_date](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/str_to_date.md) but has a different return type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATE str2date(VARCHAR str, VARCHAR format); ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: the time expression you want to convert. It must be of the VARCHAR type. `format`: the format used to return the value. For the supported format, see [date\_format](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_format.md). #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DATE type. NULL is returned if `str` or `format` is NULL. #### Examples[​](#examples "Direct link to Examples") ```plain select str2date('2010-11-30 23:59:59', '%Y-%m-%d %H:%i:%s'); +------------------------------------------------------+ | str2date('2010-11-30 23:59:59', '%Y-%m-%d %H:%i:%s') | +------------------------------------------------------+ | 2010-11-30 | +------------------------------------------------------+ 1 row in set (0.01 sec) ``` --- ### time_format Formats TIME-type values in the specified format. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR TIME_FORMAT(TIME time, VARCHAR format) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `time` (Required): The time value (of the TIME type) to be formatted. * `format` (Required): The format to use. Valid values: ```plain %f Microseconds (000000 to 999999) %H Hour (00 to 23) %h Hour (00 to 12) %i Minutes (00 to 59) %p AM or PM %S Seconds (00 to 59) %s Seconds (00 to 59) ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> SELECT TIME_FORMAT("19:30:10", "%h %i %s %p"); +----------------------------------------+ | time_format('19:30:10', '%h %i %s %p') | +----------------------------------------+ | 12 00 00 AM | +----------------------------------------+ 1 row in set (0.01 sec) ``` --- ### time_slice Converts a given time into the beginning or end of a time interval based on the specified time granularity. This function is supported from v2.3. v2.5 supports converting a given time into the end of a time interval. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME time_slice(DATETIME dt, INTERVAL N type[, boundary]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `dt`: the time to convert, DATETIME. * `INTERVAL N type`: the time granularity, for example, `interval 5 second`. * `N` is the length of time interval. It must be an INT value. * `type` is the unit, which can be YEAR, QUARTER, MONTH, WEEK, DAY, HOUR, MINUTE, SECOND, MILLISECOND (since 3.1.7), and MICROSECOND (since 3.1.7). * `boundary`: optional. It is used to specify whether to return the beginning (`FLOOR`) or end (`CEIL`) of the time interval. Valid values: FLOOR, CEIL. If this parameter is not specified, `FLOOR` is the default. This parameter is supported from v2.5. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DATETIME type. #### Usage notes[​](#usage-notes "Direct link to Usage notes") The time interval starts from A.D. `0001-01-01 00:00:00`. #### Examples[​](#examples "Direct link to Examples") The following examples are provided based on the `test_all_type_select` table. ```plaintext select * from test_all_type_select order by id_int; +------------+---------------------+--------+ | id_date | id_datetime | id_int | +------------+---------------------+--------+ | 2052-12-26 | 1691-12-23 04:01:09 | 0 | | 2168-08-05 | 2169-12-18 15:44:31 | 1 | | 1737-02-06 | 1840-11-23 13:09:50 | 2 | | 2245-10-01 | 1751-03-21 00:19:04 | 3 | | 1889-10-27 | 1861-09-12 13:28:18 | 4 | +------------+---------------------+--------+ 5 rows in set (0.06 sec) ``` Example 1: Convert a given DATETIME value to the beginning of a 5-second time interval without specifying the `boundary` parameter. ```plaintext select time_slice(id_datetime, interval 5 second) from test_all_type_select order by id_int; +---------------------------------------------------+ | time_slice(id_datetime, INTERVAL 5 second, floor) | +---------------------------------------------------+ | 1691-12-23 04:01:05 | | 2169-12-18 15:44:30 | | 1840-11-23 13:09:50 | | 1751-03-21 00:19:00 | | 1861-09-12 13:28:15 | +---------------------------------------------------+ 5 rows in set (0.16 sec) ``` Example 2: Convert a given DATETIME value to the beginning of a 5-day time interval with `boundary` set to `FLOOR`. ```plaintext select time_slice(id_datetime, interval 5 day, FLOOR) from test_all_type_select order by id_int; +------------------------------------------------+ | time_slice(id_datetime, INTERVAL 5 day, floor) | +------------------------------------------------+ | 1691-12-22 00:00:00 | | 2169-12-16 00:00:00 | | 1840-11-21 00:00:00 | | 1751-03-18 00:00:00 | | 1861-09-12 00:00:00 | +------------------------------------------------+ 5 rows in set (0.15 sec) ``` Example 3: Convert a given DATETIME value to the end of a 5-day time interval. ```plaintext select time_slice(id_datetime, interval 5 day, CEIL) from test_all_type_select order by id_int; +-----------------------------------------------+ | time_slice(id_datetime, INTERVAL 5 day, ceil) | +-----------------------------------------------+ | 1691-12-27 00:00:00 | | 2169-12-21 00:00:00 | | 1840-11-26 00:00:00 | | 1751-03-23 00:00:00 | | 1861-09-17 00:00:00 | +-----------------------------------------------+ ``` Example 4: Convert a given DATETIME value to the end of a 1-millisecond time interval. ```plaintext select id_datetime, time_slice(id_datetime, interval 1 millisecond, CEIL) from test_all_type_select order by id_int; +---------------------+-------------------------------------------------------+ | id_datetime | time_slice(id_datetime, INTERVAL 1 millisecond, ceil) | +---------------------+-------------------------------------------------------+ | 1691-12-23 04:01:09 | 1691-12-23 04:01:09.001000 | | 2169-12-18 15:44:31 | 2169-12-18 15:44:31.001000 | | 1840-11-23 13:09:50 | 1840-11-23 13:09:50.001000 | | 1751-03-21 00:19:04 | 1751-03-21 00:19:04.001000 | | 1861-09-12 13:28:18 | 1861-09-12 13:28:18.001000 | +---------------------+-------------------------------------------------------+ ``` Example 5: Convert a given DATETIME value to the end of a 1-microsecond time interval. ```plaintext select id_datetime, time_slice(id_datetime, interval 1 microsecond, CEIL) from test_all_type_select order by id_int; +---------------------+-------------------------------------------------------+ | id_datetime | time_slice(id_datetime, INTERVAL 1 microsecond, ceil) | +---------------------+-------------------------------------------------------+ | 1691-12-23 04:01:09 | 1691-12-23 04:01:09.000001 | | 2169-12-18 15:44:31 | 2169-12-18 15:44:31.000001 | | 1840-11-23 13:09:50 | 1840-11-23 13:09:50.000001 | | 1751-03-21 00:19:04 | 1751-03-21 00:19:04.000001 | | 1861-09-12 13:28:18 | 1861-09-12 13:28:18.000001 | +---------------------+-------------------------------------------------------+ ``` --- ### time_to_sec Converts a time value into the number of seconds. The formula used for the conversion is as follows: Hour x 3600 + Minute x 60 + Second #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT time_to_sec(TIME time) ``` #### Parameters[​](#parameters "Direct link to Parameters") `time`: It must be of the TIME type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BIGINT type. If the input is invalid, NULL is returned. #### Examples[​](#examples "Direct link to Examples") ```plain select time_to_sec('12:13:14'); +-----------------------------+ | time_to_sec('12:13:14') | +-----------------------------+ | 43994| +-----------------------------+ ``` --- ### timediff Returns the difference between two DATETIME expressions. The return value must be of the TIME type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext TIME TIMEDIFF(DATETIME expr1, DATETIME expr2)` ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT TIMEDIFF(now(),utc_timestamp()); +----------------------------------+ | timediff(now(), utc_timestamp()) | +----------------------------------+ | 08:00:00 | +----------------------------------+ MySQL > SELECT TIMEDIFF('2019-07-11 16:59:30','2019-07-11 16:59:21'); +--------------------------------------------------------+ | timediff('2019-07-11 16:59:30', '2019-07-11 16:59:21') | +--------------------------------------------------------+ | 00:00:09 | +--------------------------------------------------------+ MySQL > SELECT TIMEDIFF('2019-01-01 00:00:00', NULL); +---------------------------------------+ | timediff('2019-01-01 00:00:00', NULL) | +---------------------------------------+ | NULL | +---------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") TIMEDIFF --- ### timestamp Returns the DATETIME value of a date or datetime expression. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME timestamp(DATETIME|DATE expr); ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: the time expression you want to convert. It must be of the DATETIME or DATE type. #### Return value[​](#return-value "Direct link to Return value") Returns a DATETIME value. If the input time is empty or does not exist, such as `2021-02-29`, NULL is returned. #### Examples[​](#examples "Direct link to Examples") ```plain select timestamp("2019-05-27"); +-------------------------+ | timestamp('2019-05-27') | +-------------------------+ | 2019-05-27 00:00:00 | +-------------------------+ 1 row in set (0.00 sec) ``` --- ### timestampadd Adds an integer expression interval to the date or datetime expression `datetime_expr`. The unit for the interval as mentioned must be one of the following: MILLISECOND (since 3.2), SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, or YEAR. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME TIMESTAMPADD(unit, interval, DATETIME datetime_expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `datetime_expr`: the DATE or DATETIME value to which you want to add a time interval. * `interval`: an integer expression that specifies the number of intervals to add. * `unit`: the unit of the time interval to add. Supported units include MILLISECOND (since 3.2), SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, and YEAR. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the same data type as `datetime_expr`. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT TIMESTAMPADD(MINUTE,1,'2019-01-02'); +------------------------------------------------+ | timestampadd(MINUTE, 1, '2019-01-02 00:00:00') | +------------------------------------------------+ | 2019-01-02 00:01:00 | +------------------------------------------------+ MySQL > SELECT TIMESTAMPADD(WEEK,1,'2019-01-02'); +----------------------------------------------+ | timestampadd(WEEK, 1, '2019-01-02 00:00:00') | +----------------------------------------------+ | 2019-01-09 00:00:00 | +----------------------------------------------+ MySQL > SELECT TIMESTAMPADD(MILLISECOND,1,'2019-01-02'); +--------------------------------------------+ | timestampadd(MILLISECOND, 1, '2019-01-02') | +--------------------------------------------+ | 2019-01-02 00:00:00.001000 | +--------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") TIMESTAMPADD --- ### timestampdiff Returns the interval from `datetime_expr2` to `datetime_expr1`. `datetime_expr1` and `datetime_expr2` must be of the DATE or DATETIME type. The unit for the integer result and the interval should be one of the following: MILLISECOND (since 3.2), SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, or YEAR. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT TIMESTAMPDIFF(unit,DATETIME datetime_expr1, DATETIME datetime_expr2) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `datetime_expr`: the two DATE or DATETIME expressions you want to compare. * `unit`: the unit of the time difference. Supported units include MILLISECOND (since 3.2), SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, and YEAR. #### Return value[​](#return-value "Direct link to Return value") Returns a BIGINT value. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL> SELECT TIMESTAMPDIFF(MONTH,'2003-02-01','2003-05-01'); +--------------------------------------------------------------------+ | timestampdiff(MONTH, '2003-02-01 00:00:00', '2003-05-01 00:00:00') | +--------------------------------------------------------------------+ | 3 | +--------------------------------------------------------------------+ MySQL> SELECT TIMESTAMPDIFF(YEAR,'2002-05-01','2001-01-01'); +-------------------------------------------------------------------+ | timestampdiff(YEAR, '2002-05-01 00:00:00', '2001-01-01 00:00:00') | +-------------------------------------------------------------------+ | -1 | +-------------------------------------------------------------------+ MySQL> SELECT TIMESTAMPDIFF(MINUTE,'2003-02-01','2003-05-01 12:05:55'); +---------------------------------------------------------------------+ | timestampdiff(MINUTE, '2003-02-01 00:00:00', '2003-05-01 12:05:55') | +---------------------------------------------------------------------+ | 128885 | +---------------------------------------------------------------------+ MySQL > SELECT TIMESTAMPDIFF(MILLISECOND,'2003-02-01','2003-05-01'); +--------------------------------------------------------+ | timestampdiff(MILLISECOND, '2003-02-01', '2003-05-01') | +--------------------------------------------------------+ | 7689600000 | +--------------------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") TIMESTAMPDIFF --- ### to_date Converts a DATETIME value into a date. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATE TO_DATE(DATETIME datetime) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select to_date("2020-02-02 00:00:00"); +--------------------------------+ | to_date('2020-02-02 00:00:00') | +--------------------------------+ | 2020-02-02 | +--------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") TO\_DATE --- ### to_datetime Converts a Unix timestamp to a DATETIME type value **based on the current time zone settings**. For detailed instructions on setting a time zone, see [Configure a time zone](https://docs.starrocks.io/docs/administration/management/timezone.md). If you want to convert a Unix timestamp to a DATETIME type value independent of the current session's time zone settings, you can use [to\_datetime\_ntz](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/to_datetime_ntz.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql DATETIME to_datetime(BIGINT unix_ts, INT scale) ``` #### Parameters[​](#parameters "Direct link to Parameters") | Name | Type | Required | Description | | --------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `unix_ts` | BIGINT | Yes | The Unix timestamp to convert. For example, `1598306400` (seconds) and `1598306400123` (milliseconds). | | `scale` | INT | No | Time precision. Valid values:- `0` indicates seconds (Default).
- `3` indicates milliseconds.
- `6` indicates microseconds. | #### Return Value[​](#return-value "Direct link to Return Value") * On success: returns a `DATETIME` value based on the current session's time zone. * On failure: returns `NULL`. Common reasons include: * Invalid `scale` (not 0, 3, or 6) * Value out of DATETIME range (0001-01-01 to 9999-12-31) #### Example[​](#example "Direct link to Example") ```sql SET time_zone = 'Asia/Shanghai'; SELECT to_datetime(1598306400); -- Returns: 2020-08-25 06:00:00 SELECT to_datetime(1598306400123, 3); -- Returns: 2020-08-25 06:00:00.123000 ``` --- ### to_datetime_ntz Converts a Unix timestamp to a DATETIME type value. **This function is always based on UTC+0, regardless of the time zone settings.** If you want to convert a Unix timestamp to a DATETIME type value based on the current time zone, you can use [to\_datetime](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/to_datetime.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql DATETIME to_datetime_ntz(BIGINT unix_ts, INT scale) ``` #### Parameters[​](#parameters "Direct link to Parameters") | Name | Type | Required | Description | | --------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `unix_ts` | BIGINT | Yes | The Unix timestamp to convert. For example, `1598306400` (seconds) and `1598306400123` (milliseconds). | | `scale` | INT | No | Time precision. Valid values:- `0` indicates seconds (Default).
- `3` indicates milliseconds.
- `6` indicates microseconds. | #### Return Value[​](#return-value "Direct link to Return Value") * On success: returns a `DATETIME` value based on UTC+0. * On failure: returns `NULL`. Common reasons include: * Invalid `scale` (not 0, 3, or 6) * Value out of DATETIME range (0001-01-01 to 9999-12-31) #### Example[​](#example "Direct link to Example") ```sql SELECT to_datetime_ntz(1598306400); -- Returns: 2020-08-24 22:00:00 SELECT to_datetime_ntz(1598306400123456, 6); -- Returns: 2020-08-24 22:00:00.123456 ``` --- ### to_days Returns the number of days between a date and 0000-01-01. The `date` parameter must be of the DATE or DATETIME type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT TO_DAYS(DATETIME date) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select to_days('2007-10-07'); +-----------------------+ | to_days('2007-10-07') | +-----------------------+ | 733321 | +-----------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") TO\_DAYS,TO,DAYS --- ### to_iso8601 Converts the specified date into a string in ISO 8601 format. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR TO_ISO8601(DATETIME | DATE date) ``` #### Parameters[​](#parameters "Direct link to Parameters") `date`: the date you want to convert. It must be a valid date or date expression. #### Examples[​](#examples "Direct link to Examples") Example 1: Convert the date `2020-01-01` into a string in ISO 8601 format. ```sql MySQL > select to_iso8601(date'2020-01-01'); +--------------------------+ | to_iso8601('2020-01-01') | +--------------------------+ | 2020-01-01 | +--------------------------+ ``` Example 2: Convert the date `2020-01-01 00:00:00.01` into a string in ISO 8601 format. ```sql MySQL > select to_iso8601(datetime'2020-01-01 00:00:00.01'); +------------------------------------------+ | to_iso8601('2020-01-01 00:00:00.010000') | +------------------------------------------+ | 2020-01-01T00:00:00.010000 | +------------------------------------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") TO\_ISO8601 --- ### to_tera_date Parses a date or time string according to the specified format and converts the string to a DATE value. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATE to_tera_date(VARCHAR str, VARCHAR format) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str`: the time expression to convert. It must be of the VARCHAR type. * `format`: the date format specifier for `str`. It is used to parse and convert the input string. `format` must match `string`. Otherwise, NULL is returned. If `format` is invalid, an error is returned. The following table describes the format elements. | **Element** | **Description** | | ---------------------- | ----------------------------------------------------- | | \[ \r \n \t - / , . ;] | Punctuation characters that are ignored in conversion | | dd | Day of month (1 - 31) | | hh | Hour of day (1 - 12) | | hh24 | Hour of day (0 - 23) | | mi | Minute (0 - 59) | | mm | Month (01 - 12) | | ss | Second (0 - 59) | | yyyy | 4-digit year. | | yy | 2-digit year. | | am | Meridian indicator. | | pm | Meridian indicator. | #### Examples[​](#examples "Direct link to Examples") ```sql select to_tera_date("1988/04/08","yyyy/mm/dd"); +------------------------------------------+ | to_tera_date('1988/04/08', 'yyyy/mm/dd') | +------------------------------------------+ | 1988-04-08 | +------------------------------------------+ select to_tera_date("04-08-1988","mm-dd-yyyy"); +------------------------------------------+ | to_tera_date('04-08-1988', 'mm-dd-yyyy') | +------------------------------------------+ | 1988-04-08 | +------------------------------------------+ select to_tera_date(";198804:08",";yyyymm:dd"); +------------------------------------------+ | to_tera_date(';198804:08', ';yyyymm:dd') | +------------------------------------------+ | 1988-04-08 | +------------------------------------------+ select to_tera_date("2020-02-02 00:00:00", "yyyy-mm-dd"); +---------------------------------------------------+ | to_tera_date('2020-02-02 00:00:00', 'yyyy-mm-dd') | +---------------------------------------------------+ | 2020-02-02 | +---------------------------------------------------+ -- The input is year and does not contain the month or date part. The first day in that year is returned. select to_tera_date("1988","yyyy"); +------------------------------+ | to_tera_date('1988', 'yyyy') | +------------------------------+ | 1988-01-01 | +------------------------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") TO\_TERA\_DATE --- ### to_tera_timestamp Parses a date or time string according to the specified format and converts the string to a DATETIME value. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME to_tera_timestamp(VARCHAR str, VARCHAR format) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str`: the time expression to convert. It must be of the VARCHAR type. * `format`: the time format specifier for `str`. It is used to parse and convert the input string. `format` must match `string`. Otherwise, NULL is returned. If `format` is invalid, an error is returned. The following table describes the format elements. | **Element** | **Description** | | ---------------------- | ----------------------------------------------------- | | \[ \r \n \t - / , . ;] | Punctuation characters that are ignored in conversion | | dd | Day of month (1 - 31) | | hh | Hour of day (1 - 12) | | hh24 | Hour of day (0 - 23) | | mi | Minute (0 - 59) | | mm | Month (01 - 12) | | ss | Second (0 - 59) | | yyyy | 4-digit year. | | yy | 2-digit year. | | am | Meridian indicator. | | pm | Meridian indicator. | #### Examples[​](#examples "Direct link to Examples") ```sql select to_tera_timestamp("1988/04/08","yyyy/mm/dd"); +-----------------------------------------------+ | to_tera_timestamp('1988/04/08', 'yyyy/mm/dd') | +-----------------------------------------------+ | 1988-04-08 00:00:00 | +-----------------------------------------------+ select to_tera_timestamp("04-08-1988","mm-dd-yyyy"); +-----------------------------------------------+ | to_tera_timestamp('04-08-1988', 'mm-dd-yyyy') | +-----------------------------------------------+ | 1988-04-08 00:00:00 | +-----------------------------------------------+ select to_tera_timestamp("04.1988,08","mm.yyyy,dd"); +-----------------------------------------------+ | to_tera_timestamp('04.1988,08', 'mm.yyyy,dd') | +-----------------------------------------------+ | 1988-04-08 00:00:00 | +-----------------------------------------------+ select to_tera_timestamp("1988/04/08 2","yyyy/mm/dd hh"); +----------------------------------------------------+ | to_tera_timestamp('1988/04/08 2', 'yyyy/mm/dd hh') | +----------------------------------------------------+ | 1988-04-08 02:00:00 | +----------------------------------------------------+ select to_tera_timestamp("1988/04/08 14","yyyy/mm/dd hh24"); +-------------------------------------------------------+ | to_tera_timestamp('1988/04/08 14', 'yyyy/mm/dd hh24') | +-------------------------------------------------------+ | 1988-04-08 14:00:00 | +-------------------------------------------------------+ select to_tera_timestamp("1988/04/08 14:15","yyyy/mm/dd hh24:mi"); +-------------------------------------------------------------+ | to_tera_timestamp('1988/04/08 14:15', 'yyyy/mm/dd hh24:mi') | +-------------------------------------------------------------+ | 1988-04-08 14:15:00 | +-------------------------------------------------------------+ select to_tera_timestamp("1988/04/08 2:3:4","yyyy/mm/dd hh24:mi:ss"); +----------------------------------------------------------------+ | to_tera_timestamp('1988/04/08 2:3:4', 'yyyy/mm/dd hh24:mi:ss') | +----------------------------------------------------------------+ | 1988-04-08 02:03:04 | +----------------------------------------------------------------+ select to_tera_timestamp("1988/04/08 02 am:3:4","yyyy/mm/dd hh am:mi:ss"); +---------------------------------------------------------------------+ | to_tera_timestamp('1988/04/08 02 am:3:4', 'yyyy/mm/dd hh am:mi:ss') | +---------------------------------------------------------------------+ | 1988-04-08 02:03:04 | +---------------------------------------------------------------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") TO\_TERA\_TIMESTAMP --- ### unix_timestamp Converts a DATE or DATETIME value into a UNIX timestamp. If no parameter is specified, this function converts the current time into a UNIX timestamp. The `date` parameter must be of the DATE or DATETIME type. For time before 1970-01-01 00:00:00 or after 2038-01-19 11:14:07, this function returns 0. For more information about the date format, see [date\_format](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_format.md). This function may return different results for different time zones. For more information, see [Configure a time zone](https://docs.starrocks.io/docs/administration/management/timezone.md). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT UNIX_TIMESTAMP() BIGINT UNIX_TIMESTAMP(DATETIME date) BIGINT UNIX_TIMESTAMP(DATETIME date, STRING fmt) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select unix_timestamp(); +------------------+ | unix_timestamp() | +------------------+ | 1558589570 | +------------------+ MySQL > select unix_timestamp('2007-11-30 10:30:19'); +---------------------------------------+ | unix_timestamp('2007-11-30 10:30:19') | +---------------------------------------+ | 1196389819 | +---------------------------------------+ MySQL > select unix_timestamp('2007-11-30 10:30-19', '%Y-%m-%d %H:%i-%s'); +---------------------------------------+ | unix_timestamp('2007-11-30 10:30-19') | +---------------------------------------+ | 1196389819 | +---------------------------------------+ MySQL > select unix_timestamp('2007-11-30 10:30%3A19', '%Y-%m-%d %H:%i%%3A%s'); +---------------------------------------+ |unix_timestamp('2007-11-30 10:30%3A19')| +---------------------------------------+ | 1196389819 | +---------------------------------------+ MySQL > select unix_timestamp('1969-01-01 00:00:00'); +---------------------------------------+ | unix_timestamp('1969-01-01 00:00:00') | +---------------------------------------+ | 0 | +---------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") UNIX\_TIMESTAMP,UNIX,TIMESTAMP --- ### utc_time Returns the current UTC time as a value in 'HH:MM:SS'. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext TIME UTC_TIME() ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select utc_time(); +----------------+ | utc_time() | +----------------+ | 12:31:18 | +----------------+ ``` #### keyword[​](#keyword "Direct link to keyword") UTC\_TIME,UTC,TIME --- ### utc_timestamp Returns the current UTC date and time as a value in 'YYYY-MM-DD HH:MM:SS' or 'YYYYMMDDHHMMSS' format depending on the usage of the function, for example, in a string or numeric context. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME UTC_TIMESTAMP() ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select utc_timestamp(),utc_timestamp() + 1; +---------------------+---------------------+ | utc_timestamp() | utc_timestamp() + 1 | +---------------------+---------------------+ | 2019-07-10 12:31:18 | 20190710123119 | +---------------------+---------------------+ ``` `utc_timestamp() + N` means adding `N` seconds to the current time. #### keyword[​](#keyword "Direct link to keyword") UTC\_TIMESTAMP,UTC,TIMESTAMP --- ### week Returns the week number for a given date. This function works in the same way as the WEEK function in MySQL. This function is supported from v2.3. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT WEEK(DATETIME|DATE date, INT mode) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `Date`: The supported data types are DATETIME and DATE. * `Mode`: optional. The supported data type is INT. This parameter is used to specify the logic for calculating the week number, that is, whether the week starts on Sunday or Monday, and whether the return value is in the range of 0–53 or 1–53. Value range: 0–7. Default value: `0`. If this parameter is not specified, mode `0` is used by default. The following table describes how this parameter works. | Mode | First day of week | Range | Week 1 is the first week … | | ---- | ----------------- | ----- | ----------------------------- | | 0 | Sunday | 0-53 | with a Sunday in this year | | 1 | Monday | 0-53 | with 4 or more days this year | | 2 | Sunday | 1-53 | with a Sunday in this year | | 3 | Monday | 1-53 | with 4 or more days this year | | 4 | Sunday | 0-53 | with 4 or more days this year | | 5 | Monday | 0-53 | with a Monday in this year | | 6 | Sunday | 1-53 | with 4 or more days this year | | 7 | Monday | 1-53 | with a Monday in this year | #### Return value[​](#return-value "Direct link to Return value") Returns a value of the INT type. Value range: 0–53. The specific range is determined based on the `mode` parameter. `NULL` is returned if the value of `date` is invalid or the input value is empty. #### Examples[​](#examples "Direct link to Examples") Calculate the week number for `2007-01-01`. `2007-01-01` is Monday on the calendar. * `Mode` is set to `0` and `0` is returned. The week starts on Sunday. `2007-01-01` is Monday and cannot be week 1. Therefore, `0` is returned. ```plaintext mysql> SELECT WEEK('2007-01-01', 0); +-----------------------+ | week('2007-01-01', 0) | +-----------------------+ | 0 | +-----------------------+ 1 row in set (0.02 sec) ``` * `Mode` is set to `1` and `1` is returned. The week starts on Monday and `2007-01-01` is Monday. ```plaintext mysql> SELECT WEEK('2007-01-01', 1); +-----------------------+ | week('2007-01-01', 1) | +-----------------------+ | 1 | +-----------------------+ 1 row in set (0.02 sec) ``` * `Mode` is set to `2` and `53` is returned. The week starts on Sunday. However, `2007-01-01` is Monday and the value range is 1–53. Therefore, `53` is returned, indicating that this is the last week of the previous year. ```plaintext mysql> SELECT WEEK('2007-01-01', 2); +-----------------------+ | week('2007-01-01', 2) | +-----------------------+ | 53 | +-----------------------+ 1 row in set (0.01 sec) ``` --- ### week_iso Returns the ISO standard week of the year for the specified date as an integer within the range of `1` to `53`. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT WEEK_ISO(DATETIME date) ``` #### Parameters[​](#parameters "Direct link to Parameters") `date`: the date you want to convert. It must be of the DATE or DATETIME type. #### Examples[​](#examples "Direct link to Examples") The following example returns the ISO standard week of the year for the date `2008-02-20 00:00:00`: ```sql MySQL > select week_iso ('2008-02-20 00:00:00'); +-----------------------------------+ | week_iso('2008-02-20 00:00:00') | +-----------------------------------+ | 8 | +-----------------------------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") WEEK\_ISO --- ### weekday Returns the weekday index for a given date. For example, the index for Monday is 0, for Sunday is 6. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT WEEKDAY(DATETIME date) ``` #### Parameters[​](#parameters "Direct link to Parameters") `date`: The `date` parameter must be of the DATE or DATETIME type, or a valid expression that can be cast into a DATE or DATETIME value. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select weekday('2023-01-01'); +-----------------------+ | weekday('2023-01-01') | +-----------------------+ | 6 | +-----------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") WEEKDAY --- ### weekofyear Returns the week number for a given date within a year. The `date` parameter must be of the DATE or DATETIME type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT WEEKOFYEAR(DATETIME date) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select weekofyear('2008-02-20 00:00:00'); +-----------------------------------+ | weekofyear('2008-02-20 00:00:00') | +-----------------------------------+ | 8 | +-----------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") WEEKOFYEAR --- ### weeks_add Returns the value with the number of weeks added to date. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME weeks_add(DATETIME expr1, INT expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr1`: the original date. It must be of the `DATETIME` type. * `expr2`: the number of weeks. It must be of the `INT` type. #### Return value[​](#return-value "Direct link to Return value") returns `DATETIME`. `NULL` is returned if the date does not exist. #### Examples[​](#examples "Direct link to Examples") ```plain select weeks_add('2022-12-20',2); +----------------------------+ | weeks_add('2022-12-20', 2) | +----------------------------+ | 2023-01-03 00:00:00 | +----------------------------+ ``` --- ### weeks_diff Returns the week difference between two date expressions (*`expr1`* − *`expr2`*), accurate to the week. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT weeks_diff(DATETIME expr1,DATETIME expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr1`: the end time. It must be of the DATETIME or DATE type. * `expr2`: the start time. It must be of the DATETIME or DATE type. #### Return value[​](#return-value "Direct link to Return value") Returns a BIGINT value. NULL is returned if the date does not exist, for example, 2022-02-29. #### Examples[​](#examples "Direct link to Examples") ```plain select weeks_diff('2010-11-30 23:59:59', '2010-1-1 23:59:59'); +--------------------------------------------------------+ | weeks_diff('2010-11-30 23:59:59', '2010-1-1 23:59:59') | +--------------------------------------------------------+ | 47 | +--------------------------------------------------------+ select weeks_diff(current_time(), '2010-11-30 23:59:59'); +---------------------------------------------------+ | weeks_diff(current_time(), '2010-11-30 23:59:59') | +---------------------------------------------------+ | 619 | +---------------------------------------------------+ select weeks_diff('2010-11-30 23:59:59', '2010-11-24 23:59:59'); +----------------------------------------------------------+ | weeks_diff('2010-11-30 23:59:59', '2010-11-24 23:59:59') | +----------------------------------------------------------+ | 0 | +----------------------------------------------------------+ ``` --- ### weeks_sub Subtracts a specified number of weeks from a datetime or date value. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME weeks_sub(DATETIME expr1, INT expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr1`: the original date. It must be of the `DATETIME` type. * `expr2`: the number of weeks. It must be of the `INT` type. #### Return value[​](#return-value "Direct link to Return value") Returns a `DATETIME` value. `NULL` is returned if the date does not exist. #### Examples[​](#examples "Direct link to Examples") ```plain select weeks_sub('2022-12-22',2); +----------------------------+ | weeks_sub('2022-12-22', 2) | +----------------------------+ | 2022-12-08 00:00:00 | +----------------------------+ ``` --- ### year Returns the year part in a date and returns a value that ranges from 1000 to 9999. The `date` parameter must be of the DATE or DATETIME type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT YEAR(DATETIME date) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select year('1987-01-01'); +-----------------------------+ | year('1987-01-01 00:00:00') | +-----------------------------+ | 1987 | +-----------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") YEAR --- ### years_add #### description[​](#description "Direct link to description") ##### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME YEARS_ADD(DATETIME expr1,INT expr2) ``` Add a specified time interval to the date. Unit is year. The expr1 parameter is a valid datetime expression. The expr2 parameter is the years you want to add. #### example[​](#example "Direct link to example") ```plain select years_add('2010-11-30 23:50:50', 2); +-------------------------------------+ | years_add('2010-11-30 23:50:50', 2) | +-------------------------------------+ | 2012-11-30 23:50:50 | +-------------------------------------+ select years_add('2010-11-30', 2); +----------------------------+ | years_add('2010-11-30', 2) | +----------------------------+ | 2012-11-30 00:00:00 | +----------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") YEARS\_ADD,ADD --- ### years_diff Returns the year difference between two date expressions (`expr1` − `expr2`), accurate to the year. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT years_diff(DATETIME expr1,DATETIME expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr1`: the end time. It must be of the DATETIME or DATE type. * `expr2`: the start time. It must be of the DATETIME or DATE type. #### Return value[​](#return-value "Direct link to Return value") Returns a BIGINT value. NULL is returned if the date does not exist, for example, 2022-02-29. #### Examples[​](#examples "Direct link to Examples") ```plain select years_diff('2010-11-30 23:59:59', '2000-11-1 23:59:59'); +---------------------------------------------------------+ | years_diff('2010-11-30 23:59:59', '2000-11-1 23:59:59') | +---------------------------------------------------------+ | 10 | +---------------------------------------------------------+ select years_diff('2010-11-30', '2000-11-1'); +---------------------------------------+ | years_diff('2010-11-30', '2000-11-1') | +---------------------------------------+ | 10 | +---------------------------------------+ ``` --- ### years_sub Subtracts the specified number of years from the specified datetime or date. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DATETIME YEARS_SUB(DATETIME date, INT years) ``` #### Parameters[​](#parameters "Direct link to Parameters") `date`: The original date time, of type DATETIME or DATE. `years`: The number of years to subtract. The value can be negative, but date year minus years can't exceed 10000. For example, if the year of date is 2022, then years can't be less than -7979. At the same time, the years cannot exceed the year value of date, for example, if the year value of date is 2022, then years can't be greater than 2022. #### Return value[​](#return-value "Direct link to Return value") The return value type is the same as the parameter `date`. Returns NULL if the result year is out of range \[0, 9999]. #### Examples[​](#examples "Direct link to Examples") ```plain select years_sub("2022-12-20 15:50:21", 2); +-------------------------------------+ | years_sub('2022-12-20 15:50:21', 2) | +-------------------------------------+ | 2020-12-20 15:50:21 | +-------------------------------------+ ``` --- ### yearweek Returns year and week number for a given date. This function works in the same way as the yearweek() function in MySQL. This function is supported from v3.3 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT YEARWEEK(DATETIME|DATE date[, INT mode]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `date`: The supported data types are DATETIME and DATE. * `mode`: optional. The supported data type is INT. This parameter is used to specify the logic for calculating the yearweek number, that is, whether the week starts on Sunday or Monday, and whether the return value is in the range of 0~~53 or 1~~53. Value range: 0~7. Default value: `0`. If this parameter is not specified, mode `0` is used by default. The following table describes how this parameter works. | Mode | First day of week | Range | Week 1 is the first week … | | ---- | ----------------- | ----- | ----------------------------- | | 0 | Sunday | 0-53 | with a Sunday in this year | | 1 | Monday | 0-53 | with 4 or more days this year | | 2 | Sunday | 1-53 | with a Sunday in this year | | 3 | Monday | 1-53 | with 4 or more days this year | | 4 | Sunday | 0-53 | with 4 or more days this year | | 5 | Monday | 0-53 | with a Monday in this year | | 6 | Sunday | 1-53 | with 4 or more days this year | | 7 | Monday | 1-53 | with a Monday in this year | #### Return value[​](#return-value "Direct link to Return value") Returns a value of the INT type. Value range: 0~53. The specific range is determined based on the `mode` parameter. `NULL` is returned if the value of `date` is invalid or the input value is empty. #### Examples[​](#examples "Direct link to Examples") Returns year and week for a date. The value of the mode argument defaults to 0 ```plaintext mysql> SELECT YEARWEEK('2007-01-01', 0); +---------------------------+ | yearweek('2007-01-01', 0) | +---------------------------+ | 200653 | +---------------------------+ ``` ```plaintext mysql> SELECT YEARWEEK('2007-01-01', 1); +---------------------------+ | yearweek('2007-01-01', 1) | +---------------------------+ | 200701 | +---------------------------+ ``` ```plaintext mysql> SELECT YEARWEEK('2007-01-01', 2); +---------------------------+ | yearweek('2007-01-01', 2) | +---------------------------+ | 200653 | +---------------------------+ 1 row in set (0.01 sec) ``` --- ### Dictionary #### [📄️ dict\_mapping](https://docs.starrocks.io/docs/sql-reference/sql-functions/dict-functions/dict_mapping.md) [Returns the value mapped to the specified key in a dictionary table.](https://docs.starrocks.io/docs/sql-reference/sql-functions/dict-functions/dict_mapping.md) #### [📄️ dictionary\_get](https://docs.starrocks.io/docs/sql-reference/sql-functions/dict-functions/dictionary_get.md) [Query the value mapped to the key in a dictionary object.](https://docs.starrocks.io/docs/sql-reference/sql-functions/dict-functions/dictionary_get.md) --- ### dict_mapping Returns the value mapped to the specified key in a dictionary table. This function is mainly used to simplify the application of a global dictionary table. During data loading into a target table, StarRocks automatically obtains the value mapped to the specified key from the dictionary table by using the input parameters in this function, and then loads the value into the target table. Since v3.2.5, StarRocks supports this function. Also, note that currently StarRocks's shared-data mode does not support this function. #### Syntax[​](#syntax "Direct link to Syntax") ```sql dict_mapping("[.]", key_column_expr_list [, ] [, ] ) key_column_expr_list ::= key_column_expr [, key_column_expr ... ] key_column_expr ::= | ``` #### Parameters[​](#parameters "Direct link to Parameters") * Required parameters: * `[.]`: The name of the dictionary table, which needs to be a Primary Key table. The supported data type is VARCHAR. * `key_column_expr_list`: The expression list for key columns in the dictionary table, including one or multiple `key_column_exprs`. The `key_column_expr` can be the name of a key column in the dictionary table, or a specific key or key expression. This expression list needs to include all Primary Key columns of the dictionary table, which means the total number of expressions needs to match the total number of Primary Key columns in the dictionary table. So when the dictionary table uses Composite Primary Key, the expressions in this list needs to correspond to the Primary Key columns defined in the table schema by sequence. Multiple expressions in this list are separated by commas (`,`). And if a `key_column_expr` is a specific key or key expression, its type must match the type of the corresponding Primary Key column in the dictionary table. * Optional parameters: * ``: The name of the value column, which is also the mapping column. If the value column is not specified, the default value column is the AUTO\_INCREMENT column of the dictionary table. The value column can also be defined as any column in the dictionary table excluding auto-incremented columns and primary keys. The column's data type has no restrictions. * `` (Optional): Whether to return if the key does not exist in the dictionary table. Valid values: * `true`: Null is returned if the key does not exist. * `false` (Default): An exception is thrown if the key does not exist. #### Usage Notes[​](#usage-notes "Direct link to Usage Notes") By default, the system will allow the optimizer to choose the `COUNT(DISTINCT)` implementation. When deduplicating low- and medium-cardinality columns, you can set `count_distinct_implementation` to `multi_count_distinct` for a query via query hint to verify the `multi_distinct_count` implementation for precise counting. However, do not use the `multi_distinct_count` implementation when deduplicating high-cardinality columns, because its HashSet status and final merging can cause excessive memory consumption and even OOM. #### Return Value[​](#return-value "Direct link to Return Value") The data type of the returned values remains consistent with the data type of the value column. If the value column is the auto-incremented column of the dictionary table, the data type of the returned values is BIGINT. However, when the value mapped to the specified key is not found, if the `` parameter is set to `true`, `NULL` is returned. If the parameter is set to `false`(default), an error `query failed if record not exist in dict table` is returned. #### Example[​](#example "Direct link to Example") **Example 1: Directly query the value mapped to a key from a dictionary table.** 1. Create a dictionary table and load simulated data. ```sql MySQL [test]> CREATE TABLE dict ( order_uuid STRING, order_id_int BIGINT AUTO_INCREMENT ) PRIMARY KEY (order_uuid) DISTRIBUTED BY HASH (order_uuid); Query OK, 0 rows affected (0.02 sec) MySQL [test]> INSERT INTO dict (order_uuid) VALUES ('a1'), ('a2'), ('a3'); Query OK, 3 rows affected (0.12 sec) {'label':'insert_9e60b0e4-89fa-11ee-a41f-b22a2c00f66b', 'status':'VISIBLE', 'txnId':'15029'} MySQL [test]> SELECT * FROM dict; +------------+--------------+ | order_uuid | order_id_int | +------------+--------------+ | a1 | 1 | | a3 | 3 | | a2 | 2 | +------------+--------------+ 3 rows in set (0.01 sec) ``` > **NOTICE** > > Currently the `INSERT INTO` statement does not support partial updates. So please make sure that the values inserted into the `dict`'s key column are not duplicated. Otherwise, inserting the same key column value in the dictionary table multiple times causes its mapped value in the value column to change. 2. Query the value mapped to key `a1` in the dictionary table. ```sql MySQL [test]> SELECT dict_mapping('dict', 'a1'); +----------------------------+ | dict_mapping('dict', 'a1') | +----------------------------+ | 1 | +----------------------------+ 1 row in set (0.01 sec) ``` **Example 2: The mapping column in the table is configured as a generated column using the `dict_mapping` function. So StarRocks can automatically obtain the values mapped to the keys when loading data into this table.** 1. Create a data table and configure the mapping column as a generated column by using `dict_mapping('dict', order_uuid)`. ```sql CREATE TABLE dest_table1 ( id BIGINT, -- This column records the STRING type order number, corresponding to the order_uuid column in the dict table in Example 1. order_uuid STRING, batch int comment 'used to distinguish different batch loading', -- This column records the BIGINT type order number which mapped with the order_uuid column. -- Because this column is a generated column configured with dict_mapping, the values in this column are automatically obtained from the dict table in Example 1 during data loading. -- Subsequently, this column can be directly used for deduplication and JOIN queries. order_id_int BIGINT AS dict_mapping('dict', order_uuid) ) DUPLICATE KEY (id, order_uuid) DISTRIBUTED BY HASH(id); ``` 2. When loading simulated data into this table where the `order_id_int` column is configured as `dict_mapping('dict', 'order_uuid')`, StarRocks automatically loads values into the `order_id_int` column based on the mapping relationship between keys and values in the `dict` table. ```sql MySQL [test]> INSERT INTO dest_table1(id, order_uuid, batch) VALUES (1, 'a1', 1), (2, 'a1', 1), (3, 'a3', 1), (4, 'a3', 1); Query OK, 4 rows affected (0.05 sec) {'label':'insert_e191b9e4-8a98-11ee-b29c-00163e03897d', 'status':'VISIBLE', 'txnId':'72'} MySQL [test]> SELECT * FROM dest_table1; +------+------------+-------+--------------+ | id | order_uuid | batch | order_id_int | +------+------------+-------+--------------+ | 1 | a1 | 1 | 1 | | 4 | a3 | 1 | 3 | | 2 | a1 | 1 | 1 | | 3 | a3 | 1 | 3 | +------+------------+-------+--------------+ 4 rows in set (0.02 sec) ``` The usage of `dict_mapping` in this example can accelerate [deduplication calculations and JOIN queries](https://docs.starrocks.io/docs/using_starrocks/query_acceleration_with_auto_increment.md). Compared to the previous solutions for building a global dictionary to accelerate precise deduplication, the solution by using `dict_mapping` is more flexible and user-friendly. Because the mapping values are directly obtained from the dictionary table at the stage "loading mapping relationships between keys and values into the table". You do not need to write statements to join the dictionary table to obtain mapping values. Additionally, this solution supports various data loading methods. **Example 3: If the mapping column in the table is not configured as a generated column, you need to explicitly configure the `dict_mapping` function for the mapping column when loading data into the table, obtain the values mapped to the keys.** > **NOTICE** > > The difference between Example 3 and Example 2 is that when importing into the data table, you need to modify the import command to explicitly configure the `dict_mapping` expression for the mapping column. 1. Create a table. ```sql CREATE TABLE dest_table2 ( id BIGINT, order_uuid STRING, order_id_int BIGINT NULL, batch int comment 'used to distinguish different batch loading' ) DUPLICATE KEY (id, order_uuid, order_id_int) DISTRIBUTED BY HASH(id); ``` 2. When simulated data loads into this table, you obtain the mapped values from the dictionary table by configuring `dict_mapping`. ```sql MySQL [test]> INSERT INTO dest_table2 VALUES (1, 'a1', dict_mapping('dict', 'a1'), 1); Query OK, 1 row affected (0.35 sec) {'label':'insert_19872ab6-8a96-11ee-b29c-00163e03897d', 'status':'VISIBLE', 'txnId':'42'} MySQL [test]> SELECT * FROM dest_table2; +------+------------+--------------+-------+ | id | order_uuid | order_id_int | batch | +------+------------+--------------+-------+ | 1 | a1 | 1 | 1 | +------+------------+--------------+-------+ 1 row in set (0.02 sec) ``` **Example 4: Enable null\_if\_not\_exist mode** When `` mode is disabled and the value mapped to the key that doesn't exist in the dictionary table is queried , an error, instead of `NULL`, is returned. It makes sure that a data row's key is first loaded into the dictionary table and its mapped value (dictionary ID) is generated before that data row is loaded into the target table. ```sql MySQL [test]> SELECT dict_mapping('dict', 'b1', true); ERROR 1064 (HY000): Query failed if record not exist in dict table. ``` **Example 5: If the dictionary table uses composite primary keys, all primary keys must be specified when querying.** 1. Create a dictionary table with Composite Primary Keys and load simulated data into it. ```sql MySQL [test]> CREATE TABLE dict2 ( order_uuid STRING, order_date DATE, order_id_int BIGINT AUTO_INCREMENT ) PRIMARY KEY (order_uuid,order_date) -- Composite Primary Key DISTRIBUTED BY HASH (order_uuid,order_date) ; Query OK, 0 rows affected (0.02 sec) MySQL [test]> INSERT INTO dict2 VALUES ('a1','2023-11-22',default), ('a2','2023-11-22',default), ('a3','2023-11-22',default); Query OK, 3 rows affected (0.12 sec) {'label':'insert_9e60b0e4-89fa-11ee-a41f-b22a2c00f66b', 'status':'VISIBLE', 'txnId':'15029'} MySQL [test]> select * from dict2; +------------+------------+--------------+ | order_uuid | order_date | order_id_int | +------------+------------+--------------+ | a1 | 2023-11-22 | 1 | | a3 | 2023-11-22 | 3 | | a2 | 2023-11-22 | 2 | +------------+------------+--------------+ 3 rows in set (0.01 sec) ``` 2. Query the value mapped to the key in the dictionary table. Because the dictionary table has Composite Primary Keys, all primary keys need to be specified in `dict_mapping`. ```sql SELECT dict_mapping('dict2', 'a1', cast('2023-11-22' as DATE)); ``` Note that an error occurs when only one Primary Key is specified. ```sql MySQL [test]> SELECT dict_mapping('dict2', 'a1'); ERROR 1064 (HY000): Getting analyzing error. Detail message: dict_mapping function param size should be 3 - 5. ``` --- ### dictionary_get Query the value mapped to the key in a dictionary object. #### Syntax[​](#syntax "Direct link to Syntax") ```sql dictionary_get('dictionary_object_name', key_expression_list, [NULL_IF_NOT_EXIST]) key_expression_list ::= key_expression [, ...] key_expression ::= column_name | const_value ``` #### Parameters[​](#parameters "Direct link to Parameters") * `dictionary_name`: The name of the dictionary object. * `key_expression_list`: A list of expressions for all key columns. It can be a list of column names or a list of values. * `NULL_IF_NOT_EXIST` (Optional): Whether to return if the key does not exist in the dictionary cache. Valid values: * `true`: Null is returned if the key does not exist. * `false` (Default): An exception is thrown if the key does not exist. #### Returns[​](#returns "Direct link to Returns") Returns the values of value columns as a STRUCT type. Therefore, you can use `[N]` or `.` to specify a particular column's value. `N` represents the column's position, starting from 1. #### Examples[​](#examples "Direct link to Examples") The following examples uses the dataset from the examples of [dict\_mapping](https://docs.starrocks.io/docs/sql-reference/sql-functions/dict-functions/dict_mapping.md). * Example 1: Query the values of the value column mapped to the key column `order_uuid` in the dictionary object `dict_obj`. ```plain MySQL > SELECT dictionary_get('dict_obj', order_uuid) FROM dict; +--------------------+ | DICTIONARY_GET | +--------------------+ | {"order_id_int":1} | | {"order_id_int":3} | | {"order_id_int":2} | +--------------------+ 3 rows in set (0.02 sec) ``` * Example 2: Query the value of the value column mapped to key `a1` in the dictionary object `dict_obj`. ```plain MySQL > SELECT dictionary_get("dict_obj", "a1"); +--------------------+ | DICTIONARY_GET | +--------------------+ | {"order_id_int":1} | +--------------------+ 1 row in set (0.01 sec) ``` * Example 3: Query the values of the value columns mapped to key `1` in the dictionary object `dimension_obj`. ```plain MySQL > SELECT dictionary_get("dimension_obj", 1); +-----------------------------------------------------------------------------------------------------------------+ | DICTIONARY_GET | +-----------------------------------------------------------------------------------------------------------------+ | {"ProductName":"T-Shirt","Category":"Apparel","SubCategory":"Shirts","Brand":"BrandA","Color":"Red","Size":"M"} | +-----------------------------------------------------------------------------------------------------------------+ 1 row in set (0.01 sec) ``` * Example 4: Query the value of the first value column mapped to key `1` in the dictionary object `dimension_obj`. ```plain MySQL > SELECT dictionary_get("dimension_obj", 1)[1]; +-------------------+ | DICTIONARY_GET[1] | +-------------------+ | T-Shirt | +-------------------+ 1 row in set (0.01 sec) ``` * Example 5: Query the value of the second value column mapped to key `1` in the dictionary object `dimension_obj`. ```plain MySQL > SELECT dictionary_get("dimension_obj", 1)[2]; +-------------------+ | DICTIONARY_GET[2] | +-------------------+ | Apparel | +-------------------+ 1 row in set (0.01 sec) ``` * Example 6: Query the value of `ProductName` value column mapped to key `1` in the dictionary object `dimension_obj`. ```plain MySQL > SELECT dictionary_get("dimension_obj", 1).ProductName; +----------------------------+ | DICTIONARY_GET.ProductName | +----------------------------+ | T-Shirt | +----------------------------+ 1 row in set (0.01 sec) ``` --- ### Hash #### [📄️ murmur\_hash3\_32](https://docs.starrocks.io/docs/sql-reference/sql-functions/hash-functions/murmur_hash3_32.md) [Returns the 32-bit murmur3 hash value of the input string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/hash-functions/murmur_hash3_32.md) #### [📄️ xx\_hash3\_128](https://docs.starrocks.io/docs/sql-reference/sql-functions/hash-functions/xx_hash3_128.md) [Returns the 128-bit xxhash3 hash value of the input string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/hash-functions/xx_hash3_128.md) #### [📄️ xx\_hash3\_64](https://docs.starrocks.io/docs/sql-reference/sql-functions/hash-functions/xx_hash3_64.md) [Returns the 64-bit xxhash3 hash value of the input string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/hash-functions/xx_hash3_64.md) --- ### murmur_hash3_32 Returns the 32-bit murmur3 hash value of the input string. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT MURMUR_HASH3_32(VARCHAR input, ...) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select murmur_hash3_32(null); +-----------------------+ | murmur_hash3_32(NULL) | +-----------------------+ | NULL | +-----------------------+ MySQL > select murmur_hash3_32("hello"); +--------------------------+ | murmur_hash3_32('hello') | +--------------------------+ | 1321743225 | +--------------------------+ MySQL > select murmur_hash3_32("hello", "world"); +-----------------------------------+ | murmur_hash3_32('hello', 'world') | +-----------------------------------+ | 984713481 | +-----------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") MURMUR\_HASH3\_32,HASH --- ### xx_hash3_128 Returns the 128-bit xxhash3 hash value of the input string. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext LARGEINT XX_HASH3_128(VARCHAR input, ...) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select xx_hash3_128(null); +-------------------+ | xx_hash3_128(NULL) | +-------------------+ | NULL | +-------------------+ MySQL > select xx_hash3_128("hello"); +-----------------------------------------+ | xx_hash3_128('hello') | +-----------------------------------------+ | -98478366302105124680504504609445627880 | +-----------------------------------------+ MySQL > select xx_hash3_128("hello", "world"); +-----------------------------------------+ | xx_hash3_128('hello', 'world') | +-----------------------------------------+ | -45235302294609689438180196264627189905 | +-----------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") XX\_HASH3\_128,HASH,xxHash3 --- ### xx_hash3_64 Returns the 64-bit xxhash3 hash value of the input string. xx\_hash3\_64 has better performance than [murmur\_hash3\_32](https://docs.starrocks.io/docs/sql-reference/sql-functions/hash-functions/murmur_hash3_32.md) by using AVX2 instruction and has state-of-art hash quality which is broadly integrated with many software. This function is supported from v3.2.0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT XX_HASH3_64(VARCHAR input, ...) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select xx_hash3_64(null); +-------------------+ | xx_hash3_64(NULL) | +-------------------+ | NULL | +-------------------+ MySQL > select xx_hash3_64("hello"); +----------------------+ | xx_hash3_64('hello') | +----------------------+ | -7685981735718036227 | +----------------------+ MySQL > select xx_hash3_64("hello", "world"); +-------------------------------+ | xx_hash3_64('hello', 'world') | +-------------------------------+ | 7001965798170371843 | +-------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") XX\_HASH3\_64,HASH,xxHash3 --- ### Hive Bitmap UDF Hive Bitmap UDF provides UDFs that can be directly used in Hive. They can be used to generate Bitmap data and perform Bitmap-related calculations. The Bitmap format defined by Hive Bitmap UDF is consistent with the format in StarRocks and can be directly used for loading Bitmap data into StarRocks and unloading Bitmap data from StarRocks to Hive. Applicable scenarios: * The amount of raw data is large and directly loading these data into StarRocks for computing will cause tremendous pressure on StarRocks clusters. The desired solution is generating Bitmap data in Hive and then loading Bitmap into StarRocks. * Export the Bitmap data generated in StarRocks to Hive for other systems to use. Supported source and target data types: * v3.1 and later support loading and unloading data of these types: String, Base64, and Binary. * v2.5 and v3.0 only support loading and unloading of String and Base64 data. #### Hive Bitmap UDFs that can be generated[​](#hive-bitmap-udfs-that-can-be-generated "Direct link to Hive Bitmap UDFs that can be generated") * com.starrocks.hive.udf.UDAFBitmapAgg Combines multiple rows of non-null values in a column into one row of Bitmap values, which is equivalent to StarRocks' built-in aggregate function [bitmap\_agg](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_agg.md). * com.starrocks.hive.udf.UDAFBitmapUnion Calculates the union of a set of bitmaps, which is equivalent to StarRocks' built-in aggregate function [bitmap\_union](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_union.md). * com.starrocks.hive.udf.UDFBase64ToBitmap Converts a base64-encoded string into a bitmap, which is equivalent to StarRocks' built-in function [base64\_to\_bitmap](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/base64_to_bitmap.md). * com.starrocks.hive.udf.UDFBitmapAnd Calculates the intersection of two bitmaps, which is equivalent to StarRocks' built-in function [bitmap\_and](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_and.md). * com.starrocks.hive.udf.UDFBitmapCount Counts the number of values in the bitmap, which is equivalent to StarRocks' built-in function [bitmap\_count](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_count.md). * com.starrocks.hive.udf.UDFBitmapFromString Converts a comma-separated string to a bitmap, equivalent to StarRocks' built-in function [bitmap\_from\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_from_string.md). * com.starrocks.hive.udf.UDFBitmapOr Calculates the union of two bitmaps, equivalent to StarRocks' built-in function [bitmap\_or](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_or.md). * com.starrocks.hive.udf.UDFBitmapToBase64 Converts Bitmap to Base64 string, equivalent to StarRocks' built-in function [bitmap\_to\_base64](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_to_base64.md). * com.starrocks.hive.udf.UDFBitmapToString Converts a bitmap to a comma-separated string, equivalent to StarRocks' built-in function [bitmap\_to\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_to_string.md). * com.starrocks.hive.udf.UDFBitmapXor Calculates the set of unique elements in two Bitmaps, which is equivalent to StarRocks' built-in function [bitmap\_xor](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_xor.md). #### How to use[​](#how-to-use "Direct link to How to use") 1. Compile and generate Hive UDF on the FE. ```bash ./build.sh --hive-udf ``` A JAR package `hive-udf-*.jar` will be generated in the `fe/hive-udf/` directory. 2. Upload the JAR package to HDFS. ```bash hadoop fs -put -f ./hive-udf-*.jar hdfs://:/hive-udf-*.jar ``` 3. Load the JAR package to Hive. ```bash hive> add jar hdfs://:/hive-udf-*.jar; ``` 4. Load UDF functions. ```sql hive> create temporary function bitmap_agg as 'com.starrocks.hive.udf.UDAFBitmapAgg'; hive> create temporary function bitmap_union as 'com.starrocks.hive.udf.UDAFBitmapUnion'; hive> create temporary function base64_to_bitmap as 'com.starrocks.hive.udf.UDFBase64ToBitmap'; hive> create temporary function bitmap_and as 'com.starrocks.hive.udf.UDFBitmapAnd'; hive> create temporary function bitmap_count as 'com.starrocks.hive.udf.UDFBitmapCount'; hive> create temporary function bitmap_from_string as 'com.starrocks.hive.udf.UDFBitmapFromString'; hive> create temporary function bitmap_or as 'com.starrocks.hive.udf.UDFBitmapOr'; hive> create temporary function bitmap_to_base64 as 'com.starrocks.hive.udf.UDFBitmapToBase64'; hive> create temporary function bitmap_to_string as 'com.starrocks.hive.udf.UDFBitmapToString'; hive> create temporary function bitmap_xor as 'com.starrocks.hive.udf.UDFBitmapXor'; ``` #### Usage examples[​](#usage-examples "Direct link to Usage examples") ##### Generate Bitmap in Hive and load it into StarRocks in Binary format[​](#generate-bitmap-in-hive-and-load-it-into-starrocks-in-binary-format "Direct link to Generate Bitmap in Hive and load it into StarRocks in Binary format") 1. Create a Hive source table. ```sql hive> create table t_src(c1 bigint, c2 bigint) stored as parquet; hive> insert into t_src values (1,1), (1,2), (1,3), (2,4), (2,5); hive> select * from t_src; 1 1 1 2 1 3 2 4 2 5 ``` 2. Create a Hive bitmap table. ```sql hive> create table t_bitmap(c1 bigint, c2 binary) stored as parquet; ``` Hive generates bitmap through UDFBitmapAgg and writes it into the bitmap table. ```sql hive> insert into t_bitmap select c1, bitmap_agg(c2) from t_src group by c1; ``` 3. Create a StarRocks Bitmap table. ```sql mysql> create table t1(c1 int, c2 bitmap bitmap_union) aggregate key(c1) distributed by hash(c1); ``` 4. Load Bitmap data into StarRocks in different ways. * Load data via the [files](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) function. ```sql mysql> insert into t1 select c1, bitmap_from_binary(c2) from files ( "path" = "hdfs://://t_bitmap/*", "format"="parquet", "compression" = "uncompressed" ); ``` * Load data via [Hive Catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md). ```sql mysql> insert into t1 select c1, bitmap_from_binary(c2) from hive_catalog_hms.xxx_db.t_bitmap; ``` 5. View the results. ```sql mysql> select c1, bitmap_to_string(c2) from t1; +------+----------------------+ | c1 | bitmap_to_string(c2) | +------+----------------------+ | 1 | 1,2,3 | | 2 | 4,5 | +------+----------------------+ ``` ##### Export Bitmap from StarRocks to Hive[​](#export-bitmap-from-starrocks-to-hive "Direct link to Export Bitmap from StarRocks to Hive") 1. Create a Bitmap table in StarRocks and write data into this table. ```sql mysql> create table t1(c1 int, c2 bitmap bitmap_union) aggregate key(c1) buckets 3 distributed by hash(c1); mysql> select c1, bitmap_to_string(c2) from t1; +------+----------------------+ | c1 | bitmap_to_string(c2) | +------+----------------------+ | 1 | 1,2,3 | | 2 | 4,5 | +------+----------------------+ ``` 2. Create a Bitmap table in Hive. ```sql hive> create table t_bitmap(c1 bigint, c2 binary) stored as parquet; ``` 3. Export data in different ways. * Export data via INSERT INTO FILES (Binary format). ```sql mysql> insert into files ( "path" = "hdfs://://t_bitmap/", "format"="parquet", "compression" = "uncompressed" ) select c1, bitmap_to_binary(c2) as c2 from t1; ``` * Export data via [Hive Catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md) (Binary format). ```sql mysql> insert into hive_catalog_hms..t_bitmap select c1, bitmap_to_binary(c2) from t1; ``` 4. View results in Hive. ```plain hive> select c1, bitmap_to_string(c2) from t_bitmap; 1 1,2,3 2 4,5 ``` --- ### Java UDF From v2.2.0 onwards, you can compile user-defined functions (UDFs) to suit your specific business needs by using the Java programming language. From v3.0 onwards, StarRocks supports global UDFs, and you only need to include the `GLOBAL` keyword in the related SQL statements (CREATE/SHOW/DROP). This topic how to develop and use various UDFs. Currently, StarRocks supports scalar UDFs, user-defined aggregate functions (UDAFs), user-defined window functions (UDWFs), and user-defined table functions (UDTFs). #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * You have installed [Apache Maven](https://maven.apache.org/download.cgi), so you can create and compile Java projects. * You have installed JDK 17 on your servers. * The Java UDF feature is enabled. You can set the FE configuration item `enable_udf` to `true` in the FE configuration file **fe/conf/fe.conf** to enable this feature, and then restart the FE nodes to make the settings take effect. For more information, see [Parameter configuration](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md). #### Develop and use UDFs[​](#develop-and-use-udfs "Direct link to Develop and use UDFs") You need to create a Maven project and compile the UDF you need by using the Java programming language. ##### Step 1: Create a Maven project[​](#step-1-create-a-maven-project "Direct link to Step 1: Create a Maven project") Create a Maven project, whose basic directory structure is as follows: ```plain project |--pom.xml |--src | |--main | | |--java | | |--resources | |--test |--target ``` ##### Step 2: Add dependencies[​](#step-2-add-dependencies "Direct link to Step 2: Add dependencies") Add the following dependencies to the **pom.xml** file: ```xml 4.0.0 org.example udf 1.0-SNAPSHOT 17 17 com.alibaba fastjson 1.2.76 org.apache.maven.plugins maven-dependency-plugin 2.10 copy-dependencies package copy-dependencies ${project.build.directory}/lib org.apache.maven.plugins maven-assembly-plugin 3.3.0 make-assembly package single jar-with-dependencies ``` ##### Step 3: Compile a UDF[​](#step-3-compile-a-udf "Direct link to Step 3: Compile a UDF") Use the Java programming language to compile a UDF. ###### Compile a scalar UDF[​](#compile-a-scalar-udf "Direct link to Compile a scalar UDF") A scalar UDF operates on a single row of data and returns a single value. When you use a scalar UDF in a query, each row corresponds to a single value in the result set. Typical scalar functions include `UPPER`, `LOWER`, `ROUND`, and `ABS`. Suppose that the values of a field in your JSON data are JSON strings rather than JSON objects. When you use an SQL statement to extract JSON strings, you need to run `GET_JSON_STRING` twice, for example, `GET_JSON_STRING(GET_JSON_STRING('{"key":"{\\"k0\\":\\"v0\\"}"}', "$.key"), "$.k0")`. To simplify the SQL statement, you can compile a scalar UDF that can directly extract JSON strings, for example, `MY_UDF_JSON_GET('{"key":"{\\"k0\\":\\"v0\\"}"}', "$.key.k0")`. ```java package com.starrocks.udf.sample; import com.alibaba.fastjson.JSONPath; public class UDFJsonGet { public final String evaluate(String obj, String key) { if (obj == null || key == null) return null; try { // The JSONPath library can be fully expanded even if the values of a field are JSON strings. return JSONPath.read(obj, key).toString(); } catch (Exception e) { return null; } } } ``` The user-defined class must implement the method described in the following table. > **NOTE** > > The data types of the request parameters and return parameters in the method must be the same as those declared in the `CREATE FUNCTION` statement that is to be executed in [Step 6](#step-6-create-the-udf-in-starrocks) and conform to the mapping that is provided in the "[Mapping between SQL data types and Java data types](#mapping-between-sql-data-types-and-java-data-types)" section of this topic. | Method | Description | | -------------------------- | ---------------------------------------------------------------------------- | | TYPE1 evaluate(TYPE2, ...) | Runs the UDF. The evaluate() method requires the public member access level. | ###### Compile a UDAF[​](#compile-a-udaf "Direct link to Compile a UDAF") A UDAF operates on multiple rows of data and returns a single value. Typical aggregate functions include `SUM`, `COUNT`, `MAX`, and `MIN`, which aggregate multiple rows of data specified in each GROUP BY clause and return a single value. Suppose that you want to compile a UDAF named `MY_SUM_INT`. Unlike the built-in aggregate function `SUM`, which returns BIGINT-type values, the `MY_SUM_INT` function supports only request parameters and returns parameters of the INT data type. ```java package com.starrocks.udf.sample; public class SumInt { public static class State { int counter = 0; public int serializeLength() { return 4; } } public State create() { return new State(); } public void destroy(State state) { } public final void update(State state, Integer val) { if (val != null) { state.counter+= val; } } public void serialize(State state, java.nio.ByteBuffer buff) { buff.putInt(state.counter); } public void merge(State state, java.nio.ByteBuffer buffer) { int val = buffer.getInt(); state.counter += val; } public Integer finalize(State state) { return state.counter; } } ``` The user-defined class must implement the methods described in the following table. > **NOTE** > > The data types of the request parameters and return parameters in the methods must be the same as those declared in the `CREATE FUNCTION` statement that is to be executed in [Step 6](#step-6-create-the-udf-in-starrocks) and conform to the mapping that is provided in the "[Mapping between SQL data types and Java data types](#mapping-between-sql-data-types-and-java-data-types)" section of this topic. | Method | Description | | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | State create() | Creates a state. | | void destroy(State) | Destroys a state. | | void update(State, ...) | Updates a state. In addition to the first parameter `State`, you can also specify one or more request parameters in the UDF declaration. | | void serialize(State, ByteBuffer) | Serializes a state into the byte buffer. | | void merge(State, ByteBuffer) | Deserializes a state from the byte buffer, and merges the byte buffer into the state as the first parameter. | | TYPE finalize(State) | Obtains the final result of the UDF from a state. | During compilation, you must also use the buffer class `java.nio.ByteBuffer` and the local variable `serializeLength`, which are described in the following table. | Class and local variable | Description | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | java.nio.ByteBuffer() | The buffer class, which stores intermediate results. Intermediate results may be serialized or deserialized when they are transmitted between nodes for execution. Therefore, you must also use the `serializeLength` variable to specify the length that is allowed for the deserialization of intermediate results. | | serializeLength() | The length that is allowed for the deserialization of intermediate results. Unit: bytes. Set this local variable to an INT-type value. For example, `State { int counter = 0; public int serializeLength() { return 4; }}` specifies that intermediate results are of the INT data type and the length for deserialization is 4 bytes. You can adjust these settings based on your business requirements. For example, if you want to specify the data type of intermediate results as LONG and the length for deserialization as 8 bytes, pass `State { long counter = 0; public int serializeLength() { return 8; }}`. | Take note of the following points for the deserialization of intermediate results stored in the `java.nio.ByteBuffer` class: * The remaining() method that is dependent on the `ByteBuffer` class cannot be called to deserialize a state. * The clear() method cannot be called on the `ByteBuffer` class. * The value of `serializeLength` must be the same as the length of the written-in data. Otherwise, incorrect results are generated during serialization and deserialization. ###### Compile a UDWF[​](#compile-a-udwf "Direct link to Compile a UDWF") Unlike regular aggregate functions, a UDWF operates on a set of multiple rows, which are collectively called a window, and returns a value for each row. A typical window function includes an `OVER` clause that divides rows into multiple sets. It performs a calculation across each set of rows and returns a value for each row. Suppose that you want to compile a UDWF named `MY_WINDOW_SUM_INT`. Unlike the built-in aggregate function `SUM`, which returns BIGINT-type values, the `MY_WINDOW_SUM_INT` function supports only request parameters and returns parameters of the INT data type. ```java package com.starrocks.udf.sample; public class WindowSumInt { public static class State { int counter = 0; public int serializeLength() { return 4; } @Override public String toString() { return "State{" + "counter=" + counter + '}'; } } public State create() { return new State(); } public void destroy(State state) { } public void update(State state, Integer val) { if (val != null) { state.counter+=val; } } public void serialize(State state, java.nio.ByteBuffer buff) { buff.putInt(state.counter); } public void merge(State state, java.nio.ByteBuffer buffer) { int val = buffer.getInt(); state.counter += val; } public Integer finalize(State state) { return state.counter; } public void reset(State state) { state.counter = 0; } public void windowUpdate(State state, int peer_group_start, int peer_group_end, int frame_start, int frame_end, Integer[] inputs) { for (int i = (int)frame_start; i < (int)frame_end; ++i) { state.counter += inputs[i]; } } } ``` The user-defined class must implement the method required by UDAFs (because a UDWF is a special aggregate function) and the windowUpdate() method described in the following table. > **NOTE** > > The data types of the request parameters and return parameters in the method must be the same as those declared in the `CREATE FUNCTION` statement that is to be executed in [Step 6](#step-6-create-the-udf-in-starrocks) and conform to the mapping that is provided in the "[Mapping between SQL data types and Java data types](#mapping-between-sql-data-types-and-java-data-types)" section of this topic. | Method | Description | | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | void windowUpdate(State state, int, int, int , int, ...) | Updates the data of the window. For more information about UDWFs, see [Window functions](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function.md). Every time when you enter a row as input, this method obtains the window information and updates intermediate results accordingly.- `peer_group_start`: the start position of the current partition. `PARTITION BY` is used in the OVER clause to specify a partition column. Rows with the same values in the partition column are considered to be in the same partition.
- `peer_group_end`: the end position of the current partition.
- `frame_start`: the start position of the current window frame. The window frame clause specifies a calculation range, which covers the current row and the rows that are within a specified distance to the current row. For example, `ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING` specifies a calculation range that covers the current row, the previous row before the current row, and the following row after the current row.
- `frame_end`: the end position of the current window frame.
- `inputs`: the data that is entered as the input to a window. The data is an array package that supports only specific data types. In this example, INT values are entered as input, and the array package is Integer\[]. | ###### Compile a UDTF[​](#compile-a-udtf "Direct link to Compile a UDTF") A UDTF reads one row of data and returns multiple values that can be considered to be a table. Table-valued functions are typically used to transform rows into columns. > **NOTE** > > StarRocks allows a UDTF to return a table that consists of multiple rows and one column. Suppose that you want to compile a UDTF named `MY_UDF_SPLIT`. The `MY_UDF_SPLIT` function allows you to use spaces as delimiters and supports request parameters and return parameters of the STRING data type. ```java package com.starrocks.udf.sample; public class UDFSplit{ public String[] process(String in) { if (in == null) return null; return in.split(" "); } } ``` The method defined by the user-defined class must meet the following requirements: > **NOTE** > > The data types of the request parameters and return parameters in the method must be the same as those declared in the `CREATE FUNCTION` statement that is to be executed in [Step 6](#step-6-create-the-udf-in-starrocks) and conform to the mapping that is provided in the "[Mapping between SQL data types and Java data types](#mapping-between-sql-data-types-and-java-data-types)" section of this topic. | Method | Description | | ----------------- | ----------------------------------- | | TYPE\[] process() | Runs the UDTF and returns an array. | ##### Step 4: Package the Java project[​](#step-4-package-the-java-project "Direct link to Step 4: Package the Java project") Run the following command to package the Java project: ```bash mvn package ``` The following JAR files are generated in the **target** folder: **udf-1.0-SNAPSHOT.jar** and **udf-1.0-SNAPSHOT-jar-with-dependencies.jar**. ##### Step 5: Upload the Java project[​](#step-5-upload-the-java-project "Direct link to Step 5: Upload the Java project") Upload the JAR file **udf-1.0-SNAPSHOT-jar-with-dependencies.jar** to an HTTP server that keeps up and running and is accessible to all FEs and BEs in your StarRocks cluster. Then, run the following command to deploy the file: ```bash mvn deploy ``` You can set up a simple HTTP server by using Python and upload the JAR file to that HTTP server. > **NOTE** > > In [Step 6](#step-6-create-the-udf-in-starrocks), the FEs will check the JAR file that contains the code for the UDF and calculate the checksum, and the BEs will download and execute the JAR file. ##### Step 6: Create the UDF in StarRocks[​](#step-6-create-the-udf-in-starrocks "Direct link to Step 6: Create the UDF in StarRocks") StarRocks allows you to create UDFs in two types of namespaces: database namespaces and global namespaces. * If you do not have visibility or isolation requirements for a UDF, you can create it as a global UDF. Then, you can reference the global UDF by using the function name without including the catalog and database names as prefixes to the function name. * If you have visibility or isolation requirements for a UDF, or if you need to create the same UDF in different databases, you can create it in each individual database. As such, if your session is connected to the target database, you can reference the UDF by using the function name. If your session is connected to a different catalog or database other than the target database, you need to reference the UDF by including the catalog and database names as prefixes to the function name, for example, `catalog.database.function`. > **NOTICE** > > Before you create and use a global UDF, you must contact the system administrator to grant you the required permissions. For more information, see [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md). After you upload the JAR package, you can create UDFs in StarRocks. For a global UDF, you must include the `GLOBAL` keyword in the creation statement. ###### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE [GLOBAL][AGGREGATE | TABLE] FUNCTION function_name (arg_type [, ...]) RETURNS return_type PROPERTIES ("key" = "value" [, ...]) ``` ###### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | -------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GLOBAL | No | Whether to create a global UDF, supported from v3.0. | | AGGREGATE | No | Whether to create a UDAF or UDWF. | | TABLE | No | Whether to create a UDTF. If both `AGGREGATE` and `TABLE` are not specified, a Scalar function is created. | | function\_name | Yes | The name of the function you want to create. You can include the name of the database in this parameter, for example,`db1.my_func`. If `function_name` includes the database name, the UDF is created in that database. Otherwise, the UDF is created in the current database. The name of the new function and its parameters cannot be the same as an existing name in the destination database. Otherwise, the function cannot be created. The creation succeeds if the function name is the same but the parameters are different. | | arg\_type | Yes | Argument type of the function. The added argument can be represented by `, ...`. For the supported data types, see [Mapping between SQL data types and Java data types](#mapping-between-sql-data-types-and-java-data-types). | | return\_type | Yes | The return type of the function. For the supported data types, see [Java UDF](#mapping-between-sql-data-types-and-java-data-types). | | PROPERTIES | Yes | Properties of the function, which vary depending on the type of the UDF to create. | ###### Create a scalar UDF[​](#create-a-scalar-udf "Direct link to Create a scalar UDF") Run the following command to create the scalar UDF you have compiled in the preceding example: ```sql CREATE [GLOBAL] FUNCTION MY_UDF_JSON_GET(string, string) RETURNS string PROPERTIES ( "symbol" = "com.starrocks.udf.sample.UDFJsonGet", "type" = "StarrocksJar", "file" = "http://http_host:http_port/udf-1.0-SNAPSHOT-jar-with-dependencies.jar" ); ``` | Parameter | Description | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | symbol | The name of the class for the Maven project to which the UDF belongs. The value of this parameter is in the `.` format. | | type | The type of the UDF. Set the value to `StarrocksJar`, which specifies that the UDF is a Java-based function. | | file | The HTTP URL from which you can download the JAR file that contains the code for the UDF. The value of this parameter is in the `http://:/` format. | | isolation | (Optional) To share function instances across UDF executions and support static variables, set this to "shared". | ###### Create a UDAF[​](#create-a-udaf "Direct link to Create a UDAF") Run the following command to create the UDAF you have compiled in the preceding example: ```sql CREATE [GLOBAL] AGGREGATE FUNCTION MY_SUM_INT(INT) RETURNS INT PROPERTIES ( "symbol" = "com.starrocks.udf.sample.SumInt", "type" = "StarrocksJar", "file" = "http://http_host:http_port/udf-1.0-SNAPSHOT-jar-with-dependencies.jar" ); ``` The descriptions of the parameters in PROPERTIES are the same as those in [Create a scalar UDF](#create-a-scalar-udf). ###### Create a UDWF[​](#create-a-udwf "Direct link to Create a UDWF") Run the following command to create the UDWF you have compiled in the preceding example: ```sql CREATE [GLOBAL] AGGREGATE FUNCTION MY_WINDOW_SUM_INT(Int) RETURNS Int properties ( "analytic" = "true", "symbol" = "com.starrocks.udf.sample.WindowSumInt", "type" = "StarrocksJar", "file" = "http://http_host:http_port/udf-1.0-SNAPSHOT-jar-with-dependencies.jar" ); ``` `analytic`: Whether the UDF is a window function. Set the value to `true`. The descriptions of other properties are the same as those in [Create a scalar UDF](#create-a-scalar-udf). ###### Create a UDTF[​](#create-a-udtf "Direct link to Create a UDTF") Run the following command to create the UDTF you have compiled in the preceding example: ```sql CREATE [GLOBAL] TABLE FUNCTION MY_UDF_SPLIT(string) RETURNS string properties ( "symbol" = "com.starrocks.udf.sample.UDFSplit", "type" = "StarrocksJar", "file" = "http://http_host:http_port/udf-1.0-SNAPSHOT-jar-with-dependencies.jar" ); ``` The descriptions of the parameters in PROPERTIES are the same as those in [Create a scalar UDF](#create-a-scalar-udf). ##### Step 7: Use the UDF[​](#step-7-use-the-udf "Direct link to Step 7: Use the UDF") After you create the UDF, you can test and use it based on your business needs. ###### Use a scalar UDF[​](#use-a-scalar-udf "Direct link to Use a scalar UDF") Run the following command to use the scalar UDF you have created in the preceding example: ```sql SELECT MY_UDF_JSON_GET('{"key":"{\\"in\\":2}"}', '$.key.in'); ``` ###### Use a UDAF[​](#use-a-udaf "Direct link to Use a UDAF") Run the following command to use the UDAF you have created in the preceding example: ```sql SELECT MY_SUM_INT(col1); ``` ###### Use a UDWF[​](#use-a-udwf "Direct link to Use a UDWF") Run the following command to use the UDWF you have created in the preceding example: ```sql SELECT MY_WINDOW_SUM_INT(intcol) OVER (PARTITION BY intcol2 ORDER BY intcol3 ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) FROM test_basic; ``` ###### Use a UDTF[​](#use-a-udtf "Direct link to Use a UDTF") Run the following command to use the UDTF you have created in the preceding example: ```plain -- Suppose that you have a table named t1, and the information about its columns a, b, and c1 is as follows: SELECT t1.a,t1.b,t1.c1 FROM t1; > output: 1,2.1,"hello world" 2,2.2,"hello UDTF." -- Run the MY_UDF_SPLIT() function. SELECT t1.a,t1.b, MY_UDF_SPLIT FROM t1, MY_UDF_SPLIT(t1.c1); > output: 1,2.1,"hello" 1,2.1,"world" 2,2.2,"hello" 2,2.2,"UDTF." ``` > **NOTE** > > * The first `MY_UDF_SPLIT` in the preceding code snippet is the alias of the column that is returned by the second `MY_UDF_SPLIT`, which is a function. > * You cannot use `AS t2(f1)` to specify the aliases of the table and its columns that are to be returned. #### View UDFs[​](#view-udfs "Direct link to View UDFs") Run the following command to query UDFs: ```sql SHOW [GLOBAL] FUNCTIONS; ``` For more information, see [SHOW FUNCTIONS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Function/SHOW_FUNCTIONS.md). #### Drop a UDF[​](#drop-a-udf "Direct link to Drop a UDF") Run the following command to drop a UDF: ```sql DROP [GLOBAL] FUNCTION (arg_type [, ...]); ``` For more information, see [DROP FUNCTION](https://docs.starrocks.io/docs/sql-reference/sql-statements/Function/DROP_FUNCTION.md). #### Mapping between SQL data types and Java data types[​](#mapping-between-sql-data-types-and-java-data-types "Direct link to Mapping between SQL data types and Java data types") > **NOTE** > > Scalar UDFs, UDAFs, and UDTFs all support nested `ARRAY`, `MAP`, and `STRUCT` parameter/return types — including arbitrary nesting such as `ARRAY>`, `ARRAY>`, `MAP>`, `STRUCT>`, and `ARRAY>`. The leaf element type must still be one of the scalar types listed below. Because of Java type erasure, the Java method signature only needs the raw `java.util.List` / `java.util.Map` for ARRAY / MAP slots whose subtree contains no STRUCT; StarRocks drives the per-row conversion from the SQL signature. STRUCT slots, on the other hand, must be bound to a concrete Java `record` class so the analyzer can preserve the formal record type through the JNI boundary (see [STRUCT type binding](#struct-type-binding) below). | SQL TYPE | Java TYPE | | ------------------------------------------ | ------------------------------------------------ | | BOOLEAN | java.lang.Boolean | | TINYINT | java.lang.Byte | | SMALLINT | java.lang.Short | | INT | java.lang.Integer | | BIGINT | java.lang.Long | | FLOAT | java.lang.Float | | DOUBLE | java.lang.Double | | STRING/VARCHAR | java.lang.String | | DECIMAL(p, s) (DECIMAL32 / 64 / 128 / 256) | java.math.BigDecimal | | DATE | java.time.LocalDate | | DATETIME | java.time.LocalDateTime | | ARRAY | java.util.List | | Map | java.util.Map | | STRUCT | a Java `record` class declared by the UDF author | > **NOTE** > > For `DECIMAL` parameters, BigDecimal values produced by the UDF are rescaled to the column's declared `(precision, scale)` using `RoundingMode.HALF_UP` before they are written back. If the rescaled value does not fit in the declared `(precision, scale)`, the behavior follows the session `overflow_mode`: > > * `OUTPUT_NULL` (default): the offending row is written as `NULL`. > * `REPORT_ERROR`: the query aborts with an `ArithmeticException`. ##### STRUCT type binding[​](#struct-type-binding "Direct link to STRUCT type binding") `STRUCT` parameters and return types must be bound to a Java `record` class (JDK 14+) declared by the UDF author. The mapping is positional: * The record's component count must match the SQL `STRUCT` field count. * Each component's type must match the corresponding SQL field type by position; component names are not enforced (Java identifiers cannot represent every legal SQL field name, and CREATE FUNCTION binds by position regardless of the session's `STRUCT_CAST_BY_NAME` setting). * Nested `STRUCT` is supported in any position: as a record component, as an `ARRAY` element (`List`), or as a `MAP` key/value (`Map`). The same record-class binding applies to scalar UDF, UDAF, and UDTF. For UDAF, the record class is supplied for the `update(State, ...)` arguments and the `finalize(State)` return type. For UDTF, the record class is supplied for the `process(...)` arguments and the `TYPE[] process(...)` element return type. ###### Scalar UDF example[​](#scalar-udf-example "Direct link to Scalar UDF example") ```java public record Address(String street, Integer zip) {} public record AddressOut(String full, Integer region) {} public class AddrUdf { public AddressOut evaluate(Address addr) { return new AddressOut(addr.street() + " #" + addr.zip(), addr.zip() / 1000); } } ``` ```sql CREATE FUNCTION addr_udf(struct) RETURNS struct<`full` string, region int> PROPERTIES ( "symbol" = "com.example.AddrUdf", "type" = "StarrocksJar", "file" = "http://localhost:8080/addr_udf.jar" ); ``` ###### UDAF example[​](#udaf-example "Direct link to UDAF example") ```java public record Item(String name, Integer qty) {} public record TopItem(String name, Long total) {} public class TopItemAgg { public static class State { // serialization elided for brevity public int serializeLength() { return 0; } } public State create() { return new State(); } public void destroy(State state) {} public void update(State state, Item item) { /* ... */ } public void serialize(State state, java.nio.ByteBuffer buf) { /* ... */ } public void merge(State state, java.nio.ByteBuffer buf) { /* ... */ } public TopItem finalize(State state) { return new TopItem("a", 0L); } } ``` ```sql CREATE AGGREGATE FUNCTION top_item_agg(struct) RETURNS struct PROPERTIES ( "symbol" = "com.example.TopItemAgg", "type" = "StarrocksJar", "file" = "http://localhost:8080/top_item_agg.jar" ); ``` ###### UDTF example[​](#udtf-example "Direct link to UDTF example") ```java public record Pair(String key, Integer value) {} public class ExplodePairs { public Pair[] process(java.util.Map m) { return m.entrySet().stream() .map(e -> new Pair(e.getKey(), e.getValue())) .toArray(Pair[]::new); } } ``` ```sql CREATE TABLE FUNCTION explode_pairs(map) RETURNS struct<`key` string, `value` int> PROPERTIES ( "symbol" = "com.example.ExplodePairs", "type" = "StarrocksJar", "file" = "http://localhost:8080/explode_pairs.jar" ); ``` #### Parameter settings[​](#parameter-settings "Direct link to Parameter settings") Configure the following environment variable in the **be/conf/be.conf** file of each Java virtual machine (JVM) in your StarRocks cluster to control memory usage. ```bash JAVA_OPTS="-Xmx12G" ``` #### FAQ[​](#faq "Direct link to FAQ") Can I use static variables when I create UDFs? Do the static variables of different UDFs have mutual impacts on each other? Yes, you can use static variables when you compile UDFs. The static variables of different UDFs are isolated from each other and do not affect each other even if the UDFs have classes with identical names. --- ### JSON #### [🗃️ JSON Constructor](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-constructor-functions.md) [3 items](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-constructor-functions.md) #### [🗃️ JSON query and processing](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions.md) [17 items](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions.md) #### [📄️ JSON operators](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-operators.md) [StarRocks supports the following JSON comparison operators: <, <=, >, >=, =, and !=.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-operators.md) #### [📄️ Overview of JSON functions and operators](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) [Accelerate your queries with generated columns](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) --- ### JSON Constructor #### [📄️ json\_array](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-constructor-functions/json_array.md) [Converts each element of an SQL array to a JSON value and returns a JSON array that consists of the JSON values.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-constructor-functions/json_array.md) #### [📄️ json\_object](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-constructor-functions/json_object.md) [Converts one or more key-value pairs to a JSON object that consists of the key-value pairs.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-constructor-functions/json_object.md) #### [📄️ parse\_json](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-constructor-functions/parse_json.md) [Converts a string to a JSON value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-constructor-functions/parse_json.md) --- ### json_array Converts each element of an SQL array to a JSON value and returns a JSON array that consists of the JSON values. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext json_array(value, ...) ``` #### Parameters[​](#parameters "Direct link to Parameters") `value`: an element in the SQL array. Only `NULL` values and the following data types are supported: STRING, VARCHAR, CHAR, JSON, TINYINT, SMALLINT, INT, BIGINT, LARGEINT, DOUBLE, FLOAT, and BOOLEAN. #### Return value[​](#return-value "Direct link to Return value") Returns a JSON array. #### Examples[​](#examples "Direct link to Examples") Example 1: Construct a JSON array that consists of values of different data types. ```plaintext mysql> SELECT json_array(1, true, 'starrocks', 1.1); -> [1, true, "starrocks", 1.1] ``` Example 2: Construct an empty JSON array. ```plaintext mysql> SELECT json_array(); -> [] ``` --- ### json_object Converts one or more key-value pairs to a JSON object that consists of the key-value pairs. The key-value pairs are sorted by key in dictionary order. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext json_object(key, value, ...) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `key`: a key in the JSON object. Only the VARCHAR data type is supported. * `value`: a value in the JSON object. Only `NULL` values and the following data types are supported: STRING, VARCHAR, CHAR, JSON, TINYINT, SMALLINT, INT, BIGINT, LARGEINT, DOUBLE, FLOAT, and BOOLEAN. #### Return value[​](#return-value "Direct link to Return value") Returns a JSON object. > If the total number of keys and values is an odd number, the JSON\_OBJECT function fills `NULL` in the last field. #### Examples[​](#examples "Direct link to Examples") Example 1: Construct a JSON object that consists of values of different data types. ```plaintext mysql> SELECT json_object('name', 'starrocks', 'active', true, 'published', 2020); -> {"active": true, "name": "starrocks", "published": 2020} ``` Example 2: Construct a JSON object by using nested JSON\_OBJECT functions. ```plaintext mysql> SELECT json_object('k1', 1, 'k2', json_object('k2', 2), 'k3', json_array(4, 5)); -> {"k1": 1, "k2": {"k2": 2}, "k3": [4, 5]} ``` Example 3: Construct an empty JSON object. ```plaintext mysql> SELECT json_object(); -> {} ``` --- ### parse_json Converts a string to a JSON value. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext parse_json(string_expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") `string_expr`: the expression that represents the string. Only the STRING, VARCHAR, and CHAR data types are supported. #### Return value[​](#return-value "Direct link to Return value") Returns a JSON value. > Note: If the string cannot be parsed into a standard JSON value, the PARSE\_JSON function returns `NULL` (see Example 5). For information about the JSON specifications, see [RFC 7159](https://tools.ietf.org/html/rfc7159?spm=a2c63.p38356.0.0.14d26b9fcp7fcf#page-4). #### Examples[​](#examples "Direct link to Examples") Example 1: Convert a STRING value of `1` to a JSON value of `1`. ```plaintext mysql> SELECT parse_json('1'); +-----------------+ | parse_json('1') | +-----------------+ | "1" | +-----------------+ ``` Example 2: Convert an array of the STRING data type to a JSON array. ```plaintext mysql> SELECT parse_json('[1,2,3]'); +-----------------------+ | parse_json('[1,2,3]') | +-----------------------+ | [1, 2, 3] | +-----------------------+ ``` Example 3: Convert an object of the STRING data type to a JSON object. ```plaintext mysql> SELECT parse_json('{"star": "rocks"}'); +---------------------------------+ | parse_json('{"star": "rocks"}') | +---------------------------------+ | {"star": "rocks"} | +---------------------------------+ ``` Example 4: Construct a JSON value of `NULL`. ```plaintext mysql> SELECT parse_json('null'); +--------------------+ | parse_json('null') | +--------------------+ | "null" | +--------------------+ ``` Example 5: If the string cannot be parsed into a standard JSON value, the PARSE\_JSON function returns `NULL`. In this example, `star` is not enclosed in double quotation marks ("). Therefore, the PARSE\_JSON function returns `NULL`. ```plaintext mysql> SELECT parse_json('{star: "rocks"}'); +-------------------------------+ | parse_json('{star: "rocks"}') | +-------------------------------+ | NULL | +-------------------------------+ ``` Example 6: If a JSON key contains a '.', for example, 'a.1', it must be escaped with '\\' or you need to enclose the entire key value along with double quotes in single quotes. ```plaintext mysql> select parse_json('{"b":4, "a.1": "1"}')->"a\\.1"; +--------------------------------------------+ | parse_json('{"b":4, "a.1": "1"}')->'a\\.1' | +--------------------------------------------+ | "1" | +--------------------------------------------+ mysql> select parse_json('{"b":4, "a.1": "1"}')->'"a.1"'; +--------------------------------------------+ | parse_json('{"b":4, "a.1": "1"}')->'"a.1"' | +--------------------------------------------+ | "1" | +--------------------------------------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") parse\_json, parse json --- ### JSON operators StarRocks supports the following JSON comparison operators: `<`, `<=`, `>`, `>=`, `=`, and `!=`. You can use these operators to query JSON data. However, StarRocks does not allow you to use `IN` to query JSON data. * > The operands of an operator must both be JSON values. * > If one operand of an operator is a JSON value while the other is not, the operand that is not a JSON value is converted to a JSON value during the arithmetic operation. For more information about the conversion rules, see [CAST](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/cast.md). tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) #### Arithmetic rules[​](#arithmetic-rules "Direct link to Arithmetic rules") JSON operators comply with the following arithmetic rules: * When the operands of an operator are JSON values of the same data type: * If both operands are JSONs values of a basic data type, such as NUMBER, STRING, or BOOLEAN, the operator performs the arithmetic operation in compliance with the arithmetic rules for the basic data type. > Note: If both operands are numbers but one is a DOUBLE value and the other is an INT value, the operator converts the INT value to a DOUBLE value. * If both operands are JSON values of a composite data type, such as OBJECT or ARRAY, the operator sorts the keys in the operands in dictionary order based on the sequence of the keys in the first operand and then compares the values of the keys between the operands. Example 1: The first operand is `{"a": 1, "c": 2}`, and the second operand is `{"b": 1, "a": 2}`. In this example, the operator compares the values of the keys `a` between the operands. The value of the key `a` in the first operand is `1`, whereas the value of the key `a` in the second operand is `2`. The value `1` is greater than the value `2`. Therefore, the operator concludes that the first operand `{"a": 1, "c": 2}` is less than the second operand `{"b": 1, "``a``": 2}`. ```plaintext mysql> SELECT PARSE_JSON('{"a": 1, "c": 2}') < PARSE_JSON('{"b": 1, "a": 2} '); -> 1 ``` Example 2: The first operand is `{"a": 1, "c": 2}`, and the second operand is `{"b": 1, "a": 1}`. In this example, the operator first compares the values of the keys `a` between the operands. The values of the keys `a` in the operands are both `1`. Then, the operator compares the values of the keys `c` in the operands. The second operand does not contain the key `c`. Therefore, the operator concludes that the first operand `{"a": 1, "c": 2}` is greater than the second operand `{"b": 1, "a": 1}`. ```plaintext mysql> SELECT PARSE_JSON('{"a": 1, "c": 2}') < PARSE_JSON('{"b": 1, "a": 1}'); -> 0 ``` * When the operands of an operator are JSON values of two distinct data types, the operator compares the operands in compliance with the following arithmetic rules: NULL < BOOLEAN < ARRAY < OBJECT < DOUBLE < INT < STRING. ```plaintext mysql> SELECT PARSE_JSON('"a"') < PARSE_JSON('{"a": 1, "c": 2}'); -> 0 ``` --- ### JSON query and processing #### [📄️ Arrow function](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/arrow-function.md) [Queries an element that can be located by the json\_path expression in a JSON object and returns a JSON value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/arrow-function.md) #### [📄️ cast](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/cast.md) [Converts a value between the JSON type and SQL type.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/cast.md) #### [📄️ get\_json\_bool](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/get_json_bool.md) [Parses and gets the boolean value from a specified JSON path in a JSON string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/get_json_bool.md) #### [📄️ get\_json\_double](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/get_json_double.md) [Extracts a floating-point value from a JSON string at the specified path.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/get_json_double.md) #### [📄️ get\_json\_int](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/get_json_int.md) [Extracts an integer value from a JSON string at the specified path.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/get_json_int.md) #### [📄️ get\_json\_string,get\_json\_object](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/get_json_string.md) [Analyzes and gets strings from the specified path (json\_path) in a JSON string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/get_json_string.md) #### [📄️ json\_contains](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_contains.md) [Checks whether a JSON document contains a specific value or subdocument.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_contains.md) #### [📄️ json\_each](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_each.md) [Expands the outermost elements of a JSON object into a set of key-value pairs held in two columns and returns a table that consists of one row for each element.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_each.md) #### [📄️ json\_exists](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_exists.md) [Checks whether a JSON object contains an element that can be located by the json\_path expression.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_exists.md) #### [📄️ json\_keys](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_keys.md) [Returns the top-level keys from a JSON object as a JSON array, or, if a path is specified, the top-level keys from the path.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_keys.md) #### [📄️ json\_length](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_length.md) [Returns the length of a JSON document.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_length.md) #### [📄️ json\_pretty](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_pretty.md) [Formats a JSON document into an easy-to-read, indented string format.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_pretty.md) #### [📄️ json\_query](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_query.md) [Queries the value of an element that can be located by the json\_path expression in a JSON object and returns a JSON value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_query.md) #### [📄️ json\_remove](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_remove.md) [Removes data from a JSON document at one or more specified JSON paths and returns the modified JSON document.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_remove.md) #### [📄️ json\_set](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_set.md) [Inserts or updates data in a JSON document at one or more specified JSON paths and returns the modified JSON document.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_set.md) #### [📄️ json\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_string.md) [Converting JSON object to JSON string](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_string.md) #### [📄️ to\_json](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/to_json.md) [Converts a Map or Struct value into a JSON string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/to_json.md) --- ### Arrow function Queries an element that can be located by the `json_path` expression in a JSON object and returns a JSON value. The arrow function `->` is more compact and easier to use than the [json\_query](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_query.md) function. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext json_object_expr -> json_path ``` #### Parameters[​](#parameters "Direct link to Parameters") * `json_object_expr`: the expression that represents the JSON object. The object can be a JSON column, or a JSON object that is produced by a JSON constructor function such as PARSE\_JSON. * `json_path`: the expression that represents the path to an element in the JSON object. The value of this parameter is a string. For information about the JSON path syntax that is supported by StarRocks, see [Overview of JSON functions and operators](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md). #### Return value[​](#return-value "Direct link to Return value") Returns a JSON value. > If the element does not exist, the arrow function returns an SQL value of `NULL`. #### Examples[​](#examples "Direct link to Examples") Example 1: Query an element that can be located by the `'$.a.b'` expression in the specified JSON object. ```plaintext mysql> SELECT parse_json('{"a": {"b": 1}}') -> '$.a.b'; -> 1 ``` Example 2: Use nested arrow functions to query an element. The arrow function in which another arrow function is nested queries an element based on the result that is returned by the nested arrow function. > In this example, the root element $ is omitted from the `json_path` expression. ```plaintext mysql> SELECT parse_json('{"a": {"b": 1}}')->'a'->'b'; -> 1 ``` Example 3: Query an element that can be located by the `'a'` expression in the specified JSON object. > In this example, the root element $ is omitted from the `json_path` expression. ```plaintext mysql> SELECT parse_json('{"a": "b"}') -> 'a'; -> "b" ``` --- ### cast Converts a value between the JSON type and SQL type. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) #### Syntax[​](#syntax "Direct link to Syntax") * Conversion from JSON to SQL ```plaintext cast(json_expr AS sql_data_type) ``` * Conversion from SQL to JSON ```plaintext cast(sql_expr AS JSON) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `json_expr`: the expression that represents the JSON value you want to convert to an SQL value. * `sql_data_type`: the SQL data type to which you want to convert the JSON value. Only the STRING, VARCHAR, CHAR, BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, LARGEINT, DOUBLE, and FLOAT data types are supported. * `sql_expr`: the expression that represents the SQL value you want to convert to a JSON value. This parameter supports all SQL data types that are supported by the `sql_data_type` parameter. #### Return value[​](#return-value "Direct link to Return value") * If you use the `cast(json_expr AS sql_data_type)` syntax, the cast function returns a value of the SQL data type that is specified by the `sql_data_type` parameter. * If you use the `cast(sql_expr AS JSON)` syntax, the cast function returns a JSON value. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * Conversion from SQL to JSON * If the SQL value exceeds the precision that is supported by JSON, the cast function returns `NULL` to prevent an arithmetic overflow. * If the SQL value is `NULL`, the cast function does not convert the SQL value `NULL` to a JSON value of `NULL`. The return value is still an SQL value of `NULL`. * Conversion from JSON to SQL * The cast function supports only conversions between compatible JSON and SQL data types. For example, you can convert a JSON string to an SQL string. * The cast function does not support conversions between incompatible JSON and SQL data types. For example, if you convert a JSON number to an SQL string, the function returns `NULL`. * If an arithmetic overflow occurs, the cast function returns an SQL value of `NULL`. * If you convert a JSON value of `NULL` to an SQL value, the function returns an SQL value of `NULL`. * If you convert a JSON string to a VARCHAR value, the function returns a VARCHAR value that is not enclosed in double quotation marks ("). #### Examples[​](#examples "Direct link to Examples") Example 1: Convert a JSON value to an SQL value. ```plaintext -- Convert a JSON value to an INT value. mysql> select cast(parse_json('{"a": 1}') -> 'a' as int); +--------------------------------------------+ | CAST((parse_json('{"a": 1}')->'a') AS INT) | +--------------------------------------------+ | 1 | +--------------------------------------------+ -- Convert a JSON string to a VARCHAR value. mysql> select cast(parse_json('"star"') as varchar); +---------------------------------------+ | cast(parse_json('"star"') AS VARCHAR) | +---------------------------------------+ | star | +---------------------------------------+ -- Convert a JSON object to a VARCHAR value. mysql> select cast(parse_json('{"star": 1}') as varchar); +--------------------------------------------+ | cast(parse_json('{"star": 1}') AS VARCHAR) | +--------------------------------------------+ | {"star": 1} | +--------------------------------------------+ -- Convert a JSON array to a VARCHAR value. mysql> select cast(parse_json('[1,2,3]') as varchar); +----------------------------------------+ | cast(parse_json('[1,2,3]') AS VARCHAR) | +----------------------------------------+ | [1, 2, 3] | +----------------------------------------+ ``` Example 2: Convert an SQL value to a JSON value. ```plaintext -- Convert an INT value to a JSON value. mysql> select cast(1 as json); +-----------------+ | cast(1 AS JSON) | +-----------------+ | 1 | +-----------------+ -- Convert a VARCHAR value to a JSON value. mysql> select cast("star" as json); +----------------------+ | cast('star' AS JSON) | +----------------------+ | "star" | +----------------------+ -- Convert a BOOLEAN value to a JSON value. mysql> select cast(true as json); +--------------------+ | cast(TRUE AS JSON) | +--------------------+ | true | +--------------------+ ``` --- ### get_json_bool Parses and gets the boolean value from a specified JSON path in a JSON string. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) If the format of `json_str` or `json_path` is invalid, or if no matching content can be found, this function will return NULL. This function is supported from v3.3. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BOOLEAN get_json_bool(VARCHAR json_str, VARCHAR json_path) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `json_str`: the JSON string. The supported data type is VARCHAR. * `json_path`: the JSON path. The supported data type is VARCHAR. * `json_path` must start with `$` and use `.` as the path separator. If the path includes `.`, it can be enclosed in a pair of `"`. * `[ ]` is used as the array subscripts which starts from 0. #### Examples[​](#examples "Direct link to Examples") 1. Get the value whose key is "k1". The value is `true` and `1` is returned. ```plain MySQL > SELECT get_json_bool('{"k1":true, "k2":"false"}', "$.k1"); +----------------------------------------------------+ | get_json_bool('{"k1":true, "k2":"false"}', '$.k1') | +----------------------------------------------------+ | 1 | +----------------------------------------------------+ ``` 2. Get the second element in the array whose key is "my.key". The second element is `false` and `0` is returned. ```plain SELECT get_json_bool('{"k1":"v1", "my.key":[true, false, 3]}', '$."my.key"[1]'); +--------------------------------------------------------------------------+ | get_json_bool('{"k1":"v1", "my.key":[true, false, 3]}', '$."my.key"[1]') | +--------------------------------------------------------------------------+ | 0 | +--------------------------------------------------------------------------+ ``` 3. Get the first element in the array whose path is `k1.key -> k2`. The first element is `false` and `0` is returned. ```plain MYSQL > SELECT get_json_bool('{"k1.key":{"k2":[false, true]}}', '$."k1.key".k2[0]'); +----------------------------------------------------------------------+ | get_json_bool('{"k1.key":{"k2":[false, true]}}', '$."k1.key".k2[0]') | +----------------------------------------------------------------------+ | 0 | +----------------------------------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") GET\_JSON\_BOOL,GET,JSON,BOOL --- ### get_json_double This function analyzes and gets the floating point value from a specified path in json string. json\_path must start with `$` and use `.` as the path separator. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) If the path includes `.`, it could be circled by `"` and `"`. `[ ]` is used as the array subscripts which start from 0. Content in the path should not contain `"` , `[` and `]`. If the format of json\_string or json\_path is wrong, this function will return NULL. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DOUBLE get_json_double(VARCHAR json_str, VARCHAR json_path) ``` #### Examples[​](#examples "Direct link to Examples") 1. Get the value whose key is "k1" ```plain MySQL > SELECT get_json_double('{"k1":1.3, "k2":"2"}', "$.k1"); +-------------------------------------------------+ | get_json_double('{"k1":1.3, "k2":"2"}', '$.k1') | +-------------------------------------------------+ | 1.3 | +-------------------------------------------------+ ``` 2. Get the second element in the array whose key is "my.key" ```plain MySQL > SELECT get_json_double('{"k1":"v1", "my.key":[1.1, 2.2, 3.3]}', '$."my.key"[1]'); +---------------------------------------------------------------------------+ | get_json_double('{"k1":"v1", "my.key":[1.1, 2.2, 3.3]}', '$."my.key"[1]') | +---------------------------------------------------------------------------+ | 2.2 | +---------------------------------------------------------------------------+ ``` 3. Get the first element in the array whose path is k1.key -> k2. ```plain MySQL > SELECT get_json_double('{"k1.key":{"k2":[1.1, 2.2]}}', '$."k1.key".k2[0]'); +---------------------------------------------------------------------+ | get_json_double('{"k1.key":{"k2":[1.1, 2.2]}}', '$."k1.key".k2[0]') | +---------------------------------------------------------------------+ | 1.1 | +---------------------------------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") GET\_JSON\_DOUBLE,GET,JSON,DOUBLE --- ### get_json_int This function analyzes and gets the integer value from a specified path in json string. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) json\_path must start with `$` and use `.` as the path separator. If the path includes `.`, it could be circled by `"` and `"`. `[ ]` is used as the array subscripts which start from 0. Content in the path should not contain `"` , `[` and `]`. If the format of json\_string or json\_path is wrong, this function will return NULL. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT get_json_int(VARCHAR json_str, VARCHAR json_path) ``` #### Examples[​](#examples "Direct link to Examples") 1. Get the value whose key is "k1" ```plain MySQL > SELECT get_json_int('{"k1":1, "k2":"2"}', "$.k1"); +--------------------------------------------+ | get_json_int('{"k1":1, "k2":"2"}', '$.k1') | +--------------------------------------------+ | 1 | +--------------------------------------------+ ``` 2. Get the second element in the array whose key is "my.key" ```plain MySQL > SELECT get_json_int('{"k1":"v1", "my.key":[1, 2, 3]}', '$."my.key"[1]'); +------------------------------------------------------------------+ | get_json_int('{"k1":"v1", "my.key":[1, 2, 3]}', '$."my.key"[1]') | +------------------------------------------------------------------+ | 2 | +------------------------------------------------------------------+ ``` 3. Get the first element in the array whose path is k1.key -> k2. ```plain MySQL > SELECT get_json_int('{"k1.key":{"k2":[1, 2]}}', '$."k1.key".k2[0]'); +--------------------------------------------------------------+ | get_json_int('{"k1.key":{"k2":[1, 2]}}', '$."k1.key".k2[0]') | +--------------------------------------------------------------+ | 1 | +--------------------------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") GET\_JSON\_INT,GET,JSON,INT --- ### get_json_string,get_json_object Analyzes and gets strings from the specified path (`json_path`) in a JSON string. If the format of `json_string` or `json_path` is wrong or if no matching value is found, this function will return NULL. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) The alias is get\_json\_object. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR get_json_string(VARCHAR json_str, VARCHAR json_path) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `json_str`: the JSON string. Supported data type is VARCHAR. * `json_path`: the JSON path. Supported data type is VARCHAR. `json_path` starts with `$` and uses `.` as the path separator. `[ ]` is used as the array subscripts which starts from 0. For example, `$."my.key"[1]` indicates to obtain the second value form element `my.key`. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. If no matching object is found, NULL is returned. #### Examples[​](#examples "Direct link to Examples") ##### Example 1: Get the value whose key is `k1`[​](#example-1-get-the-value-whose-key-is-k1 "Direct link to example-1-get-the-value-whose-key-is-k1") ```sql SELECT get_json_string('{"k1":"v1", "k2":"v2"}', "$.k1"); ``` ```sql +---------------------------------------------------+ | get_json_string('{"k1":"v1", "k2":"v2"}', '$.k1') | +---------------------------------------------------+ | v1 | +---------------------------------------------------+ ``` ##### Example 2: Get the value whose key is `a` from the first element[​](#example-2-get-the-value-whose-key-is-a-from-the-first-element "Direct link to example-2-get-the-value-whose-key-is-a-from-the-first-element") ```sql SELECT get_json_object('[{"a":"123", "b": "456"},{"a":"23", "b": "56"}]', '$[0].a'); ``` ```sql +------------------------------------------------------------------------------+ | get_json_object('[{"a":"123", "b": "456"},{"a":"23", "b": "56"}]', '$[0].a') | +------------------------------------------------------------------------------+ | 123 | +------------------------------------------------------------------------------+ ``` ##### Example 3: Get the second element in the array whose key is `my.key`[​](#example-3-get-the-second-element-in-the-array-whose-key-is-mykey "Direct link to example-3-get-the-second-element-in-the-array-whose-key-is-mykey") ```sql SELECT get_json_string('{"k1":"v1", "my.key":["e1", "e2", "e3"]}', '$."my.key"[1]'); ``` ```sql +------------------------------------------------------------------------------+ | get_json_string('{"k1":"v1", "my.key":["e1", "e2", "e3"]}', '$."my.key"[1]') | +------------------------------------------------------------------------------+ | e2 | +------------------------------------------------------------------------------+ ``` ##### Example 4: Get the first element in the array whose path is `k1.key -> k2`[​](#example-4-get-the-first-element-in-the-array-whose-path-is-k1key---k2 "Direct link to example-4-get-the-first-element-in-the-array-whose-path-is-k1key---k2") ```sql SELECT get_json_string('{"k1.key":{"k2":["v1", "v2"]}}', '$."k1.key".k2[0]'); ``` ```sql +-----------------------------------------------------------------------+ | get_json_string('{"k1.key":{"k2":["v1", "v2"]}}', '$."k1.key".k2[0]') | +-----------------------------------------------------------------------+ | v1 | +-----------------------------------------------------------------------+ ``` ##### Example 5: Get all values whose key is `k1` from the array[​](#example-5-get-all-values-whose-key-is-k1-from-the-array "Direct link to example-5-get-all-values-whose-key-is-k1-from-the-array") ```sql SELECT get_json_string('[{"k1":"v1"}, {"k2":"v2"}, {"k1":"v3"}, {"k1":"v4"}]', '$.[*].k1'); ``` ```sql +-------------------------------------------------------------------------------------+ | get_json_string('[{"k1":"v1"}, {"k2":"v2"}, {"k1":"v3"}, {"k1":"v4"}]', '$.[*].k1') | +-------------------------------------------------------------------------------------+ | ["v1", "v3", "v4"] | +-------------------------------------------------------------------------------------+ 1 row in set (0.01 sec) ``` #### keyword[​](#keyword "Direct link to keyword") GET\_JSON\_STRING,GET,JSON,STRING --- ### json_contains Checks whether a JSON document contains a specific value or subdocument. If the target JSON document contains the candidate JSON value, the JSON\_CONTAINS function returns `1`. Otherwise, the JSON\_CONTAINS function returns `0`. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext json_contains(json_target, json_candidate) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `json_target`: the expression that represents the target JSON document. The document can be a JSON column, or a JSON object that is produced by a JSON constructor function such as PARSE\_JSON. * `json_candidate`: the expression that represents the candidate JSON value or subdocument to search for within the target. The value can be a JSON column, or a JSON object that is produced by a JSON constructor function such as PARSE\_JSON. #### Return value[​](#return-value "Direct link to Return value") Returns a BOOLEAN value. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * For scalar values (strings, numbers, booleans, null), the function returns true if the values are equal. * For JSON objects, the function returns true if the target object contains all key-value pairs from the candidate object. * For JSON arrays, the function returns true if the target array contains all elements from the candidate array, or if the candidate is a single value contained in the target array. * The function performs deep containment checking for nested structures. #### Examples[​](#examples "Direct link to Examples") Example 1: Check if a JSON object contains a specific key-value pair. ```plaintext mysql> SELECT json_contains(PARSE_JSON('{"a": 1, "b": 2}'), PARSE_JSON('{"a": 1}')); -> 1 ``` Example 2: Check if a JSON object contains a key-value pair that doesn't exist. ```plaintext mysql> SELECT json_contains(PARSE_JSON('{"a": 1, "b": 2}'), PARSE_JSON('{"c": 3}')); -> 0 ``` Example 3: Check if a JSON array contains specific elements. ```plaintext mysql> SELECT json_contains(PARSE_JSON('[1, 2, 3, 4]'), PARSE_JSON('[2, 3]')); -> 1 ``` Example 4: Check if a JSON array contains a single scalar value. ```plaintext mysql> SELECT json_contains(PARSE_JSON('[1, 2, 3, 4]'), PARSE_JSON('2')); -> 1 ``` Example 5: Check if a JSON array contains elements that don't exist. ```plaintext mysql> SELECT json_contains(PARSE_JSON('[1, 2, 3, 4]'), PARSE_JSON('[5, 6]')); -> 0 ``` Example 6: Check containment with nested JSON structures. ```plaintext mysql> SELECT json_contains(PARSE_JSON('{"users": [{"id": 1, "name": "John"}, {"id": 2, "name": "Jane"}]}'), PARSE_JSON('{"users": [{"id": 1}]}')); -> 0 ``` Note: In the last example, the result is 0 because array containment requires exact element matching, not partial object matching within arrays. --- ### json_each Expands the outermost elements of a JSON object into a set of key-value pairs held in two columns and returns a table that consists of one row for each element. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext json_each(json_object_expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") `json_object_expr`: the expression that represents the JSON object. The object can be a JSON column, or a JSON object that is produced by a JSON constructor function such as PARSE\_JSON. #### Return value[​](#return-value "Direct link to Return value") Returns two columns: one named key and one named value. The key column stores VARCHAR values, and the value column stores JSON values. #### Usage notes[​](#usage-notes "Direct link to Usage notes") The json\_each function is a table function that returns a table. The returned table is a result set that consists of multiple rows. Therefore, a lateral join must be used in the FROM clause to join the returned table to the original table. The lateral join is mandatory, but the LATERAL keyword is optional. The json\_each function cannot be used in the SELECT clause. #### Examples[​](#examples "Direct link to Examples") ```plaintext -- A table named tj is used as an example. In the tj table, the j column is a JSON object. mysql> SELECT * FROM tj; +------+------------------+ | id | j | +------+------------------+ | 1 | {"a": 1, "b": 2} | | 3 | {"a": 3} | +------+------------------+ -- Expand the j column of the tj table into two columns by key and value to obtain a result set that consists of multiple rows. In this example, the LATERAL keyword is used to join the result set to the tj table. mysql> SELECT * FROM tj, LATERAL json_each(j); +------+------------------+------+-------+ | id | j | key | value | +------+------------------+------+-------+ | 1 | {"a": 1, "b": 2} | a | 1 | | 1 | {"a": 1, "b": 2} | b | 2 | | 3 | {"a": 3} | a | 3 | +------+------------------+------+-------+ ``` --- ### json_exists Checks whether a JSON object contains an element that can be located by the `json_path` expression. If the element exists, the JSON\_EXISTS function returns `1`. Otherwise, the JSON\_EXISTS function returns `0`. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext json_exists(json_object_expr, json_path) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `json_object_expr`: the expression that represents the JSON object. The object can be a JSON column, or a JSON object that is produced by a JSON constructor function such as PARSE\_JSON. * `json_path`: the expression that represents the path to an element in the JSON object. The value of this parameter is a string. For more information about the JSON path syntax that is supported by StarRocks, see [Overview of JSON functions and operators](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md). #### Return value[​](#return-value "Direct link to Return value") Returns a BOOLEAN value. #### Examples[​](#examples "Direct link to Examples") Example 1: Check whether the specified JSON object contains an element that can be located by the `'$.a.b'` expression. In this example, the element exists in the JSON object. Therefore, the json\_exists function returns `1`. ```plaintext mysql> SELECT json_exists(PARSE_JSON('{"a": {"b": 1}}'), '$.a.b') ; -> 1 ``` Example 2: Check whether the specified JSON object contains an element that can be located by the `'$.a.c'` expression. In this example, the element does not exist in the JSON object. Therefore, the json\_exists function returns `0`. ```plaintext mysql> SELECT json_exists(PARSE_JSON('{"a": {"b": 1}}'), '$.a.c') ; -> 0 ``` Example 3: Check whether the specified JSON object contains an element that can be located by the `'$.a[2]'` expression. In this example, the JSON object, which is an array named a, contains an element at index 2. Therefore, the json\_exists function returns `1`. ```plaintext mysql> SELECT json_exists(PARSE_JSON('{"a": [1,2,3]}'), '$.a[2]') ; -> 1 ``` Example 4: Check whether the specified JSON object contains an element that can be located by the `'$.a[3]'` expression. In this example, the JSON object, which is an array named a, does not contain an element at index 3. Therefore, the json\_exists function returns `0`. ```plaintext mysql> SELECT json_exists(PARSE_JSON('{"a": [1,2,3]}'), '$.a[3]') ; -> 0 ``` --- ### json_keys Returns the top-level keys from a JSON object as a JSON array, or, if a `path` is specified, the top-level keys from the path. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext json_keys(json_doc[, path]) ``` #### Parameters[​](#parameters "Direct link to Parameters") `json_doc`: required. The JSON document for which to return the keys. It must be a JSON object. `path`: optional. The path generally starts with `$` and uses `.` as the path separator. `[]` is used as the array subscript, which starts from 0. #### Return value[​](#return-value "Direct link to Return value") Returns a JSON array. An empty array is returned if the JSON object is empty. `NULL` is returned if the JSON document is not a JSON object or the path does not identify a value in the document. If the JSON document is an array nested with a JSON object, you can use the `path` parameter to obtain keys from that object. #### Examples[​](#examples "Direct link to Examples") Example 1: Return an empty array because the input JSON object is empty. ```plain select json_keys('{}'); +-----------------+ | json_keys('{}') | +-----------------+ | [] | +-----------------+ ``` Example 2: Return the keys of a JSON object. ```plain select json_keys('{"a": 1, "b": 2, "c": 3}'); +----------------+ | json_keys('1') | +----------------+ |["a", "b", "c"] | +----------------+ ``` Example 3: Return the keys of a JSON object that matches the specified path. ```plain select json_keys('{"a": 1, "b": 2, "c": {"d": 3, "e": 4, "f": 5}}', '$.c'); +---------------------------------------------------------------------+ | json_keys('{"a": 1, "b": 2, "c": {"d": 3, "e": 4, "f": 5}}', '$.c') | +---------------------------------------------------------------------+ | ["d", "e", "f"] | +---------------------------------------------------------------------+ ``` Example 4: The path does not exist. ```plain select json_keys('{"a": 1, "b": 2, "c": {"d": 3, "e": 4, "f": 5}}', '$.e'); +---------------------------------------------------------------------+ | json_keys('{"a": 1, "b": 2, "c": {"d": 3, "e": 4, "f": 5}}', '$.e') | +---------------------------------------------------------------------+ | NULL | +---------------------------------------------------------------------+ ``` Example 5: The JSON document is not a JSON object. ```plain select json_keys('[1, 2, {"a": 1, "b": 2}]'); +---------------------------------------+ | json_keys('[1, 2, {"a": 1, "b": 2}]') | +---------------------------------------+ | NULL | +---------------------------------------+ ``` Example 6: The JSON document is an array nested with a JSON object. A path is specified to obtain keys from that object. ```plain select json_keys('[0, 1, {"a": 1, "b": 2}]', '$[2]'); +-----------------------------------------------+ | json_keys('[0, 1, {"a": 1, "b": 2}]', '$[2]') | +-----------------------------------------------+ | ["a", "b"] | +-----------------------------------------------+ ``` --- ### json_length Returns the length of a JSON document. If a path is specified, this function returns the length of the value identified by the path. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) The length of a document is determined according to the following rules: * The length of a scalar value is 1. For example, the length of `1`, `"a"`, `true`, `false`, and `null` is 1. * The length of an array is the number of array elements. For example, the length of `[1, 2]` is 2. * The length of an object is the number of object members. For example, the length of `{"a": 1}` is 1. * The length of nested arrays or objects does not count. For example, the length of `{"a": [1, 2]}` is 1, because the nested array `[1, 2]` is not calculated into length. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext json_length(json_doc[, path]) ``` #### Parameters[​](#parameters "Direct link to Parameters") `json_doc`: required, the JSON document for which to return the length. `path`: optional. It is used to return the length of a value within the document. The path generally starts with `$` and uses `.` as the path separator. `[]` is used as the array subscript, which starts from 0. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the INT type. An error is returned if the JSON document is not a valid document. 0 is returned in any of the following scenarios: * The **`path`** does not identify a value in the document. * The path is not a valid path expression. * The path contains the `*` or `**` wildcard. #### Examples[​](#examples "Direct link to Examples") Example 1: Return the length of a scalar value. ```plain select json_length('1'); +------------------+ | json_length('1') | +------------------+ | 1 | +------------------+ ``` Example 2: Return the length of an empty object. ```plain select json_length('{}'); +-------------------+ | json_length('{}') | +-------------------+ | 0 | +-------------------+ ``` Example 3: Return the length of an object that has data. ```plain select json_length('{"Name": "Homer"}'); +----------------------------------+ | json_length('{"Name": "Homer"}') | +----------------------------------+ | 1 | +----------------------------------+ ``` Example 4: Return the length of a JSON array. ```plain select json_length('[1, 2, 3]'); +--------------------------+ | json_length('[1, 2, 3]') | +--------------------------+ | 3 | +--------------------------+ ``` Example 5: Return the length of a JSON array in which one element has a nested array. The nested array `[3, 4]` is not calculated into length. ```plain select json_length('[1, 2, [3, 4]]'); +-------------------------------+ | json_length('[1, 2, [3, 4]]') | +-------------------------------+ | 3 | +-------------------------------+ ``` Example 6: Return the length of an object specified by path `$.Person`. ```sql SET @file = '{ "Person": { "Name": "Homer", "Age": 39, "Hobbies": ["Eating", "Sleeping"] } }'; select json_length(@file, '$.Person') 'Result'; ``` Example 7: Return the length of the value specified by path `$.y`. ```plain select json_length('{"x": 1, "y": [1, 2]}', '$.y'); +---------------------------------------------+ | json_length('{"x": 1, "y": [1, 2]}', '$.y') | +---------------------------------------------+ | 2 | +---------------------------------------------+ ``` --- ### json_pretty Formats a JSON document into an easy-to-read, indented string format. This function is useful for debugging or displaying JSON data in a human-readable structure. tip All of the JSON functions and operators are listed in the navigation and on the overview page. #### Syntax[​](#syntax "Direct link to Syntax") ```sql json_pretty(json_object_expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `json_object_expr`: The expression that represents the JSON object. The object can be a JSON column, a string containing valid JSON, or a JSON object produced by a JSON constructor function such as PARSE\_JSON. #### Return value[​](#return-value "Direct link to Return value") Returns the formatted JSON document as a string. > * Returns NULL if the argument is NULL. > * The returned string includes newlines and spaces for indentation. > * Object keys are sorted alphabetically in the output. #### Examples[​](#examples "Direct link to Examples") Example 1: Format a simple JSON object. ```plaintext mysql> SELECT json_pretty('{"b": 2, "a": 1}'); -> { "a": 1, "b": 2 } ``` Example 2: Format a JSON array. ```plaintext mysql> SELECT json_pretty('[1, 2, 3]'); -> [ 1, 2, 3 ] ``` Example 3: Format a nested JSON structure. ```plaintext mysql> SELECT json_pretty('{"level1": {"level2": {"level3": "value"}}}'); -> { "level1": { "level2": { "level3": "value" } } } ``` Example 4: Use with a table column containing JSON data. ```plaintext mysql> CREATE TABLE json_test (id INT, data JSON); mysql> INSERT INTO json_test VALUES (1, parse_json('{"name": "Alice", "details": {"age": 25, "city": "NYC"}}')); mysql> SELECT json_pretty(data) FROM json_test; -> { "details": { "age": 25, "city": "NYC" }, "name": "Alice" } ``` #### Usage notes[​](#usage-notes "Direct link to Usage notes") * **Indentation:** The function adds standard indentation (spaces) and newlines to make the JSON structure visual. * **Key Sorting:** JSON object keys are sorted alphabetically in the output string. This is standard behavior for the underlying JSON processing library (VelocyPack). * **Null Handling:** If the input is SQL NULL, the function returns NULL. * **Data Types:** It supports formatting of standard JSON types including Objects, Arrays, Strings, Numbers, Booleans, and Nulls. --- ### json_query Queries the value of an element that can be located by the `json_path` expression in a JSON object and returns a JSON value. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext json_query(json_object_expr, json_path) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `json_object_expr`: the expression that represents the JSON object. The object can be a JSON column, or a JSON object that is produced by a JSON constructor function such as PARSE\_JSON. * `json_path`: the expression that represents the path to an element in the JSON object. The value of this parameter is a string. For information about the JSON path syntax that is supported by StarRocks, see [Overview of JSON functions and operators](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md). #### Return value[​](#return-value "Direct link to Return value") Returns a JSON value. > If the element does not exist, the json\_query function returns an SQL value of `NULL`. #### Examples[​](#examples "Direct link to Examples") Example 1: Query the value of an element that can be located by the `'$.a.b'` expression in the specified JSON object. In this example, the json\_query function returns a JSON value of `1`. ```plaintext mysql> SELECT json_query(PARSE_JSON('{"a": {"b": 1}}'), '$.a.b') ; -> 1 ``` Example 2: Query the value of an element that can be located by the `'$.a.c'` expression in the specified JSON object. In this example, the element does not exist. Therefore, the json\_query function returns an SQL value of `NULL`. ```plaintext mysql> SELECT json_query(PARSE_JSON('{"a": {"b": 1}}'), '$.a.c') ; -> NULL ``` Example 3: Query the value of an element that can be located by the `'$.a[2]'` expression in the specified JSON object. In this example, the JSON object, which is an array named a, contains an element at index 2, and the value of the element is 3. Therefore, the JSON\_QUERY function returns a JSON value of `3`. ```plaintext mysql> SELECT json_query(PARSE_JSON('{"a": [1,2,3]}'), '$.a[2]') ; -> 3 ``` Example 4: Query an element that can be located by the `'$.a[3]'` expression in the specified JSON object. In this example, the JSON object, which is an array named a, does not contain an element at index 3. Therefore, the json\_query function returns an SQL value of `NULL`. ```plaintext mysql> SELECT json_query(PARSE_JSON('{"a": [1,2,3]}'), '$.a[3]') ; -> NULL ``` --- ### json_remove Removes data from a JSON document at one or more specified JSON paths and returns the modified JSON document. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext json_remove(json_object_expr, json_path[, json_path] ...) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `json_object_expr`: the expression that represents the JSON object. The object can be a JSON column, or a JSON object that is produced by a JSON constructor function such as PARSE\_JSON. * `json_path`: one or more expressions that represent the paths to elements in the JSON object that should be removed. The value of each parameter is a string. For information about the JSON path syntax that is supported by StarRocks, see [Overview of JSON functions and operators](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md). #### Return value[​](#return-value "Direct link to Return value") Returns a JSON document with the specified paths removed. > * If a path does not exist in the JSON document, it is ignored. > * If an invalid path is provided, it is ignored. > * If all paths are invalid or non-existent, the original JSON document is returned unchanged. #### Examples[​](#examples "Direct link to Examples") Example 1: Remove a single key from a JSON object. ```plaintext mysql> SELECT json_remove('{"a": 1, "b": [10, 20, 30]}', '$.a'); -> {"b": [10, 20, 30]} ``` Example 2: Remove multiple keys from a JSON object. ```plaintext mysql> SELECT json_remove('{"a": 1, "b": [10, 20, 30], "c": "test"}', '$.a', '$.c'); -> {"b": [10, 20, 30]} ``` Example 3: Remove array elements from a JSON object. ```plaintext mysql> SELECT json_remove('{"a": 1, "b": [10, 20, 30]}', '$.b[1]'); -> {"a": 1, "b": [10, 30]} ``` Example 4: Remove nested object properties. ```plaintext mysql> SELECT json_remove('{"a": {"x": 1, "y": 2}, "b": 3}', '$.a.x'); -> {"a": {"y": 2}, "b": 3} ``` Example 5: Attempt to remove non-existent paths (ignored). ```plaintext mysql> SELECT json_remove('{"a": 1, "b": 2}', '$.c', '$.d'); -> {"a": 1, "b": 2} ``` Example 6: Remove multiple paths including non-existent ones. ```plaintext mysql> SELECT json_remove('{"a": 1, "b": 2, "c": 3}', '$.a', '$.nonexistent', '$.c'); -> {"b": 2} ``` #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The `json_remove` function follows MySQL-compatible behavior. * Invalid JSON paths are silently ignored rather than causing errors. * The function supports removing multiple paths in a single operation, which is more efficient than multiple separate operations. * Currently, the function supports simple object key removal (e.g., `$.key`). Support for complex nested paths and array element removal may be limited in the current implementation. --- ### json_set Inserts or updates data in a JSON document at one or more specified JSON paths and returns the modified JSON document. tip All of the JSON functions and operators are listed in the navigation and on the overview page. #### Syntax[​](#syntax "Direct link to Syntax") ```sql json_set(json_object_expr, json_path, value[, json_path, value] ...) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `json_object_expr`: the expression that represents the JSON object. The object can be a JSON column, or a JSON object that is produced by a JSON constructor function such as **PARSE\_JSON**. * `json_path`: the path to the element in the JSON object that you want to insert or update. The value must be a string. For information about the JSON path syntax that is supported by StarRocks, see Overview of JSON functions and operators. * `value`: the value to be inserted or updated at the specified path. It can be a string, number, boolean, null, or a JSON object. #### Return value[​](#return-value "Direct link to Return value") Returns the modified JSON document. > * Returns *NULL* if any argument is *NULL*. > * If the path exists in the JSON document, the existing value is updated (replaced). > * If the path does not exist, the new value is inserted (Upsert behavior). > * Arguments are evaluated from left to right. The result of the first path-value pair becomes the input for the second pair. #### Examples[​](#examples "Direct link to Examples") Example 1: Insert a new key into a JSON object. ```plaintext mysql> SELECT json_set('{"a": 1}', '$.b', 2); -> {"a": 1, "b": 2} ``` Example 2: Update an existing key in a JSON object. ```plaintext mysql> SELECT json_set('{"a": 1}', '$.a', 10); -> {"a": 10} ``` Example 3: Perform multiple operations (Update one existing key, Insert one new key). ```plaintext mysql> SELECT json_set('{"a": 1, "b": 2}', '$.a', 10, '$.c', 3); -> {"a": 10, "b": 2, "c": 3} ``` Example 4: Update a value inside a nested JSON object. ```plaintext mysql> SELECT json_set('{"a": {"x": 1, "y": 2}}', '$.a.x', 100); -> {"a": {"x": 100, "y": 2}} ``` Example 5: Update an element in an array by index. ```plaintext mysql> SELECT json_set('{"arr": [10, 20, 30]}', '$.arr[1]', 99); -> {"arr": [10, 99, 30]} ``` Example 6: Append to an array (using an index larger than the array length). ```plaintext mysql> SELECT json_set('{"arr": [10, 20]}', '$.arr[5]', 30); -> {"arr": [10, 20, 30]} ``` Example 7: Insert different data types (Boolean and JSON Null). To insert a JSON `null` value, use `parse_json('null')`. Passing a raw SQL `NULL` will return `NULL` for the entire result. ```plaintext mysql> SELECT json_set('{"a": 1}', '$.b', true, '$.c', parse_json('null')); -> {"a": 1, "b": true, "c": null} ``` #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The `json_set` function follows MySQL-compatible behavior. * It operates as an **Upsert** (Update or Insert): * **INSERT:** If the path does not exist, the value is added to the document. * **UPDATE:** If the path already exists, the old value is replaced with the new value. * If you specifically want to insert *only* (without updating existing values), use `json_insert`. * If you specifically want to update *only* (without inserting new values), use `json_replace`. * **Null Handling:** To insert a JSON null value, use parse\_json('null'). Passing a raw SQL NULL as an argument will cause the function to return NULL. * **Note:** Wildcard characters (e.g., `*` or `**`) and array slices (e.g., `[1:3]`) are not currently supported in `json_path` for modification. If a path contains these, the update for that path will be ignored to ensure safety. --- ### json_string Converting JSON object to JSON string tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) #### Syntax[​](#syntax "Direct link to Syntax") ```sql json_string(json_object_expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `json_object_expr`: the expression that represents the JSON object. The object can be a JSON column, or a JSON object that is produced by a JSON constructor function such as PARSE\_JSON. #### Return value[​](#return-value "Direct link to Return value") Returns a VARCHAR value. #### Examples[​](#examples "Direct link to Examples") Example 1: Converting the JSON object to a JSON string ```plain select json_string('{"Name": "Alice"}'); +----------------------------------+ | json_string('{"Name": "Alice"}') | +----------------------------------+ | {"Name": "Alice"} | +----------------------------------+ ``` Example 1: Convert the result of PARSE\_JSON to a JSON string ```plain select json_string(parse_json('{"Name": "Alice"}')); +----------------------------------+ | json_string('{"Name": "Alice"}') | +----------------------------------+ | {"Name": "Alice"} | +----------------------------------+ ``` --- ### to_json Converts a Map or Struct value into a JSON string. If the input value is NULL, NULL is returned. tip All of the JSON functions and operators are listed in the navigation and on the [overview page](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md) Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) If you want to cast values of other data types, see [cast](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/cast.md). This function is supported from v3.1 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext to_json(any_value) ``` #### Parameters[​](#parameters "Direct link to Parameters") `any_value`: the Map or Struct expression you want to convert. If the input value is invalid, an error is returned. The value in each key-value pair of the Map or Struct value is nullable. See the last example. #### Return value[​](#return-value "Direct link to Return value") Returns a JSON value. #### Examples[​](#examples "Direct link to Examples") ```plaintext select to_json(map{1:'a',2:'b'}); +---------------------------+ | to_json(map{1:'a',2:'b'}) | +---------------------------+ | {"1": "a", "2": "b"} | +---------------------------+ select to_json(row('asia','eu')); +--------------------------------+ | to_json(row('asia', 'eu')) | +--------------------------------+ | {"col1": "asia", "col2": "eu"} | +--------------------------------+ select to_json(map('a', named_struct('b', 1))); +----------------------------------------+ | to_json(map{'a':named_struct('b', 1)}) | +----------------------------------------+ | {"a": {"b": 1}} | +----------------------------------------+ select to_json(named_struct("k1", cast(null as string), "k2", "v2")); +-----------------------------------------------------------------------+ | to_json(named_struct('k1', CAST(NULL AS VARCHAR(65533)), 'k2', 'v2')) | +-----------------------------------------------------------------------+ | {"k1": null, "k2": "v2"} | +-----------------------------------------------------------------------+ ``` #### See also[​](#see-also "Direct link to See also") * [Map data type](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/Map.md) * [Struct data type](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/STRUCT.md) * [Map functions](https://docs.starrocks.io/docs/sql-reference/sql-functions#map-functions) * [Struct functions](https://docs.starrocks.io/docs/sql-reference/sql-functions#struct-functions) --- ### Overview of JSON functions and operators This topic provides an overview of the JSON constructor functions, query functions, and processing functions, operators, and path expressions that are supported by StarRocks. tip Accelerate your queries with [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) #### JSON constructor functions[​](#json-constructor-functions "Direct link to JSON constructor functions") JSON constructor functions are used to construct JSON data, such as JSON objects and JSON arrays. | Function | Description | Example | Return value | | ----------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | ---------------------------------------- | | [json\_object](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-constructor-functions/json_object.md) | Converts one or more key-value pairs to a JSON object that consists of the key-value pairs, which are sorted by key in dictionary order. | `SELECT JSON_OBJECT('Daniel Smith', 26, 'Lily Smith', 25);` | `{"Daniel Smith": 26, "Lily Smith": 25}` | | [json\_array](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-constructor-functions/json_array.md) | Converts each element of an SQL array to a JSON value and returns a JSON array that consists of those JSON values. | `SELECT JSON_ARRAY(1, 2, 3);` | `[1,2,3]` | | [parse\_json](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-constructor-functions/parse_json.md) | Converts a string to a JSON value. | `SELECT PARSE_JSON('{"a": 1}');` | `{"a": 1}` | #### JSON query functions and processing functions[​](#json-query-functions-and-processing-functions "Direct link to JSON query functions and processing functions") JSON query functions and processing functions are used to query and process JSON data. For example, you can use a path expression to locate an element in a JSON object. | Function | Description | Example | Return value | | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | [arrow function](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/arrow-function.md) | Queries the element that can be located by a path expression in a JSON object. | `SELECT parse_json('{"a": {"b": 1}}') -> '$.a.b';` | `1` | | [cast](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/cast.md) | Converts data between a JSON data type and an SQL data type. | `SELECT CAST(1 AS JSON);` | `1` | | [get\_json\_double](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/get_json_double.md) | Analyzes and gets the floating point value from a specified path in a JSON string. | `SELECT get_json_double('{"k1":1.3, "k2":"2"}', "$.k1");` | `1.3` | | [get\_json\_int](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/get_json_int.md) | Analyzes and gets the integer value from a specified path in a JSON string. | `SELECT get_json_int('{"k1":1, "k2":"2"}', "$.k1");` | `1` | | [get\_json\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/get_json_string.md) | Analyzes and gets the strings from a specified path in a JSON string. | `SELECT get_json_string('{"k1":"v1", "k2":"v2"}', "$.k1");` | `v1` | | [json\_query](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_query.md) | Queries the value of an element that can be located by a path expression in a JSON object. | `SELECT JSON_QUERY('{"a": 1}', '$.a');` | `1` | | [json\_remove](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_remove.md) | Removes data from a JSON document at one or more specified JSON paths. | `SELECT JSON_REMOVE('{"a": 1, "b": [10, 20, 30]}', '$.a', '$.b[1]');` | `{"b": [10, 30]}` | | [json\_each](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_each.md) | Expands the top-level elements of a JSON object into key-value pairs. | `SELECT * FROM tj_test, LATERAL JSON_EACH(j);` | `!`[json\_each](https://docs.starrocks.io/assets/files/json_each-acfb20b2e8c25e0f1889a081f532f10f.png) | | [json\_exists](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_exists.md) | Checks whether a JSON object contains an element that can be located by a path expression. If the element exists, this function returns 1. If the element does not exist, the function returns 0. | `SELECT JSON_EXISTS('{"a": 1}', '$.a');` | `1` | | [json\_keys](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_keys.md) | Returns the top-level keys from a JSON object as a JSON array, or, if a path is specified, the top-level keys from the path. | `SELECT JSON_KEYS('{"a": 1, "b": 2, "c": 3}');` | `["a", "b", "c"]` | | [json\_length](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_length.md) | Returns the length of a JSON document. | `SELECT json_length('{"Name": "Alice"}');` | `1` | | [json\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_string.md) | Converts the JSON object to a JSON string | `SELECT json_string(parse_json('{"Name": "Alice"}'));` | `{"Name": "Alice"}` | #### JSON operators[​](#json-operators "Direct link to JSON operators") StarRocks supports the following JSON comparison operators: `<`, `<=`, `>`, `>=`, `=`, and `!=`. You can use these operators to query JSON data. However, it does not allow you to use `IN` to query JSON data. For more information about JSON operators, see [JSON operators](https://docs.starrocks.io/docs/sql-reference/sql-functions/json-functions/json-operators.md). #### JSON path expressions[​](#json-path-expressions "Direct link to JSON path expressions") You can use a JSON path expression to query an element in a JSON object. JSON path expressions are of the STRING data type. In most cases, they are used with various JSON functions, such as JSON\_QUERY. In StarRocks, JSON path expressions do not completely comply with the [SQL/JSON path specifications](https://modern-sql.com/blog/2017-06/whats-new-in-sql-2016#json-path). For information about the JSON path syntax that is supported in StarRocks, see the following table, in which the following JSON object is used as an example. ```json { "people": [{ "name": "Daniel", "surname": "Smith" }, { "name": "Lily", "surname": "Smith", "active": true }] } ``` | JSON path symbol | Description | JSON path example | Return value | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------------ | | `$` | Denotes a root JSON object. | `'$'` | `{ "people": [ { "name": "Daniel", "surname": "Smith" }, { "name": "Lily", "surname": Smith, "active": true } ] }` | | `.` | Denotes a child JSON object. | `' $.people'` | `[ { "name": "Daniel", "surname": "Smith" }, { "name": "Lily", "surname": Smith, "active": true } ]` | | `[]` | Denotes one or more array indexes. `[n]` denotes the nth element in an array. Indexes start from 0.
StarRocks 2.5 supports querying multi-dimensional arrays, for example, `["Lucy", "Daniel"], ["James", "Smith"]`. To query the "Lucy" element, you can use `$.people[0][0]`. | `'$.people [0]'` | `{ "name": "Daniel", "surname": "Smith" }` | | `[*]` | Denotes all elements in an array. | `'$.people[*].name'` | `["Daniel", "Lily"]` | | `[start: end]` | Denotes a subset of elements from an array. The subset is specified by the `[start, end]` interval, which excludes the element that is denoted by the end index. | `'$.people[0: 1].name'` | `["Daniel"]` | --- ### Lambda expression Lambda expressions are anonymous functions that can be passed as parameters into higher-order SQL functions. Lambda expressions allow you to develop code that is more concise, elegant, and extensible. Lambda expressions are written with the `->` operator, which reads "goes to". The left side of `->` is the input parameters (if any) and the right side is an expression. From v2.5 onwards, StarRocks supports using lambda expressions in the following higher-order SQL functions: [array\_map()](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_map.md), [array\_filter()](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_filter.md), [array\_sum()](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_sum.md), and [array\_sortby()](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_sortby.md). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext parameter -> expression ``` #### Parameters[​](#parameters "Direct link to Parameters") * `parameter`: the input parameters for the lambda expression, which can accept zero, one, or more parameters. Two or more input parameters are enclosed in parentheses. * `expression`: a simple expression that references `parameter`. The expression must be valid for the input parameters. #### Return value[​](#return-value "Direct link to Return value") The type of the return value is determined by the result type of `expression`. #### Usage notes[​](#usage-notes "Direct link to Usage notes") Almost all scalar functions can be used in a lambda body. But there are a few exceptions: * Subqueries are not supported, for example, `x -> 5 + (SELECT 3)`. * Aggregate functions are not supported, for example, `x -> min(y)`. * Window functions are not supported. * Table functions are not supported. * Correlated columns cannot occur in lambda functions. #### Examples[​](#examples "Direct link to Examples") Simple examples of lambda expressions: ```sql -- Accepts no parameters and returns 5. () -> 5 -- Takes x and returns the value of (x + 2). x -> x + 2 -- Takes x and y, and returns their sum. (x, y) -> x + y -- Takes x and applies a function to x. x -> COALESCE(x, 0) x -> day(x) x -> split(x,",") x -> if(x>0,"positive","negative") ``` An example of using lambda expressions in higher-order functions: ```plaintext select array_map((x,y,z) -> x + y, [1], [2], [4]); +----------------------------------------------+ | array_map((x, y, z) -> x + y, [1], [2], [4]) | +----------------------------------------------+ | [3] | +----------------------------------------------+ 1 row in set (0.01 sec) ``` --- ### Pattern matching #### [📄️ like](https://docs.starrocks.io/docs/sql-reference/sql-functions/like-predicate-functions/like.md) [Checks whether a given expression fuzzy matches the specified pattern.](https://docs.starrocks.io/docs/sql-reference/sql-functions/like-predicate-functions/like.md) #### [📄️ regexp](https://docs.starrocks.io/docs/sql-reference/sql-functions/like-predicate-functions/regexp.md) [Checks whether a given expression matches the regular expression specified by pattern.](https://docs.starrocks.io/docs/sql-reference/sql-functions/like-predicate-functions/regexp.md) #### [📄️ regexp\_count](https://docs.starrocks.io/docs/sql-reference/sql-functions/like-predicate-functions/regexp_count.md) [Counts the number of times a pattern occurs in a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/like-predicate-functions/regexp_count.md) #### [📄️ regexp\_extract](https://docs.starrocks.io/docs/sql-reference/sql-functions/like-predicate-functions/regexp_extract.md) [Returns the first substring in a string that matches the specified regular expression pattern.](https://docs.starrocks.io/docs/sql-reference/sql-functions/like-predicate-functions/regexp_extract.md) #### [📄️ regexp\_extract\_all](https://docs.starrocks.io/docs/sql-reference/sql-functions/like-predicate-functions/regexp_extract_all.md) [Extracts all substrings from a string that match a regular expression pattern at the specified capture group index.](https://docs.starrocks.io/docs/sql-reference/sql-functions/like-predicate-functions/regexp_extract_all.md) #### [📄️ regexp\_replace](https://docs.starrocks.io/docs/sql-reference/sql-functions/like-predicate-functions/regexp_replace.md) [Replaces substrings matching a regular expression pattern with a replacement string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/like-predicate-functions/regexp_replace.md) --- ### like Checks whether a given expression fuzzy matches the specified pattern. If yes, 1 is returned. Otherwise, 0 is returned. NULL is returned if any of the input parameter is NULL. LIKE is usually used together with characters such as the percent sign (%) and underscore (\_). `%` matches 0, 1, or more characters. `_` matches any single character. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BOOLEAN like(VARCHAR expr, VARCHAR pattern); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr`: the string expression. The supported data type is VARCHAR. * `pattern`: the pattern to match. The supported data type is VARCHAR. #### Return value[​](#return-value "Direct link to Return value") Returns a BOOLEAN value. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select like("star","star"); +----------------------+ | like('star', 'star') | +----------------------+ | 1 | +----------------------+ mysql> select like("starrocks","star%"); +----------------------+ | like('star', 'star') | +----------------------+ | 1 | +----------------------+ mysql> select like("starrocks","star_"); +----------------------------+ | like('starrocks', 'star_') | +----------------------------+ | 0 | +----------------------------+ ``` --- ### regexp Checks whether a given expression matches the regular expression specified by `pattern`. If yes, 1 is returned. Otherwise, 0 is returned. NULL is returned if any of the input parameter is NULL. regexp() supports more complex matching conditions than [like()](https://docs.starrocks.io/docs/sql-reference/sql-functions/like-predicate-functions/like.md). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BOOLEAN regexp(VARCHAR expr, VARCHAR pattern); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr`: the string expression. The supported data type is VARCHAR. * `pattern`: the pattern to match. The supported data type is VARCHAR. #### Return value[​](#return-value "Direct link to Return value") Returns a BOOLEAN value. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select regexp("abc123","abc*"); +--------------------------+ | regexp('abc123', 'abc*') | +--------------------------+ | 1 | +--------------------------+ 1 row in set (0.06 sec) select regexp("abc123","xyz*"); +--------------------------+ | regexp('abc123', 'xyz*') | +--------------------------+ | 0 | +--------------------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") regexp, regular --- ### regexp_count Counts the number of times a pattern occurs in a string. It returns the number of occurrences of the regular expression pattern in the target string. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT regexp_count(VARCHAR str, VARCHAR pattern) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str`: The string to search in. Supported data type is VARCHAR. If the input is NULL, NULL is returned. * `pattern`: The regular expression pattern to search for. Supported data type is VARCHAR. If the pattern is NULL, NULL is returned. #### Return value[​](#return-value "Direct link to Return value") Returns an INT representing the count of occurrences. Returns 0 if no matches are found or if the input string is empty. #### Examples[​](#examples "Direct link to Examples") tip This example uses the Local Climatological Data(LCD) dataset featured in the [StarRocks Basics](https://docs.starrocks.io/docs/quick_start/shared-nothing.md) Quick Start. You can load the data and try the example yourself. ```sql -- Count occurrences of digits SELECT regexp_count('abc123def456', '[0-9]'); ``` ```plaintext +---------------------------------------+ | regexp_count('abc123def456', '[0-9]') | +---------------------------------------+ | 6 | +---------------------------------------+ ``` ```sql -- Count occurrences of dots SELECT regexp_count('test.com test.net test.org', '\\.'); ``` ```plaintext +---------------------------------------------------+ | regexp_count('test.com test.net test.org', '\\.') | +---------------------------------------------------+ | 3 | +---------------------------------------------------+ ``` ```sql -- Count occurrences of whitespace sequences SELECT regexp_count('a b c d', '\\s+'); ``` ```plaintext +----------------------------------------+ | regexp_count('a b c d', '\\s+') | +----------------------------------------+ | 3 | +----------------------------------------+ ``` ```sql -- Count occurrences of a repeated pattern SELECT regexp_count('ababababab', 'ab'); ``` ```plaintext +------------------------------------+ | regexp_count('ababababab', 'ab') | +------------------------------------+ | 5 | +------------------------------------+ ``` ```sql -- Using with NULL and empty values SELECT regexp_count('', '.') AS empty_str, regexp_count(NULL, '.') AS null_str, regexp_count('abc', NULL) AS null_pattern; ``` ```plaintext +------------+----------+--------------+ | empty_str | null_str | null_pattern | +------------+----------+--------------+ | 0 | NULL | NULL | +------------+----------+--------------+ ``` ```sql -- Count occurrence of Unicode/Chinese characters SELECT regexp_count('abc中文def', '[\\p{Han}]+'); ``` ```plaintext +-----------------------------------------------+ | regexp_count('abc中文def', '[\\p{Han}]+') | +-----------------------------------------------+ | 1 | +-----------------------------------------------+ ``` ##### With table data[​](#with-table-data "Direct link to With table data") ```sql CREATE TABLE sample_text ( str VARCHAR(65533) NULL, regex VARCHAR(65533) NULL ); INSERT INTO sample_text VALUES ('abc123def456', '[0-9]'), ('test.com test.net test.org', '\\.'), ('a b c d', '\\s+'), ('ababababab', 'ab'); SELECT str, regex, regexp_count(str, regex) AS count FROM sample_text ORDER BY str; ``` ```plaintext +-----------------------------+------+-------+ | str | regex| count | +-----------------------------+------+-------+ | a b c d | \s+ | 3 | | abc123def456 | [0-9]| 6 | | ababababab | ab | 5 | | test.com test.net test.org | \. | 3 | +-----------------------------+------+-------+ ``` #### keyword[​](#keyword "Direct link to keyword") REGEXP\_COUNT,REGEXP,COUNT --- ### regexp_extract This function returns the first matching substring in the target value which matches the regular expression pattern. It extracts the item in pos that matches the pattern. The pattern must completely match some parts of str so that the function can return parts needed to be matched in the pattern. If no matches are found, it will return an empty string. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR regexp_extract(VARCHAR str, VARCHAR pattern, int pos) ``` #### Examples[​](#examples "Direct link to Examples") tip This example uses the Local Climatological Data(LCD) dataset featured in the [StarRocks Basics](https://docs.starrocks.io/docs/quick_start/shared-nothing.md) Quick Start. You can load the data and try the example yourself. Given this data: ```sql SELECT HourlySkyConditions FROM quickstart.weatherdata WHERE HourlySkyConditions LIKE '%OVC%' LIMIT 10; ``` ```plaintext +---------------------+ | HourlySkyConditions | +---------------------+ | OVC:08 110 | | OVC:08 120 | | OVC:08 120 | | OVC:08 30 | | OVC:08 29 | | OVC:08 27 | | OVC:08 26 | | OVC:08 22 | | OVC:08 23 | | OVC:08 22 | +---------------------+ 10 rows in set (0.03 sec) ``` ##### Return the two sets of digits following the string `OVC: `[​](#return-the-two-sets-of-digits-following-the-string-ovc- "Direct link to return-the-two-sets-of-digits-following-the-string-ovc-") ```sql SELECT regexp_extract(HourlySkyConditions, 'OVC:(\\d+ \\d+)', 1) FROM quickstart.weatherdata WHERE HourlySkyConditions LIKE '%OVC%' LIMIT 10; ``` ```plaintext +-----------------------------------------------------------+ | regexp_extract(HourlySkyConditions, 'OVC:(\\d+ \\d+)', 1) | +-----------------------------------------------------------+ | 08 110 | | 08 120 | | 08 120 | | 08 30 | | 08 29 | | 08 27 | | 08 26 | | 08 22 | | 08 23 | | 08 22 | +-----------------------------------------------------------+ 10 rows in set (0.01 sec) ``` ##### Return only the second set of digits following the string `OVC: `[​](#return-only-the-second-set-of-digits-following-the-string-ovc- "Direct link to return-only-the-second-set-of-digits-following-the-string-ovc-") ```sql SELECT regexp_extract(HourlySkyConditions, 'OVC:(\\d+) (\\d+)', 2) FROM quickstart.weatherdata WHERE HourlySkyConditions LIKE '%OVC%' LIMIT 10; ``` ```plaintext +-------------------------------------------------------------+ | regexp_extract(HourlySkyConditions, 'OVC:(\\d+) (\\d+)', 2) | +-------------------------------------------------------------+ | 110 | | 120 | | 120 | | 30 | | 29 | | 27 | | 26 | | 22 | | 23 | | 22 | +-------------------------------------------------------------+ 10 rows in set (0.01 sec) ``` tip The same result could be returned by not wrapping the first set of digits in a matching group `()` and returning the first group: ```sql regexp_extract(HourlySkyConditions, 'OVC:\\d+ (\\d+)', 1) ``` #### keyword[​](#keyword "Direct link to keyword") REGEXP\_EXTRACT,REGEXP,EXTRACT --- ### regexp_extract_all Extracts all substrings from the target string (`str`) that matches a regular expression pattern (`pattern`) and corresponds to the regex group index specified by `pos`. This function returns an array. In regex, groups are enclosed within the parentheses () and numbered by counting their opening parentheses from left to right, starting from 1. For example, `([[:lower:]]+)C([[:lower:]]+)` is to match lowercase letters to the left or right side of the uppercase letter `C`. This pattern contains two groups: `([[:lower:]]+)` to the left of `C` is the first group and `([[:lower:]]+)` to the right of `C` is the second group. The pattern must completely match some parts of `str`. If no matches are found, an empty string is returned. This function is supported from v3.1.10. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext ARRAY regexp_extract_all(VARCHAR str, VARCHAR pattern, BIGINT pos) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str`: the string to be matched. * `pattern`: the regular expression pattern used to match substrings. * `pos`: `pattern` may contain multiple groups. `pos` indicates which regex group to extract. #### Return value[​](#return-value "Direct link to Return value") Returns an ARRAY that consists of VARCHAR elements. #### Examples[​](#examples "Direct link to Examples") ```plain -- Return all the letters that match group 1 in the pattern. MySQL > SELECT regexp_extract_all('AbCdE', '([[:lower:]]+)C([[:lower:]]+)', 1); +-------------------------------------------------------------------+ | regexp_extract_all('AbCdE', '([[:lower:]]+)C([[:lower:]]+)', 1) | +-------------------------------------------------------------------+ | ['b'] | +-------------------------------------------------------------------+ -- Return all the letters that match group 2 in the pattern. MySQL > SELECT regexp_extract_all('AbCdExCeF', '([[:lower:]]+)C([[:lower:]]+)', 2); +---------------------------------------------------------------------+ | regexp_extract_all('AbCdExCeF', '([[:lower:]]+)C([[:lower:]]+)', 2) | +---------------------------------------------------------------------+ | ['d','e'] | +---------------------------------------------------------------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") REGEXP\_EXTRACT\_ALL,REGEXP,EXTRACT --- ### regexp_replace This function uses repl to replace a sequence of characters in str that matches a regular expression pattern. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR regexp_replace(VARCHAR str, VARCHAR pattern, VARCHAR repl) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT regexp_replace('a b c', " ", "-"); +-----------------------------------+ | regexp_replace('a b c', ' ', '-') | +-----------------------------------+ | a-b-c | +-----------------------------------+ MySQL > SELECT regexp_replace('a b c','(b)','<\\1>'); +----------------------------------------+ | regexp_replace('a b c', '(b)', '<\1>') | +----------------------------------------+ | a c | +----------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") REGEXP\_REPLACE,REGEXP,REPLACE --- ### Map #### [📄️ cardinality](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/cardinality.md) [Returns the number of elements in a MAP value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/cardinality.md) #### [📄️ distinct\_map\_keys](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/distinct_map_keys.md) [Removes duplicate keys from a map, because keys in a map must be unique in terms of semantics.](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/distinct_map_keys.md) #### [📄️ element\_at](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/element_at.md) [Returns the value for the specified key from a key-value pair of a map.](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/element_at.md) #### [📄️ map\_apply](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_apply.md) [Applies a Lambda expression to the keys and values of the original Map and generates a new Map.](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_apply.md) #### [📄️ map\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_concat.md) [Returns the union of the input maps.](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_concat.md) #### [📄️ map\_filter](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_filter.md) [Filters key-value pairs in a map by applying a Boolean array or a Lambda expression to each key-value pair.](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_filter.md) #### [📄️ map\_from\_arrays](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_from_arrays.md) [Creates a MAP value from the given pair of key item array and value item array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_from_arrays.md) #### [📄️ map\_keys](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_keys.md) [Returns an array of all the keys in the specified map.](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_keys.md) #### [📄️ map\_size](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_size.md) [Returns the number of elements in a MAP value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_size.md) #### [📄️ map\_values](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_values.md) [Returns an array of all the values in the specified map.](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_values.md) #### [📄️ transform\_keys](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/transform_keys.md) [Transforms keys in a map using a Lambda expression and produces a new key for each entry in the map.](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/transform_keys.md) #### [📄️ transform\_values](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/transform_values.md) [Transforms values in a map using a Lambda expression and produces a new value for each entry in the map.](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/transform_values.md) --- ### cardinality Returns the number of elements in a MAP value. MAP is an unordered collection of key-value pairs, for example, `{"a":1, "b":2}`. One key-value pair constitutes one element. `{"a":1, "b":2}` contains two elements. This function is supported from v3.0 onwards. It is the alias of [map\_size()](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_size.md). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT cardinality(any_map) ``` #### Parameters[​](#parameters "Direct link to Parameters") `any_map`: the MAP value from which you want to retrieve the number of elements. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the INT value. If the input is NULL, NULL is returned. If a key or value in the MAP value is NULL, NULL is processed as a normal value. #### Examples[​](#examples "Direct link to Examples") ##### Query MAP data from a StarRocks native table[​](#query-map-data-from-a-starrocks-native-table "Direct link to Query MAP data from a StarRocks native table") From v3.1 onwards, StarRocks supports defining MAP columns when you create a table. This example uses table `test_map`, which contains the following data: ```plain CREATE TABLE test_map( col_int INT, col_map MAP ) DUPLICATE KEY(col_int); INSERT INTO test_map VALUES (1,map{"a":1,"b":2}), (2,map{"c":3}), (3,map{"d":4,"e":5}); SELECT * FROM test_map ORDER BY col_int; +---------+---------------+ | col_int | col_map | +---------+---------------+ | 1 | {"a":1,"b":2} | | 2 | {"c":3} | | 3 | {"d":4,"e":5} | +---------+---------------+ 3 rows in set (0.05 sec) ``` Obtain the number of elements in each row of the `col_map` column. ```plaintext select cardinality(col_map) from test_map order by col_int; +----------------------+ | cardinality(col_map) | +----------------------+ | 2 | | 1 | | 2 | +----------------------+ 3 rows in set (0.05 sec) ``` ##### Query MAP data from data lake[​](#query-map-data-from-data-lake "Direct link to Query MAP data from data lake") This example uses Hive table `hive_map`, which contains the following data: ```plaintext SELECT * FROM hive_map ORDER BY col_int; +---------+---------------+ | col_int | col_map | +---------+---------------+ | 1 | {"a":1,"b":2} | | 2 | {"c":3} | | 3 | {"d":4,"e":5} | +---------+---------------+ ``` After a [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md#create-a-hive-catalog) is created in your cluster, you can use this catalog and the cardinality() function to obtain the number of elements in each row of the `col_map` column. ```plaintext SELECT cardinality(col_map) FROM hive_map ORDER BY col_int; +----------------------+ | cardinality(col_map) | +----------------------+ | 2 | | 1 | | 2 | +----------------------+ 3 rows in set (0.05 sec) ``` --- ### distinct_map_keys Removes duplicate keys from a map, because keys in a map must be unique in terms of semantics. This function keeps only the last value for identical keys, called LAST WIN. This function is used when querying MAP data from external tables if there are duplicate keys in maps. StarRocks internal tables natively remove duplicate keys in maps. This function is supported from v3.1 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext distinct_map_keys(any_map) ``` #### Parameters[​](#parameters "Direct link to Parameters") `any_map`: the MAP value from which you want to remove duplicate keys. #### Return value[​](#return-value "Direct link to Return value") Returns a new map without duplicate keys in each map. If the input is NULL, NULL is returned. #### Examples[​](#examples "Direct link to Examples") Example 1: Simple usage. ```plain select distinct_map_keys(map{"a":1,"a":2}); +-------------------------------------+ | distinct_map_keys(map{'a':1,'a':2}) | +-------------------------------------+ | {"a":2} | +-------------------------------------+ ``` Example 2: Query MAP data from external tables and remove duplicate keys from the `col_map` column. ```plain select distinct_map_keys(col_map) as unique, col_map from external_table; +---------------+---------------+ | unique | col_map | +---------------+---------------+ | {"c":2} | {"c":1,"c":2} | | NULL| NULL | | {"e":4,"d":5} | {"e":4,"d":5} | +---------------+---------------+ 3 rows in set (0.05 sec) ``` --- ### element_at Returns the value for the specified key from a key-value pair of a map. If any input parameter is NULL or if the key does not exist in the map, the result is NULL. If you want to retrieve an element from an array, see [element\_at](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/element_at.md). This function is supported from v3.0 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext element_at(any_map, any_key) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `any_map`: a MAP expression from which to retrieve values. * `any_key`: a key in the map. #### Return value[​](#return-value "Direct link to Return value") If `any_key` exists in `any_map`, the value corresponding to the key will be returned. Otherwise, NULL is returned. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select element_at(map{1:3,2:4},1); +-----------------+ | map{1:3,2:4}[1] | +-----------------+ | 3 | +-----------------+ mysql> select element_at(map{1:3,2:4},3); +-----------------+ | map{1:3,2:4}[3] | +-----------------+ | NULL | +-----------------+ mysql> select element_at(map{'a':1,'b':2},'a'); +-----------------------+ | map{'a':1,'b':2}['a'] | +-----------------------+ | 1 | +-----------------------+ ``` #### keywords[​](#keywords "Direct link to keywords") ELEMENT\_AT, MAP --- ### map_apply Applies a [Lambda expression](https://docs.starrocks.io/docs/sql-reference/sql-functions/Lambda_expression.md) to the keys and values of the original Map and generates a new Map. This function is supported from v3.0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext MAP map_apply(lambda_func, any_map) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `lambda_func`: the Lambda expression. * `any_map`: the map to which the Lambda expression is applied. #### Return value[​](#return-value "Direct link to Return value") Returns a map value. The data types of keys and values in the result map are determined by the result of the Lambda expression. If any input parameter is NULL, NULL is returned. If a key or value in the original map is NULL, NULL is processed as a normal value. The Lambda expression must have two parameters. The first parameter represents the key. The second parameter represents the value. #### Examples[​](#examples "Direct link to Examples") The following example uses [map\_from\_arrays()](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_from_arrays.md) to generate a map value `{1:"ab",3:"cd"}`. Then the Lambda expression increments each key by 1 and calculates the length of each value. For example, the length of "ab" is 2. ```sql mysql> select map_apply((k,v)->(k+1,length(v)), col_map) from (select map_from_arrays([1,3],["ab","cd"]) as col_map)A; +--------------------------------------------------+ | map_apply((k, v) -> (k + 1, length(v)), col_map) | +--------------------------------------------------+ | {2:2,4:2} | +--------------------------------------------------+ 1 row in set (0.01 sec) mysql> select map_apply((k,v)->(k+1,length(v)), col_map) from (select map_from_arrays(null,null) as col_map)A; +--------------------------------------------------+ | map_apply((k, v) -> (k + 1, length(v)), col_map) | +--------------------------------------------------+ | NULL | +--------------------------------------------------+ 1 row in set (0.02 sec) mysql> select map_apply((k,v)->(k+1,length(v)), col_map) from (select map_from_arrays([1,null],["ab","cd"]) as col_map)A; +--------------------------------------------------+ | map_apply((k, v) -> (k + 1, length(v)), col_map) | +--------------------------------------------------+ | NULL | +--------------------------------------------------+ ``` --- ### map_concat Returns the union of the input maps. If a key is found in multiple maps, this function keeps only the last value among these maps, called LAST WIN. For example, `SELECT map_concat(map{1:3},map{1:'4'});` returns `{1:"4"}`. This function is supported from v3.1 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext ANY_MAP map_concat(any_map0, any_map1...) ``` #### Parameters[​](#parameters "Direct link to Parameters") `any_mapN`: the map values you want to union. All maps must share a common type. If data types of the input maps are not the same, the return type is the common supertype of the input maps. #### Return value[​](#return-value "Direct link to Return value") Returns a MAP of the common supertype of the input maps. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> SELECT map_concat(map('a',1, 'b',2), map('c',3)); +------------------------------------------+ | map_concat(map{'a':1,'b':2}, map{'c':3}) | +------------------------------------------+ | {"c":3,"a":1,"b":2} | +------------------------------------------+ mysql> select map_concat(map{1:3},map{'3.323':3}); +--------------------------------------+ | map_concat(map{1:3}, map{'3.323':3}) | +--------------------------------------+ | {"3.323":3,"1":3} | +--------------------------------------+ mysql> select map_concat(map{1:3},map{1:'4', 3:'5',null:null}, null); +--------------------------------------------------------+ | map_concat(map{1:3}, map{1:'4',3:'5',NULL:NULL}, NULL) | +--------------------------------------------------------+ | {null:null,1:"4",3:"5"} | +--------------------------------------------------------+ ``` --- ### map_filter Filters key-value pairs in a map by applying a Boolean array or a [Lambda expression](https://docs.starrocks.io/docs/sql-reference/sql-functions/Lambda_expression.md) to each key-value pair. The pair that evaluates to `true` is returned. This function is supported from v3.1 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext MAP map_filter(any_map, array) MAP map_filter(lambda_func, any_map) ``` * `map_filter(any_map, array)` Evaluates key-value pairs in `any_map` one by one against `array` and returns key-value pairs that evaluate to `true`. * `map_filter(lambda_func, any_map)` Applies `lambda_func` to the key-value pairs in `any_map` one by one and returns key-value pairs whose result is `true`. #### Parameters[​](#parameters "Direct link to Parameters") * `any_map`: the map value. * `array`: the Boolean array used to evaluate the map value. * `lambda_func`: the Lambda expression used to evaluate the map value. #### Return value[​](#return-value "Direct link to Return value") Returns a map whose data type is the same as `any_map`. If `any_map` is NULL, NULL is returned. If `array` is null, an empty map is returned. If a key or value in the map value is NULL, NULL is processed as a normal value. The Lambda expression must have two parameters. The first parameter represents the key. The second parameter represents the value. #### Examples[​](#examples "Direct link to Examples") ##### Use `array`[​](#use-arrayboolean "Direct link to use-arrayboolean") The following example uses [map\_from\_arrays()](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_from_arrays.md) to generate a map value `{1:"ab",3:"cdd",2:null,null:"abc"}`. Then each key-value pair is evaluated against `array` and the pair whose result is `true` is returned. ```sql mysql> select map_filter(col_map, array[0,0,0,1,1]) from (select map_from_arrays([1,3,null,2,null],['ab','cdd',null,null,'abc']) as col_map)A; +----------------------------------------------------+ | map_filter(col_map, ARRAY[0, 0, 0, 1, 1]) | +----------------------------------------------------+ | {null:"abc"} | +----------------------------------------------------+ 1 row in set (0.02 sec) mysql> select map_filter(null, array[0,0,0,1,1]); +-------------------------------------------------+ | map_filter(NULL, ARRAY[0, 0, 0, 1, 1]) | +-------------------------------------------------+ | NULL | +-------------------------------------------------+ 1 row in set (0.02 sec) mysql> select map_filter(col_map, null) from (select map_from_arrays([1,3,null,2,null],['ab','cdd',null,null,'abc']) as col_map)A; +---------------------------+ | map_filter(col_map, NULL) | +---------------------------+ | {} | +---------------------------+ 1 row in set (0.01 sec) ``` ##### Use Lambda expression[​](#use-lambda-expression "Direct link to Use Lambda expression") The following example uses map\_from\_arrays() to generate a map value `{1:"ab",3:"cdd",2:null,null:"abc"}`. Then each key-value pair is evaluated against the Lambda expression and the key-value pair whose value is not null is returned. ```sql mysql> select map_filter((k,v) -> v is not null,col_map) from (select map_from_arrays([1,3,null,2,null],['ab','cdd',null,null,'abc']) as col_map)A; +------------------------------------------------+ | map_filter((k,v) -> v is not null, col_map) | +------------------------------------------------+ | {1:"ab",3:"cdd",null:'abc'} | +------------------------------------------------+ 1 row in set (0.02 sec) ``` --- ### map_from_arrays Creates a MAP value from the given pair of key item array and value item array. This function is supported from v3.1 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext MAP map_from_arrays(ARRAY keys, ARRAY values) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `keys`: uses to construct the keys of the result MAP. Make sure the elements of keys are unique. * `values`: used to construct the values of the result MAP. #### Return value[​](#return-value "Direct link to Return value") Returns a MAP that is constructed from the input keys and values. * Keys and values must have the same length, otherwise it will return an error. * If key or value is NULL, this function returns NULL. * The returned MAP value has distinct keys. #### Examples[​](#examples "Direct link to Examples") ```plaintext select map_from_arrays([1, 2], ['Star', 'Rocks']); +--------------------------------------------+ | map_from_arrays([1, 2], ['Star', 'Rocks']) | +--------------------------------------------+ | {1:"Star",2:"Rocks"} | +--------------------------------------------+ ``` ```plaintext select map_from_arrays([1, 2], NULL); +-------------------------------+ | map_from_arrays([1, 2], NULL) | +-------------------------------+ | NULL | +-------------------------------+ select map_from_arrays([1,3,null,2,null],['ab','cdd',null,null,'abc']); +--------------------------------------------------------------------------+ | map_from_arrays([1, 3, NULL, 2, NULL], ['ab', 'cdd', NULL, NULL, 'abc']) | +--------------------------------------------------------------------------+ | {1:"ab",3:"cdd",2:null,null:"abc"} | +--------------------------------------------------------------------------+ ``` --- ### map_keys Returns an array of all the keys in the specified map. This function is supported from v2.5. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext map_keys(any_map) ``` #### Parameters[​](#parameters "Direct link to Parameters") `any_map`: the MAP value from which you want to retrieve keys. #### Return value[​](#return-value "Direct link to Return value") The return value is in the format of `array`. The element type in the array matches the key type in the map. If the input is NULL, NULL is returned. If a key or value in the MAP value is NULL, NULL is processed as a normal value and contained in the result. #### Examples[​](#examples "Direct link to Examples") ##### Query MAP data from a StarRocks native table[​](#query-map-data-from-a-starrocks-native-table "Direct link to Query MAP data from a StarRocks native table") From v3.1 onwards, StarRocks supports defining MAP columns when you create a table. This example uses table `test_map`, which contains the following data: ```plain CREATE TABLE test_map( col_int INT, col_map MAP ) DUPLICATE KEY(col_int); INSERT INTO test_map VALUES (1,map{"a":1,"b":2}), (2,map{"c":3}), (3,map{"d":4,"e":5}); SELECT * FROM test_map ORDER BY col_int; +---------+---------------+ | col_int | col_map | +---------+---------------+ | 1 | {"a":1,"b":2} | | 2 | {"c":3} | | 3 | {"d":4,"e":5} | +---------+---------------+ 3 rows in set (0.05 sec) ``` Obtain all the keys from each row of the `col_map` column. ```plaintext select map_keys(col_map) from test_map order by col_int; +-------------------+ | map_keys(col_map) | +-------------------+ | ["a","b"] | | ["c"] | | ["d","e"] | +-------------------+ 3 rows in set (0.05 sec) ``` ##### Query MAP data from data lake[​](#query-map-data-from-data-lake "Direct link to Query MAP data from data lake") This example uses Hive table `hive_map`, which contains the following data: ```plaintext SELECT * FROM hive_map ORDER BY col_int; +---------+---------------+ | col_int | col_map | +---------+---------------+ | 1 | {"a":1,"b":2} | | 2 | {"c":3} | | 3 | {"d":4,"e":5} | +---------+---------------+ ``` After a [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md#create-a-hive-catalog) is created in your cluster, you can use this catalog and the map\_keys() function to obtain all the keys from each row of the `col_map` column. ```plaintext select map_keys(col_map) from hive_map order by col_int; +-------------------+ | map_keys(col_map) | +-------------------+ | ["a","b"] | | ["c"] | | ["d","e"] | +-------------------+ 3 rows in set (0.05 sec) ``` --- ### map_size Returns the number of elements in a MAP value. MAP is an unordered collection of key-value pairs, for example, `{"a":1, "b":2}`. One key-value pair constitutes one element, for example, `{"a":1, "b":2}` contains two elements. This function has an alias [cardinality()](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/cardinality.md). It is supported from v2.5. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT map_size(any_map) ``` #### Parameters[​](#parameters "Direct link to Parameters") `any_map`: the MAP value from which you want to retrieve the number of elements. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the INT type. If the input is NULL, NULL is returned. If a key or value in the MAP value is NULL, NULL is processed as a normal value. #### Examples[​](#examples "Direct link to Examples") ##### Query MAP data from a StarRocks native table[​](#query-map-data-from-a-starrocks-native-table "Direct link to Query MAP data from a StarRocks native table") From v3.1 onwards, StarRocks supports defining MAP columns when you create a table. This example uses table `test_map`, which contains the following data: ```plain CREATE TABLE test_map( col_int INT, col_map MAP ) DUPLICATE KEY(col_int); INSERT INTO test_map VALUES (1,map{"a":1,"b":2}), (2,map{"c":3}), (3,map{"d":4,"e":5}); SELECT * FROM test_map ORDER BY col_int; +---------+---------------+ | col_int | col_map | +---------+---------------+ | 1 | {"a":1,"b":2} | | 2 | {"c":3} | | 3 | {"d":4,"e":5} | +---------+---------------+ 3 rows in set (0.05 sec) ``` Obtain the number of elements in each row of the `col_map` column. ```plaintext select map_size(col_map) from test_map order by col_int; +-------------------+ | map_size(col_map) | +-------------------+ | 2 | | 1 | | 2 | +-------------------+ 3 rows in set (0.05 sec) ``` ##### Query MAP data from data lake[​](#query-map-data-from-data-lake "Direct link to Query MAP data from data lake") This example uses Hive table `hive_map`, which contains the following data: ```plaintext SELECT * FROM hive_map ORDER BY col_int; +---------+---------------+ | col_int | col_map | +---------+---------------+ | 1 | {"a":1,"b":2} | | 2 | {"c":3} | | 3 | {"d":4,"e":5} | +---------+---------------+ ``` After a [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md#create-a-hive-catalog) is created in your cluster, you can use this catalog and the map\_size() function to obtain the number of elements in each row of the `col_map` column. ```plaintext select map_size(col_map) from hive_map order by col_int; +-------------------+ | map_size(col_map) | +-------------------+ | 2 | | 1 | | 2 | +-------------------+ 3 rows in set (0.05 sec) ``` --- ### map_values Returns an array of all the values in the specified map. This function is supported from v2.5. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext map_values(any_map) ``` #### Parameters[​](#parameters "Direct link to Parameters") `any_map`: the MAP value from which you want to retrieve values. #### Return value[​](#return-value "Direct link to Return value") The return value is in the format of `array`. The element type in the array matches the value type in the map. If the input is NULL, NULL is returned. If a key or value in the MAP value is NULL, NULL is processed as a normal value and contained in the result. #### Examples[​](#examples "Direct link to Examples") ##### Query MAP data from a StarRocks native table[​](#query-map-data-from-a-starrocks-native-table "Direct link to Query MAP data from a StarRocks native table") From v3.1 onwards, StarRocks supports defining MAP columns when you create a table. This example uses table `test_map`, which contains the following data: ```plain CREATE TABLE test_map( col_int INT, col_map MAP ) DUPLICATE KEY(col_int); INSERT INTO test_map VALUES (1,map{"a":1,"b":2}), (2,map{"c":3}), (3,map{"d":4,"e":5}); SELECT * FROM test_map ORDER BY col_int; +---------+---------------+ | col_int | col_map | +---------+---------------+ | 1 | {"a":1,"b":2} | | 2 | {"c":3} | | 3 | {"d":4,"e":5} | +---------+---------------+ 3 rows in set (0.05 sec) ``` Obtain all the values from each row of the `col_map` column. ```sql select map_values(col_map) from test_map order by col_int; +---------------------+ | map_values(col_map) | +---------------------+ | [1,2] | | [3] | | [4,5] | +---------------------+ 3 rows in set (0.04 sec) ``` ##### Query MAP data from data lake[​](#query-map-data-from-data-lake "Direct link to Query MAP data from data lake") This example uses Hive table `hive_map`, which contains the following data: ```plaintext SELECT * FROM hive_map ORDER BY col_int; +---------+---------------+ | col_int | col_map | +---------+---------------+ | 1 | {"a":1,"b":2} | | 2 | {"c":3} | | 3 | {"d":4,"e":5} | +---------+---------------+ ``` After a [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md#create-a-hive-catalog) is created in your cluster, you can use this catalog and the map\_values() function to obtain all the values from each row of the `col_map` column. ```sql select map_values(col_map) from hive_map order by col_int; +---------------------+ | map_values(col_map) | +---------------------+ | [1,2] | | [3] | | [4,5] | +---------------------+ 3 rows in set (0.04 sec) ``` --- ### transform_keys Transforms keys in a map using a [Lambda expression](https://docs.starrocks.io/docs/sql-reference/sql-functions/Lambda_expression.md) and produces a new key for each entry in the map. This function is supported from v3.1 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext MAP transform_keys(lambda_func, any_map) ``` `lambda_func` can also be placed after `any_map`: ```plaintext MAP transform_keys(any_map, lambda_func) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `any_map`: the Map. * `lambda_func`: the Lambda expression you want to apply to `any_map`. #### Return value[​](#return-value "Direct link to Return value") Returns a map value where the data types of keys are determined by the result of the Lambda expression and the data types of values are the same as values in `any_map`. If any input parameter is NULL, NULL is returned. If a key or value in the original map is NULL, NULL is processed as a normal value. The Lambda expression must have two parameters. The first parameter represents the key. The second parameter represents the value. #### Examples[​](#examples "Direct link to Examples") The following example uses [map\_from\_arrays](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_from_arrays.md) to generate a map value `{1:"ab",3:"cdd",2:null,null:"abc"}`. Then the Lambda expression is applied to each key to increment the key by 1. ```sql mysql> select transform_keys((k,v)->(k+1), col_map) from (select map_from_arrays([1,3,null,2,null],['ab','cdd',null,null,'abc']) as col_map)A; +------------------------------------------+ | transform_keys((k, v) -> k + 1, col_map) | +------------------------------------------+ | {2:"ab",4:"cdd",3:null,null:"abc"} | +------------------------------------------+ ``` --- ### transform_values Transforms values in a map using a [Lambda expression](https://docs.starrocks.io/docs/sql-reference/sql-functions/Lambda_expression.md) and produces a new value for each entry in the map. This function is supported from v3.1 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext MAP transform_values(lambda_func, any_map) ``` `lambda_func` can also be placed after `any_map`: ```plaintext MAP transform_values(any_map, lambda_func) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `any_map`: the Map. * `lambda_func`: the Lambda expression you want to apply to `any_map`. #### Return value[​](#return-value "Direct link to Return value") Returns a map value where the data types of values are determined by the result of the Lambda expression and the data types of keys are the same as keys in `any_map`. If any input parameter is NULL, NULL is returned. If a key or value in the original map is NULL, NULL is processed as a normal value. The Lambda expression must have two parameters. The first parameter represents the key. The second parameter represents the value. #### Examples[​](#examples "Direct link to Examples") The following example uses [map\_from\_arrays](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_from_arrays.md) to generate a map value `{1:"ab",3:"cdd",2:null,null:"abc"}`. Then the Lambda expression is applied to each value of the map. The first example changes the value of each key-value pair to 1. The second example changes the value of each key-value pair to null. ```sql mysql> select transform_values((k,v)->1, col_map) from (select map_from_arrays([1,3,null,2,null],['ab','cdd',null,null,'abc']) as col_map)A; +----------------------------------------+ | transform_values((k, v) -> 1, col_map) | +----------------------------------------+ | {1:1,3:1,2:1,null:1} | +----------------------------------------+ 1 row in set (0.02 sec) mysql> select transform_values((k,v)->null, col_map) from (select map_from_arrays([1,3,null,2,null],['ab','cdd',null,null,'abc']) as col_map)A; +--------------------------------------------+ | transform_values((k, v) -> NULL, col_map) | +--------------------------------------------+ | {1:null,3:null,2:null,null:null} | +--------------------------------------------+ 1 row in set (0.01 sec) ``` --- ### Math #### [📄️ abs](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/abs.md) [Returns the absolute value of the numeric value x.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/abs.md) #### [📄️ acos](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/acos.md) [Computes the arc cosine of an argument.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/acos.md) #### [📄️ asin](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/asin.md) [Computes the arc sine of the argument.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/asin.md) #### [📄️ atan](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/atan.md) [Computes the arc tangent of the argument.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/atan.md) #### [📄️ atan2](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/atan2.md) [Returns the arc tangent of x divided by y, that is, the arc tangent of x/y.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/atan2.md) #### [📄️ bin](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/bin.md) [Transforms the input arg into a binary.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/bin.md) #### [📄️ cbrt](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cbrt.md) [Computes the cube root of the argument.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cbrt.md) #### [📄️ ceil, dceil](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/ceil.md) [Returns values from the input arg rounded to the nearest equal or larger integer.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/ceil.md) #### [📄️ ceiling](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/ceiling.md) [Returns values from the input arg rounded to the nearest equal or larger integer.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/ceiling.md) #### [📄️ conv](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/conv.md) [Converts the number x from one numeric base system to another, and returns the result as a string value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/conv.md) #### [📄️ cos](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cos.md) [Computes the cosine of the argument.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cos.md) #### [📄️ cosine\_similarity](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cos_similarity.md) [Measures the similarity of two vectors by calculating the cosine of the angle between them.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cos_similarity.md) #### [📄️ cosine\_similarity\_norm](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cos_similarity_norm.md) [Measures the similarity of two normalized vectors by calculating the cosine of the angle between them.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cos_similarity_norm.md) #### [📄️ cosh](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cosh.md) [Computes the hyperbolic cosine of the argument.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cosh.md) #### [📄️ cot](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cot.md) [Returns the cotangent of the angle in radians x.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cot.md) #### [📄️ degrees](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/degrees.md) [Converts the angle in radians x to degrees.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/degrees.md) #### [📄️ divide](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/divide.md) [Return the quotient of x divide y.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/divide.md) #### [📄️ e](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/e.md) [Returns the constant e (the base of natural logarithms).](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/e.md) #### [📄️ exp,dexp](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/exp.md) [Returns the value of e raised to the power of x.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/exp.md) #### [📄️ floor, dfloor](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/floor.md) [Returns the largest integer that is not more than x.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/floor.md) #### [📄️ fmod](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/fmod.md) [Returns the floating point remainder of the division ( dividend/divisor ).](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/fmod.md) #### [📄️ greatest](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/greatest.md) [Returns the largest value from a list of one or more parameters.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/greatest.md) #### [📄️ least](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/least.md) [Returns the smallest value from a list of one or more parameters.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/least.md) #### [📄️ ln, dlog1, log](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/ln.md) [Computes the natural (base e) logarithm of a number.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/ln.md) #### [📄️ log](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/log.md) [Calculates the logarithm of a number to a specified base (or radix).](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/log.md) #### [📄️ log10, dlog10](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/log10.md) [Calculates the base 10 logarithm of a number.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/log10.md) #### [📄️ log2](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/log2.md) [Calculates the base 2 logarithm of a number.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/log2.md) #### [📄️ mod](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/mod.md) [The modulus function that returns the remainder of dividend divided by divisor.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/mod.md) #### [📄️ multiply](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/multiply.md) [Computes the product of the arguments.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/multiply.md) #### [📄️ negative](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/negative.md) [Returns the negative of input arg.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/negative.md) #### [📄️ pi](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/pi.md) [Returns the value of Pi.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/pi.md) #### [📄️ pmod](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/pmod.md) [Returns the positive remainder of dividend divided bydivisor.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/pmod.md) #### [📄️ positive](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/positive.md) [Returns x as a value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/positive.md) #### [📄️ pow, power, dpow, fpow](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/pow.md) [Returns the result of x raised to the power of y.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/pow.md) #### [📄️ radians](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/radians.md) [Converts x from an angle to a radian.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/radians.md) #### [📄️ rand, random](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/rand.md) [Returns a random floating-point number between 0 (inclusive) and 1 (exclusive).](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/rand.md) #### [📄️ round, dround](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/round.md) [Rounds a number to a specified number of digits.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/round.md) #### [📄️ sign](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/sign.md) [Returns the sign of x.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/sign.md) #### [📄️ sin](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/sin.md) [Returns the sine of x, where x is in radians.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/sin.md) #### [📄️ sinh](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/sinh.md) [Computes the hyperbolic sine of the argument.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/sinh.md) #### [📄️ sqrt, dsqrt](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/sqrt.md) [Calculates the square root of a value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/sqrt.md) #### [📄️ square](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/square.md) [Calculates the square of a value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/square.md) #### [📄️ tan](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/tan.md) [Returns the tangent of x, where x is in radians.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/tan.md) #### [📄️ tanh](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/tanh.md) [Computes the hyperbolic tangent of the argument.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/tanh.md) #### [📄️ truncate](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/truncate.md) [Rounds the input down to the nearest equal or smaller value with the specified number of places after the decimal point.](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/truncate.md) --- ### abs Returns the absolute value of the numeric value `x`. If the input value is NULL, NULL is returned. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext ABS(x); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: the numeric value or expression. Supported data types: DOUBLE, FLOAT, LARGEINT, BIGINT, INT, SMALLINT, TINYINT, DECIMALV2, DECIMAL32, DECIMAL64, DECIMAL128. #### Return value[​](#return-value "Direct link to Return value") The data type of the return value is the same as the type of `x`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select abs(-1); +---------+ | abs(-1) | +---------+ | 1 | +---------+ 1 row in set (0.00 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") abs, absolute --- ### acos Computes the arc cosine of an argument. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DOUBLE acos(DOUBLE arg) ``` ##### Parameters[​](#parameters "Direct link to Parameters") `arg`: You can specify only a numeric value. This function converts the numeric value into a DOUBLE value before it computes the arc cosine of the value. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Usage notes[​](#usage-notes "Direct link to Usage notes") If you specify a non-numeric value, this function returns `NULL`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select acos(-1); +-------------------+ | acos(-1) | +-------------------+ | 3.141592653589793 | +-------------------+ mysql> select acos(0); +--------------------+ | acos(0) | +--------------------+ | 1.5707963267948966 | +--------------------+ mysql> select acos(1); +---------+ | acos(1) | +---------+ | 0 | +---------+ mysql> select acos(""); +----------+ | acos('') | +----------+ | NULL | +----------+ ``` #### keyword[​](#keyword "Direct link to keyword") ACOS --- ### asin Computes the arc sine of the argument. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DOUBLE asin(DOUBLE arg) ``` ##### Parameters[​](#parameters "Direct link to Parameters") `arg`: You can specify only a numeric value. This function converts the numeric value into a DOUBLE value before it computes the arc sine of the value. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Usage notes[​](#usage-notes "Direct link to Usage notes") If you specify a non-numeric value, this function returns `NULL`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select asin(1); +--------------------+ | asin(1) | +--------------------+ | 1.5707963267948966 | +--------------------+ mysql> select asin(-0.5); +---------------------+ | asin(-0.5) | +---------------------+ | -0.5235987755982989 | +---------------------+ mysql> select asin(0); +---------+ | asin(0) | +---------+ | 0 | +---------+ mysql> select asin(""); +----------+ | asin('') | +----------+ | NULL | +----------+ ``` #### keyword[​](#keyword "Direct link to keyword") ASIN --- ### atan Computes the arc tangent of the argument. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DOUBLE atan(DOUBLE arg) ``` ##### Parameters[​](#parameters "Direct link to Parameters") `arg`: You can specify only a numeric value. This function converts the numeric value into a DOUBLE value before it computes the arc tangent of the value. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Usage notes[​](#usage-notes "Direct link to Usage notes") If you specify a non-numeric value, this function returns `NULL`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select atan(1); +--------------------+ | atan(1) | +--------------------+ | 0.7853981633974483 | +--------------------+ mysql> select atan(0); +---------+ | atan(0) | +---------+ | 0 | +---------+ mysql> select atan(-1); +---------------------+ | atan(-1) | +---------------------+ | -0.7853981633974483 | +---------------------+ mysql> select atan(""); +----------+ | atan('') | +----------+ | NULL | +----------+ ``` #### keyword[​](#keyword "Direct link to keyword") ATAN --- ### atan2 Returns the arc tangent of `x` divided by `y`, that is, the arc tangent of `x/y`. The signs of the two parameters are used to determine the quadrant of the result. The return value is in the range of \[-π, π]. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext ATAN2(x,y); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: The supported data type is DOUBLE. `y`: The supported data type is DOUBLE. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE type. Returns NULL if `x` or `y` is NULL. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select atan2(-0.8,2); +---------------------+ | atan2(-0.8, 2) | +---------------------+ | -0.3805063771123649 | +---------------------+ 1 row in set (0.01 sec) ``` --- ### bin Transforms the input `arg` into a binary. #### Syntax[​](#syntax "Direct link to Syntax") ```shell bin(arg) ``` #### Parameter[​](#parameter "Direct link to Parameter") `arg`: the input you want to transform into a binary. It supports the BIGINT data type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR data type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select bin(3); +--------+ | bin(3) | +--------+ | 11 | +--------+ 1 row in set (0.02 sec) ``` --- ### cbrt Computes the cube root of the argument. This function is supported from v3.3 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DOUBLE cbrt(DOUBLE arg) ``` #### Parameters[​](#parameters "Direct link to Parameters") `arg`: You can specify only a numeric value. This function converts the numeric value into a DOUBLE value before it computes the cube root of the value. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. If you specify a non-numeric value, this function returns `NULL`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select cbrt(8); +---------+ | cbrt(8) | +---------+ | 2 | +---------+ mysql> select cbrt(-8); +----------+ | cbrt(-8) | +----------+ | -2 | +----------+ mysql> select cbrt(0); +---------+ | cbrt(0) | +---------+ | 0 | +---------+ mysql> select cbrt(""); +----------+ | cbrt('') | +----------+ | NULL | +----------+ ``` #### keywords[​](#keywords "Direct link to keywords") cbrt, cube root --- ### ceil, dceil Returns values from the input `arg` rounded to the nearest equal or larger integer. #### Syntax[​](#syntax "Direct link to Syntax") ```shell ceil(arg) ``` #### Parameter[​](#parameter "Direct link to Parameter") `arg` supports the DOUBLE data type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BIGINT data type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select ceil(3.14); +------------+ | ceil(3.14) | +------------+ | 4 | +------------+ 1 row in set (0.15 sec) ``` --- ### ceiling Returns values from the input `arg` rounded to the nearest equal or larger integer. #### Syntax[​](#syntax "Direct link to Syntax") ```shell ceiling(arg) ``` #### Parameter[​](#parameter "Direct link to Parameter") `arg` supports the DOUBLE data type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BIGINT data type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select ceiling(3.14); +---------------+ | ceiling(3.14) | +---------------+ | 4 | +---------------+ 1 row in set (0.00 sec) ``` --- ### conv Converts the number `x` from one numeric base system to another, and returns the result as a string value. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext CONV(x,y,z); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `x`: the number to be converted. VARCHAR or BIGINT is supported. * `y`: the source base. TINYINT is supported. * `z`: the target base. TINYINT is supported. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR data type. #### Examples[​](#examples "Direct link to Examples") Convert the number 8 from numeric base system 2 to numeric base system 10: ```plain mysql> select conv(8,10,2); +----------------+ | conv(8, 10, 2) | +----------------+ | 1000 | +----------------+ 1 row in set (0.00 sec) ``` --- ### cos Computes the cosine of the argument. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DOUBLE cos(DOUBLE arg) ``` ##### Parameters[​](#parameters "Direct link to Parameters") `arg`: You can specify only a numeric value. This function converts the numeric value into a DOUBLE value before it computes the cosine of the value. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Usage notes[​](#usage-notes "Direct link to Usage notes") If you specify a non-numeric value, this function returns `NULL`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select cos(-1); +--------------------+ | cos(-1) | +--------------------+ | 0.5403023058681398 | +--------------------+ mysql> select cos(0); +--------+ | cos(0) | +--------+ | 1 | +--------+ mysql> select cos(1); +--------------------+ | cos(1) | +--------------------+ | 0.5403023058681398 | +--------------------+ mysql> select cos(""); +---------+ | cos('') | +---------+ | NULL | +---------+ ``` #### keyword[​](#keyword "Direct link to keyword") COS --- ### cosine_similarity Measures the similarity of two vectors by calculating the cosine of the angle between them. The angle is formed by the direction of the vectors while the difference in their magnitude is ignored. The similarity is between -1 and 1. Smaller angles between vectors indicate greater cosine similarity. * If two vectors have the same direction, they have a 0-degree angle and the cosine similarity is 1. * Perpendicular vectors have a 90-degree angle and a cosine similarity of 0. * Opposite vectors have a 180-degree angle and a cosine similarity of -1. `cosine_similarity` is commonly used for assessing text and video similarity. This function normalizes vectors before measuring the cosine similarity. If the input vectors have been normalized, you can use [cosine\_similarity\_norm](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cos_similarity_norm.md). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext cosine_similarity(a, b) ``` #### Parameters[​](#parameters "Direct link to Parameters") `a` and `b` are the vectors to compare. They must have the same dimension. The supported data type is `Array`. The two arrays must have the same number of elements. Otherwise, an error is returned. #### Return value[​](#return-value "Direct link to Return value") Returns a FLOAT value within the range of \[-1, 1]. If any input parameter is null or invalid, an error is reported. #### Examples[​](#examples "Direct link to Examples") 1. Create a table to store vectors and insert data into this table. ```sql CREATE TABLE t1_similarity (id int, data array) DISTRIBUTED BY HASH(id); INSERT INTO t1_similarity VALUES (1, array[0.1, 0.2, 0.3]), (2, array[0.2, 0.1, 0.3]), (3, array[0.3, 0.2, 0.1]); ``` 2. Calculate the similarity of each row in the `data` column compared to the array `[0.1, 0.2, 0.3]` and list the result in descending order. ```plain SELECT id, data, cosine_similarity([0.1, 0.2, 0.3], data) as dist FROM t1_similarity ORDER BY dist DESC; +------+---------------+-----------+ | id | data | dist | +------+---------------+-----------+ | 1 | [0.1,0.2,0.3] | 0.9999999 | | 2 | [0.2,0.1,0.3] | 0.9285713 | | 3 | [0.3,0.2,0.1] | 0.7142856 | +------+---------------+-----------+ ``` --- ### cosine_similarity_norm Measures the similarity of two normalized vectors by calculating the cosine of the angle between them. The angle is formed by the direction of the vectors while the difference in their magnitude is ignored. This function assumes that the input vectors have been normalized. If you need to normalize vectors before calculating the cosine similarity, use [cosine\_similarity](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cos_similarity.md). The similarity is between -1 and 1. Smaller angles between vectors indicate greater cosine similarity. * If two vectors have the same direction, they have a 0-degree angle and the cosine similarity is 1. * Perpendicular vectors have a 90-degree angle and a cosine similarity of 0. * Opposite vectors have a 180-degree angle and a cosine similarity of -1. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext cosine_similarity_norm(a, b) ``` #### Parameters[​](#parameters "Direct link to Parameters") `a` and `b` are the vectors to compare. They must have the same dimension. The supported data type is `Array`. The two arrays must have the same number of elements. Otherwise, an error is returned. #### Return value[​](#return-value "Direct link to Return value") Returns a FLOAT value within the range of \[-1, 1]. If any input parameter is null or invalid, an error is reported. #### Examples[​](#examples "Direct link to Examples") 1. Create a table to store vectors and insert data into this table. ```sql CREATE TABLE t1_similarity (id int, data array) DISTRIBUTED BY HASH(id); INSERT INTO t1_similarity VALUES (1, array[0.1, 0.2, 0.3]), (2, array[0.2, 0.1, 0.3]), (3, array[0.3, 0.2, 0.1]); ``` 2. Calculate the similarity of each row in the `data` column compared to the array `[0.1, 0.2, 0.3]` and list the result in descending order. ```plain SELECT id, data, cosine_similarity_norm([0.1, 0.2, 0.3], data) as dist FROM t1_similarity ORDER BY dist DESC; +------+---------------+------------+ | id | data | dist | +------+---------------+------------+ | 1 | [0.1,0.2,0.3] | 0.14000002 | | 2 | [0.2,0.1,0.3] | 0.13000001 | | 3 | [0.3,0.2,0.1] | 0.10000001 | +------+---------------+------------+ ``` --- ### cosh Computes the hyperbolic cosine of the argument. This function is supported from v3.0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DOUBLE cosh(DOUBLE arg) ``` ##### Parameters[​](#parameters "Direct link to Parameters") `arg`: You can specify only a numeric value. This function converts the numeric value into a DOUBLE value before it computes the hyperbolic cosine of the value. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Usage notes[​](#usage-notes "Direct link to Usage notes") If you specify a non-numeric value, this function returns `NULL`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select cosh(-1); +--------------------+ | cosh(-1) | +--------------------+ | 1.5430806348152437 | +--------------------+ mysql> select cosh(0); +---------+ | cosh(0) | +---------+ | 1 | +---------+ mysql> select cosh(1); +--------------------+ | cosh(1) | +--------------------+ | 1.5430806348152437 | +--------------------+ mysql> select cosh(""); +----------+ | cosh('') | +----------+ | NULL | +----------+ ``` #### keyword[​](#keyword "Direct link to keyword") COSH --- ### cot Returns the cotangent of the angle in radians `x`. If the angle in radians `x` is `0` or `NULL`, `NULL` is returned. #### **Syntax**[​](#syntax "Direct link to syntax") ```sql COT(x); ``` #### **Parameters**[​](#parameters "Direct link to parameters") `x`: the angle in radians. DOUBLE is supported. #### **Return value**[​](#return-value "Direct link to return-value") Returns a value of the DOUBLE data type. #### **Examples**[​](#examples "Direct link to examples") ```plaintext mysql> select cot(3.1415926/2); +------------------------+ | cot(3.1415926 / 2) | +------------------------+ | 2.6794896585028646e-08 | +------------------------+ 1 row in set (0.25 sec) mysql> select cot(0); +--------+ | cot(0) | +--------+ | NULL | +--------+ 1 row in set (0.03 sec) ``` --- ### degrees Converts the angle in radians `x` to degrees. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DEGREES(x); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: the angle in radians. DOUBLE is supported. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Examples[​](#examples "Direct link to Examples") ```plaintext mysql> select degrees(3.1415926535898); +--------------------------+ | degrees(3.1415926535898) | +--------------------------+ | 180.0000000000004 | +--------------------------+ 1 row in set (0.07 sec) ``` --- ### divide Return the quotient of x divide y. If y is 0, return null. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext divide(x, y) ``` ##### Parameters[​](#parameters "Direct link to Parameters") * `x`: The supported types are DOUBLE, FLOAT, LARGEINT, BIGINT, INT, SMALLINT, TINYINT, DECIMALV2, DECIMAL32, DECIMAL64, DECIMAL128. * `y`: The supported types are the same as `x`. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Usage notes[​](#usage-notes "Direct link to Usage notes") If you specify a non-numeric value, this function returns `NULL`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select divide(3, 2); +--------------+ | divide(3, 2) | +--------------+ | 1.5 | +--------------+ 1 row in set (0.00 sec) ``` --- ### e Returns the constant e (the base of natural logarithms). #### Syntax[​](#syntax "Direct link to Syntax") ```sql E(); ``` #### Parameters[​](#parameters "Direct link to Parameters") This function takes no parameters. #### Return value[​](#return-value "Direct link to Return value") Returns the constant e as an DOUBLE. #### Examples[​](#examples "Direct link to Examples") ```plaintext mysql> select e(); +-------------------+ | e() | +-------------------+ | 2.718281828459045 | +-------------------+ 1 row in set (0.01 sec) ``` --- ### exp,dexp Returns the value of e raised to the power of `x`. This function is called natural logarithms function. #### Syntax[​](#syntax "Direct link to Syntax") ```sql EXP(x); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: the power number. DOUBLE is supported. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Examples[​](#examples "Direct link to Examples") Return e raised to the power of 3.14: ```plaintext mysql> select exp(3.14); +--------------------+ | exp(3.14) | +--------------------+ | 23.103866858722185 | +--------------------+ 1 row in set (0.01 sec) ``` --- ### floor, dfloor Returns the largest integer that is not more than `x`. #### Syntax[​](#syntax "Direct link to Syntax") ```sql FLOOR(x); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: DOUBLE is supported. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BIGINT data type. #### Examples[​](#examples "Direct link to Examples") ```plaintext mysql> select floor(3.14); +-------------+ | floor(3.14) | +-------------+ | 3 | +-------------+ 1 row in set (0.01 sec) ``` --- ### fmod Returns the floating point remainder of the division ( `dividend`/`divisor` ). It is a modulo function. #### Syntax[​](#syntax "Direct link to Syntax") ```sql fmod(dividend,devisor); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `dividend`: DOUBLE or FLOAT is supported. * `devisor`: DOUBLE or FLOAT is supported. > **Note** > > The data type of `devisor` needs to be the same as the data type of `dividend`. Otherwise, StarRocks performs implicit conversion to convert the data type. #### Return value[​](#return-value "Direct link to Return value") The data type and sign of output need to be the same as the data type and sign of `dividend`. If `divisor` is `0`, `NULL` is returned. #### Examples[​](#examples "Direct link to Examples") ```plaintext mysql> select fmod(3.14,3.14); +------------------+ | fmod(3.14, 3.14) | +------------------+ | 0 | +------------------+ mysql> select fmod(11.5,3); +---------------+ | fmod(11.5, 3) | +---------------+ | 2.5 | +---------------+ mysql> select fmod(3,6); +------------+ | fmod(3, 6) | +------------+ | 3 | +------------+ mysql> select fmod(3,0); +------------+ | fmod(3, 0) | +------------+ | NULL | +------------+ ``` --- ### greatest Returns the largest value from a list of one or more parameters. Generally, the return value has the same data type as the input. The parameters are compared based on the following rules: * NULL is returned if any of the input parameters is NULL. * If at least one parameter is of the DOUBLE type, all the parameters are compared as DOUBLE values. The same rule applies to the DECIMAL and FLOAT data types. * If the parameters are a mix of numbers and strings but the strings can be converted into numbers, the parameters are compared as numbers. If the strings cannot be converted into numbers, the parameters are compared as strings. * If the parameters are characters, they are compared based on the alphabetical order of the first letter. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext GREATEST(expr1,...); ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr1`: the expression to compare. It supports the following data types: * SMALLINT * TINYINT * INT * BIGINT * LARGEINT * FLOAT * DOUBLE * DECIMALV2 * DECIMAL32 * DECIMAL64 * DECIMAL128 * DATETIME * VARCHAR #### Examples[​](#examples "Direct link to Examples") Example 1: Return the largest value for a single input. ```plain mysql> select greatest(3); +-------------+ | greatest(3) | +-------------+ | 3 | +-------------+ 1 row in set (0.01 sec) ``` Example 2: Return the largest value from a list of values. ```plain mysql> select greatest(3,4,5,5,6); +-------------------------+ | greatest(3, 4, 5, 5, 6) | +-------------------------+ | 6 | +-------------------------+ 1 row in set (0.00 sec) ``` Example 3: One parameter is of the DOUBLE type and a DOUBLE value is returned. ```plain mysql> select greatest(7,4.5); +------------------+ | greatest(7, 4.5) | +------------------+ | 7.0 | +------------------+ 1 row in set (0.05 sec) ``` Example 4: The input parameters are a mix of number and string but the string can be converted into a number. The parameters are compared as numbers. ```plain mysql> select greatest(7,'9'); +------------------+ | greatest(7, '9') | +------------------+ | 9 | +------------------+ 1 row in set (0.04 sec) ``` Example 5: The input parameters are a mix of number and string but the string cannot be converted into a number. The parameters are compared as strings. The string `'1'` is less than `'at'`. ```plain select greatest(1,'at'); +-------------------+ | greatest(1, 'at') | +-------------------+ | at | +-------------------+ ``` Example 6: The input parameters are characters. The letter `Z` has the largest value. ```plain mysql> select greatest('A','B','Z'); +-------------------------+ | greatest('A', 'B', 'Z') | +-------------------------+ | Z | +-------------------------+ 1 row in set (0.00 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") GREATEST, greatest --- ### least Returns the smallest value from a list of one or more parameters. Generally, the return value has the same data type as the input. The comparison rules are the same as the [greatest](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/greatest.md) function. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext LEAST(expr1,...); ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr1`: the expression to compare. It supports the following data types: * SMALLINT * TINYINT * INT * BIGINT * LARGEINT * FLOAT * DOUBLE * DECIMALV2 * DECIMAL32 * DECIMAL64 * DECIMAL128 * DATETIME * VARCHAR #### Examples[​](#examples "Direct link to Examples") Example 1: Return the smallest value for a single input. ```plain select least(3); +----------+ | least(3) | +----------+ | 3 | +----------+ 1 row in set (0.00 sec) ``` Example 2: Return the smallest value from a list of values. ```plain select least(3,4,5,5,6); +----------------------+ | least(3, 4, 5, 5, 6) | +----------------------+ | 3 | +----------------------+ 1 row in set (0.01 sec) ``` Example 3: One parameter is of the DOUBLE type and a DOUBLE value is returned. ```plain select least(4,4.5,5.5); +--------------------+ | least(4, 4.5, 5.5) | +--------------------+ | 4.0 | +--------------------+ ``` Example 4: The input parameters are a mix of number and string but the string can be converted into a number. The parameters are compared as numbers. ```plain select least(7,'5'); +---------------+ | least(7, '5') | +---------------+ | 5 | +---------------+ 1 row in set (0.01 sec) ``` Example 5: The input parameters are a mix of number and string but the string cannot be converted into a number. The parameters are compared as strings. The string `'1'` is less than `'at'`. ```plain select least(1,'at'); +----------------+ | least(1, 'at') | +----------------+ | 1 | +----------------+ ``` Example 6: The input parameters are characters. ```plain mysql> select least('A','B','Z'); +----------------------+ | least('A', 'B', 'Z') | +----------------------+ | A | +----------------------+ 1 row in set (0.00 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") LEAST, least --- ### ln, dlog1, log Computes the natural (base `e`) logarithm of a number. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ln(arg) log(arg) dlog1(arg) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `arg`: the value whose logarithm you want to calculate. Only the DOUBLE data type is supported. > **NOTE** > > StarRocks returns NULL if `arg` is specified as a negative or 0. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Example[​](#example "Direct link to Example") Example 1: Calculate the natural logarithm of 3. ```sql mysql> select ln(3); +--------------------+ | ln(3) | +--------------------+ | 1.0986122886681098 | +--------------------+ 1 row in set (0.00 sec) mysql> select dlog1(3); +--------------------+ | dlog1(3) | +--------------------+ | 1.0986122886681098 | +--------------------+ 1 row in set (0.00 sec) mysql> select log(3); +--------------------+ | log(3) | +--------------------+ | 1.0986122886681098 | +--------------------+ 1 row in set (0.00 sec) ``` --- ### log Calculates the logarithm of a number to a specified base (or radix). If the base is not specified, this function is equivalent to [ln](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/ln.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql log([base,] arg) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `base`: Optional. The base. Only DOUBLE data type is supported. If this parameter is not specified, this function is equivalent to [ln](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/ln.md). > **NOTE** > > StarRocks returns NULL if `base` is specified as a negative, 0, or 1. * `arg`: The value whose logarithm you want to calculate. Only DOUBLE data type is supported. > **NOTE** > > StarRocks returns NULL if `arg` is specified as a negative or 0. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Example[​](#example "Direct link to Example") Example 1: Calculate the logarithm of 8 to the base 2. ```plain mysql> select log(2,8); +-----------+ | log(2, 8) | +-----------+ | 3 | +-----------+ 1 row in set (0.01 sec) ``` Example 2: Calculate the logarithm of 10 to the base *e* (with the base not specified). ```plain mysql> select log(10); +-------------------+ | log(10) | +-------------------+ | 2.302585092994046 | +-------------------+ 1 row in set (0.09 sec) ``` --- ### log10, dlog10 Calculates the base 10 logarithm of a number. #### Syntax[​](#syntax "Direct link to Syntax") ```sql log10(arg) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `arg`: The value whose logarithm you want to calculate. Only DOUBLE data type is supported. > **NOTE** > > StarRocks returns NULL if `arg` is specified as a negative or 0. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Example[​](#example "Direct link to Example") Example 1: Calculate the base 10 logarithm of 100. ```plain select log10(100); +------------+ | log10(100) | +------------+ | 2 | +------------+ 1 row in set (0.02 sec) ``` --- ### log2 Calculates the base 2 logarithm of a number. #### Syntax[​](#syntax "Direct link to Syntax") ```sql log2(arg) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `arg`: The value whose logarithm you want to calculate. Only DOUBLE data type is supported. > **NOTE** > > StarRocks returns NULL if `arg` is specified as a negative or 0. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Example[​](#example "Direct link to Example") Example 1: Calculate the base 2 logarithm of 8. ```plain mysql> select log2(8); +---------+ | log2(8) | +---------+ | 3 | +---------+ 1 row in set (0.00 sec) ``` --- ### mod The modulus function that returns the remainder of `dividend` divided by `divisor`. #### Syntax[​](#syntax "Direct link to Syntax") ```sql mod(dividend, divisor) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `dividend`: The number to be divided. * `divisor`: The number that divides. Both `dividend` and `divisor` support the following data types: * TINYINT * SMALLINT * INT * BIGINT * LARGEINT * FLOAT * DOUBLE * DECIMALV2 * DECIMAL32 * DECIMAL64 * DECIMAL128 > **NOTE** > > `dividend` and `divisor` must agree in the data type. StarRocks performs an implicit conversion if they do not agree in the data type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the same data type as the `dividend`. StarRocks returns NULL if `divisor` is specified as 0. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select mod(3.14,3.14); +-----------------+ | mod(3.14, 3.14) | +-----------------+ | 0 | +-----------------+ mysql> select mod(3.14, 3); +--------------+ | mod(3.14, 3) | +--------------+ | 0.14 | +--------------+ select mod(11,-5); +------------+ | mod(11, -5)| +------------+ | 1 | +------------+ select mod(-11,5); +-------------+ | mod(-11, 5) | +-------------+ | -1 | +-------------+ ``` --- ### multiply Computes the product of the arguments. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext multiply(arg1, arg2) ``` ##### Parameters[​](#parameters "Direct link to Parameters") `arg1`: numeric source column or literal. `arg2`: numeric source column or literal. #### Return value[​](#return-value "Direct link to Return value") Returns the product of two arguments. The return type depends on the arguments. #### Usage notes[​](#usage-notes "Direct link to Usage notes") If you specify a non-numeric value, this function will fail. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL [test]> select multiply(10,2); +-----------------+ | multiply(10, 2) | +-----------------+ | 20 | +-----------------+ 1 row in set (0.01 sec) MySQL [test]> select multiply(1,2.1); +------------------+ | multiply(1, 2.1) | +------------------+ | 2.1 | +------------------+ 1 row in set (0.01 sec) MySQL [test]> select * from t; +------+------+------+------+ | id | name | job1 | job2 | +------+------+------+------+ | 2 | 2 | 2 | 2 | +------+------+------+------+ 1 row in set (0.08 sec) MySQL [test]> select multiply(1.0,id) from t; +-------------------+ | multiply(1.0, id) | +-------------------+ | 2 | +-------------------+ 1 row in set (0.01 sec) ``` #### keyword[​](#keyword "Direct link to keyword") multiply --- ### negative Returns the negative of input `arg`. #### Syntax[​](#syntax "Direct link to Syntax") ```plain negative(arg) ``` #### Parameter[​](#parameter "Direct link to Parameter") `arg` supports the following data types: * BIGINT * DOUBLE * DECIMALV2 * DECIMAL32 * DECIMAL64 * DECIMAL128 #### Return value[​](#return-value "Direct link to Return value") Returns a value of the same data type as the input. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select negative(3); +-------------+ | negative(3) | +-------------+ | -3 | +-------------+ 1 row in set (0.00 sec) mysql> select negative(cast(3.14 as decimalv2)); +--------------------------------------+ | negative(CAST(3.14 AS DECIMAL(9,0))) | +--------------------------------------+ | -3.14 | +--------------------------------------+ 1 row in set (0.01 sec) ``` --- ### pi Returns the value of Pi. #### Syntax[​](#syntax "Direct link to Syntax") ```plain pi() ``` #### Parameter[​](#parameter "Direct link to Parameter") N/A #### Return value[​](#return-value "Direct link to Return value") Returns the value of Pi as a DOUBLE type value. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select pi(); +-------------------+ | pi() | +-------------------+ | 3.141592653589793 | +-------------------+ 1 row in set (0.00 sec) ``` --- ### pmod Returns the positive remainder of `dividend` divided by`divisor`. #### Syntax[​](#syntax "Direct link to Syntax") ```sql pmod(dividend, divisor) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `dividend`: the number to be divided. * `divisor`: the number that divides. Both `arg1` and `arg2` support the following data types: * BIGINT * DOUBLE > **NOTE** > > `dividend` and `divisor` must agree in the data type. StarRocks performs an implicit conversion if they do not agree in the data type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the same data type as the `dividend`. StarRocks returns NULL if `divisor` is specified as 0. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select pmod(3.14,3.14); +------------------+ | pmod(3.14, 3.14) | +------------------+ | 0 | +------------------+ mysql> select pmod(3,6); +------------+ | pmod(3, 6) | +------------+ | 3 | +------------+ mysql> select pmod(11,5); +-------------+ | pmod(11, 5) | +-------------+ | 1 | +-------------+ mysql> select pmod(-11,5); +--------------+ | pmod(-11, 5) | +--------------+ | 4 | +--------------+ mysql> SELECT pmod(11,-5); +--------------+ | pmod(11, -5) | +--------------+ | -4 | +--------------+ mysql> SELECT pmod(-11,-5); +---------------+ | pmod(-11, -5) | +---------------+ | -1 | +---------------+ ``` --- ### positive Returns `x` as a value. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext POSITIVE(x); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: supports the BIGINT, DOUBLE, DECIMALV2, DECIMAL32, DECIMAL64, and DECIMAL128 data types. #### Return value[​](#return-value "Direct link to Return value") Returns a value whose data type is the same as the data type of the `x` value. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select positive(3); +-------------+ | positive(3) | +-------------+ | 3 | +-------------+ 1 row in set (0.01 sec) mysql> select positive(cast(3.14 as decimalv2)); +--------------------------------------+ | positive(CAST(3.14 AS DECIMAL(9,0))) | +--------------------------------------+ | 3.14 | +--------------------------------------+ 1 row in set (0.01 sec) ``` --- ### pow, power, dpow, fpow Returns the result of `x` raised to the power of `y`. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext POW(x,y);POWER(x,y); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: supports the DOUBLE data type. `y`: supports the DOUBLE data type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select pow(2,2); +-----------+ | pow(2, 2) | +-----------+ | 4 | +-----------+ 1 row in set (0.00 sec) mysql> select power(4,3); +-------------+ | power(4, 3) | +-------------+ | 64 | +-------------+ 1 row in set (0.00 sec) ``` --- ### radians Converts `x` from an angle to a radian. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext REDIANS(x); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: supports the DOUBLE data type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select radians(90); +--------------------+ | radians(90) | +--------------------+ | 1.5707963267948966 | +--------------------+ 1 row in set (0.00 sec) ``` --- ### rand, random Returns a random floating-point number between 0 (inclusive) and 1 (exclusive). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext RAND(x); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: optional. The data type is BIGINT. Whether \`x is specified or not, this function returns a completely random number. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE type. #### Examples[​](#examples "Direct link to Examples") ```plain select rand(); +--------------------+ | rand() | +--------------------+ | 0.9393535880089522 | +--------------------+ 1 row in set (0.01 sec) select rand(3); +--------------------+ | rand(3) | +--------------------+ | 0.6659865964511347 | +--------------------+ 1 row in set (0.00 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") RAND, RANDOM --- ### round, dround Rounds a number to a specified number of digits. * If `n` is not specified, `x` is rounded to the nearest integer. * If `n` is specified, `x` is rounded to `n` decimal places. If `n` is negative, `x` is rounded to the left of the decimal point. If overflow occurs, an error is returned. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext ROUND(x [,n]); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: the number to be rounded. It supports the DOUBLE and DECIMAL128 data types. `n`: the number of decimal places to round the number to. It supports the INT data type. This parameter is optional. #### Return value[​](#return-value "Direct link to Return value") * If only `x` is specified, the return value is of the following data type: \["DECIMAL128"] -> "DECIMAL128" \["DOUBLE"] -> "BIGINT" * If both `x` and `n` are specified, the return value is of the following data type: \["DECIMAL128", "INT"] -> "DECIMAL128" \["DOUBLE", "INT"] -> "DOUBLE" #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select round(3.14); +-------------+ | round(3.14) | +-------------+ | 3 | +-------------+ mysql> select round(3.14,1); +----------------+ | round(3.14, 1) | +----------------+ | 3.1 | +----------------+ mysql> select round(13.14,-1); +------------------+ | round(13.14, -1) | +------------------+ | 10 | +------------------+ mysql> select round(122.14,-1); +-------------------+ | round(122.14, -1) | +-------------------+ | 120 | +-------------------+ mysql> select round(122.14,-2); +-------------------+ | round(122.14, -2) | +-------------------+ | 100 | +-------------------+ mysql> select round(122.14,-3); +-------------------+ | round(122.14, -3) | +-------------------+ | 0 | +-------------------+ ``` --- ### sign Returns the sign of `x`. A negative number, 0, or a positive number as input corresponds to `-1`, `0`, or `1` as output, respectively. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext SIGN(x); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: supports the DOUBLE data type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the FLOAT data type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select sign(3.14159); +---------------+ | sign(3.14159) | +---------------+ | 1 | +---------------+ 1 row in set (0.02 sec) ``` --- ### sin Returns the sine of `x`, where `x` is in radians. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext SIN(x); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: supports the DOUBLE data type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select sin(3.14); +-----------------------+ | sin(3.14) | +-----------------------+ | 0.0015926529164868282 | +-----------------------+ 1 row in set (0.21 sec) ``` --- ### sinh Computes the hyperbolic sine of the argument. This function is supported from v3.0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DOUBLE sinh(DOUBLE arg) ``` ##### Parameters[​](#parameters "Direct link to Parameters") `arg`: You can specify only a numeric value. This function converts the numeric value into a DOUBLE value before it computes the hyperbolic sine of the value. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Usage notes[​](#usage-notes "Direct link to Usage notes") If you specify a non-numeric value, this function returns `NULL`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select sinh(-1); +---------------------+ | sinh(-1) | +---------------------+ | -1.1752011936438014 | +---------------------+ mysql> select sinh(0); +---------+ | sinh(0) | +---------+ | 0 | +---------+ mysql> select sinh(1); +--------------------+ | sinh(1) | +--------------------+ | 1.1752011936438014 | +--------------------+ mysql> select sinh(""); +----------+ | sinh('') | +----------+ | NULL | +----------+ ``` #### keyword[​](#keyword "Direct link to keyword") SINH --- ### sqrt, dsqrt Calculates the square root of a value. dsqrt is same as sqrt. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DOUBLE SQRT(DOUBLE x); DOUBLE DSQRT(DOUBLE x); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: You can specify only a numeric value. This function converts the numeric value into a DOUBLE value before calculation. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Usage notes[​](#usage-notes "Direct link to Usage notes") If you specify a non-numeric value, this function returns `NULL`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select sqrt(3.14); +-------------------+ | sqrt(3.14) | +-------------------+ | 1.772004514666935 | +-------------------+ 1 row in set (0.01 sec) mysql> select dsqrt(3.14); +-------------------+ | dsqrt(3.14) | +-------------------+ | 1.772004514666935 | +-------------------+ ``` --- ### square Calculates the square of a value. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext square(arg) ``` ##### Parameters[​](#parameters "Direct link to Parameters") `arg`: the value whose square you want to calculate. You can specify only a numeric value. This function converts the numeric value into a DOUBLE value before it calculates the square of the value. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Usage notes[​](#usage-notes "Direct link to Usage notes") If you specify a non-numeric value, this function returns `NULL`. #### Examples[​](#examples "Direct link to Examples") Example 1: Calculate the square of a numeric value. ```plain mysql> select square(11); +------------+ | square(11) | +------------+ | 121 | +------------+ ``` Example 2: Calculate the square of a non-numeric value. The return value is `NULL`. ```plain mysql> select square('2021-01-01'); +----------------------+ | square('2021-01-01') | +----------------------+ | NULL | +----------------------+ ``` --- ### tan Returns the tangent of `x`, where `x` is in radians. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext TAN(x); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: supports the DOUBLE data type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select tan(3.14); +-----------------------+ | tan(3.14) | +-----------------------+ | -0.001592654936407223 | +-----------------------+ 1 row in set (0.12 sec) ``` --- ### tanh Computes the hyperbolic tangent of the argument. This function is supported from v3.0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DOUBLE tanh(DOUBLE arg) ``` ##### Parameters[​](#parameters "Direct link to Parameters") `arg`: You can specify only a numeric value. This function converts the numeric value into a DOUBLE value before it computes the hyperbolic tangent of the value. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the DOUBLE data type. #### Usage notes[​](#usage-notes "Direct link to Usage notes") If you specify a non-numeric value, this function returns `NULL`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select tanh(-1); +---------------------+ | tanh(-1) | +---------------------+ | -0.7615941559557649 | +---------------------+ mysql> select tanh(0); +---------+ | tanh(0) | +---------+ | 0 | +---------+ mysql> select tanh(1); +--------------------+ | tanh(1) | +--------------------+ | 0.7615941559557649 | +--------------------+ mysql> select tanh(""); +----------+ | tanh('') | +----------+ | NULL | +----------+ ``` #### keyword[​](#keyword "Direct link to keyword") TANH --- ### truncate Rounds the input down to the nearest equal or smaller value with the specified number of places after the decimal point. #### Syntax[​](#syntax "Direct link to Syntax") ```shell truncate(arg1,arg2); ``` #### Parameter[​](#parameter "Direct link to Parameter") * `arg1`: the input to be rounded. It supports the following data types: * DOUBLE * DECIMAL128 * `arg2`: the number of places to keep after the decimal point. It supports the INT data type. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the same data type as `arg1`. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select truncate(3.14,1); +-------------------+ | truncate(3.14, 1) | +-------------------+ | 3.1 | +-------------------+ 1 row in set (0.00 sec) ``` --- ### get_query_dump `get_query_dump(query)` `get_query_dump(query, enable_mock)` These functions return a dump of the query for debugging purposes. #### Arguments[​](#arguments "Direct link to Arguments") `query`: The SQL query string (VARCHAR). `enable_mock`: (Optional) A boolean value indicating whether to enable mock data for the dump. Defaults to `FALSE`. #### Return Value[​](#return-value "Direct link to Return Value") Returns a VARCHAR string containing the query dump. #### Examples[​](#examples "Direct link to Examples") Example 1: Get a simple query's dump without mocking data ```text mysql> select get_query_dump('select * from ss limit 1'); 1 row in set (0.04 sec) ``` --- ### get_query_dump_from_query_id `get_query_dump_from_query_id(query_id)` `get_query_dump_from_query_id(query_id, enable_mock)` Returns a dump of a previously executed query, located by its `query_id`. The dump contains the table schemas, statistics, session variables and other context needed to reproduce planning of the original query, in the same JSON shape produced by [`get_query_dump`](https://docs.starrocks.io/docs/sql-reference/sql-functions/meta-functions/get_query_dump.md) and the `/api/query_dump` HTTP endpoint. This function is intended for debugging and post-mortem analysis. It looks up the query in StarRocks' in-memory query detail queue on the current FE, recovers the original SQL plus its catalog and database, and re-runs the dumper against that SQL. #### Arguments[​](#arguments "Direct link to Arguments") `query_id`: The query identifier, in the standard StarRocks UUID format (`xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`). Type: VARCHAR. `enable_mock`: (Optional) When `TRUE`, the dump uses mocked table / column names instead of the originals (handy for sharing the dump without leaking schema). Defaults to `FALSE`. Type: BOOLEAN. #### Return Value[​](#return-value "Direct link to Return Value") VARCHAR. JSON-encoded query dump. Throws an error if the query cannot be located, the recorded SQL was desensitized, or the caller is not allowed to read the query. #### Operational requirements[​](#operational-requirements "Direct link to Operational requirements") The query detail queue is the source of truth, so the same constraints that apply to that queue apply here. The function validates the two FE-config gates upfront and returns a clear error when either is mis-set: * `enable_collect_query_detail_info` must be `true` (default `false`). When unset, the function refuses immediately with `query detail collection is disabled. Set FE config enable_collect_query_detail_info=true ...` instead of silently producing a "not found" error. * `enable_sql_desensitize_in_log` must be `false`. When set to `true` the recorded SQL is rewritten to a digest form and cannot be re-dumped; the function refuses upfront with `SQL desensitization is enabled ...`. Additional runtime constraints, surfaced after the upfront checks pass: * The query must have been executed on the same FE that the function is invoked on. Detail data is per-FE in-memory state, not synchronized. * The detail row must still be in the cache window controlled by `query_detail_cache_time_nanosecond` (default 30 seconds). Older queries are evicted by a background sweep. * As defense in depth, even with desensitization currently off, if the recorded SQL was captured while desensitization was on (so the detail row holds the placeholder), the function refuses with `original sql not retained`. #### Privilege[​](#privilege "Direct link to Privilege") The caller must either match the original executor's full account identity (`user`@`host`, e.g. `'alice'@'10.0.0.1'`) or hold system-level `OPERATE` privilege. Username-only matches are rejected, so two distinct accounts that share a username but differ on host cannot read each other's queries. #### Examples[​](#examples "Direct link to Examples") ```sql -- A user looking up one of their own queries (assuming -- enable_collect_query_detail_info = true on the FE). mysql> SELECT get_query_dump_from_query_id('a1b2c3d4-e5f6-7890-abcd-ef0123456789')\G -- Same, but with mocked schema for safer sharing. mysql> SELECT get_query_dump_from_query_id('a1b2c3d4-e5f6-7890-abcd-ef0123456789', TRUE)\G -- Typical error when the query is not in the cache window. mysql> SELECT get_query_dump_from_query_id('00000000-0000-0000-0000-000000000000'); ERROR 1064 (HY000): Getting analyzing error. Detail message: Invalid parameter get_query_dump_from_query_id: query_id not found in query detail queue: ... ``` #### See also[​](#see-also "Direct link to See also") * [`get_query_dump`](https://docs.starrocks.io/docs/sql-reference/sql-functions/meta-functions/get_query_dump.md) — dump a SQL string supplied directly by the caller, without going through the query detail queue. --- ### inspect_all_pipes `inspect_all_pipes()` This function returns meta data of all pipes in the current database. #### Arguments[​](#arguments "Direct link to Arguments") None. #### Return Value[​](#return-value "Direct link to Return Value") Returns a VARCHAR string containing the meta data of all pipes in JSON format. #### Examples[​](#examples "Direct link to Examples") Example 1: Get the curent all ```text mysql> select inspect_all_pipes(); +---------------------+ | inspect_all_pipes() | +---------------------+ | [] | +---------------------+ 1 row in set (0.01 sec) ``` --- ### inspect_hive_part_info `inspect_hive_part_info(table_name)` This function returns Hive partition information for a table. #### Arguments[​](#arguments "Direct link to Arguments") `table_name`: The name of the table (VARCHAR). #### Return Value[​](#return-value "Direct link to Return Value") Returns a VARCHAR string containing the Hive partition information in JSON format. #### Examples[​](#examples "Direct link to Examples") Example 1: Inspect a partitioned Hive table's partition information: ```text mysql> select inspect_hive_part_info('t1'); +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | inspect_hive_part_info('t1') | +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | {"k3=9999-12-03/k4=3":{"parameters":{"totalSize":"625","numRows":"1","starrocks_version":"bc39130-bc39130","numFiles":"1","starrocks_query_id":"0197ca18-dd10-76fd-bb85-253226af8365","transient_lastDdlTime":"1751442316","STATS_GENERATED_VIA_STATS_TASK":"workaround for potential lack of HIVE-12730"},"inputFormat":"PARQUET","textFileFormatDesc":{},"fullPath":"hdfs://emr-header-1.cluster-49091:9000/user/hive/warehouse/hive_db_b4425ea7d8184049a2b1e039c0a8f595.db/t1/k3=9999-12-03/k4=3","isSplittable":true}} | +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) ``` --- ### inspect_memory `inspect_memory(module_name)` This function returns the estimated memory usage for a specified module. #### Arguments[​](#arguments "Direct link to Arguments") `module_name`: The name of the module (VARCHAR). NOTE: The following module names are supported (for more details, see the `MemoryUsageTracker.java` class): * Load * Compaction * Export * Delete * Transaction * Backup * Task * TabletInvertedIndex * LocalMetastore * Report * MV * Query * Profile * Agent * Statistics * Coordinator * Dict #### Return Value[​](#return-value "Direct link to Return Value") Returns a VARCHAR string representing the estimated memory size (e.g., "100MB"). #### Examples[​](#examples "Direct link to Examples") Example 1: Inspect current FE's Load module memory usage: ```text mysql> SELECT inspect_memory('Load'); +------------------------+ | inspect_memory('Load') | +------------------------+ | 251.8KB | +------------------------+ 1 row in set (0.00 sec) ``` --- ### inspect_memory_detail `inspect_memory_detail(module_name, class_info)` This function returns the estimated memory usage for a specific class or field within a module. #### Arguments[​](#arguments "Direct link to Arguments") `module_name`: The name of the module (VARCHAR). `class_info`: The name of the class or 'class\_name.field\_name' (VARCHAR). NOTE: 1. The following module names are supported (for more details, see the `MemoryUsageTracker.java` class): * Load * Compaction * Export * Delete * Transaction * Backup * Task * TabletInvertedIndex * LocalMetastore * Report * MV * Query * Profile * Agent * Statistics * Coordinator * Dict 2. `class_info` can be a class or a class's field of the specific class(for more details, see the `MemoryUsageTracker.java` class). #### Return Value[​](#return-value "Direct link to Return Value") Returns a VARCHAR string representing the estimated memory size (e.g., "100MB"). #### Examples[​](#examples "Direct link to Examples") Example 1: Inspect Task module TaskManager's memory usage: ```text mysql> select inspect_memory_detail('Task', 'TaskManager'); +----------------------------------------------+ | inspect_memory_detail('Task', 'TaskManager') | +----------------------------------------------+ | 0B | +----------------------------------------------+ 1 row in set (0.00 sec) ``` Example 2: Inspect Task module TaskManager's taskRunManager field's memory usage: ```text mysql> select inspect_memory_detail('Task', 'TaskManager.taskRunManager'); +-------------------------------------------------------------+ | inspect_memory_detail('Task', 'TaskManager.taskRunManager') | +-------------------------------------------------------------+ | 592B | +-------------------------------------------------------------+ 1 row in set (0.01 sec) ``` --- ### inspect_mv_meta `inspect_mv_meta(mv_name)` This function returns verbose metadata of a materialized view. #### Arguments[​](#arguments "Direct link to Arguments") `mv_name`: The name of the materialized view (VARCHAR). #### Return Value[​](#return-value "Direct link to Return Value") Returns a VARCHAR string containing the verbose metadata of the materialized view. #### Examples[​](#examples "Direct link to Examples") Example 1: Inspect a materialized view's internal metadata: ```text mysql> select inspect_mv_meta('mv_on_view_1'); +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | inspect_mv_meta('mv_on_view_1') | +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | {"clazz":"LakeMaterializedView","dbId":28651,"refreshScheme":{"moment":"IMMEDIATE","type":"ASYNC","asyncRefreshContext":{"baseTableVisibleVersionMap":{"28673":{"ss":{"id":28672,"version":4,"lastRefreshTime":1751439875145,"lastFileModifiedTime":-1,"fileNumber":-1}}},"baseTableInfoVisibleVersionMap":{},"mvPartitionNameRefBaseTablePartitionMap":{},"defineStartTime":false,"starTime":1751442848,"step":0},"lastRefreshTime":1751439875145},"baseTableInfos":[{"catalogName":"default_catalog","dbId":28651,"tableId":28673,"dbName":"test","tableName":"ss"},{"catalogName":"default_catalog","dbId":28651,"tableId":28805,"dbName":"test","tableName":"view1"}],"active":true,"viewDefineSql":"SELECT `test`.`view1`.`event_day`, `test`.`view1`.`sum_pv`\nFROM `test`.`view1`","simpleDefineSql":"SELECT `view1`.`event_day`, `view1`.`sum_pv`\nFROM `test`.`view1`","originalViewDefineSql":"select * from view1","partitionRefTableExprs":[],"maxMVRewriteStaleness":0,"partitionExprMaps":{},"queryOutputIndices":[],"warehouseId":0,"state":"NORMAL","indexIdToMeta":{"28807":{"indexId":28807,"schema":[{"name":"event_day","columnId":"event_day","uniqueId":-1,"type":{"clazz":"ScalarType","type":"DATE","len":-1,"precision":0,"scale":0},"isAggregationTypeImplicit":false,"isKey":true,"isAllowNull":true,"isAutoIncrement":false,"comment":"","stats":{"avgSerializedSize":-1.0,"maxSize":-1,"numDistinctValues":-1,"numNulls":-1},"physicalName":""},{"name":"sum_pv","columnId":"sum_pv","uniqueId":-1,"type":{"clazz":"ScalarType","type":"BIGINT","len":-1,"precision":0,"scale":0},"isAggregationTypeImplicit":false,"isKey":true,"isAllowNull":true,"isAutoIncrement":false,"comment":"","stats":{"avgSerializedSize":-1.0,"maxSize":-1,"numDistinctValues":-1,"numNulls":-1},"physicalName":""}],"schemaVersion":0,"schemaHash":1933831807,"schemaId":28807,"shortKeyColumnCount":2,"storageType":"COLUMN","keysType":"DUP_KEYS","dbId":0,"isColocateMVIndex":false}},"indexNameToId":{"mv_on_view_1":28807},"keysType":"DUP_KEYS","partitionInfo":{"clazz":"SinglePartitionInfo","type":"UNPARTITIONED","idToDataProperty":{"28820":{"storageMedium":"HDD","cooldownTimeMs":253402271999000}},"idToReplicationNum":{"28820":1},"isMultiColumnPartition":false,"idToInMemory":{"28820":false},"idToStorageCacheInfo":{"28820":{"cacheInfoBytes":[8,1,16,-1,-1,-1,-1,-1,-1,-1,-1,-1,1]}}},"idToPartition":{"28820":{"id":28820,"name":"mv_on_view_1","state":"NORMAL","dpid":28821,"idToSubPartition":{"28821":{"id":28821,"name":"mv_on_view_1_28819_28821","parentId":28820,"shardGroupId":8170,"isImmutable":false,"baseIndex":{"id":28807,"state":"NORMAL","rowCount":0,"virtualBuckets":[28822,28823,28824,28825,28826,28827,28828,28829],"tablets":[{"clazz":"LakeTablet","dataSize":0,"rowCount":0,"dataSizeUpdateTime":0,"id":28822,"signature":-1,"lastCheckTime":-1},{"clazz":"LakeTablet","dataSize":0,"rowCount":0,"dataSizeUpdateTime":0,"id":28823,"signature":-1,"lastCheckTime":-1},{"clazz":"LakeTablet","dataSize":0,"rowCount":0,"dataSizeUpdateTime":0,"id":28824,"signature":-1,"lastCheckTime":-1},{"clazz":"LakeTablet","dataSize":0,"rowCount":0,"dataSizeUpdateTime":0,"id":28825,"signature":-1,"lastCheckTime":-1},{"clazz":"LakeTablet","dataSize":0,"rowCount":0,"dataSizeUpdateTime":0,"id":28826,"signature":-1,"lastCheckTime":-1},{"clazz":"LakeTablet","dataSize":0,"rowCount":0,"dataSizeUpdateTime":0,"id":28827,"signature":-1,"lastCheckTime":-1},{"clazz":"LakeTablet","dataSize":0,"rowCount":0,"dataSizeUpdateTime":0,"id":28828,"signature":-1,"lastCheckTime":-1},{"clazz":"LakeTablet","dataSize":0,"rowCount":0,"dataSizeUpdateTime":0,"id":28829,"signature":-1,"lastCheckTime":-1}],"shardGroupId":8170,"signature":-1,"lastCheckTime":-1},"idToVisibleRollupIndex":{},"idToShadowIndex":{},"visibleVersion":2,"visibleVersionTime":1751442848573,"nextVersion":3,"dataVersion":2,"nextDataVersion":3,"versionEpoch":364078271679168512,"versionTxnType":"TXN_NORMAL","metadataSwitchVersion":0,"bucketNum":8,"signature":-1,"lastCheckTime":-1}},"distributionInfo":{"clazz":"RandomDistributionInfo","b":8,"typeStr":"RANDOM","type":"RANDOM"},"shardGroupId":-1,"isImmutable":false,"idToVisibleRollupIndex":{},"idToShadowIndex":{},"visibleVersion":0,"visibleVersionTime":0,"nextVersion":0,"dataVersion":0,"nextDataVersion":0,"versionEpoch":0,"signature":-1,"lastCheckTime":-1}},"defaultDistributionInfo":{"clazz":"RandomDistributionInfo","b":0,"typeStr":"RANDOM","type":"RANDOM"},"tempPartitions":{"idToPartition":{}},"bfFpp":0.0,"indexes":{"indexes":[],"properties":{}},"baseIndexId":28807,"tableProperty":{"properties":{"replicated_storage":"true","replication_num":"1"},"compressionLevel":-1,"hasDelete":false,"hasForbitGlobalDict":false,"storageInfo":{"storeInfoBytes":[10,-123,2,8,1,18,32,115,116,97,114,114,111,99,107,115,45,99,108,111,117,100,45,100,97,116,97,45,122,104,97,110,103,106,105,97,107,111,117,26,-108,1,10,32,115,116,97,114,114,111,99,107,115,45,99,108,111,117,100,45,100,97,116,97,45,122,104,97,110,103,106,105,97,107,111,117,26,40,111,115,115,45,99,110,45,122,104,97,110,103,106,105,97,107,111,117,45,105,110,116,101,114,110,97,108,46,97,108,105,121,117,110,99,115,46,99,111,109,34,60,18,58,10,24,76,84,65,73,53,116,81,68,100,114,87,77,54,81,56,54,111,68,90,122,77,69,98,102,18,30,73,82,109,109,106,115,50,119,117,121,102,104,108,67,121,55,121,122,86,74,88,81,106,73,90,104,73,100,114,73,42,8,88,88,88,88,88,88,88,88,50,22,98,117,105,108,116,105,110,95,115,116,111,114,97,103,101,95,118,111,108,117,109,101,56,1,82,46,115,51,58,47,47,115,116,97,114,114,111,99,107,115,45,99,108,111,117,100,45,100,97,116,97,45,122,104,97,110,103,106,105,97,107,111,117,47,88,88,88,88,88,88,88,88,18,97,115,51,58,47,47,115,116,97,114,114,111,99,107,115,45,99,108,111,117,100,45,100,97,116,97,45,122,104,97,110,103,106,105,97,107,111,117,47,88,88,88,88,88,88,88,88,47,57,53,99,48,56,55,49,53,45,55,102,53,57,45,52,48,57,50,45,98,50,101,48,45,102,97,49,53,48,57,56,101,98,100,101,102,47,100,98,50,56,54,53,49,47,50,56,56,48,54],"cacheInfoBytes":[8,1,16,-1,-1,-1,-1,-1,-1,-1,-1,-1,1]}},"maxColUniqueId":-1,"maxIndexId":-1,"id":28806,"name":"mv_on_view_1","type":"LAKE_MATERIALIZED_VIEW","createTime":1751442848,"fullSchema":[{"name":"event_day","columnId":"event_day","uniqueId":-1,"type":{"clazz":"ScalarType","type":"DATE","len":-1,"precision":0,"scale":0},"isAggregationTypeImplicit":false,"isKey":true,"isAllowNull":true,"isAutoIncrement":false,"comment":"","stats":{"avgSerializedSize":-1.0,"maxSize":-1,"numDistinctValues":-1,"numNulls":-1},"physicalName":""},{"name":"sum_pv","columnId":"sum_pv","uniqueId":-1,"type":{"clazz":"ScalarType","type":"BIGINT","len":-1,"precision":0,"scale":0},"isAggregationTypeImplicit":false,"isKey":true,"isAllowNull":true,"isAutoIncrement":false,"comment":"","stats":{"avgSerializedSize":-1.0,"maxSize":-1,"numDistinctValues":-1,"numNulls":-1},"physicalName":""}],"comment":"","mvs":[],"signature":-1,"lastCheckTime":-1} | +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.01 sec) ``` --- ### inspect_mv_plan `inspect_mv_plan(mv_name)` `inspect_mv_plan(mv_name, use_cache)` These functions return the logical plan of a materialized view. #### Arguments[​](#arguments "Direct link to Arguments") `mv_name`: The name of the materialized view (VARCHAR). `use_cache`: (Optional) A boolean value indicating whether to use the materialized view plan cache. Defaults to `TRUE`. #### Return Value[​](#return-value "Direct link to Return Value") Returns a VARCHAR string containing the logical plan of the materialized view. #### Examples[​](#examples "Direct link to Examples") Example 1: Inspect a materialized view's logical plan using the plan cache: ```text mysql> select inspect_mv_plan('mv_on_view_1'); +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | inspect_mv_plan('mv_on_view_1') | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | plan 0: LogicalAggregation {type=GLOBAL ,aggregations={3: sum=sum(2: pv)} ,groupKeys=[1: event_day] ,projection=null ,predicate=null} -> LogicalOlapScanOperator {table=28673, selectedPartitionId=null, selectedIndexId=28674, outputColumns=[1: event_day, 2: pv], predicate=null, prunedPartitionPredicates=[], limit=-1} plan 1: LogicalViewScanOperator {table='view1', outputColumns='[4: event_day, 5: sum_pv]'} | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.01 sec) ``` Example 2: Inspect a materialized view's logical plan without using the plan cache: ```text mysql> select inspect_mv_plan('mv_on_view_1', false); +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | inspect_mv_plan('mv_on_view_1', FALSE) | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | plan 0: LogicalAggregation {type=GLOBAL ,aggregations={3: sum=sum(2: pv)} ,groupKeys=[1: event_day] ,projection=null ,predicate=null} -> LogicalOlapScanOperator {table=28673, selectedPartitionId=null, selectedIndexId=28674, outputColumns=[1: event_day, 2: pv], predicate=null, prunedPartitionPredicates=[], limit=-1} plan 1: LogicalViewScanOperator {table='view1', outputColumns='[4: event_day, 5: sum_pv]'} | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.01 sec) ``` --- ### inspect_mv_refresh_info `inspect_mv_refresh_info(mv_name)` This function returns refresh information of a materialized view. #### Arguments[​](#arguments "Direct link to Arguments") `mv_name`: The name of the materialized view (VARCHAR). #### Return Value[​](#return-value "Direct link to Return Value") Returns a VARCHAR string containing the refresh information of the materialized view in JSON format including: * `tableToUpdatePartitions`: the mv's base tables' to-refresh partition meta infos; * `baseOlapTableVisibleVersionMap`: the mv's olap base tables' already refreshed version map; * `baseExternalTableInfoVisibleVersionMap`: the mv's external base tables' already refreshed version map; #### Examples[​](#examples "Direct link to Examples") Example 1: Inspect a materialized view's refresh information to check its current refresh status: ```text mysql> select inspect_mv_refresh_info('test_mv1'); +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | inspect_mv_refresh_info('test_mv1') | +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | {"tableToUpdatePartitions":{},"baseOlapTableVisibleVersionMap":{"ss":{"ss":{"id":28672,"version":4,"lastRefreshTime":1751439875145,"lastFileModifiedTime":-1,"fileNumber":-1}}},"baseExternalTableInfoVisibleVersionMap":{}} | +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) ``` --- ### inspect_mv_relationships `inspect_mv_relationships()` This function returns the content in `ConnectorTblMetaInfoMgr`, which contains mapping information from base external table to materialized view. #### Arguments[​](#arguments "Direct link to Arguments") None. #### Return Value[​](#return-value "Direct link to Return Value") Returns a VARCHAR string containing the mapping information in JSON format. #### Examples[​](#examples "Direct link to Examples") Example 1: Inspect current mv relations in ConnectorTblMetaInfoMgr: ```text mysql> select inspect_mv_relationships(); +----------------------------+ | inspect_mv_relationships() | +----------------------------+ | {} | +----------------------------+ 1 row in set (0.01 sec) ``` --- ### inspect_related_mv `inspect_related_mv(table_name)` This function returns related materialized views of a table in JSON array format. #### Arguments[​](#arguments "Direct link to Arguments") `table_name`: The name of the table (VARCHAR). #### Return Value[​](#return-value "Direct link to Return Value") Returns a VARCHAR string containing a JSON array of related materialized views. #### Examples[​](#examples "Direct link to Examples") Example 1: Inspect `ss` table related mvs: ```text mysql> select inspect_related_mv('ss'); +---------------------------------------------------------------------+ | inspect_related_mv('ss') | +---------------------------------------------------------------------+ | [{"id":28806,"name":"mv_on_view_1"},{"id":28844,"name":"test_mv1"}] | +---------------------------------------------------------------------+ 1 row in set (0.01 sec) ``` --- ### inspect_table_partition_info `inspect_table_partition_info(table_name)` This function returns partition information of a table. #### Arguments[​](#arguments "Direct link to Arguments") `table_name`: The name of the table (VARCHAR). #### Return Value[​](#return-value "Direct link to Return Value") Returns a VARCHAR string containing the partition information of the table in JSON format. #### Examples[​](#examples "Direct link to Examples") Example 1: Inspect a table's partitino information: ```text mysql> select inspect_table_partition_info('ss'); +-----------------------------------------------------------------------------------------------------------+ | inspect_table_partition_info('ss') | +-----------------------------------------------------------------------------------------------------------+ | {"ss":{"id":28672,"version":4,"lastRefreshTime":1751439875145,"lastFileModifiedTime":-1,"fileNumber":-1}} | +-----------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) ``` --- ### inspect_task_runs `inspect_task_runs()` This function returns all status about the TaskManager. #### Arguments[​](#arguments "Direct link to Arguments") None. #### Return Value[​](#return-value "Direct link to Return Value") Returns a VARCHAR string containing the TaskManager status in JSON format. #### Examples[​](#examples "Direct link to Examples") Example 1: Inspect current TaskManger global pending/running task runs information: ```text mysql> select inspect_task_runs(); +-----------------------------------------------------------------------------------------------------------------------------+ | inspect_task_runs() | +-----------------------------------------------------------------------------------------------------------------------------+ | {"pendingTaskRunQueue":{"pendingTaskRunMap":{},"pendingTaskRunQueue":[]},"runningTaskRunMap":{},"runningSyncTaskRunMap":{}} | +-----------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) ``` --- ### lookup_string `lookup_string(table_name, lookup_key, return_column)` This function looks up a value from a primary key table and evaluates it in the optimizer. #### Arguments[​](#arguments "Direct link to Arguments") `table_name`: The name of the table to lookup. Must be a primary-key table (VARCHAR). `lookup_key`: The key to lookup. Must be a string type (VARCHAR). `return_column`: The name of the column to return (VARCHAR). #### Return Value[​](#return-value "Direct link to Return Value") Returns a VARCHAR string containing the looked-up value. Returns `NULL` if not found. #### Examples[​](#examples "Direct link to Examples") Example 1: Return `t2` table's `event_day` column value of `t2` table with its primay key value equals to `1`; ```text mysql> select lookup_string('t2', '1', 'event_day'); +---------------------------------------+ | lookup_string('t2', '1', 'event_day') | +---------------------------------------+ | 2020-01-14 | +---------------------------------------+ 1 row in set (0.02 sec) ``` --- ### Percentile #### [📄️ percentile\_approx](https://docs.starrocks.io/docs/sql-reference/sql-functions/percentile-functions/percentile_approx.md) #### [📄️ percentile\_approx\_raw](https://docs.starrocks.io/docs/sql-reference/sql-functions/percentile-functions/percentile_approx_raw.md) [Returns the value that corresponds to a specified percentile from x.](https://docs.starrocks.io/docs/sql-reference/sql-functions/percentile-functions/percentile_approx_raw.md) #### [📄️ percentile\_cont](https://docs.starrocks.io/docs/sql-reference/sql-functions/percentile-functions/percentile_cont.md) #### [📄️ percentile\_disc](https://docs.starrocks.io/docs/sql-reference/sql-functions/percentile-functions/percentile_disc.md) #### [📄️ percentile\_empty](https://docs.starrocks.io/docs/sql-reference/sql-functions/percentile-functions/percentile_empty.md) [Constructs an empty PERCENTILE value used as a placeholder during Stream Load or INSERT INTO data loading.](https://docs.starrocks.io/docs/sql-reference/sql-functions/percentile-functions/percentile_empty.md) #### [📄️ percentile\_hash](https://docs.starrocks.io/docs/sql-reference/sql-functions/percentile-functions/percentile_hash.md) [Constructs DOUBLE values as PERCENTILE values.](https://docs.starrocks.io/docs/sql-reference/sql-functions/percentile-functions/percentile_hash.md) #### [📄️ percentile\_union](https://docs.starrocks.io/docs/sql-reference/sql-functions/percentile-functions/percentile_union.md) [Aggregates PERCENTILE data.](https://docs.starrocks.io/docs/sql-reference/sql-functions/percentile-functions/percentile_union.md) --- ### percentile_approx ### percentile\_approx Returns the approximate value for a given percentile p, or an array of values for corresponding percentiles if p is an array. All percentile values must be in the range \[0,1]. Compression parameter is optional and has a setting range of \[2048, 10000]. The larger the value, the higher the accuracy, the larger the memory consumption, and the longer the calculation time. If it is not specified or not beyond the range of \[2048, 10000], the function runs with a default compression parameter of 10000. This function uses fixed size memory, so less memory can be used for columns with high cardinality, and can be used to calculate statistics such as tp99. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DOUBLE PERCENTILE_APPROX(expr, DOUBLE|ARRAY p[, DOUBLE compression]) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select `table`, percentile_approx(cost_time,0.99) from log_statis group by `table`; +----------+--------------------------------------+ | table | percentile_approx(`cost_time`, 0.99) | +----------+--------------------------------------+ | test | 54.22 | +----------+--------------------------------------+ MySQL > select `table`, percentile_approx(cost_time,0.99, 4096) from log_statis group by `table`; +----------+----------------------------------------------+ | table | percentile_approx(`cost_time`, 0.99, 4096.0) | +----------+----------------------------------------------+ | test | 54.21 | +----------+----------------------------------------------+ MySQL > select percentile_approx(c2, [0.1, 0.5, 0.9], 10000) from t1; +-----------------------------------------------+ | percentile_approx(c2, [0.1, 0.5, 0.9], 10000) | +-----------------------------------------------+ | [4999.6005859375,25000,45000.3984375] | +-----------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") PERCENTILE\_APPROX,PERCENTILE,APPROX --- ### percentile_approx_raw Returns the value that corresponds to a specified percentile from `x`. If `x` is a column, this function first sorts values in `x` in ascending order and returns the value that corresponds to percentile `y`. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext PERCENTILE_APPROX_RAW(x, y); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `x`: It can be a column or a set of values. It must evaluate to PERCENTILE. * `y`: the percentile. The supported data type is DOUBLE. Value range: \[0.0,1.0]. #### Return value[​](#return-value "Direct link to Return value") Returns a PERCENTILE value. #### Examples[​](#examples "Direct link to Examples") Create table `aggregate_tbl`, where the `percent` column is the input of percentile\_approx\_raw(). ```sql CREATE TABLE `aggregate_tbl` ( `site_id` largeint(40) NOT NULL COMMENT "id of site", `date` date NOT NULL COMMENT "time of event", `city_code` varchar(20) NULL COMMENT "city_code of user", `pv` bigint(20) SUM NULL DEFAULT "0" COMMENT "total page views", `percent` PERCENTILE PERCENTILE_UNION COMMENT "others" ) ENGINE=OLAP AGGREGATE KEY(`site_id`, `date`, `city_code`) COMMENT "OLAP" DISTRIBUTED BY HASH(`site_id`) PROPERTIES ("replication_num" = "1"); ``` Insert data into the table. ```sql insert into aggregate_tbl values (5, '2020-02-23', 'city_code', 555, percentile_hash(1)); insert into aggregate_tbl values (5, '2020-02-23', 'city_code', 555, percentile_hash(2)); insert into aggregate_tbl values (5, '2020-02-23', 'city_code', 555, percentile_hash(3)); insert into aggregate_tbl values (5, '2020-02-23', 'city_code', 555, percentile_hash(4)); ``` Calculate the value corresponding to percentile 0.5. ```plain mysql> select percentile_approx_raw(percent, 0.5) from aggregate_tbl; +-------------------------------------+ | percentile_approx_raw(percent, 0.5) | +-------------------------------------+ | 2.5 | +-------------------------------------+ 1 row in set (0.03 sec) ``` --- ### percentile_cont ### percentile\_cont Computes the percentile value of `expr` with linear interpolation. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext PERCENTILE_CONT (expr, percentile) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr`: the expression by which to order the values. It must be of numeric data types, DATE, or DATETIME. For example, if you want to find the median score of physics, specify the column that contains the physics scores. * `percentile`: the percentile of the value you want to find. It is a constant floating-point number from 0 to 1. For example, if you want to find the median value, set this parameter to `0.5`. #### Return value[​](#return-value "Direct link to Return value") Returns a value that is at the specified percentile. If no input value lies exactly at the desired percentile, the result is calculated using linear interpolation of the two nearest input values. The data type is the same as `expr`. #### Usage notes[​](#usage-notes "Direct link to Usage notes") This function ignores NULLs. #### Examples[​](#examples "Direct link to Examples") Suppose there is a table named `exam` with the following data. ```plain select * from exam order by Subject; +-----------+-------+ | Subject | Score | +-----------+-------+ | chemistry | 80 | | chemistry | 100 | | chemistry | NULL | | math | 60 | | math | 70 | | math | 85 | | physics | 75 | | physics | 80 | | physics | 85 | | physics | 99 | +-----------+-------+ ``` Calculate the median score of each subject while ignoring NULLs. Query: ```sql SELECT Subject, PERCENTILE_CONT (Score, 0.5) FROM exam group by Subject; ``` Result: ```plain +-----------+-----------------------------+ | Subject | percentile_cont(Score, 0.5) | +-----------+-----------------------------+ | chemistry | 90 | | math | 70 | | physics | 82.5 | +-----------+-----------------------------+ ``` --- ### percentile_disc ### percentile\_disc Returns a percentile value based on a discrete distribution of the input column `expr`. If the exact percentile value cannot be found, this function returns the larger value between the two closest values. This function is supported from v2.5 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```sql PERCENTILE_DISC (expr, percentile) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr`: the column for which you want to calculate the percentile value. The column can be of any data type that is sortable. * `percentile`: the percentile of the value you want to find. It must be a constant floating-point number between 0 and 1. For example, if you want to find the median value, set this parameter to `0.5`. If you want to find the value at the 70th percentile, specify 0.7. #### Return value[​](#return-value "Direct link to Return value") The data type of the return value is the same as `expr`. #### Usage notes[​](#usage-notes "Direct link to Usage notes") NULL values are ignored in the calculation. #### Examples[​](#examples "Direct link to Examples") Create table `exam` and insert data into this table. ```sql CREATE TABLE exam ( subject STRING, score INT ) DISTRIBUTED BY HASH(`subject`); INSERT INTO exam VALUES ('chemistry',80), ('chemistry',100), ('chemistry',null), ('math',60), ('math',70), ('math',85), ('physics',75), ('physics',80), ('physics',85), ('physics',99); ``` ```plain select * from exam order by subject; +-----------+-------+ | subject | score | +-----------+-------+ | chemistry | 80 | | chemistry | 100 | | chemistry | NULL | | math | 60 | | math | 70 | | math | 85 | | physics | 75 | | physics | 80 | | physics | 85 | | physics | 99 | +-----------+-------+ ``` Calculate the median of each subject. ```sql select subject, percentile_disc(score, 0.5) from exam group by subject; ``` Output ```plain +-----------+-----------------------------+ | subject | percentile_disc(score, 0.5) | +-----------+-----------------------------+ | chemistry | 100 | | math | 70 | | physics | 85 | +-----------+-----------------------------+ ``` --- ### percentile_empty Constructs a PERCENTILE value, which is used to fill in null values for data loading using [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) or [INSERT INTO](https://docs.starrocks.io/docs/loading/InsertInto.md). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext PERCENTILE_EMPTY(); ``` #### Parameters[​](#parameters "Direct link to Parameters") None #### Return value[​](#return-value "Direct link to Return value") Returns a PERCENTILE value. #### Examples[​](#examples "Direct link to Examples") Create a table. The `percent` column is a PERCENTILE column. ```sql CREATE TABLE `aggregate_tbl` ( `site_id` largeint(40) NOT NULL COMMENT "id of site", `date` date NOT NULL COMMENT "time of event", `city_code` varchar(20) NULL COMMENT "city_code of user", `pv` bigint(20) SUM NULL DEFAULT "0" COMMENT "total page views", `percent` PERCENTILE PERCENTILE_UNION COMMENT "others" ) ENGINE=OLAP AGGREGATE KEY(`site_id`, `date`, `city_code`) COMMENT "OLAP" DISTRIBUTED BY HASH(`site_id`) PROPERTIES ("replication_num" = "3"); ``` Insert data into the table. ```sql INSERT INTO aggregate_tbl VALUES (5, '2020-02-23', 'city_code', 555, percentile_empty()); ``` --- ### percentile_hash Constructs DOUBLE values as PERCENTILE values. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext PERCENTILE_HASH(x); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: The supported data type is DOUBLE. #### Return value[​](#return-value "Direct link to Return value") Returns a PERCENTILE value. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select percentile_approx_raw(percentile_hash(234.234), 0.99); +-------------------------------------------------------+ | percentile_approx_raw(percentile_hash(234.234), 0.99) | +-------------------------------------------------------+ | 234.23399353027344 | +-------------------------------------------------------+ 1 row in set (0.00 sec) ``` --- ### percentile_union Aggregates PERCENTILE data. #### Syntax[​](#syntax "Direct link to Syntax") ```sql percentile_union(expr); ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr`: The supported data type is PERCENTILE. #### Return value[​](#return-value "Direct link to Return value") Returns a PERCENTILE value. #### Examples[​](#examples "Direct link to Examples") Example 1: Use percentile data in materialized views. Create a table. ```sql CREATE TABLE sales_records( record_id int, seller_id int, store_id int, sale_date date, sale_amt bigint ) distributed BY hash(record_id) PROPERTIES ("replication_num" = "3"); ``` Create a materialized view based on the `sale_amt` column of the table. ```sql create materialized view mv as select store_id, percentile_union(percentile_hash(sale_amt)) from sales_records group by store_id; ``` Example 2: Load PERCENTILE data. Create an Aggregate table that contains a PERCENTILE column `sale_amt_per`. ```sql CREATE TABLE sales_records( record_id int, seller_id int, store_id int, sale_amt_per percentile percentile_union ) ENGINE=OLAP AGGREGATE KEY(`record_id`, `seller_id`, `store_id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`record_id`) PROPERTIES ( "replication_num" = "3", "storage_format" = "DEFAULT" ); ``` Query data from `sale_amt_per`. ```sql select percentile_approx_raw(percentile_union(sale_amt_per), 0.99) from sales_records; ``` Load data that contains PERCENTILE data into the `sales_records` table. ```sql curl --location-trusted -u root -H "columns: record_id, seller_id, store_id,tmp, sale_amt_per =percentile_hash(tmp)" -H "column_separator:," -T a http:///api/test/sales_records/_stream_load ``` --- ### Python UDF Experimental feature [Advice on use of experimental features](https://docs.starrocks.io/docs/maturity/features.md) This topic describes how to develop User Defined Functions (UDF) using Python. Since v3.4.0, StarRocks supports creating Python UDFs. Currently, StarRocks only supports scalar UDFs in Python. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Make sure the following requirements are met before proceeding: * Install [Python 3.8](https://www.python.org/downloads/release/python-380/) or later. * Enable UDF in StarRocks by setting `enable_udf` to `true` in the FE configuration file **fe/conf/fe.conf**, and then restart the FE nodes to allow the configuration to take effect. For more information, see [FE configuration - enable\_udf](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#enable_udf). * Python UDF requires the following packages: pyarrow * Set the location of the Python interpreter environment in the BE instance using environment variable. Add the variable item `python_envs` and set it to the location of the Python interpreter installation, for example, `/opt/Python-3.8/`. #### Develop and use Python UDFs[​](#develop-and-use-python-udfs "Direct link to Develop and use Python UDFs") Syntax: ```sql CREATE [GLOBAL] FUNCTION function_name(arg_type [, ...]) RETURNS return_type {PROPERTIES ("key" = "value" [, ...]) | key="value" [...] } [AS $$ $$] ``` | **Parameter** | **Required** | **Description** | | -------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GLOBAL | No | Whether to create a global UDF. | | function\_name | Yes | The name of the function you want to create. You can include the name of the database in this parameter, for example,`db1.my_func`. If `function_name` includes the database name, the UDF is created in that database. Otherwise, the UDF is created in the current database. The name of the new function and its parameters cannot be the same as an existing name in the destination database. Otherwise, the function cannot be created. The creation succeeds if the function name is the same but the parameters are different. | | arg\_type | Yes | Argument type of the function. The added argument can be represented by `, ...`. For the supported data types, see [Mapping between SQL data types and Python data types](#mapping-between-sql-data-types-and-python-data-types). | | return\_type | Yes | The return type of the function. For the supported data types, see [Mapping between SQL data types and Python data types](#mapping-between-sql-data-types-and-python-data-types). | | PROPERTIES | Yes | Properties of the function, which vary depending on the type of the UDF to create. | | AS $$ $$ | No | Specify the inline UDF code between `$$` marks. | Properties include: | **Property** | **Required** | **Description** | | ------------ | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | type | Yes | The type of the UDF. Setting it to `Python` indicates to create a Python-based UDF. | | symbol | Yes | The name of the class for the Python project to which the UDF belongs. The value of this parameter is in the `.` format. | | input | No | Type of input. Valid values: `scalar` (Default) and `arrow` (vectorized input). | | file | No | The HTTP URL from which you can download the Python package file that contains the code for the UDF. The value of this parameter is in the `http://:/` format. Note the package name must have the suffix `.py.zip`. Default value: `inline`, which indicates to create an inline UDF. | ##### Create an inline scalar input UDF using Python[​](#create-an-inline-scalar-input-udf-using-python "Direct link to Create an inline scalar input UDF using Python") The following example creates an inline `echo` function with scalar input using Python. ```sql CREATE FUNCTION python_echo(INT) RETURNS INT type = 'Python' symbol = 'echo' file = 'inline' AS $$ def echo(x): return x $$ ; ``` ##### Create an inline vectorized input UDF using Python[​](#create-an-inline-vectorized-input-udf-using-python "Direct link to Create an inline vectorized input UDF using Python") Vectorized input is supported to enhance the UDF processing performance. The following example creates an inline `add` function with vectorized input using Python. ```sql CREATE FUNCTION python_add(INT) RETURNS INT type = 'Python' symbol = 'add' input = "arrow" AS $$ import pyarrow.compute as pc def add(x): return pc.add(x, 1) $$ ; ``` ##### Create a packaged UDF using Python[​](#create-a-packaged-udf-using-python "Direct link to Create a packaged UDF using Python") When creating the Python package, you must package the module into `.py.zip` files, which need to meet the [zipimport format](https://docs.python.org/3/library/zipimport.html). ```plain > tree . . ├── main.py └── yaml ├── composer.py ├── constructor.py ├── cyaml.py ├── dumper.py ├── emitter.py ├── error.py ├── events.py ├── __init__.py ├── loader.py ├── nodes.py ├── parser.py ``` ```plain > cat main.py import numpy import yaml def echo(a): return yaml.__version__ ``` ```sql CREATE FUNCTION py_pack(string) RETURNS string symbol = "add" type = "Python" file = "http://HTTP_IP:HTTP_PORT/m1.py.zip" symbol = "main.echo" ; ``` #### Appendix[​](#appendix "Direct link to Appendix") ##### Mapping between SQL data types and Python data types[​](#mapping-between-sql-data-types-and-python-data-types "Direct link to Mapping between SQL data types and Python data types") | SQL Type | Python 3 Type | | ------------------------------------ | ------------------------------- | | **SCALAR** | | | TINYINT/SMALLINT/INT/BIGINT/LARGEINT | INT | | STRING | string | | DOUBLE | FLOAT | | BOOLEAN | BOOL | | DATETIME | DATETIME.DATETIME | | FLOAT | FLOAT | | CHAR | STRING | | VARCHAR | STRING | | DATE | DATETIME.DATE | | DECIMAL | DECIMAL.DECIMAL | | ARRAY | list | | MAP | dict | | STRUCT | dict (keyed by field name) | | JSON | str (use `json.loads` to parse) | | **VECTORIZED** | | | TYPE\_BOOLEAN | pyarrow.lib.BoolArray | | TYPE\_TINYINT | pyarrow.lib.Int8Array | | TYPE\_SMALLINT | pyarrow.lib.Int15Array | | TYPE\_INT | pyarrow.lib.Int32Array | | TYPE\_BIGINT | pyarrow.lib.Int64Array | | TYPE\_FLOAT | pyarrow.FloatArray | | TYPE\_DOUBLE | pyarrow.DoubleArray | | VARCHAR | pyarrow.StringArray | | DECIMAL | pyarrow.Decimal128Array | | DATE | pyarrow.Date32Array | | TYPE\_TIME | pyarrow.TimeArray | | ARRAY | pyarrow.ListArray | | MAP | pyarrow.MapArray | | STRUCT | pyarrow.StructArray | Nested types (`ARRAY`, `MAP`, `STRUCT`) may be composed arbitrarily — for example, `array>`, `map>`, `struct, b map>`, or `array>`. In `scalar` mode each row is delivered as plain Python `list` / `dict` values recursively. To return a nested type, return matching Python `list` / `dict` values (with `dict` keys equal to struct field names); `pyarrow` performs the recursive conversion automatically. ##### Compile Python[​](#compile-python "Direct link to Compile Python") Follow these steps to compile Python: 1. Get the OpenSSL package. ```bash wget 'https://github.com/openssl/openssl/archive/OpenSSL_1_1_1m.tar.gz' ``` 2. Decompress the package. ```bash tar -zxf OpenSSL_1_1_1m.tar.gz ``` 3. Navigate to the decompressed folder. ```bash cd openssl-OpenSSL_1_1_1m ``` 4. Set the environment variable `OPENSSL_DIR`. ```bash export OPENSSL_DIR=`pwd`/install ``` 5. Prepare source code for compilation. ```bash ./Configure --prefix=`pwd`/install ./config --prefix=`pwd`/install ``` 6. Compile OpenSSL. ```bash make -j 16 && make install ``` 7. Set the environment variable `LD_LIBRARY_PATH`. ```bash LD_LIBRARY_PATH=$OPENSSL_DIR/lib:$LD_LIBRARY_PATH ``` 8. Navigate back to the working directory and get the Python package. ```bash wget 'https://www.python.org/ftp/python/3.12.9/Python-3.12.9.tgz' ``` 9. Decompress the package. ```bash tar -zxf ./Python-3.12.9.tgz ``` 10. Navigate to the decompressed folder. ```bash cd Python-3.12.9 ``` 11. Make and navigate to the directory `build`. ```bash mkdir build && cd build ``` 12. Prepare source code for compilation. ```bash ../configure --prefix=`pwd`/install --with-openssl=$OPENSSL_DIR ``` 13. Compile Python. ```bash make -j 16 && make install ``` 14. Install PyArrow and grpcio. ```bash ./install/bin/pip3 install pyarrow grpcio ``` 15. Compress the files into the package. ```bash tar -zcf ./Python-3.12.9.tar.gz install ``` 16. Distribute the package to the target BE server, and decompress the package. ```bash tar -zxf ./Python-3.12.9.tar.gz ``` 17. Modify the BE configuration file **be.conf** to add the following configuration item. ```properties python_envs=/home/disk1/sr/install ``` --- ### Scalar #### [📄️ hll\_cardinality](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_cardinality.md) [Calculates the cardinality of a single HLL type value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_cardinality.md) #### [📄️ hll\_empty](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_empty.md) [Generates an empty HLL column to supplement the default values when inserting or loading data.](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_empty.md) #### [📄️ hll\_hash](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_hash.md) [Converts a value to an hll type.](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_hash.md) --- ### hll_cardinality Calculates the cardinality of a single HLL type value. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext HLL_CARDINALITY(hll) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select HLL_CARDINALITY(uv_set) from test_uv; +---------------------------+ | hll_cardinality(`uv_set`) | +---------------------------+ | 3 | +---------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") HLL,HLL\_CARDINALITY --- ### hll_empty Generates an empty HLL column to supplement the default values when inserting or loading data. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext HLL_EMPTY() ``` #### Return value[​](#return-value "Direct link to Return value") Returns an empty HLL. #### Examples[​](#examples "Direct link to Examples") Supplement the default values when inserting data. ```plain insert into hllDemo(k1,v1) values(10,hll_empty()); ``` Supplement the default values when loading data. ```plain curl --location-trusted -u : \ -H "columns: temp1, temp2, col1=hll_hash(temp1), col2=hll_empty()" \ -T example7.csv -XPUT \ http://:/api/test_db/table7/_stream_load ``` --- ### hll_hash Converts a value to an hll type. Typically used in imports to map a value in the source data to an HLL column type in the StarRocks table. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext HLL_HASH(column_name) ``` #### Parameters[​](#parameters "Direct link to Parameters") `column_name`: The name of the generated HLL column. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the HLL type. #### Example[​](#example "Direct link to Example") ```plain mysql> select hll_cardinality(hll_hash("a")); +--------------------------------+ | hll_cardinality(hll_hash('a')) | +--------------------------------+ | 1 | +--------------------------------+ ``` --- ### Spatial #### [📄️ ST\_AsText,ST\_AsWKT](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_astext.md) [Converts a geometric figure to WKT (Well Known Text) format.](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_astext.md) #### [📄️ ST\_Circle](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_circle.md) [Converts a WKT (WEll Known Text) to a circle on the sphere of the earth.](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_circle.md) #### [📄️ ST\_Contains](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_contains.md) [Checks whether the geometric figure shape1 can fully contain shape2.](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_contains.md) #### [📄️ ST\_Distance\_Sphere](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_distance_sphere.md) [Calculates the spherical distance between two points on the Earth in 'meters'.](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_distance_sphere.md) #### [📄️ ST\_GeometryFromText,ST\_GeomFromText](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_geometryfromtext.md) [Converts a WKT (Well Known Text) to the corresponding memory geometry.](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_geometryfromtext.md) #### [📄️ ST\_LineFromText,ST\_LineStringFromText](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_linefromtext.md) [Converts a WKT (Well Known Text) to a memory representation in the form of Line.](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_linefromtext.md) #### [📄️ ST\_Point](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_point.md) [Returns the corresponding Point with the given X-coordinate and Y-coordinate.](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_point.md) #### [📄️ ST\_Polygon,ST\_PolyFromText,ST\_PolygonFromText](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_polygon.md) [Converts a WKT (Well Known Text) to a corresponding polygon memory form.](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_polygon.md) #### [📄️ ST\_X](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_x.md) [If point is of a valid Point type, return the corresponding X-coordinate value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_x.md) #### [📄️ ST\_Y](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_y.md) [If point is of a valid Point type, return the corresponding Y-coordinate value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/spatial-functions/st_y.md) --- ### ST_AsText,ST_AsWKT Converts a geometric figure to WKT (Well Known Text) format. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR ST_AsText(GEOMETRY geo) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT ST_AsText(ST_Point(24.7, 56.7)); +---------------------------------+ | st_astext(st_point(24.7, 56.7)) | +---------------------------------+ | POINT (24.7 56.7) | +---------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ST\_ASTEXT,ST\_ASWKT,ST,ASTEXT,ASWKT --- ### ST_Circle Converts a WKT (WEll Known Text) to a circle on the sphere of the earth. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext GEOMETRY ST_Circle(DOUBLE center_lng, DOUBLE center_lat, DOUBLE radius) ``` #### Parameters[​](#parameters "Direct link to Parameters") `center_lng` indicates the longitude of the center of the circle. `center_lat` indicates the latitude of the center of the circle. `radius` indicates the radius of a circle, in meters. A maximum of 9999999 radius is supported. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT ST_AsText(ST_Circle(111, 64, 10000)); +--------------------------------------------+ | st_astext(st_circle(111.0, 64.0, 10000.0)) | +--------------------------------------------+ | CIRCLE ((111 64), 10000) | +--------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ST\_CIRCLE,ST,CIRCLE --- ### ST_Contains Checks whether the geometric figure shape1 can fully contain shape2. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BOOL ST_Contains(GEOMETRY shape1, GEOMETRY shape2) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT ST_Contains(ST_Polygon("POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))"), ST_Point(5, 5)); +----------------------------------------------------------------------------------------+ | st_contains(st_polygon('POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))'), st_point(5.0, 5.0)) | +----------------------------------------------------------------------------------------+ | 1 | +----------------------------------------------------------------------------------------+ MySQL > SELECT ST_Contains(ST_Polygon("POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))"), ST_Point(50, 50)); +------------------------------------------------------------------------------------------+ | st_contains(st_polygon('POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))'), st_point(50.0, 50.0)) | +------------------------------------------------------------------------------------------+ | 0 | +------------------------------------------------------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ST\_CONTAINS,ST,CONTAINS --- ### ST_Distance_Sphere Calculates the spherical distance between two points on the Earth in "meters". The parameters inputted are longitude at X, latitude at X, longitude at Y, latitude at Y. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DOUBLE ST_Distance_Sphere(DOUBLE x_lng, DOUBLE x_lat, DOUBLE y_lng, DOUBLE x_lat) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select st_distance_sphere(116.35620117, 39.939093, 116.4274406433, 39.9020987219); +----------------------------------------------------------------------------+ | st_distance_sphere(116.35620117, 39.939093, 116.4274406433, 39.9020987219) | +----------------------------------------------------------------------------+ | 7336.9135549995917 | +----------------------------------------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ST\_DISTANCE\_SPHERE,ST,DISTANCE,SPHERE --- ### ST_GeometryFromText,ST_GeomFromText Converts a WKT (Well Known Text) to the corresponding memory geometry. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext GEOMETRY ST_GeometryFromText(VARCHAR wkt) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT ST_AsText(ST_GeometryFromText("LINESTRING (1 1, 2 2)")); +---------------------------------------------------------+ | st_astext(st_geometryfromtext('LINESTRING (1 1, 2 2)')) | +---------------------------------------------------------+ | LINESTRING (1 1, 2 2) | +---------------------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ST\_GEOMETRYFROMTEXT,ST\_GEOMFROMTEXT,ST,GEOMETRYFROMTEXT,GEOMFROMTEXT --- ### ST_LineFromText,ST_LineStringFromText Converts a WKT (Well Known Text) to a memory representation in the form of Line. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext GEOMETRY ST_LineFromText(VARCHAR wkt) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT ST_AsText(ST_LineFromText("LINESTRING (1 1, 2 2)")); +---------------------------------------------------------+ | st_astext(st_linefromtext('LINESTRING (1 1, 2 2)')) | +---------------------------------------------------------+ | LINESTRING (1 1, 2 2) | +---------------------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ST\_LINEFROMTEXT,ST\_LINESTRINGFROMTEXT,ST,LINEFROMTEXT,LINESTRINGFROMTEXT --- ### ST_Point Returns the corresponding Point with the given X-coordinate and Y-coordinate. At the moment this value only makes sense on a spherical set. X/Y corresponds to longitude/latitude. > **Caution** > > If you directly select ST\_Point(), it may get stuck. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext POINT ST_Point(DOUBLE x, DOUBLE y) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT ST_AsText(ST_Point(24.7, 56.7)); +---------------------------------+ | st_astext(st_point(24.7, 56.7)) | +---------------------------------+ | POINT (24.7 56.7) | +---------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ST\_POINT,ST,POINT --- ### ST_Polygon,ST_PolyFromText,ST_PolygonFromText Converts a WKT (Well Known Text) to a corresponding polygon memory form. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext GEOMETRY ST_Polygon(VARCHAR wkt) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT ST_AsText(ST_Polygon("POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))")); +------------------------------------------------------------------+ | st_astext(st_polygon('POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))')) | +------------------------------------------------------------------+ | POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0)) | +------------------------------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ST\_POLYGON,ST\_POLYFROMTEXT,ST\_POLYGONFROMTEXT,ST,POLYGON,POLYFROMTEXT,POLYGONFROMTEXT --- ### ST_X If point is of a valid Point type, return the corresponding X-coordinate value. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DOUBLE ST_X(POINT point) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT ST_X(ST_Point(24.7, 56.7)); +----------------------------+ | st_x(st_point(24.7, 56.7)) | +----------------------------+ | 24.7 | +----------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ST\_X,ST,X --- ### ST_Y If point is of a valid Point type, return the corresponding Y-coordinate value. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext DOUBLE ST_Y(POINT point) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT ST_Y(ST_Point(24.7, 56.7)); +----------------------------+ | st_y(st_point(24.7, 56.7)) | +----------------------------+ | 56.7 | +----------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ST\_Y,ST,Y --- ### SQL UDF Since version 4.1, StarRocks supports creating SQL user-define-functions(UDFs), allowing users to create functions using SQL expressions. SQL UDF is a lightweight function definition method that encapsulates a SQL expression into a reusable function, which is dynamically expanded into the actual SQL expression at query time. #### Overview[​](#overview "Direct link to Overview") Main features of SQL UDF: * **Dynamic Expansion**: Expands function calls into actual SQL expressions during the query optimization phase * **Type Safety**: Supports explicit declaration of parameter types and return types * **Parameterized**: Supports named parameters to improve function readability Applicable scenarios: * Complex SQL expressions that need to be reused * Simple data transformation and calculation logic * Standardized data processing rules #### Syntax[​](#syntax "Direct link to Syntax") ##### Creating SQL UDF[​](#creating-sql-udf "Direct link to Creating SQL UDF") ```sql CREATE [GLOBAL] FUNCTION function_name(arg1_name arg1_type, arg2_name arg2_type, ...) RETURNS expression ``` ###### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | -------------- | ----------------------------------------------------------------------------------- | | GLOBAL | Optional. If specified, creates a global function visible to all databases | | function\_name | Function name. Can include database name, e.g., `db1.my_func` | | arg\_name | Parameter name used to reference in the expression | | arg\_type | Parameter type, supports all StarRocks basic data types | | expression | SQL expression that will be expanded into actual expression when function is called | ##### Dropping SQL UDF[​](#dropping-sql-udf "Direct link to Dropping SQL UDF") ```sql DROP FUNCTION [IF EXISTS] function_name(arg_type [, ...]) ``` ##### Showing SQL UDF[​](#showing-sql-udf "Direct link to Showing SQL UDF") ```sql SHOW [GLOBAL] FUNCTIONS; ``` #### Usage Examples[​](#usage-examples "Direct link to Usage Examples") ##### Example 1: String Processing Function[​](#example-1-string-processing-function "Direct link to Example 1: String Processing Function") ```sql -- Create a function that converts string to uppercase and adds a prefix CREATE FUNCTION format_username(name STRING) RETURNS concat('USER_', upper(name)); -- Use the function SELECT format_username('alice'); -- Result: USER_ALICE -- Use in queries SELECT format_username(username) as display_name FROM users; ``` ##### Example 2: Multi-parameter Calculation Function[​](#example-2-multi-parameter-calculation-function "Direct link to Example 2: Multi-parameter Calculation Function") ```sql -- Create a function that calculates discounted price CREATE FUNCTION calculate_discount_price(original_price DECIMAL(10,2), discount_rate DOUBLE) RETURNS original_price * (1 - discount_rate); -- Use the function SELECT calculate_discount_price(100.00, 0.2); -- Result: 80.00 SELECT calculate_discount_price(price, 0.15) as final_price FROM products; ``` ##### Example 3: Complex Expression Encapsulation[​](#example-3-complex-expression-encapsulation "Direct link to Example 3: Complex Expression Encapsulation") ```sql -- Create a function that extracts JSON and transforms CREATE FUNCTION extract_user_info(json_str STRING, field_name STRING) RETURNS get_json_string(get_json_string(json_str, concat('$.', field_name)), '$.value'); -- Simplify complex nested calls SELECT extract_user_info(user_data, 'email') as user_email FROM events; ``` ##### Example 4: Conditional Logic Function[​](#example-4-conditional-logic-function "Direct link to Example 4: Conditional Logic Function") ```sql -- Create a function with conditional logic CREATE FUNCTION classify_temperature(temp DOUBLE) RETURNS CASE WHEN temp >= 30 THEN 'hot' WHEN temp >= 20 THEN 'warm' WHEN temp >= 10 THEN 'cool' ELSE 'cold' END; -- Use the function SELECT classify_temperature(25); -- Result: warm ``` ##### Example 5: Global SQL UDF[​](#example-5-global-sql-udf "Direct link to Example 5: Global SQL UDF") ```sql -- Create a global function visible to all databases CREATE GLOBAL FUNCTION format_date_display(dt DATETIME) RETURNS concat(year(dt), '-', lpad(month(dt), 2, '0'), '-', lpad(day(dt), 2, '0')); -- Can be used directly in any database SELECT format_date_display(create_time) from my_table; ``` #### Advanced Features[​](#advanced-features "Direct link to Advanced Features") ##### 1. Nested Usage[​](#1-nested-usage "Direct link to 1. Nested Usage") SQL UDF supports nested calls: ```sql CREATE FUNCTION func_a(x INT, y INT) RETURNS x + y; CREATE FUNCTION func_b(a INT, b INT) RETURNS func_a(a, b) * 2; SELECT func_b(3, 4); -- Result: 14 ``` ##### 2. Type Conversion[​](#2-type-conversion "Direct link to 2. Type Conversion") SQL UDF supports implicit type conversion: ```sql CREATE FUNCTION convert_and_add(a STRING, b INT) RETURNS cast(a AS INT) + b; SELECT convert_and_add('100', 50); -- Result: 150 ``` ##### 3. Combination with Built-in Functions[​](#3-combination-with-built-in-functions "Direct link to 3. Combination with Built-in Functions") SQL UDF can be freely combined with StarRocks built-in functions: ```sql CREATE FUNCTION get_year_month(dt DATETIME) RETURNS concat(year(dt), '-', lpad(month(dt), 2, '0')); SELECT get_year_month(create_time) as ym FROM events GROUP BY ym; ``` #### Viewing and Managing[​](#viewing-and-managing "Direct link to Viewing and Managing") ##### Showing All Functions[​](#showing-all-functions "Direct link to Showing All Functions") ```sql -- Show functions in current database SHOW FUNCTIONS; -- Show all global functions SHOW GLOBAL FUNCTIONS; ``` ##### Dropping Functions[​](#dropping-functions "Direct link to Dropping Functions") ```sql -- Drop function in current database DROP FUNCTION format_username(STRING); -- Drop global function DROP GLOBAL FUNCTION format_date_display(DATETIME); -- Use IF EXISTS to avoid errors DROP FUNCTION IF EXISTS my_function(INT, STRING); ``` ##### Replacing Functions[​](#replacing-functions "Direct link to Replacing Functions") Use the `OR REPLACE` keyword to replace an existing function: ```sql -- Modify function definition CREATE OR REPLACE FUNCTION calculate_tax(amount DECIMAL(10,2)) RETURNS amount * 0.1 + 5.0; ``` #### Performance Characteristics[​](#performance-characteristics "Direct link to Performance Characteristics") SQL UDF is expanded into actual SQL expressions during the query optimization phase, with the following performance characteristics: 1. **Zero Function Call Overhead**: No function call overhead like traditional UDFs 2. **Optimizer Visibility**: The optimizer can see the complete expression for better optimization 3. **Predicate Pushdown**: Supports predicate pushdown and other optimization rules For example, the following query: ```sql CREATE FUNCTION my_func(x INT) RETURNS x * 2 + 1; SELECT * FROM t WHERE my_func(a) > 10; ``` Will be expanded by the optimizer as: ```sql SELECT * FROM t WHERE a * 2 + 1 > 10; ``` #### Limitations and Considerations[​](#limitations-and-considerations "Direct link to Limitations and Considerations") 1. **Parameter Count Limitation**: Parameter count should match variables used in expression 2. **Type Matching**: Parameter types should match usage in expression --- ### String functions #### [📄️ append\_trailing\_char\_if\_absent](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/append_trailing_char_if_absent.md) [If the str string is not empty and does not contain trailing\_char character in the end, it appends trailing\_char character to the end.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/append_trailing_char_if_absent.md) #### [📄️ ascii](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/ascii.md) [Returns the ASCII code of the leftmost character of a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/ascii.md) #### [📄️ char](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/char.md) [CHAR() returns the character value of the given integer value according to the ASCII table.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/char.md) #### [📄️ char\_length](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/char_length.md) [Returns the number of characters in a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/char_length.md) #### [📄️ character\_length](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/character_length.md) [Alias of char\_length; returns the number of characters in a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/character_length.md) #### [📄️ concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/concat.md) [Concatenates multiple strings into a single string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/concat.md) #### [📄️ concat\_ws](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/concat_ws.md) [Concatenates strings using a specified separator between each argument.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/concat_ws.md) #### [📄️ crc32](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/crc32.md) [Returns the 32-bit cyclic redundancy check (CRC) value of a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/crc32.md) #### [📄️ ends\_with](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/ends_with.md) [Returns true if a string ends with a specified suffix.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/ends_with.md) #### [📄️ field](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/field.md) [Returns the index (position) of a value in a list of values.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/field.md) #### [📄️ find\_in\_set](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/find_in_set.md) [Returns the 1-based position of a string within a comma-separated list.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/find_in_set.md) #### [📄️ format\_bytes](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/format_bytes.md) [Converts a byte count into a human-readable string with appropriate units (B, KB, MB, GB, TB, PB, EB).](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/format_bytes.md) #### [📄️ group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat.md) [Concatenates non-null values from a group into a single string, with a sep argument, which is , by default if not specified.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat.md) #### [📄️ hex](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/hex.md) [Converts a numeric or string value to its hexadecimal representation.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/hex.md) #### [📄️ hex\_decode\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/hex_decode_binary.md) [Decodes a hex encoded string to a binary.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/hex_decode_binary.md) #### [📄️ hex\_decode\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/hex_decode_string.md) [Converts a hexadecimal-encoded string back to its original string value; inverse of hex.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/hex_decode_string.md) #### [📄️ inet\_aton](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/inet_aton.md) [Takes a string containing an IPv4 address in the format A.B.C.D.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/inet_aton.md) #### [📄️ initcap](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/initcap.md) [Converts the first letter of each word in a string to uppercase and the remaining characters to lowercase.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/initcap.md) #### [📄️ instr](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/instr.md) [Returns the 1-based character position of the first occurrence of a substring within a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/instr.md) #### [📄️ lcase](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/lcase.md) [Converts a string to lowercase; alias of lower.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/lcase.md) #### [📄️ left](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/left.md) [Returns a specified number of characters from the left side of a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/left.md) #### [📄️ length](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/length.md) [Returns the byte length of a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/length.md) #### [📄️ locate](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/locate.md) [Returns the 1-based character position of a substring within a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/locate.md) #### [📄️ lower](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/lower.md) [Converts all strings in an argument to lower-case.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/lower.md) #### [📄️ lpad](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/lpad.md) [Left-pads a string with a pad string to reach the specified length.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/lpad.md) #### [📄️ ltrim](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/ltrim.md) [Removes the leading spaces or specified characters from the beginning (left) of the str argument.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/ltrim.md) #### [📄️ money\_format](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/money_format.md) [Formats a numeric value as a currency string with two decimal places and thousands separators.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/money_format.md) #### [📄️ ngram\_search](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/ngram_search.md) [Calculate the ngram similarity of the two strings.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/ngram_search.md) #### [📄️ null\_or\_empty](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/null_or_empty.md) [Returns true if the string is NULL or an empty string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/null_or_empty.md) #### [📄️ parse\_url](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/parse_url.md) [Parses a URL string and extracts a specified component such as host, path, or query parameter.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/parse_url.md) #### [📄️ regexp\_split](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/regexp_split.md) [Split string str by regexp expression pattern, return maximum max\_split elements in ARRAY\ type.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/regexp_split.md) #### [📄️ repeat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/repeat.md) [Returns a string repeated a specified number of times.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/repeat.md) #### [📄️ replace](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/replace.md) [Replaces all occurrences of characters in a string with another string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/replace.md) #### [📄️ reverse](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/reverse.md) [Reverses a string or array.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/reverse.md) #### [📄️ right](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/right.md) [Returns a specified number of characters from the right side of a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/right.md) #### [📄️ rpad](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/rpad.md) [Right-pads a string with a pad string to reach the specified length.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/rpad.md) #### [📄️ rtrim](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/rtrim.md) [Removes the trailing spaces or specified characters from the end (right) of the str argument.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/rtrim.md) #### [📄️ space](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/space.md) [Returns a string of the specified number of spaces.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/space.md) #### [📄️ split](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/split.md) [Splits a string by a delimiter and returns the resulting parts as an ARRAY.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/split.md) #### [📄️ split\_part](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/split_part.md) [Splits a string by a delimiter and returns the Nth part.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/split_part.md) #### [📄️ starts\_with](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/starts_with.md) [Returns 1 if a string starts with the specified prefix, or 0 otherwise.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/starts_with.md) #### [📄️ str\_to\_map](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/str_to_map.md) [Splits a given string into key-value pairs using two delimiters and returns a map of the split pairs.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/str_to_map.md) #### [📄️ strcmp](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/strcmp.md) [Compares two strings and returns 0 if equal, -1 if the first is smaller, or 1 if larger.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/strcmp.md) #### [📄️ strleft](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/strleft.md) [Returns a specified number of characters from the left of a string; alias of left.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/strleft.md) #### [📄️ strpos](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/strpos.md) [Returns the position of the N-th substring in a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/strpos.md) #### [📄️ strright](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/strright.md) [Returns a specified number of characters from the right of a string; alias of right.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/strright.md) #### [📄️ substring, substr](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/substring.md) [Extracts characters staring from the specified position and returns a substring of specified length.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/substring.md) #### [📄️ substring\_index](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/substring_index.md) [Extracts a substring that precedes or follows the count occurrences of the delimiter.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/substring_index.md) #### [📄️ tokenize](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/tokenize.md) [Splits and parses text into tokens based on the specified tokenizer.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/tokenize.md) #### [📄️ translate](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/translate.md) [Substitutes specified characters within a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/translate.md) #### [📄️ trim](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/trim.md) [Removes consecutive spaces or specified characters from the beginning and end of the str argument.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/trim.md) #### [📄️ ucase](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/ucase.md) [Converts a string to uppercase; alias of upper.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/ucase.md) #### [📄️ unhex](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/unhex.md) [Converts a hexadecimal string to its original binary string; inverse of hex.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/unhex.md) #### [📄️ upper](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/upper.md) [Converts a string to upper-case.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/upper.md) #### [📄️ url\_decode](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/url_decode.md) [Translates a string back from the application/x-www-form-urlencoded format.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/url_decode.md) #### [📄️ url\_encode](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/url_encode.md) [Translates a string into the application/x-www-form-urlencoded format.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/url_encode.md) #### [📄️ url\_extract\_host](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/url_extract_host.md) [Extracts the host section from a URL.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/url_extract_host.md) #### [📄️ url\_extract\_parameter](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/url_extract_parameter.md) [Extracts the value of the requested name parameter from the query string of a URL.](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/url_extract_parameter.md) --- ### append_trailing_char_if_absent If the str string is not empty and does not contain trailing\_char character in the end, it appends trailing\_char character to the end. trailing\_char can only contain one character. If it contains multiple characters, this function will return NULL. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR append_trailing_char_if_absent(VARCHAR str, VARCHAR trailing_char) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL [test]> select append_trailing_char_if_absent('a','c'); +------------------------------------------+ |append_trailing_char_if_absent('a', 'c') | +------------------------------------------+ | ac | +------------------------------------------+ 1 row in set (0.02 sec) MySQL [test]> select append_trailing_char_if_absent('ac','c'); +-------------------------------------------+ |append_trailing_char_if_absent('ac', 'c') | +-------------------------------------------+ | ac | +-------------------------------------------+ 1 row in set (0.00 sec) ``` #### keyword[​](#keyword "Direct link to keyword") APPEND\_TRAILING\_CHAR\_IF\_ABSENT --- ### ascii This function returns the ascii value of the leftmost character of a given string. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT ascii(VARCHAR str) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select ascii('1'); +------------+ | ascii('1') | +------------+ | 49 | +------------+ MySQL > select ascii('234'); +--------------+ | ascii('234') | +--------------+ | 50 | +--------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ASCII --- ### char CHAR() returns the character value of the given integer value according to the ASCII table. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext char(n) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `n`: integer value #### Return value[​](#return-value "Direct link to Return value") Returns a VARCHAR value. #### Examples[​](#examples "Direct link to Examples") ```plain > select char(77); +----------+ | char(77) | +----------+ | M | +----------+ ``` #### keyword[​](#keyword "Direct link to keyword") CHAR --- ### char_length This function returns the length of a string. For multi-byte characters, it returns the number of characters. It currently only supports utf8 coding. Note: This function is also named as character\_length. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT char_length(VARCHAR str) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select char_length("abc"); +--------------------+ | char_length('abc') | +--------------------+ | 3 | +--------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") CHAR\_LENGTH, CHARACTER\_LENGTH --- ### character_length This function is an alias of `char_length`, please check [char\_length](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/char_length.md) function for detailed explanation. #### keyword[​](#keyword "Direct link to keyword") CHAR\_LENGTH,CHARACTER\_LENGTH --- ### concat This function combines multiple strings. If any of the parameter value is NULL, it will return NULL. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR concat(VARCHAR,...) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select concat("a", "b"); +------------------+ | concat('a', 'b') | +------------------+ | ab | +------------------+ MySQL > select concat("a", "b", "c"); +-----------------------+ | concat('a', 'b', 'c') | +-----------------------+ | abc | +-----------------------+ MySQL > select concat("a", null, "c"); +------------------------+ | concat('a', NULL, 'c') | +------------------------+ | NULL | +------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") CONCAT --- ### concat_ws This function uses the first argument sep as the separator which combines the second with the rest to form a string. If the separator is NULL, the result is NULL. concat\_ws does not skip empty strings, but it will skip NULL values. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR concat_ws(VARCHAR sep, VARCHAR str,...) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select concat_ws("Rock", "Star", "s"); +--------------------------------+ | concat_ws('Rock', 'Star', 's') | +--------------------------------+ | StarRocks | +--------------------------------+ MySQL > select concat_ws(NULL, "Star", "s"); +------------------------------+ | concat_ws(NULL, 'Star', 's') | +------------------------------+ | NULL | +------------------------------+ 1 row in set (0.01 sec) MySQL > StarRocks > select concat_ws("Rock", "Star", NULL, "s"); +--------------------------------------+ | concat_ws('Rock', 'Star', NULL, 's') | +--------------------------------------+ | StarRocks | +--------------------------------------+ 1 row in set (0.04 sec) ``` #### keyword[​](#keyword "Direct link to keyword") CONCAT\_WS,CONCAT,WS --- ### crc32 Returns the 32-bit cyclic redundancy check (CRC) value of a string. If the input is NULL, NULL is returned. This function is used for error detection. It uses the CRC32 algorithm to detect changes between source and target data. This function is supported from v3.3 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT crc32(VARCHAR str) ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: the string for which you want to calculate the CRC32 value. #### Examples[​](#examples "Direct link to Examples") ```plain mysql > select crc32("starrocks"); +--------------------+ | crc32('starrocks') | +--------------------+ | 2312449062 | +--------------------+ mysql > select crc32(null); +-------------+ | crc32(NULL) | +-------------+ | NULL | +-------------+ 1 row in set (0.18 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") CRC32 --- ### ends_with Returns `true` if a string ends with a specified suffix. Otherwise, it returns `false`. If the argument is NULL, the result is NULL. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BOOLEAN ENDS_WITH (VARCHAR str, VARCHAR suffix) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select ends_with("Hello starrocks", "starrocks"); +-----------------------------------+ | ends_with('Hello starrocks', 'starrocks') | +-----------------------------------+ | 1 | +-----------------------------------+ MySQL > select ends_with("Hello starrocks", "Hello"); +-----------------------------------+ | ends_with('Hello starrocks', 'Hello') | +-----------------------------------+ | 0 | +-----------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") ENDS\_WITH --- ### field Returns the index (position) of a value in a list of values. This function is supported from v3.5. #### Syntax[​](#syntax "Direct link to Syntax") ```sql INT field(VARCHAR val, VARCHAR val1, val2, ...); ``` * `val`: The value to search for in the list. * `val1`, `val2`, ...: The values in the list. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * If the specified value is not found in the list of values, this function will return `0`. If the specified value is NULL, this function will return `0`. * If all arguments to the function are strings, all arguments are compared as strings. If all arguments are numbers, they are compared as numbers. Otherwise, the arguments are compared as DOUBLE. #### Examples[​](#examples "Direct link to Examples") ```sql MYSQL > select field('a', 'b', 'a', 'd'); +---------------------------+ | field('a', 'b', 'a', 'd') | +---------------------------+ | 2 | +---------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") FIELD --- ### find_in_set This function returns the position of the first str in strlist (start counting with 1). Strlist is a sting separated by commas. If it does not find any str, it returns 0. When the argument is NULL, the result is NULL. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT find_in_set(VARCHAR str, VARCHAR strlist) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select find_in_set("b", "a,b,c"); +---------------------------+ | find_in_set('b', 'a,b,c') | +---------------------------+ | 2 | +---------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") FIND\_IN\_SET,FIND,IN,SET --- ### format_bytes Converts a byte count into a human-readable string with appropriate units (B, KB, MB, GB, TB, PB, EB). This function is useful for displaying file sizes, table sizes, memory usage, and other storage-related metrics in a user-friendly format. It uses 1024-based calculations (binary prefixes) but displays units as KB, MB, GB for simplicity and familiarity. This function is supported from v3.4. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR format_bytes(BIGINT bytes) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `bytes`: the number of bytes to format. BIGINT is supported. Must be a non-negative integer value. #### Return value[​](#return-value "Direct link to Return value") Returns a VARCHAR value representing the formatted byte size with appropriate units. * For values less than 1024: returns the exact number with "B" (bytes) * For larger values: returns a formatted string with 2 decimal places and the appropriate unit (KB, MB, GB, TB, PB, EB) * Returns NULL if the input is negative or NULL #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The function uses 1024-based (binary) calculations internally (1 KB = 1024 bytes, 1 MB = 1024² bytes, etc.) * Units are displayed as KB, MB, GB, TB, PB, EB for user familiarity, though they represent binary prefixes (KiB, MiB, GiB, etc.) * Values greater than or equal to 1 KB are displayed with exactly 2 decimal places * Byte values (less than 1024) are displayed as whole numbers without decimal places * Negative values return NULL * Supports values up to 8 EB (exabytes), covering the full range of BIGINT #### Examples[​](#examples "Direct link to Examples") Example 1: Format various byte sizes. ```sql SELECT format_bytes(0); -- 0 B SELECT format_bytes(123); -- 123 B SELECT format_bytes(1024); -- 1.00 KB SELECT format_bytes(4096); -- 4.00 KB SELECT format_bytes(123456789); -- 117.74 MB SELECT format_bytes(10737418240); -- 10.00 GB ``` Example 2: Handle edge cases and null values. ```sql SELECT format_bytes(-1); -- NULL SELECT format_bytes(NULL); -- NULL SELECT format_bytes(0); -- 0 B ``` Example 3: Practical usage for table size monitoring. ```sql -- Create a sample table with size information CREATE TABLE storage_info ( table_name VARCHAR(64), size_bytes BIGINT ); INSERT INTO storage_info VALUES ('user_profiles', 1073741824), ('transaction_logs', 5368709120), ('product_catalog', 524288000), ('analytics_data', 2199023255552); -- Display table sizes in human-readable format SELECT table_name, size_bytes, format_bytes(size_bytes) as formatted_size FROM storage_info ORDER BY size_bytes DESC; -- Expected output: -- +------------------+---------------+----------------+ -- | table_name | size_bytes | formatted_size | -- +------------------+---------------+----------------+ -- | analytics_data | 2199023255552 | 2.00 TB | -- | transaction_logs | 5368709120 | 5.00 GB | -- | user_profiles | 1073741824 | 1.00 GB | -- | product_catalog | 524288000 | 500.00 MB | -- +------------------+---------------+----------------+ ``` Example 4: Precision and rounding behavior. ```sql SELECT format_bytes(1536); -- 1.50 KB (exactly 1.5 KB) SELECT format_bytes(1025); -- 1.00 KB (rounded to 2 decimals) SELECT format_bytes(1048576 + 52429); -- 1.05 MB (rounded to 2 decimals) ``` Example 5: Full range of supported units. ```sql SELECT format_bytes(512); -- 512 B SELECT format_bytes(2048); -- 2.00 KB SELECT format_bytes(1572864); -- 1.50 MB SELECT format_bytes(2147483648); -- 2.00 GB SELECT format_bytes(1099511627776); -- 1.00 TB SELECT format_bytes(1125899906842624); -- 1.00 PB SELECT format_bytes(1152921504606846976); -- 1.00 EB ``` #### keyword[​](#keyword "Direct link to keyword") FORMAT\_BYTES --- ### group_concat Concatenates non-null values from a group into a single string, with a `sep` argument, which is `,` by default if not specified. This function can be used to concatenate values from multiple rows of a column into one string. group\_concat supports DISTINCT and ORDER BY in 3.0 versions later than 3.0.6 and 3.1 versions later than 3.1.3. #### Syntax[​](#syntax "Direct link to Syntax") ```sql VARCHAR GROUP_CONCAT([DISTINCT] expr [,expr ...] [ORDER BY {unsigned_integer | col_name | expr} [ASC | DESC] [,col_name ...]] [SEPARATOR sep]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `expr`: the values to concatenate, with null values ignored. It must evaluate to VARCHAR. You can optionally specify `DISTINCT` to eliminate duplicate values from the output string. If you want to concatenate multiple `expr` directly, use [concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/concat.md) or [concat\_ws](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/concat_ws.md) to specify formats. * Items in ORDER BY can be unsigned integers (starting from 1), column names, or normal expressions. The results are sorted in ascending order by default. You can also explicitly specify the ASC keyword. If you want to sort results in descending order, add the DESC keyword to the name of the column you are sorting. * `sep`: the optional separator used to concatenate non-null values from different rows. If it is not specified, `,` (a comma) is used by default. To eliminate the separator, specify an empty string `''`. > **NOTE** > > From v3.0.6 and v3.1.3 onwards, there is a behavior change when you specify the separator. You must use `SEPARATOR` to declare the separator, for example, `select group_concat(name SEPARATOR '-') as res from ss;`. #### Return value[​](#return-value "Direct link to Return value") Returns a string value for each group and returns NULL if there are no non-NULL values. You can limit the length of the string returned by group\_concat by setting the [session variable](https://docs.starrocks.io/docs/sql-reference/System_variable.md) `group_concat_max_len`, which defaults to 1024. Minimum value: 4. Unit: characters. Example: ```sql SET [GLOBAL | SESSION] group_concat_max_len = ; ``` #### Examples[​](#examples "Direct link to Examples") 1. Create a table `ss` which contains subject scores. ```sql CREATE TABLE `ss` ( `id` int(11) NULL COMMENT "", `name` varchar(255) NULL COMMENT "", `subject` varchar(255) NULL COMMENT "", `score` int(11) NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`id`) DISTRIBUTED BY HASH(`id`) BUCKETS 4 PROPERTIES ( "replication_num" = "1" ); insert into ss values (1,"Tom","English",90); insert into ss values (1,"Tom","Math",80); insert into ss values (2,"Tom","English",NULL); insert into ss values (2,"Tom",NULL,NULL); insert into ss values (3,"May",NULL,NULL); insert into ss values (3,"Ti","English",98); insert into ss values (4,NULL,NULL,NULL); insert into ss values (NULL,"Ti","Phy",98); select * from ss order by id; +------+------+---------+-------+ | id | name | subject | score | +------+------+---------+-------+ | NULL | Ti | Phy | 98 | | 1 | Tom | English | 90 | | 1 | Tom | Math | 80 | | 2 | Tom | English | NULL | | 2 | Tom | NULL | NULL | | 3 | May | NULL | NULL | | 3 | Ti | English | 98 | | 4 | NULL | NULL | NULL | +------+------+---------+-------+ ``` 2. Use group\_concat. Example 1: Concatenate names into a string with the default separator and with null values ignored. Duplicate names are retained. ```sql select group_concat(name) as res from ss; +---------------------------+ | res | +---------------------------+ | Tom,Tom,Ti,Tom,Tom,May,Ti | +---------------------------+ ``` Example 2: Concatenate names into a string, connected by the separator `-` and with null values ignored. Duplicate names are retained. ```sql select group_concat(name SEPARATOR '-') as res from ss; +---------------------------+ | res | +---------------------------+ | Ti-May-Ti-Tom-Tom-Tom-Tom | +---------------------------+ ``` Example 3: Concatenate distinct names into a string with the default separator and with null values ignored. Duplicate names are removed. ```sql select group_concat(distinct name) as res from ss; +---------------------------+ | res | +---------------------------+ | Ti,May,Tom | +---------------------------+ ``` Example 4: Concatenate the name-subject strings of the same ID in ascending order of `score`. For example, `TomMath` and `TomEnglish` share ID 1 and they are concatenated with a comma in ascending order of `score`. ```sql select id, group_concat(distinct name,subject order by score) as res from ss group by id order by id; +------+--------------------+ | id | res | +------+--------------------+ | NULL | TiPhy | | 1 | TomMath,TomEnglish | | 2 | TomEnglish | | 3 | TiEnglish | | 4 | NULL | +------+--------------------+ ``` Example 5: group\_concat is nested with concat(), which is used to combine `name`, `-`, and `subject` as a string. The strings in the same row are sorted in ascending order of `score`. ```sql select id, group_concat(distinct concat(name, '-',subject) order by score) as res from ss group by id order by id; +------+----------------------+ | id | res | +------+----------------------+ | NULL | Ti-Phy | | 1 | Tom-Math,Tom-English | | 2 | Tom-English | | 3 | Ti-English | | 4 | NULL | +------+----------------------+ ``` Example 6: No matching result is found and NULL is returned. ```sql select group_concat(distinct name) as res from ss where id < 0; +------+ | res | +------+ | NULL | +------+ ``` Example 7: Limit the length of the returned string to six characters. ```sql set group_concat_max_len = 6; select id, group_concat(distinct name,subject order by score) as res from ss group by id order by id; +------+--------+ | id | res | +------+--------+ | NULL | TiPhy | | 1 | TomMat | | 2 | NULL | | 3 | TiEngl | | 4 | NULL | +------+--------+ ``` #### keyword[​](#keyword "Direct link to keyword") GROUP\_CONCAT,CONCAT,ARRAY\_AGG --- ### hex If `x` is a numerical value, this function returns a hexadecimal string representation of the value. If `x` is a string, this function returns a hexadecimal string representation of the string where each character in the string is converted into two hexadecimal digits. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext HEX(x); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: the string or number to convert. The supported data types are BIGINT, VARCHAR, and VARBINARY (v3.0 and later). #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select hex(3); +--------+ | hex(3) | +--------+ | 3 | +--------+ 1 row in set (0.00 sec) mysql> select hex('3'); +----------+ | hex('3') | +----------+ | 33 | +----------+ 1 row in set (0.00 sec) mysql> select hex('apple'); +--------------+ | hex('apple') | +--------------+ | 6170706C65 | +--------------+ -- The input is a binary value. mysql> select hex(x'abab'); +-------------+ | hex('ABAB') | +-------------+ | ABAB | +-------------+ 1 row in set (0.01 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") HEX --- ### hex_decode_binary Decodes a hex encoded string to a binary. This function is supported from v3.0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext hex_decode_binary(str); ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: the string to convert. The supported data type is VARCHAR. An empty binary is returned if any of the following situations occur: * The length of the string is 0 or the number of characters in the string is an odd number. * The string contains characters other than `[0-9]`, `[a-z]`, and `[A-Z]`. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARBINARY type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select hex(hex_decode_binary(hex("Hello StarRocks"))); +------------------------------------------------+ | hex(hex_decode_binary(hex('Hello StarRocks'))) | +------------------------------------------------+ | 48656C6C6F2053746172526F636B73 | +------------------------------------------------+ mysql> select hex_decode_binary(NULL); +--------------------------------------------------+ | hex_decode_binary(NULL) | +--------------------------------------------------+ | NULL | +--------------------------------------------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") HEX\_DECODE\_BINARY --- ### hex_decode_string This function performs the opposite operation of [hex()](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/hex.md). It interprets each pair of hexadecimal digits in the input string as a number and converts it to the byte represented by the number. The return value is a binary string. This function is supported from v3.0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext hex_decode_string(str); ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: the string to convert. The supported data type is VARCHAR. An empty string is returned if any of the following situations occur: * The length of the string is 0 or the number of characters in the string is an odd number. * The string contains characters other than `[0-9]`, `[a-z]`, and `[A-Z]`. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select hex_decode_string(hex("Hello StarRocks")); +-------------------------------------------+ | hex_decode_string(hex('Hello StarRocks')) | +-------------------------------------------+ | Hello StarRocks | +-------------------------------------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") HEX\_DECODE\_STRING --- ### inet_aton Takes a string containing an IPv4 address in the format A.B.C.D. Returns a BIGINT number representing the corresponding IPv4 address in big endian. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT inet_aton(VARCHAR ipv4) ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select inet_aton('192.168.1.1'); +--------------------------------------+ | inet_aton('192.168.1.1') | +--------------------------------------+ | 3232235777 | +--------------------------------------+ mysql> select stringIp, inet_aton(stringIp) from ipv4; +-----------------+----------------------------+ |stringIp | inet_aton(stringIp) | +-----------------+----------------------------+ | 0.0.0.0 | 0 | | 255.255.255.255 | 4294967295 | | invalid | NULL | +-----------------+----------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") INET\_ATON --- ### initcap * Converts the first letter of each word in a string to uppercase and the remaining characters to lowercase. * This function is useful for normalizing text data, such as names, addresses, or titles, where consistent Title Casing is required. #### Syntax[​](#syntax "Direct link to Syntax") ```sql initcap(str) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str:` The string expression to be formatted. This can be a VARCHAR column, a literal string, or the result of another string function. #### Return value[​](#return-value "Direct link to Return value") Returns the formatted string as a VARCHAR. > * Returns NULL if the argument is NULL. > * Words are delimited by whitespace or non-alphanumeric characters (e.g., punctuation, symbols, numbers). > * Characters immediately following a delimiter are uppercased; all other letters are lowercased. #### Examples[​](#examples "Direct link to Examples") Example 1: Format basic strings (lowercase or uppercase). ```plaintext mysql> SELECT initcap('hello world'); +------------------------+ | initcap('hello world') | +------------------------+ | Hello World | +------------------------+ ``` ```text mysql> SELECT initcap('HELLO WORLD'); +------------------------+ | initcap('HELLO WORLD') | +------------------------+ | Hello World | +------------------------+ ``` Example 2: Format mixed-case strings. ```plaintext mysql> SELECT initcap('sTaRroCks dAtAbAsE'); +-------------------------------+ | initcap('sTaRroCks dAtAbAsE') | +-------------------------------+ | Starrocks Database | +-------------------------------+ ``` Example 3: Handling delimiters (numbers, symbols, and punctuation). ```plaintext mysql> SELECT initcap('1st place, in-the-world!'); +-------------------------------------+ | initcap('1st place, in-the-world!') | +-------------------------------------+ | 1st Place, In-The-World! | +-------------------------------------+ ``` #### Usage notes[​](#usage-notes "Direct link to Usage notes") **Word Boundaries:** The function identifies the start of a "word" based on any character that is not a letter or a number. For example, in `'abc-def'`, the hyphen - is a delimiter, so d becomes D. **Numbers:** Numbers are treated as word characters if they appear at the start, but they do not change case. However, symbols following numbers will reset the word boundary (e.g., 1st -> 1st because 1 starts the word, but 1-st -> 1-St because - is a delimiter). --- ### instr This function returns the position where str first appeared in substr (start counting from 1 and measured in characters). If str is not found in substr, then this function will return 0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT instr(VARCHAR str, VARCHAR substr) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select instr("abc", "b"); +-------------------+ | instr('abc', 'b') | +-------------------+ | 2 | +-------------------+ MySQL > select instr("abc", "d"); +-------------------+ | instr('abc', 'd') | +-------------------+ | 0 | +-------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") INSTR --- ### lcase This function converts a string to lower-case. It is analogous to the function lower. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR lcase(VARCHAR str) ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> SELECT lcase("AbC123"); +-----------------+ |lcase('AbC123') | +-----------------+ |abc123 | +-----------------+ ``` #### keyword[​](#keyword "Direct link to keyword") LCASE --- ### left This function returns a specified number of characters from the left side of a given string. The unit for length: utf8 character. Note: This function is also named as [strleft](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/strleft.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql VARCHAR left(VARCHAR str,INT len) ``` #### Examples[​](#examples "Direct link to Examples") ```sql MySQL > select left("Hello starrocks",5); +----------------------------+ | left('Hello starrocks', 5) | +----------------------------+ | Hello | +----------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") LEFT --- ### length This function returns the length of a string (in bytes). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT length(VARCHAR str) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select length("abc"); +---------------+ | length('abc') | +---------------+ | 3 | +---------------+ ``` #### keyword[​](#keyword "Direct link to keyword") LENGTH --- ### locate This function is used for finding the location of a substring in a string (starting counting from 1 and measured in characters). If the third argument pos is specified, it will start to find positions of substr in strings below pos. If str is not found, it will return 0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT locate(VARCHAR substr, VARCHAR str[, INT pos]) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT LOCATE('bar', 'foobarbar'); +----------------------------+ | locate('bar', 'foobarbar') | +----------------------------+ | 4 | +----------------------------+ MySQL > SELECT LOCATE('xbar', 'foobar'); +--------------------------+ | locate('xbar', 'foobar') | +--------------------------+ | 0 | +--------------------------+ MySQL > SELECT LOCATE('bar', 'foobarbar', 5); +-------------------------------+ | locate('bar', 'foobarbar', 5) | +-------------------------------+ | 7 | +-------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") LOCATE --- ### lower Converts all strings in an argument to lower-case. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT lower(VARCHAR str) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT lower("AbC123"); +-----------------+ | lower('AbC123') | +-----------------+ | abc123 | +-----------------+ ``` #### keyword[​](#keyword "Direct link to keyword") LOWER --- ### lpad This function returns strings with a length of `len` (starting counting from the first syllable) in `str`. If `len` is longer than `str`, the return value is lengthened to `len` characters by adding pad characters in front of `str`. If `str` is longer than `len`, the return value is shortened to `len` characters. `len` means the length of characters, not bytes. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR lpad(VARCHAR str, INT len[, VARCHAR pad]) ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: required, the string to be padded, which must evaluate to a VARCHAR value. `len`: required, the length of return value, it means the length of characters, not bytes, which must evaluate to an INT value. `pad`: optional, the characters to be added in front of str, which must be a VARCHAR value. If this parameter is not specified, spaces are added by default. #### Return value[​](#return-value "Direct link to Return value") Returns a VARCHAR value. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT lpad("hi", 5, "xy"); +---------------------+ | lpad('hi', 5, 'xy') | +---------------------+ | xyxhi | +---------------------+ MySQL > SELECT lpad("hi", 1, "xy"); +---------------------+ | lpad('hi', 1, 'xy') | +---------------------+ | h | +---------------------+ MySQL > SELECT lpad("hi", 5); +---------------------+ | lpad('hi', 5, ' ') | +---------------------+ | hi | +---------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") LPAD --- ### ltrim Removes the leading spaces or specified characters from the beginning (left) of the `str` argument. Removing specified characters are supported from StarRocks 2.5.0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR ltrim(VARCHAR str[, VARCHAR characters]) ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: required, the string to trim, which must evaluate to a VARCHAR value. `characters`: optional, the characters to remove, which must be a VARCHAR value. If this parameter is not specified, spaces are removed from the string by default. If this parameter is set to an empty string, an error is returned. #### Return value[​](#return-value "Direct link to Return value") Returns a VARCHAR value. #### Examples[​](#examples "Direct link to Examples") Example 1: Remove spaces from the beginning of the string. ```plain MySQL > SELECT ltrim(' ab d'); +------------------+ | ltrim(' ab d') | +------------------+ | ab d | +------------------+ ``` Example 2: Remove specified characters from the beginning of the string. ```plain MySQL > SELECT ltrim("xxabcdxx", "x"); +------------------------+ | ltrim('xxabcdxx', 'x') | +------------------------+ | abcdxx | +------------------------+ ``` #### References[​](#references "Direct link to References") * [trim](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/trim.md) * [rtrim](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/rtrim.md) #### keyword[​](#keyword "Direct link to keyword") LTRIM --- ### money_format This function returns a string formatted as a currency string. The integer part is separated by comma every three bits and the decimal part is reserved for two bits. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR money_format(Number) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select money_format(17014116); +------------------------+ | money_format(17014116) | +------------------------+ | 17,014,116.00 | +------------------------+ MySQL > select money_format(1123.456); +------------------------+ | money_format(1123.456) | +------------------------+ | 1,123.46 | +------------------------+ MySQL > select money_format(1123.4); +----------------------+ | money_format(1123.4) | +----------------------+ | 1,123.40 | +----------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") MONEY\_FORMAT,MONEY,FORMAT --- ### ngram_search Calculate the ngram similarity of the two strings. info * Currently, the character encoding only supports ASCII encoding and does not support UTF-8 encoding. * The function `ngram_search` is case-sensitive. Another function `ngram_search_case_insensitive` is case-insensitive. Other than that, these two functions are the same. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DOUBLE ngram_search(VARCHAR haystack, VARCHAR needle, INT gram_num) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `haystack`: required, the first string to compare. It must be a VARCHAR value. It can be a column name or a const value. If `haystack` is a column name, and an N-Gram bloom filter index is created for that column in the table, the index can accelerate the computation speed of the `ngram_search` function. * `needle`: required, the second string to compare. It must be a VARCHAR value. It can only be a const value. tip * The length of the `needle` value can not be larger than 2^15. Otherwise an error will be thrown. * If the length of the `haystack` value is larger than 2^15, this function will return 0. * If the length of the `haystack` or `needle` value is smaller than `gram_num`, this function will return 0. * `gram_num`: required, used for specifying the number of grams. The recommended value is `4`. #### Return value[​](#return-value "Direct link to Return value") Returns a value describing how similar these two strings are. The range of returned value is between 0 and 1. The larger this value, the more similar the two strings are. #### Examples[​](#examples "Direct link to Examples") ```sql -- The values of haystack and needle are const values. mysql> select ngram_search('English', 'England',4); +---------------------------------------+ | ngram_search('English', 'England', 4) | +---------------------------------------+ | 0.25 | +---------------------------------------+ -- The value of haystack is a column name and the value of needle is a const value. mysql> select rowkey,ngram_search(rowkey,"31dc496b-760d-6f1a-4521-050073a70000",4) as string_similarity from string_table order by string_similarity desc limit 5; +--------------------------------------+-------------------+ | rowkey | string_similarity | +--------------------------------------+-------------------+ | 31dc496b-760d-6f1a-4521-050073a70000 | 1 | | 31dc496b-760d-6f1a-4521-050073a40000 | 0.8787879 | | 31dc496b-760d-6f1a-4521-05007fa70000 | 0.8787879 | | 31dc496b-760d-6f1a-4521-050073a30000 | 0.8787879 | | 31dc496b-760d-6f1a-4521-0500c3a70000 | 0.8787879 | +--------------------------------------+-------------------+ ``` --- ### null_or_empty This function returns true when the string is empty or NULL. Otherwise, it returns false. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BOOLEAN NULL_OR_EMPTY (VARCHAR str) BOOLEAN NULL_OR_EMPTY (any_array) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select null_or_empty(null); +---------------------+ | null_or_empty(NULL) | +---------------------+ | 1 | +---------------------+ MySQL > select null_or_empty(""); +-------------------+ | null_or_empty('') | +-------------------+ | 1 | +-------------------+ MySQL > select null_or_empty("a"); +--------------------+ | null_or_empty('a') | +--------------------+ | 0 | +--------------------+ MySQL > select null_or_empty([]); +-------------------+ | null_or_empty('') | +-------------------+ | 1 | +-------------------+ MySQL > select null_or_empty([1]); +--------------------+ | null_or_empty('a') | +--------------------+ | 0 | +--------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") NULL\_OR\_EMPTY --- ### parse_url Parses a URL and extracts a component from this URL. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext parse_url(expr1,expr2); ``` #### Parameters[​](#parameters "Direct link to Parameters") `expr1`: the URL. The supported data type is VARCHAR. `expr2`: the component to extract from this URL. The supported data type is VARCHAR. Valid values: * PROTOCOL * HOST * PATH * REF * AUTHORITY * FILE * USERINFO * QUERY. Parameters in QUERY cannot be returned. If you want to return specific parameters, use parse\_url with [trim](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/trim.md) to achieve this implementation. For details, see [Examples](#examples). `expr2` is **case-sensitive**. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. If the URL is invalid, an error is returned. If the requested information cannot be find, NULL is returned. #### Examples[​](#examples "Direct link to Examples") ```plain select parse_url('http://facebook.com/path/p1.php?query=1', 'HOST'); +--------------------------------------------------------------+ | parse_url('http://facebook.com/path/p1.php?query=1', 'HOST') | +--------------------------------------------------------------+ | facebook.com | +--------------------------------------------------------------+ select parse_url('http://facebook.com/path/p1.php?query=1', 'AUTHORITY'); +-------------------------------------------------------------------+ | parse_url('http://facebook.com/path/p1.php?query=1', 'AUTHORITY') | +-------------------------------------------------------------------+ | facebook.com | +-------------------------------------------------------------------+ select parse_url('http://facebook.com/path/p1.php?query=1', 'QUERY'); +---------------------------------------------------------------+ | parse_url('http://facebook.com/path/p1.php?query=1', 'QUERY') | +---------------------------------------------------------------+ | query=1 | +---------------------------------------------------------------+ select trim(parse_url('http://facebook.com/path/p1.php?query=1', 'QUERY'),'query='); +-------------------------------------------------------------------------------+ | trim(parse_url('http://facebook.com/path/p1.php?query=1', 'QUERY'), 'query=') | +-------------------------------------------------------------------------------+ | 1 | +-------------------------------------------------------------------------------+ ``` --- ### regexp_split Split string `str` by regexp expression `pattern`, return maximum `max_split` elements in `ARRAY` type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext regexp_split(str, pattern[, max_split]) ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: required, the string to split, which must evaluate to a `VARCHAR` value. `pattern`: required, the regexp expression pattern used to split, which must evaluate to a `VARCHAR` value. `max_split`: optional, the number of maximum elements in the output value, which must evaluate to an `INT` value. #### Return value[​](#return-value "Direct link to Return value") Returns an `ARRAY` value. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select regexp_split('StarRocks', ''); +---------------------------------------+ | regexp_split('StarRocks', '') | +---------------------------------------+ | ["S","t","a","r","R","o","c","k","s"] | +---------------------------------------+ mysql> select regexp_split('StarRocks', '[SR]'); +-----------------------------------+ | regexp_split('StarRocks', '[SR]') | +-----------------------------------+ | ["","tar","ocks"] | +-----------------------------------+ mysql> select regexp_split('StarRocks', '[SR]', 1); +--------------------------------------+ | regexp_split('StarRocks', '[SR]', 1) | +--------------------------------------+ | ["StarRocks"] | +--------------------------------------+ mysql> select regexp_split('StarRocks', '[SR]', 2); +--------------------------------------+ | regexp_split('StarRocks', '[SR]', 1) | +--------------------------------------+ | ["","tarRocks"] | +--------------------------------------+ ``` --- ### repeat This function repeats `str` by a number of times according to `count`. When `count` is below 1, it returns an empty string. When `str` or `count` is NULL, it returns NULL. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR repeat(VARCHAR str, INT count) ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT repeat("a", 3); +----------------+ | repeat('a', 3) | +----------------+ | aaa | +----------------+ MySQL > SELECT repeat("a", -1); +-----------------+ | repeat('a', -1) | +-----------------+ | | +-----------------+ ``` #### keyword[​](#keyword "Direct link to keyword") REPEAT, --- ### replace Replaces all occurrences of characters in a string with another string. This function performs a case-sensitive match when searching for `pattern`. This function is supported from v3.0. Note: Prior to 3.0, this function was implemented as [regexp\_replace](https://docs.starrocks.io/docs/sql-reference/sql-functions/like-predicate-functions/regexp_replace.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql VARCHAR replace(VARCHAR str, VARCHAR pattern, VARCHAR repl) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str`: the original string. * `pattern`: the characters to replace. Note that this is not a regular expression. * `repl`: the string used to replace characters in `pattern`. #### Return value[​](#return-value "Direct link to Return value") Returns a string with the specified characters replaced. If any argument is NULL, the result is NULL. If no matching characters are found, the original string is returned. #### Examples[​](#examples "Direct link to Examples") ```plain -- Replace '.' in 'a.b.c' with '+'. MySQL > SELECT replace('a.b.c', '.', '+'); +----------------------------+ | replace('a.b.c', '.', '+') | +----------------------------+ | a+b+c | +----------------------------+ -- No matching characters are found and the original string is returned. MySQL > SELECT replace('a b c', '', '*'); +----------------------------+ | replace('a b c', '', '*') | +----------------------------+ | a b c | +----------------------------+ -- Replace 'like' with an empty string. MySQL > SELECT replace('We like StarRocks', 'like', ''); +------------------------------------------+ | replace('We like StarRocks', 'like', '') | +------------------------------------------+ | We StarRocks | +------------------------------------------+ -- No matching characters are found and the original string is returned. MySQL > SELECT replace('He is awesome', 'handsome', 'smart'); +-----------------------------------------------+ | replace('He is awesome', 'handsome', 'smart') | +-----------------------------------------------+ | He is awesome | +-----------------------------------------------+ ``` #### keywords[​](#keywords "Direct link to keywords") REPLACE, replace --- ### reverse Reverses a string or array. Returns a string or array with the characters in the string or array elements in reverse order. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext reverse(param) ``` #### Parameters[​](#parameters "Direct link to Parameters") `param`: the string or array to reverse. It can be of the VARCHAR, CHAR, or ARRAY type. Currently, this function supports only one-dimensional arrays and the array elements cannot be of the DECIMAL type. This function supports the following types of array elements: BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, VARCHAR, DECIMALV2, DATETIME, DATE, and JSON. **JSON is supported from 2.5.** #### Return value[​](#return-value "Direct link to Return value") The return type is the same as `param`. #### Examples[​](#examples "Direct link to Examples") Example 1: Reverse a string. ```plain MySQL > SELECT REVERSE('hello'); +------------------+ | REVERSE('hello') | +------------------+ | olleh | +------------------+ 1 row in set (0.00 sec) ``` Example 2: Reverse an array. ```plain MYSQL> SELECT REVERSE([4,1,5,8]); +--------------------+ | REVERSE([4,1,5,8]) | +--------------------+ | [8,5,1,4] | +--------------------+ ``` --- ### right This function returns a specified length of characters from the right side of a given string. Length unit: utf8 character. Note: This function is also named as [strright](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/strright.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql VARCHAR right(VARCHAR str,INT len) ``` #### Examples[​](#examples "Direct link to Examples") ```sql MySQL > select right("Hello starrocks",9); +-----------------------------+ | right('Hello starrocks', 9) | +-----------------------------+ | starrocks | +-----------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") RIGHT --- ### rpad This function returns strings with a length of `len` (starting counting from the first syllable) in `str`. If `len` is longer than `str`, the return value is lengthened to `len` characters by adding pad characters behind `str`. If `str` is longer than `len`, the return value is shortened to `len` characters. `len` means the length of characters, not bytes. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR rpad(VARCHAR str, INT len[, VARCHAR pad]) ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: required, the string to be padded, which must evaluate to a VARCHAR value. `len`: required, the length of return value, it means the length of characters, not bytes, which must evaluate to an INT value. `pad`: optional, the characters to be added behind str, which must be a VARCHAR value. If this parameter is not specified, spaces are added by default. #### Return value[​](#return-value "Direct link to Return value") Returns a VARCHAR value. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SELECT rpad("hi", 5, "xy"); +---------------------+ | rpad('hi', 5, 'xy') | +---------------------+ | hixyx | +---------------------+ MySQL > SELECT rpad("hi", 1, "xy"); +---------------------+ | rpad('hi', 1, 'xy') | +---------------------+ | h | +---------------------+ MySQL > SELECT rpad("hi", 5); +---------------------+ | rpad('hi', 5, ' ') | +---------------------+ | hi | +---------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") RPAD --- ### rtrim Removes the trailing spaces or specified characters from the end (right) of the `str` argument. Removing specified characters are supported from StarRocks 2.5.0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR rtrim(VARCHAR str[, VARCHAR characters]); ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: required, the string to trim, which must evaluate to a VARCHAR value. `characters`: optional, the characters to remove, which must be a VARCHAR value. If this parameter is not specified, spaces are removed from the string by default. If this parameter is set to an empty string, an error is returned. #### Return value[​](#return-value "Direct link to Return value") Returns a VARCHAR value. #### Examples[​](#examples "Direct link to Examples") Example 1: Remove the three trailing spaces from the string. ```plain mysql> SELECT rtrim(' ab d '); +---------------------+ | rtrim(' ab d ') | +---------------------+ | ab d | +---------------------+ 1 row in set (0.00 sec) ``` Example 2: Remove specified characters from the end of the string. ```plain MySQL > SELECT rtrim("xxabcdxx", "x"); +------------------------+ | rtrim('xxabcdxx', 'x') | +------------------------+ | xxabcd | +------------------------+ ``` #### References[​](#references "Direct link to References") * [trim](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/trim.md) * [ltrim](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/ltrim.md) --- ### space Returns a string of the specified number of spaces. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext space(x); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: the number of spaces to return. The supported data type is INT. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select space(6); +----------+ | space(6) | +----------+ | | +----------+ 1 row in set (0.00 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") SPACE --- ### split This function splits a given string according to the separators, and returns the split parts in ARRAY. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ARRAY split(VARCHAR content, VARCHAR delimiter) ``` #### Examples[​](#examples "Direct link to Examples") ```sql mysql> select split("a,b,c",","); +---------------------+ | split('a,b,c', ',') | +---------------------+ | ["a","b","c"] | +---------------------+ mysql> select split("a,b,c",",b,"); +-----------------------+ | split('a,b,c', ',b,') | +-----------------------+ | ["a","c"] | +-----------------------+ mysql> select split("abc",""); +------------------+ | split('abc', '') | +------------------+ | ["a","b","c"] | +------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") SPLIT --- ### split_part This function splits a given string according to the separators and returns the requested part. (start counting from the beginning) #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR split_part(VARCHAR content, VARCHAR delimiter, INT field) ``` #### Parameters[​](#parameters "Direct link to Parameters") `content`: The string to be split. Data type: VARCHAR. `delimiter`: The separator used to split the string. Data type: VARCHAR. `field`: The position of the part to return. Positive values count from the beginning, negative values count from the end. Data type: INT. #### Return Value[​](#return-value "Direct link to Return Value") Returns the specified part of the split string. Data type: VARCHAR. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select split_part("hello world", " ", 1); +----------------------------------+ |split_part('hello world', ' ', 1) | +----------------------------------+ | hello | +----------------------------------+ MySQL > select split_part("hello world", " ", 2); +-----------------------------------+ | split_part('hello world', ' ', 2) | +-----------------------------------+ | world | +-----------------------------------+ MySQL > select split_part("hello world", " ", -1); +----------------------------------+ |split_part('hello world', ' ', -1) | +----------------------------------+ | world | +----------------------------------+ MySQL > select split_part("hello world", " ", -2); +-----------------------------------+ | split_part('hello world', ' ', -2) | +-----------------------------------+ | hello | +-----------------------------------+ MySQL > select split_part("hello world", "|", 1); +----------------------------------+ | split_part('hello world', '|', 1) | +----------------------------------+ | hello world | +----------------------------------+ MySQL > select split_part("hello world", "|", -1); +-----------------------------------+ | split_part('hello world', '|', -1) | +-----------------------------------+ | hello world | +-----------------------------------+ MySQL > select split_part("hello world", "|", 2); +----------------------------------+ | split_part('hello world', '|', 2) | +----------------------------------+ | | +----------------------------------+ MySQL > select split_part("abca", "a", 1); +----------------------------+ | split_part('abca', 'a', 1) | +----------------------------+ | | +----------------------------+ MySQL > select split_part("abca", "a", -1); +-----------------------------+ | split_part('abca', 'a', -1) | +-----------------------------+ | | +-----------------------------+ MySQL > select split_part("abca", "a", -2); +-----------------------------+ | split_part('abca', 'a', -2) | +-----------------------------+ | bc | +-----------------------------+ MySQL > select split_part("2019年7月8日", "月", 1); +-----------------------------------------+ | split_part('2019年7月8日', '月', 1) | +-----------------------------------------+ | 2019年7 | +-----------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") SPLIT\_PART,SPLIT,PART --- ### starts_with This function returns 1 when a string starts with a specified prefix. Otherwise, it returns 0. When the argument is NULL, the result is NULL. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BOOLEAN starts_with(VARCHAR str, VARCHAR prefix) ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select starts_with("hello world","hello"); +-------------------------------------+ |starts_with('hello world', 'hello') | +-------------------------------------+ | 1 | +-------------------------------------+ mysql> select starts_with("hello world","world"); +-------------------------------------+ |starts_with('hello world', 'world') | +-------------------------------------+ | 0 | +-------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") START\_WITH --- ### str_to_map Splits a given string into key-value pairs using two delimiters and returns a map of the split pairs. This function is supported from v3.1 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext MAP str_to_map(VARCHAR content[, VARCHAR delimiter[, VARCHAR map_delimiter]]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `content`: required, the string expression to split. * `delimiter`: optional, the delimiter used to split `content` into key-value pairs, defaults to `,`. * `map_delimiter`: optional, the delimiter used to separate each key-value pair, defaults to `:`. #### Return value[​](#return-value "Direct link to Return value") Returns a MAP of STRING elements. Any null input results in NULL. #### Examples[​](#examples "Direct link to Examples") ```sql mysql> SELECT str_to_map('a:1|b:2|c:3', '|', ':') as map; +---------------------------+ | map | +---------------------------+ | {"a":"1","b":"2","c":"3"} | +---------------------------+ mysql> SELECT str_to_map('a:1;b:2;c:3', ';', ':') as map; +---------------------------+ | map | +---------------------------+ | {"a":"1","b":"2","c":"3"} | +---------------------------+ mysql> SELECT str_to_map('a:1,b:2,c:3', ',', ':') as map; +---------------------------+ | map | +---------------------------+ | {"a":"1","b":"2","c":"3"} | +---------------------------+ mysql> SELECT str_to_map('a') as map; +------------+ | map | +------------+ | {"a":null} | +------------+ mysql> SELECT str_to_map('a:1,b:2,c:3',null, ':') as map; +------+ | map | +------+ | NULL | +------+ mysql> SELECT str_to_map('a:1,b:2,c:null') as map; +------------------------------+ | map | +------------------------------+ | {"a":"1","b":"2","c":"null"} | +------------------------------+ ``` #### keywords[​](#keywords "Direct link to keywords") STR\_TO\_MAP --- ### strcmp This function compares two strings. Returns 0 if lhs and rhs compare equal. Return -1 if lhs appears before rhs in lexicographical order. Returns 1 if lhs appears after rhs in lexicographical order. When the argument is NULL, the result is NULL. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT strcmp(VARCHAR lhs, VARCHAR rhs) ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select strcmp("test1", "test1"); +--------------------------+ | strcmp('test1', 'test1') | +--------------------------+ | 0 | +--------------------------+ mysql> select strcmp("test1", "test2"); +--------------------------+ | strcmp('test1', 'test2') | +--------------------------+ | -1 | +--------------------------+ mysql> select strcmp("test2", "test1"); +--------------------------+ | strcmp('test2', 'test1') | +--------------------------+ | 1 | +--------------------------+ mysql> select strcmp("test1", NULL); +-----------------------+ | strcmp('test1', NULL) | +-----------------------+ | NULL | +-----------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") STRCMP --- ### strleft This function extracts a number of characters from a string with specified length (starting from left). The unit for length: utf8 character. Note: This function is also named as [left](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/left.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql VARCHAR strleft(VARCHAR str,INT len) ``` #### Examples[​](#examples "Direct link to Examples") ```sql MySQL > select strleft("Hello starrocks",5); +-------------------------------+ | strleft('Hello starrocks', 5) | +-------------------------------+ | Hello | +-------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") STRLEFT --- ### strpos Returns the position of the N-th substring in a string. If N is a negative number, the search will start from the end of the string. Positions start with 1. If not found, 0 is returned. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext INT strpos(VARCHAR str, VARCHAR substr [, INT instance]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str`: The string in which to find the substring. * `substr`: The substring to find. * `instance`: The N-th substring instance to find. If this item is set to a negative value, the search will start from the end of the string. Default Value: `1`. #### Return value[​](#return-value "Direct link to Return value") Returns an integer. `0` is returned if the substring is not found. #### Examples[​](#examples "Direct link to Examples") ```sql SELECT strpos('hello world', 'world'); +-----------------------------+ | strpos('hello world', 'world') | +-----------------------------+ | 7 | +-----------------------------+ SELECT strpos('Hello World', 'world'); +-----------------------------+ | strpos('Hello World', 'world') | +-----------------------------+ | 0 | +-----------------------------+ SELECT strpos('hello world hello', 'hello', 2); +--------------------------------------+ | strpos('hello world hello', 'hello', 2) | +--------------------------------------+ | 13 | +--------------------------------------+ SELECT strpos('StarRocks', 'Spark'); +----------------------------+ | strpos('StarRocks', 'Spark') | +----------------------------+ | 0 | +----------------------------+ SELECT strpos(NULL, 'test'); +--------------------+ | strpos(NULL, 'test') | +--------------------+ | NULL | +--------------------+ ``` #### Keywords[​](#keywords "Direct link to Keywords") STRPOS, STRING, POSITION, SUBSTRING --- ### strright This function extracts a number of characters from a string with specified length (starting from right). The unit for length: utf-8 character. Note: This function is also named as [right](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/right.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql VARCHAR strright(VARCHAR str,INT len) ``` #### Examples[​](#examples "Direct link to Examples") ```sql MySQL > select strright("Hello starrocks",9); +--------------------------------+ | strright('Hello starrocks', 9) | +--------------------------------+ | starrocks | +--------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") STRRIGHT --- ### substring, substr Extracts characters staring from the specified position and returns a substring of specified length. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR substr(VARCHAR str, pos[, len]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str`: required, the string to extract characters. It must be a VARCHAR value. * `pos`: required, an integer that specifies the starting position. Note that the first character in a string is 1, not 0. * If `pos` is 0, an empty string is returned. * `pos` can be a negative integer. In this case, this function extracts characters starting from the end of the string. See Example 2. * If the position specified by `pos` exceeds the range of the string, an empty string is returned. See Example 3. * `len`: optional, a positive integer that specifies the number of characters to extract. * If `len` is specified, this function extracts `len` characters starting from the position specified by `pos`. * If `len` is not specified, this function extracts all the characters starting at `pos`. See Example 1. * If `len` exceeds the actual length of the matching characters, all the matching characters are returned. See Example 4. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. #### Examples[​](#examples "Direct link to Examples") ```plain -- Extract all characters starting from the first character "s". MySQL > select substring("starrockscluster", 1); +-------------------------------------+ | substring('starrockscluster', 1) | +-------------------------------------+ | starrockscluster | +-------------------------------------+ -- Extract the first 9 characters starting from the first character "s". MySQL > select substring("starrockscluster", 1, 9); +-------------------------------------+ | substring('starrockscluster', 1, 9) | +-------------------------------------+ | starrocks | +-------------------------------------+ -- The position is negative and the counting is from the end of the string. MySQL > select substring("starrocks", -5, 5); +-------------------------------+ | substring('starrocks', -5, 5) | +-------------------------------+ | rocks | +-------------------------------+ -- The position exceeds the length of the string and an empty string is returned. MySQL > select substring("apple", 8, 2); +--------------------------------+ | substring('apple', 8, 2) | +--------------------------------+ | | +--------------------------------+ -- There are 5 matching characters. The length 9 exceeds the length of the matching characters and all the matching characters are returned. MySQL > select substring("apple", 1, 9); +--------------------------+ | substring('apple', 1, 9) | +--------------------------+ | apple | +--------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") substring,string,sub --- ### substring_index Extracts a substring that precedes or follows the `count` occurrences of the delimiter. * If `count` is positive, counting starts from the beginning of the string and this function returns the substring that precedes the `count`th delimiter. For example, `select substring_index('https://www.starrocks.io', '.', 2);` returns the substring before the second `.` delimiter, which is `https://www.starrocks`. * If `count` is negative, counting starts from the end of the string and this function returns the substring that follows the `count`th delimiter. For example, `select substring_index('https://www.starrocks.io', '.', -2);` returns the substring after the second `.` delimiter, which is `starrocks.io`. If any input parameter is null, NULL is returned. This function is supported from v3.2. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR substring_index(VARCHAR str, VARCHAR delimiter, INT count) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str`: required, the string you want to split. * `delimiter`: required, the delimiter used to split the string. * `count`: required, the position of the delimiter. The value cannot be 0. Otherwise, NULL is returned. If the value is greater than the actual number of delimiters in the string, the entire string is returned. #### Return value[​](#return-value "Direct link to Return value") Returns a VARCHAR value. #### Examples[​](#examples "Direct link to Examples") ```plain -- Return the substring that precedes the second "." delimiter. mysql> select substring_index('https://www.starrocks.io', '.', 2); +-----------------------------------------------------+ | substring_index('https://www.starrocks.io', '.', 2) | +-----------------------------------------------------+ | https://www.starrocks | +-----------------------------------------------------+ -- The count is negative. mysql> select substring_index('https://www.starrocks.io', '.', -2); +------------------------------------------------------+ | substring_index('https://www.starrocks.io', '.', -2) | +------------------------------------------------------+ | starrocks.io | +------------------------------------------------------+ mysql> select substring_index("hello world", " ", 1); +----------------------------------------+ | substring_index("hello world", " ", 1) | +----------------------------------------+ | hello | +----------------------------------------+ mysql> select substring_index("hello world", " ", -1); +-----------------------------------------+ | substring_index('hello world', ' ', -1) | +-----------------------------------------+ | world | +-----------------------------------------+ -- Count is 0 and NULL is returned. mysql> select substring_index("hello world", " ", 0); +----------------------------------------+ | substring_index('hello world', ' ', 0) | +----------------------------------------+ | NULL | +----------------------------------------+ -- Count is greater than the number of spaces in the string and the entire string is returned. mysql> select substring_index("hello world", " ", 2); +----------------------------------------+ | substring_index("hello world", " ", 2) | +----------------------------------------+ | hello world | +----------------------------------------+ -- Count is greater than the number of spaces in the string and the entire string is returned. mysql> select substring_index("hello world", " ", -2); +-----------------------------------------+ | substring_index("hello world", " ", -2) | +-----------------------------------------+ | hello world | +-----------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") substring\_index --- ### tokenize Splits and parses text into tokens based on the specified tokenizer. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ARRRY tokenize(VARCHAR tokenizer_name, VARCHAR content); ``` #### Parameters[​](#parameters "Direct link to Parameters") * `tokenizer_name`: The tokenizer to use. Valid values: `english`, `standard`, and `chinese`. * `content`: The text to be tokenized. This item can be a constant string or a column name. If a column is specified, it must be of the STRING or VARCHAR type. #### Return value[​](#return-value "Direct link to Return value") Returns a array of VARCHAR. #### Examples[​](#examples "Direct link to Examples") ```sql MYSQL > SELECT tokenize('english', 'Today is saturday'); +------------------------------------------+ | tokenize('english', 'Today is saturday') | +------------------------------------------+ | ["today","is","saturday"] | +------------------------------------------+ ``` #### keyword[​](#keyword "Direct link to keyword") TOKENIZE --- ### translate Substitutes specified characters within a string. It works by taking a string (`source`) as the input and replaces the `from_string` characters in `source` with `to_string`. This function is supported from v3.2. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext TRANSLATE(source, from_string, to_string) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `source`: supports the `VARCHAR` type. The source string to be translated. If a character in the `source` is not found in `from_string`, it is simply included in the result string. * `from_string`: supports the `VARCHAR` type. Each character in `from_string` is either replaced by its corresponding character in the `to_string`, or if there is no corresponding character (i.e. if `to_string` has fewer characters than the `from_string`, the character is excluded from the result string). See Examples 2 and 3. If a character appears multiple times in `from_string`, only its first occurrence is effective. See Example 5. * `to_string`: supports the `VARCHAR` type. The string used to replace characters. If more characters are specified in `to_string` than in the `from_string` argument, extra characters from `to_string` are ignored. See Example 4. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the `VARCHAR` type. Scenarios where the result is `NULL`: * Any of the input parameters is `NULL`. * The length of the result string after translation exceeds the maximum length (1048576) of `VARCHAR`. #### Examples[​](#examples "Direct link to Examples") ```plaintext -- Replace 'ab' in the source string with '12'. mysql > select translate('abcabc', 'ab', '12') as test; +--------+ | test | +--------+ | 12c12c | +--------+ -- Replace 'mf1' in the source string with 'to'. 'to' has less characters than 'mf1' and '1' is excluded from the result string. mysql > select translate('s1m1a1rrfcks','mf1','to') as test; +-----------+ | test | +-----------+ | starrocks | +-----------+ -- Replace 'ab' in the source string with '1'. '1' has less characters than 'ab' and 'b' is excluded from the result string. mysql > select translate('abcabc', 'ab', '1') as test; +------+ | test | +------+ | 1c1c | +------+ -- Replace 'ab' in the source string with '123'. '123' has more characters than 'ab' and '3' is ignored. mysql > select translate('abcabc', 'ab', '123') as test; +--------+ | test | +--------+ | 12c12c | +--------+ -- Replace 'aba' in the source string with '123'. 'a' appears twice and only the first occurrence of 'a' is replaced. mysql > select translate('abcabc', 'aba', '123') as test; +--------+ | test | +--------+ | 12c12c | +--------+ -- Use this function with repeat() and concat(). The result string exceeds the maximum length of VARCHAR and NULL is returned. mysql > select translate(concat('bcde', repeat('a', 1024*1024-3)), 'a', 'z') as test; +--------+ | test | +--------+ | NULL | +--------+ -- Use this function with length(), repeat(), and concat() to calculate the length of the result string. mysql > select length(translate(concat('bcd', repeat('a', 1024*1024-3)), 'a', 'z')) as test; +---------+ | test | +---------+ | 1048576 | +---------+ ``` #### See also[​](#see-also "Direct link to See also") * [concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/concat.md) * [length](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/length.md) * [repeat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/repeat.md) --- ### trim Removes consecutive spaces or specified characters from the beginning and end of the `str` argument. Removing specified characters are supported from StarRocks 2.5.0. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR trim(VARCHAR str[, VARCHAR characters]) ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: required, the string to trim, which must evaluate to a VARCHAR value. `characters`: optional, the characters to remove, which must be a VARCHAR value. If this parameter is not specified, spaces are removed from the string by default. If this parameter is set to an empty string, an error is returned. #### Return value[​](#return-value "Direct link to Return value") Returns a VARCHAR value. #### Examples[​](#examples "Direct link to Examples") Example 1: Remove the five spaces from the beginning and end of the string. ```plain MySQL > SELECT trim(" ab c "); +-------------------+ | trim(' ab c ') | +-------------------+ | ab c | +-------------------+ 1 row in set (0.00 sec) ``` Example 2: Remove specified characters from the beginning and end of the string. ```plain MySQL > SELECT trim("abcd", "ad"); +--------------------+ | trim('abcd', 'ad') | +--------------------+ | bc | +--------------------+ MySQL > SELECT trim("xxabcdxx", "x"); +-----------------------+ | trim('xxabcdxx', 'x') | +-----------------------+ | abcd | +-----------------------+ ``` #### References[​](#references "Direct link to References") * [ltrim](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/ltrim.md) * [rtrim](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/rtrim.md) --- ### ucase This function converts a string to upper-case. It is analogous to the function upper. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR ucase(VARCHAR str) ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> SELECT ucase("AbC123"); +-----------------+ |ucase('AbC123') | +-----------------+ |ABC123 | +-----------------+ ``` #### keyword[​](#keyword "Direct link to keyword") UCASE --- ### unhex This function performs the opposite operation of hex(). It interprets each pair of hexadecimal digits in the input string as a number and converts it to the byte represented by the number. The return value is a binary string. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext UNHEX(str); ``` #### Parameters[​](#parameters "Direct link to Parameters") `str`: the string to convert. The supported data type is VARCHAR. An empty string is returned if any of the following situations occur: * The length of the string is 0 or the number of characters in the string is an odd number. * The string contains characters other than `[0-9]`, `[a-z]`, and `[A-Z]`. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select unhex('33'); +-------------+ | unhex('33') | +-------------+ | 3 | +-------------+ mysql> select unhex('6170706C65'); +---------------------+ | unhex('6170706C65') | +---------------------+ | apple | +---------------------+ mysql> select unhex('4142@'); +----------------+ | unhex('4142@') | +----------------+ | | +----------------+ 1 row in set (0.01 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") UNHEX --- ### upper Converts a string to upper-case. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext upper(str) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str`: the string to convert. If `str` is not a string type, it will try implicit cast first. #### Return values[​](#return-values "Direct link to Return values") Return an upper-case string. #### Examples[​](#examples "Direct link to Examples") ```plaintext MySQL [test]> select C_String, upper(C_String) from ex_iceberg_tbl; +-------------------+-------------------+ | C_String | upper(C_String) | +-------------------+-------------------+ | Hello, StarRocks! | HELLO, STARROCKS! | | Hello, World! | HELLO, WORLD! | +-------------------+-------------------+ ``` --- ### url_decode Translates a string back from the [application/x-www-form-urlencoded](https://www.w3.org/TR/html4/interact/forms.html#h-17.13.4.1) format. This function is an inverse of [url\_encode](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/url_encode.md). This functions is supported from v3.2. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR url_decode(VARCHAR str) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str`: the string to decode. If `str` is not a string, the system will try implicit cast first. #### Return value[​](#return-value "Direct link to Return value") Returns a decoded string. #### Examples[​](#examples "Direct link to Examples") ```plaintext mysql> select url_decode('https%3A%2F%2Fdocs.starrocks.io%2Fdocs%2Fintroduction%2FStarRocks_intro%2F'); +------------------------------------------------------------------------------------------+ | url_decode('https%3A%2F%2Fdocs.starrocks.io%2Fdocs%2Fintroduction%2FStarRocks_intro%2F') | +------------------------------------------------------------------------------------------+ | https://docs.starrocks.io/docs/introduction/StarRocks_intro/ | +------------------------------------------------------------------------------------------+ ``` --- ### url_encode Translates a string into the [application/x-www-form-urlencoded](https://www.w3.org/TR/html4/interact/forms.html#h-17.13.4.1) format. This functions is supported from v3.2. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR url_encode(VARCHAR str) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str`: the string to encode. If `str` is not a string, this function will try implicit cast first. #### Return value[​](#return-value "Direct link to Return value") Returns an encoded string compliant with the [application/x-www-form-urlencoded](https://www.w3.org/TR/html4/interact/forms.html#h-17.13.4.1) format. #### Examples[​](#examples "Direct link to Examples") ```plaintext mysql> select url_encode('https://docs.starrocks.io/docs/introduction/StarRocks_intro/'); +----------------------------------------------------------------------------+ | url_encode('https://docs.starrocks.io/docs/introduction/StarRocks_intro/') | +----------------------------------------------------------------------------+ | https%3A%2F%2Fdocs.starrocks.io%2Fdocs%2Fintroduction%2FStarRocks_intro%2F | +----------------------------------------------------------------------------+ ``` --- ### url_extract_host Extracts the host section from a URL. This function is supported from v3.3 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR url_extract_host(VARCHAR str) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str`: the string to extract its host string. If `str` is not a string, this function will try implicit cast first. #### Return value[​](#return-value "Direct link to Return value") Returns the host string. #### Examples[​](#examples "Direct link to Examples") ```plaintext mysql> select url_extract_host('httpa://starrocks.com/test/api/v1'); +-------------------------------------------------------+ | url_extract_host('httpa://starrocks.com/test/api/v1') | +-------------------------------------------------------+ | starrocks.com | +-------------------------------------------------------+ ``` --- ### url_extract_parameter Extracts the value of the requested `name` parameter from the query string of a URL. Parameter extraction is handled in the manner as specified in [RFC 1866#section-8.2.1](https://datatracker.ietf.org/doc/html/rfc1866.html#section-8.2.1). If the specified parameter name does not exist, NULL is returned. This function is supported from v3.2. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR url_extract_parameter(VARCHAR str, VARCHAR name) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `str`: the URL string to extract parameters. * `name`: the name of the parameter in the query string. #### Return value[​](#return-value "Direct link to Return value") Returns a VARCHAR value. #### Examples[​](#examples "Direct link to Examples") ```plaintext mysql> select url_extract_parameter("https://starrocks.io/doc?k1=10&k2=3&k1=100", "k1"); +---------------------------------------------------------------------------+ | url_extract_parameter('https://starrocks.io/doc?k1=10&k2=3&k1=100', 'k1') | +---------------------------------------------------------------------------+ | 10 | +---------------------------------------------------------------------------+ mysql> select url_extract_parameter('https://starrocks.com/doc?k0=10&k1=%21%23%24%26%27%28%29%2A%2B%2C%2F%3A%3B%3D%3F%40%5B%5D%20%22%25%2D%2E%3C%3E%5C%5E%5F%60%7B%7C%7D%7E&k2','k1'); +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | url_extract_parameter('https://starrocks.com/doc?k0=10&k1=%21%23%24%26%27%28%29%2A%2B%2C%2F%3A%3B%3D%3F%40%5B%5D%20%22%25%2D%2E%3C%3E%5C%5E%5F%60%7B%7C%7D%7E&k2', 'k1') | +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | !#$&'()*+,/:;=?@[] "%-.<>\^_`{|}~ | +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` --- ### Struct #### [📄️ named\_struct](https://docs.starrocks.io/docs/sql-reference/sql-functions/struct-functions/named_struct.md) [Creates a struct with the specified field names and values.](https://docs.starrocks.io/docs/sql-reference/sql-functions/struct-functions/named_struct.md) #### [📄️ row](https://docs.starrocks.io/docs/sql-reference/sql-functions/struct-functions/row.md) [Creates a named STRUCT or ROW value from the given values.](https://docs.starrocks.io/docs/sql-reference/sql-functions/struct-functions/row.md) --- ### named_struct Creates a struct with the specified field names and values. This function is supported from v3.1 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext STRUCT named_struct({STRING name1, ANY val1} [, ...] ) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `nameN`: A STRING field. * `valN`: An expression of any type that specifies the value for field N. Values are nullable. The expressions of names and values must be in pairs. Otherwise, the struct cannot be created. You must pass at least one pair of field name and value, separated by a comma (`,`). #### Return value[​](#return-value "Direct link to Return value") Returns a STRUCT value. #### Examples[​](#examples "Direct link to Examples") ```plain SELECT named_struct('a', 1, 'b', 2, 'c', 3); +--------------------------------------+ | named_struct('a', 1, 'b', 2, 'c', 3) | +--------------------------------------+ | {"a":1,"b":2,"c":3} | +--------------------------------------+ SELECT named_struct('a', null, 'b', 2, 'c', 3); +-----------------------------------------+ | named_struct('a', null, 'b', 2, 'c', 3) | +-----------------------------------------+ | {"a":null,"b":2,"c":3} | +-----------------------------------------+ ``` #### References[​](#references "Direct link to References") * [STRUCT data type](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/STRUCT.md) * [row/struct](https://docs.starrocks.io/docs/sql-reference/sql-functions/struct-functions/row.md) --- ### row Creates a named STRUCT or ROW value from the given values. It supports unnamed struct. You do not need to specify the field names. StarRocks automatically generates column names, such as `col1, col2,...`. This function is supported from v3.1 onwards. struct() is the alias of row(). #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext STRUCT row(ANY val, ...) ``` #### Parameters[​](#parameters "Direct link to Parameters") `val`: an expression of any supported type. This function is a variable argument function. You must pass at least one argument. `value` is nullable. Separate multiple values with a comma (`,`). #### Return value[​](#return-value "Direct link to Return value") Returns a STRUCT value which consists of the input values. #### Examples[​](#examples "Direct link to Examples") ```plaintext select row(1,"Apple","Pear"); +-----------------------------------------+ | row(1, 'Apple', 'Pear') | +-----------------------------------------+ | {"col1":1,"col2":"Apple","col3":"Pear"} | +-----------------------------------------+ select row("Apple", NULL); +------------------------------+ | row('Apple', NULL) | +------------------------------+ | {"col1":"Apple","col2":null} | +------------------------------+ select struct(1,2,3); +------------------------------+ | row(1, 2, 3) | +------------------------------+ | {"col1":1,"col2":2,"col3":3} | +------------------------------+ ``` #### References[​](#references "Direct link to References") * [STRUCT data type](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/STRUCT.md) * [named\_struct](https://docs.starrocks.io/docs/sql-reference/sql-functions/struct-functions/named_struct.md) --- ### Table #### [📄️ FILES](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) [Table function for defining data files in remote storage (HDFS, S3, GCS) for use in loading data into or unloading data from StarRocks.](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) #### [📄️ generate\_series](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/generate_series.md) [Generates a series of values within the interval specified by start and end, and with an optional step.](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/generate_series.md) #### [📄️ native\_query](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/native_query.md) [native\_query is a JDBC catalog table function.](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/native_query.md) --- ### FILES ### `FILES` Defines data files in remote storage, used for loading and unloading data: * [Load or query data from a remote storage system](#files-for-loading) * [Unload data into a remote storage system](#files-for-unloading) `FILES()` supports the following data sources and file formats: * **Data sources:** * HDFS * AWS S3 * Google Cloud Storage * Other S3-compatible storage system * Microsoft Azure Blob Storage * NFS(NAS) * **File formats:** * Parquet * ORC (Supported from v3.3 onwards) * CSV (Supported from v3.3 onwards) * Avro (Supported from v3.4.4 onwards and for loading only) From v3.2 onwards, FILES() further supports complex data types including `ARRAY`, `JSON`, `MAP`, and `STRUCT` in addition to basic data types. #### `FILES()` for loading[​](#files-for-loading "Direct link to files-for-loading") From v3.1.0 onwards, StarRocks supports defining read-only files in remote storage using the table function `FILES()`. It can access remote storage with the path-related properties of the files, infers the table schema of the data in the files, and returns the data rows. You can directly query the data rows using [`SELECT`](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md), load the data rows into an existing table using [`INSERT`](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md), or create a new table and load the data rows into it using [`CREATE TABLE AS SELECT`](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md). From v3.3.4, you can also view the schema of a data file using `FILES()` with [`DESC`](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md). ##### Syntax[​](#syntax "Direct link to Syntax") ```sql FILES( data_location , [data_format] [, schema_detect ] [, StorageCredentialParams ] [, columns_from_path ] [, list_files_only ] [, list_recursively]) ``` ##### Parameters[​](#parameters "Direct link to Parameters") All parameters are in the `"key" = "value"` pairs. ###### `data_location`[​](#data_location "Direct link to data_location") The URI used to access the files. You can specify a path or a file. For example, you can specify this parameter as `"hdfs://:/user/data/tablename/20210411"` to load a data file named `20210411` from the path `/user/data/tablename` on the HDFS server. You can also specify this parameter as the save path of multiple data files by using wildcards `?`, `*`, `[]`, or `^`. For example, you can specify this parameter as `"hdfs://:/user/data/tablename/*/*"` or `"hdfs://:/user/data/tablename/dt=202104*/*"` to load the data files from all partitions or only `202104` partitions in the path `/user/data/tablename` on the HDFS server. note Wildcards can also be used to specify intermediate paths. * To access HDFS, you need to specify this parameter as: ```sql "path" = "hdfs://:/" -- Example: "path" = "hdfs://127.0.0.1:9000/path/file.parquet" ``` * To access AWS S3: * If you use the S3 protocol, you need to specify this parameter as: ```sql "path" = "s3://" -- Example: "path" = "s3://path/file.parquet" ``` * If you use the S3A protocol, you need to specify this parameter as: ```sql "path" = "s3a://" -- Example: "path" = "s3a://path/file.parquet" ``` * To access Google Cloud Storage, you need to specify this parameter as: ```sql "path" = "s3a://" -- Example: "path" = "s3a://path/file.parquet" ``` * To access Azure Blob Storage: * If your storage account allows access over HTTP, you need to specify this parameter as: ```sql "path" = "wasb://@.blob.core.windows.net/" -- Example: "path" = "wasb://testcontainer@testaccount.blob.core.windows.net/path/file.parquet" ``` * If your storage account allows access over HTTPS, you need to specify this parameter as: ```sql "path" = "wasbs://@.blob.core.windows.net/" -- Example: "path" = "wasbs://testcontainer@testaccount.blob.core.windows.net/path/file.parquet" ``` * To access NFS(NAS): ```sql "path" = "file:///" -- Example: "path" = "file:///home/ubuntu/parquetfile/file.parquet" ``` note To access NFS(NAS) via the `file://` protocol, mount the same NAS device as NFS under the same directory on the nodes that need to access the path: * For read/write operations, mount it on each FE node and each BE or CN node. FE nodes list the files and infer the file schema, and BE/CN nodes read the data. * For write only operations, mount it on each BE or CN node. ###### `data_format`[​](#data_format "Direct link to data_format") The format of the data file. Valid values: * `parquet` * `orc` (Supported from v3.3 onwards) * `csv` (Supported from v3.3 onwards) * `avro` (Supported from v3.4.4 onwards and for loading only) You must set detailed options for specific data file formats. When `list_files_only` is set to `true`, you do not need to specify `data_format`. ###### Parquet[​](#parquet "Direct link to Parquet") Example of the Parquet format: ```sql "format"="parquet", "parquet.use_legacy_encoding" = "true", -- for unloading only "parquet.version" = "2.6" -- for unloading only ``` When reading Parquet files (for example, via `FILES()` or Broker Load), StarRocks maps the Parquet TIMESTAMP logical type to DATETIME according to the type's `isAdjustedToUTC` attribute: * **Instant Semantics**: If `isAdjustedToUTC` is `true`, the value identifies an instant on the timeline normalized to UTC. StarRocks converts it to the wall-clock reading in the current session time zone. * **Local Semantics**: If `isAdjustedToUTC` is `false`, the value is a wall-clock reading without a time zone. StarRocks returns it as written, regardless of the session time zone. * The legacy INT96 physical type carries no `isAdjustedToUTC` attribute. StarRocks treats an INT96 column as an instant normalized to UTC and converts it to the session time zone, whether the INT96 timestamp is a top-level column or nested inside a STRUCT, ARRAY, or MAP. note **Behavior change**: In earlier versions, StarRocks shifted timestamps with Local Semantics (`isAdjustedToUTC` = `false`) by the session time zone offset on read. Such values are now returned as written. If your session time zone is not UTC, the same file now returns different values than in earlier versions. The new values are correct per the Parquet specification. ###### `parquet.use_legacy_encoding`[​](#parquetuse_legacy_encoding "Direct link to parquetuse_legacy_encoding") Controls the encoding technique used for DATETIME and DECIMAL data types. Valid values: `true` and `false` (default). This property is only supported for data unloading. If this item is set to `true`: * For DECIMAL type, the system uses `fixed_len_byte_array` encoding. * For DATETIME type, the system uses `INT96` encoding. If this item is set to `false`: * For DECIMAL type, the system uses `INT32` or `INT64` encoding. * For DATETIME type, the system uses `INT64` encoding. * **Instant Semantics**: If `isAdjustedToUTC` is set to `true` for the Parquet TIMESTAMP type, the system outputs a timestamp normalized to UTC. Each value unambiguously identifies a single instant on the timeline, and can be transferred into a specific timezone. * **Local Semantics**: If `isAdjustedToUTC` is set to `false` for the Parquet TIMESTAMP type, the system outputs a timestamp that represents the year, month, day, hour, minute, second, and sub-second in a local timezone, regardless of what specific timezone is considered local. Such values are always displayed the same way, regardless of the local timezone in effect, and do not identify instants on the timeline. note For DECIMAL 128 data type, only `fixed_len_byte_array` encoding is available. `parquet.use_legacy_encoding` does not take effect. ###### `parquet.version`[​](#parquetversion "Direct link to parquetversion") Controls the Parquet version into which the system unloads data. Supported from v3.4.6 onwards. Valid values: `1.0`, `2.4`, and `2.6` (default). This property is only supported for data unloading. ###### CSV[​](#csv "Direct link to CSV") Example for the CSV format: ```sql "format"="csv", "csv.column_separator"="\\t", "csv.enclose"='"', "csv.skip_header"="1", -- for loading only "csv.escape"="\\" ``` ###### `csv.column_separator`[​](#csvcolumn_separator "Direct link to csvcolumn_separator") Specifies the column separator used when the data file is in CSV format. If you do not specify this parameter, this parameter defaults to `\\t`, indicating tab. The column separator you specify using this parameter must be the same as the column separator that is actually used in the data file. Otherwise, the load job will fail due to inadequate data quality. Tasks that use Files() are submitted according to the MySQL protocol. StarRocks and MySQL both escape characters in the load requests. Therefore, if the column separator is an invisible character such as tab, you must add a backslash (`\`) preceding the column separator. For example, you must input `\\t` if the column separator is `\t`, and you must input `\\n` if the column separator is `\n`. Apache Hive™ files use `\x01` as their column separator, so you must input `\\x01` if the data file is from Hive. note * For CSV data, you can use a UTF-8 string, such as a comma (,), tab, or pipe (|), whose length does not exceed 50 bytes as a text delimiter. > - Null values are denoted by using `\N`. For example, a data file consists of three columns, and a record from that data file holds data in the first and third columns but no data in the second column. In this situation, you need to use `\N` in the second column to denote a null value. This means the record must be compiled as `a,\N,b` instead of `a,,b`. `a,,b` denotes that the second column of the record holds an empty string. ###### `csv.enclose`[​](#csvenclose "Direct link to csvenclose") Specifies the character that is used to wrap the field values in the data file according to RFC4180 when the data file is in CSV format. Type: single-byte character. Default value: `NONE`. The most prevalent characters are single quotation mark (`'`) and double quotation mark (`"`). All special characters (including row separators and column separators) wrapped by using the `enclose`-specified character are considered normal symbols. StarRocks can do more than RFC4180 as it allows you to specify any single-byte character as the `enclose`-specified character. If a field value contains an `enclose`-specified character, you can use the same character to escape that `enclose`-specified character. For example, you set `enclose` to `"`, and a field value is `a "quoted" c`. In this case, you can enter the field value as `"a ""quoted"" c"` into the data file. ###### `csv.skip_header`[​](#csvskip_header "Direct link to csvskip_header") Specifies the number of header rows to skip in the CSV-formatted data. Type: `INTEGER`. Default value: `0`. This property is only supported for data loading. In some CSV-formatted data files, a number of header rows are used to define metadata such as column names and column data types. By setting the `skip_header` parameter, you can enable StarRocks to skip these header rows. For example, if you set this parameter to `1`, StarRocks skips the first row of the data file during data loading. The header rows in the data file must be separated by using the row separator that you specify in the load statement. ###### `csv.escape`[​](#csvescape "Direct link to csvescape") Specifies the character that is used to escape various special characters, such as row separators, column separators, escape characters, and `enclose`-specified characters, which are then considered by StarRocks to be common characters and are parsed as part of the field values in which they reside. Type: single-byte character. Default value: `NONE`. The most prevalent character is slash (`\`), which must be written as double slashes (`\\`) in SQL statements. note The character specified by `escape` is applied to both inside and outside of each pair of `enclose`-specified characters. > Two examples are as follows: > > * When you set `enclose` to `"` and `escape` to `\`, StarRocks parses `"say \"Hello world\""` into `say "Hello world"`. > * Assume that the column separator is comma (`,`). When you set `escape` to `\`, StarRocks parses `a, b\, c` into two separate field values: `a` and `b, c`. ###### `schema_detect`[​](#schema_detect "Direct link to schema_detect") From v3.2 onwards, `FILES()` supports automatic schema detection and unionization of the same batch of data files. StarRocks first detects the schema of the data by sampling certain data rows of a random data file in the batch. Then, StarRocks unionizes the columns from all the data files in the batch. You can configure the sampling rule using the following parameters: * `auto_detect_sample_files`: the number of random data files to sample in each batch. By default, the first and last files are selected. Range: `[0, + ∞]`. Default: `2`. * `auto_detect_sample_rows`: the number of data rows to scan in each sampled data file. Range: `[0, + ∞]`. Default: `500`. * `auto_detect_types`: (valid for CSV only) - whether to guess the data types of sampled columns, or just assume String. `{true | false}`. Default: `true`. After the sampling, StarRocks unionizes the columns from all the data files according to these rules: * For columns with different column names or indices, each column is identified as an individual column, and, eventually, the union of all individual columns is returned. * For columns with the same column name but different data types, they are identified as the same column but with a general data type on a relative fine granularity level. For example, if the column `col1` in file A is `INT` but `DECIMAL` in file B, `DOUBLE` is used in the returned column. * All integer columns will be unionized as an integer type on an overall rougher granularity level. * Integer columns together with `FLOAT` type columns will be unionized as the DECIMAL type. * String types are used for unionizing other types. * Generally, the `STRING` type can be used to unionize all data types. * If type auto-detection is turned off, all columns will return as `STRING` You can refer to Example 5. If StarRocks fails to unionize all the columns, it generates a schema error report that includes the error information and all the file schemas. important All data files in a single batch must be of the same file format. ###### Push down target table column types / schema[​](#push-down-target-table-column-types--schema "Direct link to Push down target table column types / schema") From v3.4.0 onwards, the system supports pushing down target table column types to the Scan stage of `FILES()` to improve type inference accuracy. Schema detection of `FILES()` is not fully strict. For example, any integer column in CSV files is inferred as the BIGINT type when the function is reading the files. In this case, if the corresponding column in the target table is the `TINYINT` type, the CSV data records that exceed the TINYINT range will not be filtered. Instead, they will be filled with `NULL` implicitly. To address this issue, the system introduces the dynamic FE configuration item `files_enable_insert_push_down_column_type` (alias: `files_enable_insert_push_down_schema`) to control whether to push down the target table column types to the Scan stage of `FILES()`. By setting `files_enable_insert_push_down_column_type` to `true`, the system will rewrite the types of matched file columns with the target table column types at the file reading stage. Only columns that already exist in the inferred file schema are affected; no columns are added or removed. For full schema push-down (both column names and types), you can set the INSERT property `enable_push_down_schema` to `true`. Unlike the FE configuration, this is specified on the INSERT statement itself rather than in the `FILES()` properties: ```sql INSERT INTO target_table PROPERTIES ("enable_push_down_schema" = "true") SELECT * FROM FILES( "path" = "s3://...", "format" = "parquet" ); ``` When `enable_push_down_schema` is `true`, StarRocks reshapes the `FILES()` schema to match the target table columns — adding columns that are missing from the inferred schema and trimming the schema to only the columns actually read by the SELECT list. Note that in `BY NAME` mode with `SELECT *`, the `*` is expanded to the target table's column names rather than the file's original columns, so extra file columns are dropped even with `SELECT *`. ###### Union files with different schema[​](#union-files-with-different-schema "Direct link to Union files with different schema") From v3.4.0 onwards, the system supports unionizing files with different schema, and by default, an error will be returned if there are non-existent columns. By setting the property `fill_mismatch_column_with` to `null`, you can allow the system to assign `NULL` values to the non-existent columns instead of returning an error. `fill_mismatch_column_with`: The behavior of the system after a non-existent column is detected when unionizing files with different schema. Valid values: * `none`: An error will be returned if a non-existent column is detected. * `null`: NULL values will be assigned to the non-existent column. For example, the files to read are from different partitions of a Hive table, and Schema Change has been performed on the newer partitions. When reading both new and old partitions, you can set `fill_mismatch_column_with` to `null`, and the system will unionize the schema of the new and old partition files, and assign NULL values to the non-existent columns. The system unionizes the schema of Parquet and ORC files based on the column names, and that of CSV files based on the position (order) of the columns. ###### Infer STRUCT type from Parquet[​](#infer-struct-type-from-parquet "Direct link to Infer STRUCT type from Parquet") From v3.4.0 onwards, `FILES()` supports inferring the `STRUCT` type data from Parquet files. ###### `StorageCredentialParams`[​](#storagecredentialparams "Direct link to storagecredentialparams") The authentication information used by StarRocks to access your storage system. StarRocks currently supports accessing HDFS with the simple authentication, accessing AWS S3 and GCS with the IAM user-based authentication, and accessing Azure Blob Storage with Shared Key, SAS Token, Managed Identity, and Service Principal. ###### HDFS[​](#hdfs "Direct link to HDFS") * Use the simple authentication to access HDFS: ```sql "hadoop.security.authentication" = "simple", "username" = "xxxxxxxxxx", "password" = "yyyyyyyyyy" ``` | **Key** | **Required** | **Description** | | -------------------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------- | | `hadoop.security.authentication` | No | The authentication method. Valid value: `simple` (Default). `simple` represents simple authentication, meaning no authentication. | | `username` | Yes | The username of the account that you want to use to access the NameNode of the HDFS cluster. | | `password` | Yes | The password of the account that you want to use to access the NameNode of the HDFS cluster. | * Use the Kerberos authentication to access HDFS: Currently, FILES() supports Kerberos authentication with HDFS only via the configuration file **`hdfs-site.xml`** placed under the **`fe/conf`**, **`be/conf`**, and **`cn/conf`** directories. In addition, you need to append the following option in the configuration item `JAVA_OPTS` in each FE configuration file **`fe.conf`**, BE configuration file **`be.conf`**, and CN configuration file **`cn.conf`**: ```plain # Specify the local path to which the Kerberos configuration file is stored. -Djava.security.krb5.conf= ``` Example: ```properties JAVA_OPTS="-Xlog:gc*:${LOG_DIR}/be.gc.log.$DATE:time -XX:ErrorFile=${LOG_DIR}/hs_err_pid%p.log -Djava.security.krb5.conf=/etc/krb5.conf" ``` You also need to run the `kinit` command on each FE, BE, and CN node to obtain Ticket Granting Ticket (TGT) from Key Distribution Center (KDC). ```bash kinit -kt ``` To run this command, the principal you use must have the write access to your HDFS cluster. In addition, you need to set a crontab for the command to schedule the task by a specific interval, thus preventing the authentication from expiring. Example: ```bash # Renew TGT every 6 hours. 0 */6 * * * kinit -kt sr.keytab sr/test.starrocks.com@STARROCKS.COM > /tmp/kinit.log ``` * Access HDFS with HA mode enabled: Currently, `FILES()` supports access to HDFS with HA mode enabled only via the configuration file **`hdfs-site.xml`** placed under the **`fe/conf`**, **`be/conf`**, and **`cn/conf`** directories. ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") If you choose AWS S3 as your storage system, take one of the following actions: * To choose the instance profile-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "" ``` * To choose the assumed role-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.region" = "" ``` * To choose the IAM user-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ----------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `aws.s3.use_instance_profile` | Yes | Specifies whether to enable the credential methods instance profile and assumed role. Valid values: `true` and `false`. Default value: `false`. | | `aws.s3.iam_role_arn` | No | The ARN of the IAM role that has privileges on your AWS S3 bucket. If you choose assumed role as the credential method for accessing AWS S3, you must specify this parameter. | | `aws.s3.region` | Yes | The region in which your AWS S3 bucket resides. Example: `us-west-1`. | | `aws.s3.access_key` | No | The access key of your IAM user. If you choose IAM user as the credential method for accessing AWS S3, you must specify this parameter. | | `aws.s3.secret_key` | No | The secret key of your IAM user. If you choose IAM user as the credential method for accessing AWS S3, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in AWS IAM Console, see [Authentication parameters for accessing AWS S3](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-s3). ###### AWS STS Regional endpoints[​](#aws-sts-regional-endpoints "Direct link to AWS STS Regional endpoints") [AWS Security Token Service](https://docs.aws.amazon.com/sdkref/latest/guide/feature-sts-regionalized-endpoints.html) (AWS STS) is available both as a global and Regional service. | Parameter | Required | Description | | --------------------- | -------- | ------------------------------------------------------------------------ | | `aws.s3.sts.region` | No | The region of the AWS Security Token Service to access. | | `aws.s3.sts.endpoint` | No | Used to override the default endpoint of the AWS Security Token Service. | important When using AWS STS endpoints for authentication and accessing data in S3 compatible storage outside of S3, you must set `aws.s3.use_instance_profile` to `false`. ###### Google GCS[​](#google-gcs "Direct link to Google GCS") If you choose Google GCS as your storage system, take one of the following actions: * To choose the VM-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.use_compute_engine_service_account" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | -------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | `gcp.gcs.use_compute_engine_service_account` | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | * To choose the service account-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------- | ----------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------- | | `gcp.gcs.service_account_email` | "" | `"user@hello.iam.gserviceaccount.com"` | The email address in the JSON file generated at the creation of the service account. | | `gcp.gcs.service_account_private_key_id` | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the service account. | | `gcp.gcs.service_account_private_key` | "" | "`-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n`" | The private key in the JSON file generated at the creation of the service account. | * To choose the impersonation-based authentication method, configure `StorageCredentialParams` as follows: * Make a VM instance impersonate a service account: ```sql "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | -------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | `gcp.gcs.use_compute_engine_service_account` | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | | `gcp.gcs.impersonation_service_account` | "" | "hello" | The service account that you want to impersonate. | * Make a service account (named as meta service account) impersonate another service account (named as data service account): ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------- | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | `gcp.gcs.service_account_email` | "" | `"user@hello.iam.gserviceaccount.com"` | The email address in the JSON file generated at the creation of the meta service account. | | `gcp.gcs.service_account_private_key_id` | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the meta service account. | | `gcp.gcs.service_account_private_key` | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the meta service account. | | `gcp.gcs.impersonation_service_account` | "" | "hello" | The data service account that you want to impersonate. | ###### Azure Blob Storage[​](#azure-blob-storage "Direct link to Azure Blob Storage") * Use Shared Key to access Azure Blob Storage: ```sql "azure.blob.shared_key" = "" ``` | **Key** | **Required** | **Description** | | ----------------------- | ------------ | ------------------------------------------------------------------------- | | `azure.blob.shared_key` | Yes | The Shared Key that you can use to access the Azure Blob Storage account. | * Use SAS Token to access Azure Blob Storage: ```sql "azure.blob.sas_token" = "" ``` | **Key** | **Required** | **Description** | | ---------------------- | ------------ | ------------------------------------------------------------------------ | | `azure.blob.sas_token` | Yes | The SAS token that you can use to access the Azure Blob Storage account. | * Use Managed Identity to access Azure Blob Storage (Supported from v3.4.4 onwards): note * Only User-assigned Managed Identities with Client ID credentials are supported. * The FE dynamic configuration `azure_use_native_sdk` (Default: `true`) controls whether to allow the system to use authentication with Managed Identity and Service Principals. ```sql "azure.blob.oauth2_use_managed_identity" = "true", "azure.blob.oauth2_client_id" = "" ``` | **Key** | **Required** | **Description** | | ---------------------------------------- | ------------ | ------------------------------------------------------------------------------------------------ | | `azure.blob.oauth2_use_managed_identity` | Yes | Whether to use Managed Identity to access the Azure Blob Storage account. Set it to `true`. | | `azure.blob.oauth2_client_id` | Yes | The Client ID of the Managed Identity that you can use to access the Azure Blob Storage account. | * Use Service Principal to access Azure Blob Storage (Supported from v3.4.4 onwards): note * Only Client Secret credentials are supported. * The FE dynamic configuration `azure_use_native_sdk` (Default: `true`) controls whether to allow the system to use authentication with Managed Identity and Service Principals. ```sql "azure.blob.oauth2_client_id" = "", "azure.blob.oauth2_client_secret" = "", "azure.blob.oauth2_tenant_id" = "" ``` | **Key** | **Required** | **Description** | | --------------------------------- | ------------ | ----------------------------------------------------------------------------------------------------- | | `azure.blob.oauth2_client_id` | Yes | The Client ID of the Service Principal that you can use to access the Azure Blob Storage account. | | `azure.blob.oauth2_client_secret` | Yes | The Client Secret of the Service Principal that you can use to access the Azure Blob Storage account. | | `azure.blob.oauth2_tenant_id` | Yes | The Tenant ID of the Service Principal that you can use to access the Azure Blob Storage account. | ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2 "Direct link to Azure Data Lake Storage Gen2") If you choose Data Lake Storage Gen2 as your storage system, take one of the following actions: * To choose the Managed Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ----------------------------------------- | ------------ | ------------------------------------------------------------------------------------------------ | | `azure.adls2.oauth2_use_managed_identity` | Yes | Specifies whether to enable the Managed Identity authentication method. Set the value to `true`. | | `azure.adls2.oauth2_tenant_id` | Yes | The ID of the tenant whose data you want to access. | | `azure.adls2.oauth2_client_id` | Yes | The client (application) ID of the managed identity. | * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ----------------------------- | ------------ | -------------------------------------------------------------- | | `azure.adls2.storage_account` | Yes | The username of your Data Lake Storage Gen2 storage account. | | `azure.adls2.shared_key` | Yes | The shared key of your Data Lake Storage Gen2 storage account. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------ | ------------ | -------------------------------------------------------------------------- | | `azure.adls2.oauth2_client_id` | Yes | The client (application) ID of the service principal. | | `azure.adls2.oauth2_client_secret` | Yes | The value of the new client (application) secret created. | | `azure.adls2.oauth2_client_endpoint` | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | * To choose the Workload Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_token_file" = "", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_token\_file | Yes | The absolute file path to the OAuth2 token file projected into the pod by the Azure Workload Identity webhook. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client ID (application ID) of the Azure AD application (user-assigned managed identity or app registration) associated with the workload identity. | ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1 "Direct link to Azure Data Lake Storage Gen1") If you choose Data Lake Storage Gen1 as your storage system, take one of the following actions: * To choose the Managed Service Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.use_managed_service_identity" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------ | ------------ | -------------------------------------------------------------------------------------------------------- | | `azure.adls1.use_managed_service_identity` | Yes | Specifies whether to enable the Managed Service Identity authentication method. Set the value to `true`. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------- | ------------ | -------------------------------------------------------------------------- | | `azure.adls1.oauth2_client_id` | Yes | The client (application) ID of the . | | `azure.adls1.oauth2_credential` | Yes | The value of the new client (application) secret created. | | `azure.adls1.oauth2_endpoint` | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | ###### Other S3-compatible storage system[​](#other-s3-compatible-storage-system "Direct link to Other S3-compatible storage system") If you choose other S3-compatible storage system, such as MinIO, configure `StorageCredentialParams` as follows: ```sql "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | --------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `aws.s3.enable_ssl` | Yes | Specifies whether to enable SSL connection. Valid values: `true` and `false`. Default value: `true`. | | `aws.s3.enable_path_style_access` | Yes | Specifies whether to enable path-style URL access. Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`. | | `aws.s3.endpoint` | Yes | The endpoint that is used to connect to your S3-compatible storage system instead of AWS S3. | | `aws.s3.access_key` | Yes | The access key of your IAM user. | | `aws.s3.secret_key` | Yes | The secret key of your IAM user. | ###### `columns_from_path`[​](#columns_from_path "Direct link to columns_from_path") From v3.2 onwards, StarRocks can extract the value of a key/value pair from the file path as the value of a column. ```sql "columns_from_path" = " [, ...]" ``` Suppose the data file **file1** is stored under a path in the format of `/geo/country=US/city=LA/`. You can specify the `columns_from_path` parameter as `"columns_from_path" = "country, city"` to extract the geographic information in the file path as the value of columns that are returned. For further instructions, see Example 4. ###### `schema`[​](#schema "Direct link to schema") From v4.1.2 onwards, `FILES()` supports an explicit `schema` parameter that lets you declare exactly which columns to read and their StarRocks types, bypassing BE-side schema inference. ```sql "schema" = "col_name TYPE[, col_name TYPE ...]" ``` When `schema` is set, `FILES()` reads only the declared columns with the declared types. Automatic schema detection (sampling-based inference) is skipped, which makes query behavior predictable when the underlying files evolve differently across batches. ###### Supported types[​](#supported-types "Direct link to Supported types") All StarRocks scalar and complex types (`ARRAY`, `MAP`, `STRUCT`) are supported inside `schema`, except StarRocks-only aggregate types (`HLL`, `BITMAP`, `PERCENTILE`), which have no representation in Parquet/ORC/Avro/CSV and are rejected at any depth (top level or inside `ARRAY` / `MAP` value / `STRUCT` field). For `STRUCT`, you may declare only a subset of subfields; undeclared subfields are ignored during projection. Example of a partial nested declaration: ```sql "schema" = "request_data STRUCT, now BIGINT>" ``` ###### Forbidden tokens inside the schema string[​](#forbidden-tokens-inside-the-schema-string "Direct link to Forbidden tokens inside the schema string") The following tokens are rejected and produce a validation error: * `NULL` / `NOT NULL` * `DEFAULT` * `COMMENT` * `KEY` (and related key-type descriptors) * `AUTO_INCREMENT` * Charset specifiers (for example `CHARACTER SET`) * Aggregation descriptors (for example `SUM`, `REPLACE`) * Generated-column clauses (`AS (...)`) Example (invalid): ```sql "schema" = "id BIGINT NOT NULL, dt DATE DEFAULT '2026-01-01'" ``` ###### Column matching semantics per format[​](#column-matching-semantics-per-format "Direct link to Column matching semantics per format") * **Parquet / ORC / Avro**: schema columns are matched against file columns **by name**, and matching is **case-sensitive**. For example, `UserId` in `schema` does not match `userid` in the file. * **CSV**: schema columns are matched **by position**. The names in `schema` act only as aliases for the ordinal columns in the CSV file. The first schema item maps to the first CSV column, the second to the second, and so on. ###### Mutual exclusions[​](#mutual-exclusions "Direct link to Mutual exclusions") `schema` cannot be combined with any of the auto-detection parameters. Using any of the following together with `schema` is a validation error: * `auto_detect_sample_files` * `auto_detect_sample_rows` * `auto_detect_types` ###### Interactions with other properties and statements[​](#interactions-with-other-properties-and-statements "Direct link to Interactions with other properties and statements") * **`fill_mismatch_column_with`**: Declared columns that are missing from some files follow the existing `fill_mismatch_column_with` behavior — `none` fails the query, `null` fills `NULL` for the missing column. * **`columns_from_path`**: Path-extracted columns are appended **after** the `schema` columns. If a `columns_from_path` name collides with a name in `schema`, the query fails with a validation error. * **`list_files_only = true`**: `schema` is silently ignored when `list_files_only` is `true` (only file metadata is returned). * **`DESC FILES(..., "schema" = ...)`**: Explicitly rejected. Use `DESC FILES(...)` without `schema` to inspect the inferred file schema. * **`INSERT INTO FILES(..., "schema" = ...)` (unload)**: Explicitly rejected. `schema` is a read-path parameter only. * **INSERT push-down interactions**: * The FE configuration `files_enable_insert_push_down_column_type` (alias `files_enable_insert_push_down_schema`) is **silently skipped** when `schema` is set, because the user-declared types already determine column types. * Combining `schema` with the INSERT property `enable_push_down_schema = true` is a validation error. ###### Examples[​](#examples "Direct link to Examples") Parquet matched by name: ```sql SELECT user_id, event_time FROM FILES( "path" = "s3://bucket/path/*.parquet", "format" = "parquet", "schema" = "user_id BIGINT, event_time DATETIME" ); ``` CSV matched by position (the names `a` and `b` are aliases for the first and second CSV columns): ```sql SELECT * FROM FILES( "path" = "s3://bucket/path/*.csv", "format" = "csv", "csv.column_separator" = ",", "schema" = "a BIGINT, b VARCHAR(64)" ); ``` Partial nested declaration (only two subfields of `request_data` are projected): ```sql SELECT request_data.device_data.platform, request_data.now FROM FILES( "path" = "s3://bucket/path/*.parquet", "format" = "parquet", "schema" = "request_data STRUCT, now BIGINT>" ); ``` ###### `list_files_only`[​](#list_files_only "Direct link to list_files_only") From v3.4.0 onwards, `FILES()` supports only list the files when reading them. ```sql "list_files_only" = "true" ``` Please note that you do not need to specify `data_format` when `list_files_only` is set to `true`. For more information, see [Return](#return). ###### `list_recursively`[​](#list_recursively "Direct link to list_recursively") StarRocks further supports `list_recursively` to list the files and directories recursively. `list_recursively` only takes effect when `list_files_only` is set to `true`. The default value is `false`. ```sql "list_files_only" = "true", "list_recursively" = "true" ``` When both `list_files_only` and `list_recursively` are set to `true`, StarRocks will do the follows: * If the specified `path` is a file (whether it is specified specifically or represented by wildcards), StarRocks will show the information of the file. * If the specified `path` is a directory (whether it is specified specifically or represented by wildcards, and whether or not it is suffixed by `/`), StarRocks will show all the files and sub-directories under this directory. For more information, see [Return](#return). ##### Return[​](#return "Direct link to Return") ###### `SELECT FROM FILES()`[​](#select-from-files "Direct link to select-from-files") When used with SELECT, FILES() returns the data in the file as a table. * When querying CSV files, you can use `$1`, `$2`, and so on, to represent each column in the SELECT statement, or specify `*` to obtain data from all columns. ```sql SELECT * FROM FILES( "path" = "s3://inserttest/csv/file1.csv", "format" = "csv", "csv.column_separator"=",", "csv.row_delimiter"="\n", "csv.enclose"='"', "csv.skip_header"="1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.region" = "us-west-2" ) WHERE $1 > 5; +------+---------+------------+ | $1 | $2 | $3 | +------+---------+------------+ | 6 | 0.34413 | 2017-11-25 | | 7 | 0.40055 | 2017-11-26 | | 8 | 0.42437 | 2017-11-27 | | 9 | 0.67935 | 2017-11-27 | | 10 | 0.22783 | 2017-11-29 | +------+---------+------------+ 5 rows in set (0.30 sec) SELECT $1, $2 FROM FILES( "path" = "s3://inserttest/csv/file1.csv", "format" = "csv", "csv.column_separator"=",", "csv.row_delimiter"="\n", "csv.enclose"='"', "csv.skip_header"="1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.region" = "us-west-2" ); +------+---------+ | $1 | $2 | +------+---------+ | 1 | 0.71173 | | 2 | 0.16145 | | 3 | 0.80524 | | 4 | 0.91852 | | 5 | 0.37766 | | 6 | 0.34413 | | 7 | 0.40055 | | 8 | 0.42437 | | 9 | 0.67935 | | 10 | 0.22783 | +------+---------+ 10 rows in set (0.38 sec) ``` * When querying Parquet or ORC files, you can directly specify the name of the desired columns in the SELECT statement, or specify `*` to obtain data from all columns. ```sql SELECT * FROM FILES( "path" = "s3://inserttest/parquet/file2.parquet", "format" = "parquet", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.region" = "us-west-2" ) WHERE c1 IN (101,105); +------+------+---------------------+ | c1 | c2 | c3 | +------+------+---------------------+ | 101 | 9 | 2018-05-15T18:30:00 | | 105 | 6 | 2018-05-15T18:30:00 | +------+------+---------------------+ 2 rows in set (0.29 sec) SELECT c1, c3 FROM FILES( "path" = "s3://inserttest/parquet/file2.parquet", "format" = "parquet", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.region" = "us-west-2" ); +------+---------------------+ | c1 | c3 | +------+---------------------+ | 101 | 2018-05-15T18:30:00 | | 102 | 2018-05-15T18:30:00 | | 103 | 2018-05-15T18:30:00 | | 104 | 2018-05-15T18:30:00 | | 105 | 2018-05-15T18:30:00 | | 106 | 2018-05-15T18:30:00 | | 107 | 2018-05-15T18:30:00 | | 108 | 2018-05-15T18:30:00 | | 109 | 2018-05-15T18:30:00 | | 110 | 2018-05-15T18:30:00 | +------+---------------------+ 10 rows in set (0.55 sec) ``` * When you query files with `list_files_only` set to `true`, the system will return `PATH`, `SIZE`, `IS_DIR` (whether the given path is a directory), and `MODIFICATION_TIME`. ```sql SELECT * FROM FILES( "path" = "s3://bucket/*.parquet", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "list_files_only" = "true" ); +-----------------------+------+--------+---------------------+ | PATH | SIZE | IS_DIR | MODIFICATION_TIME | +-----------------------+------+--------+---------------------+ | s3://bucket/1.parquet | 5221 | 0 | 2024-08-15 20:47:02 | | s3://bucket/2.parquet | 5222 | 0 | 2024-08-15 20:54:57 | | s3://bucket/3.parquet | 5223 | 0 | 2024-08-20 15:21:00 | | s3://bucket/4.parquet | 5224 | 0 | 2024-08-15 11:32:14 | +-----------------------+------+--------+---------------------+ 4 rows in set (0.03 sec) ``` * When you query files with `list_files_only` and `list_recursively` set to `true`, the system will list the files and directories recursively. Suppose the path `s3://bucket/list/` contains the following files and sub-directories: ```plain s3://bucket/list/ ├── basic1.csv ├── basic2.csv ├── orc0 │   └── orc1 │   └── basic_type.orc ├── orc1 │   └── basic_type.orc └── parquet └── basic_type.parquet ``` List the files and directories recursively: ```plain SELECT * FROM FILES( "path"="s3://bucket/list/", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "list_files_only" = "true", "list_recursively" = "true" ); +---------------------------------------------+------+--------+---------------------+ | PATH | SIZE | IS_DIR | MODIFICATION_TIME | +---------------------------------------------+------+--------+---------------------+ | s3://bucket/list | 0 | 1 | 2024-12-24 22:15:59 | | s3://bucket/list/basic1.csv | 52 | 0 | 2024-12-24 11:35:53 | | s3://bucket/list/basic2.csv | 34 | 0 | 2024-12-24 11:35:53 | | s3://bucket/list/orc0 | 0 | 1 | 2024-12-24 11:35:53 | | s3://bucket/list/orc0/orc1 | 0 | 1 | 2024-12-24 11:35:53 | | s3://bucket/list/orc0/orc1/basic_type.orc | 1027 | 0 | 2024-12-24 11:35:53 | | s3://bucket/list/orc1 | 0 | 1 | 2024-12-24 22:16:00 | | s3://bucket/list/orc1/basic_type.orc | 1027 | 0 | 2024-12-24 22:16:00 | | s3://bucket/list/parquet | 0 | 1 | 2024-12-24 11:35:53 | | s3://bucket/list/parquet/basic_type.parquet | 2281 | 0 | 2024-12-24 11:35:53 | +---------------------------------------------+------+--------+---------------------+ 10 rows in set (0.04 sec) ``` Lists files and directories matching `orc*` in this path in a non-recursive way: ```plain SELECT * FROM FILES( "path"="s3://bucket/list/orc*", "list_files_only" = "true", "list_recursively" = "false" ); +--------------------------------------+------+--------+---------------------+ | PATH | SIZE | IS_DIR | MODIFICATION_TIME | +--------------------------------------+------+--------+---------------------+ | s3://bucket/list/orc0/orc1 | 0 | 1 | 2024-12-24 11:35:53 | | s3://bucket/list/orc1/basic_type.orc | 1027 | 0 | 2024-12-24 22:16:00 | +--------------------------------------+------+--------+---------------------+ 2 rows in set (0.03 sec) ``` ###### `DESC FILES()`[​](#desc-files "Direct link to desc-files") When used with `DESC`, `FILES()` returns the schema of the file. ```plain DESC FILES( "path" = "s3://inserttest/lineorder.parquet", "format" = "parquet", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.region" = "us-west-2" ); +------------------+------------------+------+ | Field | Type | Null | +------------------+------------------+------+ | lo_orderkey | int | YES | | lo_linenumber | int | YES | | lo_custkey | int | YES | | lo_partkey | int | YES | | lo_suppkey | int | YES | | lo_orderdate | int | YES | | lo_orderpriority | varchar(1048576) | YES | | lo_shippriority | int | YES | | lo_quantity | int | YES | | lo_extendedprice | int | YES | | lo_ordtotalprice | int | YES | | lo_discount | int | YES | | lo_revenue | int | YES | | lo_supplycost | int | YES | | lo_tax | int | YES | | lo_commitdate | int | YES | | lo_shipmode | varchar(1048576) | YES | +------------------+------------------+------+ 17 rows in set (0.05 sec) ``` When you viewing files with `list_files_only` set to `true`, the system will return the `Type` and `Null` properties of `PATH`, `SIZE`, `IS_DIR` (whether the given path is a directory), and `MODIFICATION_TIME`. ```plain DESC FILES( "path" = "s3://bucket/*.parquet", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "list_files_only" = "true" ); +-------------------+------------------+------+ | Field | Type | Null | +-------------------+------------------+------+ | PATH | varchar(1048576) | YES | | SIZE | bigint | YES | | IS_DIR | boolean | YES | | MODIFICATION_TIME | datetime | YES | +-------------------+------------------+------+ 4 rows in set (0.00 sec) ``` #### `FILES()` for unloading[​](#files-for-unloading "Direct link to files-for-unloading") From v3.2.0 onwards, `FILES()` supports writing data into files in remote storage. You can use `INSERT INTO FILES()` to unload data from StarRocks to remote storage. ##### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql FILES( data_location , data_format [, StorageCredentialParams ] , unload_data_param ) ``` ##### Parameters[​](#parameters-1 "Direct link to Parameters") All parameters are in the `"key" = "value"` pairs. ###### data\_location[​](#data_location-1 "Direct link to data_location") See [`FILES()` for loading - Parameters - data\_location](#data_location). ###### data\_format[​](#data_format-1 "Direct link to data_format") See [`FILES()` for loading - Parameters - data\_format](#data_format). ###### StorageCredentialParams[​](#storagecredentialparams-1 "Direct link to StorageCredentialParams") See [`FILES()` for loading - Parameters - StorageCredentialParams](#storagecredentialparams). ###### `unload_data_param`[​](#unload_data_param "Direct link to unload_data_param") ```sql unload_data_param ::= "compression" = { "uncompressed" | "gzip" | "snappy" | "zstd | "lz4" }, "partition_by" = " [, ...]", "single" = { "true" | "false" } , "target_max_file_size" = "", "csv.column_separator" = "", "csv.row_delimiter" = "", "csv.include_header" = { "true" | "false" }, "csv.enclose" = "", "csv.escape" = "" ``` | **Key** | **Required** | **Description** | | ---------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `compression` | Yes | The compression method to use when unloading data. Valid values:- `uncompressed`: No compression algorithm is used.
- `gzip`: Use the gzip compression algorithm.
- `snappy`: Use the SNAPPY compression algorithm.
- `zstd`: Use the Zstd compression algorithm.
- `lz4`: Use the LZ4 compression algorithm.**NOTE**
Unloading into CSV files does not support data compression. You must set this item as `uncompressed`. | | `partition_by` | No | The list of columns that are used to partition data files into different storage paths. Multiple columns are separated by commas (,). `FILES()` extracts the key/value information of the specified columns and stores the data files under the storage paths featured with the extracted key/value pair. For further instructions, see Example 7. | | `single` | No | Whether to unload the data into a single file. Valid values:- `true`: The data is stored in a single data file.
- `false` (Default): The data is stored in multiple files if the amount of data unloaded exceeds 512 MB. | | `target_max_file_size` | No | The best-effort maximum size of each file in the batch to be unloaded. Unit: Bytes. Default value: 1073741824 (1 GB). When the size of data to be unloaded exceeds this value, the data will be divided into multiple files, and the size of each file will not significantly exceed this value. Introduced in v3.2.7. | | `csv.column_separator` | No | The column separator used in CSV-format output files. Default value: `\t`. Only applicable when `format` is `csv`. | | `csv.row_delimiter` | No | The row delimiter used in CSV-format output files. Default value: `\n`. Only applicable when `format` is `csv`. | | `csv.include_header` | No | Whether to include column names as the first row of CSV-format output files. Valid values: `true`, `false` (default). Only applicable when `format` is `csv`. | | `csv.enclose` | No | The character used to enclose each field value in CSV-format output files. When specified, all non-NULL fields are wrapped with this character, and occurrences of the enclose/escape characters within field values are escaped using the `csv.escape` character. NULL values are output as `\N` without enclosing. Type: single-byte ASCII character. Multi-character or multi-byte (non-ASCII) values are rejected with a semantic error. Default: `NONE` (disabled). Only applicable when `format` is `csv`. | | `csv.escape` | No | The character used to escape the enclose character and the escape character itself within field values. Type: single-byte ASCII character. Multi-character or multi-byte (non-ASCII) values are rejected with a semantic error. Default: `NONE`. Common combinations:- `"csv.enclose"="\""`, `"csv.escape"="\""`: RFC 4180 style (doubled quotes).
- `"csv.enclose"="\""`, `"csv.escape"="\\"`: backslash escape style.**NOTE**
When re-importing RFC 4180 doubled-quote output (where `escape` equals `enclose`) back into StarRocks, set only `csv.enclose` without `csv.escape` on the read side. StarRocks' CSV reader natively handles doubled quotes via its ENCLOSE state. | #### Examples[​](#examples-1 "Direct link to Examples") ###### Example 1: Query the data from a file[​](#example-1-query-the-data-from-a-file "Direct link to Example 1: Query the data from a file") Query the data from the Parquet file **parquet/par-dup.parquet** within the AWS S3 bucket `inserttest`: ```sql SELECT * FROM FILES( "path" = "s3://inserttest/parquet/par-dup.parquet", "format" = "parquet", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.region" = "us-west-2" ); +------+---------------------------------------------------------+ | c1 | c2 | +------+---------------------------------------------------------+ | 1 | {"1": "key", "1": "1", "111": "1111", "111": "aaaa"} | | 2 | {"2": "key", "2": "NULL", "222": "2222", "222": "bbbb"} | +------+---------------------------------------------------------+ 2 rows in set (22.335 sec) ``` Query the data from the Parquet files in NFS(NAS): ```sql SELECT * FROM FILES( 'path' = 'file:///home/ubuntu/parquetfile/*.parquet', 'format' = 'parquet' ); ``` ###### Example 2: Insert the data rows from a file[​](#example-2-insert-the-data-rows-from-a-file "Direct link to Example 2: Insert the data rows from a file") Insert the data rows from the Parquet file **parquet/insert\_wiki\_edit\_append.parquet** within the AWS S3 bucket `inserttest` into the table `insert_wiki_edit`: ```sql INSERT INTO insert_wiki_edit SELECT * FROM FILES( "path" = "s3://inserttest/parquet/insert_wiki_edit_append.parquet", "format" = "parquet", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.region" = "us-west-2" ); Query OK, 2 rows affected (23.03 sec) {'label':'insert_d8d4b2ee-ac5c-11ed-a2cf-4e1110a8f63b', 'status':'VISIBLE', 'txnId':'2440'} ``` Insert the data rows from the CSV files in NFS(NAS) into the table `insert_wiki_edit`: ```sql INSERT INTO insert_wiki_edit SELECT * FROM FILES( 'path' = 'file:///home/ubuntu/csvfile/*.csv', 'format' = 'csv', 'csv.column_separator' = ',', 'csv.row_delimiter' = '\n' ); ``` ###### Example 3: CTAS with data rows from a file[​](#example-3-ctas-with-data-rows-from-a-file "Direct link to Example 3: CTAS with data rows from a file") Create a table named `ctas_wiki_edit` and insert the data rows from the Parquet file **parquet/insert\_wiki\_edit\_append.parquet** within the AWS S3 bucket `inserttest` into the table: ```plain CREATE TABLE ctas_wiki_edit AS SELECT * FROM FILES( "path" = "s3://inserttest/parquet/insert_wiki_edit_append.parquet", "format" = "parquet", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.region" = "us-west-2" ); Query OK, 2 rows affected (22.09 sec) {'label':'insert_1a217d70-2f52-11ee-9e4a-7a563fb695da', 'status':'VISIBLE', 'txnId':'3248'} ``` ###### Example 4: Query the data from a file and extract the key/value information in its path[​](#example-4-query-the-data-from-a-file-and-extract-the-keyvalue-information-in-its-path "Direct link to Example 4: Query the data from a file and extract the key/value information in its path") Query the data from the Parquet file **/geo/country=US/city=LA/file1.parquet** (which only contains two columns -`id` and `user`), and extract the key/value information in its path as columns returned. ```plain SELECT * FROM FILES( "path" = "hdfs://xxx.xx.xxx.xx:9000/geo/country=US/city=LA/file1.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "xxxxx", "password" = "xxxxx", "columns_from_path" = "country, city" ); +------+---------+---------+------+ | id | user | country | city | +------+---------+---------+------+ | 1 | richard | US | LA | | 2 | amber | US | LA | +------+---------+---------+------+ 2 rows in set (3.84 sec) ``` ###### Example 5: Automatic schema detection and Unionization[​](#example-5-automatic-schema-detection-and-unionization "Direct link to Example 5: Automatic schema detection and Unionization") The following example is based on two Parquet files in the S3 bucket: * File 1 contains three columns - INT column `c1`, FLOAT column `c2`, and DATE column `c3`. ```plain c1,c2,c3 1,0.71173,2017-11-20 2,0.16145,2017-11-21 3,0.80524,2017-11-22 4,0.91852,2017-11-23 5,0.37766,2017-11-24 6,0.34413,2017-11-25 7,0.40055,2017-11-26 8,0.42437,2017-11-27 9,0.67935,2017-11-27 10,0.22783,2017-11-29 ``` * File 2 contains three columns - INT column `c1`, INT column `c2`, and DATETIME column `c3`. ```plain c1,c2,c3 101,9,2018-05-15T18:30:00 102,3,2018-05-15T18:30:00 103,2,2018-05-15T18:30:00 104,3,2018-05-15T18:30:00 105,6,2018-05-15T18:30:00 106,1,2018-05-15T18:30:00 107,8,2018-05-15T18:30:00 108,5,2018-05-15T18:30:00 109,6,2018-05-15T18:30:00 110,8,2018-05-15T18:30:00 ``` Use a CTAS statement to create a table named `test_ctas_parquet` and insert the data rows from the two Parquet files into the table: ```sql CREATE TABLE test_ctas_parquet AS SELECT * FROM FILES( "path" = "s3://inserttest/parquet/*", "format" = "parquet", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.region" = "us-west-2" ); ``` View the table schema of `test_ctas_parquet`: ```sql SHOW CREATE TABLE test_ctas_parquet\G ``` ```plain *************************** 1. row *************************** Table: test_ctas_parquet Create Table: CREATE TABLE `test_ctas_parquet` ( `c1` bigint(20) NULL COMMENT "", `c2` decimal(38, 9) NULL COMMENT "", `c3` varchar(1048576) NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`c1`, `c2`) COMMENT "OLAP" DISTRIBUTED BY RANDOM PROPERTIES ( "bucket_size" = "4294967296", "compression" = "LZ4", "replication_num" = "3" ); ``` The result shows that the `c2` column, which contains both FLOAT and INT data, is merged as a DECIMAL column, and `c3`, which contains both DATE and DATETIME data, is merged as a VARCHAR column. The above result stays the same when the Parquet files are changed to CSV files that contain the same data: ```sql CREATE TABLE test_ctas_csv AS SELECT * FROM FILES( "path" = "s3://inserttest/csv/*", "format" = "csv", "csv.column_separator"=",", "csv.row_delimiter"="\n", "csv.enclose"='"', "csv.skip_header"="1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.region" = "us-west-2" ); Query OK, 0 rows affected (30.90 sec) SHOW CREATE TABLE test_ctas_csv\G *************************** 1. row *************************** Table: test_ctas_csv Create Table: CREATE TABLE `test_ctas_csv` ( `c1` bigint(20) NULL COMMENT "", `c2` decimal(38, 9) NULL COMMENT "", `c3` varchar(1048576) NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`c1`, `c2`) COMMENT "OLAP" DISTRIBUTED BY RANDOM PROPERTIES ( "bucket_size" = "4294967296", "compression" = "LZ4", "replication_num" = "3" ); 1 row in set (0.27 sec) ``` * Unionize the schema of Parquet files and allow the system to assign NULL values to non-existent columns by setting `fill_mismatch_column_with` to `null`: ```sql SELECT * FROM FILES( "path" = "s3://inserttest/basic_type.parquet,s3://inserttest/basic_type_k2k5k7.parquet", "format" = "parquet", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.region" = "us-west-2", "fill_mismatch_column_with" = "null" ); +------+------+------+-------+------------+---------------------+------+------+ | k1 | k2 | k3 | k4 | k5 | k6 | k7 | k8 | +------+------+------+-------+------------+---------------------+------+------+ | NULL | 21 | NULL | NULL | 2024-10-03 | NULL | c | NULL | | 0 | 1 | 2 | 3.20 | 2024-10-01 | 2024-10-01 12:12:12 | a | 4.3 | | 1 | 11 | 12 | 13.20 | 2024-10-02 | 2024-10-02 13:13:13 | b | 14.3 | +------+------+------+-------+------------+---------------------+------+------+ 3 rows in set (0.03 sec) ``` ###### Example 6: View the schema of a file[​](#example-6-view-the-schema-of-a-file "Direct link to Example 6: View the schema of a file") View the schema of the Parquet file `lineorder` stored in AWS S3 using DESC. ```plain DESC FILES( "path" = "s3://inserttest/lineorder.parquet", "format" = "parquet", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.region" = "us-west-2" ); +------------------+------------------+------+ | Field | Type | Null | +------------------+------------------+------+ | lo_orderkey | int | YES | | lo_linenumber | int | YES | | lo_custkey | int | YES | | lo_partkey | int | YES | | lo_suppkey | int | YES | | lo_orderdate | int | YES | | lo_orderpriority | varchar(1048576) | YES | | lo_shippriority | int | YES | | lo_quantity | int | YES | | lo_extendedprice | int | YES | | lo_ordtotalprice | int | YES | | lo_discount | int | YES | | lo_revenue | int | YES | | lo_supplycost | int | YES | | lo_tax | int | YES | | lo_commitdate | int | YES | | lo_shipmode | varchar(1048576) | YES | +------------------+------------------+------+ 17 rows in set (0.05 sec) ``` ###### Example 7: Unload data[​](#example-7-unload-data "Direct link to Example 7: Unload data") Unload all data rows in `sales_records` as multiple Parquet files under the path **/unload/partitioned/** in the HDFS cluster. These files are stored in different subpaths distinguished by the values in the column `sales_time`. ```sql INSERT INTO FILES( "path" = "hdfs://xxx.xx.xxx.xx:9000/unload/partitioned/", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "xxxxx", "password" = "xxxxx", "compression" = "lz4", "partition_by" = "sales_time" ) SELECT * FROM sales_records; ``` Unload the query results into CSV and Parquet files in NFS(NAS): ```sql -- CSV INSERT INTO FILES( 'path' = 'file:///home/ubuntu/csvfile/', 'format' = 'csv', 'csv.column_separator' = ',', 'csv.row_delimitor' = '\n' ) SELECT * FROM sales_records; -- Parquet INSERT INTO FILES( 'path' = 'file:///home/ubuntu/parquetfile/', 'format' = 'parquet' ) SELECT * FROM sales_records; ``` ###### Example 8: Avro files[​](#example-8-avro-files "Direct link to Example 8: Avro files") Load an Avro file: ```sql INSERT INTO avro_tbl SELECT * FROM FILES( "path" = "hdfs://xxx.xx.xx.x:yyyy/avro/primitive.avro", "format" = "avro" ); ``` Query the data from an Avro file: ```sql SELECT * FROM FILES("path" = "hdfs://xxx.xx.xx.x:yyyy/avro/complex.avro", "format" = "avro")\G *************************** 1. row *************************** record_field: {"id":1,"name":"avro"} enum_field: HEARTS array_field: ["one","two","three"] map_field: {"a":1,"b":2} union_field: 100 fixed_field: 0x61626162616261626162616261626162 1 row in set (0.05 sec) ``` View the schema of an Avro file: ```sql DESC FILES("path" = "hdfs://xxx.xx.xx.x:yyyy/avro/logical.avro", "format" = "avro"); +------------------------+------------------+------+ | Field | Type | Null | +------------------------+------------------+------+ | decimal_bytes | decimal(10,2) | YES | | decimal_fixed | decimal(10,2) | YES | | uuid_string | varchar(1048576) | YES | | date | date | YES | | time_millis | int | YES | | time_micros | bigint | YES | | timestamp_millis | datetime | YES | | timestamp_micros | datetime | YES | | local_timestamp_millis | bigint | YES | | local_timestamp_micros | bigint | YES | | duration | varbinary(12) | YES | +------------------------+------------------+------+ ``` ###### Example 9: Access Azure Blob Storage using Managed Identity and Service Principal[​](#example-9-access-azure-blob-storage-using-managed-identity-and-service-principal "Direct link to Example 9: Access Azure Blob Storage using Managed Identity and Service Principal") ```sql -- Managed Identity SELECT * FROM FILES( "path" = "wasbs://storage-container@storage-account.blob.core.windows.net/ssb_1g/customer/*", "format" = "parquet", "azure.blob.oauth2_use_managed_identity" = "true", "azure.blob.oauth2_client_id" = "1d6bfdec-dd34-4260-b8fd-aaaaaaaaaaaa" ); -- Service Principal SELECT * FROM FILES( "path" = "wasbs://storage-container@storage-account.blob.core.windows.net/ssb_1g/customer/*", "format" = "parquet", "azure.blob.oauth2_client_id" = "1d6bfdec-dd34-4260-b8fd-bbbbbbbbbbbb", "azure.blob.oauth2_client_secret" = "C2M8Q~ZXXXXXX_5XsbDCeL2dqP7hIR60xxxxxxxx", "azure.blob.oauth2_tenant_id" = "540e19cc-386b-4a44-a7b8-cccccccccccc" ); ``` ###### Example 10: CSV file[​](#example-10-csv-file "Direct link to Example 10: CSV file") Query the data from a CSV file: ```sql SELECT * FROM FILES( "path" = "s3://test-bucket/file1.csv", "format" = "csv", "csv.column_separator"=",", "csv.row_delimiter"="\r\n", "csv.enclose"='"', "csv.skip_header"="1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.region" = "us-west-2" ); +------+---------+--------------+ | $1 | $2 | $3 | +------+---------+--------------+ | 1 | 0.71173 | 2017-11-20 | | 2 | 0.16145 | 2017-11-21 | | 3 | 0.80524 | 2017-11-22 | | 4 | 0.91852 | 2017-11-23 | | 5 | 0.37766 | 2017-11-24 | | 6 | 0.34413 | 2017-11-25 | | 7 | 0.40055 | 2017-11-26 | | 8 | 0.42437 | 2017-11-27 | | 9 | 0.67935 | 2017-11-27 | | 10 | 0.22783 | 2017-11-29 | +------+---------+--------------+ 10 rows in set (0.33 sec) ``` Load a CSV file: ```sql INSERT INTO csv_tbl SELECT * FROM FILES( "path" = "s3://test-bucket/file1.csv", "format" = "csv", "csv.column_separator"=",", "csv.row_delimiter"="\r\n", "csv.enclose"='"', "csv.skip_header"="1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.region" = "us-west-2" ); ``` ###### Example 11: Using AWS STS Regional endpoints[​](#example-11-using-aws-sts-regional-endpoints "Direct link to Example 11: Using AWS STS Regional endpoints") There are two cases shown here: 1. Using a STS regional endpoint outside an AWS environment. 2. Using STS within an AWS environment (for example, EC2). ###### Outside an AWS environment[​](#outside-an-aws-environment "Direct link to Outside an AWS environment") important When working outside an AWS environment and using regional STS requires setting `"aws.s3.use_instance_profile" = "false"`. ```sql SELECT COUNT(*) FROM FILES("path" = "s3://aws-bucket/path/file.csv.gz", "format" = "csv", "compression" = "gzip", "aws.s3.endpoint"="https://s3.us-east-1.amazonaws.com", "aws.s3.region"="us-east-1", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "****", "aws.s3.secret_key" = "****", "aws.s3.iam_role_arn"="arn:aws:iam::1234567890:role/access-role", "aws.s3.sts.region" = "{sts_region}", "aws.s3.sts.endpoint" = "{sts_endpoint}" ); ``` ###### Inside an AWS environment[​](#inside-an-aws-environment "Direct link to Inside an AWS environment") ```sql SELECT COUNT(*) FROM FILES("path" = "s3://aws-bucket/path/file.csv.gz", "format" = "csv", "compression" = "gzip", "aws.s3.endpoint"="https://s3.us-east-1.amazonaws.com", "aws.s3.region"="us-east-1", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "true", "aws.s3.access_key" = "****", "aws.s3.secret_key" = "****", "aws.s3.iam_role_arn"="arn:aws:iam::1234567890:role/access-role", "aws.s3.sts.region" = "{sts_region}", "aws.s3.sts.endpoint" = "{sts_endpoint}" ); ``` --- ### generate_series Generates a series of values within the interval specified by `start` and `end`, and with an optional `step`. generate\_series() is a table function. A table function can return a row set for each input row. The row set can contain zero, one, or multiple rows. Each row can contain one or more columns. To use generate\_series() in StarRocks, you must enclose it in the TABLE keyword if the input parameters are constants. If the input parameters are expressions, such as column names, the TABLE keyword is not required (See Example 5). This function is supported from v3.1. #### Syntax[​](#syntax "Direct link to Syntax") ```sql generate_series(start, end [,step]) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `start`: the starting value of the series, required. Supported data types are INT, BIGINT, and LARGEINT. * `end`: the ending value of the series, required. Supported data types are INT, BIGINT, and LARGEINT. * `step`: the value to increment or decrement, optional. Supported data types are INT, BIGINT, and LARGEINT. If not specified, the default step is 1. `step` can be either negative or positive, but cannot be zero. The three parameters must have the same data type, for example, `generate_series(INT start, INT end [, INT step])`. It supports named arguments from v3.3, where all parameters are input with names, like `name=>expr`, for example, `generate_series(start=>3, end=>7, step=>2)`. Named arguments can disorder arguments and optionally set default arguments, but can't mix with positional arguments. #### Return value[​](#return-value "Direct link to Return value") Returns a series of values that have the same as the input parameters `start` and `end`. * When `step` is positive, zero rows are returned if `start` is greater than `end`. Conversely, when `step` is negative, zero rows are returned if `start` is less than `end`. * An error is returned if `step` is 0. * This function deals with nulls as follows: If any input parameter is a literal null, an error is reported. If any input parameter is an expression and the result of the expression is null, 0 rows are returned (See Example 5). #### Examples[​](#examples "Direct link to Examples") Example 1: Generate a sequence of values within the range \[2,5] in ascending order with the default step `1`. ```sql MySQL > select * from TABLE(generate_series(2, 5)); +-----------------+ | generate_series | +-----------------+ | 2 | | 3 | | 4 | | 5 | +-----------------+ ``` Example 2: Generate a sequence of values within the range \[2,5] in ascending order with the specified step `2`. ```sql MySQL > select * from TABLE(generate_series(2, 5, 2)); +-----------------+ | generate_series | +-----------------+ | 2 | | 4 | +-----------------+ ``` Example 3: Generate a sequence of values within the range \[5,2] in descending order with the specified step `-1`. ```sql MySQL > select * from TABLE(generate_series(5, 2, -1)); +-----------------+ | generate_series | +-----------------+ | 5 | | 4 | | 3 | | 2 | +-----------------+ ``` Example 4: Zero rows are returned when `step` is negative and `start` is less than `end`. ```sql MySQL > select * from TABLE(generate_series(2, 5, -1)); Empty set (0.01 sec) ``` Example 5: Use table columns as the input parameters of generate\_series(). In this use case, you do not need to use `TABLE()` with generate\_series(). ```sql CREATE TABLE t_numbers(start INT, end INT) DUPLICATE KEY (start) DISTRIBUTED BY HASH(start) BUCKETS 1; INSERT INTO t_numbers VALUES (1, 3), (5, 2), (NULL, 10), (4, 7), (9,6); SELECT * FROM t_numbers; +-------+------+ | start | end | +-------+------+ | NULL | 10 | | 1 | 3 | | 4 | 7 | | 5 | 2 | | 9 | 6 | +-------+------+ -- Generate multiple rows for rows (1,3) and (4,7) with step 1. SELECT * FROM t_numbers, generate_series(t_numbers.start, t_numbers.end); +-------+------+-----------------+ | start | end | generate_series | +-------+------+-----------------+ | 1 | 3 | 1 | | 1 | 3 | 2 | | 1 | 3 | 3 | | 4 | 7 | 4 | | 4 | 7 | 5 | | 4 | 7 | 6 | | 4 | 7 | 7 | +-------+------+-----------------+ -- Generate multiple rows for rows (5,2) and (9,6) with step -1. SELECT * FROM t_numbers, generate_series(t_numbers.start, t_numbers.end, -1); +-------+------+-----------------+ | start | end | generate_series | +-------+------+-----------------+ | 5 | 2 | 5 | | 5 | 2 | 4 | | 5 | 2 | 3 | | 5 | 2 | 2 | | 9 | 6 | 9 | | 9 | 6 | 8 | | 9 | 6 | 7 | | 9 | 6 | 6 | +-------+------+-----------------+ ``` Example 6: Generate a sequence of values within the range \[2,5] in ascending order with the specified step `2`, using named arguments. ```sql MySQL > select * from TABLE(generate_series(start=>2, end=>5, step=>2)); +-----------------+ | generate_series | +-----------------+ | 2 | | 4 | +-----------------+ ``` #### keywords[​](#keywords "Direct link to keywords") table function, generate series --- ### native_query `native_query` is a JDBC catalog table function. It executes a database-native `SELECT` statement through a JDBC catalog and exposes the result as a StarRocks relation. Use this function when the source database must run SQL that is hard to express against a single JDBC external table, such as a pre-filtered subquery, a join in the source database, or vendor-specific SQL syntax. After `native_query` returns a relation, you can use StarRocks to apply additional filters, joins, aggregations, projections, or load the result with `INSERT INTO`. This function is supported from v4.1 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SELECT ... FROM TABLE(.native_query('')) [AS] [WHERE ...]; ``` #### Parameters[​](#parameters "Direct link to Parameters") ##### `jdbc_catalog`[​](#jdbc_catalog "Direct link to jdbc_catalog") The name of a JDBC catalog. Only JDBC catalogs support this function. ##### `select_sql`[​](#select_sql "Direct link to select_sql") A string literal that contains the pass-through SQL statement executed by the source database. After StarRocks removes leading comments and trailing semicolons from `select_sql`, the statement must start with `SELECT`. Use the SQL dialect, object names, quoting rules, and functions of the source database. If `select_sql` contains a single quotation mark (`'`), escape it as two single quotation marks (`''`) in the StarRocks SQL string. ##### `alias`[​](#alias "Direct link to alias") An optional table alias for the result relation. Column aliases after the table alias, such as `AS q(c1, c2)`, are not supported. Define column aliases inside `select_sql`, for example `SELECT id AS id_alias FROM ...`. #### Return value[​](#return-value "Direct link to Return value") `native_query` returns a relation whose columns are inferred from the JDBC result set metadata of `select_sql`. StarRocks maps JDBC column types to StarRocks column types by using the schema resolver of the JDBC catalog. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The StarRocks user needs the `USAGE` privilege on the JDBC catalog. Permissions on source database objects are checked by the remote database with the user configured in the JDBC catalog. * `select_sql` must be a string literal and must be the only argument. Named arguments are not supported. * `select_sql` must start with `SELECT` after leading comments are removed. Statements that start with `WITH`, `INSERT`, `UPDATE`, `DELETE`, or other non-`SELECT` keywords are not supported. * The legacy `.system.query(...)` form is not supported. * If the source query returns no columns, StarRocks returns an error during analysis. #### Examples[​](#examples "Direct link to Examples") The following examples assume that the source MySQL database contains a table named `app.orders` and that a JDBC catalog named `jdbc0` has been created in StarRocks. Example 1: Run a source-side subquery and apply an outer StarRocks filter. ```sql SELECT id, name, doubled_score FROM TABLE(jdbc0.native_query( 'SELECT id, name, score * 2 AS doubled_score FROM app.orders WHERE score >= 20' )) q WHERE doubled_score < 70 ORDER BY id; ``` Example 2: Aggregate the result returned by the pass-through query. ```sql SELECT category, SUM(score) AS total_score FROM TABLE(jdbc0.native_query( 'SELECT category, score FROM app.orders WHERE status = ''PAID''' )) q GROUP BY category ORDER BY category; ``` Example 3: Load the result of a native query into a StarRocks table. ```sql INSERT INTO paid_order_summary SELECT category, SUM(score) AS total_score FROM TABLE(jdbc0.native_query( 'SELECT category, score FROM app.orders WHERE status = ''PAID''' )) q GROUP BY category; ``` #### Unsupported forms[​](#unsupported-forms "Direct link to Unsupported forms") ```sql -- Named arguments are not supported. SELECT * FROM TABLE(jdbc0.native_query(query => 'SELECT id FROM app.orders')); -- The legacy system.query alias is not supported. SELECT * FROM TABLE(jdbc0.system.query('SELECT id FROM app.orders')); -- WITH queries are not supported because the SQL must start with SELECT. SELECT * FROM TABLE(jdbc0.native_query('WITH q AS (SELECT id FROM app.orders) SELECT * FROM q')); -- Column aliases after the table alias are not supported. SELECT * FROM TABLE(jdbc0.native_query('SELECT id FROM app.orders')) q(id_alias); ``` --- ### Utility #### [📄️ bar](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/bar.md) [Draw a bar graph like histogram to inspect the data distribution.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/bar.md) #### [📄️ catalog](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/catalog.md) [Returns the name of the current catalog.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/catalog.md) #### [📄️ column\_size & column\_compressed\_size](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/column_size_functions.md) [These functions return the size information of table columns for storage analysis and optimization.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/column_size_functions.md) #### [📄️ current\_role](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/current_role.md) [Queries roles that are activated for the current user.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/current_role.md) #### [📄️ current\_version](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/current_version.md) [Returns the current version of StarRocks.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/current_version.md) #### [📄️ CURRENT\_WAREHOUSE](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/current_warehouse.md) [Returns the effective warehouse name for the current session, reflecting the warehouse set via SET WAREHOUSE or the default warehouse.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/current_warehouse.md) #### [📄️ database](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/database.md) [Returns the name of the current database.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/database.md) #### [📄️ encode\_sort\_key](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/encode_sort_key.md) [Creates an order-preserving composite binary key from multiple heterogeneous input columns.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/encode_sort_key.md) #### [📄️ equiwidth\_bucket](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/equiwidth_bucket.md) [Calculate the equi-width histogram bucket.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/equiwidth_bucket.md) #### [📄️ get\_query\_profile](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/get_query_profile.md) [Obtains the profile of a query by using its query\_id.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/get_query_profile.md) #### [📄️ host\_name](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/host_name.md) [Obtains the hostname of the node on which the computation is performed.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/host_name.md) #### [📄️ is\_role\_in\_session](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/is_role_in_session.md) [Verifies whether a role (or a nested role) is active in the current session.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/is_role_in_session.md) #### [📄️ isnotnull](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/isnotnull.md) [Checks whether the value is not NULL, returns 1 if it is not NULL, and returns 0 if it is NULL.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/isnotnull.md) #### [📄️ isnull](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/isnull.md) [Checks whether the value is NULL, returns 1 if it is NULL, and returns 0 if it is not NULL.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/isnull.md) #### [📄️ last\_query\_id](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/last_query_id.md) [Obtains the ID of the most recently executed query in the current session.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/last_query_id.md) #### [📄️ sleep](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/sleep.md) [Delays query execution for a specified number of seconds and returns whether the sleep completed without interruption.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/sleep.md) #### [📄️ uuid](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/uuid.md) [Returns a random UUID of the VARCHAR type.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/uuid.md) #### [📄️ uuid\_numeric](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/uuid_numeric.md) [Returns a random UUID of the LARGEINT type.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/uuid_numeric.md) #### [📄️ uuid\_v7\_numeric](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/uuid_v7_numeric.md) [Returns a time-ordered UUID v7 of the LARGEINT type.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/uuid_v7_numeric.md) #### [📄️ uuid\_v7](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/uuidv7.md) [Returns a time-ordered UUID v7 of the VARCHAR type.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/uuidv7.md) #### [📄️ version](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/version.md) [Returns the current version of the MySQL database.](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/version.md) --- ### bar Draw a bar graph like histogram to inspect the data distribution. #### Syntax[​](#syntax "Direct link to Syntax") ```sql bar(size, min, max, width) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `size`: size of the bar, must be within the `[min, max]` * `min`: min value of the bar * `max`: max value of the bar * `width`: width of the bar #### Example[​](#example "Direct link to Example") ```sql select r, bar(r, 0, 10, 20) as x from table(generate_series(0, 10)) as s(r); ``` ```plaintext 0 1 ▓▓ 2 ▓▓▓▓ 3 ▓▓▓▓▓▓ 4 ▓▓▓▓▓▓▓▓ 5 ▓▓▓▓▓▓▓▓▓▓ 6 ▓▓▓▓▓▓▓▓▓▓▓▓ 7 ▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 8 ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 9 ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 10 ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ ``` --- ### catalog Returns the name of the current catalog. The catalog can be a StarRocks internal catalog or an external catalog that is mapped to an external data source. For more information about catalogs, see [Catalog overview](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md). If no catalog is selected, the StarRocks internal catalog `default_catalog` is returned. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext catalog() ``` #### Parameters[​](#parameters "Direct link to Parameters") This function does not require parameters. #### Return value[​](#return-value "Direct link to Return value") Returns the name of the current catalog as a string. #### Examples[​](#examples "Direct link to Examples") Example 1: The current catalog is StarRocks internal catalog `default_catalog`. ```plaintext select catalog(); +-----------------+ | CATALOG() | +-----------------+ | default_catalog | +-----------------+ 1 row in set (0.01 sec) ``` Example 2: The current catalog is an external catalog `hudi_catalog`. ```sql -- Switch to an external catalog. set catalog hudi_catalog; -- Return the name of the current catalog. select catalog(); +--------------+ | CATALOG() | +--------------+ | hudi_catalog | +--------------+ ``` #### See also[​](#see-also "Direct link to See also") [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md): Switches to a destination catalog. --- ### column_size & column_compressed_size These functions return the size information of table columns for storage analysis and optimization. Both functions work with the `[_META_]` hint to inspect segment file metadata. #### column\_size[​](#column_size "Direct link to column_size") Returns the decompressed size of a column in bytes. ##### Syntax[​](#syntax "Direct link to Syntax") ```sql -- Do not omit the brackets [] in the hint. SELECT column_size(column_name) FROM table_name [_META_]; ``` ##### Parameters[​](#parameters "Direct link to Parameters") * `column_name`: The name of the column for which you want to get the decompressed size. ##### Return value[​](#return-value "Direct link to Return value") Returns the decompressed size of the column in bytes as a BIGINT value. #### column\_compressed\_size[​](#column_compressed_size "Direct link to column_compressed_size") Returns the compressed size of a column in bytes. ##### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql -- Do not omit the brackets [] in the hint. SELECT column_compressed_size(column_name) FROM table_name [_META_]; ``` ##### Parameters[​](#parameters-1 "Direct link to Parameters") * `column_name`: The name of the column for which you want to get the compressed size. ##### Return value[​](#return-value-1 "Direct link to Return value") Returns the compressed size of the column in bytes as a BIGINT value. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * Both functions must be used with the `[_META_]` hint to access metadata information. * The functions scan the metadata of underlying segment files using the META\_SCAN operator. * For complex data types (JSON, ARRAY, MAP, STRUCT), the functions recursively calculate the size of all sub-columns. * `column_size` returns the uncompressed column size. * `column_compressed_size` returns the compressed on-disk size. #### Examples[​](#examples "Direct link to Examples") ```sql -- Get both decompressed and compressed sizes of columns SELECT column_size(name) as name_decompressed_size, column_compressed_size(name) as name_compressed_size, column_size(description) as desc_decompressed_size, column_compressed_size(description) as desc_compressed_size FROM products [_META_]; ``` --- ### current_role Queries roles that are activated for the current user. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext current_role(); current_role; ``` #### Parameters[​](#parameters "Direct link to Parameters") None. #### Return value[​](#return-value "Direct link to Return value") Returns a VARCHAR value. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select current_role(); +----------------+ | current_role() | +----------------+ | db_admin | +----------------+ ``` --- ### current_version Returns the current version of StarRocks. Two syntaxes are provided for compatibility with different clients. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext current_version(); @@version_comment; ``` #### Parameters[​](#parameters "Direct link to Parameters") None #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select current_version(); +-------------------+ | current_version() | +-------------------+ | 2.1.2 0782ad7 | +-------------------+ 1 row in set (0.00 sec) mysql> select @@version_comment; +-------------------------+ | @@version_comment | +-------------------------+ | StarRocks version 2.1.2 | +-------------------------+ 1 row in set (0.01 sec) ``` --- ### CURRENT_WAREHOUSE Returns the effective warehouse name for the current session. The result reflects the warehouse set via `SET WAREHOUSE` or the default warehouse if none was explicitly selected. #### Syntax[​](#syntax "Direct link to Syntax") ```sql current_warehouse() ``` #### Return value[​](#return-value "Direct link to Return value") `VARCHAR`. The warehouse name currently in use for the session. #### Examples[​](#examples "Direct link to Examples") ```sql mysql> SET WAREHOUSE = 'compute_pool_x'; mysql> SELECT current_warehouse(); +---------------------+ | CURRENT_WAREHOUSE() | +---------------------+ | compute_pool_x | +---------------------+ 1 row in set (0.00 sec) ``` --- ### database Returns the name of the current database. If no database is selected, an empty value is returned. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext database() ``` #### Parameters[​](#parameters "Direct link to Parameters") This function does not require parameters. #### Return value[​](#return-value "Direct link to Return value") Returns the name of the current database as a string. #### Examples[​](#examples "Direct link to Examples") ```sql -- Select a destination database. use db_test -- Query the name of the current database. select database(); +------------+ | DATABASE() | +------------+ | db_test | +------------+ ``` #### See also[​](#see-also "Direct link to See also") [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md): Switches to a destination database. --- ### encode_sort_key Creates an order-preserving composite binary key from multiple heterogeneous input columns. This function is essential for creating efficient sort keys that maintain the original sort order when compared lexicographically. #### Syntax[​](#syntax "Direct link to Syntax") ```sql encode_sort_key(column1, column2, ..., columnN) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `column1, column2, ..., columnN`: One or more columns of any supported data type. The function accepts a variable number of arguments. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARBINARY type representing the composite sort key. #### Supported data types[​](#supported-data-types "Direct link to Supported data types") The following data types are supported: | Data Type | Description | | ----------- | ---------------------- | | `TINYINT` | 8-bit signed integer | | `SMALLINT` | 16-bit signed integer | | `INT` | 32-bit signed integer | | `BIGINT` | 64-bit signed integer | | `LARGEINT` | 128-bit signed integer | | `FLOAT` | 32-bit floating point | | `DOUBLE` | 64-bit floating point | | `VARCHAR` | Variable-length string | | `CHAR` | Fixed-length string | | `DATE` | Date value | | `DATETIME` | Date and time value | | `TIMESTAMP` | Timestamp value | The following complex types are not supported and will return an error: * `JSON` * `ARRAY` * `MAP` * `STRUCT` * `HLL` * `BITMAP` * `PERCENTILE` #### Encoding strategy[​](#encoding-strategy "Direct link to Encoding strategy") The function uses different encoding strategies for different data types to ensure that lexicographic comparison of the resulting binary keys preserves the original sort order: * **Integer types**: Big-endian byte order with sign bit flipped for signed types * **Floating-point types**: Custom encoding to ensure correct sort order * **String types**: Special encoding with 0x00 byte escaping and 0x00 0x00 terminator for non-last fields * **Date/Time types**: Internal integer representation encoded as integral value ##### NULL handling[​](#null-handling "Direct link to NULL handling") * Each column gets a NULL marker (0x00 for NULL, 0x01 for NOT NULL) for every row * NULL markers are added even for non-nullable columns to ensure consistent encoding * A separator byte (0x00) is used between columns (except after the last column) #### Examples[​](#examples "Direct link to Examples") ##### Generated sort key column[​](#generated-sort-key-column "Direct link to Generated sort key column") ```sql CREATE TABLE user_analytics ( user_id INT, region VARCHAR(50), score DOUBLE, created_date DATE, sort_key VARBINARY(1024) AS ( encode_sort_key(region, score, created_date) ) ) ORDER BY (sort_key); ``` ##### JSON field extraction[​](#json-field-extraction "Direct link to JSON field extraction") ```sql CREATE TABLE json_data ( id INT, json_content JSON, sort_key VARBINARY(1024) AS ( encode_sort_key( get_json_int(json_content, '$.priority'), get_json_string(json_content, '$.category'), get_json_double(json_content, '$.score') ) ) ) ORDER BY (sort_key); ``` #### Limitations[​](#limitations "Direct link to Limitations") ##### Data type restrictions[​](#data-type-restrictions "Direct link to Data type restrictions") Complex types like JSON, ARRAY, MAP, and STRUCT cannot be directly encoded. Use JSON extraction functions to get primitive values: ```sql -- Instead of: encode_sort_key(json_col) -- Use: encode_sort_key(get_json_int(json_col, '$.field1'), get_json_string(json_col, '$.field2')) ``` ##### Performance considerations[​](#performance-considerations "Direct link to Performance considerations") * Each call to `encode_sort_key` requires encoding all input columns * Binary keys can be significantly larger than original data * Use generated columns to avoid repeated encoding ##### Key size limitations[​](#key-size-limitations "Direct link to Key size limitations") * Keep the number of columns reasonable (typically < 10) * Prefer shorter string columns when possible * Consider using hash functions for very long strings --- ### equiwidth_bucket Calculate the equi-width histogram bucket. #### Syntax[​](#syntax "Direct link to Syntax") ```sql equiwidth_bucket(value, min, max, buckets) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `value`: row value, must be within the `[min, max]` * `min`: min value of the histogram * `max`: max value of the histogram * `buckets`: number of histogram buckets #### Example[​](#example "Direct link to Example") ```sql select r, equiwidth_bucket(r, 0, 10, 20) as x from table(generate_series(0, 10)) as s(r); ``` ```plaintext 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 ``` --- ### get_query_profile Obtains the profile of a query by using its `query_id`. This function returns empty if the `query_id` does not exist or is incorrect. To use this function, you must enable the profiling feature, that is, set the session variable `enable_profile` to `true` (`set enable_profile = true;`). If this feature is not enabled, an empty profile is returned. This function is supported from v3.0. If the FE configuration item `authorization_enable_query_profile_access_check` is set to `true`, this function returns only the profiles of queries run by the current user, unless the user has the SYSTEM-level OPERATE privilege. In that case, passing a non-constant `query_id`, or a `query_id` whose profile is not cached on the FE the session is connected to, also requires the OPERATE privilege. When `authorization_enable_admin_user_protection` is also enabled, only `root` can pass a `query_id` that is not cached on the connected FE. To read your own profile without the OPERATE privilege, obtain the id first with `select last_query_id();` and pass the returned literal, as shown in the examples below. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext get_query_profile(x) ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: the query\_id string. The supported data type is VARCHAR. #### Return value[​](#return-value "Direct link to Return value") The query profile contains the following fields. For more information about the query profile, see [Query Profile](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md). ```sql Query: Summary: Planner: Execution Profile 7de16a85-761c-11ed-917d-00163e14d435: Fragment 0: Pipeline (id=2): EXCHANGE_SINK (plan_node_id=18): LOCAL_MERGE_SOURCE (plan_node_id=17): Pipeline (id=1): LOCAL_SORT_SINK (plan_node_id=17): AGGREGATE_BLOCKING_SOURCE (plan_node_id=16): Pipeline (id=0): AGGREGATE_BLOCKING_SINK (plan_node_id=16): EXCHANGE_SOURCE (plan_node_id=15): Fragment 1: ... Fragment 2: ... ``` #### Examples[​](#examples "Direct link to Examples") ```sql -- Enable the profiling feature. set enable_profile = true; -- Run a simple query. select 1; -- Get the query_id of the query. select last_query_id(); +--------------------------------------+ | last_query_id() | +--------------------------------------+ | bd3335ce-8dde-11ee-92e4-3269eb8da7d1 | +--------------------------------------+ -- Obtain the query profile. select get_query_profile('502f3c04-8f5c-11ee-a41f-b22a2c00f66b'); -- Use the regexp_extract function to obtain the QueryPeakMemoryUsage in the profile that matches the specified pattern. select regexp_extract(get_query_profile('bd3335ce-8dde-11ee-92e4-3269eb8da7d1'), 'QueryPeakMemoryUsage: [0-9\.]* [KMGB]*', 0); +-----------------------------------------------------------------------------------------------------------------------+ | regexp_extract(get_query_profile('bd3335ce-8dde-11ee-92e4-3269eb8da7d1'), 'QueryPeakMemoryUsage: [0-9.]* [KMGB]*', 0) | +-----------------------------------------------------------------------------------------------------------------------+ | QueryPeakMemoryUsage: 3.828 KB | +-----------------------------------------------------------------------------------------------------------------------+ ``` #### Related functions[​](#related-functions "Direct link to Related functions") * [last\_query\_id](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/last_query_id.md) * [regexp\_extract](https://docs.starrocks.io/docs/sql-reference/sql-functions/like-predicate-functions/regexp_extract.md) --- ### host_name Obtains the hostname of the node on which the computation is performed. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext host_name(); ``` #### Parameters[​](#parameters "Direct link to Parameters") None #### Return value[​](#return-value "Direct link to Return value") Returns a VARCHAR value. #### Examples[​](#examples "Direct link to Examples") ```plaintext select host_name(); +-------------+ | host_name() | +-------------+ | sandbox-sql | +-------------+ 1 row in set (0.01 sec) ``` --- ### is_role_in_session Verifies whether a role (or a nested role) is active in the current session. This function is supported from v3.1.4 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BOOLEAN is_role_in_session(VARCHAR role_name); ``` #### Parameters[​](#parameters "Direct link to Parameters") `role_name`: the role you want to verify (can also be a nested role). Supported data type is VARCHAR. #### Return value[​](#return-value "Direct link to Return value") Returns a BOOLEAN value. `1` indicates the role is active in the current session. `0` indicates the opposite. #### Examples[​](#examples "Direct link to Examples") 1. Create roles and a user. ```sql -- Create three roles. create role r1; create role r2; create role r3; -- Create user u1. create user u1; -- Pass roles r2 and r3 to r1, and grant r1 to user u1. This way, user u1 has three roles: r1, r2, and r3. grant r3 to role r2; grant r2 to role r1; grant r1 to user u1; -- Switch to user u1 and perform operations as u1. execute as u1 with no revert; ``` 2. Verify whether `r1` is active. The result shows this role is not active. ```plaintext select is_role_in_session("r1"); +--------------------------+ | is_role_in_session('r1') | +--------------------------+ | 0 | +--------------------------+ ``` 3. Run the [SET ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_ROLE.md) command to activate `r1` and use `is_role_in_session` to verify whether the role is active. The result shows `r1` is active and roles `r2` and `r3` that are nested in `r1` are also active. ```sql set role "r1"; select is_role_in_session("r1"); +--------------------------+ | is_role_in_session('r1') | +--------------------------+ | 1 | +--------------------------+ select is_role_in_session("r2"); +--------------------------+ | is_role_in_session('r2') | +--------------------------+ | 1 | +--------------------------+ select is_role_in_session("r3"); +--------------------------+ | is_role_in_session('r3') | +--------------------------+ | 1 | +--------------------------+ ``` --- ### isnotnull Checks whether the value is not `NULL`, returns `1` if it is not `NULL`, and returns `0` if it is `NULL`. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext ISNOTNULL(v) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `v`: the value to check. All date types is supported. #### Return value[​](#return-value "Direct link to Return value") Returns 1 if it is not `NULL`, and returns 0 if it is `NULL`. #### Examples[​](#examples "Direct link to Examples") ```plain MYSQL > SELECT c1, isnotnull(c1) FROM t1; +------+--------------+ | c1 | `c1` IS NULL | +------+--------------+ | NULL | 0 | | 1 | 1 | +------+--------------+ ``` --- ### isnull Checks whether the value is `NULL`, returns `1` if it is `NULL`, and returns `0` if it is not `NULL`. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext ISNULL(v) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `v`: the value to check. All date types is supported. #### Return value[​](#return-value "Direct link to Return value") Returns 1 if it is `NULL`, and returns 0 if it is not `NULL`. #### Example[​](#example "Direct link to Example") ```plain MYSQL > SELECT c1, isnull(c1) FROM t1; +------+--------------+ | c1 | `c1` IS NULL | +------+--------------+ | NULL | 1 | | 1 | 0 | +------+--------------+ ``` --- ### last_query_id Obtains the ID of the most recently executed query in the current session. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR last_query_id(); ``` #### Parameters[​](#parameters "Direct link to Parameters") None #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select last_query_id(); +--------------------------------------+ | last_query_id() | +--------------------------------------+ | 7c1d8d68-bbec-11ec-af65-00163e1e238f | +--------------------------------------+ 1 row in set (0.00 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") LAST\_QUERY\_ID --- ### sleep Delays the execution of an operation for a specified period of time (in seconds) and returns a BOOLEAN value to indicate whether the sleep is completed without interruption. `1` is returned if the sleep is completed without interruption. Otherwise, `0` is returned. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BOOLEAN sleep(INT x); ``` #### Parameters[​](#parameters "Direct link to Parameters") `x`: the duration for which you want to delay the execution of an operation. It must be of the INT type. Unit: seconds. If the input is is NULL, NULL is returned immediately without sleeping. #### Return value[​](#return-value "Direct link to Return value") Returns a value of the BOOLEAN type. #### Examples[​](#examples "Direct link to Examples") ```plain select sleep(3); +----------+ | sleep(3) | +----------+ | 1 | +----------+ 1 row in set (3.00 sec) select sleep(NULL); +-------------+ | sleep(NULL) | +-------------+ | NULL | +-------------+ 1 row in set (0.00 sec) ``` #### Keywords[​](#keywords "Direct link to Keywords") SLEEP, sleep --- ### uuid Returns a random UUID of the VARCHAR type. Two calls to this function can generate two different numbers. The UUID is 36 characters in length. It contains 5 hexadecimal numbers which are connected with four hyphens in the aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee format. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext uuid(); ``` #### Parameters[​](#parameters "Direct link to Parameters") None #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select uuid(); +--------------------------------------+ | uuid() | +--------------------------------------+ | 74a2ed19-9d21-4a99-a67b-aa5545f26454 | +--------------------------------------+ 1 row in set (0.01 sec) ``` --- ### uuid_numeric Returns a random UUID of the LARGEINT type. This function has an execution performance 2 orders of magnitude better than the `uuid` function. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext uuid_numeric(); ``` #### Parameters[​](#parameters "Direct link to Parameters") None #### Return value[​](#return-value "Direct link to Return value") Returns a value of the LARGEINT type. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > select uuid_numeric(); +--------------------------+ | uuid_numeric() | +--------------------------+ | 558712445286367898661205 | +--------------------------+ 1 row in set (0.00 sec) ``` --- ### uuid_v7_numeric Returns a time-ordered UUID v7 of the LARGEINT type. UUID v7 is defined in RFC 9562 and provides better database performance compared to random UUIDs (v4) because it maintains temporal ordering, which improves index locality and reduces fragmentation. This function returns the UUID v7 as a 128-bit integer instead of a string format, which can be more efficient for storage and comparison operations. This function is non-deterministic. Two calls to this function generate two different UUIDs. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext uuid_v7_numeric() ``` #### Return value[​](#return-value "Direct link to Return value") Returns a value of the LARGEINT type (128-bit integer). #### Examples[​](#examples "Direct link to Examples") ```plain mysql> SELECT uuid_v7_numeric(); +---------------------------------------+ | uuid_v7_numeric() | +---------------------------------------+ | 2088748395792837468371928374619283746 | +---------------------------------------+ 1 row in set (0.01 sec) ``` #### Benefits[​](#benefits "Direct link to Benefits") 1. **Time-ordered**: UUID values generated later will be larger than earlier ones. 2. **Improved Index Performance**: Better locality in B-tree indexes. 3. **Reduced Fragmentation**: Sequential inserts cause fewer page splits. 4. **Unique**: Random bits ensure uniqueness even within the same millisecond. 5. **Efficient Storage**: Numeric representation can be more compact than string format. #### Comparison with uuid\_numeric[​](#comparison-with-uuid_numeric "Direct link to Comparison with uuid_numeric") | Feature | uuid\_numeric() (v4) | uuid\_v7\_numeric() (v7) | | ----------------- | -------------------- | ------------------------ | | Ordering | Random | Time-ordered | | Index Performance | Poor | Good | | Fragmentation | High | Low | | Timestamp Info | No | Yes (48-bit ms) | --- ### uuid_v7 Returns a time-ordered UUID v7 of the VARCHAR type. UUID v7 is defined in RFC 9562 and provides better database performance compared to random UUIDs (v4) because it maintains temporal ordering, which improves index locality and reduces fragmentation. The UUID is 36 characters in length and contains 5 hexadecimal numbers connected with four hyphens in the `xxxxxxxx-xxxx-7xxx-xxxx-xxxxxxxxxxxx` format, where the version field is always 7. This function is non-deterministic. Two calls to this function generate two different UUIDs. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext uuid_v7() ``` #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> SELECT uuid_v7(); +--------------------------------------+ | uuid_v7() | +--------------------------------------+ | 019ba290-15ce-7f13-86b4-ebc97d2f72c0 | +--------------------------------------+ 1 row in set (0.01 sec) ``` #### UUID v7 Format[​](#uuid-v7-format "Direct link to UUID v7 Format") UUID v7 follows RFC 9562 specification with the following structure: * **48 bits**: Unix timestamp in milliseconds (time-ordered component). * **4 bits**: Version field, constant value `7`. * **12 bits**: Random data * **2 bits**: Variant field (RFC 4122) * **62 bits**: Random data This structure ensures that UUIDs generated later will sort after earlier ones, improving index performance. #### Benefits[​](#benefits "Direct link to Benefits") 1. **Time-ordered**: UUIDs generated later will sort after earlier ones. 2. **Improved Index Performance**: Better locality in B-tree indexes. 3. **Reduced Fragmentation**: Sequential inserts cause less page splits. 4. **Unique**: Random bits ensure uniqueness even within the same millisecond. #### Comparison with UUID v4[​](#comparison-with-uuid-v4 "Direct link to Comparison with UUID v4") | Feature | uuid() (v4) | uuid\_v7() (v7) | | ----------------- | ----------- | --------------- | | Ordering | Random | Time-ordered | | Index Performance | Poor | Good | | Fragmentation | High | Low | | Timestamp Info | No | Yes (48-bit ms) | --- ### version Returns the current version of the MySQL database. You can use [current\_version](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/current_version.md) to query StarRocks version. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR version(); ``` #### Parameters[​](#parameters "Direct link to Parameters") None #### Return value[​](#return-value "Direct link to Return value") Returns a value of the VARCHAR type. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select version(); +-----------+ | version() | +-----------+ | 5.1.0 | +-----------+ 1 row in set (0.00 sec) ``` #### References[​](#references "Direct link to References") [current\_version](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/current_version.md) --- ### Variant functions #### [📄️ get\_variant](https://docs.starrocks.io/docs/sql-reference/sql-functions/variant-functions/get_variant.md) [A family of functions that extract typed values from VARIANT objects at specified paths.](https://docs.starrocks.io/docs/sql-reference/sql-functions/variant-functions/get_variant.md) #### [📄️ variant\_query](https://docs.starrocks.io/docs/sql-reference/sql-functions/variant-functions/variant_query.md) [Queries the value of an element that can be located by the path expression in a VARIANT object and returns a VARIANT value.](https://docs.starrocks.io/docs/sql-reference/sql-functions/variant-functions/variant_query.md) #### [📄️ variant\_typeof](https://docs.starrocks.io/docs/sql-reference/sql-functions/variant-functions/variant_typeof.md) [Returns the type name of a VARIANT value as a string.](https://docs.starrocks.io/docs/sql-reference/sql-functions/variant-functions/variant_typeof.md) --- ### get_variant A family of functions that extract typed values from VARIANT objects at specified paths. These functions navigate to a specified path in a VARIANT value and return the value as a specific data type. If the value at the path does not exist, the path is invalid, or the value cannot be converted to the requested type, the functions return NULL. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext BIGINT get_variant_int(variant_expr, path) DOUBLE get_variant_double(variant_expr, path) VARCHAR get_variant_string(variant_expr, path) BOOLEAN get_variant_bool(variant_expr, path) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `variant_expr`: the expression that represents the VARIANT object. This is typically a VARIANT column from an Iceberg table. * `path`: the expression that represents the path to an element in the VARIANT object. The value of this parameter is a string. The path syntax is similar to JSON path: * `$` represents the root element * `.` is used to access object fields * `[index]` is used to access array elements (0-based indexing) * Field names with special characters can be quoted: `$."field.name"` #### Return values[​](#return-values "Direct link to Return values") * `get_variant_int`: Returns a BIGINT value, or NULL if the value cannot be converted to an integer. * `get_variant_double`: Returns a DOUBLE value, or NULL if the value cannot be converted to a double. * `get_variant_string`: Returns a VARCHAR value, or NULL if the value cannot be converted to a string. * `get_variant_bool`: Returns a BOOLEAN value (1 or 0), or NULL if the value cannot be converted to a boolean. #### Examples[​](#examples "Direct link to Examples") ##### Example 1: Extract basic fields from root level[​](#example-1-extract-basic-fields-from-root-level "Direct link to Example 1: Extract basic fields from root level") Extract string and integer values from the Bluesky firehose data: ```sql SELECT get_variant_string(data, '$.kind') AS kind, get_variant_string(data, '$.did') AS did, get_variant_int(data, '$.time_us') AS timestamp_us FROM bluesky LIMIT 3; ``` ```plaintext +--------+--------------------------------------+------------------+ | kind | did | timestamp_us | +--------+--------------------------------------+------------------+ | commit | did:plc:gw3yid42zz6qx6gxukvkgxgq | 1733267476040329 | | commit | did:plc:jbaujn4dd466ebt6pdf5vxod | 1733267476040803 | | commit | did:plc:ytgql26s6zoifhlnvf7qheea | 1733267476041472 | +--------+--------------------------------------+------------------+ ``` ##### Example 2: Extract nested fields[​](#example-2-extract-nested-fields "Direct link to Example 2: Extract nested fields") Extract operation details from nested commit structures: ```sql SELECT get_variant_string(data, '$.commit.collection') AS collection, get_variant_string(data, '$.commit.operation') AS operation, get_variant_string(data, '$.commit.rkey') AS record_key FROM bluesky WHERE get_variant_string(data, '$.commit.operation') = 'create' LIMIT 5; ``` ```plaintext +------------------------+-----------+---------------+ | collection | operation | record_key | +------------------------+-----------+---------------+ | app.bsky.feed.post | create | 3lckw3k2abc2d | | app.bsky.graph.follow | create | 3lckw3k2def3e | | app.bsky.feed.repost | create | 3lckw3k2ghi4f | | app.bsky.feed.repost | create | 3lckw3k2jkl5g | | app.bsky.feed.like | create | 3lckw3k2mno6h | +------------------------+-----------+---------------+ ``` ##### Example 3: Extract deeply nested content[​](#example-3-extract-deeply-nested-content "Direct link to Example 3: Extract deeply nested content") Extract post text and metadata from deeply nested structures: ```sql SELECT get_variant_string(data, '$.commit.record.text') AS post_text, get_variant_string(data, '$.commit.record.langs[0]') AS language, get_variant_int(data, '$.commit.record.createdAt') AS created_timestamp FROM bluesky WHERE get_variant_string(data, '$.commit.collection') = 'app.bsky.feed.post' AND get_variant_string(data, '$.commit.record.text') IS NOT NULL LIMIT 3; ``` ```plaintext +---------------------------------------------------------------------------------+----------+-------------------+ | post_text | language | created_timestamp | +---------------------------------------------------------------------------------+----------+-------------------+ | I went on a walk today | en | 1733267476 | | Are you interested in developing your own graph sequence model based on a sp... | en | 1733267476 | | Tam im tiež celkom slušne práši... | sk | 1733267476 | +---------------------------------------------------------------------------------+----------+-------------------+ ``` ##### Example 4: Extract boolean values[​](#example-4-extract-boolean-values "Direct link to Example 4: Extract boolean values") Extract boolean flags from post records: ```sql SELECT get_variant_string(data, '$.commit.record.text') AS post_text, get_variant_bool(data, '$.commit.record.bridgyOriginalUrl') AS is_bridged, get_variant_bool(data, '$.commit.record.selfLabel.settings.adult') AS has_adult_label FROM bluesky WHERE get_variant_string(data, '$.commit.collection') = 'app.bsky.feed.post' LIMIT 5; ``` ```plaintext +------------------------------------+-------------+------------------+ | post_text | is_bridged | has_adult_label | +------------------------------------+-------------+------------------+ | Great day for coding! | NULL | 0 | | Check out this amazing view | 1 | 0 | | Working on my new project | NULL | NULL | | Another beautiful sunset | NULL | 0 | | Weekend plans anyone? | 0 | NULL | +------------------------------------+-------------+------------------+ ``` ##### Example 5: Extract numeric values with calculations[​](#example-5-extract-numeric-values-with-calculations "Direct link to Example 5: Extract numeric values with calculations") Convert microseconds to seconds and perform timestamp calculations: ```sql SELECT get_variant_int(data, '$.time_us') AS timestamp_us, get_variant_int(data, '$.time_us') / 1000000 AS timestamp_seconds, FROM_UNIXTIME(get_variant_int(data, '$.time_us') / 1000000) AS readable_time FROM bluesky LIMIT 3; ``` ```plaintext +------------------+-------------------+---------------------+ | timestamp_us | timestamp_seconds | readable_time | +------------------+-------------------+---------------------+ | 1733267476040329 | 1733267476 | 2024-12-03 20:17:56 | | 1733267476040803 | 1733267476 | 2024-12-03 20:17:56 | | 1733267476041472 | 1733267476 | 2024-12-03 20:17:56 | +------------------+-------------------+---------------------+ ``` ##### Example 6: Use in aggregation queries[​](#example-6-use-in-aggregation-queries "Direct link to Example 6: Use in aggregation queries") Count records by collection type: ```sql SELECT get_variant_string(data, '$.commit.collection') AS collection, COUNT(*) AS count FROM bluesky GROUP BY collection ORDER BY count DESC LIMIT 5; ``` ```plaintext +------------------------+---------+ | collection | count | +------------------------+---------+ | app.bsky.feed.like | 5067917 | | app.bsky.graph.follow | 2931757 | | app.bsky.feed.post | 960723 | | app.bsky.feed.repost | 666364 | | app.bsky.graph.block | 196053 | +------------------------+---------+ ``` ##### Example 7: Count records by operation type[​](#example-7-count-records-by-operation-type "Direct link to Example 7: Count records by operation type") ```sql SELECT get_variant_string(data, '$.commit.operation') AS operation, COUNT(*) AS count FROM bluesky GROUP BY operation; ``` ```plaintext +-----------+---------+ | operation | count | +-----------+---------+ | delete | 420223 | | update | 40283 | | NULL | 39361 | | create | 9500118 | +-----------+---------+ ``` ##### Example 8: Extract and analyze engagement metrics[​](#example-8-extract-and-analyze-engagement-metrics "Direct link to Example 8: Extract and analyze engagement metrics") Extract numeric engagement metrics from posts: ```sql SELECT get_variant_string(data, '$.commit.record.text') AS post_text, get_variant_int(data, '$.commit.record.likeCount') AS likes, get_variant_int(data, '$.commit.record.replyCount') AS replies, get_variant_int(data, '$.commit.record.repostCount') AS reposts, get_variant_double(data, '$.commit.record.engagementRate') AS engagement_rate FROM bluesky WHERE get_variant_string(data, '$.commit.collection') = 'app.bsky.feed.post' AND get_variant_int(data, '$.commit.record.likeCount') > 100 LIMIT 5; ``` ```plaintext +----------------------------------------+-------+---------+---------+-----------------+ | post_text | likes | replies | reposts | engagement_rate | +----------------------------------------+-------+---------+---------+-----------------+ | This is amazing! | 245 | 23 | 45 | 0.12 | | Can't believe this happened | 189 | 12 | 34 | 0.08 | | Just finished my project | 156 | 8 | 19 | 0.05 | | Beautiful day at the park | 134 | 15 | 28 | 0.07 | | Check out this cool thing I made | 112 | 6 | 11 | 0.04 | +----------------------------------------+-------+---------+---------+-----------------+ ``` ##### Example 9: Combine multiple variant functions in filtering[​](#example-9-combine-multiple-variant-functions-in-filtering "Direct link to Example 9: Combine multiple variant functions in filtering") Find verified accounts that are actively posting: ```sql SELECT get_variant_string(data, '$.did') AS user_did, get_variant_string(data, '$.commit.record.text') AS post_text, get_variant_bool(data, '$.commit.record.verified') AS is_verified FROM bluesky WHERE get_variant_string(data, '$.commit.collection') = 'app.bsky.feed.post' AND get_variant_bool(data, '$.commit.record.verified') = TRUE AND get_variant_int(data, '$.time_us') > 1733267476000000 LIMIT 5; ``` ```plaintext +--------------------------------------+----------------------------------+-------------+ | user_did | post_text | is_verified | +--------------------------------------+----------------------------------+-------------+ | did:plc:rhqpfbdmabrwrd3o552tlsy7 | Official announcement coming up | 1 | | did:plc:vlq5rxxanqrzrqdcjfoirgfz | New feature release today | 1 | | did:plc:iefkhz4tg3vldzmtgyqkj3xb | Thank you all for your support | 1 | | did:plc:2phsrz5r5kynlq6pnrrzvrno | Excited to share this news | 1 | | did:plc:hxz45fftjaa62vnwiiwh6fhj | Join us for the live stream | 1 | +--------------------------------------+----------------------------------+-------------+ ``` ##### Example 10: NULL handling[​](#example-10-null-handling "Direct link to Example 10: NULL handling") Returns NULL when paths don't exist or values can't be converted: ```sql SELECT get_variant_int(data, '$.nonexistent.field') AS missing_int, get_variant_double(data, '$.kind') AS string_as_double, get_variant_bool(data, '$.time_us') AS int_as_bool, get_variant_string(data, '$.commit.record.nonexistent') AS missing_string FROM bluesky LIMIT 1; ``` ```plaintext +-------------+------------------+--------------+----------------+ | missing_int | string_as_double | int_as_bool | missing_string | +-------------+------------------+--------------+----------------+ | NULL | NULL | NULL | NULL | +-------------+------------------+--------------+----------------+ ``` #### keyword[​](#keyword "Direct link to keyword") GET\_VARIANT\_INT,GET\_VARIANT\_DOUBLE,GET\_VARIANT\_STRING,GET\_VARIANT\_BOOL,VARIANT --- ### variant_query Queries the value of an element that can be located by the path expression in a VARIANT object and returns a VARIANT value. This function is used to navigate and extract sub-elements from VARIANT data read from Iceberg tables in Parquet format. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext variant_query(variant_expr, path) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `variant_expr`: the expression that represents the VARIANT object. This is typically a VARIANT column from an Iceberg table. * `path`: the expression that represents the path to an element in the VARIANT object. The value of this parameter is a string. The path syntax is similar to JSON path: * `$` represents the root element * `.` is used to access object fields * `[index]` is used to access array elements (0-based indexing) * Field names with special characters can be quoted: `$."field.name"` #### Return value[​](#return-value "Direct link to Return value") Returns a VARIANT value. If the element does not exist or the path is invalid, the function returns NULL. #### Examples[​](#examples "Direct link to Examples") Example 1: Query the root element of a VARIANT value. ```sql SELECT variant_query(data, '$') AS root_data FROM bluesky LIMIT 1; ``` ```plaintext +-------------------------------------------------------------------------------------------+ | root_data | +-------------------------------------------------------------------------------------------+ | {"commit":{"collection":"app.bsky.graph.follow","operation":"delete","rev":"3lcgs4mw...} | +-------------------------------------------------------------------------------------------+ ``` Example 2: Query a nested field in a VARIANT object. ```sql SELECT variant_query(data, '$.commit') AS commit_info FROM bluesky LIMIT 1; ``` ```plaintext +--------------------------------------------------------------------------------+ | commit_info | +--------------------------------------------------------------------------------+ | {"collection":"app.bsky.graph.follow","operation":"delete","rev":"3lcgs4mw...} | +--------------------------------------------------------------------------------+ ``` Example 3: Check the type of nested fields using variant\_typeof. ```sql SELECT variant_typeof(variant_query(data, '$.commit')) AS commit_type, variant_typeof(variant_query(data, '$.time_us')) AS time_type FROM bluesky LIMIT 1; ``` ```plaintext +-------------+-----------+ | commit_type | time_type | +-------------+-----------+ | Object | Int64 | +-------------+-----------+ ``` Example 4: Cast the result to a SQL type. ```sql SELECT CAST(variant_query(data, '$.commit') AS STRING) AS commit_info FROM bluesky LIMIT 1; ``` ```plaintext +-----------------------------------------------------------------------------------+ | commit_info | +-----------------------------------------------------------------------------------+ | {"cid":"bafyreia3k...","collection":"app.bsky.feed.repost","operation":"create"...} | +-----------------------------------------------------------------------------------+ ``` Example 5: Filter using variant\_query results. ```sql SELECT COUNT(*) AS total FROM bluesky WHERE variant_query(data, '$.commit.record') IS NOT NULL; ``` ```plaintext +---------+ | total | +---------+ | 9500118 | +---------+ ``` #### keyword[​](#keyword "Direct link to keyword") VARIANT\_QUERY,VARIANT --- ### variant_typeof Returns the type name of a VARIANT value as a string. This function inspects a VARIANT value and returns its type as a string. This is useful for understanding the structure of VARIANT data or for conditional processing based on the data type. #### Syntax[​](#syntax "Direct link to Syntax") ```plaintext VARCHAR variant_typeof(variant_expr) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `variant_expr`: the expression that represents the VARIANT object. This is typically a VARIANT column from an Iceberg table or a VARIANT value returned by `variant_query`. #### Return value[​](#return-value "Direct link to Return value") Returns a VARCHAR value representing the type name. Possible return values include: * `"Null"` - for NULL values * `"Boolean(true)"` - for boolean true values * `"Boolean(false)"` - for boolean false values * `"Int8"`, `"Int16"`, `"Int32"`, `"Int64"` - for integer values * `"Float"`, `"Double"` - for floating-point values * `"Decimal4"`, `"Decimal8"`, `"Decimal16"` - for decimal values with different precision * `"String"` - for string values * `"Binary"` - for binary data * `"Date"` - for date values * `"TimestampTz"`, `"TimestampNtz"` - for timestamp values with or without timezone * `"TimestampTzNanos"`, `"TimestampNtzNanos"` - for nanosecond-precision timestamp values * `"TimeNtz"` - for time values without timezone * `"Uuid"` - for UUID values * `"Object"` - for struct or map values * `"Array"` - for array values #### Examples[​](#examples "Direct link to Examples") Example 1: Get the type of a VARIANT value at the root level. ```sql SELECT variant_typeof(data) AS type FROM bluesky LIMIT 1; ``` ```plaintext +--------+ | type | +--------+ | Object | +--------+ ``` Example 2: Get the type of nested fields. ```sql SELECT variant_typeof(variant_query(data, '$.kind')) AS kind_type, variant_typeof(variant_query(data, '$.did')) AS did_type, variant_typeof(variant_query(data, '$.commit')) AS commit_type FROM bluesky LIMIT 1; ``` ```plaintext +-----------+----------+-------------+ | kind_type | did_type | commit_type | +-----------+----------+-------------+ | String | String | Object | +-----------+----------+-------------+ ``` Example 3: Check types of multiple fields. ```sql SELECT variant_typeof(variant_query(data, '$.commit')) AS commit_type, variant_typeof(variant_query(data, '$.time_us')) AS time_type FROM bluesky LIMIT 1; ``` ```plaintext +-------------+-----------+ | commit_type | time_type | +-------------+-----------+ | Object | Int64 | +-------------+-----------+ ``` Example 4: Use in conditional logic to handle different types. ```sql SELECT CASE variant_typeof(variant_query(data, '$.time_us')) WHEN 'Int64' THEN get_variant_int(data, '$.time_us') ELSE NULL END AS timestamp_value FROM bluesky LIMIT 3; ``` ```plaintext +------------------+ | timestamp_value | +------------------+ | 1733267476040329 | | 1733267476040803 | | 1733267476041472 | +------------------+ ``` Example 5: Filter rows based on type. ```sql SELECT COUNT(*) AS object_count FROM bluesky WHERE variant_typeof(variant_query(data, '$.commit')) = 'Object'; ``` ```plaintext +--------------+ | object_count | +--------------+ | 9960624 | +--------------+ ``` #### keyword[​](#keyword "Direct link to keyword") VARIANT\_TYPEOF,VARIANT --- ### Window functions #### Background[​](#background "Direct link to Background") The window function is a special class of built-in functions. Similar to the aggregation function, it also does calculations on multiple input rows to get a single data value. The difference is that the window function processes the input data within a specific window, rather than using the "group by" method. The data in each window can be sorted and grouped using the over() clause. The window function **computes a separate value for each row**, rather than computing one value for each group. This flexibility allows users to add additional columns to the select clause and further filter the result set. The window function can only appear in the select list and the outermost position of a clause. It takes effect at the end of the query, that is, after the `join`, `where`, and `group by` operations are performed. The window function is often used to analyze trends, calculate outliers, and perform bucketing analyses on large-scale data. #### Usage[​](#usage "Direct link to Usage") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql function(args) OVER([partition_by_clause] [order_by_clause] [order_by_clause window_clause]) partition_by_clause ::= PARTITION BY expr [, expr ...] order_by_clause ::= ORDER BY expr [ASC | DESC] [, expr [ASC | DESC] ...] ``` ##### PARTITION BY clause[​](#partition-by-clause "Direct link to PARTITION BY clause") The Partition By clause is similar to Group By. It groups the input rows by one or more specified columns. Rows with the same value are grouped together. ##### ORDER BY clause[​](#order-by-clause "Direct link to ORDER BY clause") The `Order By` clause is basically the same as the outer `Order By`. It defines the order of the input rows. If `Partition By` is specified, `Order By` defines the order within each Partition grouping. The only difference is that `Order By n` (n is a positive integer) in the `OVER` clause is equivalent to no operation, whereas `n` in the outer `Order By` indicates sorting by the nth column. Example: This example shows adding an id column to the select list with values of 1, 2, 3, etc., sorted by the `date_and_time` column in the events table. ```sql SELECT row_number() OVER (ORDER BY date_and_time) AS id, c1, c2, c3, c4 FROM events; ``` ##### Window clause[​](#window-clause "Direct link to Window clause") The window clause is used to specify a range of rows for operations (the preceding and following rows based on the current row). It supports the following syntaxes: AVG(), COUNT(), FIRST\_VALUE(), LAST\_VALUE(), and SUM(). For MAX() and MIN(), the window clause can specify the start to `UNBOUNDED PRECEDING`. **Syntax:** ```sql ROWS BETWEEN [ { m | UNBOUNDED } PRECEDING | CURRENT ROW] [ AND [CURRENT ROW | { UNBOUNDED | n } FOLLOWING] ] RANGE BETWEEN [ { m | UNBOUNDED } PRECEDING | CURRENT ROW] [ AND [CURRENT ROW | { UNBOUNDED | n } FOLLOWING] ] ``` note **ARRAY\_AGG() window frame limitation:** When using ARRAY\_AGG() as a window function, only RANGE frames are supported. ROWS frames are NOT supported. For example: ```sql -- Supported: RANGE frame array_agg(col) OVER (PARTITION BY x ORDER BY y RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) -- NOT Supported: ROWS frame (will cause error) array_agg(col) OVER (PARTITION BY x ORDER BY y ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) ``` #### Window function sample table[​](#window-function-sample-table "Direct link to Window function sample table") This section creates a sample table `scores`. You can use this table to test many window functions below. ```sql CREATE TABLE `scores` ( `id` int(11) NULL, `name` varchar(11) NULL, `subject` varchar(11) NULL, `score` int(11) NULL ) DISTRIBUTED BY HASH(`score`) BUCKETS 10; INSERT INTO `scores` VALUES (1, "lily", "math", NULL), (1, "lily", "english", 100), (1, "lily", "physics", 60), (2, "tom", "math", 80), (2, "tom", "english", 98), (2, "tom", "physics", NULL), (3, "jack", "math", 95), (3, "jack", "english", NULL), (3, "jack", "physics", 99), (4, "amy", "math", 80), (4, "amy", "english", 92), (4, "amy", "physics", 99), (5, "mike", "math", 70), (5, "mike", "english", 85), (5, "mike", "physics", 85), (6, "amber", "math", 92), (6, "amber", NULL, 90), (6, "amber", "physics", 100); ``` #### Function examples[​](#function-examples "Direct link to Function examples") This section describes the window functions supported in StarRocks. ##### AVG()[​](#avg "Direct link to AVG()") Calculates the average value of a field in a given window. This function ignores NULL values. **Syntax:** ```sql AVG([DISTINCT] expr) [OVER (*analytic_clause*)] ``` `DISTINCT` is supported from StarRocks v4.0. When specified, AVG() calculates the average of only distinct values in the window. note **Window frame limitation:** When using AVG(DISTINCT) as a window function, only RANGE frames are supported. ROWS frames are NOT supported. **Examples** **Example 1: Basic usage** The following example uses stock data as an example. ```sql CREATE TABLE stock_ticker ( stock_symbol STRING, closing_price DECIMAL(8,2), closing_date DATETIME ) DUPLICATE KEY(stock_symbol) COMMENT "OLAP" DISTRIBUTED BY HASH(closing_date); INSERT INTO stock_ticker VALUES ("JDR", 12.86, "2014-10-02 00:00:00"), ("JDR", 12.89, "2014-10-03 00:00:00"), ("JDR", 12.94, "2014-10-04 00:00:00"), ("JDR", 12.55, "2014-10-05 00:00:00"), ("JDR", 14.03, "2014-10-06 00:00:00"), ("JDR", 14.75, "2014-10-07 00:00:00"), ("JDR", 13.98, "2014-10-08 00:00:00") ; ``` Calculate the average closing price in the current row and each row before and after it. ```sql select stock_symbol, closing_date, closing_price, avg(closing_price) over (partition by stock_symbol order by closing_date rows between 1 preceding and 1 following ) as moving_average from stock_ticker; ``` Output: ```plaintext +--------------+---------------------+---------------+----------------+ | stock_symbol | closing_date | closing_price | moving_average | +--------------+---------------------+---------------+----------------+ | JDR | 2014-10-02 00:00:00 | 12.86 | 12.87500000 | | JDR | 2014-10-03 00:00:00 | 12.89 | 12.89666667 | | JDR | 2014-10-04 00:00:00 | 12.94 | 12.79333333 | | JDR | 2014-10-05 00:00:00 | 12.55 | 13.17333333 | | JDR | 2014-10-06 00:00:00 | 14.03 | 13.77666667 | | JDR | 2014-10-07 00:00:00 | 14.75 | 14.25333333 | | JDR | 2014-10-08 00:00:00 | 13.98 | 14.36500000 | +--------------+---------------------+---------------+----------------+ ``` For example, `12.87500000` in the first row is the average value of closing prices on "2014-10-02" (`12.86`), its previous day "2014-10-01" (null), and its following day "2014-10-03" (`12.89`). **Example 2: Using AVG(DISTINCT) over overall window** This example uses the data in the [Sample table](#window-function-sample-table) `scores`. Calculate the average of distinct scores across all rows: ```sql SELECT id, subject, score, AVG(DISTINCT score) OVER () AS distinct_avg FROM test_scores; ``` Output: ```plaintext +----+---------+-------+-------------+ | id | subject | score | distinct_avg| +----+---------+-------+-------------+ | 1 | math | 80 | 85.00 | | 2 | math | 85 | 85.00 | | 3 | math | 80 | 85.00 | | 4 | english | 90 | 85.00 | | 5 | english | 85 | 85.00 | | 6 | english | 90 | 85.00 | +----+---------+-------+-------------+ ``` The distinct average is 85.00 (`(80 + 85 + 90) / 3`). **Example 3: Using AVG(DISTINCT) over framed window with RANGE frame** This example uses the data in the [Sample table](#window-function-sample-table) `scores`. Calculate the average of distinct scores within each subject partition using a RANGE frame: ```sql SELECT id, subject, score, AVG(DISTINCT score) OVER ( PARTITION BY subject ORDER BY score RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS distinct_avg FROM test_scores; ``` Output: ```plaintext +----+---------+-------+-------------+ | id | subject | score | distinct_avg| +----+---------+-------+-------------+ | 1 | math | 80 | 80.00 | | 3 | math | 80 | 80.00 | | 2 | math | 85 | 82.50 | | 5 | english | 85 | 85.00 | | 4 | english | 90 | 87.50 | | 6 | english | 90 | 87.50 | +----+---------+-------+-------------+ ``` For each row, the function calculates the average of distinct scores from the beginning of the partition up to and including the current row's score value. ##### ARRAY\_AGG()[​](#array_agg "Direct link to ARRAY_AGG()") Aggregates values (including NULL values) in a window into an array. You can use the optional `ORDER BY` clause to sort the elements within the array. This function is supported from v3.4. tip **Window frame limitation:** ARRAY\_AGG() as a window function only supports RANGE window frames. ROWS window frames are NOT supported. If no window frame is specified, the default `RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW` is used. **Syntax:** ```sql ARRAY_AGG([DISTINCT] expr [ORDER BY expr [ASC | DESC]]) OVER([partition_by_clause] [order_by_clause] [window_clause]) ``` **Parameters:** * `expr`: The expression to aggregate. It can be a column of any supported data type. * `DISTINCT`: Optional. Eliminates duplicate values from the result array. * `ORDER BY`: Optional. Specifies the order of elements within the array. **Return value:** Returns an ARRAY containing all values in the window. **Usage notes:** * **ROWS frames are NOT supported.** Only RANGE frames can be used with ARRAY\_AGG() as a window function. Using ROWS frames will result in an error. * NULL values are included in the result array. * When `DISTINCT` is specified, duplicate values are removed from the array. * When `ORDER BY` is specified within ARRAY\_AGG(), the elements in the resulting array are sorted accordingly. **Examples** These examples use the data in the [Sample table](#window-function-sample-table) `scores`. **Example 1: Basic usage** Collect all scores within each subject partition: ```sql SELECT *, array_agg(score) OVER ( PARTITION BY subject ORDER BY score ) AS score_array FROM scores WHERE subject = 'math'; ``` Output: ```plaintext +------+-------+---------+-------+----------------------+ | id | name | subject | score | score_array | +------+-------+---------+-------+----------------------+ | 1 | lily | math | NULL | [null] | | 5 | mike | math | 70 | [null,70] | | 2 | tom | math | 80 | [null,70,80,80] | | 4 | amy | math | 80 | [null,70,80,80] | | 6 | amber | math | 92 | [null,70,80,80,92] | | 3 | jack | math | 95 | [null,70,80,80,92,95]| +------+-------+---------+-------+----------------------+ ``` Note: Rows with the same `score` value (tom and amy both have 80) receive the same array due to the RANGE frame semantics. **Example 2: ARRAY\_AGG(DISTINCT) over window** Collect distinct scores within each subject partition: ```sql SELECT *, array_agg(DISTINCT score) OVER ( PARTITION BY subject ORDER BY score ) AS distinct_scores FROM scores WHERE subject = 'math'; ``` Output: ```plaintext +------+-------+---------+-------+-------------------+ | id | name | subject | score | distinct_scores | +------+-------+---------+-------+-------------------+ | 1 | lily | math | NULL | [null] | | 5 | mike | math | 70 | [null,70] | | 2 | tom | math | 80 | [null,70,80] | | 4 | amy | math | 80 | [null,70,80] | | 6 | amber | math | 92 | [null,70,80,92] | | 3 | jack | math | 95 | [null,70,80,92,95]| +------+-------+---------+-------+-------------------+ ``` **Example 3: ARRAY\_AGG() with ORDER BY** Collect scores sorted in descending order within the array: ```sql SELECT *, array_agg(score ORDER BY score DESC) OVER ( PARTITION BY subject ) AS scores_desc FROM scores WHERE subject = 'math'; ``` Output: ```plaintext +------+-------+---------+-------+----------------------+ | id | name | subject | score | scores_desc | +------+-------+---------+-------+----------------------+ | 1 | lily | math | NULL | [95,92,80,80,70,null]| | 5 | mike | math | 70 | [95,92,80,80,70,null]| | 2 | tom | math | 80 | [95,92,80,80,70,null]| | 4 | amy | math | 80 | [95,92,80,80,70,null]| | 6 | amber | math | 92 | [95,92,80,80,70,null]| | 3 | jack | math | 95 | [95,92,80,80,70,null]| +------+-------+---------+-------+----------------------+ ``` **Example 4: ARRAY\_AGG() with RANGE frame** Collect all scores in the entire partition using RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING: ```sql SELECT *, array_agg(score) OVER ( PARTITION BY subject ORDER BY score RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING ) AS all_scores FROM scores WHERE subject = 'math'; ``` Output: ```plaintext +------+-------+---------+-------+----------------------+ | id | name | subject | score | all_scores | +------+-------+---------+-------+----------------------+ | 1 | lily | math | NULL | [null,70,80,80,92,95]| | 5 | mike | math | 70 | [null,70,80,80,92,95]| | 2 | tom | math | 80 | [null,70,80,80,92,95]| | 4 | amy | math | 80 | [null,70,80,80,92,95]| | 6 | amber | math | 92 | [null,70,80,80,92,95]| | 3 | jack | math | 95 | [null,70,80,80,92,95]| +------+-------+---------+-------+----------------------+ ``` **Example 5: Collect names partitioned by score ranges** Using the `stock_ticker` table, collect stock symbols in a moving window: ```sql SELECT stock_symbol, closing_date, closing_price, array_agg(closing_price) OVER ( PARTITION BY stock_symbol ORDER BY closing_date ) AS price_history FROM stock_ticker; ``` Output: ```plaintext +--------------+---------------------+---------------+---------------------------------------+ | stock_symbol | closing_date | closing_price | price_history | +--------------+---------------------+---------------+---------------------------------------+ | JDR | 2014-10-02 00:00:00 | 12.86 | [12.86] | | JDR | 2014-10-03 00:00:00 | 12.89 | [12.86,12.89] | | JDR | 2014-10-04 00:00:00 | 12.94 | [12.86,12.89,12.94] | | JDR | 2014-10-05 00:00:00 | 12.55 | [12.86,12.89,12.94,12.55] | | JDR | 2014-10-06 00:00:00 | 14.03 | [12.86,12.89,12.94,12.55,14.03] | | JDR | 2014-10-07 00:00:00 | 14.75 | [12.86,12.89,12.94,12.55,14.03,14.75] | | JDR | 2014-10-08 00:00:00 | 13.98 | [12.86,12.89,12.94,12.55,14.03,14.75,13.98] | +--------------+---------------------+---------------+---------------------------------------+ ``` **Example 6: Invalid usage - ROWS frame (will cause error)** The following query will fail because ROWS frames are not supported: ```sql -- This will cause an error! SELECT *, array_agg(score) OVER ( PARTITION BY subject ORDER BY score ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING -- NOT SUPPORTED! ) AS score_array FROM scores; ``` Error message: ```plaintext ERROR: array_agg as window function does not support ROWS frame type. Please use RANGE frame instead. ``` ##### COUNT()[​](#count "Direct link to COUNT()") Calculates the total number of rows that meet the specified conditions in a give window. **Syntax:** ```sql COUNT([DISTINCT] expr) [OVER (analytic_clause)] ``` `DISTINCT` is supported from StarRocks v4.0. When specified, COUNT() counts only distinct values in the window. note **Window frame limitation:** When using COUNT(DISTINCT) as a window function, only RANGE frames are supported. ROWS frames are NOT supported. For example: ```sql -- Supported: RANGE frame count(distinct col) OVER (PARTITION BY x ORDER BY y RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) -- NOT Supported: ROWS frame (will cause error) count(distinct col) OVER (PARTITION BY x ORDER BY y ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) ``` **Examples** These examples use the data in the [Sample table](#window-function-sample-table) `scores`. **Example 1: Basic usage** Count the occurrence of math scores that are greater than 90 from the current row to the first row in the math partition. This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```sql select *, count(score) over ( partition by subject order by score rows between unbounded preceding and current row ) as 'score_count' from scores where subject in ('math') and score > 90; ``` ```plaintext +------+-------+---------+-------+-------------+ | id | name | subject | score | score_count | +------+-------+---------+-------+-------------+ | 6 | amber | math | 92 | 1 | | 3 | jack | math | 95 | 2 | +------+-------+---------+-------+-------------+ ``` **Example 2: Using COUNT(DISTINCT) over overall window** Count distinct scores across all rows: ```sql CREATE TABLE test_scores ( id INT, subject VARCHAR(20), score INT ) DISTRIBUTED BY HASH(id); INSERT INTO test_scores VALUES (1, 'math', 80), (2, 'math', 85), (3, 'math', 80), (4, 'english', 90), (5, 'english', 85), (6, 'english', 90); ``` ```sql SELECT id, subject, score, COUNT(DISTINCT score) OVER () AS distinct_count FROM test_scores; ``` Output: ```plaintext +----+---------+-------+---------------+ | id | subject | score | distinct_count| +----+---------+-------+---------------+ | 1 | math | 80 | 4 | | 2 | math | 85 | 4 | | 3 | math | 80 | 4 | | 4 | english | 90 | 4 | | 5 | english | 85 | 4 | | 6 | english | 90 | 4 | +----+---------+-------+---------------+ ``` The distinct count is 4 (values: 80, 85, 90, and NULL if any). **Example 3: Using COUNT(DISTINCT) over framed window with RANGE frame** Count distinct scores within each subject partition using a RANGE frame: ```sql SELECT id, subject, score, COUNT(DISTINCT score) OVER ( PARTITION BY subject ORDER BY score RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS distinct_count FROM test_scores; ``` Output: ```plaintext +----+---------+-------+---------------+ | id | subject | score | distinct_count| +----+---------+-------+---------------+ | 1 | math | 80 | 1 | | 3 | math | 80 | 1 | | 2 | math | 85 | 2 | | 5 | english | 85 | 1 | | 4 | english | 90 | 2 | | 6 | english | 90 | 2 | +----+---------+-------+---------------+ ``` For each row, the function counts distinct scores from the beginning of the partition up to and including the current row's score value. ##### CUME\_DIST()[​](#cume_dist "Direct link to CUME_DIST()") The CUME\_DIST() function calculates the cumulative distribution of a value within a partition or window, indicating its relative position as a percentage in the partition. It is often used to calculate the distribution of highest or lowest values in a group. * If data is sorted in ascending order, this function calculates the percentage of values less than or equal to the value in the current row. * If data is sorted in descending order, this function calculates the percentage of values greater than or equal to the value in the current row. The cumulative distribution is in the range of 0 to 1. It is useful for percentile calculation and data distribution analysis. This function is supported from v3.2. **Syntax:** ```sql CUME_DIST() OVER (partition_by_clause order_by_clause) ``` * `partition_by_clause`: optional. If this clause is not specified, the entire result set is processed as a single partition. * `order_by_clause`: **This function must be used with ORDER BY to sort partition rows into the desired order.** CUME\_DIST() contains NULL values and treats them as the lowest values. **Examples** The following example shows the cumulative distribution of each score within each `subject` group. This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```plaintext SELECT *, cume_dist() OVER ( PARTITION BY subject ORDER BY score ) AS cume_dist FROM scores; +------+-------+---------+-------+---------------------+ | id | name | subject | score | cume_dist | +------+-------+---------+-------+---------------------+ | 6 | amber | NULL | 90 | 1 | | 3 | jack | english | NULL | 0.2 | | 5 | mike | english | 85 | 0.4 | | 4 | amy | english | 92 | 0.6 | | 2 | tom | english | 98 | 0.8 | | 1 | lily | english | 100 | 1 | | 1 | lily | math | NULL | 0.16666666666666666 | | 5 | mike | math | 70 | 0.3333333333333333 | | 2 | tom | math | 80 | 0.6666666666666666 | | 4 | amy | math | 80 | 0.6666666666666666 | | 6 | amber | math | 92 | 0.8333333333333334 | | 3 | jack | math | 95 | 1 | | 2 | tom | physics | NULL | 0.16666666666666666 | | 1 | lily | physics | 60 | 0.3333333333333333 | | 5 | mike | physics | 85 | 0.5 | | 4 | amy | physics | 99 | 0.8333333333333334 | | 3 | jack | physics | 99 | 0.8333333333333334 | | 6 | amber | physics | 100 | 1 | +------+-------+---------+-------+---------------------+ ``` * For `cume_dist` in the first row, the `NULL` group has only one row, and only this row itself meets the condition of "less than or equal to the current row". The cumulative distribution is 1。 * For `cume_dist` in the second row, the `english` group has five rows, and only this row itself (NULL) meets the condition of "less than or equal to the current row". The cumulative distribution is 0.2. * For `cume_dist` in the third row, the `english` group has five rows, and two rows (85 and NULL) meet the condition of "less than or equal to the current row". The cumulative distribution is 0.4. ##### DENSE\_RANK()[​](#dense_rank "Direct link to DENSE_RANK()") The DENSE\_RANK() function is used to represent rankings. Unlike RANK(), DENSE\_RANK() **does not have vacant** numbers. For example, if there are two 1s, the third number of DENSE\_RANK() is still 2, whereas the third number of RANK() is 3. **Syntax:** ```sql DENSE_RANK() OVER(partition_by_clause order_by_clause) ``` **Examples** The following example shows the ranking of math scores (sorted in descending order). This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```sql select *, dense_rank() over ( partition by subject order by score desc ) as `rank` from scores where subject in ('math'); ``` ```plaintext +------+-------+---------+-------+------+ | id | name | subject | score | rank | +------+-------+---------+-------+------+ | 3 | jack | math | 95 | 1 | | 6 | amber | math | 92 | 2 | | 2 | tom | math | 80 | 3 | | 4 | amy | math | 80 | 3 | | 5 | mike | math | 70 | 4 | | 1 | lily | math | NULL | 5 | +------+-------+---------+-------+------+ ``` The result data has two rows whose score is 80. They all rank 3. The rank for the next score 70 is 4. This shows DENSE\_RANK() **does not have vacant** numbers. ##### FIRST\_VALUE()[​](#first_value "Direct link to FIRST_VALUE()") FIRST\_VALUE() returns the **first** value of the window range. **Syntax:** ```sql FIRST_VALUE(expr [IGNORE NULLS]) OVER(partition_by_clause order_by_clause [window_clause]) ``` `IGNORE NULLS` is supported from v2.5.0. It is used to determine whether NULL values of `expr` are eliminated from the calculation. By default, NULL values are included, which means NULL is returned if the first value in the filtered result is NULL. If you specify IGNORE NULLS, the first non-null value in the filtered result is returned. If all the values are NULL, NULL is returned even if you specify IGNORE NULLS. ARRAY types are supported from StarRocks v3.5. You can use FIRST\_VALUE() with ARRAY columns to get the first array value in the window. **Examples** **Example 1: Basic usage** Return the first `score` value for each member in each group (descending order), grouping by `subject`. This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```sql select *, first_value(score IGNORE NULLS) over ( partition by subject order by score desc ) as first from scores; ``` ```plaintext +------+-------+---------+-------+-------+ | id | name | subject | score | first | +------+-------+---------+-------+-------+ | 1 | lily | english | 100 | 100 | | 2 | tom | english | 98 | 100 | | 4 | amy | english | 92 | 100 | | 5 | mike | english | 85 | 100 | | 3 | jack | english | NULL | 100 | | 6 | amber | physics | 100 | 100 | | 3 | jack | physics | 99 | 100 | | 4 | amy | physics | 99 | 100 | | 5 | mike | physics | 85 | 100 | | 1 | lily | physics | 60 | 100 | | 2 | tom | physics | NULL | 100 | | 6 | amber | NULL | 90 | 90 | | 3 | jack | math | 95 | 95 | | 6 | amber | math | 92 | 95 | | 2 | tom | math | 80 | 95 | | 4 | amy | math | 80 | 95 | | 5 | mike | math | 70 | 95 | | 1 | lily | math | NULL | 95 | +------+-------+---------+-------+-------+ ``` **Example 2: Using FIRST\_VALUE() with ARRAY types** Create a table with ARRAY columns: ```sql CREATE TABLE test_array_value ( col_1 INT, arr1 ARRAY ) DISTRIBUTED BY HASH(col_1); INSERT INTO test_array_value (col_1, arr1) VALUES (1, [1, 11]), (2, [2, 22]), (3, [3, 33]), (4, NULL), (5, [5, 55]); ``` Query data using FIRST\_VALUE() with ARRAY types: ```sql SELECT col_1, arr1, FIRST_VALUE(arr1) OVER (ORDER BY col_1) AS first_array FROM test_array_value; ``` Output: ```plaintext +-------+--------+------------+ | col_1 | arr1 | first_array| +-------+--------+------------+ | 1 | [1,11] | [1,11] | | 2 | [2,22] | [1,11] | | 3 | [3,33] | [1,11] | | 4 | NULL | [1,11] | | 5 | [5,55] | [1,11] | +-------+--------+------------+ ``` The first array value `[1,11]` is returned for all rows in the window. ##### LAST\_VALUE()[​](#last_value "Direct link to LAST_VALUE()") LAST\_VALUE() returns the **last** value of the window range. It is the opposite of FIRST\_VALUE(). **Syntax:** ```sql LAST_VALUE(expr [IGNORE NULLS]) OVER(partition_by_clause order_by_clause [window_clause]) ``` `IGNORE NULLS` is supported from v2.5.0. It is used to determine whether NULL values of `expr` are eliminated from the calculation. By default, NULL values are included, which means NULL is returned if the last value in the filtered result is NULL. If you specify IGNORE NULLS, the last non-null value in the filtered result is returned. If all the values are NULL, NULL is returned even if you specify IGNORE NULLS. By default, LAST\_VALUE() calculates `rows between unbounded preceding and current row`, which compares the current row with all its preceding rows. If you want to show only one value for each partition, use `rows between unbounded preceding and unbounded following` after ORDER BY. ARRAY types are supported from StarRocks v3.5. You can use LAST\_VALUE() with ARRAY columns to get the last array value in the window. **Examples** **Example 1: Basic usage** Returns the last `score` for each member in the group (descending order), grouping by `subject`. This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```sql select *, last_value(score IGNORE NULLS) over ( partition by subject order by score desc rows between unbounded preceding and unbounded following ) as last from scores; ``` ```plaintext +------+-------+---------+-------+------+ | id | name | subject | score | last | +------+-------+---------+-------+------+ | 1 | lily | english | 100 | 85 | | 2 | tom | english | 98 | 85 | | 4 | amy | english | 92 | 85 | | 5 | mike | english | 85 | 85 | | 3 | jack | english | NULL | 85 | | 6 | amber | physics | 100 | 60 | | 3 | jack | physics | 99 | 60 | | 4 | amy | physics | 99 | 60 | | 5 | mike | physics | 85 | 60 | | 1 | lily | physics | 60 | 60 | | 2 | tom | physics | NULL | 60 | | 6 | amber | NULL | 90 | 90 | | 3 | jack | math | 95 | 70 | | 6 | amber | math | 92 | 70 | | 2 | tom | math | 80 | 70 | | 4 | amy | math | 80 | 70 | | 5 | mike | math | 70 | 70 | | 1 | lily | math | NULL | 70 | +------+-------+---------+-------+------+ ``` **Example 2: Using LAST\_VALUE() with ARRAY types** Use the same table from FIRST\_VALUE() Example 2: ```sql SELECT col_1, arr1, LAST_VALUE(arr1) OVER ( ORDER BY col_1 ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING ) AS last_array FROM test_array_value; ``` Output: ```plaintext +-------+--------+-----------+ | col_1 | arr1 | last_array| +-------+--------+-----------+ | 1 | [1,11] | [5,55] | | 2 | [2,22] | [5,55] | | 3 | [3,33] | [5,55] | | 4 | NULL | [5,55] | | 5 | [5,55] | [5,55] | +-------+--------+-----------+ ``` The last array value `[5,55]` is returned for all rows in the window. ##### LAG()[​](#lag "Direct link to LAG()") Returns the value of the row that lags the current row by `offset` rows. This function is often used to compare values between rows and filter data. `LAG()` can be used to query data of the following types: * Numeric: TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, DECIMAL * String: CHAR, VARCHAR * Date: DATE, DATETIME * BITMAP and HLL are supported from StarRocks v2.5. * ARRAY types are supported from StarRocks v3.5. **Syntax:** ```sql LAG(expr [IGNORE NULLS] [, offset[, default]]) OVER([] []) ``` **Parameters:** * `expr`: the field you want to compute. * `offset`: the offset. It must be a **positive integer**. If this parameter is not specified, 1 is the default. * `default`: the default value returned if no matching row is found. If this parameter is not specified, NULL is the default. `default` supports any expression whose type is compatible with `expr`, since version 4.0, default no longer has to be a constant—it can be a column name. * `IGNORE NULLS` is supported from v3.0. It is used to determine whether NULL values of `expr` are included in the result. By default, NULL values are included when `offset` rows are counted, which means NULL is returned if the value of the destination row is NULL. See Example 1. If you specify IGNORE NULLS, NULL values are ignored when `offset` rows are counted and the system continues to search for `offset` non-null values. If `offset` non-null values cannot be found, NULL or `default` (if specified) is returned. See Example 2. **Examples** **Example 1: IGNORE NULLS is not specified** Create a table and insert values: ```sql CREATE TABLE test_tbl (col_1 INT, col_2 INT) DISTRIBUTED BY HASH(col_1); INSERT INTO test_tbl VALUES (1, NULL), (2, 4), (3, NULL), (4, 2), (5, NULL), (6, 7), (7, 6), (8, 5), (9, NULL), (10, NULL); ``` Query data from this table, where `offset` is 2, which means traversing the previous two rows; `default` is 0, which means 0 is returned if no matching rows are found. Output: ```plaintext SELECT col_1, col_2, LAG(col_2,2,0) OVER (ORDER BY col_1) FROM test_tbl ORDER BY col_1; +-------+-------+---------------------------------------------+ | col_1 | col_2 | lag(col_2, 2, 0) OVER (ORDER BY col_1 ASC ) | +-------+-------+---------------------------------------------+ | 1 | NULL | 0 | | 2 | 4 | 0 | | 3 | NULL | NULL | | 4 | 2 | 4 | | 5 | NULL | NULL | | 6 | 7 | 2 | | 7 | 6 | NULL | | 8 | 5 | 7 | | 9 | NULL | 6 | | 10 | NULL | 5 | +-------+-------+---------------------------------------------+ ``` For the first two rows, no previous two rows exist and the default value 0 is returned. For NULL in row 3, the value two rows backward is NULL and NULL is returned because NULL values are allowed. **Example 2: IGNORE NULLS is specified** Use the preceding table and parameter settings. ```sql SELECT col_1, col_2, LAG(col_2 IGNORE NULLS,2,0) OVER (ORDER BY col_1) FROM test_tbl ORDER BY col_1; +-------+-------+---------------------------------------------+ | col_1 | col_2 | lag(col_2, 2, 0) OVER (ORDER BY col_1 ASC ) | +-------+-------+---------------------------------------------+ | 1 | NULL | 0 | | 2 | 4 | 0 | | 3 | NULL | 0 | | 4 | 2 | 0 | | 5 | NULL | 4 | | 6 | 7 | 4 | | 7 | 6 | 2 | | 8 | 5 | 7 | | 9 | NULL | 6 | | 10 | NULL | 6 | +-------+-------+---------------------------------------------+ ``` For rows 1 to 4, the system cannot find two non-NULL values for each of them in the previous rows and the default value 0 is returned. For value 6 in row 7, the value two rows backward is NULL and NULL is ignored because IGNORE NULLS is specified. The system continues to search for non-null values and 2 in row 4 is returned. **Example 3: Setting the default value in LAG() to a column name** Use the preceding table and parameter settings. ```sql SELECT col_1, col_2, LAG(col_2 ,2,col_1) OVER (ORDER BY col_1) FROM test_tbl ORDER BY col_1; +-------+-------+-------------------------------------------------+ | col_1 | col_2 | lag(col_2, 2, col_1) OVER (ORDER BY col_1 ASC ) | +-------+-------+-------------------------------------------------+ | 1 | NULL | 1 | | 2 | 4 | 2 | | 3 | NULL | NULL | | 4 | 2 | 4 | | 5 | NULL | NULL | | 6 | 7 | 2 | | 7 | 6 | NULL | | 8 | 5 | 7 | | 9 | NULL | 6 | | 10 | NULL | 5 | +-------+-------+-------------------------------------------------+ ``` As you can see, for rows 1 and 2 there are not two non-NULL values when scanning backward, so the default returned is the current row’s col\_1 value. All other rows behave the same as in Example 1. **Example 4: Using LAG() with ARRAY types** Create a table with ARRAY columns: ```sql CREATE TABLE test_array_value ( col_1 INT, arr1 ARRAY, arr2 ARRAY NOT NULL ) DISTRIBUTED BY HASH(col_1); INSERT INTO test_array_value (col_1, arr1, arr2) VALUES (1, [1, 11], [101, 111]), (2, [2, 22], [102, 112]), (3, [3, 33], [103, 113]), (4, NULL, [104, 114]), (5, [5, 55], [105, 115]), (6, [6, 66], [106, 116]); ``` Query data using LAG() with ARRAY types: ```sql SELECT col_1, arr1, LAG(arr1, 2, arr2) OVER (ORDER BY col_1) AS lag_result FROM test_array_value; ``` Output: ```plaintext +-------+--------+-------------+ | col_1 | arr1 | lag_result | +-------+--------+-------------+ | 1 | [1,11] | [101,111] | | 2 | [2,22] | [102,112] | | 3 | [3,33] | [1,11] | | 4 | NULL | [2,22] | | 5 | [5,55] | [3,33] | | 6 | [6,66] | NULL | +-------+--------+-------------+ ``` For the first two rows, no previous two rows exist, so the default value from `arr2` is returned. ##### LEAD()[​](#lead "Direct link to LEAD()") Returns the value of the row that leads the current row by `offset` rows. This function is often used to compare values between rows and filter data. Data types that can be queried by `LEAD()` are the same as those supported by [LAG()](#lag). **Syntax:** ```sql LEAD(expr [IGNORE NULLS] [, offset[, default]]) OVER([] []) ``` **Parameters:** * `expr`: the field you want to compute. * `offset`: the offset. It must be a positive integer. If this parameter is not specified, 1 is the default. * `default`: the default value returned if no matching row is found. If this parameter is not specified, NULL is the default. `default` supports any expression whose type is compatible with `expr`, since version 4.0, default no longer has to be a constant—it can be a column name. * `IGNORE NULLS` is supported from v3.0. It is used to determine whether NULL values of `expr` are included in the result. By default, NULL values are included when `offset` rows are counted, which means NULL is returned if the value of the destination row is NULL. See Example 1. If you specify IGNORE NULLS, NULL values are ignored when `offset` rows are counted and the system continues to search for `offset` non-null values. If `offset` non-null values cannot be found, NULL or `default` (if specified) is returned. See Example 2. **Examples** **Example 1: IGNORE NULLS is not specified** Create a table and insert values: ```sql CREATE TABLE test_tbl (col_1 INT, col_2 INT) DISTRIBUTED BY HASH(col_1); INSERT INTO test_tbl VALUES (1, NULL), (2, 4), (3, NULL), (4, 2), (5, NULL), (6, 7), (7, 6), (8, 5), (9, NULL), (10, NULL); ``` Query data from this table, where `offset` is 2, which means traversing the subsequent two rows; `default` is 0, which means 0 is returned if no matching rows are found. Output: ```plaintext SELECT col_1, col_2, LEAD(col_2,2,0) OVER (ORDER BY col_1) FROM test_tbl ORDER BY col_1; +-------+-------+----------------------------------------------+ | col_1 | col_2 | lead(col_2, 2, 0) OVER (ORDER BY col_1 ASC ) | +-------+-------+----------------------------------------------+ | 1 | NULL | NULL | | 2 | 4 | 2 | | 3 | NULL | NULL | | 4 | 2 | 7 | | 5 | NULL | 6 | | 6 | 7 | 5 | | 7 | 6 | NULL | | 8 | 5 | NULL | | 9 | NULL | 0 | | 10 | NULL | 0 | +-------+-------+----------------------------------------------+ ``` For the first row, the value two rows forward is NULL and NULL is returned because NULL values are allowed. For the last two rows, no subsequent two rows exist and the default value 0 is returned. **Example 2: IGNORE NULLS is specified** Use the preceding table and parameter settings. ```sql SELECT col_1, col_2, LEAD(col_2 IGNORE NULLS,2,0) OVER (ORDER BY col_1) FROM test_tbl ORDER BY col_1; +-------+-------+----------------------------------------------+ | col_1 | col_2 | lead(col_2, 2, 0) OVER (ORDER BY col_1 ASC ) | +-------+-------+----------------------------------------------+ | 1 | NULL | 2 | | 2 | 4 | 7 | | 3 | NULL | 7 | | 4 | 2 | 6 | | 5 | NULL | 6 | | 6 | 7 | 5 | | 7 | 6 | 0 | | 8 | 5 | 0 | | 9 | NULL | 0 | | 10 | NULL | 0 | +-------+-------+----------------------------------------------+ ``` For rows 7 to 10, the system cannot find two non-null values in the subsequent rows and the default value 0 is returned. For the first row, the value two rows forward is NULL and NULL is ignored because IGNORE NULLS is specified. The system continues to search for the second non-null value and 2 in row 4 is returned. **Example 3: Setting the default value in LEAD() to a column name** Use the preceding table and parameter settings. ```sql SELECT col_1, col_2, LEAD(col_2 ,2,col_1) OVER (ORDER BY col_1) FROM test_tbl ORDER BY col_1; +-------+-------+--------------------------------------------------+ | col_1 | col_2 | lead(col_2, 2, col_1) OVER (ORDER BY col_1 ASC ) | +-------+-------+--------------------------------------------------+ | 1 | NULL | NULL | | 2 | 4 | 2 | | 3 | NULL | NULL | | 4 | 2 | 7 | | 5 | NULL | 6 | | 6 | 7 | 5 | | 7 | 6 | NULL | | 8 | 5 | NULL | | 9 | NULL | 9 | | 10 | NULL | 10 | +-------+-------+--------------------------------------------------+ ``` As you can see, for rows 9 and 10 there are not two non-NULL values when scanning forward, so the default returned is the current row’s col\_1 value. All other rows behave the same as in Example 1. **Example 4: Using LEAD() with ARRAY types** Use the same table from LAG() Example 4: ```sql SELECT col_1, arr1, LEAD(arr1, 2, arr2) OVER (ORDER BY col_1) AS lead_result FROM test_array_value; ``` Output: ```plaintext +-------+--------+-------------+ | col_1 | arr1 | lead_result | +-------+--------+-------------+ | 1 | [1,11] | [3,33] | | 2 | [2,22] | NULL | | 3 | [3,33] | [5,55] | | 4 | NULL | [6,66] | | 5 | [5,55] | [105,115] | | 6 | [6,66] | [106,116] | +-------+--------+-------------+ ``` For the last two rows, no subsequent two rows exist, so the default value from `arr2` is returned. ##### MAX()[​](#max "Direct link to MAX()") Returns the maximum value of the specified rows in the current window. **Syntax:** ```sql MAX(expr) [OVER (analytic_clause)] ``` **Examples** Calculate the maximum value of rows from the first row to the row after the current row. This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```sql select *, max(scores) over ( partition by subject order by score rows between unbounded preceding and 1 following ) as max from scores where subject in ('math'); ``` ```plain +------+-------+---------+-------+------+ | id | name | subject | score | max | +------+-------+---------+-------+------+ | 1 | lily | math | NULL | 70 | | 5 | mike | math | 70 | 80 | | 2 | tom | math | 80 | 80 | | 4 | amy | math | 80 | 92 | | 6 | amber | math | 92 | 95 | | 3 | jack | math | 95 | 95 | +------+-------+---------+-------+------+ ``` The following example calculates the maximum score among all rows for the `math` subject. ```sql select *, max(score) over ( partition by subject order by score rows between unbounded preceding and unbounded following ) as max from scores where subject in ('math'); ``` From StarRocks 2.4 onwards, you can specify the row range as `rows between n preceding and n following`, which means you can capture `n` rows before the current row and `n` rows after the current row. Example statement: ```sql select *, max(score) over ( partition by subject order by score rows between 3 preceding and 2 following) as max from scores where subject in ('math'); ``` ##### MIN()[​](#min "Direct link to MIN()") Returns the minimum value of the specified rows in the current window. **Syntax:** ```sql MIN(expr) [OVER (analytic_clause)] ``` **Examples** Calculate the lowest score among all rows for the math subject. This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```sql select *, min(score) over ( partition by subject order by score rows between unbounded preceding and unbounded following) as min from scores where subject in ('math'); ``` ```plaintext +------+-------+---------+-------+------+ | id | name | subject | score | min | +------+-------+---------+-------+------+ | 1 | lily | math | NULL | 70 | | 5 | mike | math | 70 | 70 | | 2 | tom | math | 80 | 70 | | 4 | amy | math | 80 | 70 | | 6 | amber | math | 92 | 70 | | 3 | jack | math | 95 | 70 | +------+-------+---------+-------+------+ ``` From StarRocks 2.4 onwards, you can specify the row range as `rows between n preceding and n following`, which means you can capture `n` rows before the current row and `n` rows after the current row. Example statement: ```sql select *, min(score) over ( partition by subject order by score rows between 3 preceding and 2 following) as max from scores where subject in ('math'); ``` ##### NTILE()[​](#ntile "Direct link to NTILE()") NTILE() function divides the sorted rows in a partition by the specified number of `num_buckets` as equally as possible, stores the divided rows in the respective buckets, starting from 1 `[1, 2, ..., num_buckets]`, and returns the bucket number that each row is in. About the size of the bucket: * If the row counts can be divided by the specified number of `num_buckets` exactly, all the buckets will be of the same size. * If the row counts cannot be divided by the specified number of `num_buckets` exactly, there will be buckets of two different sizes. The difference between sizes is 1. The buckets with more rows will be listed ahead of the one with fewer rows. **Syntax:** ```sql NTILE (num_buckets) OVER (partition_by_clause order_by_clause) ``` `num_buckets`: Number of the buckets to be created. The value must be a constant positive integer whose maximum is `2^63 - 1`. Window clause is not allowed in NTILE() function. NTILE() function returns BIGINT type of data. **Examples** The following example divides all rows in the partition into two buckets. This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```sql select *, ntile(2) over ( partition by subject order by score ) as bucket_id from scores; ``` Output: ```plaintext +------+-------+---------+-------+-----------+ | id | name | subject | score | bucket_id | +------+-------+---------+-------+-----------+ | 6 | amber | NULL | 90 | 1 | | 1 | lily | math | NULL | 1 | | 5 | mike | math | 70 | 1 | | 2 | tom | math | 80 | 1 | | 4 | amy | math | 80 | 2 | | 6 | amber | math | 92 | 2 | | 3 | jack | math | 95 | 2 | | 3 | jack | english | NULL | 1 | | 5 | mike | english | 85 | 1 | | 4 | amy | english | 92 | 1 | | 2 | tom | english | 98 | 2 | | 1 | lily | english | 100 | 2 | | 2 | tom | physics | NULL | 1 | | 1 | lily | physics | 60 | 1 | | 5 | mike | physics | 85 | 1 | | 3 | jack | physics | 99 | 2 | | 4 | amy | physics | 99 | 2 | | 6 | amber | physics | 100 | 2 | +------+-------+---------+-------+-----------+ ``` As the above example shown, when `num_buckets` is `2`: * For the first row, this partition has only this record and it is assigned to only one bucket. * For rows 2 to 7, the partition has 6 records and the first 3 records are assigned to bucket 1 and other 3 records are assigned to bucket 2. ##### PERCENT\_RANK()[​](#percent_rank "Direct link to PERCENT_RANK()") Calculates the relative rank of a row within a result set as a percentage. PERCENT\_RANK() is calculated using the following formula, where `Rank` represents the rank of the current row in the partition. ```plaintext (Rank - 1)/(Rows in partition - 1) ``` The return values range from 0 to 1. This function is useful for percentile calculation and analyzing data distribution. It is supported from v3.2. **Syntax:** ```sql PERCENT_RANK() OVER (partition_by_clause order_by_clause) ``` note PERCENT\_RANK() must be used with ORDER BY to sort partition rows into the desired order. **Examples** The following example shows the relative rank of each `score` within the group of `math`. This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```sql SELECT *, PERCENT_RANK() OVER ( PARTITION BY subject ORDER BY score ) AS `percent_rank` FROM scores where subject in ('math'); ``` ```plaintext +------+-------+---------+-------+--------------+ | id | name | subject | score | percent_rank | +------+-------+---------+-------+--------------+ | 1 | lily | math | NULL | 0 | | 5 | mike | math | 70 | 0.2 | | 2 | tom | math | 80 | 0.4 | | 4 | amy | math | 80 | 0.4 | | 6 | amber | math | 92 | 0.8 | | 3 | jack | math | 95 | 1 | +------+-------+---------+-------+--------------+ ``` ##### RANK()[​](#rank "Direct link to RANK()") The RANK() function is used to represent rankings. Unlike DENSE\_RANK(), RANK() will **appear as a vacant** number. For example, if two tied 1s appear, the third number of RANK() will be 3 instead of 2. **Syntax:** ```sql RANK() OVER(partition_by_clause order_by_clause) ``` **Examples** Ranking math scores in the group. This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```sql select *, rank() over( partition by subject order by score desc ) as `rank` from scores where subject in ('math'); ``` ```plain +------+-------+---------+-------+------+ | id | name | subject | score | rank | +------+-------+---------+-------+------+ | 3 | jack | math | 95 | 1 | | 6 | amber | math | 92 | 2 | | 4 | amy | math | 80 | 3 | | 2 | tom | math | 80 | 3 | | 5 | mike | math | 70 | 5 | | 1 | lily | math | NULL | 6 | +------+-------+---------+-------+------+ ``` The result data has two rows whose score is 80. They all rank 3. The rank for the next score 70 is 5. ##### ROW\_NUMBER()[​](#row_number "Direct link to ROW_NUMBER()") Returns a continuously increasing integer starting from 1 for each row of a Partition. Unlike RANK() and DENSE\_RANK(), the value returned by ROW\_NUMBER() **does not repeat or have gaps** and is **continuously incremented**. **Syntax:** ```sql ROW_NUMBER() OVER(partition_by_clause order_by_clause) ``` **Examples** Rank math scores in the group. This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```sql select *, row_number() over( partition by subject order by score desc) as `rank` from scores where subject in ('math'); ``` ```plaintext +------+-------+---------+-------+------+ | id | name | subject | score | rank | +------+-------+---------+-------+------+ | 3 | jack | math | 95 | 1 | | 6 | amber | math | 92 | 2 | | 2 | tom | math | 80 | 3 | | 4 | amy | math | 80 | 4 | | 5 | mike | math | 70 | 5 | | 1 | lily | math | NULL | 6 | +------+-------+---------+-------+------+ ``` ##### QUALIFY()[​](#qualify "Direct link to QUALIFY()") The QUALIFY clause filters the results of window functions. In a SELECT statement, you can use the QUALIFY clause to apply conditions to a column to filter results. QUALIFY is analogous to the HAVING clause in aggregate functions. This function is supported from v2.5. QUALIFY simplifies the writing of SELECT statements. Before QUALIFY is used, a SELECT statement may go like this: ```sql SELECT * FROM (SELECT DATE, PROVINCE_CODE, TOTAL_SCORE, ROW_NUMBER() OVER(PARTITION BY PROVINCE_CODE ORDER BY TOTAL_SCORE) AS SCORE_ROWNUMBER FROM example_table) T1 WHERE T1.SCORE_ROWNUMBER = 1; ``` After QUALIFY is used, the statement is shortened to: ```sql SELECT DATE, PROVINCE_CODE, TOTAL_SCORE FROM example_table QUALIFY ROW_NUMBER() OVER(PARTITION BY PROVINCE_CODE ORDER BY TOTAL_SCORE) = 1; ``` QUALIFY supports only the following three window functions: ROW\_NUMBER(), RANK(), and DENSE\_RANK(). **Syntax:** ```sql SELECT FROM [GROUP BY ...] [HAVING ...] QUALIFY [ ... ] ``` **Parameters:** ``: columns from which you want to obtain data. ``: The data source is generally a table. ``: The `QUALIFY` clause can only be followed by a window function, including ROW\_NUMBER(), RANK(), and DENSE\_RANK(). **Examples** ```sql -- Create a table. CREATE TABLE sales_record ( city_id INT, item STRING, sales INT ) DISTRIBUTED BY HASH(`city_id`); -- Insert data into the table. insert into sales_record values (1,'fruit',95), (2,'drinks',70), (3,'fruit',87), (4,'drinks',98); -- Query data from the table. select * from sales_record order by city_id; +---------+--------+-------+ | city_id | item | sales | +---------+--------+-------+ | 1 | fruit | 95 | | 2 | drinks | 70 | | 3 | fruit | 87 | | 4 | drinks | 98 | +---------+--------+-------+ ``` **Example 1: Obtain records whose row number is greater than 1 from the table** ```sql SELECT city_id, item, sales FROM sales_record QUALIFY row_number() OVER (ORDER BY city_id) > 1; +---------+--------+-------+ | city_id | item | sales | +---------+--------+-------+ | 2 | drinks | 70 | | 3 | fruit | 87 | | 4 | drinks | 98 | +---------+--------+-------+ ``` **Example 2: Obtain records whose row number is 1 from each partition of the table** The table is divided into two partitions by `item` and the first row in each partition is returned. ```sql SELECT city_id, item, sales FROM sales_record QUALIFY ROW_NUMBER() OVER (PARTITION BY item ORDER BY city_id) = 1 ORDER BY city_id; +---------+--------+-------+ | city_id | item | sales | +---------+--------+-------+ | 1 | fruit | 95 | | 2 | drinks | 70 | +---------+--------+-------+ 2 rows in set (0.01 sec) ``` Example 3: Obtain records whose sales rank No.1 from each partition of the table. The table is divided into two partitions by `item` and the row with the highest sales in each partition is returned. ```sql SELECT city_id, item, sales FROM sales_record QUALIFY RANK() OVER (PARTITION BY item ORDER BY sales DESC) = 1 ORDER BY city_id; +---------+--------+-------+ | city_id | item | sales | +---------+--------+-------+ | 1 | fruit | 95 | | 4 | drinks | 98 | +---------+--------+-------+ ``` **Usage notes:** * QUALIFY supports only the following three window functions: ROW\_NUMBER(), RANK(), and DENSE\_RANK(). * The execution order of clauses in a query with QUALIFY is evaluated in the following order: 1. FROM 2. WHERE 3. GROUP BY 4. HAVING 5. Window 6. QUALIFY 7. DISTINCT 8. ORDER BY 9. LIMIT ##### SUM()[​](#sum "Direct link to SUM()") Calculates the sum of specified rows. **Syntax:** ```sql SUM([DISTINCT] expr) [OVER (analytic_clause)] ``` `DISTINCT` is supported from StarRocks v4.0. When specified, SUM() sums only distinct values in the window. note **Window frame limitation:** When using SUM(DISTINCT) as a window function, only RANGE frames are supported. ROWS frames are NOT supported. **Examples** These examples use the data in the [Sample table](#window-function-sample-table) `scores`. **Example 1: Basic usage** Group data by `subject` and calculate the sum of scores of all rows within the group. ```sql select *, sum(score) over ( partition by subject order by score rows between unbounded preceding and unbounded following ) as 'sum' from scores; ``` ```plaintext +------+-------+---------+-------+------+ | id | name | subject | score | sum | +------+-------+---------+-------+------+ | 6 | amber | NULL | 90 | 90 | | 1 | lily | math | NULL | 417 | | 5 | mike | math | 70 | 417 | | 2 | tom | math | 80 | 417 | | 4 | amy | math | 80 | 417 | | 6 | amber | math | 92 | 417 | | 3 | jack | math | 95 | 417 | | 3 | jack | english | NULL | 375 | | 5 | mike | english | 85 | 375 | | 4 | amy | english | 92 | 375 | | 2 | tom | english | 98 | 375 | | 1 | lily | english | 100 | 375 | | 2 | tom | physics | NULL | 443 | | 1 | lily | physics | 60 | 443 | | 5 | mike | physics | 85 | 443 | | 3 | jack | physics | 99 | 443 | | 4 | amy | physics | 99 | 443 | | 6 | amber | physics | 100 | 443 | +------+-------+---------+-------+------+ ``` **Example 2: Using SUM(DISTINCT) over overall window** Sum distinct scores across all rows: ```sql SELECT id, subject, score, SUM(DISTINCT score) OVER () AS distinct_sum FROM test_scores; ``` Output: ```plaintext +----+---------+-------+-------------+ | id | subject | score | distinct_sum| +----+---------+-------+-------------+ | 1 | math | 80 | 255| | 2 | math | 85 | 255| | 3 | math | 80 | 255| | 4 | english | 90 | 255| | 5 | english | 85 | 255| | 6 | english | 90 | 255| +----+---------+-------+-------------+ ``` The distinct sum is 255 (80 + 85 + 90). **Example 3: Using SUM(DISTINCT) over framed window with RANGE frame** Sum distinct scores within each subject partition using a RANGE frame: ```sql SELECT id, subject, score, SUM(DISTINCT score) OVER ( PARTITION BY subject ORDER BY score RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS distinct_sum FROM test_scores; ``` Output: ```plaintext +----+---------+-------+-------------+ | id | subject | score | distinct_sum| +----+---------+-------+-------------+ | 1 | math | 80 | 80| | 3 | math | 80 | 80| | 2 | math | 85 | 165| | 5 | english | 85 | 85| | 4 | english | 90 | 175| | 6 | english | 90 | 175| +----+---------+-------+-------------+ ``` For each row, the function sums distinct scores from the beginning of the partition up to and including the current row's score value. ##### VARIANCE, VAR\_POP, VARIANCE\_POP[​](#variance-var_pop-variance_pop "Direct link to VARIANCE, VAR_POP, VARIANCE_POP") Returns the population variance of an expression. VAR\_POP and VARIANCE\_POP are aliases of VARIANCE. These functions can be used as window functions since v2.5.10. **Syntax:** ```sql VARIANCE(expr) OVER([partition_by_clause] [order_by_clause] [order_by_clause window_clause]) ``` tip From 2.5.13, 3.0.7, 3.1.4 onwards, this window function supports the ORDER BY and Window clauses. **Parameters:** If `expr` is a table column, it must evaluate to TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, or DECIMAL. **Examples** This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```plaintext select *, variance(score) over ( partition by subject order by score ) as 'variance' from scores where subject in ('math'); +------+-------+---------+-------+--------------------+ | id | name | subject | score | variance | +------+-------+---------+-------+--------------------+ | 1 | lily | math | NULL | NULL | | 5 | mike | math | 70 | 0 | | 2 | tom | math | 80 | 22.222222222222225 | | 4 | amy | math | 80 | 22.222222222222225 | | 6 | amber | math | 92 | 60.74999999999997 | | 3 | jack | math | 95 | 82.23999999999998 | +------+-------+---------+-------+--------------------+ ``` ##### VAR\_SAMP, VARIANCE\_SAMP[​](#var_samp-variance_samp "Direct link to VAR_SAMP, VARIANCE_SAMP") Returns the sample variance of an expression. These functions can be used as window functions since v2.5.10. **Syntax:** ```sql VAR_SAMP(expr) OVER([partition_by_clause] [order_by_clause] [order_by_clause window_clause]) ``` tip From 2.5.13, 3.0.7, 3.1.4 onwards, this window function supports the ORDER BY and Window clauses. **Parameters:** If `expr` is a table column, it must evaluate to TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, or DECIMAL. **Examples** This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```plaintext select *, VAR_SAMP(score) over (partition by subject order by score) as VAR_SAMP from scores where subject in ('math'); +------+-------+---------+-------+--------------------+ | id | name | subject | score | VAR_SAMP | +------+-------+---------+-------+--------------------+ | 1 | lily | math | NULL | NULL | | 5 | mike | math | 70 | 0 | | 2 | tom | math | 80 | 33.333333333333336 | | 4 | amy | math | 80 | 33.333333333333336 | | 6 | amber | math | 92 | 80.99999999999996 | | 3 | jack | math | 95 | 102.79999999999997 | +------+-------+---------+-------+--------------------+ ``` ##### STD, STDDEV, STDDEV\_POP[​](#std-stddev-stddev_pop "Direct link to STD, STDDEV, STDDEV_POP") Returns the standard deviation of an expression. These functions can be used as window functions since v2.5.10. **Syntax:** ```sql STD(expr) OVER([partition_by_clause] [order_by_clause] [order_by_clause window_clause]) ``` tip From 2.5.13, 3.0.7, 3.1.4 onwards, this window function supports the ORDER BY and Window clauses. **Parameters:** If `expr` is a table column, it must evaluate to TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, or DECIMAL. **Examples** This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```plaintext select *, STD(score) over ( partition by subject order by score) as std from scores where subject in ('math'); +------+-------+---------+-------+-------------------+ | id | name | subject | score | std | +------+-------+---------+-------+-------------------+ | 1 | lily | math | NULL | NULL | | 5 | mike | math | 70 | 0 | | 4 | amy | math | 80 | 4.714045207910317 | | 2 | tom | math | 80 | 4.714045207910317 | | 6 | amber | math | 92 | 7.794228634059946 | | 3 | jack | math | 95 | 9.068627239003707 | +------+-------+---------+-------+-------------------+ ``` ##### STDDEV\_SAMP[​](#stddev_samp "Direct link to STDDEV_SAMP") Returns the sample standard deviation of an expression. This function can be used as a window function since v2.5.10. **Syntax:** ```sql STDDEV_SAMP(expr) OVER([partition_by_clause] [order_by_clause] [order_by_clause window_clause]) ``` tip From 2.5.13, 3.0.7, 3.1.4 onwards, this window function supports the ORDER BY and Window clauses. **Parameters:** If `expr` is a table column, it must evaluate to TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, or DECIMAL. **Examples** This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```plaintext select *, STDDEV_SAMP(score) over ( partition by subject order by score ) as STDDEV_SAMP from scores where subject in ('math'); +------+-------+---------+-------+--------------------+ | id | name | subject | score | STDDEV_SAMP | +------+-------+---------+-------+--------------------+ | 1 | lily | math | NULL | NULL | | 5 | mike | math | 70 | 0 | | 2 | tom | math | 80 | 5.773502691896258 | | 4 | amy | math | 80 | 5.773502691896258 | | 6 | amber | math | 92 | 8.999999999999998 | | 3 | jack | math | 95 | 10.139033484509259 | +------+-------+---------+-------+--------------------+ select *, STDDEV_SAMP(score) over ( partition by subject order by score rows between unbounded preceding and 1 following) as STDDEV_SAMP from scores where subject in ('math'); +------+-------+---------+-------+--------------------+ | id | name | subject | score | STDDEV_SAMP | +------+-------+---------+-------+--------------------+ | 1 | lily | math | NULL | 0 | | 5 | mike | math | 70 | 7.0710678118654755 | | 2 | tom | math | 80 | 5.773502691896258 | | 4 | amy | math | 80 | 8.999999999999998 | | 6 | amber | math | 92 | 10.139033484509259 | | 3 | jack | math | 95 | 10.139033484509259 | +------+-------+---------+-------+--------------------+ ``` ##### COVAR\_SAMP[​](#covar_samp "Direct link to COVAR_SAMP") Returns the sample covariance of two expressions. This function is supported from v2.5.10. It is also an aggregate function. **Syntax:** ```sql COVAR_SAMP(expr1,expr2) OVER([partition_by_clause] [order_by_clause] [order_by_clause window_clause]) ``` tip From 2.5.13, 3.0.7, 3.1.4 onwards, this window function supports the ORDER BY and Window clauses. **Parameters:** If `expr` is a table column, it must evaluate to TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, or DECIMAL. **Examples** This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```plaintext select *, COVAR_SAMP(id, score) over ( partition by subject order by score) as covar_samp from scores where subject in ('math'); +------+-------+---------+-------+----------------------+ | id | name | subject | score | covar_samp | +------+-------+---------+-------+----------------------+ | 1 | lily | math | NULL | NULL | | 5 | mike | math | 70 | 0 | | 2 | tom | math | 80 | -6.666666666666668 | | 4 | amy | math | 80 | -6.666666666666668 | | 6 | amber | math | 92 | 4.5 | | 3 | jack | math | 95 | -0.24999999999999822 | +------+-------+---------+-------+----------------------+ select *, COVAR_SAMP(id,score) over ( partition by subject order by score rows between unbounded preceding and 1 following) as COVAR_SAMP from scores where subject in ('math'); +------+-------+---------+-------+----------------------+ | id | name | subject | score | COVAR_SAMP | +------+-------+---------+-------+----------------------+ | 1 | lily | math | NULL | 0 | | 5 | mike | math | 70 | -5 | | 4 | amy | math | 80 | -6.666666666666661 | | 2 | tom | math | 80 | 4.500000000000004 | | 6 | amber | math | 92 | -0.24999999999999467 | | 3 | jack | math | 95 | -0.24999999999999467 | ``` ##### COVAR\_POP[​](#covar_pop "Direct link to COVAR_POP") Returns the population covariance of two expressions. This function is supported from v2.5.10. It is also an aggregate function. **Syntax:** ```sql COVAR_POP(expr1, expr2) OVER([partition_by_clause] [order_by_clause] [order_by_clause window_clause]) ``` tip From 2.5.13, 3.0.7, 3.1.4 onwards, this window function supports the ORDER BY and Window clauses. **Parameters:** If `expr` is a table column, it must evaluate to TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, or DECIMAL. **Examples** This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```plaintext select *, COVAR_POP(id, score) over ( partition by subject order by score) as covar_pop from scores where subject in ('math'); +------+-------+---------+-------+----------------------+ | id | name | subject | score | covar_pop | +------+-------+---------+-------+----------------------+ | 1 | lily | math | NULL | NULL | | 5 | mike | math | 70 | 0 | | 2 | tom | math | 80 | -4.4444444444444455 | | 4 | amy | math | 80 | -4.4444444444444455 | | 6 | amber | math | 92 | 3.375 | | 3 | jack | math | 95 | -0.19999999999999857 | +------+-------+---------+-------+----------------------+ ``` ##### CORR[​](#corr "Direct link to CORR") Returns the Pearson correlation coefficient between two expressions. This function is supported from v2.5.10. It is also an aggregate function. **Syntax:** ```sql CORR(expr1, expr2) OVER([partition_by_clause] [order_by_clause] [order_by_clause window_clause]) ``` tip From 2.5.13, 3.0.7, 3.1.4 onwards, this window function supports the ORDER BY and Window clauses. **Parameters:** If `expr` is a table column, it must evaluate to TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, or DECIMAL. **Examples** This example uses the data in the [Sample table](#window-function-sample-table) `scores`. ```plaintext select *, CORR(id, score) over ( partition by subject order by score) as corr from scores where subject in ('math'); +------+-------+---------+-------+-----------------------+ | id | name | subject | score | corr | +------+-------+---------+-------+-----------------------+ | 5 | mike | math | 70 | -0.015594571538795355 | | 1 | lily | math | NULL | -0.015594571538795355 | | 2 | tom | math | 80 | -0.015594571538795355 | | 4 | amy | math | 80 | -0.015594571538795355 | | 3 | jack | math | 95 | -0.015594571538795355 | | 6 | amber | math | 92 | -0.015594571538795355 | +------+-------+---------+-------+-----------------------+ select *, CORR(id,score) over ( partition by subject order by score rows between unbounded preceding and 1 following) as corr from scores where subject in ('math'); +------+-------+---------+-------+-------------------------+ | id | name | subject | score | corr | +------+-------+---------+-------+-------------------------+ | 1 | lily | math | NULL | 1.7976931348623157e+308 | | 5 | mike | math | 70 | -1 | | 2 | tom | math | 80 | -0.7559289460184546 | | 4 | amy | math | 80 | 0.29277002188455997 | | 6 | amber | math | 92 | -0.015594571538795024 | | 3 | jack | math | 95 | -0.015594571538795024 | +------+-------+---------+-------+-------------------------+ ``` --- ### SQL statements #### [🗃️ User account management](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management.md) [16 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management.md) #### [🗃️ Cluster management](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management.md) [9 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management.md) #### [🗃️ Loading and unloading](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading.md) [12 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading.md) #### [🗃️ Catalog](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog.md) [5 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog.md) #### [🗃️ Database](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database.md) [7 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database.md) #### [🗃️ Resource](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource.md) [4 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource.md) #### [🗃️ Table, bucket, partition, and index](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index.md) [24 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index.md) #### [🗃️ View](https://docs.starrocks.io/docs/sql-reference/sql-statements/View.md) [4 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/View.md) #### [🗃️ Materialized View](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view.md) [8 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view.md) #### [🗃️ Function](https://docs.starrocks.io/docs/sql-reference/sql-statements/Function.md) [4 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/Function.md) #### [🗃️ CBO statistics](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats.md) [8 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats.md) #### [🗃️ Backup and restore](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore.md) [11 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore.md) #### [📄️ Translate Trino SQL](https://docs.starrocks.io/docs/sql-reference/sql-statements/TRANSLATE_TRINO.md) [From v3.3.9, StarRocks supports translating Trino SQL statements into StarRocks SQL statements.](https://docs.starrocks.io/docs/sql-reference/sql-statements/TRANSLATE_TRINO.md) #### [🗃️ dictionary](https://docs.starrocks.io/docs/sql-reference/sql-statements/dictionary/CANCEL_REFRESH_DICTIONARY.md) [5 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/dictionary/CANCEL_REFRESH_DICTIONARY.md) #### [📄️ Generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) [Since v3.1, StarRocks supports generated columns.](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) #### [📄️ Keywords](https://docs.starrocks.io/docs/sql-reference/sql-statements/keywords.md) [Reference list of reserved and non-reserved keywords in StarRocks SQL.](https://docs.starrocks.io/docs/sql-reference/sql-statements/keywords.md) #### [📄️ Prepared statements](https://docs.starrocks.io/docs/sql-reference/sql-statements/prepared_statement.md) [From v3.2 onwards, StarRocks provides prepared statements for executing SQL statements multiple times with the same structure but different variables.](https://docs.starrocks.io/docs/sql-reference/sql-statements/prepared_statement.md) --- ### User account management #### [📄️ ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md) [ALTER USER modifies user information, including password, authentication method, default roles, and user properties (supported from v3.3.3 onwards).](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md) #### [📄️ CREATE ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_ROLE.md) [CREATE ROLE creates a role.](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_ROLE.md) #### [📄️ CREATE USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_USER.md) [CREATE USER creates a StarRocks user.](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_USER.md) #### [📄️ DROP ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/DROP_ROLE.md) [DROP ROLE drops a role.](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/DROP_ROLE.md) #### [📄️ DROP USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/DROP_USER.md) [DROP USER drops a specified user identity.](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/DROP_USER.md) #### [📄️ EXECUTE AS](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/EXECUTE_AS.md) [Use the IMPERSONATE privilege with EXECUTE AS statements to switch the execution context of the current session to the impersonated user.](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/EXECUTE_AS.md) #### [📄️ GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) [Grants roles or privileges on specific objects to a user, a role, or an external group.](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) #### [📄️ REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md) [Use REVOKE to remove specific privileges or roles from a user or a role.](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md) #### [📄️ SET DEFAULT ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_DEFAULT_ROLE.md) [SET DEFAULT ROLE sets the roles that are activated by default when the user connects to the server.](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_DEFAULT_ROLE.md) #### [📄️ SET PASSWORD](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_PASSWORD.md) [SET PASSWORD changes the login password for users.](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_PASSWORD.md) #### [📄️ SET ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_ROLE.md) [SET ROLE activates roles, along with all of its associated privileges and nested roles, for the current session.](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_ROLE.md) #### [📄️ SHOW AUTHENTICATION](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_AUTHENTICATION.md) [SHOW AUTHENTICATION displays the authentication information of the current user or all users in the current cluster.](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_AUTHENTICATION.md) #### [📄️ SHOW GRANTS](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_GRANTS.md) [SHOW GRANTS displays all the privileges that have been granted to a user or role.](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_GRANTS.md) #### [📄️ SHOW PROPERTY](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_PROPERTY.md) [SHOW PROPERTY displays properties of a user, including the maximum number of connections, the default catalog, and the default database.](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_PROPERTY.md) #### [📄️ SHOW ROLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_ROLES.md) [SHOW ROLES displays all roles in the system.](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_ROLES.md) #### [📄️ SHOW USERS](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_USERS.md) [SHOW USERS displays all users in the system.](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_USERS.md) --- ### ALTER USER ALTER USER modifies user information, including password, authentication method, default roles, and user properties (supported from v3.3.3 onwards). tip Common users can use this command to modify information for themselves. Only users with the `user_admin` role can modify information for other users. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ALTER USER user_identity [auth_option] [default_role] [DEFAULT ROLE [, , ...]] [SET PROPERTIES ("key"="value", ...)] ``` #### Parameters[​](#parameters "Direct link to Parameters") * `user_identity` consists of two parts, "user\_name" and "host", in the format of `username@'userhost'`. For the "host" part, you can use `%` for fuzzy match. If "host" is not specified, "%" is used by default, meaning that the user can connect to StarRocks from any host. However, when you use this statement with `SET PROPERTIES` to modify the user properties, you must specify the `username` instead of `user_identity`. * `auth_option` specifies the authentication method. Currently, three authentication methods are supported: StarRocks native password, mysql\_native\_password, and "authentication\_ldap\_simple". StarRocks native password is the same as mysql\_native\_password in logic but slightly differs in syntax. One user identity can use only one authentication method. You can use ALTER USER to modify users' passwords and authentication methods. ```sql auth_option: { IDENTIFIED BY 'auth_string' IDENTIFIED WITH mysql_native_password BY 'auth_string' IDENTIFIED WITH mysql_native_password AS 'auth_string' IDENTIFIED WITH authentication_ldap_simple AS 'auth_string' } ``` | **Authentication method** | **Password for user creation** | **Password for login** | | ---------------------------- | ------------------------------ | ---------------------- | | Native password | Plaintext or ciphertext | Plaintext | | `mysql_native_password BY` | Plaintext | Plaintext | | `mysql_native_password WITH` | Ciphertext | Plaintext | | `authentication_ldap_simple` | Plaintext | Plaintext | > Note: StarRocks encrypts users' passwords before storing them. * `DEFAULT ROLE` sets the default role for the user. ```sql -- Set specified roles as default roles. DEFAULT ROLE [, , ...] -- Set all roles of the user, including roles that will be assigned to this user, as default roles. DEFAULT ROLE ALL -- No default role is set but the public role is still enabled after a user login. DEFAULT ROLE NONE ``` Before you run ALTER USER to set default roles, make sure that all the roles have been assigned to users. The roles are automatically activated after the user logs in again. * `SET PROPERTIES` sets user properties, including the maximum user connection number (`max_user_connections`), catalog, database or session variables on the user level. User-level session variables take effect as the user logs in. This feature is supported from v3.3.3. ```sql -- Set the maximum user connection number. SET PROPERTIES ("max_user_connections" = "") -- Set the catalog. SET PROPERTIES ("catalog" = "") -- Set the database. SET PROPERTIES ("catalog" = "", "database" = "") -- Set session variables. SET PROPERTIES ("session." = "", ...) -- Clear the properties set for the user. SET PROPERTIES ("catalog" = "", "database" = "", "session." = ""); ``` tip * `SET PROPERTIES` works on user instead of user identity. Therefore, when modifying the user properties, you must specify the `username` instead of `user_identity` in the `ALTER USER` statement. * Global variables and read-only variables cannot be set for a specific user. * Variables take effect in the following order: SET\_VAR > Session > User property > Global. * You can use [SHOW PROPERTY](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_PROPERTY.md) to view the properties of a specific user. #### Examples[​](#examples "Direct link to Examples") Example 1: Change user's password to a plaintext password. ```sql ALTER USER 'jack' IDENTIFIED BY '123456'; ALTER USER jack@'172.10.1.10' IDENTIFIED WITH mysql_native_password BY '123456'; ``` Example 2: Change user's password to a ciphertext password. ```sql ALTER USER jack@'172.10.1.10' IDENTIFIED BY PASSWORD '*6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9'; ALTER USER jack@'172.10.1.10' IDENTIFIED WITH mysql_native_password AS '*6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9'; ``` > You can get the encrypted password using the password() function. Example 3: Change the authentication method to LDAP. ```sql ALTER USER jack@'172.10.1.10' IDENTIFIED WITH authentication_ldap_simple; ``` Example 4: Change the authentication method to LDAP and specify the distinguished name (DN) of the user in LDAP. ```sql ALTER USER jack@'172.10.1.10' IDENTIFIED WITH authentication_ldap_simple AS 'uid=jack,ou=company,dc=example,dc=com'; ``` Example 5: Change the default roles of the user to `db_admin` and `user_admin`. Note that the user must have been assigned these two roles. ```sql ALTER USER 'jack'@'192.168.%' DEFAULT ROLE db_admin, user_admin; ``` Example 6: Set all roles of the user, including roles that will be assigned to this user as default roles. ```sql ALTER USER 'jack'@'192.168.%' DEFAULT ROLE ALL; ``` Example 7: Clear all the default roles of the user. ```sql ALTER USER 'jack'@'192.168.%' DEFAULT ROLE NONE; ``` > Note: By default, the `public` role is still activated for the user. Example 8: Set the maximum user connection number to `600`. ```sql ALTER USER 'jack' SET PROPERTIES ("max_user_connections" = "600"); ``` Example 9: Set the catalog of the user to `hive_catalog`. ```sql ALTER USER 'jack' SET PROPERTIES ('catalog' = 'hive_catalog'); ``` Example 10: Set the database of the user to `test_db` in the default catalog. ```sql ALTER USER 'jack' SET PROPERTIES ('catalog' = 'default_catalog', 'database' = 'test_db'); ``` Example 11: Set the session variable `query_timeout` to `600` for the user. ```sql ALTER USER 'jack' SET PROPERTIES ('session.query_timeout' = '600'); ``` Example 12: Clear the properties set for the user. ```sql ALTER USER 'jack' SET PROPERTIES ('catalog' = '', 'database' = '', 'session.query_timeout' = ''); ``` #### References[​](#references "Direct link to References") * [CREATE USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_USER.md) * [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) * [SHOW USERS](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_USERS.md) * [DROP USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/DROP_USER.md) --- ### CREATE ROLE CREATE ROLE creates a role. After a role is created, you can grant privileges to the role and then assign this role to a user or another role. This way, the privileges associated with this role are passed on to users or roles. tip Only users with the `user_admin` role has the privilege to perform this operation. #### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE ROLE ``` #### Parameters[​](#parameters "Direct link to Parameters") `role_name`: the name of the role. For the naming conventions, see [System limits](https://docs.starrocks.io/docs/sql-reference/System_limit.md). Note that the created role name cannot be the same as [system-defined roles](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md#system-defined-roles): `root`, `cluster_admin`, `db_admin`, `user_admin`, and `public`. #### Limits[​](#limits "Direct link to Limits") By default, a user can have a maximum of 64 roles. You can adjust this setting by using the FE dynamic parameter `privilege_max_total_roles_per_user`. A role can have a maximum of 16 inheritance levels. You can adjust this setting by using the FE dynamic parameter `privilege_max_role_depth`. #### Examples[​](#examples "Direct link to Examples") Create a role. ```sql CREATE ROLE role1; ``` #### References[​](#references "Direct link to References") * [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) * [SHOW ROLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_ROLES.md) * [DROP ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/DROP_ROLE.md) --- ### CREATE USER CREATE USER creates a StarRocks user. In StarRocks, a "user\_identity" uniquely identifies a user. From v3.3.3, StarRocks supports setting user properties when creating a user. tip Only users with the `user_admin` role has the privilege to perform this operation. ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE USER [IF NOT EXISTS] [auth_option] [DEFAULT ROLE [, , ...]] [PROPERTIES ("key"="value", ...)] ``` #### Parameters[​](#parameters "Direct link to Parameters") * `user_identity` consists of two parts, "user\_name" and "host", in the format of `username@'userhost'`. For the "host" part, you can use `%` for fuzzy match. If "host" is not specified, "%" is used by default, meaning that the user can connect to StarRocks from any host. For the naming conventions of usernames, see [System limits](https://docs.starrocks.io/docs/sql-reference/System_limit.md). - `auth_option` specifies the authentication method. Currently, five authentication methods are supported: StarRocks native password, `mysql_native_password`, `authentication_ldap_simple`, JSON Web Token (JWT) Authentication, and OAuth 2.0 Authentication. StarRocks native password is the same as `mysql_native_password` in logic but slightly differs in syntax. One user identity can use only one authentication method. ```sql auth_option: { IDENTIFIED BY 'auth_string' IDENTIFIED WITH mysql_native_password BY 'auth_string' IDENTIFIED WITH mysql_native_password AS 'auth_string' IDENTIFIED WITH authentication_ldap_simple AS 'auth_string' IDENTIFIED WITH authentication_jwt [AS 'auth_properties'] IDENTIFIED WITH authentication_oauth2 [AS 'auth_properties'] } ``` | **Authentication method** | **Password for user creation** | **Password for login** | | ---------------------------- | ------------------------------ | ---------------------- | | Native password | Plaintext or ciphertext | Plaintext | | `mysql_native_password BY` | Plaintext | Plaintext | | `mysql_native_password WITH` | Ciphertext | Plaintext | | `authentication_ldap_simple` | Plaintext | Plaintext | note StarRocks encrypts users' passwords before storing them. For details of `auth_properties` for JSON Web Token (JWT) Authentication and OAuth 2.0 Authentication, see the corresponding documents: * [JSON Web Token Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/jwt_authentication.md) * [OAuth 2.0 Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/oauth2_authentication.md) * `DEFAULT ROLE [, , ...]`: If this parameter is specified, the roles are automatically assigned to the user and activated by default when the user logs in. If not specified, this user does not have any privileges. Make sure that all the roles that are specified already exist. * `PROPERTIES` sets user properties, including the maximum user connection number (`max_user_connections`), catalog, database or session variables on the user level. User-level session variables take effect as the user logs in. This feature is supported from v3.3.3. ```sql -- Set the maximum user connection number. PROPERTIES ("max_user_connections" = "") -- Set the catalog. PROPERTIES ("catalog" = "") -- Set the database. PROPERTIES ("catalog" = "", "database" = "") -- Set session variables. PROPERTIES ("session." = "", ...) ``` tip * `PROPERTIES` works on user instead of user identity. * Global variables and read-only variables cannot be set for a specific user. * Variables take effect in the following order: SET\_VAR > Session > User property > Global. * You can use [SHOW PROPERTY](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_PROPERTY.md) to view the properties of a specific user. #### Examples[​](#examples "Direct link to Examples") Example 1: Create a user using a plaintext password with no host specified, which is equivalent to `jack@'%'`. ```sql CREATE USER 'jack' IDENTIFIED BY '123456'; ``` Example 2: Create a user using a plaintext password and allow the user to log in from `'172.10.1.10'`. ```sql CREATE USER jack@'172.10.1.10' IDENTIFIED WITH mysql_native_password BY '123456'; ``` Example 3: Create a user with a ciphertext password and allow the user to log in from `'172.10.1.10'`. ```sql CREATE USER jack@'172.10.1.10' IDENTIFIED BY PASSWORD '*6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9'; CREATE USER jack@'172.10.1.10' IDENTIFIED WITH mysql_native_password AS '*6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9'; ``` > Note: You can get the encrypted password using the password() function. Example 4: Create a user who is allowed to log in from a domain name 'example\_domain'. ```sql CREATE USER 'jack'@['example_domain'] IDENTIFIED BY '123456'; ``` Example 5: Create a user that uses LDAP authentication. ```sql CREATE USER jack@'172.10.1.10' IDENTIFIED WITH authentication_ldap_simple; ``` Example 6: Create a user that uses LDAP authentication and specify the distinguished name (DN) of the user in LDAP. ```sql CREATE USER jack@'172.10.1.10' IDENTIFIED WITH authentication_ldap_simple AS 'uid=jack,ou=company,dc=example,dc=com'; ``` Example 7: Create a user who is allowed to log in from the '192.168' subnet and set `db_admin` and `user_admin` as the default roles for the user. ```sql CREATE USER 'jack'@'192.168.%' DEFAULT ROLE db_admin, user_admin; ``` Example 8: Create a user and set its maximum user connection number to `600`. ```sql CREATE USER 'jack'@'192.168.%' PROPERTIES ("max_user_connections" = "600"); ``` Example 9: Create a user and set the catalog of the user to `hive_catalog`. ```sql CREATE USER 'jack'@'192.168.%' PROPERTIES ('catalog' = 'hive_catalog'); ``` Example 10: Create a user and set the database of the user to `test_db` in the default catalog. ```sql CREATE USER 'jack'@'192.168.%' PROPERTIES ('catalog' = 'default_catalog', 'database' = 'test_db'); ``` Example 11: Create a user and set the session variable `query_timeout` to `600` for the user. ```sql CREATE USER 'jack'@'192.168.%' PROPERTIES ('session.query_timeout' = '600'); ``` Example 12: Create a user with JSON Web Token Authentication. ```sql CREATE USER tom IDENTIFIED WITH authentication_jwt AS '{ "jwks_url": "http://localhost:38080/realms/master/protocol/jwt/certs", "principal_field": "preferred_username", "required_issuer": "http://localhost:38080/realms/master", "required_audience": "starrocks" }'; ``` Example 13: Create a user with OAuth 2.0 Authentication. ```sql CREATE USER tom IDENTIFIED WITH authentication_oauth2 AS '{ "auth_server_url": "http://localhost:38080/realms/master/protocol/openid-connect/auth", "token_server_url": "http://localhost:38080/realms/master/protocol/openid-connect/token", "client_id": "12345", "client_secret": "LsWyD9vPcM3LHxLZfzJsuoBwWQFBLcoR", "redirect_url": "http://localhost:8030/api/oauth2", "jwks_url": "http://localhost:38080/realms/master/protocol/openid-connect/certs", "principal_field": "preferred_username", "required_issuer": "http://localhost:38080/realms/master", "required_audience": "12345" }'; ``` --- ### DROP ROLE DROP ROLE drops a role. If a role has been granted to a user, the user will lose the privileges associated with this role after the role is dropped. tip * Only users with the `user_admin` role can drop a role. * [StarRocks system-defined roles](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md#system-defined-roles) cannot be dropped. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DROP ROLE ``` #### Examples[​](#examples "Direct link to Examples") Drop a role. ```sql DROP ROLE role1; ``` --- ### DROP USER DROP USER drops a specified user identity. tip Only users with the `user_admin` role has the privilege to perform this operation. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DROP USER '' `user_identity`: user@'host' user@['domain'] ``` #### Examples[​](#examples "Direct link to Examples") Drop user `jack@'192.%'`. ```sql DROP USER 'jack'@'192.%' ``` --- ### EXECUTE AS Use the IMPERSONATE privilege with EXECUTE AS statements to switch the execution context of the current session to the impersonated user. This command is supported from v2.4. #### Syntax[​](#syntax "Direct link to Syntax") ```sql EXECUTE AS user WITH NO REVERT ``` #### Parameters[​](#parameters "Direct link to Parameters") `user`: The user must already exist. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The current login user (who calls the EXECUTE AS statement) must be granted the privilege to impersonate another user. For more information, see [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md). * The EXECUTE AS statement must contain the WITH NO REVERT clause, which means the execution context of the current session cannot be switched back to the original login user before the current session ends. #### Examples[​](#examples "Direct link to Examples") Switch the execution context of the current session to the user `test2`. ```sql EXECUTE AS test2 WITH NO REVERT; ``` After the switch succeeds, you can run the `select current_user()` command to obtain the current user. ```sql select current_user(); +-----------------------------+ | CURRENT_USER() | +-----------------------------+ | 'default_cluster:test2'@'%' | +-----------------------------+ ``` --- ### GRANT Grants roles or privileges on specific objects to a user, a role, or an external group. For more information about the privileges that can be granted, see [Privilege items](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md). For more information about creating an external group, see [Group Provider](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). After a GRANT operation is performed, you can run [SHOW GRANTS](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_GRANTS.md) to view detailed privilege information or run [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md) to revoke a privilege or role. Before a GRANT operation is performed, make sure that the related user or role has been created. For more information, see [CREATE USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_USER.md) and [CREATE ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_ROLE.md). tip * Only users with the `user_admin` role can grant any privilege to other users and roles. * After a role is granted to a user, you must run [SET ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_ROLE.md) to activate this role before you perform operations as this role. If you want all default roles to be activated upon login, run [ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md) or [SET DEFAULT ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_DEFAULT_ROLE.md). If you want all privileges in the system to be activated for all users upon login, set the global variable `SET GLOBAL activate_all_roles_on_login = TRUE;`. * Common users can only grant privileges that have the `WITH GRANT OPTION` keyword to other users and roles. #### Syntax[​](#syntax "Direct link to Syntax") ##### Grant privileges to roles or users[​](#grant-privileges-to-roles-or-users "Direct link to Grant privileges to roles or users") ###### System[​](#system "Direct link to System") ```sql GRANT { CREATE RESOURCE GROUP | CREATE RESOURCE | CREATE EXTERNAL CATALOG | REPOSITORY | BLACKLIST | FILE | OPERATE | CREATE STORAGE VOLUME | SECURITY } ON SYSTEM TO { ROLE | USER } { | } [ WITH GRANT OPTION ] ``` ###### Resource group[​](#resource-group "Direct link to Resource group") ```sql GRANT { ALTER | DROP | ALL [PRIVILEGES] } ON { RESOURCE GROUP [, ,...] ｜ ALL RESOURCE GROUPS} TO { ROLE | USER } { | } [ WITH GRANT OPTION ] ``` ###### Resource[​](#resource "Direct link to Resource") ```sql GRANT { USAGE | ALTER | DROP | ALL [PRIVILEGES] } ON { RESOURCE [, < resource_name >,...] ｜ ALL RESOURCES} TO { ROLE | USER } { | } [ WITH GRANT OPTION ] ``` ###### Global UDF[​](#global-udf "Direct link to Global UDF") ```sql GRANT { USAGE | DROP | ALL [PRIVILEGES]} ON { GLOBAL FUNCTION (input_data_type) [, (input_data_type),...] | ALL GLOBAL FUNCTIONS } TO { ROLE | USER } { | } [ WITH GRANT OPTION ] ``` Example: `GRANT usage ON GLOBAL FUNCTION a(string) to kevin;` ###### Internal catalog[​](#internal-catalog "Direct link to Internal catalog") ```sql GRANT { USAGE | CREATE DATABASE | ALL [PRIVILEGES]} ON CATALOG default_catalog TO { ROLE | USER } { | } [ WITH GRANT OPTION ] ``` ###### External catalog[​](#external-catalog "Direct link to External catalog") ```sql GRANT { USAGE | DROP | ALL [PRIVILEGES] } ON { CATALOG [, ,...] | ALL CATALOGS} TO { ROLE | USER } { | } [ WITH GRANT OPTION ] ``` ###### Database[​](#database "Direct link to Database") ```sql GRANT { ALTER | DROP | CREATE TABLE | CREATE VIEW | CREATE FUNCTION | CREATE MATERIALIZED VIEW | ALL [PRIVILEGES] } ON { DATABASE [, ,...] | ALL DATABASES } TO { ROLE | USER } { | } [ WITH GRANT OPTION ] ``` * You must first run SET CATALOG before you run this command. * For databases in an External Catalog, you can grant the CREATE TABLE privilege only on Hive (since v3.1) and Iceberg databases (since v3.2). ###### Table[​](#table "Direct link to Table") ```sql -- Grant privileges on SPECIFIC TABLES. GRANT { ALTER | DROP | SELECT | INSERT | EXPORT | UPDATE | DELETE | ALL [PRIVILEGES]} ON TABLE [, < table_name >,...] TO { ROLE | USER } { | } [ WITH GRANT OPTION ] -- Grant privileges on ALL TABLES in a specific database or all databases. GRANT { ALTER | DROP | SELECT | INSERT | EXPORT | UPDATE | DELETE | ALL [PRIVILEGES]} ON ALL TABLES IN { { DATABASE } | ALL DATABASES } TO { ROLE | USER } { | } [ WITH GRANT OPTION ] ``` * You must first run SET CATALOG before you run this command. * You can also use `.` to represent a table. * You can grant the SELECT privilege on all tables in Internal and External Catalogs to read data from these tables. For tables in Hive and Iceberg Catalogs, you can grant the INSERT privilege to write data into such tables (supported since v3.1 for Iceberg and v3.2 for Hive) ```sql GRANT ON TABLE . TO { ROLE | USER } ``` ###### View[​](#view "Direct link to View") ```sql GRANT { ALTER | DROP | SELECT | ALL [PRIVILEGES]} ON { VIEW [, < view_name >,...] ｜ ALL VIEWS} IN { { DATABASE } | ALL DATABASES } TO { ROLE | USER } { | } [ WITH GRANT OPTION ] ``` * You must first run SET CATALOG before you run this command. * You can also use `.` to represent a view. * For tables in an External Catalog, you can only grant the SELECT privilege on Hive table views (since v3.1). ```sql GRANT ON VIEW . TO { ROLE | USER } ``` ###### Materialized view[​](#materialized-view "Direct link to Materialized view") ```sql GRANT { SELECT | ALTER | REFRESH | DROP | ALL [PRIVILEGES]} ON { MATERIALIZED VIEW [, < mv_name >,...] ｜ ALL MATERIALIZED VIEWS} IN { { DATABASE } | ALL DATABASES } TO { ROLE | USER } { | } [ WITH GRANT OPTION ] ``` * You must first run SET CATALOG before you run this command. * You can also use `.` to represent an mv. ```sql GRANT ON MATERIALIZED VIEW . TO { ROLE | USER } ``` ###### Function[​](#function "Direct link to Function") ```sql GRANT { USAGE | DROP | ALL [PRIVILEGES]} ON { FUNCTION (input_data_type) [, < function_name >(input_data_type),...] ｜ ALL FUNCTIONS} IN { { DATABASE } | ALL DATABASES } TO { ROLE | USER } { | } [ WITH GRANT OPTION ] ``` * You must first run SET CATALOG before you run this command. * You can also use `.` to represent a function. ```sql GRANT ON FUNCTION .(input_data_type) TO { ROLE | USER } ``` ###### User[​](#user "Direct link to User") ```sql GRANT IMPERSONATE ON USER TO USER [ WITH GRANT OPTION ] ``` ###### Storage volume[​](#storage-volume "Direct link to Storage volume") ```sql GRANT { USAGE | ALTER | DROP | ALL [PRIVILEGES] } ON { STORAGE VOLUME < name > [, < name >,...] ｜ ALL STORAGE VOLUMES} TO { ROLE | USER } { | } [ WITH GRANT OPTION ] ``` ##### Grant roles to roles, users, or external groups[​](#grant-roles-to-roles-users-or-external-groups "Direct link to Grant roles to roles, users, or external groups") ```sql GRANT [,, ...] TO ROLE GRANT [,, ...] TO USER GRANT [,, ...] TO EXTERNAL GROUP ``` #### Examples[​](#examples "Direct link to Examples") Example 1: Grant the privilege to read data from all tables in all databases to user `jack`. ```sql GRANT SELECT ON *.* TO 'jack'@'%'; ``` Example 2: Grant the privilege to load data into all tables of database `db1` to role `my_role`. ```sql GRANT INSERT ON db1.* TO ROLE 'my_role'; ``` Example 3: Grant the privileges to read, update, and load data into table `tbl1` of database `db1` to user `jack`. ```sql GRANT SELECT,ALTER,INSERT ON db1.tbl1 TO 'jack'@'192.8.%'; ``` Example 4: Grant the privilege to use all the resources to user `jack`. ```sql GRANT USAGE ON RESOURCE * TO 'jack'@'%'; ``` Example 5: Grant the privilege to use resource `spark_resource` to user `jack`. ```sql GRANT USAGE ON RESOURCE 'spark_resource' TO 'jack'@'%'; ``` Example 6: Grant the privilege to use resource `spark_resource` to role `my_role`. ```sql GRANT USAGE ON RESOURCE 'spark_resource' TO ROLE 'my_role'; ``` Example 7: Grant the privilege to read data from table `sr_member` to user `jack` and allow user `jack` to grant this privilege to other users or roles (by specifying WITH GRANT OPTION). ```sql GRANT SELECT ON TABLE sr_member TO USER jack@'172.10.1.10' WITH GRANT OPTION; ``` Example 8: Grant system-defined roles `db_admin`, `user_admin`, and `cluster_admin` to user `user_platform`. ```sql GRANT db_admin, user_admin, cluster_admin TO USER user_platform; ``` Example 9: Allow user `jack` to perform operations as user `rose`. ```sql GRANT IMPERSONATE ON USER 'rose'@'%' TO USER 'jack'@'%'; ``` Example 10: Grant system-defined roles `db_admin`, `user_admin`, and `cluster_admin` to the external group `admin_group`. ```sql GRANT db_admin, user_admin, cluster_admin TO EXTERNAL GROUP admin_group; ``` #### Best practices[​](#best-practices "Direct link to Best practices") ##### Customize roles based on scenarios[​](#customize-roles-based-on-scenarios "Direct link to Customize roles based on scenarios") We recommend you customize roles to manage privileges and users. The following examples classify a few combinations of privileges for some common scenarios. ###### Grant global read-only privileges on StarRocks tables[​](#grant-global-read-only-privileges-on-starrocks-tables "Direct link to Grant global read-only privileges on StarRocks tables") ```sql -- Create a role. CREATE ROLE read_only; -- Grant the USAGE privilege on all catalogs to the role. GRANT USAGE ON ALL CATALOGS TO ROLE read_only; -- Grant the privilege to query all tables to the role. GRANT SELECT ON ALL TABLES IN ALL DATABASES TO ROLE read_only; -- Grant the privilege to query all views to the role. GRANT SELECT ON ALL VIEWS IN ALL DATABASES TO ROLE read_only; -- Grant the privilege to query all materialized views and the privilege to accelerate queries with them to the role. GRANT SELECT ON ALL MATERIALIZED VIEWS IN ALL DATABASES TO ROLE read_only; ``` And you can further grant the privilege to use UDFs in queries: ```sql -- Grant the USAGE privilege on all database-level UDF to the role. GRANT USAGE ON ALL FUNCTIONS IN ALL DATABASES TO ROLE read_only; -- Grant the USAGE privilege on global UDF to the role. GRANT USAGE ON ALL GLOBAL FUNCTIONS TO ROLE read_only; ``` ###### Grant global write privileges on StarRocks tables[​](#grant-global-write-privileges-on-starrocks-tables "Direct link to Grant global write privileges on StarRocks tables") ```sql -- Create a role. CREATE ROLE write_only; -- Grant the USAGE privilege on all catalogs to the role. GRANT USAGE ON ALL CATALOGS TO ROLE write_only; -- Grant the INSERT and UPDATE privileges on all tables to the role. GRANT INSERT, UPDATE ON ALL TABLES IN ALL DATABASES TO ROLE write_only; -- Grant the REFRESH privilege on all materialized views to the role. GRANT REFRESH ON ALL MATERIALIZED VIEWS IN ALL DATABASES TO ROLE write_only; ``` ###### Grant read-only privileges on a specific external catalog[​](#grant-read-only-privileges-on-a-specific-external-catalog "Direct link to Grant read-only privileges on a specific external catalog") ```sql -- Create a role. CREATE ROLE read_catalog_only; -- Grant the USAGE privilege on the destination catalog to the role. GRANT USAGE ON CATALOG hive_catalog TO ROLE read_catalog_only; -- Switch to the corresponding catalog. SET CATALOG hive_catalog; -- Grant the privileges to query all tables and all views in the external catalog. GRANT SELECT ON ALL TABLES IN ALL DATABASES TO ROLE read_catalog_only; ``` tip For views in external catalogs, you can query only Hive table views (since v3.1). ###### Grant write-only privileges on a specific external catalog[​](#grant-write-only-privileges-on-a-specific-external-catalog "Direct link to Grant write-only privileges on a specific external catalog") You can only write data into Iceberg tables (since v3.1) and Hive tables (since v3.2). ```sql -- Create a role. CREATE ROLE write_catalog_only; -- Grant the USAGE privilege on the destination catalog to the role. GRANT USAGE ON CATALOG iceberg_catalog TO ROLE read_catalog_only; -- Switch to the corresponding catalog. SET CATALOG iceberg_catalog; -- Grant the privilege to write data into Iceberg tables. GRANT INSERT ON ALL TABLES IN ALL DATABASES TO ROLE write_catalog_only; ``` ###### Grant admin privileges on a specific database[​](#grant-admin-privileges-on-a-specific-database "Direct link to Grant admin privileges on a specific database") ```sql -- Create a role. CREATE ROLE db1_admin; -- Grant ALL privileges on the destination database to the role. This role can create tables, views, materialized views, and UDFs in this database. And it also can drop or modify this database. GRANT ALL ON DATABASE db1 TO ROLE db1_admin; -- Switch to the corresponding catalog. SET CATALOG iceberg_catalog; -- Grant all privileges on tables, views, materialized views, and UDFs in this database to the role. GRANT ALL ON ALL TABLES IN DATABASE db1 TO ROLE db1_admin; GRANT ALL ON ALL VIEWS IN DATABASE db1 TO ROLE db1_admin; GRANT ALL ON ALL MATERIALIZED VIEWS IN DATABASE db1 TO ROLE db1_admin; GRANT ALL ON ALL FUNCTIONS IN DATABASE db1 TO ROLE db1_admin; ``` ###### Grant privileges to perform backup and restore operations on global, database, table, and partition levels[​](#grant-privileges-to-perform-backup-and-restore-operations-on-global-database-table-and-partition-levels "Direct link to Grant privileges to perform backup and restore operations on global, database, table, and partition levels") * Grant privileges to perform global backup and restore operations: The privileges to perform global backup and restore operations allow the role to back up and restore any database, table, or partition. It requires the REPOSITORY privilege on the SYSTEM level, the privileges to create databases in the default catalog, to create tables in any database, and to load and export data on any table. ```sql -- Create a role. CREATE ROLE recover; -- Grant the REPOSITORY privilege on the SYSTEM level. GRANT REPOSITORY ON SYSTEM TO ROLE recover; -- Grant the privilege to create databases in the default catalog. GRANT CREATE DATABASE ON CATALOG default_catalog TO ROLE recover; -- Grant the privilege to create tables in any database. GRANT CREATE TABLE ON ALL DATABASES TO ROLE recover; -- Grant the privilege to load and export data on any table. GRANT INSERT, EXPORT ON ALL TABLES IN ALL DATABASES TO ROLE recover; ``` * Grant the privileges to perform database-level backup and restore operations: The privileges to perform database-level backup and restore operations require the REPOSITORY privilege on the SYSTEM level, the privilege to create databases in the default catalog, the privilege to create tables in any database, the privilege to load data into any table, and the privilege export data from any table in the database to be backed up. ```sql -- Create a role. CREATE ROLE recover_db; -- Grant the REPOSITORY privilege on the SYSTEM level. GRANT REPOSITORY ON SYSTEM TO ROLE recover_db; -- Grant the privilege to create databases. GRANT CREATE DATABASE ON CATALOG default_catalog TO ROLE recover_db; -- Grant the privilege to create tables. GRANT CREATE TABLE ON ALL DATABASES TO ROLE recover_db; -- Grant the privilege to load data into any table. GRANT INSERT ON ALL TABLES IN ALL DATABASES TO ROLE recover_db; -- Grant the privilege to export data from any table in the database to be backed up. GRANT EXPORT ON ALL TABLES IN DATABASE TO ROLE recover_db; ``` * Grant the privileges to perform table-level backup and restore operations: The privileges to perform table-level backup and restore operations require the REPOSITORY privilege on the SYSTEM level, the privilege to create tables in corresponding databases, the privilege to load data into any table in the database, and the privilege to export data from the table to be backed up. ```sql -- Create a role. CREATE ROLE recover_tbl; -- Grant the REPOSITORY privilege on the SYSTEM level. GRANT REPOSITORY ON SYSTEM TO ROLE recover_tbl; -- Grant the privilege to create tables in corresponding databases. GRANT CREATE TABLE ON DATABASE TO ROLE recover_tbl; -- Grant the privilege to load data into any table in a database. GRANT INSERT ON ALL TABLES IN DATABASE TO ROLE recover_db; -- Grant the privilege to export data from the table you want to back up. GRANT EXPORT ON TABLE TO ROLE recover_tbl; ``` * Grant the privileges to perform partition-level backup and restore operations: The privileges to perform partition-level backup and restore operations require the REPOSITORY privilege on the SYSTEM level, and the privilege to load and export data on the corresponding table. ```sql -- Create a role. CREATE ROLE recover_par; -- Grant the REPOSITORY privilege on the SYSTEM level. GRANT REPOSITORY ON SYSTEM TO ROLE recover_par; -- Grant the privilege to load and export data on the corresponding table. GRANT INSERT, EXPORT ON TABLE TO ROLE recover_par; ``` For the best practices of multi-service access control, see [Multi-service access control](https://docs.starrocks.io/docs/administration/user_privs/authorization/User_privilege.md#multi-service-access-control). --- ### REVOKE Use REVOKE to remove specific privileges or roles from a user or a role. For the privileges supported by StarRocks, see [Privileges supported by StarRocks](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md). tip * Common users can only revoke their privileges that have the `WITH GRANT OPTION` keyword from other users and roles. For information about `WITH GRANT OPTION`, see [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md). * Only users with the `user_admin` role has the privilege to revoke privileges from other users. #### Syntax[​](#syntax "Direct link to Syntax") ##### Revoke privileges[​](#revoke-privileges "Direct link to Revoke privileges") The privileges that can be revoked are object-specific. The following part describes syntax based on objects. ###### System[​](#system "Direct link to System") ```sql REVOKE { CREATE RESOURCE GROUP | CREATE RESOURCE | CREATE EXTERNAL CATALOG | REPOSITORY | BLACKLIST | FILE | OPERATE } ON SYSTEM FROM { ROLE | USER} {|} ``` ###### Resource group[​](#resource-group "Direct link to Resource group") ```sql REVOKE { ALTER | DROP | ALL [PRIVILEGES] } ON { RESOURCE GROUP [, ,...] ｜ ALL RESOURCE GROUPS} FROM { ROLE | USER} {|} ``` ###### Resource[​](#resource "Direct link to Resource") ```sql REVOKE { USAGE | ALTER | DROP | ALL [PRIVILEGES] } ON { RESOURCE [, ,...] ｜ ALL RESOURCES} FROM { ROLE | USER} {|} ``` ###### User[​](#user "Direct link to User") ```sql REVOKE IMPERSONATE ON USER FROM USER ; ``` ###### Global UDF[​](#global-udf "Direct link to Global UDF") ```sql REVOKE { USAGE | DROP | ALL [PRIVILEGES]} ON { GLOBAL FUNCTION (input_data_type) [, (input_data_type),...] | ALL GLOBAL FUNCTIONS } FROM { ROLE | USER} {|} ``` ###### Internal catalog[​](#internal-catalog "Direct link to Internal catalog") ```sql REVOKE { USAGE | CREATE DATABASE | ALL [PRIVILEGES]} ON CATALOG default_catalog FROM { ROLE | USER} {|} ``` ###### External catalog[​](#external-catalog "Direct link to External catalog") ```sql REVOKE { USAGE | DROP | ALL [PRIVILEGES] } ON { CATALOG [, ,...] | ALL CATALOGS} FROM { ROLE | USER} {|} ``` ###### Database[​](#database "Direct link to Database") ```sql REVOKE { ALTER | DROP | CREATE TABLE | CREATE VIEW | CREATE FUNCTION | CREATE MATERIALIZED VIEW | ALL [PRIVILEGES] } ON {{ DATABASE [, ,...]} | ALL DATABASES } FROM { ROLE | USER} {|} ``` * You must first run SET CATALOG before you run this command. ###### Table[​](#table "Direct link to Table") ```sql REVOKE { ALTER | DROP | SELECT | INSERT | EXPORT | UPDATE | DELETE | ALL [PRIVILEGES]} ON { TABLE [, < table_name >,...] | ALL TABLES} IN { { DATABASE [, ,...]} | ALL DATABASES } FROM { ROLE | USER} {|} ``` * You must first run SET CATALOG before you run this command. * You can also use db.tbl to represent a table. ```sql REVOKE ON TABLE db.tbl FROM {ROLE | USER } ``` ###### View[​](#view "Direct link to View") ```sql REVOKE { ALTER | DROP | SELECT | ALL [PRIVILEGES]} ON { VIEW [, < view_name >,...] ｜ ALL VIEWS} IN { { DATABASE [, ,...]} | ALL DATABASES } FROM { ROLE | USER} {|} ``` * You must first run SET CATALOG before you run this command. * You can also use db.view to represent a view. ```sql REVOKE ON VIEW db.view FROM {ROLE | USER } ``` ###### Materialized view[​](#materialized-view "Direct link to Materialized view") ```sql REVOKE { SELECT | ALTER | REFRESH | DROP | ALL [PRIVILEGES]} ON { MATERIALIZED VIEW [, < mv_name >,...] ｜ ALL MATERIALIZED VIEWS} IN { { DATABASE [, ,...] } | ALL [DATABASES] } FROM { ROLE | USER} {|} ``` * You must first run SET CATALOG before you run this command. * You can also use db.mv to represent an mv. ```sql REVOKE ON MATERIALIZED VIEW db.mv FROM {ROLE | USER } ``` ###### Function[​](#function "Direct link to Function") ```sql REVOKE { USAGE | DROP | ALL [PRIVILEGES]} ON { FUNCTION (input_data_type) [, (input_data_type),...] ｜ ALL FUNCTIONS} IN { { DATABASE [, ,...] } | ALL DATABASES } FROM { ROLE | USER} {|} ``` * You must first run SET CATALOG before you run this command. * You can also use db.function to represent a function. ```sql REVOKE ON FUNCTION .(input_data_type) FROM {ROLE | USER } ``` ###### Storage volume[​](#storage-volume "Direct link to Storage volume") ```sql REVOKE CREATE STORAGE VOLUME ON SYSTEM FROM { ROLE | USER} {|} REVOKE { USAGE | ALTER | DROP | ALL [PRIVILEGES] } ON { STORAGE VOLUME < name > [, < name >,...] ｜ ALL STORAGE VOLUME} FROM { ROLE | USER} {|} ``` ##### Revoke roles[​](#revoke-roles "Direct link to Revoke roles") ```sql REVOKE [,, ...] FROM ROLE REVOKE [,, ...] FROM USER ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | ------------------- | ----------------------------------------------- | | role\_name | The role name. | | user\_identity | The user identity, for example, 'jack'@'192.%'. | | resourcegroup\_name | The resource group name | | resource\_name | The resource name. | | function\_name | The function name. | | catalog\_name | The name of the external catalog. | | database\_name | The database name. | | table\_name | The table name. | | view\_name | The view name. | | mv\_name | The name of the materialized view. | #### Examples[​](#examples "Direct link to Examples") ##### Revoke privileges[​](#revoke-privileges-1 "Direct link to Revoke privileges") Revoke the SELECT privilege on table `sr_member` from user `jack`: ```sql REVOKE SELECT ON TABLE sr_member FROM USER 'jack'@'192.%' ``` Revoke the USAGE privilege on resource `spark_resource` from role `test_role`: ```sql REVOKE USAGE ON RESOURCE 'spark_resource' FROM ROLE 'test_role'; ``` ##### Revoke roles[​](#revoke-roles-1 "Direct link to Revoke roles") Revoke the role `example_role` from user `jack`: ```sql REVOKE example_role FROM 'jack'@'%'; ``` Revoke the role `example_role` from role `test_role`: ```sql REVOKE example_role FROM ROLE 'test_role'; ``` #### References[​](#references "Direct link to References") [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) --- ### SET DEFAULT ROLE SET DEFAULT ROLE sets the roles that are activated by default when the user connects to the server. This command is supported from v3.0. tip Common users can set default roles for themselves. Users with the `user_admin` role can set default roles for other users. Before you perform this operation, make sure that the user has already been assigned these roles. You can query the roles of a user using [SHOW GRANTS](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_GRANTS.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql -- Set specified roles as default roles. SET DEFAULT ROLE [,,..] TO ; -- Set all roles of the user, including roles that will be assigned to this user, as default roles. SET DEFAULT ROLE ALL TO ; -- No default role is set but the public role is still enabled after a user login. SET DEFAULT ROLE NONE TO ; ``` #### Parameters[​](#parameters "Direct link to Parameters") `role_name`: the role name `user_identity`: the user identity #### Examples[​](#examples "Direct link to Examples") Query the roles of the current user. ```sql SHOW GRANTS FOR test; +--------------+---------+----------------------------------------------+ | UserIdentity | Catalog | Grants | +--------------+---------+----------------------------------------------+ | 'test'@'%' | NULL | GRANT 'db_admin', 'user_admin' TO 'test'@'%' | +--------------+---------+----------------------------------------------+ ``` Example 1: Set roles `db_admin` and `user_admin` as the default roles for the user `test`. ```sql SET DEFAULT ROLE db_admin TO test; ``` Example 2: Set all roles of the user `test`, including roles that will be assigned to this user as default roles. ```sql SET DEFAULT ROLE ALL TO test; ``` Example 3: Clear all the default roles of the user `test`. ```sql SET DEFAULT ROLE NONE TO test; ``` --- ### SET PASSWORD SET PASSWORD changes the login password for users. The [ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md) command can also be used to change a password. tip * All users can reset their own password. * Only users with the `user_admin` role can change the password of other users. * Only the `root` user itself can change its password. For more information, see [the priviege overview](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md). ##### Syntax[​](#syntax "Direct link to Syntax") ```sql SET PASSWORD [FOR user_identity] = [PASSWORD('plain password')]|['hashed password'] ``` The `user_identity` must match exactly the `user_identity` specified when creating a user by using CREATE USER. Otherwise, the user will be reported as non-existent. If `user_identity` is not specified, the password of the current user will be changed. The current user can be viewed through [SHOW GRANTS](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_GRANTS.md). `PASSWORD()` inputs a plaintext password. If you directly input a string without using `PASSWORD()`, the string must be encrypted. #### Examples[​](#examples "Direct link to Examples") 1. Set the password for the current user. ```sql SET PASSWORD = PASSWORD('123456') SET PASSWORD = '*6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9' ``` 2. Set the password for a specified user. ```sql SET PASSWORD FOR 'jack'@'192.%' = PASSWORD('123456') SET PASSWORD FOR 'jack'@['domain'] = '*6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9' ``` --- ### SET ROLE SET ROLE activates roles, along with all of its associated privileges and nested roles, for the current session. After the role is activated, users can use this role to perform operations. After running this command, you can run `select is_role_in_session("");` to verify whether this role is active in the current session. This command is supported from v3.0. #### Usage notes[​](#usage-notes "Direct link to Usage notes") Users can only activate roles that have been assigned to them. You can query the roles of a user using [SHOW GRANTS](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_GRANTS.md). You can query the active roles of the current user using `SELECT CURRENT_ROLE()`. For more information, see [current\_role](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/current_role.md). The activated role affects all authorization checks, including queries against `information_schema` system views. For example, after executing `SET ROLE role1`, queries like `SELECT * FROM information_schema.tables` will only return tables accessible through `role1`. This ensures that role-based access control is consistently applied across all system catalogs and metadata queries. #### Syntax[​](#syntax "Direct link to Syntax") ```sql -- Active specific roles and perform operations as this role. SET ROLE [,,..]; -- Activate all roles of a user, except for specific roles. SET ROLE ALL EXCEPT [,,..]; -- Activate all roles of a user. SET ROLE ALL; ``` #### Parameters[​](#parameters "Direct link to Parameters") `role_name`: the role name #### Examples[​](#examples "Direct link to Examples") Query all the roles of the current user. ```sql SHOW GRANTS; +--------------+---------+----------------------------------------------+ | UserIdentity | Catalog | Grants | +--------------+---------+----------------------------------------------+ | 'test'@'%' | NULL | GRANT 'db_admin', 'user_admin' TO 'test'@'%' | +--------------+---------+----------------------------------------------+ ``` Activate the `db_admin` role. ```sql SET ROLE db_admin; ``` Query active roles of the current user. ```sql SELECT CURRENT_ROLE(); +--------------------+ | CURRENT_ROLE() | +--------------------+ | db_admin | +--------------------+ ``` #### References[​](#references "Direct link to References") * [CREATE ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_ROLE.md): creates a role. * [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md): assigns roles to users or other roles. * [ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md): modifies roles. * [SHOW ROLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_ROLES.md): show all roles in the system. * [current\_role](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/current_role.md): show roles of the current user. * [is\_role\_in\_session](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/is_role_in_session.md): verifies whether a role (or a nested role) is active in the current session. * [DROP ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/DROP_ROLE.md): drops a role. --- ### SHOW AUTHENTICATION SHOW AUTHENTICATION displays the authentication information of the current user or all users in the current cluster. tip All users have the privilege to view their own authentication information. Only users with the `user_admin` role can view the authentication information of all users or the authentication information of specified users. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW [ALL] AUTHENTICATION [FOR USERNAME] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ALL | No | If this keyword is specified, the authentication information of all users in the current cluster is returned. If this keyword is not specified, only the authentication information of the current user is returned. | | USERNAME | No | If this parameter is specified, the authentication information of a specified user can be viewed. If this parameter is not specified, only the authentication information of the current user can be viewed. | #### Output[​](#output "Direct link to Output") ```sql +---------------+----------+-------------+-------------------+ | UserIdentity | Password | AuthPlugin | UserForAuthPlugin | +---------------+----------+-------------+-------------------+ ``` | **Field** | **Description** | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | UserIdentity | The user identity. | | Password | Whether a password is used to log in to the StarRocks cluster.- `Yes`: A password is used.
- `No`: No password is used. | | AuthPlugin | The interface that is used for authentication. Valid values: `MYSQL_NATIVE_PASSWORD` and `AUTHENTICATION_LDAP_SIMPLE`. If no interface is used, `NULL` is returned. | | UserForAuthPlugin | The name of the user using the LDAP or Kerberos authentication. If no authentication is used, `NULL` is returned. | #### Examples[​](#examples "Direct link to Examples") Example 1: Display the authentication information of the current user. ```plain SHOW AUTHENTICATION; +--------------+----------+------------+-------------------+ | UserIdentity | Password | AuthPlugin | UserForAuthPlugin | +--------------+----------+------------+-------------------+ | 'root'@'%' | No | NULL | NULL | +--------------+----------+------------+-------------------+ ``` Example 2: Display the authentication information of all users in the current cluster. ```plain SHOW ALL AUTHENTICATION; +---------------+----------+----------------------------+-------------------+ | UserIdentity | Password | AuthPlugin | UserForAuthPlugin | +---------------+----------+----------------------------+-------------------+ | 'root'@'%' | Yes | NULL | NULL | | 'chelsea'@'%' | No | AUTHENTICATION_LDAP_SIMPLE | NULL | +---------------+----------+----------------------------+-------------------+ ``` Example 3: Display the authentication information of a specified user. ```plain SHOW AUTHENTICATION FOR root; +--------------+----------+------------+-------------------+ | UserIdentity | Password | AuthPlugin | UserForAuthPlugin | +--------------+----------+------------+-------------------+ | 'root'@'%' | Yes | NULL | NULL | +--------------+----------+------------+-------------------+ ``` --- ### SHOW GRANTS SHOW GRANTS displays all the privileges that have been granted to a user or role. For more information about roles and privileges, see [Overview of privileges](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md). tip All roles and users can view the privileges granted to them or the roles assigned to them. Only users with the `user_admin` role can view the privileges of a specified user or role. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW GRANTS; -- View the privileges of the current user. SHOW GRANTS FOR CURRENT_USER[()]; -- View the privileges of the current user (MySQL-compatible syntax). SHOW GRANTS FOR ROLE ; -- View the privileges of a specific role. SHOW GRANTS FOR ; -- View the privileges of a specific user. ``` #### Parameters[​](#parameters "Direct link to Parameters") * `CURRENT_USER[()]`: Returns the privileges of the current session user. The parentheses are optional. This form is equivalent to `SHOW GRANTS` and is provided for compatibility with MySQL clients (such as Metabase and DBeaver) that issue this statement automatically during connection initialization. * role\_name * user\_identity Return fields: ```sql -- View the privileges of a specific user. +--------------+--------+---------------------------------------------+ |UserIdentity |Catalog | Grants | +--------------+--------+---------------------------------------------+ -- View the privileges of a specific role. +-------------+--------+-------------------------------------------------------+ |RoleName |Catalog | Grants | +-------------+-----------------+----------------------------------------------+ ``` | **Field** | **Description** | | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | UserIdentity | The user identity, which is displayed when you query the privileges of a user. | | RoleName | The role name, which is displayed when you query the privileges of a role. | | Catalog | The catalog name.
`default` is returned if the GRANT operation is performed on the StarRocks internal catalog.
The name of the external catalog is returned if the GRANT operation is performed on an external catalog.
`NULL` is returned if the operation shown in the `Grants` column is assigning roles. | | Grants | The specific GRANT operation. | #### Examples[​](#examples "Direct link to Examples") ```sql mysql> SHOW GRANTS; +--------------+---------+----------------------------------------+ | UserIdentity | Catalog | Grants | +--------------+---------+----------------------------------------+ | 'root'@'%' | NULL | GRANT 'root', 'testrole' TO 'root'@'%' | +--------------+---------+----------------------------------------+ mysql> SHOW GRANTS FOR CURRENT_USER(); +--------------+---------+----------------------------------------+ | UserIdentity | Catalog | Grants | +--------------+---------+----------------------------------------+ | 'root'@'%' | NULL | GRANT 'root', 'testrole' TO 'root'@'%' | +--------------+---------+----------------------------------------+ mysql> SHOW GRANTS FOR 'user_g'@'%'; +-------------+-------------+-----------------------------------------------------------------------------------------------+ |UserIdentity |Catalog |Grants | +-------------+-------------------------------------------------------------------------------------------------------------+ |'user_g'@'%' |NULL |GRANT role_g, public to `user_g`@`%`; | |'user_g'@'%' |NULL |GRANT IMPERSONATE ON USER `user_a`@`%` TO USER `user_g`@`%`; | |'user_g'@'%' |default |GRANT CREATE_DATABASE ON CATALOG default_catalog TO USER `user_g`@`%`; | |'user_g'@'%' |default |GRANT ALTER, DROP, CREATE_TABLE ON DATABASE db1 TO USER `user_g`@`%`; | |'user_g'@'%' |default |GRANT CREATE_VIEW ON DATABASE db1 TO USER `user_g`@`%` WITH GRANT OPTION; | |'user_g'@'%' |default |GRANT ALTER, DROP, SELECT, INGEST, EXPORT, DELETE, UPDATE ON TABLE db.* TO USER `user_g`@`%`; | |'user_g'@'%' |default |GRANT ALTER, DROP, SELECT ON VIEW db2.view TO USER `user_g`@`%`; | |'user_g'@'%' |Hive_catalog |GRANT USAGE ON CATALOG Hive_catalog TO USER `user_g`@`%` | +-------------+--------------+-----------------------------------------------------------------------------------------------+ mysql> SHOW GRANTS FOR ROLE role_g; +-------------+--------+-------------------------------------------------------+ |RoleName |Catalog | Grants | +-------------+-----------------+----------------------------------------------+ |role_g |NULL | GRANT role_p, role_test TO ROLE role_g; | |role_g |default | GRANT SELECT ON *.* TO ROLE role_g WITH GRANT OPTION; | +-------------+--------+--------------------------------------------------------+ ``` #### References[​](#references "Direct link to References") [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) --- ### SHOW PROPERTY SHOW PROPERTY displays properties of a user, including the maximum number of connections, the default catalog, and the default database. tip The current user can view their own properties. Only users with the `user_admin` role can view the properties of other users. info To set properties such as `database` or `catalog`, use the [ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md) command with `SET PROPERTIES`. For `max_user_connections`, you can use the `SET PROPERTY` syntax. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW PROPERTY [FOR 'user_name'] [LIKE ''] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | -------------- | ------------ | ---------------------------------------------------------------------------- | | user\_name | No | The user name. If not specified, the property of the current user is viewed. | | property\_name | No | The user property name. | #### Examples[​](#examples "Direct link to Examples") Example 1: View the properties of the current user. ```sql SHOW PROPERTY; ``` ```plain +----------------------+-----------------+ | Key | Value | +----------------------+-----------------+ | max_user_connections | 1024 | | catalog | default_catalog | | database | | +----------------------+-----------------+ ``` Example 2: View the properties of the user `jack`. ```sql SHOW PROPERTY FOR 'jack'; ``` ```plain +----------------------+------------------+ | Key | Value | +----------------------+------------------+ | max_user_connections | 100 | | catalog | default_catalog | | database | sales_db | +----------------------+------------------+ ``` Example 3: Filter properties using `LIKE`. ```sql SHOW PROPERTY FOR 'jack' LIKE 'max_user_connections'; ``` ```plain +----------------------+-------+ | Key | Value | +----------------------+-------+ | max_user_connections | 100 | +----------------------+-------+ ``` #### See also[​](#see-also "Direct link to See also") [ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md): Sets properties for a user. --- ### SHOW ROLES SHOW ROLES displays all roles in the system. You can use `SHOW GRANTS FOR ROLE ;` to view the privileges of a specific role. For more information, see [SHOW GRANTS](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_GRANTS.md). This command is supported from v3.0. tip Only users with the `user_admin` role has the privilege to perform this operation. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW ROLES ``` Return fields: | **Field** | **Description** | | --------- | --------------------- | | Name | The name of the role. | #### Examples[​](#examples "Direct link to Examples") Display all roles in the system. ```sql mysql> SHOW ROLES; +---------------+ | Name | +---------------+ | root | | db_admin | | cluster_admin | | user_admin | | public | | testrole | +---------------+ ``` #### References[​](#references "Direct link to References") * [CREATE ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_ROLE.md) * [ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md) * [DROP ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/DROP_ROLE.md) --- ### SHOW USERS SHOW USERS displays all users in the system. Users mentioned here are user identities, not user names. For more information about user identities, see [CREATE USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_USER.md). This command is supported from v3.0. You can use `SHOW GRANTS FOR ;` to view the privileges of a specific user. For more information, see [SHOW GRANTS](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_GRANTS.md). tip Only users with the `user_admin` role has the privilege to perform this operation. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW USERS ``` Return fields: | **Field** | **Description** | | --------- | ------------------ | | User | The user identity. | #### Examples[​](#examples "Direct link to Examples") Display all users in the system. ```sql mysql> SHOW USERS; +-----------------+ | User | +-----------------+ | 'wybing5'@'%' | | 'root'@'%' | | 'admin'@'%' | | 'star'@'%' | | 'wybing_30'@'%' | | 'simo'@'%' | | 'wybing1'@'%' | | 'wybing2'@'%' | +-----------------+ ``` #### References[​](#references "Direct link to References") [CREATE USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_USER.md), [ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md), [DROP USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/DROP_USER.md) --- ### Backup and restore #### [📄️ BACKUP](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/BACKUP.md) [Backs up databases, tables, materialized views, logical views, and UDFs in StarRocks to a remote storage repository.](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/BACKUP.md) #### [📄️ CANCEL BACKUP](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/CANCEL_BACKUP.md) [Cancels an ongoing BACKUP task in a specified database.](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/CANCEL_BACKUP.md) #### [📄️ CANCEL RESTORE](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/CANCEL_RESTORE.md) [Cancels an ongoing RESTORE task in a specified database.](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/CANCEL_RESTORE.md) #### [📄️ CREATE REPOSITORY](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/CREATE_REPOSITORY.md) [Creates a repository in a remote storage system that is used to store data snapshots for backing up and restoring data.](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/CREATE_REPOSITORY.md) #### [📄️ DROP REPOSITORY](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/DROP_REPOSITORY.md) [Deletes a repository.](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/DROP_REPOSITORY.md) #### [📄️ RECOVER](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/RECOVER.md) [Recovers a database, table, or partition that was dropped by using the DROP command.](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/RECOVER.md) #### [📄️ RESTORE](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/RESTORE.md) [Restores databases, tables, materialized views, logical views, and UDFs in StarRocks from a snapshot created by BACKUP.](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/RESTORE.md) #### [📄️ SHOW BACKUP](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_BACKUP.md) [Views the last BACKUP task in a specified database.](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_BACKUP.md) #### [📄️ SHOW REPOSITORIES](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_REPOSITORIES.md) [Views the repositories created in StarRocks.](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_REPOSITORIES.md) #### [📄️ SHOW RESTORE](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_RESTORE.md) [Views the last RESTORE task in a specified database.](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_RESTORE.md) #### [📄️ SHOW SNAPSHOT](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_SNAPSHOT.md) [Views data snapshots in a specified repository.](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_SNAPSHOT.md) --- ### BACKUP StarRocks supports backing up and restoring the following objects: * Internal databases, tables (of all types and partitioning strategies), and partitions * Metadata of external catalogs (supported from v3.4.0 onwards) * Synchronous materialized views and asynchronous materialized views * Logical views (supported from v3.4.0 onwards) * User-defined functions (supported from v3.4.0 onwards) tip For an overview of backup and restore please see the [backup and restore guide](https://docs.starrocks.io/docs/administration/management/Backup_and_restore.md). BACKUP is an asynchronous operation. You can check the status of a BACKUP job status using [SHOW BACKUP](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_BACKUP.md), or cancel a BACKUP job using [CANCEL BACKUP](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/CANCEL_BACKUP.md). You can view the snapshot information using [SHOW SNAPSHOT](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_SNAPSHOT.md). > **CAUTION** > > * Shared-data StarRocks clusters do not support data BACKUP and RESTORE. > * In each database, only one running BACKUP or RESTORE job is allowed each time. Otherwise, StarRocks returns an error. > * StarRocks does not support specifying data compression algorithm for data backup. #### Privilege requirement[​](#privilege-requirement "Direct link to Privilege requirement") In versions earlier than v3.0, users with the `admin_priv` privilege can perform this operation. In v3.0 and later versions, to back up a specific object, users must have the REPOSITORY privilege at the System level and the EXPORT privilege for the corresponding table or all tables under the corresponding database. For example: * Grant the role permission to export data from the specified table. ```sql GRANT REPOSITORY ON SYSTEM TO ROLE backup_tbl; GRANT EXPORT ON TABLE TO ROLE backup_tbl; ``` * Grant the role permission to export data from all tables under the specified data. ```sql GRANT REPOSITORY ON SYSTEM TO ROLE backup_db; GRANT EXPORT ON ALL TABLES IN DATABASE TO ROLE backup_db; ``` * Grant the role permission to export data from all tables in all databases. ```sql GRANT REPOSITORY ON SYSTEM TO ROLE backup; GRANT EXPORT ON ALL TABLES IN ALL DATABASES TO ROLE backup; ``` #### Syntax compatible with earlier versions[​](#syntax-compatible-with-earlier-versions "Direct link to Syntax compatible with earlier versions") ```sql BACKUP SNAPSHOT . TO [ ON ( [ PARTITION ( [, ...] ) ] [, ...] ) ] [ PROPERTIES ("key"="value" [, ...] ) ] ``` ##### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | db\_name | Name of the database that stores the data to be backed up. | | snapshot\_name | Specify a name for the data snapshot. Globally unique. | | repository\_name | Repository name. You can create a repository using [CREATE REPOSITORY](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/CREATE_REPOSITORY.md). | | ON | Name of the tables to be backed up. The whole database is backed up if this parameter is not specified. | | PARTITION | Name of the partitions to be backed up. The whole table is backed up if this parameter is not specified. | | PROPERTIES | Properties of the data snapshot. Valid keys:- `type`: Backup type. Currently, only full backup `FULL` is supported. Default: `FULL`.
- `timeout`: Task timeout. Unit: second. Default: `86400`. | #### Syntax supported from v3.4.0 onwards[​](#syntax-supported-from-v340-onwards "Direct link to Syntax supported from v3.4.0 onwards") ```sql -- Back up external catalog metadata. BACKUP { ALL EXTERNAL CATALOGS | EXTERNAL CATALOG[S] ( [, ...]) } [ DATABASE ] SNAPSHOT [.] TO [ PROPERTIES ("key"="value" [, ...] ) ] -- Back up databases, tables, partitions, materialized views, logical views, or UDFs. BACKUP [ DATABASE ] SNAPSHOT [.] TO [ ON ( backup_object [, ...] )] [ PROPERTIES ("key"="value" [, ...] ) ] backup_object ::= [ { ALL TABLE[S] | TABLE[S] [, TABLE[S] ...] } | { ALL MATERIALIZED VIEW[S] | MATERIALIZED VIEW[S] [, MATERIALIZED VIEW[S] ...] } | { ALL VIEW[S] | VIEW[S] [, VIEW[S] ...] } | { ALL FUNCTION[S] | FUNCTION[S] [, FUNCTION[S] ...] } | PARTITION ([, ...]) ] ``` ##### Parameters[​](#parameters-1 "Direct link to Parameters") | **Parameter** | **Description** | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ALL EXTERNAL CATALOGS | Backs up the metadata of all external catalogs. | | catalog\_name | Name of the external catalog(s) that need to be backed up. | | DATABASE db\_name | Name of the database to which the object(s) or snapshot belongs. You can only specify either `DATABASE ` or `.`. | | db\_name. | Name of the database to which the object(s) or snapshot belongs. You can only specify either `DATABASE ` or `.`. | | snapshot\_name | Name of the data snapshot. Globally unique. | | repository\_name | Repository name. You can create a repository using [CREATE REPOSITORY](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/CREATE_REPOSITORY.md). | | ON | The object to be backed up. The whole database is backed up if this parameter is not specified. | | table\_name | Name of the table(s) to be backed up. | | mv\_name | Name of the materialized view(s) to be backed up. | | view\_name | Name of the logical view(s) to be backed up. | | udf\_name | Name of the UDF(s) to be backed up. | | PARTITION | Name of the partitions to be backed up. The whole table is backed up if this parameter is not specified. | | PROPERTIES | Properties of the data snapshot. Valid keys:- `type`: Backup type. Currently, only full backup `FULL` is supported. Default: `FULL`.
- `timeout`: Task timeout. Unit: second. Default: `86400`. | #### Examples[​](#examples "Direct link to Examples") ##### Examples with syntax compatible with earlier versions[​](#examples-with-syntax-compatible-with-earlier-versions "Direct link to Examples with syntax compatible with earlier versions") Example 1: Backs up the database `example_db` to the repository `example_repo`. ```sql BACKUP SNAPSHOT example_db.snapshot_label1 TO example_repo PROPERTIES ("type" = "full"); ``` Example 2: Backs up the table `example_tbl` in `example_db` to `example_repo`. ```sql BACKUP SNAPSHOT example_db.snapshot_label2 TO example_repo ON (example_tbl); ``` Example 2: Backs up the partitions `p1` and `p2` of `example_tbl` and the table `example_tbl2` in `example_db` to `example_repo`. ```sql BACKUP SNAPSHOT example_db.snapshot_label3 TO example_repo ON( example_tbl PARTITION (p1, p2), example_tbl2 ); ``` ##### Examples with syntax supported from v3.4.0 onwards[​](#examples-with-syntax-supported-from-v340-onwards "Direct link to Examples with syntax supported from v3.4.0 onwards") Example 1: Backs up a database. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_hub_backup TO test_repo; ``` Example 2: Backs up the table(s) in the database. ```sql -- Backs up one table. BACKUP DATABASE sr_hub SNAPSHOT sr_member_backup TO test_repo ON (TABLE sr_member); -- Backs up multiple tables. BACKUP DATABASE sr_hub SNAPSHOT sr_core_backup TO test_repo ON (TABLE sr_member, TABLE sr_pmc); -- Backs up all tables. BACKUP DATABASE sr_hub SNAPSHOT sr_all_backup TO test_repo ON (ALL TABLES); ``` Example 3: Backs up the partition(s) in a table. ```sql -- Backs up one partition. BACKUP DATABASE sr_hub SNAPSHOT sr_par_backup TO test_repo ON (TABLE sr_member PARTITION (p1)); -- Backs up multiple partitions. BACKUP DATABASE sr_hub SNAPSHOT sr_par_backup TO test_repo ON (TABLE sr_member PARTITION (p1,p2,p3)); ``` Example 4: Backs up the materialized view(s) in the database. ```sql -- Backs up one materialized view. BACKUP DATABASE sr_hub SNAPSHOT sr_mv1_backup TO test_repo ON (MATERIALIZED VIEW sr_mv1); -- Backs up multiple materialized views. BACKUP DATABASE sr_hub SNAPSHOT sr_mv2_backup TO test_repo ON (MATERIALIZED VIEW sr_mv1, MATERIALIZED VIEW sr_mv2); -- Backs up all materialized views. BACKUP DATABASE sr_hub SNAPSHOT sr_mv3_backup TO test_repo ON (ALL MATERIALIZED VIEWS); ``` Example 5: Backs up the logical view(s) in the database. ```sql -- Backs up one logical view. BACKUP DATABASE sr_hub SNAPSHOT sr_view1_backup TO test_repo ON (VIEW sr_view1); -- Backs up multiple logical views. BACKUP DATABASE sr_hub SNAPSHOT sr_view2_backup TO test_repo ON (VIEW sr_view1, VIEW sr_view2); -- Backs up all logical views. BACKUP DATABASE sr_hub SNAPSHOT sr_view3_backup TO test_repo ON (ALL VIEWS); ``` Example 6: Backs up the UDF(s) in the database. ```sql -- Backs up one UDF. BACKUP DATABASE sr_hub SNAPSHOT sr_udf1_backup TO test_repo ON (FUNCTION sr_udf1); -- Backs up multiple UDFs. BACKUP DATABASE sr_hub SNAPSHOT sr_udf2_backup TO test_repo ON (FUNCTION sr_udf1, FUNCTION sr_udf2); -- Backs up all UDFs. BACKUP DATABASE sr_hub SNAPSHOT sr_udf3_backup TO test_repo ON (ALL FUNCTIONS); ``` #### Usage notes[​](#usage-notes "Direct link to Usage notes") * Performing backup and restore operations on global, database, table, and partition levels requires different privileges. * In each database, only one running BACKUP or RESTORE job is allowed each time. Otherwise, StarRocks returns an error. * Because BACKUP and RESTORE jobs occupy many resources of your StarRocks cluster, you can back up and restore your data while your StarRocks cluster is not heavily loaded. * StarRocks does not support specifying data compression algorithms for data backup. * Because data is backed up as snapshots, the data loaded upon snapshot generation is not included in the snapshot. Therefore, if you load data into the old cluster after the snapshot is generated and before the RESTORE job is completed, you also need to load the data into the cluster that data is restored into. It is recommended that you load data into both clusters in parallel for a period of time after the data migration is complete, and then migrate your application to the new cluster after verifying the correctness of the data and services. * Before the RESTORE job is completed, you cannot operate the table to be restored. * Primary Key tables cannot be restored to a StarRocks cluster earlier than v2.5. * You do not need to create the table to be restored in the new cluster before restoring it. The RESTORE job automatically creates it. * If there is an existing table that has a duplicated name with the table to be restored, StarRocks first checks whether or not the schema of the existing table matches that of the table to be restored. If the schemas match, StarRocks overwrites the existing table with the data in the snapshot. If the schema does not match, the RESTORE job fails. You can either rename the table to be restored using the keyword `AS`, or delete the existing table before restoring data. * If the RESTORE job overwrites an existing database, table, or partition, the overwritten data cannot be restored after the job enters the COMMIT phase. If the RESTORE job fails or is canceled at this point, the data may be corrupted and inaccessible. In this case, you can only perform the RESTORE operation again and wait for the job to complete. Therefore, we recommend that you do not restore data by overwriting unless you are sure that the current data is no longer used. The overwrite operation first checks metadata consistency between the snapshot and the existing database, table, or partition. If an inconsistency is detected, the RESTORE operation cannot be performed. * Currently, StarRocks does not support backing up and restoring the configuration data related to user accounts, privileges, and resource groups. * Currently, StarRocks does not support backing up and restoring the Colocate Join relationship among tables. --- ### CANCEL BACKUP Cancels an ongoing BACKUP task in a specified database. #### Syntax[​](#syntax "Direct link to Syntax") ```sql CANCEL BACKUP { FROM | FOR EXTERNAL CATALOG } ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | -------------------- | ------------------------------------------------------------------ | | db\_name | Name of the database that the BACKUP task belongs to. | | FOR EXTERNAL CATALOG | Cancels the ongoing BACKUP task for the external catalog metadata. | #### Examples[​](#examples "Direct link to Examples") Example 1: Cancels the BACKUP task under the database `example_db`. ```sql CANCEL BACKUP FROM example_db; ``` --- ### CANCEL RESTORE Cancels an ongoing RESTORE task in a specified database. > **CAUTION** > > If a RESTORE task is canceled during the COMMIT phase, the restored data will be corrupted and inaccessible. In this case, you can only perform the RESTORE operation again and wait for the job to complete. #### Syntax[​](#syntax "Direct link to Syntax") ```sql CANCEL RESTORE { FROM | FOR EXTERNAL CATALOG } ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | -------------------- | ------------------------------------------------------------------- | | db\_name | Name of the database that the RESTORE task belongs to. | | FOR EXTERNAL CATALOG | Cancels the ongoing RESTORE task for the external catalog metadata. | #### Examples[​](#examples "Direct link to Examples") Example 1: Cancels the RESTORE task under the database `example_db`. ```sql CANCEL RESTORE FROM example_db; ``` --- ### CREATE REPOSITORY Creates a repository in a remote storage system that is used to store data snapshots for backing up and restoring data. tip For an overview of backup and restore please see the [backup and restore guide](https://docs.starrocks.io/docs/administration/management/Backup_and_restore.md). > **CAUTION** > > Only users with the ADMIN privilege can create a repository. For detailed instructions on deleting a repository, see [DROP REPOSITORY](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/DROP_REPOSITORY.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE [READ ONLY] REPOSITORY WITH BROKER ON LOCATION "" PROPERTIES ("key"="value", ...) ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | READ ONLY | Create a read-only repository. Note that you can only restore data from a read-only repository. When creating the same repository for two clusters to migrate data, you can create a read-only repository for the new cluster and only grant it RESTORE permissions. | | repository\_name | Repository name. For the naming conventions, see [System limits](https://docs.starrocks.io/docs/sql-reference/System_limit.md). | | repository\_location | Location of the repository in the remote storage system. | | PROPERTIES | The credential method for accessing the remote storage system. | **PROPERTIES**: StarRocks supports creating repositories in HDFS, AWS S3, and Google GCS. * For HDFS: * "username": The username of the account that you want to use to access the NameNode of the HDFS cluster. * "password": The password of the account that you want to use to access the NameNode of the HDFS cluster. * For AWS S3: * "aws.s3.use\_instance\_profile": Whether or not to allow instance profile and assumed role as credential methods for accessing AWS S3. Default: `false`. * If you use IAM user-based credential (Access Key and Secret Key) to access AWS S3, you don't need to specify this parameter, and you need to specify "aws.s3.access\_key", "aws.s3.secret\_key", and "aws.s3.region". * If you use Instance Profile to access AWS S3, you need to set this parameter to `true` and specify "aws.s3.region". * If you use Assumed Role to access AWS S3, you need to set this parameter to `true` and specify "aws.s3.iam\_role\_arn" and "aws.s3.region". * "aws.s3.access\_key": The Access Key ID that you can use to access the Amazon S3 bucket. * "aws.s3.secret\_key": The Secret Access Key that you can use to access the Amazon S3 bucket. * "aws.s3.iam\_role\_arn": The ARN of the IAM role that has privileges on the AWS S3 bucket in which your data files are stored. If you want to use assumed role as the credential method for accessing AWS S3, you must specify this parameter. Then, StarRocks will assume this role when it analyzes your Hive data by using a Hive catalog. * "aws.s3.region": The region in which your AWS S3 bucket resides. Example: `us-west-1`. > **NOTE** > > StarRocks supports creating repositories in AWS S3 only according to the S3A protocol. Therefore, when you create repositories in AWS S3, you must replace `s3://` in the S3 URI you pass as a repository location in `ON LOCATION` with `s3a://`. * For Google GCS: * "fs.s3a.access.key": The Access Key that you can use to access the Google GCS bucket. * "fs.s3a.secret.key": The Secret Key that you can use to access the Google GCS bucket. * "fs.s3a.endpoint": The endpoint that you can use to access the Google GCS bucket. Do not specify `https` in the endpoint address. > **NOTE** > > StarRocks supports creating repositories in Google GCS only according to the S3A protocol. Therefore, when you create repositories in Google GCS, you must replace the prefix in the GCS URI you pass as a repository location in `ON LOCATION` with `s3a://`. * For MinIO: * "aws.s3.access\_key": The Access Key that you can use to access the MinIO bucket. * "aws.s3.secret\_key": The Secret Key that you can use to access the MinIO bucket. * "aws.s3.endpoint": The endpoint that you can use to access the MinIO bucket. #### Examples[​](#examples "Direct link to Examples") Example 1: Create a repository named `hdfs_repo` in an Apache™ Hadoop® cluster. ```sql CREATE REPOSITORY hdfs_repo WITH BROKER ON LOCATION "hdfs://x.x.x.x:yyyy/repo_dir/backup" PROPERTIES( "username" = "xxxxxxxx", "password" = "yyyyyyyy" ); ``` Example 2: Create a read-only repository named `s3_repo` in the Amazon S3 bucket `bucket_s3`. ```sql CREATE READ ONLY REPOSITORY s3_repo WITH BROKER ON LOCATION "s3a://bucket_s3/backup" PROPERTIES( "aws.s3.access_key" = "XXXXXXXXXXXXXXXXX", "aws.s3.secret_key" = "yyyyyyyyyyyyyyyyy", "aws.s3.region" = "us-east-1" ); ``` Example 3: Create a repository named `gcs_repo` in the Google GCS bucket `bucket_gcs`. ```sql CREATE REPOSITORY gcs_repo WITH BROKER ON LOCATION "s3a://bucket_gcs/backup" PROPERTIES( "fs.s3a.access.key" = "xxxxxxxxxxxxxxxxxxxx", "fs.s3a.secret.key" = "yyyyyyyyyyyyyyyyyyyy", "fs.s3a.endpoint" = "storage.googleapis.com" ); ``` Example 4: Create a repository named `minio_repo` in the MinIO bucket `bucket_minio`. ```sql CREATE REPOSITORY minio_repo WITH BROKER ON LOCATION "s3://bucket_minio/backup" PROPERTIES( "aws.s3.access_key" = "XXXXXXXXXXXXXXXXX", "aws.s3.secret_key" = "yyyyyyyyyyyyyyyyy", "aws.s3.endpoint" = "http://minio:9000" ); ``` Example 5: Create a repository named `kerberos_repo` in an Apache™ Hadoop® cluster with Kerberos enabled. ```sql CREATE REPOSITORY kerberos_repo WITH BROKER ON LOCATION "hdfs://nameservices1/user/hive/backup" PROPERTIES( "hadoop.security.authentication" = "kerberos", "kerberos_principal" = "sr/cs01.starrocks.com@STARROCKS.COM", "kerberos_keytab" = "/home/disk1/user1/sr.keytab" ); ``` --- ### DROP REPOSITORY Deletes a repository. Repositories are used to store data snapshots for data backup and restoration. > **CAUTION** > > * Only root user and superuser can delete a repository. > * This operation only deletes the mapping of the repository in StarRocks, and does not delete the actual data. You need to delete it manually in the remote storage system. After deletion, you can map to that repository again by specifying the same remote storage system path. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DROP REPOSITORY ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | ---------------- | ------------------------------------- | | repository\_name | Name of the repository to be deleted. | #### Example[​](#example "Direct link to Example") Example 1: deletes a repository named `oss_repo`. ```sql DROP REPOSITORY `oss_repo`; ``` --- ### RECOVER Recovers a database, table, or partition that was dropped by using the DROP command. The dropped database, table, or partition can be recovered within the period specified by the FE parameter `catalog_trash_expire_second` (1 day by default). Data deleted by using [TRUNCATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/TRUNCATE_TABLE.md) cannot be recovered. #### Syntax[​](#syntax "Direct link to Syntax") 1. Recover a database. ```sql RECOVER DATABASE ``` 2. Recover a table. ```sql RECOVER TABLE [.] ``` 3. Recover a partition. ```sql RECOVER PARTITION FROM [.] ``` Note: 1. This command can only recover metadata deleted some time ago (1 day by default). You can change the duration by adjusting the FE parameter `catalog_trash_expire_second`. 2. If the metadata is deleted with identical metadata created, the previous one will not be recovered. #### Examples[​](#examples "Direct link to Examples") 1. Recover database `example_db`. ```sql RECOVER DATABASE example_db; ``` 2. Recover table `example_tbl`. ```sql RECOVER TABLE example_db.example_tbl; ``` 3. Recover partition `p1` in the `example_tbl` table. ```sql RECOVER PARTITION p1 FROM example_tbl; ``` --- ### RESTORE StarRocks supports backing up and restoring the following objects: * Internal databases, tables (of all types and partitioning strategies), and partitions * Metadata of external catalogs (supported from v3.4.0 onwards) * Synchronous materialized views and asynchronous materialized views * Logical views (supported from v3.4.0 onwards) * User-defined functions (supported from v3.4.0 onwards) tip For an overview of backup and restore please see the [backup and restore guide](https://docs.starrocks.io/docs/administration/management/Backup_and_restore.md). RESTORE is an asynchronous operation. You can check the status of a RESTORE job using [SHOW RESTORE](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_RESTORE.md), or cancel a RESTORE job using [CANCEL RESTORE](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/CANCEL_RESTORE.md). > **CAUTION** > > * Shared-data StarRocks clusters do not support data BACKUP and RESTORE. > * In each database, only one running BACKUP or RESTORE job is allowed each time. Otherwise, StarRocks returns an error. #### Privilege requirement[​](#privilege-requirement "Direct link to Privilege requirement") In versions earlier than v3.0, users with the `admin_priv` privilege can perform this operation. In v3.0 and later versions, to back up a specific object, users must have the REPOSITORY privilege at the System level and the EXPORT privilege for the corresponding table or all tables under the corresponding database. For example: * Grant the role permission to export data from the specified table. ```sql GRANT REPOSITORY ON SYSTEM TO ROLE backup_tbl; GRANT EXPORT ON TABLE TO ROLE backup_tbl; ``` * Grant the role permission to export data from all tables under the specified data. ```sql GRANT REPOSITORY ON SYSTEM TO ROLE backup_db; GRANT EXPORT ON ALL TABLES IN DATABASE TO ROLE backup_db; ``` * Grant the role permission to export data from all tables in all databases. ```sql GRANT REPOSITORY ON SYSTEM TO ROLE backup; GRANT EXPORT ON ALL TABLES IN ALL DATABASES TO ROLE backup; ``` #### Syntax compatible with earlier versions[​](#syntax-compatible-with-earlier-versions "Direct link to Syntax compatible with earlier versions") ```sql RESTORE SNAPSHOT . FROM [ ON ( [ PARTITION ( [, ...] ) ] [ AS ] [, ...] ) ] PROPERTIES ("key"="value", ...) ``` ##### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | db\_name | Name of the database that the data is restored to. | | snapshot\_name | Name for the data snapshot. | | repository\_name | Repository name. | | ON | Name of the tables to restore. The whole database is restored if this parameter is not specified. | | PARTITION | Name of the partitions to be restored. The whole table is restored if this parameter is not specified. You can view the partition name using [SHOW PARTITIONS](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md). | | PROPERTIES | Properties of the RESTORE operation. Valid keys:- `backup_timestamp`: Backup timestamp. **Required**. You can view backup timestamps using [SHOW SNAPSHOT](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_SNAPSHOT.md).
- `replication_num`: Specify the number of replicas to be restored. Default: `3`.
- `meta_version`: This parameter is only used as a temporary solution to restore the data backed up by the earlier version of StarRocks. The latest version of the backed up data already contains `meta version`, and you do not need to specify it.
- `timeout`: Task timeout. Unit: second. Default: `86400`. | #### Syntax supported from v3.4.0 onwards[​](#syntax-supported-from-v340-onwards "Direct link to Syntax supported from v3.4.0 onwards") ```sql -- Restore external catalog metadata. RESTORE SNAPSHOT [.] FROM { ALL EXTERNAL CATALOGS | EXTERNAL CATALOG[S] [, EXTERNAL CATALOG[S] ...] [ AS ] } [ DATABASE [AS ] ] [ PROPERTIES ("key"="value" [, ...] ) ] -- Restore databases, tables, partitions, materialized views, logical views, or UDFs. RESTORE SNAPSHOT [.] FROM [ DATABASE [AS ] ] [ ON ( restore_object [, ...] ) ] [ PROPERTIES ("key"="value" [, ...] ) ] restore_object ::= [ { ALL TABLE[S] | TABLE[S] [, TABLE[S] ...] [AS ] } | { ALL MATERIALIZED VIEW[S] | MATERIALIZED VIEW[S] [, MATERIALIZED VIEW[S] ...] [AS ] } | { ALL VIEW[S] | VIEW[S] [, VIEW[S] ...] [AS ] } | { ALL FUNCTION[S] | FUNCTION[S] [, FUNCTION[S] ...] [AS ] } | PARTITION ([, ...]) [AS ] ] ``` ##### Parameters[​](#parameters-1 "Direct link to Parameters") | **Parameter** | **Description** | | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | db\_name. | Name of the database to which the object(s) or snapshot is restored in the target cluster. If the database does not exist, the system will create it. You can only specify either `AS ` or `.`. | | snapshot\_name | Name of the data snapshot. | | repository\_name | Repository name. | | ALL EXTERNAL CATALOGS | Restores the metadata of all external catalogs. | | catalog\_name | Name of the external catalog(s) that need to be restored. | | DATABASE db\_name\_in\_snapshot | Name of the database to which the object(s) or snapshot belongs when it was backed up in the source cluster. | | AS target\_db | Name of the database to which the object(s) or snapshot is restored in the target cluster. If the database does not exist, the system will create it. You can only specify either `AS ` or `.`. | | ON | The object to be restored. The whole database is restored if this parameter is not specified. | | table\_name | Name of the table(s) to be restored. | | mv\_name | Name of the materialized view(s) to be restored. | | view\_name | Name of the logical view(s) to be restored. | | udf\_name | Name of the UDF(s) to be restored. | | PARTITION | Name of the partitions to be restored. The whole table is restored if this parameter is not specified. | | AS alias | Sets a new name for the object to be restored in the target cluster. | | PROPERTIES | Properties of the RESTORE operation. Valid keys:- `backup_timestamp`: Backup timestamp. **Required**. You can view backup timestamps using [SHOW SNAPSHOT](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_SNAPSHOT.md).
- `replication_num`: Specify the number of replicas to be restored. Default: `3`.
- `meta_version`: This parameter is only used as a temporary solution to restore the data backed up by the earlier version of StarRocks. The latest version of the backed up data already contains `meta version`, and you do not need to specify it.
- `timeout`: Task timeout. Unit: second. Default: `86400`. | #### Examples[​](#examples "Direct link to Examples") ##### Examples with syntax compatible with earlier versions[​](#examples-with-syntax-compatible-with-earlier-versions "Direct link to Examples with syntax compatible with earlier versions") Example 1: Restores the table `backup_tbl` in the snapshot `snapshot_label1` from the `example_repo` repository to the database `example_db`, and the backup timestamp is `2018-05-04-16-45-08`. Restores one replica. ```sql RESTORE SNAPSHOT example_db.snapshot_label1 FROM example_repo ON ( backup_tbl ) PROPERTIES ( "backup_timestamp"="2018-05-04-16-45-08", "replication_num" = "1" ); ``` Example 2: Restores partitions `p1` and `p2` of table `backup_tbl` in `snapshot_label2` and table `backup_tbl2` from `example_repo` to database `example_db`, and rename `backup_tbl2` to `new_tbl`. The backup timestamp is `2018-05-04-17-11-01`. Restores three replicas by default. ```sql RESTORE SNAPSHOT example_db.snapshot_label2 FROM example_repo ON( backup_tbl PARTITION (p1, p2), backup_tbl2 AS new_tbl ) PROPERTIES ( "backup_timestamp"="2018-05-04-17-11-01" ); ``` ##### Examples with syntax supported from v3.4.0 onwards[​](#examples-with-syntax-supported-from-v340-onwards "Direct link to Examples with syntax supported from v3.4.0 onwards") Example 1: Restores a database. ```sql -- Restores the database with its original name. RESTORE SNAPSHOT sr_hub_backup FROM test_repo DATABASE sr_hub PROPERTIES("backup_timestamp" = "2024-12-09-10-25-58-842"); -- Restores the database with a new name. RESTORE SNAPSHOT sr_hub_backup FROM test_repo DATABASE sr_hub AS sr_hub_new PROPERTIES("backup_timestamp" = "2024-12-09-10-25-58-842"); ``` Example 2: Restores the table(s) in the database. ```sql -- Restores one table with its original name. RESTORE SNAPSHOT sr_member_backup FROM test_repo DATABASE sr_hub ON (TABLE sr_member) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Restores one table with a new name. RESTORE SNAPSHOT sr_member_backup FROM test_repo DATABASE sr_hub AS sr_hub_new ON (TABLE sr_member AS sr_member_new) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Restores multiple tables. RESTORE SNAPSHOT sr_core_backup FROM test_repo DATABASE sr_hub ON (TABLE sr_member, TABLE sr_pmc) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Restores all tables. RESTORE SNAPSHOT sr_all_backup FROM test_repo DATABASE sr_hub ON (ALL TABLES); -- Restores one from all tables. RESTORE SNAPSHOT sr_all_backup FROM test_repo DATABASE sr_hub ON (TABLE sr_member) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` Example 3: Restores the partition(s) in a table. ```sql -- Restores one partition. RESTORE SNAPSHOT sr_par_backup FROM test_repo DATABASE sr_hub ON (TABLE sr_member PARTITION (p1)) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Restores multiple partitions. RESTORE SNAPSHOT sr_par_backup FROM test_repo DATABASE sr_hub ON (TABLE sr_member PARTITION (p1,p2)) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` Example 4: Restores the materialized view(s) in the database. ```sql -- Restores one materialized view. RESTORE SNAPSHOT sr_mv1_backup FROM test_repo DATABASE sr_hub ON (MATERIALIZED VIEW sr_mv1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Restores multiple materialized views. RESTORE SNAPSHOT sr_mv2_backup FROM test_repo DATABASE sr_hub ON (MATERIALIZED VIEW sr_mv1, MATERIALIZED VIEW sr_mv2) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Restores all materialized views. RESTORE SNAPSHOT sr_mv3_backup FROM test_repo DATABASE sr_hub ON (ALL MATERIALIZED VIEWS) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Restores one of all materialized views. RESTORE SNAPSHOT sr_mv3_backup FROM test_repo DATABASE sr_hub ON (MATERIALIZED VIEW sr_mv1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` Example 5: Restores the logical view(s) in the database. ```sql -- Restores one logical view. RESTORE SNAPSHOT sr_view1_backup FROM test_repo DATABASE sr_hub ON (VIEW sr_view1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Restores multiple logical views. RESTORE SNAPSHOT sr_view2_backup FROM test_repo DATABASE sr_hub ON (VIEW sr_view1, VIEW sr_view2) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Restores all logical views. RESTORE SNAPSHOT sr_view3_backup FROM test_repo DATABASE sr_hub ON (ALL VIEWS) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Restores one of all logical views. RESTORE SNAPSHOT sr_view3_backup FROM test_repo DATABASE sr_hub ON (VIEW sr_view1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` Example 6: Restores the UDF(s) in the database. ```sql -- Restores one UDF. RESTORE SNAPSHOT sr_udf1_backup FROM test_repo DATABASE sr_hub ON (FUNCTION sr_udf1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Restores multiple UDFs. RESTORE SNAPSHOT sr_udf2_backup FROM test_repo DATABASE sr_hub ON (FUNCTION sr_udf1, FUNCTION sr_udf2) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Restores all UDFs. RESTORE SNAPSHOT sr_udf3_backup FROM test_repo DATABASE sr_hub ON (ALL FUNCTIONS) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Restores one of all UDFs. RESTORE SNAPSHOT sr_udf3_backup FROM test_repo DATABASE sr_hub ON (FUNCTION sr_udf1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` #### Usage notes[​](#usage-notes "Direct link to Usage notes") * Performing backup and restore operations on global, database, table, and partition levels requires different privileges. * In each database, only one running BACKUP or RESTORE job is allowed each time. Otherwise, StarRocks returns an error. * Because BACKUP and RESTORE jobs occupy many resources of your StarRocks cluster, you can back up and restore your data while your StarRocks cluster is not heavily loaded. * StarRocks does not support specifying data compression algorithms for data backup. * Because data is backed up as snapshots, the data loaded upon snapshot generation is not included in the snapshot. Therefore, if you load data into the old cluster after the snapshot is generated and before the RESTORE job is completed, you also need to load the data into the cluster that data is restored into. It is recommended that you load data into both clusters in parallel for a period of time after the data migration is complete, and then migrate your application to the new cluster after verifying the correctness of the data and services. * Before the RESTORE job is completed, you cannot operate the table to be restored. * Primary Key tables cannot be restored to a StarRocks cluster earlier than v2.5. * You do not need to create the table to be restored in the new cluster before restoring it. The RESTORE job automatically creates it. * If there is an existing table that has a duplicated name with the table to be restored, StarRocks first checks whether or not the schema of the existing table matches that of the table to be restored. If the schemas match, StarRocks overwrites the existing table with the data in the snapshot. If the schema does not match, the RESTORE job fails. You can either rename the table to be restored using the keyword `AS`, or delete the existing table before restoring data. * If the RESTORE job overwrites an existing database, table, or partition, the overwritten data cannot be restored after the job enters the COMMIT phase. If the RESTORE job fails or is canceled at this point, the data may be corrupted and inaccessible. In this case, you can only perform the RESTORE operation again and wait for the job to complete. Therefore, we recommend that you do not restore data by overwriting unless you are sure that the current data is no longer used. The overwrite operation first checks metadata consistency between the snapshot and the existing database, table, or partition. If an inconsistency is detected, the RESTORE operation cannot be performed. * Currently, StarRocks does not support backing up and restoring the configuration data related to user accounts, privileges, and resource groups. * Currently, StarRocks does not support backing up and restoring the Colocate Join relationship among tables. --- ### SHOW BACKUP Views the last BACKUP task in a specified database. > **NOTE** > > Only the information of the last BACKUP task is saved in StarRocks. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW BACKUP [FROM ] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | ------------- | ----------------------------------------------------- | | db\_name | Name of the database that the BACKUP task belongs to. | #### Return[​](#return "Direct link to Return") | **Return** | **Description** | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | JobId | Unique job ID. | | SnapshotName | Name of the data snapshot. | | DbName | Name of the database that the BACKUP task belongs to. | | State | Current state of the BACKUP task:- PENDING: Initial state after submitting a job.
- SNAPSHOTING: Creating snapshot.
- UPLOAD\_SNAPSHOT: Snapshot complete, ready for upload.
- UPLOADING: Uploading snapshot.
- SAVE\_META: Creating local metadata files.
- UPLOAD\_INFO: Uploading metadata files and information of the BACKUP task.
- FINISHED: BACKUP task finished.
- CANCELLED: BACKUP task failed or cancelled. | | BackupObjs | Backed up objects. | | CreateTime | Task submission time. | | SnapshotFinishedTime | Snapshot completion time. | | UploadFinishedTime | Snapshot upload completion time. | | FinishedTime | Task completion Time. | | UnfinishedTasks | Unfinished subtask IDs in the SNAPSHOTING and UPLOADING phases. | | Progress | The progress of snapshot uploading tasks. | | TaskErrMsg | Error messages. | | Status | Status information. | | Timeout | Task timeout. Unit: second. | #### Examples[​](#examples "Direct link to Examples") Example 1: Views the last BACKUP task in the database `example_db`. ```sql SHOW BACKUP FROM example_db; ``` --- ### SHOW REPOSITORIES Views the repositories created in StarRocks. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW REPOSITORIES ``` #### Return[​](#return "Direct link to Return") | **Return** | **Description** | | ---------- | -------------------------------------------------------- | | RepoId | Repository ID. | | RepoName | Repository name. | | CreateTime | Repository creation time. | | IsReadOnly | If the repository is read-only. | | Location | Location of the repository in the remote storage system. | | Broker | Broker used to create the repository. | | ErrMsg | Error message during connection to the repository. | #### Examples[​](#examples "Direct link to Examples") Example 1: Views the repositories created in StarRocks. ```sql SHOW REPOSITORIES; ``` --- ### SHOW RESTORE Views the last RESTORE task in a specified database. > **NOTE** > > Only the information of the last RESTORE task is saved in StarRocks. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW RESTORE [FROM ] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | ------------- | ------------------------------------------------------ | | db\_name | Name of the database that the RESTORE task belongs to. | #### Return[​](#return "Direct link to Return") | **Return** | **Description** | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | JobId | Unique job ID. | | Label | Name of the data snapshot. | | Timestamp | Backup timestamp. | | DbName | Name of the database that the RESTORE task belongs to. | | State | Current state of the RESTORE task:- PENDING: Initial state after submitting a job.
- SNAPSHOTING: Executing the local snapshot.
- DOWNLOAD: Submitting snapshot download task.
- DOWNLOADING: Downloading the snapshot.
- COMMIT: To commit the downloaded snapshot.
- COMMITTING: Committing the downloaded snapshot.
- FINISHED: RESTORE task finished.
- CANCELLED: RESTORE task failed or cancelled. | | AllowLoad | If loading data is allowed during the RESTORE task. | | ReplicationNum | Number of replicas to be restored. | | RestoreObjs | The restored objects (tables and partitions). | | CreateTime | Task submission time. | | MetaPreparedTime | Local metadata completion time. | | SnapshotFinishedTime | Snapshot completion time. | | DownloadFinishedTime | Snapshot download completion time. | | FinishedTime | Task completion Time. | | UnfinishedTasks | Unfinished subtask IDs in the SNAPSHOTTING, DOWNLOADING, and COMMITTING phases. | | Progress | The progress of snapshot downloading tasks. | | TaskErrMsg | Error messages. | | Status | Status information. | | Timeout | Task timeout. Unit: second. | #### Examples[​](#examples "Direct link to Examples") Example 1: Views the last RESTORE task in the database `example_db`. ```sql SHOW RESTORE FROM example_db; ``` --- ### SHOW SNAPSHOT Views data snapshots in a specified repository. For more information, see [data backup and restoration](https://docs.starrocks.io/docs/administration/management/Backup_and_restore.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW SNAPSHOT ON [WHERE SNAPSHOT = [AND TIMESTAMP = ]] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | ----------------- | ---------------------------------------------------- | | repo\_name | Name of the repository that the snapshot belongs to. | | snapshot\_nam | Name of the snapshot. | | backup\_timestamp | Backup timestamp of the snapshot. | #### Return[​](#return "Direct link to Return") | **Return** | **Description** | | ---------- | ------------------------------------------------------------------------------------------ | | Snapshot | Name of the snapshot. | | Timestamp | Backup timestamp of the snapshot. | | Status | Displays `OK` if the snapshot is okay. Displays error message if the snapshot is not okay. | | Database | Name of the database that the snapshot belongs to. | | Details | JSON-formatted directory and structure of the snapshot. | #### Example[​](#example "Direct link to Example") Example 1: Views snapshots in repository `example_repo`. ```sql SHOW SNAPSHOT ON example_repo; ``` Example 2: Views the snapshot named `backup1` in repository `example_repo`. ```sql SHOW SNAPSHOT ON example_repo WHERE SNAPSHOT = "backup1"; ``` Example 3: Views the snapshot named `backup1` with backup timestamp `2018-05-05-15-34-26` in repository `example_repo`. ```sql SHOW SNAPSHOT ON example_repo WHERE SNAPSHOT = "backup1" AND TIMESTAMP = "2018-05-05-15-34-26"; ``` --- ### Catalog #### [📄️ CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) [CREATE EXTERNAL CATALOG creates an external catalog.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) #### [📄️ DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) [DROP CATALOG deletes an external catalog.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) #### [📄️ SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) [Switches to a specified catalog in the current session.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) #### [📄️ SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) [SHOW CATALOGS queries all catalogs in the current StarRocks cluster, including the internal catalog and external catalogs.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) #### [📄️ SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) [SHOW CREATE CATALOG queries the creation statement of an external catalog.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) --- ### CREATE EXTERNAL CATALOG CREATE EXTERNAL CATALOG creates an external catalog. You can use external catalogs to query data in external data sources without loading data into StarRocks or creating external tables. Currently, you can create the following types of external catalogs: * [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md): used for querying data from Apache Hive™. * [Iceberg catalog](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md): used for querying data from Apache Iceberg. * [Hudi catalog](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md): used for querying data from Apache Hudi. * [Delta Lake catalog](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md): used to query data from Delta Lake. * [JDBC catalog](https://docs.starrocks.io/docs/data_source/catalog/jdbc_catalog.md): used to query data from JDBC-compatible data sources. * [Elasticsearch catalog](https://docs.starrocks.io/docs/data_source/catalog/elasticsearch_catalog.md): used to query data from Elasticsearch. Elasticsearch catalogs are supported from v3.1 onwards. * [Paimon catalog](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog.md): used to query data from Apache Paimon. Paimon catalogs are supported from v3.1 onwards. * [Unified catalog](https://docs.starrocks.io/docs/data_source/catalog/unified_catalog.md): used to query data from from Hive, Iceberg, Hudi, and Delta Lake data sources as a unified data source. Unified catalogs are supported from v3.2 onwards. > **NOTE** > > * In v3.0 and later, this statement requires SYSTEM-level CREATE EXTERNAL CATALOG privilege. > * Before you create external catalogs, configure your StarRocks cluster to meet the requirements of the data storage system (such as Amazon S3), metadata service (such as Hive metastore), and authenticating service (such as Kerberos) of external data sources. For more information, see the "Before you begin" section in each [external catalog topic](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [IF NOT EXISTS] [COMMENT ] PROPERTIES ("key"="value", ...) ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | catalog\_name | Yes | The name of the external catalog. For the naming conventions, see [System limits](https://docs.starrocks.io/docs/sql-reference/System_limit.md). | | comment | No | The description of the external catalog. | | PROPERTIES | Yes | The properties of an external catalog. Configure properties based on the types of external catalogs. For more information, see [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md), [Iceberg catalog](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md), [Hudi catalog](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md), [Delta Lake catalog](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md), and [JDBC Catalog](https://docs.starrocks.io/docs/data_source/catalog/jdbc_catalog.md). | #### Examples[​](#examples "Direct link to Examples") Example 1: Create a Hive catalog named `hive_metastore_catalog`. The corresponding Hive cluster uses Hive metastore as its metadata service. ```sql CREATE EXTERNAL CATALOG hive_metastore_catalog COMMENT "External catalog to Hive" PROPERTIES( "type"="hive", "hive.metastore.uris"="thrift://xx.xx.xx.xx:9083" ); ``` Example 2: Create a Hive catalog named `hive_glue_catalog`. The corresponding Hive cluster uses AWS Glue as its metadata service. ```sql CREATE EXTERNAL CATALOG hive_glue_catalog COMMENT "External catalog to Hive" PROPERTIES( "type"="hive", "hive.metastore.type"="glue", "aws.hive.metastore.glue.aws-access-key"="xxxxxx", "aws.hive.metastore.glue.aws-secret-key"="xxxxxxxxxxxx", "aws.hive.metastore.glue.endpoint"="https://glue.x-x-x.amazonaws.com" ); ``` Example 3: Create an Iceberg catalog named `iceberg_metastore_catalog`. The corresponding Iceberg cluster uses Hive metastore as its metadata service. ```sql CREATE EXTERNAL CATALOG iceberg_metastore_catalog COMMENT "External catalog to Iceberg" PROPERTIES( "type"="iceberg", "iceberg.catalog.type"="hive", "iceberg.catalog.hive.metastore.uris"="thrift://xx.xx.xx.xx:9083" ); ``` Example 4: Create an Iceberg catalog named `iceberg_glue_catalog`. The corresponding Iceberg cluster uses AWS Glue as its metadata service. ```sql CREATE EXTERNAL CATALOG iceberg_glue_catalog COMMENT "External catalog to Iceberg" PROPERTIES( "type"="iceberg", "iceberg.catalog.type"="glue", "aws.hive.metastore.glue.aws-access-key"="xxxxx", "aws.hive.metastore.glue.aws-secret-key"="xxxxxxxxxxxx", "aws.hive.metastore.glue.endpoint"="https://glue.x-x-x.amazonaws.com" ); ``` Example 5: Create a Hudi catalog named `hudi_metastore_catalog`. The corresponding Hudi cluster uses Hive metastore as its metadata service. ```sql CREATE EXTERNAL CATALOG hudi_metastore_catalog COMMENT "External catalog to Hudi" PROPERTIES( "type"="hudi", "hive.metastore.uris"="thrift://xx.xx.xx.xx:9083" ); ``` Example 6: Create a Hudi catalog named `hudi_glue_catalog`. The corresponding Hudi cluster uses AWS Glue as its metadata service. ```sql CREATE EXTERNAL CATALOG hudi_glue_catalog COMMENT "External catalog to Hudi" PROPERTIES( "type"="hudi", "hive.metastore.type"="glue", "aws.hive.metastore.glue.aws-access-key"="xxxxxx", "aws.hive.metastore.glue.aws-secret-key"="xxxxxxxxxxxx", "aws.hive.metastore.glue.endpoint"="https://glue.x-x-x.amazonaws.com" ); ``` Example 7: Create a Delta Lake catalog named `delta_metastore_catalog`. The corresponding Delta Lake service uses Hive metastore as its metadata service. ```sql CREATE EXTERNAL CATALOG delta_metastore_catalog COMMENT "External catalog to Delta" PROPERTIES( "type"="deltalake", "hive.metastore.uris"="thrift://xx.xx.xx.xx:9083" ); ``` Example 8: Create a Delta Lake catalog named `delta_glue_catalog`. The corresponding Delta Lake service uses AWS Glue as its metadata service. ```sql CREATE EXTERNAL CATALOG delta_glue_catalog COMMENT "External catalog to Delta" PROPERTIES( "type"="deltalake", "hive.metastore.type"="glue", "aws.hive.metastore.glue.aws-access-key"="xxxxxx", "aws.hive.metastore.glue.aws-secret-key"="xxxxxxxxxxxx", "aws.hive.metastore.glue.endpoint"="https://glue.x-x-x.amazonaws.com" ); ``` #### References[​](#references "Direct link to References") * To view all catalogs in your StarRocks cluster, see [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md). * To view the creation statement of an external catalog, see [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md). * To delete an external catalog from your StarRocks cluster, see [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md). --- ### DROP CATALOG DROP CATALOG deletes an external catalog. The internal catalog cannot be deleted. A StarRocks cluster has only one internal catalog named `default_catalog`. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DROP CATALOG [IF EXISTS] ``` #### Parameters[​](#parameters "Direct link to Parameters") `catalog_name`: The name of an external catalog. #### Examples[​](#examples "Direct link to Examples") Create a Hive catalog named `hive1`. ```sql CREATE EXTERNAL CATALOG hive1 PROPERTIES( "type"="hive", "hive.metastore.uris"="thrift://xx.xx.xx.xx:9083" ); ``` Delete the Hive catalog. ```sql DROP CATALOG hive1; ``` --- ### SET CATALOG Switches to a specified catalog in the current session. This command is supported from v3.0 onwards. > **NOTE** > > For a newly deployed StarRocks v3.1 cluster, you must have the USAGE privilege on the destination external catalog if you want to run SET CATALOG to switch to that catalog. You can use [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the required privileges. For a v3.1 cluster upgraded from an earlier version, you can run SET CATALOG with the inherited privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SET CATALOG ``` #### Parameter[​](#parameter "Direct link to Parameter") `catalog_name`: the name of the catalog to use in the current session. You can switch to an internal or external catalog. If the catalog that you specify does not exist, an exception is thrown. #### Examples[​](#examples "Direct link to Examples") Run the following command to switch to a Hive catalog named `hive_metastore` in the current session: ```sql SET CATALOG hive_metastore; ``` Run the following command to switch to the internal catalog `default_catalog` in the current session: ```sql SET CATALOG default_catalog; ``` --- ### SHOW CATALOGS SHOW CATALOGS queries all catalogs in the current StarRocks cluster, including the internal catalog and external catalogs. > **NOTE** > > SHOW CATALOGS returns external catalogs to users who have the USAGE privilege on that external catalog. If users or roles do not have this privilege on any external catalog, this command returns only the default\_catalog. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW CATALOGS [LIKE ''] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | pattern | Optional. The pattern used to match catalog names by using the `LIKE` clause. `%` matches any number of characters and `_` matches a single character. Only the catalogs whose names match the pattern are returned. | #### Output[​](#output "Direct link to Output") ```sql +----------+--------+----------+ | Catalog | Type | Comment | +----------+--------+----------+ ``` The following table describes the fields returned by this statement. | **Field** | **Description** | | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Catalog | The catalog name. | | Type | The catalog type. `Internal` is returned if the catalog is `default_catalog`. The corresponding catalog type is returned if the catalog is an external catalog, such as `Hive`, `Hudi`, or `Iceberg`. | | Comment | The comments of a catalog. StarRocks does not support adding comments to an external catalog. Therefore, the value is `NULL` for an external catalog. If the catalog is `default_catalog`, the comment is `An internal catalog contains this cluster's self-managed tables.` by default. `default_catalog` is the only internal catalog in a StarRocks cluster. | #### Examples[​](#examples "Direct link to Examples") Query all catalogs in the current cluster. ```sql SHOW CATALOGS\G *************************** 1. row *************************** Catalog: default_catalog Type: Internal Comment: An internal catalog contains this cluster's self-managed tables. *************************** 2. row *************************** Catalog: hudi_catalog Type: Hudi Comment: NULL *************************** 3. row *************************** Catalog: iceberg_catalog Type: Iceberg Comment: NULL ``` Query the catalogs whose names match a specific pattern. ```sql SHOW CATALOGS LIKE 'iceberg%'\G *************************** 1. row *************************** Catalog: iceberg_catalog Type: Iceberg Comment: NULL ``` --- ### SHOW CREATE CATALOG SHOW CREATE CATALOG queries the creation statement of an [external catalog](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md). Note that authentication-related information in the return result will be anonymized. This command is supported from v3.0 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW CREATE CATALOG ; ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | ------------------------------------------------------------------ | | catalog\_name | Yes | The name of the catalog whose creation statement you want to view. | #### Return result[​](#return-result "Direct link to Return result") ```plain +------------+-----------------+ | Catalog | Create Catalog | +------------+-----------------+ ``` | **Field** | **Description** | | -------------- | ------------------------------------------------------ | | Catalog | The name of the catalog. | | Create Catalog | The statement that was executed to create the catalog. | #### Examples[​](#examples "Direct link to Examples") The following example queries the creation statement of a Hive catalog named `hive_catalog_hms`: ```sql SHOW CREATE CATALOG hive_catalog_hms; ``` The return result is as follows: ```sql CREATE EXTERNAL CATALOG `hive_catalog_hms` PROPERTIES ("aws.s3.access_key" = "AK******M4", "hive.metastore.type" = "glue", "aws.s3.secret_key" = "iV******iD", "aws.glue.secret_key" = "iV******iD", "aws.s3.use_instance_profile" = "false", "aws.s3.region" = "us-west-1", "aws.glue.region" = "us-west-1", "type" = "hive", "aws.glue.access_key" = "AK******M4" ) ``` --- ### CBO statistics #### [📄️ ANALYZE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/ANALYZE_TABLE.md) [ANALYZE TABLE creates a manual collection task for collecting CBO statistics.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/ANALYZE_TABLE.md) #### [📄️ CREATE ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/CREATE_ANALYZE.md) [CREATE ANALYZE customizes an automatic collection task for collecting CBO statistics.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/CREATE_ANALYZE.md) #### [📄️ DROP ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/DROP_ANALYZE.md) [DROP ANALYZE deletes a custom collection task.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/DROP_ANALYZE.md) #### [📄️ DROP STATS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/DROP_STATS.md) [DROP STATS deletes CBO statistics, which include basic statistics and histograms.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/DROP_STATS.md) #### [📄️ KILL ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/KILL_ANALYZE.md) [KILL ANALYZE cancels a running collection task, including manual and custom automatic tasks.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/KILL_ANALYZE.md) #### [📄️ SHOW ANALYZE JOB](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/SHOW_ANALYZE_JOB.md) [SHOW ANALYZE JOB views the information and status of custom collection tasks.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/SHOW_ANALYZE_JOB.md) #### [📄️ SHOW ANALYZE STATUS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/SHOW_ANALYZE_STATUS.md) [SHOW ANALYZE STATUS views the status of collection tasks.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/SHOW_ANALYZE_STATUS.md) #### [📄️ SHOW META](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/SHOW_META.md) [SHOW META views metadata of CBO statistics, including basic statistics and histograms.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/SHOW_META.md) --- ### ANALYZE TABLE ANALYZE TABLE creates a manual collection task for collecting CBO statistics. By default, manual collection is a synchronous operation. You can also set it to an asynchronous operation. In asynchronous mode, after you run ANALYZE TABLE, the system immediately returns whether this statement is successful. However, the collection task will be running in the background and you do not have to wait for the result. You can check the status of the task by running SHOW ANALYZE STATUS. Asynchronous collection is suitable for tables with large data volume, whereas synchronous collection is suitable for tables with small data volume. **Manual collection tasks are run only once after creation. You do not need to delete manual collection tasks.** This statement is supported from v2.4. ##### Manually collect basic statistics[​](#manually-collect-basic-statistics "Direct link to Manually collect basic statistics") For more information about basic statistics, see [Gather statistics for CBO](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md#basic-statistics). ###### Syntax[​](#syntax "Direct link to Syntax") ```sql ANALYZE [FULL|SAMPLE] TABLE tbl_name (col_name [,col_name]) [WITH SYNC | ASYNC MODE] PROPERTIES (property [,property]) ``` ###### Parameter description[​](#parameter-description "Direct link to Parameter description") * Collection type * FULL: indicates full collection. * SAMPLE: indicates sampled collection. * If no collection type is specified, full collection is used by default. * `col_name`: columns from which to collect statistics. Separate multiple columns with commas (`,`). If this parameter is not specified, the entire table is collected. * \[WITH SYNC | ASYNC MODE]: whether to run the manual collection task in synchronous or asynchronous mode. Synchronous collection is used by default if you do not specify this parameter. * `PROPERTIES`: custom parameters. If `PROPERTIES` is not specified, the default settings in the `fe.conf` file are used. The properties that are actually used can be viewed via the `Properties` column in the output of SHOW ANALYZE STATUS. | **PROPERTIES** | **Type** | **Default value** | **Description** | | -------------------------------- | -------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | statistic\_sample\_collect\_rows | INT | 200000 | The minimum number of rows to collect for sampled collection.If the parameter value exceeds the actual number of rows in your table, full collection is performed. | ###### Examples[​](#examples "Direct link to Examples") Example 1: Manual full collection ```sql -- Manually collect full stats of a table using default settings. ANALYZE TABLE tbl_name; -- Manually collect full stats of a table using default settings. ANALYZE FULL TABLE tbl_name; -- Manually collect stats of specified columns in a table using default settings. ANALYZE TABLE tbl_name(c1, c2, c3); ``` Example 2: Manual sampled collection ```sql -- Manually collect partial stats of a table using default settings. ANALYZE SAMPLE TABLE tbl_name; -- Manually collect stats of specified columns in a table, with the number of rows to collect specified. ANALYZE SAMPLE TABLE tbl_name (v1, v2, v3) PROPERTIES( "statistic_sample_collect_rows" = "1000000" ); ``` ##### Manually collect histograms[​](#manually-collect-histograms "Direct link to Manually collect histograms") For more information about histograms, see [Gather statistics for CBO](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md#histogram). ###### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql ANALYZE TABLE tbl_name UPDATE HISTOGRAM ON col_name [, col_name] [WITH SYNC | ASYNC MODE] [WITH N BUCKETS] PROPERTIES (property [,property]); ``` ###### Parameter description[​](#parameter-description-1 "Direct link to Parameter description") * `col_name`: columns from which to collect statistics. Separate multiple columns with commas (`,`). If this parameter is not specified, the entire table is collected. This parameter is required for histograms. * \[WITH SYNC | ASYNC MODE]: whether to run the manual collection task in synchronous or asynchronous mode. Synchronous collection is used by default if you do not specify this parameter. * `WITH N BUCKETS`: `N` is the number of buckets for histogram collection. If not specified, the default value in `fe.conf` is used. * PROPERTIES: custom parameters. If `PROPERTIES` is not specified, the default settings in `fe.conf` are used. The properties that are actually used can be viewed via the `Properties` column in the output of SHOW ANALYZE STATUS. | **PROPERTIES** | **Type** | **Default value** | **Description** | | ---------------------------------- | -------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | statistic\_sample\_collect\_rows | INT | 200000 | The minimum number of rows to collect. If the parameter value exceeds the actual number of rows in your table, full collection is performed. | | histogram\_buckets\_size | LONG | 64 | The default bucket number for a histogram. | | histogram\_mcv\_size | INT | 100 | The number of most common values (MCV) for a histogram. | | histogram\_sample\_ratio | FLOAT | 0.1 | The sampling ratio for a histogram. | | histogram\_max\_sample\_row\_count | LONG | 10000000 | The maximum number of rows to collect for a histogram. | The number of rows to collect for a histogram is controlled by multiple parameters. It is the larger value between `statistic_sample_collect_rows` and table row count \* `histogram_sample_ratio`. The number cannot exceed the value specified by `histogram_max_sample_row_count`. If the value is exceeded, `histogram_max_sample_row_count` takes precedence. ###### Examples[​](#examples-1 "Direct link to Examples") ```sql -- Manually collect histograms on v1 using the default settings. ANALYZE TABLE tbl_name UPDATE HISTOGRAM ON v1; -- Manually collect histograms on v1 and v2, with 32 buckets, 32 MCVs, and 50% sampling ratio. ANALYZE TABLE tbl_name UPDATE HISTOGRAM ON v1,v2 WITH 32 BUCKETS PROPERTIES( "histogram_mcv_size" = "32", "histogram_sample_ratio" = "0.5" ); ``` #### References[​](#references "Direct link to References") [SHOW ANALYZE STATUS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/SHOW_ANALYZE_STATUS.md): view the status of a manual collection task. [KILL ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/KILL_ANALYZE.md): cancel a manual collection task that is running. For more information about collecting statistics for CBO, see [Gather statistics for CBO](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md). --- ### CREATE ANALYZE CREATE ANALYZE customizes an automatic collection task for collecting CBO statistics. By default, StarRocks automatically collects full statistics of a table. It checks for any data updates every 5 minutes. If data change is detected, data collection will be automatically triggered. If you do not want to use automatic full collection, you can set the FE configuration item `enable_collect_full_statistic` to `false` and customize a collection task. Behaviour difference: * Before 3.2.12 and 3.3.4, to create a custom automatic collection task, you must disable automatic full collection (`enable_collect_full_statistic = false`). Otherwise, custom tasks cannot take effect. * Since these version, you can create analyze job and keep system tasks. The user-created tasks would overwrite the system tasks if they have conflicts. This statement is supported from v2.4. #### Syntax[​](#syntax "Direct link to Syntax") ```sql -- Automatically collect stats of all databases. CREATE ANALYZE [FULL|SAMPLE] ALL PROPERTIES (property [,property]) -- Automatically collect stats of all tables in a database. CREATE ANALYZE [FULL|SAMPLE] DATABASE db_name PROPERTIES (property [,property]) -- Automatically collect stats of specified columns in a table. CREATE ANALYZE [FULL|SAMPLE] [IF NOT EXISTS] TABLE tbl_name (col_name [,col_name]) PROPERTIES (property [,property]) ``` #### Parameter description[​](#parameter-description "Direct link to Parameter description") * Collection type * FULL: indicates full collection. * SAMPLE: indicates sampled collection. * If no collection type is specified, sampled collection is used by default. * `col_name`: columns from which to collect statistics. Separate multiple columns with commas (`,`). If this parameter is not specified, the entire table is collected. * `PROPERTIES`: custom parameters. If `PROPERTIES` is not specified, the default settings in `fe.conf` are used. The properties that are actually used can be viewed via the `Properties` column in the output of SHOW ANALYZE JOB. | **PROPERTIES** | **Type** | **Default value** | **Description** | | ------------------------------------------ | -------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | statistic\_auto\_collect\_ratio | FLOAT | 0.8 | The threshold for determining whether the statistics for automatic collection are healthy. If the statistics health is below this threshold, automatic collection is triggered. | | statistics\_max\_full\_collect\_data\_size | INT | 100 | The size of the largest partition for automatic collection to collect data. Unit: GB.If a partition exceeds this value, full collection is discarded and sampled collection is performed instead. | | statistic\_sample\_collect\_rows | INT | 200000 | The minimum number of rows to collect.If the parameter value exceeds the actual number of rows in your table, full collection is performed. | #### Examples[​](#examples "Direct link to Examples") Example 1: Automatic full collection ```sql -- Automatically collect full stats of all databases. CREATE ANALYZE ALL; -- Automatically collect full stats of a database. CREATE ANALYZE DATABASE db_name; -- Automatically collect full stats of all tables in a database. CREATE ANALYZE FULL DATABASE db_name; -- Automatically collect full stats of specified columns in a table. CREATE ANALYZE TABLE tbl_name(c1, c2, c3); ``` Example 2: Automatic sampled collection ```sql -- Automatically collect stats of all tables in a database using default settings. CREATE ANALYZE SAMPLE DATABASE db_name; -- Automatically collect stats of specified columns in a table, with statistics health and the number of rows to collect specified. CREATE ANALYZE SAMPLE TABLE tbl_name(c1, c2, c3) PROPERTIES( "statistic_auto_collect_ratio" = "0.5", "statistic_sample_collect_rows" = "1000000" ); ``` #### References[​](#references "Direct link to References") [SHOW ANALYZE JOB](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/SHOW_ANALYZE_JOB.md): view the status of a custom collection task. [DROP ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/DROP_ANALYZE.md): delete a custom collection task. [KILL ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/KILL_ANALYZE.md): cancel a custom collection task that is running. For more information about collecting statistics for CBO, see [Gather statistics for CBO](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md). --- ### DROP ANALYZE DROP ANALYZE deletes a custom collection task. By default, StarRocks automatically collects full statistics of a table. It checks for any data updates every 5 minutes. If data change is detected, data collection will be automatically triggered. If you do not want to use automatic full collection, you can set the FE configuration item `enable_collect_full_statistic` to `false` and customize a collection task. This statement is supported from v2.4. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DROP ANALYZE ``` The task ID can be obtained by using the SHOW ANALYZE JOB statement. #### Examples[​](#examples "Direct link to Examples") ```sql DROP ANALYZE 266030; ``` #### References[​](#references "Direct link to References") [CREATE ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/CREATE_ANALYZE.md): customize an automatic collection task. [SHOW ANALYZE JOB](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/SHOW_ANALYZE_JOB.md): view the status of an automatic collection task. [KILL ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/KILL_ANALYZE.md): cancel a custom collection task that is running. For more information about collecting statistics for CBO, see [Gather statistics for CBO](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md). --- ### DROP STATS DROP STATS deletes CBO statistics, which include basic statistics and histograms. For more information, see [Gather statistics for CBO](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md#basic-statistics). You can delete statistical information you do not need. When you delete statistics, both the data and metadata of the statistics are deleted, as well as the statistics in expired cache. Note that if an automatic collection task is ongoing, previously deleted statistics may be collected again. You can use `SHOW ANALYZE STATUS` to view the history of collection tasks. This statement is supported from v2.4. #### Syntax[​](#syntax "Direct link to Syntax") ##### Delete basic statistics[​](#delete-basic-statistics "Direct link to Delete basic statistics") ```sql DROP STATS tbl_name ``` ##### Delete histograms[​](#delete-histograms "Direct link to Delete histograms") ```sql ANALYZE TABLE tbl_name DROP HISTOGRAM ON col_name [, col_name]; ``` #### References[​](#references "Direct link to References") For more information about collecting statistics for CBO, see [Gather statistics for CBO](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md). --- ### KILL ANALYZE KILL ANALYZE cancels a **running** collection task, including manual and custom automatic tasks. This statement is supported from v2.4. #### Syntax[​](#syntax "Direct link to Syntax") ```sql KILL ANALYZE ``` The task ID for a manual collection task can be obtained from SHOW ANALYZE STATUS. The task ID for a custom collection task can be obtained from SHOW ANALYZE SHOW ANALYZE JOB. #### References[​](#references "Direct link to References") [SHOW ANALYZE STATUS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/SHOW_ANALYZE_STATUS.md) [SHOW ANALYZE JOB](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/SHOW_ANALYZE_JOB.md) For more information about collecting statistics for CBO, see [Gather statistics for CBO](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md). --- ### SHOW ANALYZE JOB SHOW ANALYZE JOB views the information and status of custom collection tasks. By default, StarRocks automatically collects full statistics of a table. It checks for any data updates every 5 minutes. If data change is detected, data collection will be automatically triggered. If you do not want to use automatic full collection, you can set the FE configuration item `enable_collect_full_statistic` to `false` and customize a collection task. This statement is supported from v2.4. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW ANALYZE JOB [WHERE] ``` You can filter results by using the WHERE clause. The statement returns the following columns. | **Column** | **Description** | | ------------ | ---------------------------------------------------------------------------------- | | Id | The ID of the collection task. | | Database | The database name. | | Table | The table name. | | Columns | The column names. | | Type | The type of statistics, including `FULL` and `SAMPLE`. | | Schedule | The type of scheduling. The type is `SCHEDULE` for an automatic task. | | Properties | Custom parameters. | | Status | The task status, including PENDING, RUNNING, SUCCESS, and FAILED. | | LastWorkTime | The time of the last collection. | | Reason | The reason why the task failed. NULL is returned if task execution was successful. | #### Examples[​](#examples "Direct link to Examples") ```sql -- View all the custom collection tasks. SHOW ANALYZE JOB; -- View custom collection tasks of database `test`. SHOW ANALYZE JOB where `database` = 'test'; ``` #### References[​](#references "Direct link to References") [CREATE ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/CREATE_ANALYZE.md): customize an automatic collection task. [DROP ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/DROP_ANALYZE.md): delete a custom collection task. [KILL ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/KILL_ANALYZE.md): cancel a custom collection task that is running. For more information about collecting statistics for CBO, see [Gather statistics for CBO](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md). --- ### SHOW ANALYZE STATUS SHOW ANALYZE STATUS views the status of collection tasks. This statement cannot be used to view the status of custom collection tasks. To view the status of custom collection tasks, use SHOW ANALYZE JOB. This statement is supported from v2.4. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW ANALYZE STATUS [WHERE] ``` You can use `LIKE` or `WHERE` to filter the information to return. This statement returns the following columns. | **List name** | **Description** | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Id | The ID of the collection task. | | Database | The database name. | | Table | The table name. | | Columns | The column names. | | Type | The type of statistics, including FULL, SAMPLE, and HISTOGRAM. | | Schedule | The type of scheduling. `ONCE` means manual and `SCHEDULE` means automatic. | | Status | The status of the task. | | StartTime | The time when the task starts executing. | | EndTime | The time when the task execution ends. | | Properties | Custom parameters. For external table (for example, Iceberg, Hive, or Paimon) collection tasks, this also includes collection metadata such as `table_format`, `partition_count`, `column_count`, and, for Iceberg tables, `snapshot_id`, `total_files`, and `total_rows`. | | Reason | The reason why the task failed. NULL is returned if the execution was successful. | #### References[​](#references "Direct link to References") [ANALYZE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/ANALYZE_TABLE.md): create a manual collection task. [KILL ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/KILL_ANALYZE.md): cancel a custom collection task that is running. For more information about collecting statistics for CBO, see [Gather statistics for CBO](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md). --- ### SHOW META SHOW META views metadata of CBO statistics, including basic statistics and histograms. This statement is supported from v2.4. ##### View metadata of basic statistics[​](#view-metadata-of-basic-statistics "Direct link to View metadata of basic statistics") ###### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW STATS META [WHERE] ``` This statement returns the following columns. | **Column** | **Description** | | ---------- | ------------------------------------------------------------------------------------------- | | Database | The database name. | | Table | The table name. | | Columns | The column names. | | Type | The type of statistics. `FULL` means full collection and `SAMPLE` means sampled collection. | | UpdateTime | The latest statistics update time for the current table. | | Properties | Custom parameters. | | Healthy | The health of statistical information. | ##### View metadata of histograms[​](#view-metadata-of-histograms "Direct link to View metadata of histograms") ###### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql SHOW HISTOGRAM META [WHERE]; ``` This statement returns the following columns. | **Column** | **Description** | | ---------- | ------------------------------------------------------------ | | Database | The database name. | | Table | The table name. | | Column | The columns. | | Type | Type of statistics. The value is `HISTOGRAM` for histograms. | | UpdateTime | The latest statistics update time for the current table. | | Properties | Custom parameters. | #### References[​](#references "Direct link to References") For more information about collecting statistics for CBO, see [Gather statistics for CBO](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md). --- ### Cluster management #### [🗃️ Nodes and processes](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes.md) [14 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes.md) #### [🗃️ Configuration variables](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars.md) [5 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars.md) #### [🗃️ Resource group](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group.md) [5 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group.md) #### [🗃️ Storage volume](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume.md) [6 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume.md) #### [🗃️ Tablet and replica](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica.md) [10 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica.md) #### [🗃️ File](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/file.md) [3 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/file.md) #### [🗃️ SQL blacklist](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/sql_blacklist.md) [3 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/sql_blacklist.md) #### [🗃️ Plugin](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plugin.md) [3 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plugin.md) #### [🗃️ Execution plan and profile](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile.md) [4 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile.md) --- ### Configuration variables #### [📄️ ADMIN SET CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) [ADMIN SET CONFIG sets FE dynamic configuration items for the StarRocks cluster at runtime.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) #### [📄️ ADMIN SHOW CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md) [ADMIN SHOW CONFIG displays the configuration of the current cluster (Currently, only FE configuration items can be displayed).](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md) #### [📄️ REFRESH CONNECTIONS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/REFRESH_CONNECTIONS.md) [Refreshes all active connections to apply global variables that have been changed by SET GLOBAL.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/REFRESH_CONNECTIONS.md) #### [📄️ SET](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/SET.md) [Sets the specified system variables or user-defined variables for StarRocks.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/SET.md) #### [📄️ SHOW VARIABLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/SHOW_VARIABLES.md) [SHOW VARIABLES shows the system variables of StarRocks.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/SHOW_VARIABLES.md) --- ### ADMIN SET CONFIG ADMIN SET CONFIG is used to set configuration items for the cluster (Currently, only FE dynamic configuration items can be set using this command). You can view these configuration items using the [ADMIN SHOW FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. The configurations will be restored to the default values in the `fe.conf` file after the FE restarts. Therefore, we recommend that you also modify the configuration items in `fe.conf` to prevent the loss of modifications. tip This operation requires the SYSTEM-level OPERATE privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ADMIN SET FRONTEND CONFIG ("key" = "value") ``` #### Examples[​](#examples "Direct link to Examples") 1. Set `disable_balance` to `true`. ```sql ADMIN SET FRONTEND CONFIG ("disable_balance" = "true"); ``` --- ### ADMIN SHOW CONFIG ADMIN SHOW CONFIG displays the configuration of the current cluster (Currently, only FE configuration items can be displayed). For detailed description of these configuration items, see [Configuration](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md). If you want to set or modify a configuration item, use [ADMIN SET CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md). tip This operation requires the SYSTEM-level OPERATE privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ADMIN SHOW FRONTEND CONFIG [LIKE "pattern"] ``` Note: Description of the return parameters: ```plain 1. Key: Configuration item name 2. Value: Configuration item value 3. Type: Configuration item type 4. IsMutable: Whether it can be set through the ADMIN SET CONFIG command 5. MasterOnly: Whether it only applies to leader FE 6. Comment: Configuration item description ``` For configuration items that hold credentials, such as passwords, secret keys, and tokens, `Value` does not show the actual value. If the item is set, `Value` is `******`. If the item is not set, `Value` is empty. Masking affects only the output. The FE still uses the actual value. #### Examples[​](#examples "Direct link to Examples") 1. View the configuration of the current FE node. ```sql ADMIN SHOW FRONTEND CONFIG; ``` 2. Search for the configuration of the current FE node by using the `like` predicate. ```plain mysql> ADMIN SHOW FRONTEND CONFIG LIKE '%check_java_version%'; +--------------------+-------+---------+-----------+------------+---------+ | Key | Value | Type | IsMutable | MasterOnly | Comment | +--------------------+-------+---------+-----------+------------+---------+ | check_java_version | true | boolean | false | false | | +--------------------+-------+---------+-----------+------------+---------+ 1 row in set (0.00 sec) ``` --- ### REFRESH CONNECTIONS Refreshes all active connections to apply global variables that have been changed by `SET GLOBAL`. This command allows administrators to update long-lived connections (such as those from connection pools) when runtime parameters are adjusted, without requiring connections to reconnect. #### Background[​](#background "Direct link to Background") By default, `SET GLOBAL` in StarRocks only affects newly created connections. Existing sessions keep their local copies of session variables until they disconnect or explicitly run `SET SESSION`. This behavior is consistent with MySQL but makes it difficult to update long-lived connections when administrators adjust runtime parameters. The `REFRESH CONNECTIONS` command solves this problem by allowing administrators to refresh all active connections to apply the latest global variable values. tip This command requires the OPERATE privilege on the SYSTEM object. #### Syntax[​](#syntax "Direct link to Syntax") ```sql REFRESH CONNECTIONS [FORCE]; ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | FORCE | No | If specified, forces refresh of all variables even if they have been modified by the session using `SET SESSION`. When `FORCE` is not specified, only variables that haven't been modified by the session are refreshed. | #### Behavior[​](#behavior "Direct link to Behavior") * **Selective refresh**: By default, the command only refreshes variables that have not been modified by the session itself. Variables that were explicitly set using `SET SESSION` are preserved. When `FORCE` is specified, all variables are refreshed regardless of session modifications. * **Immediate execution**: Running statements are not affected immediately. The refresh takes effect before the next command is executed. * **Distribution**: The command is automatically distributed to all FE nodes via RPC, ensuring consistency across the cluster. * **Privilege requirement**: Only users with OPERATE privilege on the SYSTEM object can execute this command. #### Examples[​](#examples "Direct link to Examples") Example 1: Refresh all connections after changing a global variable. ```plain mysql> SET GLOBAL query_timeout = 600; Query OK, 0 rows affected (0.00 sec) mysql> REFRESH CONNECTIONS; Query OK, 0 rows affected (0.00 sec) ``` After executing `REFRESH CONNECTIONS`, all active connections (except those that have modified `query_timeout` in their session) will have their `query_timeout` value updated to 600. Example 2: Refresh connections after changing multiple global variables. ```plain mysql> SET GLOBAL query_timeout = 600; Query OK, 0 rows affected (0.00 sec) mysql> SET GLOBAL exec_mem_limit = 2147483648; Query OK, 0 rows affected (0.00 sec) mysql> REFRESH CONNECTIONS; Query OK, 0 rows affected (0.00 sec) ``` Example 3: Force refresh all connections, including those with session-modified variables. ```plain mysql> SET GLOBAL query_timeout = 600; Query OK, 0 rows affected (0.00 sec) mysql> REFRESH CONNECTIONS FORCE; Query OK, 0 rows affected (0.00 sec) ``` After executing `REFRESH CONNECTIONS FORCE`, all active connections will have their `query_timeout` value updated to 600, even if they previously modified this variable using `SET SESSION query_timeout = 500`. #### Notes[​](#notes "Direct link to Notes") * By default, variables that have been modified in a session using `SET SESSION` are not refreshed. For example, if a connection has executed `SET SESSION query_timeout = 500`, the `query_timeout` value for that connection will remain 500 even after `REFRESH CONNECTIONS` is executed. However, if you use `REFRESH CONNECTIONS FORCE`, all variables will be refreshed regardless of session modifications. * When `FORCE` is specified, variables that were previously modified by the session are reset to the global default values, and their modification flags are cleared. This allows future non-forced refreshes to update these variables. * The command affects all active connections on all FE nodes in the cluster. * Session-only variables (variables that cannot be set globally) are not affected by this command. * Variables with the `DISABLE_FORWARD_TO_LEADER` flag are not refreshed. #### Related Statements[​](#related-statements "Direct link to Related Statements") * [SET](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/SET.md): Set system variables or user-defined variables * [SHOW VARIABLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/SHOW_VARIABLES.md): Show system variables --- ### SET Sets the specified system variables or user-defined variables for StarRocks. You can view the system variables of StarRocks using [SHOW VARIABLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/SHOW_VARIABLES.md). For details about system variables, see [System Variables](https://docs.starrocks.io/docs/sql-reference/System_variable.md). For details about user-defined variables, see [User-defined variables](https://docs.starrocks.io/docs/sql-reference/user_defined_variables.md). tip This operation does not require privileges. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SET [ GLOBAL | SESSION ] = [, = ] ... ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Modifier:- GLOBAL
- SESSION | - With a `GLOBAL` modifier, the statement sets the variables globally.
- With a `SESSION` modifier, the statement sets the variables within the session. `LOCAL` is a synonym for `SESSION`.
- If no modifier is present, the default is `SESSION`.For details about global and session variables, see [System Variables](https://docs.starrocks.io/docs/sql-reference/System_variable.md).
**NOTE**
Only users with the ADMIN privilege can set the variables globally. | | variable\_name | The name of the variable. | | value | The value of the variable. | #### Examples[​](#examples "Direct link to Examples") Example 1: Set the `time_zone` to `Asia/Shanghai` within the session. ```plain mysql> SET time_zone = "Asia/Shanghai"; Query OK, 0 rows affected (0.00 sec) ``` Example 2: Set the `exec_mem_limit` to `2147483648` globally. ```plain mysql> SET GLOBAL exec_mem_limit = 2147483648; Query OK, 0 rows affected (0.00 sec) ``` Example 3: Set multiple global variables. Add the `GLOBAL` keyword before each variable. ```plain mysql> SET GLOBAL exec_mem_limit = 2147483648, GLOBAL time_zone = "Asia/Shanghai"; Query OK, 0 rows affected (0.00 sec) ``` --- ### SHOW VARIABLES SHOW VARIABLES shows the system variables of StarRocks. For details about system variables, see [System Variables](https://docs.starrocks.io/docs/sql-reference/System_variable.md). tip This operation does not require privileges. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW [ GLOBAL | SESSION ] VARIABLES [ LIKE | WHERE ] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Modifier:- GLOBAL
- SESSION | - With a `GLOBAL` modifier, the statement displays global system variable values. These are the values used to initialize the corresponding session variables for new connections to StarRocks. If a variable has no global value, no value is displayed.
- With a `SESSION` modifier, the statement displays the system variable values that are in effect for the current connection. If a variable has no session value, the global value is displayed. `LOCAL` is a synonym for `SESSION`.
- If no modifier is present, the default is `SESSION`. | | pattern | The pattern used to match the variable by the variable name with the LIKE clause. You can use the % wildcard in this parameter. | | expr | The expression used to match the variable by the variable name `variable_name` or the variable value `value` with the WHERE clause. | #### Return value[​](#return-value "Direct link to Return value") | **Return** | **Description** | | -------------- | -------------------------- | | Variable\_name | The name of the variable. | | Value | The value of the variable. | #### Examples[​](#examples "Direct link to Examples") Example 1: Show a variable by exactly matching the variable name with the LIKE clause. ```plain mysql> SHOW VARIABLES LIKE 'wait_timeout'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | wait_timeout | 28800 | +---------------+-------+ 1 row in set (0.01 sec) ``` Example 2: Show variables by approximately matching the variable name with the LIKE clause and the wildcard (%). ```plain mysql> SHOW VARIABLES LIKE '%imeou%'; +------------------------------------+-------+ | Variable_name | Value | +------------------------------------+-------+ | interactive_timeout | 3600 | | net_read_timeout | 60 | | net_write_timeout | 60 | | new_planner_optimize_timeout | 3000 | | query_delivery_timeout | 300 | | query_queue_pending_timeout_second | 300 | | query_timeout | 300 | | tx_visible_wait_timeout | 10 | | wait_timeout | 28800 | +------------------------------------+-------+ 9 rows in set (0.00 sec) ``` Example 3: Show a variable by exactly matching the variable name with the WHERE clause. ```plain mysql> show variables where variable_name = 'wait_timeout'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | wait_timeout | 28800 | +---------------+-------+ 1 row in set (0.17 sec) ``` Example 4: Show a variable by exactly matching the value of the variable with the WHERE clause. ```plain mysql> show variables where value = '28800'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | wait_timeout | 28800 | +---------------+-------+ 1 row in set (0.70 sec) ``` --- ### File #### [📄️ CREATE FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/file/CREATE_FILE.md) [You can execute the CREATE FILE statement to create a file.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/file/CREATE_FILE.md) #### [📄️ DROP FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/file/DROP_FILE.md) [Use the DROP FILE statement to delete a file.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/file/DROP_FILE.md) #### [📄️ SHOW FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/file/SHOW_FILE.md) [You can execute the SHOW FILE statement to view the information about a file stored in a database.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/file/SHOW_FILE.md) --- ### CREATE FILE You can execute the CREATE FILE statement to create a file. After a file is created, the file is uploaded and persisted in StarRocks. In a database, only an admin user can create and delete files, and all users who have permission to access the database can use the files that belong to the database. tip This operation requires the SYSTEM-level FILE privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Basic concepts[​](#basic-concepts "Direct link to Basic concepts") **File**: refers to the file that is created and saved in StarRocks. After a file is created and stored in StarRocks, StarRocks assigns a unique ID to the file. You can find a file based on the database name, catalog, and file name. #### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE FILE "file_name" [IN database] [properties] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | file\_name | Yes | The name of the file. | | database | No | The database to which the file belongs. If you do not specify this parameter, this parameter defaults to the name of the database that you access in the current session. | | properties | Yes | The properties of the file. The following table describes the configuration items of properties. | **Configuration items of** **`properties`** | **Configuration item** | **Required** | **Description** | | ---------------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | url | Yes | The URL from which you can download the file. Only an unauthenticated HTTP URL is supported. After the file is stored in StarRocks, the URL is no longer needed. | | catalog | Yes | The category to which the file belongs. You can specify a catalog based on your business requirements. However, in some situations, you must set this parameter to a specific catalog. For example, if you load data from Kafka, StarRocks searches for files in the catalog from the Kafka data source. | | MD5 | No | The message-digest algorithm that is used to check a file. If you specify this parameter, StarRocks checks the file after the file is downloaded. | #### Examples[​](#examples "Direct link to Examples") * Create a file named **test.pem** under the category named kafka. ```sql CREATE FILE "test.pem" PROPERTIES ( "url" = "https://starrocks-public.oss-cn-xxxx.aliyuncs.com/key/test.pem", "catalog" = "kafka" ); ``` * Create a file named **client.key** under the category named my\_catalog. ```sql CREATE FILE "client.key" IN my_database PROPERTIES ( "url" = "http://test.bj.bcebos.com/kafka-key/client.key", "catalog" = "my_catalog", "md5" = "b5bb901bf10f99205b39a46ac3557dd9" ); ``` --- ### DROP FILE Use the DROP FILE statement to delete a file. When you use this statement to delete a file, the file is deleted both in frontend (FE) memory and in Berkeley DB Java Edition (BDBJE). tip This operation requires the SYSTEM-level FILE privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DROP FILE "file_name" [FROM database] [properties] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | ------------------------------------------------------------------------------------------------ | | file\_name | Yes | The name of the file. | | database | No | The database to which a file belongs. | | properties | Yes | The properties of the file. The following table describes the configuration items of properties. | **Configuration items of** **`properties`** | **Configuration items** | **Required** | **Description** | | ----------------------- | ------------ | ------------------------------------- | | catalog | Yes | The category to which a file belongs. | #### Examples[​](#examples "Direct link to Examples") Delete a file named **ca.pem**. ```sql DROP FILE "ca.pem" properties("catalog" = "kafka"); ``` --- ### SHOW FILE You can execute the SHOW FILE statement to view the information about a file stored in a database. tip Users with any privilege on the database to which a file belongs can perform this operation. If a file belongs to a database, all users that have access to this database can use this file. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW FILE [FROM database] ``` The file information returned by this statement is as follows: * `FileId`: the globally unique ID of the file. * `DbName`: the database to which the file belongs. * `Catalog`: the category to which the file belongs. * `FileName`: the name of the file. * `FileSize`: the size of the file. The unit is bytes. * `MD5`: the message-digest algorithm that is used to check the file. #### Examples[​](#examples "Direct link to Examples") View the file stored in `my_database`. ```sql SHOW FILE FROM my_database; ``` --- ### Nodes and processes #### [📄️ ADD BACKEND/COMPUTE NOTE BLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ADD_BACKEND_BLACKLIST.md) [Adds a BE or CN node to the BE and CN Blacklist.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ADD_BACKEND_BLACKLIST.md) #### [📄️ ALTER SYSTEM](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md) [ALTER SYSTEM Manages FE, BE, CN, Broker nodes, and metadata snapshots in a cluster.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md) #### [📄️ CANCEL DECOMMISSION](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/CANCEL_DECOMMISSION.md) [CANCEL DECOMMISSION is used to undo a node decommission.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/CANCEL_DECOMMISSION.md) #### [📄️ DELETE BACKEND/COMPUTE NOT BLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/DELETE_BACKEND_BLACKLIST.md) [Removes a BE/CN node from the BE Blacklist.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/DELETE_BACKEND_BLACKLIST.md) #### [📄️ KILL](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/KILL.md) [Terminates a connection or a query currently being performed by threads executing within StarRocks.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/KILL.md) #### [📄️ SHOW BACKENDS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md) [SHOW BACKENDS shows the information of all BE nodes in the cluster.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md) #### [📄️ SHOW BACKEND/COMPUTE NODE BLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKEND_BLACKLIST.md) [Shows the BE/CN nodes in the BE and CN Blacklist.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKEND_BLACKLIST.md) #### [📄️ SHOW BROKER](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BROKER.md) [SHOW BROKER views the information of all brokers that have been added to the StarRocks cluster.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BROKER.md) #### [📄️ SHOW COMPUTE NODES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md) [SHOW COMPUTE NODES shows the information of all CN nodes in the cluster.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md) #### [📄️ SHOW FRONTENDS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_FRONTENDS.md) [SHOW FRONTENDS views the information of all FE nodes in the cluster, including their IP addresses, ports, roles, and status.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_FRONTENDS.md) #### [📄️ SHOW PROC](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md) [SHOW PROC shows certain indicators of the StarRocks cluster.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md) #### [📄️ SHOW PROCESSLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROCESSLIST.md) [SHOW PROCESSLIST lists the operations currently being performed by threads executing within the server.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROCESSLIST.md) #### [📄️ SHOW RUNNING QUERIES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_RUNNING_QUERIES.md) [SHOW RUNNING QUERIES displays information of all queries that are running or pending in the query queue.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_RUNNING_QUERIES.md) #### [📄️ SYNC](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SYNC.md) [Synchronizes the data consistency among different sessions.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SYNC.md) --- ### ADD BACKEND/COMPUTE NOTE BLACKLIST Adds a BE or CN node to the BE and CN Blacklist. You can manually add BE/CN nodes to the blacklist to forbid the usage of the nodes in query execution, thereby avoiding frequent query failures or other unexpected behaviors caused by failed connections to the nodes. BE Blacklist is supported from v3.3.0 onwards and CN Blacklist is supported from v4.0 onwards. For more information, see [Manage BE and CN Blacklist](https://docs.starrocks.io/docs/administration/management/BE_blacklist.md). note Only users with the SYSTEM-level BLACKLIST privilege can perform this operation. By default, StarRocks can automatically manage the BE and CN Blacklist, adding the BE/CN nodes that have lost connection to the blacklist and removing them from the blacklist when the connection is reestablished. However, StarRocks will not remove the node from the blacklist if it is manually blacklisted. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ADD { BACKEND | COMPUTE NODE } BLACKLIST { [, ...] | [, ...] } ``` #### Parameters[​](#parameters "Direct link to Parameters") `be_id` or `cn_id`: ID of the BE or CN node to be blacklisted. You can obtain the BE ID by executing [SHOW BACKENDS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md) and CN ID by executing [SHOW COMPUTE NODES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md). #### Examples[​](#examples "Direct link to Examples") ```sql -- Obtain BE ID. SHOW BACKENDS\G *************************** 1. row *************************** BackendId: 10001 IP: xxx.xx.xx.xxx ... -- Add BE to the blacklist. ADD BACKEND BLACKLIST 10001; -- Obtain CN ID. SHOW COMPUTE NODES\G *************************** 1. row *************************** ComputeNodeId: 10005 IP: xxx.xx.xx.xxx ... -- Add CN to the blacklist. ADD COMPUTE NODE BLACKLIST 10005; ``` #### Relevant SQLs[​](#relevant-sqls "Direct link to Relevant SQLs") * [DELETE BACKEND/COMPUTE NODE BLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/DELETE_BACKEND_BLACKLIST.md) * [SHOW BACKEND/COMPUTE NODE BLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKEND_BLACKLIST.md) * [SHOW BACKENDS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md) * [SHOW COMPUTE NODES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md) --- ### ALTER SYSTEM ALTER SYSTEM Manages FE, BE, CN, Broker nodes, and metadata snapshots in a cluster. > **NOTE** > > Only the `cluster_admin` role has the privilege to perform this operation. #### Syntax and parameters[​](#syntax-and-parameters "Direct link to Syntax and parameters") ##### FE[​](#fe "Direct link to FE") * Add a Follower FE. ```sql ALTER SYSTEM ADD FOLLOWER ":"[, ...] ``` You can check the status of the new Follower FE by executing `SHOW PROC '/frontends'\G`. * Drop a Follower FE. ```sql ALTER SYSTEM DROP FOLLOWER ":"[, ...] ``` * Add an Observer FE. ```sql ALTER SYSTEM ADD OBSERVER ":"[, ...] ``` You can check the status of the new Observer FE by executing `SHOW PROC '/frontends'\G`. * Drop an Observer FE. ```sql ALTER SYSTEM DROP OBSERVER ":"[, ...] ``` | **Parameter** | **Required** | **Description** | | --------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | | fe\_host | Yes | The host name or IP address of the FE instance. Use the value of configuration item `priority_networks` if your instance has multiple IP addresses. | | edit\_log\_port | Yes | BDB JE communication port of the FE node. Default: `9010`. | ##### BE[​](#be "Direct link to BE") * Add a BE node. ```sql ALTER SYSTEM ADD BACKEND ":"[, ...] ``` You can check the status of the new BE by executing [SHOW BACKENDS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md). * Drop a BE node. > **NOTE** > > You cannot drop the BE node that stores the tablets of single-replica tables. ```sql ALTER SYSTEM DROP BACKEND ":"[, ...] ``` * Decommission a BE node. ```sql ALTER SYSTEM DECOMMISSION BACKEND ":"[, ...] ``` Unlike dropping a BE node, which is removing it forcibly from the cluster, decommissioning a BE means removing it safely. It is an asynchronous operation. When a BE is decommissioned, the data on the BE is first migrated to other BEs, and then the BE is removed from the cluster. Data loading and query will not be affected during the data migration. You can check whether the operation is successful using [SHOW BACKENDS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md). If the operation is successful, the decommissioned BE will not be returned. If the operation fails, the BE will still be online. You can manually cancel the operation using [CANCEL DECOMMISSION](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/CANCEL_DECOMMISSION.md). | **Parameter** | **Required** | **Description** | | ------------------------ | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | | be\_host | Yes | The host name or IP address of the BE instance. Use the value of configuration item `priority_networks` if your instance has multiple IP addresses. | | heartbeat\_service\_port | Yes | BE heartbeat service port. BE uses this port to receive heartbeat from FE. Default: `9050`. | ##### CN[​](#cn "Direct link to CN") * Add a CN node. ```sql ALTER SYSTEM ADD COMPUTE NODE ":"[, ...] ``` You can check the status of the new CN by executing [SHOW COMPUTE NODES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md). * Drop a CN node. ```sql ALTER SYSTEM DROP COMPUTE NODE ":"[, ...] ``` > **NOTE** > > You cannot decommission a CN node using the `ALTER SYSTEM DECOMMISSION` command. | **Parameter** | **Required** | **Description** | | ------------------------ | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | | cn\_host | Yes | The host name or IP address of the CN instance. Use the value of configuration item `priority_networks` if your instance has multiple IP addresses. | | heartbeat\_service\_port | Yes | CN heartbeat service port. CN uses this port to receive heartbeat from FE. Default: `9050`. | ##### Broker[​](#broker "Direct link to Broker") * Add Broker nodes. You can use Broker nodes to load data from HDFS or cloud storage into StarRocks. For more information, see [Loading](https://docs.starrocks.io/docs/loading/loading_introduction.md). ```sql ALTER SYSTEM ADD BROKER ":"[, ...] ``` You can add multiple Broker nodes with one SQL. Each `:` pair represents one Broker node. And they share a common `broker_name`. You can check the status of the new Broker node by executing [SHOW BROKER](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BROKER.md). * Drop Broker nodes. > **CAUTION** > > Dropping a Broker node terminates the tasks currently running on it. * Drop one or multiple Broker nodes with the same `broker_name`. ```sql ALTER SYSTEM DROP BROKER ":"[, ...] ``` * Drop all Broker nodes with the same `broker_name`. ```sql ALTER SYSTEM DROP ALL BROKER ``` | **Parameter** | **Required** | **Description** | | ----------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | broker\_name | Yes | The name of the Broker node(s). Multiple Broker nodes can use the same name. | | broker\_host | Yes | The host name or IP address of the Broker instance. Use the value of configuration item `priority_networks` if your instance has multiple IP addresses. | | broker\_ipc\_port | Yes | The thrift server port on the Broker node. The Broker node uses it to receive requests from FE or BE. Default: `8000`. | ##### Create image[​](#create-image "Direct link to Create image") Create an image file. An image file is a snapshot of the FE metadata. ```sql ALTER SYSTEM CREATE IMAGE ``` Creating an image is an asynchronous operation on the Leader FE. You can check the start time and end time of the operation in the FE log file **fe.log**. A log like `triggering a new checkpoint manually...` indicates that the image creation has started, and a log like `finished save image...` indicates the image has been created. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * Adding and dropping FE, BE, CN, or Broker nodes are synchronous operations. You cannot cancel the node dropping operations. * You cannot drop the FE node in a single-FE cluster. * You cannot directly drop the Leader FE node in a multi-FE cluster. To drop it, you must first restart it. After StarRocks elects a new Leader FE, you can then drop the previous one. * You cannot drop BE nodes if the number of the remained BE nodes is less than the number of data replicas. For example, if you have three BE nodes in your cluster and you store your data in three replicas, you cannot drop any of the BE nodes. And if you have four BE nodes and three replicas, you can drop one BE node. * The difference between dropping and decommissioning a BE node is that, when you drop a BE node, StarRocks removes it forcibly from the cluster and make up the dropped tablets after the removal, and when you decommission a BE node, StarRocks first migrates the tablets on the decommissioned BE node to others, and then removes the node. #### Examples[​](#examples "Direct link to Examples") Example 1: Add a Follower FE node. ```sql ALTER SYSTEM ADD FOLLOWER "x.x.x.x:9010"; ``` Example 2: Drop two Observer FE nodes simultaneously. ```sql ALTER SYSTEM DROP OBSERVER "x.x.x.x:9010","x.x.x.x:9010"; ``` Example 3: Add a BE node. ```sql ALTER SYSTEM ADD BACKEND "x.x.x.x:9050"; ``` Example 4: Drop two BE nodes simultaneously. ```sql ALTER SYSTEM DROP BACKEND "x.x.x.x:9050", "x.x.x.x:9050"; ``` Example 5: Decommission two BE nodes simultaneously. ```sql ALTER SYSTEM DECOMMISSION BACKEND "x.x.x.x:9050", "x.x.x.x:9050"; ``` Example 6: Add two Broker nodes with the same `broker_name` - `hdfs`. ```sql ALTER SYSTEM ADD BROKER hdfs "x.x.x.x:8000", "x.x.x.x:8000"; ``` Example 7: Drop two Broker nodes from `amazon_s3`. ```sql ALTER SYSTEM DROP BROKER amazon_s3 "x.x.x.x:8000", "x.x.x.x:8000"; ``` Example 8: Drop all Broker nodes in `amazon_s3`. ```sql ALTER SYSTEM DROP ALL BROKER amazon_s3; ``` --- ### CANCEL DECOMMISSION CANCEL DECOMMISSION is used to undo a node decommission. tip Only the `cluster_admin` role has the privilege to perform this operation. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. Syntax: ```sql CANCEL DECOMMISSION BACKEND ":"[,":"...] ``` #### Examples[​](#examples "Direct link to Examples") 1. Cancel decommission of two nodes. ```sql CANCEL DECOMMISSION BACKEND "host1:port", "host2:port"; ``` --- ### DELETE BACKEND/COMPUTE NOT BLACKLIST Removes a BE/CN node from the BE Blacklist. Please note that StarRocks will not remove the BE/CN nodes that are manually blacklisted by users. BE Blacklist is supported from v3.3.0 onwards and CN Blacklist is supported from v4.0 onwards. For more information, see [Manage BE and CN Blacklist](https://docs.starrocks.io/docs/administration/management/BE_blacklist.md). note Only users with the SYSTEM-level BLACKLIST privilege can perform this operation. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DELETE { BACKEND | COMPUTE NODE } BLACKLIST { [, ...] | [, ...] } ``` #### Parameters[​](#parameters "Direct link to Parameters") `be_id` or `cn_id`: ID of the BE or CN node to be removed from the blacklist. You can obtain the ID of the blacklisted BE/CN by executing [SHOW BACKEND/COMPUTE NODE BLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKEND_BLACKLIST.md). #### Examples[​](#examples "Direct link to Examples") ```sql DELETE BACKEND BLACKLIST 10001; ``` #### Relevant SQLs[​](#relevant-sqls "Direct link to Relevant SQLs") * [ADD BACKEND/COMPUTE NODE BLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ADD_BACKEND_BLACKLIST.md) * [SHOW BACKEND/COMPUTE NODE BLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKEND_BLACKLIST.md) * [SHOW BACKENDS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md) * [SHOW COMPUTE NODES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md) --- ### KILL Terminates a connection or a query currently being performed by threads executing within StarRocks. tip This operation does not require privileges. #### Syntax[​](#syntax "Direct link to Syntax") ```sql KILL [ CONNECTION | QUERY ] { | } ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Modifier:- CONNECTION
- QUERY | - With a `CONNECTION` modifier, the KILL statement terminates the connection associated with the given `processlist_id`, after terminating any statement the connection is executing.
- With a `QUERY` modifier, the KILL statement terminates the statement the connection is currently executing, but leaves the connection itself intact.
- If no modifier is present, the default is `CONNECTION`. | | processlist\_id | The ID of the thread you want to terminate. You can get the IDs of the threads that are being executed using [SHOW PROCESSLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROCESSLIST.md). | | query\_id | The ID of the query you want to terminate. You can get the IDs of the queries that are pending or being executed using SHOW RUNNING QUERIES or [SHOW PROC](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md) with `'/current_queries'` and `'/global_current_queries'`. | #### Examples[​](#examples "Direct link to Examples") ```plain mysql> SHOW FULL PROCESSLIST; +------+------+---------------------+--------+---------+---------------------+------+-------+-----------------------+-----------+ | Id | User | Host | Db | Command | ConnectionStartTime | Time | State | Info | IsPending | +------+------+---------------------+--------+---------+---------------------+------+-------+-----------------------+-----------+ | 20 | root | xxx.xx.xxx.xx:xxxxx | sr_hub | Query | 2023-01-05 16:30:19 | 0 | OK | show full processlist | false | +------+------+---------------------+--------+---------+---------------------+------+-------+-----------------------+-----------+ 1 row in set (0.01 sec) mysql> KILL 20; Query OK, 0 rows affected (0.00 sec) ``` --- ### SHOW BACKEND/COMPUTE NODE BLACKLIST Shows the BE/CN nodes in the BE and CN Blacklist. BE Blacklist is supported from v3.3.0 onwards and CN Blacklist is supported from v4.0 onwards. For more information, see [Manage BE and CN Blacklist](https://docs.starrocks.io/docs/administration/management/BE_blacklist.md). note Only users with the SYSTEM-level BLACKLIST privilege can perform this operation. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW { BACKEND | COMPUTE NODE } BLACKLIST ``` #### Return value[​](#return-value "Direct link to Return value") | **Return** | **Description** | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | AddBlackListType | How the BE/CN node was added to the blacklist. `MANUAL` indicates it is manually blacklisted by the user. `AUTO` indicates it is automatically blacklisted by StarRocks. | | LostConnectionTime | For the `MANUAL` type, it indicates the time when the BE/CN node was manually added to the blacklist.
For the `AUTO` type, it indicates the time when the last successful connection was established. | | LostConnectionNumberInPeriod | The number of disconnections detected within `CheckTimePeriod(s)`. | | CheckTimePeriod(s) | The interval at which StarRocks checks the connection status of the blacklisted BE/CN nodes. Its value is evaluated to the value you specified for the FE configuration item `black_host_history_sec`. Unit: Seconds. | #### Examples[​](#examples "Direct link to Examples") ```sql -- View the BE Blacklist. SHOW BACKEND BLACKLIST; +-----------+------------------+---------------------+------------------------------+--------------------+ | BackendId | AddBlackListType | LostConnectionTime | LostConnectionNumberInPeriod | CheckTimePeriod(s) | +-----------+------------------+---------------------+------------------------------+--------------------+ | 10001 | MANUAL | 2024-04-28 11:52:09 | 0 | 5 | +-----------+------------------+---------------------+------------------------------+--------------------+ -- View the CN Blacklist. SHOW COMPUTE NODE BLACKLIST; +---------------+------------------+---------------------+------------------------------+--------------------+ | ComputeNodeId | AddBlackListType | LostConnectionTime | LostConnectionNumberInPeriod | CheckTimePeriod(s) | +---------------+------------------+---------------------+------------------------------+--------------------+ | 10005 | MANUAL | 2025-08-18 10:47:51 | 0 | 5 | +---------------+------------------+---------------------+------------------------------+--------------------+ ``` #### Relevant SQLs[​](#relevant-sqls "Direct link to Relevant SQLs") * [ADD BACKEND/COMPUTE NODE BLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ADD_BACKEND_BLACKLIST.md) * [DELETE BACKEND/COMPUTE NODE BLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/DELETE_BACKEND_BLACKLIST.md) * [SHOW BACKENDS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md) * [SHOW COMPUTE NODES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md) --- ### SHOW BACKENDS SHOW BACKENDS shows the information of all BE nodes in the cluster. tip Only users with the SYSTEM-level OPERATE privilege or the `cluster_admin` role can perform this operation. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW BACKENDS ``` #### Return[​](#return "Direct link to Return") ```sql +-----------+--------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+--------------+--------------------------------------------------------+-------------------+-------------+----------+----------+-------------------+------------+------------+--------------------------------------------------------+----------+------------+ | BackendId | IP | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime | LastHeartbeat | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | ErrMsg | Version | Status | DataTotalCapacity | DataUsedPct | CpuCores | MemLimit | NumRunningQueries | MemUsedPct | CpuUsedPct | DataCacheMetrics | Location | StatusCode | +-----------+--------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+--------------+--------------------------------------------------------+-------------------+-------------+----------+----------+-------------------+------------+------------+--------------------------------------------------------+----------+------------+ ``` | **Return** | **Description** | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | BackendId | The ID of the BE node. | | IP | The IP address of the BE node. | | HeartbeatPort | The heartbeat port of the BE node. It is used to receive heartbeats from the FE node. | | BePort | Thrift server port of the BE node. It is used to receive requests from the FE node. | | HttpPort | HTTP server port of the BE node. It is used to access the BE node via web page. | | BrpcPort | bRPC port of the BE node. It is used for communication across BE nodes. | | LastStartTime | The last time when the BE node is started. | | LastHeartbeat | The last time when the FE node sends a heartbeat and the BE responds. | | Alive | Whether the BE node is alive or not.- `true`: the BE node is alive.
- `false`: the BE node is not alive. | | SystemDecommissioned | Whether the BE node is decommissioned or not. Before it is decommissioned, the BE node migrates the data to other BE nodes. Data loading and queries are not affected during migration.- `true`: the cluster has marked the BE node as decommissioned. The data could be successfully migrated or migrating.
- `false`: the BE node is running. | | ClusterDecommissioned | This parameter is used for system compatibility. | | TabletNum | The number of tablets on the BE node. | | DataUsedCapacity | The storage capacity occupied by the data file. | | AvailCapacity | Available storage capacity in the BE node. | | TotalCapacity | The total storage capacity. It is equivalent to `DataUsedCapacity` + `AvailCapacity` + Storage capacity occupied by the non-data files. | | UsedPct | The percentage of the used storage capacity. | | MaxDiskUsedPct | If a BE node has multiple directories to store data, this parameter shows the maximum percentage of the used storage capacity among all directories. | | ErrMsg | The error message returned when the BE node fails to receive the heartbeat from the FE node. | | Version | The StarRocks version of cluster. | | Status | The last time when the BE node reports the tablet number to the FE node. Displayed in JSON format. | | DataTotalCapacity | It is equivalent to `DataUsedCapacity` + `AvailCapacity`. It indicates the sum of the storage capacity occupied by the data file and the available storage capacity in the BE node. | | DataUsedPct | It is equivalent to `DataUsedCapacity`/`DataTotalCapacity`. It indicates the proportion of the storage capacity occupied by the data file to the sum of the data-occupied storage capacity and the available storage capacity. | | CpuCores | The number of CPU cores in the BE node. | | MemLimit | The memory limit of the BE node. | | NumRunningQueries | The number of running queries on the BE node. | | MemUsedPct | The percentage of used memory. | | CpuUsedPct | The percentage of used CPU cores. | | DataCacheMetrics | The status of Data Cache:- `Disabled`: Disabled
- `Normal`: Normal
- `Abnormal`: Abnormal
- `Updating`: Data Cache is being updating, for example, scaling. | | Location | The label of the BE node. | | StatusCode | BE status code. Valid values: - `CONNECTING`: The BE node is being added to the cluster for the first time.
- `OK`: The BE node is running.
- `SHUTDOWN`: The BE node is being shut down gracefully.
- `DISCONNECTED`: The BE node is disconnected. | #### Example[​](#example "Direct link to Example") View the information of all BE nodes in the cluster. ```plain SHOW BACKENDS; +-----------+---------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+----------------------+--------------------------------------------------------+-------------------+-------------+----------+ | BackendId | IP | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime | LastHeartbeat | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | ErrMsg | Version | Status | DataTotalCapacity | DataUsedPct | CpuCores | +-----------+---------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+----------------------+--------------------------------------------------------+-------------------+-------------+----------+ | 10002 | x.x.x.x | 9254 | 9260 | 8238 | 8260 | 2022-09-08 14:37:10 | 2022-09-08 15:14:21 | true | false | false | 21753 | 25.122 GB | 1.088 TB | 1.968 TB | 44.72 % | 44.72 % | | MAIN-RELEASE-d094052 | {"lastSuccessReportTabletsTime":"2022-09-08 15:14:13"} | 1.113 TB | 2.20 % | 16 | | 10003 | x.x.x.x | 9254 | 9260 | 8238 | 8260 | 2022-09-08 14:37:20 | 2022-09-08 15:14:21 | true | false | false | 21754 | 25.121 GB | 1.169 TB | 1.968 TB | 40.61 % | 40.61 % | | MAIN-RELEASE-d094052 | {"lastSuccessReportTabletsTime":"2022-09-08 15:14:22"} | 1.194 TB | 2.06 % | 16 | | 10004 | x.x.x.x | 9254 | 9260 | 8238 | 8260 | 2022-09-08 14:37:39 | 2022-09-08 15:14:21 | true | false | false | 21754 | 25.112 GB | 1.172 TB | 1.968 TB | 40.46 % | 40.46 % | | MAIN-RELEASE-d094052 | {"lastSuccessReportTabletsTime":"2022-09-08 15:13:42"} | 1.197 TB | 2.05 % | 16 | +-----------+---------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+----------------------+--------------------------------------------------------+-------------------+-------------+----------+ ``` --- ### SHOW BROKER SHOW BROKER views the information of all brokers that have been added to the StarRocks cluster. tip Only users with the SYSTEM-level OPERATE privilege or the `cluster_admin` role can perform this operation. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW BROKER ``` #### Return[​](#return "Direct link to Return") ```sql +-----------+-----------+------+-------+---------------------+---------------------+--------+ | Name | IP | Port | Alive | LastStartTime | LastUpdateTime | ErrMsg | +-----------+-----------+------+-------+---------------------+---------------------+--------+ ``` The following table describes the parameters returned by this statement. | **Parameter** | **Description** | | -------------- | ------------------------------------------------------------------------------------------------------------- | | Name | The name of the broker. | | IP | The IP address where the broker is running. | | Port | The port used by the broker (default: 8000). | | Alive | Whether the broker node is reachable. - true: The broker is alive.
- false: The broker is not reachable. | | LastStartTime | The last time when the broker process was started. | | LastUpdateTime | The last time the broker reported a heartbeat to the Frontend (FE). | | ErrMsg | Displays error messages if the heartbeat fails or connection is lost. | #### Example[​](#example "Direct link to Example") View the information of all brokers in the cluster. ```plain SHOW BROKER; +-----------+-----------+------+-------+---------------------+---------------------+--------+ | Name | IP | Port | Alive | LastStartTime | LastUpdateTime | ErrMsg | +-----------+-----------+------+-------+---------------------+---------------------+--------+ | my_broker | 127.0.0.1 | 8000 | true | 2026-01-21 12:40:00 | 2026-01-21 12:40:05 | | +-----------+-----------+------+-------+---------------------+---------------------+--------+ ``` --- ### SHOW COMPUTE NODES SHOW COMPUTE NODES shows the information of all CN nodes in the cluster. tip Only users with the SYSTEM-level OPERATE privilege or the `cluster_admin` role can perform this operation. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW COMPUTE NODES ``` #### Return[​](#return "Direct link to Return") ```sql +---------------+--------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+--------+--------------------+----------+-------------------+------------+------------+----------------+-------------+----------+-------------------+-----------+------------+ | ComputeNodeId | IP | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime | LastHeartbeat | Alive | SystemDecommissioned | ClusterDecommissioned | ErrMsg | Version | CpuCores | NumRunningQueries | MemUsedPct | CpuUsedPct | HasStoragePath | StarletPort | WorkerId | WarehouseName | TabletNum | StatusCode | +---------------+--------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+--------+--------------------+----------+-------------------+------------+------------+----------------+-------------+----------+-------------------+-----------+------------+ ``` The following table describes the parameters returned by this statement. | **Parameter** | **Description** | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ComputeNodeId | The ID of the CN node. | | IP | The IP address of the CN node. | | HeartbeatPort | The heartbeat port of the CN node. It is used to receive heartbeats from the FE node. | | BePort | Thrift server port of the CN node. It is used to receive requests from the FE node. | | HttpPort | HTTP server port of the CN node. It is used to access the CN node via web page. | | BrpcPort | bRPC port of the CN node. It is used for communication across CN nodes. | | LastStartTime | The last time at which the CN node starts. | | LastHeartbeat | The last time at which the CN node sends a heartbeat. | | Alive | Whether the CN node is alive or not.- `true`: the CN node is alive.
- `false`: the CN node is not alive. | | SystemDecommissioned | If the value of the parameter is `true`, the CN node is removed from your StarRocks cluster. | | ClusterDecommissioned | This parameter is used for system compatibility. | | ErrMsg | The error message if the CN node fails to send a heartbeat. | | Version | The StarRocks version of the CN node. | | CpuCores | The number of CPU cores in the CN node. | | NumRunningQueries | The number of running queries on the CN node. | | MemUsedPct | The percentage of used memory. | | CpuUsedPct | The percentage of used CPU cores. | | HasStoragePath | Whether the CN node has storage paths configured. | | StarletPort | The `starlet_port` of the CN node. It is an extra agent service port. | | WorkerId | The ID of the CN node for internal scheduling. | | WarehouseName | The name of the warehouse to which the CN node belongs. The value is always `default_warehouse`. | | TabletNum | The number of tablets (of cached data) on the CN node. | | StatusCode | CN status code. Valid values: - `CONNECTING`: The CN node is being added to the cluster for the first time.
- `OK`: The CN node is running.
- `SHUTDOWN`: The CN node is being shut down gracefully.
- `DISCONNECTED`: The CN node is disconnected. | #### Example[​](#example "Direct link to Example") View the information of all CN nodes in the cluster. ```plain MySQL > SHOW COMPUTE NODES\G *************************** 1. row *************************** ComputeNodeId: 10001 IP: x.x.x.x HeartbeatPort: 9050 BePort: 9060 HttpPort: 8040 BrpcPort: 8060 LastStartTime: 2024-05-14 15:45:34 LastHeartbeat: 2024-05-14 15:47:59 Alive: true SystemDecommissioned: false ClusterDecommissioned: false ErrMsg: Version: 3.3.0-rc01-3b8cb0c CpuCores: 4 NumRunningQueries: 0 MemUsedPct: 1.95 % CpuUsedPct: 0.0 % HasStoragePath: true StarletPort: 8167 WorkerId: 1 WarehouseName: default_warehouse TabletNum: 58 1 row in set (0.00 sec) ``` --- ### SHOW FRONTENDS SHOW FRONTENDS views the information of all FE nodes in the cluster, including their IP addresses, ports, roles, and status. tip Only users with the SYSTEM-level OPERATE privilege or the `cluster_admin` role can perform this operation. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW FRONTENDS ``` #### Return[​](#return "Direct link to Return") ```sql +------+------------------------------+-----------+-------------+----------+-----------+---------+--------+-----------+------+-------+-------------------+---------------------+----------+--------+---------------------+---------------+ | Id | Name | IP | EditLogPort | HttpPort | QueryPort | RpcPort | Role | ClusterId | Join | Alive | ReplayedJournalId | LastHeartbeat | IsHelper | ErrMsg | StartTime | Version | +------+------------------------------+-----------+-------------+----------+-----------+---------+--------+-----------+------+-------+-------------------+---------------------+----------+--------+---------------------+---------------+ ``` The following table describes the parameters returned by this statement. | **Parameter** | **Description** | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Id | The unique ID of the FE node. | | Name | The name of the FE node in BDBJE (typically IP\_EditLogPort\_Timestamp). | | IP | The IP address of the FE node. | | EditLogPort | The port used for BDBJE communication (default: 9010). | | HttpPort | The HTTP server port of the FE node (default: 8030). Used for accessing the FE web UI. | | QueryPort | The MySQL protocol port (default: 9030). Used by MySQL clients to connect to StarRocks. | | RpcPort | The Thrift server port (default: 9020). Used for internal RPC communication. | | Role | The role of the FE node. - LEADER: The master node, handles metadata writes.
- FOLLOWER: Participates in elections and metadata syncing.
- OBSERVER: Syncs metadata but does not participate in elections. | | ClusterId | The unique ID of the cluster. All nodes in the same cluster must have the same ClusterId. | | Join | Whether the node has joined the BDBJE group. - true: Joined.
- false: Not joined (note: a node might be removed but still show true temporarily). | | Alive | Whether the node is alive and responding. - true: Alive.
- false: Not alive. | | ReplayedJournalId | The maximum metadata journal ID that the node has currently replayed. Used to check if the node is in sync with the Leader. | | LastHeartbeat | The timestamp of the last heartbeat received. | | IsHelper | Whether the node is a Helper node (specified during cluster startup to help other nodes join). | | ErrMsg | Error message displayed if the heartbeat fails | | StartTime | The time when this FE process was started. | | Version | The StarRocks version running on this FE node. | #### Example[​](#example "Direct link to Example") View the information of all FE nodes in the cluster. ```plain mysql> SHOW FRONTENDS\G; *************************** 1. row *************************** Id: 1 Name: 127.0.0.1_9010_1766983864084 IP: 127.0.0.1 EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: LEADER ClusterId: 391198626 Join: true Alive: true ReplayedJournalId: 59720 LastHeartbeat: 2026-01-21 13:24:20 IsHelper: true ErrMsg: StartTime: 2026-01-20 12:58:48 Version: 4.0.2-1f1aa9c 1 row in set (0.01 sec) ``` --- ### SHOW PROC SHOW PROC shows certain indicators of the StarRocks cluster. SHOW PROC is equivalent to the the `/proc` filesystem in Linux. Similar to the paths in `/proc`, the parameters in SHOW PROC have a tree-like hierarchical structure, and each layer displays different information. Take `SHOW PROC '/dbs'` as an example: * `SHOW PROC '/dbs'` will return the information of all databases in the cluster. * `SHOW PROC '/dbs/'` will return the information of the specified database. * `SHOW PROC '/dbs//tables'` will return the information of all tables in the specified database. * `SHOW PROC '/dbs//'` will return the information of the specified table. * `SHOW PROC '/dbs///partitions'` will return the information of all partitions in the specified table. It can be specified all the way to the leaf node. tip This operation requires the SYSTEM-level OPERATE privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW PROC { '/backends' | '/compute_nodes' | '/dbs' | '/jobs' | '/statistic' | '/tasks' | '/frontends' | '/brokers' | '/resources' | '/load_error_hub' | '/transactions' | '/monitor' | '/current_queries' | '/current_backend_instances' | '/cluster_balance' | '/routine_loads' | '/colocation_group' | '/catalog' | '/replications' | '/global_current_queries' } ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------- | | '/backends' | Shows the information of BE nodes in the cluster. | | '/compute\_nodes' | Shows the information of CN nodes in the cluster. | | '/dbs' | Shows the information of databases, tables, partitions, and tablets in the cluster. | | '/jobs' | Shows the information of jobs in the cluster. | | '/statistic' | Shows the statistics of each database in the cluster. | | '/tasks' | Shows the total number of all generic tasks and the failed tasks in the cluster. | | '/frontends' | Shows the information of FE nodes in the cluster. | | '/brokers' | Shows the information of Broker nodes in the cluster. | | '/resources' | Shows the information of resources in the cluster. | | '/load\_error\_hub' | Shows the configuration of the cluster's Load Error Hub, which is used to manage error messages of loading jobs. | | '/transactions' | Shows the information of transactions in the cluster. | | '/monitor' | Shows the monitoring information in the cluster. | | '/current\_queries' | Shows the information of running queries on the current FE node. | | '/current\_backend\_instances' | Shows the BE nodes that are processing requests in the cluster. | | '/cluster\_balance' | Shows the load balance information in the cluster. | | '/routine\_loads' | Shows the information of Routine Load in the cluster. | | '/colocation\_group' | Shows the information of Colocate Join groups in the cluster. | | '/catalog' | Shows the information of catalogs in the cluster. | | '/replications' | Shows the information of data replication tasks in the cluster. | | '/global\_current\_queries' | Shows the information of running queries on all FE nodes in the cluster. | #### Examples[​](#examples "Direct link to Examples") Example 1: Shows the information of BE nodes in the cluster. ```plain mysql> SHOW PROC '/backends'\G *************************** 1. row *************************** BackendId: 10004 IP: xxx.xx.92.200 HeartbeatPort: 9354 BePort: 9360 HttpPort: 8338 BrpcPort: 8360 LastStartTime: 2023-04-21 09:56:10 LastHeartbeat: 2023-04-21 09:56:10 Alive: true SystemDecommissioned: false ClusterDecommissioned: false TabletNum: 2199 DataUsedCapacity: 0.000 AvailCapacity: 584.578 GB TotalCapacity: 1.968 TB UsedPct: 71.00 % MaxDiskUsedPct: 71.00 % ErrMsg: Version: BRANCH-3.0-RELEASE-8eb8705 Status: {"lastSuccessReportTabletsTime":"N/A"} DataTotalCapacity: 584.578 GB DataUsedPct: 0.00 % CpuCores: 16 NumRunningQueries: 0 MemUsedPct: 0.52 % CpuUsedPct: 0.0 % ``` | **Return** | **Description** | | --------------------- | ----------------------------------------------------------------------------------------------------------- | | BackendId | ID of the BE node. | | IP | IP address of the BE node. | | HeartbeatPort | Heartbeat service port of the BE node. | | BePort | Thrift Server port of the BE node. | | HttpPort | HTTP Server port of the BE node. | | BrpcPort | bRPC port of the BE node. | | LastStartTime | The last time when the BE node was started. | | LastHeartbeat | The last time when the BE node received a heartbeat. | | Alive | If the BE node is alive. | | SystemDecommissioned | If the BE node is decommissioned. | | ClusterDecommissioned | If the BE node is decommissioned within the cluster. | | TabletNum | Number of tablets in the BE node. | | DataUsedCapacity | Storage capacity that is used for data in the BE node. | | AvailCapacity | Available storage capacity in the BE node. | | TotalCapacity | Total storage capacity in the BE node. | | UsedPct | Percentage at which the storage capacity is used in the BE node. | | MaxDiskUsedPct | The maximum percentage at which the storage capacity is used in the BE node. | | ErrMsg | Error messages in the BE node. | | Version | StarRocks version of the BE node. | | Status | Status information of the BE node, including the last time when the BE node reported tablets. | | DataTotalCapacity | Total of used and available data storage capacity. The sum of `DataUsedCapacity` and `AvailCapacity`. | | DataUsedPct | Percentage at which the data storage takes up the total data capacity (DataUsedCapacity/DataTotalCapacity). | | CpuCores | Number of CPU cores in the BE node. | | NumRunningQueries | The number of queries currently running in the cluster. | | MemUsedPct | The current memory usage percentage. | | CpuUsedPct | The current CPU usage percentage. | Example 2: Shows the information of databases in the cluster. ```plain mysql> show proc "/dbs"; +-------+--------------------+----------+----------------+--------------------------+---------------------+ | DbId | DbName | TableNum | Quota | LastConsistencyCheckTime | ReplicaQuota | +-------+--------------------+----------+----------------+--------------------------+---------------------+ | 1 | information_schema | 54 | 8388608.000 TB | NULL | 9223372036854775807 | | 100 | sys | 6 | 8388608.000 TB | NULL | 9223372036854775807 | | 10001 | _statistics_ | 12 | 8388608.000 TB | NULL | 9223372036854775807 | | 10015 | test | 1 | 8388608.000 TB | NULL | 9223372036854775807 | +-------+--------------------+----------+----------------+--------------------------+---------------------+ 4 rows in set (0.00 sec) mysql> show proc "/dbs/test"; +---------+------------------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+--------------------------------------------------------------------------------------------------------+ | TableId | TableName | IndexNum | PartitionColumnName | PartitionNum | State | Type | LastConsistencyCheckTime | ReplicaCount | PartitionType | StoragePath | +---------+------------------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+--------------------------------------------------------------------------------------------------------+ | 10207 | source_wiki_edit | 1 | event_time | 4 | NORMAL | CLOUD_NATIVE | NULL | 8 | RANGE | s3://test/xxx | +---------+------------------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+--------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) mysql> show proc "/dbs/test/source_wiki_edit"; +-----------------+ | Nodes | +-----------------+ | partitions | | temp_partitions | | index_schema | +-----------------+ 3 rows in set (0.00 sec) ``` | **Return** | **Description** | | ------------------------ | ------------------------------------------------------- | | DbId | Database ID. | | DbName | Database name. | | TableNum | Number of tables in the database. | | Quota | Storage quota of the database. | | LastConsistencyCheckTime | The last time when consistency check is executed. | | ReplicaQuota | Data replica quota of the database. | | TableId | Table ID. | | TableName | Table name. | | PartitionColumnName | Partition column name. | | PartitionNum | Number of partitions in the table. | | State | State of the table. | | Type | Type of the table. | | LastConsistencyCheckTime | The time when the last consistency check was performed. | | ReplicaCount | Number of replicas for the table. | | PartitionType | Type of the partitions in the table. | | StoragePath | Storage path of the table. | | partitions | Partitions in the table. | | temp\_partitions | Temporary partitions in the table. | | index\_schema | Synchronous materialized view in the table. | Example 3: Shows the information of jobs in the cluster. ```plain mysql> SHOW PROC '/jobs'; +-------+--------------------------------------+ | DbId | DbName | +-------+--------------------------------------+ | 10005 | default_cluster:_statistics_ | | 0 | default_cluster:information_schema | | 12711 | default_cluster:starrocks_audit_db__ | +-------+--------------------------------------+ 3 rows in set (0.00 sec) mysql> SHOW PROC '/jobs/10005'; +---------------+---------+---------+----------+-----------+-------+ | JobType | Pending | Running | Finished | Cancelled | Total | +---------------+---------+---------+----------+-----------+-------+ | load | 0 | 0 | 3 | 0 | 3 | | rollup | 0 | 0 | 0 | 0 | 0 | | schema_change | 0 | 0 | 0 | 0 | 0 | | export | 0 | 0 | 0 | 0 | 0 | +---------------+---------+---------+----------+-----------+-------+ 4 rows in set (0.00 sec) ``` | **Return** | **Description** | | ---------- | ---------------------------------- | | DbId | Database ID. | | DbName | Database name. | | JobType | Job type. | | Pending | Number of jobs that are pending. | | Running | Number of jobs that are running. | | Finished | Number of jobs that are finished. | | Cancelled | Number of jobs that are cancelled. | | Total | Total number of jobs. | Example 4: Shows the statistics of each database in the cluster. ```plain mysql> SHOW PROC '/statistic'; +--------+----------------------------------------------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+---------------------+ | DbId | DbName | TableNum | PartitionNum | IndexNum | TabletNum | ReplicaNum | UnhealthyTabletNum | InconsistentTabletNum | CloningTabletNum | ErrorStateTabletNum | +--------+----------------------------------------------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+---------------------+ | 10004 | _statistics_ | 3 | 3 | 3 | 30 | 60 | 0 | 0 | 0 | 0 | | 1 | information_schema | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | 92498 | stream_load_test_db_03afc714_b1cb_11ed_a82c_00163e237e98 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | 92542 | stream_load_test_db_79876e92_b1da_11ed_b50e_00163e237e98 | 1 | 1 | 1 | 3 | 3 | 0 | 0 | 0 | 0 | | 115476 | testdb | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | 10002 | zq_test | 8 | 8 | 8 | 5043 | 7063 | 0 | 0 | 0 | 2 | | Total | 6 | 12 | 12 | 12 | 5076 | 7126 | 0 | 0 | 0 | 2 | +--------+----------------------------------------------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+---------------------+ 7 rows in set (0.01 sec) mysql> show proc '/statistic/10002'; +------------------+---------------------+----------------+-------------------+ | UnhealthyTablets | InconsistentTablets | CloningTablets | ErrorStateTablets | +------------------+---------------------+----------------+-------------------+ | [] | [] | [] | [116703, 116706] | +------------------+---------------------+----------------+-------------------+ ``` | **Return** | **Description** | | --------------------- | ------------------------------------------------------------------------------------ | | DbId | Database ID. | | DbName | Database name. | | TableNum | Number of tables in the database. | | PartitionNum | Number of partitions in the database. | | IndexNum | Number of indexes in the database. | | TabletNum | Number of tablets in the database. | | ReplicaNum | Number of replicas in the database. | | UnhealthyTabletNum | Number of unfinished (unhealthy) tablets in the database during data redistribution. | | InconsistentTabletNum | Number of inconsistent tablets in the database. | | CloningTabletNum | Number of tablets that are being cloned in the database. | | ErrorStateTabletNum | In a Primary Key type table, the number of tablets in Error state. | | ErrorStateTablets | In a Primary Key type table, the IDs of the tablets in Error state. | Example 5: Shows the total number of all generic tasks and the failed tasks in the cluster. ```plain mysql> SHOW PROC '/tasks'; +-------------------------+-----------+----------+ | TaskType | FailedNum | TotalNum | +-------------------------+-----------+----------+ | CREATE | 0 | 0 | | DROP | 0 | 0 | | PUSH | 0 | 0 | | CLONE | 0 | 0 | | STORAGE_MEDIUM_MIGRATE | 0 | 0 | | ROLLUP | 0 | 0 | | SCHEMA_CHANGE | 0 | 0 | | CANCEL_DELETE | 0 | 0 | | MAKE_SNAPSHOT | 0 | 0 | | RELEASE_SNAPSHOT | 0 | 0 | | CHECK_CONSISTENCY | 0 | 0 | | UPLOAD | 0 | 0 | | DOWNLOAD | 0 | 0 | | CLEAR_REMOTE_FILE | 0 | 0 | | MOVE | 0 | 0 | | REALTIME_PUSH | 0 | 0 | | PUBLISH_VERSION | 0 | 0 | | CLEAR_ALTER_TASK | 0 | 0 | | CLEAR_TRANSACTION_TASK | 0 | 0 | | RECOVER_TABLET | 0 | 0 | | STREAM_LOAD | 0 | 0 | | UPDATE_TABLET_META_INFO | 0 | 0 | | ALTER | 0 | 0 | | INSTALL_PLUGIN | 0 | 0 | | UNINSTALL_PLUGIN | 0 | 0 | | NUM_TASK_TYPE | 0 | 0 | | Total | 0 | 0 | +-------------------------+-----------+----------+ ``` | **Return** | **Description** | | ---------- | ----------------------- | | TaskType | Task type. | | FailedNum | Number of failed tasks. | | TotalNum | Total number of tasks. | Example 6: Shows the information of FE nodes in the cluster. ```plain mysql> SHOW PROC '/frontends'; +----------------------------------+---------------+-------------+----------+-----------+---------+----------+------------+-------+-------+-------------------+---------------+----------+---------------+-----------+---------+ | Name | IP | EditLogPort | HttpPort | QueryPort | RpcPort | Role | ClusterId | Join | Alive | ReplayedJournalId | LastHeartbeat | IsHelper | ErrMsg | StartTime | Version | +----------------------------------+---------------+-------------+----------+-----------+---------+----------+------------+-------+-------+-------------------+---------------+----------+---------------+-----------+---------+ | xxx.xx.xx.xxx_9009_1600088918395 | xxx.xx.xx.xxx | 9009 | 7390 | 0 | 0 | FOLLOWER | 1747363037 | false | false | 0 | NULL | true | got exception | NULL | NULL | +----------------------------------+---------------+-------------+----------+-----------+---------+----------+------------+-------+-------+-------------------+---------------+----------+---------------+-----------+---------+ ``` | **Return** | **Description** | | ----------------- | ------------------------------------------------------ | | Name | FE node name. | | IP | IP address of the FE node. | | EditLogPort | Port for communication between FE nodes. | | HttpPort | HTTP Server port of the FE node. | | QueryPort | MySQL Server port of the FE node. | | RpcPort | RPC port of the FE node. | | Role | Role of the FE node (Leader, Follower, or Observer). | | ClusterId | Cluster ID. | | Join | If the FE node has joined a cluster. | | Alive | If the FE node is alive. | | ReplayedJournalId | The largest metadata ID that the FE node has replayed. | | LastHeartbeat | The last time when the FE node sent a heartbeat. | | IsHelper | If the FE node is the BDBJE helper node. | | ErrMsg | Error messages in the FE node. | | StartTime | Time when the FE node is started. | | Version | StarRocks version of the FE node. | Example 7: Shows the information of Broker nodes in the cluster. ```plain mysql> SHOW PROC '/brokers'; +-------------+---------------+------+-------+---------------+---------------------+--------+ | Name | IP | Port | Alive | LastStartTime | LastUpdateTime | ErrMsg | +-------------+---------------+------+-------+---------------+---------------------+--------+ | hdfs_broker | xxx.xx.xx.xxx | 8500 | true | NULL | 2022-10-10 16:37:59 | | | hdfs_broker | xxx.xx.xx.xxx | 8500 | true | NULL | 2022-10-10 16:37:59 | | | hdfs_broker | xxx.xx.xx.xxx | 8500 | true | NULL | 2022-10-10 16:37:59 | | +-------------+---------------+------+-------+---------------+---------------------+--------+ ``` | **Return** | **Description** | | -------------- | ---------------------------------------------------------------------------- | | Name | Broker node name. | | IP | IP address of the broker node. | | Port | Thrift Server port of the broker node. The port is used to receive requests. | | Alive | If the broker node is alive. | | LastStartTime | The last time when the broker node was started. | | LastUpdateTime | The last time when the broker node was updated. | | ErrMsg | Error message in the broker node. | Example 8: Shows the information of resources in the cluster. ```plain mysql> SHOW PROC '/resources'; +-------------------------+--------------+---------------------+------------------------------+ | Name | ResourceType | Key | Value | +-------------------------+--------------+---------------------+------------------------------+ | hive_resource_stability | hive | hive.metastore.uris | thrift://xxx.xx.xxx.xxx:9083 | | hive2 | hive | hive.metastore.uris | thrift://xxx.xx.xx.xxx:9083 | +-------------------------+--------------+---------------------+------------------------------+ ``` | **Return** | **Description** | | ------------ | --------------- | | Name | Resource name. | | ResourceType | Resource type. | | Key | Resource key. | | Value | Resource value. | Example 9: Shows the information of transactions in the cluster. ```plain mysql> SHOW PROC '/transactions'; +-------+--------------------------------------+ | DbId | DbName | +-------+--------------------------------------+ | 10005 | default_cluster:_statistics_ | | 12711 | default_cluster:starrocks_audit_db__ | +-------+--------------------------------------+ 2 rows in set (0.00 sec) mysql> SHOW PROC '/transactions/10005'; +----------+--------+ | State | Number | +----------+--------+ | running | 0 | | finished | 4 | +----------+--------+ 2 rows in set (0.00 sec) ``` | **Return** | **Description** | | ---------- | ----------------------------- | | DbId | Database ID. | | DbName | Database name. | | State | The state of the transaction. | | Number | Number of transactions. | Example 10: Shows the monitoring information in the cluster. ```plain mysql> SHOW PROC '/monitor'; +------+------+ | Name | Info | +------+------+ | jvm | | +------+------+ ``` | **Return** | **Description** | | ---------- | ---------------- | | Name | JVM name. | | Info | JVM information. | Example 11: Shows the load balance information in the cluster. ```plain mysql> SHOW PROC '/cluster_balance'; +-------------------+--------+ | Item | Number | +-------------------+--------+ | balance_stat | 1 | | cluster_load_stat | 1 | | working_slots | 3 | | sched_stat | 1 | | priority_repair | 0 | | pending_tablets | 2001 | | running_tablets | 0 | | history_tablets | 1000 | +-------------------+--------+ ``` | **Return** | **Description** | | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Item | Sub-command item in `cluster_balance`. - balance\_stat: The current balance status of the cluster.
- cluster\_load\_stat: The current load status of the cluster.
- working\_slots: The number of currently available working slots.
- sched\_stat: The current status of the scheduler.
- priority\_repair: The number of Tablet repair tasks that are prioritized.
- pending\_tablets: The number of Tablets waiting to be processed.
- running\_tablets: The number of Tablets currently being repaired.
- history\_tablets: The total number of Tablets repaired historically. | | Number | Number of each sub-command in `cluster_balance`. | ```plain mysql> SHOW PROC '/cluster_balance/balance_stat'; +---------------+--------------------------------+----------+----------------+----------------+ | StorageMedium | BalanceType | Balanced | PendingTablets | RunningTablets | +---------------+--------------------------------+----------+----------------+----------------+ | HDD | inter-node disk usage | true | 0 | 0 | | HDD | inter-node tablet distribution | true | 0 | 0 | | HDD | intra-node disk usage | true | 0 | 0 | | HDD | intra-node tablet distribution | true | 0 | 0 | | HDD | colocation group | true | 0 | 0 | | HDD | label-aware location | true | 0 | 0 | +---------------+--------------------------------+----------+----------------+----------------+ ``` | **Return** | **Description** | | -------------- | ------------------------------------------- | | StorageMedium | Storage medium. | | BalanceType | Type of balance. | | Balanced | Whether the balanced state is achieved. | | PendingTablets | Number of tablets with task status Pending. | | RunningTablets | Number of tablets with task status Running. | Example 12: Shows the information of Colocate Join groups in the cluster. ```plain mysql> SHOW PROC '/colocation_group'; +-----------------+----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+------------+----------------+-------------+----------+ | GroupId | GroupName | TableIds | BucketsNum | ReplicationNum | DistCols | IsStable | +-----------------+----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+------------+----------------+-------------+----------+ | 24010.177354 | 24010_lineitem_str_g1 | 177672 | 12 | 1 | varchar(-1) | true | | 24010.182146 | 24010_lineitem_str_g2 | 182144 | 192 | 1 | varchar(-1) | true | | 1439318.1735496 | 1439318_group_agent_uid | 1735677, 1738390 | 12 | 2 | bigint(20) | true | | 24010.37804 | 24010_gsdaf2449s9e | 37802 | 192 | 1 | int(11) | true | | 174844.175370 | 174844_groupa4 | 175368, 591307, 591362, 591389, 591416 | 12 | 1 | int(11) | true | | 24010.30587 | 24010_group2 | 30585, 30669 | 12 | 1 | int(11) | true | | 10005.181366 | 10005_lineorder_str_normal | 181364 | 192 | 1 | varchar(-1) | true | | 1904968.5973175 | 1904968_groupa2 | 5973173 | 12 | 1 | int(11) | true | | 24010.182535 | 24010_lineitem_str_g3 | 182533 | 192 | 1 | varchar(-1) | true | +-----------------+----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+------------+----------------+-------------+----------+ ``` | **Return** | **Description** | | -------------- | ----------------------------------------------- | | GroupId | Colocate Join Group ID. | | GroupName | Colocate Join Group name. | | TableIds | IDs of tables in the Colocate Join Group. | | BucketsNum | Buckets in the Colocate Join Group. | | ReplicationNum | Replications in the Colocate Join Group. | | DistCols | Distribution column of the Colocate Join Group. | | IsStable | If the Colocate Join Group is stable. | Example 13: Shows the information of catalogs in the cluster. ```plain mysql> SHOW PROC '/catalog'; +--------------------------------------------------------------+----------+----------------------+ | Catalog | Type | Comment | +--------------------------------------------------------------+----------+----------------------+ | resource_mapping_inside_catalog_hive_hive2 | hive | mapping hive catalog | | resource_mapping_inside_catalog_hive_hive_resource_stability | hive | mapping hive catalog | | default_catalog | Internal | Internal Catalog | +--------------------------------------------------------------+----------+----------------------+ ``` | **Return** | **Description** | | ---------- | ------------------------- | | Catalog | Catalog name. | | Type | Catalog type. | | Comment | Comments for the catalog. | Example 14: Shows the information of replication tasks in the cluster. ```plain mysql> SHOW PROC '/replications'; +-------------------------------------------------+------------+---------+-------+---------------------+---------------------+-----------+----------+-------+ | JobID | DatabaseID | TableID | TxnID | CreatedTime | FinishedTime | State | Progress | Error | +-------------------------------------------------+------------+---------+-------+---------------------+---------------------+-----------+----------+-------+ | FAILOVER_GROUP_group1-11006-11010-1725593360156 | 11006 | 11010 | 99 | 2024-09-06 11:29:20 | 2024-09-06 11:29:21 | COMMITTED | | | | FAILOVER_GROUP_group1-11006-11009-1725593360161 | 11006 | 11009 | 98 | 2024-09-06 11:29:20 | 2024-09-06 11:29:21 | COMMITTED | | | | FAILOVER_GROUP_group1-11006-11074-1725593360161 | 11006 | 11074 | 100 | 2024-09-06 11:29:20 | 2024-09-06 11:29:21 | COMMITTED | | | | FAILOVER_GROUP_group1-11006-12474-1725593360250 | 11006 | 12474 | 102 | 2024-09-06 11:29:20 | 2024-09-06 11:29:24 | COMMITTED | | | | FAILOVER_GROUP_group1-11006-11024-1725593360293 | 11006 | 11024 | 101 | 2024-09-06 11:29:20 | 2024-09-06 11:29:24 | COMMITTED | | | | FAILOVER_GROUP_group1-11006-13861-1725607270963 | 11006 | 13861 | 627 | 2024-09-06 15:21:10 | 2024-09-06 15:21:14 | COMMITTED | | | +-------------------------------------------------+------------+---------+-------+---------------------+---------------------+-----------+----------+-------+ ``` | **Return** | **Description** | | ------------ | --------------------------------------------------------------------------------------------- | | JobID | Job ID. | | DatabaseID | Database ID. | | TableID | Table ID. | | TxnID | Transaction ID. | | CreatedTime | Time when the task was created. | | FinishedTime | Time when the task finished. | | State | Status of the task. Valid values: INITIALIZING, SNAPSHOTING, REPLICATING, COMMITTED, ABORTED. | | Progress | Progress of the task. | | Error | Error message (if any). | **Example 15: Shows the information of running queries on the current FE node.** ```sql MySQL > show proc '/current_queries'; +---------------------+---------------+--------------------------------------+--------------+----------+------+------------+--------------+-------------+---------------+----------+----------+-----------+-------------------+---------------+---------------+ | StartTime | feIp | QueryId | ConnectionId | Database | User | ScanBytes | ScanRows | MemoryUsage | DiskSpillSize | CPUTime | ExecTime | ExecState | Warehouse | CustomQueryId | ResourceGroup | +---------------------+---------------+--------------------------------------+--------------+----------+------+------------+--------------+-------------+---------------+----------+----------+-----------+-------------------+---------------+---------------+ | 2025-03-07 02:00:19 | 172.26.92.227 | ddbd69b9-fab4-11ef-8063-461f20abc3f0 | 11 | tpcds_2 | root | 120.573 MB | 5859503 rows | 296.432 MB | 0.000 B | 27.888 s | 3.153 s | RUNNING | default_warehouse | | rg1 | +---------------------+---------------+--------------------------------------+--------------+----------+------+------------+--------------+-------------+---------------+----------+----------+-----------+-------------------+---------------+---------------+ ``` **Example 16: Shows the information of running queries on all FE nodes in the cluster.** ```sql MySQL > show proc '/global_current_queries'; +---------------------+---------------+--------------------------------------+--------------+----------+------+------------+--------------+-------------+---------------+---------+----------+-----------+-------------------+---------------+---------------+ | StartTime | feIp | QueryId | ConnectionId | Database | User | ScanBytes | ScanRows | MemoryUsage | DiskSpillSize | CPUTime | ExecTime | ExecState | Warehouse | CustomQueryId | ResourceGroup | +---------------------+---------------+--------------------------------------+--------------+----------+------+------------+--------------+-------------+---------------+---------+----------+-----------+-------------------+---------------+---------------+ | 2025-03-07 02:02:47 | 172.26.92.227 | 3603d566-fab5-11ef-8063-461f20abc3f0 | 12 | tpcds_2 | root | 100.886 MB | 4899036 rows | 114.491 MB | 0.000 B | 5.700 s | 0.713 s | RUNNING | default_warehouse | | rg1 | +---------------------+---------------+--------------------------------------+--------------+----------+------+------------+--------------+-------------+---------------+---------+----------+-----------+-------------------+---------------+---------------+ | **Return** | **Description** | | ------------ | ------------------------------------------------------------------- | | StartTime | The time when the query started execution. | | feIp | The IP address of the FE node processing the query. | | QueryId | The unique identifier for the query. | | ConnectionId | The connection ID associated with the query. | | Database | The database where the query is executed. | | User | The user who submitted the query. | | ScanBytes | The amount of data scanned by the query. | | ScanRows | The number of rows scanned by the query. | | MemoryUsage | The current memory usage of the query. | | DiskSpillSize| The amount of data spilled to disk during query execution. | | CPUTime | The total CPU time consumed by the query. | | ExecTime | The total execution time of the query. | | ExecState | The execution state of the query. Valid values: `RUNNING`, `PENDING`. | | Warehouse | The compute warehouse processing the query. | | CustomQueryId| Custom query identifier if specified. | | ResourceGroup| The resource group assigned to the query. | | QueryType | The type of the query: `Query`, `Statistics` (such as ANALYZE), `Task`, `MV`, or `Internal`. | ``` --- ### SHOW PROCESSLIST SHOW PROCESSLIST lists the operations currently being performed by threads executing within the server. The current version of StarRocks only supports listing queries. tip This operation does not require privileges. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW [FULL] PROCESSLIST ``` #### Parameter[​](#parameter "Direct link to Parameter") | Parameter | Required | Description | | --------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | FULL | No | If you specify this parameter, the complete SQL statement will be displayed. Otherwise, only the first 100 characters of the statement are displayed. | #### Return[​](#return "Direct link to Return") | Return | Description | | ------------------- | ---------------------------------------------------------------------------- | | Server | Server ID. | | Id | Connection ID. | | User | The name of the user who runs the operation. | | Host | The hostname of the client which runs the operation. | | Db | The name of the database where the operation is executed. | | Command | The type of the command. | | ConnectionStartTime | Time when the connection starts. | | Time | The time (in second) since the operation has entered the current state. | | State | The state of the operation. | | Info | The command that the operation is executing. | | IsPending | Whether the query is pending in the queue. Valid values: `true` and `false`. | | Warehouse | The name of the warehouse where the query is executed. | | CNGroup | The name of the compute node group where the query is run. | | Catalog | The name of the catalog. | | QueryId | The qury ID if the Command is "Query". | #### Usage note[​](#usage-note "Direct link to Usage note") If the current user is `root`, this statement lists the operations of all users in the cluster. Otherwise, only operations of the current user are listed. The `IsPending`, `Warehouse`, and `CNGroup` fields provide additional information about query execution in warehouse environments: * `IsPending`: Shows whether a query is waiting in the queue (`true`) or actively executing (`false`) * `Warehouse`: Displays the warehouse name where the query is being executed * `CNGroup`: Shows the compute node group name responsible for executing the query #### Example[​](#example "Direct link to Example") Example 1: lists the operations state via the user `root`. ```sql SHOW PROCESSLIST\G *************************** 6. row *************************** ServerName: starrocks-fe_9010_1782850099498 Id: 33560554 User: root Host: 172.18.0.4:54818 Db: Command: Query ConnectionStartTime: 2026-07-02 01:21:14 Time: 0 State: OK Info: show processlist IsPending: false Warehouse: default_warehouse CNGroup: Catalog: default_catalog QueryId: 019f1eb3-246c-7899-ab3e-40a018645bba ``` --- ### SHOW RUNNING QUERIES ### `SHOW RUNNING QUERIES` `SHOW RUNNING QUERIES` displays information of all queries that are running or pending in the query queue. This feature is supported from v3.1.4 onwards. tip Query queues must be configured, they are off by default. Learn more in the [query queues documentation](https://docs.starrocks.io/docs/administration/management/resource_management/query_queues.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW RUNNING QUERIES ``` #### Return[​](#return "Direct link to Return") * `QueryId`: The ID of the query. * `ResourceGroupId`: The ID of the resource group that the query hit. When there is no hit on a user-defined resource group, it will be displayed as "-". * `StartTime`: The start time of the query. * `PendingTimeout`: The time when the PENDING query will time out in the queue. * `QueryTimeout`: The time when the query times out. * `State`: The queue state of the query, where "PENDING" indicates it is in the queue, and "RUNNING" indicates it is currently executing. * `Slots`: The logical resource quantity requested by the query, currently fixed at `1`. * `Frontend`: The FE node that initiated the query. * `FeStartTime`: The start time of the FE node that initiated the query. #### Example[​](#example "Direct link to Example") ```plain MySQL [(none)]> SHOW RUNNING QUERIES; +--------------------------------------+-----------------+---------------------+---------------------+---------------------+-----------+-------+---------------------------------+---------------------+ | QueryId | ResourceGroupId | StartTime | PendingTimeout | QueryTimeout | State | Slots | Frontend | FeStartTime | +--------------------------------------+-----------------+---------------------+---------------------+---------------------+-----------+-------+---------------------------------+---------------------+ | a46f68c6-3b49-11ee-8b43-00163e10863a | - | 2023-08-15 16:56:37 | 2023-08-15 17:01:37 | 2023-08-15 17:01:37 | RUNNING | 1 | 127.00.00.01_9010_1692069711535 | 2023-08-15 16:37:03 | | a6935989-3b49-11ee-935a-00163e13bca3 | 12003 | 2023-08-15 16:56:40 | 2023-08-15 17:01:40 | 2023-08-15 17:01:40 | RUNNING | 1 | 127.00.00.02_9010_1692069658426 | 2023-08-15 16:37:03 | | a7b5e137-3b49-11ee-8b43-00163e10863a | 12003 | 2023-08-15 16:56:42 | 2023-08-15 17:01:42 | 2023-08-15 17:01:42 | PENDING | 1 | 127.00.00.03_9010_1692069711535 | 2023-08-15 16:37:03 | +--------------------------------------+-----------------+---------------------+---------------------+---------------------+-----------+-------+---------------------------------+---------------------+ ``` --- ### SYNC Synchronizes the data consistency among different sessions. Currently, StarRocks can only guarantee consistency within the same session, that is, only the session that initiated the data write operation can read the latest data immediately after the operation succeeds. If you want to read the latest data immediately from other sessions, you must synchronize the data consistency using the SYNC statement. Generally, there will be a millisecond-level latency among sessions if you do not execute this statement. tip This operation does not require privileges. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SYNC ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> SYNC; Query OK, 0 rows affected (0.00 sec) ``` --- ### Execution plan and profile #### [📄️ ANALYZE PROFILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/ANALYZE_PROFILE.md) [ANALYZE PROFILE analyzes a specific query profile on a per-fragment basis, and displays it in a tree structure.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/ANALYZE_PROFILE.md) #### [📄️ EXPLAIN](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/EXPLAIN.md) [EXPLAIN shows the logical or physical execution plans for a query statement.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/EXPLAIN.md) #### [📄️ EXPLAIN ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/EXPLAIN_ANALYZE.md) [EXPLAIN ANALYZE runs a SQL statement and returns its query profile with per-operator execution statistics.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/EXPLAIN_ANALYZE.md) #### [📄️ SHOW PROFILELIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/SHOW_PROFILELIST.md) [SHOW PROFILELIST lists the query profile records cached in your StarRocks cluster.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/SHOW_PROFILELIST.md) --- ### ANALYZE PROFILE ANALYZE PROFILE analyzes a specific query profile on a per-fragment basis, and displays it in a tree structure. For more information about query profile, see [Query Profile Overview](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md). This feature is supported from v3.1 onwards. > **CAUTION** > > By default, no privilege is required to perform this operation. If the FE configuration item `authorization_enable_query_profile_access_check` is set to `true`, a user can analyze only the profiles of the queries they ran, and analyzing the profile of a query run by another user requires the SYSTEM-level OPERATE privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ANALYZE PROFILE FROM '', [[, ...] ] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | query\_id | The ID of the query. You can obtain it using [SHOW PROFILELIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/SHOW_PROFILELIST.md). | | plan\_node\_id | The ID of the plan node in the profile. You can specify this parameter to view the detailed metrics of the corresponding plan node(s). If this parameter is not specified, only the summary metrics of all plan nodes are displayed. | #### Examples[​](#examples "Direct link to Examples") Example 1: Querying the Query Profile without specifying node ID. ![img](/assets/images/text_based_profile_without_node_id-07ed49a240ec43e9f3a498dcefba5c70.jpeg) Example 2: Querying the Query Profile and specifying node ID as `0`. StarRocks returns all detailed metrics for Node ID `0` and highlights metrics with high usage for easier problem identification. ![img](/assets/images/text_based_profile_with_node_id-ed71ee1e7f25bdbb97752e03e7be34d1.jpeg) #### Relevant SQLs[​](#relevant-sqls "Direct link to Relevant SQLs") * [SHOW PROFILELIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/SHOW_PROFILELIST.md) * [EXPLAIN ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/EXPLAIN_ANALYZE.md) --- ### EXPLAIN EXPLAIN shows the logical or physical execution plans for a query statement. For instructions on how to analyze a query plan, refer to [Plan analysis](https://docs.starrocks.io/docs/best_practices/query_tuning/query_planning.md#plan-analysis). tip This operation does not require privileges. #### Syntax[​](#syntax "Direct link to Syntax") ```sql EXPLAIN [ LOGICAL | VERBOSE | COSTS ] ``` tip The level of execution plan detail returned: EXPLAIN LOGICAL < EXPLAIN < EXPLAIN VERBOSE < EXPLAIN COSTS. From v3.3.5, the default detail level is changed to `COSTS`. You can configure the level of details returned by EXPLAIN using the dynamic FE parameter `query_detail_explain_level`. In most cases, using EXPLAIN is sufficient. EXPLAIN VERBOSE and EXPLAIN COSTS will print a lot of internal information, which is mainly used for debugging the execution plan. #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | --------------- | ------------------------------------------------------------------------------------------------------------------ | | LOGICAL | Shows the simple logical execution plans. | | VERBOSE | Shows the detailed logical execution plans, including data types, nullable information, and optimization strategy. | | COSTS (Default) | Shows the detailed logical execution plans, including column statistics. | | query | The query statement whose execution plan you want to view. | > **NOTE** > > You can configure the level of details returned by EXPLAIN using the dynamic FE parameter `query_detail_explain_level`. #### Return value[​](#return-value "Direct link to Return value") Returns the execution plan of the input query statement. | **Return** | **Description** | | ----------------- | ---------------------------------------------------------------------------------------------- | | avgRowSize | The average size of the scanned data rows. | | cardinality | The total number of data rows in the scanned table. | | colocate | Whether the table is in colocate mode. | | numNodes | The number of nodes to scan. | | rollup | The Rollup materialized view that is used. | | preaggregation | The status of preaggregation. | | predicates | Predicates in the query. | | column statistics | The statictics of the column，sequence: MIN, MAX, NULL numbers，average size，and cardinality. | #### Examples[​](#examples "Direct link to Examples") ##### EXPLAIN[​](#explain-1 "Direct link to EXPLAIN") ```plain MySQL tpcds> explain select sr_customer_sk as ctr_customer_sk, sr_store_sk as ctr_store_sk, sum(SR_RETURN_AMT) as ctr_total_return from store_returns, date_dim where sr_returned_date_sk = d_date_sk and d_year = 2000 group by sr_customer_sk, sr_store_sk +-----------------------------------------------------------------------------+ | Explain String | +-----------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS:5: sr_customer_sk | 9: sr_store_sk | 49: sum | | PARTITION: UNPARTITIONED | | | | RESULT SINK | | | | 9:EXCHANGE | | | | PLAN FRAGMENT 1 | | OUTPUT EXPRS: | | PARTITION: HASH_PARTITIONED: 5: sr_customer_sk, 9: sr_store_sk | | | | STREAM DATA SINK | | EXCHANGE ID: 09 | | UNPARTITIONED | | | | 8:AGGREGATE (merge finalize) | | | output: sum(49: sum) | | | group by: 5: sr_customer_sk, 9: sr_store_sk | | | | | 7:EXCHANGE | | | | PLAN FRAGMENT 2 | | OUTPUT EXPRS: | | PARTITION: RANDOM | | | | STREAM DATA SINK | | EXCHANGE ID: 07 | | HASH_PARTITIONED: 5: sr_customer_sk, 9: sr_store_sk | | | | 6:AGGREGATE (update serialize) | | | STREAMING | | | output: sum(12: sr_return_amt) | | | group by: 5: sr_customer_sk, 9: sr_store_sk | | | | | 5:Project | | | : 5: sr_customer_sk | | | : 9: sr_store_sk | | | : 12: sr_return_amt | | | | | 4:HASH JOIN | | | join op: INNER JOIN (BROADCAST) | | | colocate: false, reason: | | | equal join conjunct: 3: sr_returned_date_sk = 21: d_date_sk | | | | | |----3:EXCHANGE | | | | | 0:OlapScanNode | | TABLE: store_returns | | PREAGGREGATION: ON | | PREDICATES: 3: sr_returned_date_sk IS NOT NULL | | partitions=1/1 | | rollup: store_returns | | tabletRatio=10/10 | | tabletList=10241,10243,10245,10247,10249,10251,10253,10255,10257,10259 | | cardinality=277502 | | avgRowSize=20.0 | | | | PLAN FRAGMENT 3 | | OUTPUT EXPRS: | | PARTITION: RANDOM | | | | STREAM DATA SINK | | EXCHANGE ID: 03 | | UNPARTITIONED | | | | 2:Project | | | : 21: d_date_sk | | | | | 1:OlapScanNode | | TABLE: date_dim | | PREAGGREGATION: ON | | PREDICATES: 27: d_year = 2000 | | partitions=1/1 | | rollup: date_dim | | tabletRatio=5/5 | | tabletList=11543,11545,11547,11549,11551 | | cardinality=362 | | avgRowSize=8.0 | +-----------------------------------------------------------------------------+ 79 rows in set ``` ##### EXPLAIN LOGICAL[​](#explain-logical "Direct link to EXPLAIN LOGICAL") ```plain MySQL tpcds> explain logical select sr_customer_sk as ctr_customer_sk, sr_store_sk as ctr_store_sk, sum(SR_RETURN_AMT) as ctr_total_return from store_returns, date_dim where sr_returned_date_sk = d_date_sk and d_year = 2000 group by sr_customer_sk, sr_store_sk +---------------------------------------------------------------------------------------------------------------------------------+ | Explain String | +---------------------------------------------------------------------------------------------------------------------------------+ | - Output => [5:sr_customer_sk, 9:sr_store_sk, 49:sum] | | - AGGREGATE(GLOBAL) [5:sr_customer_sk, 9:sr_store_sk] | | Estimates: {row: 49728, cpu: 795660.58, memory: 795660.58, network: 0.00, cost: 8405375.45} | | 49:sum := sum(49:sum) | | - EXCHANGE(SHUFFLE) [5, 9] | | Estimates: {row: 49728, cpu: 79566.06, memory: 0.00, network: 0.00, cost: 6416223.99} | | - AGGREGATE(LOCAL) [5:sr_customer_sk, 9:sr_store_sk] | | Estimates: {row: 49728, cpu: 119349.09, memory: 79566.06, network: 0.00, cost: 6376440.96} | | 49:sum := sum(12:sr_return_amt) | | - HASH/INNER JOIN [3:sr_returned_date_sk = 21:d_date_sk] => [5:sr_customer_sk, 9:sr_store_sk, 12:sr_return_amt] | | Estimates: {row: 49728, cpu: 6744977.39, memory: 1446.51, network: 0.00, cost: 6157634.30} | | - SCAN [store_returns] => [3:sr_returned_date_sk, 5:sr_customer_sk, 9:sr_store_sk, 12:sr_return_amt] | | Estimates: {row: 277502, cpu: 5550040.00, memory: 0.00, network: 0.00, cost: 2775020.00} | | partitionRatio: 1/1, tabletRatio: 10/10 | | predicate: 3:sr_returned_date_sk IS NOT NULL | | - EXCHANGE(BROADCAST) | | Estimates: {row: 361, cpu: 1446.51, memory: 1446.51, network: 1446.51, cost: 7232.57} | | - SCAN [date_dim] => [21:d_date_sk] | | Estimates: {row: 361, cpu: 2893.03, memory: 0.00, network: 0.00, cost: 1446.51} | | partitionRatio: 1/1, tabletRatio: 5/5 | | predicate: 27:d_year = 2000 | +---------------------------------------------------------------------------------------------------------------------------------+ 21 rows in set ``` ##### EXPLAIN VERBOSE[​](#explain-verbose "Direct link to EXPLAIN VERBOSE") ```plain MySQL tpcds> explain verbose select sr_customer_sk as ctr_customer_sk, sr_store_sk as ctr_store_sk, sum(SR_RETURN_AMT) as ctr_total_return from store_returns, date_dim where sr_returned_date_sk = d_date_sk and d_year = 2000 group by sr_customer_sk, sr_store_sk +---------------------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String | +---------------------------------------------------------------------------------------------------------------------------------------------------------+ | RESOURCE GROUP: default_wg | | | | PLAN FRAGMENT 0(F04) | | Fragment Cost: 0.0 | | Output Exprs:5: sr_customer_sk | 9: sr_store_sk | 49: sum | | Input Partition: UNPARTITIONED | | RESULT SINK | | | | 9:EXCHANGE | | cardinality: 49729 | | | | PLAN FRAGMENT 1(F03) | | Fragment Cost: 2028934.4876271961 | | | | Input Partition: HASH_PARTITIONED: 5: sr_customer_sk, 9: sr_store_sk | | OutPut Partition: UNPARTITIONED | | OutPut Exchange Id: 09 | | | | 8:AGGREGATE (merge finalize) | | | aggregate: sum[([49: sum, DECIMAL128(38,2), true]); args: DECIMAL64; result: DECIMAL128(38,2); args nullable: true; result nullable: true] | | | group by: [5: sr_customer_sk, INT, true], [9: sr_store_sk, INT, true] | | | cardinality: 49729 | | | | | 7:EXCHANGE | | distribution type: SHUFFLE | | partition exprs: [5: sr_customer_sk, INT, true], [9: sr_store_sk, INT, true] | | cardinality: 49729 | | | | PLAN FRAGMENT 2(F00) | | Fragment Cost: 6376440.959353934 | | | | Input Partition: RANDOM | | OutPut Partition: HASH_PARTITIONED: 5: sr_customer_sk, 9: sr_store_sk | | OutPut Exchange Id: 07 | | | | 6:AGGREGATE (update serialize) | | | STREAMING | | | aggregate: sum[([12: sr_return_amt, DECIMAL64(7,2), true]); args: DECIMAL64; result: DECIMAL128(38,2); args nullable: true; result nullable: true] | | | group by: [5: sr_customer_sk, INT, true], [9: sr_store_sk, INT, true] | | | cardinality: 49729 | | | | | 5:Project | | | output columns: | | | 5 <-> [5: sr_customer_sk, INT, true] | | | 9 <-> [9: sr_store_sk, INT, true] | | | 12 <-> [12: sr_return_amt, DECIMAL64(7,2), true] | | | cardinality: 49729 | | | | | 4:HASH JOIN | | | join op: INNER JOIN (BROADCAST) | | | equal join conjunct: [3: sr_returned_date_sk, INT, true] = [21: d_date_sk, INT, false] | | | build runtime filters: | | | - filter_id = 0, build_expr = (21: d_date_sk), remote = false | | | output columns: 5, 9, 12 | | | can local shuffle: true | | | cardinality: 49729 | | | | | |----3:EXCHANGE | | | distribution type: BROADCAST | | | cardinality: 362 | | | | | 0:OlapScanNode | | table: store_returns, rollup: store_returns | | preAggregation: on | | Predicates: 3: sr_returned_date_sk IS NOT NULL | | partitionsRatio=1/1, tabletsRatio=10/10 | | tabletList=10241,10243,10245,10247,10249,10251,10253,10255,10257,10259 | | actualRows=287514, avgRowSize=20.0 | | cardinality: 277502 | | probe runtime filters: | | - filter_id = 0, probe_expr = (3: sr_returned_date_sk) | | | | PLAN FRAGMENT 3(F01) | | Fragment Cost: 1446.5148514851485 | | | | Input Partition: RANDOM | | OutPut Partition: UNPARTITIONED | | OutPut Exchange Id: 03 | | | | 2:Project | | | output columns: | | | 21 <-> [21: d_date_sk, INT, false] | | | cardinality: 362 | | | | | 1:OlapScanNode | | table: date_dim, rollup: date_dim | | preAggregation: on | | Predicates: [27: d_year, INT, false] = 2000 | | partitionsRatio=1/1, tabletsRatio=5/5 | | tabletList=11543,11545,11547,11549,11551 | | actualRows=73049, avgRowSize=8.0 | | cardinality: 362 | +---------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` ##### EXPLAIN COSTS[​](#explain-costs "Direct link to EXPLAIN COSTS") ```plain +---------------------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String | +---------------------------------------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0(F04) | | Output Exprs:5: sr_customer_sk | 9: sr_store_sk | 49: sum | | Input Partition: UNPARTITIONED | | RESULT SINK | | | | 9:EXCHANGE | | cardinality: 49729 | | | | PLAN FRAGMENT 1(F03) | | | | Input Partition: HASH_PARTITIONED: 5: sr_customer_sk, 9: sr_store_sk | | OutPut Partition: UNPARTITIONED | | OutPut Exchange Id: 09 | | | | 8:AGGREGATE (merge finalize) | | | aggregate: sum[([49: sum, DECIMAL128(38,2), true]); args: DECIMAL64; result: DECIMAL128(38,2); args nullable: true; result nullable: true] | | | group by: [5: sr_customer_sk, INT, true], [9: sr_store_sk, INT, true] | | | cardinality: 49729 | | | column statistics: | | | * sr_customer_sk-->[1.0, 100000.0, 2.010867271218169E-5, 4.0, 49728.786461450894] ESTIMATE | | | * sr_store_sk-->[1.0, 10.0, 0.14285714285714285, 4.0, 6.0] ESTIMATE | | | * sum-->[0.0, 16917.12, 0.034878301578357924, 8.0, 49728.786461450894] ESTIMATE | | | | | 7:EXCHANGE | | distribution type: SHUFFLE | | partition exprs: [5: sr_customer_sk, INT, true], [9: sr_store_sk, INT, true] | | cardinality: 49729 | | | | PLAN FRAGMENT 2(F00) | | | | Input Partition: RANDOM | | OutPut Partition: HASH_PARTITIONED: 5: sr_customer_sk, 9: sr_store_sk | | OutPut Exchange Id: 07 | | | | 6:AGGREGATE (update serialize) | | | STREAMING | | | aggregate: sum[([12: sr_return_amt, DECIMAL64(7,2), true]); args: DECIMAL64; result: DECIMAL128(38,2); args nullable: true; result nullable: true] | | | group by: [5: sr_customer_sk, INT, true], [9: sr_store_sk, INT, true] | | | cardinality: 49729 | | | column statistics: | | | * sr_customer_sk-->[1.0, 100000.0, 2.010867271218169E-5, 4.0, 49728.786461450894] ESTIMATE | | | * sr_store_sk-->[1.0, 10.0, 0.14285714285714285, 4.0, 6.0] ESTIMATE | | | * sum-->[0.0, 16917.12, 0.034878301578357924, 8.0, 49728.786461450894] ESTIMATE | | | | | 5:Project | | | output columns: | | | 5 <-> [5: sr_customer_sk, INT, true] | | | 9 <-> [9: sr_store_sk, INT, true] | | | 12 <-> [12: sr_return_amt, DECIMAL64(7,2), true] | | | cardinality: 49729 | | | column statistics: | | | * sr_customer_sk-->[1.0, 100000.0, 0.034836564480338346, 4.0, 49728.786461450894] ESTIMATE | | | * sr_store_sk-->[1.0, 10.0, 0.034996556689413386, 4.0, 6.0] ESTIMATE | | | * sr_return_amt-->[0.0, 16917.12, 0.034878301578357924, 8.0, 49728.786461450894] ESTIMATE | | | | | 4:HASH JOIN | | | join op: INNER JOIN (BROADCAST) | | | equal join conjunct: [3: sr_returned_date_sk, INT, true] = [21: d_date_sk, INT, false] | | | build runtime filters: | | | - filter_id = 0, build_expr = (21: d_date_sk), remote = false | | | output columns: 5, 9, 12 | | | can local shuffle: true | | | cardinality: 49729 | | | column statistics: | | | * sr_returned_date_sk-->[2450820.0, 2452822.0, 0.0, 4.0, 361.6287128712871] ESTIMATE | | | * sr_customer_sk-->[1.0, 100000.0, 0.034836564480338346, 4.0, 49728.786461450894] ESTIMATE | | | * sr_store_sk-->[1.0, 10.0, 0.034996556689413386, 4.0, 6.0] ESTIMATE | | | * sr_return_amt-->[0.0, 16917.12, 0.034878301578357924, 8.0, 49728.786461450894] ESTIMATE | | | * d_date_sk-->[2450820.0, 2452822.0, 0.0, 4.0, 361.6287128712871] ESTIMATE | | | | | |----3:EXCHANGE | | | distribution type: BROADCAST | | | cardinality: 362 | | | | | 0:OlapScanNode | | table: store_returns, rollup: store_returns | | preAggregation: on | | Predicates: 3: sr_returned_date_sk IS NOT NULL | | partitionsRatio=1/1, tabletsRatio=10/10 | | tabletList=10241,10243,10245,10247,10249,10251,10253,10255,10257,10259 | | actualRows=287514, avgRowSize=20.0 | | cardinality: 277502 | | probe runtime filters: | | - filter_id = 0, probe_expr = (3: sr_returned_date_sk) | | column statistics: | | * sr_returned_date_sk-->[2450820.0, 2452822.0, 0.0, 4.0, 2018.0] ESTIMATE | | * sr_customer_sk-->[1.0, 100000.0, 0.034836564480338346, 4.0, 86524.0] ESTIMATE | | * sr_store_sk-->[1.0, 10.0, 0.034996556689413386, 4.0, 6.0] ESTIMATE | | * sr_return_amt-->[0.0, 16917.12, 0.034878301578357924, 8.0, 113968.0] ESTIMATE | | | | PLAN FRAGMENT 3(F01) | | | | Input Partition: RANDOM | | OutPut Partition: UNPARTITIONED | | OutPut Exchange Id: 03 | | | | 2:Project | | | output columns: | | | 21 <-> [21: d_date_sk, INT, false] | | | cardinality: 362 | | | column statistics: | | | * d_date_sk-->[2415022.0, 2488070.0, 0.0, 4.0, 361.6287128712871] ESTIMATE | | | | | 1:OlapScanNode | | table: date_dim, rollup: date_dim | | preAggregation: on | | Predicates: [27: d_year, INT, false] = 2000 | | partitionsRatio=1/1, tabletsRatio=5/5 | | tabletList=11543,11545,11547,11549,11551 | | actualRows=73049, avgRowSize=8.0 | | cardinality: 362 | | column statistics: | | * d_date_sk-->[2415022.0, 2488070.0, 0.0, 4.0, 361.6287128712871] ESTIMATE | | * d_year-->[2000.0, 2000.0, 0.0, 4.0, 202.0] ESTIMATE | +---------------------------------------------------------------------------------------------------------------------------------------------------------+ 114 rows in set ``` --- ### EXPLAIN ANALYZE EXPLAIN ANALYZE executes the specified SQL statement, and shows the query profile of this statement. For more information about query profile, see [Query Profile Overview](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md). This feature is supported from v3.1 onwards. > **CAUTION** > > Only users with the SELECT or INSERT privilege on a specific table can perform this operation. #### Syntax[​](#syntax "Direct link to Syntax") ```sql EXPLAIN ANALYZE ``` #### Parameters[​](#parameters "Direct link to Parameters") `statement`: the SQL statement of which to show the query profile. Both [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) and [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) are supported. #### Usage notes[​](#usage-notes "Direct link to Usage notes") Please note that when you analyze the profiles of an INSERT INTO statement, no data will actually be inserted. By default, the transaction is aborted, ensuring that no unintended changes are made to the data in the process of profile analysis. #### Examples[​](#examples "Direct link to Examples") Example 1: Simulate and analyze a SELECT statement. The query result are discarded. ![img](/assets/images/text_based_explain_analyze_select-54f611db544479a1b4df4aadab264388.jpeg) Example 2: Simulate and analyze an INSERT INTO statement. The loading transaction will be aborted. ![img](/assets/images/text_based_explain_analyze_insert-178eb5df5ee41c8a33f648ece8a9bff8.jpeg) #### Relevant SQLs[​](#relevant-sqls "Direct link to Relevant SQLs") * [SHOW PROFILELIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/SHOW_PROFILELIST.md) * [ANALYZE PROFILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/ANALYZE_PROFILE.md) --- ### SHOW PROFILELIST SHOW PROFILELIST lists the query profile records cached in your StarRocks cluster. For more information about query profile, see [Query Profile Overview](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md). This feature is supported from v3.1 onwards. By default, no privilege is required to perform this operation. If the FE configuration item `authorization_enable_query_profile_access_check` is set to `true`, a user can list only the profiles of the queries they ran, and listing the profiles of queries run by other users requires the SYSTEM-level OPERATE privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW PROFILELIST [LIMIT n] ``` #### Parameters[​](#parameters "Direct link to Parameters") `LIMIT n`: lists n most recent records. #### Return value[​](#return-value "Direct link to Return value") | **Return** | **Description** | | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | QueryId | The ID of the query. | | StartTime | The start time of the query. | | Time | The latency of the query. | | State | The status of the query, including:`Error`: The query encounters an error.`Finished`: The query is finished.`Running`: The query is running. | | Statement | The statement of the query. | #### Examples[​](#examples "Direct link to Examples") Example 1: Show five most recent query profile records. ```sql SHOW PROFILELIST LIMIT 5; ``` #### Relevant SQLs[​](#relevant-sqls "Direct link to Relevant SQLs") * [ANALYZE PROFILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/ANALYZE_PROFILE.md) * [EXPLAIN ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/EXPLAIN_ANALYZE.md) --- ### Plugin #### [📄️ INSTALL PLUGIN](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md) [INSTALL PLUGIN is used to install a plugin.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md) #### [📄️ SHOW PLUGINS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plugin/SHOW_PLUGINS.md) [SHOW PLUGINS is used to view the installed plugins.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plugin/SHOW_PLUGINS.md) #### [📄️ UNINSTALL PLUGIN](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plugin/UNINSTALL_PLUGIN.md) [UNINSTALL PLUGIN is used to uninstall a plugin.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plugin/UNINSTALL_PLUGIN.md) --- ### INSTALL PLUGIN INSTALL PLUGIN is used to install a plugin. tip This operation requires the SYSTEM-level PLUGIN privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql INSTALL PLUGIN FROM [source] [PROPERTIES ("key"="value", ...)] ``` 3 types of sources are supported: ```plain 1. An absolute path that directs to a zip file 2. An absolute path that directs to a plugin directory 3. A http or https download link that directs to a zip file ``` PROPERTIES supports setting some configurations of plugins, such as setting the md5sum value of the zip file, etc. #### Examples[​](#examples "Direct link to Examples") 1. Install a plugin from local zip file: ```sql INSTALL PLUGIN FROM "/home/users/starrocks/auditdemo.zip"; ``` 2. Install a plugin from local inpath: ```sql INSTALL PLUGIN FROM "/home/users/starrocks/auditdemo/"; ``` 3. Download and install a plugin: ```sql INSTALL PLUGIN FROM "http://mywebsite.com/plugin.zip"; ``` 4. Download and install a plugin. Meanwhile, set the md5sum value of the zip file: ```sql INSTALL PLUGIN FROM "http://mywebsite.com/plugin.zip" PROPERTIES("md5sum" = "73877f6029216f4314d712086a146570"); ``` --- ### SHOW PLUGINS SHOW PLUGINS is used to view the installed plugins. tip This operation requires the SYSTEM-level PLUGIN privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW PLUGINS ``` This command will display all built-in plugins and custom plugins. #### Examples[​](#examples "Direct link to Examples") 1. Show the installed plugins: ```sql SHOW PLUGINS; ``` --- ### UNINSTALL PLUGIN UNINSTALL PLUGIN is used to uninstall a plugin. tip This operation requires the SYSTEM-level PLUGIN privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql UNINSTALL PLUGIN ``` plugin\_name can be viewed through SHOW PLUGINS command Only non-builtin plugins can be uninstalled. #### Examples[​](#examples "Direct link to Examples") 1. Uninstall a plugin: ```sql UNINSTALL PLUGIN auditdemo; ``` --- ### Resource group #### [📄️ ALTER RESOURCE GROUP](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group/ALTER_RESOURCE_GROUP.md) [ALTER RESOURCE GROUP alters the configuration of a resource group.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group/ALTER_RESOURCE_GROUP.md) #### [📄️ CREATE RESOURCE GROUP](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md) [CREATE RESOURCE GROUP creates a resource group.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md) #### [📄️ DROP RESOURCE GROUP](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group/DROP_RESOURCE_GROUP.md) [DROP RESOURCE GROUP \[IF EXISTS\] drops the specified resource group.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group/DROP_RESOURCE_GROUP.md) #### [📄️ SHOW RESOURCE GROUP](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group/SHOW_RESOURCE_GROUP.md) [SHOW RESOURCE GROUP shows the information of resource groups.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group/SHOW_RESOURCE_GROUP.md) #### [📄️ SHOW USAGE RESOURCE GROUPS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group/SHOW_USAGE_RESOURCE_GROUPS.md) [SHOW USAGE RESOURCE GROUPS shows the usage information of resource groups.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group/SHOW_USAGE_RESOURCE_GROUPS.md) --- ### ALTER RESOURCE GROUP ALTER RESOURCE GROUP alters the configuration of a resource group. tip This operation requires the ALTER privilege on the target resource group. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ALTER RESOURCE GROUP resource_group_name { ADD CLASSIFIER1, CLASSIFIER2, ... | DROP (CLASSIFIER_ID_1, CLASSIFIER_ID_2, ...) | DROP ALL | WITH resource_limit }; ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | resource\_group\_name | Name of the resource group to be altered. | | ADD | Add classifiers to the resource group. See [CREATE RESOURCE GROUP - Parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md) for more information on how to define a classifier. | | DROP | Drop classifiers from the resource group via classifier IDs. You can check the ID of a classifier via [SHOW RESOURCE GROUP](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group/SHOW_RESOURCE_GROUP.md) statement. | | DROP ALL | Drop all classifiers from the resource group. | | WITH | Modify the resource limits of the resource group. See [CREATE RESOURCE GROUP - Parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md) for more information on how to set resource limits. | #### Examples[​](#examples "Direct link to Examples") Example 1: Adds a new classifier to the resource group `rg1`. ```sql ALTER RESOURCE GROUP rg1 ADD (user='root', query_type in ('select')); ``` Example 2: Drops classifiers with ID `300040`, `300041`, and `300041` from the resource group `rg1`. ```sql ALTER RESOURCE GROUP rg1 DROP (300040, 300041, 300041); ``` Example 3: Drops all classifiers from the resource group `rg1`. ```sql ALTER RESOURCE GROUP rg1 DROP ALL; ``` Example 4: modifies the resource limits of the resource group `rg1`. ```sql ALTER RESOURCE GROUP rg1 WITH ( 'cpu_weight' = '10' ); ``` --- ### CREATE RESOURCE GROUP CREATE RESOURCE GROUP creates a resource group. For more information, see [Resource group](https://docs.starrocks.io/docs/administration/management/resource_management/resource_group.md). tip This operation requires the SYSTEM-level CREATE RESOURCE GROUP privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE RESOURCE GROUP resource_group_name TO CLASSIFIER1, CLASSIFIER2, ... WITH resource_limit ``` #### Parameters[​](#parameters "Direct link to Parameters") * `resource_group_name`: The name of the resource group to be created. * `CLASSIFIER`: Classifiers used to filter the queries on which resource limits are imposed. You must specify a classifier using `"key"="value"` pairs. You can set multiple classifiers for a resource group. Parameters for classifiers are listed below: | **Parameter** | **Required** | **Description** | | ---------------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | user | No | The name of user. | | role | No | The role of user. | | query\_type | No | The type of query. `SELECT` and `INSERT` (from v2.5) are supported. When INSERT tasks hit a resource group with `query_type` as `insert`, the BE node reserves the specified CPU resources for the tasks. | | source\_ip | No | The CIDR block from which the query is initiated. | | db | No | The database which the query accesses. It can be specified by strings separated by commas (,). | | plan\_cpu\_cost\_range | No | The estimated CPU cost range of the query. Its value has an equivalent meaning with the field `PlanCpuCost` in **fe.audit.log**, and has no unit. The format is `[DOUBLE, DOUBLE)`. The default value is NULL, indicating no such restriction. This parameter is supported from v3.1.4 onwards. | | plan\_mem\_cost\_range | No | The estimated memory cost range of the query. Its value has an equivalent meaning with the field `PlanMemCost` in **fe.audit.log**, and has no unit. The format is `[DOUBLE, DOUBLE)`. The default value is NULL, indicating no such restriction. This parameter is supported from v3.1.4 onwards. | * `resource_limit`: Resource limits to be imposed on the resource group. You must specify resource limits using `"key"="value"` pairs. You can set multiple resource limits for a resource group. Parameters for resource limits are listed below: | **Parameter** | **Required** | **Description** | | ------------------------------ | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | cpu\_core\_limit | No | The soft limit for the number of CPU cores that can be allocated to the resource group on the BE. In actual business scenarios, CPU cores that are allocated to the resource group proportionally scale based on the availability of CPU cores on the BE. Valid values: any non-zero positive integer. | | cpu\_weight | No | Required parameter for creating Shared resource groups. It specifies the CPU scheduling weight of a resource group on a single BE node, determining the relative share of CPU time allocated to tasks from this group. In actual business scenarios, CPU cores that are allocated to the resource group proportionally scale based on the availability of CPU cores on the BE. Value range: (0, `avg_be_cpu_cores`], where `avg_be_cpu_cores` is the average number of CPU cores across all BE nodes. The parameter is effective only when it is set to greater than 0. Only one of `cpu_weight` or `exclusive_cpu_cores` can be set to greater than 0. | | exclusive\_cpu\_cores | No | Required parameter for creating Exclusive resource groups (with CPU hard limit). It means to reserve `exclusive_cpu_cores` CPU cores exclusively for this resource group, making them unavailable to other groups, even when idle, and to limit the resource group to only using these reserved CPU cores, preventing it from using available CPU resources from other groups. Value range: (0, `min_be_cpu_cores - 1`], where `min_be_cpu_cores` is the minimum number of CPU cores across all BE nodes. It takes effect only when greater than 0. Only one of `cpu_weight` or `exclusive_cpu_cores` can be set to greater than 0. | | mem\_limit | No | The percentage of memory that can be used for queries in the total memory that is provided by the BE. Unit: %. Valid values: (0, 1). | | concurrency\_limit | No | The upper limit of concurrent queries in a resource group. It is used to avoid system overload caused by too many concurrent queries. | | max\_cpu\_cores | No | The CPU core limit for this resource group on a single BE node. It takes effect only when it is set to greater than `0`. Range: \[0, `avg_be_cpu_cores`], where `avg_be_cpu_cores` represents the average number of CPU cores across all BE nodes. Default: 0. | | big\_query\_cpu\_second\_limit | No | The upper time limit of CPU occupation for a big query. Concurrent queries add up the time. The unit is second. | | big\_query\_scan\_rows\_limit | No | The upper limit of row counts that can be scanned by a big query. | | big\_query\_mem\_limit | No | The upper limit of memory usage of a big query. The unit is byte. | > **NOTE** > > Before v3.3.5, StarRocks allowed setting the `type` of a resource group to `short_query`. However, the parameter `type` has been deprecated and replaced by `exclusive_cpu_cores`. For any existing resource groups of this type, the system will automatically convert them to an Exclusive resource group where the `exclusive_cpu_cores` value equals the `cpu_weight` after upgrading to v3.3.5. #### Example[​](#example "Direct link to Example") Example 1: Creates a Shared resource group `rg1` based on multiple classifiers. ```sql CREATE RESOURCE GROUP rg1 TO (user='rg1_user1', role='rg1_role1', query_type in ('select'), source_ip='192.168.x.x/24'), (user='rg1_user2', query_type in ('select'), source_ip='192.168.x.x/24'), (user='rg1_user3', source_ip='192.168.x.x/24'), (user='rg1_user4'), (db='db1') WITH ('cpu_weight' = '10', 'mem_limit' = '20%', 'big_query_cpu_second_limit' = '100', 'big_query_scan_rows_limit' = '100000', 'big_query_mem_limit' = '1073741824' ); ``` Example 2: Creates an Exclusive resource group `rg2` based on multiple classifiers. ```sql CREATE RESOURCE GROUP rg2 TO (user='rg1_user5', role='rg1_role5', query_type in ('select'), source_ip='192.168.x.x/24'), (user='rg1_user6', query_type in ('select'), source_ip='192.168.x.x/24'), (user='rg1_user7', source_ip='192.168.x.x/24'), (user='rg1_user8'), (db='db2') WITH ('exclusive_cpu_cores' = '10', 'mem_limit' = '20%', 'type' = 'normal', 'big_query_cpu_second_limit' = '100', 'big_query_scan_rows_limit' = '100000', 'big_query_mem_limit' = '1073741824' ); ``` --- ### DROP RESOURCE GROUP DROP RESOURCE GROUP \[IF EXISTS] drops the specified resource group. tip This operation requires the DROP privilege on the target resource group. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DROP RESOURCE GROUP ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | --------------------- | ----------------------------------------- | | resource\_group\_name | Name of the resource group to be dropped. | #### Example[​](#example "Direct link to Example") Example 1: Drops the resource group `rg1`. ```sql DROP RESOURCE GROUP rg1; ``` --- ### SHOW RESOURCE GROUP SHOW RESOURCE GROUP shows the information of resource groups. tip This operation does not require privileges. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW { RESOURCE GROUPS [ALL] | RESOURCE GROUP } ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | --------------------- | ------------------------------------------------------------------ | | RESOURCE GROUPS | Show the resource groups that match the current user. | | ALL | Specify this parameter to show all resource groups in the cluster. | | RESOURCE GROUP | Show the specified resource group. | | resource\_group\_name | Name of the resource group to be shown. | #### Return[​](#return "Direct link to Return") | **Return** | **Description** | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | | name | Name of the resource group. | | id | ID of the resource group. | | cpu\_weight | CPU scheduling weight of this resource group on a BE node. | | exclusive\_cpu\_cores | CPU hard isolation parameter for this resource group. | | mem\_limit | Memory limit of the resource group. | | big\_query\_cpu\_second\_limit | Big query upper time limit of the resource group. | | big\_query\_scan\_rows\_limit | Big query scan row limit of the resource group. | | big\_query\_mem\_limit | Big query memory limit of the resource group. | | concurrency\_limit | Concurrency limit of the resource group. | | spill\_mem\_limit\_threshold | Memory usage threshold that triggers spilling to disk. | | classifiers | Classifiers that are associated with the resource group. `id` is the ID of the classifier, and `weight` is the degree of matching. | #### Examples[​](#examples "Direct link to Examples") Example 1: Shows all resource groups in the cluster. ```plain mysql> SHOW RESOURCE GROUPS ALL; +---------------+------+------------+---------------------+-----------+----------------------------+---------------------------+---------------------+-------------------+---------------------------+--------------------+ | name | id | cpu_weight | exclusive_cpu_cores | mem_limit | big_query_cpu_second_limit | big_query_scan_rows_limit | big_query_mem_limit | concurrency_limit | spill_mem_limit_threshold | classifiers | +---------------+------+------------+---------------------+-----------+----------------------------+---------------------------+---------------------+-------------------+---------------------------+--------------------+ | default_mv_wg | 3 | 1 | 0 | 80.0% | 0 | 0 | 0 | null | 80% | (id=0, weight=0.0) | | default_wg | 2 | 32 | 0 | 100.0% | 0 | 0 | 0 | null | 100% | (id=0, weight=0.0) | +---------------+------+------------+---------------------+-----------+----------------------------+---------------------------+---------------------+-------------------+---------------------------+--------------------+ ``` --- ### SHOW USAGE RESOURCE GROUPS SHOW USAGE RESOURCE GROUPS shows the usage information of resource groups. This feature is supported from v3.1.4 onwards. tip This operation does not require privileges. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW USAGE RESOURCE GROUPS ``` #### Return[​](#return "Direct link to Return") * `Name`: The name of the resource group. * `Id`: The ID of the resource group. * `Backend`: The BE's IP or FQDN. * `BEInUseCpuCores`: The number of CPU cores currently in use by this resource group on this BE. This value is an approximate estimate. * `BEInUseMemBytes`: The number of memory bytes currently in use by this resource group on this BE. * `BERunningQueries`: The number of queries from this resource group that are still running on this BE. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * BEs periodically report this resource usage information to the Leader FE at the interval specified in `report_resource_usage_interval_ms`, which is by default set to 1 second. * The results will only show rows where at least one of `BEInUseCpuCores`/`BEInUseMemBytes`/`BERunningQueries` is a positive number. In other words, the information is displayed only when a resource group is actively using some resources on a BE. #### Example[​](#example "Direct link to Example") ```plain MySQL [(none)]> SHOW USAGE RESOURCE GROUPS; +------------+----+-----------+-----------------+-----------------+------------------+ | Name | Id | Backend | BEInUseCpuCores | BEInUseMemBytes | BERunningQueries | +------------+----+-----------+-----------------+-----------------+------------------+ | default_wg | 0 | 127.0.0.1 | 0.100 | 1 | 5 | +------------+----+-----------+-----------------+-----------------+------------------+ | default_wg | 0 | 127.0.0.2 | 0.200 | 2 | 6 | +------------+----+-----------+-----------------+-----------------+------------------+ | wg1 | 0 | 127.0.0.1 | 0.300 | 3 | 7 | +------------+----+-----------+-----------------+-----------------+------------------+ | wg2 | 0 | 127.0.0.1 | 0.400 | 4 | 8 | +------------+----+-----------+-----------------+-----------------+------------------+ ``` --- ### SQL blacklist #### [📄️ ADD SQLBLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/sql_blacklist/ADD_SQLBLACKLIST.md) [ADD SQLBLACKLIST adds a regular expression to the SQL blacklist to forbid certain SQL patterns.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/sql_blacklist/ADD_SQLBLACKLIST.md) #### [📄️ DELETE SQLBLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/sql_blacklist/DELETE_SQLBLACKLIST.md) [DELETE SQLBLACKLIST deletes an SQL regular expression from the SQL blacklist.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/sql_blacklist/DELETE_SQLBLACKLIST.md) #### [📄️ SHOW SQLBLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/sql_blacklist/SHOW_SQLBLACKLIST.md) [SHOW SQLBLACKLIST shows the SQL regular expressions in the SQL blacklist.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/sql_blacklist/SHOW_SQLBLACKLIST.md) --- ### ADD SQLBLACKLIST ADD SQLBLACKLIST adds a regular expression to the SQL blacklist to forbid certain SQL patterns. When the SQL Blacklist feature is enabled, StarRocks compares all SQL statements to be executed against the SQL regular expressions in the blacklist. StarRocks does not execute SQLs that match any regular expression in the blacklist and returns an error. This prevents certain SQLs from triggering cluster crashes or unexpected behavior. For more about SQL Blacklist, see [Manage SQL Blacklist](https://docs.starrocks.io/docs/administration/management/resource_management/Blacklist.md). tip * This operation requires the SYSTEM-level BLACKLIST privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. * The blacklist is only for SELECT statements, INSERT statements (from v3.1 onwards), and CTAS statements (from v3.4 onwards). #### Syntax[​](#syntax "Direct link to Syntax") ```sql ADD SQLBLACKLIST "" ``` #### Parameter[​](#parameter "Direct link to Parameter") `sql_reg_expr`: the regular expression that is used to specify a certain SQL pattern. To distinguish the special characters in the SQL statement and those in the regular expression, you need to use the escape character `\` as a prefix for the special characters in the SQL statement, such as `(`, `)`, and `+`. Whereas `(` and `)` have been often used in SQL statements, StarRocks can identify `(` and `)` in SQL statements directly. You do not need to use the escape character for `(` and `)`. #### Examples[​](#examples "Direct link to Examples") Example 1: Add `count(\*)` to the SQL blacklist. ```plain mysql> ADD SQLBLACKLIST "select count(\\*) from .+"; ``` Example 2: Add `count(distinct )` to the SQL blacklist. ```plain mysql> ADD SQLBLACKLIST "select count(distinct .+) from .+"; ``` Example 3: Add `order by limit x, y, 1 <= x <=7, 5 <=y <=7` to the SQL blacklist. ```plain mysql> ADD SQLBLACKLIST "select id_int from test_all_type_select1 order by id_int limit [1-7], [5-7]"; ``` Example 4: Add a complex SQL regular expression to the SQL blacklist. This example is to demonstrate how to use escape characters for `*` and `-` in SQL statements. ```plain mysql> ADD SQLBLACKLIST "select id_int \\* 4, id_tinyint, id_varchar from test_all_type_nullable except select id_int, id_tinyint, id_varchar from test_basic except select (id_int \\* 9 \\- 8) \\/ 2, id_tinyint, id_varchar from test_all_type_nullable2 except select id_int, id_tinyint, id_varchar from test_basic_nullable"; ``` Example 5: Prohibit all INSERT INTO statements: ```sql ADD SQLBLACKLIST "(?i)^insert\\s+into\\s+.*"; ``` Example 6: Prohibit all INSERT INTO ... VALUES statements: ```sql ADD SQLBLACKLIST "(?i)^insert\\s+into\\s+.*values\\s*\\("; ``` Example 7: Prohibit all INSERT INTO ... VALUES statements except those against the system-defined view `_statistics_.column_statistics`: ```sql ADD SQLBLACKLIST "(?i)^insert\\s+into\\s+(?!column_statistics\\b).*values\\s*\\("; ``` --- ### DELETE SQLBLACKLIST DELETE SQLBLACKLIST deletes an SQL regular expression from the SQL blacklist. For more about SQL Blacklist, see [Manage SQL Blacklist](https://docs.starrocks.io/docs/administration/management/resource_management/Blacklist.md). tip This operation requires the SYSTEM-level BLACKLIST privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DELETE SQLBLACKLIST ``` #### Parameter[​](#parameter "Direct link to Parameter") `sql_index_number`: the index number of the SQL regular expression in the blacklist. Separate multiple index numbers with a comma (,) and a space. You can obtain the index number using [SHOW SQLBLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/sql_blacklist/SHOW_SQLBLACKLIST.md). #### Examples[​](#examples "Direct link to Examples") ```plain mysql> DELETE SQLBLACKLIST 3, 4; mysql> SHOW SQLBLACKLIST; +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Index | Forbidden SQL | +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 1 | select count\(\*\) from .+ | | 2 | select id_int \* 4, id_tinyint, id_varchar from test_all_type_nullable except select id_int, id_tinyint, id_varchar from test_basic except select \(id_int \* 9 \- 8\) \/ 2, id_tinyint, id_varchar from test_all_type_nullable2 except select id_int, id_tinyint, id_varchar from test_basic_nullable | +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` --- ### SHOW SQLBLACKLIST SHOW SQLBLACKLIST shows the SQL regular expressions in the SQL blacklist. For more about SQL Blacklist, see [Manage SQL Blacklist](https://docs.starrocks.io/docs/administration/management/resource_management/Blacklist.md). tip This operation requires the SYSTEM-level BLACKLIST privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW SQLBLACKLIST ``` #### Return value[​](#return-value "Direct link to Return value") | **Return** | **Description** | | ------------- | ------------------------------------------------------------------------------------ | | Index | The index number of the SQL regular expression that has been added to the blacklist. | | Forbidden SQL | The SQL regular expression that has been added to the blacklist. | #### Examples[​](#examples "Direct link to Examples") ```plain mysql> SHOW SQLBLACKLIST; +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Index | Forbidden SQL | +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 1 | select count\(\*\) from .+ | | 2 | select id_int \* 4, id_tinyint, id_varchar from test_all_type_nullable except select id_int, id_tinyint, id_varchar from test_basic except select \(id_int \* 9 \- 8\) \/ 2, id_tinyint, id_varchar from test_all_type_nullable2 except select id_int, id_tinyint, id_varchar from test_basic_nullable | | 3 | select id_int from test_all_type_select1 order by id_int limit [1-7], [5-7] | | 4 | select count\(distinct .+\) from .+ | +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` --- ### Storage volume #### [📄️ ALTER STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/ALTER_STORAGE_VOLUME.md) [ALTER STORAGE VOLUME alters the credential properties, comment, or status (enabled) of a storage volume.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/ALTER_STORAGE_VOLUME.md) #### [📄️ CREATE STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md) [CREATE STORAGE VOLUME creates a storage volume for a remote storage system.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md) #### [📄️ DESC STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/DESC_STORAGE_VOLUME.md) [DESC STORAGE VOLUME describes a storage volume.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/DESC_STORAGE_VOLUME.md) #### [📄️ DROP STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/DROP_STORAGE_VOLUME.md) [DROP STORAGE VOLUME drops a storage volume.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/DROP_STORAGE_VOLUME.md) #### [📄️ SET DEFAULT STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/SET_DEFAULT_STORAGE_VOLUME.md) [SET DEFAULT STORAGE VOLUME sets a storage volume as the default storage volume.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/SET_DEFAULT_STORAGE_VOLUME.md) #### [📄️ SHOW STORAGE VOLUMES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/SHOW_STORAGE_VOLUMES.md) [SHOW STORAGE VOLUMES shows the storage volumes in your StarRocks cluster.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/SHOW_STORAGE_VOLUMES.md) --- ### ALTER STORAGE VOLUME ALTER STORAGE VOLUME alters the credential properties, comment, or status (`enabled`) of a storage volume. For more about the properties of a storage volume, see [CREATE STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md). This feature is supported from v3.1. > **CAUTION** > > * Only users with the ALTER privilege on a specific storage volume can perform this operation. > * The `TYPE`, `LOCATIONS`, and other path-related properties of an existing storage volume cannot be altered. You can only alter its credential-related properties. If you changed the path-related configuration items, the databases and tables you created before the change become read-only, and you cannot load data into them. > * When `enabled` is `false`, the corresponding storage volume cannot be referenced. > * In shared-data mode, StarRocks performs a storage accessibility check during `ALTER STORAGE VOLUME` only when the `enable_storage_volume_access_check` configuration is enabled (enabled by default). If this check is enabled and the updated credentials or endpoint cannot access remote storage, the statement fails. You can turn off this check by disabling `enable_storage_volume_access_check`. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ALTER STORAGE VOLUME [ IF EXISTS ] { COMMENT = '' | SET ("key" = "value"[,...]) } ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | --------------------- | ---------------------------------------- | | storage\_volume\_name | The name of the storage volume to alter. | | COMMENT | The comment on the storage volume. | For detailed information on the properties that can be altered or added, see [CREATE STORAGE VOLUME - PROPERTIES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md#properties). #### Examples[​](#examples "Direct link to Examples") Example 1: Disable the storage volume `my_s3_volume`. ```plain MySQL > ALTER STORAGE VOLUME my_s3_volume -> SET ("enabled" = "false"); Query OK, 0 rows affected (0.01 sec) ``` Example 2: Alter the credential information of the storage volume `my_s3_volume`. ```plain MySQL > ALTER STORAGE VOLUME my_s3_volume -> SET ( -> "aws.s3.use_instance_profile" = "true" -> ); Query OK, 0 rows affected (0.00 sec) ``` #### Relevant SQL statements[​](#relevant-sql-statements "Direct link to Relevant SQL statements") * [CREATE STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md) * [DROP STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/DROP_STORAGE_VOLUME.md) * [SET DEFAULT STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/SET_DEFAULT_STORAGE_VOLUME.md) * [DESC STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/DESC_STORAGE_VOLUME.md) * [SHOW STORAGE VOLUMES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/SHOW_STORAGE_VOLUMES.md) --- ### CREATE STORAGE VOLUME CREATE STORAGE VOLUME creates a storage volume for a remote storage system. This feature is supported from v3.1. A storage volume consists of the properties and credential information of the remote data storage. You can reference a storage volume when you create databases and cloud-native tables in a shared-data StarRocks cluster. > **CAUTION** > > Only users with the CREATE STORAGE VOLUME privilege on the SYSTEM level can perform this operation. > > In shared-data mode, StarRocks performs a storage accessibility check for each `LOCATION` during `CREATE STORAGE VOLUME` when the FE configuration item `enable_storage_volume_access_check` is enabled (enabled by default). If this check is enabled and credential, endpoint, or network access is invalid, the statement fails immediately. You can turn off this check by disabling `enable_storage_volume_access_check`. #### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE STORAGE VOLUME [IF NOT EXISTS] TYPE = { S3 | HDFS | AZBLOB | ADLS2 | GS } LOCATIONS = ('') [ COMMENT '' ] PROPERTIES ("key" = "value",...) ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | storage\_volume\_name | The name of the storage volume. Please note that you cannot create a storage volume named `builtin_storage_volume` because it is used to create the builtin storage volume. For the naming conventions, see [System limits](https://docs.starrocks.io/docs/sql-reference/System_limit.md). | | TYPE | The type of the remote storage system. Valid values: `S3`, `HDFS`, `AZBLOB`, `ADLS2`, and `GS`. `S3` indicates AWS S3 or S3-compatible storage systems. `AZBLOB` indicates Azure Blob Storage (supported from v3.1.1 onwards). `ADLS2` indicates Azure Data Lake Storage Gen2 (supported from v3.4.1 onwards). `HDFS` indicates an HDFS cluster. `GS` indicates Google Storage (access with native SDK, supported from v3.5.1 onwards). | | LOCATIONS | The storage locations. The format is as follows:- For AWS S3 or S3 protocol-compatible storage systems: `s3://`. `` must be an absolute path, for example, `s3://testbucket/subpath`. Note that if you want to enable the [Partitioned Prefix](#partitioned-prefix) feature for the storage volume, you can only specify the bucket name, and specifying a sub-path is not allowed.
- For Azure Blob Storage: `azblob://`. `` must be an absolute path, for example, `azblob://testcontainer/subpath`.
- For Azure Data Lake Storage Gen2: `adls2:///`. Example: `adls2://testfilesystem/starrocks`.
- For GS with native SDK: `gs://`. `` must be an absolute path, for example, `gs://testcbucket/subpath`.
- For HDFS: `hdfs://:/`. `` must be an absolute path, for example, `hdfs://127.0.0.1:9000/user/xxx/starrocks`.
- For WebHDFS: `webhdfs://:/`, where `` is the HTTP port of the NameNode. `` must be an absolute path, for example, `webhdfs://127.0.0.1:50070/user/xxx/starrocks`.
- For ViewFS：`viewfs:///`, where `` is the ViewFS cluster name. `` must be an absolute path, for example, `viewfs://myviewfscluster/user/xxx/starrocks`. | | COMMENT | The comment on the storage volume. | | PROPERTIES | Parameters in the `"key" = "value"` pairs used to specify the properties and credential information to access the remote storage system. For detailed information, see [PROPERTIES](#properties). | ##### PROPERTIES[​](#properties "Direct link to PROPERTIES") The table below lists all available properties of storage volumes. Following the table are the usage instructions of these properties, categorized by different scenarios from the perspectives of [Credential information](#credential-information) and [Features](#features). | **Property** | **Description** | | ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enabled | Whether to enable this storage volume. Default: `false`. Disabled storage volume cannot be referenced. | | aws.s3.region | The region in which your S3 bucket resides, for example, `us-west-2`. | | aws.s3.endpoint | The endpoint URL used to access your S3 bucket, for example, `https://s3.us-west-2.amazonaws.com`. \[Preview] From v3.3.0 onwards, the Amazon S3 Express One Zone storage class is supported, for example, `https://s3express.us-west-2.amazonaws.com`.Beta feature[Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) | | aws.s3.use\_aws\_sdk\_default\_behavior | Whether to use the default authentication credential of AWS SDK. Valid values: `true` and `false` (Default). | | aws.s3.use\_instance\_profile | Whether to use Instance Profile and Assumed Role as credential methods for accessing S3. Valid values: `true` and `false` (Default).- If you use IAM user-based credential (Access Key and Secret Key) to access S3, you must specify this item as `false`, and specify `aws.s3.access_key` and `aws.s3.secret_key`.
- If you use Instance Profile to access S3, you must specify this item as `true`.
- If you use Assumed Role to access S3, you must specify this item as `true`, and specify `aws.s3.iam_role_arn`.
- And if you use an external AWS account, you must specify this item as `true`, and specify `aws.s3.iam_role_arn` and `aws.s3.external_id`. | | aws.s3.access\_key | The Access Key ID used to access your S3 bucket. | | aws.s3.secret\_key | The Secret Access Key used to access your S3 bucket. | | aws.s3.iam\_role\_arn | The ARN of the IAM role that has privileges on your S3 bucket in which your data files are stored. | | aws.s3.external\_id | The external ID of the AWS account that is used for cross-account access to your S3 bucket. | | aws.s3.enable\_path\_style\_access | Whether to use the Path Style to access S3.
Valid values: `true` and `false`. Default: `false`. | | azure.blob.endpoint | The endpoint of your Azure Blob Storage Account, for example, `https://test.blob.core.windows.net`. | | azure.blob.shared\_key | The Shared Key used to authorize requests for your Azure Blob Storage. | | azure.blob.sas\_token | The shared access signatures (SAS) used to authorize requests for your Azure Blob Storage. | | azure.adls2.endpoint | The endpoint of your Azure Data Lake Storage Gen2 Account, for example, `https://test.dfs.core.windows.net`. | | azure.adls2.shared\_key | The Shared Key used to authorize requests for your Azure Data Lake Storage Gen2e. | | azure.adls2.sas\_token | The shared access signatures (SAS) used to authorize requests for your Azure Data Lake Storage Gen2. | | azure.adls2.oauth2\_use\_managed\_identity | Whether to use Managed Identity to authorize requests for your Azure Data Lake Storage Gen2. Default: `false`. | | azure.adls2.oauth2\_tenant\_id | The Tenant ID of the Managed Identity used to authorize requests for your Azure Data Lake Storage Gen2. | | azure.adls2.oauth2\_client\_id | - For Managed Identity Authentication: The Client ID of the Managed Identity used to authorize requests for your Azure Data Lake Storage Gen2.
- For Workload Identity: The client ID (application ID) of the Azure AD application (user-assigned managed identity or app registration) associated with the workload identity. | | azure.adls2.oauth2\_token\_file | Path to the federated token file for Workload Identity authentication. Mount the token file at the same path and make it readable on every FE and CN. | | gcp.gcs.service\_account\_email | The email address in the JSON file generated at the creation of the Service Account, for example, `user@hello.iam.gserviceaccount.com`. | | gcp.gcs.service\_account\_private\_key\_id | The Private Key ID in the JSON file generated at the creation of the Service Account. | | gcp.gcs.service\_account\_private\_key | The Private Key in the JSON file generated at the creation of the Service Account, for example, `-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n`. | | gcp.gcs.impersonation\_service\_account | The Service Account that you want to impersonate if you use the impersonation-based authentication. | | gcp.gcs.use\_compute\_engine\_service\_account | Whether to use the Service Account that is bound to your Compute Engine. | | hadoop.security.authentication | The authentication method. Valid values: `simple`(Default) and `kerberos`. `simple` indicates simple authentication, that is, username. `kerberos` indicates Kerberos authentication. | | username | Username used to access the NameNode in the HDFS cluster. | | hadoop.security.kerberos.ticket.cache.path | The path that stores the kinit-generated Ticket Cache. | | dfs.nameservices | Name of the HDFS cluster. | | dfs.ha.namenodes.`` | Name of the NameNode. Multiple names must be separated by commas (,). No space is allowed in the double quotes. `` is the name of the HDFS service specified in `dfs.nameservices`. | | dfs.namenode.rpc-address.``.`` | The RPC address information of the NameNode. `` is the name of the NameNode specified in `dfs.ha.namenodes.`. | | dfs.client.failover.proxy.provider | The provider of the NameNode for client connection. The default value is `org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider`. | | fs.viewfs.mounttable.``.link./`` | The path to the ViewFS cluster to be mounted. Multiple paths must be separated by commas (,). `` is the ViewFS cluster name specified in `LOCATIONS`. | | aws.s3.enable\_partitioned\_prefix | Whether to enable the Partitioned Prefix feature for the storage volume. Default: `false`. For more information about this feature, see [Partitioned Prefix](#partitioned-prefix). | | aws.s3.num\_partitioned\_prefix | The number of prefixes to be created for the storage volume. Default: `256`. Valid range: \[4, 1024]. | ###### Credential information[​](#credential-information "Direct link to Credential information") ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") * If you use the default authentication credential of AWS SDK to access S3, set the following properties: ```sql "enabled" = "{ true | false }", "aws.s3.region" = "", "aws.s3.endpoint" = "", "aws.s3.use_aws_sdk_default_behavior" = "true" ``` * If you use IAM user-based credential (Access Key and Secret Key) to access S3, set the following properties: ```sql "enabled" = "{ true | false }", "aws.s3.region" = "", "aws.s3.endpoint" = "", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` * If you use Instance Profile to access S3, set the following properties: ```sql "enabled" = "{ true | false }", "aws.s3.region" = "", "aws.s3.endpoint" = "", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "true" ``` * If you use Assumed Role to access S3, set the following properties: ```sql "enabled" = "{ true | false }", "aws.s3.region" = "", "aws.s3.endpoint" = "", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "" ``` * If you use Assumed Role to access S3 from an external AWS account, set the following properties: ```sql "enabled" = "{ true | false }", "aws.s3.region" = "", "aws.s3.endpoint" = "", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.external_id" = "" ``` ###### MinIO[​](#minio "Direct link to MinIO") If you use MinIO, set the following properties: ```sql "enabled" = "{ true | false }", -- For example: us-east-1 "aws.s3.region" = "", -- For example: http://172.26.xx.xxx:39000 "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` ###### Azure Blob Storage[​](#azure-blob-storage "Direct link to Azure Blob Storage") Creating a storage volume on Azure Blob Storage is supported from v3.1.1 onwards. * If you use Shared Key to access Azure Blob Storage, set the following properties: ```sql "enabled" = "{ true | false }", "azure.blob.endpoint" = "", "azure.blob.shared_key" = "" ``` * If you use shared access signatures (SAS) to access Azure Blob Storage, set the following properties: ```sql "enabled" = "{ true | false }", "azure.blob.endpoint" = "", "azure.blob.sas_token" = "" ``` note The hierarchical namespace must be disabled when you create the Azure Blob Storage Account. ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2 "Direct link to Azure Data Lake Storage Gen2") Creating a storage volume on Azure Data Lake Storage Gen2 is supported from v3.4.1 onwards. * If you use Shared Key to access Azure Data Lake Storage Gen2, set the following properties: ```sql "enabled" = "{ true | false }", "azure.adls2.endpoint" = "", "azure.adls2.shared_key" = "" ``` * If you use shared access signatures (SAS) to access Azure Data Lake Storage Gen2, set the following properties: ```sql "enabled" = "{ true | false }", "azure.adls2.endpoint" = "", "azure.adls2.sas_token" = "" ``` * If you use Managed Identity to access Azure Data Lake Storage Gen2, set the following properties: ```sql "enabled" = "{ true | false }", "azure.adls2.endpoint" = "", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` * If you use Workload Identity to access Azure Data Lake Storage Gen2, set the following properties: ```sql "enabled" = "{ true | false }", "azure.adls2.endpoint" = "", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_use_managed_identity" = "false" ``` Path to the federated token file for Workload Identity authentication. Mount the token file at the same path and make it readable on every FE and CN. For Workload Identity, set `azure.adls2.oauth2_use_managed_identity` to `false`. All FEs and CNs must support Workload Identity for ADLS2 storage volumes. When migrating an existing volume, clear `azure.adls2.shared_key`, `azure.adls2.sas_token`, and `azure.adls2.oauth2_client_secret` in the same `ALTER STORAGE VOLUME` statement because ALTER retains properties that you do not specify. note Azure Data Lake Storage Gen1 is not supported. ###### Google Storage[​](#google-storage "Direct link to Google Storage") * If you use the Service Account that is bound to your Compute Engine to access Google Storage (supported from v3.5.1), set the following properties: ```sql "enabled" = "{ true | false }", "gcp.gcs.use_compute_engine_service_account" = "true" ``` * If you use the Service Account-based authentication method to access Google Storage (supported from v3.5.1), set the following properties: ```sql "enabled" = "{ true | false }", "gcp.gcs.use_compute_engine_service_account" = "false", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ``` * If you use the impersonation-based authentication to access Google Storage (supported from v3.5.1), set the following properties: ```sql "enabled" = "{ true | false }", "gcp.gcs.use_compute_engine_service_account" = "false", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` * If you use S3 protocol with the IAM user-based authentication to access Google Storage, set the following properties: tip Google Storage is supported using the [XML API](https://docs.cloud.google.com/storage/docs/interoperability), and the settings use the AWS S3 syntax. In this case, you must set `TYPE` as `S3` and `LOCATIONS` to an S3 protocol-compatible storage location. ```sql "enabled" = "{ true | false }", -- For example: us-east1 "aws.s3.region" = "", -- For example: https://storage.googleapis.com "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` ###### HDFS[​](#hdfs "Direct link to HDFS") * If you do not use authentication to access HDFS, set the following properties: ```sql "enabled" = "{ true | false }" ``` * If you are using simple authentication (supported from v3.2) to access HDFS, set the following properties: ```sql "enabled" = "{ true | false }", "hadoop.security.authentication" = "simple", "username" = "" ``` * If you are using Kerberos Ticket Cache authentication (supported since v3.2) to access HDFS, set the following properties: ```sql "enabled" = "{ true | false }", "hadoop.security.authentication" = "kerberos", "hadoop.security.kerberos.ticket.cache.path" = "" ``` > **CAUTION** > > * This setting only forces the system to use KeyTab to access HDFS via Kerberos. Make sure that each BE or CN node has access to the KeyTab files. Also make sure that the **/etc/krb5.conf** file is set up correctly. > * The Ticket cache is generated by an external kinit tool. Make sure you have a crontab or similar periodic task to refresh the tickets. * If your HDFS cluster is enabled for NameNode HA configuration (supported since v3.2), additionally set the following properties: ```sql "dfs.nameservices" = "", "dfs.ha.namenodes." = ", [, ...]", "dfs.namenode.rpc-address.." = ":", "dfs.namenode.rpc-address.." = ":", [...] "dfs.client.failover.proxy.provider." = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider" ``` For more information, see [HDFS HA Documentation](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html). * If you are using WebHDFS (supported since v3.2), set the following properties: ```sql "enabled" = "{ true | false }" ``` For more information, see [WebHDFS Documentation](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html). * If you are using Hadoop ViewFS (supported since v3.2), set the following properties: ```sql -- Replace with the name of the ViewFS cluster. "fs.viewfs.mounttable..link./" = "hdfs://:/", "fs.viewfs.mounttable..link./" = "hdfs://:/", [, ...] ``` For more information, see [ViewFS Documentation](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ViewFs.html). ###### Features[​](#features "Direct link to Features") ###### Partitioned Prefix[​](#partitioned-prefix "Direct link to Partitioned Prefix") From v3.2.4, StarRocks supports creating storage volumes with the Partitioned Prefix feature for S3-compatible object storage systems. When this feature is enabled, StarRocks stores the data into multiple, uniformly prefixed partitions (sub-paths) under the bucket. To enable this feature, set the following properties in addition to the above credential-related parameters: ```sql "aws.s3.enable_partitioned_prefix" = "{ true | false }", "aws.s3.num_partitioned_prefix" = "" ``` note * The Partitioned Prefix feature is only supported for S3-compatible object storage systems, that is, the `TYPE` of the storage volume must be `S3`. * `LOCATIONS` of the storage volume must only contain the bucket name, for example, `s3://testbucket`. Specifying a sub-path after the bucket name is not allowed. * Both properties are immutable once the storage volume is created. * You cannot enable this feature when create a storage volume by using the FE configuration file **fe.conf**. #### Examples[​](#examples "Direct link to Examples") Example 1: Create a storage volume `my_s3_volume` for the AWS S3 bucket `defaultbucket`, use the IAM user-based credential (Access Key and Secret Key) to access S3, and enable it. ```sql CREATE STORAGE VOLUME my_s3_volume TYPE = S3 LOCATIONS = ("s3://defaultbucket/test/") PROPERTIES ( "aws.s3.region" = "us-west-2", "aws.s3.endpoint" = "https://s3.us-west-2.amazonaws.com", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "xxxxxxxxxx", "aws.s3.secret_key" = "yyyyyyyyyy" ); ``` Example 2: Create a storage volume `my_hdfs_volume` for HDFS and enable it. ```sql CREATE STORAGE VOLUME my_hdfs_volume TYPE = HDFS LOCATIONS = ("hdfs://127.0.0.1:9000/sr/test/") PROPERTIES ( "enabled" = "true" ); ``` Example 3: Create a storage volume `hdfsvolumehadoop` for HDFS using simple authentication. ```sql CREATE STORAGE VOLUME hdfsvolumehadoop TYPE = HDFS LOCATIONS = ("hdfs://127.0.0.1:9000/sr/test/") PROPERTIES( "hadoop.security.authentication" = "simple", "username" = "starrocks" ); ``` Example 4: Use Kerberos Ticket Cache authentication to access HDFS and create storage volume `hdfsvolkerberos`. ```sql CREATE STORAGE VOLUME hdfsvolkerberos TYPE = HDFS LOCATIONS = ("hdfs://127.0.0.1:9000/sr/test/") PROPERTIES( "hadoop.security.authentication" = "kerberos", "hadoop.security.kerberos.ticket.cache.path" = "/path/to/ticket/cache/path" ); ``` Example 5: Create storage volume `hdfsvolha` for an HDFS cluster with NameNode HA configuration enabled. ```sql CREATE STORAGE VOLUME hdfsvolha TYPE = HDFS LOCATIONS = ("hdfs://myhacluster/data/sr") PROPERTIES( "dfs.nameservices" = "myhacluster", "dfs.ha.namenodes.myhacluster" = "nn1,nn2,nn3", "dfs.namenode.rpc-address.myhacluster.nn1" = "machine1.example.com:8020", "dfs.namenode.rpc-address.myhacluster.nn2" = "machine2.example.com:8020", "dfs.namenode.rpc-address.myhacluster.nn3" = "machine3.example.com:8020", "dfs.namenode.http-address.myhacluster.nn1" = "machine1.example.com:9870", "dfs.namenode.http-address.myhacluster.nn2" = "machine2.example.com:9870", "dfs.namenode.http-address.myhacluster.nn3" = "machine3.example.com:9870", "dfs.client.failover.proxy.provider.myhacluster" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider" ); ``` Example 6: Create a storage volume `webhdfsvol` for WebHDFS. ```sql CREATE STORAGE VOLUME webhdfsvol TYPE = HDFS LOCATIONS = ("webhdfs://namenode:9870/data/sr"); ``` Example 7: Create a storage volume `viewfsvol` using Hadoop ViewFS. ```sql CREATE STORAGE VOLUME viewfsvol TYPE = HDFS LOCATIONS = ("viewfs://clusterX/data/sr") PROPERTIES( "fs.viewfs.mounttable.clusterX.link./data" = "hdfs://nn1-clusterx.example.com:8020/data", "fs.viewfs.mounttable.clusterX.link./project" = "hdfs://nn2-clusterx.example.com:8020/project" ); ``` Example 8: Create a storage volume `adls2` for Azure Data Lake Storage Gen2 using SAS token. ```sql CREATE STORAGE VOLUME adls2 TYPE = ADLS2 LOCATIONS = ("adls2://testfilesystem/starrocks") PROPERTIES ( "azure.adls2.endpoint" = "https://test.dfs.core.windows.net", "azure.adls2.sas_token" = "xxx" ); ``` Example 9: Create a storage volume `gs` for Google Storage using impersonated Service Account. ```sql CREATE STORAGE VOLUME gs TYPE = GS LOCATIONS = ("gs://testbucket/starrocks") PROPERTIES ( "gcp.gcs.use_compute_engine_service_account" = "false", "gcp.gcs.service_account_email" = "user@hello.iam.gserviceaccount.com", "gcp.gcs.service_account_private_key_id" = "61d257bd847xxxxxxxxxxxxxxx4f0b9b6b9ca07af3b7ea", "gcp.gcs.service_account_private_key" = "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n", "gcp.gcs.impersonation_service_account" = "admin@hello.iam.gserviceaccount.com" ); ``` #### Relevant SQL statements[​](#relevant-sql-statements "Direct link to Relevant SQL statements") * [ALTER STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/ALTER_STORAGE_VOLUME.md) * [DROP STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/DROP_STORAGE_VOLUME.md) * [SET DEFAULT STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/SET_DEFAULT_STORAGE_VOLUME.md) * [DESC STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/DESC_STORAGE_VOLUME.md) * [SHOW STORAGE VOLUMES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/SHOW_STORAGE_VOLUMES.md) --- ### DESC STORAGE VOLUME DESC STORAGE VOLUME describes a storage volume. This feature is supported from v3.1. > **CAUTION** > > Only users with the USAGE privilege on a specific storage volume can perform this operation. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DESC[RIBE] STORAGE VOLUME ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | --------------------- | ------------------------------------------- | | storage\_volume\_name | The name of the storage volume to describe. | #### Return value[​](#return-value "Direct link to Return value") | **Return** | **Description** | | ---------- | ----------------------------------------------------------------------- | | Name | The name of the storage volume. | | Type | The type of the remote storage system. Valid values: `S3` and `AZBLOB`. | | IsDefault | Whether the storage volume is the default storage volume. | | Location | The location of the remote storage system. | | Params | The credential information used to access the remote storage system. | | Enabled | Whether the storage volume is enabled. | | Comment | The comment on the storage volume. | #### Examples[​](#examples "Direct link to Examples") Example 1: Describe the storage volume `my_s3_volume`. ```plain MySQL > DESCRIBE STORAGE VOLUME my_s3_volume\G *************************** 1. row *************************** Name: my_s3_volume Type: S3 IsDefault: false Location: s3://defaultbucket/test/ Params: {"aws.s3.access_key":"xxxxxxxxxx","aws.s3.secret_key":"yyyyyyyyyy","aws.s3.endpoint":"https://s3.us-west-2.amazonaws.com","aws.s3.region":"us-west-2","aws.s3.use_instance_profile":"true","aws.s3.use_aws_sdk_default_behavior":"false"} Enabled: false Comment: 1 row in set (0.00 sec) ``` #### Relevant SQL statements[​](#relevant-sql-statements "Direct link to Relevant SQL statements") * [CREATE STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md) * [ALTER STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/ALTER_STORAGE_VOLUME.md) * [DROP STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/DROP_STORAGE_VOLUME.md) * [SET DEFAULT STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/SET_DEFAULT_STORAGE_VOLUME.md) * [SHOW STORAGE VOLUMES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/SHOW_STORAGE_VOLUMES.md) --- ### DROP STORAGE VOLUME DROP STORAGE VOLUME drops a storage volume. Dropped storage volumes cannot be referenced anymore. This feature is supported from v3.1. > **CAUTION** > > * Only users with the DROP privilege on a specific storage volume can perform this operation. > * The default storage volume and the built-in storage volume `builtin_storage_volume` cannot be dropped. You can use [DESC STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/DESC_STORAGE_VOLUME.md) to check whether a storage volume is the default storage volume. > * Storage volumes that are referenced by existing databases or cloud-native tables cannot be dropped. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DROP STORAGE VOLUME [ IF EXISTS ] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | --------------------- | --------------------------------------- | | storage\_volume\_name | The name of the storage volume to drop. | #### Examples[​](#examples "Direct link to Examples") Example 1: Drop the storage volume `my_s3_volume`. ```plain MySQL > DROP STORAGE VOLUME my_s3_volume; Query OK, 0 rows affected (0.01 sec) ``` #### Relevant SQL statements[​](#relevant-sql-statements "Direct link to Relevant SQL statements") * [CREATE STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md) * [ALTER STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/ALTER_STORAGE_VOLUME.md) * [SET DEFAULT STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/SET_DEFAULT_STORAGE_VOLUME.md) * [DESC STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/DESC_STORAGE_VOLUME.md) * [SHOW STORAGE VOLUMES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/SHOW_STORAGE_VOLUMES.md) --- ### SET DEFAULT STORAGE VOLUME SET DEFAULT STORAGE VOLUME sets a storage volume as the default storage volume. After creating a storage volume for an external data source, you can set it as the default storage volume of your StarRocks cluster. This feature is supported from v3.1. > **CAUTION** > > * Only users with the ALTER privilege on a specific storage volume can perform this operation. > * The default storage volume cannot be dropped or disabled. > * You must set the default storage volume for a shared-data StarRocks cluster because StarRocks stores the system statistics information in the default storage volume. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SET AS DEFAULT STORAGE VOLUME ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | --------------------- | ----------------------------------------------------------------------- | | storage\_volume\_name | The name of the storage volume to be set as the default storage volume. | #### Examples[​](#examples "Direct link to Examples") Example 1: Set the storage volume `my_s3_volume` as the default storage volume. ```sql MySQL > SET my_s3_volume AS DEFAULT STORAGE VOLUME; Query OK, 0 rows affected (0.01 sec) ``` #### Relevant SQL statements[​](#relevant-sql-statements "Direct link to Relevant SQL statements") * [CREATE STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md) * [ALTER STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/ALTER_STORAGE_VOLUME.md) * [DROP STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/DROP_STORAGE_VOLUME.md) * [DESC STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/DESC_STORAGE_VOLUME.md) * [SHOW STORAGE VOLUMES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/SHOW_STORAGE_VOLUMES.md) --- ### SHOW STORAGE VOLUMES SHOW STORAGE VOLUMES shows the storage volumes in your StarRocks cluster. This feature is supported from v3.1. tip This operation does not require privileges. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW STORAGE VOLUMES [ LIKE '' ] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | ------------- | ---------------------------------------------- | | pattern | The pattern used to match the storage volumes. | #### Return value[​](#return-value "Direct link to Return value") | **Return** | **Description** | | -------------- | ------------------------------- | | Storage Volume | The name of the storage volume. | #### Examples[​](#examples "Direct link to Examples") Example 1: Show all storage volumes in the StarRocks cluster. ```plain MySQL > SHOW STORAGE VOLUMES; +----------------+ | Storage Volume | +----------------+ | my_s3_volume | +----------------+ 1 row in set (0.01 sec) ``` #### Relevant SQL statements[​](#relevant-sql-statements "Direct link to Relevant SQL statements") * [CREATE STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md) * [ALTER STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/ALTER_STORAGE_VOLUME.md) * [DROP STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/DROP_STORAGE_VOLUME.md) * [SET DEFAULT STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/SET_DEFAULT_STORAGE_VOLUME.md) * [DESC STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/DESC_STORAGE_VOLUME.md) --- ### Tablet and replica #### [📄️ ADMIN CANCEL REPAIR](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_CANCEL_REPAIR.md) [Cancels the prioritized schedule of repairing operations on specified tables or partitions.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_CANCEL_REPAIR.md) #### [📄️ ADMIN CHECK TABLET](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_CHECK_TABLET.md) [ADMIN CHECK TABLET is used to check a group of tablets.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_CHECK_TABLET.md) #### [📄️ ADMIN REPAIR](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md) [Attempts to repair the specified table or partitions.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md) #### [📄️ ADMIN SET PARTITION VERSION](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_PARTITION_VERSION.md) [ADMIN SET PARTITION VERSION sets a partition to a specific data version.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_PARTITION_VERSION.md) #### [📄️ ADMIN SET REPLICA STATUS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_REPLICA_STATUS.md) [ADMIN SET REPLICA STATUS is used to set the status of the specified replicas.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_REPLICA_STATUS.md) #### [📄️ ADMIN SHOW REPLICA DISTRIBUTION](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_REPLICA_DISTRIBUTION.md) [ADMIN SHOW REPLICA DISTRIBUTION is used to show the distribution status of a table or a partition replica.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_REPLICA_DISTRIBUTION.md) #### [📄️ ADMIN SHOW REPLICA STATUS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_REPLICA_STATUS.md) [ADMIN SHOW REPLICA STATUS is used to show the status of replicas for a table or a partition.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_REPLICA_STATUS.md) #### [📄️ ADMIN SHOW TABLET STATUS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_TABLET_STATUS.md) [Shows the status of Tablets in a cloud-native table or cloud-native materialized view in a shared-data cluster, including whether metadata or data files are...](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_TABLET_STATUS.md) #### [📄️ ADMIN SKIP COMMITTED TRANSACTION](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SKIP_COMMITTED_TRANSACTION.md) [Force-abort a transaction stuck in the COMMITTED state on a shared-data table, discarding its data while advancing the partition version.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SKIP_COMMITTED_TRANSACTION.md) #### [📄️ SHOW TABLE STATUS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/SHOW_TABLE_STATUS.md) [SHOW TABLE STATUS is used to view some of the information in Table.](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/SHOW_TABLE_STATUS.md) --- ### ADMIN CANCEL REPAIR Cancels the prioritized schedule of repairing operations on specified tables or partitions. This statement only indicates that the system will no longer repair sharding replicas of specified tables or partitions with high priority. It still repairs these copies by default scheduling. ADMIN CANCEL REPAIR applies only to native tables in shared-nothing clusters. tip This operation requires the SYSTEM-level OPERATE privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ADMIN CANCEL REPAIR TABLE table_name[ PARTITION (p1,...)] ``` #### Examples[​](#examples "Direct link to Examples") 1. Cancel the high-priority repairing schedule for the partition `p1` in the native table `tbl1`. ```sql ADMIN CANCEL REPAIR TABLE tbl PARTITION(p1); ``` --- ### ADMIN CHECK TABLET ADMIN CHECK TABLET is used to check a group of tablets. tip This operation requires the SYSTEM-level OPERATE privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ADMIN CHECK TABLET (tablet_id1, tablet_id2, ...) PROPERTIES("type" = "...") ``` Note: 1. The "type" property in tablet id and PROPERTIES must be specified. 2. Currently, "type" only supports: Consistency: Check the consistency of replicas of the tablet. This command is asynchronous. After sending it, StarRocks will start checking the consistency among corresponding tablets. The final results will be shown in the InconsistentTabletNum column in the result of SHOW PROC "/statistic". #### Examples[​](#examples "Direct link to Examples") 1. Check the consistency of replicas on a group of specified tablets ```sql ADMIN CHECK TABLET (10000, 10001) PROPERTIES("type" = "consistency"); ``` --- ### ADMIN REPAIR Attempts to repair the specified table or partitions. For native tables in shared-nothing clusters, this statement attempts to prioritize scheduling the replica repairing operation. For cloud-native tables in shared-data clusters, it attempts to rollback to a historical available version when metadata or data files are lost. Please note that **this may result in the loss of the latest data for some tablets**. tip This operation requires the SYSTEM-level OPERATE privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ADMIN REPAIR TABLE table_name [PARTITION (p1,...)] [PROPERTIES ("key" = "value", ...)] ``` Note: 1. This statement only means that the system attempts to repair sharding replicas of specified tables or partitions with high priority without the guarantee that the repair will be successful. Users can view the repair status through ADMIN SHOW REPLICA STATUS command. 2. The default timeout is 14400 seconds (4 hours). Timeout means the system will not repair sharding replicas of specified tables or partitions with high priority. In case of timeout, the command needs to be used again for the intended settings. 3. You can set repair behavior by specifying `PROPERTIES` in the statement. **Currently, only cloud-native tables in shared-data clusters support `PROPERTIES`**. **PROPERTIES** * `enforce_consistent_version`: Whether to enforce all tablets in a partition to roll back to a consistent version. Default: `true`. If this item is set to `true`, the system will search for a consistent version in the history that is valid for all tablets to perform the rollback, ensuring data version alignment across the partition. If it is set to `false`, each tablet in the partition is allowed to rollback to its latest available valid version. The versions of different tablets may be inconsistent, but this maximizes data preservation. * `allow_empty_tablet_recovery`: Whether to allow recovery by creating empty tablets. Default: `false`. This item takes effect only when `enforce_consistent_version` is `false`. If this item is set to `true`, when metadata is missing for all versions of some tablets but valid metadata exists for at least one tablet, the system attempts to create empty tablets to fill the missing versions. If metadata for all versions of all tablets is lost, recovery is impossible. * `dry_run`: Whether to return the repair plan without actually executing the repair. Default: `false`. If set to `true`, the system evaluates the recoverability of each partition and returns the repair plan without performing any actual metadata rollback. The result set contains the following columns: * `PartitionId`: Partition ID. * `VisibleVersion`: Current visible version. * `RepairStatus`: Repair status. Valid values: `NORMAL` (all tablets are healthy, no repair needed), `RECOVERABLE` (missing metadata or data files but recoverable), `UNRECOVERABLE` (missing metadata or data files and unrecoverable), `UNKNOWN` (an exception occurred during probing). * `TabletRecoverInfo`: JSON array listing the version each tablet will roll back to. Only populated when status is `RECOVERABLE`. * `ErrorMsg`: Error message. Only populated when status is `UNRECOVERABLE` or `UNKNOWN`. #### Examples[​](#examples "Direct link to Examples") 1. Attempt to repair specified tables ```sql ADMIN REPAIR TABLE tbl1; ``` 2. Attempt to fix specified partition ```sql ADMIN REPAIR TABLE tbl1 PARTITION (p1, p2); ``` 3. Attempt to repair a shared-data table, allowing inconsistent versions and creating empty tablets for recovery ```sql ADMIN REPAIR TABLE cloud_tbl PROPERTIES ( "enforce_consistent_version" = "false", "allow_empty_tablet_recovery" = "true" ); ``` 4. Preview the repair plan for a shared-data table without executing it ```sql ADMIN REPAIR TABLE cloud_tbl PARTITION (p1) PROPERTIES ("dry_run" = "true"); ``` --- ### ADMIN SET PARTITION VERSION ADMIN SET PARTITION VERSION sets a partition to a specific data version. Note that manually setting the partition version is a high-risk operation and is recommended only when problems occur in the cluster metadata. In normal circumstances, the version of a partition is consistent with that of the tablets within. tip This operation requires the SYSTEM-level OPERATE privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ADMIN SET TABLE PARTITION ( | ) VERSION TO ``` #### Parameters[​](#parameters "Direct link to Parameters") * `table_name`: The name of the table to which the partition belongs. * `partition_name`: The name of the partition. You need to specify a partition using either `partition_name` or `partition_id`. For non-partitioned tables, `partition_name` is the same as the table name. * `partition_id`: The ID of the partition. You need to specify a partition using either `partition_name` or `partition_id`. For tables with the random bucketing strategy, you can only use `partition_id` to specify the partition. * `version`: The version you want to set for the partition. #### Examples[​](#examples "Direct link to Examples") 1. Set the version of the non-partitioned table `t1` to `10`. ```sql ADMIN SET TABLE t1 PARTITION(t1) VERSION TO 10; ``` 2. Set the version of partition `p1` in table `t2` to `10`. ```sql ADMIN SET TABLE t2 PARTITION(p1) VERSION TO 10; ``` 3. Set the version of the partition whose ID is `123456` to `10`. `t3` is a table with the random bucketing strategy. ```sql ADMIN SET TABLE t3 PARTITION('123456') VERSION TO 10; ``` --- ### ADMIN SET REPLICA STATUS ADMIN SET REPLICA STATUS is used to set the status of the specified replicas. This command currently only used to manually set the status of some replicas to BAD or OK, allowing the system to automatically repair these replicas. tip This operation requires the SYSTEM-level OPERATE privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ADMIN SET REPLICA STATUS PROPERTIES ("key" = "value", ...) ``` The following attributes are currently supported: "table\_id': required. Specify a Tablet Id. "backend\_id": required. Specify a Backend Id. "status": required. Specify the status. Currently only "bad" and "ok" are supported. If the specified replica does not exist or its status is bad, the replica will be ignored. Note: Replicas set to Bad status may be dropped immediately, please proceed with caution. #### Examples[​](#examples "Direct link to Examples") 1. Set the replica status of tablet 10003 on BE 10001 to bad. ```sql ADMIN SET REPLICA STATUS PROPERTIES("tablet_id" = "10003", "backend_id" = "10001", "status" = "bad"); ``` 2. Set the replica status of tablet 10003 on BE 10001 to ok. ```sql ADMIN SET REPLICA STATUS PROPERTIES("tablet_id" = "10003", "backend_id" = "10001", "status" = "ok"); ``` --- ### ADMIN SHOW REPLICA DISTRIBUTION ADMIN SHOW REPLICA DISTRIBUTION is used to show the distribution status of a table or a partition replica. tip This operation requires the SYSTEM-level OPERATE privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ADMIN SHOW REPLICA DISTRIBUTION FROM [db_name.]tbl_name [PARTITION (p1, ...)] ``` Note: The Graph column in the result shows the distribution ratio of replicas graphically. #### Examples[​](#examples "Direct link to Examples") 1. View the replica distribution of tables ```sql ADMIN SHOW REPLICA DISTRIBUTION FROM tbl1; ``` 2. View the replica distribution of partitions in the table ```sql ADMIN SHOW REPLICA DISTRIBUTION FROM db1.tbl1 PARTITION(p1, p2); ``` --- ### ADMIN SHOW REPLICA STATUS ADMIN SHOW REPLICA STATUS is used to show the status of replicas for a table or a partition. tip This operation requires the SYSTEM-level OPERATE privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ADMIN SHOW REPLICA STATUS FROM [db_name.]tbl_name [PARTITION (p1, ...)] [where_clause] ``` ```sql where_clause: WHERE STATUS [!]= "replica_status" ``` ```plain replica_status: OK: The replica is healthy DEAD: The Backend of replica is not available VERSION_ERROR: The replica data version is missing SCHEMA_ERROR: The schema hash of replica is incorrect MISSING: The replica does not exist ``` #### Examples[​](#examples "Direct link to Examples") 1. View the status of all replicas of the table. ```sql ADMIN SHOW REPLICA STATUS FROM db1.tbl1; ``` 2. View the replica of a partition with the status of VERSION\_ERROR. ```sql ADMIN SHOW REPLICA STATUS FROM tbl1 PARTITION (p1, p2) WHERE STATUS = "VERSION_ERROR"; ``` 3. View all unhealthy replicas of the table. ```sql ADMIN SHOW REPLICA STATUS FROM tbl1 WHERE STATUS != "OK"; ``` --- ### ADMIN SHOW TABLET STATUS #### Description[​](#description "Direct link to Description") Shows the status of Tablets in a cloud-native table or cloud-native materialized view in a shared-data cluster, including whether metadata or data files are missing. tip This operation requires the SYSTEM-level OPERATE privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ADMIN SHOW TABLET STATUS FROM [.] [PARTITION ( [, , ...])] [WHERE STATUS [=|!=] {'NORMAL'|'MISSING_META'|'MISSING_DATA'}] [PROPERTIES ("max_missing_data_files_to_show" = "")] ``` #### Parameters[​](#parameters "Direct link to Parameters") | Parameter | Description | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | db\_name | Database name. If omitted, the current database is used. | | table\_name | Table or materialized view name. Only cloud-native tables and cloud-native materialized views in shared-data clusters are supported. | | PARTITION | Partitions to check. If omitted, all partitions in the table are checked. | | WHERE STATUS \[=|!=] | Filter tablets by status. Supports `=` (equals) and `!=` (not equals). See **Return columns** for valid status values. | | max\_missing\_data\_files\_to\_show | Maximum number of missing data files to display per tablet. Default: `5`. Set to `-1` to show all missing files. Set to `0` to suppress the file list while still counting missing files. | #### Return Columns[​](#return-columns "Direct link to Return Columns") | Column | Description | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | TabletId | Tablet ID. | | PartitionId | Physical partition ID of the partition the Tablet belongs to. | | Version | Current visible version of the Tablet. | | Status | Tablet status: `NORMAL` (metadata and data files are all intact), `MISSING_META` (metadata file is missing), `MISSING_DATA` (data file(s) are missing). | | MissingDataFileCount | Number of missing data files. Populated when `Status` is `MISSING_DATA`. | | MissingDataFiles | List of missing data file paths. Limited by `max_missing_data_files_to_show`. Populated when `Status` is `MISSING_DATA`. | #### Examples[​](#examples "Direct link to Examples") 1. Show the status of all tablets in a table. ```sql ADMIN SHOW TABLET STATUS FROM my_cloud_table; ``` 2. Show the status of tablets in specified partitions. ```sql ADMIN SHOW TABLET STATUS FROM my_cloud_table PARTITION (p20250101, p20250102); ``` 3. Show all abnormal tablets (non-NORMAL) in a table. ```sql ADMIN SHOW TABLET STATUS FROM my_cloud_table WHERE STATUS != "NORMAL"; ``` 4. Show tablets with missing metadata files. ```sql ADMIN SHOW TABLET STATUS FROM my_cloud_table WHERE STATUS = "MISSING_META"; ``` 5. Show tablets with missing data files and display all missing file paths. ```sql ADMIN SHOW TABLET STATUS FROM my_cloud_table WHERE STATUS = "MISSING_DATA" PROPERTIES ("max_missing_data_files_to_show" = "-1"); ``` --- ### ADMIN SKIP COMMITTED TRANSACTION Force-abort a transaction that is stuck in the `COMMITTED` state on a shared-data (lake) table by performing a no-op publish: the transaction's data contribution is discarded, while the partition's visible version still advances by writing a new tablet metadata file that carries no data changes from this transaction. This is an **operator-only escape hatch** for unblocking a publish queue that is permanently stalled (for example by a missing `txnlog`, a lost segment / SST file, or a persistent storage outage). It will **discard the affected transaction's data**. Use only when normal retry cannot recover, and prefer it over destructive recovery paths such as dropping the partition. warning Aborting a `COMMITTED` transaction is **irreversible** and causes data loss for that transaction. The transaction record is still finalized as `VISIBLE` to keep the partition's version sequence intact, but it is internally marked as a no-op publish. tip This operation requires the SYSTEM-level OPERATE privilege. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ADMIN SKIP COMMITTED TRANSACTION [REASON ''] ``` #### Parameters[​](#parameters "Direct link to Parameters") | Parameter | Required | Description | | ----------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `txn_id` | Yes | The numeric transaction ID. Find it with [SHOW TRANSACTION](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION.md) or in the FE log. | | `REASON ''` | No | A free-form reason for the abort, recorded in the FE audit log. Strongly recommended so the action can be tracked back during incident review. | #### Constraints (phase 1)[​](#constraints-phase-1 "Direct link to Constraints (phase 1)") 1. **Feature is gated by FE config**. The statement is rejected unless the FE config `enable_admin_skip_committed_txn` is set to `true`. The default is `false` to prevent accidental data loss. Set it via: ```sql ADMIN SET FRONTEND CONFIG ("enable_admin_skip_committed_txn" = "true"); ``` Set it back to `false` after the recovery is finished. 2. **The transaction must be in the `COMMITTED` state**. Transactions in `PREPARED` time out on their own (driven by the txn `timeout` setting) or can be cancelled via the loader-specific path — for example `CANCEL LOAD` for broker / spark / routine load, or `POST /api/transaction/rollback` for stream load. Transactions already in `VISIBLE` cannot be undone. 3. **Only shared-data (cloud-native) tables with `file_bundling=true` are supported**. With `file_bundling`, the partition's metadata is written atomically by the aggregator, so the no-op publish either takes effect for the whole partition or not at all. Tables without `file_bundling` may end up in a partial-publish state and are rejected at the entrypoint. 4. **Only load and lake-compaction transactions are supported in this release**. Alter / schema-change transactions are not supported yet and return an error if attempted. #### Behavior[​](#behavior "Direct link to Behavior") 1. FE validates the state, source type, and table properties. 2. FE persists a no-op-publish marker on the transaction in the edit log. 3. The next tick of the publish daemon picks up the marker and sends the publish RPC to BE with the no-op-publish flag set. 4. BE writes a tablet metadata file at the transaction's target version whose content equals the base version (no data changes from this transaction). The aggregator under `file_bundling` makes this write atomic across the whole partition. 5. The partition's visible version advances to the transaction's target version. Subsequent transactions on the table can now publish normally. 6. The original transaction is recorded as `VISIBLE`, but its no-op-publish marker is preserved for audit. Race resolution: if the in-flight publish RPC happened to succeed first, the transaction becomes `VISIBLE` normally and the marker is a no-op (the data lands safely instead of being discarded). This is recorded in the FE log. #### Auditing[​](#auditing "Direct link to Auditing") `SHOW PROC '/transactions//finished'` exposes two trailing columns for inspecting whether the abort actually took effect: * `NoOpPublish` — `true`/`false`, whether the transaction was finalized via the no-op publish path * `NoOpPublishReason` — the free-form `REASON` text supplied to the `ADMIN SKIP COMMITTED TRANSACTION` statement Use these to retroactively confirm whether a recovery action succeeded and what justification was recorded. #### Examples[​](#examples "Direct link to Examples") 1. Find the stuck transaction: ```sql SHOW PROC '/transactions//running'; ``` 2. Enable the feature switch: ```sql ADMIN SET FRONTEND CONFIG ("enable_admin_skip_committed_txn" = "true"); ``` 3. Abort the stuck transaction: ```sql ADMIN SKIP COMMITTED TRANSACTION 12345 REASON 'txnlog file lost on OSS, manual recovery'; ``` 4. Disable the feature switch again to prevent accidental use: ```sql ADMIN SET FRONTEND CONFIG ("enable_admin_skip_committed_txn" = "false"); ``` #### Related[​](#related "Direct link to Related") * [`SHOW TRANSACTION`](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION.md) — inspect transaction state. * [`ADMIN REPAIR`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md) — rollback a lake partition to a historical version when metadata is unrecoverable; a more destructive option used when this command is not applicable. --- ### SHOW TABLE STATUS SHOW TABLE STATUS is used to view some of the information in Table. tip This operation does not require privileges. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW TABLE STATUS [FROM db] [LIKE "pattern"] ``` > Note > > This statement is mainly compatible with MySQL syntax. At present, it only shows a few information, such as Comment. #### Examples[​](#examples "Direct link to Examples") 1. View all the information of tables under the current database. ```sql SHOW TABLE STATUS; ``` 2. View all the information of tables whose names contain example and who are under specified databases. ```sql SHOW TABLE STATUS FROM db LIKE "%example%"; ``` --- ### Database #### [📄️ ALTER DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/ALTER_DATABASE.md) [ALTER DATABASE configures the properties of the specified database.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/ALTER_DATABASE.md) #### [📄️ CREATE DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/CREATE_DATABASE.md) [CREATE DATABASE is used to create databases.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/CREATE_DATABASE.md) #### [📄️ DROP DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/DROP_DATABASE.md) [DROP DATABASE is used to delete a database in StarRocks.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/DROP_DATABASE.md) #### [📄️ SHOW CREATE DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_CREATE_DATABASE.md) [Shows the SQL command used to create a database.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_CREATE_DATABASE.md) #### [📄️ SHOW DATA](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATA.md) [SHOW DATA is used to display the amount of data, the number of copies, and the number of statistical rows in a database or a database table.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATA.md) #### [📄️ SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) [SHOW DATABASES lists databases in your current StarRocks cluster or an external data source.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) #### [📄️ USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) [USE specifies the active database for your session.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) --- ### ALTER DATABASE ALTER DATABASE configures the properties of the specified database. tip This operation requires the ALTER privilege on the target database. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") 1. Set database data quota in B/K/KB/M/MB/G/GB/T/TB/P/PB. ```sql ALTER DATABASE SET DATA QUOTA ; ``` 2. Rename a database. ```sql ALTER DATABASE RENAME ; ``` 3. Set database replica quota. ```sql ALTER DATABASE SET REPLICA QUOTA ; ``` 4. Set database storage volume ```sql ALTER DATABASE SET ("storage_volume" = ""); ``` Note: ```plain - After renaming the database, use REVOKE and GRANT commands to modify the corresponding user permission if necessary. - The database's default data quota and the default replica quota are 2^63-1. - The database's default storage volume is the default storage volume when the database is created if the "storage_volume" property is not explicitly provided. ``` #### Examples[​](#examples "Direct link to Examples") 1. Set data quota for a database. ```sql ALTER DATABASE example_db SET DATA QUOTA 10995116277760B; -- The above unit is bytes, equivalent to the following statement. ALTER DATABASE example_db SET DATA QUOTA 10T; ALTER DATABASE example_db SET DATA QUOTA 100G; ALTER DATABASE example_db SET DATA QUOTA 200M; ``` 2. Rename the database `example_db` as `example_db2`. ```sql ALTER DATABASE example_db RENAME example_db2; ``` 3. Set database replica quota. ```sql ALTER DATABASE example_db SET REPLICA QUOTA 102400; ``` 4. Set database storage volume. ```sql ALTER DATABASE example_db SET ("storage_volume" = "aws_s3_storage_volume"); ``` #### References[​](#references "Direct link to References") * [CREATE DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/CREATE_DATABASE.md) * [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) * [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) * [DESC](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md) * [DROP DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/DROP_DATABASE.md) --- ### CREATE DATABASE CREATE DATABASE is used to create databases. tip This operation requires the CREATE DATABASE privilege on the target catalog. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE DATABASE [IF NOT EXISTS] [PROPERTIES ("key"="value", ...)] ``` #### Parameters[​](#parameters "Direct link to Parameters") `db_name`: the name of the database to create. For the naming conventions, see [System limits](https://docs.starrocks.io/docs/sql-reference/System_limit.md). **PROPERTIES (Optional)** `storage_volume`: Specifies the name of the storage volume that is used to store the database in a shared-data cluster. If not provided, the default storage volume will be attached to the database. The property can be changed via `ALTER DATABASE` statement. #### Examples[​](#examples "Direct link to Examples") 1. Create the database `db_test`. ```sql CREATE DATABASE db_test; ``` 2. Create the cloud-native database `cloud_db` with the storage volume `s3_storage_volume`. ```sql CREATE DATABASE cloud_db PROPERTIES ("storage_volume"="s3_storage_volume"); ``` #### References[​](#references "Direct link to References") * [SHOW CREATE DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_CREATE_DATABASE.md) * [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) * [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) * [DESC](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md) * [DROP DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/DROP_DATABASE.md) * [ALTER DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/ALTER_DATABASE.md) --- ### DROP DATABASE DROP DATABASE is used to delete a database in StarRocks. > **NOTE** > > This operation requires the DROP privilege on the destination database. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DROP DATABASE [IF EXISTS] [FORCE] ``` Take note of the following points: * After executing DROP DATABASE to drop a database, you can restore the dropped database by using the [RECOVER](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/RECOVER.md) statement within a specified retention period (the default retention period spans one day), but the pipes (supported from v3.2 onwards) that have been dropped along with the database cannot be recovered. * If you execute `DROP DATABASE FORCE` to drop a database, the database is deleted directly without any checks on whether there are unfinished activities in it and cannot be recovered. Generally, this operation is not recommended. * If you drop a database, all pipes (supported from v3.2 onwards) that belong to the database are dropped along with the database. #### Examples[​](#examples "Direct link to Examples") 1. Drop database db\_text. ```sql DROP DATABASE db_test; ``` #### References[​](#references "Direct link to References") * [CREATE DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/CREATE_DATABASE.md) * [SHOW CREATE DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_CREATE_DATABASE.md) * [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) * [DESC](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md) * [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) --- ### SHOW CREATE DATABASE Shows the SQL command used to create a database. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW CREATE DATABASE ``` #### Parameters[​](#parameters "Direct link to Parameters") `db_name`: the database name, required. #### Returns[​](#returns "Direct link to Returns") * `Database`: the database name * `Create Database`: the SQL command used to create the database #### Examples[​](#examples "Direct link to Examples") ```sql mysql > show create database zj_test; +----------+---------------------------+ | Database | Create Database | +----------+---------------------------+ | zj_test | CREATE DATABASE `zj_test` | +----------+---------------------------+ ``` #### References[​](#references "Direct link to References") * [CREATE DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/CREATE_DATABASE.md) * [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) * [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) * [DESC](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md) * [DROP DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/DROP_DATABASE.md) --- ### SHOW DATA SHOW DATA is used to display the amount of data, the number of copies, and the number of statistical rows in a database or a database table. Syntax: ```sql SHOW DATA [FROM [.]] ``` Note: 1. If the FROM clause is not specified, the amount of data and copies subdivided into each table in the current db will be displayed. Where the data volume is the total data volume of all replicas. The number of copies is the number of copies of all partitions of the table and all materialized views. 2. If the FROM clause is specified, the amount of data, the number of copies and the number of statistical rows subdivided into each materialized view under the table are displayed. Where the data volume is the total data volume of all replicas. The number of copies is the number of copies of all partitions corresponding to the materialized view. The number of statistical rows is the number of statistical rows of all partitions corresponding to the materialized view. 3. When counting the number of rows, the copy with the largest number of rows among multiple copies shall prevail. 4. The Total row in the result set represents the summary row. The Quota row represents the quota set by the current database. The Left line represents the remaining quota. #### Examples[​](#examples "Direct link to Examples") Example 1: Display the data volume, copy quantity, summary data volume and summary copy quantity of each table in the current database. ```plain SHOW DATA; +-----------+-------------+--------------+ | TableName | Size | ReplicaCount | +-----------+-------------+--------------+ | tbl1 | 900.000 B | 6 | | tbl2 | 500.000 B | 3 | | Total | 1.400 KB | 9 | | Quota | 1024.000 GB | 1073741824 | | Left | 1021.921 GB | 1073741815 | +-----------+-------------+--------------+ ``` Example 2: Display the amount of breakdown data, the number of copies, and the number of statistical rows in the specified table. ```plain SHOW DATA FROM example_db.test; +-----------+-----------+-----------+--------------+----------+ | TableName | IndexName | Size | ReplicaCount | RowCount | +-----------+-----------+-----------+--------------+----------+ | test | r1 | 10.000MB | 30 | 10000 | | | r2 | 20.000MB | 30 | 20000 | | | test2 | 50.000MB | 30 | 50000 | | | Total | 80.000 | 90 | | +-----------+-----------+-----------+--------------+----------+ ``` --- ### SHOW DATABASES SHOW DATABASES lists databases in your current StarRocks cluster or an external data source. StarRocks supports viewing databases of an external data source from v2.3 onwards. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW DATABASES [FROM ] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | catalog\_name | No | The name of the internal catalog or an external catalog.- If you do not specify the parameter or specify the name of the internal catalog, which is `default_catalog`, you can view databases in your current StarRocks cluster.
- If you set the value of the parameter to the name of an external catalog, you can view databases in the corresponding external data source. You can run [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to view internal and external catalogs. | #### Examples[​](#examples "Direct link to Examples") Example 1: View databases in your current StarRocks cluster. ```sql SHOW DATABASES; ``` Or ```sql SHOW DATABASES FROM default_catalog; ``` The output of the preceding statements is as follows. ```sql +----------+ | Database | +----------+ | db1 | | db2 | | db3 | +----------+ ``` Example 2: View databases in a Hive cluster by using the `Hive1` external catalog. ```sql SHOW DATABASES FROM hive1; +-----------+ | Database | +-----------+ | hive_db1 | | hive_db2 | | hive_db3 | +-----------+ ``` #### References[​](#references "Direct link to References") * [CREATE DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/CREATE_DATABASE.md) * [SHOW CREATE DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_CREATE_DATABASE.md) * [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) * [DESC](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md) * [DROP DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/DROP_DATABASE.md) --- ### USE USE specifies the active database for your session. You can then perform operations, such as creating tables and executing queries. #### Syntax[​](#syntax "Direct link to Syntax") ```sql USE [.] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | catalog\_name | No | The catalog name.- If this parameter is not specified, a database in `default_catalog` is used by default.
- You must specify this parameter when you use a database from an external catalog. For more information, see Example 2.
- You must specify this parameter when you switch databases between different catalogs. For more information, see Example 3.For more information about catalogs, see [Overview](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md). | | db\_name | Yes | The database name. The database must exist. | #### Examples[​](#examples "Direct link to Examples") Example 1: Use `example_db` from `default_catalog` as the active database of your session. ```sql USE default_catalog.example_db; ``` Or ```sql USE example_db; ``` Example 2: Use `example_db` from `hive_catalog` as the active database of your session. ```sql USE hive_catalog.example_db; ``` Example 3: Switch the active database of your session from `hive_catalog.example_table1` to `iceberg_catalog.example_table2`. ```sql USE iceberg_catalog.example_table2; ``` --- ### CANCEL REFRESH DICTIONARY Cancels the refresh of a dictionary object. #### Syntax[​](#syntax "Direct link to Syntax") ```sql CANCEL REFRESH DICTIONARY ``` #### Parameters[​](#parameters "Direct link to Parameters") * **dictionary\_object\_name**: The name of the dictionary object that is in the REFRESHING state. #### Examples[​](#examples "Direct link to Examples") Cancel the refresh of the dictionary object `dict_obj`. ```plain MySQL > CANCEL REFRESH DICTIONARY dict_obj; Query OK, 0 rows affected (0.01 sec) ``` --- ### CREATE DICTIONARY Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) Creates a dictionary object based on an original object. The dictionary object organizes the key-value mappings from the original object in the form of a hash table and is cached in the memory of all BE nodes. It can be viewed as a cached table. **Advantages** * **Richer original objects for dictionary objects**: When using `dictionary_get()` to query dictionary objects, the original object can be a table of any type, asynchronous materialized view, or logical view. However, when using `dict_mapping()` to query dictionary tables, the dictionary tables can only be primary key tables. * **Fast query speed**: Since the dictionary object is a hash table and fully cached in the memory of all BE nodes, querying the dictionary object to get the mapping is realized by looking up the hash table in memory. Therefore, the query speed is very fast. * **Supports multiple value columns**: Internally, the dictionary object encodes multiple value columns into a single STRUCT type column. For queries based on a key, multiple values are returned together. Therefore, the dictionary object can serve as a dimension table where each key (usually a unique identifier) corresponds to multiple values (descriptive attributes). * **Ensures consistent snapshot reads**: The dictionary snapshot obtained within the same transaction is consistent, ensuring that the query results from the dictionary object do not change during the same query or load process. #### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE DICTIONARY USING ( column_name KEY, [..., column_name KEY,] column_name VALUE[, ..., column_name VALUE] ) [PROPERTIES ("key"="value", ...)]; ``` #### Parameters[​](#parameters "Direct link to Parameters") * `dictionary_object_name`: The name of the dictionary object. The dictionary object is effective globally and does not belong to a specific database. * `dictionary_source`: The name of the original object on which the dictionary object is based. The original object can be a table of any type, asynchronous materialized view, or logical view. * Definition of columns in the dictionary object: To preserve the key-value mapping maintained in the dictionary table, you need to use the `KEY` and `VALUE` keywords in the dictionary object's columns to specify the keys and their mapped values. * The column names `column_name` in the dictionary object must be consistent with those in the dictionary table. * The data types for key and value columns in the dictionary object are limited to boolean, integer, string, and date types. * The key column in the original object must ensure uniqueness. * Related properties of dictionary objects (`PROPERTIES`): * `dictionary_warm_up`: The method to cache data into the dictionary object on each BE node. Valid values: `TRUE` (default) or `FALSE`. If the parameter is set to `TRUE`, data is automatically cached into the dictionary object after its creation; if the parameter is set to `FALSE`, you need to manually refresh the dictionary object to cache the data. * `dictionary_memory_limit`: The maximum memory the dictionary object can occupy on each BE node. Unit: bytes. Default value: 2,000,000,000 bytes (2 GB). * `dictionary_refresh_interval`: The interval for periodically refreshing the dictionary object. Unit: seconds. Default value: `0`. A value `<=0` means no automatic refresh. * `dictionary_read_latest`: Whether to only query the latest dictionary object, mainly affecting the dictionary object queried during refresh. Valid values: `TRUE` or `FALSE` (default). If the parameter is set to `TRUE`, the dictionary object cannot be queried during refresh because the latest dictionary object is still being refreshed. If the parameter is set to `FALSE`, the previously successfully cached dictionary object can be queried during refresh. * `dictionary_ignore_failed_refresh`: Whether to automatically roll back to the last successfully cached dictionary object if the refresh fails. Valid values: `TRUE` or `FALSE` (default). If the parameter is set to `TRUE`, it automatically rolls back to the last successfully cached dictionary object when the refresh fails. If the parameter is set to `FALSE`, the dictionary object status is set to `CANCELLED` when the refresh fails. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The dictionary object is fully cached in the memory of each BE node, so it consumes relatively more memory. * Even if the original object is deleted, the dictionary object created based on it still exists. You need to manually DROP the dictionary object. #### Examples[​](#examples "Direct link to Examples") **Example 1: Create a simple dictionary object to replace the original dictionary table.** Take the following dictionary table as an example and insert test data. ```sql CREATE TABLE dict ( order_uuid STRING, order_id_int BIGINT AUTO_INCREMENT ) PRIMARY KEY (order_uuid) DISTRIBUTED BY HASH (order_uuid); ``` ```sql INSERT INTO dict (order_uuid) VALUES ('a1'), ('a2'), ('a3'); ``` ```sql Query OK, 3 rows affected (0.12 sec) {'label':'insert_9e60b0e4-89fa-11ee-a41f-b22a2c00f66b', 'status':'VISIBLE', 'txnId':'15029'} ``` ```sql SELECT * FROM dict; ``` ```sql +------------+--------------+ | order_uuid | order_id_int | +------------+--------------+ | a1 | 1 | | a2 | 2 | | a3 | 3 | +------------+--------------+ 3 rows in set (0.01 sec) ``` Create a dictionary object based on the mappings in this dictionary table. ```sql CREATE DICTIONARY dict_obj USING dict (order_uuid KEY, order_id_int VALUE); ``` For future queries of the mappings in the dictionary table, you can directly query the dictionary object instead of the dictionary table. For example, query the value mapped by key `a1`. ```sql SELECT dictionary_get("dict_obj", "a1"); ``` ```sql +--------------------+ | DICTIONARY_GET | +--------------------+ | {"order_id_int":1} | +--------------------+ 1 row in set (0.01 sec) ``` **Example 2: Create a dictionary object to replace the original dimension table** Take the following dimension table as an example and insert test data. ```sql CREATE TABLE ProductDimension ( ProductKey BIGINT AUTO_INCREMENT, ProductName VARCHAR(100) NOT NULL, Category VARCHAR(50), SubCategory VARCHAR(50), Brand VARCHAR(50), Color VARCHAR(20), Size VARCHAR(20) ) PRIMARY KEY (ProductKey) DISTRIBUTED BY HASH (ProductKey); ``` ```sql INSERT INTO ProductDimension (ProductName, Category, SubCategory, Brand, Color, Size) VALUES ('T-Shirt', 'Apparel', 'Shirts', 'BrandA', 'Red', 'M'), ('Jeans', 'Apparel', 'Pants', 'BrandB', 'Blue', 'L'), ('Running Shoes', 'Footwear', 'Athletic', 'BrandC', 'Black', '10'), ('Jacket', 'Apparel', 'Outerwear', 'BrandA', 'Green', 'XL'), ('Baseball Cap', 'Accessories', 'Hats', 'BrandD', 'White', 'OneSize'); ``` ```sql Query OK, 5 rows affected (0.48 sec) {'label':'insert_e938481f-181e-11ef-a6a9-00163e19e14e', 'status':'VISIBLE', 'txnId':'50'} ``` ```sql SELECT * FROM ProductDimension; ``` ```sql +------------+---------------+-------------+-------------+--------+-------+---------+ | ProductKey | ProductName | Category | SubCategory | Brand | Color | Size | +------------+---------------+-------------+-------------+--------+-------+---------+ | 1 | T-Shirt | Apparel | Shirts | BrandA | Red | M | | 2 | Jeans | Apparel | Pants | BrandB | Blue | L | | 3 | Running Shoes | Footwear | Athletic | BrandC | Black | 10 | | 4 | Jacket | Apparel | Outerwear | BrandA | Green | XL | | 5 | Baseball Cap | Accessories | Hats | BrandD | White | OneSize | +------------+---------------+-------------+-------------+--------+-------+---------+ 5 rows in set (0.02 sec) ``` Create a dictionary object to replace the original dimension table. ```sql CREATE DICTIONARY dimension_obj USING ProductDimension (ProductKey KEY, ProductName VALUE, Category VALUE, SubCategory VALUE, Brand VALUE, Color VALUE, Size VALUE); ``` ```sql Query OK, 0 rows affected (0.00 sec) ``` For future queries of dimension values, you can directly query the dictionary object instead of the dimension table to obtain dimension values. For example, query the value mapped by key `1`. ```sql SELECT dictionary_get("dict_obj", "a1"); ``` ```sql 1 row in set (0.01 sec) +-----------------------------------------+ | DICTIONARY_GET('dict_obj', 'a1', false) | +-----------------------------------------+ | {"order_id_int":1} | +-----------------------------------------+ ``` --- ### DROP DICTIONARY Delete a dictionary object or clear the cached data within a dictionary object. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DROP DICTIONARY [ CACHE ] ``` #### Parameters[​](#parameters "Direct link to Parameters") * `dictionary_object_name`: The name of the dictionary object. * `CACHE`: If the keyword `CACHE` is specified, only the cached data within the dictionary object will be cleared. To restore the cached data later, you can manually refresh it. If the keyword `CACHE` is not specified, the dictionary object will be deleted. #### Examples[​](#examples "Direct link to Examples") * Example 1: Clear only the cached data within the dictionary object. ```plain DROP DICTIONARY dict_obj CACHE; ``` The dictionary object still exists. ```plain MySQL > SHOW DICTIONARY dict_obj\G *************************** 1. row *************************** DictionaryId: 5 DictionaryName: dict_obj DbName: example_db dictionaryObject: dict dictionaryKeys: [order_uuid] dictionaryValues: [order_id_int] status: UNINITIALIZED lastSuccessRefreshTime: 2024-05-24 12:59:10 lastSuccessFinishedTime: 2024-05-24 12:59:20 nextSchedulableTime: disable auto schedule for refreshing ErrorMessage: approximated dictionaryMemoryUsage (Bytes): 172.26.80.55:8060 : 0 172.26.80.56:8060 : 0 172.26.80.57:8060 : 0 1 row in set (0.00 sec) ``` * Example 2: Delete the dictionary object `dict_obj`. ```plain DROP DICTIONARY dict_obj; ``` The dictionary object is completely deleted and no longer exists. ```plain MySQL > SHOW DICTIONARY dict_obj; Empty set (0.00 sec) ``` --- ### REFRESH DICTIONARY Manually refreshes a dictionary object. Internally, the system will query the latest data from the original object and write it into the dictionary object. #### Syntax[​](#syntax "Direct link to Syntax") ```sql REFRESH DICTIONARY ``` #### Parameters[​](#parameters "Direct link to Parameters") * **dictionary\_object\_name**: The name of the dictionary object. #### Examples[​](#examples "Direct link to Examples") Manually refresh the dictionary object `dict_obj`. ```plain MySQL > REFRESH DICTIONARY dict_obj; Query OK, 0 rows affected (0.01 sec) ``` --- ### SHOW DICTIONARY Shows information about dictionary objects. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW DICTIONARY [ ] ``` #### Parameters[​](#parameters "Direct link to Parameters") * `dictionary_object_name`: The name of the dictionary object. #### Returns[​](#returns "Direct link to Returns") * `DictionaryId`: The unique ID of each dictionary object. * `DictionaryName`: The name of the dictionary object. * `DbName`: The database to which the original object belongs. * `dictionarySource`: The name of the original object. * `dictionaryKeys`: The key columns of the dictionary object. * `dictionaryValues`: The value columns of the dictionary object. * `state`: The refresh state of the dictionary object. * `UNINITIALIZED`: The dictionary object has just been created and has not yet been refreshed. * `REFRESHING`: The dictionary object is currently being refreshed. * `COMMITTING`: Data has been cached in each BE node's dictionary object, and some final tasks are being performed, such as updating metadata. * `FINISHED`: The refresh was successful and has ended. * `CANCELLED`: An error occurred during the refresh process, and the refresh was canceled. You need to fix the issue based on the error information in `ErrorMessage`, and then refresh the dictionary object again. * `lastSuccessRefreshTime`: The start time of the last successful refresh of the dictionary object. * `lastSuccessFinishedTime`: The end time of the last successful refresh of the dictionary object. * `nextSchedulableTime`: The next time the dictionary object data will be automatically refreshed. * `ErrorMessage`: Error message when refreshing the dictionary object fails. * `approximated dictionaryMemoryUsage (Bytes)`: The estimated memory usage of the dictionary object cached on each BE node. #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SHOW DICTIONARY\G *************************** 1. row *************************** DictionaryId: 3 DictionaryName: dict_obj DbName: example_db dictionaryObject: dict dictionaryKeys: [order_uuid] dictionaryValues: [order_id_int] status: FINISHED lastSuccessRefreshTime: 2024-05-23 16:12:03 lastSuccessFinishedTime: 2024-05-23 16:12:12 nextSchedulableTime: disable auto schedule for refreshing ErrorMessage: approximated dictionaryMemoryUsage (Bytes): 172.26.82.208:8060 : 30 172.26.82.210:8060 : 30 172.26.82.209:8060 : 30 *************************** 2. row *************************** DictionaryId: 4 DictionaryName: dimension_obj DbName: example_db dictionaryObject: ProductDimension dictionaryKeys: [ProductKey] dictionaryValues: [ProductName, Category, SubCategory, Brand, Color, Size] status: FINISHED lastSuccessRefreshTime: 2024-05-23 16:12:41 lastSuccessFinishedTime: 2024-05-23 16:12:42 nextSchedulableTime: disable auto schedule for refreshing ErrorMessage: approximated dictionaryMemoryUsage (Bytes): 172.26.82.208:8060 : 270 172.26.82.210:8060 : 270 172.26.82.209:8060 : 270 2 rows in set (0.00 sec) ``` --- ### Function #### [📄️ CREATE FUNCTION](https://docs.starrocks.io/docs/sql-reference/sql-statements/Function/CREATE_FUNCTION.md) [CREATE FUNCTION creates a user-defined function (UDF).](https://docs.starrocks.io/docs/sql-reference/sql-statements/Function/CREATE_FUNCTION.md) #### [📄️ DROP FUNCTION](https://docs.starrocks.io/docs/sql-reference/sql-statements/Function/DROP_FUNCTION.md) [DROP FUNCTION deletes a custom function.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Function/DROP_FUNCTION.md) #### [📄️ SHOW CREATE FUNCTION](https://docs.starrocks.io/docs/sql-reference/sql-statements/Function/SHOW_CREATE_FUNCTION.md) [SHOW CREATE FUNCTION returns the CREATE FUNCTION DDL for a user-defined function.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Function/SHOW_CREATE_FUNCTION.md) #### [📄️ SHOW FUNCTIONS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Function/SHOW_FUNCTIONS.md) [SHOW FUNCTIONS queries all the custom (or built-in) functions under a database.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Function/SHOW_FUNCTIONS.md) --- ### CREATE FUNCTION CREATE FUNCTION creates a user-defined function (UDF). Currently, you can only create Java UDFs, including Scalar functions, user-defined aggregate functions (UDAFs), user-defined window functions (UDWFs), and user-defined table functions (UDTFs). **For details about how to compile, create, and use a Java UDF, see [Java UDF](https://docs.starrocks.io/docs/sql-reference/sql-functions/JAVA_UDF.md).** > **NOTE** > > To create a global UDF, you must have the SYSTEM-level CREATE GLOBAL FUNCTION privilege. To create a database-wide UDF, you must have the DATABASE-level CREATE FUNCTION privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE [OR REPLACE] [GLOBAL] [AGGREGATE | TABLE] FUNCTION function_name (arg_type [, ...]) RETURNS return_type PROPERTIES ("key" = "value" [, ...]) ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | -------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | OR REPLACE | No | If a function exists sharing the same function signature, it will be replaced. Supported from v3.4. | | GLOBAL | No | Whether to create a global UDF, supported from v3.0. | | AGGREGATE | No | Whether to create a UDAF or UDWF. | | TABLE | No | Whether to create a UDTF. If both `AGGREGATE` and `TABLE` are not specified, a Scalar function is created. | | function\_name | Yes | The name of the function you want to create. You can include the name of the database in this parameter, for example,`db1.my_func`. If `function_name` includes the database name, the UDF is created in that database. Otherwise, the UDF is created in the current database. The name of the new function and its parameters cannot be the same as an existing name in the destination database. Otherwise, the function cannot be created. The creation succeeds if the function name is the same but the parameters are different. | | arg\_type | Yes | Argument type of the function. The added argument can be represented by `, ...`. For the supported data types, see [Java UDF](https://docs.starrocks.io/docs/sql-reference/sql-functions/JAVA_UDF.md#mapping-between-sql-data-types-and-java-data-types). | | return\_type | Yes | The return type of the function. For the supported data types, see [Java UDF](https://docs.starrocks.io/docs/sql-reference/sql-functions/JAVA_UDF.md#mapping-between-sql-data-types-and-java-data-types). | | PROPERTIES | Yes | Properties of the function, which vary depending on the type of the UDF to create. For details, see [Java UDF](https://docs.starrocks.io/docs/sql-reference/sql-functions/JAVA_UDF.md#step-6-create-the-udf-in-starrocks). | --- ### DROP FUNCTION #### Description[​](#description "Direct link to Description") DROP FUNCTION deletes a custom function. The function can only be deleted when its name and parameter type are consistent. Only the owner of the custom function have the permissions to delete the function. ##### Syntax[​](#syntax "Direct link to Syntax") ```sql DROP FUNCTION function_name(arg_type [, ...]) ``` ##### Parameters[​](#parameters "Direct link to Parameters") `function_name`: the name of function to be dropped. `arg_type`: the argument type of function to be dropped. #### Examples[​](#examples "Direct link to Examples") 1. Drop a function. ```sql DROP FUNCTION my_add(INT, INT) ``` --- ### SHOW CREATE FUNCTION SHOW CREATE FUNCTION returns the `CREATE FUNCTION` DDL for a user-defined function. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW CREATE [GLOBAL] FUNCTION ( [, ...] ) ``` #### Example[​](#example "Direct link to Example") ```sql SHOW CREATE FUNCTION default_db.python_add(BIGINT); ``` ```sql CREATE FUNCTION default_db.python_add(BIGINT) RETURNS BIGINT PROPERTIES ( "type" = "Python", "file" = "inline", "symbol" = "add", "input" = "arrow" ) AS $$ import pyarrow.compute as pc def add(x): return pc.add(x, 1) $$ 1 row in set (0.01 sec) ``` --- ### SHOW FUNCTIONS SHOW FUNCTIONS queries all the custom (or built-in) functions under a database. If no database is specified, the current database is used by default. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW [FULL] [BUILTIN] FUNCTIONS [IN|FROM db] [LIKE 'function_pattern'] ``` #### Parameters[​](#parameters "Direct link to Parameters") ```plain full: indicates showing all the functions. builtin: indicates showing functions provided by the system. db: the name of the database to query. function_pattern: the pattern used to filter function name. ``` #### Examples[​](#examples "Direct link to Examples") ```plain mysql> show full functions in testDb\G *************************** 1. row *************************** Signature: my_add(INT,INT) Return Type: INT Function Type: Scalar Intermediate Type: NULL Properties: {"symbol":"_ZN9starrocks_udf6AddUdfEPNS_15FunctionContextERKNS_6IntValES4_","object_file":"http://host:port/libudfsample.so","md5":"cfe7a362d10f3aaf6c49974ee0f1f878"} *************************** 2. row *************************** Signature: my_count(BIGINT) Return Type: BIGINT Function Type: Aggregate Intermediate Type: NULL Properties: {"object_file":"http://host:port/libudasample.so","finalize_fn":"_ZN9starrocks_udf13CountFinalizeEPNS_15FunctionContextERKNS_9BigIntValE","init_fn":"_ZN9starrocks_udf9CountInitEPNS_15FunctionContextEPNS_9BigIntValE","merge_fn":"_ZN9starrocks_udf10CountMergeEPNS_15FunctionContextERKNS_9BigIntValEPS2_","md5":"37d185f80f95569e2676da3d5b5b9d2f","update_fn":"_ZN9starrocks_udf11CountUpdateEPNS_15FunctionContextERKNS_6IntValEPNS_9BigIntValE"} 2 rows in set (0.00 sec) mysql> show builtin functions in testDb like 'year%'; +---------------+ | Function Name | +---------------+ | year | | years_add | | years_diff | | years_sub | +---------------+ 2 rows in set (0.00 sec) ``` --- ### Generated columns Since v3.1, StarRocks supports generated columns. Generated columns can be used to accelerate queries with complex expressions. This feature supports precomputing and storing the results of expressions and [query rewrites](#query-rewrites), which significantly accelerates queries with the same complex expressions. You can define one or more generated columns to store the results of expressions at table creation. As such, when executing queries that contain the expression whose results are stored in the generated column you have defined, the CBO rewrites the query to read data directly from the generated column. Alternatively, you can directly query the data in the generated column. It is also recommended to **evaluate the impact of generated columns on loading performance because computing expressions takes some time**. Additionally, it is advised to **[create generated columns at table creation](#create-generated-columns-at-table-creation-recommended) rather than adding or modifying them after table creation**. Because it is time-consuming and costly to add or modify generated columns after table creation. However, note that when data is loaded into tables with a generated column, the time and overhead may increase because StarRocks needs to perform additional computation to calculate the results based on the expression and writes the results into the generated column. From v3.5.0, StarRocks shared-data clusters support generated columns. #### Basic operations[​](#basic-operations "Direct link to Basic operations") ##### Create generated columns[​](#create-generated-columns "Direct link to Create generated columns") ###### Syntax[​](#syntax "Direct link to Syntax") ```sql [NULL] AS [COMMENT 'string'] ``` ###### Create generated columns at table creation[​](#create-generated-columns-at-table-creation "Direct link to Create generated columns at table creation") Create a table named `test_tbl1` with five columns of which columns `newcol1` and `newcol2` are generated columns whose values are computed by using the specified expressions and referencing the values of the regular columns `data_array` and `data_json` respectively. ```sql CREATE TABLE test_tbl1 ( id INT NOT NULL, data_array ARRAY NOT NULL, data_json JSON NOT NULL, newcol1 DOUBLE AS array_avg(data_array), newcol2 String AS json_string(json_query(data_json, "$.a")) ) PRIMARY KEY (id) DISTRIBUTED BY HASH(id); ``` **NOTICE**: * Generated columns must be defined after regular columns. * Aggregate functions cannot be used in the expressions for generated columns. * The expressions for generated column cannot reference other generated columns or [auto-increment columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/auto_increment.md), but the expressions can reference multiple regular columns. * The data type of a generated column must match the data type of the result generated by the expression for the generated column. * Generated columns cannot be created on Aggregate tables. ###### Add generated columns after table creation[​](#add-generated-columns-after-table-creation "Direct link to Add generated columns after table creation") tip In most cases, the expressions frequently used during querying are determined after table creation, so generated columns are often added after table creation. For performance purposes, the underlying logic for StarRocks to add generated columns after table creation is optimized. As such, when adding a generated column, StarRocks does not need to rewrite all the data. Instead, StarRocks only needs to write the data of the newly added generated column and associate that data with the existing physical data files, which greatly improves the efficiency of adding generated columns after table creation. 1. Create a table named `test_tbl2` with three regular columns `id`, `data_array`, and `data_json`. Insert a data row into the table. ```sql -- Create a table. CREATE TABLE test_tbl2 ( id INT NOT NULL, data_array ARRAY NOT NULL, data_json JSON NOT NULL ) PRIMARY KEY (id) DISTRIBUTED BY HASH(id); -- Insert a data row. INSERT INTO test_tbl2 VALUES (1, [1,2], parse_json('{"a" : 1, "b" : 2}')); -- Query the table. MySQL [example_db]> select * from test_tbl2; +------+------------+------------------+ | id | data_array | data_json | +------+------------+------------------+ | 1 | [1,2] | {"a": 1, "b": 2} | +------+------------+------------------+ 1 row in set (0.04 sec) ``` 2. Execute ALTER TABLE ... ADD COLUMN ... to add generated columns `newcol1` and `newcol2`, which are created by evaluating the expressions based on the values of regular columns `data_array` and `data_json`. ```sql ALTER TABLE test_tbl2 ADD COLUMN newcol1 DOUBLE AS array_avg(data_array); ALTER TABLE test_tbl2 ADD COLUMN newcol2 String AS json_string(json_query(data_json, "$.a")); ``` **NOTICE**: * Adding generated columns to Aggregate tables is not supported. * Regular columns need to be defined before generated columns. When you use the ALTER TABLE ... ADD COLUMN ... statement to add a regular column without specifying the position of the new regular column, the system automatically places it before the generated columns. Moreover, you cannot use AFTER to explicitly place the regular column after a generated column. 3. Query the table data. ```sql MySQL [example_db]> SELECT * FROM test_tbl2; +------+------------+------------------+---------+---------+ | id | data_array | data_json | newcol1 | newcol2 | +------+------------+------------------+---------+---------+ | 1 | [1,2] | {"a": 1, "b": 2} | 1.5 | 1 | +------+------------+------------------+---------+---------+ 1 row in set (0.04 sec) ``` The result shows that the generated columns `newcol1` and `newcol2` are added to the table, and StarRocks automatically computes their values based on the expression. ##### Load data into generated columns[​](#load-data-into-generated-columns "Direct link to Load data into generated columns") During data loading, StarRocks automatically calculates the values for generated columns based on the expressions. You cannot specify the values of generated columns. The following example uses the [INSERT INTO](https://docs.starrocks.io/docs/loading/InsertInto.md) statement to load data: 1. Use INSERT INTO to insert a record into the `test_tbl1` table. Note that you cannot specify the values for the generated columns within the `VALUES ()` clause. ```sql INSERT INTO test_tbl1 (id, data_array, data_json) VALUES (1, [1,2], parse_json('{"a" : 1, "b" : 2}')); ``` 2. Query the table data. ```sql MySQL [example_db]> SELECT * FROM test_tbl1; +------+------------+------------------+---------+---------+ | id | data_array | data_json | newcol1 | newcol2 | +------+------------+------------------+---------+---------+ | 1 | [1,2] | {"a": 1, "b": 2} | 1.5 | 1 | +------+------------+------------------+---------+---------+ 1 row in set (0.01 sec) ``` The results show that StarRocks automatically computes the values for the generated columns `newcol1` and `newcol2` based on the expressions. **NOTICE**: The following error is returned if you specify values for the generated columns during data loading: ```sql MySQL [example_db]> INSERT INTO test_tbl1 (id, data_array, data_json, newcol1, newcol2) VALUES (2, [3,4], parse_json('{"a" : 3, "b" : 4}'), 3.5, "3"); ERROR 1064 (HY000): Getting analyzing error. Detail message: materialized column 'newcol1' can not be specified. MySQL [example_db]> INSERT INTO test_tbl1 VALUES (2, [3,4], parse_json('{"a" : 3, "b" : 4}'), 3.5, "3"); ERROR 1064 (HY000): Getting analyzing error. Detail message: Column count doesn't match value count. ``` ##### Modify generated columns[​](#modify-generated-columns "Direct link to Modify generated columns") tip When modifying a generated column, StarRocks needs to rewrite all the data, which is time-consuming and resource-intensive. If it is unavoidable to use ALTER TABLE to modify generated columns, it is recommended to evaluate the cost and time involved in advance. You can modify the data type and expression of a generated column. 1. Create a table `test_tbl3` with five columns of which columns `newcol1` and `newcol2` are generated columns whose values are computed by using the specified expressions and referencing the values of the regular columns `data_array` and `data_json` respectively. Insert a data row into the table. ```sql -- Create a table. MySQL [example_db]> CREATE TABLE test_tbl3 ( id INT NOT NULL, data_array ARRAY NOT NULL, data_json JSON NOT NULL, -- The data types and expressions of generated columns are specified as follows: newcol1 DOUBLE AS array_avg(data_array), newcol2 String AS json_string(json_query(data_json, "$.a")) ) PRIMARY KEY (id) DISTRIBUTED BY HASH(id); -- Insert a data row. INSERT INTO test_tbl3 (id, data_array, data_json) VALUES (1, [1,2], parse_json('{"a" : 1, "b" : 2}')); -- Query the table. MySQL [example_db]> select * from test_tbl3; +------+------------+------------------+---------+---------+ | id | data_array | data_json | newcol1 | newcol2 | +------+------------+------------------+---------+---------+ | 1 | [1,2] | {"a": 1, "b": 2} | 1.5 | 1 | +------+------------+------------------+---------+---------+ 1 row in set (0.01 sec) ``` 2. Modified the generated columns `newcol1` and `newcol2`: * Change the data type of the generated column `newcol1` to `ARRAY` and change its expression to `data_array`. ```sql ALTER TABLE test_tbl3 MODIFY COLUMN newcol1 ARRAY AS data_array; ``` * Modify the expression of the generated column `newcol2` to extract the values of field `b` from the regular column `data_json`. ```sql ALTER TABLE test_tbl3 MODIFY COLUMN newcol2 String AS json_string(json_query(data_json, "$.b")); ``` 3. View the modified schema and the data in the table. * View the modified schema. ```sql MySQL [example_db]> show create table test_tbl3\G **** 1. row **** Table: test_tbl3 Create Table: CREATE TABLE test_tbl3 ( id int(11) NOT NULL COMMENT "", data_array array NOT NULL COMMENT "", data_json json NOT NULL COMMENT "", -- After modification, the data types and expressions of generated columns are as follows: newcol1 array NULL AS example_db.test_tbl3.data_array COMMENT "", newcol2 varchar(65533) NULL AS json_string(json_query(example_db.test_tbl3.data_json, '$.b')) COMMENT "" ) ENGINE=OLAP PRIMARY KEY(id) DISTRIBUTED BY HASH(id) PROPERTIES (...); 1 row in set (0.00 sec) ``` * Query the table data after modification. The result shows that StarRocks recalculates the values of the generated columns `newcol1` and `newcol2` based on the modified expressions. ```sql MySQL [example_db]> select * from test_tbl3; +------+------------+------------------+---------+---------+ | id | data_array | data_json | newcol1 | newcol2 | +------+------------+------------------+---------+---------+ | 1 | [1,2] | {"a": 1, "b": 2} | [1,2] | 2 | +------+------------+------------------+---------+---------+ 1 row in set (0.01 sec) ``` ##### Drop a generated column[​](#drop-a-generated-column "Direct link to Drop a generated column") Drop column `newcol1` from the table `test_tbl3` ```sql ALTER TABLE test_tbl3 DROP COLUMN newcol1; ``` > **NOTICE**: > > If a generated colum references a regular column in the expression, you cannot directly drop or modify that regular column. Instead, you need to first drop the generated column and then drop or modify the regular column. ##### Query rewrites[​](#query-rewrites "Direct link to Query rewrites") If the expression in a query matches the expression of a generated column, the optimizer automatically rewrites the query to directly read the values of the generated column. 1. Suppose that you create a table `test_tbl4` with the following schema: ```sql CREATE TABLE test_tbl4 ( id INT NOT NULL, data_array ARRAY NOT NULL, data_json JSON NOT NULL, newcol1 DOUBLE AS array_avg(data_array), newcol2 String AS json_string(json_query(data_json, "$.a")) ) PRIMARY KEY (id) DISTRIBUTED BY HASH(id); ``` 2. If you query the data in the table `test_tbl4` by using the `SELECT array_avg(data_array), json_string(json_query(data_json, "$.a")) FROM test_tbl4;` statement, the query involves only the regular columns `data_array` and `data_json`. However, the expressions in the query match the expressions of the generated columns `newcol1` and `newcol2`. In this case, the execution plan shows that the CBO automatically rewrites the query to read the values of the generated columns `newcol1` and `newcol2`. ```sql MySQL [example_db]> EXPLAIN SELECT array_avg(data_array), json_string(json_query(data_json, "$.a")) FROM test_tbl4; +---------------------------------------+ | Explain String | +---------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS:4: newcol1 | 5: newcol2 | -- The query is rewritten to read data from the generated columns newcol1 and newcol2 are accessed. | PARTITION: RANDOM | | | | RESULT SINK | | | | 0:OlapScanNode | | TABLE: test_tbl4 | | PREAGGREGATION: ON | | partitions=0/1 | | rollup: test_tbl4 | | tabletRatio=0/0 | | tabletList= | | cardinality=1 | | avgRowSize=2.0 | +---------------------------------------+ 15 rows in set (0.00 sec) ``` ##### Partial updates and generated columns[​](#partial-updates-and-generated-columns "Direct link to Partial updates and generated columns") To perform partial updates on a Primary Key table, you must specify all the regular columns referenced by the generated columns in the `columns` parameter. The following example uses Stream Load to perform partial updates. 1. Create a table `test_tbl5` with five columns of which columns `newcol1` and `newcol2` are generated columns whose values are computed by using the specified expressions and referencing the values of the regular columns `data_array` and `data_json` respectively. Insert a data row into the table. ```sql -- Create a table. CREATE TABLE test_tbl5 ( id INT NOT NULL, data_array ARRAY NOT NULL, data_json JSON NULL, newcol1 DOUBLE AS array_avg(data_array), newcol2 String AS json_string(json_query(data_json, "$.a")) ) PRIMARY KEY (id) DISTRIBUTED BY HASH(id); -- Insert into a data row. INSERT INTO test_tbl5 (id, data_array, data_json) VALUES (1, [1,2], parse_json('{"a" : 1, "b" : 2}')); -- Query the table. MySQL [example_db]> select * from test_tbl5; +------+------------+------------------+---------+---------+ | id | data_array | data_json | newcol1 | newcol2 | +------+------------+------------------+---------+---------+ | 1 | [1,2] | {"a": 1, "b": 2} | 1.5 | 1 | +------+------------+------------------+---------+---------+ 1 row in set (0.01 sec) ``` 2. Prepare a CSV file `my_data1.csv` to update some columns in the `test_tbl5` table. ```sql 1|[3,4]|{"a": 3, "b": 4} 2|[3,4]|{"a": 3, "b": 4} ``` 3. Use [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) with the `my_data1.csv` file to update some columns of the `test_tbl5` table. You need to set `partial_update:true` and specify all the regular columns referenced by the generated columns in the `columns` parameter. ```bash curl --location-trusted -u : -H "label:1" \ -H "column_separator:|" \ -H "partial_update:true" \ -H "columns:id,data_array,data_json" \ -T my_data1.csv -XPUT \ http://:/api/example_db/test_tbl5/_stream_load ``` 4. Query the table data. ```sql [example_db]> select * from test_tbl5; +------+------------+------------------+---------+---------+ | id | data_array | data_json | newcol1 | newcol2 | +------+------------+------------------+---------+---------+ | 1 | [3,4] | {"a": 3, "b": 4} | 3.5 | 3 | | 2 | [3,4] | {"a": 3, "b": 4} | 3.5 | 3 | +------+------------+------------------+---------+---------+ 2 rows in set (0.01 sec) ``` An error is returned by Stream Load if you perform partial updates without specifying all the regular columns referenced by the generated columns. 1. Prepare a CSV file `my_data2.csv`. ```csv 1|[3,4] 2|[3,4] ``` 2. When partial column updates are performed by using [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) with the `my_data2.csv` file, if the values for the `data_json` column are not provided in `my_data2.csv` and the `columns` parameter in the Stream Load job does not include the `data_json` column, even if the `data_json` column allows null values, an error is returned by Stream Load because the column `data_json` is referenced by the generated column `newcol2`. --- ### Keywords This topic describes non-reserved keywords and reserved keywords. It provides a list of reserved keywords in StarRocks. #### Introduction[​](#introduction "Direct link to Introduction") Keywords in SQL statements, such as `CREATE` and `DROP`, have special meanings when parsed by StarRocks. Keywords are classified into non-reserved keywords and reserved keywords. * **Non-reserved keywords** can be directly used as identifiers without special treatment, such as table names and column names. For example, `DB` is a non-reserved keyword. You can create a database named `DB`. ```sql CREATE DATABASE DB; Query OK, 0 rows affected (0.00 sec) ``` * **Reserved keywords** can be used as identifiers only after special treatment. For example, `LIKE` is a reserved keyword. If you want to use it to identify a database, enclose it in a pair of backticks (\`). ```sql CREATE DATABASE `LIKE`; Query OK, 0 rows affected (0.01 sec) ``` If it is not enclosed in backticks, an error is returned: ```sql CREATE DATABASE LIKE; ERROR 1064 (HY000): Getting syntax error at line 1, column 16. Detail message: Unexpected input 'like', the most similar input is {a legal identifier}. ``` #### Reserved keywords[​](#reserved-keywords "Direct link to Reserved keywords") Following are StarRocks reserved keywords arranged in alphabetical order. You must enclose them in a pair of backticks (\`) if you want to use them as identifiers. The reserved keywords may vary with StarRocks versions. ##### A[​](#a "Direct link to A") * ADD * ALL * ALTER * ANALYZE * AND * ARRAY * AS * ASC ##### B[​](#b "Direct link to B") * BETWEEN * BIGINT * BITMAP * BOTH * BY ##### C[​](#c "Direct link to C") * CASE * CHAR * CHARACTER * CHECK * COLLATE * COLUMN * COMPACTION (v3.1 and later) * CONVERT * CREATE * CROSS * CUBE * CURRENT\_DATE * CURRENT\_TIME * CURRENT\_TIMESTAMP * CURRENT\_USER * CURRENT\_ROLE (v3.0 and later) ##### D[​](#d "Direct link to D") * DATABASE * DATABASES * DECIMAL * DECIMALV2 * DECIMAL32 * DECIMAL64 * DECIMAL128 * DEFAULT * DELETE * DENSE\_RANK * DESC * DESCRIBE * DISTINCT * DOUBLE * DROP * DUAL * DEFERRED (v3.0 and later) ##### E[​](#e "Direct link to E") * ELSE * EXCEPT * EXCLUDE * EXISTS * EXPLAIN ##### F[​](#f "Direct link to F") * FALSE * FIRST\_VALUE * FLOAT * FOR * FORCE * FROM * FULL * FUNCTION ##### G[​](#g "Direct link to G") * GRANT * GROUP * GROUPS * GROUPING * GROUPING\_ID ##### H[​](#h "Direct link to H") * HAVING * HLL * HOST ##### I[​](#i "Direct link to I") * IF * IGNORE * IN * INDEX * INFILE * INNER * INSERT * INT * INTEGER * INTERSECT * INTO * IS * IMMEDIATE (v3.0 and later) ##### J[​](#j "Direct link to J") * JOIN * JSON ##### K[​](#k "Direct link to K") * KEY * KEYS * KILL ##### L[​](#l "Direct link to L") * LAG * LARGEINT * LAST\_VALUE * LATERAL * LEAD * LEFT * LIKE * LIMIT * LOAD * LOCALTIME * LOCALTIMESTAMP ##### M[​](#m "Direct link to M") * MAXVALUE * MINUS * MOD ##### N[​](#n "Direct link to N") * NTILE * NOT * NULL ##### O[​](#o "Direct link to O") * ON * OR * ORDER * OUTER * OUTFILE * OVER ##### P[​](#p "Direct link to P") * PARTITION * PERCENTILE * PRIMARY * PROCEDURE ##### Q[​](#q "Direct link to Q") * QUALIFY ##### R[​](#r "Direct link to R") * RANGE * RANK * READ * REGEXP * RELEASE * RENAME * REPLACE * REVOKE * RIGHT * RLIKE * ROW * ROWS * ROW\_NUMBER ##### S[​](#s "Direct link to S") * SCHEMA * SCHEMAS * SELECT * SET * SET\_VAR * SHOW * SMALLINT * SYSTEM ##### T[​](#t "Direct link to T") * TABLE * TERMINATED * TEXT (v3.1 and later) * THEN * TINYINT * TO * TRUE ##### U[​](#u "Direct link to U") * UNION * UNIQUE * UNSIGNED * UPDATE * USE * USING ##### V[​](#v "Direct link to V") * VALUES * VARCHAR ##### W[​](#w "Direct link to W") * WHEN * WHERE * WITH --- ### Loading and unloading #### [📄️ ALTER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ALTER_LOAD.md) [ALTER LOAD changes the priority of a Broker Load job that is in the QUEUEING or LOADING state.](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ALTER_LOAD.md) #### [📄️ BROKER\_LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) [StarRocks provides the MySQL-based loading method Broker Load.](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) #### [📄️ CANCEL LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/CANCEL_LOAD.md) [CANCEL LOAD cancels a given load job: Broker Load, Spark Load, or INSERT.](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/CANCEL_LOAD.md) #### [🗃️ ETL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/ALTER_TASK.md) [3 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/ALTER_TASK.md) #### [📄️ INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) [Inserts data into a specific table or overwrites a specific table with data.](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) #### [📄️ SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md) [SHOW LOAD displays information of all load jobs or given load jobs in a database.](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md) #### [📄️ SHOW TRANSACTION](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION.md) [SHOW TRANSACTION is used to view the transaction details of the specified transaction id.](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION.md) #### [📄️ SPARK LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SPARK_LOAD.md) [SPARK LOAD imports data into StarRocks using Spark for preprocessing, improving performance for large-scale initial data migrations.](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SPARK_LOAD.md) #### [📄️ STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) [STREAM LOAD allows you to load data from a local file system or a streaming data source.](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) #### [🗃️ pipe](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/ALTER_PIPE.md) [6 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/ALTER_PIPE.md) #### [🗃️ routine\_load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/ALTER_ROUTINE_LOAD.md) [7 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/ALTER_ROUTINE_LOAD.md) #### [🗃️ unloading](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/unloading/CANCEL_EXPORT.md) [3 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/unloading/CANCEL_EXPORT.md) --- ### ALTER LOAD ALTER LOAD changes the priority of a Broker Load job that is in the **QUEUEING** or **LOADING** state. This statement is supported since v2.5. > **NOTE** > > Changing the priority of a Broker Load job that is in the **LOADING** state does not affect the execution of the job. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ALTER LOAD FOR PROPERTIES ( 'priority'='{LOWEST | LOW | NORMAL | HIGH | HIGHEST}' ) ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | Description | | ------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | label\_name | Yes | The label of the load job. Format: `[.]`. See [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#database_name-and-label_name). | | priority | Yes | The new priority that you want to specify for the load job. Valid values: `LOWEST`, `LOW`, `NORMAL`, `HIGH`, and `HIGHEST`. See [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). | #### Examples[​](#examples "Direct link to Examples") Suppose that you have a Broker Load job whose label is `test_db.label1` and the job is in the **QUEUEING** state. If you want to run the job at the soonest, you can run the following command to change the priority of the job to `HIGHEST`: ```sql ALTER LOAD FOR test_db.label1 PROPERTIES ( 'priority'='HIGHEST' ); ``` --- ### BROKER LOAD StarRocks provides the MySQL-based loading method Broker Load. After you submit a load job, StarRocks asynchronously runs the job. You can use `SELECT * FROM information_schema.loads` to query the job result. This feature is supported from v3.1 onwards. For more information about the background information, principles, supported data file formats, how to perform single-table loads and multi-table loads, and how to view job results, see [loading overview](https://docs.starrocks.io/docs/loading/loading_introduction.md). You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. #### Syntax[​](#syntax "Direct link to Syntax") ```sql LOAD LABEL [.] ( data_desc[, data_desc ...] ) WITH BROKER ( StorageCredentialParams ) [PROPERTIES ( opt_properties ) ] ``` Note that in StarRocks some literals are used as reserved keywords by the SQL language. Do not directly use these keywords in SQL statements. If you want to use such a keyword in an SQL statement, enclose it in a pair of backticks (\`). See [Keywords](https://docs.starrocks.io/docs/sql-reference/sql-statements/keywords.md). #### Parameters[​](#parameters "Direct link to Parameters") ##### database\_name and label\_name[​](#database_name-and-label_name "Direct link to database_name and label_name") `label_name` specifies the label of the load job. For the naming conventions, see [System limits](https://docs.starrocks.io/docs/sql-reference/System_limit.md). `database_name` optionally specifies the name of the database to which the destination table belongs. Each load job has a label that is unique across the entire database. You can use the label of a load job to view the execution status of the load job and prevent repeatedly loading the same data. When a load job enters the **FINISHED** state, its label cannot be reused. Only the label of a load job that has entered the **CANCELLED** state can be reused. In most cases, the label of a load job is reused to retry that load job and load the same data, thereby implementing Exactly-Once semantics. For label naming conventions, see [System limits](https://docs.starrocks.io/docs/sql-reference/System_limit.md). ##### data\_desc[​](#data_desc "Direct link to data_desc") The description of a batch of data to be loaded. Each `data_desc` descriptor declares information such as the data source, ETL functions, destination StarRocks table, and destination partitions. Broker Load supports loading multiple data files at a time. In one load job, you can use multiple `data_desc` descriptors to declare multiple data files you want to load, or use one `data_desc` descriptor to declare one file path from which you want to load all data files in it. Broker Load can also ensure the transactional atomicity of each load job that is run to load multiple data files. Atomicity means that the loading of multiple data files in one load job must all succeed or fail. It never happens that the loading of some data files succeeds while the loading of the other files fails. `data_desc` supports the following syntax: ```sql DATA INFILE (""[, "" ...]) [NEGATIVE] INTO TABLE [PARTITION ([, ...])] [TEMPORARY PARTITION ([, ...])] [COLUMNS TERMINATED BY ""] [ROWS TERMINATED BY ""] [FORMAT AS "CSV | Parquet | ORC"] [(format_type_options)] [(column_list)] [COLUMNS FROM PATH AS ([, ...])] [SET [, ...]] [WHERE predicate] ``` `data_desc` must include the following parameters: * `file_path` Specifies the save path of one or more data files you want to load. You can specify this parameter as the save path of one data file. For example, you can specify this parameter as `"hdfs://:/user/data/tablename/20210411"` to load a data file named `20210411` from the path `/user/data/tablename` on the HDFS server. You can also specify this parameter as the save path of multiple data files by using wildcards `?`, `*`, `[]`, `{}`, or `^`. See [Wildcard reference](https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html#globStatus-org.apache.hadoop.fs.Path-). For example, you can specify this parameter as `"hdfs://:/user/data/tablename/*/*"` or `"hdfs://:/user/data/tablename/dt=202104*/*"` to load the data files from all partitions or only `202104` partitions in the path `/user/data/tablename` on the HDFS server. > **NOTE** > > Wildcards can also be used to specify intermediate paths. In the preceding examples, the `hdfs_host` and `hdfs_port` parameters are described as follows: * `hdfs_host`: the IP address of the NameNode host in the HDFS cluster. * `hdfs_host`: the FS port of the NameNode host in the HDFS cluster. The default port number is `9000`. > **NOTICE** > > * Broker Load supports accessing AWS S3 according to the S3 or S3A protocol. Therefore, when you load data from AWS S3, you can include `s3://` or `s3a://` as the prefix in the S3 URI that you pass as the file path. > > * Broker Load supports accessing Google GCS only according to the gs protocol. Therefore, when you load data from Google GCS, you must include `gs://` as the prefix in the GCS URI that you pass as the file path. > > * When you load data from Blob Storage, you must use the wasb or wasbs protocol to access your data: > > > > * If your storage account allows access over HTTP, use the wasb protocol and write the file path as `wasb://@.blob.core.windows.net///*`. > * If your storage account allows access over HTTPS, use the wasbs protocol and write the file path as `wasbs://@.blob.core.windows.net///*`. > > * When you load data from Data Lake Storage Gen2, you must use the abfs or abfss protocol to access your data: > > > > * If your storage account allows access over HTTP, use the abfs protocol and write the file path as `abfs://@.dfs.core.windows.net/`. > * If your storage account allows access over HTTPS, use the abfss protocol and write the file path as `abfss://@.dfs.core.windows.net/`. > > * When you load data from Data Lake Storage Gen1, you must use the adl protocol to access your data and write the file path as `adl://.azuredatalakestore.net//`. * `INTO TABLE` Specifies the name of the destination StarRocks table. `data_desc` can also optionally include the following parameters: * `NEGATIVE` Revokes the loading of a specific batch of data. To achieve this, you need to load the same batch of data with the `NEGATIVE` keyword specified. > **NOTE** > > This parameter is valid only when the StarRocks table is an Aggregate table and all its value columns are computed by the `sum` function. * `PARTITION` Specifies the partitions into which you want to load data. By default, if you do not specify this parameter, the source data will be loaded into all partitions of the StarRocks table. * `TEMPORARY PARTITION` Specifies the name of the [temporary partition](https://docs.starrocks.io/docs/table_design/data_distribution/Temporary_partition.md) into which you want to load data. You can specify multiple temporary partitions, which must be separated by commas (,). * `COLUMNS TERMINATED BY` Specifies the column separator used in the data file. By default, if you do not specify this parameter, this parameter defaults to `\t`, indicating tab. The column separator you specify using this parameter must be the same as the column separator that is actually used in the data file. Otherwise, the load job will fail due to inadequate data quality, and its `State` will be `CANCELLED`. Broker Load jobs are submitted according to the MySQL protocol. StarRocks and MySQL both escape characters in the load requests. Therefore, if the column separator is an invisible character such as tab, you must add a backslash () preceding the column separator. For example, you must input `\\t` if the column separator is `\t`, and you must input `\\n` if the column separator is `\n`. Apache Hive™ files use `\x01` as their column separator, so you must input `\\x01` if the data file is from Hive. > **NOTE** > > * For CSV data, you can use a UTF-8 string, such as a comma (,), tab, or pipe (|), whose length does not exceed 50 bytes as a text delimiter. > * Null values are denoted by using `\N`. For example, a data file consists of three columns, and a record from that data file holds data in the first and third columns but no data in the second column. In this situation, you need to use `\N` in the second column to denote a null value. This means the record must be compiled as `a,\N,b` instead of `a,,b`. `a,,b` denotes that the second column of the record holds an empty string. * `ROWS TERMINATED BY` Specifies the row separator used in the data file. By default, if you do not specify this parameter, this parameter defaults to `\n`, indicating line break. The row separator you specify using this parameter must be the same as the row separator that is actually used in the data file. Otherwise, the load job will fail due to inadequate data quality, and its `State` will be `CANCELLED`. This parameter is supported from v2.5.4 onwards. For the usage notes about the row separator, see the usage notes for the preceding `COLUMNS TERMINATED BY` parameter. * `FORMAT AS` Specifies the format of the data file. Valid values: `CSV`, `Parquet`, and `ORC`. By default, if you do not specify this parameter, StarRocks determines the data file format based on the filename extension **.csv**, **.parquet**, or **.orc** specified in the `file_path` parameter. * `format_type_options` Specifies CSV format options when `FORMAT AS` is set to `CSV`. Syntax: ```json ( key = value key = value ... ) ``` > **NOTE** > > `format_type_options` is supported in v3.0 and later. The following parameters describe the options. ###### skip\_header[​](#skip_header "Direct link to skip_header") Specifies whether to skip the first rows of the data file when the data file is in CSV format. Type: INTEGER. Default value: `0`. In some CSV-formatted data files, the first rows at the beginning are used to define metadata such as column names and column data types. By setting the `skip_header` parameter, you can enable StarRocks to skip the first rows of the data file during data loading. For example, if you set this parameter to `1`, StarRocks skips the first row of the data file during data loading. The first rows at the beginning in the data file must be separated by using the row separator that you specify in the load statement. ###### trim\_space[​](#trim_space "Direct link to trim_space") Specifies whether to remove spaces preceding and following column separators from the data file when the data file is in CSV format. Type: BOOLEAN. Default value: `false`. For some databases, spaces are added to column separators when you export data as a CSV-formatted data file. Such spaces are called leading spaces or trailing spaces depending on their locations. By setting the `trim_space` parameter, you can enable StarRocks to remove such unnecessary spaces during data loading. Note that StarRocks does not remove the spaces (including leading spaces and trailing spaces) within a field wrapped in a pair of `enclose`-specified characters. For example, the following field values use pipe (`|`) as the column separator and double quotation marks (`"`) as the `enclose`-specified character: ```plain |"Love StarRocks"| |" Love StarRocks "| | "Love StarRocks" | ``` If you set `trim_space` to `true`, StarRocks processes the preceding field values as follows: ```plain |"Love StarRocks"| |" Love StarRocks "| |"Love StarRocks"| ``` ###### enclose[​](#enclose "Direct link to enclose") Specifies the character that is used to wrap the field values in the data file according to [RFC4180](https://www.rfc-editor.org/rfc/rfc4180) when the data file is in CSV format. Type: single-byte character. Default value: `NONE`. The most prevalent characters are single quotation mark (`'`) and double quotation mark (`"`). All special characters (including row separators and column separators) wrapped by using the `enclose`-specified character are considered normal symbols. StarRocks can do more than RFC4180 as it allows you to specify any single-byte character as the `enclose`-specified character. If a field value contains an `enclose`-specified character, you can use the same character to escape that `enclose`-specified character. For example, you set `enclose` to `"`, and a field value is `a "quoted" c`. In this case, you can enter the field value as `"a ""quoted"" c"` into the data file. ###### escape[​](#escape "Direct link to escape") Specifies the character that is used to escape various special characters, such as row separators, column separators, escape characters, and `enclose`-specified characters, which are then considered by StarRocks to be common characters and are parsed as part of the field values in which they reside. Type: single-byte character. Default value: `NONE`. The most prevalent character is slash (`\`), which must be written as double slashes (`\\`) in SQL statements. > **NOTE** > > The character specified by `escape` is applied to both inside and outside of each pair of `enclose`-specified characters. Two examples are as follows: * When you set `enclose` to `"` and `escape` to `\`, StarRocks parses `"say \"Hello world\""` into `say "Hello world"`. * Assume that the column separator is comma (`,`). When you set `escape` to `\`, StarRocks parses `a, b\, c` into two separate field values: `a` and `b, c`. * `column_list` Specifies the column mapping between the data file and the StarRocks table. Syntax: `([, ...])`. The columns declared in `column_list` are mapped by name onto the StarRocks table columns. > **NOTE** > > If the columns of the data file are mapped in sequence onto the columns of the StarRocks table, you do not need to specify `column_list`. If you want to skip a specific column of the data file, you only need to temporarily name that column as different from any of the StarRocks table columns. For more information, see [loading overview](https://docs.starrocks.io/docs/loading/loading_introduction.md). * `COLUMNS FROM PATH AS` Extracts the information about one or more partition fields from the file path you specify. This parameter is valid only when the file path contains partition fields. For example, if the data file is stored in the path `/path/col_name=col_value/file1` in which `col_name` is a partition field and can be mapped onto a column of the StarRocks table, you can specify this parameter as `col_name`. As such, StarRocks extracts `col_value` values from the path and loads them into the StarRocks table column onto which `col_name` is mapped. > **NOTE** > > This parameter is available only when you load data from HDFS. * `SET` Specifies one or more functions that you want to use to convert a column of the data file. Examples: * The StarRocks table consists of three columns, which are `col1`, `col2`, and `col3` in sequence. The data file consists of four columns, among which the first two columns are mapped in sequence onto `col1` and `col2` of the StarRocks table and the sum of the last two columns is mapped onto `col3` of the StarRocks table. In this case, you need to specify `column_list` as `(col1,col2,tmp_col3,tmp_col4)` and specify `(col3=tmp_col3+tmp_col4)` in the SET clause to implement data conversion. * The StarRocks table consists of three columns, which are `year`, `month`, and `day` in sequence. The data file consists of only one column that accommodates date and time values in `yyyy-mm-dd hh:mm:ss` format. In this case, you need to specify `column_list` as `(tmp_time)` and specify `(year = year(tmp_time), month=month(tmp_time), day=day(tmp_time))` in the SET clause to implement data conversion. * `WHERE` Specifies the conditions based on which you want to filter the source data. StarRocks loads only the source data that meets the filter conditions specified in the WHERE clause. ##### WITH BROKER[​](#with-broker "Direct link to WITH BROKER") In v2.3 and earlier, input `WITH BROKER ""` to specify the broker you want to use. From v2.5 onwards, you no longer need to specify a broker, but you still need to retain the `WITH BROKER` keyword. ##### StorageCredentialParams[​](#storagecredentialparams "Direct link to StorageCredentialParams") The authentication information used by StarRocks to access your storage system. ###### HDFS[​](#hdfs "Direct link to HDFS") Open-source HDFS supports two authentication methods: simple authentication and Kerberos authentication. Broker Load uses simple authentication by default. Open-source HDFS also supports configuring an HA mechanism for the NameNode. If you choose open-source HDFS as your storage system, you can specify the authentication configuration and HA configuration as follows: * Authentication configuration * If you use simple authentication, configure `StorageCredentialParams` as follows: ```plain "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ``` The following table describes the parameters in `StorageCredentialParams`. | Parameter | Description | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hadoop.security.authentication | The authentication method. Valid values: `simple` and `kerberos`. Default value: `simple`. `simple` represents simple authentication, meaning no authentication, and `kerberos` represents Kerberos authentication. | | username | The username of the account that you want to use to access the NameNode of the HDFS cluster. | | password | The password of the account that you want to use to access the NameNode of the HDFS cluster. | * If you use Kerberos authentication, configure `StorageCredentialParams` as follows: ```plain "hadoop.security.authentication" = "kerberos", "kerberos_principal" = "nn/zelda1@ZELDA.COM", "kerberos_keytab" = "/keytab/hive.keytab", "kerberos_keytab_content" = "YWFhYWFh" ``` The following table describes the parameters in `StorageCredentialParams`. | Parameter | Description | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hadoop.security.authentication | The authentication method. Valid values: `simple` and `kerberos`. Default value: `simple`. `simple` represents simple authentication, meaning no authentication, and `kerberos` represents Kerberos authentication. | | kerberos\_principal | The Kerberos principal to be authenticated. Each principal consists of the following three parts to ensure that it is unique across the HDFS cluster:- `username` or `servicename`: The name of the principal.
- `instance`: the name of the server that hosts the node to be authenticated in the HDFS cluster. The server name helps ensure that the principal is unique, for example, when the HDFS cluster consists of multiple DataNodes that each are independently authenticated.
- `realm`: The name of the realm. The realm name must be capitalized.Example: `nn/zelda1@ZELDA.COM`. | | kerberos\_keytab | The save path of the Kerberos keytab file. | | kerberos\_keytab\_content | The Base64-encoded content of the the Kerberos keytab file. You can choose to specify either `kerberos_keytab` or `kerberos_keytab_content`. | * HA configuration You can configure an HA mechanism for the NameNode of the HDFS cluster. This way, if the NameNode is switched over to another node, StarRocks can automatically identify the new node that serves as the NameNode. This includes the following scenarios: * If you load data from a single HDFS cluster that has one Kerberos user configured, both broker-based loading and broker-free loading are supported. * To perform broker-based loading, make sure that at least one independent broker group is deployed, and place the `hdfs-site.xml` file to the `{deploy}/conf` path on the broker node that serves the HDFS cluster. StarRocks will add the `{deploy}/conf` path to the environment variable `CLASSPATH` upon broker startup, allowing the brokers to read information about the HDFS cluster nodes. * To perform broker-free loading, you only need to set `hadoop.security.authentication = kerberos` in `conf/core-site.xml` under the deployment directories of all FE, BE, and CN nodes in your cluster, and use the `kinit` command to configure the Kerberos account. * If you load data from a single HDFS cluster that has multiple Kerberos users configured, only broker-based loading is supported. Make sure that at least one independent broker group is deployed, and place the `hdfs-site.xml` file to the `{deploy}/conf` path on the broker node that serves the HDFS cluster. StarRocks will add the `{deploy}/conf` path to the environment variable `CLASSPATH` upon broker startup, allowing the brokers to read information about the HDFS cluster nodes. * If you load data from multiple HDFS clusters (regardless of whether one or multiple Kerberos users are configured), only broker-based loading is supported. Make sure that at least one independent broker group is deployed for each of these HDFS clusters, and take one of the following actions to enable the brokers to read information about the HDFS cluster nodes: * Place the `hdfs-site.xml` file to the `{deploy}/conf` path on the broker node that serves each HDFS cluster. StarRocks will add the `{deploy}/conf` path to the environment variable `CLASSPATH` upon broker startup, allowing the brokers to read information about the nodes in that HDFS cluster. * Add the following HA configuration at job creation: ```plain "dfs.nameservices" = "ha_cluster", "dfs.ha.namenodes.ha_cluster" = "ha_n1,ha_n2", "dfs.namenode.rpc-address.ha_cluster.ha_n1" = ":", "dfs.namenode.rpc-address.ha_cluster.ha_n2" = ":", "dfs.client.failover.proxy.provider" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider" ``` The following table describes the parameters in the HA configuration. | Parameter | Description | | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | dfs.nameservices | The name of the HDFS cluster. | | dfs.ha.namenodes.XXX | The name of the NameNode in the HDFS cluster. If you specify multiple NameNode names, separate them with commas (`,`). `xxx` is the HDFS cluster name that you have specified in `dfs.nameservices`. | | dfs.namenode.rpc-address.XXX.NN | The RPC address of the NameNode in the HDFS cluster. `NN` is the NameNode name that you have specified in `dfs.ha.namenodes.XXX`. | | dfs.client.failover.proxy.provider | The provider of the NameNode to which the client will connect. Default value: `org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider`. | > **NOTE** > > You can use the [SHOW BROKER](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BROKER.md) statement to check for brokers that are deployed in your StarRocks cluster. ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") If you choose AWS S3 as your storage system, take one of the following actions: * To choose the instance profile-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "" ``` * To choose the assumed role-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.region" = "" ``` * To choose the IAM user-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ----------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.use\_instance\_profile | Yes | Specifies whether to enable the credential methods instance profile and assumed role. Valid values: `true` and `false`. Default value: `false`. | | aws.s3.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS S3 bucket. If you choose assumed role as the credential method for accessing AWS S3, you must specify this parameter. | | aws.s3.region | Yes | The region in which your AWS S3 bucket resides. Example: `us-west-1`. | | aws.s3.access\_key | No | The access key of your IAM user. If you choose IAM user as the credential method for accessing AWS S3, you must specify this parameter. | | aws.s3.secret\_key | No | The secret key of your IAM user. If you choose IAM user as the credential method for accessing AWS S3, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in AWS IAM Console, see [Authentication parameters for accessing AWS S3](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-s3). ###### Google GCS[​](#google-gcs "Direct link to Google GCS") If you choose Google GCS as your storage system, take one of the following actions: * To choose the VM-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.use_compute_engine_service_account" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | * To choose the service account-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------- | | gcp.gcs.service\_account\_email | "" | `"user@hello.iam.gserviceaccount.com"` | The email address in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the service account. | * To choose the impersonation-based authentication method, configure `StorageCredentialParams` as follows: * Make a VM instance impersonate a service account: ```sql "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The service account that you want to impersonate. | * Make a service account (named as meta service account) impersonate another service account (named as data service account): ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | gcp.gcs.service\_account\_email | "" | `"user@hello.iam.gserviceaccount.com"` | The email address in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the meta service account. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The data service account that you want to impersonate. | ###### Other S3-compatible storage system[​](#other-s3-compatible-storage-system "Direct link to Other S3-compatible storage system") If you choose other S3-compatible storage system, such as MinIO, configure `StorageCredentialParams` as follows: ```sql "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ---------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.enable\_ssl | Yes | Specifies whether to enable SSL connection. Valid values: `true` and `false`. Default value: `true`. | | aws.s3.enable\_path\_style\_access | Yes | Specifies whether to enable path-style URL access. Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`. | | aws.s3.endpoint | Yes | The endpoint that is used to connect to your S3-compatible storage system instead of AWS S3. | | aws.s3.access\_key | Yes | The access key of your IAM user. | | aws.s3.secret\_key | Yes | The secret key of your IAM user. | ###### Microsoft Azure Storage[​](#microsoft-azure-storage "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage "Direct link to Azure Blob Storage") If you choose Blob Storage as your storage system, take one of the following actions: * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | -------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.shared\_key | Yes | The shared key of your Blob Storage account. | * To choose the SAS Token authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | --------------------------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.container | Yes | The name of the blob container that stores your data. | | azure.blob.sas\_token | Yes | The SAS token that is used to access your Blob Storage account. | ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2 "Direct link to Azure Data Lake Storage Gen2") If you choose Data Lake Storage Gen2 as your storage system, take one of the following actions: * To choose the Managed Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_use\_managed\_identity | Yes | Specifies whether to enable the Managed Identity authentication method. Set the value to `true`. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the managed identity. | * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ---------------------------- | ------------ | -------------------------------------------------------------- | | azure.adls2.storage\_account | Yes | The username of your Data Lake Storage Gen2 storage account. | | azure.adls2.shared\_key | Yes | The shared key of your Data Lake Storage Gen2 storage account. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ``` The following table describes the parameters you need to configure `in StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls2.oauth2\_client\_secret | Yes | The value of the new client (application) secret created. | | azure.adls2.oauth2\_client\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | * To choose the Workload Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_token_file" = "", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_token\_file | Yes | The absolute file path to the OAuth2 token file projected into the pod by the Azure Workload Identity webhook. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client ID (application ID) of the Azure AD application (user-assigned managed identity or app registration) associated with the workload identity. | ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1 "Direct link to Azure Data Lake Storage Gen1") If you choose Data Lake Storage Gen1 as your storage system, take one of the following actions: * To choose the Managed Service Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.use_managed_service_identity" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------- | | azure.adls1.use\_managed\_service\_identity | Yes | Specifies whether to enable the Managed Service Identity authentication method. Set the value to `true`. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls1.oauth2\_client\_id | Yes | The client (application) ID of the . | | azure.adls1.oauth2\_credential | Yes | The value of the new client (application) secret created. | | azure.adls1.oauth2\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | ##### opt\_properties[​](#opt_properties "Direct link to opt_properties") Specifies some optional parameters whose settings are applied to the entire load job. Syntax: ```plain PROPERTIES ("" = ""[, "" = "" ...]) ``` The following parameters are supported: * `timeout` Specifies the timeout period of the load job. Unit: second. The default timeout period is 4 hours. We recommend that you specify a timeout period shorter than 6 hours. If the load job does not finish within the timeout period, StarRocks cancels the load job and the status of the load job becomes **CANCELLED**. > **NOTE** > > In most cases, you do not need to set the timeout period. We recommend that you set the timeout period only when the load job cannot finish within the default timeout period. Use the following formula to infer the timeout period: **Timeout period > (Total size of the data files to be loaded x Total number of the data files to be loaded and the materialized views created on the data files)/Average load speed** > **NOTE** > > "Average load speed" is the average load speed for your entire StarRocks cluster. The average load speed varies for each cluster depending on the server configuration and the maximum number of concurrent query tasks allowed for the cluster. You can infer the average load speed based on the load speeds of historical load jobs. Suppose that you want to load a 1-GB data file on which two materialized views are created into a StarRocks cluster whose average load speed is 10 MB/s. The amount of time required for the data load is approximately 102 seconds. (1 x 1024 x 3)/10 = 307.2 (second) For this example, we recommend that you set the timeout period to a value greater than 308 seconds. * `max_filter_ratio` Specifies the maximum error tolerance of the load job. The maximum error tolerance is the maximum percentage of rows that can be filtered out as a result of inadequate data quality. Valid values: `0`~`1`. Default value: `0`. * If you set this parameter to `0`, StarRocks does not ignore unqualified rows during loading. As such, if the source data contains unqualified rows, the load job fails. This helps ensure the correctness of the data loaded into StarRocks. * If you set this parameter to a value greater than `0`, StarRocks can ignore unqualified rows during loading. As such, the load job can succeed even if the source data contains unqualified rows. > **NOTE** > > Rows that are filtered out due to inadequate data quality do not include rows that are filtered out by the WHERE clause. If the load job fails because the maximum error tolerance is set to `0`, you can use [SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md) to view the job result. Then, determine whether unqualified rows can be filtered out. If unqualified rows can be filtered out, calculate the maximum error tolerance based on the values returned for `dpp.abnorm.ALL` and `dpp.norm.ALL` in the job result, adjust the maximum error tolerance, and submit the load job again. The formula for calculating the maximum error tolerance is as follows: `max_filter_ratio` = \[`dpp.abnorm.ALL`/(`dpp.abnorm.ALL` + `dpp.norm.ALL`)] The sum of the values returned for `dpp.abnorm.ALL` and `dpp.norm.ALL` is the total number of rows to be loaded. * `log_rejected_record_num` Specifies the maximum number of unqualified data rows that can be logged. This parameter is supported from v3.1 onwards. Valid values: `0`, `-1`, and any non-zero positive integer. Default value: `0`. * The value `0` specifies that data rows that are filtered out will not be logged. * The value `-1` specifies that all data rows that are filtered out will be logged. * A non-zero positive integer such as `n` specifies that up to `n` data rows that are filtered out can be logged on each BE. * `load_mem_limit` Specifies the maximum amount of memory that can be provided to the load job. The value of this parameter cannot exceed the upper memory limit supported by each BE or CN node. Unit: bytes: The default memory limit is 2 GB. * `strict_mode` Specifies whether to enable the [strict mode](https://docs.starrocks.io/docs/loading/strict_mode.md). Valid values: `true` and `false`. Default value: `false`. `true` specifies to enable the strict mode, and `false` specifies to disable the strict mode. * `timezone` Specifies the time zone of the load job. Default value: `Asia/Shanghai`. The time zone setting affects the results returned by functions such as strftime, alignment\_timestamp, and from\_unixtime. For more information, see [Configure a time zone](https://docs.starrocks.io/docs/administration/management/timezone.md). The time zone specified in the `timezone` parameter is a session-level time zone. * `priority` Specifies the priority of the load job. Valid values: `LOWEST`, `LOW`, `NORMAL`, `HIGH`, and `HIGHEST`. Default value: `NORMAL`. Broker Load provides the FE parameter `max_broker_load_job_concurrency`, determines the maximum number of Broker Load jobs that can be concurrently run within your StarRocks cluster. If the number of Broker Load jobs that are submitted within the specified time period exceeds the maximum number, excessive jobs will be waiting to be scheduled based on their priorities. You can use the [ALTER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ALTER_LOAD.md) statement to change the priority of an existing load job that is in the `QUEUEING` or `LOADING` state. StarRocks allows setting the `priority` parameter for a Broker Load job since v2.5. * `partial_update` Whether to use partial updates. Valid values: `TRUE` and `FALSE`. Default value: `FALSE`, indicating to disable this feature. * `partial_update_mode` Specifies the mode for partial updates. Valid values: `row` and `column`. * The value `row` (default) means partial updates in row mode, which is more suitable for real-time updates with many columns and small batches. * The value `column` means partial updates in column mode, which is more suitable for batch updates with few columns and many rows. In such scenarios, enabling the column mode offers faster update speeds. For example, in a table with 100 columns, if only 10 columns (10% of the total) are updated for all rows, the update speed of the column mode is 10 times faster. * `merge_condition` Specifies the name of the column you want to use as the condition to determine whether updates can take effect. The update from a source record to a destination record takes effect only when the source data record has a greater or equal value than the destination data record in the specified column. > **NOTE** > > The column that you specify cannot be a primary key column. Additionally, only tables that use the Primary Key table support conditional updates. StarRocks supports loading JSON data from v3.2.3 onwards. The parameters are as follows: * jsonpaths The names of the keys that you want to load from the JSON data file. You need to specify this parameter only when you load JSON data by using the matched mode. The value of this parameter is in JSON format. See [Configure column mapping for JSON data loading](#configure-column-mapping-for-json-data-loading). * strip\_outer\_array Specifies whether to strip the outermost array structure. Valid values: `true` and `false`. Default value: `false`. In real-world business scenarios, the JSON data may have an outermost array structure as indicated by a pair of square brackets `[]`. In this situation, we recommend that you set this parameter to `true`, so StarRocks removes the outermost square brackets `[]` and loads each inner array as a separate data record. If you set this parameter to `false`, StarRocks parses the entire JSON data file into one array and loads the array as a single data record. For example, the JSON data is `[ {"category" : 1, "author" : 2}, {"category" : 3, "author" : 4} ]`. If you set this parameter to `true`, `{"category" : 1, "author" : 2}` and `{"category" : 3, "author" : 4}` are parsed into separate data records that are loaded into separate StarRocks table rows. * json\_root The root element of the JSON data that you want to load from the JSON data file. You need to specify this parameter only when you load JSON data by using the matched mode. The value of this parameter is a valid JsonPath string. By default, the value of this parameter is empty, indicating that all data of the JSON data file will be loaded. For more information, see the "[Load JSON data using matched mode with root element specified](#load-json-data-using-matched-mode-with-root-element-specified)" section of this topic. When you load JSON data, also note that the size per JSON object cannot exceed 4 GB. If an individual JSON object in the JSON data file exceeds 4 GB in size, an error "This parser can't support a document that big." is reported. #### Column mapping[​](#column-mapping "Direct link to Column mapping") ##### Configure column mapping for CSV data loading[​](#configure-column-mapping-for-csv-data-loading "Direct link to Configure column mapping for CSV data loading") If the columns of the data file can be mapped one on one in sequence to the columns of the StarRocks table, you do not need to configure the column mapping between the data file and the StarRocks table. If the columns of the data file cannot be mapped one on one in sequence to the columns of the StarRocks table, you need to use the `columns` parameter to configure the column mapping between the data file and the StarRocks table. This includes the following two use cases: * **Same number of columns but different column sequence.** **Also, the data from the data file does not need to be computed by functions before it is loaded into the matching StarRocks table columns.** In the `columns` parameter, you need to specify the names of the StarRocks table columns in the same sequence as how the data file columns are arranged. For example, the StarRocks table consists of three columns, which are `col1`, `col2`, and `col3` in sequence, and the data file also consists of three columns, which can be mapped to the StarRocks table columns `col3`, `col2`, and `col1` in sequence. In this case, you need to specify `"columns: col3, col2, col1"`. * **Different number of columns and different column sequence. Also, the data from the data file needs to be computed by functions before it is loaded into the matching StarRocks table columns.** In the `columns` parameter, you need to specify the names of the StarRocks table columns in the same sequence as how the data file columns are arranged and specify the functions you want to use to compute the data. Two examples are as follows: * The StarRocks table consists of three columns, which are `col1`, `col2`, and `col3` in sequence. The data file consists of four columns, among which the first three columns can be mapped in sequence to the StarRocks table columns `col1`, `col2`, and `col3` and the fourth column cannot be mapped to any of the StarRocks table columns. In this case, you need to temporarily specify a name for the fourth column of the data file, and the temporary name must be different from any of the StarRocks table column names. For example, you can specify `"columns: col1, col2, col3, temp"`, in which the fourth column of the data file is temporarily named `temp`. * The StarRocks table consists of three columns, which are `year`, `month`, and `day` in sequence. The data file consists of only one column that accommodates date and time values in `yyyy-mm-dd hh:mm:ss` format. In this case, you can specify `"columns: col, year = year(col), month=month(col), day=day(col)"`, in which `col` is the temporary name of the data file column and the functions `year = year(col)`, `month=month(col)`, and `day=day(col)` are used to extract data from the data file column `col` and loads the data into the mapping StarRocks table columns. For example, `year = year(col)` is used to extract the `yyyy` data from the data file column `col` and loads the data into the StarRocks table column `year`. For detailed examples, see [Configure column mapping](#configure-column-mapping). ##### Configure column mapping for JSON data loading[​](#configure-column-mapping-for-json-data-loading "Direct link to Configure column mapping for JSON data loading") If the keys of the JSON document have the same names as the columns of the StarRocks table, you can load the JSON-formatted data by using the simple mode. In simple mode, you do not need to specify the `jsonpaths` parameter. This mode requires that the JSON-formatted data must be an object as indicated by curly brackets `{}`, such as `{"category": 1, "author": 2, "price": "3"}`. In this example, `category`, `author`, and `price` are key names, and these keys can be mapped one on one by name to the columns `category`, `author`, and `price` of the StarRocks table. If the keys of the JSON document have different names than the columns of the StarRocks table, you can load the JSON-formatted data by using the matched mode. In matched mode, you need to use the `jsonpaths` and `COLUMNS` parameters to specify the column mapping between the JSON document and the StarRocks table: * In the `jsonpaths` parameter, specify the JSON keys in the sequence as how they are arranged in the JSON document. * In the `COLUMNS` parameter, specify the mapping between the JSON keys and the StarRocks table columns: * The column names specified in the `COLUMNS` parameter are mapped one on one in sequence to the JSON keys. * The column names specified in the `COLUMNS` parameter are mapped one on one by name to the StarRocks table columns. For examples about loading JSON-formatted data by using the matched mode, see [Load JSON data using matched mode](#load-json-data-using-matched-mode). #### Related configuration items[​](#related-configuration-items "Direct link to Related configuration items") The FE configuration item `max_broker_load_job_concurrency` specifies the maximum number of Broker Load jobs that can be concurrently run within your StarRocks cluster. In StarRocks v2.4 and earlier, if the total number of Broker Load jobs that are submitted within a specific period of time exceeds the maximum number, excessive jobs will be queued and scheduled based on their submission time. Since StarRocks v2.5, if the total number of Broker Load jobs that are submitted within a specific period of time exceeds the maximum number, excessive jobs are queued and scheduled based on their priorities. You can specify a priority for a job by using the `priority` parameter described above. You can use [ALTER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ALTER_LOAD.md) to modify the priority of an existing job that is in the **QUEUEING** or **LOADING** state. #### Job splitting and concurrent running[​](#job-splitting-and-concurrent-running "Direct link to Job splitting and concurrent running") A Broker Load job can be split into one or more tasks that concurrently run. The tasks within a load job are run within a single transaction. They must all succeed or fail. StarRocks splits each load job based on how you declare `data_desc` in the `LOAD` statement: * If you declare multiple `data_desc` parameters, each of which specifies a distinct table, a task is generated to load the data of each table. * If you declare multiple `data_desc` parameters, each of which specifies a distinct partition for the same table, a task is generated to load the data of each partition. Additionally, each task can be further split into one or more instances, which are evenly distributed to and concurrently run on the BEs or CNs of your StarRocks cluster. StarRocks splits each task based on the FE parameter `min_bytes_per_broker_scanner` and the number of BE or CN nodes. You can use the following formula to calculate the number of instances in an individual task: **Number of instances in an individual task = min(Amount of data to be loaded by an individual task/`min_bytes_per_broker_scanner`, Number of BE/CN nodes)** In most cases, only one `data_desc` is declared for each load job, each load job is split into only one task, and the task is split into the same number of instances as the number of BE or CN nodes. #### Examples[​](#examples "Direct link to Examples") This section uses HDFS as an example to describe various load configurations. ##### Load CSV data[​](#load-csv-data "Direct link to Load CSV data") This section uses CSV as an example to explain the various parameter configurations that you can use to meet your diverse load requirements. ###### Set timeout period[​](#set-timeout-period "Direct link to Set timeout period") Your StarRocks database `test_db` contains a table named `table1`. The table consists of three columns, which are `col1`, `col2`, and `col3` in sequence. Your data file `example1.csv` also consists of three columns, which are mapped in sequence onto `col1`, `col2`, and `col3` of `table1`. If you want to load all data from `example1.csv` into `table1` within up to 3600 seconds, run the following command: ```sql LOAD LABEL test_db.label1 ( DATA INFILE("hdfs://:/user/starrocks/data/input/example1.csv") INTO TABLE table1 ) WITH BROKER ( "username" = "", "password" = "" ) PROPERTIES ( "timeout" = "3600" ); ``` ###### Set maximum error tolerance[​](#set-maximum-error-tolerance "Direct link to Set maximum error tolerance") Your StarRocks database `test_db` contains a table named `table2`. The table consists of three columns, which are `col1`, `col2`, and `col3` in sequence. Your data file `example2.csv` also consists of three columns, which are mapped in sequence onto `col1`, `col2`, and `col3` of `table2`. If you want to load all data from `example2.csv` into `table2` with a maximum error tolerance of `0.1`, run the following command: ```sql LOAD LABEL test_db.label2 ( DATA INFILE("hdfs://:/user/starrocks/data/input/example2.csv") INTO TABLE table2 ) WITH BROKER ( "username" = "", "password" = "" ) PROPERTIES ( "max_filter_ratio" = "0.1" ); ``` ###### Load all data files from a file path[​](#load-all-data-files-from-a-file-path "Direct link to Load all data files from a file path") Your StarRocks database `test_db` contains a table named `table3`. The table consists of three columns, which are `col1`, `col2`, and `col3` in sequence. All data files stored in the `/user/starrocks/data/input/` path of your HDFS cluster also each consist of three columns, which are mapped in sequence onto `col1`, `col2`, and `col3` of `table3`. The column separator used in these data files is `\x01`. If you want to load data from all these data files stored in the `hdfs://:/user/starrocks/data/input/` path into `table3`, run the following command: ```sql LOAD LABEL test_db.label3 ( DATA INFILE("hdfs://:/user/starrocks/data/input/*") INTO TABLE table3 COLUMNS TERMINATED BY "\\x01" ) WITH BROKER ( "username" = "", "password" = "" ); ``` ###### Set NameNode HA mechanism[​](#set-namenode-ha-mechanism "Direct link to Set NameNode HA mechanism") Your StarRocks database `test_db` contains a table named `table4`. The table consists of three columns, which are `col1`, `col2`, and `col3` in sequence. Your data file `example4.csv` also consists of three columns, which are mapped onto `col1`, `col2`, and `col3` of `table4`. If you want to load all data from `example4.csv` into `table4` with an HA mechanism configured for the NameNode, run the following command: ```sql LOAD LABEL test_db.label4 ( DATA INFILE("hdfs://:/user/starrocks/data/input/example4.csv") INTO TABLE table4 ) WITH BROKER ( "username" = "", "password" = "", "dfs.nameservices" = "my_ha", "dfs.ha.namenodes.my_ha" = "my_namenode1, my_namenode2","dfs.namenode.rpc-address.my_ha.my_namenode1" = "nn1_host:rpc_port", "dfs.namenode.rpc-address.my_ha.my_namenode2" = "nn2_host:rpc_port", "dfs.client.failover.proxy.provider" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider" ); ``` ###### Set Kerberos authentication[​](#set-kerberos-authentication "Direct link to Set Kerberos authentication") Your StarRocks database `test_db` contains a table named `table5`. The table consists of three columns, which are `col1`, `col2`, and `col3` in sequence. Your data file `example5.csv` also consists of three columns, which are mapped in sequence onto `col1`, `col2`, and `col3` of `table5`. If you want to load all data from `example5.csv` into `table5` with Kerberos authentication configured and the keytab file path specified, run the following command: ```sql LOAD LABEL test_db.label5 ( DATA INFILE("hdfs://:/user/starrocks/data/input/example5.csv") INTO TABLE table5 COLUMNS TERMINATED BY "\t" ) WITH BROKER ( "hadoop.security.authentication" = "kerberos", "kerberos_principal" = "starrocks@YOUR.COM", "kerberos_keytab" = "/home/starRocks/starRocks.keytab" ); ``` ###### Revoke data loading[​](#revoke-data-loading "Direct link to Revoke data loading") Your StarRocks database `test_db` contains a table named `table6`. The table consists of three columns, which are `col1`, `col2`, and `col3` in sequence. Your data file `example6.csv` also consists of three columns, which are mapped in sequence onto `col1`, `col2`, and `col3` of `table6`. You have loaded all data from `example6.csv` into `table6` by running a Broker Load job. If you want to revoke the data you have loaded, run the following command: ```sql LOAD LABEL test_db.label6 ( DATA INFILE("hdfs://:/user/starrocks/data/input/example6.csv") NEGATIVE INTO TABLE table6 COLUMNS TERMINATED BY "\t" ) WITH BROKER ( "hadoop.security.authentication" = "kerberos", "kerberos_principal" = "starrocks@YOUR.COM", "kerberos_keytab" = "/home/starRocks/starRocks.keytab" ); ``` ###### Specify destination partitions[​](#specify-destination-partitions "Direct link to Specify destination partitions") Your StarRocks database `test_db` contains a table named `table7`. The table consists of three columns, which are `col1`, `col2`, and `col3` in sequence. Your data file `example7.csv` also consists of three columns, which are mapped in sequence onto `col1`, `col2`, and `col3` of `table7`. If you want to load all data from `example7.csv` into two partitions, `p1` and `p2`, of `table7`, run the following command: ```sql LOAD LABEL test_db.label7 ( DATA INFILE("hdfs://:/user/starrocks/data/input/example7.csv") INTO TABLE table7 PARTITION (p1, p2) COLUMNS TERMINATED BY "," ) WITH BROKER ( "username" = "", "password" = "" ); ``` ###### Configure column mapping[​](#configure-column-mapping "Direct link to Configure column mapping") Your StarRocks database `test_db` contains a table named `table8`. The table consists of three columns, which are `col1`, `col2`, and `col3` in sequence. Your data file `example8.csv` also consists of three columns, which are mapped in sequence onto `col2`, `col1`, and `col3` of `table8`. If you want to load all data from `example8.csv` into `table8`, run the following command: ```sql LOAD LABEL test_db.label8 ( DATA INFILE("hdfs://:/user/starrocks/data/input/example8.csv") INTO TABLE table8 COLUMNS TERMINATED BY "," (col2, col1, col3) ) WITH BROKER ( "username" = "", "password" = "" ); ``` > **NOTE** > > In the preceding example, the columns of `example8.csv` cannot be mapped onto the columns of `table8` in the same sequence as how these columns are arranged in `table8`. Therefore, you need to use `column_list` to configure the column mapping between `example8.csv` and `table8`. ###### Set filter conditions[​](#set-filter-conditions "Direct link to Set filter conditions") Your StarRocks database `test_db` contains a table named `table9`. The table consists of three columns, which are `col1`, `col2`, and `col3` in sequence. Your data file `example9.csv` also consists of three columns, which are mapped in sequence onto `col1`, `col2`, and `col3` of `table9`. If you want to load only the data records whose values in the first column are greater than `20180601` from `example9.csv` into `table9`, run the following command: ```sql LOAD LABEL test_db.label9 ( DATA INFILE("hdfs://:/user/starrocks/data/input/example9.csv") INTO TABLE table9 (col1, col2, col3) where col1 > 20180601 ) WITH BROKER ( "username" = "", "password" = "" ); ``` > **NOTE** > > In the preceding example, the columns of `example9.csv` can be mapped onto the columns of `table9` in the same sequence as how these columns are arranged in `table9`, but you need to use the WHERE clause to specify column-based filter conditions. Therefore, you need to use `column_list` to configure the column mapping between `example9.csv` and `table9`. ###### Load data into tables containing HLL-type columns[​](#load-data-into-tables-containing-hll-type-columns "Direct link to Load data into tables containing HLL-type columns") Your StarRocks database `test_db` contains a table named `table10`. The table consists of four columns, which are `id`, `col1`, `col2`, and `col3` in sequence. `col1` and `col2` are defined as HLL-type columns. Your data file `example10.csv` consists of three columns, among which the first column is mapped onto `id` of `table10` and the second and third columns are mapped in sequence onto `col1` and `col2` of `table10`. The values in the second and third columns of `example10.csv` can be converted into HLL-type data by using functions before they are loaded into `col1` and `col2` of `table10`. If you want to load all data from `example10.csv` into `table10`, run the following command: ```sql LOAD LABEL test_db.label10 ( DATA INFILE("hdfs://:/user/starrocks/data/input/example10.csv") INTO TABLE table10 COLUMNS TERMINATED BY "," (id, temp1, temp2) SET ( col1 = hll_hash(temp1), col2 = hll_hash(temp2), col3 = empty_hll() ) ) WITH BROKER ( "username" = "", "password" = "" ); ``` > **NOTE** > > In the preceding example, the three columns of `example10.csv` are named `id`, `temp1`, and `temp2` in sequence by using `column_list`. Then, functions are used to convert data as follows: > > * The `hll_hash` function is used to convert the values in `temp1` and `temp2` of `example10.csv` into HLL-type data and map `temp1` and `temp2` of `example10.csv` onto `col1` and `col2` of `table10`. > > * The `hll_empty` function is used to fill the specified default value into `col3` of `table10`. For usage of the functions `hll_hash` and `hll_empty`, see [hll\_hash](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_hash.md) and [hll\_empty](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_empty.md). ###### Extract partition field values from file path[​](#extract-partition-field-values-from-file-path "Direct link to Extract partition field values from file path") Broker Load supports parsing the values of specific partition fields contained in a file path based on the column definitions of the destination StarRocks table. This feature of StarRocks is similar to the Partition Discovery feature of Apache Spark™. Your StarRocks database `test_db` contains a table named `table11`. The table consists of five columns, which are `col1`, `col2`, `col3`, `city`, and `utc_date` in sequence. The file path `/user/starrocks/data/input/dir/city=beijing` of your HDFS cluster contains the following data files: * `/user/starrocks/data/input/dir/city=beijing/utc_date=2019-06-26/0000.csv` * `/user/starrocks/data/input/dir/city=beijing/utc_date=2019-06-26/0001.csv` These data files each consist of three columns, which are mapped in sequence onto `col1`, `col2`, and `col3` of `table11`. If you want to load data from all data files from the file path `/user/starrocks/data/input/dir/city=beijing/utc_date=*/*` into `table11` and, at the same time, you want to extract the values of the partition fields `city` and `utc_date` contained in the file path and load the extracted values into `city` and `utc_date` of `table11`, run the following command: ```sql LOAD LABEL test_db.label11 ( DATA INFILE("hdfs://:/user/starrocks/data/input/dir/city=beijing/*/*") INTO TABLE table11 FORMAT AS "csv" (col1, col2, col3) COLUMNS FROM PATH AS (city, utc_date) SET (uniq_id = md5sum(k1, city)) ) WITH BROKER ( "username" = "", "password" = "" ); ``` ###### Extract partition field values from `%3A`-included file path[​](#extract-partition-field-values-from-3a-included-file-path "Direct link to extract-partition-field-values-from-3a-included-file-path") In HDFS, file paths cannot contain colons (:). All colons (:) will be converted into `%3A`. Your StarRocks database `test_db` contains a table named `table12`. The table consists of three columns, which are `data_time`, `col1`, and `col2` in sequence. The table schema is as follows: ```sql data_time DATETIME, col1 INT, col2 INT ``` The file path `/user/starrocks/data` of your HDFS cluster contains the following data files: * `/user/starrocks/data/data_time=2020-02-17 00%3A00%3A00/example12.csv` * `/user/starrocks/data/data_time=2020-02-18 00%3A00%3A00/example12.csv` If you want to load all data from `example12.csv` into `table12` and, at the same time, you want to extract the values of the partition field `data_time` from the file path and load the extracted values into `data_time` of `table12`, run the following command: ```sql LOAD LABEL test_db.label12 ( DATA INFILE("hdfs://:/user/starrocks/data/*/example12.csv") INTO TABLE table12 COLUMNS TERMINATED BY "," FORMAT AS "csv" (col1,col2) COLUMNS FROM PATH AS (data_time) SET (data_time = str_to_date(data_time, '%Y-%m-%d %H%%3A%i%%3A%s')) ) WITH BROKER ( "username" = "", "password" = "" ); ``` > **NOTE** > > In the preceding example, the values extracted from the partition field `data_time` are strings that contain `%3A`, such as `2020-02-17 00%3A00%3A00`. Therefore, you need to use the `str_to_date` function to convert the strings into DATETIME-type data before they are loaded into `data_time` of `table8`. ###### Setting `format_type_options`[​](#setting-format_type_options "Direct link to setting-format_type_options") Your StarRocks database `test_db` contains a table named `table13`. The table consists of three columns, which are `col1`, `col2`, and `col3` in sequence. Your data file `example13.csv` also consists of three columns, which are mapped in sequence onto `col2`, `col1`, and `col3` of `table13`. If you want to load all data from `example13.csv` into `table13`, with the intention of skipping the first two rows of `example13.csv`, removing the spaces preceding and following column separators, and setting `enclose` to `\` and `escape` to `\`, run the following command: ```sql LOAD LABEL test_db.label13 ( DATA INFILE("hdfs://:/user/starrocks/data/*/example13.csv") INTO TABLE table13 COLUMNS TERMINATED BY "," FORMAT AS "CSV" ( skip_header = 2 trim_space = TRUE enclose = "\"" escape = "\\" ) (col2, col1, col3) ) WITH BROKER ( "username" = "hdfs_username", "password" = "hdfs_password" ) PROPERTIES ( "timeout" = "3600" ); ``` ##### Load Parquet data[​](#load-parquet-data "Direct link to Load Parquet data") This section describes some parameters settings you need to take note of when you load Parquet data. Your StarRocks database `test_db` contains a table named `table13`. The table consists of three columns, which are `col1`, `col2`, and `col3` in sequence. Your data file `example13.parquet` also consists of three columns, which are mapped in sequence onto `col1`, `col2`, and `col3` of `table13`. If you want to load all data from `example13.parquet` into `table13`, run the following command: ```sql LOAD LABEL test_db.label13 ( DATA INFILE("hdfs://:/user/starrocks/data/input/example13.parquet") INTO TABLE table13 FORMAT AS "parquet" (col1, col2, col3) ) WITH BROKER ( "username" = "", "password" = "" ); ``` > **NOTE** > > By default, when you load Parquet data, StarRocks determines the data file format based on whether the filename contains the extension **.parquet**. If the filename does not contain the extension **.parquet**, you must use `FORMAT AS` to specify the data file format as `Parquet`. ##### Load ORC data[​](#load-orc-data "Direct link to Load ORC data") This section describes some parameters settings you need to take note of when you load ORC data. Your StarRocks database `test_db` contains a table named `table14`. The table consists of three columns, which are `col1`, `col2`, and `col3` in sequence. Your data file `example14.orc` also contains three columns, which are mapped in sequence onto `col1`, `col2`, and `col3` of `table14`. If you want to load all data from `example14.orc` into `table14`, run the following command: ```sql LOAD LABEL test_db.label14 ( DATA INFILE("hdfs://:/user/starrocks/data/input/example14.orc") INTO TABLE table14 FORMAT AS "orc" (col1, col2, col3) ) WITH BROKER ( "username" = "", "password" = "" ); ``` > **NOTE** > > * By default, when you load ORC data, StarRocks determines the data file format based on whether the filename contains the extension **.orc**. If the filename does not contain the extension **.orc**, you must use `FORMAT AS` to specify the data file format as `ORC`. > > * In StarRocks v2.3 and earlier, if the data file contains ARRAY-type columns, you must make sure that the columns of the ORC data file have the same names as their mapping columns in the StarRocks table and the columns cannot be specified in the SET clause. ##### Load JSON data[​](#load-json-data "Direct link to Load JSON data") This section describes the parameter settings that you need to take note of when you load JSON data. Your StarRocks database `test_db` contains a table named `tbl1`, whose schema is as follows: ```sql `category` varchar(512) NULL COMMENT "", `author` varchar(512) NULL COMMENT "", `title` varchar(512) NULL COMMENT "", `price` double NULL COMMENT "" ``` ###### Load JSON data using simple mode[​](#load-json-data-using-simple-mode "Direct link to Load JSON data using simple mode") Suppose that your data file `example1.json` consists of the following data: ```json {"category":"C++","author":"avc","title":"C++ primer","price":895} ``` To load all data from `example1.json` into `tbl1`, run the following command: ```sql LOAD LABEL test_db.label15 ( DATA INFILE("hdfs://:/user/starrocks/data/input/example1.csv") INTO TABLE tbl1 FORMAT AS "json" ) WITH BROKER ( "username" = "", "password" = "" ); ``` > **NOTE** > > In the preceding example, the parameters `columns` and `jsonpaths` are not specified. Therefore, the keys in `example1.json` are mapped by name onto the columns of `tbl1`. ###### Load JSON data using matched mode[​](#load-json-data-using-matched-mode "Direct link to Load JSON data using matched mode") StarRocks performs the following steps to match and process JSON data: 1. (Optional) Strips the outermost array structure as instructed by the `strip_outer_array` parameter setting. > **NOTE** > > This step is performed only when the outermost layer of the JSON data is an array structure as indicated by a pair of square brackets `[]`. You need to set `strip_outer_array` to `true`. 2. (Optional) Matches the root element of the JSON data as instructed by the `json_root` parameter setting. > **NOTE** > > This step is performed only when the JSON data has a root element. You need to specify the root element by using the `json_root` parameter. 3. Extracts the specified JSON data as instructed by the `jsonpaths` parameter setting. ###### Load JSON data using matched mode without root element specified[​](#load-json-data-using-matched-mode-without-root-element-specified "Direct link to Load JSON data using matched mode without root element specified") Suppose that your data file `example2.json` consists of the following data: ```json [ {"category":"xuxb111","author":"1avc","title":"SayingsoftheCentury","price":895}, {"category":"xuxb222","author":"2avc","title":"SayingsoftheCentury","price":895}, {"category":"xuxb333","author":"3avc","title":"SayingsoftheCentury","price":895} ] ``` To load only `category`, `author`, and `price` from `example2.json`, run the following command: ```sql LOAD LABEL test_db.label16 ( DATA INFILE("hdfs://:/user/starrocks/data/input/example2.csv") INTO TABLE tbl1 FORMAT AS "json" (category, price, author) ) WITH BROKER ( "username" = "", "password" = "" ) PROPERTIES ( "strip_outer_array" = "true", "jsonpaths" = "[\"$.category\",\"$.price\",\"$.author\"]" ); ``` > **NOTE** > > In the preceding example, the outermost layer of the JSON data is an array structure as indicated by a pair of square brackets `[]`. The array structure consists of multiple JSON objects that each represent a data record. Therefore, you need to set `strip_outer_array` to `true` to strip the outermost array structure. The key **title** that you do not want to load is ignored during loading. ###### Load JSON data using matched mode with root element specified[​](#load-json-data-using-matched-mode-with-root-element-specified "Direct link to Load JSON data using matched mode with root element specified") Suppose your data file `example3.json` consists of the following data: ```json { "id": 10001, "RECORDS":[ {"category":"11","title":"SayingsoftheCentury","price":895,"timestamp":1589191587}, {"category":"22","author":"2avc","price":895,"timestamp":1589191487}, {"category":"33","author":"3avc","title":"SayingsoftheCentury","timestamp":1589191387} ], "comments": ["3 records", "there will be 3 rows"] } ``` To load only `category`, `author`, and `price` from `example3.json`, run the following command: ```sql LOAD LABEL test_db.label17 ( DATA INFILE("hdfs://:/user/starrocks/data/input/example3.csv") INTO TABLE tbl1 FORMAT AS "json" (category, price, author) ) WITH BROKER ( "username" = "", "password" = "" ) PROPERTIES ( "json_root"="$.RECORDS", "strip_outer_array" = "true", "jsonpaths" = "[\"$.category\",\"$.price\",\"$.author\"]" ); ``` > **NOTE** > > In the preceding example, the outermost layer of the JSON data is an array structure as indicated by a pair of square brackets `[]`. The array structure consists of multiple JSON objects that each represent a data record. Therefore, you need to set `strip_outer_array` to `true` to strip the outermost array structure. The keys `title` and `timestamp` that you do not want to load are ignored during loading. Additionally, the `json_root` parameter is used to specify the root element, which is an array, of the JSON data. --- ### CANCEL LOAD CANCEL LOAD cancels a given load job: [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md), [Spark Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SPARK_LOAD.md), or [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). A load job in the `PREPARED`, `CANCELLED` or `FINISHED` state cannot be canceled. Canceling a load job is an asynchronous process. You can use the [SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md) statement to check whether a load job is successfully canceled. The load job is successfully canceled if the value of `State` is `CANCELLED` and the value of `type` (displayed in `ErrorMsg`) is `USER_CANCEL`. #### Syntax[​](#syntax "Direct link to Syntax") ```sql CANCEL LOAD [FROM db_name] WHERE LABEL = "label_name" ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | db\_name | No | The name of the database to which the load job belongs. If this parameter is not specified, a load job in your current database is canceled by default. | | label\_name | Yes | The label of the load job. | #### Examples[​](#examples "Direct link to Examples") Example 1: Cancel the load job whose label is `example_label` in the current database. ```sql CANCEL LOAD WHERE LABEL = "example_label"; ``` Example 2: Cancel the load job whose label is `example_label` in the `example_db` database. ```sql CANCEL LOAD FROM example_db WHERE LABEL = "example_label"; ``` --- ### ALTER TASK Modifies an asynchronous ETL task submitted using [SUBMIT TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md). This feature has been supported from v4.1 onwards. You can use this statement to: * Suspend a running task * Resume a suspended task * Update task properties #### Syntax[​](#syntax "Direct link to Syntax") ```sql ALTER TASK [IF EXISTS] { RESUME | SUSPEND | SET ('key' = 'value'[, ...]) } ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | IF EXISTS | No | If this parameter is specified, StarRocks will not throw an exception when modifying a task that does not exist. If this parameter is not specified, the system will throw an exception when modifying a task that does not exist. | | task\_name | Yes | The name of the task to modify. | | RESUME | No | Resumes a suspended task. The task will be scheduled according to its original schedule (for periodic tasks) or be available for manual execution (for manual tasks). | | SUSPEND | No | Suspends a running task. For periodic tasks, this stops the task scheduler and kills any running task runs. | | SET | No | Updates the properties of the task. The properties will be merged with existing properties and applied to subsequent task executions. | #### Usage notes[​](#usage-notes "Direct link to Usage notes") * You can check the information of asynchronous tasks by querying the metadata views `tasks` and `task_runs` in Information Schema. ```sql SELECT * FROM INFORMATION_SCHEMA.tasks; SELECT * FROM information_schema.tasks WHERE task_name = ''; SELECT * FROM information_schema.task_runs; SELECT * FROM information_schema.task_runs WHERE task_name = ''; ``` * The `SUSPEND` action stops the task scheduler for periodic tasks and kills any currently running task runs. The task state will be changed to `PAUSE`. * The `RESUME` action restarts the task scheduler for periodic tasks. The task state will be changed to `ACTIVE`. * The `SET` action updates task properties that will be applied to subsequent task executions. You can use `session.` prefix with session variables to change the task running connect context configurations. #### Examples[​](#examples "Direct link to Examples") Example 1: Suspend a task named `etl_task`: ```sql ALTER TASK etl_task SUSPEND; ``` Example 2: Resume a suspended task named `etl_task`: ```sql ALTER TASK etl_task RESUME; ``` Example 3: Update the query timeout for a task named `etl_task`: ```sql ALTER TASK etl_task SET ('session.query_timeout' = '5000'); ``` Example 4: Update multiple properties for a task: ```sql ALTER TASK etl_task SET ( 'session.query_timeout' = '5000', 'session.enable_profile' = 'true' ); ``` Example 5: Suspend a task only if it exists (avoid error if task does not exist): ```sql ALTER TASK IF EXISTS etl_task SUSPEND; ``` --- ### DROP TASK DROP TASK drops an asynchronous ETL task submitted using [SUBMIT TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md). This feature has been supported since StarRocks v2.5.7. > **NOTE** > > Dropping a task with DROP TASK simultaneously cancels the corresponding TaskRun. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DROP TASK [IF EXISTS] `` [FORCE] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | IF EXISTS | no | If this parameter is specified, StarRocks will not throw an exception when deleting a task that does not exist. If this parameter is not specified, the system will throw an exception when deleting a task that does not exist. | | task\_name | Yes | The name of the task to drop. Please wrap the task name with backticks (\`) to prevent any parse failure. | | FORCE | No | Forces to drop the task. | #### Usage notes[​](#usage-notes "Direct link to Usage notes") You can check the information of asynchronous tasks by querying the metadata views `tasks` and `task_runs` in Information Schema. ```sql SELECT * FROM INFORMATION_SCHEMA.tasks; SELECT * FROM information_schema.tasks WHERE task_name = ''; SELECT * FROM information_schema.task_runs; SELECT * FROM information_schema.task_runs WHERE task_name = ''; ``` You can obtain the `task_name` of materialized view refresh tasks by using the SHOW MATERIALIZED VIEWS statement. ```sql SHOW MATERIALIZED VIEWS; SHOW MATERIALIZED VIEWS WHERE name = ''; ``` #### Examples[​](#examples "Direct link to Examples") ```plain MySQL > SUBMIT /*+set_var(insert_timeout=100000)*/ TASK `ctas` AS -> CREATE TABLE insert_wiki_edit_new -> AS SELECT * FROM source_wiki_edit; +----------+-----------+ | TaskName | Status | +----------+-----------+ | ctas | SUBMITTED | +----------+-----------+ 1 row in set (1.19 sec) MySQL > DROP TASK `ctas`; Query OK, 0 rows affected (0.35 sec) ``` Drop a non-existing task * If the `IF EXISTS` parameter is not specified, dropping a non-existing task `test_task` will result in an error. ```plain MySQL > DROP TASK test_task; Query 1 ERROR: Getting analyzing error. Detail message: Task test_task is not exist. ``` * If the `IF EXISTS` parameter is specified, dropping a non-existing `test_task` will not result in an error. ```plain MySQL > DROP TASK IF EXISTS test_task; Query OK, 0 rows affected (0.00 sec) ``` --- ### SUBMIT TASK SUBMIT TASK submits an ETL statement as an asynchronous task. You can use this statement to: * execute long-running tasks in the background (supported from v2.5 onwards) * schedule a task at a regular interval (supported from v3.3 onwards) Supported statements include: * [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) (from v3.0 onwards) * [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) (from v3.0 onwards) * [CACHE SELECT](https://docs.starrocks.io/docs/data_source/data_cache/block_cache_warmup.md) (from v3.3 onwards) You can view the list of tasks by querying `INFORMATION_SCHEMA.tasks`, or view the execution history of tasks by querying `INFORMATION_SCHEMA.task_runs`. For more information, see [Usage Notes](#usage-notes). You can modify an asynchronous task using [ALTER TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/ALTER_TASK.md), or drop an asynchronous task using [DROP TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/DROP_TASK.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql SUBMIT TASK [SCHEDULE [START()] EVERY(INTERVAL ) ] [PROPERTIES(<"key" = "value"[, ...]>)] AS ``` #### PROPERTIES[​](#properties "Direct link to PROPERTIES") You can add the `session.` prefix before session variables to change the task running context configurations. | **Property** | **Type** | **Description** | | -------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `session.query_timeout` | Integer | The timeout duration of a query. Unit: Seconds. Value range: 1 to 259200. Default: `300`. From v3.4.0 onwards, this variable does not apply to INSERT operations. | | `session.insert_timeout` | Integer | The timeout duration of an INSERT operation. Unit: Seconds. Default: `14400`. Supported from v3.4.0 onwards. | | `session.enable_profile` | Boolean | Whether to enable query profiling for the task. Default: `false`. | | `session.new_planner_optimize_timeout` | Integer | The timeout duration of the query optimizer. Unit: Milliseconds. Default: `3000`. | For example, the following statement submits a task named `test_task` with session properties which enables query profile and increase query timeout: ```sql SUBMIT TASK test_task PROPERTIES ( "session.enable_profile" = "true", "session.insert_timeout" = "10000" ) AS insert into t2 select * from t1; ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------------ | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | task\_name | Yes | The name of the task. | | schedule\_start | No | The start time for the scheduled task. | | schedule\_interval | No | The interval at which the scheduled task is executed, with a minimum interval of 10 seconds. | | etl\_statement | Yes | The ETL statement that you want to submit as an asynchronous task. StarRocks currently supports submitting asynchronous tasks for [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) and [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). | #### Return value[​](#return-value "Direct link to Return value") * `TaskName`: The name of the task. * `Status`: The status of the task. Valid values: * `SUBMITTED`: The task has been submitted. * `REJECTED`: The task has been rejected. * `FAILED`: The task has failed. #### Usage notes[​](#usage-notes "Direct link to Usage notes") This statement creates a Task, which is a template for storing a task that executes the ETL statement. You can check the information of the Task by querying the metadata view [`tasks` in Information Schema](https://docs.starrocks.io/docs/sql-reference/information_schema/tasks.md). ```sql SELECT * FROM INFORMATION_SCHEMA.tasks; SELECT * FROM information_schema.tasks WHERE task_name = ''; ``` After you run the Task, a TaskRun is generated accordingly. A TaskRun indicates a task that executes the ETL statement. A TaskRun has the following states: * `PENDING`: The task waits to be run. * `RUNNING`: The task is running. * `FAILED`: The task failed. * `SUCCESS`: The task runs successfully. You can check the state of a TaskRun by querying the metadata view [`task_runs` in Information Schema](https://docs.starrocks.io/docs/sql-reference/information_schema/task_runs.md). ```sql SELECT * FROM INFORMATION_SCHEMA.task_runs; SELECT * FROM information_schema.task_runs WHERE task_name = ''; ``` #### Configure via FE configuration items[​](#configure-via-fe-configuration-items "Direct link to Configure via FE configuration items") You can configure asynchronous ETL tasks using the following FE configuration items: | **Parameter** | **Default value** | **Description** | | -------------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | task\_ttl\_second | 86400 | The period during which a Task is valid. Unit: seconds. Tasks that exceed the validity period are deleted. Here, a Task indicates a non-periodic task that is performed only once. | | task\_check\_interval\_second | 3600 | The time interval to delete invalid Tasks. Unit: seconds. | | task\_runs\_ttl\_second | 86400 | The period during which a TaskRun is valid. Unit: seconds. TaskRuns that exceed the validity period are deleted automatically. Additionally, TaskRuns in the `FAILED` and `SUCCESS` states are also deleted automatically. A `TaskRun` is an individual run of a periodic task. | | task\_runs\_concurrency | 4 | The maximum number of TaskRuns that can be run in parallel. | | task\_runs\_queue\_length | 500 | The maximum number of TaskRuns that are pending for running. If the number exceeds the default value, the incoming tasks will be suspended. | | task\_runs\_max\_history\_number | 10000 | The maximum number of TaskRun records to retain. | | task\_min\_schedule\_interval\_s | 10 | The minimum interval for Task execution. Unit: seconds. | #### Examples[​](#examples "Direct link to Examples") Example 1: Submit an asynchronous task for `CREATE TABLE tbl1 AS SELECT * FROM src_tbl`, and specify the task name as `etl0`: ```sql SUBMIT TASK etl0 AS CREATE TABLE tbl1 AS SELECT * FROM src_tbl; ``` Example 2: Submit an asynchronous task for `INSERT INTO tbl2 SELECT * FROM src_tbl`, and specify the task name as `etl1`: ```sql SUBMIT TASK etl1 AS INSERT INTO tbl2 SELECT * FROM src_tbl; ``` Example 3: Submit an asynchronous task for `INSERT OVERWRITE tbl3 SELECT * FROM src_tbl`: ```sql SUBMIT TASK AS INSERT OVERWRITE tbl3 SELECT * FROM src_tbl; ``` Example 4: Submit an asynchronous task for `INSERT OVERWRITE insert_wiki_edit SELECT * FROM source_wiki_edit` without specifying the task name, and extend the query timeout to `100000` seconds using the hint: ```sql SUBMIT /*+set_var(insert_timeout=100000)*/ TASK AS INSERT OVERWRITE insert_wiki_edit SELECT * FROM source_wiki_edit; ``` Example 5: Create an asynchronous task for an INSERT OVERWRITE statement. The task will be regularly executed at an interval of 1 minute. ```sql SUBMIT TASK SCHEDULE EVERY(INTERVAL 1 MINUTE) AS INSERT OVERWRITE insert_wiki_edit SELECT dt, user_id, count(*) FROM source_wiki_edit GROUP BY dt, user_id; ``` Example 6: Create a task with custom session properties: ```sql SUBMIT TASK test_task PROPERTIES ( "session.enable_profile" = "true", "session.insert_timeout" = "10000" ) AS insert into t2 select * from t1; ``` --- ### INSERT Inserts data into a specific table or overwrites a specific table with data. From v3.2.0 onwards, INSERT supports writing data into files in remote storage. You can use INSERT INTO FILES() to unload data from StarRocks to remote storage. You can submit an asynchronous INSERT task using [SUBMIT TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md). #### Syntax[​](#syntax "Direct link to Syntax") * **Data loading**: ```sql INSERT { INTO | OVERWRITE } [db_name.] [ PARTITION ( [, ...] ) ] [ TEMPORARY PARTITION ( [, ...] ) ] [ WITH LABEL ` | `` | | hour | `hour()` | `date_trunc('hour', )` | | day | `day()` | `date_trunc('day', )` | | month | `month()` | `date_trunc('month', )` | | year | `year()` | `date_trunc('year', )` | | bucket | `bucket(, )` | Not supported | | truncate | `truncate()` | Not supported | * For non-Iceberg partition columns, where partition expression computation is not involved, additional partition expression handling is not required. You can map them directly. See [Example -5](#examples) for detailed instructions on multi-column partition expressions. **order\_by\_expression** (optional) The sort key of the asynchronous materialized view. If you do not specify the sort key, StarRocks chooses some of the prefix columns from SELECT columns as the sort keys. For example, in `select a, b, c, d`, sort keys can be `a` and `b`. This parameter is supported from StarRocks v3.0 onwards. > **NOTE** There are two different uses of `ORDER BY` in materialized views: > > * `ORDER BY` in the CREATE MATERIALIZED VIEW statement defines the sort key of the materialized view, which helps accelerate queries based on the sort key. This does not affect the materialized view's SPJG-based transparent acceleration capability but does not guarantee global ordering of the materialized view's query results. > * `ORDER BY` in the materialized view's query definition guarantees global ordering of the query results, but prevents the materialized view from being used for SPJG-based transparent query rewrite. Therefore, `ORDER BY` should not be used in the materialized view's query definition if the MV is used for query rewrite usage. **INDEX** (optional) Asynchronous materialized views support ​Bitmap​ and ​BloomFilter​ indexes to accelerate query performance, and their usage is the same as in regular tables. For details on the use cases and information about ​Bitmap​ and ​BloomFilter​ indexes, please refer to：[Bitmap Index](https://docs.starrocks.io/docs/table_design/indexes/Bitmap_index.md) and [Bloom filter Index](https://docs.starrocks.io/docs/table_design/indexes/Bloomfilter_index.md). Using Bitmap Indexes: ```sql -- Create an index CREATE INDEX ON () USING BITMAP COMMENT ''; -- Check index creation progress SHOW ALTER TABLE COLUMN; -- View indexes SHOW INDEXES FROM ; -- Drop an index DROP INDEX ON ; ``` Using BloomFilter Indexes: ```sql -- Create an index ALTER MATERIALIZED VIEW SET ("bloom_filter_columns" = ""); -- View indexes SHOW CREATE MATERIALIZED VIEW ; -- Drop an index ALTER MATERIALIZED VIEW SET ("bloom_filter_columns" = ""); ``` ###### PROPERTIES (optional)[​](#properties-optional "Direct link to PROPERTIES (optional)") Properties of the asynchronous materialized view. You can modify the properties of an existing materialized view using [ALTER MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/ALTER_MATERIALIZED_VIEW.md). * `session.`: If you want to alter a session variable-related property of the materialized view, you must add a `session.` prefix to the property, for example, `session.insert_timeout`. You do not need to specify the prefix for non-session properties, for example, `mv_rewrite_staleness_second`. * `replication_num`: The number of materialized view replicas to create. * `storage_medium`: Storage medium type. Valid values: `HDD` and `SSD`. * `storage_cooldown_time`: the storage cooldown time for a partition. If both HDD and SSD storage mediums are used, data in the SSD storage is moved to the HDD storage after the time specified by this property. Format: "yyyy-MM-dd HH:mm:ss". The specified time must be later than the current time. If this property is not explicitly specified, the storage cooldown is not performed by default. * `bloom_filter_columns`: An array of column names that enable Bloom filter indexing. For details about Bloom filter indexes, see [Bloom filter Index](https://docs.starrocks.io/docs/table_design/indexes/Bloomfilter_index.md). * `partition_ttl`: The time-to-live (TTL) for partitions. Partitions whose data is within the specified time range are retained. Expired partitions are deleted automatically. Unit: `YEAR`, `MONTH`, `DAY`, `HOUR`, and `MINUTE`. For example, you can specify this property as `2 MONTH`. This property is recommended over `partition_ttl_number`. It is supported from v3.1.5 onwards. * `partition_ttl_number`: The number of most recent materialized view partitions to retain. For the partitions with a start time earlier than the current time, after the number of these partitions exceeds this value, less recent partitions will be deleted. StarRocks will periodically check materialized view partitions according to the time interval specified in the FE configuration item `dynamic_partition_check_interval_seconds`, and automatically delete expired partitions. If you enabled the [dynamic partitioning](https://docs.starrocks.io/docs/table_design/data_distribution/dynamic_partitioning.md) strategy, the partitions created in advance are not counted in. When the value is `-1`, all partitions of the materialized view will be preserved. Default: `-1`. * `partition_refresh_number`: In a single refresh, the maximum number of partitions to refresh. If the number of partitions to be refreshed exceeds this value, StarRocks will split the refresh task and complete it in batches. Only when the previous batch of partitions is refreshed successfully, StarRocks will continue to refresh the next batch of partitions until all partitions are refreshed. If any of the partitions fail to be refreshed, no subsequent refresh tasks will be generated. When the value is `-1`, the refresh task will not be split. The default value is changed from `-1` to `1` since v3.3, meaning StarRocks refeshes partitions one by one. * `partition_refresh_strategy`：The refresh strategy for a materialized view during a single refresh operation. When set to `adaptive`, the number of partitions to refresh will be automatically determined based on the data volume in the base table partitions, significantly improving refresh efficiency. If this property is not specified, the default strategy is `strict`, meaning the number of partitions refreshed in a single operation is strictly controlled by `partition_refresh_number`. Setting it to `force` refreshes every partition on each refresh; it has no effect on a materialized view whose `refresh_mode` is `INCREMENTAL`, which is maintained from base table changes by definition. * `excluded_trigger_tables`: If a base table of the materialized view is listed here, the automatic refresh task will not be triggered when the data in the base table is changed. This parameter only applies to load-triggered refresh strategy, and is usually used together with the property `auto_refresh_partitions_limit`. Format: `[db_name.]table_name`. When the value is an empty string, any data change in all base tables triggers the refresh of the corresponding materialized view. The default value is an empty string. * `excluded_refresh_tables`: The base tables listed in this property will not be updated to the materialized view when their data changes. Format: `[db_name.]table_name`. The default value is an empty string. When the value is an empty string, any base table data change will trigger the corresponding materialized view refresh. tip The difference between `excluded_trigger_tables` and `excluded_refresh_tables` is: * `excluded_trigger_tables` controls whether to trigger a refresh, not whether to participate in the refresh. For example, a partitioned materialized view is obtained by joining two partitioned tables A and B, and the partitions of the two tables A and B correspond one to one. `excluded_trigger_table` contains table A. During a period of time, table A updated partitions `[1,2,3]`, but because it is an `excluded_trigger_table`, the refresh of the materialized view is not triggered. At this time, table B updates partition `[3]`, and the materialized view triggers a refresh, which will refresh the three partitions `[1, 2, 3]`. Here you can see that `excluded_trigger_table` only controls whether to trigger a refresh. Although the update of table A cannot trigger a materialized view refresh, when the update of table B triggers a materialized view refresh, the partition updated by table A will also be added to the refresh task. * `excluded_refresh_tables` controls whether to participate in the refresh. In the above example, if table A exists in both `excluded_trigger_table` and `excluded_refresh_tables`, when the update of table B triggers a materialized view refresh, only partition `[3]` will be refreshed. * `auto_refresh_partitions_limit`: The number of most recent materialized view partitions that need to be refreshed when a materialized view refresh is triggered. You can use this property to limit the refresh range and reduce the refresh cost. However, because not all the partitions are refreshed, the data in the materialized view may not be consistent with the base table. Default: `-1`. When the value is `-1`, all partitions will be refreshed. When the value is a positive integer N, StarRocks sorts the existing partitions in chronological order, and refreshes the current partition and N-1 most recent partitions. If the number of partitions is less than N, StarRocks refreshes all existing partitions. If there are dynamic partitions created in advance in your materialized view, StarRocks refreshes all pre-created partitions. This property is not supported on a materialized view whose `refresh_mode` is `INCREMENTAL`, because incremental maintenance requires a complete baseline. To batch the initial full refresh of such a view, use `partition_refresh_number` instead; refreshes after that process only the base table changes and are not affected by either property. * `mv_rewrite_staleness_second`: This materialized view can be used directly for query rewrite, regardless of whether the base table data has changed, as long as the gap between the latest base-table modification time and the materialized view's last confirmed complete refresh does not exceed the time interval specified in this property. Because the comparison is against the latest base-table modification time (not the current wall-clock time), a materialized view whose base tables have not changed since its last complete refresh stays usable for query rewrite no matter how much time passes. Otherwise, StarRocks checks whether the base tables have been updated to determine whether the materialized view can be used for query rewrite. Note that a refresh covering only part of the partitions (for example, a manual `REFRESH ... PARTITION START ... END`, or a refresh limited by `auto_refresh_partitions_limit`) does not renew this freshness baseline. Unit: Second. This property is supported from v3.0. * `colocate_with`: The colocation group of the asynchronous materialized view. See [Colocate Join](https://docs.starrocks.io/docs/using_starrocks/Colocate_join.md) for further information. This property is supported from v3.0. * `unique_constraints` and `foreign_key_constraints`: The Unique Key constraints and Foreign Key constraints when you create an asynchronous materialized view for query rewrite in the View Delta Join scenario. See [Asynchronous materialized view - Rewrite queries in View Delta Join scenario](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md) for further information. This property is supported from v3.0. * `excluded_refresh_tables`：The base tables listed in this property will not trigger data refresh to the materialized view when their data changes. This property is usually used together with the `excluded_trigger_tables` property. Format: `[db_name.]table_name`. The default value is an empty string. When the value is an empty string, any data change in all base tables will trigger the corresponding materialized view refresh. > **CAUTION** > > The Unique Key constraints and Foreign Key constraints are only used for query rewrite. The Foreign Key constraint checks are not guaranteed when data is loaded into the table. You must ensure the data loaded into the table meets the constraints. * `resource_group`: The resource group to which the refresh tasks of the materialized view belong. The default value of this property is `default_mv_wg`, which is a system-defined resource group specifically used for materialized view refresh. The `cpu_core_limit` of `default_mv_wg` is `1`, `mem_limit` is `0.8`. For more about resource groups, see [Resource group](https://docs.starrocks.io/docs/administration/management/resource_management/resource_group.md). * `query_rewrite_consistency`: The query rewrite rule for the asynchronous materialized views. This property is supported from v3.2. Valid values: * `disable`: Disable automatic query rewrite of the asynchronous materialized view. * `checked` (Default value): Enable automatic query rewrite only when the materialized view meets the timeliness requirement, which means: * If `mv_rewrite_staleness_second` is not specified, the materialized view can be used for query rewrite only when its data is consistent with the data in all base tables. * If `mv_rewrite_staleness_second` is specified, the materialized view can be used for query rewrite when its last confirmed complete refresh is within the staleness time interval. * `loose`: Enable automatic query rewrite directly, and no consistency check is required. * `force_mv`: From v3.5.0 onwards, StarRocks materialized views support Common Partition Expression TTL. The `force_mv` semantic is specifically designed for this scenario. When this semantic is enabled: * If the materialized view does not have the `partition_retention_condition` property, it will always force the use of the materialized view for query rewrite, regardless of whether the base table has been updated. * If the materialized view has the `partition_retention_condition` property: * For partitions within the TTL range, query rewrite based on the materialized view is always available, regardless of whether the base table has been updated. * For partitions outside the TTL range, a Union compensation between the materialized view and the base table is required, regardless of whether the base table has been updated. For example, if the materialized view has the `partition_retention_condition` property defined and the partition for `20241131` has expired, but the base table data for `20241203` has been updated while the materialized view data for `20241203` has not been refreshed, the following applies when the `query_rewrite_consistency` property is set to `force_mv`: * The materialized view guarantees that queries against the partitions within the TTL range (for example, from `20241201` to `20241203`) defined in `partition_retention_condition` can always be transparently rewritten. * For queries against partitions outside the `partition_retention_condition` range, compensation will occur automatically based on the Union of the materialized view and the base table. See [Example 6](#examples) for detailed instructions on the `force_mv` semantic and `partition_retention_condition`. * `storage_volume`: The name of the storage volume used to store the asynchronous materialized view you want to create if you are using a shared-data cluster. This property is supported from v3.1 onwards. If this property is not specified, the default storage volume is used. Example: `"storage_volume" = "def_volume"`. * `force_external_table_query_rewrite`: Whether to enable query rewrite for external catalog-based materialized views. This property is supported from v3.2. Valid values: * `true`(Default value since v3.3): Enable query rewrite for external catalog-based materialized views. * `false`: Disable query rewrite for external catalog-based materialized views. Because strong data consistency is not guaranteed between base tables and external catalog-based materialized views, this feature is set to `false` by default. When this feature is enabled, the materialized view is used for query rewrite in accordance with the rule specified in `query_rewrite_consistency`. * `enable_query_rewrite`: Whether to use the materialized view for query rewrite. When there are many materialized views, query rewrite based on materialized views can impact the optimizer's time consumption. With this property, you can control whether the materialized view can be used for query rewrite. This feature is supported from v3.3.0 onwards. Valid values: * `default` (Default): The system will not perform semantic checks on the materialized view, but only the SPJG-type materialized views can be used for query rewrite. Note that if the text-based query rewrite is enabled, non-SPJG-type materialized views can also be used for query rewrite. * `true`: The system will perform semantic checks when creating or modifying the materialized view. If the materialized view is not eligible for query rewrite (that is, the definition of the materialized view is not an SPJG-type query), a failure will be returned. * `false`: The materialized view will not be used for query rewrite. * \[Preview] `transparent_mv_rewrite_mode`: Specifies the transparent rewrite mode for **queries directly against the materialized view**. This feature is supported from v3.3.0 onwards. Valid values: * `false` (Default, compatible with the behavior in earlier versions): Queries directly against the materialized view will not be rewritten, and are only returned with the existing data in the materialized view. Their query results may be different from those of queries based on the definition of the materialized view according to the refresh status (data consistency) of the materialized view. * `true`: Queries directly against the materialized view will be rewritten and returned with the most updated data, which is consistent with the result of the materialized view definition query. Please note that when the materialized view is inactive or does not support transparent query rewrite, these queries will be executed as the materialized view definition query. * `transparent_or_error`: Queries directly against the materialized view will be rewritten whenever they are eligible. If the materialized view is inactive or does not support transparent query rewrite, these queries will be returned with an error. * `transparent_or_default` Queries directly against the materialized view will be rewritten whenever they are eligible. If the materialized view is inactive or does not support transparent query rewrite, these queries will be returned with the existing data in the materialized view. * `partition_retention_condition`: From v3.5.0 onwards, StarRocks materialized views support Common Partition Expression TTL. This property is the expression that declares the partitions to be retained dynamically. Partitions that do not meet the condition in the expression will be dropped regularly. Example: `'partition_retention_condition' = 'dt >= CURRENT_DATE() - INTERVAL 3 MONTH'`. * The expression can only contain partition columns and constants. Non-partition columns are not supported. * Common Partition Expression applies to List partitions and Range partitions differently: * For materialized views with List partitions, StarRocks supports deleting partitions filtered by the Common Partition Expression. * For materialized views with Range partitions, StarRocks can only filter and delete partitions using the partition pruning capability of FE. Partitions correspond to predicates that are not supported by partition pruning cannot be filtered and deleted. See [Example 6](#examples) for detailed instructions on the `force_mv` semantic and `partition_retention_condition`. * `refresh_mode`: Controls how a materialized view is refreshed. Introduced in StarRocks v4.1, and only supported for Iceberg append-only tables. Valid values: * `PCT`: (Default) For partitioned materialized views, only the affected partition is refreshed when there is a data change, ensuring result consistency for that partition. For non-partitioned materialized views, any data change in the base table triggers a full refresh of the materialized view. * `INCREMENTAL`: Ensures that only incremental refreshes are performed. If the materialized view does not support incremental refresh based on its definition or encounters non-incremental data, creation or refresh will fail. **query\_statement** (required) The query statement to create the asynchronous materialized view. From v3.1.6 onwards, StarRocks supports creating asynchronous materialized views with Common Table Expression (CTE). ##### Query an asynchronous materialized view[​](#query-an-asynchronous-materialized-view "Direct link to Query an asynchronous materialized view") An asynchronous materialized view is a physical table. You can operate it as any regular table **except that you cannot directly load data into an asynchronous materialized view**. ##### Automatic query rewrite with asynchronous materialized view[​](#automatic-query-rewrite-with-asynchronous-materialized-view "Direct link to Automatic query rewrite with asynchronous materialized view") StarRocks v2.5 supports automatic and transparent query rewrite based on the SPJG-type asynchronous materialized views. The SPJG-type materialized views refer to materialized views whose plan only includes Scan, Filter, Project, and Aggregate types of operators. The SPJG-type materialized views query rewrite includes single table query rewrite, Join query rewrite, aggregation query rewrite, Union query rewrite and query rewrite based on nested materialized views. See [Asynchronous materialized view - Rewrite queries with the asynchronous materialized view](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md) for further information. ##### Supported data types[​](#supported-data-types "Direct link to Supported data types") * Asynchronous materialized views created based on the StarRocks default catalog support the following data types: * **Date**: DATE, DATETIME * **String**: CHAR, VARCHAR * **Numeric**: BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, DECIMAL, PERCENTILE * **Semi-structured**: ARRAY, JSON, MAP (from v3.1 onwards), STRUCT (from v3.1 onwards) * **Other**: BITMAP, HLL > **NOTE** > > BITMAP, HLL, and PERCENTILE have been supported since v2.4.5. * Asynchronous materialized views created based on the StarRocks external catalogs support the following data types: * Hive Catalog * **Numeric**: INT/INTEGER, BIGINT, DOUBLE, FLOAT, DECIMAL * **Date**: TIMESTAMP * **String**: STRING, VARCHAR, CHAR * **Semi-structured**: ARRAY * Hudi Catalog * **Numeric**: BOOLEAN, INT, LONG, FLOAT, DOUBLE, DECIMAL * **Date**: DATE, TimeMillis/TimeMicros, TimestampMillis/TimestampMicros * **String**: STRING * **Semi-structured**: ARRAY * Iceberg Catalog * **Numeric**: BOOLEAN, INT, LONG, FLOAT, DOUBLE, DECIMAL(P, S) * **Date**: DATE, TIME, TIMESTAMP * **String**: STRING, UUID, FIXED(L), BINARY * **Semi-structured**: LIST ##### Incremental Materialized View[​](#incremental-materialized-view "Direct link to Incremental Materialized View") StarRocks v4.1 introduced the `refresh_mode` parameter to control the refresh behavior of materialized views. You can specify `refresh_mode` when creating each materialized view. If `refresh_mode` is not set during materialized view creation, the system uses the default value `PCT`, governed by the `default_mv_refresh_mode` parameter (Default: `pct`, valid values: `pct`, `incremental`). Please note the following usage guidance: * There are restrictions when adjusting `refresh_mode`: * You cannot change legacy materialized views (for example, those of type `PCT`) to use `INCREMENTAL` refresh modes. To do so, you must rebuild the materialized view. * When modifying a materialized view from `INCREMENTAL` types, the system will check if incremental refresh is possible. If not, the operation fails. * Materialized views with `refresh_mode` set to `INCREMENTAL` do not support specifying partition refresh. An exception is thrown if you attempt a partition refresh. ###### Sort Key[​](#sort-key "Direct link to Sort Key") A materialized view whose `refresh_mode` is `INCREMENTAL` is a Primary Key table keyed by an internal row-id column, so `ORDER BY` defines a sort key of its own instead of becoming part of the primary key. * On a hash- or random-distributed materialized view, the sort key is the `ORDER BY` columns. * On a range-distributed materialized view, `ORDER BY` is not supported: the sort key defines the tablet boundaries and must equal the primary key, which is a column you cannot name. Add `DISTRIBUTED BY HASH(...)` if you need a sort key. * If you omit `ORDER BY`, the materialized view is sorted by its primary key. ###### Supported Incremental Operators[​](#supported-incremental-operators "Direct link to Supported Incremental Operators") Incremental refresh supports only append-only operations on base tables. If unsupported operations such as `UPDATE`, `MERGE`, or `OVERWRITE` are performed, the refresh of materialized views whose `refresh_mode` is set to `INCREMENTAL` will fail. The following operators are currently supported for incremental refresh: | Operator | Incremental Refresh Support | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Select | Supported | | From `
` | Supported only for Iceberg tables; not yet available for other table types. | | Filter | Supported | | Aggregate with Group By | Supported - Aggregation functions with `distinct` are not yet supported.
- Aggregation without GROUP BY is not supported yet.
- A GROUP BY or SELECT DISTINCT key of a complex type (ARRAY, MAP, STRUCT) is not supported; such a key is rejected at CREATE. | | Inner Join | Supported | | Union All | Supported | | Left/Right/Full Outer Join | Not supported yet | note While operators listed above generally support incremental refresh, certain operator combinations currently have limitations: * Incremental computation is supported for aggregation after Join and aggregation after UNION. * However, incremental computation is **not** supported when performing Join after aggregation or UNION ALL after aggregation. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The current version of StarRocks does not support creating multiple materialized views at the same time. A new materialized view can only be created when the one before is completed. * About synchronous materialized views: * Synchronous materialized views only support aggregate functions on a single column. Query statements in the form of `sum(a+b)` are not supported. * Synchronous materialized views support only one aggregate function for each column of the base table. Query statements such as `select sum(a), min(a) from table` are not supported. * When creating a synchronous materialized view with an aggregate function, you must specify the GROUP BY clause, and specify at least one GROUP BY column in SELECT. * Synchronous materialized views do not support clauses such as JOIN, and the HAVING clause of GROUP BY. * When using ALTER TABLE DROP COLUMN to drop a specific column in a base table, you must ensure that all synchronous materialized views of the base table do not contain the dropped column, otherwise, the drop operation will fail. Before you drop the column, you must first drop all synchronous materialized views that contain the column. * Creating too many synchronous materialized views for a table will affect the data load efficiency. When data is being loaded to the base table, the data in the synchronous materialized view and the base table will be updated synchronously. If a base table contains `n` synchronous materialized views, the efficiency of loading data into the base table is about the same as the efficiency of loading data into `n` tables. * About nested asynchronous materialized views: * The refresh strategy for each materialized view only applies to the corresponding materialized view. * Currently, StarRocks does not limit the number of nesting levels. In a production environment, we recommend that the number of nesting layers not exceed THREE. * About external catalog asynchronous materialized views: * External catalog materialized view only support asynchronous fixed-interval refresh and manual refresh. * Strict consistency is not guaranteed between the materialized view and the base tables in the external catalog. * Currently, building materialized views based on external resources is not supported. * Currently, StarRocks cannot perceive if the base table data in the external catalog has changed, so all partitions will be refreshed by default every time the base table is refreshed. You can manually refresh only some of partitions using [REFRESH MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/REFRESH_MATERIALIZED_VIEW.md). #### Examples[​](#examples "Direct link to Examples") ##### Examples of synchronous materialized views[​](#examples-of-synchronous-materialized-views "Direct link to Examples of synchronous materialized views") The schema of the base table is as follows: ```plain mysql> desc duplicate_table; +-------+--------+------+------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+--------+------+------+---------+-------+ | k1 | INT | Yes | true | N/A | | | k2 | INT | Yes | true | N/A | | | k3 | BIGINT | Yes | true | N/A | | | k4 | BIGINT | Yes | true | N/A | | +-------+--------+------+------+---------+-------+ ``` Example 1: Create a synchronous materialized view that only contains the columns of the original table (k1, k2). ```sql create materialized view k1_k2 as select k1, k2 from duplicate_table; ``` The materialized view contains only two columns k1 and k2 without any aggregation. ```plain +-----------------+-------+--------+------+------+---------+-------+ | IndexName | Field | Type | Null | Key | Default | Extra | +-----------------+-------+--------+------+------+---------+-------+ | k1_k2 | k1 | INT | Yes | true | N/A | | | | k2 | INT | Yes | true | N/A | | +-----------------+-------+--------+------+------+---------+-------+ ``` Example 2: Create a synchronous materialized view sorted by k2. ```sql create materialized view k2_order as select k2, k1 from duplicate_table order by k2; ``` The materialized view's schema is shown below. The materialized view contains only two columns k2 and k1, where column k2 is a sort column without any aggregation. ```plain +-----------------+-------+--------+------+-------+---------+-------+ | IndexName | Field | Type | Null | Key | Default | Extra | +-----------------+-------+--------+------+-------+---------+-------+ | k2_order | k2 | INT | Yes | true | N/A | | | | k1 | INT | Yes | false | N/A | NONE | +-----------------+-------+--------+------+-------+---------+-------+ ``` Example 3: Create a synchronous materialized view grouped by k1 and k2, and a SUM aggregation on k3. ```sql create materialized view k1_k2_sumk3 as select k1, k2, sum(k3) from duplicate_table group by k1, k2; ``` The materialized view's schema is shown below. The materialized view contains three columns k1, k2 and sum (k3), where k1, k2 are grouped columns, and sum (k3) is the sum of the k3 columns grouped according to k1 and k2. ```plain +-----------------+-------+--------+------+-------+---------+-------+ | IndexName | Field | Type | Null | Key | Default | Extra | +-----------------+-------+--------+------+-------+---------+-------+ | k1_k2_sumk3 | k1 | INT | Yes | true | N/A | | | | k2 | INT | Yes | true | N/A | | | | k3 | BIGINT | Yes | false | N/A | SUM | +-----------------+-------+--------+------+-------+---------+-------+ ``` Because the materialized view does not declare a sort column, and it adopts an aggregation function, StarRocks supplements the grouped columns k1 and k2 by default. Example 4: Create a synchronous materialized view to remove duplicate rows. ```sql create materialized view deduplicate as select k1, k2, k3, k4 from duplicate_table group by k1, k2, k3, k4; ``` The materialized view's schema is shown below. The materialized view contains k1, k2, k3, and k4 columns, and there are no duplicate rows. ```plain +-----------------+-------+--------+------+-------+---------+-------+ | IndexName | Field | Type | Null | Key | Default | Extra | +-----------------+-------+--------+------+-------+---------+-------+ | deduplicate | k1 | INT | Yes | true | N/A | | | | k2 | INT | Yes | true | N/A | | | | k3 | BIGINT | Yes | true | N/A | | | | k4 | BIGINT | Yes | true | N/A | | +-----------------+-------+--------+------+-------+---------+-------+ ``` Example 5: Create a non-aggregated synchronous materialized view that does not declare a sort column. The schema of the base table is shown below: ```plain +-------+--------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+--------------+------+-------+---------+-------+ | k1 | TINYINT | Yes | true | N/A | | | k2 | SMALLINT | Yes | true | N/A | | | k3 | INT | Yes | true | N/A | | | k4 | BIGINT | Yes | true | N/A | | | k5 | DECIMAL(9,0) | Yes | true | N/A | | | k6 | DOUBLE | Yes | false | N/A | NONE | | k7 | VARCHAR(20) | Yes | false | N/A | NONE | +-------+--------------+------+-------+---------+-------+ ``` The materialized view contains k3, k4, k5, k6, and k7 columns, and no sort column is declared. Create the materialized view with the following statement: ```sql create materialized view mv_1 as select k3, k4, k5, k6, k7 from all_type_table; ``` StarRocks automatically uses k3, k4, and k5 as the sort columns by default. The sum of the bytes occupied by these three column types is 4 (INT) + 8 (BIGINT) + 16 (DECIMAL) = 28 < 36. So these three columns are added as sort columns. The materialized view's schema is as follows. ```plain +----------------+-------+--------------+------+-------+---------+-------+ | IndexName | Field | Type | Null | Key | Default | Extra | +----------------+-------+--------------+------+-------+---------+-------+ | mv_1 | k3 | INT | Yes | true | N/A | | | | k4 | BIGINT | Yes | true | N/A | | | | k5 | DECIMAL(9,0) | Yes | true | N/A | | | | k6 | DOUBLE | Yes | false | N/A | NONE | | | k7 | VARCHAR(20) | Yes | false | N/A | NONE | +----------------+-------+--------------+------+-------+---------+-------+ ``` It can be observed that the `key` field of the k3, k4, and k5 columns is `true`, which indicates that they are the sort keys. The key field of the k6, and k7 columns is `false`, which indicates that they are not the sort keys. Example 6: Create a synchronous materialized view that contains the WHERE clause and complex expressions. ```sql -- Create the base table: user_event CREATE TABLE user_event ( ds date NOT NULL, id varchar(256) NOT NULL, user_id int DEFAULT NULL, user_id1 varchar(256) DEFAULT NULL, user_id2 varchar(256) DEFAULT NULL, column_01 int DEFAULT NULL, column_02 int DEFAULT NULL, column_03 int DEFAULT NULL, column_04 int DEFAULT NULL, column_05 int DEFAULT NULL, column_06 DECIMAL(12,2) DEFAULT NULL, column_07 DECIMAL(12,3) DEFAULT NULL, column_08 JSON DEFAULT NULL, column_09 DATETIME DEFAULT NULL, column_10 DATETIME DEFAULT NULL, column_11 DATE DEFAULT NULL, column_12 varchar(256) DEFAULT NULL, column_13 varchar(256) DEFAULT NULL, column_14 varchar(256) DEFAULT NULL, column_15 varchar(256) DEFAULT NULL, column_16 varchar(256) DEFAULT NULL, column_17 varchar(256) DEFAULT NULL, column_18 varchar(256) DEFAULT NULL, column_19 varchar(256) DEFAULT NULL, column_20 varchar(256) DEFAULT NULL, column_21 varchar(256) DEFAULT NULL, column_22 varchar(256) DEFAULT NULL, column_23 varchar(256) DEFAULT NULL, column_24 varchar(256) DEFAULT NULL, column_25 varchar(256) DEFAULT NULL, column_26 varchar(256) DEFAULT NULL, column_27 varchar(256) DEFAULT NULL, column_28 varchar(256) DEFAULT NULL, column_29 varchar(256) DEFAULT NULL, column_30 varchar(256) DEFAULT NULL, column_31 varchar(256) DEFAULT NULL, column_32 varchar(256) DEFAULT NULL, column_33 varchar(256) DEFAULT NULL, column_34 varchar(256) DEFAULT NULL, column_35 varchar(256) DEFAULT NULL, column_36 varchar(256) DEFAULT NULL, column_37 varchar(256) DEFAULT NULL ) PARTITION BY date_trunc("day", ds) DISTRIBUTED BY hash(id); -- Create the materialized view with the WHERE clause and complex expresssions. CREATE MATERIALIZED VIEW test_mv1 AS SELECT ds, column_19, column_36, sum(column_01) as column_01_sum, bitmap_union(to_bitmap( user_id)) as user_id_dist_cnt, bitmap_union(to_bitmap(case when column_01 > 1 and column_34 IN ('1','34') then user_id2 else null end)) as filter_dist_cnt_1, bitmap_union(to_bitmap( case when column_02 > 60 and column_35 IN ('11','13') then user_id2 else null end)) as filter_dist_cnt_2, bitmap_union(to_bitmap(case when column_03 > 70 and column_36 IN ('21','23') then user_id2 else null end)) as filter_dist_cnt_3, bitmap_union(to_bitmap(case when column_04 > 20 and column_27 IN ('31','27') then user_id2 else null end)) as filter_dist_cnt_4, bitmap_union(to_bitmap( case when column_05 > 90 and column_28 IN ('41','43') then user_id2 else null end)) as filter_dist_cnt_5 FROM user_event WHERE ds >= '2023-11-02' GROUP BY ds, column_19, column_36; ``` ##### Examples of asynchronous materialized views[​](#examples-of-asynchronous-materialized-views "Direct link to Examples of asynchronous materialized views") The following examples are based on the base tables below: ```sql CREATE TABLE `lineorder` ( `lo_orderkey` int(11) NOT NULL COMMENT "", `lo_linenumber` int(11) NOT NULL COMMENT "", `lo_custkey` int(11) NOT NULL COMMENT "", `lo_partkey` int(11) NOT NULL COMMENT "", `lo_suppkey` int(11) NOT NULL COMMENT "", `lo_orderdate` int(11) NOT NULL COMMENT "", `lo_orderpriority` varchar(16) NOT NULL COMMENT "", `lo_shippriority` int(11) NOT NULL COMMENT "", `lo_quantity` int(11) NOT NULL COMMENT "", `lo_extendedprice` int(11) NOT NULL COMMENT "", `lo_ordtotalprice` int(11) NOT NULL COMMENT "", `lo_discount` int(11) NOT NULL COMMENT "", `lo_revenue` int(11) NOT NULL COMMENT "", `lo_supplycost` int(11) NOT NULL COMMENT "", `lo_tax` int(11) NOT NULL COMMENT "", `lo_commitdate` int(11) NOT NULL COMMENT "", `lo_shipmode` varchar(11) NOT NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`lo_orderkey`) COMMENT "OLAP" PARTITION BY RANGE(`lo_orderdate`) (PARTITION p1 VALUES [("-2147483648"), ("19930101")), PARTITION p2 VALUES [("19930101"), ("19940101")), PARTITION p3 VALUES [("19940101"), ("19950101")), PARTITION p4 VALUES [("19950101"), ("19960101")), PARTITION p5 VALUES [("19960101"), ("19970101")), PARTITION p6 VALUES [("19970101"), ("19980101")), PARTITION p7 VALUES [("19980101"), ("19990101"))) DISTRIBUTED BY HASH(`lo_orderkey`); CREATE TABLE IF NOT EXISTS `customer` ( `c_custkey` int(11) NOT NULL COMMENT "", `c_name` varchar(26) NOT NULL COMMENT "", `c_address` varchar(41) NOT NULL COMMENT "", `c_city` varchar(11) NOT NULL COMMENT "", `c_nation` varchar(16) NOT NULL COMMENT "", `c_region` varchar(13) NOT NULL COMMENT "", `c_phone` varchar(16) NOT NULL COMMENT "", `c_mktsegment` varchar(11) NOT NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`c_custkey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`c_custkey`); CREATE TABLE IF NOT EXISTS `dates` ( `d_datekey` int(11) NOT NULL COMMENT "", `d_date` varchar(20) NOT NULL COMMENT "", `d_dayofweek` varchar(10) NOT NULL COMMENT "", `d_month` varchar(11) NOT NULL COMMENT "", `d_year` int(11) NOT NULL COMMENT "", `d_yearmonthnum` int(11) NOT NULL COMMENT "", `d_yearmonth` varchar(9) NOT NULL COMMENT "", `d_daynuminweek` int(11) NOT NULL COMMENT "", `d_daynuminmonth` int(11) NOT NULL COMMENT "", `d_daynuminyear` int(11) NOT NULL COMMENT "", `d_monthnuminyear` int(11) NOT NULL COMMENT "", `d_weeknuminyear` int(11) NOT NULL COMMENT "", `d_sellingseason` varchar(14) NOT NULL COMMENT "", `d_lastdayinweekfl` int(11) NOT NULL COMMENT "", `d_lastdayinmonthfl` int(11) NOT NULL COMMENT "", `d_holidayfl` int(11) NOT NULL COMMENT "", `d_weekdayfl` int(11) NOT NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`d_datekey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`d_datekey`); CREATE TABLE IF NOT EXISTS `supplier` ( `s_suppkey` int(11) NOT NULL COMMENT "", `s_name` varchar(26) NOT NULL COMMENT "", `s_address` varchar(26) NOT NULL COMMENT "", `s_city` varchar(11) NOT NULL COMMENT "", `s_nation` varchar(16) NOT NULL COMMENT "", `s_region` varchar(13) NOT NULL COMMENT "", `s_phone` varchar(16) NOT NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`s_suppkey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`s_suppkey`); CREATE TABLE IF NOT EXISTS `part` ( `p_partkey` int(11) NOT NULL COMMENT "", `p_name` varchar(23) NOT NULL COMMENT "", `p_mfgr` varchar(7) NOT NULL COMMENT "", `p_category` varchar(8) NOT NULL COMMENT "", `p_brand` varchar(10) NOT NULL COMMENT "", `p_color` varchar(12) NOT NULL COMMENT "", `p_type` varchar(26) NOT NULL COMMENT "", `p_size` int(11) NOT NULL COMMENT "", `p_container` varchar(11) NOT NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`p_partkey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`p_partkey`); create table orders ( dt date NOT NULL, order_id bigint NOT NULL, user_id int NOT NULL, merchant_id int NOT NULL, good_id int NOT NULL, good_name string NOT NULL, price int NOT NULL, cnt int NOT NULL, revenue int NOT NULL, state tinyint NOT NULL ) PRIMARY KEY (dt, order_id) PARTITION BY RANGE(`dt`) ( PARTITION p20210820 VALUES [('2021-08-20'), ('2021-08-21')), PARTITION p20210821 VALUES [('2021-08-21'), ('2021-08-22')) ) DISTRIBUTED BY HASH(order_id) PROPERTIES ( "replication_num" = "3", "enable_persistent_index" = "true" ); ``` Example 1: Create a non-partitioned materialized view. ```sql -- create an unpartitioned materialized view sorted by lo_custkey CREATE MATERIALIZED VIEW lo_mv1 DISTRIBUTED BY HASH(`lo_orderkey`) ORDER BY (`lo_custkey`) REFRESH ASYNC AS select lo_orderkey, lo_custkey, sum(lo_quantity) as total_quantity, sum(lo_revenue) as total_revenue, count(lo_shipmode) as shipmode_count from lineorder group by lo_orderkey, lo_custkey; ``` Example 2: Create a partitioned materialized view. ```sql -- create a partitioned materialized view partitioned by `lo_orderdate` and sorted by `lo_custkey`. CREATE MATERIALIZED VIEW lo_mv2 PARTITION BY `lo_orderdate` DISTRIBUTED BY HASH(`lo_orderkey`) ORDER BY (`lo_custkey`) REFRESH ASYNC START('2023-07-01 10:00:00') EVERY (interval 1 day) AS select lo_orderkey, lo_orderdate, lo_custkey, sum(lo_quantity) as total_quantity, sum(lo_revenue) as total_revenue, count(lo_shipmode) as shipmode_count from lineorder group by lo_orderkey, lo_orderdate, lo_custkey; -- Use the date_trunc() function to partition the materialized view by month. CREATE MATERIALIZED VIEW order_mv1 PARTITION BY date_trunc('month', `dt`) DISTRIBUTED BY HASH(`order_id`) REFRESH ASYNC START('2023-07-01 10:00:00') EVERY (interval 1 day) AS select dt, order_id, user_id, sum(cnt) as total_cnt, sum(revenue) as total_revenue, count(state) as state_count from orders group by dt, order_id, user_id; ``` Example 3: Create an asynchronous materialized view. ```sql CREATE MATERIALIZED VIEW flat_lineorder DISTRIBUTED BY HASH(`lo_orderkey`) REFRESH MANUAL AS SELECT l.LO_ORDERKEY AS LO_ORDERKEY, l.LO_LINENUMBER AS LO_LINENUMBER, l.LO_CUSTKEY AS LO_CUSTKEY, l.LO_PARTKEY AS LO_PARTKEY, l.LO_SUPPKEY AS LO_SUPPKEY, l.LO_ORDERDATE AS LO_ORDERDATE, l.LO_ORDERPRIORITY AS LO_ORDERPRIORITY, l.LO_SHIPPRIORITY AS LO_SHIPPRIORITY, l.LO_QUANTITY AS LO_QUANTITY, l.LO_EXTENDEDPRICE AS LO_EXTENDEDPRICE, l.LO_ORDTOTALPRICE AS LO_ORDTOTALPRICE, l.LO_DISCOUNT AS LO_DISCOUNT, l.LO_REVENUE AS LO_REVENUE, l.LO_SUPPLYCOST AS LO_SUPPLYCOST, l.LO_TAX AS LO_TAX, l.LO_COMMITDATE AS LO_COMMITDATE, l.LO_SHIPMODE AS LO_SHIPMODE, c.C_NAME AS C_NAME, c.C_ADDRESS AS C_ADDRESS, c.C_CITY AS C_CITY, c.C_NATION AS C_NATION, c.C_REGION AS C_REGION, c.C_PHONE AS C_PHONE, c.C_MKTSEGMENT AS C_MKTSEGMENT, s.S_NAME AS S_NAME, s.S_ADDRESS AS S_ADDRESS, s.S_CITY AS S_CITY, s.S_NATION AS S_NATION, s.S_REGION AS S_REGION, s.S_PHONE AS S_PHONE, p.P_NAME AS P_NAME, p.P_MFGR AS P_MFGR, p.P_CATEGORY AS P_CATEGORY, p.P_BRAND AS P_BRAND, p.P_COLOR AS P_COLOR, p.P_TYPE AS P_TYPE, p.P_SIZE AS P_SIZE, p.P_CONTAINER AS P_CONTAINER FROM lineorder AS l INNER JOIN customer AS c ON c.C_CUSTKEY = l.LO_CUSTKEY INNER JOIN supplier AS s ON s.S_SUPPKEY = l.LO_SUPPKEY INNER JOIN part AS p ON p.P_PARTKEY = l.LO_PARTKEY; ``` Example 4: Create a partitioned materialized view and use `str2date` to transform the STRING type partition key of the base table into the date type as for the materialized view. ```sql -- Hive Table with string partition column. CREATE TABLE `part_dates` ( `d_date` varchar(20) DEFAULT NULL, `d_dayofweek` varchar(10) DEFAULT NULL, `d_month` varchar(11) DEFAULT NULL, `d_year` int(11) DEFAULT NULL, `d_yearmonthnum` int(11) DEFAULT NULL, `d_yearmonth` varchar(9) DEFAULT NULL, `d_daynuminweek` int(11) DEFAULT NULL, `d_daynuminmonth` int(11) DEFAULT NULL, `d_daynuminyear` int(11) DEFAULT NULL, `d_monthnuminyear` int(11) DEFAULT NULL, `d_weeknuminyear` int(11) DEFAULT NULL, `d_sellingseason` varchar(14) DEFAULT NULL, `d_lastdayinweekfl` int(11) DEFAULT NULL, `d_lastdayinmonthfl` int(11) DEFAULT NULL, `d_holidayfl` int(11) DEFAULT NULL, `d_weekdayfl` int(11) DEFAULT NULL, `d_datekey` varchar(11) DEFAULT NULL ) partition by (d_datekey); -- Create the materialied view with `str2date`. CREATE MATERIALIZED VIEW IF NOT EXISTS `test_mv` PARTITION BY str2date(`d_datekey`,'%Y%m%d') DISTRIBUTED BY HASH(`d_date`, `d_month`, `d_month`) REFRESH MANUAL AS SELECT `d_date` , `d_dayofweek`, `d_month` , `d_yearmonthnum` , `d_yearmonth` , `d_daynuminweek`, `d_daynuminmonth`, `d_daynuminyear` , `d_monthnuminyear` , `d_weeknuminyear` , `d_sellingseason`, `d_lastdayinweekfl`, `d_lastdayinmonthfl`, `d_holidayfl` , `d_weekdayfl`, `d_datekey` FROM `hive_catalog`.`ssb_1g_orc`.`part_dates` ; ``` Example 5: Create a partitioned materialized view with multi-column partition expression upon the base table from Iceberg Catalog (Spark). The definition of the base table in Spark: ```sql -- The partition expression of the base table contains multiple columns and a `days` transform. CREATE TABLE lineitem_days ( l_orderkey BIGINT, l_partkey INT, l_suppkey INT, l_linenumber INT, l_quantity DECIMAL(15, 2), l_extendedprice DECIMAL(15, 2), l_discount DECIMAL(15, 2), l_tax DECIMAL(15, 2), l_returnflag VARCHAR(1), l_linestatus VARCHAR(1), l_shipdate TIMESTAMP, l_commitdate TIMESTAMP, l_receiptdate TIMESTAMP, l_shipinstruct VARCHAR(25), l_shipmode VARCHAR(10), l_comment VARCHAR(44) ) USING ICEBERG PARTITIONED BY (l_returnflag, l_linestatus, days(l_shipdate)); ``` Create a materialized view with its partition columns mapped one-to-one with those of the base table: ```sql CREATE MATERIALIZED VIEW test_days PARTITION BY (l_returnflag, l_linestatus, date_trunc('day', l_shipdate)) REFRESH DEFERRED MANUAL AS SELECT * FROM iceberg_catalog.test_db.lineitem_days; ``` Example 6: Create a partitioned materialized view, define Common Partition Expression TTL for it, and enable the `force_mv` semantic for query rewrite. ```sql CREATE MATERIALIZED VIEW test_mv1 PARTITION BY (dt, province) REFRESH MANUAL PROPERTIES ( "partition_retention_condition" = "dt >= CURRENT_DATE() - INTERVAL 3 MONTH", "query_rewrite_consistency" = "force_mv" ) AS SELECT * from t1; ``` Example 7: Create a partition materialized view with a specific sort key: ```sql CREATE MATERIALIZED VIEW lo_mv2 PARTITION BY `lo_orderdate` DISTRIBUTED BY HASH(`lo_orderkey`) ORDER BY (`lo_custkey`) REFRESH ASYNC START('2023-07-01 10:00:00') EVERY (interval 1 day) AS select lo_orderkey, lo_orderdate, lo_custkey, sum(lo_quantity) as total_quantity, sum(lo_revenue) as total_revenue, count(lo_shipmode) as shipmode_count from lineorder group by lo_orderkey, lo_orderdate, lo_custkey; ``` Example 8: Create an incremental materialized view. ```sql CREATE MATERIALIZED VIEW test_mv1 PARTITION BY dt REFRESH DEFERRED MANUAL properties ( "refresh_mode" = "INCREMENTAL" ) AS SELECT t1.dt, t1.col1 as col11, t2.col1 as col21, t3.col1 as col31, t4.col1 as col41, t5.col1 as col51, sum(t1.col2) as col12, sum(t2.col2) as col22, sum(t3.col2) as col32, sum(t4.col2) as col42, sum(t5.col2) as col52, avg(t1.col2) as col13, avg(t2.col2) as col23, avg(t3.col2) as col33, avg(t4.col2) as col43, avg(t5.col2) as col53, min(t1.col2) as col14, min(t2.col2) as col24, min(t3.col2) as col34, min(t4.col2) as col44, min(t5.col2) as col54, max(t1.col2) as col15, max(t2.col2) as col25, max(t3.col2) as col35, max(t4.col2) as col45, max(t5.col2) as col55, count(t1.col2) as col16, count(t2.col2) as col26, count(t3.col2) as col36, count(t4.col2) as col46, count(t5.col2) as col56, approx_count_distinct(t1.col2) as col17, approx_count_distinct(t2.col2) as col27, approx_count_distinct(t3.col2) as col37, approx_count_distinct(t4.col2) as col47, approx_count_distinct(t5.col2) as col57 FROM iceberg_catalog.iceberg_test_dbt1 JOIN iceberg_catalog.iceberg_test_dbt2 ON t1.dt = t2.dt JOIN iceberg_catalog.iceberg_test_dbt3 ON t1.dt = t3.dt JOIN iceberg_catalog.iceberg_test_dbt4 ON t1.dt = t4.dt JOIN iceberg_catalog.iceberg_test_dbt5 ON t1.dt = t5.dt GROUP BY t1.dt, t1.col1, t2.col1, t3.col1, t4.col1, t5.col1; REFRESH MATERIALIZED VIEW test_mv1 WITH SYNC MODE; ``` The `refreshMode` field has been added to the `EXTRA_MESSAGE` column in `information_schema.task_runs` to indicate the refresh mode of the `TaskRun`. For more details, see [materialized\_view\_task\_run\_details](https://docs.starrocks.io/docs/using_starrocks/async_mv/materialized_view_task_run_details.md). ```sql mysql> select * from information_schema.task_runs order by CREATE_TIME desc limit 1\G; QUERY_ID: 0199f00e-2152-70a8-83da-26d6a8321ac6 TASK_NAME: mv-78190 CREATE_TIME: 2025-10-17 10:44:41 FINISH_TIME: 2025-10-17 10:44:44 STATE: SUCCESS CATALOG: NULL DATABASE: test_mv_async_db_621c29ff_ab02_11f0_9e41_00163e09349d DEFINITION: insert overwrite `test_mv_case_iceberg_transform_day_44` SELECT `t1`.`id`, `t1`.`v1`, `t1`.`v2`, `t1`.`dt` FROM `iceberg_catalog_621c2b62_ab02_11f0_a703_00163e09349d`.`iceberg_db_621c2bc9_ab02_11f0_885d_00163e09349d`.`t1` WHERE (`t1`.`id` > 1) AND (`t1`.`dt` >= '2025-06-01') EXPIRE_TIME: 2025-10-24 10:44:41 ERROR_CODE: 0 ERROR_MESSAGE: NULL PROGRESS: 100% EXTRA_MESSAGE: {"forceRefresh":false,"mvPartitionsToRefresh":["p20250718000000","p20250715000000","p20250721000000","p20250615000000","p20250618000000","p20250524000000","p20250621000000","p20250518000000"],"refBasePartitionsToRefreshMap":{"t1":["p20250718000000","p20250721000000","p20250618000000","p20250524000000","p20250621000000","p20250518000000","p20250715000000","p20250615000000","pNULL","p20250521000000","p20250624000000","p20250724000000","p20250515000000"]},"basePartitionsToRefreshMap":{},"processStartTime":1760669082430,"executeOption":{"priority":80,"taskRunProperties":{"FORCE":"false","mvId":"78190","warehouse":"default_warehouse"},"isMergeRedundant":false,"isManual":true,"isSync":true,"isReplay":false},"planBuilderMessage":{},"refreshMode":"INCREMENTAL"} PROPERTIES: {"FORCE":"false","mvId":"78190","warehouse":"default_warehouse"} JOB_ID: 0199f00e-2152-76b0-987c-76a9a19e77f9 ``` --- ### DROP MATERIALIZED VIEW DROP MATERIALIZED VIEW drops a materialized view. You cannot drop a synchronous materialized view that is being created in process with this command unless you use the **FORCE** option. With **FORCE**, StarRocks cancels any stuck sync MV build jobs, restores the base table state to NORMAL, and then drops the materialized view. Without FORCE, see [Synchronous materialized View - Drop an unfinished materialized view](https://docs.starrocks.io/docs/using_starrocks/Materialized_view-single_table.md#drop-an-unfinished-synchronous-materialized-view) for further instructions. tip This operation requires the DROP privilege on the target materialized view. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DROP MATERIALIZED VIEW [IF EXISTS] [database.]mv_name [FORCE] ``` Parameters in brackets \[] are optional. #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | IF EXISTS | no | If this parameter is specified, StarRocks will not throw an exception when deleting a materialized view that does not exist. If this parameter is not specified, the system will throw an exception when deleting a materialized view that does not exist. | | mv\_name | yes | The name of the materialized view to delete. | | FORCE | no | When the base table is not in NORMAL state (e.g., a sync MV build is stuck), FORCE cancels the related alter jobs, sets the table to NORMAL, and then drops the materialized view. Omit FORCE for normal drops. | #### Examples[​](#examples "Direct link to Examples") Example 1: Drop an existing materialized view 1. View all existing materialized views in the database. ```plain MySQL > SHOW MATERIALIZED VIEWS\G *************************** 1. row *************************** id: 470740 name: order_mv1 database_name: default_cluster:sr_hub text: SELECT `sr_hub`.`orders`.`dt` AS `dt`, `sr_hub`.`orders`.`order_id` AS `order_id`, `sr_hub`.`orders`.`user_id` AS `user_id`, sum(`sr_hub`.`orders`.`cnt`) AS `total_cnt`, sum(`sr_hub`.`orders`.`revenue`) AS `total_revenue`, count(`sr_hub`.`orders`.`state`) AS `state_count` FROM `sr_hub`.`orders` GROUP BY `sr_hub`.`orders`.`dt`, `sr_hub`.`orders`.`order_id`, `sr_hub`.`orders`.`user_id` rows: 0 1 rows in set (0.00 sec) ``` 2. Drop the materialized view `order_mv1`. ```sql DROP MATERIALIZED VIEW order_mv1; ``` 3. Check if the dropped materialized view exists. ```plain MySQL > SHOW MATERIALIZED VIEWS; Empty set (0.01 sec) ``` Example 2: Drop a non-existing materialized view * If the parameter `IF EXISTS` is specified, StarRocks will not throw an exception when deleting a materialized view that does not exist. ```plain MySQL > DROP MATERIALIZED VIEW IF EXISTS k1_k2; Query OK, 0 rows affected (0.00 sec) ``` * If the parameter `IF EXISTS` is not specified, the system will throw an exception when deleting a materialized view that does not exist. ```plain MySQL > DROP MATERIALIZED VIEW k1_k2; ERROR 1064 (HY000): Materialized view k1_k2 is not find ``` --- ### REFRESH MATERIALIZED VIEW REFRESH MATERIALIZED VIEW manually refreshes a specific asynchronous materialized view or partitions within. > **CAUTION** > > You can only manually refresh materialized views that adopt ASYNC or MANUAL refresh strategy. You can check the refresh strategy of an asynchronous materialized view using [SHOW MATERIALIZED VIEWS](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/SHOW_MATERIALIZED_VIEW.md). This operation requires the REFRESH privilege on the target materialized view. #### Syntax[​](#syntax "Direct link to Syntax") ```sql REFRESH MATERIALIZED VIEW [database.]mv_name [PARTITION START ("") END ("")] [FORCE] [WITH { SYNC | ASYNC } MODE] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------------------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | mv\_name | yes | The name of the materialized view to refresh manually. | | PARTITION START () END () | no | Manually refresh partitions within a certain time interval. | | partition\_start\_date | no | The start date of the partitions to refresh manually. | | partition\_end\_date | no | The end date of the partitions to refresh manually. | | FORCE | no | If you specify this parameter, StarRocks forcibly refreshes the corresponding materialized view or partitions. If you do not specify this parameter, StarRocks automatically judges if the data is updated and refreshes the partition only when needed. | | WITH ... MODE | no | Make a synchronous or asynchronous call of the refresh task. `SYNC` indicates making a synchronous call of the refresh task, and StarRocks returns the task result only when the task succeeds or fails. `ASYNC` indicates making an asynchronous call of the refresh task, and StarRocks returns success right after the task is submitted, leaving the task to be executed asynchronously in the background. You can check the refresh task status of an asynchronous materialized view by querying the `tasks` and `task_runs` metadata views in StarRocks' Information Schema. For more information, see [Check the execution status of asynchronous materialized view](https://docs.starrocks.io/docs/using_starrocks/async_mv/Materialized_view.md#check-the-execution-status-of-asynchronous-materialized-view). Default: `ASYNC`. Supported from v2.5.8 and v3.1.0 onwards. | > **CAUTION** > > When refreshing materialized views created based on the external catalogs, StarRocks refreshes all partitions in the materialized views. #### Monitoring[​](#monitoring "Direct link to Monitoring") After executing `REFRESH MATERIALIZED VIEW`, you can monitor the status and result of the refresh operation in the following ways: * Execute [`SHOW MATERIALIZED VIEWS`](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/SHOW_MATERIALIZED_VIEW.md) to view the status of the materialized view, including the last refresh time, refresh state, and any error messages. * For more detailed execution information, especially for asynchronous refresh tasks, query the system view [`information_schema.task_runs`](https://docs.starrocks.io/docs/sql-reference/information_schema/task_runs.md). This view provides task execution records and states (PENDING, RUNNING, SUCCESS, FAILED). For materialized view refreshes, the `EXTRA_MESSAGE` field includes detailed information about the partitions and execution details for each refresh sub-task. See [Understand Materialized View Task Runs](https://docs.starrocks.io/docs/using_starrocks/async_mv/materialized_view_task_run_details.md) for details. #### Examples[​](#examples "Direct link to Examples") Example 1: Manually refresh a specific materialized view via an asynchronous call. ```plain REFRESH MATERIALIZED VIEW lo_mv1; REFRESH MATERIALIZED VIEW lo_mv1 WITH ASYNC MODE; ``` Example 2: Manually refresh certain partitions of a specific materialized view. ```plain REFRESH MATERIALIZED VIEW lo_mv1 PARTITION START ("2020-02-01") END ("2020-03-01"); ``` Example 3: Forcibly refresh certain partitions of a specific materialized view. ```plain REFRESH MATERIALIZED VIEW lo_mv1 PARTITION START ("2020-02-01") END ("2020-03-01") FORCE; ``` Example 4: Manually refresh a materialized view via a synchronous call. ```plain REFRESH MATERIALIZED VIEW lo_mv1 WITH SYNC MODE; ``` --- ### SHOW ALTER MATERIALIZED VIEW SHOW ALTER MATERIALIZED VIEW shows the building status of synchronous materialized views. tip This operation does not require privileges. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW ALTER MATERIALIZED VIEW [ { FROM | IN } db_name] [WHERE TableName|CreateTime|FinishedTime|State] [ORDER BY] [LIMIT] ``` Parameters in brackets \[] is optional. #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ---------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | db\_name | no | The name of the database to which the materialized view resides. If this parameter is not specified, the current database is used by default. | | WHERE, ORDER BY, LIMIT | no | Filter, sort and truncate the jobs returned. The columns they accept are `TableName`, `CreateTime`, `FinishedTime` and `State`. `FinishTime` is accepted as an alias of `FinishedTime`. | #### Returns[​](#returns "Direct link to Returns") | **Return** | **Description** | | --------------- | ------------------------------------------------ | | JobId | The ID of refresh job. | | TableName | The name of the table. | | CreateTime | The time when refresh job is created. | | FinishedTime | The time when refresh job is finished. | | BaseIndexName | The name of the base table. | | RollupIndexName | The name of the materialized view. | | RollupId | The ID of the materialized view rollup. | | TransactionId | The ID of transaction that waits to be executed. | | State | The state of the job. | | Msg | Error message. | | Progress | Progress of the refresh job. | | Timeout | Timeout for the refresh job. | #### Examples[​](#examples "Direct link to Examples") Example 1: building status of synchronous materialized views ```plain MySQL > SHOW ALTER MATERIALIZED VIEW\G *************************** 1. row *************************** JobId: 475991 TableName: lineorder CreateTime: 2022-08-24 19:46:53 FinishedTime: 2022-08-24 19:47:15 BaseIndexName: lineorder RollupIndexName: lo_mv_sync_1 RollupId: 475992 TransactionId: 33067 State: FINISHED Msg: Progress: NULL Timeout: 86400 *************************** 2. row *************************** JobId: 477337 TableName: lineorder CreateTime: 2022-08-24 19:47:25 FinishedTime: 2022-08-24 19:47:45 BaseIndexName: lineorder RollupIndexName: lo_mv_sync_2 RollupId: 477338 TransactionId: 33068 State: FINISHED Msg: Progress: NULL Timeout: 86400 2 rows in set (0.00 sec) ``` --- ### SHOW CREATE MATERIALIZED VIEW SHOW CREATE MATERIALIZED VIEW shows the definition of a specific asynchronous materialized view. tip This operation does not require privileges. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW CREATE MATERIALIZED VIEW [database.] ``` Parameters in brackets \[] is optional. #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | ------------------------------------------ | | mv\_name | yes | The name of the materialized view to show. | #### Returns[​](#returns "Direct link to Returns") | **Return** | **Description** | | ------------------------ | ---------------------------------------- | | Materialized View | The name of the materialized view. | | Create Materialized View | The definition of the materialized view. | #### Examples[​](#examples "Direct link to Examples") Example 1: Show the definition of a specific materialized view ```plain MySQL > SHOW CREATE MATERIALIZED VIEW lo_mv1\G *************************** 1. row *************************** Materialized View: lo_mv1 Create Materialized View: CREATE MATERIALIZED VIEW `lo_mv1` COMMENT "MATERIALIZED_VIEW" DISTRIBUTED BY HASH(`lo_orderkey`) REFRESH ASYNC PROPERTIES ( "replication_num" = "3", "storage_medium" = "HDD" ) AS SELECT `wlc_test`.`lineorder`.`lo_orderkey` AS `lo_orderkey`, `wlc_test`.`lineorder`.`lo_custkey` AS `lo_custkey`, sum(`wlc_test`.`lineorder`.`lo_quantity`) AS `total_quantity`, sum(`wlc_test`.`lineorder`.`lo_revenue`) AS `total_revenue`, count(`wlc_test`.`lineorder`.`lo_shipmode`) AS `shipmode_count` FROM `wlc_test`.`lineorder` GROUP BY `wlc_test`.`lineorder`.`lo_orderkey`, `wlc_test`.`lineorder`.`lo_custkey` ORDER BY `wlc_test`.`lineorder`.`lo_orderkey` ASC ; 1 row in set (0.01 sec) ``` --- ### SHOW MATERIALIZED VIEWS SHOW MATERIALIZED VIEWS shows all or one specific asynchronous materialized view. Since v3.0, the name of this statement is changed from SHOW MATERIALIZED VIEW to SHOW MATERIALIZED VIEWS. tip This operation does not require privileges. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW MATERIALIZED VIEWS [FROM db_name] [ WHERE NAME { = "mv_name" | LIKE "mv_name_matcher"} ] ``` note Since v3.3, `SHOW MATERIALIZED VIEWS` command will track the state of all task\_runs if a refresh task consists of multiple partitions/task\_runs to refresh. Only when all task\_runs are success, `last_refresh_state` will return `SUCCESS`. #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ----------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | | db\_name | no | The name of the database to which the materialized view resides. If this parameter is not specified, the current database is used by default. | | mv\_name | no | The name of the materialized view to show. | | mv\_name\_matcher | no | The matcher used to filter materialized views. | #### Returns[​](#returns "Direct link to Returns") | **Return** | **Description** | | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | id | The ID of the materialized view. | | database\_name | The name of the database in which the materialized view resides. | | name | The name of the materialized view. | | refresh\_type | The refresh type of the materialized view. Valid values: `SYNC` (synchronous materialized view) and `ASYNC` (asynchronous materialized view, regardless of how the refresh is triggered). | | is\_active | Whether the materialized view state is active. Valid Value: `true` and `false`. | | inactive\_reason | The reason why the materialized view is inactive. | | partition\_type | The partition type of the materialized view, including RANGE and UNPARTITIONED. | | task\_id | ID of the materialized view refresh task. | | task\_name | Name of the materialized view refresh task. | | last\_refresh\_start\_time | The start time of the last refresh of the materialized view. | | last\_refresh\_finished\_time | The end time of the last refresh of the materialized view. | | last\_refresh\_duration | The time taken by the last refresh. Unit: seconds. | | last\_refresh\_state | The status of the last refresh, including PENDING, RUNNING, FAILED, SUCCESS, and SKIPPED. When no data changes are detected on the base table partition, the refresh for the corresponding materialized view partition is skipped. | | last\_refresh\_force\_refresh | Whether the last refresh is a FORCE refresh. | | last\_refresh\_start\_partition | The start partition of the last refresh in the materialized view. | | last\_refresh\_end\_partition | The end partition of the last refresh in the materialized view. | | last\_refresh\_base\_refresh\_partitions | The base table partitions that were refreshed in the last refresh. | | last\_refresh\_mv\_refresh\_partitions | The materialized view partitions that were refreshed in the last refresh. | | last\_refresh\_error\_code | The error code for the last failed refresh of the materialized view (if the materialized view state is not active). | | last\_refresh\_error\_message | The reason why the last refresh failed (if the materialized view state is not active). | | rows | The number of data rows in the materialized view. | | text | The statement used to create the materialized view. | | extra\_message | Extra information about the latest refresh task. | | query\_rewrite\_status | Query rewrite status of the materialized view. | | creator | Creator of the materialized view refresh task. | | last\_refresh\_process\_time | The process start time of the latest refresh task. | | last\_refresh\_job\_id | Job ID of the latest refresh task. | | last\_refresh\_time | Time up to which base table updates are reflected in the materialized view. | | warehouse | Name of the warehouse that the asynchronous materialized view uses for its refresh tasks. Empty in shared-nothing mode, or for synchronous (rollup) materialized views. | | refresh\_mode | Configured refresh mode of the asynchronous materialized view. Valid values: `PCT` (partition change tracking, where only changed partitions are refreshed) and `INCREMENTAL` (incremental view maintenance). Empty for synchronous materialized views. | | refresh\_trigger | How a refresh is triggered. Valid values: `NONE` (synchronous materialized view), `MANUAL` (only via REFRESH MATERIALIZED VIEW), `SCHEDULED` (periodic, via an EVERY interval), and `ON_BASE_TABLE_CHANGE` (automatically when a base table loads or changes). | | refresh\_policy | Human-readable refresh policy. Valid values: `NONE`, `MANUAL`, `ON_BASE_TABLE_CHANGE`, or a schedule such as `START("yyyy-MM-dd HH:mm:ss") EVERY(INTERVAL n unit)` (the `START` clause is present only if a start time was defined). | | resource\_group | Resource group used for the materialized view's refresh tasks (from the materialized view's `resource_group` property). Defaults to `default_mv_wg` when not set. | | query\_rewrite\_status\_reason | The reason behind `query_rewrite_status`. Valid values: `OK`, `MV_INACTIVE`, `QUERY_REWRITE_DISABLED`, `UNSUPPORTED_DEFINITION`, and `UNKNOWN`. | | base\_table\_refresh\_version\_times | Per-base-table data version time, as a JSON object mapping each base table's `catalog.database.table` name to the latest data version time observed for it. This is the per-table detail behind `last_refresh_time` (their single maximum): external/data lake base tables report the partition source modified time, and OLAP (internal) base tables report the visible-version commit time. `{}` when no base table has a recorded time. | | last\_freshness\_confirmed\_at | Start time of the most recent successful refresh, recorded once the entire refresh (all of its task runs) completes; a refresh that found no base-table changes to apply also confirms freshness. The materialized view reflects base-table data as of this moment. Unlike `last_refresh_time` (the base-table data version), this is wall-clock time. Empty until the first successful refresh, and for synchronous materialized views. A partition-scoped (partial) REFRESH does not advance it. | #### Examples[​](#examples "Direct link to Examples") The following examples is based on this business scenario: ```plain -- Create Table: customer CREATE TABLE customer ( C_CUSTKEY INTEGER NOT NULL, C_NAME VARCHAR(25) NOT NULL, C_ADDRESS VARCHAR(40) NOT NULL, C_NATIONKEY INTEGER NOT NULL, C_PHONE CHAR(15) NOT NULL, C_ACCTBAL double NOT NULL, C_MKTSEGMENT CHAR(10) NOT NULL, C_COMMENT VARCHAR(117) NOT NULL, PAD char(1) NOT NULL) ENGINE=OLAP DUPLICATE KEY(`c_custkey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`c_custkey`) PROPERTIES ( "replication_num" = "3", "storage_format" = "DEFAULT" ); -- Create MV: customer_mv CREATE MATERIALIZED VIEW customer_mv DISTRIBUTED BY HASH(c_custkey) REFRESH MANUAL PROPERTIES ( "replication_num" = "3" ) AS SELECT c_custkey, c_phone, c_acctbal, count(1) as c_count, sum(c_acctbal) as c_sum FROM customer GROUP BY c_custkey, c_phone, c_acctbal; -- Refresh the MV REFRESH MATERIALIZED VIEW customer_mv; ``` Example 1: Show a specific materialized view. ```plain mysql> SHOW MATERIALIZED VIEWS WHERE NAME='customer_mv'\G *************************** 1. row *************************** id: 10142 name: customer_mv database_name: test refresh_type: ASYNC is_active: true last_refresh_start_time: 2023-02-17 10:27:33 last_refresh_finished_time: 2023-02-17 10:27:33 last_refresh_duration: 0 last_refresh_state: SUCCESS inactive_code: 0 inactive_reason: text: CREATE MATERIALIZED VIEW `customer_mv` COMMENT "MATERIALIZED_VIEW" DISTRIBUTED BY HASH(`c_custkey`) REFRESH MANUAL PROPERTIES ( "replication_num" = "3", "storage_medium" = "HDD" ) AS SELECT `customer`.`c_custkey`, `customer`.`c_phone`, `customer`.`c_acctbal`, count(1) AS `c_count`, sum(`customer`.`c_acctbal`) AS `c_sum` FROM `test`.`customer` GROUP BY `customer`.`c_custkey`, `customer`.`c_phone`, `customer`.`c_acctbal`; rows: 0 warehouse: refresh_mode: PCT refresh_trigger: MANUAL refresh_policy: MANUAL resource_group: default_mv_wg 1 row in set (0.11 sec) ``` Example 2: Show materialized views by matching the name. ```plain mysql> SHOW MATERIALIZED VIEWS WHERE NAME LIKE 'customer_mv'\G *************************** 1. row *************************** id: 10142 name: customer_mv database_name: test refresh_type: ASYNC is_active: true last_refresh_start_time: 2023-02-17 10:27:33 last_refresh_finished_time: 2023-02-17 10:27:33 last_refresh_duration: 0 last_refresh_state: SUCCESS inactive_code: 0 inactive_reason: text: CREATE MATERIALIZED VIEW `customer_mv` COMMENT "MATERIALIZED_VIEW" DISTRIBUTED BY HASH(`c_custkey`) REFRESH MANUAL PROPERTIES ( "replication_num" = "3", "storage_medium" = "HDD" ) AS SELECT `customer`.`c_custkey`, `customer`.`c_phone`, `customer`.`c_acctbal`, count(1) AS `c_count`, sum(`customer`.`c_acctbal`) AS `c_sum` FROM `test`.`customer` GROUP BY `customer`.`c_custkey`, `customer`.`c_phone`, `customer`.`c_acctbal`; rows: 0 warehouse: refresh_mode: PCT refresh_trigger: MANUAL refresh_policy: MANUAL resource_group: default_mv_wg 1 row in set (0.12 sec) ``` --- ### Prepared statements From v3.2 onwards, StarRocks provides prepared statements for executing SQL statements multiple times with the same structure but different variables. This feature significantly improves execution efficiency and prevents SQL injection. #### Description[​](#description "Direct link to Description") The prepared statements basically work as follows: 1. **Preparation**: Users prepare a SQL statement where variables are represented by placeholders `?`. The FE parses the SQL statement and generates an execution plan. 2. **Execution**: After declaring variables, users pass these variables to the statement and execute the statement. Users can execute the same statement multiple times with different variables. **Benefits** * **Saves overhead of parsing**: In real-world business scenarios, an application often executes a statement multiple times with the same structure but different variables. With prepared statements supported, StarRocks needs to parse the statement only once during the preparation phase. Subsequent executions of the same statement with different variables can directly use the pre-generated parsing result. As such, statement execution performance is significantly improved, especially for complex queries. * **Prevents SQL injection attacks**: By separating the statement from the variables and passing user-input data as parameters rather than directly concatenating the variables into the statement, StarRocks can prevent malicious users from executing malicious SQL codes. **Usages** Prepared statements are effective only in the current session and cannot be used in other sessions. After the current session exits, the prepared statements created in that session are automatically dropped. #### Syntax[​](#syntax "Direct link to Syntax") The execution of a prepared statement consists of the following phases: * PREPARE: Prepare the statement where variables are represented by placeholders `?`. * SET: Declare variables within the statement. * EXECUTE: Pass the declared variables to the statement and execute it. * DROP PREPARE or DEALLOCATE PREPARE: Delete the prepared statement. ##### PREPARE[​](#prepare "Direct link to PREPARE") **Syntax:** ```sql PREPARE FROM ``` **Parameters:** * `stmt_name`: the name given to the prepared statement, which is subsequently used to execute or deallocate that prepared statement. The name must be unique within a single session. * `preparable_stmt`: the SQL statement to be prepared, where the placeholder for variables is a question mark (`?`). Currently, **only the `SELECT` statement** is supported. **Example:** Prepare a `SELECT` statement with specific values represented by placeholders `?`. ```sql PREPARE select_by_id_stmt FROM 'SELECT * FROM users WHERE id = ?'; ``` ##### SET[​](#set "Direct link to SET") **Syntax:** ```sql SET @var_name = expr [, ...]; ``` **Parameters:** * `var_name`: the name of a user-defined variable. * `expr`: a user-defined variable. **Example:** Declare variables. ```sql SET @id1 = 1, @id2 = 2; ``` For more information, see [user-defined variables](https://docs.starrocks.io/docs/sql-reference/user_defined_variables.md). ##### EXECUTE[​](#execute "Direct link to EXECUTE") **Syntax:** ```sql EXECUTE [USING @var_name [, @var_name] ...] ``` **Parameters:** * `var_name`: the name of a variable declared in the `SET` statement. * `stmt_name`: the name of the prepared statement. **Example:** Pass variables to a `SELECT` statement and execute that statement. ```sql EXECUTE select_by_id_stmt USING @id1; ``` ##### DROP PREPARE or DEALLOCATE PREPARE[​](#drop-prepare-or-deallocate-prepare "Direct link to DROP PREPARE or DEALLOCATE PREPARE") **Syntax:** ```sql {DEALLOCATE | DROP} PREPARE ``` **Parameters:** * `stmt_name`: The name of the prepared statement. **Example:** Delete a prepared statement. ```sql DROP PREPARE select_by_id_stmt; ``` #### Examples[​](#examples "Direct link to Examples") ##### Use prepared statements[​](#use-prepared-statements "Direct link to Use prepared statements") The following example demonstrates how to use prepared statements to insert, delete, update, and query data from a StarRocks table: Assuming the following database named `demo` and table named `users` are already created: ```sql CREATE DATABASE IF NOT EXISTS demo; USE demo; CREATE TABLE users ( id BIGINT NOT NULL, country STRING, city STRING, revenue BIGINT ) PRIMARY KEY (id) DISTRIBUTED BY HASH(id); ``` 1. Prepare statements for execution. ```sql PREPARE select_all_stmt FROM 'SELECT * FROM users'; PREPARE select_by_id_stmt FROM 'SELECT * FROM users WHERE id = ?'; ``` 2. Declare variables in these statements. ```sql SET @id1 = 1, @id2 = 2; ``` 3. Use the declared variables to execute the statements. ```sql -- Query all data from the table. EXECUTE select_all_stmt; -- Query data with ID 1 or 2 separately. EXECUTE select_by_id_stmt USING @id1; EXECUTE select_by_id_stmt USING @id2; ``` ##### Using Prepared Statements in Java application[​](#using-prepared-statements-in-java-application "Direct link to Using Prepared Statements in Java application") The following example demonstrates how a Java application can use a JDBC driver to insert, delete, update, and query data from a StarRocks table: 1. When specifying StarRocks' connection URL in JDBC, you need to enable server-side prepared statements: ```plaintext jdbc:mysql://:/useServerPrepStmts=true ``` 2. The StarRocks GitHub project provides a [Java code example](https://github.com/StarRocks/starrocks/blob/main/fe/fe-core/src/test/java/com/starrocks/analysis/PreparedStmtTest.java) that explains how to insert, delete, update, and query data from a StarRocks table through a JDBC driver. --- ### Resource #### [📄️ ALTER RESOURCE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/ALTER_RESOURCE.md) [Use the ALTER RESOURCE statement to modify the properties of a resource.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/ALTER_RESOURCE.md) #### [📄️ CREATE RESOURCE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/CREATE_RESOURCE.md) [Creates resources.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/CREATE_RESOURCE.md) #### [📄️ DROP RESOURCE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/DROP_RESOURCE.md) [Use DROP RESOURCE to drop an existing resource.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/DROP_RESOURCE.md) #### [📄️ SHOW RESOURCES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/SHOW_RESOURCES.md) [SHOW RESOURCES shows the resources that the users have permissions to use.](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/SHOW_RESOURCES.md) --- ### ALTER RESOURCE Use the ALTER RESOURCE statement to modify the properties of a resource. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ALTER RESOURCE 'resource_name' SET PROPERTIES ("key"="value", ...) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `resource_name`: the name of the resource to be modified. * `PROPERTIES ("key"="value", ...)`: the properties of the resource. You can modify different properties based on resource types. Currently, StarRocks supports modifying the URI of the Hive metastore of the following resources. * Apache Iceberg resources support modifying the following properties: * `iceberg.catalog-impl`: the fully qualified class name of [custom catalog](https://docs.starrocks.io/docs/data_source/External_table.md). * `iceberg.catalog.hive.metastore.uris`: the URI of the Hive metastore. * Apache Hive™ resources and Apache Hudi resources support modifying `hive.metastore.uris`, which indicates the URI of the Hive metastore. #### Usage notes[​](#usage-notes "Direct link to Usage notes") After you reference a resource to create an external table, if you modify the URI of the Hive metastore of this resource, the external table becomes unavailable. If you still want to use the external table to query data, make sure that the new metastore contains a table whose name and schema are the same as that in the original metastore. #### Examples[​](#examples "Direct link to Examples") Modify the URI of the Hive metastore of Hive resource `hive0`. ```sql ALTER RESOURCE 'hive0' SET PROPERTIES ("hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083") ``` --- ### CREATE RESOURCE Creates resources. The following types of resources can be created: Apache Spark™, Apache Hive™, Apache Iceberg, Apache Hudi, and JDBC. Spark resources are used in Spark Load to manage loading information, such as YARN configurations, storage path of intermediate data, and Broker configurations. Hive, Iceberg, Hudi, and JDBC resources are used for managing data source access information involved in querying [External tables](https://docs.starrocks.io/docs/data_source/External_table.md). tip * Only users with the SYSTEM-level CREATE RESOURCE privilege can perform this operation. * You can create JDBC resources only in StarRocks v2.3 and later. #### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE [EXTERNAL] RESOURCE "resource_name" PROPERTIES ("key"="value", ...) ``` #### Parameters[​](#parameters "Direct link to Parameters") * `resource_name`: the name of the resource to create. For the naming conventions, see [System limits](https://docs.starrocks.io/docs/sql-reference/System_limit.md). * `PROPERTIES`: specifies the properties of the resource type. The PROPERTIES vary depending on the resource type. See Examples for details. #### Examples[​](#examples "Direct link to Examples") 1. Create a Spark resource named spark0 in yarn Cluster mode. ```sql CREATE EXTERNAL RESOURCE "spark0" PROPERTIES ( "type" = "spark", "spark.master" = "yarn", "spark.submit.deployMode" = "cluster", "spark.jars" = "xxx.jar,yyy.jar", "spark.files" = "/tmp/aaa,/tmp/bbb", "spark.executor.memory" = "1g", "spark.yarn.queue" = "queue0", "spark.hadoop.yarn.resourcemanager.address" = "127.0.0.1:9999", "spark.hadoop.fs.defaultFS" = "hdfs://127.0.0.1:10000", "working_dir" = "hdfs://127.0.0.1:10000/tmp/starrocks", "broker" = "broker0", "broker.username" = "user0", "broker.password" = "password0" ); ``` Parameters related to Spark are as follows: ```plain 1. spark.master: required. Currently, yarn and spark ://host:port. are supported. 2. spark.submit.deployMode: deployment mode of the Spark program is required. Support cluster and client. 3. spark.hadoop.yarn.resourcemanager.address: required when master is yarn. 4. spark.hadoop.fs.defaultFS: required when master is yarn. 5. Other parameters are optional. Please refer to http://spark.apache.org/docs/latest/configuration.html ``` If Spark is used for ETL, working\_DIR and broker need to be specified. The instructions are as follows: ```plain working_dir: Directory used by ETL. It is required when spark is used as ETL resource. For example: hdfs://host:port/tmp/starrocks. broker: Name of broker. It is required when spark is used as ETL resource and needs to be configured beforehand by using `ALTER SYSTEM ADD BROKER` command. broker.property_key: It is the property information needed to be specified when broker reads the intermediate files created by ETL. ``` 2. Create a Hive resource named hive0. ```sql CREATE EXTERNAL RESOURCE "hive0" PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://10.10.44.98:9083" ); ``` --- ### DROP RESOURCE Use DROP RESOURCE to drop an existing resource. Only root or superuser can drop resources. Syntax: ```sql DROP RESOURCE 'resource_name' ``` #### Examples[​](#examples "Direct link to Examples") 1. Drop Spark resource named spark0. ```sql DROP RESOURCE 'spark0'; ``` 2. Drop Hive resource named hive0. ```sql DROP RESOURCE 'hive0'; ``` --- ### SHOW RESOURCES SHOW RESOURCES shows the resources that the users have permissions to use. Ordinary users can only use the resources they have permission to use, while root or admin users can show all resources. Syntax: ```sql SHOW RESOURCES ``` --- ### Table, bucket, partition, and index #### [🗃️ SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) [16 items](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) #### [📄️ ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md) [Modifies an existing StarRocks table: renaming, partitioning, bucketing, columns, rollup indexes, and table properties.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md) #### [📄️ CANCEL ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CANCEL_ALTER_TABLE.md) [Cancels an in-progress ALTER TABLE operation such as column modification, schema optimization, or rollup index creation.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CANCEL_ALTER_TABLE.md) #### [📄️ CREATE INDEX](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_INDEX.md) [Creates indexes.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_INDEX.md) #### [📄️ CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) [Create a new table in StarRocks.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) #### [📄️ CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) [CREATE TABLE AS SELECT (CTAS) creates a new table and populates it with query results, supporting synchronous and asynchronous modes.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) #### [📄️ CREATE TABLE LIKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_LIKE.md) [CREATE TABLE LIKE creates an identical empty table based on the definition of another table.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_LIKE.md) #### [📄️ DELETE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DELETE.md) [Deletes data rows from a table based on the specified conditions.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DELETE.md) #### [📄️ DESC](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md) [Views the schema of a StarRocks table or external table, including its sort key and materialized view information.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md) #### [📄️ DROP INDEX](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_INDEX.md) [DROP INDEX is used to drop a specified index on a table.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_INDEX.md) #### [📄️ DROP TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md) [DROP TABLE is used to delete a table.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md) #### [📄️ REFRESH EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/REFRESH_EXTERNAL_TABLE.md) [REFRESH EXTERNAL TABLE metadata cached in StarRocks.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/REFRESH_EXTERNAL_TABLE.md) #### [📄️ SHOW ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_ALTER.md) [Shows the execution status of ongoing ALTER TABLE operations including column changes, schema optimization, and rollup index changes.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_ALTER.md) #### [📄️ SHOW CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_CREATE_TABLE.md) [Returns the CREATE TABLE statement that was used to create a given table.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_CREATE_TABLE.md) #### [📄️ SHOW DELETE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_DELETE.md) [SHOW DELETE queries historical DELETE operations that were successfully performed on Duplicate Key, Unique Key, and Aggregate tables in a specified database.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_DELETE.md) #### [📄️ SHOW DYNAMIC PARTITION TABLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_DYNAMIC_PARTITION_TABLES.md) [SHOW DYNAMIC PARTITION TABLES is used to display the status of all the partitioned tables for which dynamic partitioning properties are configured in a...](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_DYNAMIC_PARTITION_TABLES.md) #### [📄️ SHOW FULL COLUMNS](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_FULL_COLUMNS.md) [SHOW FULL COLUMNS is used to show content in columns from specified tables.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_FULL_COLUMNS.md) #### [📄️ SHOW INDEX](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_INDEX.md) [SHOW INDEX is used to show information related to index in a table.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_INDEX.md) #### [📄️ SHOW PARTITIONS](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md) [SHOW PARTITIONS displays partition information for a table, including common and temporary partitions.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md) #### [📄️ SHOW TABLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLES.md) [SHOW TABLES displays all tables in a StarRocks database or a database in an external data source, for example, Hive, Iceberg, Hudi, or Delta Lake.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLES.md) #### [📄️ SHOW TABLET](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLET.md) [SHOW TABLET displays tablet related information.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLET.md) #### [📄️ TRUNCATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/TRUNCATE_TABLE.md) [TRUNCATE TABLE is used to truncate the specified table and partition data.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/TRUNCATE_TABLE.md) #### [📄️ UPDATE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/UPDATE.md) [Updates rows in a Primary Key table.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/UPDATE.md) #### [📄️ AUTO\_INCREMENT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/auto_increment.md) [Since version 3.0, StarRocks supports the AUTO\_INCREMENT column attribute, which can simplify data management.](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/auto_increment.md) --- ### ALTER TABLE ALTER TABLE Modifies an existing table, including: * [Rename table, partition, rollup, or column](#rename) * [Modify table comment](#alter-table-comment-from-v31) * [Modify partitions (add/delete partitions and modify partition attributes)](#modify-partition) * [Modify the bucketing method and number of buckets](#modify-the-bucketing-method-and-number-of-buckets-from-v32) * [Modify the tablet size](#modify-the-tablet-size) * [Modify columns (add/delete columns, change column order, and modify column comment)](#modify-columns-adddelete-columns-change-column-order-modify-column-comment) * [Create/delete rollup](#modify-rollup) * [Create/delete index](#modify-indexes) * [Modify table properties](#modify-table-properties) * [Atomic swap](#swap) * [Manual data version compaction](#manual-compaction-from-31) * [Drop Primary Key Persistent Index](#drop-primary-key-persistent-index-from-339) tip This operation requires the ALTER privilege on the destination table. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ALTER TABLE [.] alter_clause1[, alter_clause2, ...] ``` `alter_clause` can held the following operations: rename, comment, partition, bucket, column, rollup, index, table property, swap, and compaction. * rename: renames a table, rollup, partition, or column (supported from **v3.3.2 onwards**). * comment: modifies the table comment (supported from **v3.1 onwards**). * partition: modifies partition properties, drops a partition, or adds a partition. * bucket: modifies the bucketing method and number of buckets. * column: adds, drops, or reorders columns, modifies column type or comment * rollup: creates or drops a rollup. * index: modifies indexes. * swap: atomic exchange of two tables. * compaction: performs manual compaction to merge versions of loaded data (supported from **v3.1 onwards**). * drop persistent index: Drop persistent index for Primary Key table in shared-data cluster (supported from **v3.3.9 onwards**). #### Limits and usage notes[​](#limits-and-usage-notes "Direct link to Limits and usage notes") * Operations on partition, column, and rollup index cannot be performed in one ALTER TABLE statement. * One table can have only one ongoing schema change operation at a time. You cannot run two schema change commands on a table at the same time. * Operations on bucket, column and rollup are asynchronous operations. A success message is return immediately after the task is submitted. You can run the [SHOW ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_ALTER.md) command to check the progress, and run the [CANCEL ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CANCEL_ALTER_TABLE.md) command to cancel the operation. * Operations on rename, comment, partition, index and swap are synchronous operations, and a command return indicates that the execution is finished. ##### Rename[​](#rename "Direct link to Rename") Rename supports modification of table name, rollup, and partition name. ###### Rename a table[​](#rename-a-table "Direct link to Rename a table") ```sql ALTER TABLE RENAME ``` ###### Rename a rollup[​](#rename-a-rollup "Direct link to Rename a rollup") ```sql ALTER TABLE [.] RENAME ROLLUP ``` ###### Rename a partition[​](#rename-a-partition "Direct link to Rename a partition") ```sql ALTER TABLE [.] RENAME PARTITION ``` ###### Rename a column[​](#rename-a-column "Direct link to Rename a column") From v3.3.2 onwards, StarRocks supports renaming columns. ```sql ALTER TABLE [.] RENAME COLUMN [ TO ] ``` note * After renaming a column from A to B, adding a new column named A is not supported. * Materialized views built on a renamed column will not take effect. You must rebuild them upon the column with the new name. ##### Alter table comment (from v3.1)[​](#alter-table-comment-from-v31 "Direct link to Alter table comment (from v3.1)") Syntax: ```sql ALTER TABLE [.] COMMENT = ""; ``` ##### Modify partition[​](#modify-partition "Direct link to Modify partition") ###### ADD PARTITION(S)[​](#add-partitions "Direct link to ADD PARTITION(S)") You must strictly follow the respective syntax to add range partitions or list partitions. note * Adding expression partitions is not supported. * Please note that `PARTITION BY date_trunc(column)` and `PARTITION BY time_slice(column)` are considered range partitioning, despite their format of expression partitioning. Therefore, you can use the following syntax for range partitions to add new partitions to tables use such partitioning strategies. Syntax： * Range partitions ```sql ALTER TABLE ADD { single_range_partition | multi_range_partitions } [distribution_desc] ["key"="value"]; single_range_partition ::= PARTITION [IF NOT EXISTS] VALUES partition_key_desc partition_key_desc ::= { LESS THAN { MAXVALUE | value_list } | [ value_list , value_list ) } -- Note that [ represents a left-closed interval. value_list ::= ( [, ...] ) multi_range_partitions ::= { PARTITIONS START ("") END ("") EVERY ( INTERVAL ) | PARTITIONS START ("") END ("") EVERY ( ) } -- The partition column values still need to be enclosed in double quotes even if the partition column values specified by START and END are integers. However, the interval values in the EVERY clause do not need to be enclosed in double quotes. ``` * List partitions ```sql ALTER TABLE ADD PARTITION VALUES IN (value_list) [distribution_desc] ["key"="value"]; value_list ::= value_item [, ...] value_item ::= { | ( [, ...] ) } ``` Parameters: * Partition-related parameters: * For range partitions, you can add a single range partition (`single_range_partition`) or multiple range partitions in batch (`multi_range_partitions`). * For list partitions, you can only add a single list partition. * `distribution_desc`: You can set the number of buckets for the new partition separately, but you cannot set the bucketing method separately. * `"key"="value"`: You can set properties for the new partition. For details, see [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md#properties). Examples: * Range partitions * If the partition column is specified as `event_day` at table creation, for example `PARTITION BY RANGE(event_day)`, and a new partition needs to be added after table creation, you can execute: ```sql ALTER TABLE site_access ADD PARTITION p4 VALUES LESS THAN ("2020-04-30"); ``` * If the partition column is specified as `datekey` at table creation, for example `PARTITION BY RANGE (datekey)`, and multiple partitions need to be added in batch after table creation, you can execute: ```sql ALTER TABLE site_access ADD PARTITIONS START ("2021-01-05") END ("2021-01-10") EVERY (INTERVAL 1 DAY); ``` * List partitions * If a single partition column is specified at table creation, for example `PARTITION BY LIST (city)`, and a new partition needs to be added after table creation, you can execute: ```sql ALTER TABLE t_recharge_detail2 ADD PARTITION pCalifornia VALUES IN ("Los Angeles","San Francisco","San Diego"); ``` * If multiple partition columns are specified at table creation, for example `PARTITION BY LIST (dt,city)`, and a new partition needs to be added after table creation, you can execute: ```sql ALTER TABLE t_recharge_detail4 ADD PARTITION p202204_California VALUES IN ( ("2022-04-01", "Los Angeles"), ("2022-04-01", "San Francisco"), ("2022-04-02", "Los Angeles"), ("2022-04-02", "San Francisco") ); ``` ###### DROP PARTITION(S)[​](#drop-partitions "Direct link to DROP PARTITION(S)") * Drop a single partition: ```sql ALTER TABLE [.] DROP PARTITION [ IF EXISTS ] [ FORCE ] ``` * Drop partitions in batch (Supported from v3.4.0): ```sql ALTER TABLE [.] DROP PARTITIONS [ IF EXISTS ] { partition_name_list | multi_range_partitions } [ FORCE ] partition_name_list ::= ( [, ... ] ) multi_range_partitions ::= { START ("") END ("") EVERY ( INTERVAL ) | START ("") END ("") EVERY ( ) } -- The partition column values still need to be enclosed in double quotes even if the partition column values are integers. However, the interval values in the EVERY clause do not need to be enclosed in double quotes. ``` Notes for `multi_range_partitions`: * It only applies to Range Partitioning. * The parameters involved is consistent with those in [ADD PARTITION(S)](#add-partitions). * It only supports partitions with a single Partition Key. * Drop partitions with Common Partition Expression (Supported from v3.5.0): ```sql ALTER TABLE [.] DROP PARTITIONS WHERE ``` From v3.5.0 onwards, StarRocks supports dropping partitions using Common Partition Expression. You can specify a WHERE clause with an expression to filter the partitions to drop. * The expression declares the partitions to be dropped. Partitions that meet the condition in the expression will be dropped in batch. Be cautious when proceeding. * The expression can only contain partition columns and constants. Non-partition columns are not supported. * Common Partition Expression applies to List partitions and Range partitions differently: * For tables with List partitions, StarRocks supports deleting partitions filtered by the Common Partition Expression. * For tables with Range partitions, StarRocks can only filter and delete partitions using the partition pruning capability of FE. Partitions correspond to predicates that are not supported by partition pruning cannot be filtered and deleted. Example: ```sql -- Drop the data earlier than the last three months. Column `dt` is the partition column of the table. ALTER TABLE t1 DROP PARTITIONS WHERE dt < CURRENT_DATE() - INTERVAL 3 MONTH; ``` note * Keep at least one partition for partitioned tables. * If FORCE is not specified, you can recover the dropped partitions by using the [RECOVER](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/RECOVER.md) command within a specified period (1 day by default). * If FORCE is specified, the partitions will be deleted directly regardless of whether there are any unfinished operations on the partitions, and they cannot be recovered. Thus, generally, this operation is not recommended. ###### Add a temporary partition[​](#add-a-temporary-partition "Direct link to Add a temporary partition") Syntax: ```sql ALTER TABLE [.] ADD TEMPORARY PARTITION [IF NOT EXISTS] { single_range_partition | multi_range_partitions | list_partitions } [DISTRIBUTED BY HASH (k1[,k2 ...]) [BUCKETS num]] -- For details of single_range_partition and multi_range_partitions, see ADD PARTITION(S) section in this page. list_partitions::= PARTITION VALUES IN (value_list) value_list ::= value_item [, ...] value_item ::= { | ( [, ...] ) } ``` ###### Use a temporary partition to replace the current partition[​](#use-a-temporary-partition-to-replace-the-current-partition "Direct link to Use a temporary partition to replace the current partition") Syntax: ```sql ALTER TABLE [.] REPLACE PARTITION partition_desc ["key"="value"] WITH TEMPORARY PARTITION partition_desc ["key"="value"] [PROPERTIES ("key"="value", ...)] ``` ###### Drop a temporary partition[​](#drop-a-temporary-partition "Direct link to Drop a temporary partition") Syntax: ```sql ALTER TABLE [.] DROP TEMPORARY PARTITION ``` ###### Modify partition properties[​](#modify-partition-properties "Direct link to Modify partition properties") **Syntax** ```sql ALTER TABLE [.] MODIFY PARTITION { | ( [, ...] ) | (*) } SET ("key" = "value", ...); ``` **Usages** * The following properties of a partition can be modified: * storage\_medium * storage\_cooldown\_ttl or storage\_cooldown\_time * replication\_num * For the table that has only one partition, the partition name is the same as the table name. If the table is divided into multiple partitions, you can use `(*)`to modify the properties of all partitions, which is more convenient. * Execute `SHOW PARTITIONS FROM ` to view the partition properties after modification. ##### Modify the bucketing method and number of buckets (from v3.2)[​](#modify-the-bucketing-method-and-number-of-buckets-from-v32 "Direct link to Modify the bucketing method and number of buckets (from v3.2)") Syntax: ```sql ALTER TABLE [.] [ partition_names ] [ distribution_desc ] partition_names ::= (PARTITION | PARTITIONS) ( [, ...] ) distribution_desc ::= DISTRIBUTED BY RANDOM [ BUCKETS ] | DISTRIBUTED BY HASH ( [, ...] ) [ [ DEFAULT ] BUCKETS ] ``` Example: For example, the original table is a Duplicate Key table where hash bucketing is used and the number of buckets is automatically set by StarRocks. ```sql CREATE TABLE IF NOT EXISTS details ( event_time DATETIME NOT NULL COMMENT "datetime of event", event_type INT NOT NULL COMMENT "type of event", user_id INT COMMENT "id of user", device_code INT COMMENT "device code", channel INT COMMENT "" ) DUPLICATE KEY(event_time, event_type) PARTITION BY date_trunc('day', event_time) DISTRIBUTED BY HASH(user_id); -- Insert data of several days INSERT INTO details (event_time, event_type, user_id, device_code, channel) VALUES -- Data of November 26th ('2023-11-26 08:00:00', 1, 101, 12345, 2), ('2023-11-26 09:15:00', 2, 102, 54321, 3), ('2023-11-26 10:30:00', 1, 103, 98765, 1), -- Data of November 27th ('2023-11-27 08:30:00', 1, 104, 11111, 2), ('2023-11-27 09:45:00', 2, 105, 22222, 3), ('2023-11-27 11:00:00', 1, 106, 33333, 1), -- Data of November 28th ('2023-11-28 08:00:00', 1, 107, 44444, 2), ('2023-11-28 09:15:00', 2, 108, 55555, 3), ('2023-11-28 10:30:00', 1, 109, 66666, 1); ``` ###### Modify the bucketing method only[​](#modify-the-bucketing-method-only "Direct link to Modify the bucketing method only") > **NOTICE** > > * The modification is applied to all partitions in the table and cannot be applied to specific partitions only. > * Although only the bucketing method needs to be modified, the number of buckets still needs to be specified in the command using `BUCKETS `. If `BUCKETS ` is not specified, it means that the number of buckets is automatically determined by StarRocks. * The bucketing method is modified to random bucketing from hash bucketing and the number of buckets remains automatically set by StarRocks. ```sql ALTER TABLE details DISTRIBUTED BY RANDOM; ``` * The keys for hash bucketing are modified to `user_id, event_time` from `event_time, event_type`. And the number of buckets remains automatically set by StarRocks. ```sql ALTER TABLE details DISTRIBUTED BY HASH(user_id, event_time); ``` ###### Modify the number of buckets only[​](#modify-the-number-of-buckets-only "Direct link to Modify the number of buckets only") > **NOTICE** > > Although only the number of buckets needs to be modified, the bucketing method still needs to be specified in the command, for example, `HASH(user_id)`. * Modify the number of buckets for all partitions to 10 from being automatically set by StarRocks. ```sql ALTER TABLE details DISTRIBUTED BY HASH(user_id) BUCKETS 10; ``` * Modify the default number of buckets of the table to 10 from being automatically set by StarRocks **without changing that of existing partitions** (supported from v3.5.8 and v4.0.1 onwards). ```sql ALTER TABLE details DISTRIBUTED BY HASH(user_id) DEFAULT BUCKETS 10; ``` > **NOTICE** > > You cannot specify both `partition_names` and `DEFAULT`. * Modify the number of buckets for specified partitions to 15 from being automatically set by StarRocks. ```sql ALTER TABLE details PARTITIONS (p20231127, p20231128) DISTRIBUTED BY HASH(user_id) BUCKETS 15 ; ``` > **NOTE** > > Partition names can be viewed by executing `SHOW PARTITIONS FROM ;`. ###### Modify both the bucketing method and the number of buckets[​](#modify-both-the-bucketing-method-and-the-number-of-buckets "Direct link to Modify both the bucketing method and the number of buckets") > **NOTICE** > > The modification is applied to all partitions in the table and cannot be applied to specific partitions only. * Modify the bucketing method from hash bucketing to random bucketing, and change the number of buckets to 10 from being automatically set by StarRocks. ```sql ALTER TABLE details DISTRIBUTED BY RANDOM BUCKETS 10; ``` * Modify the key for hash bucketing, and change the number of buckets to 10 from being automatically set by StarRocks. The key used for hashing bucketing is modified to `user_id, event_time` from the original `event_time, event_type`. The number of buckets is modified to 10 from automatically set by StarRocks. ```sql ALTER TABLE details DISTRIBUTED BY HASH(user_id, event_time) BUCKETS 10; ``` ##### Modify the tablet size[​](#modify-the-tablet-size "Direct link to Modify the tablet size") StarRocks supports splitting and merging tablets from v4.1 onwards, allowing dynamic management of tablet size, thus an adaptive mechanism for handling data skew. Syntax for SPLIT: ```sql ALTER TABLE SPLIT { TABLET | TABLETS } [ { PARTITION () | PARTITIONS (, , ...) } ｜ { () | (, , , ...) } ] [PROPERTIES ( "tablet_reshard_target_size"="") ] ``` Syntax for MERGE: ```sql ALTER TABLE MERGE { TABLET | TABLETS } [ { PARTITION () | PARTITIONS (, , ...) } ｜ { (, , ...) | (, , ...) (, , ...) ... } ] [PROPERTIES ( "tablet_reshard_target_size"="") ] ``` Parameter: * `tablet_reshard_target_size`: The target size of the tablets after the SPLIT or MERGE operation. Default: 10 GB. You do not need to specify this parameter if you have explicitly specified tablet IDs. * A tablet will be split if both the following conditions are met: * The size of the tablet is **larger** than `tablet_reshard_target_size`. * The number of tablets that are running tablet SPLIT or MERGE is less than the FE configuration `tablet_reshard_max_parallel_tablets` (Default: 10240). * A tablet will be merged if both the following conditions are met: * The total size of two adjacent tablets is **smaller** than `tablet_reshard_target_size`. * The number of tablets that are running tablet SPLIT or MERGE is less than the FE configuration `tablet_reshard_max_parallel_tablets` (Default: 10240). For detailed examples, see [Split or merge tablets](#split-or-merge-tablets). ##### Modify columns (add/delete columns, change column order, modify column comment)[​](#modify-columns-adddelete-columns-change-column-order-modify-column-comment "Direct link to Modify columns (add/delete columns, change column order, modify column comment)") ###### Add a column to the specified location of the specified index[​](#add-a-column-to-the-specified-location-of-the-specified-index "Direct link to Add a column to the specified location of the specified index") Syntax: ```sql ALTER TABLE [.] ADD COLUMN column_name column_type [KEY | agg_type] [DEFAULT "default_value"] [AFTER column_name|FIRST] [TO rollup_index_name] [PROPERTIES ("key"="value", ...)] ``` Note: 1. If you add a value column to an Aggregate table, you need to specify agg\_type. 2. If you add a key column to a non-Aggregate table (such as a Duplicate Key table), you need to specify the KEY keyword. On an Aggregate table, a column that specifies neither `agg_type` nor `KEY` is created as a key column, which changes the table's aggregation key and rewrites existing data. Set the FE configuration item `allow_implicit_key_column_in_agg_add_column` to `false` to reject such a statement instead, so that you are asked to specify one of the two. 3. You cannot add a column that already exists in the base index to the rollup. (You can recreate a rollup if needed.) ###### Add multiple columns to specified index[​](#add-multiple-columns-to-specified-index "Direct link to Add multiple columns to specified index") Syntax: * Add multiple columns ```sql ALTER TABLE [.] ADD COLUMN (column_name1 column_type [KEY | agg_type] DEFAULT "default_value", ...) [TO rollup_index_name] [PROPERTIES ("key"="value", ...)] ``` * Add multiple columns and use AFTER to specify locations of the added columns ```sql ALTER TABLE [.] ADD COLUMN column_name1 column_type [KEY | agg_type] DEFAULT "default_value" AFTER column_name, ADD COLUMN column_name2 column_type [KEY | agg_type] DEFAULT "default_value" AFTER column_name [, ...] [TO rollup_index_name] [PROPERTIES ("key"="value", ...)] ``` Note: 1. If you add a value column to an Aggregate table, you need to specify `agg_type`. 2. If you add a key column to a non-Aggregate table, you need to specify the KEY keyword. On an Aggregate table, a column that specifies neither `agg_type` nor `KEY` is created as a key column. Set the FE configuration item `allow_implicit_key_column_in_agg_add_column` to `false` to reject such a statement instead. 3. You cannot add a column that already exists in the base index to the rollup. (You can create another rollup if needed.) ###### Add a generated column (from v3.1)[​](#add-a-generated-column-from-v31 "Direct link to Add a generated column (from v3.1)") Syntax: ```sql ALTER TABLE [.] ADD COLUMN col_name data_type [NULL] AS generation_expr [COMMENT 'string'] ``` You can add a generated column and specify its expression. [The generated column](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) can be used to precompute and store the results of expressions, which significantly accelerates queries with the same complex expressions. Since v3.1, StarRocks supports generated columns. ###### Drop a column from specified index[​](#drop-a-column-from-specified-index "Direct link to Drop a column from specified index") Syntax: ```sql ALTER TABLE [.] DROP COLUMN column_name [FROM rollup_index_name]; ``` Note: 1. You cannot drop partition column. 2. If the column is dropped from the base index, it will also be dropped if it is included in the rollup. ###### Modify the column type, position, comment, and other properties[​](#modify-the-column-type-position-comment-and-other-properties "Direct link to Modify the column type, position, comment, and other properties") Syntax: ```sql ALTER TABLE [.] MODIFY COLUMN [ column_type [ KEY | agg_type ] ] [ NULL | NOT NULL ] [ DEFAULT ""] [ COMMENT "" ] [ AFTER | FIRST ] [ FROM rollup_index_name ] [ PROPERTIES ("key"="value", ...) ] ``` Note: 1. If you modify the value column in aggregation models, you need to specify agg\_type. 2. If you modify the key column in non-aggregation models, you need to specify the KEY keyword. 3. While modifying the type, default value, nullability, and position, you must specify the full definition of the column in the statement. 4. Modifying only the column comment — whether via `MODIFY COLUMN COMMENT ""` or via a full column definition where only the comment differs — changes only the metadata and does not initiate a Schema Change task. This applies to Primary Key columns, key columns, and regular columns. If the full definition also changes any other attribute of the column, the statement initiates a Schema Change task as usual. 5. The partition column cannot be modified. 6. The following types of conversions are currently supported (accuracy loss is guaranteed by the user). * Convert TINYINT/SMALLINT/INT/BIGINT to TINYINT/SMALLINT/INT/BIGINT/DOUBLE. * Convert TINYINT/SMALLINT/INT/BIGINT/LARGEINT/FLOAT/DOUBLE/DECIMAL to VARCHAR. VARCHAR supports modification of maximum length. * Convert VARCHAR to TINYINT/SMALLINT/INT/BIGINT/LARGEINT/FLOAT/DOUBLE. * Convert VARCHAR to DATE (currently support six formats: "%Y-%m-%d", "%y-%m-%d", "%Y%m%d", "%y%m%d", "%Y/%m/%d, "%y/%m/%d") * Convert DATETIME to DATE(only year-month-day information is retained, i.e. `2019-12-09 21:47:05` `<-->` `2019-12-09`) * Convert DATE to DATETIME (set hour, minute, second to zero, For example: `2019-12-09` `<-->` `2019-12-09 00:00:00`) * Convert FLOAT to DOUBLE * Convert INT to DATE (If the INT data fails to convert, the original data remains the same) 7. Conversion from NULL to NOT NULL is not supported. 8. You can modify several properties in a single MODIFY COLUMN clause. However, some combination of properties are not supported. ###### Reorder the columns of specified index[​](#reorder-the-columns-of-specified-index "Direct link to Reorder the columns of specified index") Syntax: ```sql ALTER TABLE [.] ORDER BY (column_name1, column_name2, ...) [FROM rollup_index_name] [PROPERTIES ("key"="value", ...)] ``` Note: * All columns in the index must be written. * The value column is listed after the key column. ###### Modify the sort key[​](#modify-the-sort-key "Direct link to Modify the sort key") Since v3.0, the sort keys for the Primary Key tables can be modified. v3.3 extends this support to Duplicate Key tables, Aggregate tables, and Unique Key tables. The sort keys in Duplicate Key tables and Primary Key tables can be combination of any sort columns. The sort keys in Aggregate tables and Unique Key tables must include all key columns, but the order of the columns does not need to be same as the key columns. Syntax: ```sql ALTER TABLE [.] [ order_desc ] order_desc ::= ORDER BY [, ...] ``` Example: modify the sort keys in Primary Key tables. For example, the original table is a Primary Key table where the sort key and the primary key are coupled, which is `dt, order_id`. ```sql create table orders ( dt date NOT NULL, order_id bigint NOT NULL, user_id int NOT NULL, merchant_id int NOT NULL, good_id int NOT NULL, good_name string NOT NULL, price int NOT NULL, cnt int NOT NULL, revenue int NOT NULL, state tinyint NOT NULL ) PRIMARY KEY (dt, order_id) PARTITION BY date_trunc('day', dt) DISTRIBUTED BY HASH(order_id); ``` Decouple the sort key from the primary key, and modify the sort key to `dt, revenue, state`. ```sql ALTER TABLE orders ORDER BY (dt, revenue, state); ``` ###### Modify a STRUCT column to add or drop a field[​](#modify-a-struct-column-to-add-or-drop-a-field "Direct link to Modify a STRUCT column to add or drop a field") Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) From v3.2.10 and v3.3.2 onwards, StarRocks supports modifying a STRUCT column to add or drop a field, which can be nested or within an ARRAY type. Syntax: ```sql -- Add a field ALTER TABLE [.] MODIFY COLUMN ADD FIELD field_path field_desc -- Drop a field ALTER TABLE [.] MODIFY COLUMN DROP FIELD field_path field_path ::= [ { . | [*]. } [ ... ] ] -- Note that here `[*]` as a whole is a pre-defined symbol and represents all elements in the ARRAY field -- when adding or removing a field in a STRUCT type nested within an ARRAY type. -- For detailed information, see the parameter description and examples of `field_path`. field_desc ::= [ AFTER | FIRST ] ``` Parameters: * `field_path`: The field to be added or removed. This can be a simple field name, indicating a top-dimension field, for example, `new_field_name`, or a Column Access Path that represents a nested field, for example, `lv1_k1.lv2_k2.[*].new_field_name`. * `[*]`: When a STRUCT type is nested within an ARRAY type, `[*]` represents all elements in the ARRAY field. It is used to add or remove a field in all STRUCT elements nested under the ARRAY field. * `prior_field_name`: The field preceding the newly added field. Used in conjunction with the AFTER keyword to specify the order of the new field. You do not need to specify this parameter if the FIRST keyword is used, indicating the new field should be the first field. The dimension of `prior_field_name` is determined by `field_path` (specifically, the part preceding `new_field_name`, that is, `level1_k1.level2_k2.[*]`) and does not need to be specified explicitly. Examples of `field_path`: * Add or drop a sub-field in a STRUCT field nested within a STRUCT column. Suppose there is a column `fx stuct>`. The syntax to add a `v3` field under `c2` is: ```sql ALTER TABLE tbl MODIFY COLUMN fx ADD FIELD c2.v3 INT ``` After the operation, the column becomes `fx stuct>`. * Add or drop a sub-field in every STRUCT field nested within a ARRAY field. Suppose there is a column `fx struct>>`. The field `c2` is an ARRAY type, which contains a STRUCT with two fields `v1` and `v2`. The syntax to add a `v3` field to the nested STRUCT is: ```sql ALTER TABLE tbl MODIFY COLUMN fx ADD FIELD c2.[*].v3 INT ``` After the operation, the column becomes `fx struct>>`. For more usage instructions, see [Example - Column -14](#column). note * Currently, this feature is only supported in shared-nothing clusters. * The table must have the `fast_schema_evolution` property enabled. * Modifying the Value type of a MAP subfield in a STRUCT type is not supported, regardless of whether the Value type is ARRAY, STRUCT, or MAP. * Newly added fields cannot have default values or attributes such as Nullable specified. They default to being Nullable, with a default value of null. * After this feature is used, downgrading the cluster directly to a version that does not support this feature is not allowed. ##### Modify rollup[​](#modify-rollup "Direct link to Modify rollup") ###### Create a rollup[​](#create-a-rollup "Direct link to Create a rollup") Syntax: ```sql ALTER TABLE [.] ADD ROLLUP rollup_name (column_name1, column_name2, ...) [FROM from_index_name] [PROPERTIES ("key"="value", ...)] ``` PROPERTIES: Support setting timeout time and the default timeout time is one day. Example: ```sql ALTER TABLE [.] ADD ROLLUP r1(col1,col2) from r0; ``` ###### Create rollups in batches[​](#create-rollups-in-batches "Direct link to Create rollups in batches") Syntax: ```sql ALTER TABLE [.] ADD ROLLUP [rollup_name (column_name1, column_name2, ...) [FROM from_index_name] [PROPERTIES ("key"="value", ...)],...]; ``` Example: ```sql ALTER TABLE [.] ADD ROLLUP r1(col1,col2) from r0, r2(col3,col4) from r0; ``` Note: 1. If from\_index\_name is not specified, then create from base index by default. 2. The columns in the rollup table must be existing columns in from\_index. 3. In properties, user can specify the storage format. See CREATE TABLE for details. ###### Drop a rollup[​](#drop-a-rollup "Direct link to Drop a rollup") Syntax: ```sql ALTER TABLE [.] DROP ROLLUP rollup_name [PROPERTIES ("key"="value", ...)]; ``` Example: ```sql ALTER TABLE [.] DROP ROLLUP r1; ``` ###### Batch drop rollups[​](#batch-drop-rollups "Direct link to Batch drop rollups") Syntax: ```sql ALTER TABLE [.] DROP ROLLUP [rollup_name [PROPERTIES ("key"="value", ...)],...]; ``` Example: ```sql ALTER TABLE [.] DROP ROLLUP r1, r2; ``` Note: You cannot drop the base index. ##### Modify indexes[​](#modify-indexes "Direct link to Modify indexes") You can create or drop the following indexes: * [Bitmap index](https://docs.starrocks.io/docs/table_design/indexes/Bitmap_index.md) * [N-Gram bloom filter index](https://docs.starrocks.io/docs/table_design/indexes/Ngram_Bloom_Filter_Index.md) * [Full-Text inverted index](https://docs.starrocks.io/docs/table_design/indexes/inverted_index.md) * [Vector index](https://docs.starrocks.io/docs/table_design/indexes/vector_index.md) ###### Create an index[​](#create-an-index "Direct link to Create an index") Syntax: ```sql ALTER TABLE [.] ADD INDEX index_name (column) [USING { BITMAP | NGRAMBF | GIN | VECTOR } ] [COMMENT ''] ``` For detailed instructions and examples on creating these indexes, see the corresponding tutorials listed above. ###### Drop an index[​](#drop-an-index "Direct link to Drop an index") Syntax: ```sql ALTER TABLE [.] DROP INDEX index_name; ``` ##### Modify table properties[​](#modify-table-properties "Direct link to Modify table properties") Syntax: ```sql ALTER TABLE [.] SET ("key" = "value") ``` Currently, StarRocks supports modifying the following table properties: * `replication_num` * `default.replication_num` * `default.storage_medium` * Dynamic partitioning related properties * `enable_persistent_index` * `bloom_filter_columns` * `colocate_with` * `bucket_size` (supported since 3.2) * `base_compaction_forbidden_time_ranges` (supported since v3.2.13) * `enable_statistic_auto_collect` note * In most cases, you are only allowed to modify one property at a time. You can only modify multiple properties at a time only if these properties have the same prefix. Currently, only `dynamic_partition.` and `binlog.` are supported. * You can also modify the properties by merging into the above operation on column. See the [following examples](#examples). ##### Swap[​](#swap "Direct link to Swap") Swap supports atomic exchange of two tables. Syntax: ```sql ALTER TABLE [.] SWAP WITH ; ``` note * Unique Key and Foreign Key constraints between OLAP tables will be validated during Swap to ensure that the constraints of the two tables being swapped are consistent. An error will be returned if inconsistencies are detected. If no inconsistencies are detected, Unique Key and Foreign Key constraints will be automatically swapped. * Materialized views that are depended on the tables being swapped will be automatically set to inactive, and their Unique Key and Foreign Key constraints will be removed and no longer available. ##### Manual compaction (from 3.1)[​](#manual-compaction-from-31 "Direct link to Manual compaction (from 3.1)") StarRocks uses a compaction mechanism to merge different versions of loaded data. This feature can combine small files into large files, which effectively improves query performance. Before v3.1, compaction is performed in two ways: * Automatic compaction by system: Compaction is performed at the BE level in the background. Users cannot specify database or table for compaction. * Users can perform compaction by calling an HTTP interface. Starting from v3.1, StarRocks offers a SQL interface for users to manually perform compaction by running SQL commands. They can choose a specific table or partition for compaction. This provides more flexibility and control over the compaction process. Shared-data clusters support this feature from v3.3.0 onwards. > **NOTE** > > From v3.2.13 onwards, you can forbid Base Compaction within certain time range using the property [`base_compaction_forbidden_time_ranges`](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md#forbid-base-compaction). Syntax: ```sql ALTER TABLE [ BASE | CUMULATIVE ] COMPACT [ | ( [, ...] ) ] ``` That is: ```sql -- Perform compaction on the entire table. ALTER TABLE COMPACT -- Perform compaction on a single partition. ALTER TABLE COMPACT -- Perform compaction on multiple partitions. ALTER TABLE COMPACT ([,,...]) -- Perform cumulative compaction. ALTER TABLE CUMULATIVE COMPACT ([,,...]) -- Perform base compaction. ALTER TABLE BASE COMPACT ([,,...]) ``` The `be_compactions` table in the `information_schema` database records compaction results. You can run `SELECT * FROM information_schema.be_compactions;` to query data versions after compaction. ##### Drop Primary Key Persistent Index (From 3.3.9)[​](#drop-primary-key-persistent-index-from-339 "Direct link to Drop Primary Key Persistent Index (From 3.3.9)") Syntax: ```sql ALTER TABLE [.] DROP PERSISTENT INDEX ON TABLETS([, , ...]); ``` > **NOTE** > > StarRocks only supports deleting persistent indexes for cloud-native Primary Key tables in shared-data clusters. #### Examples[​](#examples "Direct link to Examples") ##### Table[​](#table "Direct link to Table") 1. Modify the default number of replicas of the table, which is used as the default number of replicas for newly added partitions. ```sql ALTER TABLE example_db.my_table SET ("default.replication_num" = "2"); ``` 2. Modify the actual number of replicas of a single-partition table. ```sql ALTER TABLE example_db.my_table SET ("replication_num" = "3"); ``` 3. Modify data writing and replication mode among replicas. ```sql ALTER TABLE example_db.my_table SET ("replicated_storage" = "false"); ``` This example sets the data writing and replication mode among replicas to "leaderless replication", which means data is written to multiple replicas at the same time without differentiating primary and secondary replicas. For more information, see the `replicated_storage` parameter in [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md). ##### Partition[​](#partition "Direct link to Partition") 1. Add a partition and use the default bucketing mode. The existing partition is \[MIN, 2013-01-01). The added partition is \[2013-01-01, 2014-01-01). ```sql ALTER TABLE example_db.my_table ADD PARTITION p1 VALUES LESS THAN ("2014-01-01"); ``` 2. Add a partition and use the new number of buckets. ```sql ALTER TABLE example_db.my_table ADD PARTITION p1 VALUES LESS THAN ("2015-01-01") DISTRIBUTED BY HASH(k1); ``` 3. Add a partition and use the new number of replicas. ```sql ALTER TABLE example_db.my_table ADD PARTITION p1 VALUES LESS THAN ("2015-01-01") ("replication_num"="1"); ``` 4. Alter the number of replicas for a partition. ```sql ALTER TABLE example_db.my_table MODIFY PARTITION p1 SET("replication_num"="1"); ``` 5. Batch alter the number of replicas for specified partitions. ```sql ALTER TABLE example_db.my_table MODIFY PARTITION (p1, p2, p4) SET("replication_num"="1"); ``` 6. Batch alter the storage medium of all partitions. ```sql ALTER TABLE example_db.my_table MODIFY PARTITION (*) SET("storage_medium"="HDD"); ``` 7. Drop a partition. ```sql ALTER TABLE example_db.my_table DROP PARTITION p1; ``` 8. Add a partition that has upper and lower boundaries. ```sql ALTER TABLE example_db.my_table ADD PARTITION p1 VALUES [("2014-01-01"), ("2014-02-01")); ``` ##### Rollup[​](#rollup "Direct link to Rollup") 1. Create a rollup `example_rollup_index` based on the base index (k1,k2,k3,v1,v2). Column-based storage is used. ```sql ALTER TABLE example_db.my_table ADD ROLLUP example_rollup_index(k1, k3, v1, v2) PROPERTIES("storage_type"="column"); ``` 2. Create a rollup `example_rollup_index2` based on `example_rollup_index(k1,k3,v1,v2)`. ```sql ALTER TABLE example_db.my_table ADD ROLLUP example_rollup_index2 (k1, v1) FROM example_rollup_index; ``` 3. Create a rollup `example_rollup_index3` based on the base index (k1, k2, k3, v1). The rollup timeout time is set to one hour. ```sql ALTER TABLE example_db.my_table ADD ROLLUP example_rollup_index3(k1, k3, v1) PROPERTIES("storage_type"="column", "timeout" = "3600"); ``` 4. Drop the rollup `example_rollup_index2`. ```sql ALTER TABLE example_db.my_table DROP ROLLUP example_rollup_index2; ``` ##### Column[​](#column "Direct link to Column") 1. Add a key column `new_col` (non-aggregate column) after the `col1` column of `example_rollup_index`. ```sql ALTER TABLE example_db.my_table ADD COLUMN new_col INT KEY DEFAULT "0" AFTER col1 TO example_rollup_index; ``` 2. Add a value column `new_col` (non-aggregate column) after the `col1` column of `example_rollup_index`. ```sql ALTER TABLE example_db.my_table ADD COLUMN new_col INT DEFAULT "0" AFTER col1 TO example_rollup_index; ``` 3. Add a key column `new_col` (aggregate column) after the `col1` column of `example_rollup_index`. ```sql ALTER TABLE example_db.my_table ADD COLUMN new_col INT KEY DEFAULT "0" AFTER col1 TO example_rollup_index; ``` 4. Add a value column `new_col SUM` (aggregate column) after the `col1` column of `example_rollup_index`. ```sql ALTER TABLE example_db.my_table ADD COLUMN new_col INT SUM DEFAULT "0" AFTER col1 TO example_rollup_index; ``` 5. Add multiple columns to `example_rollup_index` (aggregate). ```sql ALTER TABLE example_db.my_table ADD COLUMN (col1 INT DEFAULT "1", col2 FLOAT SUM DEFAULT "2.3") TO example_rollup_index; ``` 6. Add multiple columns to `example_rollup_index` (aggregate) and specify the locations of the added columns using `AFTER`. ```sql ALTER TABLE example_db.my_table ADD COLUMN col1 INT DEFAULT "1" AFTER `k1`, ADD COLUMN col2 FLOAT SUM AFTER `v2` TO example_rollup_index; ``` 7. Drop a column from `example_rollup_index`. ```sql ALTER TABLE example_db.my_table DROP COLUMN col2 FROM example_rollup_index; ``` 8. Modify the column type of col1 of the base index to BIGINT and put it after `col2`. ```sql ALTER TABLE example_db.my_table MODIFY COLUMN col1 BIGINT DEFAULT "1" AFTER col2; ``` 9. Modify the maximum length of the `val1` column of the base index to 64. The original length is 32. ```sql ALTER TABLE example_db.my_table MODIFY COLUMN val1 VARCHAR(64) REPLACE DEFAULT "abc"; ``` 10. Reorder the columns in `example_rollup_index`. The original column order is k1, k2, k3, v1, v2. ```sql ALTER TABLE example_db.my_table ORDER BY (k3,k1,k2,v2,v1) FROM example_rollup_index; ``` 11. Perform two operations (ADD COLUMN and ORDER BY) at one time. ```sql ALTER TABLE example_db.my_table ADD COLUMN v2 INT MAX DEFAULT "0" AFTER k2 TO example_rollup_index, ORDER BY (k3,k1,k2,v2,v1) FROM example_rollup_index; ``` 12. Alter the bloomfilter columns of the table. ```sql ALTER TABLE example_db.my_table SET ("bloom_filter_columns"="k1,k2,k3"); ``` This operation can also be merged into the above column operation (note that the syntax of multiple clauses is slightly different). ```sql ALTER TABLE example_db.my_table DROP COLUMN col2 PROPERTIES ("bloom_filter_columns"="k1,k2,k3"); ``` 13. Modify the data type of multiple columns in a single statement. ```sql ALTER TABLE example_db.my_table MODIFY COLUMN k1 VARCHAR(100) KEY NOT NULL, MODIFY COLUMN v2 DOUBLE DEFAULT "1" AFTER v1; ``` 14. Add and drop fields in STRUCT-type data. **Prerequisites**: Create a table and insert a row of data. ```sql CREATE TABLE struct_test( c0 INT, c1 STRUCT, v3 INT>, c2 STRUCT>>> ) DUPLICATE KEY(c0) DISTRIBUTED BY HASH(`c0`) BUCKETS 1 PROPERTIES ( "fast_schema_evolution" = "true" ); INSERT INTO struct_test VALUES ( 1, ROW(1, ROW(2, 3), 4), ROW(5, [ROW(6, ROW(7, 8)), ROW(9, ROW(10, 11))]) ); ``` ```plain mysql> SELECT * FROM struct_test\G *************************** 1. row *************************** c0: 1 c1: {"v1":1,"v2":{"v4":2,"v5":3},"v3":4} c2: {"v1":5,"v2":[{"v3":6,"v4":{"v5":7,"v6":8}},{"v3":9,"v4":{"v5":10,"v6":11}}]} ``` * Add a new field to a STRUCT-type column. ```sql ALTER TABLE struct_test MODIFY COLUMN c1 ADD FIELD v4 INT AFTER v2; ``` ```plain mysql> SELECT * FROM struct_test\G *************************** 1. row *************************** c0: 1 c1: {"v1":1,"v2":{"v4":2,"v5":3},"v4":null,"v3":4} c2: {"v1":5,"v2":[{"v3":6,"v4":{"v5":7,"v6":8}},{"v3":9,"v4":{"v5":10,"v6":11}}]} ``` * Add a new field to a nested STRUCT type. ```sql ALTER TABLE struct_test MODIFY COLUMN c1 ADD FIELD v2.v6 INT FIRST; ``` ```plain mysql> SELECT * FROM struct_test\G *************************** 1. row *************************** c0: 1 c1: {"v1":1,"v2":{"v6":null,"v4":2,"v5":3},"v4":null,"v3":4} c2: {"v1":5,"v2":[{"v3":6,"v4":{"v5":7,"v6":8}},{"v3":9,"v4":{"v5":10,"v6":11}}]} ``` * Add a new field to a STRUCT type in an array. ```sql ALTER TABLE struct_test MODIFY COLUMN c2 ADD FIELD v2.[*].v7 INT AFTER v3; ``` ```plain mysql> SELECT * FROM struct_test\G *************************** 1. row *************************** c0: 1 c1: {"v1":1,"v2":{"v6":null,"v4":2,"v5":3},"v4":null,"v3":4} c2: {"v1":5,"v2":[{"v3":6,"v7":null,"v4":{"v5":7,"v6":8}},{"v3":9,"v7":null,"v4":{"v5":10,"v6":11}}]} ``` * Drop a field from a STRUCT-type column. ```sql ALTER TABLE struct_test MODIFY COLUMN c1 DROP FIELD v3; ``` ```plain mysql> SELECT * FROM struct_test\G *************************** 1. row *************************** c0: 1 c1: {"v1":1,"v2":{"v6":null,"v4":2,"v5":3},"v4":null} c2: {"v1":5,"v2":[{"v3":6,"v7":null,"v4":{"v5":7,"v6":8}},{"v3":9,"v7":null,"v4":{"v5":10,"v6":11}}]} ``` * Drop a field from a nested STRUCT type. ```sql ALTER TABLE struct_test MODIFY COLUMN c1 DROP FIELD v2.v4; ``` ```plain mysql> SELECT * FROM struct_test\G *************************** 1. row *************************** c0: 1 c1: {"v1":1,"v2":{"v6":null,"v5":3},"v4":null} c2: {"v1":5,"v2":[{"v3":6,"v7":null,"v4":{"v5":7,"v6":8}},{"v3":9,"v7":null,"v4":{"v5":10,"v6":11}}]} ``` * Drop a field from a STRUCT type in an array. ```sql ALTER TABLE struct_test MODIFY COLUMN c2 DROP FIELD v2.[*].v3; ``` ```plain mysql> SELECT * FROM struct_test\G *************************** 1. row *************************** c0: 1 c1: {"v1":1,"v2":{"v6":null,"v5":3},"v4":null} c2: {"v1":5,"v2":[{"v7":null,"v4":{"v5":7,"v6":8}},{"v7":null,"v4":{"v5":10,"v6":11}}]} ``` ##### Table property[​](#table-property "Direct link to Table property") 1. Alter the Colocate property of the table. ```sql ALTER TABLE example_db.my_table SET ("colocate_with" = "t1"); ``` 2. Alter the dynamic partition property of the table. ```sql ALTER TABLE example_db.my_table SET ("dynamic_partition.enable" = "false"); ``` If you need to add dynamic partition properties to a table with no dynamic partition properties configured, you need to specify all the dynamic partition properties. ```sql ALTER TABLE example_db.my_table SET ( "dynamic_partition.enable" = "true", "dynamic_partition.time_unit" = "DAY", "dynamic_partition.end" = "3", "dynamic_partition.prefix" = "p", "dynamic_partition.buckets" = "32" ); ``` 3. Alter the storage medium property of the table. ```sql ALTER TABLE example_db.my_table SET("default.storage_medium"="SSD"); ``` ##### Rename[​](#rename-1 "Direct link to Rename") 1. Rename `table1` to `table2`. ```sql ALTER TABLE table1 RENAME table2; ``` 2. Rename rollup `rollup1` of `example_table` to `rollup2`. ```sql ALTER TABLE example_table RENAME ROLLUP rollup1 rollup2; ``` 3. Rename partition `p1` of `example_table` to `p2`. ```sql ALTER TABLE example_table RENAME PARTITION p1 p2; ``` ##### Index[​](#index "Direct link to Index") 1. Create a bitmap index for column `siteid` in `table1`. ```sql ALTER TABLE table1 ADD INDEX index_1 (siteid) USING BITMAP COMMENT 'site_id_bitmap'; ``` 2. Drop the bitmap index of column `siteid` in `table1`. ```sql ALTER TABLE table1 DROP INDEX index_1; ``` ##### Swap[​](#swap-1 "Direct link to Swap") Atomic swap between `table1` and `table2`. ```sql ALTER TABLE table1 SWAP WITH table2 ``` ##### Manual compaction[​](#manual-compaction "Direct link to Manual compaction") ```sql CREATE TABLE compaction_test( event_day DATE, pv BIGINT) DUPLICATE KEY(event_day) PARTITION BY date_trunc('month', event_day) DISTRIBUTED BY HASH(event_day) BUCKETS 8 PROPERTIES("replication_num" = "3"); INSERT INTO compaction_test VALUES ('2023-02-14', 2), ('2033-03-01',2); {'label':'insert_734648fa-c878-11ed-90d6-00163e0dcbfc', 'status':'VISIBLE', 'txnId':'5008'} INSERT INTO compaction_test VALUES ('2023-02-14', 2),('2033-03-01',2); {'label':'insert_85c95c1b-c878-11ed-90d6-00163e0dcbfc', 'status':'VISIBLE', 'txnId':'5009'} ALTER TABLE compaction_test COMPACT; ALTER TABLE compaction_test COMPACT p203303; ALTER TABLE compaction_test COMPACT (p202302,p203303); ALTER TABLE compaction_test CUMULATIVE COMPACT (p202302,p203303); ALTER TABLE compaction_test BASE COMPACT (p202302,p203303); ``` ##### Drop Primary Key Persistent Index[​](#drop-primary-key-persistent-index "Direct link to Drop Primary Key Persistent Index") Drop persistent index on tablets `100` and `101` for Primary Key table `db1.test_tbl` in a shared-data cluster. ```sql ALTER TABLE db1.test_tbl DROP PERSISTENT INDEX ON TABLETS (100, 101); ``` ##### SPLIT or MERGE tablets[​](#split-or-merge-tablets "Direct link to SPLIT or MERGE tablets") * Split all tablets that meet the conditions in the table to a target size of 10 GB (Default). ```sql ALTER TABLE table1 SPLIT TABLETS; ``` * Split all tablets that meet the conditions in a partition. ```sql ALTER TABLE table1 SPLIT TABLETS PARTITION (p1); ``` * Split specific tablets by ID. ```sql ALTER TABLE table1 SPLIT TABLETS (9588955, 9588956, 9588957); ``` * Merge all tablets that meet the conditions in the table to a target size of 2 GB. ```sql ALTER TABLE table1 MERGE TABLETS PROPERTIES ( "tablet_reshard_target_size"="2147483648"); ``` * Merge all tablets that meet the conditions in specific partitions. ```sql ALTER TABLE table1 MERGE TABLETS PARTITIONS (p1, p2, p3) PROPERTIES ( "tablet_reshard_target_size"="2147483648"); ``` * Merge specific tablets by ID. ```sql ALTER TABLE table1 MERGE TABLETS (9588955, 9588956, 9588957) (9588958, 9588959); ``` #### References[​](#references "Direct link to References") * [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) * [SHOW CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_CREATE_TABLE.md) * [SHOW TABLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLES.md) * [SHOW ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_ALTER.md) * [DROP TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md) --- ### AUTO_INCREMENT Since version 3.0, StarRocks supports the `AUTO_INCREMENT` column attribute, which can simplify data management. This topic introduces the application scenarios, usage and features of the `AUTO_INCREMENT` column attribute. #### Introduction[​](#introduction "Direct link to Introduction") When a new data row is loaded into a table and values are not specified for the `AUTO_INCREMENT` column, StarRocks automatically assigns an integer value for the row's `AUTO_INCREMENT` column as its unique ID across the table. The subsequent values for the `AUTO_INCREMENT` column automatically increase at a specific step starting from the ID of the row. An `AUTO_INCREMENT` column can be used to simplify data management and speed up some queries. Here are some application scenarios of an `AUTO_INCREMENT` column: * Serve as primary keys: An `AUTO_INCREMENT` column can be used as the primary key to ensure that each row has a unique ID and make it easy to query and manage data. * Join tables: When multiple tables are joined, an `AUTO_INCREMENT` column can be used as the Join Key, which can expedite queries compared to using a column whose data type is STRING, for example, UUID. * Count the number of distinct values in a high-cardinality column: An `AUTO_INCREMENT` column can be used to represent the unique value column in a dictionary. Compared to directly counting distinct STRING values, counting distinct integer values of the `AUTO_INCREMENT` column can sometimes improve the query speed by several times or even tens of times. You need to specify an `AUTO_INCREMENT` column in the CREATE TABLE statement. The data types of an `AUTO_INCREMENT` column must be BIGINT. The value for an AUTO\_INCREMENT column can be [implicitly assigned or explicitly specified](#assign-values-for-auto_increment-column). It starts from 1, and increments by 1 for each new row. #### Basic operations[​](#basic-operations "Direct link to Basic operations") ##### Specify `AUTO_INCREMENT` column at table creation[​](#specify-auto_increment-column-at-table-creation "Direct link to specify-auto_increment-column-at-table-creation") Create a table named `test_tbl1` with two columns, `id` and `number`. Specify the column `number` as the `AUTO_INCREMENT` column. ```sql CREATE TABLE test_tbl1 ( id BIGINT NOT NULL, number BIGINT NOT NULL AUTO_INCREMENT ) PRIMARY KEY (id) DISTRIBUTED BY HASH(id) PROPERTIES("replicated_storage" = "true"); ``` ##### Assign values for `AUTO_INCREMENT` column[​](#assign-values-for-auto_increment-column "Direct link to assign-values-for-auto_increment-column") ###### Assign values implicitly[​](#assign-values-implicitly "Direct link to Assign values implicitly") When you load data into a StarRocks table, you do not need to specify the values for the `AUTO_INCREMENT` column. StarRocks automatically assigns unique integer values for that column and inserts them into the table. ```sql INSERT INTO test_tbl1 (id) VALUES (1); INSERT INTO test_tbl1 (id) VALUES (2); INSERT INTO test_tbl1 (id) VALUES (3),(4),(5); ``` View data in the table. ```sql mysql > SELECT * FROM test_tbl1 ORDER BY id; +------+--------+ | id | number | +------+--------+ | 1 | 1 | | 2 | 2 | | 3 | 3 | | 4 | 4 | | 5 | 5 | +------+--------+ 5 rows in set (0.02 sec) ``` When you load data into a StarRocks table, you can also specify the values as `DEFAULT` for the `AUTO_INCREMENT` column. StarRocks automatically assigns unique integer values for that column and inserts them into the table. ```sql INSERT INTO test_tbl1 (id, number) VALUES (6, DEFAULT); ``` View data in the table. ```sql mysql > SELECT * FROM test_tbl1 ORDER BY id; +------+--------+ | id | number | +------+--------+ | 1 | 1 | | 2 | 2 | | 3 | 3 | | 4 | 4 | | 5 | 5 | | 6 | 6 | +------+--------+ 6 rows in set (0.02 sec) ``` In actual usage, the following result may be returned when you view the data in the table. This is because StarRocks cannot guarantee that the values for the `AUTO_INCREMENT` column are strictly monotonic. But StarRocks can guarantee that the values roughly increase in chronological order. For more information, see [Monotonicity](#monotonicity). ```sql mysql > SELECT * FROM test_tbl1 ORDER BY id; +------+--------+ | id | number | +------+--------+ | 1 | 1 | | 2 | 100001 | | 3 | 200001 | | 4 | 200002 | | 5 | 200003 | | 6 | 200004 | +------+--------+ 6 rows in set (0.01 sec) ``` ###### Specify values explicitly[​](#specify-values-explicitly "Direct link to Specify values explicitly") You can also explicitly specify the values for the `AUTO_INCREMENT` column and insert them into the table. ```sql INSERT INTO test_tbl1 (id, number) VALUES (7, 100); -- view data in the table. mysql > SELECT * FROM test_tbl1 ORDER BY id; +------+--------+ | id | number | +------+--------+ | 1 | 1 | | 2 | 100001 | | 3 | 200001 | | 4 | 200002 | | 5 | 200003 | | 6 | 200004 | | 7 | 100 | +------+--------+ 7 rows in set (0.01 sec) ``` Moreover, explicitly specifying values does not affect the subsequent values generated by StarRocks for newly inserted data rows. ```sql INSERT INTO test_tbl1 (id) VALUES (8); -- view data in the table. mysql > SELECT * FROM test_tbl1 ORDER BY id; +------+--------+ | id | number | +------+--------+ | 1 | 1 | | 2 | 100001 | | 3 | 200001 | | 4 | 200002 | | 5 | 200003 | | 6 | 200004 | | 7 | 100 | | 8 | 2 | +------+--------+ 8 rows in set (0.01 sec) ``` **NOTICE** We recommend that you do not use implicitly assigned values and explicitly specified values for the `AUTO_INCREMENT` column at the same time. Because the specified values may be the same as the values generated by StarRocks, breaking the [global uniqueness of auto-incremented IDs](#uniqueness). #### Basic features[​](#basic-features "Direct link to Basic features") ##### Uniqueness[​](#uniqueness "Direct link to Uniqueness") In general, StarRocks guarantees that the values for a `AUTO_INCREMENT` column are globally unique across a table. We recommend that you do not implicitly assign and explicitly specify the values for the `AUTO_INCREMENT` column at the same time. If you do so, it may break the global uniqueness of auto-incremented IDs. Here is a simple example: Create a table named `test_tbl2` with two columns, `id` and `number`. Specify the column `number` as the `AUTO_INCREMENT` column. ```sql CREATE TABLE test_tbl2 ( id BIGINT NOT NULL, number BIGINT NOT NULL AUTO_INCREMENT ) PRIMARY KEY (id) DISTRIBUTED BY HASH(id) PROPERTIES("replicated_storage" = "true"); ``` Implicitly assign and explicitly specify the values for the `AUTO_INCREMENT` column `number` in the table `test_tbl2`. ```sql INSERT INTO test_tbl2 (id, number) VALUES (1, DEFAULT); INSERT INTO test_tbl2 (id, number) VALUES (2, 2); INSERT INTO test_tbl2 (id) VALUES (3); ``` Query the table `test_tbl2`. ```sql mysql > SELECT * FROM test_tbl2 ORDER BY id; +------+--------+ | id | number | +------+--------+ | 1 | 1 | | 2 | 2 | | 3 | 100001 | +------+--------+ 3 rows in set (0.08 sec) ``` ##### Monotonicity[​](#monotonicity "Direct link to Monotonicity") In order to improve the performance of allocating auto-incremented IDs, BEs cache some auto-incremented IDs locally. In this situation, StarRocks cannot guarantee that the values for the `AUTO_INCREMENT` column are strictly monotonic. It can only be ensured that the values roughly increase in chronological order. > **NOTE** > > The number of auto-incremented IDs cached by the BEs is determined by the FE dynamic parameter `auto_increment_cache_size`, which defaults to `100,000`. You can modify the value by using `ADMIN SET FRONTEND CONFIG ("auto_increment_cache_size" = "xxx");` For example, a StarRocks cluster has one FE node and two BE nodes. Create a table named `test_tbl3` and insert five rows of data as follows: ```sql CREATE TABLE test_tbl3 ( id BIGINT NOT NULL, number BIGINT NOT NULL AUTO_INCREMENT ) PRIMARY KEY (id) DISTRIBUTED BY HASH(id) PROPERTIES("replicated_storage" = "true"); INSERT INTO test_tbl3 VALUES (1, DEFAULT); INSERT INTO test_tbl3 VALUES (2, DEFAULT); INSERT INTO test_tbl3 VALUES (3, DEFAULT); INSERT INTO test_tbl3 VALUES (4, DEFAULT); INSERT INTO test_tbl3 VALUES (5, DEFAULT); ``` The auto-incremented IDs in the table `test_tbl3` do not monotonically increase, because the two BE nodes cache auto-incremented IDs, \[1, 100000] and \[100001, 200000], respectively. When data is loaded by using multiple INSERT statements, the data is sent to different BE nodes which allocate auto-incremented IDs independently. Therefore, it cannot be guaranteed that auto-incremented IDs are strictly monotonic. ```sql mysql > SELECT * FROM test_tbl3 ORDER BY id; +------+--------+ | id | number | +------+--------+ | 1 | 1 | | 2 | 100001 | | 3 | 200001 | | 4 | 2 | | 5 | 100002 | +------+--------+ 5 rows in set (0.07 sec) ``` #### Partial updates and `AUTO_INCREMENT` column[​](#partial-updates-and-auto_increment-column "Direct link to partial-updates-and-auto_increment-column") This section explains how to update only a few specified columns in a table that contains an `AUTO_INCREMENT` column. > **NOTE** > > Currently, only Primary Key tables support partial updates. ##### `AUTO_INCREMENT` column is primary key[​](#auto_increment-column-is-primary-key "Direct link to auto_increment-column-is-primary-key") You need to specify the primary key during partial updates. Therefore, if the `AUTO_INCREMENT` column is the primary key or part of the primary key, the user behavior for partial updates is exactly the same as when the `AUTO_INCREMENT` column is not defined. 1. Create a table `test_tbl4` in the database `example_db` and insert one data row. ```sql -- Create a table. CREATE TABLE test_tbl4 ( id BIGINT AUTO_INCREMENT, name BIGINT NOT NULL, job1 BIGINT NOT NULL, job2 BIGINT NOT NULL ) PRIMARY KEY (id, name) DISTRIBUTED BY HASH(id) PROPERTIES("replicated_storage" = "true"); -- Prepared data. mysql > INSERT INTO test_tbl4 (id, name, job1, job2) VALUES (0, 0, 1, 1); Query OK, 1 row affected (0.04 sec) {'label':'insert_6af28e77-7d2b-11ed-af6e-02424283676b', 'status':'VISIBLE', 'txnId':'152'} -- Query the table. mysql > SELECT * FROM test_tbl4 ORDER BY id; +------+------+------+------+ | id | name | job1 | job2 | +------+------+------+------+ | 0 | 0 | 1 | 1 | +------+------+------+------+ 1 row in set (0.01 sec) ``` 2. Prepare the CSV file **my\_data4.csv** to update table `test_tbl4`. The CSV file includes values for the `AUTO_INCREMENT` column and does not include values for the column `job1`. The primary key of the first row already exists in table `test_tbl4`, while the primary key of the second row does not exist in the table. ```plaintext 0,0,99 1,1,99 ``` 3. Run a [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) job and use the CSV file to update table `test_tbl4`. ```bash curl --location-trusted -u : -H "label:1" \ -H "column_separator:," \ -H "partial_update:true" \ -H "columns:id,name,job2" \ -T my_data4.csv -XPUT \ http://:/api/example_db/test_tbl4/_stream_load ``` 4. Query the updated table. The first row of data already exists in table `test_tbl4`, and the value for the column `job1` remains unchanged. The second row of data is newly inserted, and because the default value for the column `job1` is not specified, the partial update framework directly sets the value for this column to `0`. ```sql mysql > SELECT * FROM test_tbl4 ORDER BY id; +------+------+------+------+ | id | name | job1 | job2 | +------+------+------+------+ | 0 | 0 | 1 | 99 | | 1 | 1 | 0 | 99 | +------+------+------+------+ 2 rows in set (0.01 sec) ``` ##### `AUTO_INCREMENT` column is not primary key[​](#auto_increment-column-is-not-primary-key "Direct link to auto_increment-column-is-not-primary-key") If the `AUTO_INCREMENT` column is not a primary key or a part of the primary key, and auto-incremented IDs are not provided in a Stream Load job, the following situations occur: * If the row already exists in the table, StarRocks does not update the auto-incremented ID. * If the row is newly loaded into the table, StarRocks generates a new auto-incremented ID. This feature can be used to build a dictionary table for quickly computing distinct STRING values. 1. In the database `example_db`, create a table `test_tbl5` and specify the column `job1` as the `AUTO_INCREMENT` column and insert a data row into the table `test_tbl5`. ```sql -- Create a table. CREATE TABLE test_tbl5 ( id BIGINT NOT NULL, name BIGINT NOT NULL, job1 BIGINT NOT NULL AUTO_INCREMENT, job2 BIGINT NOT NULL ) PRIMARY KEY (id, name) DISTRIBUTED BY HASH(id) PROPERTIES("replicated_storage" = "true"); -- Prepare data. mysql > INSERT INTO test_tbl5 VALUES (0, 0, -1, -1); Query OK, 1 row affected (0.04 sec) {'label':'insert_458d9487-80f6-11ed-ae56-aa528ccd0ebf', 'status':'VISIBLE', 'txnId':'94'} -- Query the table. mysql > SELECT * FROM test_tbl5 ORDER BY id; +------+------+------+------+ | id | name | job1 | job2 | +------+------+------+------+ | 0 | 0 | -1 | -1 | +------+------+------+------+ 1 row in set (0.01 sec) ``` 2. Prepare a CSV file **my\_data5.csv** to update table `test_tbl5`. The CSV file does not contain values for the `AUTO_INCREMENT` column `job1`. The primary key of the first row already exists in the table while the primary keys of the second and third rows do not. ```plaintext 0,0,99 1,1,99 2,2,99 ``` 3. Run a [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) job to load data from the CSV file into table `test_tbl5`. ```bash curl --location-trusted -u : -H "label:2" \ -H "column_separator:," \ -H "partial_update:true" \ -H "columns: id,name,job2" \ -T my_data5.csv -XPUT \ http://:/api/example_db/test_tbl5/_stream_load ``` 4. Query the updated table. The first row of data already exists in table `test_tbl5`, so the `AUTO_INCREMENT` column `job1` retains its original value. The second and third rows of data are newly inserted, so StarRocks generate new values for the `AUTO_INCREMENT` column `job1`. ```sql mysql > SELECT * FROM test_tbl5 ORDER BY id; +------+------+--------+------+ | id | name | job1 | job2 | +------+------+--------+------+ | 0 | 0 | -1 | 99 | | 1 | 1 | 1 | 99 | | 2 | 2 | 100001 | 99 | +------+------+--------+------+ 3 rows in set (0.01 sec) ``` #### Usage Notes[​](#usage-notes "Direct link to Usage Notes") By default, the system will allow the optimizer to choose the `COUNT(DISTINCT)` implementation. When deduplicating low- and medium-cardinality columns, you can set `count_distinct_implementation` to `multi_count_distinct` for a query via query hint to verify the `multi_distinct_count` implementation for precise counting. However, do not use the `multi_distinct_count` implementation when deduplicating high-cardinality columns, because its HashSet status and final merging can cause excessive memory consumption and even OOM. #### Limits[​](#limits "Direct link to Limits") * When a table with an `AUTO_INCREMENT` column is created, `'replicated_storage' = 'true'` must be set to ensure that all replicas have the same auto-incremented IDs. * Each table can have only one `AUTO_INCREMENT` column. * The data type of the `AUTO_INCREMENT` column must be BIGINT. * The `AUTO_INCREMENT` column must be `NOT NULL` and does not have a default value. * You can delete data from a Primary Key table with an `AUTO_INCREMENT` column. However, if the `AUTO_INCREMENT` column is not the primary key or part of the primary key, you need to note the following limits when you delete data in the following scenarios: * During the DELETE operation, there is also a load job for partial updates, which only contains UPSERT operations. If both the UPSERT and DELETE operations hit the same data row and the UPSERT operation is executed after the DELETE operation, the UPSERT operation may not take effect. * There is a load job for partial updates, which includes multiple UPSERT and DELETE operations on the same data row. If a certain UPSERT operation is executed after the DELETE operation, the UPSERT operation may not take effect. * Adding the `AUTO_INCREMENT` attribute by using ALTER TABLE is not supported. * Since version 3.1, StarRocks's shared-data mode supports the `AUTO_INCREMENT` attribute. * StarRocks does not support specifying the starting value and step size for the `AUTO_INCREMENT` column. #### Keywords[​](#keywords "Direct link to Keywords") AUTO\_INCREMENT, AUTO INCREMENT --- ### CANCEL ALTER TABLE CANCEL ALTER TABLE cancels the execution of the ongoing ALTER TABLE operation, including: * Modify columns. * Optimize table schema (from v3.2), including modifying the bucketing method and the number of buckets. * Create and delete the rollup index. note * This statement is a synchronous operation. * This statement requires you to have the `ALTER_PRIV` privilege on the table. * This statement only supports canceling asynchronous operations using ALTER TABLE (as mentioned above) and does not support canceling synchronous operations using ALTER TABLE, such as rename. #### Syntax[​](#syntax "Direct link to Syntax") ```sql CANCEL ALTER TABLE { COLUMN | OPTIMIZE | ROLLUP } FROM [db_name.]table_name [ (rollup_job_id [, rollup_job_id]) ] ``` #### Parameters[​](#parameters "Direct link to Parameters") * `{COLUMN ｜ OPTIMIZE | ROLLUP}` * If `COLUMN` is specified, this statement cancels operations of modifying columns. * If `OPTIMIZE` is specified, this statement cancels operations of optimizing table schema. * If `ROLLUP` is specified, this statement cancels operations of adding or deleting the rollup index. You can further specify `rollup_job_id` to cancel specific rollup jobs. * `db_name`: optional. The name of the database to which the table belongs. If this parameter is not specified, your current database is used by default. * `table_name`: required. The table name. * `rollup_job_id`: optional. The ID of the rollup job. You can get the rollup job ID using [SHOW ALTER MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/SHOW_ALTER_MATERIALIZED_VIEW.md). #### Examples[​](#examples "Direct link to Examples") 1. Cancel the operation of modifying columns for `example_table` in the database `example_db`. ```sql CANCEL ALTER TABLE COLUMN FROM example_db.example_table; ``` 2. Cancel the operation of optimizing table schema for `example_table` in the database `example_db`. ```sql CANCEL ALTER TABLE OPTIMIZE FROM example_db.example_table; ``` 3. Cancel the operation of adding or deleting the rollup index for `example_table` in the current database. ```sql CANCEL ALTER TABLE ROLLUP FROM example_table; ``` 4. Cancel specific rollup alterations for `example_table` in the current database using their job IDs. ```sql CANCEL ALTER TABLE ROLLUP FROM example_table (12345, 12346); ``` --- ### CREATE INDEX Creates indexes. You can create the following indexes: * [Bitmap index](https://docs.starrocks.io/docs/table_design/indexes/Bitmap_index.md) * [N-Gram bloom filter index](https://docs.starrocks.io/docs/table_design/indexes/Ngram_Bloom_Filter_Index.md) * [Full-Text inverted index](https://docs.starrocks.io/docs/table_design/indexes/inverted_index.md) * [Vector index](https://docs.starrocks.io/docs/table_design/indexes/vector_index.md) For detailed instructions and examples on creating these indexes, see the corresponding tutorials listed above. tip This operation requires the ALTER privilege on the target table. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. #### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE INDEX index_name ON table_name (column_name) [USING { BITMAP | NGRAMBF | GIN | VECTOR } ] [(index_property)] [COMMENT ''] ``` #### Parameter[​](#parameter "Direct link to Parameter") | **Parameter** | **Required** | **Description** | | --------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | index\_name | Yes | The index name. For naming conventions, see [System Limits](https://docs.starrocks.io/docs/sql-reference/System_limit.md). | | table\_name | Yes | The name of the table. | | column\_name | Yes | The name of the column to build index on. One column can have only one index. If a column already has an index, you cannot create one more index on it. | | USING | No | The type of the index to create. Valid values: - BITMAP (Default)
- NGRAMBF
- GIN
- VECTOR | | index\_property | No | The properties of the index to create. For `NGRAMBF`, `GIN`, and `VECTOR`, you must specify the corresponding properties. For detailed instructions, see the corresponding tutorials. | | COMMENT | No | The comment for the index. | #### Examples[​](#examples "Direct link to Examples") Create a table `sales_records` as follows: ```sql CREATE TABLE sales_records ( record_id int, seller_id int, item_id int ) DISTRIBUTED BY hash(record_id) PROPERTIES ( "replication_num" = "3" ); ``` Create a bitmap index `index3` on the `item_id` column of `sales_records`. ```sql CREATE INDEX index3 ON sales_records (item_id) USING BITMAP COMMENT ''; ``` #### Relevant SQLs[​](#relevant-sqls "Direct link to Relevant SQLs") * [SHOW INDEX](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_INDEX.md) * [DROP INDEX](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_INDEX.md) --- ### CREATE TABLE Create a new table in StarRocks. tip This operation requires the CREATE TABLE privilege on the destination database. #### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE [EXTERNAL] [TEMPORARY] TABLE [IF NOT EXISTS] [database.]table_name (column_definition1[, column_definition2, ...] [, index_definition1[, index_definition12,]]) [ENGINE = [olap|mysql|elasticsearch|hive|hudi|iceberg|jdbc]] [key_desc] [COMMENT "table comment"] [partition_desc] [distribution_desc] [rollup_index] [ORDER BY (column_name1,...)] [PROPERTIES ("key"="value", ...)] ``` tip * The table name, partition name, column name, and index name you create must follow the naming conventions in [System limits](https://docs.starrocks.io/docs/sql-reference/System_limit.md). * When you specify database name, table name, column name, or partition name, note that some literals are used as reserved keywords in StarRocks. Do not directly use these keywords in SQL statements. If you want to use such a keyword in an SQL statement, enclose it in a pair of backticks (\`). See [Keywords](https://docs.starrocks.io/docs/sql-reference/sql-statements/keywords.md) for these reserved keywords. #### Keywords[​](#keywords "Direct link to Keywords") ##### `EXTERNAL`[​](#external "Direct link to external") caution The `EXTERNAL` keyword is deprecated. We recommend that you use [external catalogs](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md) to query data from Hive, Iceberg, Hudi, and JDBC data sources instead of using the `EXTERNAL` keyword to create external tables. tip **Recommendation** From v3.1 onwards, StarRocks supports creating Parquet-formatted tables in Iceberg catalogs, and supports sinking data to these Parquet-formatted Iceberg tables by using INSERT INTO. From v3.2 onwards, StarRocks supports creating Parquet-formatted tables in Hive catalogs, and supports sinking data to these Parquet-formatted Hive tables by using INSERT INTO. From v3.3 onwards, StarRocks supports creating ORC- and Textfile-formatted tables in Hive catalogs, and supports sinking data to these ORC- and Textfile-formatted Hive tables by using INSERT INTO. If you would like to use the deprecated `EXTERNAL` keyword, please expand **`EXTERNAL` keyword details** `EXTERNAL` keyword details To create an external table to query external data sources, specify `CREATE EXTERNAL TABLE` and set `ENGINE` to any of these values. You can refer to [External table](https://docs.starrocks.io/docs/data_source/External_table.md) for more information. * For MySQL external tables, specify the following properties: ```plaintext PROPERTIES ( "host" = "mysql_server_host", "port" = "mysql_server_port", "user" = "your_user_name", "password" = "your_password", "database" = "database_name", "table" = "table_name" ) ``` Note: "table\_name" in MySQL should indicate the real table name. In contrast, "table\_name" in CREATE TABLE statement indicates the name of this MySQL table on StarRocks. They can either be different or the same. The aim of creating MySQL tables in StarRocks is to access MySQL database. StarRocks itself does not maintain or store any MySQL data. * For Elasticsearch external tables, specify the following properties: ```plaintext PROPERTIES ( "hosts" = "http://192.168.xx.xx:8200,http://192.168.xx0.xx:8200", "user" = "root", "password" = "root", "index" = "tindex", "type" = "doc" ) ``` * `hosts`: the URL that is used to connect your Elasticsearch cluster. You can specify one or more URLs. * `user`: the account of the root user that is used to log in to your Elasticsearch cluster for which basic authentication is enabled. * `password`: the password of the preceding root account. * `index`: the index of the StarRocks table in your Elasticsearch cluster. The index name is the same as the StarRocks table name. You can set this parameter to the alias of the StarRocks table. * `type`: the type of index. The default value is `doc`. * For Hive external tables, specify the following properties: ```plaintext PROPERTIES ( "database" = "hive_db_name", "table" = "hive_table_name", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ) ``` Here, database is the name of the corresponding database in Hive table. Table is the name of Hive table. `hive.metastore.uris` is the server address. * For JDBC external tables, specify the following properties: ```plaintext PROPERTIES ( "resource"="jdbc0", "table"="dest_tbl" ) ``` `resource` is the JDBC resource name and `table` is the destination table. * For Iceberg external tables, specify the following properties: ```plaintext PROPERTIES ( "resource" = "iceberg0", "database" = "iceberg", "table" = "iceberg_table" ) ``` `resource` is the Iceberg resource name. `database` is the Iceberg database. `table` is the Iceberg table. * For Hudi external tables, specify the following properties: ```plaintext PROPERTIES ( "resource" = "hudi0", "database" = "hudi", "table" = "hudi_table" ) ``` ##### `TEMPORARY`[​](#temporary "Direct link to temporary") Creates a temporary table. From v3.3.1, StarRocks supports creating temporary tables in the Default Catalog. For more information, see [Temporary Table](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md#temporary-table). note When creating a temporary table, you must set `ENGINE` to `olap`. #### Column definition[​](#column-definition "Direct link to Column definition") ```sql col_name col_type [agg_type] [NULL | NOT NULL] [DEFAULT "default_value"] [AUTO_INCREMENT] [AS generation_expr] ``` ##### `col_name`[​](#col_name "Direct link to col_name") Note that normally you cannot create a column whose name is initiated with `__op` or `__row` because these name formats are reserved for special purposes in StarRocks and creating such columns may result in undefined behavior. If you do need to create such column, set the FE dynamic parameter [`allow_system_reserved_names`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#allow_system_reserved_names) to `TRUE`. ##### `col_type`[​](#col_type "Direct link to col_type") Specific column information, such as types and ranges: * TINYINT (1 byte): Ranges from -2^7 + 1 to 2^7 - 1. * SMALLINT (2 bytes): Ranges from -2^15 + 1 to 2^15 - 1. * INT (4 bytes): Ranges from -2^31 + 1 to 2^31 - 1. * BIGINT (8 bytes): Ranges from -2^63 + 1 to 2^63 - 1. * LARGEINT (16 bytes): Ranges from -2^127 + 1 to 2^127 - 1. * FLOAT (4 bytes): Supports scientific notation. * DOUBLE (8 bytes): Supports scientific notation. * DECIMAL\[(precision, scale)] (16 bytes) * Default value: DECIMAL(10, 0) * precision: 1 ~ 38 * scale: 0 ~ precision * Integer part: precision - scale Scientific notation is not supported. * DATE (3 bytes): Ranges from 0000-01-01 to 9999-12-31. * DATETIME (8 bytes): Ranges from 0000-01-01 00:00:00 to 9999-12-31 23:59:59. * CHAR\[(length)]: Fixed length string. Range: 1 ~ 255. Default value: 1. * VARCHAR\[(length)]: A variable-length string. The default value is 1. Unit: bytes. In versions earlier than StarRocks 2.1, the value range of `length` is 1–65533. \[Preview] In StarRocks 2.1 and later versions, the value range of `length` is 1–1048576. * HLL (1~16385 bytes): For HLL type, there's no need to specify length or default value. The length will be controlled within the system according to data aggregation. HLL column can only be queried or used by [hll\_union\_agg](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/hll_union_agg.md), [Hll\_cardinality](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_cardinality.md), and [hll\_hash](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_hash.md). * BITMAP: Bitmap type does not require specified length or default value. It represents a set of unsigned bigint numbers. The largest element could be up to 2^64 - 1. ##### `agg_type`[​](#agg_type "Direct link to agg_type") Aggregation type. If not specified, this column is a key column. If specified, it is a value column. The aggregation types supported are as follows: * `SUM`, `MAX`, `MIN`, `REPLACE` * `HLL_UNION` (only for `HLL` type) * `BITMAP_UNION` (only for `BITMAP`) * `REPLACE_IF_NOT_NULL`: This means the imported data will only be replaced when it is of non-null value. If it is of null value, StarRocks will retain the original value. note * When the column of aggregation type BITMAP\_UNION is imported, its original data types must be TINYINT, SMALLINT, INT, and BIGINT. * If NOT NULL is specified by REPLACE\_IF\_NOT\_NULL column when the table was created, StarRocks will still convert the data to NULL without sending an error report to the user. With this, the user can import selected columns. This aggregation type applies ONLY to the Aggregate table whose key\_desc type is AGGREGATE KEY. Since v3.1.9, `REPLACE_IF_NOT_NULL` newly supports the columns of the BITMAP type. ##### `NULL` | `NOT NULL`[​](#null--not-null "Direct link to null--not-null") Whether the column is allowed to be `NULL`. By default, `NULL` is specified for all columns in a table that uses the Duplicate Key, Aggregate, or Unique Key table. In a table that uses the Primary Key table, by default, value columns are specified with `NULL`, whereas key columns are specified with `NOT NULL`. If `NULL` values are included in the raw data, present them with `\N`. StarRocks treats `\N` as `NULL` during data loading. ##### `DEFAULT`[​](#default "Direct link to default") The default value of a column. When you load data into StarRocks, if the source field mapped onto the column is empty, StarRocks automatically fills the default value in the column. You can specify a default value in one of the following ways: * **DEFAULT current\_timestamp**: Use the current time as the default value. For more information, see [current\_timestamp()](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/current_timestamp.md). * **DEFAULT (``)**: Use the result returned by a given expression or function as the default value. The following expressions are supported: * [uuid()](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/uuid.md) and [uuid\_numeric()](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/uuid_numeric.md): Generate unique identifiers. * ARRAY literal expressions (e.g., `[1, 2, 3]`): For ARRAY type columns. * MAP expressions (e.g., `map{key: value}`): For MAP type columns. * row() function (e.g., `row(val1, val2)`): For STRUCT type columns. * **DEFAULT ``**: Use a given value of the column data type as the default value. StarRocks supports specifying default values for different types: **Basic types**: Use string literals to specify default values. ```sql -- Numeric types age INT DEFAULT '18' price DECIMAL(10,2) DEFAULT '99.99' -- String types name VARCHAR(50) DEFAULT 'Anonymous' -- Date/time types created_at DATETIME DEFAULT '2024-01-01 00:00:00' -- Boolean type is_active BOOLEAN DEFAULT 'true' -- Supports 'true'/'false'/'1'/'0' ``` **JSON type**: Use JSON-formatted strings to specify default values. ```sql metadata JSON DEFAULT '{"status": "active"}' tags JSON DEFAULT '[1, 2, 3]' ``` **VARBINARY type**: Only empty string is supported as default value. ```sql binary_data VARBINARY DEFAULT '' ``` **BITMAP and HLL types**: Only empty string is supported as default value, only for AGGREGATE KEY tables. ```sql -- In AGGREGATE KEY tables bm BITMAP BITMAP_UNION DEFAULT '' h HLL HLL_UNION DEFAULT '' ``` **Complex types (ARRAY/MAP/STRUCT)**: Use expression syntax to specify default values, only supported for OLAP tables. note Default values for complex types are **only supported when `fast_schema_evolution = true`**. If the table's `fast_schema_evolution` property is explicitly set to `false`, adding default values for complex types will result in an error. ```sql -- ARRAY type tags ARRAY DEFAULT ['tag1', 'tag2'] scores ARRAY DEFAULT [90, 85, 92] -- MAP type attrs MAP DEFAULT map{'age': 25, 'score': 100} -- STRUCT type person STRUCT DEFAULT row('John', 30) -- Complex nesting: STRUCT with nested STRUCT, ARRAY, and MAP user_profile STRUCT< id INT, name VARCHAR(50), contact STRUCT, tags ARRAY, attributes MAP > DEFAULT row(1, 'Alice', row('alice@example.com', '123-456-7890'), ['admin', 'user'], map{'level': 'premium', 'status': 'active'}) ``` **Limitations**: * TIME and VARIANT types do not support default values yet. * Default values for complex types (ARRAY/MAP/STRUCT) are only supported for OLAP tables and require the `fast_schema_evolution` property to be enabled. ##### `AUTO_INCREMENT`[​](#auto_increment "Direct link to auto_increment") Specifies an `AUTO_INCREMENT` column. The data types of `AUTO_INCREMENT` columns must be BIGINT. Auto-incremented IDs start from 1 and increase at a step of 1. For more information about `AUTO_INCREMENT` columns, see [AUTO\_INCREMENT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/auto_increment.md). Since v3.0, StarRocks supports `AUTO_INCREMENT` columns. ##### `AS`[​](#as "Direct link to as") Specifies the generated column and its expression. [The generated column](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) can be used to precompute and store the results of expressions, which significantly accelerates queries with the same complex expressions. Since v3.1, StarRocks supports generated columns. #### Index definition[​](#index-definition "Direct link to Index definition") ```sql INDEX index_name (col_name[, col_name, ...]) [USING BITMAP] COMMENT 'xxxxxx' ``` For more information about parameter descriptions and usage notes, see [Bitmap indexing](https://docs.starrocks.io/docs/table_design/indexes/Bitmap_index.md#create-an-index). #### `ENGINE`[​](#engine "Direct link to engine") Default value: `olap`. If this parameter is not specified, an OLAP table (StarRocks native table) is created by default. Optional value: `mysql`, `elasticsearch`, `hive`, `jdbc`, `iceberg`, and `hudi`. #### Key[​](#key "Direct link to Key") Syntax: ```sql key_type(k1[,k2 ...]) ``` Data is sequenced in specified key columns and has different attributes for different key types: * AGGREGATE KEY: Identical content in key columns will be aggregated into value columns according to the specified aggregation type. It usually applies to business scenarios such as financial statements and multi-dimensional analysis. * UNIQUE KEY/PRIMARY KEY: Identical content in key columns will be replaced in value columns according to the import sequence. It can be applied to make addition, deletion, modification and query on key columns. * DUPLICATE KEY: Identical content in key columns co-exists in StarRocks. It can be used to store detailed data or data with no aggregation attributes. note DUPLICATE KEY is the default type. Data will be sequenced according to key columns. note Value columns do not need to specify aggregation types when other key type is used to create tables with the exception of AGGREGATE KEY. ##### Range-based Distribution[​](#range-based-distribution "Direct link to Range-based Distribution") From v4.1 onwards, StarRocks supports the **Range-based Distribution semantic** (disabled by default), controlled by the FE configuration `enable_range_distribution`. The data will be sequenced according to the data range of the key columns, and each tablet contains the data from a certain range. The range-based distribution semantic is different from the default semantic in the following aspects: * If the key type (AGGREGATE KEY/UNIQUE KEY/PRIMARY KEY/DUPLICATE KEY) is explicitly specified, and a DISTRIBUTED BY clause is not specified, the data will be distributed by range by default. * If none of the key type, a DISTRIBUTED BY clause, or an ORDER BY is specified, a Duplicate Key table with the random bucketing strategy will be created. * If the key type and a DISTRIBUTED BY clause are not specified, but an ORDER BY clause is specified, a Duplicate Key table with the range-based distribution strategy will be created. In this case, DUPLICATE KEY is equivalent to an ORDER BY clause, and vice versa. * If both DUPLICATE KEY and an ORDER BY clause are specified, only the ORDER BY clause will take effect, and DUPLICATE KEY will be ignored. #### `COMMENT`[​](#comment "Direct link to comment") You can add a table comment when you create a table, optional. Note that COMMENT must be placed after `key_desc`. Otherwise, the table cannot be created. From v3.1 onwards, you can modify the table comment suing `ALTER TABLE COMMENT = "new table comment"`. #### Partition[​](#partition "Direct link to Partition") Partitions can be managed in the following ways: ##### Create partitions dynamically[​](#create-partitions-dynamically "Direct link to Create partitions dynamically") [Dynamic partitioning](https://docs.starrocks.io/docs/table_design/data_distribution/dynamic_partitioning.md) provides a time-to-live (TTL) management for partitions. StarRocks automatically creates new partitions in advance and removes expired partitions to ensure data freshness. To enable this feature, you can configure Dynamic partitioning related properties at table creation. ##### Create partitions one by one[​](#create-partitions-one-by-one "Direct link to Create partitions one by one") ###### Specify only the upper bound for a partition[​](#specify-only-the-upper-bound-for-a-partition "Direct link to Specify only the upper bound for a partition") Syntax: ```sql PARTITION BY RANGE ( [, , ... ] ) PARTITION VALUES LESS THAN ("" [ , "", ... ] ) [ , PARTITION VALUES LESS THAN ("" [ , "", ... ] ) , ... ] ) ``` note Please use specified key columns and specified value ranges for partitioning. * For the naming conventions of partitions, see [System limits](https://docs.starrocks.io/docs/sql-reference/System_limit.md). * Before v3.3.0, columns for the range partitioning only support the following types: TINYINT, SMALLINT, INT, BIGINT, LARGEINT, DATE, and DATETIME. Since v3.3.0, three specific time functions can be used as columns for the range partitioning. For detailed usage, see [Data distribution](https://docs.starrocks.io/docs/table_design/data_distribution.md#manually-create-partitions). * Partitions are left closed and right open. The left boundary of the first partition is of minimum value. * NULL value is stored only in partitions that contain minimum values. When the partition containing the minimum value is deleted, NULL values can no longer be imported. * Partition columns can either be single columns or multiple columns. The partition values are the default minimum values. * When only one column is specified as the partitioning column, you can set `MAXVALUE` as the upper bound for the partitioning column of the most recent partition. ```sql PARTITION BY RANGE (pay_dt) ( PARTITION p1 VALUES LESS THAN ("20210102"), PARTITION p2 VALUES LESS THAN ("20210103"), PARTITION p3 VALUES LESS THAN MAXVALUE ) ``` note * Partitions are often used for managing data related to time. * When data backtracking is needed, you may want to consider emptying the first partition for adding partitions later when necessary. ###### Specify both the lower and upper bounds for a partition[​](#specify-both-the-lower-and-upper-bounds-for-a-partition "Direct link to Specify both the lower and upper bounds for a partition") Syntax: ```sql PARTITION BY RANGE ( [, , ... ] ) ( PARTITION VALUES [( "" [ , "", ... ] ), ( "", ... ] ) ) [, PARTITION VALUES [( "" [ , "", ... ] ), ( "" [ , "", ... ] ) ) , ...] ) ``` note * Fixed Range is more flexible than LESS THAN. You can customize the left and right partitions. * Fixed Range is the same as LESS THAN in the other aspects. * When only one column is specified as the partitioning column, you can set `MAXVALUE` as the upper bound for the partitioning column of the most recent partition. ```sql PARTITION BY RANGE (pay_dt) ( PARTITION p202101 VALUES [("20210101"), ("20210201")), PARTITION p202102 VALUES [("20210201"), ("20210301")), PARTITION p202103 VALUES [("20210301"), (MAXVALUE)) ) ``` ##### Create multiple partitions in a batch[​](#create-multiple-partitions-in-a-batch "Direct link to Create multiple partitions in a batch") Syntax * If the partitioning column is of a date type. ```sql PARTITION BY RANGE () ( START ("") END ("") EVERY (INTERVAL ) ) ``` * If the partitioning column is of an integer type. ```sql PARTITION BY RANGE () ( START ("") END ("") EVERY () ) ``` Description You can specify the start and end values in `START()` and `END()` and the time unit or partitioning granularity in `EVERY()` to create multiple partitions in a batch. * Before v3.3.0, columns for the range partitioning only support the following types: TINYINT, SMALLINT, INT, BIGINT, LARGEINT, DATE, and DATETIME. Since v3.3.0, three specific time functions can be used as columns for the range partitioning. For detailed usage, see [Data distribution](https://docs.starrocks.io/docs/table_design/data_distribution.md#manually-create-partitions). * If the partitioning column is of a date type, you need to use the `INTERVAL` keyword to specify the time interval. You can specify the time unit as hour (since v3.0), day, week, month, or year. The naming conventions of partitions are the same as those for dynamic partitions. For more information, see [Data distribution](https://docs.starrocks.io/docs/table_design/data_distribution.md). #### Distribution[​](#distribution "Direct link to Distribution") StarRocks supports hash bucketing and random bucketing. If you do not configure bucketing, StarRocks uses random bucketing and automatically sets the number of buckets by default. * Random bucketing (since v3.1) For data in a partition, StarRocks distributes the data randomly across all buckets, which is not based on specific column values. And if you want StarRocks to automatically set the number of buckets, you do not need to specify any bucketing configurations. If you choose to manually specify the number of buckets, the syntax is as follows: ```sql DISTRIBUTED BY RANDOM BUCKETS ``` However, note that the query performance provided by random bucketing may not be ideal when you query massive amounts of data and frequently use certain columns as conditional columns. In this scenario, it is recommended to use hash bucketing. Because only a small number of buckets need to be scanned and computed, significantly improving query performance. **Precautions** * You can only use random bucketing to create Duplicate Key tables. * You can not specify a [Colocation Group](https://docs.starrocks.io/docs/using_starrocks/Colocate_join.md) for a table bucketed randomly. * Spark Load cannot be used to load data into tables bucketed randomly. * Since StarRocks v2.5.7, you do not need to set the number of buckets when you create a table. StarRocks automatically sets the number of buckets. If you want to set this parameter, see [Set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). For more information, see [Random bucketing](https://docs.starrocks.io/docs/table_design/data_distribution.md#random-bucketing-since-v31). * Hash bucketing Syntax: ```sql DISTRIBUTED BY HASH (k1[,k2 ...]) [BUCKETS num] ``` Data in partitions can be subdivided into buckets based on the hash values of the bucketing columns and the number of buckets. We recommend that you choose the column that meets the following two requirements as the bucketing column. * High cardinality column such as ID * Column that is often used as a filter in queries If such a column does not exist, you can determine the bucketing column according to the complexity of queries. * If the query is complex, we recommend that you select a high cardinality column as the bucketing column to ensure balanced data distribution among buckets and improve cluster resource utilization. * If the query is relatively simple, we recommend that you select the column that is often used as the query condition as the bucketing column to improve query efficiency. If partition data cannot be evenly distributed into each bucket by using one bucketing column, you can choose multiple bucketing columns (at most three). For more information, see [Choose bucketing columns](https://docs.starrocks.io/docs/table_design/data_distribution.md#hash-bucketing). **Precautions**: * **When you create a table, you must specify its bucketing columns**. * The values of bucketing columns cannot be updated. * Bucketing columns cannot be modified after they are specified. * Since StarRocks v2.5.7, you do not need to set the number of buckets when you create a table. StarRocks automatically sets the number of buckets. If you want to set this parameter, see [Set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). * Range-based distribution From v4.1 onwards, StarRocks supports the **Range-based Distribution semantic** (disabled by default), controlled by the FE configuration `enable_range_distribution`. For detailed information, see [Range-based distribution](#range-based-distribution). #### Rollup index[​](#rollup-index "Direct link to Rollup index") You can create rollups in bulk when you create a table. Syntax: ```sql ROLLUP (rollup_name (column_name1, column_name2, ...) [FROM from_index_name] [PROPERTIES ("key"="value", ...)],...) ``` #### `ORDER BY`[​](#order-by "Direct link to order-by") Since v3.0, Primary Key tables support defining sort keys using `ORDER BY`. Since v3.3, Duplicate Key tables, Aggregate tables, and Unique Key tables support defining sort keys using `ORDER BY`. For more descriptions of sort keys, see [Sort keys and prefix indexes](https://docs.starrocks.io/docs/table_design/indexes/Prefix_index_sort_key.md). #### `PROPERTIES`[​](#properties "Direct link to properties") ##### Storage and replicas[​](#storage-and-replicas "Direct link to Storage and replicas") If the engine type is `OLAP`, you can specify initial storage medium (`storage_medium`), automatic storage cooldown time (`storage_cooldown_time`) or time interval (`storage_cooldown_ttl`), and replica number (`replication_num`) when you create a table. The scope where the properties take effect: If the table has only one partition, the properties belong to the table. If the table is divided into multiple partitions, the properties belong to each partition. And when you need to configure different properties for specified partitions, you can execute [ALTER TABLE ... ADD PARTITION or ALTER TABLE ... MODIFY PARTITION](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md) after table creation. ###### Set initial storage medium and automatic storage cooldown time[​](#set-initial-storage-medium-and-automatic-storage-cooldown-time "Direct link to Set initial storage medium and automatic storage cooldown time") ```sql PROPERTIES ( "storage_medium" = "[SSD|HDD]", { "storage_cooldown_ttl" = " { YEAR | MONTH | DAY | HOUR } " | "storage_cooldown_time" = "yyyy-MM-dd HH:mm:ss" } ) ``` **Properties** * `storage_medium`: the initial storage medium, which can be set to `SSD` or `HDD`. Make sure that the type of storage medium you explicitly specified is consistent with the BE disk types for your StarRocks cluster specified in the BE static parameter `storage_root_path`.
If the FE configuration item `enable_strict_storage_medium_check` is set to `true`, the system strictly checks BE disk type when you create a table. If the storage medium you specified in CREATE TABLE is inconsistent with BE disk type, an error "Failed to find enough host in all backends with storage medium is SSD|HDD." is returned and table creation fails. If `enable_strict_storage_medium_check` is set to `false`, the system ignores this error and forcibly creates the table. However, cluster disk space may be unevenly distributed after data is loaded.
From v2.3.6, v2.4.2, v2.5.1, and v3.0 onwards, the system automatically infers storage medium based on BE disk type if `storage_medium` is not explicitly specified.
* The system automatically sets this parameter to SSD in the following scenarios: * The disk types reported by BEs (`storage_root_path`) contain only SSD. * The disk types reported by BEs (`storage_root_path`) contain both SSD and HDD. Note that from v2.3.10, v2.4.5, v2.5.4, and v3.0 onwards, the system sets `storage_medium` to SSD when `storage_root_path` reported by BEs contain both SSD and HDD and the property `storage_cooldown_time` is specified. * The system automatically sets this parameter to HDD in the following scenarios: * The disk types reported by BEs (`storage_root_path`) contain only HDD. * From 2.3.10, 2.4.5, 2.5.4, and 3.0 onwards, the system sets `storage_medium` to HDD when `storage_root_path` reported by BEs contain both SSD and HDD and the property `storage_cooldown_time` is not specified. * `storage_cooldown_ttl` or `storage_cooldown_time`: the automatic storage cooldown time or time interval. Automatic storage cooldown refers to automatically migrate data from SSD to HDD. This feature is only effective when the initial storage medium is SSD. * `storage_cooldown_ttl`: the **time interval** of automatic storage cooldown for the partitions in this table. If you need to retain the most recent partitions on SSD and automatically cool down older partitions to HDD after a certain time interval, you can use this parameter. The automatic storage cooldown time for each partition is calculated using the value of this parameter plus the upper time bound of the partition. The supported values are ` YEAR`, ` MONTH`, ` DAY`, and ` HOUR`. `` is a non-negative integer. The default value is null, indicating that storage cooldown is not automatically performed. For example, you specify the value as `"storage_cooldown_ttl"="1 DAY"` when creating the table, and the partition `p20230801` with a range of `[2023-08-01 00:00:00,2023-08-02 00:00:00)` exists. The automatic storage cooldown time for this partition is `2023-08-03 00:00:00`, which is `2023-08-02 00:00:00 + 1 DAY`. If you specify the value as `"storage_cooldown_ttl"="0 DAY"` when creating the table, the automatic storage cooldown time for this partition is `2023-08-02 00:00:00`. * `storage_cooldown_time`: the automatic storage cooldown time (**absolute time**) when the table is cooled down from SSD to HDD. The specified time needs to be later than the current time. Format: "yyyy-MM-dd HH:mm :ss ". When you need to configure different properties for specified partitions, you can execute [ALTER TABLE ... ADD PARTITION or ALTER TABLE ... MODIFY PARTITION](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md). ###### Usage[​](#usage "Direct link to Usage") * The comparison between the parameters related to automatic storage cooldown is as follows: * `storage_cooldown_ttl`: A table property that specifies the time interval of automatic storage cooldown for partitions in the table. The system automatically cools down a partition at the time `the value of this parameter plus the upper time bound of the partition`. So automatic storage cooldown is performed at the partition granularity, which is more flexible. * `storage_cooldown_time`: A table property that specifies the automatic storage cooldown time (**absolute time**) for this table. Also, you can configure different properties for specified partitions after table creation. * `storage_cooldown_second`: A static FE parameter that specifies the automatic storage cooldown latency for all tables within the cluster. * The table property `storage_cooldown_ttl` or `storage_cooldown_time` takes precedence over the FE static parameter `storage_cooldown_second`. * When configuring these parameters, you need to specify `"storage_medium = "SSD"`. * If you do not configure these parameters, automatic storage cooldown is not be automatically performed. * Execute `SHOW PARTITIONS FROM ` to view the automatic storage cooldown time for each partition. ###### Limits[​](#limits "Direct link to Limits") * Expression and List partitioning are not supported. * The partition column need to be of date type. * Multiple partition columns are not supported. * Primary Key tables are not supported. ###### Set the number of replicas for each tablet in partitions[​](#set-the-number-of-replicas-for-each-tablet-in-partitions "Direct link to Set the number of replicas for each tablet in partitions") `replication_num`: number of replicas for each table in the partitions. Default number: `3`. ```sql PROPERTIES ( "replication_num" = "" ) ``` ##### Bloom filter indexes[​](#bloom-filter-indexes "Direct link to Bloom filter indexes") If the Engine type is `olap`, you can specify a column to adopt bloom filter indexes. The following limits apply when you use bloom filter index: * You can create bloom filter indexes for all columns of a Duplicate Key or Primary Key table. For an Aggregate table or Unique Key table, you can only create bloom filter indexes for key columns. * TINYINT, FLOAT, DOUBLE, and DECIMAL columns do not support creating bloom filter indexes. * Bloom filter indexes can only improve the performance of queries that contain the `in` and `=` operators, such as `Select xxx from table where x in {}` and `Select xxx from table where column = xxx`. More discrete values in this column will result in more precise queries. For more information, see [Bloom filter indexing](https://docs.starrocks.io/docs/table_design/indexes/Bloomfilter_index.md) ```sql PROPERTIES ( "bloom_filter_columns"="k1,k2,k3" ) ``` ##### Colocate Join[​](#colocate-join "Direct link to Colocate Join") If you want to use Colocate Join attributes, specify it in `properties`. ```sql PROPERTIES ( "colocate_with"="table1" ) ``` ##### Dynamic partitions[​](#dynamic-partitions "Direct link to Dynamic partitions") If you want to use dynamic partition attributes, please specify it in properties. ```sql PROPERTIES ( "dynamic_partition.enable" = "true|false", "dynamic_partition.time_unit" = "DAY|WEEK|MONTH", "dynamic_partition.start" = "${integer_value}", "dynamic_partition.end" = "${integer_value}", "dynamic_partition.prefix" = "${string_value}", "dynamic_partition.buckets" = "${integer_value}" ``` **`PROPERTIES`** | Parameter | Required | Description | | ----------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | dynamic\_partition.enable | No | Whether to enable dynamic partitioning. Valid values: `TRUE` and `FALSE`. Default value: `TRUE`. | | dynamic\_partition.time\_unit | Yes | The time granularity for dynamically created partitions. It is a required parameter. Valid values: `DAY`, `WEEK`, and `MONTH`. The time granularity determines the suffix format for dynamically created partitions.
- If the value is `DAY`, the suffix format for dynamically created partitions is `yyyyMMdd`. An example partition name suffix is `20200321`.
- If the value is `WEEK`, the suffix format for dynamically created partitions is `yyyy_ww`, for example `2020_13` for the 13th week of 2020.
- If the value is `MONTH`, the suffix format for dynamically created partitions is `yyyyMM`, for example `202003`. | | dynamic\_partition.start | No | The starting offset of dynamic partitioning. The value of this parameter must be a negative integer. The partitions before this offset will be deleted based on the current day, week, or month which is determined by `dynamic_partition.time_unit`. The default value is `Integer.MIN_VALUE`, namely, -2147483648, which means that historical partitions will not be deleted. | | dynamic\_partition.end | Yes | The end offset of dynamic partitioning. The value of this parameter must be a positive integer. The partitions from the current day, week, or month to the end offset will be created in advance. | | dynamic\_partition.prefix | No | The prefix added to the names of dynamic partitions. Default value: `p`. | | dynamic\_partition.buckets | No | The number of buckets per dynamic partition. The default value is the same as the number of buckets determined by the reserved word `BUCKETS` or automatically set by StarRocks. | note When the partition column is the INT type, its format must be `yyyyMMdd`, regardless of the partition time granularity. ##### Bucket size with random bucketing[​](#bucket-size-with-random-bucketing "Direct link to Bucket size with random bucketing") Since v3.2, for tables configured with random bucketing, you can specify the bucket size by using the `bucket_size` parameter in `PROPERTIES` at table creation to enable the on-demand and dynamic increase of the number of buckets. Unit: B. ```sql PROPERTIES ( "bucket_size" = "1073741824" ) ``` ##### Data compression algorithm[​](#data-compression-algorithm "Direct link to Data compression algorithm") You can specify a data compression algorithm for a table by adding property `compression` when you create a table. The valid values of `compression` are: * `LZ4`: the LZ4 algorithm. * `ZSTD`: the Zstandard algorithm. * `ZLIB`: the zlib algorithm. * `SNAPPY`: the Snappy algorithm. From v3.3.2 onwards, StarRocks supports specifying the compression level for zstd compression format during table creation. Syntax: ```sql PROPERTIES ("compression" = "zstd()") ``` `compression_level`: the compression level for ZSTD compression format. Type: Integer. Range: \[1,22]. Default: `3` (Recommended). The greater the number, the higher the compression ratio. The higher the compression level, the more time consumption for compression and decompression. Example: ```sql PROPERTIES ("compression" = "zstd(3)") ``` For more information about how to choose a suitable data compression algorithm, see [Data compression](https://docs.starrocks.io/docs/table_design/data_compression.md). ##### Write quorum for data loading[​](#write-quorum-for-data-loading "Direct link to Write quorum for data loading") If your StarRocks cluster has multiple data replicas, you can set different write quorum for tables, that is, how many replicas are required to return loading success before StarRocks can determine the loading task is successful. You can specify write quorum by adding the property `write_quorum` when you create a table. This property is supported from v2.5. The valid values of `write_quorum` are: * `MAJORITY`: Default value. When the **majority** of data replicas return loading success, StarRocks returns loading task success. Otherwise, StarRocks returns loading task failed. * `ONE`: When **one** of the data replicas returns loading success, StarRocks returns loading task success. Otherwise, StarRocks returns loading task failed. * `ALL`: When **all** of the data replicas return loading success, StarRocks returns loading task success. Otherwise, StarRocks returns loading task failed. caution * Setting a low write quorum for loading increases the risk of data inaccessibility and even loss. For example, you load data into a table with one write quorum in a StarRocks cluster of two replicas, and the data was successfully loaded into only one replica. Despite that StarRocks determines the loading task succeeded, there is only one surviving replica of the data. If the server which stores the tablets of loaded data goes down, the data in these tablets becomes inaccessible. And if the disk of the server is damaged, the data is lost. * StarRocks returns the loading task status only after all data replicas have returned the status. StarRocks will not return the loading task status when there are replicas whose loading status is unknown. In a replica, loading timeout is also considered as loading failed. ##### Replica data writing and replication mode[​](#replica-data-writing-and-replication-mode "Direct link to Replica data writing and replication mode") If your StarRocks cluster has multiple data replicas, you can specify the `replicated_storage` parameter in `PROPERTIES` to configure the data writing and replication mode among replicas. * `true` (default in v3.0 and later) indicates "single leader replication", which means data is written only to the primary replica. Other replicas synchronize data from the primary replica. This mode significantly reduces CPU cost caused by data writing to multiple replicas. It is supported from v2.5. * `false` (default in v2.5) indicates "leaderless replication", which means data is directly written to multiple replicas, without differentiating primary and secondary replicas. The CPU cost is multiplied by the number of replicas. In most cases, using the default value gains better data writing performance. If you want to change the data writing and replication mode among replicas, run the ALTER TABLE command. Example: ```sql ALTER TABLE example_db.my_table SET ("replicated_storage" = "false"); ``` ##### Delta Join unique and foreign key constraints[​](#delta-join-unique-and-foreign-key-constraints "Direct link to Delta Join unique and foreign key constraints") To enable query rewrite in the View Delta Join scenario, you must define the Unique Key constraints `unique_constraints` and Foreign Key constraints `foreign_key_constraints` for the table to be joined in the Delta Join. See [Asynchronous materialized view - Rewrite queries in View Delta Join scenario](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md#query-delta-join-rewrite) for further information. ```sql PROPERTIES ( "unique_constraints" = "[, ...]", "foreign_key_constraints" = " ([, ...]) REFERENCES [catalog_name].[database_name].([, ...]) [;...] " ) ``` * `child_column`: the Foreign Key of the table. You can define multiple `child_column`. * `catalog_name`: the name of the catalog where the table to join resides. The default catalog is used if this parameter is not specified. * `database_name`: the name of the database where the table to join resides. The current database is used if this parameter is not specified. * `parent_table_name`: the name of the table to join. * `parent_column`: the column to be joined. They must be the Primary Keys or Unique Keys of the corresponding tables. caution * `unique_constraints` and `foreign_key_constraints` are only used for query rewrite. Foreign Key constraints checks are not guaranteed when data is loaded into the table. You must ensure the data loaded into the table meets the constraints. * The primary keys of a Primary Key table or the unique keys of a Unique Key table are, by default, the corresponding `unique_constraints`. You do not need to set it manually. * The `child_column` in a table's `foreign_key_constraints` must be referenced to a `unique_key` in another table's `unique_constraints`. * The number of `child_column` and `parent_column` must agree. * The data types of the `child_column` and the corresponding `parent_column` must match. ##### Cloud-native tables for shared-data clusters[​](#cloud-native-tables-for-shared-data-clusters "Direct link to Cloud-native tables for shared-data clusters") To use your StarRocks Shared-data cluster, you must create cloud-native tables with the following properties: ```sql PROPERTIES ( "storage_volume" = "", "datacache.enable" = "{ true | false }", "datacache.partition_duration" = "", "file_bundling" = "{ true | false }" ) ``` * `storage_volume`: The name of the storage volume used to store the cloud-native table you want to create. If this property is not specified, the default storage volume is used. This property is supported from v3.1 onwards. * `datacache.enable`: Whether to enable the local disk cache. Default: `true`. * When this property is set to `true`, the data to be loaded is simultaneously written into the object storage and the local disk (as the cache for query acceleration). * When this property is set to `false`, the data is loaded only into the object storage. note To enable the local disk cache, you must specify the directory of the disk in the BE configuration item `storage_root_path`. * `datacache.partition_duration`: The validity duration of the hot data. When the local disk cache is enabled, all data is loaded into the cache. When the cache is full, StarRocks deletes the less recently used data from the cache. When a query needs to scan the deleted data, StarRocks checks if the data is within the duration of validity. If the data is within the duration, StarRocks loads the data into the cache again. If the data is not within the duration, StarRocks does not load it into the cache. This property is a string value that can be specified with the following units: `YEAR`, `MONTH`, `DAY`, and `HOUR`, for example, `7 DAY` and `12 HOUR`. If it is not specified, all data is cached as the hot data. note This property is available only when `datacache.enable` is set to `true`. * `file_bundling` (Optional): Whether to enable the File Bundling optimization for the cloud-native table. Supported from v4.0 onwards. When this feature is enabled (setting to `true`), the system automatically bundles the data files generated by loading, Compaction, or Publish operations, thereby reducing the API cost caused by high-frequency access to the external storage system. note * File Bundling is available only for shared-data clusters with StarRocks v4.0 or later. * File Bundling is enabled by default for tables created in v4.0 or later, controlled by FE configuration `enable_file_bundling` (Default: true). * After File Bundling is enabled, you can only downgrade the cluster to v3.5.2 or later. If you want to downgrade to versions earlier than v3.5.2, you must first drop the tables that have enabled File Bundling. * File Bundling remains disabled by default for the existing tables after the cluster is upgraded to v4.0. * You can manually enable File Bundling for existing tables using the [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md) statement with the following restrictions: * You cannot enable File Bundling for tables with Rollup Indexes created in versions earlier than v4.0. You can drop and recreate the indexes in v4.0 or later, and then enable File Bundling for the tables. * You cannot modify the `file_bundling` property **repeatedly** within a specific period of time. Otherwise, the system returns an error. You can check if the `file_bundling` property is available for modification by executing the following SQL statement: ```sql SELECT METADATA_SWITCH_VERSION FROM information_schema.partitions_meta WHERE TABLE_NAME = ''; ``` You are allowed to modify the `file_bundling` property only when `0` is returned. Non-zero values indicate that the data version corresponding to `METADATA_SWITCH_VERSION` is yet to be reclaimed by the GC mechanism. You must wait until the data version is reclaimed. You can foreshorten this interval by setting a lower value for the FE dynamic configuration `lake_autovacuum_grace_period_minutes`. However, remember to reset the configuration to its original value after you modify the `file_bundling` property. ##### Fast Schema Evolution[​](#fast-schema-evolution "Direct link to Fast Schema Evolution") * `fast_schema_evolution`: Whether to enable Fast Schema Evolution for the table. Valid values are `TRUE` or `FALSE` (default). Enabling Fast Schema Evolution can increase the speed of schema changes and reduce resource usage when columns are added or dropped. Currently, this property can only be enabled at table creation, and it cannot be modified using ALTER TABLE after table creation. note * Fast Schema Evolution is supported for shared-nothing clusters since v3.2.0. * Fast Schema Evolution is supported for shared-data clusters since v3.3 and is enabled by default. You do not need to specify this property when creating cloud-native tables in shared-data clusters. The FE dynamic parameter `enable_fast_schema_evolution` (Default: true) controls this behavior. * `cloud_native_fast_schema_evolution_v2`: Whether to enable Fast Schema Evolution v2 for the **cloud-native table**. Supported from v4.1 onwards. Valid values are `TRUE` (default) or `FALSE`. When Fast Schema Evolution v2 is enabled, schema changes become a synchronous process. When the ALTER TABLE statement returns successfully, the new schema is effective immediately. The system will only modify FE metadata rather than tablet metadata located on S3, so it can always achieve second-level latency no matter how many partitions or tablets in the table. While in the legacy behavior, schema changes are run as an asynchronous job that updates tablet metadata over time. note * Fast Schema Evolution v2 is supported from v4.1 onwards and only for **cloud-native tables** in shared-data clusters. * Default behaviors: * For new tables created in a v4.1 cluster, Fast Schema Evolution v2 is enabled by default. * For existing tables from a cluster upgraded to v4.1, Fast Schema Evolution v2 is disabled by default. You can enable it by explicitly setting this property to `true` via [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md). * Downgrade requirements: * To downgrade a shared-data cluster from v4.1 to v4.0.5 or later, you can directly downgrade it following the standard downgrade procedures. * Before downgrading a shared-data cluster from v4.1 to v3.x or patch versions earlier than v4.0.5, you must manually set `cloud_native_fast_schema_evolution_v2` to `false` for any tables that have enabled Fast Schema Evolution v2 via ALTER TABLE. You must wait until the asynchronous jobs become FINISHED. You can track the job status via SHOW ALTER. You can inspect the schema change jobs via [SHOW ALTER TABLE COLUMN](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_ALTER.md). Example: ```sql -- List recent column/schema change jobs in the table SHOW ALTER TABLE COLUMN FROM test_db WHERE TableName = "test_tbl"; ``` For cloud-native tables with Fast Schema Evolution v2 enabled, schema change jobs will typically appear as FINISHED, because the change is applied by updating FE metadata only. ##### Forbid Base Compaction[​](#forbid-base-compaction "Direct link to Forbid Base Compaction") `base_compaction_forbidden_time_ranges`: The time range within which Base Compaction is forbidden for the table. When this property is set, the system performs Base Compaction on eligible tablets only outside the specified time range. This property is supported from v3.2.13. note Make sure that the number of data loading to the table does not exceed 500 during the period when Base Compaction is forbidden. The value of `base_compaction_forbidden_time_ranges` follows the [Quartz cron syntax](https://productresources.collibra.com/docs/collibra/latest/Content/Cron/co_quartz-cron-syntax.htm), and only supports these fields: ` `, where `` must be `*`. ```sql crontab_param_value ::= [ "" | crontab ] crontab ::= * ``` * When this property is not set or set to `""` (an empty string), Base Compaction is not forbidden at any time. * When this property is set to `* * * * *`, Base Compaction is always forbidden. * Other values follow the Quartz cron syntax. * An independent value indicates the unit time of a field. For example, `8` in the `` field means 8:00-8:59. * A value range indicates the time range of a field. For example, `8-9` in the `` field means 8:00-9:59. * Multiple value ranges separated by commas indicate multiple time ranges of the field. * `` has a starting value of `1` for Sunday, and `7` stands for Saturday. Example: ```sql -- Forbid Base Compaction from 8:00 am to 9:00 pm every day. 'base_compaction_forbidden_time_ranges' = '* 8-20 * * *' -- Forbid Base Compaction from 0:00 am to 5:00 am and from 9:00 pm to 11:00 pm every day. 'base_compaction_forbidden_time_ranges' = '* 0-4,21-22 * * *' -- Forbid Base Compaction from Monday to Friday (that is, allow it on Saturday and Sunday). 'base_compaction_forbidden_time_ranges' = '* * * * 2-6' -- Forbid Base Compaction from 8:00 am to 9:00 pm every working day (that is, Monday to Friday). 'base_compaction_forbidden_time_ranges' = '* 8-20 * * 2-6' ``` ##### Specify Common Partition Expression TTL[​](#specify-common-partition-expression-ttl "Direct link to Specify Common Partition Expression TTL") From v3.5.0 onwards, StarRocks native tables support Common Partition Expression TTL. `partition_retention_condition`: The expression that declares the partitions to be retained dynamically. Partitions that do not meet the condition in the expression will be dropped regularly. * The expression can only contain partition columns and constants. Non-partition columns are not supported. * Common Partition Expression applies to List partitions and Range partitions differently: * For tables with List partitions, StarRocks supports deleting partitions filtered by the Common Partition Expression. * For tables with Range partitions, StarRocks can only filter and delete partitions using the partition pruning capability of FE. Partitions correspond to predicates that are not supported by partition pruning cannot be filtered and deleted. Example: ```sql -- Retain the data from the last three months. Column `dt` is the partition column of the table. "partition_retention_condition" = "dt >= CURRENT_DATE() - INTERVAL 3 MONTH" ``` To disable this feature, you can use the ALTER TABLE statement to set this property as an empty string: ```sql ALTER TABLE tbl SET('partition_retention_condition' = ''); ``` ##### Set Flat JSON properties on table level[​](#set-flat-json-properties-on-table-level "Direct link to Set Flat JSON properties on table level") In v3.3, StarRocks introduced the [Flat JSON](https://docs.starrocks.io/docs/using_starrocks/Flat_json.md) feature to improve JSON data query efficiency and reduce the complexity of using JSON. This feature was controlled by specific BE configuration items and system variables. As a result, it can only be enabled (or disabled) globally. From v4.0 onwards, you can set the Flat JSON-related properties on table level. ```sql PROPERTIES ( "flat_json.enable" = "{ true | false }", "flat_json.null.factor" = "", "flat_json.sparsity.factor" = "", "flat_json.column.max" = "" ) ``` * `flat_json.enable` (Optional): Whether to enable the Flat JSON feature. After this feature is enabled, newly loaded JSON data will be automatically flattened, improving JSON query performance. * `flat_json.null.factor` (Optional): The proportion threshold of NULL values in the column. A column will not be extracted by Flat JSON if its proportion of NULL values is higher than this threshold. This parameter takes effect only when `flat_json.enable` is set to `true`. Default value: `0.3`. * `flat_json.sparsity.factor` (Optional): The proportion threshold of columns with the same name. A column will not be extracted by Flat JSON if the proportion of columns with the same name is lower than this value. This parameter takes effect only when `flat_json.enable` is set to `true`. Default value: `0.3`. * `flat_json.column.max` (Optional): The maximum number of sub-fields that can be extracted by Flat JSON. This parameter takes effect only when `flat_json.enable` is set to `true`. Default value: `100`. ##### Skip a table in automatic statistics collection[​](#skip-a-table-in-automatic-statistics-collection "Direct link to Skip a table in automatic statistics collection") You can set `enable_statistic_auto_collect` to `false` to stop automatic statistics collection from collecting statistics on a table. Default: `true`. ```sql PROPERTIES ( "enable_statistic_auto_collect" = "false" ) ``` When this property is `false`: * The automatic collection task provided by StarRocks, and the automatic collection tasks created by `CREATE ANALYZE ALL` or `CREATE ANALYZE DATABASE`, skip this table. * Collection tasks that name this table, created by `CREATE ANALYZE TABLE` or run with `ANALYZE TABLE`, still collect its statistics. * Statistics collection triggered by data loading is not affected. It is controlled separately by `enable_statistic_collect_on_first_load`. You can change this property with [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md). This property is not supported for materialized views. #### Examples[​](#examples "Direct link to Examples") ##### Aggregate table with Hash bucketing and columnar storage[​](#aggregate-table-with-hash-bucketing-and-columnar-storage "Direct link to Aggregate table with Hash bucketing and columnar storage") ```sql CREATE TABLE example_db.table_hash ( k1 TINYINT, k2 DECIMAL(10, 2) DEFAULT "10.5", v1 CHAR(10) REPLACE, v2 INT SUM ) ENGINE=olap AGGREGATE KEY(k1, k2) COMMENT "my first starrocks table" DISTRIBUTED BY HASH(k1) PROPERTIES ("storage_type"="column"); ``` ##### Aggregate table with storage medium and cooldown time set[​](#aggregate-table-with-storage-medium-and-cooldown-time-set "Direct link to Aggregate table with storage medium and cooldown time set") ```sql CREATE TABLE example_db.table_hash ( k1 BIGINT, k2 LARGEINT, v1 VARCHAR(2048) REPLACE, v2 SMALLINT SUM DEFAULT "10" ) ENGINE=olap UNIQUE KEY(k1, k2) DISTRIBUTED BY HASH (k1, k2) PROPERTIES( "storage_type"="column", "storage_medium" = "SSD", "storage_cooldown_time" = "2015-06-04 00:00:00" ); ``` ##### Duplicate Key table with Range partition, Hash bucketing, column-based storage, storage medium, and cooldown time[​](#duplicate-key-table-with-range-partition-hash-bucketing-column-based-storage-storage-medium-and-cooldown-time "Direct link to Duplicate Key table with Range partition, Hash bucketing, column-based storage, storage medium, and cooldown time") LESS THAN ```sql CREATE TABLE example_db.table_range ( k1 DATE, k2 INT, k3 SMALLINT, v1 VARCHAR(2048), v2 DATETIME DEFAULT "2014-02-04 15:36:00" ) ENGINE=olap DUPLICATE KEY(k1, k2, k3) PARTITION BY RANGE (k1) ( PARTITION p1 VALUES LESS THAN ("2014-01-01"), PARTITION p2 VALUES LESS THAN ("2014-06-01"), PARTITION p3 VALUES LESS THAN ("2014-12-01") ) DISTRIBUTED BY HASH(k2) PROPERTIES( "storage_medium" = "SSD", "storage_cooldown_time" = "2015-06-04 00:00:00" ); ``` Note: This statement will create three data partitions: ```sql ( { MIN }, {"2014-01-01"} ) [ {"2014-01-01"}, {"2014-06-01"} ) [ {"2014-06-01"}, {"2014-12-01"} ) ``` Data outside these ranges will be not be loaded. Fixed Range ```sql CREATE TABLE table_range ( k1 DATE, k2 INT, k3 SMALLINT, v1 VARCHAR(2048), v2 DATETIME DEFAULT "2014-02-04 15:36:00" ) ENGINE=olap DUPLICATE KEY(k1, k2, k3) PARTITION BY RANGE (k1, k2, k3) ( PARTITION p1 VALUES [("2014-01-01", "10", "200"), ("2014-01-01", "20", "300")), PARTITION p2 VALUES [("2014-06-01", "100", "200"), ("2014-07-01", "100", "300")) ) DISTRIBUTED BY HASH(k2) PROPERTIES( "storage_medium" = "SSD" ); ``` ##### MySQL external table[​](#mysql-external-table "Direct link to MySQL external table") ```sql CREATE EXTERNAL TABLE example_db.table_mysql ( k1 DATE, k2 INT, k3 SMALLINT, k4 VARCHAR(2048), k5 DATETIME ) ENGINE=mysql PROPERTIES ( "host" = "127.0.0.1", "port" = "8239", "user" = "mysql_user", "password" = "mysql_passwd", "database" = "mysql_db_test", "table" = "mysql_table_test" ) ``` ##### Table with HLL columns[​](#table-with-hll-columns "Direct link to Table with HLL columns") ```sql CREATE TABLE example_db.example_table ( k1 TINYINT, k2 DECIMAL(10, 2) DEFAULT "10.5", v1 HLL HLL_UNION, v2 HLL HLL_UNION ) ENGINE=olap AGGREGATE KEY(k1, k2) DISTRIBUTED BY HASH(k1) PROPERTIES ("storage_type"="column"); ``` ##### Table using BITMAP\_UNION aggregation type[​](#table-using-bitmap_union-aggregation-type "Direct link to Table using BITMAP_UNION aggregation type") The original data type of `v1` and `v2` columns must be TINYINT, SMALLINT, or INT. ```sql CREATE TABLE example_db.example_table ( k1 TINYINT, k2 DECIMAL(10, 2) DEFAULT "10.5", v1 BITMAP BITMAP_UNION, v2 BITMAP BITMAP_UNION ) ENGINE=olap AGGREGATE KEY(k1, k2) DISTRIBUTED BY HASH(k1) PROPERTIES ("storage_type"="column"); ``` ##### Tables that support Colocate Join[​](#tables-that-support-colocate-join "Direct link to Tables that support Colocate Join") ```sql CREATE TABLE `t1` ( `id` int(11) COMMENT "", `value` varchar(8) COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`id`) DISTRIBUTED BY HASH(`id`) PROPERTIES ( "colocate_with" = "t1" ); CREATE TABLE `t2` ( `id` int(11) COMMENT "", `value` varchar(8) COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`id`) DISTRIBUTED BY HASH(`id`) PROPERTIES ( "colocate_with" = "t1" ); ``` ##### Table with bitmap index[​](#table-with-bitmap-index "Direct link to Table with bitmap index") ```sql CREATE TABLE example_db.table_hash ( k1 TINYINT, k2 DECIMAL(10, 2) DEFAULT "10.5", v1 CHAR(10) REPLACE, v2 INT SUM, INDEX k1_idx (k1) USING BITMAP COMMENT 'xxxxxx' ) ENGINE=olap AGGREGATE KEY(k1, k2) COMMENT "my first starrocks table" DISTRIBUTED BY HASH(k1) PROPERTIES ("storage_type"="column"); ``` ##### Dynamic partition table[​](#dynamic-partition-table "Direct link to Dynamic partition table") The dynamic partitioning function must be enabled ("dynamic\_partition.enable" = "true") in FE configuration. For more information, see [Configure dynamic partitions](#configure-dynamic-partitions). This example creates partitions for the next three days and deletes partitions created three days ago. For example, if today is 2020-01-08, partitions with the following names will be created: p20200108, p20200109, p20200110, p20200111, and their ranges are: ```plaintext [types: [DATE]; keys: [2020-01-08]; ‥types: [DATE]; keys: [2020-01-09]; ) [types: [DATE]; keys: [2020-01-09]; ‥types: [DATE]; keys: [2020-01-10]; ) [types: [DATE]; keys: [2020-01-10]; ‥types: [DATE]; keys: [2020-01-11]; ) [types: [DATE]; keys: [2020-01-11]; ‥types: [DATE]; keys: [2020-01-12]; ) ``` ```sql CREATE TABLE example_db.dynamic_partition ( k1 DATE, k2 INT, k3 SMALLINT, v1 VARCHAR(2048), v2 DATETIME DEFAULT "2014-02-04 15:36:00" ) ENGINE=olap DUPLICATE KEY(k1, k2, k3) PARTITION BY RANGE (k1) ( PARTITION p1 VALUES LESS THAN ("2014-01-01"), PARTITION p2 VALUES LESS THAN ("2014-06-01"), PARTITION p3 VALUES LESS THAN ("2014-12-01") ) DISTRIBUTED BY HASH(k2) PROPERTIES( "storage_medium" = "SSD", "dynamic_partition.enable" = "true", "dynamic_partition.time_unit" = "DAY", "dynamic_partition.start" = "-3", "dynamic_partition.end" = "3", "dynamic_partition.prefix" = "p", "dynamic_partition.buckets" = "10" ); ``` ##### Table with multiple partitions created in a batch, and an integer column as a partitioning column[​](#table-with-multiple-partitions-created-in-a-batch-and-an-integer-column-as-a-partitioning-column "Direct link to Table with multiple partitions created in a batch, and an integer column as a partitioning column") In the following example, the partitioning column `datekey` is of the INT type. All the partitions are created by only one simple partition clause `START ("1") END ("5") EVERY (1)`. The range of all the partitions starts from `1` and ends at `5`, with a partition granularity of `1`: > **NOTE** > > The partitioning column values in **START()** and **END()** need to be wrapped in quotation marks, while the partition granularity in the **EVERY()** does not need to be wrapped in quotation marks. ```sql CREATE TABLE site_access ( datekey INT, site_id INT, city_code SMALLINT, user_name VARCHAR(32), pv BIGINT DEFAULT '0' ) ENGINE=olap DUPLICATE KEY(datekey, site_id, city_code, user_name) PARTITION BY RANGE (datekey) (START ("1") END ("5") EVERY (1) ) DISTRIBUTED BY HASH(site_id) PROPERTIES ("replication_num" = "3"); ``` ##### Hive external table[​](#hive-external-table "Direct link to Hive external table") Before you create a Hive external table, you must have created a Hive resource and database. For more information, see [External table](https://docs.starrocks.io/docs/data_source/External_table.md#deprecated-hive-external-table). ```sql CREATE EXTERNAL TABLE example_db.table_hive ( k1 TINYINT, k2 VARCHAR(50), v INT ) ENGINE=hive PROPERTIES ( "resource" = "hive0", "database" = "hive_db_name", "table" = "hive_table_name" ); ``` ##### Primary Key table with specific sort key[​](#primary-key-table-with-specific-sort-key "Direct link to Primary Key table with specific sort key") Suppose that you need to analyze user behavior in real time from dimensions such as users' address and last active time. When you create a table, you can define the `user_id` column as the primary key and define the combination of the `address` and `last_active` columns as the sort key. ```sql create table users ( user_id bigint NOT NULL, name string NOT NULL, email string NULL, address string NULL, age tinyint NULL, sex tinyint NULL, last_active datetime, property0 tinyint NOT NULL, property1 tinyint NOT NULL, property2 tinyint NOT NULL, property3 tinyint NOT NULL ) PRIMARY KEY (`user_id`) DISTRIBUTED BY HASH(`user_id`) ORDER BY(`address`,`last_active`) PROPERTIES( "replication_num" = "3", "enable_persistent_index" = "true" ); ``` ##### Partitioned temporary table[​](#partitioned-temporary-table "Direct link to Partitioned temporary table") ```sql CREATE TEMPORARY TABLE example_db.temp_table ( k1 DATE, k2 INT, k3 SMALLINT, v1 VARCHAR(2048), v2 DATETIME DEFAULT "2014-02-04 15:36:00" ) ENGINE=olap DUPLICATE KEY(k1, k2, k3) PARTITION BY RANGE (k1) ( PARTITION p1 VALUES LESS THAN ("2014-01-01"), PARTITION p2 VALUES LESS THAN ("2014-06-01"), PARTITION p3 VALUES LESS THAN ("2014-12-01") ) DISTRIBUTED BY HASH(k2); ``` ##### Table with Flat JSON properties[​](#table-with-flat-json-properties "Direct link to Table with Flat JSON properties") ```sql CREATE TABLE example_db.example_table ( k1 DATE, k2 INT, v1 VARCHAR(2048), v2 JSON ) ENGINE=olap DUPLICATE KEY(k1, k2) DISTRIBUTED BY HASH(k2) PROPERTIES ( "flat_json.enable" = "true", "flat_json.null.factor" = "0.5", "flat_json.sparsity.factor" = "0.5", "flat_json.column.max" = "50" ); ``` #### References[​](#references "Direct link to References") * [SHOW CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_CREATE_TABLE.md) * [SHOW TABLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLES.md) * [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) * [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md) * [DROP TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md) --- ### CREATE TABLE AS SELECT Use the CREATE TABLE AS SELECT (CTAS) statement to synchronously or asynchronously query a table and create a new table based on the query result, and then insert the query result into the new table. You can submit an asynchronous CTAS task using [SUBMIT TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md). #### Syntax[​](#syntax "Direct link to Syntax") * Synchronously query a table and create a new table based on the query result, and then insert the query result into the new table. ```sql CREATE [TEMPORARY] TABLE [IF NOT EXISTS] [database.]table_name [column_name1 [, column_name2, ...]] [index_definition1 [, index_definition2, ...]] [key_desc] [COMMENT "table comment"] [partition_desc] [distribution_desc] [ORDER BY (column_name1 [, column_name2, ...])] [PROPERTIES ("key"="value", ...)] AS SELECT query [ ... ] ``` * Asynchronously query a table and create a new table based on the query result, and then insert the query result into the new table. ```sql SUBMIT [/*+ SET_VAR(key=value) */] TASK [[database.]]AS CREATE TABLE [IF NOT EXISTS] [database.]table_name [column_name1 [, column_name2, ...]] [index_definition1 [, index_definition2, ...]] [key_desc] [COMMENT "table comment"] [partition_desc] [distribution_desc] [ORDER BY (column_name1 [, column_name2, ...])] [PROPERTIES ("key"="value", ...)] AS SELECT query [ ... ] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------------ | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | TEMPORARY | No | Creates a temporary table. From v3.3.1, StarRocks supports creating temporary tables in the Default Catalog. For more information, see [Temporary Table](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md#temporary-table). Currently, StarRocks does not support creating temporary tables with asynchronous tasks by using SUBMIT TASK. | | column\_name | No | The name of a column in the new table. You do not need to specify the data type for the column. StarRocks automatically specifies an appropriate data type for the column. StarRocks converts FLOAT and DOUBLE data into DECIMAL(38,9) data. StarRocks also converts CHAR, VARCHAR, and STRING data into VARCHAR data that keeps the declared length (STRING is equivalent to VARCHAR(65533)). String expressions without a declared length, such as the result of `concat()`, are converted into VARCHAR(1048576). | | index\_definition | No | Since v3.1.8, a bitmap index can be created for the new table. The syntax is `INDEX index_name (col_name[, col_name, ...]) [USING BITMAP] COMMENT 'xxxxxx'`. For more information about parameter descriptions and usage notes, see [Bitmap indexes](https://docs.starrocks.io/docs/table_design/indexes/Bitmap_index.md). | | key\_desc | No | The syntax is `key_type ( [, , ...])`.
**Parameters**:- `key_type`: [the key type of the new table](https://docs.starrocks.io/docs/table_design/table_types.md). Valid values: `DUPLICATE KEY` and `PRIMARY KEY`. Default value: `DUPLICATE KEY`.
- `col_name`: the column to form the key. | | COMMENT | No | The comment of the new table. | | partition\_desc | No | The partitioning method of the new table. By default, if you do not specify this parameter, the new table has no partition. For more information about partitioning, see [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md#partition_desc). | | distribution\_desc | No | The bucketing method of the new table. If you do not specify this parameter, the bucket column defaults to the column with the highest cardinality collected by the cost-based optimizer (CBO). The number of buckets defaults to 10. If the CBO does not collect information about the cardinality, the bucket column defaults to the first column in the new table. For more information about bucketing, see [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md#distribution_desc). | | ORDER BY | No | Since v3.1.8, a sort key can be specified for the new table if the new table is a Primary Key table. The sort key can be a combination of any columns. A Primary Key table is a table for which `PRIMARY KEY (xxx)` is specified at table creation. | | Properties | No | The properties of the new table. | | AS SELECT query | Yes | The query result. You can specify columns in `... AS SELECT query`, for example, `... AS SELECT a, b, c FROM table_a;`. In this example, `a`, `b`, and `c` indicates the column names of the table that is queried. If you do not specify the column names of the new table, the column names of the new table are also `a`, `b`, and `c`. You can specify expressions in `... AS SELECT query`, for example, `... AS SELECT a+1 AS x, b+2 AS y, c*c AS z FROM table_a;`. In this example, `a+1`, `b+2`, and `c*c` indicates the column names of the table that is queried, and `x`, `y`, and `z` indicates the column names of the new table. Note: The number of columns in the new table need to be the same as the number of the columns specified in the SELECT statement . We recommend that you use column names that are easy to identify. | #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The CTAS statement can only create a new table that meets the following requirements: * `ENGINE` is `OLAP`. * The table is a Duplicate Key table by default. You can also specify it as a Primary Key table in `key_desc`. * The sort keys are the first three columns, and the storage space of the data types of these three columns does not exceed 36 bytes. * If the CTAS statement fails to execute due to reasons, such as an FE restart, one of the following issues may occur: * A new table is created successfully but does not contain data. * A new table fails to be created. * After a new table is created, if multiple methods (such as INSERT INTO) are used to insert data into the new table, the method that completes the INSERT operation first will insert its data into the new table. * After a new table is created, you need to manually grant permissions on this table to users. * If you do not specify a name for a Task when you asynchronously query a table and create a new table based on the query result, StarRocks automatically generates a name for the Task. #### Examples[​](#examples "Direct link to Examples") Example 1: Synchronously query a table `order` and create a new table `order_new` based on the query result, and then insert the query result into the new table. ```sql CREATE TABLE order_new AS SELECT * FROM `order`; ``` Example 2: Synchronously query the `k1`, `k2`, and `k3` columns in the table `order` and create a new table `order_new` based on the query result, and then insert the query result into the new table. Additionally, set the column names of the new table to `a`, `b`, and `c`. ```sql CREATE TABLE order_new (a, b, c) AS SELECT k1, k2, k3 FROM `order`; ``` or ```sql CREATE TABLE order_new AS SELECT k1 AS a, k2 AS b, k3 AS c FROM `order`; ``` Example 3: Synchronously query the largest value of the `salary` column in the table `employee` and create a new table `employee_new` based on the query result, and then insert the query result into the new table. Additionally, set the column name of the new table to `salary_max`. ```sql CREATE TABLE employee_new AS SELECT MAX(salary) AS salary_max FROM employee; ``` After data is inserted, query the new table. ```sql SELECT * FROM employee_new; +------------+ | salary_max | +------------+ | 10000 | +------------+ ``` Example 4: Synchronously query the `customer_id` and `first_name` columns in the table `customers` and create a new table `customers_new` based on the query result, and then insert the query result into the new table. Additionally, set the column names of the new table to `customer_id_new` and `first_name_new`. Also, build a bitmap index for the column `customer_id_new` in the new table. ```sql CREATE TABLE customers_new ( customer_id_new, first_name_new, INDEX idx_bitmap_customer_id (customer_id_new) USING BITMAP ) AS SELECT customer_id,first_name FROM customers; ``` Example 5: Synchronously query a table `customers` and create a new table `customers_new` based on the query result, and then insert the query result into the new table. Additionally, specify the new table as a Primary Key table, and specify its sort key as `first_name` and `last_name`. ```sql CREATE TABLE customers_pk PRIMARY KEY (customer_id) ORDER BY (first_name,last_name) AS SELECT * FROM customers; ``` Example 6: Use CTAS to create a Primary Key table. Note that the number of data rows in the Primary Key table may be less than that in the query result. It is because the [Primary Key](https://docs.starrocks.io/docs/table_design/table_types/primary_key_table.md) table only stores the most recent data row among a group of rows that have the same primary key. ```sql CREATE TABLE employee_new PRIMARY KEY(order_id) AS SELECT order_list.order_id, sum(goods.price) as total FROM order_list INNER JOIN goods ON goods.item_id1 = order_list.item_id2 GROUP BY order_id; ``` Example 7: Synchronously query four tables, including `lineorder`, `customer`, `supplier`, and `part` and create a new table `lineorder_flat` based on the query result, and then insert the query result to the new table. Additionally, specify the partitioning method and bucketing method for the new table. ```sql CREATE TABLE lineorder_flat PARTITION BY RANGE(`LO_ORDERDATE`) ( START ("1993-01-01") END ("1999-01-01") EVERY (INTERVAL 1 YEAR) ) DISTRIBUTED BY HASH(`LO_ORDERKEY`) AS SELECT l.LO_ORDERKEY AS LO_ORDERKEY, l.LO_LINENUMBER AS LO_LINENUMBER, l.LO_CUSTKEY AS LO_CUSTKEY, l.LO_PARTKEY AS LO_PARTKEY, l.LO_SUPPKEY AS LO_SUPPKEY, l.LO_ORDERDATE AS LO_ORDERDATE, l.LO_ORDERPRIORITY AS LO_ORDERPRIORITY, l.LO_SHIPPRIORITY AS LO_SHIPPRIORITY, l.LO_QUANTITY AS LO_QUANTITY, l.LO_EXTENDEDPRICE AS LO_EXTENDEDPRICE, l.LO_ORDTOTALPRICE AS LO_ORDTOTALPRICE, l.LO_DISCOUNT AS LO_DISCOUNT, l.LO_REVENUE AS LO_REVENUE, l.LO_SUPPLYCOST AS LO_SUPPLYCOST, l.LO_TAX AS LO_TAX, l.LO_COMMITDATE AS LO_COMMITDATE, l.LO_SHIPMODE AS LO_SHIPMODE, c.C_NAME AS C_NAME, c.C_ADDRESS AS C_ADDRESS, c.C_CITY AS C_CITY, c.C_NATION AS C_NATION, c.C_REGION AS C_REGION, c.C_PHONE AS C_PHONE, c.C_MKTSEGMENT AS C_MKTSEGMENT, s.S_NAME AS S_NAME, s.S_ADDRESS AS S_ADDRESS, s.S_CITY AS S_CITY, s.S_NATION AS S_NATION, s.S_REGION AS S_REGION, s.S_PHONE AS S_PHONE, p.P_NAME AS P_NAME, p.P_MFGR AS P_MFGR, p.P_CATEGORY AS P_CATEGORY, p.P_BRAND AS P_BRAND, p.P_COLOR AS P_COLOR, p.P_TYPE AS P_TYPE, p.P_SIZE AS P_SIZE, p.P_CONTAINER AS P_CONTAINER FROM lineorder AS l INNER JOIN customer AS c ON c.C_CUSTKEY = l.LO_CUSTKEY INNER JOIN supplier AS s ON s.S_SUPPKEY = l.LO_SUPPKEY INNER JOIN part AS p ON p.P_PARTKEY = l.LO_PARTKEY; ``` Example 8: Asynchronously query the table `order_detail` and create a new table `order_statistics` based on the query result, and then insert the query result into the new table. ```plaintext SUBMIT TASK AS CREATE TABLE order_statistics AS SELECT COUNT(*) as count FROM order_detail; +-------------------------------------------+-----------+ | TaskName | Status | +-------------------------------------------+-----------+ | ctas-df6f7930-e7c9-11ec-abac-bab8ee315bf2 | SUBMITTED | +-------------------------------------------+-----------+ ``` Check information of the Task. ```sql SELECT * FROM INFORMATION_SCHEMA.tasks; -- Information of the Task TASK_NAME: ctas-df6f7930-e7c9-11ec-abac-bab8ee315bf2 CREATE_TIME: 2022-06-14 14:07:06 SCHEDULE: MANUAL DATABASE: default_cluster:test DEFINITION: CREATE TABLE order_statistics AS SELECT COUNT(*) as cnt FROM order_detail EXPIRE_TIME: 2022-06-17 14:07:06 ``` Check the state of the TaskRun. ```sql SELECT * FROM INFORMATION_SCHEMA.task_runs; -- State of the TaskRun QUERY_ID: 37bd2b63-eba8-11ec-8d41-bab8ee315bf2 TASK_NAME: ctas-df6f7930-e7c9-11ec-abac-bab8ee315bf2 CREATE_TIME: 2022-06-14 14:07:06 FINISH_TIME: 2022-06-14 14:07:07 STATE: SUCCESS DATABASE: DEFINITION: CREATE TABLE order_statistics AS SELECT COUNT(*) as cnt FROM order_detail EXPIRE_TIME: 2022-06-17 14:07:06 ERROR_CODE: 0 ERROR_MESSAGE: NULL ``` Query the new table when the state of the TaskRun is `SUCCESS`. ```sql SELECT * FROM order_statistics; ``` --- ### CREATE TABLE LIKE CREATE TABLE LIKE creates an identical empty table based on the definition of another table. The definition includes column definition, partitions, and table properties. You can copy an external table such as MySQL. v3.2 allows you to specify a different partitioning method, bucketing method, and properties for the new table than the source table. tip This operation requires the CREATE TABLE privilege on the database in which you want to create a table, and the `SELECT` privilege on the source table based on which to create a table. #### Syntax[​](#syntax "Direct link to Syntax") * Syntax supported in versions earlier than v3.2. ```sql CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [database.] LIKE [database.] ``` * v3.2 supports specifying properties for the new table. ```sql CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [database.] [partition_desc] [distribution_desc] [PROPERTIES ("key" = "value",...)] LIKE [database.] ``` #### Parameters[​](#parameters "Direct link to Parameters") * `TEMPORARY`: Creates a temporary table. From v3.3.1, StarRocks supports creating temporary tables in the Default Catalog. For more information, see [Temporary Table](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md#temporary-table). * `database`: the database. * `table_name`: the name of the table you want to create. For the naming conventions, see [System limits](https://docs.starrocks.io/docs/sql-reference/System_limit.md). * `source_table_name`: the name of the source table you want to copy. * * `partition_desc`: the partitioning method. For more information, see [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md#partition_desc). * `distribution_desc`: the bucketing method. For more information, see [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md#distribution_desc). * `PROPERTIES`: the properties of the table. All the table properties are supported. For more information, see [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md#modify-table-properties). #### Examples[​](#examples "Direct link to Examples") Suppose there is a table `orders` in database `test1`. ```sql create table orders ( dt date NOT NULL, order_id bigint NOT NULL, user_id int NOT NULL, merchant_id int NOT NULL, good_id int NOT NULL, good_name string NOT NULL, price int NOT NULL, cnt int NOT NULL, revenue int NOT NULL, state tinyint NOT NULL ) PRIMARY KEY (dt, order_id) PARTITION BY RANGE(`dt`) ( PARTITION p20210820 VALUES [('2021-08-20'), ('2021-08-21')), PARTITION p20210821 VALUES [('2021-08-21'), ('2021-08-22')), PARTITION p20210929 VALUES [('2021-09-29'), ('2021-09-30')), PARTITION p20210930 VALUES [('2021-09-30'), ('2021-10-01')) ) DISTRIBUTED BY HASH(order_id) PROPERTIES ( "replication_num" = "3", "enable_persistent_index" = "true" ); ``` Example 1: In database `test1`, create an empty table `order_1` that has the same table structure as `orders`. ```sql CREATE TABLE test1.order_1 LIKE test1.orders; ``` ```plaintext show create table order_1\G *************************** 1. row *************************** Table: order_1 Create Table: CREATE TABLE `order_1` ( `dt` date NOT NULL COMMENT "", `order_id` bigint(20) NOT NULL COMMENT "", `user_id` int(11) NOT NULL COMMENT "", `merchant_id` int(11) NOT NULL COMMENT "", `good_id` int(11) NOT NULL COMMENT "", `good_name` varchar(65533) NOT NULL COMMENT "", `price` int(11) NOT NULL COMMENT "", `cnt` int(11) NOT NULL COMMENT "", `revenue` int(11) NOT NULL COMMENT "", `state` tinyint(4) NOT NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`dt`, `order_id`) PARTITION BY RANGE(`dt`) (PARTITION p20210820 VALUES [("2021-08-20"), ("2021-08-21")), PARTITION p20210821 VALUES [("2021-08-21"), ("2021-08-22")), PARTITION p20210929 VALUES [("2021-09-29"), ("2021-09-30")), PARTITION p20210930 VALUES [("2021-09-30"), ("2021-10-01"))) DISTRIBUTED BY HASH(`order_id`) PROPERTIES ( "replication_num" = "3", "in_memory" = "false", "enable_persistent_index" = "true", "replicated_storage" = "true", "compression" = "LZ4" ); ``` Example 2: Create an empty table `order_2` based on `orders` and specify properties for `order_2`. ```sql CREATE TABLE order_2 PARTITION BY date_trunc('day',dt) DISTRIBUTED BY hash(dt) PROPERTIES ("replication_num" = "1") LIKE orders; ``` ```plaintext show create table order_2\G *************************** 1. row *************************** Table: order_2 Create Table: CREATE TABLE `order_2` ( `dt` date NOT NULL COMMENT "", `order_id` bigint(20) NOT NULL COMMENT "", `user_id` int(11) NOT NULL COMMENT "", `merchant_id` int(11) NOT NULL COMMENT "", `good_id` int(11) NOT NULL COMMENT "", `good_name` varchar(65533) NOT NULL COMMENT "", `price` int(11) NOT NULL COMMENT "", `cnt` int(11) NOT NULL COMMENT "", `revenue` int(11) NOT NULL COMMENT "", `state` tinyint(4) NOT NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`dt`, `order_id`) PARTITION BY RANGE(date_trunc('day', dt)) () DISTRIBUTED BY HASH(`dt`) PROPERTIES ( "replication_num" = "1", "in_memory" = "false", "enable_persistent_index" = "true", "replicated_storage" = "true", "compression" = "LZ4" ); ``` Example 3: Create an empty table `table2` that has the same table structure as MySQL external table `table1`. ```sql CREATE TABLE test1.table2 LIKE test1.table1 ``` --- ### DELETE Deletes data rows from a table based on the specified conditions. The table can be a partitioned or non-partitioned table. For Duplicate Key tables, Aggregate tables, and Unique Key tables, you can delete data from specified partitions. From v2.3, Primary Key tables support complete `DELETE...WHERE` semantics, which allows you to delete data rows based on the primary key, any column, or the results of a subquery. From v3.0, StarRocks enriches the `DELETE...WHERE` semantics with multi-table joins and common table expressions (CTEs). If you need to join Primary Key tables with other tables in the database, you can reference these other tables in the USING clause or CTE. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * You must have privileges on the table and database you want to perform DELETE. * Frequent DELETE operations are not recommended. If needed, perform such operations during off-peak hours. * The DELETE operation only deletes data in the table. The table remains. To drop the table, run [DROP TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md). * To prevent misoperations from deleting data in the entire table, you must specify the WHERE clause in the DELETE statement. * The deleted rows are not immediately cleaned. They are marked as "deleted" and will be temporarily saved in Segment. Physically, the rows are removed only after data version merge (compaction) is completed. * This operation also deletes data of the materialized views that reference this table. * You can use the system variable `insert_timeout` to set the timeout duration for DELETE operations. #### Duplicate Key tables, Aggregate tables, and Unique Key tables[​](#duplicate-key-tables-aggregate-tables-and-unique-key-tables "Direct link to Duplicate Key tables, Aggregate tables, and Unique Key tables") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql DELETE FROM [.] [PARTITION ] WHERE column_name1 op { value | value_list } [ AND column_name2 op { value | value_list } ...] ``` ##### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ---------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------- | | `db_name` | No | The database to which the destination table belongs. If this parameter is not specified, the current database is used by default. | | `table_name` | Yes | The table from which you want to delete data. | | `partition_name` | No | The partition from which you want to delete data. | | `column_name` | Yes | The column you want to use as the DELETE condition. You can specify one or more columns. | | `op` | Yes | The operator used in the DELETE condition. The supported operators are `=`, `>`, `<`, `>=`, `<=`, `!=`, `IN`, and `NOT IN`. | ##### Limits and usage notes[​](#limits-and-usage-notes "Direct link to Limits and usage notes") * For Duplicate Key tables, you can use **any column** as the DELETE condition. For Aggregate tables and Unique Key tables, only **key columns** can be used as the DELETE condition. * The conditions that you specify must be in the AND relation. If you want to specify conditions in OR relation, you must specify the conditions in separate DELETE statements. * For Duplicate Key tables, Aggregate tables, and Unique Key tables, the DELETE statement does not support using subquery results as conditions. ##### Impacts[​](#impacts "Direct link to Impacts") After you execute the DELETE statement, the query performance of your cluster may deteriorate for a period of time (before compaction is completed). The degree of deterioration varies based on the number of conditions that you specify. A larger number of conditions indicates a higher degree of deterioration. ##### Examples[​](#examples "Direct link to Examples") ###### Create a table and insert data[​](#create-a-table-and-insert-data "Direct link to Create a table and insert data") The following example creates a partitioned Duplicate Key table. ```sql CREATE TABLE `my_table` ( `date` date NOT NULL, `k1` int(11) NOT NULL COMMENT "", `k2` varchar(65533) NULL DEFAULT "" COMMENT "") DUPLICATE KEY(`date`) PARTITION BY RANGE(`date`) ( PARTITION p1 VALUES [('2022-03-11'), ('2022-03-12')), PARTITION p2 VALUES [('2022-03-12'), ('2022-03-13')) ) DISTRIBUTED BY HASH(`date`) PROPERTIES ("replication_num" = "3"); INSERT INTO `my_table` VALUES ('2022-03-11', 3, 'abc'), ('2022-03-11', 2, 'acb'), ('2022-03-11', 4, 'abc'), ('2022-03-12', 2, 'bca'), ('2022-03-12', 4, 'cba'), ('2022-03-12', 5, 'cba'); ``` ###### Query data[​](#query-data "Direct link to Query data") ```plain select * from my_table order by date; +------------+------+------+ | date | k1 | k2 | +------------+------+------+ | 2022-03-11 | 3 | abc | | 2022-03-11 | 2 | acb | | 2022-03-11 | 4 | abc | | 2022-03-12 | 2 | bca | | 2022-03-12 | 4 | cba | | 2022-03-12 | 5 | cba | +------------+------+------+ ``` ###### Delete data[​](#delete-data "Direct link to Delete data") ###### Delete data from a specified partition[​](#delete-data-from-a-specified-partition "Direct link to Delete data from a specified partition") Delete rows whose `k1` values are `3` from the `p1` partition. ```plain DELETE FROM my_table PARTITION p1 WHERE k1 = 3; -- The query result shows rows whose `k1` values are `3` are deleted. select * from my_table partition (p1); +------------+------+------+ | date | k1 | k2 | +------------+------+------+ | 2022-03-11 | 2 | acb | | 2022-03-11 | 4 | abc | +------------+------+------+ ``` ###### Delete data from a specified partition using AND[​](#delete-data-from-a-specified-partition-using-and "Direct link to Delete data from a specified partition using AND") Delete rows whose `k1` values are greater than or equal to `3` and whose `k2` values are `"abc"` from the `p1` partition. ```plain DELETE FROM my_table PARTITION p1 WHERE k1 >= 3 AND k2 = "abc"; select * from my_table partition (p1); +------------+------+------+ | date | k1 | k2 | +------------+------+------+ | 2022-03-11 | 2 | acb | +------------+------+------+ ``` ###### Delete data from all partitions[​](#delete-data-from-all-partitions "Direct link to Delete data from all partitions") Delete rows whose `k2` values are `"abc"` or `"cba"` from all partitions. ```plain DELETE FROM my_table WHERE k2 in ("abc", "cba"); select * from my_table order by date; +------------+------+------+ | date | k1 | k2 | +------------+------+------+ | 2022-03-11 | 2 | acb | | 2022-03-12 | 2 | bca | +------------+------+------+ ``` #### Primary Key tables[​](#primary-key-tables "Direct link to Primary Key tables") From v2.3, Primary Key tables support complete `DELETE...WHERE` semantics, which allows you to delete data rows based on the primary key, any column, or a subquery. ##### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql [ WITH [, ...] ] DELETE FROM [ USING [, ...] ] [ WHERE ] ``` ##### Parameters[​](#parameters-1 "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ----------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `with_query` | No | One or more CTEs that can be referenced by name in a DELETE statement. CTEs are temporary result sets that can improve the readability of complex statements. | | `table_name` | Yes | The table from which you want to delete data. | | `from_item` | No | One or more other tables in the database. These tables can be joined with the table being operated based on the condition specified in the WHERE clause. Based on the result set of the join query, StarRocks deletes the matched rows from the table being operated. For example, if the USING clause is `USING t1 WHERE t0.pk = t1.pk;`, StarRocks converts the table expression in the USING clause to `t0 JOIN t1 ON t0.pk=t1.pk;` when executing the DELETE statement. | | `where_condition` | Yes | The condition based on which you want to delete rows. Only rows that meet the WHERE condition can be deleted. This parameter is required, because it helps prevent you from accidentally deleting the entire table. If you want to delete the entire table, you can use 'WHERE true'. | ##### Limits and usage notes[​](#limits-and-usage-notes-1 "Direct link to Limits and usage notes") * Primary Key tables do not support deleting data from a specified partition, for example, `DELETE FROM PARTITION WHERE `. * The following comparison operators are supported: `=`, `>`, `<`, `>=`, `<=`, `!=`, `IN`, `NOT IN`. * The following logical operators are supported: `AND` and `OR`. * You cannot use the DELETE statement to run concurrent DELETE operations or to delete data at data loading. If you perform such operations, the atomicity, consistency, isolation, and durability (ACID) of transactions may not be ensured. ##### Examples[​](#examples-1 "Direct link to Examples") ###### Create a table and insert data[​](#create-a-table-and-insert-data-1 "Direct link to Create a table and insert data") Create a Primary Key table named `score_board`: ```sql CREATE TABLE `score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`) PROPERTIES ( "replication_num" = "3", "storage_format" = "DEFAULT", "enable_persistent_index" = "true" ); INSERT INTO score_board VALUES (0, 'Jack', 21), (1, 'Bob', 21), (2, 'Stan', 21), (3, 'Sam', 22); ``` ###### Query data[​](#query-data-1 "Direct link to Query data") Execute the following statement to insert data into the `score_board` table: ```plain select * from score_board; +------+------+------+ | id | name | score| +------+------+------+ | 0 | Jack | 21 | | 1 | Bob | 21 | | 2 | Stan | 21 | | 3 | Sam | 22 | +------+------+------+ 4 rows in set (0.00 sec) ``` ###### Delete data[​](#delete-data-1 "Direct link to Delete data") ###### Delete data by primary key[​](#delete-data-by-primary-key "Direct link to Delete data by primary key") You can specify the primary key in the DELETE statement, so StarRocks does not need to scan the entire table. Delete rows whose `id` values are `0` from the `score_board` table. ```plain DELETE FROM score_board WHERE id = 0; select * from score_board; +------+------+------+ | id | name | score| +------+------+------+ | 1 | Bob | 21 | | 2 | Stan | 21 | | 3 | Sam | 22 | +------+------+------+ ``` ###### Delete data by condition[​](#delete-data-by-condition "Direct link to Delete data by condition") Example 1: Delete rows whose `score` values are `22` from the `score_board` table. ```plain DELETE FROM score_board WHERE score = 22; select * from score_board; +------+------+------+ | id | name | score| +------+------+------+ | 0 | Jack | 21 | | 1 | Bob | 21 | | 2 | Stan | 21 | +------+------+------+ ``` Example 2: Delete rows whose `score` values are less than `22` from the `score_board` table. ```plain DELETE FROM score_board WHERE score < 22; select * from score_board; +------+------+------+ | id | name | score| +------+------+------+ | 3 | Sam | 22 | +------+------+------+ ``` Example 3: Delete rows whose `score` values are less than `22` and whose `name` values are not `Bob` from the `score_board` table. ```plain DELETE FROM score_board WHERE score < 22 and name != "Bob"; select * from score_board; +------+------+------+ | id | name | score| +------+------+------+ | 1 | Bob | 21 | | 3 | Sam | 22 | +------+------+------+ 2 rows in set (0.00 sec) ``` ###### Delete data by subquery result[​](#delete-data-by-subquery-result "Direct link to Delete data by subquery result") You can nest one or more subqueries in the `DELETE` statement and use the subquery results as conditions. Before you delete data, execute the following statement to create another table named `users`: ```sql CREATE TABLE `users` ( `uid` int(11) NOT NULL COMMENT "", `name` varchar(65533) NOT NULL COMMENT "", `country` varchar(65533) NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`uid`) COMMENT "OLAP" DISTRIBUTED BY HASH(`uid`) PROPERTIES ( "replication_num" = "3", "storage_format" = "DEFAULT", "enable_persistent_index" = "true" ); ``` Insert data into the `users` table: ```sql INSERT INTO users VALUES (0, "Jack", "China"), (2, "Stan", "USA"), (1, "Bob", "China"), (3, "Sam", "USA"); ``` ```plain select * from users; +------+------+---------+ | uid | name | country | +------+------+---------+ | 0 | Jack | China | | 1 | Bob | China | | 2 | Stan | USA | | 3 | Sam | USA | +------+------+---------+ 4 rows in set (0.00 sec) ``` Nest a subquery to find the rows whose `country` values are `China` from the `users` table, and delete the rows, which have the same `name` values as the rows returned from the subquery, from the `score_board` table. You can use one of the following methods to fulfill the purpose: * Method 1 ```plain DELETE FROM score_board WHERE name IN (select name from users where country = "China"); select * from score_board; +------+------+------+ | id | name | score| +------+------+------+ | 2 | Stan | 21 | | 3 | Sam | 22 | +------+------+------+ ``` * Method 2 ```plain DELETE FROM score_board WHERE EXISTS (select name from users where score_board.name = users.name and country = "China"); select * from score_board; +------+------+-------+ | id | name | score | +------+------+-------+ | 2 | Stan | 21 | | 3 | Sam | 22 | +------+------+-------+ 2 rows in set (0.00 sec) ``` ###### Delete data by using multi-table join or CTE[​](#delete-data-by-using-multi-table-join-or-cte "Direct link to Delete data by using multi-table join or CTE") To delete all movies produced by the producer "foo", you can execute the following statement: ```sql DELETE FROM films USING producers WHERE producer_id = producers.id AND producers.name = 'foo'; ``` You can also use a CTE to rewrite the above statement to improve readability. ```sql WITH foo_producers as ( SELECT * from producers where producers.name = 'foo' ) DELETE FROM films USING foo_producers WHERE producer_id = foo_producers.id; ``` #### References[​](#references "Direct link to References") [SHOW DELETE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_DELETE.md): queries historical DELETE operations that were successfully performed on Duplicate Key tables, Aggregate tables, and Unique Key tables. --- ### DESC DESC is used to view table schemas: * View the schema of a table stored in your StarRocks cluster, along with the type of the [sort key](https://docs.starrocks.io/docs/table_design/indexes/Prefix_index_sort_key.md) and [materialized view](https://docs.starrocks.io/docs/using_starrocks/async_mv/Materialized_view.md) of the table. * View the schema of a table stored in the following external data sources, such as Apache Hive™. Note that you can perform this operation only in StarRocks 2.4 and later versions. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DESC[RIBE] { [[.].] [ALL] | FILES(files_loading_properties) } ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | catalog\_name | No | The name of the internal catalog or an external catalog. - If you set the value of the parameter to the name of the internal catalog, which is `default_catalog`, you can view the schema of the table stored in your StarRocks cluster.
- If you set the value of the parameter to the name of an external catalog, you can view the schema of the table stored in the external data source. | | db\_name | No | The database name. | | table\_name | Yes | The table name. | | ALL | No | - If this keyword is specified, you can view the type of the sort key, materialized view, and schema of a table stored in your StarRocks cluster. If this keyword is not specified, you only view the table schema.
- Do not specify this keyword when you view the schema of a table stored in an external data source. | | FILES | No | The FILES() table function. From v3.3.4 onwards, you can use DESC with FILES() to view the schema information of files stored in remote storage. For detailed information, see [Function reference - FILES()](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md). | #### Output[​](#output "Direct link to Output") ```plain +-----------+---------------+-------+------+------+-----+---------+-------+ | IndexName | IndexKeysType | Field | Type | Null | Key | Default | Extra | +-----------+---------------+-------+------+------+-----+---------+-------+ ``` The following table describes the parameters returned by this statement. | **Parameter** | **Description** | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | IndexName | The table name. If you view the schema of a table stored in an external data source, this parameter is not returned. | | IndexKeysType | The type of the sort key of the table. If you view the schema of a table stored in an external data source, this parameter is not returned. | | Field | The column name. | | Type | The data type of the column. | | Null | Whether the column values can be NULL. - `yes`: indicates the values can be NULL.
- `no`: indicates the values cannot be NULL. | | Key | Whether the column is used as the sort key. - `true`: indicates the column is used as the sort key.
- `false`: indicates the column is not used as the sort key. | | Default | The default value for the data type of the column. If the data type does not have a default value, a NULL is returned. | | Extra | - If you see the schema of a table stored in your StarRocks cluster, this field displays the following information about the column:- The aggregate function used by the column, such as `SUM` and `MIN`. * Whether a bloom filter index is created on the column. If so, the value of `Extra` is `BLOOM_FILTER`.
- If you see the schema of a table stored in external data sources, this field displays whether the column is the partition column. If the column is the partition column, the value of `Extra` is `partition key`. | > Note: For information about how a materialized view is displayed in the output, see Example 2. #### Examples[​](#examples "Direct link to Examples") Example 1: View the schema of `example_table` stored in your StarRocks cluster. ```sql DESC example_table; ``` Or ```sql DESC default_catalog.example_db.example_table; ``` The output of the preceding statements is as follows. ```plain +-------+---------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+---------------+------+-------+---------+-------+ | k1 | TINYINT | Yes | true | NULL | | | k2 | DECIMAL(10,2) | Yes | true | 10.5 | | | k3 | CHAR(10) | Yes | false | NULL | | | v1 | INT | Yes | false | NULL | | +-------+---------------+------+-------+---------+-------+ ``` Example 2: View the schema, type of the sort key, and materialized view of `sales_records` stored in your StarRocks cluster. In the following example, one materialized view `store_amt` is created based on `sales_records`. ```plain DESC db1.sales_records ALL; +---------------+---------------+-----------+--------+------+-------+---------+-------+ | IndexName | IndexKeysType | Field | Type | Null | Key | Default | Extra | +---------------+---------------+-----------+--------+------+-------+---------+-------+ | sales_records | DUP_KEYS | record_id | INT | Yes | true | NULL | | | | | seller_id | INT | Yes | true | NULL | | | | | store_id | INT | Yes | true | NULL | | | | | sale_date | DATE | Yes | false | NULL | NONE | | | | sale_amt | BIGINT | Yes | false | NULL | NONE | | | | | | | | | | | store_amt | AGG_KEYS | store_id | INT | Yes | true | NULL | | | | | sale_amt | BIGINT | Yes | false | NULL | SUM | +---------------+---------------+-----------+--------+------+-------+---------+-------+ ``` Example 3: View the schema of `hive_table` stored in your Hive cluster. ```plain DESC hive_catalog.hive_db.hive_table; +-------+----------------+------+-------+---------+---------------+ | Field | Type | Null | Key | Default | Extra | +-------+----------------+------+-------+---------+---------------+ | id | INT | Yes | false | NULL | | | name | VARCHAR(65533) | Yes | false | NULL | | | date | DATE | Yes | false | NULL | partition key | +-------+----------------+------+-------+---------+---------------+ ``` Example 4: View the schema of parquet file `lineorder` stored in AWS S3. > **NOTE** > > For files stored in remote storage, DESC only returns three fields: `Field`, `Type`, and `Null`. ```plain DESC FILES( "path" = "s3://inserttest/lineorder.parquet", "format" = "parquet", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.region" = "us-west-2" ); +------------------+------------------+------+ | Field | Type | Null | +------------------+------------------+------+ | lo_orderkey | int | YES | | lo_linenumber | int | YES | | lo_custkey | int | YES | | lo_partkey | int | YES | | lo_suppkey | int | YES | | lo_orderdate | int | YES | | lo_orderpriority | varchar(1048576) | YES | | lo_shippriority | int | YES | | lo_quantity | int | YES | | lo_extendedprice | int | YES | | lo_ordtotalprice | int | YES | | lo_discount | int | YES | | lo_revenue | int | YES | | lo_supplycost | int | YES | | lo_tax | int | YES | | lo_commitdate | int | YES | | lo_shipmode | varchar(1048576) | YES | +------------------+------------------+------+ 17 rows in set (0.05 sec) ``` #### References[​](#references "Direct link to References") * [CREATE DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/CREATE_DATABASE.md) * [SHOW CREATE DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_CREATE_DATABASE.md) * [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) * [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) * [DROP DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/DROP_DATABASE.md) --- ### DROP INDEX DROP INDEX is used to drop a specified index on a table. Currently, only bitmap index is supported in this version. tip This operation requires the ALTER privilege on the target table. You can follow the instructions in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant this privilege. Syntax: ```sql DROP INDEX index_name ON [db_name.]table_name ``` --- ### DROP TABLE DROP TABLE is used to delete a table. #### Syntax[​](#syntax "Direct link to Syntax") ```sql DROP [TEMPORARY] TABLE [IF EXISTS] [db_name.]table_name [FORCE] ``` Note: * After a table is dropped, you can use the [RECOVER](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/RECOVER.md) statement to restore the table within a specified period (1 day by default). * If DROP Table FORCE is executed, the table will be deleted directly and cannot be recovered without checking whether there are unfinished activities in the database. Generally, this operation is not recommended. * Once dropped, temporary tables cannot be restored using RECOVER. #### Examples[​](#examples "Direct link to Examples") 1. Drop a table. ```sql DROP TABLE my_table; ``` 2. If it exists, then drop the table on the specified database. ```sql DROP TABLE IF EXISTS example_db.my_table; ``` 3. Force to drop the table and clear its data on disk. ```sql DROP TABLE my_table FORCE; ``` #### References[​](#references "Direct link to References") * [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) * [SHOW TABLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLES.md) * [SHOW CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_CREATE_TABLE.md) * [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md) * [SHOW ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_ALTER.md) --- ### REFRESH EXTERNAL TABLE REFRESH EXTERNAL TABLE metadata cached in StarRocks. The metadata is from tables in data lakes. This statement is used in the following scenarios: * **External table**: When using a Hive external table or Hudi external table to query data in Apache Hive™ or Apache Hudi, you can execute this statement to update the metadata of a Hive table or Hudi table cached in StarRocks. * **External catalog**: When using an [external catalog](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md) (except for JDBC Catalog) to query data in the corresponding data source, you can execute this statement to update the metadata cached in StarRocks. > **NOTE** > > Only users with the ALTER privilege on the corresponding external table can perform this operation. #### Basic concepts[​](#basic-concepts "Direct link to Basic concepts") * **Hive external table**: is created and stored in StarRocks. You can use it to query Hive data. * **Hudi external table**: is created and stored in StarRocks. You can use it to query Hudi data. * **Hive table**: is created and stored in Hive. * **Hudi table**: is created and stored in Hudi. #### Syntaxes and parameters[​](#syntaxes-and-parameters "Direct link to Syntaxes and parameters") The following describes the syntaxes and parameters based on different cases: * External table ```sql REFRESH EXTERNAL TABLE table_name [PARTITION ('partition_name', ...)] ``` | **Parameter** | **Required** | **Description** | | --------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | table\_name | Yes | The name of a Hive external table or Hudi external table. | | partition\_name | No | The names of the partitions of a Hive table or Hudi table. Specifying this parameter updates the metadata of the partitions of the Hive table and Hudi table cached in StarRocks. | * External catalog ```sql REFRESH EXTERNAL TABLE [external_catalog.][db_name.]table_name [PARTITION ('partition_name', ...)] ``` | **Parameter** | **Required** | **Description** | | ----------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | | external\_catalog | No | The name of the external catalog, which supports Hive, Hudi, Delta Lake, and MaxCompute (since v3.3) catalogs. | | db\_name | No | The name of the database where the destination table resides. | | table\_name | Yes | The name of the table. | | partition\_name | No | The names of the partitions. Specifying this parameter updates the metadata of the partitions of the destination table cached in StarRocks. | #### Usage notes[​](#usage-notes "Direct link to Usage notes") Only users who have the `ALTER_PRIV` privilege can execute this statement to update the metadata of Hive tables and Hudi tables cached in StarRocks. #### Examples[​](#examples "Direct link to Examples") Usage examples in different cases are as follows: ##### External table[​](#external-table "Direct link to External table") Example 1: Update the cached metadata of the corresponding Hive table in StarRocks by specifying the external table `hive1`. ```sql REFRESH EXTERNAL TABLE hive1; ``` Example 2: Update the cached metadata of the partitions of the corresponding Hudi table in StarRocks by specifying the external table `hudi1` and the partitions in that table. ```sql REFRESH EXTERNAL TABLE hudi1 PARTITION ('date=2022-12-20', 'date=2022-12-21'); ``` ##### External catalog[​](#external-catalog "Direct link to External catalog") Example 1: Update the metadata of `hive_table` cached in StarRocks. ```sql REFRESH EXTERNAL TABLE hive_catalog.hive_db.hive_table; ``` Or ```sql USE hive_catalog.hive_db; REFRESH EXTERNAL TABLE hive_table; ``` Example 2: Update the metadata of the second-level partition `p2` in `hive_table` cached in StarRocks. ```sql USE hive_catalog.hive_db; REFRESH EXTERNAL TABLE hive_table PARTITION ('p1=${date}/p2=${hour}'); ``` Example 3: Update the metadata of the partitions of `hudi_table` cached in StarRocks. ```sql REFRESH EXTERNAL TABLE hudi_catalog.hudi_db.hudi_table PARTITION ('date=2022-12-20', 'date=2022-12-21'); ``` --- ### SELECT SELECT queries data from one or more tables, views, or materialized views. SELECT can work as an independent statement or a clause nested in other statements. The output of the SELECT clause can be used as the input of other statements. StarRocks' query statement basically conforms to the SQL92 standard. note To query data from tables, views, or materialized views in a StarRocks internal table, you must have the SELECT privilege on these objects. To query data from tables, views, or materialized views in an external data source, you must have the USAGE privilege on the corresponding external catalog. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SELECT select_list FROM table_reference [, ...] ``` The SELECT statement generally consists of the following clauses: * [Common Table Expression (CTE)](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT_CTE.md) * [JOINs](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT_JOIN.md) * [ORDER BY](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT_ORDER_BY.md) * [GROUP BY](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT_GROUP_BY.md) * [HAVING](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT_HAVING.md) * [LIMIT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT_LIMIT.md) * [OFFSET](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT_OFFSET.md) * [UNION](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT_UNION.md) * [INTERSECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT_INTERSECT.md) * [EXCEPT/MINUS](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT_EXCEPT_MINUS.md) * [DISTINCT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT_DISTINCT.md) * [Subquery](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT_subquery.md) * [WHERE and operators](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT_WHERE_operator.md) * [Alias](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT_alias.md) * [PIVOT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT_PIVOT.md) * [EXCLUDE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT_EXCLUDE.md) For detailed instructions, refer to the corresponding topics. --- ### Alias When you write the names of tables, columns, or expressions that contain columns in a query, you can assign them an alias. Aliases are usually shorter and better to remember than original names. When an alias is needed, you can simply add an AS clause after the table, column, and expression names in the select list or from list. The AS keyword is optional. You can also specify aliases directly after the original name without using AS. If an alias or other identifier has the same name as an internal [StarRocks keyword](https://docs.starrocks.io/docs/sql-reference/sql-statements/keywords.md), you need to enclose the name in a pair of backticks, for example, `rank`. Aliases are case-sensitive, but column aliases and expression aliases are not case-sensitive. Examples: ```sql select tiny_column as name, int_column as sex from big_table; select sum(tiny_column) as total_count from big_table; select one.tiny_column, two.int_column from small_table one,
big_table two where one.tiny_column = two.tiny_column; ``` --- ### Common Table Expression You can use a common table expression (CTE) to define a temporary result set that you can reference possibly multiple times within the scope of a SQL statement. #### WITH[​](#with "Direct link to WITH") A clause that can be added before a SELECT statement to define an alias for a complex expression that is referenced multiple times inside SELECT. Similar to CRATE VIEW, but the table and column names defined in the clause do not persist after the query ends and do not conflict with names in the actual table or VIEW. The benefits of using a WITH clause are: Convenient and easy to maintain, reducing duplication within queries. It is easier to read and understand SQL code by abstracting the most complex parts of a query into separate blocks. Examples: ```sql -- Define one subquery at the outer level, and another at the inner level as part of the initial stage of the UNION ALL query. with t1 as (select 1),t2 as (select 2) select * from t1 union all select * from t2; ``` #### RECURSIVE[​](#recursive "Direct link to RECURSIVE") From v4.1 onwards, StarRocks supports Recursive Common Table Expression (CTE), using an iterative execution approach to efficiently handle various hierarchical and tree-structured data. A Recursive CTE is a special type of CTE that can reference itself, enabling recursive queries. Recursive CTEs are particularly useful for processing hierarchical data structures such as organizational charts, file systems, graph traversals, and more. A Recursive CTE consists of the following components: * **Anchor Member**: A non-recursive initial query that provides the starting dataset for recursion. * **Recursive Member**: A recursive query that references the CTE itself. * **Termination Condition**: A condition to prevent infinite recursion, typically implemented through a WHERE clause. The execution process of a Recursive CTE is as follows: 1. Execute the anchor member to obtain the initial result set (level 0). 2. Use level 0 results as input, execute the recursive member to get level 1 results. 3. Use level 1 results as input, execute the recursive member again to get level 2 results. 4. Repeat this process until the recursive member returns no rows or reaches the maximum recursion depth. 5. Merge all levels of results using UNION ALL (or UNION). tip You must enable this feature by setting the system variable `enable_recursive_cte` to `true` before using it. ##### Syntax[​](#syntax "Direct link to Syntax") ```sql WITH RECURSIVE cte_name [(column_list)] AS ( -- Anchor member (non-recursive part) anchor_query UNION [ALL | DISTINCT] -- Recursive member (recursive part) recursive_query ) SELECT ... FROM cte_name ...; ``` ##### Parameters[​](#parameters "Direct link to Parameters") * `cte_name`: Name of the CTE. * `column_list` (optional): List of column names for the CTE result set. * `anchor_query`: Initial query that must be non-recursive and cannot reference the CTE itself. * `UNION`: Union operator. * `UNION ALL`: Retains all rows (including duplicates), recommended for better performance. * `UNION` or `UNION DISTINCT`: Removes duplicate rows. * `recursive_query`: Recursive query that references the CTE itself. ##### Limitations[​](#limitations "Direct link to Limitations") Recursive CTEs in StarRocks have the following limitations: * **Feature flag required** You must manually enable Recursive CTE by setting the system variable `enable_recursive_cte` to `true`. * **Structural requirements** * UNION or UNION ALL must be used to connect anchor and recursive members. * The anchor member cannot reference the CTE itself. * If the recursive member does not reference the CTE itself, it is executed as a regular CTE. * **Recursion depth limit** * By default, the maximum recursion depth is 5 (levels). * The maximum depth can be adjusted via the system variable `recursive_cte_max_depth` to prevent infinite recursion. * **Execution Constraints** * Currently, multi-level nested recursive CTEs are not supported. * Complex recursive CTEs may lead to degraded performance. * Constants in `anchor_query` should have types consistent with `recursive_query` output types. ##### Configurations[​](#configurations "Direct link to Configurations") The following system variables are required for using Recursive CTE: | Variable Name | Type | Default | Description | | ------------------------- | ------- | ------- | ------------------------------------------------------ | | `enable_recursive_cte` | BOOLEAN | false | Whether to enable Recursive CTE. | | `recursive_cte_max_depth` | INT | 5 | Maximum recursion depth to prevent infinite recursion. | ##### Examples[​](#examples "Direct link to Examples") **Example 1: Querying Organizational Hierarchy** Querying organizational hierarchy is one of the most common use cases for Recursive CTE. The following example queries employee organizational hierarchy relationships. 1. Prepare data: ```sql CREATE TABLE employees ( employee_id INT, name VARCHAR(100), manager_id INT, title VARCHAR(50) ) DUPLICATE KEY(employee_id) DISTRIBUTED BY RANDOM; INSERT INTO employees VALUES (1, 'Alicia', NULL, 'CEO'), (2, 'Bob', 1, 'CTO'), (3, 'Carol', 1, 'CFO'), (4, 'David', 2, 'VP of Engineering'), (5, 'Eve', 2, 'VP of Research'), (6, 'Frank', 3, 'VP of Finance'), (7, 'Grace', 4, 'Engineering Manager'), (8, 'Heidi', 4, 'Tech Lead'), (9, 'Ivan', 5, 'Research Manager'), (10, 'Judy', 7, 'Senior Engineer'); ``` 2. Query Organizational Hierarchy: ```sql WITH RECURSIVE org_hierarchy AS ( -- Anchor member: Start from CEO (employee with no manager) SELECT employee_id, name, manager_id, title, CAST(1 AS BIGINT) AS level, name AS path FROM employees WHERE manager_id IS NULL UNION ALL -- Recursive member: Find subordinates at next level SELECT e.employee_id, e.name, e.manager_id, e.title, oh.level + 1, CONCAT(oh.path, ' -> ', e.name) AS path FROM employees e INNER JOIN org_hierarchy oh ON e.manager_id = oh.employee_id ) SELECT /*+ SET_VAR(enable_recursive_cte=true, recursive_cte_max_depth=10) */ employee_id, name, title, level, path FROM org_hierarchy ORDER BY employee_id; ``` Result: ```plain +-------------+---------+----------------------+-------+-----------------------------------------+ | employee_id | name | title | level | path | +-------------+---------+----------------------+-------+-----------------------------------------+ | 1 | Alicia | CEO | 1 | Alicia | | 2 | Bob | CTO | 2 | Alicia -> Bob | | 3 | Carol | CFO | 2 | Alicia -> Carol | | 4 | David | VP of Engineering | 3 | Alicia -> Bob -> David | | 5 | Eve | VP of Research | 3 | Alicia -> Bob -> Eve | | 6 | Frank | VP of Finance | 3 | Alicia -> Carol -> Frank | | 7 | Grace | Engineering Manager | 4 | Alicia -> Bob -> David -> Grace | | 8 | Heidi | Tech Lead | 4 | Alicia -> Bob -> David -> Heidi | | 9 | Ivan | Research Manager | 4 | Alicia -> Bob -> Eve -> Ivan | | 10 | Judy | Senior Engineer | 5 | Alicia -> Bob -> David -> Grace -> Judy | +-------------+---------+----------------------+-------+-----------------------------------------+ ``` **Example 2: Multiple Recursive CTEs** You can define multiple recursive CTEs in a single query. ```sql WITH RECURSIVE cte1 AS ( SELECT CAST(1 AS BIGINT) AS n UNION ALL SELECT n + 1 FROM cte1 WHERE n < 5 ), cte2 AS ( SELECT CAST(10 AS BIGINT) AS n UNION ALL SELECT n + 1 FROM cte2 WHERE n < 15 ) SELECT /*+ SET_VAR(enable_recursive_cte=true, recursive_cte_max_depth=10) */ 'cte1' AS source, n FROM cte1 UNION ALL SELECT 'cte2' AS source, n FROM cte2 ORDER BY source, n; ``` --- ### DISTINCT The DISTINCT keyword deduplicates the result set. Example: ```sql -- Returns the unique values from one column. select distinct tiny_column from big_table limit 2; -- Returns the unique combinations of values from multiple columns. select distinct tiny_column, int_column from big_table limit 2; ``` DISTINCT can be used with aggregate functions (usually count functions), and count (distinct) is used to calculate how many different combinations are contained on one or more columns. ```sql -- Counts the unique values from one column. select count(distinct tiny_column) from small_table; ``` ```plain +-------------------------------+ | count(DISTINCT 'tiny_column') | +-------------------------------+ | 2 | +-------------------------------+ 1 row in set (0.06 sec) ``` ```sql -- Counts the unique combinations of values from multiple columns. select count(distinct tiny_column, int_column) from big_table limit 2; ``` StarRocks supports multiple aggregate functions using distinct at the same time. ```sql -- Count the unique value from multiple aggregation function separately. select count(distinct tiny_column, int_column), count(distinct varchar_column) from big_table; ``` --- ### EXCEPT/MINUS Returns distinct results of the left-hand query that do not exist in the right-hand query. EXCEPT is equivalent to MINUS. #### Syntax[​](#syntax "Direct link to Syntax") ```sql query_1 {EXCEPT | MINUS} [DISTINCT] query_2 ``` > **NOTE** > > * EXCEPT is equivalent to EXCEPT DISTINCT. The ALL keyword is not supported. > * Each query statement must return the same number of columns and the columns must have compatible data types. #### Examples[​](#examples "Direct link to Examples") The two tables in UNION are used. Return distinct `(id, price)` combinations in `select1` that cannot be found in `select2`. ```plaintext mysql> (select id, price from select1) except (select id, price from select2) order by id; +------+-------+ | id | price | +------+-------+ | 1 | 2 | +------+-------+ mysql> (select id, price from select1) minus (select id, price from select2) order by id; +------+-------+ | id | price | +------+-------+ | 1 | 2 | +------+-------+ ``` --- ### EXCLUDE This feature is supported starting from version 4.0. The EXCLUDE keyword is used to exclude specified columns from query results, simplifying SQL statements when certain columns can be ignored. It is particularly convenient when working with tables containing a large number of columns, avoiding the need to explicitly list all columns to be retained. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SELECT * EXCLUDE ( [, ...]) | .* EXCLUDE ( [, ...]) FROM ... ``` #### Parameters[​](#parameters "Direct link to Parameters") * **`* EXCLUDE`**
Selects all columns with the wildcard `*`, followed by `EXCLUDE` and a list of column names to exclude. * **`.* EXCLUDE`**
When a table alias exists, this allows excluding specific columns from that table (must be used with the alias). * **``**
The column name(s) to exclude. Multiple columns are separated by commas. Columns must exist in the table; otherwise, an error will be returned. #### Examples[​](#examples "Direct link to Examples") * Basic Usage: ```sql -- Create test_table. CREATE TABLE test_table ( id INT, name VARCHAR(50), age INT, email VARCHAR(100) ) DUPLICATE KEY(id); -- Exclude a single column (age). SELECT * EXCLUDE (age) FROM test_table; -- Above is equivalent to: SELECT id, name, email FROM test_table; -- Exclude multiple columns (name, email). SELECT * EXCLUDE (name, email) FROM test_table; -- Above is equivalent to: SELECT id, age FROM test_table; -- Exclude columns using a table alias. SELECT test_table.* EXCLUDE (email) FROM test_table; -- Above is equivalent to: SELECT id, name, age FROM test_table; ``` --- ### GROUP BY The GROUP BY clause is often used with aggregate functions. Columns specified in the GROUP BY clause will not participate in the aggregation operation. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SELECT ... aggregate_function() [ FILTER ( where boolean_expression ) ] ... FROM ... [ ... ] GROUP BY [ , ... | GROUPING SETS [, ...] ( groupSet [ , groupSet [ , ... ] ] ) | ROLLUP(expr [ , expr [ , ... ] ]) | CUBE(expr [ , expr [ , ... ] ]) ] [ ... ] ``` #### Parameters[​](#parameters "Direct link to Parameters") * `FILTER` can be used together with aggregate functions. Only filtered rows will participate in the calculation of the aggregate function. > **NOTE** > > * The FILTER clause is only supported in AVG, COUNT, MAX, MIN, SUM, ARRAY\_AGG, and ARRAY\_AGG\_DISTINCT functions. > * The FILTER clause is not supported for COUNT DISTINCT. > * When the FILTER clause is specified, ORDER BY clauses are not allowed within ARRAY\_AGG and ARRAY\_AGG\_DISTINCT functions. * `GROUPING SETS`, `CUBE`, and `ROLLUP` are extensions of the GROUP BY clause. In a GROUP BY clause, they can be used to achieve grouped aggregations of multiple sets. The results are equivalent to that of the UNION of multiple GROUP BY clauses. #### Examples[​](#examples "Direct link to Examples") Example 1: `FILTER` The following two queries are equivalent. ```sql SELECT COUNT(*) AS total_users, SUM(CASE WHEN gender = 'M' THEN 1 ELSE 0 END) AS male_users, SUM(CASE WHEN gender = 'F' THEN 1 ELSE 0 END) AS female_users FROM users; ``` ```sql SELECT COUNT(*) AS total_users, COUNT(*) FILTER (WHERE gender = 'M') AS male_users, COUNT(*) FILTER (WHERE gender = 'F') AS female_users FROM users; ``` Example 2: `GROUPING SETS`, `CUBE`, and `ROLLUP` `ROLLUP(a,b,c)` is equivalent to the following `GROUPING SETS` clause. ```sql GROUPING SETS ( (a,b,c), (a,b ), (a ), ( ) ) ``` `CUBE (a, b, c)` is equivalent to the following `GROUPING SETS` clause. ```sql GROUPING SETS ( ( a, b, c ), ( a, b ), ( a, c ), ( a ), ( b, c ), ( b ), ( c ), ( ) ) ``` Test in an actual dataset. ```sql SELECT * FROM t; +------+------+------+ | k1 | k2 | k3 | +------+------+------+ | a | A | 1 | | a | A | 2 | | a | B | 1 | | a | B | 3 | | b | A | 1 | | b | A | 4 | | b | B | 1 | | b | B | 5 | +------+------+------+ 8 rows in set (0.01 sec) SELECT k1, k2, SUM(k3) FROM t GROUP BY GROUPING SETS ( (k1, k2), (k2), (k1), ( ) ); +------+------+-----------+ | k1 | k2 | sum(`k3`) | +------+------+-----------+ | b | B | 6 | | a | B | 4 | | a | A | 3 | | b | A | 5 | | NULL | B | 10 | | NULL | A | 8 | | a | NULL | 7 | | b | NULL | 11 | | NULL | NULL | 18 | +------+------+-----------+ 9 rows in set (0.06 sec) SELECT k1, k2, GROUPING_ID(k1,k2), SUM(k3) FROM t GROUP BY GROUPING SETS ((k1, k2), (k1), (k2), ()); +------+------+---------------+----------------+ | k1 | k2 | grouping_id(k1,k2) | sum(`k3`) | +------+------+---------------+----------------+ | a | A | 0 | 3 | | a | B | 0 | 4 | | a | NULL | 1 | 7 | | b | A | 0 | 5 | | b | B | 0 | 6 | | b | NULL | 1 | 11 | | NULL | A | 2 | 8 | | NULL | B | 2 | 10 | | NULL | NULL | 3 | 18 | +------+------+---------------+----------------+ 9 rows in set (0.02 sec) ``` --- ### HAVING The HAVING clause does not filter row data in a table, but filters the results of aggregate functions. Generally speaking, HAVING is used with aggregate functions (such as COUNT(), SUM(), AVG(), MIN(), MAX()) and GROUP BY clauses. #### Examples[​](#examples "Direct link to Examples") ```sql select tiny_column, sum(short_column) from small_table group by tiny_column having sum(short_column) = 1; ``` ```plain +-------------+---------------------+ |tiny_column | sum('short_column') | +-------------+---------------------+ | 2 | 1 | +-------------+---------------------+ 1 row in set (0.07 sec) ``` ```sql select tiny_column, sum(short_column) from small_table group by tiny_column having tiny_column > 1; ``` ```plain +-------------+---------------------+ |tiny_column | sum('short_column') | +-------------+---------------------+ | 2 | 1 | +-------------+---------------------+ 1 row in set (0.07 sec) ``` --- ### INTERSECT Calculates the intersection of the results of multiple queries, that is, the results that appear in all the result sets. This clause returns only unique rows among the result sets. The ALL keyword is not supported. #### Syntax[​](#syntax "Direct link to Syntax") ```sql query_1 INTERSECT [DISTINCT] query_2 ``` > **NOTE** > > * INTERSECT is equivalent to INTERSECT DISTINCT. > * Each query statement must return the same number of columns and the columns must have compatible data types. #### Examples[​](#examples "Direct link to Examples") The two tables in UNION are used. Return distinct `(id, price)` combinations that are common to both tables. The following two statements are equivalent. ```plaintext mysql> (select id, price from select1) intersect (select id, price from select2) order by id; +------+-------+ | id | price | +------+-------+ | 2 | 3 | | 5 | 6 | +------+-------+ mysql> (select id, price from select1) intersect distinct (select id, price from select2) order by id; +------+-------+ | id | price | +------+-------+ | 2 | 3 | | 5 | 6 | +------+-------+ ``` --- ### JOIN Join operations combine data from two or more tables and then return a result set of some columns from some of them. StarRocks supports the following joins: * [Self Join](#self-join) * [Cross Join](#cross-join) * [Inner Join](#inner-join) * [Outer Join](#outer-join) (including Left Join, Right Join, and Full Join) * [Semi Join](#semi-join) * [Anti Join](#anti-join) * [Equi-join and Non-equi-join](#equi-join-and-non-equi-join) * [Join with the USING clause](#join-with-the-using-clause) * [ASOF Join](#asof-join) #### Syntax[​](#syntax "Direct link to Syntax") ```sql SELECT select_list FROM table_or_subquery1 [INNER] JOIN table_or_subquery2 | table_or_subquery1 {LEFT [OUTER] | RIGHT [OUTER] | FULL [OUTER]} JOIN table_or_subquery2 | table_or_subquery1 {LEFT | RIGHT} SEMI JOIN table_or_subquery2 | table_or_subquery1 {LEFT | RIGHT} ANTI JOIN table_or_subquery2 | [ ON col1 = col2 [AND col3 = col4 ...] | USING (col1 [, col2 ...]) ] [other_join_clause ...] [ WHERE where_clauses ] ``` ```sql SELECT select_list FROM table_or_subquery1, table_or_subquery2 [, table_or_subquery3 ...] [other_join_clause ...] WHERE col1 = col2 [AND col3 = col4 ...] ``` ```sql SELECT select_list FROM table_or_subquery1 CROSS JOIN table_or_subquery2 [other_join_clause ...] [ WHERE where_clauses ] ``` #### Self Join[​](#self-join "Direct link to Self Join") StarRocks supports self-joins. For example, different columns of the same table are joined. There is actually no special syntax to identify self-join. The conditions on both sides of a join in a self-join come from the same table. We need to assign them different aliases. Examples: ```sql SELECT lhs.id, rhs.parent, lhs.c1, rhs.c2 FROM tree_data lhs, tree_data rhs WHERE lhs.id = rhs.parent; ``` #### Cross Join[​](#cross-join "Direct link to Cross Join") Cross join can produce a lot of results, so cross join should be used with caution. Even if you need to use cross join, you need to use filter conditions and ensure that fewer results are returned. Example: ```sql SELECT * FROM t1, t2; SELECT * FROM t1 CROSS JOIN t2; ``` #### Inner Join[​](#inner-join "Direct link to Inner Join") Inner join is the most well-known and commonly used join. Returns results from columns requested by two similar tables, joined if the columns of both tables contain the same value. If a column name of both tables is the same, we need to use the full name (in the form of table\_name.column\_name) or alias the column name. Examples: The following three queries are equivalent. ```sql SELECT t1.id, c1, c2 FROM t1, t2 WHERE t1.id = t2.id; SELECT t1.id, c1, c2 FROM t1 JOIN t2 ON t1.id = t2.id; SELECT t1.id, c1, c2 FROM t1 INNER JOIN t2 ON t1.id = t2.id; ``` #### Outer Join[​](#outer-join "Direct link to Outer Join") Outer join returns the left or right table or all rows of both. If there is no matching data in another table, set it to NULL. Example: ```sql SELECT * FROM t1 LEFT OUTER JOIN t2 ON t1.id = t2.id; SELECT * FROM t1 RIGHT OUTER JOIN t2 ON t1.id = t2.id; SELECT * FROM t1 FULL OUTER JOIN t2 ON t1.id = t2.id; ``` #### Equal and unequal join[​](#equal-and-unequal-join "Direct link to Equal and unequal join") Usually, equal joins are the most commonly used joins. It requires the operator of the join condition to be an equal sign. Unequal join can be used on join conditions `!=`. Unequal joins produce a large number of results and may exceed the memory limit during calculation. Use with caution. Unequal join only supports Inner Join. Example: ```sql SELECT t1.id, c1, c2 FROM t1 INNER JOIN t2 ON t1.id = t2.id; SELECT t1.id, c1, c2 FROM t1 INNER JOIN t2 ON t1.id > t2.id; ``` #### Semi Join[​](#semi-join "Direct link to Semi Join") Left semi join returns only the rows in the left table that match the data in the right table, regardless of how many rows match the data in the right table. This row of the left table is returned at most once. Right semi join works similarly, except that the data returned is a right table. Examples: ```sql SELECT t1.c1, t1.c2, t1.c2 FROM t1 LEFT SEMI JOIN t2 ON t1.id = t2.id; ``` #### Anti Join[​](#anti-join "Direct link to Anti Join") Left anti join only returns rows from the left table that do not match the right table. Right anti join reverses this comparison, returning only rows from the right table that do not match the left table. Example: ```sql SELECT t1.c1, t1.c2, t1.c2 FROM t1 LEFT ANTI JOIN t2 ON t1.id = t2.id; ``` #### Equi-join and Non-equi-join[​](#equi-join-and-non-equi-join "Direct link to Equi-join and Non-equi-join") The various joins supported by StarRocks can be classified as equi-joins and non-equi-joins depending on the join conditions specified in the joins. | **Join Type** | **Variants** | | -------------- | ----------------------------------------------------------------------------- | | Equi-joins | Self joins, cross joins, inner joins, outer joins, semi joins, and anti joins | | Non-equi-joins | cross joins, inner joins, left semi joins, left anti joins, and outer joins | * Equi-joins An equi-join uses a join condition in which two join items are combined by the `=` operator. Example: `a JOIN b ON a.id = b.id`. * Non-equi-joins A non-equi-join uses a join condition in which two join items are combined by a comparison operator such as `<`, `<=`, `>`, `>=`, or `<>`. Example: `a JOIN b ON a.id < b.id`. Non-equi-joins run slower than equi-joins. We recommend that you exercise caution when you use non-equi-joins. The following two examples show how to run non-equi-joins: ```sql SELECT t1.id, c1, c2 FROM t1 INNER JOIN t2 ON t1.id < t2.id; SELECT t1.id, c1, c2 FROM t1 LEFT JOIN t2 ON t1.id > t2.id; ``` #### Join with the USING clause[​](#join-with-the-using-clause "Direct link to Join with the USING clause") From v4.0.2 onwards, StarRocks supports specifying join conditions via the `USING` clause in addition to `ON`. It helps to simplify equi-joins with columns of the same name. For example: `SELECT * FROM t1 JOIN t2 USING (id)`. **Differences between versions:** * **Versions before v4.0.2** `USING` is treated as syntactic sugar and internally converted to an `ON` condition. The result would include USING columns from both the left and right tables as separate columns, and table alias qualifiers (for example, `t1.id`) were allowed when referencing USING columns. Example: ```sql SELECT t1.id, t2.id FROM t1 JOIN t2 USING (id); -- Returns two separate id columns ``` * **v4.0.2 and later** StarRocks implements SQL-standard `USING` semantics. Key features include: * All join types are supported, including `FULL OUTER JOIN`. * USING columns appear as a single coalesced column in results. For FULL OUTER JOIN, the `COALESCE(left.col, right.col)` semantics is used. * Table alias qualifiers (for example, `t1.id`) are no longer supported for USING columns. You must use unqualified column names (for example, `id`). * For the result of `SELECT *`, the column order is `[USING columns, left non-USING columns, right non-USING columns]`. Example: ```sql SELECT t1.id FROM t1 JOIN t2 USING (id); -- ❌ Error: Column 'id' is ambiguous SELECT id FROM t1 JOIN t2 USING (id); -- ✅ Correct: Returns a single coalesced 'id' column SELECT * FROM t1 FULL OUTER JOIN t2 USING (id); -- ✅ FULL OUTER JOIN is supported ``` These changes align StarRocks' behavior with SQL-standard compliant databases. #### ASOF Join[​](#asof-join "Direct link to ASOF Join") An ASOF Join is a type of temporal or range-based join commonly used in time-series analytics. It allows joining two tables based on equality of certain keys and a non-equality condition on time or sequence fields, for example, `t1.time >= t2.time`. The ASOF Join selects the most recent matching row from the right-side table for each row on the left-side table. Supported from v4.0 onwards. In real-world scenarios, analytics involving time-series data often encounter the following challenges: * Data collection timing misalignment (for example, different sensor sampling times) * Small discrepancies between event occurrence and recording times * Need to find the closest historical record for a given timestamp Traditional equality joins (INNER Join) often result in significant data loss when handling such data, while inequality joins can lead to performance issues. The ASOF Join was designed to address these specific challenges. ASOF Joins are commonly used in the following cases: * **Financial Market Analysis** * Matching stock prices with trading volumes * Aligning data from different markets * Derivative pricing reference data matching * **IoT Data Processing** * Aligning multiple sensor data streams * Correlating device state changes * Time-series data interpolation * **Log Analysis** * Correlating system events with user actions * Matching logs from different services * Fault analysis and problem tracking Syntax: ```sql SELECT [select_list] FROM left_table [AS left_alias] ASOF LEFT JOIN right_table [AS right_alias] ON equality_condition AND asof_condition [WHERE ...] [ORDER BY ...] ``` * `ASOF LEFT JOIN`: Performs a non-equality join based on the nearest match in time or sequence. ASOF LEFT JOIN returns all rows from the left table, filling unmatched right-side rows with NULL. * `equality_condition`: A standard equality constraint (for example, matching ticker symbols or IDs). * `asof_condition`: A range condition typically written as `left.time >= right.time`, indicating to search the most recent `right.time` records that does not exceed `left.time`. note Only DATE and DATETIME types are supported in `asof_condition`. And only one `asof_condition` is supported. Example: ```sql SELECT * FROM holdings h ASOF LEFT JOIN prices p ON h.ticker = p.ticker AND h.`when` >= p.`when` ORDER BY ALL; ``` Limitations: * Currently, only Inner Join (default) and Left Outer Join are supported. * Only DATE and DATETIME types are supported in `asof_condition`. * Only one `asof_condition` is supported. --- ### LIMIT LIMIT clauses are used to limit the maximum number of rows returned. Setting the maximum number of rows returned can help StarRocks optimize memory usage. This clause is mainly used in the following scenarios: Returns the result of the top-N query. Think about what's included in the table below. The size of the query result set needs to be limited because of the large amount of data in the table or because the where clause does not filter too much data. Instructions for use: The value of the LIMIT clause must be a numeric literal constant. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select tiny_column from small_table limit 1; +-------------+ |tiny_column | +-------------+ | 1 | +-------------+ 1 row in set (0.02 sec) ``` ```plain mysql> select tiny_column from small_table limit 10000; +-------------+ |tiny_column | +-------------+ | 1 | | 2 | +-------------+ 2 rows in set (0.01 sec) ``` --- ### OFFSET The OFFSET clause causes the result set to skip the first few rows and return the following results directly. The result set defaults to start at line 0, so offset 0 and no offset return the same results. Generally speaking, OFFSET clauses need to be used with ORDER BY and LIMIT clauses to be valid. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> select varchar_column from big_table order by varchar_column limit 3; +----------------+ | varchar_column | +----------------+ | beijing | | chongqing | | tianjin | +----------------+ 3 rows in set (0.02 sec) ``` ```plain mysql> select varchar_column from big_table order by varchar_column limit 1 offset 0; +----------------+ |varchar_column | +----------------+ | beijing | +----------------+ 1 row in set (0.01 sec) ``` ```plain mysql> select varchar_column from big_table order by varchar_column limit 1 offset 1; +----------------+ |varchar_column | +----------------+ | chongqing | +----------------+ 1 row in set (0.01 sec) ``` ```plain mysql> select varchar_column from big_table order by varchar_column limit 1 offset 2; +----------------+ |varchar_column | +----------------+ | tianjin | +----------------+ 1 row in set (0.02 sec) ``` #### Usage notes[​](#usage-notes "Direct link to Usage notes") It is allowed to use offset syntax without order by, but offset does not make sense at this time. In this case, only the limit value is taken, and the offset value is ignored. So without order by. Offset exceeds the maximum number of rows in the result set and is still a result. It is recommended that users use offset with order by. --- ### ORDER BY The ORDER BY clause of a SELECT statement sorts the result set by comparing the values from one or more columns. ORDER BY is a time- and resource-consuming operation because all the results must be sent to one node for merging before the results can be sorted. Sorting consumes more memory resources than a query without ORDER BY. Therefore, if you only need the first `N` results from the sorted result set, you can use the LIMIT clause, which reduces memory usage and network overhead. If the LIMIT clause is not specified, the first 65535 results are returned by default. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ORDER BY [ASC | DESC] [NULLS FIRST | NULLS LAST] ``` #### Parameters[​](#parameters "Direct link to Parameters") * `ASC` specifies that the results should be returned in ascending order. * `DESC` specifies that the results should be returned in descending order. If the order is not specified, ASC (ascending) is the default. * `NULLS FIRST` indicates that NULL values should be returned before non-NULL values. * `NULLS LAST` indicates that NULL values should be returned after non-NULL values. #### Examples[​](#examples "Direct link to Examples") ```sql select * from big_table order by tiny_column, short_column desc; select * from sales_record order by employee_id nulls first; ``` --- ### PIVOT This feature is supported from v3.3 onwards. The PIVOT operation is an advanced feature in SQL that allows you to transform rows into columns in a table, which is particularly useful for creating pivot tables. This comes in handy when dealing with database reports or analytics, especially when you need to summarize or categorize data for presentation. Actually, the PIVOT is a syntax sugar, which can simplify the writing of the query statement like `sum(case when ... then ... end)`. #### Syntax[​](#syntax "Direct link to Syntax") ```sql pivot: SELECT ... FROM ... PIVOT ( aggregate_function() [[AS] alias] [, aggregate_function() [[AS] alias] ...] FOR IN () ) pivot_column: | ( [, ...]) pivot_value: [, ...] | (, ...) [, (, ...)] ``` #### Parameters[​](#parameters "Direct link to Parameters") In a PIVOT operation, you need to specify several key components: * aggregate\_function(): An aggregate function such as SUM, AVG, COUNT, etc., used to summarize the data. * alias: An alias for the aggregated result, making the outcome more understandable. * FOR pivot\_column: Specifies the column name on which the row-to-column transformation will be performed. * IN (pivot\_value): Specifies the specific values of the pivot\_column that will be transformed into columns. #### Examples[​](#examples "Direct link to Examples") ```sql create table t1 (c0 int, c1 int, c2 int, c3 int); SELECT * FROM t1 PIVOT (SUM(c1) AS sum_c1, AVG(c2) AS avg_c2 FOR c3 IN (1, 2, 3, 4, 5)); -- The result is equivalent to the following query: SELECT SUM(CASE WHEN c3 = 1 THEN c1 ELSE NULL END) AS sum_c1_1, AVG(CASE WHEN c3 = 1 THEN c2 ELSE NULL END) AS avg_c2_1, SUM(CASE WHEN c3 = 2 THEN c1 ELSE NULL END) AS sum_c1_2, AVG(CASE WHEN c3 = 2 THEN c2 ELSE NULL END) AS avg_c2_2, SUM(CASE WHEN c3 = 3 THEN c1 ELSE NULL END) AS sum_c1_3, AVG(CASE WHEN c3 = 3 THEN c2 ELSE NULL END) AS avg_c2_3, SUM(CASE WHEN c3 = 4 THEN c1 ELSE NULL END) AS sum_c1_4, AVG(CASE WHEN c3 = 4 THEN c2 ELSE NULL END) AS avg_c2_4, SUM(CASE WHEN c3 = 5 THEN c1 ELSE NULL END) AS sum_c1_5, AVG(CASE WHEN c3 = 5 THEN c2 ELSE NULL END) AS avg_c2_5 FROM t1 GROUP BY c0; ``` --- ### Subquery Subqueries are categorized into two types in terms of relevance: * A non-correlated subquery obtains its results independently of its outer query. * A correlated subquery requires values from its outer query. #### Non-correlated subquery[​](#non-correlated-subquery "Direct link to Non-correlated subquery") Non-correlated subqueries support \[NOT] IN and EXISTS. Examples: ```sql SELECT x FROM t1 WHERE x [NOT] IN (SELECT y FROM t2); SELECT * FROM t1 WHERE (x,y) [NOT] IN (SELECT x,y FROM t2 LIMIT 2); SELECT x FROM t1 WHERE EXISTS (SELECT y FROM t2 WHERE y = 1); ``` From v3.0 onwards, you can specify multiple fields in the WHERE clause of `SELECT... FROM... WHERE... [NOT] IN`, for example, `WHERE (x,y)` in the second SELECT statement. #### Correlated subquery[​](#correlated-subquery "Direct link to Correlated subquery") Related subqueries support \[NOT] IN and \[NOT] EXISTS. Examples: ```sql SELECT * FROM t1 WHERE x [NOT] IN (SELECT a FROM t2 WHERE t1.y = t2.b); SELECT * FROM t1 WHERE [NOT] EXISTS (SELECT a FROM t2 WHERE t1.y = t2.b); ``` Subqueries also support scalar quantum queries. It can be divided into irrelevant scalar quantum query, related scalar quantum query and scalar quantum query as parameters of the general function. #### Examples[​](#examples "Direct link to Examples") 1. Uncorrelated scalar quantum query with predicate = sign. For example, output information about the person with the highest wage. ```sql SELECT name FROM employees WHERE salary = (SELECT MAX(salary) FROM employees); ``` 2. Uncorrelated scalar quantum queries with predicates `>`, `<` etc. For example, output information about people who are paid more than average. ```sql SELECT name FROM employees WHERE salary > (SELECT AVG(salary) FROM employees); ``` 3. Related scalar quantum queries. For example, output the highest salary information for each department. ```sql SELECT name FROM employees a WHERE salary = (SELECT MAX(salary) FROM employees b WHERE b.Department= a.Department); ``` 4. Scalar quantum queries are used as parameters of ordinary functions. ```sql SELECT name FROM employees WHERE salary = abs((SELECT MAX(salary) FROM employees)); ``` --- ### UNION Combines the result of multiple queries. #### Syntax[​](#syntax "Direct link to Syntax") ```sql query_1 UNION [DISTINCT | ALL] query_2 ``` #### Parameters[​](#parameters "Direct link to Parameters") * `DISTINCT` (default): returns only unique rows. UNION is equivalent to UNION DISTINCT. * `ALL`: combines all rows, including duplicates. Since de-duplication is memory intensive, queries using UNION ALL are faster and less memory-consuming. For better performance, use UNION ALL. > **NOTE** > > Each query statement must return the same number of columns and the columns must have compatible data types. #### Examples[​](#examples "Direct link to Examples") Create tables `select1` and `select2`. ```sql CREATE TABLE select1( id INT, price INT ) DISTRIBUTED BY HASH(id); INSERT INTO select1 VALUES (1,2), (1,2), (2,3), (5,6), (5,6); CREATE TABLE select2( id INT, price INT ) DISTRIBUTED BY HASH(id); INSERT INTO select2 VALUES (2,3), (3,4), (5,6), (7,8); ``` Example 1: Return all IDs in the two tables, including duplicates. ```plaintext mysql> (select id from select1) union all (select id from select2) order by id; +------+ | id | +------+ | 1 | | 1 | | 2 | | 2 | | 3 | | 5 | | 5 | | 5 | | 7 | +------+ 11 rows in set (0.02 sec) ``` Example 2: Return all the unique IDs in the two tables. The following two statements are equivalent. ```plaintext mysql> (select id from select1) union (select id from select2) order by id; +------+ | id | +------+ | 1 | | 2 | | 3 | | 5 | | 7 | +------+ 6 rows in set (0.01 sec) mysql> (select id from select1) union distinct (select id from select2) order by id; +------+ | id | +------+ | 1 | | 2 | | 3 | | 5 | | 7 | +------+ 5 rows in set (0.02 sec) ``` Example 3: Return the first three IDs among all the unique IDs in the two tables. The following two statements are equivalent. ```sql mysql> (select id from select1) union distinct (select id from select2) order by id limit 3; ++------+ | id | +------+ | 1 | | 2 | | 3 | +------+ 4 rows in set (0.11 sec) mysql> select * from (select id from select1 union distinct select id from select2) as t1 order by id limit 3; +------+ | id | +------+ | 1 | | 2 | | 3 | +------+ 3 rows in set (0.01 sec) ``` --- ### Where and Operators SQL operators are a series of functions used for comparison and are widely used in where clauses of select statements. #### Arithmetic operator[​](#arithmetic-operator "Direct link to Arithmetic operator") Arithmetic operators usually appear in expressions that contain left, right, and most often left operands ##### `+` and `-`[​](#-and-- "Direct link to -and--") `+` and `-` can be used as a unit or as a 2-ary operator. When used as a unit operator, such as `+1`, `-2.5` or `-col_ name`, which means the value is multiplied by `+1` or `-1`. So the cell operator `+` returns an unchanged value, and the cell operator `-` changes the symbol bits of that value. Users can overlay two cell operators, such as `+5` (returning a positive value), `-+2` or `+-2` (returning a negative value), but users cannot use two consecutive `-` signs, because `--` is interpreted as a comment in the following statement (when a user can use two `-` signs, a space or parenthesis is required between the two `-` signs, such as `-(-2)` or `- -2`, which actually results in `+ 2`). When `+` or `-` is a binary operator, such as `2+2`, `3+1.5`, or `col1+col2`, it means that the left value is added or subtracted from the right value. Both left and right values must be numeric types. ##### `*` and `/`[​](#-and- "Direct link to -and-") `*` and `/` represent multiplication and division respectively. The operands on both sides must be data types. When two numbers are multiplied. Smaller operands may be promoted if needed (e.g., SMALLINT to INT or BIGINT), and the result of the expression will be promoted to the next larger type. For example, TINYINT multiplied by INT will produce a BIGINT type of result. When two numbers are multiplied, both operands and expression results are interpreted as DOUBLE types to avoid loss of precision. If the user wants to convert the result of the expression to another type, it needs to be converted using the CAST function. ##### `%` Modulation operator. Returns the remainder of the left operand divided by the right operand. Both left and right operands must be integers. ##### `&`, `|` and `^`[​](#--and- "Direct link to --and-") The bitwise operator returns the result of bitwise AND, bitwise OR, bitwise XOR operations on two operands. Both operands require an integer type. If the types of the two operands of a bitwise operator are inconsistent, the operands of a smaller type are promoted to the operands of a larger type, and the corresponding bitwise operations are performed. Multiple arithmetic operators can appear in an expression, and the user can enclose the corresponding arithmetic expression in parentheses. Arithmetic operators usually do not have corresponding mathematical functions to express the same functions as arithmetic operators. For example, we don't have the MOD() function to represent the% operator. Conversely, mathematical functions do not have corresponding arithmetic operators. For example, the power function POW() does not have a corresponding \*\* exponentiation operator. Users can find out which arithmetic functions we support through the Mathematical Functions section. #### `BETWEEN` Operator[​](#between-operator "Direct link to between-operator") In a where clause, expressions may be compared with both upper and lower bounds. If the expression is greater than or equal to the lower bound and less than or equal to the upper bound, the result of the comparison is true. Syntax: ```sql expression BETWEEN lower_bound AND upper_bound ``` Data type: Usually an expression evaluates to a numeric type, which also supports other data types. If you must ensure that both the lower and upper bounds are comparable characters, you can use the cast() function. Instructions for use: If the operand is of type string, note that a long string starting with an upper bound will not match the upper bound, which is larger than the upper bound. For example, "between'A'and'M' will not match 'MJ'. If you need to make sure the expression works correctly, you can use functions such as upper(), lower(), substr(), trim(). Example: ```sql select c1 from t1 where month between 1 and 6; ``` #### Comparison operators[​](#comparison-operators "Direct link to Comparison operators") Comparison operators are used to compare two values. `=`, `!=`, `>=` apply to all data types. The `<>` and `!=` operators are equivalent, which indicate that two values are not equal. #### `IN` Operator[​](#in-operator "Direct link to in-operator") The In operator compares to the VALUE collection and returns TRUE if it can match any of the elements in the collection. Parameters and VALUE collections must be comparable. All expressions using the IN operator can be written as equivalent comparisons connected with OR, but the syntax of IN is simpler, more precise, and easier for StarRocks to optimize. Examples: ```sql select * from small_table where tiny_column in (1,2); ``` #### `LIKE` Operator[​](#like-operator "Direct link to like-operator") This operator is used to compare to a string. '\_' (underscore) matches a single character, '%' matches multiple characters. The parameter must match the complete string. Typically, placing'%'at the end of a string is more practical. Examples: ```plain mysql> select varchar_column from small_table where varchar_column like 'm%'; +----------------+ |varchar_column | +----------------+ | milan | +----------------+ 1 row in set (0.02 sec) ``` ```plain mysql> select varchar_column from small_table where varchar_column like 'm____'; +----------------+ | varchar_column | +----------------+ | milan | +----------------+ 1 row in set (0.01 sec) ``` #### Logical Operator[​](#logical-operator "Direct link to Logical Operator") Logical operators return a BOOL value, including unit and multiple operators, each of which handles parameters that are expressions that return BOOL values. Supported operators are: AND: 2-ary operator, the AND operator returns TRUE if the parameters on the left and right are both calculated as TRUE. OR: 2-ary operator that returns TRUE if one of the parameters on the left and right is calculated as TRUE. If both parameters are FALSE, the OR operator returns FALSE. NOT: Unit operator, the result of inverting an expression. If the parameter is TRUE, the operator returns FALSE; If the parameter is FALSE, the operator returns TRUE. Examples: ```plain mysql> select true and true; +-------------------+ | (TRUE) AND (TRUE) | +-------------------+ | 1 | +-------------------+ 1 row in set (0.00 sec) ``` ```plain mysql> select true and false; +--------------------+ | (TRUE) AND (FALSE) | +--------------------+ | 0 | +--------------------+ 1 row in set (0.01 sec) ``` ```plain mysql> select true or false; +-------------------+ | (TRUE) OR (FALSE) | +-------------------+ | 1 | +-------------------+ 1 row in set (0.01 sec) ``` ```plain mysql> select not true; +----------+ | NOT TRUE | +----------+ | 0 | +----------+ 1 row in set (0.01 sec) ``` #### Regular Expression Operator[​](#regular-expression-operator "Direct link to Regular Expression Operator") Determines whether the regular expression is matched. Using POSIX standard regular expressions,'^'matches the first part of the string,'$' matches the end of the string. "." matches any single character, "\*" matches zero or more options, "+" matches one or more options, "?" means greedy representation, and so on. Regular expressions need to match complete values, not just parts of strings. If you want to match the middle part, the front part of the regular expression can be written as'^. 'or'.'. '^'and'$' are usually omitted. The RLIKE operator and the REGEXP operator are synonyms. The'|'operator is an optional operator. Regular expressions on either side of'|' only need to satisfy one side condition. The'|'operator and regular expressions on both sides usually need to be enclosed in (). Examples: ```plain mysql> select varchar_column from small_table where varchar_column regexp '(mi|MI).*'; +----------------+ | varchar_column | +----------------+ | milan | +----------------+ 1 row in set (0.01 sec) ``` ```plain mysql> select varchar_column from small_table where varchar_column regexp 'm.*'; +----------------+ | varchar_column | +----------------+ | milan | +----------------+ 1 row in set (0.01 sec) ``` --- ### SHOW ALTER TABLE SHOW ALTER TABLE shows the execution of the ongoing ALTER TABLE operations, including: * Modify columns. * Optimize table schema (from v3.2), including modifying the bucketing method and the number of buckets. * Create and delete the rollup index. * Modify metadata-only properties of cloud-native (shared-data) tables, such as `file_bundling`, `enable_persistent_index`, `persistent_index_type`, and `compaction_strategy`. These changes run asynchronously and are shown under `COLUMN`. #### Syntax[​](#syntax "Direct link to Syntax") * Show the execution of operations of modifying columns or optimizing table schema. ```sql SHOW ALTER TABLE { COLUMN | OPTIMIZE } [FROM db_name] [WHERE TableName|CreateTime|FinishTime|State] [ORDER BY] [LIMIT] ``` * Show the execution of operations of adding or deleting the rollup index. ```sql SHOW ALTER TABLE ROLLUP [FROM db_name] [WHERE TableName|CreateTime|FinishedTime|State] [ORDER BY] [LIMIT] ``` #### Parameters[​](#parameters "Direct link to Parameters") * `{COLUMN ｜ OPTIMIZE | ROLLUP}`: * If `COLUMN` is specified, this statement shows operations of modifying columns. For cloud-native (shared-data) tables, it also shows asynchronous metadata-only changes triggered by `ALTER TABLE ... SET (...)`, such as modifying `file_bundling`, `enable_persistent_index`, `persistent_index_type`, or `compaction_strategy`. * If `OPTIMIZE` is specified, this statement shows operations of optimizing table structure. * If `ROLLUP` is specified, this statement shows operations of adding or deleting the rollup index. * `db_name`: optional. If `db_name` is not specified, the current database is used by default. * `WHERE`, `ORDER BY` and `LIMIT`: optional, and supported for every alter type. The columns they accept are `TableName`, `CreateTime`, the finish time, and `State`. The finish time is returned as `FinishTime` by `COLUMN` and `OPTIMIZE`, and as `FinishedTime` by `ROLLUP`; `ROLLUP` accepts either spelling. #### Examples[​](#examples "Direct link to Examples") 1. Show the execution of operations of modifying columns, optimizing table schema, and creating or deleting the rollup index in the current database. ```sql SHOW ALTER TABLE COLUMN; SHOW ALTER TABLE OPTIMIZE; SHOW ALTER TABLE ROLLUP; ``` 2. Show the execution of operations related to modifying columns, optimizing table schema, and creating or deleting the rollup index in a specified database. ```sql SHOW ALTER TABLE COLUMN FROM example_db; SHOW ALTER TABLE OPTIMIZE FROM example_db; SHOW ALTER TABLE ROLLUP FROM example_db; ``` 3. Show the execution of the most recent operation of modifying columns, optimizing table schema, or adding a rollup index in a specified table. ```sql SHOW ALTER TABLE COLUMN WHERE TableName = "table1" ORDER BY CreateTime DESC LIMIT 1; SHOW ALTER TABLE OPTIMIZE WHERE TableName = "table1" ORDER BY CreateTime DESC LIMIT 1; SHOW ALTER TABLE ROLLUP WHERE TableName = "table1" ORDER BY CreateTime DESC LIMIT 1; ``` #### References[​](#references "Direct link to References") * [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) * [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md) * [SHOW TABLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLES.md) * [SHOW CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_CREATE_TABLE.md) --- ### SHOW CREATE TABLE Returns the CREATE TABLE statement that was used to create a given table. > **NOTE** > > In versions earlier than v3.0, the SHOW CREATE TABLE statement requires you to have the `SELECT_PRIV` privilege on the table. Since v3.0, the SHOW CREATE TABLE statement requires you to have the `SELECT` privilege on the table. Since v3.0, you can use the SHOW CREATE TABLE statement to view the CREATE TABLE statements of the tables that are managed by an external catalog and are stored in Apache Hive™, Apache Iceberg, Apache Hudi, or Delta Lake. Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). * If you specified the number of buckets when creating a table, the output of SHOW CREATE TABLE will display the number of buckets. * If you did not specify the number of buckets when creating a table, the output of SHOW CREATE TABLE will not display the number of buckets. You can run [SHOW PARTITIONS](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md) to view the number of buckets for each partition. In versions earlier than v2.5.7, you are required to set the number of buckets when creating a table. Therefore, SHOW CREATE TABLE displays the number of buckets by default. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW CREATE TABLE [db_name.]table_name ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | | db\_name | No | The database name. If this parameter is not specified, the CREATE TABLE statement of a given table in your current database is returned by default. | | table\_name | Yes | The table name. | #### Output[​](#output "Direct link to Output") ```plain +-----------+----------------+ | Table | Create Table | +-----------+----------------+ ``` The following table describes the parameters returned by this statement. | **Parameter** | **Description** | | ------------- | ---------------------------------------- | | Table | The table name. | | Create Table | The CREATE TABLE statement of the table. | #### Examples[​](#examples "Direct link to Examples") ##### Bucket number is not specified[​](#bucket-number-is-not-specified "Direct link to Bucket number is not specified") Create a table named `example_table` with no bucket number specified in DISTRIBUTED BY. ```sql CREATE TABLE example_table ( k1 TINYINT, k2 DECIMAL(10, 2) DEFAULT "10.5", v1 CHAR(10) REPLACE, v2 INT SUM ) ENGINE = olap AGGREGATE KEY(k1, k2) COMMENT "my first starrocks table" DISTRIBUTED BY HASH(k1); ``` Run SHOW CREATE TABLE to display the CREATE TABLE statement of `example_table`. No bucket number is displayed in DISTRIBUTED BY. Note that if you did not specify PROPERTIES when you create the table, the default properties are displayed in the output of SHOW CREATE TABLE. ```plain SHOW CREATE TABLE example_table\G *************************** 1. row *************************** Table: example_table Create Table: CREATE TABLE `example_table` ( `k1` tinyint(4) NULL COMMENT "", `k2` decimal64(10, 2) NULL DEFAULT "10.5" COMMENT "", `v1` char(10) REPLACE NULL COMMENT "", `v2` int(11) SUM NULL COMMENT "" ) ENGINE=OLAP AGGREGATE KEY(`k1`, `k2`) COMMENT "my first starrocks table" DISTRIBUTED BY HASH(`k1`) PROPERTIES ( "replication_num" = "3", "in_memory" = "false", "enable_persistent_index" = "true", "replicated_storage" = "true", "compression" = "LZ4" ); ``` ##### Bucket number is specified[​](#bucket-number-is-specified "Direct link to Bucket number is specified") Create a table named `example_table1` with bucket number set to 10 in DISTRIBUTED BY. ```sql CREATE TABLE example_table1 ( k1 TINYINT, k2 DECIMAL(10, 2) DEFAULT "10.5", v1 CHAR(10) REPLACE, v2 INT SUM ) ENGINE = olap AGGREGATE KEY(k1, k2) COMMENT "my first starrocks table" DISTRIBUTED BY HASH(k1) BUCKETS 10; ``` Run SHOW CREATE TABLE to display the CREATE TABLE statement of `example_table`. The bucket number (`BUCKETS 10`) is displayed in DISTRIBUTED BY. Note that if you did not specify PROPERTIES when you create the table, the default properties are displayed in the output of SHOW CREATE TABLE. ```plain SHOW CREATE TABLE example_table1\G *************************** 1. row *************************** Table: example_table1 Create Table: CREATE TABLE `example_table1` ( `k1` tinyint(4) NULL COMMENT "", `k2` decimal64(10, 2) NULL DEFAULT "10.5" COMMENT "", `v1` char(10) REPLACE NULL COMMENT "", `v2` int(11) SUM NULL COMMENT "" ) ENGINE=OLAP AGGREGATE KEY(`k1`, `k2`) COMMENT "my first starrocks table" DISTRIBUTED BY HASH(`k1`) BUCKETS 10 PROPERTIES ( "replication_num" = "3", "in_memory" = "false", "enable_persistent_index" = "true", "replicated_storage" = "true", "compression" = "LZ4" ); ``` #### References[​](#references "Direct link to References") * [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) * [SHOW TABLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLES.md) * [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md) * [DROP TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md) --- ### SHOW DELETE SHOW DELETE queries historical DELETE operations that were successfully performed on Duplicate Key, Unique Key, and Aggregate tables in a specified database. For more information about data deletion, see [DELETE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DELETE.md). Note that this command cannot be used to query DELETE operations that were performed on Primary Key tables. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW DELETE [FROM ] ``` `db_name`: the database name, optional. If this parameter is not specified, the current database is used by default. Return fields: * `TableName`: the table from which data is deleted. * `PartitionName`: the partition from which data is deleted. If the table is a non-partitioned table, `*` is displayed. * `CreateTime`: the time when the DELETE task was created. * `DeleteCondition`: the specified DELETE condition. * `State`: the status of the DELETE task. #### Examples[​](#examples "Direct link to Examples") Show all historical DELETE operations of `database`. ```sql SHOW DELETE FROM database; +------------+---------------+---------------------+-----------------+----------+ | TableName | PartitionName | CreateTime | DeleteCondition | State | +------------+---------------+---------------------+-----------------+----------+ | mail_merge | * | 2023-03-14 10:39:03 | name EQ "Peter" | FINISHED | +------------+---------------+---------------------+-----------------+----------+ ``` --- ### SHOW DYNAMIC PARTITION TABLES SHOW DYNAMIC PARTITION TABLES is used to display the status of all the partitioned tables for which dynamic partitioning properties are configured in a database. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW DYNAMIC PARTITION TABLES FROM ``` This statement returns the following fields: * TableName: the name of the table. * Enable: whether dynamic partitioning is enabled. * TimeUnit: the time granularity for the partitions. * Start: the starting offset of dynamic partitioning. * End: the end offset of dynamic partitioning. * Prefix: the prefix of the partition name. * Buckets: the number of buckets per partition. * ReplicationNum: the number of replicas for the table. * StartOf: the first day of each week/month depending on the specified TimeUnit * LastUpdateTime: the time when the table was last updated. * LastSchedulerTime: the time when data in the table was last scheduled. * State: the status of the table. * LastCreatePartitionMsg: the message for the latest partition creation operation. * LastDropPartitionMsg: the message for the latest partition dropping operation. #### Examples[​](#examples "Direct link to Examples") Display the status of all the partitioned tables for which dynamic partitioning properties are configured in `db_test`. ```sql SHOW DYNAMIC PARTITION TABLES FROM db_test; ``` --- ### SHOW FULL COLUMNS SHOW FULL COLUMNS is used to show content in columns from specified tables. tip This operation does not require privileges. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW FULL COLUMNS FROM ``` #### Examples[​](#examples "Direct link to Examples") 1. View column content from specified tables. ```sql SHOW FULL COLUMNS FROM tbl; ``` --- ### SHOW INDEX SHOW INDEX is used to show information related to index in a table. It currently only supports bitmap index. tip This operation does not require privileges. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW INDEX[ES] FROM [db_name.]table_name [FROM database] Or SHOW KEY[S] FROM [db_name.]table_name [FROM database] ``` #### Examples[​](#examples "Direct link to Examples") 1. Show all indexes under the specified table\_name: ```sql SHOW INDEX FROM example_db.table_name; ``` --- ### SHOW PARTITIONS SHOW PARTITIONS displays partition information, including common partitions and [temporary partitions](https://docs.starrocks.io/docs/table_design/data_distribution/Temporary_partition.md). #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW [TEMPORARY] PARTITIONS FROM [db_name.]table_name [WHERE] [ORDER BY] [LIMIT] ``` > NOTE > > This syntax only supports StarRocks tables (`"ENGINE" = "OLAP"`). Since v3.0, this operation requires the SELECT privilege on the specified table. For v2.5 and earlier versions, this operation requires the SELECT\_\_PRIV privilege on the specified table. #### Description of return fields[​](#description-of-return-fields "Direct link to Description of return fields") ```plaintext -- In shared-nothing cluster +-------------+---------------+----------------+---------------------+--------------------+--------+--------------+-------+--------------------+---------+----------------+---------------+---------------------+--------------------------+----------+------------+----------+ | PartitionId | PartitionName | VisibleVersion | VisibleVersionTime | VisibleVersionHash | State | PartitionKey | Range | DistributionKey | Buckets | ReplicationNum | StorageMedium | CooldownTime | LastConsistencyCheckTime | DataSize | IsInMemory | RowCount | +-------------+---------------+----------------+---------------------+--------------------+--------+--------------+-------+--------------------+---------+----------------+---------------+---------------------+--------------------------+----------+------------+----------+ -- In shared-data cluster +-------------+---------------+----------------+----------------+-------------+--------+--------------+------------------------------------------------------------------------------------------------------+-----------------+---------+----------+-------------+----------+-----------------+------------+-------+-------+-------+-------------+--------------------+----------------+ | PartitionId | PartitionName | CompactVersion | VisibleVersion | NextVersion | State | PartitionKey | Range | DistributionKey | Buckets | DataSize | StorageSize | RowCount | EnableDataCache | AsyncWrite | AvgCS | P50CS | MaxCS | DataVersion | VersionEpoch | VersionTxnType | +-------------+---------------+----------------+----------------+-------------+--------+--------------+------------------------------------------------------------------------------------------------------+-----------------+---------+----------+-------------+----------+-----------------+------------+-------+-------+-------+-------------+--------------------+----------------+ ``` | **Field** | **Description** | | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | PartitionId | The ID of the partition. | | PartitionName | The name of the partition. | | VisibleVersion | The version number of the last successful load transaction. The version number increases by 1 with each successful load transaction. | | CompactVersion | The version number of the last successful compaction. | | VisibleVersionTime | The timestamp of the last successful load transaction. | | VisibleVersionHash | The hash value for the version number of the last successful load transaction. | | State | The status of the partition. Fixed value: `Normal`. | | PartitionKey | The partition key that consists of one or more partition columns. | | Range | The range of the partition, which is a right half-open interval. | | DistributionKey | The bucket key of hash bucketing. | | Buckets | The number of buckets for the partition. For a table with range distribution, this is the actual number of tablets in the partition's base index, which changes as tablets are split and merged. | | ReplicationNum | The number of replicas per tablet in the partition. | | StorageMedium | The storage medium to store the data in the partition. The value `HHD` indicates hard disk drives, and the value `SSD` indicates solid-state drives. | | CooldownTime | The cooldown time for data in the partition. If the initial storage medium is SSD, the storage medium is switched from SSD to HDD after the time specified by this parameter. Format: "yyyy-MM-dd HH:mm:ss". | | LastConsistencyCheckTime | The time of the last consistency check. `NULL` indicates no consistency check was performed. | | DataSize | The size of data in the partition. | | IsInMemory | Whether all data in the partition is stored in memory. | | StorageSize | The size of storage the partition actually occupied. For shared-data clusters only. | | RowCount | The number of data rows of the partition. For shared-data clusters only. | | EnableDataCache | Whether Data Cache is enabled. For shared-data clusters only. | | AsyncWrite | Whether Asynchronous Write is enabled. For shared-data clusters only. | | AvgCS | The average Compaction Score of the partition. For shared-data clusters only. | | P50CS | The P50 Compaction Score of the partition. For shared-data clusters only. | | MaxCS | The maximum Compaction Score of the partition. For shared-data clusters only. | | DataVersion | The version number of loading transactions. Compaction operations are not included. | | VersionEpoch | The epoch of partition. The system assigns the version epoch when the partition was created, and changes it when the partition was swapped. | | VersionTxnType | The type of the transaction that generates the current data version. Valid values: `NORMAL` (normal transaction) and `REPLICATION` (data replication). | #### Examples[​](#examples "Direct link to Examples") 1. Display information of all regular partitions from the specified table `site_access` under the specified database `test`. ```sql MySQL > show partitions from test.site_access\G *************************** 1. row *************************** PartitionId: 20990 PartitionName: p2019 VisibleVersion: 1 VisibleVersionTime: 2023-08-08 15:45:13 VisibleVersionHash: 0 State: NORMAL PartitionKey: datekey Range: [types: [DATE]; keys: [2019-01-01]; ..types: [DATE]; keys: [2020-01-01]; ) DistributionKey: site_id Buckets: 6 ReplicationNum: 3 StorageMedium: HDD CooldownTime: 9999-12-31 23:59:59 LastConsistencyCheckTime: NULL DataSize: 4KB IsInMemory: false RowCount: 3 1 row in set (0.00 sec) ``` 2. Display information of all temporary partitions from the specified table `site_access` under the specified database `test`. ```sql SHOW TEMPORARY PARTITIONS FROM test.site_access; ``` 3. Display the information of the specified partition `p1` of the specified table `site_access` under the specified database `test`. ```sql -- Regular partition SHOW PARTITIONS FROM test.site_access WHERE PartitionName = "p1"; -- Temporary partition SHOW TEMPORARY PARTITIONS FROM test.site_access WHERE PartitionName = "p1"; ``` 4. Display the latest partition information of the specified table `site_access` under the specified database `test`. ```sql -- Regular partition SHOW PARTITIONS FROM test.site_access ORDER BY PartitionId DESC LIMIT 1; -- Temporary partition SHOW TEMPORARY PARTITIONS FROM test.site_access ORDER BY PartitionId DESC LIMIT 1; ``` --- ### SHOW TABLES SHOW TABLES displays all tables in a StarRocks database or a database in an external data source, for example, Hive, Iceberg, Hudi, or Delta Lake. > **NOTE** > > To view tables in an external data source, you must have the USAGE privilege on the external catalog that corresponds to that data source. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW TABLES [FROM .] ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | catalog\_name | No | The name of the internal catalog or an external catalog.- If you do not specify this parameter or set it to `default_catalog`, tables in StarRocks databases are returned.
- If you set this parameter to the name of an external catalog, tables in databases of an external data source are returned. You can run [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to view internal and external catalogs. | | db\_name | No | The database name. If not specified, the current database is used by default. | #### Examples[​](#examples "Direct link to Examples") Example 1: View tables in database `example_db` of the `default_catalog` after connecting to the StarRocks cluster. The following two statements are equivalent. ```plain show tables from example_db; +----------------------------+ | Tables_in_example_db | +----------------------------+ | depts | | depts_par | | emps | | emps2 | +----------------------------+ show tables from default_catalog.example_db; +----------------------------+ | Tables_in_example_db | +----------------------------+ | depts | | depts_par | | emps | | emps2 | +----------------------------+ ``` Example 2: View tables in the current database `example_db` after connecting to this database. ```plain show tables; +----------------------------+ | Tables_in_example_db | +----------------------------+ | depts | | depts_par | | emps | | emps2 | +----------------------------+ ``` Example 2: View tables in database `hudi_db` of the external catalog `hudi_catalog`. ```plain show tables from hudi_catalog.hudi_db; +----------------------------+ | Tables_in_hudi_db | +----------------------------+ | hudi_sync_mor | | hudi_table1 | +----------------------------+ ``` Alternatively, you can run SET CATALOG to switch to the external catalog `hudi_catalog` and then run `SHOW TABLES FROM hudi_db;`. #### References[​](#references "Direct link to References") * [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md): Views all catalogs in a StarRocks cluster. * [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md): Views all databases in the internal catalog or an external catalog. * [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md): Switches between catalogs. --- ### SHOW TABLET SHOW TABLET displays tablet related information. > **NOTE** > > For v3.0 and later, this operation requires the SYSTEM-level OPERATE privilege and TABLE-level SELECT privilege. For v2.5 and earlier, this operation requires the ADMIN\_PRIV privilege. #### Syntax[​](#syntax "Direct link to Syntax") ##### Query information of tablets in a table or a partition[​](#query-information-of-tablets-in-a-table-or-a-partition "Direct link to Query information of tablets in a table or a partition") ```sql SHOW TABLET FROM [.] [PARTITION(, ...)] [ WHERE [version = ] [[AND] backendid = ] [[AND] STATE = "NORMAL"|"ALTER"|"CLONE"|"DECOMMISSION"] ] [ORDER BY [ASC | DESC]] [LIMIT [,]] ``` | **Parameter** | **Required** | **Description** | | --------------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | db\_name | No | The database name. If you do not specify this parameter, the current database is used by default. | | table\_name | Yes | The name of the table from which you want to query tablet information. You must specify this parameter. Otherwise, an error is returned. | | partition\_name | No | The name of the partition from which you want to query tablet information. | | version\_number | No | The data version number. | | backend\_id | No | The ID of the BE where the replica of the tablet is located. | | STATE | No | The status of tablet replicas. - `NORMAL`: The replica is normal.
- `ALTER`: A Rollup or schema change is being performed on the replica.
- `CLONE`: The replica is being cloned. (Replicas in this state are not available for use).
- `DECOMMISSION`: The replica is being decommissioned. | | field\_name | No | The field by which the results are sorted. All the fields returned by `SHOW TABLET FROM ` are sortable.- If you want to display results in ascending order, use `ORDER BY field_name ASC`.
- If you want to display results in descending order, use `ORDER BY field_name DESC`. | | offset | No | The number of tablets to skip from the results. For example, `OFFSET 5` means to skip the first five tablets. Default value: 0. | | limit | No | The number of tablets to return. For example, `LIMIT 10` means to return only 10 tablets. If this parameter is not specified, all the tablets that meet the filter conditions are returned. | ##### Query information of a single tablet[​](#query-information-of-a-single-tablet "Direct link to Query information of a single tablet") After obtaining all tablet IDs using `SHOW TABLET FROM `, you can query the information of a single tablet. ```sql SHOW TABLET ``` | **Parameter** | **Required** | **Description** | | ------------- | ------------ | --------------- | | tablet\_id | Yes | Tablet ID | #### Description of return fields[​](#description-of-return-fields "Direct link to Description of return fields") ##### Query information of tablets in a table or a partition[​](#query-information-of-tablets-in-a-table-or-a-partition-1 "Direct link to Query information of tablets in a table or a partition") ```plain -- In shared-nothing cluster +----------+-----------+-----------+------------+---------+-------------+-------------------+-----------------------+------------------+----------------------+---------------+----------+----------+--------+-------------------------+--------------+------------------+--------------+----------+----------+-------------------+---------------+ | TabletId | ReplicaId | BackendId | SchemaHash | Version | VersionHash | LstSuccessVersion | LstSuccessVersionHash | LstFailedVersion | LstFailedVersionHash | LstFailedTime | DataSize | RowCount | State | LstConsistencyCheckTime | CheckVersion | CheckVersionHash | VersionCount | PathHash | MetaUrl | CompactionStatus | DiskRootPath | +----------+-----------+-----------+------------+---------+-------------+-------------------+-----------------------+------------------+----------------------+---------------+----------+----------+--------+-------------------------+--------------+------------------+--------------+----------+----------+-------------------+---------------+ -- In shared-data cluster +----------+-----------+----------+----------+------------+ | TabletId | BackendId | DataSize | RowCount | MinVersion | +----------+-----------+----------+----------+------------+ ``` | **Field** | **Description** | | ----------------------- | ------------------------------------------------------------------------------------------------- | | TabletId | Table ID. | | ReplicaId | Replica ID. | | BackendId | The ID of the BE where the replica is located. | | SchemaHash | Schema hash (randomly generated). | | Version | Data version number. | | VersionHash | Hash of data version number. | | LstSuccessVersion | The last successfully loaded version. | | LstSuccessVersionHash | The hash of the last successfully loaded version. | | LstFailedVersion | The version of the last failed loading. `-1` indicates no version failed to be loaded. | | LstFailedVersionHash | The hash of the last failed version. | | LstFailedTime | The time of the last failed loading. `NULL` indicates there is no load failure. | | DataSize | Data size of the tablet. | | RowCount | The number of data rows of the tablet. | | State | Replica status of the tablet. | | LstConsistencyCheckTime | The time of the last consistency check. `NULL` indicates no consistency check was performed. | | CheckVersion | The data version on which consistency check was performed. `-1` indicates no version was checked. | | CheckVersionHash | The hash of the version on which consistency check was performed. | | VersionCount | The total number of data versions. | | PathHash | The hash of the directory in which the tablet is stored. | | MetaUrl | The URL used to query more meta information. | | CompactionStatus | The URL used to query data version compaction status. | | DiskRootPath | The disk where the replica is located. | | MinVersion | The minimum data version retained for the tablet. For shared-data clusters only. | ##### Query information of a specific tablet[​](#query-information-of-a-specific-tablet "Direct link to Query information of a specific tablet") ```plain +--------+-----------+---------------+-----------+------+---------+-------------+---------+--------+-----------+ | DbName | TableName | PartitionName | IndexName | DbId | TableId | PartitionId | IndexId | IsSync | DetailCmd | +--------+-----------+---------------+-----------+------+---------+-------------+---------+--------+-----------+ ``` | **Field** | **Description** | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | DbName | The name of the database to which the tablet belongs. | | TableName | The name of the table to which the tablet belongs. | | PartitionName | The name of the partition to which the tablet belongs. | | IndexName | The index name. | | DbId | The database ID. | | TableId | The table ID. | | PartitionId | The partition ID. | | IndexId | The index ID. | | IsSync | Whether data on the tablet is consistent with table meta. `true` indicates data is consistent and the tablet is normal. `false` indicates data is missing on the tablet. | | DetailCmd | The URL used to query more information. | #### Examples[​](#examples "Direct link to Examples") Create table `test_show_tablet` in the database `example_db`. ```sql CREATE TABLE `test_show_tablet` ( `k1` date NULL COMMENT "", `k2` datetime NULL COMMENT "", `k3` char(20) NULL COMMENT "", `k4` varchar(20) NULL COMMENT "", `k5` boolean NULL COMMENT "", `k6` tinyint(4) NULL COMMENT "", `k7` smallint(6) NULL COMMENT "", `k8` int(11) NULL COMMENT "", `k9` bigint(20) NULL COMMENT "", `k10` largeint(40) NULL COMMENT "", `k11` float NULL COMMENT "", `k12` double NULL COMMENT "", `k13` decimal128(27, 9) NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`k1`, `k2`, `k3`, `k4`, `k5`) COMMENT "OLAP" PARTITION BY RANGE(`k1`) (PARTITION p20210101 VALUES [("2021-01-01"), ("2021-01-02")), PARTITION p20210102 VALUES [("2021-01-02"), ("2021-01-03")), PARTITION p20210103 VALUES [("2021-01-03"), ("2021-01-04")), PARTITION p20210104 VALUES [("2021-01-04"), ("2021-01-05")), PARTITION p20210105 VALUES [("2021-01-05"), ("2021-01-06")), PARTITION p20210106 VALUES [("2021-01-06"), ("2021-01-07")), PARTITION p20210107 VALUES [("2021-01-07"), ("2021-01-08")), PARTITION p20210108 VALUES [("2021-01-08"), ("2021-01-09")), PARTITION p20210109 VALUES [("2021-01-09"), ("2021-01-10"))) DISTRIBUTED BY HASH(`k1`, `k2`, `k3`); ``` * Example 1: Query information of all the tablets in the specified table. The following example excerpts information of only one tablet from the return information. ```plain mysql> show tablet from example_db.test_show_tablet\G *************************** 1. row *************************** TabletId: 9588955 ReplicaId: 9588956 BackendId: 10004 SchemaHash: 0 Version: 1 VersionHash: 0 LstSuccessVersion: 1 LstSuccessVersionHash: 0 LstFailedVersion: -1 LstFailedVersionHash: 0 LstFailedTime: NULL DataSize: 0B RowCount: 0 State: NORMAL LstConsistencyCheckTime: NULL CheckVersion: -1 CheckVersionHash: 0 VersionCount: 1 PathHash: 0 MetaUrl: http://172.26.92.141:8038/api/meta/header/9588955 CompactionStatus: http://172.26.92.141:8038/api/compaction/show?tablet_id=9588955 DiskRootPath: /storage/disk ``` * Example 2: Query information of tablet 9588955. ```plain mysql> show tablet 9588955\G *************************** 1. row *************************** DbName: example_db TableName: test_show_tablet PartitionName: p20210103 IndexName: test_show_tablet DbId: 11145 TableId: 9588953 PartitionId: 9588946 IndexId: 9588954 IsSync: true DetailCmd: SHOW PROC '/dbs/11145/9588953/partitions/9588946/9588954/9588955'; ``` * Example 3: Query information of tablets in partition `p20210103`. ```sql SHOW TABLET FROM test_show_tablet partition(p20210103); ``` * Example 4: Return information of 10 tablets. ```sql SHOW TABLET FROM test_show_tablet limit 10; ``` * Example 5: Return information of 10 tablets with an offset 5. ```sql SHOW TABLET FROM test_show_tablet limit 5,10; ``` * Example 6: Filter tablets by `backendid`, `version`, and `state`. ```sql SHOW TABLET FROM test_show_tablet WHERE backendid = 10004 and version = 1 and state = "NORMAL"; ``` * Example 7: Sort tablets by `version`. ```sql SHOW TABLET FROM table_name where backendid = 10004 order by version; ``` * Example 8: Return information of tablets whose index name is `test_show_tablet`. ```sql SHOW TABLET FROM test_show_tablet where indexname = "test_show_tablet"; ``` --- ### TRUNCATE TABLE TRUNCATE TABLE is used to truncate the specified table and partition data. Syntax: ```sql TRUNCATE TABLE [db.]tbl[ PARTITION(PartitionName1, PartitionName2, ...)] ``` Note: 1. This statement is used to truncate data while retaining tables or partitions. 2. Unlike DELETE, this statement can only empty the specified tables or partitions as a whole, and filtering conditions cannot be added. 3. Unlike DELETE, using this method to clear data will not affect query performance. 4. This statement directly deletes data. The deleted data cannot be recovered. 5. The table on which you perform this operation must be in the NORMAL state. For example, you cannot perform TRUNCATE TABLE on a table with SCHEMA CHANGE going on. #### Examples[​](#examples "Direct link to Examples") 1. Truncate table `tbl` under `example_db`. ```sql TRUNCATE TABLE example_db.tbl; ``` 2. Truncate partitions `PartitionName1` and `PartitionName2` in table `tbl`. ```sql TRUNCATE TABLE tbl PARTITION(PartitionName1, PartitionName2); ``` --- ### UPDATE Updates rows in a Primary Key table. StarRocks supports the UPDATE statement since v2.3, which only supports single-table UPDATE and does not support common table expressions (CTEs). Starting from version 3.0, StarRocks enriches the syntax to support multi-table joins and CTEs. If you need to join the table to be updated with other tables in the database, you can reference these other tables in the FROM clause or CTE. Since version 3.1, the UPDATE statement supports the partial updates in column mode, which is suitable for scenarios involving a small number of columns but a large number of rows, resulting in faster update speeds. This command requires the UPDATE privilege on the table you want to update. #### Usage notes[​](#usage-notes "Direct link to Usage notes") When executing the UPDATE statement involving multiple tables, StarRocks converts the table expression in the FROM clause of the UPDATE statement into an equivalent JOIN query statement. Therefore, make sure that the table expression that you specify in the FROM clause of the UPDATE statement supports this conversion. For example, the UPDATE statement is 'UPDATE t0 SET v1=t1.v1 FROM t1 WHERE t0.pk = t1.pk;'. The table expression in the FROM clause can be converted to 't0 JOIN t1 ON t0.pk=t1.pk;'. StarRocks matches the data rows to be updated based on the result set of the JOIN query. It is possible that multiple rows in the result set match a certain row in the table to be updated. In this scenario, that row is updated based on the value of a random row among these multiple rows. You can use the system variable `insert_timeout` to set the timeout duration for UPDATE operations. #### Syntax[​](#syntax "Direct link to Syntax") ##### Single-table UPDATE[​](#single-table-update "Direct link to Single-table UPDATE") If the data rows of the table to be updated meet the WHERE condition, the specified columns of these data rows are assigned new values. ```sql [ WITH [, ...] ] UPDATE SET = [, ...] WHERE ``` ##### Multi-table UPDATE[​](#multi-table-update "Direct link to Multi-table UPDATE") The result set from the multi-table join is matched against the table to be updated. If the data rows of the table to be updated match the result set and meet the WHERE condition, the specified columns of these data rows are assigned new values. ```sql [ WITH [, ...] ] UPDATE SET = [, ...] [ FROM [, ...] ] WHERE ``` #### Parameters[​](#parameters "Direct link to Parameters") `with_query` One or more CTEs that can be referenced by name in an UPDATE statement. CTEs are temporary result sets that can improve the readability of complex statements. `table_name` The name of the table to be updated. `column_name` The name of the column to be updated. It cannot include the table name. For example, 'UPDATE t1 SET col = 1' is not valid. `expression` The expression that assigns new values to the column. `from_item` One or more other tables in the database. These tables can be joined with the table to be updated based on the condition specified in the WHERE clause. The values of the rows in the result set are used to update the values for the specified columns in the matched rows in the table to be updated. For example, if the FROM clause is `FROM t1 WHERE t0.pk = t1.pk`, StarRocks converts the table expression in the FROM clause to `t0 JOIN t1 ON t0.pk=t1.pk` when executing the UPDATE statement. `where_condition` The condition based on which you want to update rows. Only rows that meet the WHERE condition can be updated. This parameter is required, because it helps prevent you from accidentally updating the entire table. If you want to update the entire table, you can use 'WHERE true'. However, this parameter is not required for [partial updates in column mode](#partial-updates-in-column-mode-since-v31). #### Partial updates in column mode (since v3.1)[​](#partial-updates-in-column-mode-since-v31 "Direct link to Partial updates in column mode (since v3.1)") Partial updates in column mode are suitable for scenarios where only a small number of columns, but a large number of rows need to be updated. In such scenarios, enabling the column mode offers faster update speeds. For example, in a table with 100 columns, if only 10 columns (10% of the total) are updated for all rows, the update speed of the column mode is 10 times faster. The system variable `partial_update_mode` controls the mode of partial updates and supports the following values: * `auto` (default): The system automatically determines the mode of partial updates by analyzing the UPDATE statement and the columns involved. If the following criteria are met, the system automatically uses the column mode: * The percentage of updated columns compared to the total number of columns is less than 30%, and the number of updated columns is fewer than 4. * The update statement does not use a WHERE condition. Otherwise, the system does not use the column mode. * `column`: The column mode is used for the partial updates, which is particularly suitable for the partial updates which involve a small number of columns and a large number of rows. You can use `EXPLAIN UPDATE xxx` to view the mode of partial updates. #### Examples[​](#examples "Direct link to Examples") ##### Single-table UPDATE[​](#single-table-update-1 "Direct link to Single-table UPDATE") Create a table `Employees` to record employee information and insert five data rows into the table. ```sql CREATE TABLE Employees ( EmployeeID INT, Name VARCHAR(50), Salary DECIMAL(10, 2) ) PRIMARY KEY (EmployeeID) DISTRIBUTED BY HASH (EmployeeID) PROPERTIES ("replication_num" = "3"); INSERT INTO Employees VALUES (1, 'John Doe', 5000), (2, 'Jane Smith', 6000), (3, 'Robert Johnson', 5500), (4, 'Emily Williams', 4500), (5, 'Michael Brown', 7000); ``` If you need to give a 10% raise to all employees, you can execute the following statement: ```sql UPDATE Employees SET Salary = Salary * 1.1 -- Increase the salary by 10%. WHERE true; ``` If you need to give a 10% raise to employees with salaries lower than the average salary, you can execute the following statement: ```sql UPDATE Employees SET Salary = Salary * 1.1 -- Increase the salary by 10%. WHERE Salary < (SELECT AVG(Salary) FROM Employees); ``` You can also use a CTE to rewrite the above statement to improve readability. ```sql WITH AvgSalary AS ( SELECT AVG(Salary) AS AverageSalary FROM Employees ) UPDATE Employees SET Salary = Salary * 1.1 -- Increase the salary by 10%. FROM AvgSalary WHERE Employees.Salary < AvgSalary.AverageSalary; ``` ##### Multi-table UPDATE[​](#multi-table-update-1 "Direct link to Multi-table UPDATE") Create a table `Accounts` to record account information and insert three data rows into the table. ```sql CREATE TABLE Accounts ( Accounts_id BIGINT NOT NULL, Name VARCHAR(26) NOT NULL, Sales_person VARCHAR(50) NOT NULL ) PRIMARY KEY (Accounts_id) DISTRIBUTED BY HASH (Accounts_id) PROPERTIES ("replication_num" = "3"); INSERT INTO Accounts VALUES (1,'Acme Corporation','John Doe'), (2,'Acme Corporation','Robert Johnson'), (3,'Acme Corporation','Lily Swift'); ``` If you need to give a 10% raise to employees in the table `Employees` who manage accounts for Acme Corporation, you can execute the following statement: ```sql UPDATE Employees SET Salary = Salary * 1.1 -- Increase the salary by 10%. FROM Accounts WHERE Accounts.name = 'Acme Corporation' AND Employees.Name = Accounts.Sales_person; ``` You can also use a CTE to rewrite the above statement to improve readability. ```sql WITH Acme_Accounts as ( SELECT * from Accounts WHERE Accounts.name = 'Acme Corporation' ) UPDATE Employees SET Salary = Salary * 1.1 -- Increase the salary by 10%. FROM Acme_Accounts WHERE Employees.Name = Acme_Accounts.Sales_person; ``` --- ### Translate Trino SQL Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) From v3.3.9, StarRocks supports translating Trino SQL statements into StarRocks SQL statements. #### Syntax[​](#syntax "Direct link to Syntax") ```sql TRANSLATE TRINO ``` #### Parameters[​](#parameters "Direct link to Parameters") `SELECT_statement`: The Trino SQL statement you want to translate. #### Return[​](#return "Direct link to Return") Returns the StarRocks SQL statement. #### Examples[​](#examples "Direct link to Examples") ```plain mysql> TRANSLATE TRINO SELECT id, name, category FROM products WHERE name = 'Dell XPS 13' AND category = "Electronics" AND `price` > 500; +---------------------------------------------------------------------------------------------------------------------------------------+ | Translated SQL | +---------------------------------------------------------------------------------------------------------------------------------------+ | SELECT `id`, `name`, `category` FROM `products` WHERE ((`name` = 'Dell XPS 13') AND (`category` = 'Electronics')) AND (`price` > 500) | +---------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.30 sec) ``` --- ### View #### [📄️ ALTER VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/View/ALTER_VIEW.md) [ALTER VIEW Modifies the definition of a view.](https://docs.starrocks.io/docs/sql-reference/sql-statements/View/ALTER_VIEW.md) #### [📄️ CREATE VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/View/CREATE_VIEW.md) [CREATE VIEW creates a logical view.](https://docs.starrocks.io/docs/sql-reference/sql-statements/View/CREATE_VIEW.md) #### [📄️ DROP VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/View/DROP_VIEW.md) [DROP VIEW is used to drop a logical view VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/View/DROP_VIEW.md) #### [📄️ SHOW CREATE VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/View/SHOW_CREATE_VIEW.md) [Returns the CREATE statement that was used to create a given view.](https://docs.starrocks.io/docs/sql-reference/sql-statements/View/SHOW_CREATE_VIEW.md) --- ### ALTER VIEW ALTER VIEW Modifies the definition of a view. #### Syntax[​](#syntax "Direct link to Syntax") ```sql ALTER VIEW [db_name.]view_name (column1[ COMMENT "col comment"][, column2, ...]) AS query_stmt ``` Note: 1. View is logical, where the data isn't stored in the physical medium. The view will be used as a subquery in the statement when queried. Therefore, modifying the definition of views is equivalent to modifying query\_stmt. 2. query\_stmt is arbitrarily supported SQL. #### Examples[​](#examples "Direct link to Examples") Alter `example_view` on `example_db`. ```sql ALTER VIEW example_db.example_view ( c1 COMMENT "column 1", c2 COMMENT "column 2", c3 COMMENT "column 3" ) AS SELECT k1, k2, SUM(v1) FROM example_table GROUP BY k1, k2 ``` --- ### CREATE VIEW CREATE VIEW creates a logical view. A view, or a logical view, is a virtual table whose data is derived from a query against other existing physical tables. Therefore, a view uses no physical storage, and all queries against the view are equivalent to sub-queries of the query statement used to build the view. For information about materialized views supported by StarRocks, see [Synchronous materialized views](https://docs.starrocks.io/docs/using_starrocks/Materialized_view-single_table.md) and [Asynchronous materialized views](https://docs.starrocks.io/docs/using_starrocks/async_mv/Materialized_view.md). From v3.4.1 onwards, StarRocks supports secure view. You can restrain users without the SELECT privilege on the base tables the view referenced from querying the view. tip Only users with the CREATE VIEW privilege on a specific database can perform this operation. #### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE [OR REPLACE] VIEW [IF NOT EXISTS] [.] ( [ COMMENT 'column comment'] [, [ COMMENT 'column comment'], ...] ) [COMMENT 'view comment'] [SECURITY {NONE | INVOKER}] AS ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Description** | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | OR REPLACE | Replace an existing view. | | database | The name of the database where the view resides. | | view\_name | The name of the view. For the naming conventions, see [System limits](https://docs.starrocks.io/docs/sql-reference/System_limit.md). | | column\_name | The name of the column(s) in the view. Note that the columns in the view and the columns queried in the `query_statement` must agree in number. | | COMMENT | The comment on the column in the view or the view itself. | | SECURITY | Specifies the security context to be used when checking access privileges at view invocation time. Valid values: - `NONE`(Default): Users with the SELECT on the view only can query the view.
- `INVOKER`: Only users with the SELECT on both the view and the base tables it referenced can query the view. | | query\_statement | The query statement used to create the view. It can be any query statement supported by StarRocks. | #### Usage notes[​](#usage-notes "Direct link to Usage notes") * Querying a view requires the SELECT privilege on the view and on its corresponding base tables. * If the query statement used to build a view cannot be executed due to the Schema Change on the base tables, StarRocks returns an error when you query the view. #### Examples[​](#examples "Direct link to Examples") Example 1: Create a view named `example_view` in `example_db` with an aggregate query against `example_table`. ```sql CREATE VIEW example_db.example_view (k1, k2, k3, v1) AS SELECT c1 as k1, k2, k3, SUM(v1) FROM example_table WHERE k1 = 20160112 GROUP BY k1,k2,k3; ``` Example 2: Create a view named `example_view` in the database `example_db` with an aggregate query against the table `example_table`, and specify comments for the view and each column in it. ```sql CREATE VIEW example_db.example_view ( k1 COMMENT 'first key', k2 COMMENT 'second key', k3 COMMENT 'third key', v1 COMMENT 'first value' ) COMMENT 'my first view' AS SELECT c1 as k1, k2, k3, SUM(v1) FROM example_table WHERE k1 = 20160112 GROUP BY k1,k2,k3; ``` Example 3: Create a secure view named `example_view` in `example_db` with a query against `example_table`. The view only allows queries submitted by users with the SELECT privilege on the base table `example_table`. ```sql CREATE VIEW example_db.example_view (k1, k2, k3, v1) COMMENT 'my secure view' SECURITY INVOKER AS SELECT c1 as k1, k2, k3, SUM(v1) FROM example_table WHERE k1 = 20160112 GROUP BY k1,k2,k3; ``` #### Relevant SQLs[​](#relevant-sqls "Direct link to Relevant SQLs") * [SHOW CREATE VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/View/SHOW_CREATE_VIEW.md) * [ALTER VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/View/ALTER_VIEW.md) * [DROP VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/View/DROP_VIEW.md) --- ### DROP VIEW DROP VIEW is used to drop a logical view VIEW #### Syntax[​](#syntax "Direct link to Syntax") ```sql DROP VIEW [IF EXISTS] [db_name.]view_name ``` #### Examples[​](#examples "Direct link to Examples") 1. If it exists, then drop the view example\_view on example\_db. ```sql DROP VIEW IF EXISTS example_db.example_view; ``` --- ### SHOW CREATE VIEW Returns the CREATE statement that was used to create a given view. The CREATE VIEW statement helps you understand how the view is defined and provides a reference for you to modify or reconstruct the view. Note that the SHOW CREATE VIEW statement requires you to have the `SELECT` privilege on the view and the table based on which the view is created. From v2.5.4 onwards, you can use SHOW CREATE VIEW to query the statement that is used to create a **materialized view**. #### Syntax[​](#syntax "Direct link to Syntax") ```sql SHOW CREATE VIEW [db_name.]view_name ``` #### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | | db\_name | No | The database name. If this parameter is not specified, the CREATE VIEW statement of a given view in your current database is returned by default. | | view\_name | Yes | The view name. | #### Output[​](#output "Direct link to Output") ```sql +---------+--------------+----------------------+----------------------+ | View | Create View | character_set_client | collation_connection | +---------+--------------+----------------------+----------------------+ ``` The following table describes the parameters returned by this statement. | **Parameter** | **Description** | | ---------------------- | ------------------------------------------------------------------ | | View | The view name. | | Create View | The CREATE VIEW statement of the view. | | character\_set\_client | The character set the client uses to send statements to StarRocks. | | collation\_connection | The rules for comparing characters in a character set. | #### Examples[​](#examples "Direct link to Examples") Create a table named `example_table`. ```sql CREATE TABLE example_table ( k1 TINYINT, k2 DECIMAL(10, 2) DEFAULT "10.5", v1 CHAR(10) REPLACE, v2 INT SUM ) ENGINE = olap AGGREGATE KEY(k1, k2) DISTRIBUTED BY HASH(k1); ``` Create a view named `example_view` based on `example_table`. ```sql CREATE VIEW example_view (k1, k2, k3, v1) AS SELECT k1, k2, k3, v1 FROM example_table; ``` Display the CREATE VIEW statement of `example_view`. ```plain SHOW CREATE VIEW example_db.example_view; +--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------+ | View | Create View | character_set_client | collation_connection | +--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------+ | example_view | CREATE VIEW `example_view` (k1, k2, k3, v1) COMMENT "VIEW" AS SELECT `default_cluster:db1`.`example_table`.`k1` AS `k1`, `default_cluster:db1`.`example_table`.`k2` AS `k2`, `default_cluster:db1`.`example_table`.`k3` AS `k3`, `default_cluster:db1`.`example_table`.`v1` AS `v1` FROM `default_cluster:db1`.`example_table`; | utf8 | utf8_general_ci | +--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------+ ``` --- ### System metadatabase This topic describes how to view the exclusive metadata of your StarRocks cluster via system-defined views. Each StarRocks cluster maintains a database `sys`, which contains several read-only, system-defined views. These metadata views provide a unified, easy-to-use interface that allows you to have an overall insight into the privilege structure, object dependencies, and other information within your StarRocks cluster. #### View metadata information via `sys`[​](#view-metadata-information-via-sys "Direct link to view-metadata-information-via-sys") You can view the exclusive metadata information within a StarRocks instance by querying the content of views in `sys`. The following example checks the privileges granted to user-defined roles by querying the view `grants_to_roles`. ```plain MySQL > SELECT * FROM sys.grants_to_roles LIMIT 5\G *************************** 1. row *************************** GRANTEE: role_test OBJECT_CATALOG: default_catalog OBJECT_DATABASE: db_test OBJECT_NAME: tbl1 OBJECT_TYPE: TABLE PRIVILEGE_TYPE: SELECT, ALTER IS_GRANTABLE: NO *************************** 2. row *************************** GRANTEE: role_test OBJECT_CATALOG: default_catalog OBJECT_DATABASE: db_test OBJECT_NAME: tbl2 OBJECT_TYPE: TABLE PRIVILEGE_TYPE: SELECT IS_GRANTABLE: YES *************************** 3. row *************************** GRANTEE: role_test OBJECT_CATALOG: default_catalog OBJECT_DATABASE: db_test OBJECT_NAME: mv_test OBJECT_TYPE: MATERIALIZED VIEW PRIVILEGE_TYPE: SELECT IS_GRANTABLE: YES *************************** 4. row *************************** GRANTEE: role_test OBJECT_CATALOG: NULL OBJECT_DATABASE: NULL OBJECT_NAME: NULL OBJECT_TYPE: SYSTEM PRIVILEGE_TYPE: CREATE RESOURCE GROUP IS_GRANTABLE: NO ``` #### Views in `sys`[​](#views-in-sys "Direct link to views-in-sys") `sys` contains the following metadata views: | **View** | **Description** | | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | [grants\_to\_roles](https://docs.starrocks.io/docs/sql-reference/sys/grants_to_roles.md) | Records the information of privileges that are granted to user-defined roles. | | [grants\_to\_users](https://docs.starrocks.io/docs/sql-reference/sys/grants_to_users.md) | Records the information of privileges that are granted to users. | | [role\_edges](https://docs.starrocks.io/docs/sql-reference/sys/role_edges.md) | Records the grantees of roles. | | [object\_dependencies](https://docs.starrocks.io/docs/sql-reference/sys/object_dependencies.md) | Records the dependency relationship of asynchronous materialized views. | | [policy\_references](https://docs.starrocks.io/docs/sql-reference/sys/policy_references.md) | Records the objects row access policies and column masking policies reference. | | [fe\_locks](https://docs.starrocks.io/docs/sql-reference/sys/fe_locks.md) | Records the information about metadata locks in StarRocks FE. | note As per their scenarios of applications, views in `sys` are only accessible to some `admin` roles by default. Depending on your specific needs, you can grant other users the SELECT privilege on these views. --- ### fe_locks `fe_locks` provides information about metadata locks in StarRocks FE. It is one of the system-defined views in the [`sys`](https://docs.starrocks.io/docs/sql-reference/sys.md) database. The following fields are provided in `fe_locks`: | **Field** | **Description** | | -------------- | ------------------------------------------------------------------- | | lock\_type | Type of the lock (e.g., "DATABASE"). | | lock\_object | Object identifier that is locked (database name, table name, etc.). | | lock\_mode | Lock mode. Valid values: `EXCLUSIVE`, `SHARED`. | | start\_time | When the lock was acquired. | | hold\_time\_ms | Duration the lock has been held (in milliseconds). | | thread\_info | JSON string containing thread information (threadId, threadName). | | granted | Whether the lock is currently granted. | | waiter\_list | Comma-separated list of threads waiting for this lock. | #### Lock Modes[​](#lock-modes "Direct link to Lock Modes") * **EXCLUSIVE**: Only one thread can hold this lock at a time. * **SHARED**: Multiple threads can hold this lock simultaneously. #### Lock Types[​](#lock-types "Direct link to Lock Types") * **DATABASE**: Database-level locks. * **TABLE**: Table-level locks. StarRocks manages metadata locks through the Lock Manager, which provides centralized lock management with table-level granularity. #### Examples[​](#examples "Direct link to Examples") ##### Find long-running locks[​](#find-long-running-locks "Direct link to Find long-running locks") ```sql SELECT lock_object, lock_mode, hold_time_ms, thread_info FROM sys.fe_locks WHERE hold_time_ms > 10000 -- Locks held for more than 10 seconds ORDER BY hold_time_ms DESC; ``` ##### Check for lock contention[​](#check-for-lock-contention "Direct link to Check for lock contention") ```sql SELECT lock_object, COUNT(*) as lock_count FROM sys.fe_locks WHERE granted = true GROUP BY lock_object HAVING COUNT(*) > 1; ``` ##### Find waiting threads[​](#find-waiting-threads "Direct link to Find waiting threads") ```sql SELECT lock_object, waiter_list FROM sys.fe_locks WHERE waiter_list != ''; ``` #### Notes[​](#notes "Direct link to Notes") * Querying `fe_locks` requires `OPERATE` privilege. * The view provides real-time information but may impact performance on systems with high lock activity. * Use appropriate WHERE clauses to filter results when dealing with large numbers of locks. --- ### grants_to_roles You can view the privileges granted to user-defined roles by querying the view `grants_to_roles`. note By default, only users or roles with the `user_admin` role can access this view. You can grant other users the SELECT privilege on this view using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md). The following fields are provided in `grants_to_roles`: | **Field** | **Description** | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GRANTEE | The role to whom this privilege is granted. Only user-defined roles will be listed in this view, and system-defined ones will not. | | OBJECT\_CATALOG | The catalog to which the object belongs. `NULL` is returned if the privilege is a SYSTEM, RESOURCE GROUP, RESOURCE, USER, or GLOBAL FUNCTION level privilege. | | OBJECT\_DATABASE | The database to which the object belongs. `NULL` is returned if the privilege is a SYSTEM, RESOURCE GROUP, RESOURCE, USER, GLOBAL FUNCTION, or CATALOG level privilege. | | OBJECT\_NAME | The table to which the object belongs. `NULL` is returned if the privilege is a SYSTEM, RESOURCE GROUP, RESOURCE, USER, GLOBAL FUNCTION, CATALOG, or DATABASE level privilege. | | OBJECT\_TYPE | The type of the object. | | PRIVILEGE\_TYPE | The type of the privilege. Different privileges on the same object will be merged and returned in a single row. As shown in the following example, `role_test` has the SELECT and ALTER privileges on `default_catalog.db_test.tbl1`. | | IS\_GRANTABLE | Whether the grantee has the grant option. | Example: ```plain MySQL > SELECT * FROM sys.grants_to_roles LIMIT 5\G *************************** 1. row *************************** GRANTEE: role_test OBJECT_CATALOG: default_catalog OBJECT_DATABASE: db_test OBJECT_NAME: tbl1 OBJECT_TYPE: TABLE PRIVILEGE_TYPE: SELECT, ALTER IS_GRANTABLE: NO *************************** 2. row *************************** GRANTEE: role_test OBJECT_CATALOG: default_catalog OBJECT_DATABASE: db_test OBJECT_NAME: tbl2 OBJECT_TYPE: TABLE PRIVILEGE_TYPE: SELECT IS_GRANTABLE: YES *************************** 3. row *************************** GRANTEE: role_test OBJECT_CATALOG: default_catalog OBJECT_DATABASE: db_test OBJECT_NAME: mv_test OBJECT_TYPE: MATERIALIZED VIEW PRIVILEGE_TYPE: SELECT IS_GRANTABLE: YES *************************** 4. row *************************** GRANTEE: role_test OBJECT_CATALOG: NULL OBJECT_DATABASE: NULL OBJECT_NAME: NULL OBJECT_TYPE: SYSTEM PRIVILEGE_TYPE: CREATE RESOURCE GROUP IS_GRANTABLE: NO ``` --- ### grants_to_users You can view the privileges granted to users by querying the view `grants_to_users`. note By default, only users or roles with the `user_admin` role can access this view. You can grant other users the SELECT privilege on this view using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md). The following fields are provided in `grants_to_users`: | **Field** | **Description** | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GRANTEE | The user to whom this privilege is granted. | | OBJECT\_CATALOG | The catalog to which the object belongs. `NULL` is returned if the privilege is a SYSTEM, RESOURCE GROUP, RESOURCE, USER, or GLOBAL FUNCTION level privilege. | | OBJECT\_DATABASE | The database to which the object belongs. `NULL` is returned if the privilege is a SYSTEM, RESOURCE GROUP, RESOURCE, USER, GLOBAL FUNCTION, or CATALOG level privilege. | | OBJECT\_NAME | The table to which the object belongs. `NULL` is returned if the privilege is a SYSTEM, RESOURCE GROUP, RESOURCE, USER, GLOBAL FUNCTION, CATALOG, or DATABASE level privilege. | | OBJECT\_TYPE | The type of the object. | | PRIVILEGE\_TYPE | The type of the privilege. Different privileges on the same object will be merged and returned in a single row. As shown in the following example, `'user1'@'%'` has the SELECT and DROP privileges on `default_catalog.db_test.view_test`. | | IS\_GRANTABLE | Whether the grantee has the grant option. | Example: ```plain MySQL > SELECT * FROM sys.grants_to_users LIMIT 5\G *************************** 1. row *************************** GRANTEE: 'user1'@'%' OBJECT_CATALOG: default_catalog OBJECT_DATABASE: db_test OBJECT_NAME: view_test OBJECT_TYPE: VIEW PRIVILEGE_TYPE: SELECT, DROP IS_GRANTABLE: NO *************************** 2. row *************************** GRANTEE: 'user2'@'%' OBJECT_CATALOG: default_catalog OBJECT_DATABASE: simo OBJECT_NAME: view_test OBJECT_TYPE: VIEW PRIVILEGE_TYPE: SELECT IS_GRANTABLE: NO ``` --- ### object_dependencies You can view the dependency relationship of asynchronous materialized views by querying the view `object_dependencies`. The following fields are provided in `object_dependencies`: | **Field** | **Description** | | --------------------- | -------------------------------------------------------------------------------- | | object\_id | ID of the object. | | object\_name | The name of the object. | | object\_database | The Database where the object is located. | | object\_catalog | The catalog where the object is located. This field is always `default_catalog`. | | object\_type | The type of the object. This field is always `MATERIALIZED_VIEW`. | | ref\_object\_id | ID of the referenced object. | | ref\_object\_name | The name of the referenced object. | | ref\_object\_database | The database where the referenced object is located. | | ref\_object\_catalog | The catalog where the referenced object is located. | | ref\_object\_type | The type of the referenced object. | Example: ```plain MySQL > SELECT * FROM sys.object_dependencies\G *************************** 1. row *************************** object_id: 11115 object_name: mv2 object_database: test_db object_catalog: default_catalog object_type: MATERIALIZED_VIEW ref_object_id: 11092 ref_object_name: mv1 ref_object_database: test_db ref_object_catalog: default_catalog ref_object_type: MATERIALIZED_VIEW *************************** 2. row *************************** object_id: 11092 object_name: mv1 object_database: test_db object_catalog: default_catalog object_type: MATERIALIZED_VIEW ref_object_id: 11074 ref_object_name: test_tbl ref_object_database: test_db ref_object_catalog: default_catalog ref_object_type: OLAP ``` The above example indicates that the materialized view `mv1` is created based on the StarRocks internal table `test_tbl`, and the materialized view `mv2` is created based on the materialized view `mv1`. --- ### policy_references note This view applies to a feature not available in StarRocks. You can view the objects row access policies and column masking policies reference by querying the view `policy_references`. The following fields are provided in `policy_references`: | **Field** | **Description** | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | POLICY\_DATABASE | The database where the policy is located. The namespace of the policy is the database, which is allowed only under `default_catalog`. | | POLICY\_NAME | The name of the policy. | | POLICY\_TYPE | The type of the policy. Valid values: `Row Access` and `Column Masking`. | | REF\_CATALOG | The catalog where the policy is referenced. | | REF\_DATABASE | The database where the policy is referenced. | | REF\_OBJECT\_NAME | The table where the policy is referenced. | | REF\_COLUMN | The column where the policy is referenced. This field is always NULL for row access policies. | Example: ```sql mysql > SELECT * FROM policy_references\G *************************** 1. row *************************** POLICY_DATABASE: test_db POLICY_NAME: region_data POLICY_TYPE: Row Access REF_CATALOG: default_catalog REF_DATABASE: test_db REF_OBJECT_NAME: test1 -- 表名 REF_COLUMN: NULL *************************** 2. row *************************** POLICY_DATABASE: test_db POLICY_NAME: phone_mask POLICY_TYPE: Column Masking REF_CATALOG: default_catalog REF_DATABASE: test_db REF_OBJECT_NAME: test REF_COLUMN: phone 2 rows in set (0.03 sec) ``` --- ### role_edges You can view the grantees of roles by querying the view `role_edges`. note By default, only users or roles with the `user_admin` role can access this view. You can grant other users the SELECT privilege on this view using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md). The following fields are provided in `role_edges`: | **Field** | **Description** | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | FROM\_ROLE | The role that is granted. Multiple records with the same `FROM_ROLE` can be returned because a role can be granted to multiple roles and users. | | TO\_ROLE | The role to whom the current `FROM_ROLE` is granted. `NULL` is returned if the current `FROM_ROLE` is granted to a user. | | TO\_USER | The user to whom the current `FROM_ROLE` is granted. `NULL` is returned if the current `FROM_ROLE` is granted to a role. | Example: ```plain MySQL > SELECT * FROM sys.role_edges; +------------+------------+---------------+ | FROM_ROLE | TO_ROLE | TO_USER | +------------+------------+---------------+ | dba_leader | director | NULL | | dba | dba_leader | NULL | | dba | NULL | 'aneesh'@'%' | | dba | NULL | 'chelsea'@'%' | | dba_leader | NULL | 'albert'@'%' | | director | NULL | 'stan'@'%' | | root | NULL | 'root'@'%' | +------------+------------+---------------+ ``` The following diagram illustrates the nesting relation of the granted roles in the preceding example: ![role\_edges](/assets/images/role_edges-f773fa6417aa7443d2d766c7209b5e9e.png) --- ### System limits This topic describes the rules and limits that apply when you use StarRocks. * StarRocks uses the MySQL protocol for communication. You can connect to the StarRocks cluster via a MySQL client or JDBC. We recommend that you use a MySQL client whose version is 5.1 or later. Versions earlier than 5.1 do not support usernames that are longer than 16 characters. * Naming conventions for objects such as catalogs, databases, tables, views, asynchronous materialized views, partitions, columns, indexes, usernames, roles, repositories, resources, storage volumes, and pipes: * The name can only consist of digits (0-9), letters (a-z or A-Z), and underscores (`_`). **Usernames can be all digits.** * The name can start with a letter or an underscore (`_`). * The name cannot exceed 64 characters in length, **among which:** * Database name cannot exceed 256 characters. * Table name and column name cannot exceed 1024 characters. * Username cannot exceed 128 characters. * Column name (column alias), partition name, and index name are **not** case-sensitive. Other names are **case-sensitive**. * The FE configuration item `enable_table_name_case_insensitive` (supported from v4.0 onwards) controls whether catalog names, database names, table names, view names, and asynchronous materialized view names are case-insensitive. This feature is **disabled by default** (these names are case-sensitive), and it can be enabled only when creating a cluster. warning **We strongly recommend that you keep this feature disabled unless you have a specific, well-understood reason to enable it.** When enabled, StarRocks stores the affected names in lowercase and forcibly converts every catalog, database, table, view, and materialized view name to lowercase during **both query and write (DDL/DML) processing**. This has the following consequences: * **It can make external tables and external catalogs unusable.** Different external catalog services follow different naming and case-sensitivity conventions. If an external schema, database, or table name is not already in lowercase, StarRocks lowercases the name in your SQL before passing it to the connector and then looks up a name that does not exist in the source, so the query fails with a "not found" error. * **It cannot be changed after the cluster is created.** After the cluster is started, the value of this configuration cannot be modified by any means. Any attempt to modify it results in an error, and FE fails to start when it detects that the value is inconsistent with the value used when the cluster was first started. Only enable this feature on a new cluster where you are certain that **all** object names — including those in every external data source you plan to access — are already in lowercase. * Naming conventions for labels: You can specify the label of a job when you load data. The label name can consist of digits (0-9), letters (a-z or A-Z), and underscores (`_`), and cannot exceed 128 characters in length. Label names can start with a letter or an underscore (`_`). * When you create a table, the key column cannot be of the FLOAT or DOUBLE type. You can use the DECIMAL type to represent decimals. * The maximum length of a VARCHAR value varies in different versions: * In versions earlier than StarRocks 2.1, the length ranges from 1 to 65533 bytes. * \[Preview] In StarRocks 2.1 and later versions, the length ranges from 1 to 1048576 bytes. Maximum length of a VARCHAR value = Maximum row size (1048578 bytes) - Length prefix (2 bytes). The length prefix indicates the number of bytes in the value. * The default length is 1 byte. * StarRocks supports only UTF-8 encoding, not GBK. * StarRocks does not support modifying the table types of an existing table. For example, you cannot change a Duplicate Key table to a Primary Key table. You must create a new table. * By default, a query can be nested with a maximum of 10,000 subqueries, which is controlled by the FE parameter `expr_children_limit`. --- ### System variables StarRocks provides many system variables that can be set and modified to suit your requirements. This section describes the variables supported by StarRocks. You can view the settings of these variables by running the [SHOW VARIABLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/SHOW_VARIABLES.md) command on your MySQL client. You can also use the [SET](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/SET.md) command to dynamically set or modify variables. You can make these variables take effect globally on the entire system, only in the current session, or only in a single query statement. The variables in StarRocks refer to the variable sets in MySQL, but **some variables are only compatible with the MySQL client protocol and do not function on the MySQL database**. > **NOTE** > > Any user has the privilege to run SHOW VARIABLES and make a variable take effect at session level. However, only users with the SYSTEM-level OPERATE privilege can make a variable take effect globally. Globally effective variables take effect on all the future sessions (excluding the current session). > > If you want to make a setting change for the current session and also make that setting change apply to all future sessions, you can make the change twice, once without the `GLOBAL` modifier and once with it. For example: > > ```sql > SET query_mem_limit = 137438953472; -- Apply to the current session. > SET GLOBAL query_mem_limit = 137438953472; -- Apply to all future sessions. > > ``` #### Variable hierarchy and types[​](#variable-hierarchy-and-types "Direct link to Variable hierarchy and types") StarRocks supports three types (levels) of variables: global variables, session variables, and `SET_VAR` hints. Their hierarchical relationship is as follows: * Global variables take effect on global level, and can be overridden by session variables and `SET_VAR` hints. * Session variables take effect only on the current session, and can be overridden by `SET_VAR` hints. * `SET_VAR` hints take effect only on the current query statement. #### View variables[​](#view-variables "Direct link to View variables") You can view all or some variables by using `SHOW VARIABLES [LIKE 'xxx']`. Example: ```sql -- Show all variables in the system. SHOW VARIABLES; -- Show variables that match a certain pattern. SHOW VARIABLES LIKE '%time_zone%'; ``` #### Set variables[​](#set-variables "Direct link to Set variables") ##### Set variables globally or for a single session[​](#set-variables-globally-or-for-a-single-session "Direct link to Set variables globally or for a single session") You can set variables to take effect **globally** or **only on the current session**. When set to global, the new value will be used for all the future sessions, while the current session still uses the original value. When set to "current session only", the variable will only take effect on the current session. A variable set by `SET = xxx;` only takes effect for the current session. Example: ```sql SET query_mem_limit = 137438953472; SET forward_to_master = true; SET time_zone = "Asia/Shanghai"; ``` A variable set by `SET GLOBAL = xxx;` takes effect globally. Example: ```sql SET GLOBAL query_mem_limit = 137438953472; ``` The following variables only take effect globally. They cannot take effect for a single session, which means you must use `SET GLOBAL = xxx;` for these variables. If you try to set such a variable for a single session (`SET = xxx;`), an error is returned. * activate\_all\_roles\_on\_login * character\_set\_database * cngroup\_low\_watermark\_cpu\_used\_permille * cngroup\_low\_watermark\_running\_query\_count * cngroup\_resource\_usage\_fresh\_ratio * cngroup\_schedule\_mode * default\_rowset\_type * enable\_reduce\_cast\_varchar\_expr\_sync\_type * enable\_reduce\_cast\_varchar\_length\_inheritance * enable\_group\_level\_query\_queue * enable\_query\_history * enable\_query\_queue\_load * enable\_query\_queue\_select * enable\_query\_queue\_statistic * enable\_table\_name\_case\_insensitive * enable\_tde * init\_connect * language * license * lower\_case\_table\_names * performance\_schema * query\_cache\_size * query\_history\_keep\_seconds * query\_history\_load\_interval\_seconds * query\_queue\_concurrency\_limit * query\_queue\_cpu\_used\_permille\_limit * query\_queue\_driver\_high\_water * query\_queue\_driver\_low\_water * query\_queue\_fresh\_resource\_usage\_interval\_ms * query\_queue\_max\_queued\_queries * query\_queue\_mem\_used\_pct\_limit * query\_queue\_pending\_timeout\_second * system\_time\_zone * version * version\_comment In addition, variable settings also support constant expressions, such as: ```sql SET query_mem_limit = 10 * 1024 * 1024 * 1024; ``` ```sql SET forward_to_master = concat('tr', 'u', 'e'); ``` ##### Set variables in a single query statement[​](#set-variables-in-a-single-query-statement "Direct link to Set variables in a single query statement") In some scenarios, you may need to set variables specifically for certain queries. By using the `SET_VAR` hint, you can set session variables that will take effect only within a single statement. StarRocks supports using `SET_VAR` in the following statements; * SELECT * INSERT (from v3.1.12 and v3.2.0 onwards) * UPDATE (from v3.1.12 and v3.2.0 onwards) * DELETE (from v3.1.12 and v3.2.0 onwards) `SET_VAR` can only be placed after the above keywords and enclosed in `/*+...*/`. Example: ```sql SELECT /*+ SET_VAR(query_mem_limit = 8589934592) */ name FROM people ORDER BY name; SELECT /*+ SET_VAR(query_timeout = 1) */ sleep(3); UPDATE /*+ SET_VAR(insert_timeout=100) */ tbl SET c1 = 2 WHERE c1 = 1; DELETE /*+ SET_VAR(query_mem_limit = 8589934592) */ FROM my_table PARTITION p1 WHERE k1 = 3; INSERT /*+ SET_VAR(insert_timeout = 10000000) */ INTO insert_wiki_edit SELECT * FROM FILES( "path" = "s3://inserttest/parquet/insert_wiki_edit_append.parquet", "format" = "parquet", "aws.s3.access_key" = "XXXXXXXXXX", "aws.s3.secret_key" = "YYYYYYYYYY", "aws.s3.region" = "us-west-2" ); ``` You can also set multiple variables in a single statement. Example: ```sql SELECT /*+ SET_VAR ( exec_mem_limit = 515396075520, query_timeout=10000000, batch_size=4096, parallel_fragment_exec_instance_num=32 ) */ * FROM TABLE; ``` ##### Set variables as user properties[​](#set-variables-as-user-properties "Direct link to Set variables as user properties") You can set session variables as user properties using the [ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md). This feature is supported from v3.3.3. Example: ```sql -- Set the session variable `query_timeout` to `600` for the user jack. ALTER USER 'jack' SET PROPERTIES ('session.query_timeout' = '600'); ``` #### Descriptions of variables[​](#descriptions-of-variables "Direct link to Descriptions of variables") The variables are described **in alphabetical order**. Variables with the `global` label can only take effect globally. Other variables can take effect either globally or for a single session. ##### activate\_all\_roles\_on\_login (global)[​](#activate_all_roles_on_login-global "Direct link to activate_all_roles_on_login (global)") * **Description**: Whether to enable all roles (including default roles and granted roles) for a StarRocks user when the user connects to the StarRocks cluster. * If enabled (`true`), all roles of the user are activated at user login. This takes precedence over the roles set by [SET DEFAULT ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_DEFAULT_ROLE.md). * If disabled (`false`), the roles set by SET DEFAULT ROLE are activated. * **Default**: false * **Introduced in**: v3.0 If you want to activate the roles assigned to you in a session, use the [SET ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_DEFAULT_ROLE.md) command. ##### array\_low\_cardinality\_optimize[​](#array_low_cardinality_optimize "Direct link to array_low_cardinality_optimize") * **Scope**: Session * **Description**: Controls whether the optimizer will consider ARRAY\ columns for low-cardinality (dictionary-based) decoding and related optimizations. When enabled, the optimizer's low-cardinality rules (for example, `DecodeCollector`) may define dictionary columns and apply dictionary decoding to expressions whose type is VARCHAR or ARRAY\. When disabled, only scalar VARCHAR columns are eligible and ARRAY\ types are ignored by those low-cardinality optimizations. * **Default**: true * **Data Type**: boolean * **Introduced in**: v3.3.0, v3.4.0, v3.5.0 ##### authentication\_policy[​](#authentication_policy "Direct link to authentication_policy") * **Scope**: Session * **Description**: Session-level variable that holds the raw authentication policy string (kept for MySQL 8.0 compatibility). It is declared and stored on the FE as part of the session state and exposed under the name `authentication_policy`. In the current codebase snapshot the value is stored as an opaque comma-separated string and there is no other usage or internal parsing of this field in FE (it is not referenced elsewhere). It is related to the `default_authentication_plugin` session variable: `authentication_policy` represents per-session authentication policy data while `default_authentication_plugin` indicates the default authentication plugin. Administrators or clients can set this variable per session (for example with `SET authentication_policy = '...'`) to preserve compatibility with MySQL clients or tooling. * **Default**: `*,,` * **Data Type**: String * **Introduced in**: - ##### binary\_encoding\_format[​](#binary_encoding_format "Direct link to binary_encoding_format") * **Scope**: Session * **Description**: Controls how `BINARY` / `VARBINARY` values are encoded when StarRocks serializes MySQL text results. Valid values are `raw`, `hex`, and `base64`. The default is `hex`. This variable works together with `binary_encoding_level`. MySQL clients can already handle top-level binary values, but nested binary values inside `ARRAY`, `MAP`, or `STRUCT` are returned through JSON-like strings, so they may need extra encoding to stay printable and well-formed. Set this variable to `base64` if you prefer a denser printable representation, or `raw` to disable extra encoding entirely. * **Default**: `hex` * **Data Type**: String * **Introduced in**: v4.1 ##### binary\_encoding\_level[​](#binary_encoding_level "Direct link to binary_encoding_level") * **Scope**: Session * **Description**: Controls which binary values are encoded for MySQL text results. Valid values are `nested` and `all`. The default is `nested`, which preserves historical behavior for top-level binary columns while still encoding nested binary values inside `ARRAY`, `MAP`, or `STRUCT`, where the result is rendered as a JSON-like string. Set this variable to `all` if your team wants a uniform convention and prefers top-level binary values to be encoded as well. If `binary_encoding_format = raw`, no additional binary encoding is applied even when this variable is set to `nested` or `all`, which may make nested output less readable. * **Default**: `nested` * **Data Type**: String * **Introduced in**: v4.1 ##### auto\_increment\_increment[​](#auto_increment_increment "Direct link to auto_increment_increment") Used for MySQL client compatibility. No practical usage. ##### avro\_use\_jni\_reader[​](#avro_use_jni_reader "Direct link to avro_use_jni_reader") * **Scope**: Session * **Description**: Controls whether StarRocks uses the JNI-based Avro reader when scanning Avro data from external catalogs such as Hive. When enabled (`true`), the FE sets this session variable on Avro scan ranges and the BE chooses `HdfsAvroScanner` instead of the native Avro scanner path. This option can be used as a compatibility fallback when the native reader does not meet your needs. Current limitation: `CHAR(n)` columns are not fully compatible between the JNI and non-JNI Avro readers. For Avro values such as `Char` read into a `CHAR(10)` column, the current native reader keeps the unpadded value instead of returning `Char` followed by six spaces, while the JNI reader may behave differently. To avoid inconsistent results, we recommend not relying on `CHAR(n)` padding semantics when switching `avro_use_jni_reader`, and using `VARCHAR` instead when possible. * **Default**: `true` * **Data Type**: boolean * **Introduced in**: v4.1.1 ##### big\_query\_profile\_threshold[​](#big_query_profile_threshold "Direct link to big_query_profile_threshold") * **Description**: Used to set the threshold for big queries. When the session variable `enable_profile` is set to `false` and the amount of time taken by a query exceeds the threshold specified by the variable `big_query_profile_threshold`, a profile is generated for that query. Note: In versions v3.1.5 to v3.1.7, as well as v3.2.0 to v3.2.2, we introduced the `big_query_profile_second_threshold` for setting the threshold for big queries. In versions v3.1.8, v3.2.3, and subsequent releases, this parameter has been replaced by `big_query_profile_threshold` to offer more flexible configuration options. * **Default**: 0 * **Unit**: Second * **Data type**: String * **Introduced in**: v3.1 ##### blacklist\_backup\_routing[​](#blacklist_backup_routing "Direct link to blacklist_backup_routing") * **Scope**: Session * **Description**: In shared-data mode, if the compute node the plan prefers for a scan is not among the workers available to the current query (for example, the node is down or appears on the host blocklist), the planner must choose a backup compute node. This variable sets how that backup is chosen among eligible nodes (other than the primary). `RANDOM` samples uniformly at random from the eligible set. `CIRCULAR` walks the sorted compute node id ring from the primary and takes the first eligible node (deterministic). Which nodes are eligible for backup also depends on `skip_black_list`: by default, nodes on the host blocklist are excluded; if `skip_black_list` is `true`, a node that is on the blocklist may still be chosen as a backup when it is otherwise available (for example, alive and in the warehouse). * **Default**: `CIRCULAR` * **Data type**: String * **Valid values**: `CIRCULAR`, `RANDOM` * **Introduced in**: - ##### broadcast\_row\_limit[​](#broadcast_row_limit "Direct link to broadcast_row_limit") * **Scope**: Session * **Description**: Limits the right-side table's output row count for a broadcast join. The optimizer (see `EnforceAndCostTask`) treats a candidate as ineligible for broadcast when the right table's row count exceeds this limit or when combined size/scale checks indicate the right table is not sufficiently smaller than the left. A non‑positive value disables this limit. `PushDownAggregateCollector` also uses this variable to identify "small broadcast" joins for aggregate push‑down: it requires the rightRows less than or equal to this limit and less than or equal to `cbo_push_down_aggregate_on_broadcast_join_row_count_limit` before allowing push‑down. It interacts with `broadcast_right_table_scale_factor` and the number of BE nodes when comparing left/right sizes. For example, when `leftOutputSize` is less than `rightOutputSize * beNum * broadcast_right_table_scale_factor` and `rightRowCount` is greater than `broadcast_row_limit`, broadcast is rejected. * **Default**: `15000000` * **Data Type**: long * **Introduced in**: v3.2.0 ##### catalog[​](#catalog "Direct link to catalog") * **Description**: Used to specify the catalog to which the session belongs. * **Default**: default\_catalog * **Data type**: String * **Introduced in**: v3.2.4 ##### cbo\_cte\_force\_reuse\_node\_count[​](#cbo_cte_force_reuse_node_count "Direct link to cbo_cte_force_reuse_node_count") * **Description**: Session-scoped threshold that controls an optimizer shortcut for Common Table Expressions (CTEs). In RelationTransformer.visitCTE the planner counts nodes in a CTE producer tree (cteContext.getCteNodeCount). If that count is greater than or equal to this threshold and the threshold is greater than 0, the transformer forces reuse of the CTE: it skips inlining/transforming the producer plan, builds a consume operator with precomputed expression mappings (no inputs) and uses generated column refs instead. This reduces optimizer time for very large CTE producer trees at the cost of potentially less optimal physical plans. Setting the value to `0` disables the force-reuse optimization. This variable has a getter/setter in SessionVariable and is applied per session. * **Scope**: Session * **Default**: `2000` * **Data Type**: int * **Introduced in**: v3.5.3 ##### cbo\_cte\_reuse[​](#cbo_cte_reuse "Direct link to cbo_cte_reuse") * **Description**: Controls whether the optimizer may rewrite multi-distinct aggregate queries by reusing a Common Table Expression (CTE) (the CBO CTE‑reuse rewrite). When enabled, the planner (RewriteMultiDistinctRule) may choose a CTE-based rewrite for multi-column distincts, skewed aggregations, or when statistics indicate the CTE rewrite is more efficient; it also respects the `prefer_cte_rewrite` hint. When disabled, CTE-based rewrite is not allowed and the planner will attempt the multi-function rewrite; if a query requires CTE (for example, multi-column DISTINCT or functions that cannot be handled by multi-function rewrite) the planner will raise a user error. Note: the effective setting checked by the optimizer is the logical AND of this flag and the pipeline engine flag — i.e. `isCboCteReuse()` returns this variable AND `enablePipelineEngine`, so CTE reuse is only effective when `enablePipelineEngine` is on. * **Default**: `true` * **Data Type**: Boolean * **Introduced in**: `v3.2.0` ##### cbo\_disabled\_rules[​](#cbo_disabled_rules "Direct link to cbo_disabled_rules") * **Description**: Comma-separated list of optimizer rule names to disable for the current session. Each name must match a `RuleType` enum value and only rules whose names start with `TF_` (transformation rules) or `GP_` (group-combination rules) may be disabled. The session variable is stored on `SessionVariable` (`getCboDisabledRules` / `setCboDisabledRules`) and is applied by the optimizer via `OptimizerOptions.applyDisableRuleFromSessionVariable()`, which parses the list and clears the corresponding rule switches so those rules are skipped during planning. When set through a SET statement, values are validated and the server will reject unknown names or names not starting with `TF_`/`GP_` with clear error messages (e.g. "Unknown rule name(s): ..." or "Only TF\_ ... and GP\_ ... can be disabled"). At planner runtime, unknown rule names are ignored with a warning (logged as "Ignoring unknown rule name: ... (may be from different version)"). Names must match enum identifiers exactly (case-sensitive). Whitespace around names is trimmed; empty entries are ignored. * **Scope**: Session * **Default**: `""` (no disabled rules) * **Data Type**: String * **Introduced in**: - ##### cbo\_enable\_low\_cardinality\_optimize[​](#cbo_enable_low_cardinality_optimize "Direct link to cbo_enable_low_cardinality_optimize") * **Description**: Whether to enable low cardinality optimization. After this feature is enabled, the performance of querying STRING columns improves by about three times. * **Default**: true ##### cbo\_enable\_low\_cardinality\_optimize\_for\_join[​](#cbo_enable_low_cardinality_optimize_for_join "Direct link to cbo_enable_low_cardinality_optimize_for_join") * **Scope**: Session * **Description**: Controls whether the optimizer rewrites join operators to take advantage of low-cardinality (dictionary-encoded) string columns. When enabled (default), the optimizer (DecodeCollector / DecodeRewriter) will: * rewrite join ON predicates, join predicates and projections to use dictionary-encoded column refs for low-cardinality string columns (affects hash joins); * extract equality groups for join columns and enable decoding-based optimizations for supported join patterns. When disabled, join-related low-cardinality rewrites are skipped and string columns are left unchanged for join processing. Note: the implementation currently targets broadcast joins and avoids rewrite when both sides use shuffle distribution. * **Default**: `true` * **Data Type**: boolean * **Introduced in**: - ##### cbo\_eq\_base\_type[​](#cbo_eq_base_type "Direct link to cbo_eq_base_type") * **Description**: Specifies the data type used for data comparison between DECIMAL data and STRING data. The default value is `DECIMAL`, and VARCHAR is also a valid value. **This variable takes effect only for `=` and `!=` comparison.** * **Data type**: String * **Introduced in**: v2.5.14 ##### cbo\_json\_v2\_dict\_opt[​](#cbo_json_v2_dict_opt "Direct link to cbo_json_v2_dict_opt") * **Description**: Whether to enable low-cardinality dictionary optimization for Flat JSON (JSON v2) extended string subcolumns created by JSON path rewrite. When enabled, the optimizer may build and use global dictionaries for those subcolumns to accelerate string expressions, GROUP BY, and JOIN operations. * **Default**: true * **Data type**: Boolean ##### cbo\_json\_v2\_rewrite[​](#cbo_json_v2_rewrite "Direct link to cbo_json_v2_rewrite") * **Description**: Whether to enable JSON v2 path rewrite in the optimizer. When enabled, JSON functions (such as `get_json_*`) can be rewritten to direct access of Flat JSON subcolumns, enabling predicate pushdown, column pruning, and dictionary optimization. * **Default**: true * **Data type**: Boolean ##### cbo\_max\_reorder\_node\_use\_dp[​](#cbo_max_reorder_node_use_dp "Direct link to cbo_max_reorder_node_use_dp") * **Description**: Session-scoped limit that controls when the cost-based optimizer (CBO) will include the DP (dynamic programming) join-reorder algorithm. The optimizer compares the number of join inputs (MultiJoinNode.atoms.size()) against this value and only runs or adds the DP reorder when `multiJoinNode.getAtoms().size() <= cbo_max_reorder_node_use_dp` and `cbo_enable_dp_join_reorder` is enabled. Used in JoinReorderFactory.createJoinReorderAdaptive (to add JoinReorderDP to the candidate algorithms) and in ReorderJoinRule.transform/rewrite (to decide whether to execute JoinReorderDP when copying plans into the memo). Default value 10 reflects a practical performance cutoff (comment in code: "10 table join reorder takes more than 100ms"). Tune this to trade optimizer runtime (DP is expensive) versus potential plan quality for larger multi-join queries. Interacts with `cbo_enable_dp_join_reorder` and the greedy threshold `cbo_max_reorder_node_use_greedy`. The comparison is inclusive (`<=`). * **Scope**: Session * **Default**: `10` * **Data Type**: long * **Introduced in**: `v3.2.0` ##### cbo\_max\_reorder\_node\_use\_exhaustive[​](#cbo_max_reorder_node_use_exhaustive "Direct link to cbo_max_reorder_node_use_exhaustive") * **Scope**: Session * **Description**: Controls the join-reorder algorithm selection threshold in the CBO. The optimizer counts inner/cross join nodes in the query; when that count is greater than this value the planner takes the transform-based (more aggressive) reorder path: it forces collection of CTE statistics and calls ReorderJoinRule.transform and related commutativity rules. When the count is less than or equal to this value the planner applies the cheaper join-transformation rules (and may add the INNER\_JOIN\_LEFT\_ASSCOM\_RULE for certain semi/anti-join cases). This session variable is read by the optimizer (`SPMOptimizer`, `QueryOptimizer`) and can be set at the session level via `setMaxTransformReorderJoins`. * **Default**: `4` * **Data Type**: int * **Introduced in**: v3.2.0 ##### cbo\_max\_reorder\_node\_use\_greedy[​](#cbo_max_reorder_node_use_greedy "Direct link to cbo_max_reorder_node_use_greedy") * **Description**: Maximum number of join inputs (atoms) in a multi-join for which the cost-based optimizer will consider the greedy join-reorder algorithm. The optimizer checks this limit (together with `cbo_enable_greedy_join_reorder`) when building the list of candidate reorder algorithms: if `multiJoinNode.getAtoms().size()` is less than or equal to this value, a `JoinReorderGreedy` instance will be added and executed. This variable is used by `JoinReorderFactory.createJoinReorderAdaptive()` and `ReorderJoinRule` to gate greedy reordering during join-reorder phases; it applies per session and affects whether greedy reordering is attempted (if statistics are available and greedy is enabled). Adjust this to control optimizer time/complexity trade-offs for queries with many joined relations. * **Scope**: Session (can be changed per-session) * **Default**: `16` * **Data type**: long * **Introduced in**: v3.4.0, v3.5.0 ##### cbo\_prune\_json\_subfield[​](#cbo_prune_json_subfield "Direct link to cbo_prune_json_subfield") * **Scope**: Session * **Description**: When enabled, the cost-based optimizer collects and prunes JSON subfield expressions so that JSON access paths (subfields) are recognized and converted into ColumnAccessPath for scan operators. This enables flat-JSON path optimizations and push-down of JSON subfield access into the scan layer (see PruneSubfieldRule and SubfieldExpressionCollector). Note: normalization of cast-from-JSON expressions is gated by the general `cbo_prune_subfield` optimization; both work together to produce `get_json_xxx(...)` or cast-wrapped calls so BE can apply flat JSON optimizations. Enabling `cbo_prune_json_subfield` without backend support for `flat json` may degrade performance; disable it if the BE does not support flat JSON path pushdown. * **Default**: `true` * **Data type**: boolean * **Introduced in**: v3.3.0, v3.4.0, v3.5.0 ##### cbo\_use\_correlated\_predicate\_estimate[​](#cbo_use_correlated_predicate_estimate "Direct link to cbo_use_correlated_predicate_estimate") * **Description**: Session flag that controls whether the optimizer applies a correlation-aware heuristic when estimating selectivity for conjunctive equality predicates across multiple columns. When enabled (default), the estimator applies exponential-decay weights to the selectivities of additional columns beyond the primary multi-column stats or most selective predicate, reducing the multiplicative impact of further predicates (weights: 0.5, 0.25, 0.125 for up to three additional columns). When disabled, no decay is applied (decay factor = 1) and the estimator multiplies full selectivities for those columns (stronger independence assumption). This flag is checked by StatisticsEstimateUtils.estimateConjunctiveEqualitySelectivity to choose the decay factor in both the multi-column-statistics path and the fallback path, thereby affecting cardinality estimates used by the CBO. * **Scope**: Session * **Default**: `true` * **Data Type**: boolean * **Introduced in**: v3.5.0 ##### character\_set\_database (global)[​](#character_set_database-global "Direct link to character_set_database (global)") * **Data type**: StringThe character set supported by StarRocks. Only UTF8 (`utf8`) is supported. * **Default**: utf8 * **Data type**: String ##### collation\_connection[​](#collation_connection "Direct link to collation_connection") * **Description**: Session-scoped variable that stores the connection collation name for the current client session. It is declared in `SessionVariable` as `collationConnection` and exposed with the show-name `collation_connection`. The variable is surfaced in metadata and SHOW outputs (for example, used when building `SHOW CREATE VIEW` rows in `ShowExecutor` and returned as a constant for `information_schema.views` via `ViewsSystemTable.CONSTANT_MAP`). It represents the collation reported by the server for the connection (paired with `character_set_client` and related character-set variables) but does not by itself indicate runtime collation enforcement beyond what other components implement. * **Scope**: Session * **Default**: `utf8_general_ci` * **Data Type**: String * **Introduced in**: v3.2.0 ##### collation\_database[​](#collation_database "Direct link to collation_database") * **Description**: Session-level variable that holds the default database collation name for the current session. It is declared in SessionVariable (annotated with `@VariableMgr.VarAttr`) and lives alongside other charset/collation session variables such as `character_set_client`, `collation_connection` and `collation_server`. The value is serialized when session variables are exported (e.g., included in the JSON produced by SessionVariable#getJsonString and in the session variable machinery), and is used to report the session's database collation. Changing this variable updates the session's reported database collation name; engine-level or object-level collation settings (server/table/column) may still take precedence for actual comparison/ordering behavior depending on context. * **Scope**: Session * **Default**: `utf8_general_ci` * **Data Type**: String * **Introduced in**: v3.2.0 ##### collation\_server[​](#collation_server "Direct link to collation_server") * **Scope**: Session * **Description**: Session-level server collation name used by the FE to present MySQL-compatible collation behavior for this session. This variable sets the default collation identifier (for example `utf8_general_ci`) that FE reports to clients and that is associated with `character_set_server` / `collation_connection` / `collation_database`. It is persisted in the session variable JSON (see SessionVariable#getJsonString / replayFromJson) and is exposed via the variable manager (`@VarAttr(name = COLLATION_SERVER)`), so it appears in SHOW VARIABLES and can be changed per-session. The value is stored as a plain String in SessionVariable and typically holds a standard MySQL collation name (e.g. `utf8_general_ci`, `utf8mb4_unicode_ci`); the code does not enforce a fixed enum or perform additional validation here, so the effective behavior depends on downstream components that interpret the collation name for comparisons, ordering and other collation-sensitive operations. * **Default Value**: `utf8_general_ci` * **Data Type**: String * **Introduced in**: `v3.2.0` ##### computation\_fragment\_scheduling\_policy[​](#computation_fragment_scheduling_policy "Direct link to computation_fragment_scheduling_policy") * **Scope**: Session * **Description**: Controls the scheduler policy used to choose execution instances for computation fragments. Valid values (case-insensitive) are: * `compute_nodes_only` — schedule fragments only on compute nodes (default). * `all_nodes` — allow scheduling on both compute nodes and traditional backend nodes. The variable is backed by the enum `SessionVariableConstants.ComputationFragmentSchedulingPolicy`. When set, the value is validated (upper-cased) against the enum; invalid values cause an error (`IllegalArgumentException` when set via API, `SemanticException` when used in a SET statement). The getter returns the corresponding enum value and falls back to `COMPUTE_NODES_ONLY` if unset or unrecognized. This setting affects how the FE chooses target nodes for fragment placement at planning/deployment time. * **Default**: `COMPUTE_NODES_ONLY` * **Data Type**: String * **Introduced in**: v3.2.7 ##### connector\_io\_tasks\_per\_scan\_operator[​](#connector_io_tasks_per_scan_operator "Direct link to connector_io_tasks_per_scan_operator") * **Description**: The maximum number of concurrent I/O tasks that can be issued by a scan operator during external table queries. The value is an integer. Currently, StarRocks can adaptively adjust the number of concurrent I/O tasks when querying external tables. This feature is controlled by the variable `enable_connector_adaptive_io_tasks`, which is enabled by default. * **Default**: 16 * **Data type**: Int * **Introduced in**: v2.5 ##### connector\_sink\_compression\_codec[​](#connector_sink_compression_codec "Direct link to connector_sink_compression_codec") * **Description**: Specifies the compression algorithm used for writing data into Hive tables or Iceberg tables, or exporting data with Files(). This parameter only takes effect in the following situations: * The `compression_codec` property does not exist in the Hive tables. * The `write.parquet.compression-codec` properties do not exist in the Iceberg tables. * The `compression` property is not set for `INSERT INTO FILES`. * **Valid values**: `uncompressed`, `snappy`, `lz4`, `zstd`, and `gzip`. * **Default**: uncompressed * **Data type**: String * **Introduced in**: v3.2.3 ##### connector\_sink\_target\_max\_file\_size[​](#connector_sink_target_max_file_size "Direct link to connector_sink_target_max_file_size") * **Description**: Specifies the maximum size of target file for writing data into Hive tables or Iceberg tables, or exporting data with Files(). The limit is not exact and is applied on a best-effort basis. * **Unit**: Bytes * **Default**: 1073741824 * **Data type**: Long * **Introduced in**: v3.3.0 ##### count\_distinct\_column\_buckets[​](#count_distinct_column_buckets "Direct link to count_distinct_column_buckets") * **Description**: The number of buckets for the COUNT DISTINCT column in a group-by-count-distinct query. This variable takes effect only when `enable_distinct_column_bucketization` is set to `true`. * **Default**: 1024 * **Introduced in**: v2.5 ##### count\_distinct\_implementation[​](#count_distinct_implementation "Direct link to count_distinct_implementation") * **Description**: Controls the function implementation when `COUNT(DISTINCT expr)` contains only one parameter. Valid values (case-insensitive): * `default`: Reserves the `COUNT(DISTINCT expr)` implementation. The optimizer chooses the suitable aggregation plan based on query form, statistics, and costs. * `multi_count_distinct`: Changes the `COUNT(DISTINCT expr)` implementation to `multi_distinct_count` for precise counting. For counting on low- and medium-cardinality columns, this implementation can reduce a shuffle and deduplication phase, and thereby increase the speed. However, it will reserve the distinct values in HashSet, causing excessive memory consumption and even OOM when deduplicating high-cardinality columns. Do not set this value globally without first verifying it using representative loads. * `ndv`:Changes the `COUNT(DISTINCT expr)` implementation to `ndv(expr)`. This function uses HyperLogLog, which returns approximate results with lower memory overhead. * **Default**: `default` * **Introduced in**: v3.3.6、v3.4.0 Usage Notes for `multi_distinct_count` `multi_distinct_count()` returns precise results. For most queries, `COUNT(DISTINCT expr)` is recommended. Set `count_distinct_implementation` to `default` to allow the optimizer to choose a suitable aggregation plan. When deduplicating low- and medium-cardinality columns, you can test and use `multi_distinct_count()`. This function uses two phases of aggregation, and can reduce a shuffle and deduplication phase for better performance. However, its HashSet status and final merging can cause excessive memory consumption and even OOM when deduplicating high-cardinality columns. If you want to test this implementation on one `COUNT(DISTINCT expr)` instead of changing the whole session, you can set `count_distinct_implementation` in a query hint: ```sql SELECT /*+ SET_VAR(count_distinct_implementation = multi_count_distinct) */ COUNT(DISTINCT category) FROM test; ``` Setting this value with hints applies only to `COUNT(DISTINCT)` with a single parameter. It will not affect multi-column deduplication expressions such as `COUNT(DISTINCT expr1, expr2)`. ##### custom\_query\_id (session)[​](#custom_query_id-session "Direct link to custom_query_id (session)") * **Description**: Used to bind some external identifier to a current query. Can be set using `SET SESSION custom_query_id = 'my-query-id';` before executing a query. The value is reset after query is finished. This value can be passed to `KILL QUERY 'my-query-id'`. Value can be found in audit logs as a `customQueryId` field. * **Default**: "" * **Data type**: String * **Introduced in**: v3.4.0 ##### custom\_session\_name (session)[​](#custom_session_name-session "Direct link to custom_session_name (session)") * **Description**: Used to specify custom name of current session, analog of `applicationName` or `program_name` in DMBS like MySQL or PostgreSQL. Can be set using `SET SESSION custom_session_name = 'my session name';`. Value can be found in audit logs in `customSessionName` field. * **Default**: "" * **Data type**: String * **Introduced in**: v4.1.0 ##### datacache\_sharing\_work\_period[​](#datacache_sharing_work_period "Direct link to datacache_sharing_work_period") * **Description**: The period of time that Cache Sharing takes effect. After each cluster scaling operation, only the requests within this period of time will try to access the cache data from other nodes if the Cache Sharing feature is enabled. * **Default**: 600 * **Unit**: Seconds * **Introduced in**: v3.5.1 ##### decimal\_overflow\_to\_double[​](#decimal_overflow_to_double "Direct link to decimal_overflow_to_double") * **Scope**: Session * **Description**: When enabled, the analyzer converts decimal arithmetic results that would overflow the maximum decimal precision into 64-bit floating point (`DOUBLE`) instead of widening to larger decimal types or failing. Concretely, in DecimalV3 arithmetic (see DecimalV3FunctionAnalyzer), if a multiplication's computed precision exceeds the engine's max decimal precision but its return scale is within the maximum, the session flag `decimal_overflow_to_double = true` causes the return type and operand target types to be set to `DOUBLE`. This yields an approximate (lossy) numeric result but avoids decimal precision overflow errors or forced use of larger decimal types. When false (default), the planner will keep decimal semantics (attempt decimal128/256 or throw on unrepresentable scale/precision). * **Default**: `false` * **Data Type**: boolean * **Introduced in**: - ##### default\_authentication\_plugin[​](#default_authentication_plugin "Direct link to default_authentication_plugin") * **Scope**: Session * **Description**: Session-scoped variable that specifies the default MySQL authentication plugin name for this session. It is stored as SessionVariable.defaultAuthenticationPlugin and is used by StarRocks' MySQL-protocol compatibility layers when the server needs to advertise or use a default authentication plugin (for example during handshake or when a plugin is not specified). Accepts standard MySQL authentication plugin identifiers (e.g. `mysql_native_password`, `caching_sha2_password`) supported by the server. This variable affects session behavior only; persistent user account authentication configuration is managed separately. See related session variable `authentication_policy`. * **Default**: `mysql_native_password` * **Data Type**: String * **Introduced in**: - ##### default\_rowset\_type (global)[​](#default_rowset_type-global "Direct link to default_rowset_type (global)") Used to set the default storage format used by the storage engine of the computing node. The currently supported storage formats are `alpha` and `beta`. ##### default\_storage\_engine[​](#default_storage_engine "Direct link to default_storage_engine") * **Scope**: Session * **Description**: Session system variable exposed as `default_storage_engine` (see SessionVariable VarAttr). It exists for MySQL 8.0 compatibility and to satisfy MySQL clients/libraries that query the session default storage engine. The variable is stored per-session in the SessionVariable object and is returned to clients (e.g., via SHOW VARIABLES). It is informational for compatibility; changing it adjusts the session-reported value but does not imply StarRocks will change internal storage implementation. * **Default**: `InnoDB` * **Data Type**: String * **Introduced in**: v3.4.2, v3.5.0 ##### default\_table\_compression[​](#default_table_compression "Direct link to default_table_compression") * **Description**: The default compression algorithm for table storage. Supported compression algorithms are `snappy, lz4, zlib, zstd`. Note that if you specified the `compression` property in a CREATE TABLE statement, the compression algorithm specified by `compression` takes effect. * **Default**: lz4\_frame * **Introduced in**: v3.0 ##### default\_tmp\_storage\_engine[​](#default_tmp_storage_engine "Direct link to default_tmp_storage_engine") * **Description**: Session variable that controls the default storage engine used for temporary tables (both explicit `CREATE TEMPORARY TABLE` and internal/implicit temporary tables created by the engine). Declared in `SessionVariable.java` with a `@VariableMgr.VarAttr` annotation, it exists primarily for MySQL 8.0 compatibility so clients and tools expecting MySQL-like behavior can observe or change the temporary-table engine per session. Changing this value affects how temporary table data is stored/managed on storage layers that honor different engines (for example, choosing between memory-backed vs. disk-backed engines). * **Scope**: Session * **Default**: `InnoDB` * **Data Type**: String * **Introduced in**: v3.4.2, v3.5.0 ##### disable\_colocate\_join[​](#disable_colocate_join "Direct link to disable_colocate_join") * **Description**: Used to control whether the Colocation Join is enabled. The default value is `false`, meaning the feature is enabled. When this feature is disabled, query planning will not attempt to execute Colocation Join. * **Default**: false ##### disable\_colocate\_set[​](#disable_colocate_set "Direct link to disable_colocate_set") * **Scope**: Session * **Description**: When false (default), the optimizer may apply "colocate set" handling for set operations (e.g., UNION / UNION DISTINCT) when the first child's hash distribution is local: the planner attempts to keep children colocated — avoiding full repartitioning by checking pairwise colocation and either converting to compatible bucket shuffles or keeping colocated execution. When true, this session flag disables that colocate-set optimization path; the planner will not rely on colocated-set guarantees and will instead fall back to converting the set operation to round-robin distribution or enforce explicit bucket shuffle conversions for non-colocated children. This flag is consulted by the planner (see ChildOutputPropertyGuarantor.visitPhysicalSetOperation) and exposed on the session via SessionVariable getter/setter (`isDisableColocateSet` / `setDisableColocateSet`). * **Default**: `false` * **Data Type**: boolean * **Introduced in**: v3.5.0 ##### disable\_join\_reorder[​](#disable_join_reorder "Direct link to disable_join_reorder") * **Scope**: Session * **Description**: Controls whether the cost-based optimizer performs join reordering. When `false` (default) the optimizer may apply join-reorder transformations (e.g. `ReorderJoinRule`, join transformation and outer-join transformation rules) during logical optimization in the new planner paths (seen in `SPMOptimizer` and `QueryOptimizer`). When `true`, join reordering and related outer-join reorder rules are skipped, preventing the optimizer from changing join order. This is useful to reduce optimization time, to obtain stable/reproducible join ordering, or to work around cases where CBO reordering produces suboptimal plans. This setting interacts with other CBO/session controls such as `cbo_max_reorder_node`, `cbo_max_reorder_node_use_exhaustive`, and `enable_outer_join_reorder`. * **Default**: `false` * **Data Type**: boolean * **Introduced in**: v3.2.0 ##### disable\_spill\_to\_local\_disk[​](#disable_spill_to_local_disk "Direct link to disable_spill_to_local_disk") * **Description**: When set to `true` for the session, FE will instruct BE to disable spilling to local disk and instead rely on remote storage spill (if remote spill is configured). This flag is only meaningful when `enable_spill` = `true`, `enable_spill_to_remote_storage` = `true`, and a valid `spill_storage_volume` is provided and found by FE. The value is serialized into TSpillToRemoteStorageOptions (sent to BE) as `disable_spill_to_local_disk`. If remote spill is not configured or the named storage volume cannot be resolved, this setting has no effect. Use with caution: disabling local-disk spill can increase network I/O and latency and requires reliable, performant remote storage. * **Scope**: Session * **Default**: false * **Data Type**: boolean * **Introduced in**: v3.3.0, v3.4.0, v3.5.0 ##### div\_precision\_increment[​](#div_precision_increment "Direct link to div_precision_increment") Used for MySQL client compatibility. No practical usage. ##### dynamic\_overwrite[​](#dynamic_overwrite "Direct link to dynamic_overwrite") * **Description**: Whether to enable the [Dynamic Overwrite](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md#dynamic-overwrite) semantic for INSERT OVERWRITE with partitioned tables. Valid values: * `true`: Enables Dynamic Overwrite. * `false`: Disables Dynamic Overwrite and uses the default semantic. * **Default**: false * **Introduced in**: v3.4.0 ##### enable\_adaptive\_sink\_dop[​](#enable_adaptive_sink_dop "Direct link to enable_adaptive_sink_dop") * **Description**: Specifies whether to enable adaptive parallelism for data loading. After this feature is enabled, the system automatically sets load parallelism for INSERT INTO and Broker Load jobs, which is equivalent to the mechanism of `pipeline_dop`. For a newly deployed v2.5 StarRocks cluster, the value is `true` by default. For a v2.5 cluster upgraded from v2.4, the value is `false`. * **Default**: false * **Introduced in**: v2.5 ##### enable\_bucket\_aware\_execution\_on\_lake[​](#enable_bucket_aware_execution_on_lake "Direct link to enable_bucket_aware_execution_on_lake") * **Description**: Whether to enable bucket-aware execution for queries against data lakes (such as Iceberg tables). When this feature is enabled, the system optimizes query execution by leveraging bucketing information to reduce data shuffling and improve performance. This optimization is particularly effective for join operations and aggregations on bucketed tables. * **Default**: true * **Data type**: Boolean * **Introduced in**: v4.0 ##### enable\_cbo\_based\_mv\_rewrite[​](#enable_cbo_based_mv_rewrite "Direct link to enable_cbo_based_mv_rewrite") * **Description**: Whether to enable materialized view rewrite in CBO phase which can maximize the likelihood of successful query rewriting (e.g., when the join order differs between materialized views and queries), but it will increase the execution time of the optimizer phase. * **Default**: true * **Introduced in**: v3.5.5, v4.0.1 ##### enable\_cbo\_table\_prune[​](#enable_cbo_table_prune "Direct link to enable_cbo_table_prune") * **Description**: When enabled, the optimizer will add the CBO table pruning rule (CboTablePruneRule) during memo optimization to perform cost-based table pruning for cardinality-preserving joins. The rule is conditionally added in the optimizer (see QueryOptimizer.memoOptimize and SPMOptimizer.memoOptimize) only when the join-node count in the join tree is small (fewer than 10 join nodes). This option complements the rule-based pruning toggle `enable_rbo_table_prune` and lets the Cost-Based Optimizer try to remove unnecessary tables or inputs from join processing to reduce planning and execution complexity. Default is off because pruning can change plan shape; enable it only after validating on representative workloads. * **Scope**: Session * **Default**: `false` * **Data Type**: boolean * **Introduced in**: v3.2.0 ##### enable\_cache\_udaf[​](#enable_cache_udaf "Direct link to enable_cache_udaf") * **Description**: When set to `true`, enables in-memory caching of the class-level Java UDAF initialization (class loading, method introspection, and batch-update stub generation). The cache is populated on first use and reused across all aggregator/analytor instances within the same BE process, eliminating the repeated per-instance initialization overhead that is otherwise proportional to pipeline DOP. Caching only applies to UDAFs and window functions that were created with `"isolation" = "shared"`. Functions created with `"isolation" = "private"` always go through the uncached path regardless of this setting. Default is `false`; enable after verifying that shared-isolation UDAFs are safe to share their class-level state across concurrent queries. The runtime profile exposes `UdafCacheHitCount`, `UdafCachePopulateCount`, and `UdafLoadTime` counters to observe cache behavior. * **Scope**: Session * **Default**: `false` * **Data Type**: boolean * **Introduced in**: v3.4.0 ##### enable\_color\_explain\_output[​](#enable_color_explain_output "Direct link to enable_color_explain_output") * **Scope**: Session * **Description**: Controls whether ANSI color escape sequences are included in textual EXPLAIN / PROFILE outputs. When enabled (`true`), StmtExecutor passes the session setting into the explain/profile pipeline (via calls to ExplainAnalyzer) so explain, EXPLAIN ANALYZE and analyze-profile outputs contain colored highlighting for readability in ANSI-capable terminals. When disabled (`false`), the output is produced without ANSI sequences (plain text), which is appropriate for logging, clients that do not support ANSI, or when piping output to files. This is a per-session toggle and does not change execution semantics—only the presentation of explain/profile text. * **Default**: `true` * **Data type**: boolean * **Introduced in**: v3.5.0 ##### enable\_connector\_adaptive\_io\_tasks[​](#enable_connector_adaptive_io_tasks "Direct link to enable_connector_adaptive_io_tasks") * **Description**: Whether to adaptively adjust the number of concurrent I/O tasks when querying external tables. Default value is `true`. If this feature is not enabled, you can manually set the number of concurrent I/O tasks using the variable `connector_io_tasks_per_scan_operator`. * **Default**: true * **Introduced in**: v2.5 ##### enable\_cost\_based\_multi\_stage\_agg[​](#enable_cost_based_multi_stage_agg "Direct link to enable_cost_based_multi_stage_agg") * **Description**: Controls whether the new planner uses cost-based decisions to generate and compare multi-stage aggregation plans for queries with DISTINCT aggregates. When enabled, the optimizer may produce alternative 3-stage and 4-stage aggregation candidates and rely on cost estimates to pick the better plan. It also enables post-processing in `PruneAggregateNodeRule` to merge or prune split aggregate nodes when beneficial (that is, reducing unnecessary serialization or deserialization). Note that the effective check in code is gated by `new_planner_agg_stage` — the helper `isEnableCostBasedMultiStageAgg()` returns true only when `new_planner_agg_stage` is set to `AUTO` and this parameter is set to `true`; if `new_planner_agg_stage` is non-`AUTO`, this parameter will not enable cost-based multi-stage behavior. Disabling this flag forces the planner to prefer the simpler 3-stage transformation for distinct aggregations and skips cost-driven candidate generation and certain aggregate-node merges. * **Scope**: Session * **Default**: `true` * **Data Type**: boolean * **Introduced in**: - ##### enable\_datacache\_async\_populate\_mode[​](#enable_datacache_async_populate_mode "Direct link to enable_datacache_async_populate_mode") * **Description**: Whether to populate the data cache in asynchronous mode. By default, the system uses the synchronous mode to populate data cache, that is, populating the cache while querying data. * **Default**: false * **Introduced in**: v3.2.7 ##### enable\_datacache\_io\_adaptor[​](#enable_datacache_io_adaptor "Direct link to enable_datacache_io_adaptor") * **Description**: Whether to enable the Data Cache I/O Adaptor. Setting this to `true` enables the feature. When this feature is enabled, the system automatically routes some cache requests to remote storage when the disk I/O load is high, reducing disk pressure. * **Default**: true * **Introduced in**: v3.3.0 ##### enable\_datacache\_sharing[​](#enable_datacache_sharing "Direct link to enable_datacache_sharing") * **Description**: Whether to enable Cache Sharing. Setting this to `true` enables the feature. Cache Sharing is used to support accessing cache data from other nodes through the network, which can help to reduce performance jitter caused by cache invalidation during cluster scaling. This variable takes effect only when the FE parameter `enable_trace_historical_node` is set to `true`. * **Default**: true * **Introduced in**: v3.5.1 ##### enable\_distinct\_agg\_over\_window[​](#enable_distinct_agg_over_window "Direct link to enable_distinct_agg_over_window") * **Description**: Controls the optimizer rewrite that transforms DISTINCT aggregate calls over WINDOW clauses into an equivalent join-based plan. When enabled (`true`, the default) QueryOptimizer.invoke convertDistinctAggOverWindowToNullSafeEqualJoin will: * detect queries containing a LogicalWindowOperator, * run project-merge rewrites, derive logical properties, * apply DistinctAggregationOverWindowRule to convert the DISTINCT-OVER-WINDOW pattern into a null-safe equality join (changing plan shape to enable further push-downs and aggregation optimizations), * then run SeparateProjectRule and re-derive properties. When disabled (`false`) the optimizer skips this transformation and leaves DISTINCT aggregates over windows unchanged. This setting is session-scoped and affects only the optimizer rewrite phase (see QueryOptimizer.convertDistinctAggOverWindowToNullSafeEqualJoin). * **Default**: `true` * **Data Type**: boolean * **Introduced in**: - ##### enable\_distinct\_column\_bucketization[​](#enable_distinct_column_bucketization "Direct link to enable_distinct_column_bucketization") * **Description**: Whether to enable bucketization for the COUNT DISTINCT colum in a group-by-count-distinct query. Use the `select a, count(distinct b) from t group by a;` query as an example. If the GROUP BY colum `a` is a low-cardinality column and the COUNT DISTINCT column `b` is a high-cardinality column which has severe data skew, performance bottleneck will occur. In this situation, you can split data in the COUNT DISTINCT column into multiple buckets to balance data and prevent data skew. You must use this variable with the variable `count_distinct_column_buckets`. You can also enable bucketization for the COUNT DISTINCT column by adding the `skew` hint to your query, for example, `select a,count(distinct [skew] b) from t group by a;`. * **Default**: false, which means this feature is disabled. * **Introduced in**: v2.5 ##### enable\_eliminate\_agg[​](#enable_eliminate_agg "Direct link to enable_eliminate_agg") * **Description**: Controls optimizer transformations that remove or simplify aggregation operators when it is safe to do so. When enabled, the planner applies rules (EliminateAggRule and EliminateAggFunctionRule) to replace a LogicalAggregationOperator with a LogicalProjectOperator (and optionally a LogicalFilterOperator) in two cases: * Whole-aggregation elimination (EliminateAggRule): when grouping keys form a unique key on the child (unique/UKFK constraints) and all aggregate calls are supported, non-distinct functions (SUM, COUNT, AVG, FIRST\_VALUE, MAX, MIN, GROUP\_CONCAT). COUNT is rewritten to an IF/CAST expression (COUNT(col) -> IF(col IS NULL, 0, 1); COUNT(\*) -> 1). * Per-function elimination (EliminateAggFunctionRule): when individual non-distinct aggregate functions over a grouped column (FIRST\_VALUE, LAST\_VALUE, ANY\_VALUE, MAX, MIN) can be replaced by the column itself while preserving other aggregations. The optimization requires non-empty group-by keys, supported function sets, and presence of relevant unique constraints or column relationships; it does not apply to DISTINCT aggregates. * **Scope**: Session * **Default**: `true` * **Data Type**: boolean * **Introduced in**: v3.3.8, v3.4.0, v3.5.0 ##### enable\_filter\_unused\_columns\_in\_scan\_stage[​](#enable_filter_unused_columns_in_scan_stage "Direct link to enable_filter_unused_columns_in_scan_stage") * **Description**: Controls pruning of columns produced by Scan nodes so the scan stage only outputs columns that are actually needed downstream (either as outputs or for non-pushable predicates). When enabled, PlanFragmentBuilder.setUnUsedOutputColumns will mark scan output columns that are exclusively used in pushdownable predicates and not required later, allowing the scan to trim those columns and reduce I/O and network transfer. The pruning is guarded: it will not apply for aggregation-family indexes in the non-skip-aggregation (non-skip-aggr) scan stage (keys/value columns must be retained to merge/aggregate), and the planner always ensures at least one column is returned from a scan. See `isEnableFilterUnusedColumnsInScanStage()` and the enable/disable helpers `enableTrimOnlyFilteredColumnsInScanStage()` / `disableTrimOnlyFilteredColumnsInScanStage()` in SessionVariable. * **Scope**: Session * **Default**: `true` * **Data Type**: boolean * **Introduced in**: v3.2.0 ##### enable\_force\_rule\_based\_mv\_rewrite[​](#enable_force_rule_based_mv_rewrite "Direct link to enable_force_rule_based_mv_rewrite") * **Description**: Whether to enable query rewrite for queries against multiple tables in the optimizer's rule-based optimization phase. Enabling this feature will improve the robustness of the query rewrite. However, it will also increase the time consumption if the query misses the materialized view. * **Default**: true * **Introduced in**: v3.3.0 ##### enable\_gin\_filter[​](#enable_gin_filter "Direct link to enable_gin_filter") * **Description**: Whether to utilize the [fulltext inverted index](https://docs.starrocks.io/docs/table_design/indexes/inverted_index.md) during queries. * **Default**: true * **Introduced in**: v3.3.0 ##### enable\_global\_runtime\_filter[​](#enable_global_runtime_filter "Direct link to enable_global_runtime_filter") Whether to enable global runtime filter (RF for short). RF filters data at runtime. Data filtering often occurs in the Join stage. During multi-table joins, optimizations such as predicate pushdown are used to filter data, in order to reduce the number of scanned rows for Join and the I/O in the Shuffle stage, thereby speeding up the query. StarRocks offers two types of RF: Local RF and Global RF. Local RF is suitable for Broadcast Hash Join and Global RF is suitable for Shuffle Join. Default value: `true`, which means global RF is enabled. If this feature is disabled, global RF does not take effect. Local RF can still work. ##### enable\_group\_by\_compressed\_key[​](#enable_group_by_compressed_key "Direct link to enable_group_by_compressed_key") * **Description**: Whether to use accurate statistical information to compress the GROUP BY Key column. Valid values: `true` and `false`. * **Default**: true * **Introduced in**: v4.0 ##### enable\_group\_execution[​](#enable_group_execution "Direct link to enable_group_execution") * **Description**: Whether to enable Colocate Group Execution. Colocate Group Execution is an execution pattern that leverages physical data partitioning, where a fixed number of threads sequentially process their respective data ranges to enhance locality and throughput. Enabling this feature can reduce memory usage. * **Default**: true * **Introduced in**: v3.3 ##### enable\_group\_level\_query\_queue (global)[​](#enable_group_level_query_queue-global "Direct link to enable_group_level_query_queue (global)") * **Description**: Whether to enable resource group-level [query queue](https://docs.starrocks.io/docs/administration/management/resource_management/query_queues.md). * **Default**: false, which means this feature is disabled. * **Introduced in**: v3.1.4 ##### enable\_incremental\_mv[​](#enable_incremental_mv "Direct link to enable_incremental_mv") * **Description**: Session flag that controls whether the server will plan and keep an in-memory plan for materialized views that use incremental refresh. When enabled, `MaterializedViewAnalyzer.planMVQuery` will proceed for create-MV statements whose refresh scheme is an `IncrementalRefreshSchemeDesc`: it builds the logical and physical plan for the view query and sets the session `enableMVPlanner` flag (`setMVPlanner(true)`). When disabled, planning for incremental-refresh MVs is skipped. Accessible via `isEnableIncrementalRefreshMV()` and `setEnableIncrementalRefreshMv(boolean)` in `SessionVariable`. * **Scope**: Session (per-connection) * **Default**: `false` * **Data Type**: boolean * **Introduced in**: v3.2.0 ##### enable\_insert\_partial\_update[​](#enable_insert_partial_update "Direct link to enable_insert_partial_update") * **Description**: Whether to enable Partial Update for INSERT statements on Primary Key tables. When this item is set to `true` (default), if an INSERT statement specifies only a subset of columns (fewer than the number of all non-generated columns in the table), the system performs a Partial Update to update only the specified columns while preserving existing values in other columns. When set to `false`, the system uses default values for unspecified columns instead of preserving existing values. This feature is particularly useful for updating specific columns in Primary Key tables without affecting other column values. * **Default**: true * **Introduced in**: v3.3.20, v3.4.9, v3.5.8, v4.0.2 ##### enable\_insert\_strict[​](#enable_insert_strict "Direct link to enable_insert_strict") * **Description**: Whether to enable strict mode while loading data using INSERT from files(). Valid values: `true` and `false` (Default). When strict mode is enabled, the system loads only qualified rows. It filters out unqualified rows and returns details about the unqualified rows. For more information, see [Strict mode](https://docs.starrocks.io/docs/loading/strict_mode.md). In versions earlier than v3.4.0, when `enable_insert_strict` is set to `true`, the INSERT jobs fails when there is an unqualified rows. * **Default**: true ##### enable\_lake\_tablet\_internal\_parallel[​](#enable_lake_tablet_internal_parallel "Direct link to enable_lake_tablet_internal_parallel") * **Description**: Whether to enable Parallel Scan for Cloud-native tables in a shared-data cluster. * **Default**: true * **Data type**: Boolean * **Introduced in**: v3.3.0 ##### enable\_lambda\_pushdown[​](#enable_lambda_pushdown "Direct link to enable_lambda_pushdown") * **Description**: Session-scoped boolean toggle that controls predicate pushdown behavior in the optimizer. Specifically, the `PushDownPredicateProjectRule` consults this flag: when `true` (default) the rule may push predicates through `Project` operators even if those projects contain `LambdaFunctionOperator` expressions; when `false` the rule inspects the project's expressions and aborts the pushdown if any lambda is present (the rule returns no transformation). This affects only the optimizer transformation phase (planning) and can be changed per session via the `SessionVariable` getter/setter (`getEnableLambdaPushDown` / `setEnableLambdaPushdown`). * **Scope**: Session * **Default**: `true` * **Data Type**: boolean * **Introduced in**: v3.3.6, v3.4.0, v3.5.0 ##### enable\_large\_in\_predicate[​](#enable_large_in_predicate "Direct link to enable_large_in_predicate") * **Scope**: Session * **Description**: When enabled, the parser will convert IN-lists whose literal count meets or exceeds `large_in_predicate_threshold` into a special `LargeInPredicate` (handled in `AstBuilder`). The optimizer rule `LargeInPredicateToJoinRule` then converts that predicate into a `LEFT_SEMI_JOIN` (for IN) or `NULL_AWARE_LEFT_ANTI_JOIN` (for NOT IN) against a `RawValues` constant table, reducing FE memory and planning cost for very large IN lists by avoiding one expression node per constant. The transformation has correctness restrictions (no OR compound predicates, only one large-IN per query); if these restrictions or other conditions cause the optimization to fail, the planner throws `LargeInPredicateException` and upper layers (via `StmtExecutor` / `ConnectProcessor`) retry the query from the parser stage with `enable_large_in_predicate` disabled so the query falls back to the traditional expression-based IN handling. Use `large_in_predicate_threshold` to control the minimum literal count that triggers this behavior. * **Default**: `true` * **Data Type**: boolean * **Introduced in**: - ##### max\_unknown\_string\_meta\_length (global)[​](#max_unknown_string_meta_length-global "Direct link to max_unknown_string_meta_length (global)") * **Description**: Fallback length for string columns in query result metadata when the max length is unknown. Clients that rely on the metadata may return empty values or truncation if the reported length is smaller than actual values. Valid range is `1` to `1048576`. * **Default**: 64 * **Data Type**: int * **Introduced in**: v3.5.16, v4.0.9, v4.1 ##### enable\_reduce\_cast\_varchar\_length\_inheritance (global)[​](#enable_reduce_cast_varchar_length_inheritance-global "Direct link to enable_reduce_cast_varchar_length_inheritance (global)") * **Description**: Whether to preserve the target `VARCHAR(N)` length when `ReduceCastRule` eliminates a same-type `VARCHAR -> VARCHAR` cast. Enable this variable to keep prepare and execute result-set metadata consistent for statements such as `CAST(col AS VARCHAR(N))`. * **Default**: false * **Data Type**: Boolean * **Introduced in**: v3.5.16, v4.0.9, v4.1 ##### enable\_reduce\_cast\_varchar\_expr\_sync\_type (global)[​](#enable_reduce_cast_varchar_expr_sync_type-global "Direct link to enable_reduce_cast_varchar_expr_sync_type (global)") * **Description**: Whether to synchronize the reused planner `Expr` type and origin type with the rewritten `VARCHAR(N)` type after `ReduceCastRule` eliminates a same-type `VARCHAR -> VARCHAR` cast. * **Default**: true * **Data Type**: Boolean * **Introduced in**: v3.5.16, v4.0.9, v4.1 ##### enable\_load\_profile[​](#enable_load_profile "Direct link to enable_load_profile") * **Scope**: Session * **Description**: When enabled, the FE requests collection of the runtime profile for load jobs and the load coordinator will collect/export the profile after a load completes. For stream load, FE sets `TQueryOptions.enable_profile = true` and passes `load_profile_collect_second` (from `stream_load_profile_collect_threshold_second`) to backends; the coordinator then conditionally calls profile collection (see StreamLoadTask.collectProfile()). The effective behavior is the logical OR of this session variable and the table-level property `enable_load_profile` on the destination table; collection is further gated by `load_profile_collect_interval_second` (FE-side sampling interval) to avoid frequent collection. The session flag is read via `SessionVariable.isEnableLoadProfile()` and can be set per-connection with `setEnableLoadProfile(...)`. * **Default**: `false` * **Data Type**: boolean * **Introduced in**: v3.2.0 ##### enable\_local\_shuffle\_agg[​](#enable_local_shuffle_agg "Direct link to enable_local_shuffle_agg") * **Description**: Controls whether the planner and cost model may produce a one-phase local aggregation plan that uses a local shuffle (Scan -> LocalShuffle -> OnePhaseAgg) instead of a two-phase/global-shuffle aggregation. When enabled (the default), the optimizer and cost model will: * allow replacing a SHUFFLE exchange between Scan and Global Agg with a local shuffle + one-phase agg on single-backend-and-compute-node clusters (see `PruneShuffleDistributionNodeRule` and `EnforceAndCostTask`), * let the cost model ignore network cost for SHUFFLE in that single-node case to favor the one-phase plan (`CostModel`). The replacement is only considered when `enable_pipeline_engine` is enabled and the cluster is a single backend+compute node. The planner still rejects local-shuffle conversion in unsafe cases (e.g., DISTINCT aggregates, detected data skew, missing/unknown column statistics, multi-input operators like joins, or other semantic restrictions). Some code paths (INSERT/UPDATE/DELETE planners and MaterializedViewOptimizer) temporarily disable this session flag because non-query sinks or certain rewrites require per-driver scan assignment that local-shuffle cannot use. * **Scope**: Session * **Default**: `true` * **Data Type**: boolean * **Introduced in**: v3.2.0 ##### enable\_materialized\_view\_agg\_pushdown\_rewrite[​](#enable_materialized_view_agg_pushdown_rewrite "Direct link to enable_materialized_view_agg_pushdown_rewrite") * **Description**: Whether to enable aggregation pushdown for materialized view query rewrite. If it is set to `true`, aggregate functions will be pushed down to Scan Operator during query execution and rewritten by the materialized view before the Join Operator is executed. This will relieve the data expansion caused by Join and thereby improve the query performance. For detailed information about the scenarios and limitations of this feature, see [Aggregation pushdown](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md#aggregation-pushdown). * **Default**: false * **Introduced in**: v3.3.0 ##### enable\_materialized\_view\_for\_insert[​](#enable_materialized_view_for_insert "Direct link to enable_materialized_view_for_insert") * **Description**: Whether to allow StarRocks to rewrite queries in INSERT INTO SELECT statements. * **Default**: false, which means Query Rewrite in such scenarios is disabled by default. * **Introduced in**: v2.5.18, v3.0.9, v3.1.7, v3.2.2 ##### enable\_materialized\_view\_text\_match\_rewrite[​](#enable_materialized_view_text_match_rewrite "Direct link to enable_materialized_view_text_match_rewrite") * **Description**: Whether to enable text-based materialized view rewrite. When this item is set to true, the optimizer will compare the query with the existing materialized views. A query will be rewritten if the abstract syntax tree of the materialized view's definition matches that of the query or its sub-query. * **Default**: true * **Introduced in**: v3.2.5, v3.3.0 ##### enable\_materialized\_view\_union\_rewrite[​](#enable_materialized_view_union_rewrite "Direct link to enable_materialized_view_union_rewrite") * **Description**: Whether to enable materialized view union rewrite. If this item is set to `true`, the system seeks to compensate the predicates using UNION ALL when the predicates in the materialized view cannot satisfy the query's predicates. * **Default**: true * **Introduced in**: v2.5.20, v3.1.9, v3.2.7, v3.3.0 ##### enable\_metadata\_profile[​](#enable_metadata_profile "Direct link to enable_metadata_profile") * **Description**: Whether to enabled Profile for Iceberg Catalog metadata. * **Default**: true * **Introduced in**: v3.3.3 ##### enable\_multicolumn\_global\_runtime\_filter[​](#enable_multicolumn_global_runtime_filter "Direct link to enable_multicolumn_global_runtime_filter") Whether to enable multi-column global runtime filter. Default value: `false`, which means multi-column global RF is disabled. If a Join (other than Broadcast Join and Replicated Join) has multiple equi-join conditions: * If this feature is disabled, only Local RF works. * If this feature is enabled, multi-column Global RF takes effect and carries `multi-column` in the partition by clause. ##### enable\_mv\_planner[​](#enable_mv_planner "Direct link to enable_mv_planner") * **Scope**: Session * **Description**: When enabled, activates the Materialized View (MV) planner mode for the current session. In this mode the optimizer: * Uses MV-specific rule set via `context.getRuleSet().addRealtimeMVRules()` instead of the regular join implementation rules (QueryOptimizer). * Allows stream implementation rules to apply (see `StreamImplementationRule.check` which returns true only when MV planner is on). * Alters scan/operator construction during logical plan transformation (e.g., RelationTransformer chooses `LogicalBinlogScanOperator` for native tables/materialized views when MV planner is enabled). * Disables or bypasses some standard transformations (for example, `SplitMultiPhaseAggRule.check` returns false when MV planner is on). Materialized view planning code (MaterializedViewAnalyzer) sets this flag around MV planning work (sets to true before planning and resets to false afterward), so it is primarily intended for MV plan generation and testing. Setting this session variable affects only the current session’s optimizer behavior. * **Default**: `false` * **Data Type**: boolean * **Introduced in**: v3.2.0 ##### enable\_optimize\_skew\_join\_v1[​](#enable_optimize_skew_join_v1 "Direct link to enable_optimize_skew_join_v1") * **Scope**: Session * **Description**: Controls which skew-join optimization strategy the optimizer uses. When set to `true`, the optimizer enables the query-rewrite based skew join optimization: QueryOptimizer checks `sessionVariable.isEnableOptimizerSkewJoinByQueryRewrite()` after join-expression pushdown and, if enabled and `enableOptimizerSkewJoinByBroadCastSkewValues` is disabled, invokes `skewJoinOptimize(...)` which applies `SkewJoinOptimizeRule`. If `isEnableStatsToOptimizeSkewJoin` is enabled, `skewJoinOptimize` first merges projects and computes statistics (`Utils.calculateStatistics`) before applying the rule. The session setters enforce mutual exclusivity between `enableOptimizerSkewJoinByQueryRewrite` and `enableOptimizerSkewJoinByBroadCastSkewValues` (setting one flips the other), so only one skew strategy is active at a time. * **Default**: `true` * **Data Type**: boolean * **Introduced in**: - ##### enable\_optimize\_skew\_join\_v2[​](#enable_optimize_skew_join_v2 "Direct link to enable_optimize_skew_join_v2") * **Description**: When enabled, the optimizer uses the broadcasted-skew-values strategy (Skew Join v2) to handle skewed joins. In the optimizer this flag activates the SkewShuffleJoinEliminationRule during dynamic rewrite and disables the query-rewrite based skew-join path (skewJoinOptimize). The two skew-join strategies are mutually exclusive: enabling this variable sets `enable_optimize_skew_join_v1` off and vice versa. This is a session-level variable intended to switch optimizer behavior for queries that benefit from using broadcasted skew statistics instead of query-rewrite transformations. Usage locations: `QueryOptimizer.dynamicRewrite(...)` and the main optimization flow in `QueryOptimizer` where skew-join optimization is applied. * **Scope**: Session * **Default**: `false` * **Data Type**: boolean * **Introduced in**: - ##### enable\_parallel\_merge[​](#enable_parallel_merge "Direct link to enable_parallel_merge") * **Description**: Whether to enable parallel merge for sorting. When this feature is enabled, the merge phase of sorting will utilize multiple threads for merge operations. * **Default**: true * **Introduced in**: v3.3 ##### enable\_parquet\_reader\_bloom\_filter[​](#enable_parquet_reader_bloom_filter "Direct link to enable_parquet_reader_bloom_filter") * **Default**: true * **Type**: Boolean * **Unit**: - * **Description**: Whether to enable Bloom Filter optimization when reading Parquet files. * `true` (Default): Enable Bloom Filter optimization when reading Parquet files. * `false`: Disable Bloom Filter optimization when reading Parquet files. * **Introduced in**: v3.5.0 ##### enable\_parquet\_reader\_page\_index[​](#enable_parquet_reader_page_index "Direct link to enable_parquet_reader_page_index") * **Default**: true * **Type**: Boolean * **Unit**: - * **Description**: Whether to enable Page Index optimization when reading Parquet files. * `true` (Default): Enable Page Index optimization when reading Parquet files. * `false`: Disable Page Index optimization when reading Parquet files. * **Introduced in**: v3.5.0 ##### enable\_partition\_hash\_join[​](#enable_partition_hash_join "Direct link to enable_partition_hash_join") * **Description**: Whether to enable adaptive Partition Hash Join. * **Default**: true * **Introduced in**: v3.4 ##### enable\_per\_bucket\_optimize[​](#enable_per_bucket_optimize "Direct link to enable_per_bucket_optimize") * **Description**: Whether to enable bucketed computation. When this feature is enabled, stage-one aggregation can be computed in bucketed order, reducing memory usage. * **Default**: true * **Introduced in**: v3.0 ##### enable\_phased\_scheduler[​](#enable_phased_scheduler "Direct link to enable_phased_scheduler") * **Description**: Whether to enable multi-phased scheduling. When multi-phased scheduling is enabled, it will schedule fragments according to their dependencies. For example, the system will first schedule the fragment on the build side of a Shuffle Join, and then the fragment on the probe side (Note that, unlike stage-by-stage scheduling, phased scheduling is still under the MPP execution mode). Enabling multi-phased scheduling can significantly reduce memory usage for a large number of UNION ALL queries. * **Default**: false * **Introduced in**: v3.3 ##### enable\_pipeline\_engine[​](#enable_pipeline_engine "Direct link to enable_pipeline_engine") * **Description**: Specifies whether to enable the pipeline execution engine. `true` indicates enabled and `false` indicates the opposite. Default value: `true`. * **Default**: true ##### enable\_plan\_advisor[​](#enable_plan_advisor "Direct link to enable_plan_advisor") * **Description**: Whether to enable Query Feedback feature for slow queries and manually marked queries. * **Default**: true * **Introduced in**: v3.4.0 ##### enable\_predicate\_reorder[​](#enable_predicate_reorder "Direct link to enable_predicate_reorder") * **Scope**: Session * **Description**: When enabled, the optimizer applies the Predicate Reorder rule to AND (conjunctive) predicates during logical/physical plan rewrite. The rule extracts conjuncts via `Utils.extractConjuncts`, estimates each conjunct's selectivity with `DefaultPredicateSelectivityEstimator`, and reorders the conjuncts in ascending order of estimated selectivity (less restrictive first) to form a new `CompoundPredicateOperator` (AND). The rule only runs when the operator has a `CompoundPredicateOperator` with more than one conjunct. Statistics are gathered from child `OptExpression` statistics when available; for `PhysicalOlapScanOperator` it will fetch column statistics from `GlobalStateMgr.getCurrentState().getStatisticStorage()`. If child statistics are missing and the scan is not an OLAP scan, the rule skips reordering. The session variable is exposed via `SessionVariable.isEnablePredicateReorder()`, with `enablePredicateReorder()` and `disablePredicateReorder()` helper methods. * **Default**: false * **Data type**: boolean * **Introduced in**: v3.2.0 ##### enable\_profile[​](#enable_profile "Direct link to enable_profile") * **Description**: Specifies whether to send the profile of a query for analysis. The default value is `false`, which means no profile is required. By default, a profile is sent to the FE only when a query error occurs in the BE. Profile sending causes network overhead and therefore affects high concurrency. If you need to analyze the profile of a query, you can set this variable to `true`. After the query is completed, the profile can be viewed on the web page of the currently connected FE (address: `fe_host:fe_http_port/query`). This page displays the profiles of the latest 100 queries with `enable_profile` turned on. * **Default**: false ##### enable\_query\_cache[​](#enable_query_cache "Direct link to enable_query_cache") * **Description**: Specifies whether to enable the Query Cache feature. Valid values: true and false. `true` specifies to enable this feature, and `false` specifies to disable this feature. When this feature is enabled, it works only for queries that meet the conditions specified in the application scenarios of [Query Cache](https://docs.starrocks.io/docs/using_starrocks/caching/query_cache.md#application-scenarios). * **Default**: false * **Introduced in**: v2.5 ##### enable\_query\_dump[​](#enable_query_dump "Direct link to enable_query_dump") * **Description**: Controls per-session query dumping. When this variable is set to true and HTTP Query Dump mode is not active, the server will collect and persist the session's Dump informatation. For non-HTTP queries, the system will serialize the session Dump information and write it to the Query Dump Log when an exception occurs. For HTTP-triggered dumps, the system uses a separate path (to add exception stack traces into the connection's Dump information). Use this variable to opt a session into FE-side query dump collection for post-mortem analysis and debugging; it is evaluated at runtime per session and does not affect global behavior. * **Default**: false * **Data Type**: boolean * **Introduced in**: v3.2.0 ##### enable\_query\_queue\_load (global)[​](#enable_query_queue_load-global "Direct link to enable_query_queue_load (global)") * **Description**: Boolean value to enable query queues for loading tasks. * **Default**: false ##### enable\_query\_queue\_select (global)[​](#enable_query_queue_select-global "Direct link to enable_query_queue_select (global)") * **Description**: Whether to enable query queues for SELECT queries. * **Default**: false ##### enable\_query\_queue\_statistic (global)[​](#enable_query_queue_statistic-global "Direct link to enable_query_queue_statistic (global)") * **Description**: Whether to enable query queues for statistics queries. * **Default**: false ##### enable\_query\_tablet\_affinity[​](#enable_query_tablet_affinity "Direct link to enable_query_tablet_affinity") * **Description**: Boolean value to control whether to direct multiple queries against the same tablet to a fixed replica. In scenarios where the table to query has a large number of tablets, this feature significantly improves query performance because the meta information and data of the tablet can be cached in memory more quickly. However, if there are some hotspot tablets, this feature may degrade the query performance because it directs the queries to the same BE, making it unable to fully use the resources of multiple BEs in high concurrency scenarios. * **Default**: false, which means the system selects a replica for each query. * **Introduced in**: v2.5.6, v3.0.8, v3.1.4, and v3.2.0. ##### enable\_query\_trigger\_analyze[​](#enable_query_trigger_analyze "Direct link to enable_query_trigger_analyze") * **Default**: true * **Type**: Boolean * **Description**: Whether to enable query-trigger ANALYZE tasks on tables under external catalogs. * **Introduced in**: v3.4.0 ##### enable\_rbo\_table\_prune[​](#enable_rbo_table_prune "Direct link to enable_rbo_table_prune") * **Description**: When enabled, the optimizer applies Rule-Based (RBO) table pruning for cardinality-preserving joins in the current session. The optimizer runs a sequence of rewrite and pruning steps (partition pruning, project merge/separate, `UniquenessBasedTablePruneRule`, join reorder, and `RboTablePruneRule`) to remove unnecessary table scan alternatives and reduce scanned partitions/rows in joins. Enabling this option also disables join-equivalence derivation (`context.setEnableJoinEquivalenceDerive(false)`) while logical rule rewrite is running to avoid conflicting transformations. The pruning flow may additionally run `PrimaryKeyUpdateTableRule` for update statements if `enable_table_prune_on_update` is set. The rule is only executed when a query contains prunable joins (checked via `Utils.hasPrunableJoin(tree)`). * **Scope**: Session * **Default**: `false` * **Data Type**: boolean * **Introduced in**: v3.2.0 ##### enable\_rewrite\_groupingsets\_to\_union\_all[​](#enable_rewrite_groupingsets_to_union_all "Direct link to enable_rewrite_groupingsets_to_union_all") * **Scope**: Session * **Description**: When enabled, the optimizer applies the RewriteGroupingSetsByCTERule to transform SQL GROUPING SETS (including ROLLUP and CUBE semantics) into equivalent plans expressed as multiple aggregation branches combined with UNION ALL (implemented via CTEs). The system will conditionally run the iterative rewrite pass. This rewrite can improve compatibility with existing aggregation planning rules and enable further rule-based optimizations, but it typically expands the plan into multiple aggregation/union branches which may increase intermediate rows, memory usage, and planning time. If `cbo_push_down_grouping_set` is also set to `true`, the optimizer may additionally attempt push-down of grouping-set aggregates after or instead of this rewrite. * **Default**: `false` * **Data Type**: boolean * **Introduced in**: v3.2.0 ##### enable\_runtime\_adaptive\_dop[​](#enable_runtime_adaptive_dop "Direct link to enable_runtime_adaptive_dop") * **Scope**: Session * **Description**: When enabled for a session, the planner and fragment builder will mark pipeline-capable fragments that support runtime adaptive DOP to use adaptive degree-of-parallelism at runtime. This option only takes effect when `enable_pipeline_engine` is true. Enabling it causes fragments to call `enableAdaptiveDop()` during plan construction, and has runtime implications: join probes may wait for all build phases to complete (which conflicts with `group_execution` behavior), and enabling runtime adaptive DOP will disable pipeline-level multi-partitioned runtime filters (the setter clears `enablePipelineLevelMultiPartitionedRf`). The flag is recorded in the query profile and can be toggled per-session. * **Default**: `false` * **Data type**: boolean * **Introduced in**: v3.2.0 ##### enable\_scan\_datacache[​](#enable_scan_datacache "Direct link to enable_scan_datacache") * **Description**: Specifies whether to enable the Data Cache feature. After this feature is enabled, StarRocks caches hot data read from external storage systems into blocks, which accelerates queries and analysis. For more information, see [Data Cache](https://docs.starrocks.io/docs/data_source/data_cache.md). In versions prior to 3.2, this variable was named as `enable_scan_block_cache`. * **Default**: true * **Introduced in**: v2.5 ##### enable\_shared\_scan[​](#enable_shared_scan "Direct link to enable_shared_scan") * **Scope**: Session * **Description**: Session-level boolean flag intended to request shared scan execution for pipeline queries. When pipeline execution is enabled, the FE will propagate this setting into the fragment execution parameters (`TExecPlanFragmentParams.enable_shared_scan`) so the BE can perform shared scanning (see `.../TFragmentInstanceFactory.java:153-159`). However, the FE currently does not honor user changes: `SessionVariable.isEnableSharedScan()` always returns `false` (see `.../SessionVariable.java:4176-4180`) and the feature has been disabled in FE since later versions due to incompatibility with event-based scheduling. As a result, setting this variable in a session has no effect in current releases. * **Default**: `false` * **Data Type**: boolean * **Introduced in**: v3.2.0 ##### enable\_short\_circuit[​](#enable_short_circuit "Direct link to enable_short_circuit") * **Description**: Whether to enable short circuiting for queries. Default: `false`. If it is set to `true`, when the query meets the criteria (to evaluate whether the query is a point query): the conditional columns in the WHERE clause include all primary key columns, and the operators in the WHERE clause are `=` or `IN`, the query takes the short circuit. * **Default**: false * **Introduced in**: v3.2.3 ##### enable\_sort\_aggregate[​](#enable_sort_aggregate "Direct link to enable_sort_aggregate") * **Description**: Specifies whether to enable sorted streaming. `true` indicates sorted streaming is enabled to sort data in data streams. * **Default**: false * **Introduced in**: v2.5 ##### enable\_spill[​](#enable_spill "Direct link to enable_spill") * **Description**: Whether to enable intermediate result spilling. Default: `false`. If it is set to `true`, StarRocks spills the intermediate results to disk to reduce the memory usage when processing aggregate, sort, or join operators in queries. * **Default**: false * **Introduced in**: v3.0 ##### enable\_spill\_to\_remote\_storage[​](#enable_spill_to_remote_storage "Direct link to enable_spill_to_remote_storage") * **Description**: Whether to enable intermediate result spilling to object storage. If it is set to `true`, StarRocks spills the intermediate results to the storage volume specified in `spill_storage_volume` after the capacity limit of the local disk is reached. For more information, see [Spill to object storage](https://docs.starrocks.io/docs/administration/management/resource_management/spill_to_disk.md#preview-spill-intermediate-result-to-object-storage). * **Default**: false * **Introduced in**: v3.3.0 ##### enable\_split\_topn\_agg[​](#enable_split_topn_agg "Direct link to enable_split_topn_agg") * **Description**: Controls whether the optimizer may apply the SplitTopNAggregate transformation (implemented in `SplitTopNAggregateRule.java`). When enabled, the optimizer can rewrite a plan that has a TopN on top of an aggregation over an OLAP scan into a plan that: * computes a restricted TopN-aggregation on a right-side partial scan (push-down), * joins the TopN result back to the full scan, * and finalizes aggregation on the join result. This rewrite is intended to reduce work when TopN orders by aggregated columns and the limit is small. The rule is gated by several precise checks: * TopN limit is set and less than or equal to `split_topn_agg_limit` (session `splitTopNAggLimit`, default 10000). * scan/agg projections are identity (no column remapping). * statistics/row-count heuristics (skip when outputRowCount < limit \* 10 and stats are reliable, unless running unit tests). * duplicated columns to be read twice ≤ 3. * duplicated columns are not long strings or complex types (string with averageRowSize ≥ 5 or missing stats is treated as long). * predicate complexity limits (≤ 2 conjuncts and ≤ 2 disjuncts). * only a subset (not zero or all) of aggregations are referenced by ORDER BY. * **Scope**: Session * **Default**: `true` * **Data Type**: boolean * **Introduced in**: - ##### enable\_split\_topn\_agg\_limit[​](#enable_split_topn_agg_limit "Direct link to enable_split_topn_agg_limit") * **Description**: Session-level threshold (row count) that controls whether the SplitTopN aggregate optimization may be applied. When `enable_split_topn_agg` is on, the optimizer's SplitTopNAggregateRule will skip the split transformation if the TopN operator's LIMIT is equal to the default unlimited value or greater than this threshold. This variable only governs the numeric cutoff (number of rows); the rule still enforces other correctness and cost checks (projections, predicates, column/type constraints and scan statistics) before applying the transformation. * **Scope**: Session * **Default**: `10000` * **Data Type**: long * **Introduced in**: - ##### enable\_spm\_rewrite[​](#enable_spm_rewrite "Direct link to enable_spm_rewrite") * **Description**: Whether to enable SQL Plan Manager (SPM) query rewrite. When enabled, StarRocks automatically rewrites queries to use bound query plans, improving query performance and stability. * **Default**: false ##### enable\_strict\_order\_by[​](#enable_strict_order_by "Direct link to enable_strict_order_by") * **Description**: Used to check whether the column name referenced in ORDER BY is ambiguous. When this variable is set to the default value `TRUE`, an error is reported for such a query pattern: Duplicate alias is used in different expressions of the query and this alias is also a sorting field in ORDER BY, for example, `select distinct t1.* from tbl1 t1 order by t1.k1;`. The logic is the same as that in v2.3 and earlier. When this variable is set to `FALSE`, a loose deduplication mechanism is used, which processes such queries as valid SQL queries. * **Default**: true * **Introduced in**: v2.5.18 and v3.1.7 ##### enable\_sync\_materialized\_view\_rewrite[​](#enable_sync_materialized_view_rewrite "Direct link to enable_sync_materialized_view_rewrite") * **Description**: Whether to enable query rewrite based on synchronous materialized views. * **Default**: true * **Introduced in**: v3.1.11, v3.2.5 ##### enable\_table\_prune\_on\_update[​](#enable_table_prune_on_update "Direct link to enable_table_prune_on_update") * **Description**: Session-level boolean that controls whether the optimizer applies primary-key-specific table pruning rules for UPDATE statements. When enabled, QueryOptimizer (during the pruneTables stage) invokes `PrimaryKeyUpdateTableRule` to rewrite/prune update plans — potentially improving pruning for primary-key update patterns. This flag is only effective when rule-based/CBO table pruning is active (see `enable_rbo_table_prune`). It is disabled by default because the transformation can change data-layout/plan shape (e.g., bucket-shuffle layout for OlapTableSink) and may cause correctness or performance regressions for concurrent updates. * **Default**: `false` * **Data type**: boolean * **Introduced in**: v3.2.4 ##### enable\_tablet\_internal\_parallel[​](#enable_tablet_internal_parallel "Direct link to enable_tablet_internal_parallel") * **Description**: Whether to enable adaptive parallel scanning of tablets. After this feature is enabled, multiple threads can be used to scan one tablet by segment, increasing the scan concurrency. * **Default**: true * **Introduced in**: v2.3 ##### enable\_tablet\_pre\_split[​](#enable_tablet_pre_split "Direct link to enable_tablet_pre_split") * **Description**: Per-session opt-out for Sample-Based Tablet Pre-Split. Defaults to `true` so the FE Config gates (`enable_tablet_pre_split_for_*`) remain the primary on/off switch. Set this to `false` for a session whose load you want to leave undisturbed. Both the matching Config flag and this session variable must be `true` for pre-split to run. * **Default**: true * **Introduced in**: v4.1.0 ##### enable\_topn\_runtime\_filter[​](#enable_topn_runtime_filter "Direct link to enable_topn_runtime_filter") * **Description**: Whether to enable TopN Runtime Filter. If this feature is enabled, a runtime filter will be dynamically constructed for ORDER BY LIMIT queries and pushed down to the scan for filtering. * **Default**: true * **Introduced in**: v3.3 ##### enable\_ukfk\_join\_reorder[​](#enable_ukfk_join_reorder "Direct link to enable_ukfk_join_reorder") * **Scope**: Session * **Description**: When enabled, the optimizer collects Unique Key (UK) / Foreign Key (FK) column constraints for the two join sides and uses that information to bias join reordering. If a join has an intact UK constraint, the optimizer may choose the FK table as the right child (probe/input) instead of using plain row-count comparison. The decision uses the helper allowFKAsRightTable(), which: * rejects reordering when the FK side is ordered by the FK column (`fkConstraint.isOrderByFK`), * computes normalized row counts by scaling row counts with summed column type sizes, * computes a scale ratio = fkNormalizedRows / max(1, ukNormalizedRows), and * allows the FK as right table only when the scale ratio and FK rows are below session thresholds (`max_ukfk_join_reorder_scale_ratio` and `max_ukfk_join_reorder_fk_rows`). If UK/FK conditions are not met, join ordering falls back to the default smaller-table-as-right-child heuristic (rowCount comparison). The variable is accessed via SessionVariable getters/setters and used in the JoinOrder optimizer rule. * **Default**: `false` * **Data Type**: boolean * **Introduced in**: v3.2.4 ##### enable\_ukfk\_opt[​](#enable_ukfk_opt "Direct link to enable_ukfk_opt") * **Description**: Enables optimizer support for Unique-Key / Foreign-Key (UK/FK) based transformations and statistics enhancements. When set, the optimizer runs `UKFKConstraintsCollector` to collect unique and foreign-key constraints bottom‑up and attach them to plan nodes (OptExpressions). The collected constraints are consumed by transformation rules such as `PruneUKFKJoinRule` (which can prune the UK-side of joins, rewrite predicates from UK to FK columns and add IS NULL checks for outer-join cases) and `PruneUKFKGroupByKeysRule` (which can remove redundant GROUP BY keys derived from UK/FK relationships). The collected UK/FK information is also used in `StatisticsCalculator` to produce tighter join cardinality estimates for UK‑FK joins and may replace default estimates when more precise. Default is conservative (`false`) because these optimizations rely on declared schema constraints and can change plan shape and predicate placement. * **Default**: `false` * **Scope**: Session * **Data Type**: boolean * **Introduced in**: v3.2.4 ##### enable\_view\_based\_mv\_rewrite[​](#enable_view_based_mv_rewrite "Direct link to enable_view_based_mv_rewrite") * **Description**: Whether to enable query rewrite for logical view-based materialized views. If this item is set to `true`, the logical view is used as a unified node to rewrite the queries against itself for better performance. If this item is set to `false`, the system transcribes the queries against logical views into queries against physical tables or materialized views and then rewrites them. * **Default**: false * **Introduced in**: v3.1.9, v3.2.5, v3.3.0 ##### enable\_wait\_dependent\_event[​](#enable_wait_dependent_event "Direct link to enable_wait_dependent_event") * **Description**: Whether Pipeline waits for a dependent operator to finish execution before continuing within the same fragment. For example, in a left join query, when this feature is enabled, the probe-side operator waits for the build-side operator to finish before it starts executing. Enabling this feature can reduce memory usage, but may increase the query latency. However, for queries reused in CTE, enabling this feature may increase memory usage. * **Default**: false * **Introduced in**: v3.3 ##### enable\_write\_hive\_external\_table[​](#enable_write_hive_external_table "Direct link to enable_write_hive_external_table") * **Description**: Whether to allow for sinking data to external tables of Hive. * **Default**: false * **Introduced in**: v3.2 ##### event\_scheduler[​](#event_scheduler "Direct link to event_scheduler") Used for MySQL client compatibility. No practical usage. ##### force\_schedule\_local (Session)[​](#force_schedule_local-session "Direct link to force_schedule_local (Session)") * **Scope**: Session * **Description**: When enabled, the query scheduler prefers assigning HDFS/file scan ranges to co‑located (local) compute backends that host the data blocks. `HDFSBackendSelector` receives this flag from the session and calls its computeForceScheduleLocalAssignment(...) path to: 1) select backends whose hostnames match scan-range locations, 2) choose among those local backends using reBalanceScanRangeForComputeNode(...) (which considers per-node assigned bytes and a max imbalance ratio), and 3) return remaining ranges for remote assignment via consistent hashing. `HiveConnectorScanRangeSource` also reads this session variable and, when set, emits block-level scan ranges (one per block) to enable local placement. Use this to improve data locality and reduce network I/O. Note it can increase skew (assignment imbalance) and may cause more rebalancing in heavy-locality scenarios. * **Default**: `false` * **Data Type**: boolean * **Introduced in**: v3.2.0 ##### forward\_to\_leader[​](#forward_to_leader "Direct link to forward_to_leader") Used to specify whether some commands will be forwarded to the leader FE for execution. Alias: `forward_to_master`. The default value is `false`, meaning not forwarding to the leader FE. There are multiple FEs in a StarRocks cluster, one of which is the leader FE. Normally, users can connect to any FE for full-featured operations. However, some information is only available on the leader FE. For example, if the SHOW BACKENDS command is not forwarded to the leader FE, only basic information (for example, whether the node is alive) can be viewed. Forwarding to the leader FE can get more detailed information including the node start time and last heartbeat time. The commands affected by this variable are as follows: * SHOW FRONTENDS: Forwarding to the leader FE allows users to view the last heartbeat message. * SHOW BACKENDS: Forwarding to the leader FE allows users to view the boot time, last heartbeat information, and disk capacity information. * SHOW BROKER: Forwarding to the leader FE allows users to view the boot time and last heartbeat information. * SHOW TABLET * ADMIN SHOW REPLICA DISTRIBUTION * ADMIN SHOW REPLICA STATUS: Forwarding to the leader FE allows users to view the tablet information stored in the metadata of the leader FE. Normally, the tablet information should be the same in the metadata of different FEs. If an error occurs, you can use this method to compare the metadata of the current FE and the leader FE. * Show PROC: Forwarding to the leader FE allows users to view the PROC information stored in the metadata. This is mainly used for metadata comparison. ##### group\_concat\_max\_len[​](#group_concat_max_len "Direct link to group_concat_max_len") * **Description**: The maximum length of string returned by the [group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat.md) function. * **Default**: 1024 * **Min value**: 4 * **Unit**: Characters * **Data type**: Long ##### group\_execution\_max\_groups[​](#group_execution_max_groups "Direct link to group_execution_max_groups") * **Description**: Maximum number of groups allowed for Group Execution. It is used to limit the granularity of splitting, preventing excessive scheduling overhead caused by an excessive number of groups. * **Default**: 128 * **Introduced in**: v3.3 ##### group\_execution\_min\_scan\_rows[​](#group_execution_min_scan_rows "Direct link to group_execution_min_scan_rows") * **Description**: Minimum number of rows processed per group for Group Execution. * **Default**: 5000000 * **Introduced in**: v3.3 ##### hash\_join\_push\_down\_right\_table[​](#hash_join_push_down_right_table "Direct link to hash_join_push_down_right_table") * **Description**: Used to control whether the data of the left table can be filtered by using the filter condition against the right table in the Join query. If so, it can reduce the amount of data that needs to be processed during the query. **Default**: `true` indicates the operation is allowed and the system decides whether the left table can be filtered. `false` indicates the operation is disabled. The default value is `true`. ##### historical\_nodes\_min\_update\_interval[​](#historical_nodes_min_update_interval "Direct link to historical_nodes_min_update_interval") * **Description**: The minimum interval between two updates of historical node records. If the nodes of a cluster change frequently in a short period of time (that is, less than the value set in this variable), some intermediate states will not be recorded as valid historical node snapshots. The historical nodes are the main basis for the Cache Sharing feature to choose the right cache nodes during cluster scaling. * **Default**: 600 * **Unit**: Seconds * **Introduced in**: v3.5.1 ##### init\_connect (global)[​](#init_connect-global "Direct link to init_connect (global)") Used for MySQL client compatibility. No practical usage. ##### innodb\_read\_only[​](#innodb_read_only "Direct link to innodb_read_only") * **Description**: Session-level flag (MySQL-compatible) that indicates the session's InnoDB read-only mode. The variable is declared and stored on the session as the Java field `innodbReadOnly` in `SessionVariable.java` and is accessible via `isInnodbReadOnly()` and `setInnodbReadOnly(boolean)`. The SessionVariable class only holds the flag; any enforcement (preventing write/DDL to InnoDB tables or altering transaction behavior) must be implemented by the transaction/storage/authorization layers which should read this session flag. Use this variable to convey client intent for read-only behavior within the current session for components that respect it. * **Scope**: Session * **Default**: `true` * **Data Type**: boolean * **Introduced in**: v3.2.0 ##### insert\_max\_filter\_ratio[​](#insert_max_filter_ratio "Direct link to insert_max_filter_ratio") * **Description**: The maximum error tolerance of INSERT from files(). It's the maximum ratio of data records that can be filtered out due to inadequate data quality. When the ratio of unqualified data records reaches this threshold, the job fails. Range: \[0, 1]. * **Default**: 0 * **Introduced in**: v3.4.0 ##### insert\_timeout[​](#insert_timeout "Direct link to insert_timeout") * **Description**: The timeout duration of the INSERT job. Unit: Seconds. From v3.4.0 onwards, `insert_timeout` applies to operations involved INSERT (for example, UPDATE, DELETE, CTAS, materialized view refresh, statistics collection, and PIPE), replacing `query_timeout`. * **Default**: 14400 * **Introduced in**: v3.4.0 ##### interactive\_timeout[​](#interactive_timeout "Direct link to interactive_timeout") Used for MySQL client compatibility. No practical usage. ##### io\_tasks\_per\_scan\_operator[​](#io_tasks_per_scan_operator "Direct link to io_tasks_per_scan_operator") * **Description**: The number of concurrent I/O tasks that can be issued by a scan operator. Increase this value if you want to access remote storage systems such as HDFS or S3 but the latency is high. However, a larger value causes more memory consumption. * **Default**: 4 * **Data type**: Int * **Introduced in**: v2.5 ##### jit\_level[​](#jit_level "Direct link to jit_level") * **Description**: The level at which JIT compilation for expressions is enabled. Valid values: * `1`: The system adaptively enables JIT compilation for compilable expressions. * `-1`: JIT compilation is enabled for all compilable, non-constant expressions. * `0`: JIT compilation is disabled. You can disable it manually if any error is returned for this feature. * **Default**: 1 * **Data type**: Int * **Introduced in**: - ##### join\_late\_materialization[​](#join_late_materialization "Direct link to join_late_materialization") * **Scope**: Session * **Description**: Controls whether the planner enables "late materialization" on join operators. When set to `true`, PlanFragmentBuilder reads the session flag and calls `joinNode.setEnableLateMaterialization(...)` so join execution can defer full row/payload materialization until after join/key-based filtering. This reduces memory usage and I/O for joins with large payload columns or highly selective join predicates by carrying only join keys through the join and materializing payloads for matched rows. The flag is defined in `SessionVariable.java` as `JOIN_LATE_MATERIALIZATION` and defaults to `false`. Enabling this may interact with column-trimming and scan-stage pruning optimizations (for example `enable_filter_unused_columns_in_scan_stage`) and can change join runtime behavior; test queries for correctness and performance before enabling broadly. * **Default**: `false` * **Data Type**: boolean * **Introduced in**: v3.3.0, v3.4.0, v3.5.0 ##### lake\_bucket\_assign\_mode[​](#lake_bucket_assign_mode "Direct link to lake_bucket_assign_mode") * **Description**: The bucket assignment mode for queries against tables in data lakes. This variable controls how buckets are distributed among worker nodes when bucket-aware execution takes effect during query execution. Valid values: * `balance`: Distributes buckets evenly across worker nodes to achieve balanced workload and better performance. * `elastic`: Uses consistent hashing to assign buckets to worker nodes, which can provide better load distribution in elastic environments. * **Default**: balance * **Data type**: String * **Introduced in**: v4.0 ##### language (global)[​](#language-global "Direct link to language (global)") Used for MySQL client compatibility. No practical usage. ##### large\_in\_predicate\_threshold[​](#large_in_predicate_threshold "Direct link to large_in_predicate_threshold") * **Scope**: Session * **Description**: Threshold (number of constants) at which the planner switches a regular IN-list predicate into the compact LargeInPredicate representation to avoid building very large ASTs and heavy FE analysis/planning overhead. When `enable_large_in_predicate` is true and an IN-list (string or integer) contains >= this many literals, the parser (AstBuilder) creates a LargeInPredicate containing the raw text and a compact value list (and keeps only a minimal representative Expr in the AST). For integer lists the parser additionally verifies all literals can be parsed as integral values and will fall back to a normal InPredicate if parsing fails. LargeInPredicate is later transformed into a left semi/anti join for execution, improving parse/analyze/deploy performance for queries with very large constant IN lists. The variable is exposed as a session-level `VariableMgr.VarAttr` and can be get/set per session. * **Default**: `100000` * **Data Type**: int * **Introduced in**: - ##### license (global)[​](#license-global "Direct link to license (global)") * **Description**: Displays the license of StarRocks. * **Default**: Apache License 2.0 ##### load\_mem\_limit[​](#load_mem_limit "Direct link to load_mem_limit") Specifies the memory limit for the import operation. The default value is 0, meaning that this variable is not used and `query_mem_limit` is used instead. This variable is only used for the `INSERT` operation which involves both query and import. If the user does not set this variable, the memory limit for both query and import will be set as `exec_mem_limit`. Otherwise, the memory limit for query will be set as `exec_mem_limit` and the memory limit for import will be as `load_mem_limit`. Other import methods such as `BROKER LOAD`, `STREAM LOAD` still use `exec_mem_limit` for memory limit. ##### log\_rejected\_record\_num (v3.1 and later)[​](#log_rejected_record_num-v31-and-later "Direct link to log_rejected_record_num (v3.1 and later)") Specifies the maximum number of unqualified data rows that can be logged. Valid values: `0`, `-1`, and any non-zero positive integer. Default value: `0`. * The value `0` specifies that data rows that are filtered out will not be logged. * The value `-1` specifies that all data rows that are filtered out will be logged. * A non-zero positive integer such as `n` specifies that up to `n` data rows that are filtered out can be logged on each BE. ##### low\_cardinality\_optimize\_on\_lake[​](#low_cardinality_optimize_on_lake "Direct link to low_cardinality_optimize_on_lake") * **Default**: true * **Type**: Boolean * **Unit**: - * **Description**: Whether to enable low cardinality optimization on data lake queries. Valid values: * `true` (Default): Enable low cardinality optimization on data lake queries. * `false`: Disable low cardinality optimization on data lake queries. * **Introduced in**: v3.5.0 ##### low\_cardinality\_optimize\_v2[​](#low_cardinality_optimize_v2 "Direct link to low_cardinality_optimize_v2") * **Scope**: Session * **Description**: Session-level boolean that selects which low-cardinality optimization rewrite strategy the optimizer applies. When `true` the optimizer will attempt the V2 rewrite strategy implemented by `LowCardinalityRewriteRule` (uses `DecodeCollector` / `DecodeRewriter` to encode/decode low-cardinality VARCHAR columns). When `false` the optimizer falls back to the legacy rewrite strategy implemented by `AddDecodeNodeForDictStringRule`. Execution of either rewrite also requires `enableLowCardinalityOptimize` to be enabled; if `enableLowCardinalityOptimize` is disabled, no low-cardinality rewrite is performed. The variable is checked in optimizer tree-rewrite paths to choose the appropriate transformation. * **Default**: `true` * **Data Type**: boolean * **Introduced in**: v3.3.0, v3.4.0, v3.5.0 ##### lower\_case\_table\_names (global)[​](#lower_case_table_names-global "Direct link to lower_case_table_names (global)") Used for MySQL client compatibility. No practical usage. Table names in StarRocks are case-sensitive. ##### materialized\_view\_rewrite\_mode (v3.2 and later)[​](#materialized_view_rewrite_mode-v32-and-later "Direct link to materialized_view_rewrite_mode (v3.2 and later)") Specifies the query rewrite mode of asynchronous materialized views. Valid values: * `disable`: Disable automatic query rewrite of asynchronous materialized views. * `default` (Default value): Enable automatic query rewrite of asynchronous materialized views, and allow the optimizer to decide whether a query can be rewritten using the materialized view based on the cost. If the query cannot be rewritten, it directly scans the data in the base table. * `default_or_error`: Enable automatic query rewrite of asynchronous materialized views, and allow the optimizer to decide whether a query can be rewritten using the materialized view based on the cost. If the query cannot be rewritten, an error is returned. * `force`: Enable automatic query rewrite of asynchronous materialized views, and the optimizer prioritizes query rewrite using the materialized view. If the query cannot be rewritten, it directly scans the data in the base table. * `force_or_error`: Enable automatic query rewrite of asynchronous materialized views, and the optimizer prioritizes query rewrite using the materialized view. If the query cannot be rewritten, an error is returned. ##### materialized\_view\_subquery\_text\_match\_max\_count[​](#materialized_view_subquery_text_match_max_count "Direct link to materialized_view_subquery_text_match_max_count") * **Description**: Specifies the maximum number of times that the system checks whether a query's sub-query matches the materialized views' definition. * **Default**: 4 * **Introduced in**: v3.2.5, v3.3.0 ##### max\_allowed\_packet[​](#max_allowed_packet "Direct link to max_allowed_packet") * **Description**: Used for compatibility with the JDBC connection pool C3P0. This variable specifies the maximum size of packets that can be transmitted between the client and server. * **Default**: 33554432 (32 MB). You can raise this value if the client reports "PacketTooBigException". * **Unit**: Byte * **Data type**: Int ##### max\_parallel\_scan\_instance\_num[​](#max_parallel_scan_instance_num "Direct link to max_parallel_scan_instance_num") * **Scope**: Session * **Description**: Session-level integer that caps the number of parallel scan instances the planner will produce for scan operators (applied to OLAP/Lake scan nodes). The value is propagated into the scan node Thrift message (`TOlapScanNode.max_parallel_scan_instance_num`) and is included in query/load runtime profiles. It is declared with `@VariableMgr.VarAttr` and can be read/set via the session variable APIs (`getMaxParallelScanInstanceNum` / `setMaxParallelScanInstanceNum`). Use this to limit scan parallelism per session for resource control or debugging. When left at the default `-1`, the planner/system default (resource- or configuration-derived) parallelism is used. * **Default**: `-1` * **Data Type**: int * **Introduced in**: v3.2.0 ##### max\_pipeline\_dop[​](#max_pipeline_dop "Direct link to max_pipeline_dop") * **Scope**: Session * **Description**: The per-session upper bound for the pipeline engine's degree-of-parallelism (DOP). Behavior: * Applies only when `enable_pipeline_engine` is enabled and `pipeline_dop` is not explicitly set (greater than 0). If `pipeline_dop` greater than 0 this variable is ignored and `pipeline_dop` is used directly. * When `pipeline_dop` less than or equal to 0 (adaptive/default mode), the effective DOP for execution is computed as min(`max_pipeline_dop`, the backend default DOP returned by BackendResourceStat). For pipeline sinks the same logic uses the sink default DOP. * If `max_pipeline_dop` less than or equal to 0 no additional cap is applied and the backend default DOP is used. * Purpose: avoid negative overhead from scheduling on machines with very large core counts by capping automatically computed parallelism. * **Default**: `64` * **Data Type**: int * **Introduced in**: v3.2.0 ##### max\_pushdown\_conditions\_per\_column[​](#max_pushdown_conditions_per_column "Direct link to max_pushdown_conditions_per_column") * **Description**: The maximum number of predicates that can be pushed down for a column. * **Default**: -1, indicating that the value in the `be.conf` file is used. If this variable is set to a value greater than 0, the value in `be.conf` is ignored. * **Data type**: Int ##### max\_scan\_key\_num[​](#max_scan_key_num "Direct link to max_scan_key_num") * **Description**: The maximum number of scan key segmented by each query. * **Default**: -1, indicating that the value in the `be.conf` file is used. If this variable is set to a value greater than 0, the value in `be.conf` is ignored. ##### metadata\_collect\_query\_timeout[​](#metadata_collect_query_timeout "Direct link to metadata_collect_query_timeout") * **Description**: The timeout duration for Iceberg Catalog metadata collection queries. * **Unit**: Second * **Default**: 60 * **Introduced in**: v3.3.3 ##### nested\_mv\_rewrite\_max\_level[​](#nested_mv_rewrite_max_level "Direct link to nested_mv_rewrite_max_level") * **Description**: The maximum levels of nested materialized views that can be used for query rewrite. * **Value range**: \[1, +∞). The value of `1` indicates that only materialized views created on base tables can be used for query rewrite. * **Default**: 3 * **Data type**: Int ##### net\_buffer\_length[​](#net_buffer_length "Direct link to net_buffer_length") Used for MySQL client compatibility. No practical usage. ##### net\_read\_timeout[​](#net_read_timeout "Direct link to net_read_timeout") Used for MySQL client compatibility. No practical usage. ##### net\_write\_timeout[​](#net_write_timeout "Direct link to net_write_timeout") Used for MySQL client compatibility. No practical usage. ##### new\_planner\_agg\_stage[​](#new_planner_agg_stage "Direct link to new_planner_agg_stage") * **Scope**: Session * **Description**: Controls how the new planner selects aggregation phase decomposition. Valid integer values (0–4): * `0` (AUTO) — allow the optimizer to choose the aggregation stage selection. When `0` is set, cost-based multi-stage decisions can be enabled via `enable_cost_based_multi_stage_agg`. * `1` (ONE\_STAGE) — force a single-stage aggregate. * `2` (TWO\_STAGE) — force a two-stage aggregate. * `3` (THREE\_STAGE) — force a three-stage aggregate (only producible for single-column DISTINCT scenarios). * `4` (FOUR\_STAGE) — force a four-stage aggregate (only producible for single-column DISTINCT scenarios). Setting a forced stage overrides automatic selection logic used by optimizer rules (for example, SplitMultiPhaseAggRule and related cost checks). When `0` is set, the planner may still be constrained by `enable_cost_based_multi_stage_agg`. The variable is consulted in cost enforcement (EnforceAndCostTask), aggregation-splitting rules, and plan-fragment construction to influence exchange/partitioning and pruning decisions. * **Default**: `0` * **Data Type**: int * **Introduced in**: v3.2.0 ##### new\_planner\_optimize\_timeout[​](#new_planner_optimize_timeout "Direct link to new_planner_optimize_timeout") * **Description**: The timeout duration of the query optimizer. When the optimizer times out, an error is returned and the query is stopped, which affects the query performance. You can set this variable to a larger value based on your query or contact StarRocks technical support for troubleshooting. A timeout often occurs when a query has too many joins. * **Default**: 3000 * **Unit**: ms ##### optimizer\_materialized\_view\_timelimit[​](#optimizer_materialized_view_timelimit "Direct link to optimizer_materialized_view_timelimit") * **Description**: Specifies the maximum time that one materialized view rewrite rule can consume. When the threshold is reached, this rule will not be used for query rewrite. * **Default**: 1000 * **Unit**: ms * **Introduced in**: v3.1.9, v3.2.5 ##### orc\_use\_column\_names[​](#orc_use_column_names "Direct link to orc_use_column_names") * **Description**: Used to specify how columns are matched when StarRocks reads ORC files from Hive. The default value is `false`, which means columns in ORC files are read based on their ordinal positions in the Hive table definition. If this variable is set to `true`, columns are read based on their names. * **Default**: false * **Introduced in**: v3.1.10 ##### parallel\_exchange\_instance\_num[​](#parallel_exchange_instance_num "Direct link to parallel_exchange_instance_num") Used to set the number of exchange nodes that an upper-level node uses to receive data from a lower-level node in the execution plan. The default value is -1, meaning the number of exchange nodes is equal to the number of execution instances of the lower-level node. When this variable is set to be greater than 0 but smaller than the number of execution instances of the lower-level node, the number of exchange nodes equals the set value. In a distributed query execution plan, the upper-level node usually has one or more exchange nodes to receive data from the execution instances of the lower-level node on different BEs. Usually the number of exchange nodes is equal to the number of execution instances of the lower-level node. In some aggregation query scenarios where the amount of data decreases drastically after aggregation, you can try to modify this variable to a smaller value to reduce the resource overhead. An example would be running aggregation queries using the Duplicate Key table. ##### parallel\_fragment\_exec\_instance\_num[​](#parallel_fragment_exec_instance_num "Direct link to parallel_fragment_exec_instance_num") Used to set the number of instances used to scan nodes on each BE. The default value is 1. A query plan typically produces a set of scan ranges. This data is distributed across multiple BE nodes. A BE node will have one or more scan ranges, and by default, each BE node's set of scan ranges is processed by only one execution instance. When machine resources suffice, you can increase this variable to allow more execution instances to process a scan range simultaneously for efficiency purposes. The number of scan instances determines the number of other execution nodes in the upper level, such as aggregation nodes and join nodes. Therefore, it increases the concurrency of the entire query plan execution. Modifying this variable will help improve efficiency, but larger values will consume more machine resources, such as CPU, memory, and disk IO. ##### parallel\_merge\_late\_materialization\_mode[​](#parallel_merge_late_materialization_mode "Direct link to parallel_merge_late_materialization_mode") * **Description**: The late materialization mode of parallel merge for sorting. Valid values: * `AUTO` * `ALWAYS` * `NEVER` * **Default**: `AUTO` * **Introduced in**: v3.3 ##### partial\_update\_mode[​](#partial_update_mode "Direct link to partial_update_mode") * **Description**: Used to control the mode of partial updates. Valid values: * `auto` (default): The system automatically determines the mode of partial updates by analyzing the UPDATE statement and the columns involved. * `column`: The column mode is used for the partial updates, which is particularly suitable for the partial updates which involve a small number of columns and a large number of rows. For more information, see [UPDATE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/UPDATE.md#partial-updates-in-column-mode-since-v31). * **Default**: auto * **Introduced in**: v3.1 ##### performance\_schema (global)[​](#performance_schema-global "Direct link to performance_schema (global)") Used for compatibility with MySQL JDBC versions 8.0.16 and above. No practical usage. ##### phased\_scheduler\_max\_concurrency[​](#phased_scheduler_max_concurrency "Direct link to phased_scheduler_max_concurrency") * **Description**: The concurrency for phased scheduler scheduling leaf node fragments. For example, the default value means that, in a large number of UNION ALL Scan queries, at most two scan fragments are allowed to be scheduled at the same time. * **Default**: 2 * **Introduced in**: v3.3 ##### pipeline\_dop[​](#pipeline_dop "Direct link to pipeline_dop") * **Description**: The parallelism of a pipeline instance, which is used to adjust the query concurrency. Default value: 0, indicating the system automatically adjusts the parallelism of each pipeline instance. This variable also controls the parallelism of loading jobs on OLAP tables. You can also set this variable to a value greater than 0. Generally, set the value to half the number of physical CPU cores. From v3.0 onwards, StarRocks adaptively adjusts this variable based on query parallelism. * **Default**: 0 * **Data type**: Int ##### pipeline\_profile\_level[​](#pipeline_profile_level "Direct link to pipeline_profile_level") * **Description**: Controls the level of the query profile. A query profile often has five layers: Fragment, FragmentInstance, Pipeline, PipelineDriver, and Operator. Different levels provide different details of the profile: * 0: StarRocks combines metrics of the profile and shows only a few core metrics. * 1: default value. StarRocks simplifies the profile and combines metrics of the profile to reduce profile layers. * 2: StarRocks retains all the layers of the profile. The profile size is large in this scenario, especially when the SQL query is complex. This value is not recommended. * **Default**: 1 * **Data type**: Int ##### pipeline\_sink\_dop[​](#pipeline_sink_dop "Direct link to pipeline_sink_dop") * **Description**: The parallelism of sink for loading data into Iceberg tables and Hive tables, and unloading data using INSERT INTO FILES(). It is used to adjust the concurrency of these loading jobs. Default value: 0, indicating the system automatically adjusts the parallelism. You can also set this variable to a value greater than 0. * **Default**: 0 * **Data type**: Int ##### plan\_mode[​](#plan_mode "Direct link to plan_mode") * **Description**: The metadata retrieval strategy of Iceberg Catalog. For more information, see [Iceberg Catalog metadata retrieval strategy](https://docs.starrocks.io/docs/data_source/catalog/iceberg.md#appendix-periodic-metadata-refresh-strategy). Valid values: * `auto`: The system will automatically select the retrieval plan. * `local`: Use the local cache plan. * `distributed`: Use the distributed plan. * **Default**: auto * **Introduced in**: v3.3.3 ###### enable\_iceberg\_column\_statistics[​](#enable_iceberg_column_statistics "Direct link to enable_iceberg_column_statistics") * **Description**: Whether to obtain column statistics, such as `min`, `max`, `null count`, `row size`, and `ndv` (if a puffin file exists). When this item is set to `false`, only the row count information will be collected. * **Default**: false * **Introduced in**: v3.4 ##### populate\_datacache\_mode[​](#populate_datacache_mode "Direct link to populate_datacache_mode") * **Description**: Specifies the population behavior of Data Cache when reading data blocks from external storage systems. Valid values: * `auto` (default): the system automatically caches data selectively based on the population rule. * `always`: Always cache the data. * `never`: Never cache the data. * **Default**: auto * **Introduced in**: v3.3.2 ##### query\_cache\_agg\_cardinality\_limit[​](#query_cache_agg_cardinality_limit "Direct link to query_cache_agg_cardinality_limit") * **Description**: The upper limit of cardinality for GROUP BY in Query Cache. Query Cache is not enabled if the rows generated by GROUP BY exceeds this value. Default value: 5000000. If `query_cache_entry_max_bytes` or `query_cache_entry_max_rows` is set to 0, the Passthrough mode is used even when no computation results are generated from the involved tablets. * **Default**: 5000000 * **Data type**: Long * **Introduced in**: v2.5 ##### query\_cache\_entry\_max\_bytes[​](#query_cache_entry_max_bytes "Direct link to query_cache_entry_max_bytes") * **Description**: The threshold for triggering the Passthrough mode. When the number of bytes or rows from the computation results of a specific tablet accessed by a query exceeds the threshold specified by `query_cache_entry_max_bytes` or `query_cache_entry_max_rows`, the query is switched to Passthrough mode. * **Valid values**: 0 to 9223372036854775807 * **Default**: 4194304 * **Unit**: Byte * **Introduced in**: v2.5 ##### query\_cache\_entry\_max\_rows[​](#query_cache_entry_max_rows "Direct link to query_cache_entry_max_rows") * **Description**: The upper limit of rows that can be cached. See the description in `query_cache_entry_max_bytes`. Default value: . * **Default**: 409600 * **Introduced in**: v2.5 ##### query\_cache\_size (global)[​](#query_cache_size-global "Direct link to query_cache_size (global)") Used for MySQL client compatibility. No practical use. ##### query\_cache\_type[​](#query_cache_type "Direct link to query_cache_type") Used for compatibility with JDBC connection pool C3P0. No practical use. ##### query\_delivery\_timeout[​](#query_delivery_timeout "Direct link to query_delivery_timeout") * **Scope**: Session * **Description**: Timeout (in seconds) for phase 1 of query execution — the delivery of all plan fragment instances from the coordinator (FE) to backend executors (BEs). StarRocks executes queries in two phases: (1) deliver fragments to BEs, and (2) pull results after fragments are prepared. `query_delivery_timeout` controls how long the coordinator waits for fragment delivery to complete before timing out. When building the query options sent to the backend, the value is assigned to the `query_delivery_timeout` field (subject to an internal cap to avoid integer overflow). If `enablePhasedScheduler` is enabled, the system uses the `query_timeout` value instead for delivery timeout. * **Default**: `300` * **Data Type**: int (seconds) * **Introduced in**: v3.2.0 ##### query\_mem\_limit[​](#query_mem_limit "Direct link to query_mem_limit") * **Description**: Used to set the memory limit of a query on each BE node. The default value is 0, which means no limit for it. This item takes effect only after Pipeline Engine is enabled. When the `Memory Exceed Limit` error happens, you could try to increase this variable. Setting it to `0` indicates no limit is imposed. * **Default**: 0 * **Unit**: Byte ##### query\_queue\_concurrency\_limit (global)[​](#query_queue_concurrency_limit-global "Direct link to query_queue_concurrency_limit (global)") * **Description**: The upper limit of concurrent queries on a BE. It takes effect only after being set greater than `0`. Setting it to `0` indicates no limit is imposed. * **Default**: 0 * **Data type**: Int ##### query\_queue\_cpu\_used\_permille\_limit (global)[​](#query_queue_cpu_used_permille_limit-global "Direct link to query_queue_cpu_used_permille_limit (global)") * **Description**: The upper limit of CPU usage permille (CPU usage \* 1000) on a BE. It takes effect only after being set greater than `0`. Setting it to `0` indicates no limit is imposed. * **Value range**: \[0, 1000] * **Default**: `0` ##### query\_queue\_max\_queued\_queries (global)[​](#query_queue_max_queued_queries-global "Direct link to query_queue_max_queued_queries (global)") * **Description**: The upper limit of queries in a queue. When this threshold is reached, incoming queries are rejected. It takes effect only after being set greater than `0`. Setting it to `0` indicates no limit is imposed. * **Default**: `1024`. ##### query\_queue\_mem\_used\_pct\_limit (global)[​](#query_queue_mem_used_pct_limit-global "Direct link to query_queue_mem_used_pct_limit (global)") * **Description**: The upper limit of memory usage percentage on a BE. It takes effect only after being set greater than `0`. Setting it to `0` indicates no limit is imposed. Under Query Queue v1, it applies to every BE and CN node of the cluster. Under Query Queue v2, new queries wait while any alive node reaches the limit. See [Memory admission](https://docs.starrocks.io/docs/administration/management/resource_management/query_queues.md#memory-admission). * **Value range**: \[0, 1] * **Default**: 0 ##### query\_queue\_pending\_timeout\_second (global)[​](#query_queue_pending_timeout_second-global "Direct link to query_queue_pending_timeout_second (global)") * **Description**: The maximum timeout of a pending query in a queue. When this threshold is reached, the corresponding query is rejected. * **Default**: 300 * **Unit**: Second ##### query\_timeout[​](#query_timeout "Direct link to query_timeout") * **Description**: Used to set the query timeout in "seconds". This variable will act on all query statements in the current connection. The default value is 300 seconds. From v3.4.0 onwards, `query_timeout` does not apply to operations involved INSERT (for example, UPDATE, DELETE, CTAS, materialized view refresh, statistics collection, and PIPE). * **Value range**: \[1, 259200] * **Default**: 300 * **Data type**: Int * **Unit**: Second ##### range\_pruner\_max\_predicate[​](#range_pruner_max_predicate "Direct link to range_pruner_max_predicate") * **Description**: The maximum number of IN predicates that can be used for Range partition pruning. Default value: 100. A value larger than 100 may cause the system to scan all tablets, which compromises the query performance. * **Default**: 100 * **Introduced in**: v3.0 ##### resource\_group[​](#resource_group "Direct link to resource_group") * **Description**: The specified resource group of this session * **Default**: "" * **Data Type**: String * **Introduced in**: 3.2.0 ##### runtime\_filter\_on\_exchange\_node[​](#runtime_filter_on_exchange_node "Direct link to runtime_filter_on_exchange_node") * **Description**: Whether to place GRF on Exchange Node after GRF is pushed down across the Exchange operator to a lower-level operator. The default value is `false`, which means GRF will not be placed on Exchange Node after it is pushed down across the Exchange operator to a lower-level operator. This prevents repetitive use of GRF and reduces the computation time. However, GRF delivery is a "try-best" process. If the lower-level operator fails to receive the GRF but the GRF is not placed on Exchange Node, data cannot be filtered, which compromises filter performance. `true` means GRF will still be placed on Exchange Node even after it is pushed down across the Exchange operator to a lower-level operator. * **Default**: false ##### runtime\_join\_filter\_push\_down\_limit[​](#runtime_join_filter_push_down_limit "Direct link to runtime_join_filter_push_down_limit") * **Description**: The maximum number of rows allowed for the Hash table based on which Bloom filter Local RF is generated. Local RF will not be generated if this value is exceeded. This variable prevents the generation of an excessively long Local RF. * **Default**: 1024000 * **Data type**: Int ##### runtime\_profile\_report\_interval[​](#runtime_profile_report_interval "Direct link to runtime_profile_report_interval") * **Description**: The time interval at which runtime profiles are reported. * **Default**: 10 * **Unit**: Second * **Data type**: Int * **Introduced in**: v3.1.0 ##### scan\_olap\_partition\_num\_limit[​](#scan_olap_partition_num_limit "Direct link to scan_olap_partition_num_limit") * **Description**: The number of partitions allowed to be scanned for a single table in the execution plan. * **Default**: 0 (No limit) * **Introduced in**: v3.3.9 ##### allow\_lake\_without\_partition\_filter[​](#allow_lake_without_partition_filter "Direct link to allow_lake_without_partition_filter") * **Description**: Whether to allow queries on lake tables (Hive, Iceberg, Delta Lake, Paimon, etc.) without a partition filter predicate. When set to `false`, queries that do not contain a valid partition predicate on these tables will be rejected to prevent accidental full-table scans. * **Scope**: Session * **Default**: `true` * **Data type**: Boolean * **Alias**: `allow_hive_without_partition_filter` ##### scan\_lake\_partition\_num\_limit[​](#scan_lake_partition_num_limit "Direct link to scan_lake_partition_num_limit") * **Description**: The maximum number of partitions allowed to be scanned for a single lake table (Hive, Iceberg, Delta Lake, Paimon, etc.). When set to `0`, no limit is applied. When exceeded, the query will return an error. Note that for catalog types that enumerate splits incrementally (Iceberg, Delta Lake), the limit check is performed during scan-range dispatch and the query may fail mid-execution rather than being rejected upfront. * **Scope**: Session * **Default**: `0` (No limit) * **Data type**: Int * **Alias**: `scan_hive_partition_num_limit` ##### skip\_local\_disk\_cache[​](#skip_local_disk_cache "Direct link to skip_local_disk_cache") * **Description**: Session flag that instructs the FE, when building scan ranges, to mark each tablet's internal scan range with `skip_disk_cache`. When set to `true`, `OlapScanNode.addScanRangeLocations()` sets `internalRange.setSkip_disk_cache(true)` on the created `TInternalScanRange` objects so downstream BE scan nodes are told to bypass the local disk cache for that scan. It is applied per-session and is evaluated at plan/scan-range construction time. Use this together with `skip_page_cache` (to control page cache skipping) and data-cache related variables (`enable_scan_datacache` / `enable_populate_datacache`) as appropriate. * **Scope**: Session * **Default**: `false` * **Data Type**: boolean * **Introduced in**: v3.3.9, v3.4.0, v3.5.0 ##### skip\_page\_cache[​](#skip_page_cache "Direct link to skip_page_cache") * **Scope**: Session * **Description**: Session-level boolean flag that instructs the planner/frontend to mark scan ranges so backends should bypass the page cache when reading data. When enabled, `OlapScanNode.addScanRangeLocations` sets `TInternalScanRange.skip_page_cache` for each scan range sent to the backend, causing storage reads to skip the OS/page caching layer. Typical use cases: large one-time scans where avoiding page-cache pollution is desired or when a user prefers direct I/O semantics. Do not confuse with `skip_local_disk_cache`, which controls the storage-layer data cache; `fill_data_cache` can also influence caching behavior. * **Default**: `false` * **Data Type**: boolean * **Introduced in**: v3.3.9, v3.4.0, v3.5.0 ##### spill\_encode\_level[​](#spill_encode_level "Direct link to spill_encode_level") * **Scope**: Session * **Description**: Controls the encoding/compression behaviour applied to operator spill files. The integer is a bit-flag level whose meanings mirror `transmission_encode_level`: * bit 1 (value `1`) — enable adaptive encoding; * bit 2 (value `2`) — encode integer-like columns with streamvbyte; * bit 4 (value `4`) — compress binary/string columns with LZ4. Example semantics from the related `transmission_encode_level` comment: `7` enables adaptive encoding for numbers and strings; `6` forces encoding of numbers and strings. Changing this value adjusts CPU vs. disk I/O trade-offs for spills (higher encoding levels increase CPU work but reduce spill size / I/O). Implemented as the session variable annotated `SPILL_ENCODE_LEVEL` in `SessionVariable.java` (getter `getSpillEncodeLevel()`), and documented adjacent to other spill tunables such as `spill_mem_table_size`. * **Default**: `7` * **Data Type**: int * **Introduced in**: v3.2.0 ##### spill\_mode (3.0 and later)[​](#spill_mode-30-and-later "Direct link to spill_mode (3.0 and later)") The execution mode of intermediate result spilling. Valid values: * `auto`: Spilling is automatically triggered when the memory usage threshold is reached. * `force`: StarRocks forcibly executes spilling for all relevant operators, regardless of memory usage. This variable takes effect only when the variable `enable_spill` is set to `true`. ##### spill\_partitionwise\_agg[​](#spill_partitionwise_agg "Direct link to spill_partitionwise_agg") * **Description**: Session-level flag that enables partition-wise aggregation behavior when spill is used for aggregation operators. When `spill_partitionwise_agg` is true (and `enable_spill` is enabled), the execution engine will partition spilled aggregation data and perform per-partition spill/merge processing. The flag is propagated to execution via `TSpillOptions.setSpill_partitionwise_agg`. Related session settings that affect its behavior are `spill_partitionwise_agg_partition_num` (number of partitions created) and `spill_partitionwise_agg_skew_elimination` (skew handling). This option reduces peak memory usage for large-group aggregations by splitting work across partitions during spill, but may increase I/O and merge overhead. * **Scope**: Session * **Default**: `false` * **Data Type**: boolean * **Introduced in**: v3.5.2 ##### spill\_revocable\_max\_bytes[​](#spill_revocable_max_bytes "Direct link to spill_revocable_max_bytes") * **Scope**: Session * **Description**: Experimental per-session threshold (in bytes) for operator revocable memory. If an operator's revocable memory exceeds this value, the operator will initiate spilling to disk "as soon as possible" to free revocable memory. Use this to tune aggressive spilling for memory‑intensive operators; the value is interpreted in bytes. * **Default**: `0` * **Data Type**: long * **Introduced in**: v3.2.0 ##### spill\_storage\_volume[​](#spill_storage_volume "Direct link to spill_storage_volume") * **Description**: The storage volume with which you want to store the intermediate results of queries that triggered spilling. For more information, see [Spill to object storage](https://docs.starrocks.io/docs/administration/management/resource_management/spill_to_disk.md#preview-spill-intermediate-result-to-object-storage). * **Default**: Empty string * **Introduced in**: v3.3.0 ##### sql\_dialect[​](#sql_dialect "Direct link to sql_dialect") * **Description**: The SQL dialect that is used. For example, you can run the `set sql_dialect = 'trino';` command to set the SQL dialect to Trino, so you can use Trino-specific SQL syntax and functions in your queries. > **NOTICE** > > After you configure StarRocks to use the Trino dialect, identifiers in queries are not case-sensitive by default. Therefore, you must specify names in lowercase for your databases and tables at database and table creation. If you specify database and table names in uppercase, queries against these databases and tables will fail. * **Data type**: StarRocks * **Introduced in**: v3.0 ##### sql\_mode[​](#sql_mode "Direct link to sql_mode") Used to specify the SQL mode to accommodate certain SQL dialects. Valid values include: * `PIPES_AS_CONCAT`: The pipe symbol `|` is used to concatenate strings, for example, `select 'hello ' || 'world'`. * `ONLY_FULL_GROUP_BY` (Default): The SELECT LIST can only contain GROUP BY columns or aggregate functions. * `ALLOW_THROW_EXCEPTION`: returns an error instead of NULL when type conversion fails. * `FORBID_INVALID_DATE`: prohibits invalid dates. * `MODE_DOUBLE_LITERAL`: interprets floating-point types as DOUBLE rather than DECIMAL. * `SORT_NULLS_LAST`: places NULL values at the end after sorting. * `ERROR_IF_OVERFLOW`: returns an error instead of NULL in the case of arithmetic overflow. Currently, only the DECIMAL data type supports this option. * `GROUP_CONCAT_LEGACY`: uses the `group_concat` syntax of v2.5 and earlier. This option is supported from v3.0.9 and v3.1.6. * `STRUCT_CAST_BY_NAME`: enables name-based field matching when casting between STRUCT types, rather than the default position-based matching. When this mode is enabled, fields in the source struct are matched to fields in the target struct by field name (case-insensitively), regardless of the order in which they are declared. Fields present in the source but absent in the target are ignored; fields present in the target but absent in the source are filled with NULL. This mode affects both the FE type resolution (common supertype computation for UNION ALL and castability checks) and the BE cast evaluation (runtime field reordering in CastStructExpr). This is particularly useful when performing UNION ALL on STRUCT columns whose fields are defined in different orders across branches. You can set only one SQL mode, for example: ```sql set sql_mode = 'PIPES_AS_CONCAT'; ``` Or, you can set multiple modes at a time, for example: ```sql set sql_mode = 'PIPES_AS_CONCAT,ERROR_IF_OVERFLOW,GROUP_CONCAT_LEGACY'; ``` ##### sql\_safe\_updates[​](#sql_safe_updates "Direct link to sql_safe_updates") Used for MySQL client compatibility. No practical usage. ##### sql\_select\_limit[​](#sql_select_limit "Direct link to sql_select_limit") * **Description**: Used to limit the maximum number of rows returned by a query, which can prevent issues such as insufficient memory or network congestion caused by the query returning too much data. * **Default**: Unlimited * **Data type**: Long ##### storage\_engine[​](#storage_engine "Direct link to storage_engine") The types of engines supported by StarRocks: * olap (default): StarRocks system-owned engine. * mysql: MySQL external tables. * broker: Access external tables through a broker program. * elasticsearch or es: Elasticsearch external tables. * hive: Hive external tables. * iceberg: Iceberg external tables, supported from v2.1. * hudi: Hudi external tables, supported from v2.2. * jdbc: external table for JDBC-compatible databases, supported from v2.3. ##### streaming\_preaggregation\_mode[​](#streaming_preaggregation_mode "Direct link to streaming_preaggregation_mode") Used to specify the preaggregation mode for the first phase of GROUP BY. If the preaggregation effect in the first phase is not satisfactory, you can use the streaming mode, which performs simple data serialization before streaming data to the destination. Valid values: * `auto`: The system first tries local preaggregation. If the effect is not satisfactory, it switches to the streaming mode. This is the default value. * `force_preaggregation`: The system directly performs local preaggregation. * `force_streaming`: The system directly performs streaming. ##### system\_time\_zone[​](#system_time_zone "Direct link to system_time_zone") Used to display the time zone of the current system. Cannot be changed. ##### time\_zone[​](#time_zone "Direct link to time_zone") Used to set the time zone of the current session. The time zone can affect the results of certain time functions. ##### transaction\_isolation[​](#transaction_isolation "Direct link to transaction_isolation") * **Description**: Session-scoped compatibility variable that records the client-requested transaction isolation level using MySQL-style names. Declared in SessionVariable as `transactionIsolation` and annotated with `@VariableMgr.VarAttr(name = TRANSACTION_ISOLATION)`; its presence ensures compatibility with MySQL 5.8 clients. There is a related `tx_isolation` variable (kept for c3p0 client compatibility). The value is stored per-session to satisfy client/libraries, e.g., `REPEATABLE-READ`. Transaction isolation semantics in the engine are managed by StarRocks' transaction subsystem and may not be changed solely by modifying this session variable. * **Scope**: Session * **Default**: `REPEATABLE-READ` * **Data Type**: String * **Introduced in**: v3.2.0 ##### transaction\_read\_only[​](#transaction_read_only "Direct link to transaction_read_only") * **Description**: Used for MySQL 5.8 compatibility. The alias is `tx_read_only`. This variable specifies the transaction access mode. `ON` indicates read only and `OFF` indicates readable and writable. * **Default**: OFF * **Introduced in**: v2.5.18, v3.0.9, v3.1.7 ##### transmission\_compression\_type[​](#transmission_compression_type "Direct link to transmission_compression_type") * **Description**: Controls the compression algorithm used for transmitting query-related data (RPC/exchange payloads). Use `AUTO` to let the system pick a suitable algorithm based on environment (trade CPU for network bandwidth when beneficial). Other valid values are names recognized by `CompressionUtils.findTCompressionByName()` (for example, codec identifiers exposed by the runtime). For load-specific transmission you can use the separate `load_transmission_compression_type` session variable or supply transmission compression in stream-load parameters (HTTP header `HTTP_TRANSMISSION_COMPRESSION_TYPE` / thrift field `transmission_compression_type`). * **Scope**: Session * **Default**: `AUTO` * **Data Type**: String * **Introduced in**: v3.2.0 ##### tx\_isolation[​](#tx_isolation "Direct link to tx_isolation") Used for MySQL client compatibility. No practical usage. The alias is `transaction_isolation`. ##### tx\_visible\_wait\_timeout[​](#tx_visible_wait_timeout "Direct link to tx_visible_wait_timeout") * **Description**: Session-scoped timeout (in seconds) that controls how long the server waits for a committed transaction to become visible (published) before proceeding. If the visible wait expires, the transaction is treated as COMMITTED but not yet VISIBLE. Materialized view refresh logic (`MVTaskRunProcessor`) temporarily sets this variable to `Long.MAX_VALUE / 1000` to wait effectively indefinitely for visibility and restores the original value after refresh. When `enable_sync_publish` is set to `true`, this variable is ignored because the publish wait is derived from the job deadline instead. * **Scope**: Session * **Default**: `10` * **Data Type**: long * **Introduced in**: v3.2.0 ##### use\_page\_cache[​](#use_page_cache "Direct link to use_page_cache") * **Description**: Session-scoped boolean that controls whether a query should use the backend page cache. If not explicitly set by FE, the query follows the BE's page-cache policy; if set in the session, FE enforces the session value. The session variable is propagated to the execution layer (e.g., `tResult.setUse_page_cache`) so BE execution honors the decision. Commonly disabled (`false`) for internal/background jobs (statistics collection, hyper queries, online optimize) to avoid polluting the shared page cache with non-user data — see usages in `StatisticsCollectJob`, `HyperQueryJob`, and `OnlineOptimizeJobV2`. * **Scope**: Session * **Default**: `true` * **Data Type**: boolean * **Introduced in**: v3.2.0 ##### version (global)[​](#version-global "Direct link to version (global)") The MySQL server version returned to the client. The value is the same as FE parameter `mysql_server_version`. ##### version\_comment (global)[​](#version_comment-global "Direct link to version_comment (global)") The StarRocks version. Cannot be changed. ##### wait\_timeout[​](#wait_timeout "Direct link to wait_timeout") * **Description**: The number of seconds the server waits for activity on a non-interactive connection before closing it. If a client does not interact with StarRocks for this length of time, StarRocks will actively close the connection. * **Default**: 28800 (8 hours). * **Unit**: Second * **Data type**: Int --- ### User-defined variables This topic describes how to declare and use user-defined variables. StarRocks 2.5 and later versions support user-defined variables. These variables are used to store specific values that are referenced in later SQL statements, thereby simplifying the writing of SQL statements and avoiding duplicate computation. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * User-defined variables are variables, which can be created by the user and exist in the session. This means that no one can access user-defined variables that have been declared by another user, and when the session is closed these variables expire. * StarRocks does not support using the SHOW statement to display existing user-defined variables. * The following types of values cannot be declared as user-defined variables: BITMAP, HLL, and PERCENTILE. User-defined variables of the JSON type are converted to the STRING type for storage. * From v3.2.7 onwards, StarRocks supports the ARRAY type as a user-defined variable. #### Declare user-defined variables[​](#declare-user-defined-variables "Direct link to Declare user-defined variables") ##### Syntax[​](#syntax "Direct link to Syntax") ```plain SET @var_name = expr [, ...]; ``` > **NOTE** > > * All variables must be preceded by a single at sign (@). > * Multiple variables can be declared in the same SET statement and need to be separated with commas (`,`). > * You can declare the same variable multiple times. The newly declared value overwrites the original value. > * If an undeclared variable is used, the value of the variable is `NULL` by default, and the NULL type is STRING. ##### Parameters[​](#parameters "Direct link to Parameters") | **Parameter** | **Required** | **Description** | | ------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | var\_name | Yes | The name of the user-defined variable. The naming conventions are as follows:- The name can contain letters, digits (0-9), and underscores (`_`).
- The name cannot exceed 64 characters in length.You can declare a string as a user-defined variable, such as @'my-var', @"my-var", and @\`my-var\`. User-defined variables of the STRING type can contain characters other than letters, digits, and underscores (\_), such as periods (.). | | expr | Yes | The value of the user-defined variable. You can specify a number (such as 43) or a complex expression (such as the value returned by a SELECT statement) for this parameter. The data type of the variable is the same as the data type of the result returned by the expression. | ##### Examples[​](#examples "Direct link to Examples") Example 1: Declare a number as a user-defined variable. ```sql SET @var = 43; ``` Example 2: Declare the value returned by a SELECT query as a user-defined variable. ```sql SET @var = (SELECT SUM(v1) FROM test); ``` Example 3: Declare multiple user-defined variables in the same SET statement. ```sql SET @v1=1, @v2=2; ``` #### Use user-defined variables in SQL[​](#use-user-defined-variables-in-sql "Direct link to Use user-defined variables in SQL") * Simplifies the writing of SQL statements. For example, when you execute the following SELECT statement, StarRocks parses `@var` as `1`. ```plain SET @var = 1; SELECT @var, v1 from test; ``` * Avoids duplicate computation. For example, when you execute the following SELECT statement, StarRocks parses `@var` as the result returned by the `select sum(c1) from tbl` command. ```plain SET @var = (select sum(c1) from tbl); SELECT @var, v1 from test; ``` --- ## Starrocks Kubernetes Operator ### Releases of Kubernetes Operator for StarRocks #### Notifications[​](#notifications "Direct link to Notifications") The Operator provided by StarRocks is used to deploy StarRocks clusters in the Kubernetes environment. The StarRocks cluster components include FE, BE, and CN. **User guide:** You can use the following methods to deploy StarRocks clusters on Kubernetes: * [Directly use StarRocks CRD to deploy StarRocks clusters](https://docs.starrocks.io/docs/deployment/sr_operator/) * [Deploying both the Operator and StarRocks clusters by using the Helm Chart](https://docs.starrocks.io/zh/docs/deployment/helm/) **Source codes:** [starrocks-kubernetes-operator and kube-starrocks Helm Chart](https://github.com/StarRocks/starrocks-kubernetes-operator) **Download URL of the resources:** * **URL prefix** `https://github.com/StarRocks/starrocks-kubernetes-operator/releases/download/v${operator_version}/${resource_name}` * **Resource name** * StarRocksCluster CRD: `starrocks.com_starrocksclusters.yaml` * Default configuration file for StarRocks Operator: `operator.yaml` * Helm Chart, including `kube-starrocks` Chart `kube-starrocks-${chart_version}.tgz`. The `kube-starrocks` Chart is divided into two subcharts: `starrocks` Chart `starrocks-${chart_version}.tgz` and `operator` Chart `operator-${chart_version}.tgz`. For example, the download URL for kube-starrocks chart v1.8.6 is: `https://github.com/StarRocks/starrocks-kubernetes-operator/releases/download/v1.8.6/kube-starrocks-1.8.6.tgz` **Version requirements** * Kubernetes: 1.18 or later * Go: 1.19 or later #### Release notes[​](#release-notes "Direct link to Release notes") ##### 1.9[​](#19 "Direct link to 1.9") ###### 1.9.1[​](#191 "Direct link to 1.9.1") **Improvements** * **\[Helm Chart]** The operator will not create PersistentVolumeClaim (PVC) for log storage when `logStorageSize` is set to `0`. [#398](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/398) * **\[Operator]** The operator can detect whether the values of `mountPath` and `name` in `storageVolumes` are duplicated. An error is returned when duplicate values exist. [#388](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/388) * **\[Operator]** The number of FE nodes cannot be scaled down to 1. [#394](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/394) * **\[Operator]** The values of the `feEnvVars`, `beEnvVars` and `cnEnvVars` fields in multiple values YAML files can be merged. [#396](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/396) * **\[Operator]** Add `spec.containers.securityContext.capabilities` in the StarRocksCluster CRD to customize the Linux capabilities of containers. [#404](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/404) **Bug Fixes** Fixed the following issues: * **\[Operator]** The `annotations` field in `service` can be updated. [#402](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/402) [#399](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/399) * **\[Operator]** The modification of statefulset and deployment is patched instead of updated. It solves the problem that upgrading CN will cause all CN pods to be terminated and restarted when CN and HPA are enabled. [#397](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/397) * **\[Operator]** The modification of service object is patched instead of updated. It prevents the operator from overwriting the modification, for example, when a Kubernetes Cloud provider is used and the service object is modified by that Kubernetes Cloud provider. [#387](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/387) ###### 1.9.0[​](#190 "Direct link to 1.9.0") **New features** * Add StarRocksWarehouse CRD to support StarRocks Warehouse. Note that StarRocks Warehouse is currently a feature of the StarRocks Enterprise Edition. [#323](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/323) **Enhancements** * Add the `status.reason` field in StarRocksCluster CRD. When the apply operation of a subcontroller fails during cluster deployment, you can execute `kubectl get starrockscluster -oyaml`, and view error logs shown in `status.reason` field in the returned result. [#359](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/359) * An empty directory can be mounted in the `storageVolumes` field. [#324](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/324) **Bug Fixes** Fixed the following issues: * The status of the StarRocks cluster was inconsistent with the status of the cluster's FE, BE and CN components. [#380](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/380) * The HPA resource is not deleted when `autoScalingPolicy` is deleted. [#379](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/379) * The HPA resource is not removed when `starRocksCnSpec` is deleted. [#357](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/357) ##### 1.8[​](#18 "Direct link to 1.8") ###### 1.8.8[​](#188 "Direct link to 1.8.8") **Bugfixes** * **\[Operator]** The operator no longer annotates on search service (which is an internal service) when annotations are added by using `StarRocksFeSpec.service`, `StarRocksBeSpec.service`, and `StarRocksCnSpec.service`. [#370](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/370) ###### 1.8.7[​](#187 "Direct link to 1.8.7") **Enhancements** * Add the `livenessProbeFailureSeconds` and `readinessProbeFailureSeconds` fields in StarRocksCluster CRD. When StarRocks is under heavy workload and the time of liveness and readiness probes still use default values, the liveness and readiness probes may fail and cause the containers to restart. In this case, you can add larger values to these two fields. [#309](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/309) ###### 1.8.6[​](#186 "Direct link to 1.8.6") **Bug Fixes** Fixed the following issue: * An error `sendfile() failed (32: Broken pipe) while sending request to upstream` is returned during a Stream Load job. After Nginx sends the request body to FE, the FE then redirects the request to the BE. At this point, the data cached in Nginx may already be lost. [#303](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/303) **Doc** * [Load data from outside the Kubernetes network to StarRocks through FE proxy](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/load_data_using_stream_load_howto.md) * [Update the root user's password using Helm](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/change_root_password_howto.md) ###### 1.8.5[​](#185 "Direct link to 1.8.5") **Improvements** * **\[Helm Chart] The `annotations` and `labels` can be customized for the service account of the operator**: The Operator creates a service account named `starrocks` by default, and users can customize the annotations and labels for the service account `starrocks` of the operator by specifying the `annotations` and `labels` fields in `serviceAccount` in **values.yaml**. The `operator.global.rbac.serviceAccountName` field is deprecated. [#291](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/291) * **\[Operator] FE service supports explicit protocol selection for Istio**: When Istio is installed in the Kubernetes environment, Istio needs to determine the protocol of the traffic from the StarRocks cluster, in order to provide additional functionality such as routing and rich metrics. So FE service explicitly defines its protocol as MySQL in the `appProtocol` field. This improvement is particularly important because the MySQL protocol is a server-first protocol that is incompatible with automatic protocol detection and sometimes may incur connection failures. [#288](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/288) **Bug Fixes** Fixed the following issue: * **\[Helm Chart]** The root user's password in StarRocks may not be initialized successfully when `starrocks.initPassword.enabled` is true and the value of `starrocks.starrocksCluster.name` is specified. It is caused by the wrong FE service domain name used by the initpwd pod to connect FE service. More specifically, in this scenario, FE service domain name uses the value specified in `starrocks.starrocksCluster.name`, while the initpwd pod still uses the value of the `starrocks.nameOverride` field to form the FE service domain name. ([#292](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/292)) **Upgrade notes** * **\[Helm Chart]** When the value specified in `starrocks.starrocksCluster.name` is different from the value of `starrocks.nameOverride`, the old configmaps for FE, BE, and CN will be deleted. New configmaps with new names for the FE, BE, and CN will be created. **This may result in the restart of the FE, BE, and CN pods.** ###### 1.8.4[​](#184 "Direct link to 1.8.4") **Features** * **\[Helm Chart]** The metrics of StarRocks clusters can be monitored by using the Prometheus and ServiceMonitor CR. For the user guide, see [Integration with Prometheus and Grafana](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/integration/integration-prometheus-grafana.md). [#284](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/284) * **\[Helm Chart]** Add the `storagespec` and more fields in `starrocksCnSpec` in **values.yaml** to configure the log volume for CN nodes in a StarRocks Cluster. [#280](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/280) * Add the `terminationGracePeriodSeconds` in the StarRocksCluster CRD to configure how long to wait before forcefully terminating a pod when a StarRocksCluster resource is being deleted or updated. [#283](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/283) * Add the `startupProbeFailureSeconds` field in the StarRocksCluster CRD to configure the startup probe failure threshold for the pods in the StarRocksCluster resource. [#271](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/271) **Bug Fixes** Fixed the following issue: * The FE Proxy cannot handle STREAM LOAD requests correctly when multiple FE pods exist in the StarRocks cluster. [#269](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/269) **Doc** * [Add a quick start on how to deploy a local StarRocks cluster](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/local_installation_how_to.md). * Add more user guides on how to deploy a StarRocks cluster with different configurations. For example, how to [deploy a StarRocks cluster with all supported features](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/examples/starrocks/deploy_a_starrocks_cluster_with_all_features.yaml). For more user guides, see [docs](https://github.com/StarRocks/starrocks-kubernetes-operator/tree/main/examples/starrocks). * Add more user guides on how to manage the StarRocks cluster. For example, how to configure [logging and related fields](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/logging_and_related_configurations_howto.md) and [mount external configmaps or secrets](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/mount_external_configmaps_or_secrets_howto.md). For more user guides, see [docs](https://github.com/StarRocks/starrocks-kubernetes-operator/tree/main/doc). ###### 1.8.3[​](#183 "Direct link to 1.8.3") **Upgrade notes** * **\[Helm Chart]** Add `JAVA_OPTS_FOR_JDK_11` to the default **fe.conf** file. When the default **fe.conf** file is used and the helm chart is upgraded to v1.8.3, **FE pods may restart**. [#257](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/257) **Features** * **\[Helm Chart]** Add the `watchNamespace` field to specify the one and only namespace that the operator needs to watch. Otherwise, the operator watches all namespaces in the Kubernetes cluster. In most cases, you do not need to use this feature. You can use this feature when the Kubernetes cluster manages too many nodes, the operator watches all namespaces and consumes too many memory resources. [#261](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/261) * **\[Helm Chart]** Add the `Ports` field in `starrocksFeProxySpec` in the **values.yaml** file to allow users to specify the NodePort of FE Proxy service. [#258](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/258) **Improvements** * The value of the `proxy_read_timeout` parameter is changed in the **nginx.conf** file to 600s from 60s, in order to avoid timeout. ###### 1.8.2[​](#182 "Direct link to 1.8.2") **Improvements** * Increase the maximum memory usage allowed for the operator pods to avoid OOM. [#254](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/254) ###### 1.8.1[​](#181 "Direct link to 1.8.1") **Features** * Support using [the subpath field in the configMaps and secrets](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/mount_external_configmaps_or_secrets_howto.md#3-mount-configmaps-to-a-subpath-by-helm-chart), allowing users to mount specific files or directories from these resources. [#249](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/249) * Add the `ports` field in the StarRocks cluster CRD to allow users to customize the ports of the services. [#244](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/244) **Improvements** * Remove the related Kubernetes resources when the `BeSpec` or `CnSpec` of the StarRocks cluster is deleted, ensuring a clean and consistent state of the cluster. [#245](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/245) ###### 1.8.0[​](#180 "Direct link to 1.8.0") **Upgrade notes and behavior changes** * **\[Operator]** To upgrade the StarRocksCluster CRD and operator you need to manually apply the new StarRocksCluster CRD **starrocks.com\_starrocksclusters.yaml** and **operator.yaml.** * **\[Helm Chart]** * To upgrade the Helm Chart, you need to perform the following: 1. Use the **values migration tool** to adjust the format of the previous **values.yaml** file to the new format. The values migration tool for different operating systems can be downloaded from the [Assets](https://github.com/StarRocks/starrocks-kubernetes-operator/releases/tag/v1.8.0) section. You can get help information of this tool by running the `migrate-chart-value --help` command. [#206](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/206) ```bash migrate-chart-value --input values.yaml --target-version v1.8.0 --output ./values-v1.8.0.yaml ``` 2. Update the Helm Chart repo. ```bash helm repo update ``` 3. Execute the `helm upgrade` command to apply the adjusted **values.yaml** file to the StarRocks helm chart kube-starrocks. ```bash helm upgrade starrocks/kube-starrocks -f values-v1.8.0.yaml ``` * Two subcharts, [operator](https://github.com/StarRocks/starrocks-kubernetes-operator/tree/main/helm-charts/charts/kube-starrocks/charts/operator) and [starrocks](https://github.com/StarRocks/starrocks-kubernetes-operator/tree/main/helm-charts/charts/kube-starrocks/charts/starrocks), are added into the kube-starrocks helm chart. You can choose to install StarRocks operator or StarRocks cluster respectively by specifying the corresponding subchart. This way, you can manage StarRocks clusters more flexibly, such as deploying one StarRocks operator and multiple StarRocks clusters. **Features** * **\[Helm Chart] Multiple StarRocks clusters in a Kubernetes cluster**. Support deploying multiple StarRocks clusters in different namespaces in a Kubernetes cluster by installing the `starrocks` Helm subchart. [#199](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/199) * **\[Helm Chart]** Support to [configure the initial password of StarRocks cluster's root users](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/initialize_root_password_howto.md) when executing the `helm install` command. Note that, the `helm upgrade` command does not support this feature. * **\[Helm Chart] Integration with Datadog:** Integrated with Datadog to collect StarRocks clusters' metrics and logs. To enable this feature, you need to configure the Datadog related fields in the **values.yaml** file. For the detailed user guide, see [Integration with Datadog](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/integration/integration-with-datadog.md). [#197](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/197) [#208](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/208) * **\[Operator] Run pods as a non-root user**. Add the runAsNonRoot field to allow pods to run as non-root users, which can enhance security. [#195](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/195) * **\[Operator] FE proxy.** Add the FE proxy to allow external clients and data load tools that support Stream Load protocol to access StarRocks clusters in Kubernetes. This way, you can use the load job based on Stream Load to load data into StarRocks clusters in Kubernetes. [#211](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/211) **Improvements** * Add the `subpath` field in StarRocksCluster CRD. [#212](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/212) * Increase the disk size allowed for the FE metadata. The FE container stops running when the available disk space that can be provisioned to store the FE metadata is less than that default value. [#210](https://github.com/StarRocks/starrocks-kubernetes-operator/pull/210) --- ## Table_design ### Catalogs, databases, and tables StarRocks uses internal catalogs to manage internal data and uses external catalogs to access external data in data lake. The data stored in StarRocks is maintained inside an internal catalog which consists of one or more databases. A database manages various objects, including tables, materialized views, and views, to store, manage, and manipulate data. StarRocks adopts a privilege system that controls users' privileges to perform operations on objects, thereby enhancing data security. ![img](/assets/images/Catalog_db_tbl-e7e05f2c4430964e96826f47a4a6224e.png) #### [Catalogs](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md)[​](#catalogs "Direct link to catalogs") Catalogs are categorized into internal catalogs and external catalogs. Each cluster has only one Internal catalog `default_catalog` that can include one or more databases. The internal catalog of a StarRocks cluster manages the data loaded into the cluster and the materialized views. StarRocks can well serve as a data warehouse to store data, which significantly enhances query performance, especially for complex query analytics on large-scale data. An external catalog, on the other hand, allows access to external data in your data lake. You can use StarRocks as a query engine to directly query data in the lake without loading the data into StarRocks. #### Databases[​](#databases "Direct link to Databases") A database is a collection of data objects, such as tables, views, and materialized views, to store, manage, and manipulate data. #### [Tables](https://docs.starrocks.io/docs/table_design/table_types.md)[​](#tables "Direct link to tables") Tables are categorized into internal tables and external tables. **Internal tables** Internal tables are maintained in internal catalogs. The data of internal tables is stored in StarRocks. Internal tables are maintained in databases under the internal catalog. An internal table consists of rows and columns, where each row represents a record and each column represents an attribute or field of a data row. note The rows and columns of internal tables here are logical concepts. StarRocks uses columnar storage for tables. Physically, the data in a column is segregated into data blocks, encoded, compressed, and then persistently stored on disk. StarRocks provides four types of internal tables, Primary Key tables, Duplicate Key tables, Aggregate tables, and Unique Key tables, to store various data, such as raw logs, realtime data, and aggregated data, to meet your varying business needs. An internal table adopts a two-tier data distribution strategy, that is partitioning + bucketing, to achieve even data distribution. What's more, the replicas of each bucket are evenly distributed across BEs to guarantee high data availability. **External tables** External tables are maintained in external catalogs. The data of external tables is actually stored in external data sources. StarRocks only stores the metadata of these external tables, and you can use these tables to query data from external data sources. #### [Materialized views](https://docs.starrocks.io/docs/using_starrocks/async_mv/Materialized_view.md)[​](#materialized-views "Direct link to materialized-views") Materialized views are special physical tables that hold precomputed query results from one or more base tables. Materialized views are categorized as synchronous and asynchronous materialized views. Asynchronous materialized views are more powerful because they can store precomputed results from multiple base tables (internal and external tables) and support various aggregation operators. When complex queries are executed on base tables, StarRocks can automatically reuse precomputed results stored in materialized views, to achieve transparent query acceleration, accelerate queries in data lakes and establish data modeling. #### [Views](https://docs.starrocks.io/docs/sql-reference/sql-statements/View/CREATE_VIEW.md)[​](#views "Direct link to views") Views, or logical views, are virtual tables that do not hold any data. Views only display data that is the results of queries against other existing physical tables. The query that defines a view is run every time the view is referenced in a query. #### [Privileges](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md)[​](#privileges "Direct link to privileges") Privileges determine which users can perform which operations on which objects. StarRocks adopts two types of privilege models: identity-based access control and role-based access control. You can first assign privileges to roles, and then assign roles to users. In this case, privileges are passed to users through roles. Or, you can directly assign privileges to user identities. #### Data storage in storage-compute separation architecture[​](#data-storage-in-storage-compute-separation-architecture "Direct link to Data storage in storage-compute separation architecture") Since v3.0, StarRocks introduces the new storage-compute separation (shared-data) architecture. Data storage is separated from BEs. Data is persistently stored in remote object storage or HDFS, while local disks are used for caching hot data to accelerate queries. --- ### Data compression StarRocks supports data compression for table and index storage. Data compression not only helps save storage space, but also improves performance of I/O intensive tasks because StarRocks can read fewer pages from disk for each request. Note that extra CPU resource is required to compress and decompress the data. #### Choose a data compression algorithm[​](#choose-a-data-compression-algorithm "Direct link to Choose a data compression algorithm") StarRocks supports four data compression algorithms: LZ4, Zstandard (or zstd), zlib, and Snappy. These data compression algorithms differ in compression ratio and compression/decompression performance. Generally, the compression ratios of these algorithms rank as follows: zlib > Zstandard > LZ4 > Snappy. Among them, zlib has shown relatively high compression ratios. As a result of the highly compressed data, the loading and query performances on tables with the zlib compression algorithm are also affected. LZ4 and Zstandard, especially, have well-balanced compression ratios and decompression performances. You can choose among these compression algorithms to cater to your business needs for less storage space or better performance. We recommend LZ4 or Zstandard if you do not have specific demands for less storage space. > **NOTE** > > Different data types can affect the compression ratio. #### Specify a data compression algorithm for a table[​](#specify-a-data-compression-algorithm-for-a-table "Direct link to Specify a data compression algorithm for a table") You can specify a data compression algorithm for a table only when creating the table, and you cannot change it afterwards. The following example creates a table `data_compression` with the Zstandard algorithm. For detailed instructions, see [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md). ```sql CREATE TABLE `data_compression` ( `id` INT(11) NOT NULL COMMENT "", `name` CHAR(200) NULL COMMENT "" ) ENGINE=OLAP UNIQUE KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`) PROPERTIES ( "compression" = "ZSTD" ); ``` > **NOTE** > > If no data compression algorithm is specified, StarRocks uses LZ4 by default. You can view the compression algorithm of a table using [SHOW CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_CREATE_TABLE.md). --- ### Data distribution Configuring appropriate partitioning and bucketing at table creation can help to achieve even data distribution. Even data distribution means dividing the data into subsets according to certain rules and distributing them evenly across different nodes. It can also reduce the amount of data scanned and make full use of the cluster's parallel processing capability, thereby improving query performance. > **NOTE** > > * After the data distribution is specified at table creation and query patterns or data characteristics in the business scenario evolves, since v3.2 StarRocks supports [modifying certain data distribution-related properties after table creation](#optimize-data-distribution-after-table-creation-since-32) to meet the requirements for query performance in the latest business scenarios. > * Since v3.1, you do not need to specify the bucketing key in the DISTRIBUTED BY clause when creating a table or adding a partition. StarRocks supports random bucketing, which randomly distributes data across all buckets. For more information, see [Random bucketing](#random-bucketing-since-v31). > * Since v2.5.7, you can choose not to manually set the number of buckets when you create a table or add a partition. StarRocks can automatically set the number of buckets (BUCKETS). However, if the performance does not meet your expectations after StarRocks automatically sets the number of buckets, and you are familiar with the bucketing mechanism, you can still [manually set the number of buckets](#set-the-number-of-buckets). #### Introduction[​](#introduction "Direct link to Introduction") ##### Distribution methods in general[​](#distribution-methods-in-general "Direct link to Distribution methods in general") Modern distributed database systems generally use the following basic distribution methods: Round-Robin, Range, List, and Hash. ![Data distribution method](/assets/images/3.3.2-1-9995f2cd30ed44b4bf771b5c5aea62f8.png) * **Round-Robin**: distributes data across different nodes in a cyclic. * **Range**: distributes data across different nodes based on the ranges of partitioning column values. As shown in the diagram, the ranges \[1-3] and \[4-6] correspond to different nodes. * **List**: distributes data across different nodes based on the discrete values of partitioning columns, such as gender and province. Each discrete value is mapped to a node, and multiple different values might be mapped to the same node. * **Hash**: distributes data across different nodes based on a hash function. To achieve more flexible data partitioning, in addition to using one of the above data distribution methods, you can also combine these methods based on specific business requirements. Common combinations include Hash+Hash, Range+Hash, and Hash+List. ##### Distribution methods in StarRocks[​](#distribution-methods-in-starrocks "Direct link to Distribution methods in StarRocks") StarRocks supports both separate and composite use of data distribution methods. > **NOTE** > > In addition to the general distribution methods, StarRocks also supports Random distribution to simplify bucketing configuration. Also, StarRocks distributes data by implementing the two-level partitioning + bucketing method. * The first level is partitioning: Data within a table can be partitioned. Supported partitioning methods are expression partitioning, range partitioning, and list partitioning. Or you can choose not to use partitioning (the entire table is regarded as one partition). * The second level is bucketing: Data in a partition needs to be further distributed into smaller buckets. Supported bucketing methods are hash and random bucketing. | **Distribution method** | **Partitioning and bucketing method** | **Description** | | ------------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Random distribution | Random bucketing | The entire table is considered a partition. The data in the table is randomly distributed into different buckets. This is the default data distribution method. | | Hash distribution | Hash bucketing | The entire table is considered a partition. The data in the table is distributed to the corresponding buckets, which is based on the hash values of the data's bucketing key by using a hash function. | | Range distribution | Range-based bucketing | Supported from v4.1 onwards. The entire table is considered a partition. The data in the table is distributed to the corresponding buckets, which is based on the value range of the table key. | | Range+Random distribution | 1. Range partitioning
2. Random bucketing | 1. The data in the table is distributed to the corresponding partitions, which is based on the ranges where partitioning column values fall in.
2. The data in the partition is randomly distributed across different buckets. | | Range+Hash distribution | 1. Range partitioning
2. Hash bucketing | 1. The data in the table is distributed to the corresponding partitions, which is based on the ranges where partitioning column values fall in.
2. The data in the partition is distributed to the corresponding buckets, which is based on the hash values of the data's bucketing key by using a hash function. | | List+Random distribution | 1. Expression partitioning or list partitioning
2. Random bucketing | 1. The data in the table is partitioned based on the value lists that the partitioning columns values belongs to.
2. The data in the partition is randomly distributed across different buckets. | | List+Hash distribution | 1. Expression partitioning or List partitioning
2. Hash bucketing | 1. The data in the table is partitioned based on the value lists that the partitioning columns values belongs to.
2. The data in the partition is distributed to the corresponding buckets, which is based on the hash values of the data's bucketing key by using a hash function. | * **Random distribution** If you do not configure partitioning and bucketing methods at table creation, random distribution is used by default. This distribution method currently can only be used to create a Duplicate Key table. ```sql CREATE TABLE site_access1 ( event_day DATE, site_id INT DEFAULT '10', pv BIGINT DEFAULT '0' , city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '' ) DUPLICATE KEY (event_day,site_id,pv); -- Because the partitioning and bucketing methods are not configured, random distribution is used by default. ``` * **Hash distribution** ```sql CREATE TABLE site_access2 ( event_day DATE, site_id INT DEFAULT '10', city_code SMALLINT, user_name VARCHAR(32) DEFAULT '', pv BIGINT SUM DEFAULT '0' ) AGGREGATE KEY (event_day, site_id, city_code, user_name) -- Use hash bucketing as the bucketing method and must specify the bucketing key. DISTRIBUTED BY HASH(event_day,site_id); ``` * **Range distribution** From v4.1 onwards, StarRocks supports the **Range-based Distribution semantic** (disabled by default), controlled by the FE configuration `enable_range_distribution`. The data will be sequenced according to the data range of the key columns, and each tablet contains the data from a certain range. note To enable this semantic, you must set the FE configuration `enable_range_distribution` to `true`. ```sql CREATE TABLE users ( user_id bigint NOT NULL, name string NOT NULL, email string NULL, address string NULL ) PRIMARY KEY (user_id); -- Because the bucketing method is not configured, while the key column is specified, range-based bucketing is used. ``` * **Range+Random distribution** (This distribution method currently can only be used to create a Duplicate Key table.) ```sql CREATE TABLE site_access3 ( event_day DATE, site_id INT DEFAULT '10', pv BIGINT DEFAULT '0' , city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '' ) DUPLICATE KEY(event_day,site_id,pv) -- Use expression partitioning as the partitioning method and configure a time function expression. -- You can also use range partitioning. PARTITION BY date_trunc('day', event_day); -- Because the bucketing method is not configured, random bucketing is used by default. ``` * **Range+Hash distribution** ```sql CREATE TABLE site_access4 ( event_day DATE, site_id INT DEFAULT '10', city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '', pv BIGINT SUM DEFAULT '0' ) AGGREGATE KEY(event_day, site_id, city_code, user_name) -- Use expression partitioning as the partitioning method and configure a time function expression. -- You can also use range partitioning. PARTITION BY date_trunc('day', event_day) -- Use hash bucketing as the bucketing method and must specify the bucketing key. DISTRIBUTED BY HASH(event_day, site_id); ``` * **List+Random distribution** (This distribution method currently can only be used to create a Duplicate Key table.) ```sql CREATE TABLE t_recharge_detail1 ( id bigint, user_id bigint, recharge_money decimal(32,2), city varchar(20) not null, dt date not null ) DUPLICATE KEY(id) -- Use expression partitioning as the partitioning method and specify the partitioning column. -- You can also use list partitioning. PARTITION BY (city); -- Because the bucketing method is not configured, random bucketing is used by default. ``` * **List+Hash distribution** ```sql CREATE TABLE t_recharge_detail2 ( id bigint, user_id bigint, recharge_money decimal(32,2), city varchar(20) not null, dt date not null ) DUPLICATE KEY(id) -- Use expression partitioning as the partitioning method and specify the partitioning column. -- You can also use list partitioning. PARTITION BY (city) -- Use hash bucketing as the bucketing method and must specify the bucketing key. DISTRIBUTED BY HASH(city,id); ``` ###### Partitioning[​](#partitioning "Direct link to Partitioning") The partitioning method divides a table into multiple partitions. Partitioning primarily is used to split a table into different management units (partitions) based on the partitioning key. You can set a storage strategy for each partition, including the number of buckets, the strategy of storing hot and cold data, the type of storage medium, and the number of replicas. StarRocks allows you to use different types of storage mediums within a cluster. For example, you can store the latest data on solid-state drives (SSDs) to improve query performance, and historical data on SATA hard drives to reduce storage costs. | **Partitioning method** | **Scenarios** | **Methods to create partitions** | | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | | Expression partitioning (recommended) | Previously known as automatic partitioning. This partitioning method is more flexible and easy-to-use. It is suitable for most scenarios including querying and managing data based on continuous date ranges or enum values. | Automatically created during data loading | | Range partitioning (Legacy) | The typical scenario is to store simple, ordered data that is often queried and managed based on continuous date/numeric ranges. For instance, in some special cases, historical data needs to be partitioned by month, while recent data needs to be partitioned by day. | Created manually, dynamically, or in batch | | List partitioning (Legacy) | A typical scenario is to query and manage data based on enum values, and a partition needs to include data with different values for each partitioning column. For example, if you frequently query and manage data based on countries and cities, you can use this method and select `city` as the partitioning column. So a partition can store data for multiple cities belonging to the same country. | Created manually | ###### How to choose partitioning columns and granularity[​](#how-to-choose-partitioning-columns-and-granularity "Direct link to How to choose partitioning columns and granularity") * **The partitioning key is composed of one or more partitioning columns**. Selecting a proper partitioning column can effectively reduce the amount of data scanned during queries. In most business systems, partitioning based on time is commonly adopted to resolve certain issues caused by the deletion of expired data and facilitate the management of tiered storage of hot and cold data. In this case, you can use expression partitioning or range partitioning and specify a time column as the partitioning column. Additionally, if the data is frequently queried and managed based on ENUM values, you can use expression partitioning or list partitioning and specify a column including these values as the partitioning column. * When choosing the partitioning granularity, you need to consider data volume, query patterns, and data management granularity. * Example 1: If the monthly data volume in a table is small, partitioning by month can reduce the amount of metadata compared to partitioning by day, thereby reducing the resource consumption of metadata management and scheduling. * Example 2: If the monthly data volume in a table is large and queries mostly request data of certain days, partitioning by day can effectively reduce the amount of data scanned during queries. * Example 3: If the data needs to expire on a daily basis, partitioning by day is recommended. ###### Bucketing[​](#bucketing "Direct link to Bucketing") The bucketing method divides a partition into multiple buckets. Data in a bucket is referred to as a tablet. The supported bucketing methods are [random bucketing](#random-bucketing-since-v31) (from v3.1), [hash bucketing](#hash-bucketing), and [range-based bucketing](#range-based-bucketing) (from v4.1). * Random bucketing: When creating a table or adding partitions, you do not need to set a bucketing key. Data within a partition is randomly distributed into different buckets. * Hash Bucketing: When creating a table or adding partitions, you need to specify a bucketing key. Data within the same partition is divided into buckets based on the values of the bucketing key, and rows with the same value in the bucketing key are distributed to the corresponding and unique bucket. * Range-based bucketing: From v4.1, if the FE configuration `enable_range_distribution` is enabled, a table that specifies an explicit key type or an `ORDER BY` clause but no `DISTRIBUTED BY` clause distributes data within a partition by the range of those columns. Tablets can be automatically split or merged to mitigate data skew. This semantic is disabled by default. The number of buckets: By default, StarRocks automatically sets the number of buckets (from v2.5.7). You can also manually set the number of buckets. For more information, please refer to [determining the number of buckets](#set-the-number-of-buckets). #### Create and manage partitions[​](#create-and-manage-partitions "Direct link to Create and manage partitions") ##### Create partitions[​](#create-partitions "Direct link to Create partitions") ###### Expression partitioning (recommended)[​](#expression-partitioning-recommended "Direct link to Expression partitioning (recommended)") > **NOTICE** > > StarRocks's shared-data mode supports the time function expression from v3.1.0 and the column expression from v3.1.1. Since v3.0, StarRocks has supported [expression partitioning](https://docs.starrocks.io/docs/table_design/data_distribution/expression_partitioning.md) (previously known as automatic partitioning) which is more flexible and easy to use. This partitioning method is suitable for most scenarios such as querying and managing data based on continuous date ranges or ENUM values. You only need to configure a partition expression at table creation, and StarRocks will automatically create partitions during data loading. You no longer need to manually create numerous partitions in advance, nor configure dynamic partition properties. From v3.4 onwards, expression partitioning is further optimized to unify all partitioning strategies and supported more complex solutions. It is recommended in most cases, and will replace the other partitioning strategies in future releases. Example 1: Use a simple time function expression with a DATETIME column. ```sql CREATE TABLE site_access( event_day DATETIME NOT NULL, site_id INT DEFAULT '10', city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '', pv BIGINT DEFAULT '0' ) DUPLICATE KEY(event_day, site_id, city_code, user_name) PARTITION BY time_slice(event_day, INTERVAL 7 day) DISTRIBUTED BY HASH(event_day, site_id) ``` Example 2: Use a column expression with multiple columns. ```sql CREATE TABLE t_recharge_detail1 ( id bigint, user_id bigint, recharge_money decimal(32,2), city varchar(20) not null, dt varchar(20) not null ) DUPLICATE KEY(id) PARTITION BY dt,city DISTRIBUTED BY HASH(`id`); ``` Example 3: Use a complex time function expression with a Unix timestamp column. ```sql CREATE TABLE orders ( ts BIGINT NOT NULL, id BIGINT NOT NULL, city STRING NOT NULL ) PARTITION BY from_unixtime(ts,'%Y%m%d'); ``` Example 4: Use a mixed expression of time function expression and column expression. ```sql CREATE TABLE orders ( ts BIGINT NOT NULL, id BIGINT NOT NULL, city STRING NOT NULL ) PARTITION BY from_unixtime(ts,'%Y%m%d'), city; ``` ###### Range partitioning[​](#range-partitioning "Direct link to Range partitioning") [Range partitioning](https://docs.starrocks.io/docs/table_design/data_distribution/dynamic_partitioning.md) is suitable for storing simple contiguous data, such as time series data, or continuous numerical data. ###### Manually create partitions[​](#manually-create-partitions "Direct link to Manually create partitions") Define the mapping relationship between each partition and the range of partitioning column values. * **The partitioning column is of date type.** ```sql CREATE TABLE site_access( event_day DATE, site_id INT, city_code VARCHAR(100), user_name VARCHAR(32), pv BIGINT SUM DEFAULT '0' ) AGGREGATE KEY(event_day, site_id, city_code, user_name) PARTITION BY RANGE(event_day)( PARTITION p1 VALUES LESS THAN ("2020-01-31"), PARTITION p2 VALUES LESS THAN ("2020-02-29"), PARTITION p3 VALUES LESS THAN ("2020-03-31") ) DISTRIBUTED BY HASH(site_id); ``` * **The partitioning column is of integer type.** ```sql CREATE TABLE site_access( datekey INT, site_id INT, city_code SMALLINT, user_name VARCHAR(32), pv BIGINT SUM DEFAULT '0' ) AGGREGATE KEY(datekey, site_id, city_code, user_name) PARTITION BY RANGE (datekey) ( PARTITION p1 VALUES LESS THAN ("20200131"), PARTITION p2 VALUES LESS THAN ("20200229"), PARTITION p3 VALUES LESS THAN ("20200331") ) DISTRIBUTED BY HASH(site_id); ``` * **Three specific time functions can be used as partitioning columns (supported since v3.3.0).** When explicitly defining the mapping relationship between partitions and the ranges of partition column values, you can use a specific time function to convert the partition column values of timestamps or strings into date values, and then divide the partitions based on the converted date values. * The partition column values are timestamps * The partition column values are strings ```sql -- A 10-digit timestamp accurate to the second, for example, 1703832553. CREATE TABLE site_access( event_time bigint, site_id INT, city_code SMALLINT, user_name VARCHAR(32), pv BIGINT SUM DEFAULT '0' ) AGGREGATE KEY(event_time, site_id, city_code, user_name) PARTITION BY RANGE(from_unixtime(event_time)) ( PARTITION p1 VALUES LESS THAN ("2021-01-01"), PARTITION p2 VALUES LESS THAN ("2021-01-02"), PARTITION p3 VALUES LESS THAN ("2021-01-03") ) DISTRIBUTED BY HASH(site_id) ; -- A 13-digit timestamp accurate to the millisecond, for example, 1703832553219. CREATE TABLE site_access( event_time bigint, site_id INT, city_code SMALLINT, user_name VARCHAR(32), pv BIGINT SUM DEFAULT '0' ) AGGREGATE KEY(event_time, site_id, city_code, user_name) PARTITION BY RANGE(from_unixtime_ms(event_time))( PARTITION p1 VALUES LESS THAN ("2021-01-01"), PARTITION p2 VALUES LESS THAN ("2021-01-02"), PARTITION p3 VALUES LESS THAN ("2021-01-03") ) DISTRIBUTED BY HASH(site_id); ``` ```sql CREATE TABLE site_access ( event_time varchar(100), site_id INT, city_code SMALLINT, user_name VARCHAR(32), pv BIGINT SUM DEFAULT '0' ) AGGREGATE KEY(event_time, site_id, city_code, user_name) PARTITION BY RANGE(str2date(event_time, '%Y-%m-%d'))( PARTITION p1 VALUES LESS THAN ("2021-01-01"), PARTITION p2 VALUES LESS THAN ("2021-01-02"), PARTITION p3 VALUES LESS THAN ("2021-01-03") ) DISTRIBUTED BY HASH(site_id); ``` ###### Dynamic partitioning[​](#dynamic-partitioning "Direct link to Dynamic partitioning") [Dynamic partitioning](https://docs.starrocks.io/docs/table_design/data_distribution/dynamic_partitioning.md#dynamic-partitioning) related properties are configured at table creation. StarRocks automatically creates new partitions in advance and removes expired partitions to ensure data freshness, which implements time-to-live (TTL) management for partitions. Different from the automatic partition creation ability provided by the expression partitioning, dynamic partitioning can only periodically create new partitions based on the properties. If the new data does not belong to these partitions, an error is returned for the load job. However, the automatic partition creation ability provided by the expression partitioning can always create corresponding new partitions based on the loaded data. ###### Create multiple partitions in batch[​](#create-multiple-partitions-in-batch "Direct link to Create multiple partitions in batch") Multiple partitions can be created in batch at and after table creation. You can specify the start and end time for all the partitions created in batch in `START()` and `END()` and the partition increment value in `EVERY()`. However, note that the range of partitions is left-closed and right-open, which includes the start time but does not include the end time. The naming rule for partitions is the same as that of dynamic partitioning. * **The partitioning column is of date type.** When the partitioning column is of date type, at table creation, you can use `START()` and `END()` to specify the start date and end date for all the partitions created in batch, and `EVERY(INTERVAL xxx)` to specify the incremental interval between two partitions. Currently, the interval granularity supports `HOUR` (since v3.0), `DAY`, `WEEK`, `MONTH`, and `YEAR`. * with the same date interval * with different date intervals In the following example, the partitions created in a batch start from `2021-01-01` and end on `2021-01-04`, with a partition increment of one day: ```sql CREATE TABLE site_access ( datekey DATE, site_id INT, city_code SMALLINT, user_name VARCHAR(32), pv BIGINT DEFAULT '0' ) DUPLICATE KEY(datekey, site_id, city_code, user_name) PARTITION BY RANGE (datekey) ( START ("2021-01-01") END ("2021-01-04") EVERY (INTERVAL 1 DAY) ) DISTRIBUTED BY HASH(site_id); ``` It is equivalent to using the following `PARTITION BY` clause in the CREATE TABLE statement: ```sql PARTITION BY RANGE (datekey) ( PARTITION p20210101 VALUES [('2021-01-01'), ('2021-01-02')), PARTITION p20210102 VALUES [('2021-01-02'), ('2021-01-03')), PARTITION p20210103 VALUES [('2021-01-03'), ('2021-01-04')) ) ``` You can create batches of date partitions with different incremental intervals by specifying different incremental intervals in `EVERY` for each batch of partitions (make sure that the partition ranges between different batches do not overlap). Partitions in each batch are created according to the `START (xxx) END (xxx) EVERY (xxx)` clause. For example: ```sql CREATE TABLE site_access ( datekey DATE, site_id INT, city_code SMALLINT, user_name VARCHAR(32), pv BIGINT DEFAULT '0' ) DUPLICATE KEY(datekey, site_id, city_code, user_name) PARTITION BY RANGE (datekey) ( START ("2019-01-01") END ("2021-01-01") EVERY (INTERVAL 1 YEAR), START ("2021-01-01") END ("2021-05-01") EVERY (INTERVAL 1 MONTH), START ("2021-05-01") END ("2021-05-04") EVERY (INTERVAL 1 DAY) ) DISTRIBUTED BY HASH(site_id); ``` It is equivalent to using the following `PARTITION BY` clause in the CREATE TABLE statement: ```sql PARTITION BY RANGE (datekey) ( PARTITION p2019 VALUES [('2019-01-01'), ('2020-01-01')), PARTITION p2020 VALUES [('2020-01-01'), ('2021-01-01')), PARTITION p202101 VALUES [('2021-01-01'), ('2021-02-01')), PARTITION p202102 VALUES [('2021-02-01'), ('2021-03-01')), PARTITION p202103 VALUES [('2021-03-01'), ('2021-04-01')), PARTITION p202104 VALUES [('2021-04-01'), ('2021-05-01')), PARTITION p20210501 VALUES [('2021-05-01'), ('2021-05-02')), PARTITION p20210502 VALUES [('2021-05-02'), ('2021-05-03')), PARTITION p20210503 VALUES [('2021-05-03'), ('2021-05-04')) ) ``` * **The partitioning column is of integer type.** When the data type of the partitioning column is INT, you specify the range of partitions in `START` and `END` and define the incremental value in `EVERY`. Example: > **NOTE** > > The partitioning column values in **START()** and **END()** need to be wrapped in double quotation marks, while the incremental value in the **EVERY()** does not need to be wrapped in double quotation marks. * with the same numerical interval * with different numerical intervals In the following example, the range of all the partition starts from `1` and ends at `5`, with a partition increment of `1`: ```sql CREATE TABLE site_access ( datekey INT, site_id INT, city_code SMALLINT, user_name VARCHAR(32), pv BIGINT DEFAULT '0' ) DUPLICATE KEY(datekey, site_id, city_code, user_name) PARTITION BY RANGE (datekey) ( START ("1") END ("5") EVERY (1) ) DISTRIBUTED BY HASH(site_id); ``` It is equivalent to using the following `PARTITION BY` clause in the CREATE TABLE statement: ```sql PARTITION BY RANGE (datekey) ( PARTITION p1 VALUES [("1"), ("2")), PARTITION p2 VALUES [("2"), ("3")), PARTITION p3 VALUES [("3"), ("4")), PARTITION p4 VALUES [("4"), ("5")) ) ``` You can create batches of numerical partitions with different incremental intervals by specifying different incremental intervals in `EVERY` for each batch of partitions (make sure that the partition ranges between different batches do not overlap). Partitions in each batch are created according to the `START (xxx) END (xxx) EVERY (xxx)` clause. For example: ```sql CREATE TABLE site_access ( datekey INT, site_id INT, city_code SMALLINT, user_name VARCHAR(32), pv BIGINT DEFAULT '0' ) DUPLICATE KEY(datekey, site_id, city_code, user_name) PARTITION BY RANGE (datekey) ( START ("1") END ("10") EVERY (1), START ("10") END ("100") EVERY (10) ) DISTRIBUTED BY HASH(site_id); ``` * **Three specific time functions can be used as partitioning columns (supported since v3.3.0).** When explicitly defining the mapping relationship between partitions and the ranges of partition column values, you can use a specific time function to convert the partition column values of timestamps or strings into date values, and then divide the partitions based on the converted date values. * The partitioning column values are timestamps * The partitioning column values are strings ```sql -- A 10-digit timestamp accurate to the second, for example, 1703832553. CREATE TABLE site_access( event_time bigint, site_id INT, city_code SMALLINT, user_name VARCHAR(32), pv BIGINT DEFAULT '0' ) PARTITION BY RANGE(from_unixtime(event_time)) ( START ("2021-01-01") END ("2021-01-10") EVERY (INTERVAL 1 DAY) ) DISTRIBUTED BY HASH(site_id); -- A 13-digit timestamp accurate to the milliseconds, for example, 1703832553219. CREATE TABLE site_access( event_time bigint, site_id INT, city_code SMALLINT, user_name VARCHAR(32), pv BIGINT DEFAULT '0' ) PARTITION BY RANGE(from_unixtime_ms(event_time))( START ("2021-01-01") END ("2021-01-10") EVERY (INTERVAL 1 DAY) ) DISTRIBUTED BY HASH(site_id); ``` ```sql CREATE TABLE site_access ( event_time varchar(100), site_id INT, city_code SMALLINT, user_name VARCHAR(32), pv BIGINT DEFAULT '0' ) PARTITION BY RANGE(str2date(event_time, '%Y-%m-%d'))( START ("2021-01-01") END ("2021-01-10") EVERY (INTERVAL 1 DAY) ) DISTRIBUTED BY HASH(site_id); ``` ###### List partitioning (since v3.1)[​](#list-partitioning-since-v31 "Direct link to List partitioning (since v3.1)") [List Partitioning](https://docs.starrocks.io/docs/table_design/data_distribution/list_partitioning.md) is suitable for accelerating queries and efficiently managing data based on enum values. It is especially useful for scenarios where a partition needs to include data with different values in a partitioning column. For example, if you frequently query and manage data based on countries and cities, you can use this partitioning method and select the `city` column as the partitioning column. In this case, one partition can contain data for various cities belonging to one country. StarRocks stores data in the corresponding partitions based on the explicit mapping of the predefined value list for each partition. ##### Manage partitions[​](#manage-partitions "Direct link to Manage partitions") ###### Add partitions[​](#add-partitions "Direct link to Add partitions") For range partitioning and list partitioning, you can manually add new partitions to store new data. However for expression partitioning, because partitions are created automatically during data loading, you do not need to do so. The following statement adds a new partition to table `site_access` to store data for a new month: ```sql ALTER TABLE site_access ADD PARTITION p4 VALUES LESS THAN ("2020-04-30") DISTRIBUTED BY HASH(site_id); ``` ###### Delete a partition[​](#delete-a-partition "Direct link to Delete a partition") The following statement deletes partition `p1` from table `site_access`. > **NOTE** > > This operation does not immediately delete data in a partition. Data is retained in the Trash for a period of time (one day by default). If a partition is mistakenly deleted, you can use the [RECOVER](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/RECOVER.md) command to restore the partition and its data. ```sql ALTER TABLE site_access DROP PARTITION p1; ``` ###### Restore a partition[​](#restore-a-partition "Direct link to Restore a partition") The following statement restores partition `p1` and its data to table `site_access`. ```sql RECOVER PARTITION p1 FROM site_access; ``` ###### View partitions[​](#view-partitions "Direct link to View partitions") The following statement returns details of all partitions in table `site_access`. ```sql SHOW PARTITIONS FROM site_access; ``` #### Configure bucketing[​](#configure-bucketing "Direct link to Configure bucketing") ##### Random bucketing (since v3.1)[​](#random-bucketing-since-v31 "Direct link to Random bucketing (since v3.1)") StarRocks distributes the data in a partition randomly across all buckets. It is suitable for scenarios with small data sizes and relatively low requirements for query performance. If you do not set a bucketing method, StarRocks uses random bucketing by default and automatically sets the number of buckets. However, note that if you query massive amounts of data and frequently use certain columns as filter conditions, the query performance provided by random bucketing may not be optimal. In such scenarios, it is recommended to use [hash bucketing](#hash-bucketing). When these columns are used as filter conditions for queries, only data in a small number of buckets that the query hits need to be scanned and computed, which can significantly improve query performance. ###### Limits[​](#limits "Direct link to Limits") * You can only use random bucketing to create a Duplicate Key table. * You cannot specify a table bucketed randomly to belong to a [Colocation Group](https://docs.starrocks.io/docs/using_starrocks/Colocate_join.md). * Spark Load cannot be used to load data into tables bucketed randomly. In the following CREATE TABLE example, the `DISTRIBUTED BY xxx` statement is not used, so StarRocks uses random bucketing by default, and automatically sets the number of buckets. ```sql CREATE TABLE site_access1( event_day DATE, site_id INT DEFAULT '10', pv BIGINT DEFAULT '0' , city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '' ) DUPLICATE KEY(event_day,site_id,pv); ``` You can also explicitly specify the clause in the statement: ```sql CREATE TABLE site_access1( event_day DATE, site_id INT DEFAULT '10', pv BIGINT DEFAULT '0' , city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '' ) DUPLICATE KEY(event_day,site_id,pv) DISTRIBUTED BY RANDOM BUCKETS; ``` However, if you are familiar with StarRocks' bucketing mechanism, you can also manually set the number of buckets when creating a table with random bucketing. ```sql CREATE TABLE site_access2( event_day DATE, site_id INT DEFAULT '10', pv BIGINT DEFAULT '0' , city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '' ) DUPLICATE KEY(event_day,site_id,pv) DISTRIBUTED BY RANDOM BUCKETS 8; -- manually set the number of buckets to 8 ``` ##### Hash bucketing[​](#hash-bucketing "Direct link to Hash bucketing") StarRocks can use hash bucketing to subdivide data in a partition into buckets based on the bucketing key and [the number of buckets](#set-the-number-of-buckets). In hash bucketing, a hash function takes data's bucketing key value as an input and calculates a hash value. Data is stored in the corresponding bucket based on the mapping between the hash values and buckets. ###### Advantages[​](#advantages "Direct link to Advantages") * Improved query performance: Rows with the same bucketing key values are stored in the same bucket, reducing the amount of data scanned during queries. * Even data distribution: By selecting columns with higher cardinality (a larger number of unique values) as the bucketing key, data can be more evenly distributed across buckets. ###### How to choose the bucketing columns[​](#how-to-choose-the-bucketing-columns "Direct link to How to choose the bucketing columns") We recommend that you choose the column that satisfies the following two requirements as the bucketing column. * high cardinality column such as ID * column that often used in a filter for queries But if no columns satisfy both requirements, you need to determine the bucketing column according to the complexity of queries. * If the query is complex, it is recommended that you select high cardinality columns as bucketing columns to ensure that the data is as evenly distributed as possible across all the buckets and improve the cluster resource utilization. * If the query is relatively simple, it is recommended to select columns that are frequently used as filer conditions in queries as bucketing columns to improve query efficiency. If partition data cannot be evenly distributed across all the buckets by using one bucketing column, you can choose multiple bucketing columns. Note that it is recommended to use no more than 3 columns. ###### Precautions[​](#precautions "Direct link to Precautions") * **When a table is created, you must specify the bucketing columns**. * The data types of bucketing columns must be INTEGER, DECIMAL, DATE/DATETIME, or CHAR/VARCHAR/STRING. * Since 3.2, bucketing columns can be modified by using ALTER TABLE after table creation. ###### Examples[​](#examples "Direct link to Examples") In the following example, the `site_access` table is created by using `site_id` as the bucketing column. Additionally, when data in the `site_access` table is queried, data is often filtered by sites. Using `site_id` as the bucketing key can prune a significant number of irrelevant buckets during queries. ```sql CREATE TABLE site_access( event_day DATE, site_id INT DEFAULT '10', city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '', pv BIGINT SUM DEFAULT '0' ) AGGREGATE KEY(event_day, site_id, city_code, user_name) PARTITION BY RANGE(event_day) ( PARTITION p1 VALUES LESS THAN ("2020-01-31"), PARTITION p2 VALUES LESS THAN ("2020-02-29"), PARTITION p3 VALUES LESS THAN ("2020-03-31") ) DISTRIBUTED BY HASH(site_id); ``` Suppose each partition of table `site_access` has 10 buckets. In the following query, 9 out of 10 buckets are pruned, so StarRocks only needs to scan 1/10 of the data in the `site_access` table: ```sql select sum(pv) from site_access where site_id = 54321; ``` However, if `site_id` is unevenly distributed and a large number of queries only request data of a few sites, using only one bucketing column can result in severe data skew, causing system performance bottlenecks. In this case, you can use a combination of bucketing columns. For example, the following statement uses `site_id` and `city_code` as bucketing columns. ```sql CREATE TABLE site_access ( site_id INT DEFAULT '10', city_code SMALLINT, user_name VARCHAR(32) DEFAULT '', pv BIGINT SUM DEFAULT '0' ) AGGREGATE KEY(site_id, city_code, user_name) DISTRIBUTED BY HASH(site_id,city_code); ``` Practically speaking, you can use one or two bucketing columns based on your business characteristics. Using one bucketing column `site_id` is highly beneficial for short queries as it reduces data exchange between nodes, thereby enhancing the overall performance of the cluster. On the other hand, adopting two bucketing columns `site_id` and `city_code` is advantageous for long queries as it can leverage the overall concurrency of the distributed cluster to significantly improve performance. > **NOTE** > > * Short queries involve scanning a small amount of data, and can be completed on a single node. > * Long queries involve scanning a large amount of data, and their performance can be significantly improved by parallel scanning across multiple nodes in a distributed cluster. ##### Range-based bucketing[​](#range-based-bucketing "Direct link to Range-based bucketing") From v4.1 onwards, StarRocks supports the **Range-based Distribution semantic** (disabled by default), controlled by the FE configuration `enable_range_distribution`. The data will be sequenced according to the data range of the key columns, and each tablet contains the data from a certain range. The range-based distribution semantic is different from the default semantic in the following aspects: * If the key type (AGGREGATE KEY/UNIQUE KEY/PRIMARY KEY/DUPLICATE KEY) is explicitly specified, and a DISTRIBUTED BY clause is not specified, the data will be distributed by range by default. * If none of the key type, a DISTRIBUTED BY clause, or an ORDER BY is specified, a Duplicate Key table with the random bucketing strategy will be created. * If the key type and a DISTRIBUTED BY clause are not specified, but an ORDER BY clause is specified, a Duplicate Key table with the range-based distribution strategy will be created. In this case, DUPLICATE KEY is equivalent to an ORDER BY clause, and vice versa. * If both DUPLICATE KEY and an ORDER BY clause are specified, only the ORDER BY clause will take effect, and DUPLICATE KEY will be ignored. ###### Advantages[​](#advantages-1 "Direct link to Advantages") **Fewer concepts and simpler usage** * In most cases, table creation should require only column definitions and an `ORDER BY` clause. * Users do not have to define a `DISTRIBUTED BY` clause. **Ability to handle data skew** * Tablets support automatic splitting and merging, enabling the system to re-balance data dynamically and mitigate skew. For more instructions on splitting and merging tablets, see [ALTER TABLE - Split or merge tablets](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md#split-or-merge-tablets). **Balanced data locality and distribution** * Data from a small tenant can be located on a single BE/CN for optimal locality. * Data from a large tenant can be distributed across multiple BEs/CNs to maintain scalability and performance. **No dependency on time-based partitioning** * Time-based partitioning is no longer a must for data management. * A single partition will be capable of storing large data volumes, and the amount of data in different partitions can vary greatly, allowing users to choose partitioning strategies based on their specific needs. ###### Precautions[​](#precautions-1 "Direct link to Precautions") * This semantic is supported from v4.1 onwards and disabled by default. * To enable this semantic, you must set the FE configuration `enable_range_distribution` to `true`. ###### Examples[​](#examples-1 "Direct link to Examples") The following examples omit the partition syntax. **Duplicate key tables:** ```sql -- When DUPLICATE KEY is specified, the data is distributed by range of the key column. CREATE TABLE dup_table ( tenant_id varchar(128), created_time datetime, id int, name varchar(64) ) DUPLICATE KEY (tenant_id, created_time, id); -- When ORDER BY is specified, a Duplicate Key table is created. The data is distributed by range of the sort key. CREATE TABLE dup_table_order ( tenant_id varchar(128), created_time datetime, id int, name varchar(64) ) ORDER BY (tenant_id, created_time, id); -- If both DUPLICATE KEY and ORDER BY are not specified, a Duplicate Key table with the random bucketing strategy is created. CREATE TABLE random_table ( tenant_id varchar(128), created_time datetime, id int, name varchar(64) ); ``` **Primary Key/Aggregate Key/Unique Key tables:** ```sql -- Take Primary Key table as an example. -- When the key type and the key column are specified, the data is distributed by range of the key column. CREATE TABLE pk_table ( tenant_id varchar(128), created_time datetime, id int, name varchar(64) ) PRIMARY KEY (tenant_id, created_time, id); ``` ###### Limitations[​](#limitations "Direct link to Limitations") The following operations are not supported on tables with range distribution: | DDL | Reason | | ------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | `ALTER TABLE ... ADD ROLLUP ...` | Synchronous rollup assumes 1-to-1 base/rollup tablet pairing with same row order, which range distribution does not provide. | | `CREATE MATERIALIZED VIEW ... AS ...` (synchronous form, no `REFRESH` and no `DISTRIBUTED BY` clause) | Same code path as `ADD ROLLUP`. | | `ALTER TABLE ... ORDER BY (...)` (modify sort key) | The sort key defines tablet boundaries, so modifying it would invalidate existing range tablets. | | `ALTER TABLE ... OPTIMIZE` | OPTIMIZE redistributes / rebuckets a partition, which is incompatible with range tablet boundaries. | | `ALTER TABLE ... ADD COLUMN KEY ...` | New key columns auto-append to the (derived) range sort key on AGG/UNIQUE tables or tables without explicit `ORDER BY`. | | `ALTER TABLE ... DROP COLUMN ` where `` is in the range sort key | Removing a sort-key column invalidates the stored 1:1-copied range tablet boundary values. | | `ALTER TABLE ... MODIFY COLUMN ...` where `` is in the range sort key | Changes the type/semantics under which the stored range tablet boundary values were recorded. | | `ALTER TABLE ... MODIFY COLUMN ... KEY` (or value-column promotion in any keys-type) that flips keyness | A keyness flip shifts the key-derived range sort key on AGG/UNIQUE / no-explicit-ORDER-BY tables. | For rollup-like aggregation use cases, use an **asynchronous materialized view** with an explicit `REFRESH` clause or a `DISTRIBUTED BY` clause, e.g.: ```sql CREATE MATERIALIZED VIEW mv DISTRIBUTED BY HASH(col) REFRESH ASYNC AS SELECT ... FROM range_table; ``` ##### Set the number of buckets[​](#set-the-number-of-buckets "Direct link to Set the number of buckets") Buckets reflect how data files are actually organized in StarRocks. ###### At table creation[​](#at-table-creation "Direct link to At table creation") * Automatically set the number of buckets (recommended) Since v2.5.7, StarRocks supports automatically setting the number of buckets based on machine resources and data volume for a partition. tip If the raw data size of a partition exceeds 100 GB, we recommend that you manually configure the number of buckets using Method 2. * Table configured with hash bucketing * Table configured with random bucketing Example: ```sql CREATE TABLE site_access ( site_id INT DEFAULT '10', city_code SMALLINT, user_name VARCHAR(32) DEFAULT '', event_day DATE, pv BIGINT SUM DEFAULT '0') AGGREGATE KEY(site_id, city_code, user_name,event_day) PARTITION BY date_trunc('day', event_day) DISTRIBUTED BY HASH(site_id,city_code); -- do not need to set the number of buckets ``` As for tables configured with random bucketing, in addition to automatically setting the number of buckets in a partition, StarRocks further optimizes the logic since v3.2.0. StarRocks also can **dynamically increase** the number of buckets in a partition **during data loading**, based on the cluster capacity and the volume of loaded data. warning * To enable the on-demand and dynamic increase of the number of buckets, you need to set the table property `PROPERTIES("bucket_size"="xxx")` to specify the size of a single bucket. If the data volume in a partition is small, you can set the `bucket_size` to 1 GB. Otherwise, you can set the `bucket_size` to 4 GB. * Once the on-demand and dynamic increase of the number of buckets is enabled, and you need to rollback to version 3.1, you have to first delete the table which enables the dynamic increase in the number of buckets. Then you need to manually execute a metadata checkpoint using [ALTER SYSTEM CREATE IMAGE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md) before rolling back. Example: ```sql CREATE TABLE details1 ( event_day DATE, site_id INT DEFAULT '10', pv BIGINT DEFAULT '0', city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '') DUPLICATE KEY (event_day,site_id,pv) PARTITION BY date_trunc('day', event_day) -- The number of buckets in a partition is automatically determined by StarRocks and the number dynamically increases on demand because the size of a bucket is set to 1 GB. PROPERTIES("bucket_size"="1073741824") ; CREATE TABLE details2 ( event_day DATE, site_id INT DEFAULT '10', pv BIGINT DEFAULT '0' , city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '') DUPLICATE KEY (event_day,site_id,pv) PARTITION BY date_trunc('day', event_day) -- The number of buckets in the table partition is automatically determined by StarRocks, and the number is fixed and does not dynamically increases on demand because the size of a bucket is not set. ; ``` * Manually set the number of buckets Since v2.4.0, StarRocks supports using multiple threads to scan a tablet in parallel during a query, thereby reducing the dependency of scanning performance on the tablet count. We recommend that each tablet contain about 10 GB of raw data. If you intend to manually set the number of buckets, you can estimate the amount of data in each partition of a table and then decide the number of tablets. To enable parallel scanning on tablets, make sure the `enable_tablet_internal_parallel` parameter is set to `TRUE` globally for the entire system (`SET GLOBAL enable_tablet_internal_parallel = true;`). * Table configured with hash bucketing * Table configured with random bucketing ```sql CREATE TABLE site_access ( site_id INT DEFAULT '10', city_code SMALLINT, user_name VARCHAR(32) DEFAULT '', event_day DATE, pv BIGINT SUM DEFAULT '0') AGGREGATE KEY(site_id, city_code, user_name,event_day) PARTITION BY date_trunc('day', event_day) DISTRIBUTED BY HASH(site_id,city_code) BUCKETS 30; -- Suppose the amount of raw data that you want to load into a partition is 300 GB. -- Because we recommend that each tablet contain 10 GB of raw data, the number of buckets can be set to 30. DISTRIBUTED BY HASH(site_id,city_code) BUCKETS 30; ``` ```sql CREATE TABLE details ( site_id INT DEFAULT '10', city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '', event_day DATE, pv BIGINT DEFAULT '0' ) DUPLICATE KEY (site_id,city_code) PARTITION BY date_trunc('day', event_day) DISTRIBUTED BY RANDOM BUCKETS 30 ; ``` ###### After table creation[​](#after-table-creation "Direct link to After table creation") * Automatically set the number of buckets (recommended) Since v2.5.7, StarRocks supports automatically setting the number of buckets based on machine resources and data volume for a partition. tip If the raw data size of a partition exceeds 100 GB, we recommend that you manually configure the number of buckets using Method 2. * Table configured with hash bucketing * Table configured with random bucketing ```sql -- Automatically set the number of buckets for all partitions. ALTER TABLE site_access DISTRIBUTED BY HASH(site_id,city_code); -- Automatically set the number of buckets for specific partitions. ALTER TABLE site_access PARTITIONS (p20230101, p20230102) DISTRIBUTED BY HASH(site_id,city_code); -- Automatically set the number of buckets for new partitions. ALTER TABLE site_access ADD PARTITION p20230106 VALUES [('2023-01-06'), ('2023-01-07')) DISTRIBUTED BY HASH(site_id,city_code); ``` As for tables configured with random bucketing, in addition to automatically setting the number of buckets in a partition, StarRocks further optimizes the logic since v3.2.0. StarRocks also can **dynamically increase** the number of buckets in a partition **during data loading**, based on the cluster capacity and the volume of loaded data. This not only makes partition creation easier but also increases bulk load performance. warning * To enable the on-demand and dynamic increase of the number of buckets, you need to set the table property `PROPERTIES("bucket_size"="xxx")` to specify the size of a single bucket. If the data volume in a partition is small, you can set the `bucket_size` to 1 GB. Otherwise, you can set the `bucket_size` to 4 GB. * Once the on-demand and dynamic increase of the number of buckets is enabled, and you need to rollback to version 3.1, you have to first delete the table which enables the dynamic increase in the number of buckets. Then you need to manually execute a metadata checkpoint using [ALTER SYSTEM CREATE IMAGE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md) before rolling back. ```sql -- The number of buckets for all partitions is automatically set by StarRocks and this number is fixed because the on-demand and dynamic increase of the number of buckets is disabled. ALTER TABLE details DISTRIBUTED BY RANDOM; -- The number of buckets for all partitions is automatically set by StarRocks and the on-demand and dynamic increase of the number of buckets is enabled. ALTER TABLE details SET("bucket_size"="1073741824"); -- Automatically set the number of buckets for specific partitions. ALTER TABLE details PARTITIONS (p20230103, p20230104) DISTRIBUTED BY RANDOM; -- Automatically set the number of buckets for new partitions. ALTER TABLE details ADD PARTITION p20230106 VALUES [('2023-01-06'), ('2023-01-07')) DISTRIBUTED BY RANDOM; ``` * Manually set the number of buckets You can also manually specify the buckets number. To calculate the number of buckets for a partition, you can refer to the approach used when manually setting the number of buckets at table creation, [as mentioned above](#at-table-creation). * Table configured with hash bucketing * Table configured with random bucketing ```sql -- Manually set the number of buckets for all partitions ALTER TABLE site_access DISTRIBUTED BY HASH(site_id,city_code) BUCKETS 30; -- Manually set the number of buckets for specific partitions. ALTER TABLE site_access partitions p20230104 DISTRIBUTED BY HASH(site_id,city_code) BUCKETS 30; -- Manually set the number of buckets for new partitions. ALTER TABLE site_access ADD PARTITION p20230106 VALUES [('2023-01-06'), ('2023-01-07')) DISTRIBUTED BY HASH(site_id,city_code) BUCKETS 30; ``` ```sql -- Manually set the number of buckets for all partitions ALTER TABLE details DISTRIBUTED BY RANDOM BUCKETS 30; -- Manually set the number of buckets for specific partitions. ALTER TABLE details partitions p20230104 DISTRIBUTED BY RANDOM BUCKETS 30; -- Manually set the number of buckets for new partitions. ALTER TABLE details ADD PARTITION p20230106 VALUES [('2023-01-06'), ('2023-01-07')) DISTRIBUTED BY RANDOM BUCKETS 30; ``` Manually set the default number of buckets for dynamic partitions. ```sql ALTER TABLE details_dynamic SET ("dynamic_partition.buckets"="xxx"); ``` ###### View the number of buckets[​](#view-the-number-of-buckets "Direct link to View the number of buckets") After creating a table, you can execute [SHOW PARTITIONS](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md) to view the number of buckets set by StarRocks for each partition. Tables configured with hash bucketing have a fixed number of buckets per partition. info * As for a table configured with random bucketing which enable the on-demand and dynamic increase of the number of buckets, the number of buckets in each partition dynamically increases. So the returned result displays the current number of buckets for each partition. * For this table type, the actual hierarchy within a partition is as follows: partition > subpartition > bucket. To increase the number of buckets, StarRocks actually adds a new subpartition which includes a certain number of buckets. As a result, the SHOW PARTITIONS statement may return multiple data rows with the same partition name, which show the information of the subpartitions within the same partition. #### Optimize data distribution after table creation (since 3.2)[​](#optimize-data-distribution-after-table-creation-since-32 "Direct link to Optimize data distribution after table creation (since 3.2)") > **NOTICE** > > StarRocks's shared-data mode currently does not support this feature. As query patterns and data volume evolve in business scenarios, the configurations specified at table creation, such as the bucketing method, the number of buckets, and the sort key, may no longer be suitable for the new business scenario and even may cause query performance to decrease. At this point, you can use `ALTER TABLE` to modify the bucketing method, the number of buckets, and the sort key to optimize data distribution. For example: * **Increase the number of buckets when data volume within partitions is significantly increased** When the data volume within partitions becomes significantly larger than before, it is necessary to modify the number of buckets to maintain tablet sizes generally within the range of 1 GB to 10 GB. * **Modify the bucketing key to avoid data skew** When the current bucketing key can cause data skew (for example, only the `k1` column is configured as the bucketing key), it is necessary to specify more suitable columns or add additional columns to the bucketing key. For example: ```sql ALTER TABLE t DISTRIBUTED BY HASH(k1, k2) BUCKETS 20; -- When the StarRocks's version is 3.1 or later, and the table is Duplicate Key table, you can consider directly using the system's default bucketing settings, that is, random bucketing and the number of buckets automatically set by StarRocks. ALTER TABLE t DISTRIBUTED BY RANDOM; ``` * **Adapting the sort key due to changes in query patterns** If the business query patterns are significantly changed and additional columns are used as conditional columns, it can be beneficial to adjust the sort key. For example: ```sql ALTER TABLE t ORDER BY k2, k1; ``` For more information, see [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md). --- ### Range partitioning (Legacy) Range partitioning is suitable for storing simple contiguous data, such as time series data, or continuous numerical data. Based on range partitioning, you can create partitions using the [dynamic partitioning strategy](#dynamic-partitioning), which allows you to manage the Time-to-live (TTL) of partitions. note Please note that from v3.4 onwards, [expression partitioning](https://docs.starrocks.io/docs/table_design/data_distribution/expression_partitioning.md) is further optimized to unify all partitioning strategies and supported more complex solutions. It is recommended in most cases, and will replace the range partitioning strategy in future releases. #### Introduction[​](#introduction "Direct link to Introduction") **Range partitioning is appropriate for frequently queried data based on continuous date/numerical ranges. Additionally, it can be applied in some special cases where historical data needs to be partitioned by month, and recent data needs to be partitioned by day.** You need to explicitly define the data partitioning columns and establish the mapping relationship between partitions and ranges of partitioning column values. During data loading, StarRocks assigns the data to the corresponding partitions based on the ranges to which the data partitioning column values belong. As for the data type of the partitioning columns, before v3.3.0, range partitioning only supports partitioning columns of date and integer types. Since v3.3.0, three specific time functions can be used as partition columns. When explicitly defining the mapping relationship between partitions and ranges of partitioning column values, you need to first use a specific time function to convert partitioning column values of timestamps or strings into date values, and then divide the partitions based on the converted date values. info * If the partitioning column value is a timestamp, you need to use the from\_unixtime or from\_unixtime\_ms function to convert a timestamp to a date value when dividing the partitions. When the from\_unixtime function is used, the partitioning column only supports INT and BIGINT types. When the from\_unixtime\_ms function is used, the partitioning column only supports BIGINT type. * If the partitioning column value is a string (STRING, VARCHAR, or CHAR type), you need to use the str2date function to convert the string to a date value when dividing the partitions. #### Usage[​](#usage "Direct link to Usage") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql PARTITION BY RANGE ( partition_columns | function_expression ) ( single_range_partition | multi_range_partitions ) partition_columns ::= [, ...] function_expression ::= from_unixtime(column) | from_unixtime_ms(column) | str2date(column) single_range_partition ::= PARTITION VALUES partition_key_desc partition_key_desc ::= LESS THAN { MAXVALUE | value_list } | value_range -- value_range is a left-closed, right-open interval. Example: `[202201, 202212)`. -- You must explicitly specify the bracket `[` in the half-closed interval. value_range ::= [value_list, value_list) value_list ::= ( [, ...] ) multi_range_partitions ::= { PARTITIONS START ("") END ("") EVERY ( INTERVAL time_unit ) | PARTITIONS START ("") END ("") EVERY ( ) } -- The partition column values still need to be enclosed in double quotes even if the partition column values specified by START and END are integers. However, the interval values in the EVERY clause do not need to be enclosed in double quotes. time_unit ::= HOUR | DAY | WEEK | MONTH | YEAR ``` ##### Parameters[​](#parameters "Direct link to Parameters") | **Parameters** | **Description** | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `partition_columns` | The names of the partitioning columns. The partitioning column values can be string (BINARY not supported), date or datetime, or integer. | | `function_expression` | The function expression that transforms the partitioning column into specific data types. Supported functions: from\_unixtime, from\_unixtime\_ms, and str2date.
**NOTE**
Range partitioning supports one and only one function expression. | | `partition_name` | Partition name. It is recommended to set appropriate partition names based on the business scenario to differentiate the data in different partitions. | ##### Example[​](#example "Direct link to Example") 1. Create partitions by manually defining value ranges based on date type partitioing column. ```sql PARTITION BY RANGE(date_col)( PARTITION p1 VALUES LESS THAN ("2020-01-31"), PARTITION p2 VALUES LESS THAN ("2020-02-29"), PARTITION p3 VALUES LESS THAN ("2020-03-31") ) ``` 2. Create partitions by manually defining value ranges based on integer type partitioing column. ```sql PARTITION BY RANGE (int_col) ( PARTITION p1 VALUES LESS THAN ("20200131"), PARTITION p2 VALUES LESS THAN ("20200229"), PARTITION p3 VALUES LESS THAN ("20200331") ) ``` 3. Create multiple partition with the same date interval. ```sql PARTITION BY RANGE (date_col) ( START ("2021-01-01") END ("2021-01-04") EVERY (INTERVAL 1 DAY) ) ``` 4. Create multiple partition with different date intervals. ```sql PARTITION BY RANGE (date_col) ( START ("2019-01-01") END ("2021-01-01") EVERY (INTERVAL 1 YEAR), START ("2021-01-01") END ("2021-05-01") EVERY (INTERVAL 1 MONTH), START ("2021-05-01") END ("2021-05-04") EVERY (INTERVAL 1 DAY) ) ``` 5. Create multiple partition with the same integer interval. ```sql PARTITION BY RANGE (int_col) ( START ("1") END ("10") EVERY (1) ) ``` 6. Create multiple partition with different integer intervals. ```sql PARTITION BY RANGE (int_col) ( START ("1") END ("10") EVERY (1), START ("10") END ("100") EVERY (10) ) ``` 7. Use from\_unixtime to transform timestamp (string) type partition column into date type. ```sql PARTITION BY RANGE(from_unixtime(timestamp_col)) ( PARTITION p1 VALUES LESS THAN ("2021-01-01"), PARTITION p2 VALUES LESS THAN ("2021-01-02"), PARTITION p3 VALUES LESS THAN ("2021-01-03") ) ``` 8. Use str2date to transform string type partition column into date type. ```sql PARTITION BY RANGE(str2date(string_col, '%Y-%m-%d'))( PARTITION p1 VALUES LESS THAN ("2021-01-01"), PARTITION p2 VALUES LESS THAN ("2021-01-02"), PARTITION p3 VALUES LESS THAN ("2021-01-03") ) ``` #### Dynamic partitioning[​](#dynamic-partitioning "Direct link to Dynamic partitioning") StarRocks supports dynamic partitioning, which can automatically manage the time to life (TTL) of partitions, such as partitioning new input data in tables and deleting expired partitions. This feature significantly reduces maintenance costs. ##### Enable dynamic partitioning[​](#enable-dynamic-partitioning "Direct link to Enable dynamic partitioning") Take table `site_access` as an example. To enable dynamic partitioning, you need to configure the PROPERTIES parameter. For information about the configuration items, see [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md). ```sql CREATE TABLE site_access( event_day DATE, site_id INT DEFAULT '10', city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '', pv BIGINT DEFAULT '0' ) DUPLICATE KEY(event_day, site_id, city_code, user_name) PARTITION BY RANGE(event_day)( PARTITION p20200321 VALUES LESS THAN ("2020-03-22"), PARTITION p20200322 VALUES LESS THAN ("2020-03-23"), PARTITION p20200323 VALUES LESS THAN ("2020-03-24"), PARTITION p20200324 VALUES LESS THAN ("2020-03-25") ) DISTRIBUTED BY HASH(event_day, site_id) PROPERTIES( "dynamic_partition.enable" = "true", "dynamic_partition.time_unit" = "DAY", "dynamic_partition.start" = "-3", "dynamic_partition.end" = "3", "dynamic_partition.prefix" = "p", "dynamic_partition.buckets" = "32", "dynamic_partition.history_partition_num" = "0" ); ``` **`PROPERTIES`**: | parameter | required | description | | ------------------------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | dynamic\_partition.enable | No | Enables dynamic partitioning. Valid values are `TRUE` and `FALSE`. The default value is `TRUE`. | | dynamic\_partition.time\_unit | Yes | The time granularity for dynamically created partitions. It is a required parameter. Valid values are `HOUR`, `DAY`, `WEEK`, `MONTH`, and `YEAR`. The time granularity determines the suffix format for dynamically created partitions.- If the value is `HOUR`, the partitioning column can only be of DATETIME type, and can not be of DATE type. The suffix format for dynamically created partitions is yyyyMMddHH, for example, `2020032101`.
- If the value is `DAY`, the suffix format for dynamically created partitions is yyyyMMdd. An example partition name suffix is `20200321`.
- If the value is `WEEK`, the suffix format for dynamically created partitions is yyyy\_ww, for example `2020_13` for the 13th week of 2020.
- If the value is `MONTH`, the suffix format for dynamically created partitions is yyyyMM, for example `202003`.
- If the value is `YEAR`, the suffix format for dynamically created partitions is yyyy, for example `2020`. | | dynamic\_partition.time\_zone | No | The time zone for dynamic partitions, which is by default the same as the system time zone. | | dynamic\_partition.start | No | The starting offset of dynamic partitioning. The value of this parameter must be a negative integer. The partitions before this offset will be deleted based on the current day, week, or month which is determined by the value of the parameter `dynamic_partition.time_unit`. The default value is `Integer.MIN_VALUE`, namely, -2147483648, which means that the history partitions will not be deleted. | | dynamic\_partition.end | Yes | The end offset of dynamic partitioning. The value of this parameter must be a positive integer. The partitions from the current day, week, or month to the end offset will be created in advance. | | dynamic\_partition.prefix | No | The prefix added to the names of dynamic partitions. The default value is `p`. | | dynamic\_partition.buckets | No | The number of buckets per dynamic partition. The default value is the same as the number of buckets determined by the reserved word BUCKETS or automatically set by StarRocks. | | dynamic\_partition.history\_partition\_num | No | The number of historical partitions created by the dynamic partitioning mechanism, with a default value of `0`. When the value is greater than 0, historical partitions are created in advance. From v2.5.2, StarRocks supports this parameter. | | dynamic\_partition.start\_day\_of\_week | No | When `dynamic_partition.time_unit` is `WEEK`, this parameter is used to specify the first day of each week. Valid values: `1` to `7`. `1` means Monday and `7` means Sunday. The default value is `1`, which means that every week starts on Monday. | | dynamic\_partition.start\_day\_of\_month | No | When `dynamic_partition.time_unit` is `MONTH`, this parameter is used to specify the first day of each month. Valid values: `1` to `28`. `1` means the 1st of every month and `28` means the 28th of every month. The default value is `1`, which means that every month starts on the 1st. The first day can not be the 29th, 30th, or 31st. | | dynamic\_partition.replication\_num | No | The number of replicas for tablets in dynamically created partitions. The default value is the same as the number of replicas configured at table creation. | note When the partition column is the INT type, its format must be `yyyyMMdd`, regardless of the partition time granularity. **FE configuration:** `dynamic_partition_check_interval_seconds`: the interval for scheduling dynamic partitioning. The default value is 600s, which means that the partition situation is checked every 10 minutes to see whether the partitions meet the dynamic partitioning conditions specified in `PROPERTIES`. If not, the partitions will be created and deleted automatically. ##### View dynamic partitions[​](#view-dynamic-partitions "Direct link to View dynamic partitions") After you enable dynamic partitions for a table, the input data is continuously and automatically partitioned. You can view the current partitions by using the following statement. For example, if the current date is 2020-03-25, you can only see partitions in the time range from 2020-03-25 to 2020-03-28. ```sql SHOW PARTITIONS FROM site_access; [types: [DATE]; keys: [2020-03-25]; ‥types: [DATE]; keys: [2020-03-26]; ) [types: [DATE]; keys: [2020-03-26]; ‥types: [DATE]; keys: [2020-03-27]; ) [types: [DATE]; keys: [2020-03-27]; ‥types: [DATE]; keys: [2020-03-28]; ) [types: [DATE]; keys: [2020-03-28]; ‥types: [DATE]; keys: [2020-03-29]; ) ``` If you want to create historical partitions when creating a table, you need to specify `dynamic_partition.history_partition_num` to define the number of historical partitions to be created. For example, if you set `dynamic_partition.history_partition_num` to `3` during table creation and the current date is 2020-03-25, you will only see partitions in the time range from 2020-03-22 to 2020-03-28. ```sql SHOW PARTITIONS FROM site_access; [types: [DATE]; keys: [2020-03-22]; ‥types: [DATE]; keys: [2020-03-23]; ) [types: [DATE]; keys: [2020-03-23]; ‥types: [DATE]; keys: [2020-03-24]; ) [types: [DATE]; keys: [2020-03-24]; ‥types: [DATE]; keys: [2020-03-25]; ) [types: [DATE]; keys: [2020-03-25]; ‥types: [DATE]; keys: [2020-03-26]; ) [types: [DATE]; keys: [2020-03-26]; ‥types: [DATE]; keys: [2020-03-27]; ) [types: [DATE]; keys: [2020-03-27]; ‥types: [DATE]; keys: [2020-03-28]; ) [types: [DATE]; keys: [2020-03-28]; ‥types: [DATE]; keys: [2020-03-29]; ) ``` ##### Modify properties of dynamic partitioning[​](#modify-properties-of-dynamic-partitioning "Direct link to Modify properties of dynamic partitioning") You can use the [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md) statement to modify properties of dynamic partitioning, such as disabling dynamic partitioning. Take the following statement as an example. ```sql ALTER TABLE site_access SET("dynamic_partition.enable"="false"); ``` note * To check the properties of dynamic partitioning of a table, execute the [SHOW CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_CREATE_TABLE.md) statement. * You can also use the ALTER TABLE statement to modify other properties of a table. --- ### Expression partitioning (recommended) Since v3.0, StarRocks has supported expression partitioning (previously known as automatic partitioning), which is more flexible and user-friendly. This partitioning method is suitable for most scenarios such as querying and managing data based on continuous time ranges or ENUM values. You only need to specify a simple partition expression at table creation. During data loading, StarRocks will automatically create partitions based on the data and the rule defined in the partition expression. You no longer need to manually create numerous partitions at table creation, nor configure dynamic partition properties. From v3.4 onwards, expression partitioning is further optimized to unify all partitioning strategies and supported more complex solutions. It is recommended in most cases, and will replace the other partitioning strategies in future releases. From v3.5 onwards, StarRocks supports merging expression partitions based on time function for optimized storage efficiency and query performance. For detailed information, see [Merge expression partitions](#merge-expression-partitions). #### Partitioning based on a simple time function expression[​](#partitioning-based-on-a-simple-time-function-expression "Direct link to Partitioning based on a simple time function expression") If you frequently query and manage data based on continuous time ranges, you only need to specify a date type (`DATE` or `DATETIME`) column as the partition column and specify year, month, day, or hour as the partition granularity in the time function expression. StarRocks will automatically create partitions and set the partitions' start and end dates or datetime based on the loaded data and partition expression. However, in some special scenarios, such as partitioning historical data into partitions by month and recent data into partitions by day, you must use [range partitioning](https://docs.starrocks.io/docs/table_design/data_distribution.md#range-partitioning) to create partitions. note `PARTITION BY date_trunc(column)` and `PARTITION BY time_slice(column)` are considered range partitioning, despite their format of expression partitioning. Therefore, you can use the `ALTER TABLE ... ADD PARTITION` statement for range partitions to add new partitions to tables using such partitioning strategies. ##### Syntax[​](#syntax "Direct link to Syntax") ```sql PARTITION BY expression ... [ PROPERTIES( { 'partition_live_number' = 'xxx' | 'partition_retention_condition' = 'expr' } ) ] expression ::= { date_trunc ( , ) | time_slice ( , INTERVAL [ , boundary ] ) } ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### `expression`[​](#expression "Direct link to expression") **Required**: YES
**Description**: A simple time function expression that use the [`date_trunc`](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_trunc.md) or [`time_slice`](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/time_slice.md) functions. If you use the function `time_slice`, you do not need to pass the `boundary` parameter. It is because in this scenario, the default and valid value for this parameter is `floor`, and the value cannot be `ceil`. ###### `time_unit`[​](#time_unit "Direct link to time_unit") **Required**: YES
**Description**: The partition granularity, which can be `hour`, `day`, `week`, `month`, or `year`. If partition granularity is `hour`, the partition column must be of the `DATETIME` data type and cannot be of the `DATE` data type. ###### `partition_column`[​](#partition_column "Direct link to partition_column") **Required**: YES
**Description**: The name of the partition column. * The partition column can only be of the `DATE` or `DATETIME` data type. The partition column allows `NULL` values. * The partition column can be of the `DATE` or `DATETIME` data type if the `date_trunc` function is used. The partition column must be of the `DATETIME` data type if the `time_slice` function is used. * If the partition column is of the `DATE` data type, the supported range is `[0000-01-01 ~ 9999-12-31]`. If the partition column is of the `DATETIME` data type, the supported range is `[0000-01-01 01:01:01 ~ 9999-12-31 23:59:59]` * Currently, you can specify only one partition column; multiple partition columns are not supported. ###### `partition_live_number`[​](#partition_live_number "Direct link to partition_live_number") **Required**: NO
**Description**: The number of the most recent partitions to be retained. Partitions are sorted in chronological order, **with the current date as a benchmark**; partitions older than the current date minus `partition_live_number` are deleted. StarRocks schedules tasks to manage the number of partitions, and the scheduling interval can be configured through the FE dynamic parameter `dynamic_partition_check_interval_seconds`, which defaults to 600 seconds (10 minutes). Suppose that the current date is April 4, 2023, `partition_live_number` is set to `2`, and the partitions include `p20230401`, `p20230402`, `p20230403`, `p20230404`. The partitions `p20230403` and `p20230404` are retained, and other partitions are deleted. If dirty data is loaded, such as data from the future dates April 5 and April 6, partitions include `p20230401`, `p20230402`, `p20230403`, `p20230404`, and `p20230405`, and `p20230406`. Then partitions `p20230403`, `p20230404`, `p20230405`, and `p20230406` are retained, and the other partitions are deleted. ###### `partition_retention_condition`[​](#partition_retention_condition "Direct link to partition_retention_condition") From v3.5.0 onwards, StarRocks native tables support Common Partition Expression TTL. `partition_retention_condition`: The expression that declares the partitions to be retained dynamically. Partitions that do not meet the condition in the expression will be dropped regularly. Example: `'partition_retention_condition' = 'dt >= CURRENT_DATE() - INTERVAL 3 MONTH'`. * The expression can only contain partition columns and constants. Non-partition columns are not supported. * Common Partition Expression applies to List partitions and Range partitions differently: * For tables with List partitions, StarRocks supports deleting partitions filtered by the Common Partition Expression. * For tables with Range partitions, StarRocks can only filter and delete partitions using the partition pruning capability of FE. Partitions correspond to predicates that are not supported by partition pruning cannot be filtered and deleted. ##### Usage notes[​](#usage-notes "Direct link to Usage notes") * During data loading, StarRocks automatically creates some partitions based on the loaded data, but if the load job fails for some reason, the partitions that are automatically created by StarRocks cannot be automatically deleted. * StarRocks sets the default maximum number of automatically created partitions for one load to 4096, which can be configured by the FE parameter `auto_partition_max_creation_number_per_load`. This parameter can prevent you from accidentally creating too many partitions. * The naming rule for partitions is consistent with the naming rule for dynamic partitioning. ##### Examples[​](#examples "Direct link to Examples") Example 1: Suppose you frequently query data by day. You can use the partition expression `date_trunc()` and set the partition column as `event_day` and the partition granularity as `day` at table creation. Data is automatically partitioned based on dates during loading. Data of the same day is stored in one partition and partition pruning can be used to significantly improve query efficiency. ```sql CREATE TABLE site_access1 ( event_day DATETIME NOT NULL, site_id INT DEFAULT '10', city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '', pv BIGINT DEFAULT '0' ) DUPLICATE KEY(event_day, site_id, city_code, user_name) PARTITION BY date_trunc('day', event_day) DISTRIBUTED BY HASH(event_day, site_id); ``` For example, when the following two data rows are loaded, StarRocks will automatically create two partitions, `p20230226` and `p20230227`, with ranges \[2023-02-26 00:00:00, 2023-02-27 00:00:00) and \[2023-02-27 00:00:00, 2023-02-28 00:00:00) respectively. If subsequent loaded data falls within these ranges, they are automatically routed to the corresponding partitions. ```sql -- insert two data rows INSERT INTO site_access1 VALUES ("2023-02-26 20:12:04",002,"New York","Sam Smith",1), ("2023-02-27 21:06:54",001,"Los Angeles","Taylor Swift",1); -- view partitions mysql > SHOW PARTITIONS FROM site_access1; +-------------+---------------+----------------+---------------------+--------------------+--------+--------------+------------------------------------------------------------------------------------------------------+--------------------+---------+----------------+---------------+---------------------+--------------------------+----------+-------------+------------+----------+-------------+--------------------+----------------+----------------+ | PartitionId | PartitionName | VisibleVersion | VisibleVersionTime | VisibleVersionHash | State | PartitionKey | Range | DistributionKey | Buckets | ReplicationNum | StorageMedium | CooldownTime | LastConsistencyCheckTime | DataSize | StorageSize | IsInMemory | RowCount | DataVersion | VersionEpoch | VersionTxnType | TabletBalanced | +-------------+---------------+----------------+---------------------+--------------------+--------+--------------+------------------------------------------------------------------------------------------------------+--------------------+---------+----------------+---------------+---------------------+--------------------------+----------+-------------+------------+----------+-------------+--------------------+----------------+----------------+ | 17138 | p20230226 | 2 | 2023-07-19 17:53:59 | 0 | NORMAL | event_day | [types: [DATETIME]; keys: [2023-02-26 00:00:00]; ..types: [DATETIME]; keys: [2023-02-27 00:00:00]; ) | event_day, site_id | 6 | 3 | HDD | 9999-12-31 23:59:59 | NULL | 0B | 0B | false | 0 | 2 | 409742105974407168 | TXN_NORMAL | true | | 17113 | p20230227 | 2 | 2023-07-19 17:53:59 | 0 | NORMAL | event_day | [types: [DATETIME]; keys: [2023-02-27 00:00:00]; ..types: [DATETIME]; keys: [2023-02-28 00:00:00]; ) | event_day, site_id | 6 | 3 | HDD | 9999-12-31 23:59:59 | NULL | 0B | 0B | false | 0 | 2 | 409742105974407169 | TXN_NORMAL | true | +-------------+---------------+----------------+---------------------+--------------------+--------+--------------+------------------------------------------------------------------------------------------------------+--------------------+---------+----------------+---------------+---------------------+--------------------------+----------+-------------+------------+----------+-------------+--------------------+----------------+----------------+ 2 rows in set (0.00 sec) ``` Example 2: If you want to implement partition lifecycle management, which is to retain only a certain number of recent partitions and delete historical partitions, you can use the `partition_live_number` property to specify the number of partitions to retain. ```sql CREATE TABLE site_access2 ( event_day DATETIME NOT NULL, site_id INT DEFAULT '10', city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '', pv BIGINT DEFAULT '0' ) DUPLICATE KEY(event_day, site_id, city_code, user_name) PARTITION BY date_trunc('month', event_day) DISTRIBUTED BY HASH(event_day, site_id) PROPERTIES( "partition_live_number" = "3" -- only retains the most recent three partitions ); ``` Example 3: Suppose you frequently query data by week. You can use the partition expression `time_slice()` and set the partition column as `event_day` and the partition granularity to one week at table creation. Data for one week is stored in one partition and partition pruning can be used to significantly improve query efficiency. ```sql CREATE TABLE site_access( event_day DATETIME NOT NULL, site_id INT DEFAULT '10', city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '', pv BIGINT DEFAULT '0' ) DUPLICATE KEY(event_day, site_id, city_code, user_name) PARTITION BY time_slice(event_day, INTERVAL 1 week) DISTRIBUTED BY HASH(event_day, site_id); ``` #### Partitioning based on the column expression (since v3.1)[​](#partitioning-based-on-the-column-expression-since-v31 "Direct link to Partitioning based on the column expression (since v3.1)") If you frequently query and manage data of a specific type, you only need to specify the column representing the type as the partition column. StarRocks will automatically create partitions based on the partition column values of the loaded data. However, in some special scenarios, such as when the table contains a column `city`, and you frequently query and manage data based on countries and cities. You must use [list partitioning](https://docs.starrocks.io/docs/table_design/data_distribution/list_partitioning.md) to store data of multiple cities within the same country in one partition. ##### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql PARTITION BY expression ... expression ::= partition_columns partition_columns ::= , [ [,...] ] ``` ##### Parameters[​](#parameters-1 "Direct link to Parameters") ###### `partition_columns`[​](#partition_columns "Direct link to partition_columns") **Required**: YES
**Description**: The names of partition columns. * The partition column values can be string (`BINARY` not supported), `date` or `datetime`, `integer`, and `boolean` values. The partition column allows `NULL` values. * Each partition can only contain data with the same value in the partition column. To include data with different values in a partition column in a partition, see [List partitioning](https://docs.starrocks.io/docs/table_design/data_distribution/list_partitioning.md). note From v3.4 onwards, you can omit the parentheses that are used to wrap the partition columns. For example, you can replace `PARTITION BY (dt,city)` with `PARTITION BY dt,city`. ##### Usage notes[​](#usage-notes-1 "Direct link to Usage notes") * During data loading, StarRocks automatically creates some partitions based on the loaded data, but if the load job fails for some reason, the partitions that are automatically created by StarRocks cannot be automatically deleted. * StarRocks sets the default maximum number of automatically created partitions for one load to 4096, which can be configured by the FE parameter `auto_partition_max_creation_number_per_load`. This parameter can prevent you from accidentally creating too many partitions. * The naming rule for partitions: if multiple partition columns are specified, the values of different partition columns are connected with an underscore `_` in the partition name, and the format is `p__...`. For example, if two columns `dt` and `province` are specified as partition columns, both of which are string types, and a data row with values `2022-04-01` and `beijing` is loaded, the corresponding partition automatically created is named `p20220401_beijing`. ##### Examples[​](#examples-1 "Direct link to Examples") Example 1: Suppose you frequently query details of the data center billing based on time ranges and specific cities. At table creation, you can use a partition expression to specify the first partition columns as `dt` and `city`. This way, data belonging to the same date and city are routed into the same partition, and partition pruning can be used to significantly improve query efficiency. ```sql CREATE TABLE t_recharge_detail1 ( id bigint, user_id bigint, recharge_money decimal(32,2), city varchar(20) not null, dt varchar(20) not null ) DUPLICATE KEY(id) PARTITION BY dt,city DISTRIBUTED BY HASH(`id`); ``` Insert a single data row into the table. ```sql INSERT INTO t_recharge_detail1 VALUES (1, 1, 1, 'Houston', '2022-04-01'); ``` View the partitions. The result shows that StarRocks automatically creates a partition `p20220401_Houston` based on the loaded data. During subsequent loading, data with the values `2022-04-01` and `Houston` in the partition columns `dt` and `city` are stored in this partition. tip Each partition can only contain data with the specified one value for the partition column. To specify multiple values for a partition column in a partition, see [List partitions](https://docs.starrocks.io/docs/table_design/data_distribution/list_partitioning.md). ```sql MySQL > SHOW PARTITIONS from t_recharge_detail1\G *************************** 1. row *************************** PartitionId: 16890 PartitionName: p20220401_Houston VisibleVersion: 2 VisibleVersionTime: 2023-07-19 17:24:53 VisibleVersionHash: 0 State: NORMAL PartitionKey: dt, city List: [["2022-04-01","Houston"]] DistributionKey: id Buckets: 6 ReplicationNum: 3 StorageMedium: HDD CooldownTime: 9999-12-31 23:59:59 LastConsistencyCheckTime: NULL DataSize: 2.5KB StorageSize: 2.5KB IsInMemory: false RowCount: 1 DataVersion: 2 VersionEpoch: 409742188174376960 VersionTxnType: TXN_NORMAL TabletBalanced: true 1 row in set (0.00 sec) ``` #### Partitioning based on a complex time function expression (since v3.4)[​](#partitioning-based-on-a-complex-time-function-expression-since-v34 "Direct link to Partitioning based on a complex time function expression (since v3.4)") From v3.4.0 onwards, expression partitioning supports any expressions that return `DATE` or `DATETIME` types to accommodate even more complex partitioning scenarios. For the supported time functions, see [Appendix - Supported time functions](#supported-time-functions). For example, you can define a Unix timestamp column, and use `from_unixtime()` directly against the column in the partition expression to define the partition key, instead of define a generated `DATE` or `DATETIME` column with the function. For more about the usage, see [Examples](#examples-2). From v3.4.4 onwards, partition pruning are supported for partitions based on most `DATETIME`-related functions. ##### Examples[​](#examples-2 "Direct link to Examples") Example 1: Suppose you assign a Unix timestamp to each data row and frequently query data by day. You can use the timestamp column with the function `from_unixtime()` in the expression to define the timestamp as the partition column and the partition granularity to a day at table creation. Data of each day is stored in one partition and partition pruning can be used to improve query efficiency. ```sql CREATE TABLE orders ( ts BIGINT NOT NULL, id BIGINT NOT NULL, city STRING NOT NULL ) PARTITION BY from_unixtime(ts,'%Y%m%d'); ``` Example 2: Suppose you assign an `INT` type timestamp to each data row and store data on a monthly basis. You can use the timestamp column with the functions `cast()` and `str_to_date()` in the expression to transform the timestamp into the `DATE` type, set it as the partition column, and the partition granularity to a month using `date_trunc()` at table creation. Data of each month is stored in one partition and partition pruning can be used to improve query efficiency. ```sql CREATE TABLE orders_new ( ts INT NOT NULL, id BIGINT NOT NULL, city STRING NOT NULL ) PARTITION BY date_trunc('month', str_to_date(CAST(ts as STRING),'%Y%m%d')); ``` ##### Usage notes[​](#usage-notes-2 "Direct link to Usage notes") Partition pruning is applicable to cases of partitioning based on a complex time function expression: * If the partition clause is `PARTITION BY from_unixtime(ts)`, queries with filters in the format `ts>1727224687` can be pruned to corresponding partitions. * If the partition clause is `PARTITION BY str2date(CAST(ts AS string),'%Y%m')`, queries with filters in the format `ts = "20240506"` can be pruned. * The above cases are also applicable to [Partitioning based on mixed expression](#partitioning-based-on-the-mixed-expression-since-v34). #### Partitioning based on the mixed expression (since v3.4)[​](#partitioning-based-on-the-mixed-expression-since-v34 "Direct link to Partitioning based on the mixed expression (since v3.4)") From v3.4.0 onwards, expression partitioning supports multiple partition columns with one of them being a time function expression. ##### Examples[​](#examples-3 "Direct link to Examples") Example 1: Suppose you assign a Unix timestamp to each data row and frequently query data by day and a specific city. You can use the timestamp column (with the `from_unixtime()` function) and the city column as the partition columns at table creation. Data of each day in each city is stored in one partition and partition pruning can be used to improve query efficiency. ```sql CREATE TABLE orders ( ts BIGINT NOT NULL, id BIGINT NOT NULL, city STRING NOT NULL ) PARTITION BY from_unixtime(ts,'%Y%m%d'), city; ``` #### Manage partitions[​](#manage-partitions "Direct link to Manage partitions") ##### Load data into partitions[​](#load-data-into-partitions "Direct link to Load data into partitions") During data loading, StarRocks will automatically create partitions based on the loaded data and partition rule defined by the partition expression. When using expression partitioning, you can use [INSERT OVERWRITE](https://docs.starrocks.io/docs/loading/InsertInto.md#overwrite-data-via-insert-overwrite-select) to dynamically overwrite data without specifying the partition name. StarRocks will automatically route and overwrite the data in the corresponding partitions. If you want to overwrite data in a *specific* partition, you can explicitly provide a partition range in `PARTITION()`. Note that for expression partitions, you need to provide the starting date or datetime of that partition (the partition granularity configured at table creation) or the specific column values, rather than just the partition name. If the partition does not exist, it will be automatically created during data loading. ```sql INSERT OVERWRITE site_access1 PARTITION(event_day='2022-06-08 20:12:04') SELECT * FROM site_access2 PARTITION(p20220608); ``` If you use column expression at table creation and want to overwrite data in a specific partition, you need to provide the partition column values that the partition contains. If the partition does not exist, it can be automatically created during data loading. ```sql INSERT OVERWRITE t_recharge_detail1 PARTITION(dt='2022-04-02',city='texas') SELECT * FROM t_recharge_detail2 PARTITION(p20220402_texas); ``` ##### View partitions[​](#view-partitions "Direct link to View partitions") When you want to view specific information about automatically created partitions, you need to use the `SHOW PARTITIONS FROM ` statement. The `SHOW CREATE TABLE ` statement only returns the syntax for expression partitioning that is configured at table creation. ```sql MySQL > SHOW PARTITIONS FROM t_recharge_detail1; +-------------+-------------------+----------------+---------------------+--------------------+--------+--------------+----------------------------+-----------------+---------+----------------+---------------+---------------------+--------------------------+----------+-------------+------------+----------+-------------+--------------------+----------------+----------------+ | PartitionId | PartitionName | VisibleVersion | VisibleVersionTime | VisibleVersionHash | State | PartitionKey | List | DistributionKey | Buckets | ReplicationNum | StorageMedium | CooldownTime | LastConsistencyCheckTime | DataSize | StorageSize | IsInMemory | RowCount | DataVersion | VersionEpoch | VersionTxnType | TabletBalanced | +-------------+-------------------+----------------+---------------------+--------------------+--------+--------------+----------------------------+-----------------+---------+----------------+---------------+---------------------+--------------------------+----------+-------------+------------+----------+-------------+--------------------+----------------+----------------+ | 11099 | p20220401_Houston | 2 | 2026-03-11 13:59:51 | 0 | NORMAL | dt, city | [["2022-04-01","Houston"]] | id | 6 | 3 | HDD | 9999-12-31 23:59:59 | NULL | 0B | 0B | false | 0 | 2 | 409743636180238336 | TXN_NORMAL | true | | 11116 | p20220402_texas | 2 | 2026-03-11 13:59:52 | 0 | NORMAL | dt, city | [["2022-04-02","texas"]] | id | 6 | 3 | HDD | 9999-12-31 23:59:59 | NULL | 0B | 0B | false | 0 | 2 | 409743639174971392 | TXN_NORMAL | true | +-------------+-------------------+----------------+---------------------+--------------------+--------+--------------+----------------------------+-----------------+---------+----------------+---------------+---------------------+--------------------------+----------+-------------+------------+----------+-------------+--------------------+----------------+----------------+ 2 rows in set (0.01 sec) ``` ##### Merge expression partitions[​](#merge-expression-partitions "Direct link to Merge expression partitions") In data management, partitioning based on different time granularity is crucial to optimizing queries and storage. To improve storage efficiency and query performance, StarRocks supports merging multiple expression partitions of a finer time granularity into one of a rougher time granularity, for example, merging partitions by day into one partition by month. By merging partitions that meet the specified condition (time range), StarRocks allows you to partition data on different time granularity. ###### Syntax[​](#syntax-2 "Direct link to Syntax") ```sql ALTER TABLE [.] PARTITION BY [WHERE ] BETWEEN AND ``` ###### Parameters[​](#parameters-2 "Direct link to Parameters") ###### `PARTITION BY `[​](#partition-by-time_expr "Direct link to partition-by-time_expr") **Required**: YES
**Description**: Specifies the new time granularity for the partitioning strategy, for example, `PARTITION BY date_trunc('month', dt)`. ###### `WHERE BETWEEN AND `[​](#where-time_range_column-between-start_time-and-end_time "Direct link to where-time_range_column-between-start_time-and-end_time") **Required**: YES
**Description**: Specifies the time range of the partitions to be merged. Partitions within this range will be merged based on the rules defined in the `PARTITION BY` clause. ###### Example[​](#example "Direct link to Example") Merge the partitions in the table `site_access1`, and change the partition time granularity from day to month. The time range of the partitions to be merged is from `2024-01-01` to `2024-03-31`. ```sql ALTER TABLE site_access1 PARTITION BY date_trunc('month', event_day) BETWEEN '2024-01-01' AND '2024-03-31'; ``` After merging: * Day-level partitions `2024-01-01` to `2024-01-31` are merged into a month-level partition of `2024-01`. * Day-level partitions `2024-02-01` to `2024-02-29` are merged into a month-level partition of `2024-02`. * Day-level partitions `2024-03-01` to `2024-03-31` are merged into a month-level partition of `2024-03`. ###### Usage notes[​](#usage-notes-3 "Direct link to Usage notes") * Merging is supported only for expression partitions based on a time function. * Merging partitions with multiple partition columns is not supported. * Parallel execution of merging and Schema Change/DML operations is not supported. #### Limits[​](#limits "Direct link to Limits") * Since v3.1.0, StarRocks's shared-data mode supports the [time function expression](#partitioning-based-on-a-simple-time-function-expression). And since v3.1.1, StarRocks's shared-data mode further supports the [column expression](#partitioning-based-on-the-column-expression-since-v31). * Currently, using CTAS to create tables configured expression partitioning is not supported. * Currently, using Spark Load to load data to tables that use expression partitioning is not supported. * When the `ALTER TABLE DROP PARTITION ` statement is used to delete a partition created by using the column expression, data in the partition is directly removed and cannot be recovered. * From v3.4.0, v3.3.8, v3.2.13, and v3.1.16 onwards, StarRocks supports [backing up and restoring](https://docs.starrocks.io/docs/administration/management/Backup_and_restore.md) tables created with the expression partitioning strategy. #### Appendix[​](#appendix "Direct link to Appendix") ##### Supported time functions[​](#supported-time-functions "Direct link to Supported time functions") Expression partitioning supports the following functions: **Time functions**: * `timediff` * `datediff` * `to_days` * `years_add`/`sub` * `quarters_add`/`sub` * `months_add`/`sub` * `weeks_add`/`sub` * `date_add`/`sub` * `days_add`/`sub` * `hours_add`/`sub` * `minutes_add`/`sub` * `seconds_add`/`sub` * `milliseconds_add`/`sub` * `date_trunc` * `date_format(YmdHiSf/YmdHisf)` * `str2date(YmdHiSf/YmdHisf)` * `str_to_date(YmdHiSf/YmdHisf)` * `to_iso8601` * `to_date` * `unix_timestamp` * `from_unixtime(YmdHiSf/YmdHisf)` * `time_slice` **Other functions**: * add * subtract * cast note * Combined usage of multiple time functions is supported. * The system default time zone is used for all the time functions listed above. * The value format of the time function, `YmdHiSf`, must start with the roughest time granularity, `%Y`. Formats that start with a finer time granularity, for example, `%m-%d`, are not allowed. **Example** `PARTITION BY from_unixtime(cast(str as INT) + 3600, '%Y-%m-%d')` --- ### Feature Support: Data Distribution This document outlines the partitioning and bucketing features supported by StarRocks. #### Supported table types[​](#supported-table-types "Direct link to Supported table types") * **Bucketing** Hash Bucketing is supported in all table types. Random Bucketing (from v3.1 onwards) is supported **only in Duplicate Key tables**. * **Partitioning** Expression Partitioning (from v3.1 onwards), Range Partitioning, and List Partitioning (from v3.1 onwards) are supported in all table types. #### Bucketing[​](#bucketing "Direct link to Bucketing") | Feature | Key point | Support status | Note | | -------------------- | ---------------------------------------------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Bucketing strategy | Hash Bucketing | Yes | | | | Random Bucketing | Yes (v3.1+) | Random Bucketing is supported **only in Duplicate Key tables**.
From v3.2, StarRocks supports dynamically adjusting the number of tablets to create according to cluster information and the data size. | | Bucket Key data type | Date, Integer, String | Yes | | | Bucket number | Automatically set the number of buckets | Yes (v3.0+) | Automatically determined by the number of BE nodes or the data volume of the largest historical partition.
The logic has been optimized separately for partitioned tables and non-partitioned tables in later versions. | | | Dynamic increase of the Bucket number for Random Bucketing | Yes (v3.2+) | | note From v4.1 onwards, the maximum size of a tablet is changed to 100 GB. #### Partitioning[​](#partitioning "Direct link to Partitioning") | Feature | Key point | Support status | Note | | ----------------------- | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Partitioning strategy | Expression Partitioning | Yes (v3.1+) | - Including Partitioning based on a time function expression (since v3.0) and Partitioning based on the column expression (since v3.1)
- Supported time functions: date\_trunc, time\_slice | | | Range Partitioning | Yes (v3.2+) | Since v3.3.0, three specific time functions can be used for Partition Keys: from\_unixtime, from\_unixtime\_ms, str2date, substr/substring. | | | List Partitioning | Yes (v3.1+) | | | Partition Key data type | Date, Integer, Boolean | Yes | | | | String | Yes | - Only Expression Partitioning and List Partitioning support String-type Partition Key.
- Range Partitioning does not support String-type Partition Key. You need to use str2date to transform the column to date types. | ##### Differences between partitioning strategies[​](#differences-between-partitioning-strategies "Direct link to Differences between partitioning strategies") | | Expression Partitioning | | Range Partitioning | List Partitioning | | --------------------------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | | Time function expression-based Partitioning | Column expression-based Partitioning | | | | Data type | Date (DATE/DATETIME) | - String (except BINARY)
- Date (DATE/DATETIME)
- Integer and Boolean | - String (except BINARY) \[1]
- Date or timestamp \[1]
- Integer | - String (except BINARY)
- Date (DATE/DATETIME)
- Integer and Boolean | | Support for multiple Partition Keys | / (Only supports one date-type Partition Key) | Yes | Yes | Yes | | Support Null values for Partition Keys | Yes | / \[2] | Yes | / \[2] | | Manual creation of partitions before data loading | / \[3] | / \[3] | - Yes if the partitions are manually created in batch
- No if the dynamic partitioning strategy is adopted | Yes | | Automatic creation of partitions while data loading | Yes | Yes | / | / | note * \[1]: You need to use from\_unixtime, str2date or other time functions to transform the column to date types. * \[2]: Null values will be supported in Partition Keys for List Partitioning from v3.3.3 onwards. * \[3]: Partitions are automatically created. For detailed comparisons between List Partitioning and Expression Partitioning, refer to [Comparison between list partitioning and expression partitioning](https://docs.starrocks.io/docs/table_design/data_distribution/list_partitioning.md). --- ### List partitioning (Legacy) Since v3.1, StarRocks supports list partitioning. Data is partitioned based on a predefined value list for each partition, which can accelerate queries and facilitate management according to enumerated values. note Please note that from v3.4 onwards, [expression partitioning](https://docs.starrocks.io/docs/table_design/data_distribution/expression_partitioning.md) is further optimized to unify all partitioning strategies and supported more complex solutions. It is recommended in most cases, and will replace the list partitioning strategy in future releases. #### Introduction[​](#introduction "Direct link to Introduction") You need to explicitly specify the column values list in each partition. These values do not need to be continuous, unlike the continuous time or numeric range required in Range Partitioning. During data loading, StarRocks will store the data in the corresponding partition based on the mapping between the data's partitioning column values and the predefined column values for each partition. ![list\_partitioning](/assets/images/list_partitioning-2e8c3baefb129ca890e75168adf2def5.png) List partitioning is suitable for storing data whose columns contain a small number of enum values, and you often query and manage data based on these enum values. For example, columns represent geographical locations, states, and categories. Each value in a column represents an independent category. By partitioning data based on enum values, you can improve query performance and facilitate data management. **List partitioning is particularly useful for scenarios where a partition needs to include multiple values for each partitioning column**. For example, if a table includes the `City` column representing the city where an individual is from, and you often query and manage data by states and cities. At table creation, you can use the `City` column as the partitioning column for List Partitioning and specify that data of various cities within the same state are placed in one partition, for example `PARTITION pCalifornia VALUES IN ("Los Angeles", "San Francisco", "San Diego")`. This method can accelerate queries based on states and cities while facilitating data management. If a partition only needs to contain data with the same value of each partitioning column, it is recommended to use [expression partitioning](https://docs.starrocks.io/docs/table_design/data_distribution/expression_partitioning.md). **Comparison between list partitioning and [expression partitioning](https://docs.starrocks.io/docs/table_design/data_distribution/expression_partitioning.md)** The main difference between list partitioning and expression partitioning (recommended) is that list partitioning requires you to manually create partitions one by one. On the other hand, expression partitioning can automatically create partitions during loading to simplify partitioning. And in most cases, expression partitioning can replace list partitioning. The specific comparison between the two is shown in the following table: | Partitioning method | **List partitioning** | **Expression partitioning** | | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Syntax | `PARTITION BY LIST (partition_columns)( PARTITION VALUES IN (value_list) [, ...] )` | `PARTITION BY ` | | Multiple values for each partitioning column in a partition | Supported. A partition can store data with different values in each partitioning column. In the following example, when the loaded data contains values `Los Angeles`, `San Francisco`, and `San Diego` in the `city` column, all the data is stored in one partition. `pCalifornia`.`PARTITION BY LIST (city) ( PARTITION pCalifornia VALUES IN ("Los Angeles","San Francisco","San Diego") [, ...] )` | Not supported. A partition stores data with the same value in the partitioning column For example, the expression `PARTITION BY (city)` is used in expression partitioning. When the loaded data contains values `Los Angeles`, `San Francisco`, and `San Diego` in the `city` column, StarRocks will automatically create three partitions `pLosAngeles`, `pSanFrancisco`, and `pSanDiego`. The three partitions respectively store data with values `Los Angeles,` `San Francisco`, and `San Diego` in the `city` column. | | Create partitions before data loading | Supported. Partitions need to be created at table creation. | No need to do so. Partitions can be automatically created during data loading. | | Automatically create List partitions during data loading | Not supported. If a partition corresponding to the data does not exist during data loading, an error is returned. | Supported. If a partition corresponding to the data does not exist during data loading, StarRocks automatically creates the partition to store the data. Each partition can only contain data with the same value for the partitioning column. | | SHOW CREATE TABLE | Returned the partitions definition in the CREATE TABLE statement. | After data is loaded, the statement returns the result with the partition clause used in the CREATE TABLE statement, that is, `PARTITION BY (partition_columns)`. However, the returned result does not show any automatically created partitions. If you need to view the automatically created partitions, execute `SHOW PARTITIONS FROM `. | #### Usage[​](#usage "Direct link to Usage") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql PARTITION BY LIST (partition_columns) ( PARTITION VALUES IN (value_list) [, ...] ) partition_columns::= [, [, ...] ] value_list ::= value_item [, value_item [, ...] ] value_item ::= { | ( [, , [, ...] ] ) } ``` ##### Parameters[​](#parameters "Direct link to Parameters") | **Parameters** | **Parameters** | **Description** | | ------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `partition_columns` | YES | the names of the partitioning columns. The partitioning column values can be string (BINARY not supported), date or datetime, integer, and boolean values. From v3.3.3, the partitioning column allows `NULL` values. | | `partition_name` | YES | Partition name. It is recommended to set appropriate partition names based on the business scenario to differentiate the data in different partitions. | | `value_list` | YES | A list of partitioning column values in a partition. | ##### Examples[​](#examples "Direct link to Examples") Example 1: Suppose you frequently query details of data center billing based on states or cities. At table creation, you can specify the partitioning column as `city` and specify that each partition stores data of cities within the same state. This method can accelerate queries for specific states or cities and facilitate data management. ```sql CREATE TABLE t_recharge_detail1 ( id bigint, user_id bigint, recharge_money decimal(32,2), city varchar(20) not null, dt varchar(20) not null ) DUPLICATE KEY(id) PARTITION BY LIST (city) ( PARTITION pLos_Angeles VALUES IN ("Los Angeles"), PARTITION pSan_Francisco VALUES IN ("San Francisco") ) DISTRIBUTED BY HASH(`id`); ``` Example 2: Suppose you frequently query details of data center billing based on time ranges and specific states or cities. During table creation, you can specify the partitioning columns as `dt` and `city`. This way, data of specific dates and specific states or cities is stored into the same partition, improving query speed and facilitating data management. ```sql CREATE TABLE t_recharge_detail4 ( id bigint, user_id bigint, recharge_money decimal(32,2), city varchar(20) not null, dt varchar(20) not null ) ENGINE=OLAP DUPLICATE KEY(id) PARTITION BY LIST (dt,city) ( PARTITION p202204_California VALUES IN ( ("2022-04-01", "Los Angeles"), ("2022-04-01", "San Francisco"), ("2022-04-02", "Los Angeles"), ("2022-04-02", "San Francisco") ), PARTITION p202204_Texas VALUES IN ( ("2022-04-01", "Houston"), ("2022-04-01", "Dallas"), ("2022-04-02", "Houston"), ("2022-04-02", "Dallas") ) ) DISTRIBUTED BY HASH(`id`); ``` #### Limits[​](#limits "Direct link to Limits") * List partitioning does support dynamic partitioning and creating multiple partitions at a time. * Currently, StarRocks's shared-data mode does not support this feature. * When the `ALTER TABLE DROP PARTITION ;` statement is used to delete a partition created by using list partitioning, data in the partition is directly removed and cannot be recovered. * From v3.4.0, v3.3.8, v3.2.13, and v3.1.16 onwards, StarRocks supports [backing up and restoring](https://docs.starrocks.io/docs/administration/management/Backup_and_restore.md) tables created with the list partitioning strategy. * From v3.3.5 onwards, StarRocks supports creating [asynchronous materialized views](https://docs.starrocks.io/docs/using_starrocks/async_mv/Materialized_view.md) with base tables created with the list partitioning strategy. --- ### Temporary partition This topic describes how to use the temporary partition feature. You can create temporary partitions on a partitioned table that already has defined partitioning rules and define new data distribution strategies for these temporary partitions. Temporary partitions can serve as temporary data carriers when you atomically overwrite data in a partition or when you adjust partitioning and bucketing strategies. For temporary partitions, you can reset data distribution strategies such as partition ranges, number of buckets, and properties such as the number of replicas, and the storage medium, to meet specific requirements. You can use the temporary partition feature in the following scenarios: * Atomic overwrite operation If you need to rewrite the data in a partition while ensuring that the data can be queried during the rewriting process, you can first create a temporary partition based on the original formal partition, and load the new data into the temporary partition. Then you can use the replace operation to atomically replace the original formal partition with the temporary partition. For atomic overwrite operations on non-partitioned tables, see [ALTER TABLE - SWAP](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md#swap). * Adjust partition data query concurrency If you need to modify the number of buckets for a partition, you can first create a temporary partition with the same partition range as the original formal partition and specify the new number of buckets. Then, you can use the `INSERT INTO` command to load the data of the original formal partition into the temporary partition. Finally, you can use the replace operation to atomically replace the original formal partition with the temporary partition. * Modify partitioning rules If you want to modify a partitioning strategy, such as merging partitions or splitting a large partition into multiple smaller partitions, you can first create temporary partitions with the expected merged or split ranges. Then, you can use the `INSERT INTO` command to load the data of the original formal partitions into the temporary partitions. Finally, you can use the replace operation to atomically replace the original formal partitions with the temporary partitions. #### Create temporary partitions[​](#create-temporary-partitions "Direct link to Create temporary partitions") You can create one or more partitions at a time by using the [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md) command. ##### Syntax[​](#syntax "Direct link to Syntax") ###### Create a single temporary partition[​](#create-a-single-temporary-partition "Direct link to Create a single temporary partition") ```sql ALTER TABLE ADD TEMPORARY PARTITION VALUES [("value1"), {MAXVALUE|("value2")})] [(partition_desc)] [DISTRIBUTED BY HASH()]; ALTER TABLE ADD TEMPORARY PARTITION VALUES LESS THAN {MAXVALUE|(<"value">)} [(partition_desc)] [DISTRIBUTED BY HASH()]; ``` ###### Create multiple partitions at a time[​](#create-multiple-partitions-at-a-time "Direct link to Create multiple partitions at a time") ```sql ALTER TABLE ADD TEMPORARY PARTITIONS START ("value1") END ("value2") EVERY {(INTERVAL )|} [(partition_desc)] [DISTRIBUTED BY HASH()]; ``` ##### Parameters[​](#parameters "Direct link to Parameters") `partition_desc`: specifies the number of buckets and properties for temporary partitions, such as the number of replicas and the storage medium. ##### Examples[​](#examples "Direct link to Examples") Create a temporary partition `tp1` in the table `site_access` and specify its range as `[2020-01-01, 2020-02-01)` by using the `VALUES [(...), (...)]` syntax. ```sql ALTER TABLE site_access ADD TEMPORARY PARTITION tp1 VALUES [("2020-01-01"), ("2020-02-01")); ``` Create a temporary partition `tp2` in the table `site_access` and specify its upper bound as `2020-03-01` by using the `VALUES LESS THAN (...)` syntax. StarRocks uses the upper bound of the previous temporary partition as the lower bound of this temporary partition, generating a temporary partition with the left-closed and right-open range of `[2020-02-01, 2020-03-01)`. ```sql ALTER TABLE site_access ADD TEMPORARY PARTITION tp2 VALUES LESS THAN ("2020-03-01"); ``` Create a temporary partition `tp3` in the table `site_access`, specify its upper bound as `2020-04-01` by using the `VALUES LESS THAN (...)` syntax, and specify the number of replicas as `1`. ```sql ALTER TABLE site_access ADD TEMPORARY PARTITION tp3 VALUES LESS THAN ("2020-04-01") ("replication_num" = "1") DISTRIBUTED BY HASH (site_id); ``` Create multiple partitions at a time in the table `site_access` by using the `START (...) END (...) EVERY (...)` syntax, and specify the range of these partitions as `[2020-04-01, 2021-01-01)` with a monthly partition granularity. ```sql ALTER TABLE site_access ADD TEMPORARY PARTITIONS START ("2020-04-01") END ("2021-01-01") EVERY (INTERVAL 1 MONTH); ``` ##### Usage notes[​](#usage-notes "Direct link to Usage notes") * The partition column for a temporary partition must be the same as the partition column for the original formal partition based on which you create the temporary partition and cannot be changed. * The name of a temporary partition cannot be the same as the name of any formal partition or other temporary partition. * The ranges of all temporary partitions in a table cannot overlap, but the ranges of a temporary partition and a formal partition can overlap. #### Show temporary partitions[​](#show-temporary-partitions "Direct link to Show temporary partitions") You can view the temporary partitions by using the [SHOW TEMPORARY PARTITIONS](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md) command. ```sql SHOW TEMPORARY PARTITIONS FROM [db_name.]table_name [WHERE] [ORDER BY] [LIMIT] ``` #### Load data into temporary partitions[​](#load-data-into-temporary-partitions "Direct link to Load data into temporary partitions") You can load data into one or more temporary partitions by using the `INSERT INTO` command, STREAM LOAD, or BROKER LOAD. ##### Load data by using `INSERT INTO` command[​](#load-data-by-using-insert-into-command "Direct link to load-data-by-using-insert-into-command") Example: ```sql INSERT INTO site_access TEMPORARY PARTITION (tp1) VALUES ("2020-01-01",1,"ca","lily",4); INSERT INTO site_access TEMPORARY PARTITION (tp2) SELECT * FROM site_access_copy PARTITION p2; INSERT INTO site_access TEMPORARY PARTITION (tp3, tp4,...) SELECT * FROM site_access_copy PARTITION (p3, p4,...); ``` For detailed syntax and parameter descriptions, see [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). ##### Load data by using STREAM LOAD[​](#load-data-by-using-stream-load "Direct link to Load data by using STREAM LOAD") Example: ```bash curl --location-trusted -u root: -H "label:123" -H "Expect:100-continue" -H "temporary_partitions: tp1, tp2, ..." -T testData \ http://host:port/api/example_db/site_access/_stream_load ``` For detailed syntax and parameter descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ##### Load data by using BROKER LOAD[​](#load-data-by-using-broker-load "Direct link to Load data by using BROKER LOAD") Example: ```sql LOAD LABEL example_db.label1 ( DATA INFILE("hdfs://hdfs_host:hdfs_port/user/starrocks/data/input/file") INTO TABLE my_table TEMPORARY PARTITION (tp1, tp2, ...) ... ) WITH BROKER ( StorageCredentialParams ); ``` Note that `StorageCredentialParams` represents a group of authentication parameters which vary depending on the authentication method you choose. For detailed syntax and parameter descriptions, see [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ##### Load data by using ROUTINE LOAD[​](#load-data-by-using-routine-load "Direct link to Load data by using ROUTINE LOAD") Example: ```sql CREATE ROUTINE LOAD example_db.site_access ON example_tbl COLUMNS(col, col2,...), TEMPORARY PARTITIONS(tp1, tp2, ...) FROM KAFKA ( "kafka_broker_list" =":,:", "kafka_topic" = "ordertest" ); ``` For detailed syntax and parameter descriptions, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). #### Query data in temporary partitions[​](#query-data-in-temporary-partitions "Direct link to Query data in temporary partitions") You can use the [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) statement to query data in specified temporary partitions. ```sql SELECT * FROM site_access TEMPORARY PARTITION (tp1); SELECT * FROM site_access TEMPORARY PARTITION (tp1, tp2, ...); SELECT event_day,site_id,pv FROM site_access TEMPORARY PARTITION (tp1, tp2, ...); ``` You can use the JOIN clause to query data in temporary partitions from two tables. ```sql SELECT * FROM site_access TEMPORARY PARTITION (tp1, tp2, ...) JOIN site_access_copy TEMPORARY PARTITION (tp1, tp2, ...) ON site_access.site_id=site_access1.site_id and site_access.event_day=site_access1.event_day; ``` #### Replace original formal partitions with temporary partitions[​](#replace-original-formal-partitions-with-temporary-partitions "Direct link to Replace original formal partitions with temporary partitions") You can use the [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md) statement to replace the original formal partition with a temporary partition, thereby creating a new formal partition. > **NOTE** > > The original formal partition and temporary partition that you operated in the ALTER TABLE statement are deleted and cannot be recovered. ##### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql ALTER TABLE table_name REPLACE PARTITION (partition_name) WITH TEMPORARY PARTITION (temporary_partition_name1, ...) PROPERTIES ("key" = "value"); ``` ##### Parameters[​](#parameters-1 "Direct link to Parameters") * **strict\_range** Default value: `true`. When this parameter is set to `true`, the unions of ranges of all original formal partitions must be exactly the same as the unions of ranges of the temporary partitions that are used for replacement. When this parameter is set to `false`, you only need to make sure that the ranges of the new formal partitions do not overlap with other formal partitions after replacement. * Example 1: In the following example, the unions of the original formal partitions `p1`, `p2`, and `p3` are the same as the unions of the temporary partitions `tp1` and `tp2`, and you can use `tp1` and `tp2` to replace `p1`, `p2`, and `p3`. ```plaintext # Ranges of original formal partitions p1, p2, and p3 => Unions of these ranges [10, 20), [20, 30), [40, 50) => [10, 30), [40, 50) # Ranges of temporary partitions tp1 and tp2 => Unions of these ranges [10, 30), [40, 45), [45, 50) => [10, 30), [40, 50) ``` * Example 2: In the following example, the union of ranges of the original formal partition differs from the union of ranges of the temporary partitions. If the value of the parameter `strict_range` is set to `true`, the temporary partitions `tp1` and `tp2` cannot replace the original formal partition `p1`. If the value is set to `false`, and the ranges \[10, 30) and \[40, 50) of the temporary partitions do not overlap with other formal partitions, the temporary partitions can replace the original formal partition. ```plaintext # Range of original formal partition p1 => Union of the range [10, 50) => [10, 50) # Ranges of temporary partitions tp1 and tp2 => Unions of these ranges [10, 30), [40, 50) => [10, 30), [40, 50) ``` * **use\_temp\_partition\_name** Default value: `false`. If the number of original formal partitions is the same as the number of temporary partitions used for replacement, the names of the new formal partitions remain unchanged after replacement when this parameter is set to `false`. When this parameter is set to `true`, the names of the temporary partitions are used as the names of the new formal partitions after replacement. In the following example, the partition name of the new formal partition remains `p1` after replacement when this parameter is set to `false`. However, its related data and properties are replaced with the data and properties of the temporary partition `tp1`. When this parameter is set to `true`, the partition name of the new formal partition is changed to `tp1` after replacement. The original formal partition `p1` no longer exists. ```sql ALTER TABLE tbl1 REPLACE PARTITION (p1) WITH TEMPORARY PARTITION (tp1); ``` If the number of formal partitions to be replaced is different from the number of temporary partitions used for replacement, and this parameter remains the default value `false`, the value `false` of this parameter is invalid. In the following example, after replacement, the name for the new formal partition is changed to `tp1`, and the original formal partitions `p1` and `p2` no longer exist. ```sql ALTER TABLE site_access REPLACE PARTITION (p1, p2) WITH TEMPORARY PARTITION (tp1); ``` ##### Examples[​](#examples-1 "Direct link to Examples") Replace the original formal partition `p1` with the temporary partition `tp1`. ```sql ALTER TABLE site_access REPLACE PARTITION (p1) WITH TEMPORARY PARTITION (tp1); ``` Replace the original formal partitions `p2` and `p3` with the temporary partitions `tp2` and `tp3`. ```sql ALTER TABLE site_access REPLACE PARTITION (p2, p3) WITH TEMPORARY PARTITION (tp2, tp3); ``` Replace the original formal partitions `p4` and `p5` with the temporary partitions `tp4` and `tp5`, and specify the parameters `strict_range` as `false` and `use_temp_partition_name` as `true`. ```sql ALTER TABLE site_access REPLACE PARTITION (p4, p5) WITH TEMPORARY PARTITION (tp4, tp5) PROPERTIES ( "strict_range" = "false", "use_temp_partition_name" = "true" ); ``` ##### Usage notes[​](#usage-notes-1 "Direct link to Usage notes") * When a table has temporary partitions, you cannot use the `ALTER` command to perform Schema Change operations on the table. * When performing Schema Change operations on a table, you cannot add temporary partitions to the table. #### Delete temporary partitions[​](#delete-temporary-partitions "Direct link to Delete temporary partitions") Use the following command to delete the temporary partition `tp1`. ```sql ALTER TABLE site_access DROP TEMPORARY PARTITION tp1; ``` Take note of the following limits: * If you use the `DROP` command to directly delete a database or table, you can recover the database or table within a limited time period by using the `RECOVER` command. However, temporary partitions cannot be recovered. * After using the `ALTER` command to delete a formal partition, you can recover it within a limited time period by using the `RECOVER` command. Temporary partitions are not bound with formal partitions, so operations on temporary partitions do not affect formal partitions. * After using the `ALTER` command to delete a temporary partition, you cannot recover it by using the `RECOVER` command. * When using the `TRUNCATE` command to delete data in a table, the temporary partitions of the table are deleted and cannot be recovered. * When using the `TRUNCATE` command to delete data in a formal partition, the temporary partitions are not affected. * The TRUNCATE command cannot be used to delete data in a temporary partition. --- ### Hybrid row-column storage Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) As an OLAP database, StarRocks originally stores data in the columnar storage, which can enhance the performance of complex queries, such as aggregate queries. Since v3.2.3, StarRocks also supports storing data in the hybrid row-column storage where data is stored in both row-by-row and column-by-column fashions. This hybrid row-column storage is well suited for various scenario such as primary key-based high concurrency, low-latency point queries and partial column updates, while delivering efficient analytical capabilities comparable to columnar storage. Additionally, hybrid row-column storage supports [prepared statements](https://docs.starrocks.io/docs/sql-reference/sql-statements/prepared_statement.md), which enhances query performance and security. #### Comparisons between columnar storage and hybrid row-column storage[​](#comparisons-between-columnar-storage-and-hybrid-row-column-storage "Direct link to Comparisons between columnar storage and hybrid row-column storage") **Hybrid row-column storage** * Storage method: Data is stored in both row-by-row and column-by-column fashions. Simply put, a table that uses hybrid row-column storage contains an additional, hidden binary-type column `__row`. When data is written to the table, all values from the value columns of each involved row are encoded and written to the `__row` column (as shown below). As the data is stored in both row-by-row and column-by-column fashions, additional storage costs are incurred. ![img](/assets/images/hybrid_table-77d0b8e2ea5481cb53c6c7107a76cc79.png) * Scenarios: supports the user scenarios of both row-by-row and column-by-column storage, but incurs additional storage costs. * User scenarios of row-by-row storage: * * High concurrency point queries based on primary keys. * Queries against most fields from tables that consist of a small number of fields. * Partial column updates (more specifically, multiple columns and a small number of data rows need to be updated) User scenarios of column-by-column storage: Complex data analysis. **Column-oriented** * Storage method: Data is stored in a column-by-column fashion. ![img](/assets/images/columnar_table-52c2a4d3c129963b8def5541d649618a.png) * Scenarios: Complex data analysis. * Complex queries and analyses on massive datasets, such as aggregate analysis and multi-table join queries. * Tables consist of many fields (such as wide tables), but queries against these tables involve only a few columns. #### Basic usages[​](#basic-usages "Direct link to Basic usages") ##### Create a table that uses hybrid row-column storage[​](#create-a-table-that-uses-hybrid-row-column-storage "Direct link to Create a table that uses hybrid row-column storage") 1. Enable the FE configuration item `enable_experimental_rowstore`. ```sql ADMIN SET FRONTEND CONFIG ("enable_experimental_rowstore" = "true"); ``` 2. Specify `"STORE_TYPE" = "column_with_row"` in the `PROPERTIES` at table creation. note * The table must be a Primary Key table. * The length of the `__row` column cannot exceed 1 MB. * Since v3.2.4, StarRocks extends support to the following column types: BITMAP, HLL, JSON, ARRAY, MAP, and STRUCT. ```sql CREATE TABLE users ( id bigint not null, country string, city string, revenue bigint ) PRIMARY KEY (id) DISTRIBUTED by HASH (id) PROPERTIES ("store_type" = "column_with_row"); ``` ##### Insert, delete, and update data[​](#insert-delete-and-update-data "Direct link to Insert, delete, and update data") Similar to a table that uses columnar storage, you can insert, delete, and update data on a table that uses hybrid row-column storage by data loading and DML statements. This section demonstrates how to run DML statements on the above table with hybrid row-column storage. 1. Insert a data row. 1. ```sql INSERT INTO users (id, country, city, revenue) VALUES (1, 'USA', 'New York', 5000), (2, 'UK', 'London', 4500), (3, 'France', 'Paris', 6000), (4, 'Germany', 'Berlin', 4000), (5, 'Japan', 'Tokyo', 7000), (6, 'Australia', 'Sydney', 7500); ``` 2. Delete a data row. ```sql DELETE FROM users WHERE id = 6; ``` 3. Update a data row. ```sql UPDATE users SET revenue = 6500 WHERE id = 4; ``` ##### Query data[​](#query-data "Direct link to Query data") This section uses a point query as an example. Point queries take a short circuit, directly querying data in row storage, which can improve query performance. The following example still uses the above table with hybrid row-column storage. After the table creation and data modification operations mentioned above, the table stores the data as follows: ```sql MySQL [example_db]> SELECT * FROM users ORDER BY id; +------+---------+----------+---------+ | id | country | city | revenue | +------+---------+----------+---------+ | 1 | USA | New York | 5000 | | 2 | UK | London | 4500 | | 3 | France | Paris | 6000 | | 4 | Germany | Berlin | 6500 | | 5 | Japan | Tokyo | 7000 | +------+---------+----------+---------+ 5 rows in set (0.03 sec) ``` 1. Make sure that the system enables short circuiting for queries. Once short circuiting for queries is enabled, queries that meet certain criteria (to evaluate whether the query is a point query) take a short circuit to scan data in row-oriented storage. ```sql SHOW VARIABLES LIKE '%enable_short_circuit%'; ``` If the short circuiting for queries is not enabled, run the `SET enable_short_circuit = true;` command to set the variable [`enable_short_circuit`](https://docs.starrocks.io/docs/sql-reference/System_variable.md) to `true`. 2. Query data. If the query meets the criteria that conditional columns in the WHERE clause include all primary key columns, and the operators in the WHERE clause are `=` or `IN`, the query takes the shortcut. note The conditional columns in the WHERE clause can include additional columns beyond all primary key columns. ```sql SELECT * FROM users WHERE id=1; ``` 3. Check the query plan to verify whether the query can use the short circuit. If the query plan includes `Short Circuit Scan: true`, the query can take the short circuit. ```sql MySQL [example_db]> EXPLAIN SELECT * FROM users WHERE id=1; +---------------------------------------------------------+ | Explain String | +---------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS:1: id | 2: country | 3: city | 4: revenue | | PARTITION: RANDOM | | | | RESULT SINK | | | | 0:OlapScanNode | | TABLE: users | | PREAGGREGATION: OFF. Reason: null | | PREDICATES: 1: id = 1 | | partitions=1/1 | | rollup: users | | tabletRatio=1/6 | | tabletList=10184 | | cardinality=-1 | | avgRowSize=0.0 | | Short Circuit Scan: true | -- The query can use the shortcut. +---------------------------------------------------------+ 17 rows in set (0.00 sec) ``` ##### Use prepared statements[​](#use-prepared-statements "Direct link to Use prepared statements") You can use the [prepared statements](https://docs.starrocks.io/docs/sql-reference/sql-statements/prepared_statement.md#use-prepared-statements) to query data in the tables with hybrid row-column storage. ```sql -- Prepare the statements for execution. PREPARE select_all_stmt FROM 'SELECT * FROM users'; PREPARE select_by_id_stmt FROM 'SELECT * FROM users WHERE id = ?'; -- Declare variables in the statements. SET @id1 = 1, @id2 = 2; -- Use the declared variables to execute the statements. -- Query data with ID 1 or 2 separately. EXECUTE select_by_id_stmt USING @id1; EXECUTE select_by_id_stmt USING @id2; ``` #### Limits[​](#limits "Direct link to Limits") * Currently, StarRocks shared-data clusters do not support hybrid row-column storage. * Since v3.2.4, the table with hybrid row-column storage can be altered by using [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md). * The short circuiting for queries is currently only suitable for queries that happen after scheduled batch data loading. Because mutual exclusion of indexes may be incurred when the short circuiting for queries happens at the apply stage of the data writing process, data writing may block short circuiting for queries, affecting the response time of point queries during data writing. * Hybrid row-column storage may significantly increase storage consumption. This is because data is stored in both row and column formats, and the data compression ratio of row storage may not be as high as that of column storage. * Hybrid row-column storage can increase the time and resource consumption during data loading. * The tables with hybrid row-column storage can be a viable solution for online services, but the performance of this type of table may not compete with mature OLTP databases. * The tables with hybrid row-column storage do not support features that rely on columnar storage, such as partial updates in column mode. * The tables with hybrid row-column storage must be Primary Key tables. --- ### Indexes Indexes can help quickly locate data that matches certain query conditions. To be specific, when an index is built on a table based on some of the table's columns, queries that use these columns as query conditions require scanning only on portions of the table data in order to locate the data that meet the conditions. As such, query efficiency is improved. StarRocks provides a variety of index types, mainly divided into the following two categories: * StarRocks provides built-in indexes which are automatically created by the system, including Prefix indexes, Ordinal indexes, and ZoneMap indexes. * StarRocks also allows users to manually create indexes, including Bitmap indexes and Bloom filter indexes. #### Built-in indexes[​](#built-in-indexes "Direct link to Built-in indexes") ##### [Prefix indexes](https://docs.starrocks.io/docs/table_design/indexes/Prefix_index_sort_key.md)[​](#prefix-indexes "Direct link to prefix-indexes") The Prefix index is automatically generated during data writing. To more specifically, when data is written into StarRocks, the data is sorted according to the specified sort key, and every 1024 rows of data are included in one logical data block. An index entry that consists of the values of sort key columns of the first data row in that logical data block is added to the Prefix index table. When the filtering condition in a query match the prefix of the Prefix index, the Prefix index can quickly locate the data that meets the conditions, and reduce the amount of data scanned, which can significantly improve query performance. ##### Ordinal indexes[​](#ordinal-indexes "Direct link to Ordinal indexes") StarRocks actually adopts columnar storage in the underlying storage. Data per column is stored in data pages, and the size of each data page's is generally 64 \* 1024 bytes (data\_page\_size = 64 \* 1024). An Ordinal index entry is added at the same time when a data page is generated. The Ordinal index entry contains information such as the starting row number of the data page. In this way, the Ordinal index can locate the physical address of the column Data Page data page by using the row number. ![img](/assets/images/3.1-2-4e9162f47c93451c8dd804335a5d7baa.png) ##### ZoneMap indexes[​](#zonemap-indexes "Direct link to ZoneMap indexes") ZoneMap index stores each data chunk's statistics that include Min (maximum value), Max (minimum value), HasNull (presence of null values), and HasNotNull (presence of non-null values) information. During queries, StarRocks can rapidly evaluate whether these data chunks can be filtered out based on their statistics, reducing the scanned data volume and improving query speed. More information Each data chunk can be a segment, or a data page of a column. So correspondingly, two types of ZoneMap indexes exist: one storing statistics for each Segment and the other for each data page of a column. #### Manually created indexes[​](#manually-created-indexes "Direct link to Manually created indexes") If the column in the query condition is not a prefix field, you can manually create an index to this column based on the characteristics of data and queries to improve query efficiency. ##### [Bitmap indexes](https://docs.starrocks.io/docs/table_design/indexes/Bitmap_index.md)[​](#bitmap-indexes "Direct link to bitmap-indexes") The bitmap index is suitable for queries on high cardinality columns or combinations of multiple low cardinality columns. And the bitmap index can exhibit ideal filtering performance for those queries, filtering out at least 999 out of 1000 rows. ##### [Bloom filter indexes](https://docs.starrocks.io/docs/table_design/indexes/Bloomfilter_index.md)[​](#bloom-filter-indexes "Direct link to bloom-filter-indexes") The bloom filter index is suitable for columns with relatively high cardinality, such as ID columns, but it may incur a certain rate of misjudgment. ##### [N-Gram bloom filter indexes](https://docs.starrocks.io/docs/table_design/indexes/Ngram_Bloom_Filter_Index.md)[​](#n-gram-bloom-filter-indexes "Direct link to n-gram-bloom-filter-indexes") The n-gram bloom filter index is a special type of Bloom filter index, typically used to speed up `LIKE` queries or the operations of the `ngram_search` and `ngram_search_case_insensitive` functions. ##### [Full-Text inverted indexes](https://docs.starrocks.io/docs/table_design/indexes/inverted_index.md)[​](#full-text-inverted-indexes "Direct link to full-text-inverted-indexes") A full-text inverted index can quickly locate data rows that match keywords, thereby speeding up full-text searches. ##### [Vector indexes](https://docs.starrocks.io/docs/table_design/indexes/vector_index.md)[​](#vector-indexes "Direct link to vector-indexes") A vector index allows you to perform an approximate nearest neighbor search (ANNS) with StarRocks. --- ### Bitmap indexes This topic describes how to create and manage a bitmap index, along with usage cases. #### Introduction[​](#introduction "Direct link to Introduction") A bitmap index is a special database index that uses bitmaps, which are an array of bits. A bit is always in one of two values: 0 and 1. Each bit in the bitmap corresponds to a single row in the table. The value of each bit depends upon the value of the corresponding row. A bitmap index can help improve the query performance on a given column. If a query's filter conditions match a prefix index, it can significantly improve query efficiency and quickly return results. However, a table can only have one prefix index. If the query's filter conditions do not include the prefix of the prefix index, a bitmap index can be created for that column to improve query efficiency. ##### How to design a bitmap index to accelerate queries[​](#how-to-design-a-bitmap-index-to-accelerate-queries "Direct link to How to design a bitmap index to accelerate queries") The primary considerations for choosing a bitmap index are **column cardinality and the filtering effect of the bitmap index on queries**. Contrary to popular belief, bitmap indexes in StarRocks are more suitable for **queries on columns with high cardinality and queries on the combination of multiple low cardinality columns. Furthermore, bitmap indexes need to effectively filter out data, potentially filtering out at least 999/1000 of the data, thereby reducing the amount of Page data read.** For queries on a single low cardinality column, the filtering effect of a bitmap index is poor, because too many rows need to be read and scattered across multiple Pages. info You need to consider the cost of loading data when evaluating the filtering effect of bitmap indexes on queries. In StarRocks, underlying data is organized and loaded by Pages (default size is 64K). The cost of loading data includes the time to load Pages from disk, decompress Pages, and decode. However, excessively high cardinality can also cause issues such as **occupying more disk space**, and because bitmap indexes need to be constructed during data loading, frequent data loading can **affect loading performance**. Additionally, the **overhead of loading bitmap indexes during queries** should be considered. During a query, bitmap indexes are loaded on demand, and the larger the value of `number of column values involved in query conditions/cardinality x bitmap index`, the greater the overhead of loading bitmap indexes during queries. To determine the appropriate cardinality and query conditions for bitmap indexes, it is recommended to refer to the [Performance test on bitmap index](#performance-test-on-bitmap-index) in this topic to conduct performance tests. You can use actual business data and queries to **create bitmap indexes on columns of different cardinalities, to analyze the filtering effect of bitmap indexes on queries (at least filtering out 999/1000 of the data), the disk space usage, the impact on loading performance, and the overhead of loading bitmap indexes during queries.** StarRocks has a built-in [adaptive selection mechanism for bitmap indexes](#adaptive-selection-of-bitmap-indexes). If a bitmap index fails to accelerate queries, for example, if it cannot filter out many Pages, or the overhead of loading bitmap indexes during queries is high, it will not be used during the query, so query performance will not be significantly affected. ##### Adaptive selection of bitmap indexes[​](#adaptive-selection-of-bitmap-indexes "Direct link to Adaptive selection of bitmap indexes") StarRocks can adaptively choose whether to use a bitmap index based on column cardinality and query conditions. If a bitmap index does not effectively filter out many Pages or the overhead of loading bitmap indexes during queries is high, StarRocks will not use the bitmap index by default to avoid degrading query performance. StarRocks determines whether to use a bitmap index based on the ratio of the number of values involved in the query condition to the column cardinality. Generally, the smaller this ratio, the better the filtering effect of the bitmap index. Thus, StarRocks uses `bitmap_max_filter_ratio/1000` as the threshold. When the `number of values in the filter condition/column cardinality` is less than `bitmap_max_filter_ratio/1000`, the bitmap index will be used. The default value of `bitmap_max_filter_ratio` is `1`. Take a query based on a single column as an example, such as `SELECT * FROM employees WHERE gender = 'male';`. The `gender` column in the `employees` table has values 'male' and 'female', so the cardinality is 2 (two distinct values). The query condition involves one value, so the ratio is 1/2, which is greater than 1/1000. Therefore, this query will not use the bitmap index. Take another query based on a combination of multiple columns as an example, such as `SELECT * FROM employees WHERE gender = 'male' AND city IN ('Beijing', 'Shanghai');`. The cardinality of the `city` column is of 10,000, and the query condition involves two values, so the ratio is calculated as `(1*2)/(2*10000)`, which is less than 1/1000. Therefore, this query will use the bitmap index. info The value range for `bitmap_max_filter_ratio` is 1-1000. If `bitmap_max_filter_ratio` is set to `1000`, any query on a column with a bitmap index will be forced to use the bitmap index. ##### Advantages[​](#advantages "Direct link to Advantages") * Bitmap indexes can quickly locate the row numbers of queried column values, suitable for point queries or small-range queries. * Bitmap indexes can optimize multi-dimensional queries involving union and intersection operations (`OR` and `AND` operations). #### Considerations[​](#considerations "Direct link to Considerations") ##### Queries that can be optimized[​](#queries-that-can-be-optimized "Direct link to Queries that can be optimized") Bitmap indexes are suitable for optimizing equality `=` queries, `[NOT] IN` range queries, `>`, `>=`, `<`, `<=` queries, and `IS NULL` queries. They are not suitable for optimizing `!=` and `[NOT] LIKE` queries. ##### Supported columns and data types[​](#supported-columns-and-data-types "Direct link to Supported columns and data types") Bitmap indexes can be created on all columns in primary key and duplicate key tables, and on key columns in aggregate and unique key tables. Bitmap indexes can be created on the columns of the following data types: * Date types: DATE, DATETIME. * Numeric types: TINYINT, SMALLINT, INT, BIGINT, LARGEINT, DECIMAL, BOOLEAN. * String types: CHAR, STRING, VARCHAR. * Other types: HLL. #### Basic operations[​](#basic-operations "Direct link to Basic operations") ##### Create an index[​](#create-an-index "Direct link to Create an index") * Create a bitmap index during table creation. ```sql CREATE TABLE `lineorder_partial` ( `lo_orderkey` int(11) NOT NULL COMMENT "", `lo_orderdate` int(11) NOT NULL COMMENT "", `lo_orderpriority` varchar(16) NOT NULL COMMENT "", `lo_quantity` int(11) NOT NULL COMMENT "", `lo_revenue` int(11) NOT NULL COMMENT "", INDEX lo_orderdate_index (lo_orderdate) USING BITMAP ) ENGINE=OLAP DUPLICATE KEY(`lo_orderkey`) DISTRIBUTED BY HASH(`lo_orderkey`) BUCKETS 1; ``` In this example, a bitmap index named `lo_orderdate_index` is created on the `lo_orderdate` column. Naming requirements for bitmap indexes can be found in [System Limits](https://docs.starrocks.io/docs/sql-reference/System_limit.md). Identical bitmap indexes cannot be created within the same table. Multiple bitmap indexes can be created for multiple columns, separated by commas (,). note For more parameters of table creation, refer to [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md). * `CREATE INDEX` can be used to create a bitmap index after table creation. For detailed parameter descriptions and examples, refer to [CREATE INDEX](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_INDEX.md). ```sql CREATE INDEX lo_quantity_index ON lineorder_partial (lo_quantity) USING BITMAP; ``` ##### Progress of creating an index[​](#progress-of-creating-an-index "Direct link to Progress of creating an index") Creating a bitmap index is an asynchronous process. After executing the index creation statement, you can check the index creation progress using the [SHOW ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_ALTER.md) command. When the `State` field in the returned value shows `FINISHED`, the index is successfully created. ```sql SHOW ALTER TABLE COLUMN; ``` info Each table can only have one ongoing Schema Change task at a time. You cannot create a new bitmap index until the current bitmap index is created. ##### View an index[​](#view-an-index "Direct link to View an index") View all bitmap indexes for a specified table. For detailed parameters and returned results, refer to [SHOW INDEX](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_INDEX.md). ```sql SHOW INDEXES FROM lineorder_partial; ``` note Creating a bitmap index is an asynchronous process. Using the above statement, you can only view indexes that have successfully finished being created. ##### Delete an index[​](#delete-an-index "Direct link to Delete an index") Delete a bitmap index for a specified table. For detailed parameters and examples, refer to [DROP INDEX](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_INDEX.md). ```sql DROP INDEX lo_orderdate_index ON lineorder_partial; ``` ##### Verify whether the bitmap index accelerates queries[​](#verify-whether-the-bitmap-index-accelerates-queries "Direct link to Verify whether the bitmap index accelerates queries") Check the `BitmapIndexFilterRows` field in the query Profile. For information on viewing the Profile, refer to [Query analysis](https://docs.starrocks.io/docs/best_practices/query_tuning/query_planning.md). #### Performance test on bitmap index[​](#performance-test-on-bitmap-index "Direct link to Performance test on bitmap index") ##### Objective of testing[​](#objective-of-testing "Direct link to Objective of testing") To analyze the filtering effect and other impacts, such as disk usage, of bitmap indexes on queries with different cardinalities: * [Query on a single column of low cardinality](#query-1-query-on-single-column-of-low-cardinality) * [Query on the combination of multiple columns of low cardinality](#query-2-query-on-the-combination-of-multiple-columns-of-low-cardinality) * [Query on a single column of high cardinality](#query-3-query-based-on-single-high-cardinality-column) This section also compares the performance between always using bitmap index and using bitmap indexes adaptively to verify the effectiveness of [StarRocks' adaptive selection of bitmap indexes](#adaptive-selection-of-bitmap-indexes). ##### Create table and bitmap index[​](#create-table-and-bitmap-index "Direct link to Create table and bitmap index") warning To avoid caching Page data affecting query performance, ensure that the BE configuration item `disable_storage` is set to `true`. This section takes the table `lineorder` (SSB 20G) as an example. * Original table (without bitmap indexes) as a reference ```sql CREATE TABLE `lineorder_without_index` ( `lo_orderkey` int(11) NOT NULL COMMENT "", `lo_linenumber` int(11) NOT NULL COMMENT "", `lo_custkey` int(11) NOT NULL COMMENT "", `lo_partkey` int(11) NOT NULL COMMENT "", `lo_suppkey` int(11) NOT NULL COMMENT "", `lo_orderdate` int(11) NOT NULL COMMENT "", `lo_orderpriority` varchar(16) NOT NULL COMMENT "", `lo_shippriority` int(11) NOT NULL COMMENT "", `lo_quantity` int(11) NOT NULL COMMENT "", `lo_extendedprice` int(11) NOT NULL COMMENT "", `lo_ordtotalprice` int(11) NOT NULL COMMENT "", `lo_discount` int(11) NOT NULL COMMENT "", `lo_revenue` int(11) NOT NULL COMMENT "", `lo_supplycost` int(11) NOT NULL COMMENT "", `lo_tax` int(11) NOT NULL COMMENT "", `lo_commitdate` int(11) NOT NULL COMMENT "", `lo_shipmode` varchar(11) NOT NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`lo_orderkey`) DISTRIBUTED BY HASH(`lo_orderkey`) BUCKETS 1; ``` * Table with bitmap indexes: Bitmap indexes created based on `lo_shipmode`, `lo_quantity`, `lo_discount`, `lo_orderdate`, `lo_tax`, and `lo_partkey`. ```sql CREATE TABLE `lineorder_with_index` ( `lo_orderkey` int(11) NOT NULL COMMENT "", `lo_linenumber` int(11) NOT NULL COMMENT "", `lo_custkey` int(11) NOT NULL COMMENT "", `lo_partkey` int(11) NOT NULL COMMENT "", `lo_suppkey` int(11) NOT NULL COMMENT "", `lo_orderdate` int(11) NOT NULL COMMENT "", `lo_orderpriority` varchar(16) NOT NULL COMMENT "", `lo_shippriority` int(11) NOT NULL COMMENT "", `lo_quantity` int(11) NOT NULL COMMENT "", `lo_extendedprice` int(11) NOT NULL COMMENT "", `lo_ordtotalprice` int(11) NOT NULL COMMENT "", `lo_discount` int(11) NOT NULL COMMENT "", `lo_revenue` int(11) NOT NULL COMMENT "", `lo_supplycost` int(11) NOT NULL COMMENT "", `lo_tax` int(11) NOT NULL COMMENT "", `lo_commitdate` int(11) NOT NULL COMMENT "", `lo_shipmode` varchar(11) NOT NULL COMMENT "", INDEX i_shipmode (`lo_shipmode`) USING BITMAP, INDEX i_quantity (`lo_quantity`) USING BITMAP, INDEX i_discount (`lo_discount`) USING BITMAP, INDEX i_orderdate (`lo_orderdate`) USING BITMAP, INDEX i_tax (`lo_tax`) USING BITMAP, INDEX i_partkey (`lo_partkey`) USING BITMAP ) ENGINE=OLAP DUPLICATE KEY(`lo_orderkey`) DISTRIBUTED BY HASH(`lo_orderkey`) BUCKETS 1; ``` ##### Disk space usage of bitmap indexes[​](#disk-space-usage-of-bitmap-indexes "Direct link to Disk space usage of bitmap indexes") * `lo_shipmode`: String type, cardinality 7, occupies 130M * `lo_quantity`: Integer type, cardinality 50, occupies 291M * `lo_discount`: Integer type, cardinality 11, occupies 198M * `lo_orderdate`: Integer type, cardinality 2406, occupies 191M * `lo_tax`: Integer type, cardinality 9, occupies 160M * `lo_partkey`: Integer type, cardinality 600,000, occupies 601M ##### Query 1: Query on single column of low cardinality[​](#query-1-query-on-single-column-of-low-cardinality "Direct link to Query 1: Query on single column of low cardinality") ###### Query table without bitmap index[​](#query-table-without-bitmap-index "Direct link to Query table without bitmap index") **Query**: ```sql SELECT count(1) FROM lineorder_without_index WHERE lo_shipmode="MAIL"; ``` **Query performance analysis**: Since the table queried does not have bitmap index, all pages containing the `lo_shipmode` column data need to be read and then predicate filtering is applied. Total Time: Approximately 0.91 seconds, **with data loading taking 0.47 seconds**, decoding dictionary for low cardinality optimization taking 0.31 seconds, and predicate filtering taking 0.23 seconds. ```bash PullRowNum: 20.566M (20566493) // Number of rows in the result set. CompressedBytesRead: 55.283 MB // Total amount of data read. RawRowsRead: 143.999M (143999468) // Number of rows read. Since there is no bitmap index, all data in this column is read. ReadPagesNum: 8.795K (8795) // Number of pages read. Since there is no bitmap index, all pages containing data for this column are read. IOTaskExecTime: 914ms // Total time for scanning data. BlockFetch: 469ms // Time for loading data. DictDecode: 311.612ms // Time for decoding dictionary for low cardinality optimization. PredFilter: 23.182ms // Time for predicate filtering. PredFilterRows: 123.433M (123432975) // Number of rows filtered out. ``` ###### Query table with bitmap index[​](#query-table-with-bitmap-index "Direct link to Query table with bitmap index") ###### Use bitmap index compulsorily[​](#use-bitmap-index-compulsorily "Direct link to Use bitmap index compulsorily") info To use the bitmap index compulsorily, according to Starrocks' configuration, `bitmap_max_filter_ratio=1000` needs to be set in the **be.conf** file of each BE node, and then the BE nodes need to be restarted. **Query**: ```sql SELECT count(1) FROM lineorder_with_index WHERE lo_shipmode="MAIL"; ``` **Query performance analysis**: Since the column queried is of low cardinality, bitmap index does not filter the data efficiently. Even though bitmap index can quickly locate the row numbers of actual data, a large number of rows need to be read, scattered across multiple pages. As a result, it cannot effectively filter out the pages that need to be read. Moreover, additional overhead for loading the bitmap index and using the bitmap index to filter data is incurred, resulting in a longer total time. Total time: 2.077 seconds, **with 0.93 seconds spent loading data and bitmap index**, 0.33 seconds on decoding dictionary for low cardinality optimization, 0.42 seconds on filtering data with bitmap index, and 0.17 seconds on filtering data with ZoneMap Index. ```bash PullRowNum: 20.566M (20566493) // Number of rows in the result set. CompressedBytesRead: 72.472 MB // Total amount of data read. The size of the bitmap index for this column is 130MB, with 7 unique values. The size of a bitmap index for each value is 18MB. The data size of Pages is 55MB = 73MB. RawRowsRead: 20.566M (20566493) // Number of rows read. 20 million rows are actually read. ReadPagesNum: 8.802K (8802) // Number of Pages read. All Pages are read because the 20 million rows filtered by bitmap index are randomly distributed across different Pages, and Page is the smallest data reading unit. Therefore, bitmap index does not filter out Pages. IOTaskExecTime: 2s77ms // Total time for scanning data, longer than without bitmap index. BlockFetch: 931.144ms // Time for loading data and bitmap index. Compared to the previous query, an additional 400ms was spent to load bitmap index (18MB). DictDecode: 329.696ms // Time for decoding dictionary for low cardinality optimization is similar since the output row count is the same. BitmapIndexFilter: 419.308ms // Time for filtering data with the bitmap index. BitmapIndexFilterRows: 123.433M (123432975) // Number of rows filtered by the bitmap index. ZoneMapIndexFilter: 171.580ms // Time for filtering data with ZoneMap Index. ``` ###### Determine whether to use bitmap index based on StarRocks default configuration[​](#determine-whether-to-use-bitmap-index-based-on-starrocks-default-configuration "Direct link to Determine whether to use bitmap index based on StarRocks default configuration") **Query**: ```sql SELECT count(1) FROM lineorder_with_index WHERE lo_shipmode="MAIL"; ``` **Query performance analysis**: According to StarRocks default configuration, bitmap index is used only if the number of distinct values in the filter condition column / column cardinality < `bitmap_max_filter_ratio/1000` (default is 1/1000). In this case, the value is greater than 1/1000, so bitmap index is not used, resulting in performance similar to querying a table without bitmap index. ```bash PullRowNum: 20.566M (20566493) // Number of rows in the result set. CompressedBytesRead: 55.283 MB // Total amount of data read. RawRowsRead: 143.999M (143999468) // Number of rows read. Since the bitmap index is not used, all data in this column is read. ReadPagesNum: 8.800K (8800) // Number of Pages read. Since the bitmap index is not used, all pages containing data for this column are read. IOTaskExecTime: 914.279ms // Total time for scanning data. BlockFetch: 475.890ms // Time for loading data. DictDecode: 312.019ms // Time for decoding dictionary for low cardinality optimization. PredFilterRows: 123.433M (123432975) // Number of rows filtered out by predicate. ``` ##### Query 2: Query on the combination of multiple columns of low cardinality[​](#query-2-query-on-the-combination-of-multiple-columns-of-low-cardinality "Direct link to Query 2: Query on the combination of multiple columns of low cardinality") ###### Query table without bitmap index[​](#query-table-without-bitmap-index-1 "Direct link to Query table without bitmap index") **Query**: ```sql SELECT count(1) FROM lineorder_without_index WHERE lo_shipmode = "MAIL" AND lo_quantity = 10 AND lo_discount = 9 AND lo_tax = 8; ``` **Query performance analysis**: Since the table queried does not have any bitmap index, all Pages containing data of `lo_shipmode`, `lo_quantity`, `lo_discount`, and `lo_tax` columns are read and then predicate filtering is applied. Total time taken: 1.76 seconds, **with 1.6 seconds spent loading data (4 columns)**, and 0.1 seconds on predicate filtering. ```bash PullRowNum: 4.092K (4092) // Number of rows in the result set. CompressedBytesRead: 305.346 MB // Total amount of data read. Data of 4 columns is read, totaling 305MB. RawRowsRead: 143.999M (143999468) // Number of rows read. Since there is no bitmap index, all data of these 4 columns are read. ReadPagesNum: 35.168K (35168) // Number of Pages read. Since there is no bitmap index, all Pages containing data of these 4 columns are read. IOTaskExecTime: 1s761ms // Total time for scanning data. BlockFetch: 1s639ms // Time for loading data. PredFilter: 96.533ms // Time for predicate filtering with 4 filter conditions. PredFilterRows: 143.995M (143995376) // Number of rows filtered out by predicates. ``` ###### Query Table with bitmap index[​](#query-table-with-bitmap-index-1 "Direct link to Query Table with bitmap index") ###### Use bitmap index compulsorily[​](#use-bitmap-index-compulsorily-1 "Direct link to Use bitmap index compulsorily") info To use bitmap index compulsorily, according to Starrocks' configuration, the `bitmap_max_filter_ratio=1000` must be set in the **be.conf** file of each BE node, then the BE nodes must be restarted. **Query**: ```sql SELECT count(1) FROM lineorder_with_index WHERE lo_shipmode="MAIL" AND lo_quantity=10 AND lo_discount=9 AND lo_tax=8; ``` **Query performance analysis**: Since this is a combination query based on multiple low cardinality columns, bitmap index is effective, filtering out a portion of the Pages and significantly reducing data read time. Total time taken: 0.68 seconds, **with 0.54 seconds spent loading data and bitmap index**, and 0.14 seconds filtering data with bitmap index. ```bash PullRowNum: 4.092K (4092) // Number of rows in the result set. CompressedBytesRead: 156.340 MB // Total amount of data read. The bitmap index effectively filters out 2/3 of the data. Out of this 156MB, 60MB is index data, and 90MB is actual data. ReadPagesNum: 11.325K (11325) // Number of Pages read. The bitmap index effectively filters out 2/3 of the Pages. IOTaskExecTime: 683.471ms // Total time for scanning data, significantly less than the total time spent without bitmap index. BlockFetch: 537.421ms // Time for loading data and the bitmap index. Although additional time is spent loading the bitmap index, it significantly reduces the time to load data. BitmapIndexFilter: 139.024ms // Time for filtering data with bitmap index. BitmapIndexFilterRows: 143.995M (143995376) // Number of rows filtered out by bitmap index. ``` ###### Determine whether to use bitmap index based on StarRocks default configuration[​](#determine-whether-to-use-bitmap-index-based-on-starrocks-default-configuration-1 "Direct link to Determine whether to use bitmap index based on StarRocks default configuration") **Query**: ```sql SELECT count(1) FROM lineorder_with_index WHERE lo_shipmode="MAIL" AND lo_quantity=10 AND lo_discount=9 AND lo_tax=8; ``` **Query performance analysis**: According to StarRocks default configuration, the bitmap index is used if the number of distinct values in the filter condition columns / column cardinality < `bitmap_max_filter_ratio/1000` (default is 1/1000). In this case, the value is less than 1/1000, so the bitmap index is used, resulting in performance similar to use the bitmap index compulsorily. Total time taken: 0.67 seconds, **with 0.54 seconds spent loading data and bitmap index**, and 0.13 seconds filtering data with bitmap index. ```bash PullRowNum: 4.092K (4092) // Number of rows in the result set. CompressedBytesRead: 154.430 MB // Total amount of data read. ReadPagesNum: 11.209K (11209) // Number of Pages read. IOTaskExecTime: 672.029ms // Total time for scanning data, significantly less than without bitmap index. BlockFetch: 535.823ms // Time for loading data and bitmap index. Although loading bitmap index takes a little more time, it greatly reduces data loading time. BitmapIndexFilter: 128.403ms // Time for filtering data with bitmap index. BitmapIndexFilterRows: 143.995M (143995376) // Number of rows filtered out by bitmap index. ``` ##### Query 3: Query on single Column of high cardinality[​](#query-3-query-on-single-column-of-high-cardinality "Direct link to Query 3: Query on single Column of high cardinality") ###### Query table without bitmap index[​](#query-table-without-bitmap-index-2 "Direct link to Query table without bitmap index") **Query**: ```sql select count(1) from lineorder_without_index where lo_partkey=10000; ``` **Query performance analysis**: Since the table queried does not have a bitmap index, pages containing the `lo_partkey` column data are read and then predicate filtering is applied. Total time: approximately 0.43 seconds, **including 0.39 seconds for data loading** and 0.02 seconds for predicate filtering. ```bash PullRowNum: 255 // Number of rows in the result set. CompressedBytesRead: 344.532 MB // Total amount of data read. RawRowsRead: 143.999M (143,999,468) // Number of rows read. Since there is no bitmap index, all data in this column is read. ReadPagesNum: 8.791K (8,791) // Number of pages read. Since there is no bitmap index, all data of this column are read. IOTaskExecTime: 428.258ms // Total time for scanning data. BlockFetch: 392.434ms // Time for loading data. PredFilter: 20.807ms // Time for predicate filtering. PredFilterRows: 143.999M (143,999,213) // Number of rows filtered out by predicates. ``` ###### Query a Table with bitmap index[​](#query-a-table-with-bitmap-index "Direct link to Query a Table with bitmap index") ###### Use bitmap index compulsorily[​](#use-bitmap-index-compulsorily-2 "Direct link to Use bitmap index compulsorily") info To use the bitmap index compulsorily, according to StarRocks' configuration, you must set `bitmap_max_filter_ratio=1000` in the **be.conf** file on each BE node, and then restart the BE nodes. **Query**: ```sql SELECT count(1) FROM lineorder_with_index WHERE lo_partkey=10000; ``` **Query performance analysis**: Since the queried column is of high cardinality, the bitmap index is effective, allowing for filtering out a portion of the pages and significantly reducing the time for reading data. Total time: 0.015 seconds, **including 0.009 seconds for loading data and bitmap index**, and 0.003 seconds for bitmap index filtering. ```bash PullRowNum: 255 // Number of rows in the result set. CompressedBytesRead: 13.600 MB // Total amount of data read. Bitmap Index filtering is effective, filtering out a large amount of data. RawRowsRead: 255 // Number of rows read. ReadPagesNum: 225 // Number of pages read. The bitmap index can effectively filter out data, filtering out a large number of pages. IOTaskExecTime: 15.354ms // Total time for scanning data, significantly less than the total time spent without Bitmap Index. BlockFetch: 9.530ms // Time for loading data and Bitmap Index. Although additional time is spent loading the Bitmap Index, it significantly reduces the time to load data. BitmapIndexFilter: 3.450ms // Time for filtering data with Bitmap Index. BitmapIndexFilterRows: 143.999M (143,999,213) // Number of rows filtered out by Bitmap Index. ``` ###### Determine whether to use bitmap index based on StarRocks default configuration[​](#determine-whether-to-use-bitmap-index-based-on-starrocks-default-configuration-2 "Direct link to Determine whether to use bitmap index based on StarRocks default configuration") **Query**: ```sql SELECT count(1) FROM lineorder_with_index WHERE lo_partkey=10000; ``` **Query performance analysis**: According to StarRocks' default configuration, Bitmap Index is used when the number of distinct values/column cardinality < `bitmap_max_filter_ratio/1000` (default 1/1000). Since this condition is met, the query uses the Bitmap Index, and the performance is similar to that when the Bitmap Index is compulsorily used.. Total time: 0.014 seconds, including **0.008 seconds for loading data and bitmap index**, and 0.003 seconds for Bitmap Index filtering. ```bash PullRowNum: 255 // Number of rows in the result set. CompressedBytesRead: 13.600 MB // Total amount of data read. The bitmap index can effectively filter out data, filtering out a large number of data. RawRowsRead: 255 // Number of rows read. ReadPagesNum: 225 // Number of pages read. The bitmap index can effectively filter out data, filtering out a large number of pages. IOTaskExecTime: 14.387ms // Total time for scanning data, significantly less than the total time spent without Bitmap Index. BitmapIndexFilter: 2.979ms // Time for filtering data with Bitmap Index. BitmapIndexFilterRows: 143.999M (143,999,213) // Number of rows filtered out by Bitmap Index. BlockFetch: 8.988ms // Time for loading data and Bitmap Index. Although additional time is spent loading the Bitmap Index, it significantly reduces the time to load data. ``` --- ### Bloom filter indexes This topic describes how to create and modify bloom filter indexes, along with how they works. A bloom filter index is a space-efficiency data structure that is used to detect the possible presence of filtered data in data files of a table. If the bloom filter index detects that the data to be filtered are not in a certain data file, StarRocks skips scanning the data file. Bloom filter indexes can reduce response time when the column (such as ID) has a relatively high cardinality. If a query hits a sort key column, StarRocks efficiently returns the query result by using the [prefix index](https://docs.starrocks.io/docs/table_design/indexes/Prefix_index_sort_key.md). However, the prefix index entry for a data block cannot exceed 36 bytes in length. If you want to improve the query performance on a column, which is not used as a sort key and has a relatively high cardinality, you can create a bloom filter index for the column. #### How it works[​](#how-it-works "Direct link to How it works") For example, you create a bloom filter index on a `column1` of a given table `table1` and run a query such as `Select xxx from table1 where column1 = something;`. Then the following situations happen when StarRocks scans the data files of `table1`. * If the bloom filter index detects that a data file does not contain the data to be filtered, StarRocks skips the data file to improve query performance. * If the bloom filter index detects that a data file may contain the data to be filtered, StarRocks reads the data file to check whether the data exists. Note that the bloom filter can tell you for sure if a value is not present, but it cannot say for sure that a value is present, only that it may be present. Using a bloom filter index to determine whether a value is present may give false positives, which means that a bloom filter index detects that a data file contains the data to be filtered, but the data file does not actually contain the data. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * You can create bloom filter indexes for all columns of a Duplicate Key or Primary Key table. For an Aggregate table or Unique Key table, you can only create bloom filter indexes for key columns. * Bloom filter indexes can be created for columns of the following data types: * Numeric types: SMALLINT, INT, BIGINT, and LARGEINT. * String types: CHAR, STRING, and VARCHAR. * Date types: DATE and DATETIME. * Bloom filter indexes can only improve the performance of queries that contain the `in` and `=` operators, such as `Select xxx from table where x in {}` and `Select xxx from table where column = xxx`. * You can check whether a query uses bloom filter indexes by viewing the `BloomFilterFilterRows` field of the query's profile. #### Create bloom filter indexes[​](#create-bloom-filter-indexes "Direct link to Create bloom filter indexes") You can create a bloom filter index for a column when you create a table by specifying the `bloom_filter_columns` parameter in `PROPERTIES`. For example, create bloom filter indexes for the `k1` and `k2` columns in `table1`. ```sql CREATE TABLE table1 ( k1 BIGINT, k2 LARGEINT, v1 VARCHAR(2048) REPLACE, v2 SMALLINT DEFAULT "10" ) ENGINE = olap PRIMARY KEY(k1, k2) DISTRIBUTED BY HASH (k1, k2) PROPERTIES("bloom_filter_columns" = "k1,k2"); ``` You can create bloom filter indexes for multiple columns at a time by specifying these column names. Note that you need to separate these column names with commas (`,`). For other parameter descriptions of the CREATE TABLE statement, see [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md). #### Display bloom filter indexes[​](#display-bloom-filter-indexes "Direct link to Display bloom filter indexes") For example, the following statement displays bloom filter indexes of `table1`. For the output description, see [SHOW CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_CREATE_TABLE.md). ```sql SHOW CREATE TABLE table1; ``` #### Modify bloom filter indexes[​](#modify-bloom-filter-indexes "Direct link to Modify bloom filter indexes") You can add, reduce, and delete bloom filter indexes by using the [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md) statement. * The following statement adds a bloom filter index on the `v1` column. ```sql ALTER TABLE table1 SET ("bloom_filter_columns" = "k1,k2,v1"); ``` * The following statement reduces the bloom filter index on the `k2` column. ```sql ALTER TABLE table1 SET ("bloom_filter_columns" = "k1"); ``` * The following statement deletes all bloom filter indexes of `table1`. ```sql ALTER TABLE table1 SET ("bloom_filter_columns" = ""); ``` > Note: Altering an index is an asynchronous operation. You can view the progress of this operation by executing [SHOW ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_ALTER.md). You can run only one alter index task on a table each time. --- ### Full-text inverted index Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) Since version 3.3.0, StarRocks supports full-text inverted indexes, which can break the text into smaller words, and create an index entry for each word that can show the mapping relationship between the word and its corresponding row number in the data file. For full-text searches, StarRocks queries the inverted index based on the search keywords, quickly locating the data rows that match the keywords. Primary Key tables support full-text inverted indexes from v4.0 onwards. Since v4.1, StarRocks supports a **built-in** inverted index implementation in addition to the default CLucene-based implementation. The built-in implementation supports both **shared-nothing** and **shared-data** clusters. For details, see [built-in inverted index](#built-in-inverted-index). #### Overview[​](#overview "Direct link to Overview") StarRocks stores its underlying data in the data files organized by columns. Each data file contains the full-text inverted index based on the indexed columns. The values in the indexed columns are tokenized into individual words. Each word after tokenization is treated as an index entry, mapping to the row number where the word appears. Currently supported tokenization methods for English tokenization, Chinese tokenization, multilingual tokenization, and no tokenization. For example, if a data row contains "hello world" and its row number is 123, the full-text inverted index builds index entries based on this tokenization result and row number: hello->123, world->123. During full-text searches, StarRocks can locate index entries containing the search keywords using full-text inverted indexes, and then quickly find the row numbers where the keywords appear, significantly reducing the number of data rows that need to be scanned. #### Basic operation[​](#basic-operation "Direct link to Basic operation") ##### Create full-text inverted index[​](#create-full-text-inverted-index "Direct link to Create full-text inverted index") Before creating a full-text inverted index, you need to enable FE configuration item `enable_experimental_gin`. ```sql ADMIN SET FRONTEND CONFIG ("enable_experimental_gin" = "true"); ``` note When creating a full-text inverted index for a table, you must disable the `replicated_storage` feature for the table. * For v4.0 and later, this feature is automatically disabled when you create the index. * for versions earlier than v4.0, you must manually set the table property `replicated_storage` to `false`. ###### Create full-text Inverted Index at table creation[​](#create-full-text-inverted-index-at-table-creation "Direct link to Create full-text Inverted Index at table creation") Creating a full-text inverted index on column `v` with English tokenization. ```sql CREATE TABLE `t` ( `k` BIGINT NOT NULL COMMENT "", `v` STRING COMMENT "", INDEX idx (v) USING GIN("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(`k`) DISTRIBUTED BY HASH(`k`) BUCKETS 1 PROPERTIES ( "replicated_storage" = "false" ); ``` * The `parser` parameter specifies the tokenization method. Supported values and descriptions are as follows: * `none` (default): no tokenization. The entire row of data in the indexed column is treated as a single index item when the full-text inverted index is constructed. * `english`: English tokenization. This tokenization method typically tokenizing at any non-alphabetic character. Also, uppercase English letters are converted to lowercase. Therefore, keywords in the query conditions need to be lowercase English rather than uppercase English to leverage the full-text inverted index to locate data rows. * `chinese`: Chinese tokenization. This tokenization method uses the [CJK Analyzer](https://lucene.apache.org/core/6_6_1/analyzers-common/org/apache/lucene/analysis/cjk/package-summary.html) in CLucene for tokenization. * `standard`: Multilingual tokenization. This tokenization method provides grammar based tokenization (based on the [Unicode Text Segmentation algorithm](https://unicode.org/reports/tr29/)) and works well for most languages and cases of mixed languages, such as Chinese and English. For example, this tokenization method can distinguishes between Chinese and English when these two languages coexist. After tokenizing English, it converts uppercase English letters to lowercase. Therefore, keywords in the query conditions need to be lowercase English rather than uppercase English to leverage the full-text inverted index to locate data rows. * The data type of the indexed column must be CHAR, VARCHAR, or STRING. ###### Add full-text inverted index after table creation[​](#add-full-text-inverted-index-after-table-creation "Direct link to Add full-text inverted index after table creation") After table creation, you can add a full-text inverted index using `ALTER TABLE ADD INDEX` or `CREATE INDEX`. ```sql ALTER TABLE t ADD INDEX idx (v) USING GIN('parser' = 'english'); CREATE INDEX idx ON t (v) USING GIN('parser' = 'english'); ``` ##### Manage full-text inverted index[​](#manage-full-text-inverted-index "Direct link to Manage full-text inverted index") ###### View full-text inverted index[​](#view-full-text-inverted-index "Direct link to View full-text inverted index") Execute `SHOW CREATE TABLE` to view full-text inverted indexes. ```sql MySQL [example_db]> SHOW CREATE TABLE t\G ``` ###### Delete full-text inverted index[​](#delete-full-text-inverted-index "Direct link to Delete full-text inverted index") Execute `ALTER TABLE ADD INDEX` or `DROP INDEX` to delete full-text inverted indexes. ```sql DROP INDEX idx on t; ALTER TABLE t DROP index idx; ``` ##### Accelerate queries by full-text inverted index[​](#accelerate-queries-by-full-text-inverted-index "Direct link to Accelerate queries by full-text inverted index") After creating a full-text inverted index, you need to ensure that the system variable `enable_gin_filter` is enabled, so the inverted index can accelerate queries. Also, you need to consider whether the index column values are tokenized to determine which queries can be accelerated. ###### Supported queries when indexed column is tokenized[​](#supported-queries-when-indexed-column-is-tokenized "Direct link to Supported queries when indexed column is tokenized") When a full-text inverted index column is enabled with tokenization (`parser` = `standard` | `english` | `chinese`), only the `MATCH`, `MATCH_ANY`, or `MATCH_ALL` predicates are supported for filtering. The supported formats are: * ` (NOT) MATCH '%keyword%'` * ` (NOT) MATCH_ANY 'keyword1, keyword2'` * ` (NOT) MATCH_ALL 'keyword1, keyword2'` Here, keyword must be a string literal; expressions are not supported. For a nullable indexed column, rows whose value is `NULL` are not returned by `NOT MATCH`, `NOT MATCH_ANY`, or `NOT MATCH_ALL`. 1. Create a table and insert a few rows of test data. ```sql CREATE TABLE `t` ( `id1` bigint(20) NOT NULL COMMENT "", `value` varchar(255) NOT NULL COMMENT "", INDEX gin_english (`value`) USING GIN ("parser" = "english") COMMENT 'english index' ) DUPLICATE KEY(`id1`) DISTRIBUTED BY HASH(`id1`) PROPERTIES ( "replicated_storage" = "false" ); INSERT INTO t VALUES (1, "starrocks is a database 1"), (2, "starrocks is a data warehouse"); ``` 2. Use the `MATCH` predicate for querying. * Query data rows whose `value` column contains the keyword `starrocks`. ```sql MySQL [example_db]> SELECT * FROM t WHERE t.value MATCH "starrocks"; ``` * Retrieve data rows whose `value` column contains the keyword starting with `data`. ```sql MySQL [example_db]> SELECT * FROM t WHERE t.value MATCH "data%"; ``` 3. Use the `MATCH_ANY` predicate for querying. * Query data rows whose `value` column contains the keyword `database` or `data`. ```sql MySQL [example_db]> SELECT * FROM t WHERE t.value MATCH_ANY "database data"; ``` 4. Use the `MATCH_ALL` predicate for querying. * Query data rows whose `value` column contains both the keyword `database` and `data`. ```sql MySQL [example_db]> SELECT * FROM t WHERE t.value MATCH_ALL "database data"; ``` **Notes:** * During queries, keywords can be matched fuzzily using `%`, in the format of `%keyword%`. However, the keyword must contain a part of a word. For example, if the keyword is `starrocks `, it cannot match the word `starrocks` because it contains spaces. ```sql MySQL [example_db]> SELECT * FROM t WHERE t.value MATCH "star%"; +------+-------------------------------+ | id1 | value | +------+-------------------------------+ | 1 | starrocks is a database1 | | 2 | starrocks is a data warehouse | +------+-------------------------------+ 2 rows in set (0.02 sec) MySQL [example_db]> SELECT * FROM t WHERE t.value MATCH "starrocks "; Empty set (0.02 sec) ``` * If English or multilingual tokenization is used to construct the full-text inverted index, uppercase English words are converted to lowercase when the full-text inverted index is actually stored. Therefore, during queries, keywords need to be lowercase instead of uppercase to utilize the full-text inverted index to locate data rows. ```sql MySQL [example_db]> INSERT INTO t VALUES (3, "StarRocks is the BEST"); MySQL [example_db]> SELECT * FROM t; +------+-------------------------------+ | id1 | value | +------+-------------------------------+ | 1 | starrocks is a database | | 2 | starrocks is a data warehouse | | 3 | StarRocks is the BEST | +------+-------------------------------+ 3 rows in set (0.02 sec) MySQL [example_db]> SELECT * FROM t WHERE t.value MATCH "BEST"; -- Keyword is uppercase English Empty set (0.02 sec) -- Returns an empty result set MySQL [example_db]> SELECT * FROM t WHERE t.value MATCH "best"; -- Keyword is lowercase English +------+-----------------------+ | id1 | value | +------+-----------------------+ | 3 | StarRocks is the BEST | -- Can locate data rows that meet the condition +------+-----------------------+ 1 row in set (0.01 sec) ``` * The `MATCH`, `MATCH_ANY`, or `MATCH_ALL` predicate in the query conditions must be used as a pushdown predicate, so it must be in the WHERE clause and be performed against the indexed column. Take the following table and test data as an example: ```sql CREATE TABLE `t_match` ( `id1` bigint(20) NOT NULL COMMENT "", `value` varchar(255) NOT NULL COMMENT "", `value_test` varchar(255) NOT NULL COMMENT "", INDEX gin_english (`value`) USING GIN("parser" = "english") COMMENT 'english index' ) ENGINE=OLAP DUPLICATE KEY(`id1`) DISTRIBUTED BY HASH (`id1`) BUCKETS 1 PROPERTIES ( "replicated_storage" = "false" ); INSERT INTO t_match VALUES (1, "test", "test"); ``` The following query statements do not meet the requirement: * Because the `MATCH`, `MATCH_ANY`, or `MATCH_ALL` predicate in the query statement is not in the WHERE clause, it can not be pushed down, resulting in a query error. ```sql MySQL [test]> SELECT value MATCH "test" FROM t_match; ERROR 1064 (HY000): Match can only be used as a pushdown predicate on a column with GIN in a single query. ``` * Because the column `value_test` against which the `MATCH`, `MATCH_ANY`, or `MATCH_ALL` predicate in the query statement is performed is not an indexed column, the query fails. ```sql MySQL [test]> SELECT * FROM t_match WHERE value_test match "test"; ERROR 1064 (HY000): Match can only be used as a pushdown predicate on a column with GIN in a single query. ``` ###### Supported queries when indexed column is not tokenized[​](#supported-queries-when-indexed-column-is-not-tokenized "Direct link to Supported queries when indexed column is not tokenized") If the full-text inverted index does not tokenize the indexed column, that is, `'parser' = 'none'`, all pushdown predicates in the query conditions listed below can be used for data filtering using the full-text inverted index: * Expression predicates: (NOT) LIKE, (NOT) MATCH, (NOT) MATCH\_ANY, (NOT) MATCH\_ALL note * In this case, `MATCH` is semantically equivalent to `LIKE`. * `MATCH` and `LIKE` only support the format `(NOT) MATCH|LIKE '%keyword%'`. The `keyword` must be a string literal and does not support expressions. Note that if `LIKE` does not meet this format, even if the query can be executed normally, it will degrade to a query that does not use the full-text inverted index to filter data. * Regular predicates: `==`, `!=`, `<=`, `>=`, `NOT IN`, `IN`, `IS NOT NULL`, `NOT NULL` #### Verify whether a query is accelerated by the full-text inverted index[​](#verify-whether-a-query-is-accelerated-by-the-full-text-inverted-index "Direct link to Verify whether a query is accelerated by the full-text inverted index") After executing the query, you can view the detailed metrics `GinFilterRows` and `GinFilter` in the scan node of the Query Profile to see the number of rows filtered and the filtering time using the full-text inverted index. #### Built-in inverted index[​](#built-in-inverted-index "Direct link to Built-in inverted index") Since v4.1, StarRocks provides a **built-in** inverted index implementation in addition to the default CLucene-based implementation. The built-in implementation is StarRocks' native inverted index built on top of bitmap index, and supports both **shared-nothing** and **shared-data** clusters. note The CLucene-based inverted index implementation is not supported in shared-data clusters. For shared-data clusters, you must use the built-in implementation. ##### Implementation comparison[​](#implementation-comparison "Direct link to Implementation comparison") | Implementation | Supported since | Shared-nothing | Shared-data | Description | | --------------------- | --------------- | -------------- | ----------- | -------------------------------------------------------------------------------------------------------------- | | **CLucene** (default) | v3.3.0 | Yes | No | Based on the CLucene full-text search library. This is the default implementation for shared-nothing clusters. | | **built-in** | v4.1.0 | Yes | Yes | StarRocks' native inverted index implementation. Supports both shared-nothing and shared-data clusters. | You can explicitly specify the implementation type using the `imp_lib` parameter when creating an index. If not specified, the system automatically selects the appropriate implementation based on the cluster mode: * In shared-nothing clusters, the default is CLucene. * In shared-data clusters, the default is built-in (CLucene is not supported). ##### Create built-in inverted index[​](#create-built-in-inverted-index "Direct link to Create built-in inverted index") ###### Create at table creation[​](#create-at-table-creation "Direct link to Create at table creation") ```sql -- Create table with built-in inverted index CREATE TABLE `t_builtin` ( `id1` bigint(20) NOT NULL COMMENT "", `value` varchar(255) NOT NULL COMMENT "", INDEX gin_english (`value`) USING GIN ("parser" = "english", "imp_lib" = "builtin") COMMENT 'builtin english index' ) DUPLICATE KEY(`id1`) DISTRIBUTED BY HASH(`id1`) PROPERTIES ( "replicated_storage" = "false" ); ``` ###### Add after table creation[​](#add-after-table-creation "Direct link to Add after table creation") ```sql ALTER TABLE t ADD INDEX idx_builtin (v) USING GIN('parser' = 'english', 'imp_lib' = 'builtin'); -- Or CREATE INDEX idx_builtin ON t (v) USING GIN('parser' = 'english', 'imp_lib' = 'builtin'); ``` ##### Built-in inverted index in shared-data clusters[​](#built-in-inverted-index-in-shared-data-clusters "Direct link to Built-in inverted index in shared-data clusters") In shared-data clusters, the built-in implementation is automatically selected even if `imp_lib` is not specified. You can create the index using the same syntax as for a regular inverted index: ```sql -- In a shared-data cluster, this automatically uses the built-in implementation CREATE TABLE `t_shared_data` ( `id1` bigint(20) NOT NULL COMMENT "", `value` varchar(255) NOT NULL COMMENT "", INDEX gin_english (`value`) USING GIN ("parser" = "english") COMMENT 'english index' ) DUPLICATE KEY(`id1`) DISTRIBUTED BY HASH(`id1`); ``` You can also explicitly specify `"imp_lib" = "builtin"` for clarity: ```sql CREATE TABLE `t_shared_data_explicit` ( `id1` bigint(20) NOT NULL COMMENT "", `value` varchar(255) NOT NULL COMMENT "", INDEX gin_english (`value`) USING GIN ("parser" = "english", "imp_lib" = "builtin") COMMENT 'builtin english index' ) DUPLICATE KEY(`id1`) DISTRIBUTED BY HASH(`id1`); ``` ##### dict\_gram\_num[​](#dict_gram_num "Direct link to dict_gram_num") The `dict_gram_num` parameter is available exclusively for the **built-in** inverted index implementation. It controls the n-gram size used to build an n-gram dictionary index on top of the inverted index dictionary, which can significantly accelerate wildcard and substring queries. ###### How it works[​](#how-it-works "Direct link to How it works") When `dict_gram_num` is set to a positive integer (e.g., `2`), the built-in inverted index splits each dictionary entry into n-grams (substrings of the specified character length) during index construction. For example, if `dict_gram_num` is set to `2` and a dictionary entry is `"starrocks"`, the following 2-grams are generated: `"st"`, `"ta"`, `"ar"`, `"rr"`, `"ro"`, `"oc"`, `"ck"`, `"ks"`. During a query like `MATCH '%rock%'`, the query string `"rock"` is also split into 2-grams (`"ro"`, `"oc"`, `"ck"`), and the n-gram index quickly narrows down candidate dictionary entries, avoiding a full dictionary scan. This greatly improves the performance of wildcard queries, especially on columns with a large number of distinct values. ###### Parameter details[​](#parameter-details "Direct link to Parameter details") | Parameter | Default | Valid range | Description | | --------------- | --------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------ | | `dict_gram_num` | `-1` (disabled) | Positive integers (e.g., `1`, `2`, `3`, ...) | The n-gram size for building the dictionary n-gram index. Only valid with `imp_lib` = `builtin`. | ###### Guidelines for choosing dict\_gram\_num[​](#guidelines-for-choosing-dict_gram_num "Direct link to Guidelines for choosing dict_gram_num") * A smaller value (e.g., `2`) generates more n-grams per dictionary entry, increasing index size but providing better filtering for short query patterns. * A larger value (e.g., `4`) generates fewer n-grams, resulting in smaller index size but less effective filtering for short query patterns. * If your queries primarily use short wildcard patterns (e.g., `%ab%`), a smaller `dict_gram_num` value (such as `2`) is recommended. * If the query pattern is shorter than the `dict_gram_num` value, the n-gram index cannot be used for that query and falls back to a full dictionary scan. ###### Examples[​](#examples "Direct link to Examples") **Create built-in inverted index with `dict_gram_num`** 1. Create a table with a built-in inverted index and `dict_gram_num` set to `2`. ```sql CREATE TABLE `t_gram` ( `id1` bigint(20) NOT NULL COMMENT "", `text_val` varchar(255) NOT NULL COMMENT "", INDEX idx_gram (`text_val`) USING GIN ( "parser" = "english", "imp_lib" = "builtin", "dict_gram_num" = "2" ) COMMENT 'builtin index with ngram' ) DUPLICATE KEY(`id1`) DISTRIBUTED BY HASH(`id1`) PROPERTIES ( "replicated_storage" = "false" ); ``` 2. Insert test data. ```sql INSERT INTO t_gram VALUES (1, "starrocks is a high performance database"), (2, "apache spark is a data processing engine"), (3, "rocksdb is an embedded key value store"); ``` 3. Query with wildcard patterns. The n-gram dictionary index accelerates these queries. ```sql -- Query for rows containing "rock" in the text MySQL [example_db]> SELECT * FROM t_gram WHERE text_val MATCH "%rock%"; +------+--------------------------------------------+ | id1 | text_val | +------+--------------------------------------------+ | 1 | starrocks is a high performance database | | 3 | rocksdb is an embedded key value store | +------+--------------------------------------------+ 2 rows in set (0.01 sec) ``` **Add `dict_gram_num` to an existing table** You can also specify `dict_gram_num` when adding a built-in inverted index after table creation: ```sql CREATE INDEX idx_gram ON t (v) USING GIN('parser' = 'english', 'imp_lib' = 'builtin', 'dict_gram_num' = '3'); ``` --- ### N-gram bloom filter index Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) The N-gram bloom filter index is a special [Bloom filter index](https://docs.starrocks.io/docs/table_design/indexes/Bloomfilter_index.md) which is typically used to accelerate the LIKE queries and the calculation speed of the `ngram_search` and `ngram_search_case_insensitive` functions. The N-gram bloom filter index is only suitable for string (`STRING`, `CHAR`, or `VARCHAR`) type columns. The difference between the N-gram bloom filter index and Bloom filter index is that the N-gram bloom filter index first tokenizes the strings and then writes the resulting substrings into the Bloom filter index. For example, a certain indexed column value is a string `Technical`. For a traditional Bloom filter index, the entire string `Technical` would be directly written into the index. However, for an N-gram bloom filter index with a specified `gram_num` of `4`, the string `Technical` would be first tokenized into the following substrings: ```sql "Tech", "echn", "chni", "hnic", "nica", "ical" ``` And then each of these six substrings is written into the Bloom filter index. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * For a Duplicate Key or Primary Key table, you can create N-gram bloom filter indexes for all the columns (of the string types). For an Aggregate table or Unique Key table, you can only create N-gram bloom filter indexes on the key columns (of the string types). * N-gram bloom filter indexes can only be created for columns of the string types (CHAR, STRING, and VARCHAR). * To determine whether a query hits an N-gram bloom filter index, you can check the `BloomFilterFilterRows` field in the query's profile. * Only one type of index (Bloom Filter or N-gram Bloom Filter) can be created for a single column. #### Basic operation[​](#basic-operation "Direct link to Basic operation") ##### Create N-gram bloom filter index[​](#create-n-gram-bloom-filter-index "Direct link to Create N-gram bloom filter index") ```sql CREATE TABLE test.table1 ( k1 CHAR(10), k2 CHAR(10), v1 INT SUM, INDEX index_name (k2) USING NGRAMBF ("gram_num" = "4", "bloom_filter_fpp" = "0.05") COMMENT '' ) ENGINE = olap AGGREGATE KEY(k1, k2) DISTRIBUTED BY HASH(k1); ``` N-gram bloom filter index-related parameters: | **Parameter** | **Required** | **Description** | | ------------------ | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `index_name` | Yes | The name of the index. Index names must be unique within a table. | | `column_name` | Yes | The name of the column for which the index is created. Only a single column name can be specified. In the example above, it is `k2`. | | `gram_num` | NO | The length of a substring after the string in the indexed column is tokenized.The default value is 2. | | `bloom_filter_fpp` | No | The false positive possibility of the Bloom filter, ranging from 0.0001 to 0.05. The default value is 0.05. A smaller value provides better filtering but incurs greater storage overhead. | | `case_sensitive` | No | Whether this index is case-sensitive or not. Default value is `true`. | | `COMMENT` | No | Index comment. | For explanations of other parameters related to table creation, see [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md). ##### View N-gram bloom filter index[​](#view-n-gram-bloom-filter-index "Direct link to View N-gram bloom filter index") You can view all indexes of a table using `SHOW CREATE TABLE` or `SHOW INDEX FROM` table. Since index creation is asynchronous, you can only see the corresponding index after the index is successfully created. ```sql SHOW CREATE TABLE table1; SHOW INDEX FROM table1; ``` ##### Modify N-gram bloom filter index[​](#modify-n-gram-bloom-filter-index "Direct link to Modify N-gram bloom filter index") You can add and delete the N-gram bloom filter index by using the [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md) statement. * Execute the following statement to add a new N-gram bloom filter index `new_index_name` for the column `k1` to the table `table1`. ```sql ALTER TABLE table1 ADD INDEX new_index_name(k1) USING NGRAMBF ("gram_num" = "4", "bloom_filter_fpp" = "0.05") COMMENT ''; ``` * Execute the following statement to remove the N-gram bloom filter index `new_index_name` from the table `table1`. ```sql ALTER TABLE table1 DROP INDEX new_index_name; ``` note Altering an index is an asynchronous operation. You can view the progress of this operation by executing [SHOW ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_ALTER.md). You can run only one task to alter the index in a table each time. #### Query that can be accelerated[​](#query-that-can-be-accelerated "Direct link to Query that can be accelerated") ##### `LIKE` queries[​](#like-queries "Direct link to like-queries") An N-gram bloom filter index can accelerate `LIKE` queries if `gram_num` is small enough (smaller than the length of the string being queried for). Otherwise, the N-gram bloom filter index can not accelerate `LIKE` queries. For example, if `gram_num` is `4`, and the query statement is `SELECT * FROM table WHERE col1 LIKE "%abc"`, the N-Gram Bloom filter index will not accelerate this query, because the queried string is `abc`, which has only three characters, less than the value `4` of `gram_num`. If the query condition is `WHERE col1 LIKE "%abcd"` or `WHERE col1 LIKE "%abcde%"`, the N-Gram Bloom filter index will accelerate the query. ##### ngram\_search[​](#ngram_search "Direct link to ngram_search") When the `ngram_search` function is used in the query, if the column specified in the function has an N-gram bloom filter index and the `gram_num` specified in the function matches the `gram_num` of the N-gram bloom filter index, the index will automatically filter out data with a string similarity of 0, significantly speeding up the function execution process. ##### ngram\_search\_case\_insensitive[​](#ngram_search_case_insensitive "Direct link to ngram_search_case_insensitive") The usage of this function is the same as `ngram_search`, except that this function is case-sensitive. So it requires `case_sensitive` to be `false` when the N-gram bloom filter index is created. ```sql CREATE TABLE test.table1 ( k1 CHAR(10), k2 CHAR(10), v1 INT SUM, INDEX index_name (k2) USING NGRAMBF ("gram_num" = "4", "bloom_filter_fpp" = "0.05", "case_sensitive" = "false") COMMENT '' ) ENGINE = olap AGGREGATE KEY(k1, k2) DISTRIBUTED BY HASH(k1); ``` If the index is already created and its parameter `case_sensitive` is set to be `true`, you can alter the parameter of this index to `false` by using: ```sql ALTER TABLE table1 ADD INDEX new_index_name(k1) USING NGRAMBF ("gram_num" = "4", "bloom_filter_fpp" = "0.05", "case_sensitive" = "false") COMMENT ''; ``` --- ### Prefix indexes Specify one or more columns to comprise the sort key at table creation. The data rows in the table will be sorted based on the sort key and then stored on the disk. **During data writing, the Prefix index is automatically generated. After the data is sorted according to the specified sort key, every 1024 rows of data are included in one logical data block. An index entry that consists of the values of sort key columns of the first data row in that logical data block is added to the Prefix index table.** With these two layers of sorting structures, queries can use binary search to quickly skip data that does not meet the query conditions, and can also avoid additional sorting operations during queries. tip The Prefix index is a sparse index, and its size is at least 1024 times smaller than the data volume. Therefore, it can generally be fully cached in memory, to accelerate query performance. #### Usage notes[​](#usage-notes "Direct link to Usage notes") Since v3.0, Primary Key tables support defining sort keys using `ORDER BY`. Since v3.3, Duplicate Key tables, Aggregate tables, and Unique Key tables support defining sort keys using `ORDER BY`. * Data in the Duplicate Key table is sorted according to the sort key `ORDER BY`. The sort key can be combination of any columns. info When both `ORDER BY` and `DUPLICATE KEY` are specified, `DUPLICATE KEY` does not take effect. * Data in the Aggregate table is first aggregated according to the aggregate key `AGGREGATE KEY`, then sorted according to the sort key `ORDER BY`. The columns in `ORDER BY` and `AGGREGATE KEY` need to be same, but the order of the columns does not need to be same. * Data in the Unique Key table is first replaced according to the unique key `UNIQUE KEY`, then sorted according to the sort key `ORDER BY`. The columns in `ORDER BY` and `UNIQUE KEY` need to be same, but the order of the columns does not need to be same. * Data in the Primary Key table is first replaced according to the primary key `PRIMARY KEY`, then sorted according to the sort key `ORDER BY`. Take the Duplicate Key table as an example. The sort key is defined as `uid` and `name` using `ORDER BY`. ```sql CREATE TABLE user_access ( uid int, name varchar(64), age int, phone varchar(16), last_access datetime, credits double ) ORDER BY (uid, name); ``` tip After table creation, you can use `SHOW CREATE TABLE ;` to view the specified sort columns and the order of these columns in the `ORDER BY` clause from the returned result. Since the maximum length of a Prefix index entry is 36 bytes, the exceeded part will be truncated. Therefore, each entry in the Prefix index of this table is uid (4 bytes) + name (only the first 32 bytes are taken), and the prefix fields are `uid` and `name`. **Note** * The number of prefix fields cannot exceed 3, and the maximum length of a Prefix index entry is 36 bytes. * Within the prefix fields, columns of the CHAR, VARCHAR, or STRING type can only appear once and must be at the end. Take the following table as example, where the first three columns are sort key columns. The prefix field of this table is `name` (20 bytes). It is because that this Prefix index begins with the VARCHAR-type column (`name`) and is truncated directly without including further columns even though the Prefix index entry does not reach 36 bytes in length. Therefore, this Prefix index only contains the `name` field. ```sql MySQL [example_db]> describe user_access2; +-------------+-------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------------+-------------+------+-------+---------+-------+ | name | varchar(20) | YES | true | NULL | | | uid | int | YES | true | NULL | | | last_access | datetime | YES | true | NULL | | | age | int | YES | false | NULL | | | phone | varchar(16) | YES | false | NULL | | | credits | double | YES | false | NULL | | +-------------+-------------+------+-------+---------+-------+ 6 rows in set (0.00 sec) ``` * If the sort key is specified in the table using `ORDER BY`, the Prefix index is formed based on the sort key. If sort key is not specified using `ORDER BY`, the Prefix index is formed based on the Key columns. ##### How to design the sort key appropriately to form the Prefix index that can accelerate queries[​](#how-to-design-the-sort-key-appropriately-to-form-the-prefix-index-that-can-accelerate-queries "Direct link to How to design the sort key appropriately to form the Prefix index that can accelerate queries") An analysis of queries and data in business scenarios helps choose appropriate sort key columns and arrange them in a proper order to form a Prefix index, which can significantly improve query performance. * The number of sort key columns is generally 3 and is not recommended to exceed 4. A sort key with too many columns can not improve query performance but increase the sorting overhead during data loading. * It is recommended to prioritize columns to form the sort key in the order below: 1. **Select columns that are frequently used in the query filter conditions as sort key columns.** If the number of the sort key columns is more than one, arrange them in descending order of their frequencies in the query filter conditions. This way, if the query filter conditions include the prefix of the Prefix index, the query performance can be significantly improved. And if the filter conditions include the entire prefix of the Prefix index, the query can fully leverage the Prefix index. Of course, as long as the filter conditions include the prefix, though not the entire prefix, the Prefix index can still optimize the query. However, the effect of the Prefix index will be weakened if the length of the prefix included in the filter conditions is too short. Still, take the [Unique Key table](#usage-notes) whose sort key is `(uid,name)` as an example. If the query filter conditions include the entire prefix, such as `select sum(credits) from user_access where uid = 123 and name = 'Jane Smith';`, the query can fully utilize the Prefix index to improve performance. If the query conditions only include part of the prefix, such as `select sum(credits) from user_access where uid = 123;`, the query can also benefit from the Prefix index to improve performance. However, if the query conditions do not include the prefix, for example, `select sum(credits) from user_access where name = 'Jane Smith';`, the query can not use the Prefix index to accelerate. * If multiple sort key columns have similar frequencies as query filter conditions, you can measure the cardinality of these columns. * If the cardinality of the column is high, it can filter more data during the query. If the cardinality is too low, such as for Boolean-type columns, its filtering effect is not ideal. tip However, considering the query characteristics in actual business scenarios, usually, slightly lower cardinality columns are more frequently used as query conditions than high cardinality columns. This is because queries whose filtering is frequently based on high cardinality columns, or even in some extreme scenarios, based on columns with UNIQUE constraints are more like point queries in OLTP databases rather than complex analytical queries in OLAP databases. * Also, consider storage compression factors. If the difference in query performance between the order of a low-cardinality column and a high-cardinality one is not readily apparent, placing the low-cardinality column before the high-cardinality one will result in a much higher storage compression rate for the sorted low cardinality column. Therefore, it is recommended to place the low cardinality column in front. ##### Considerations for defining sort key columns at table creation[​](#considerations-for-defining-sort-key-columns-at-table-creation "Direct link to Considerations for defining sort key columns at table creation") * Data types of sort columns: * The sort columns in Primary Key tables support numeric types (including integers, booleans), strings, and date/datetime types. * The sort columns in Duplicate Key tables, Aggregate tables, and Unique Key tables support numeric types (including integers, booleans, decimals), strings, and date/datetime types. * In Aggregate tables and Unique Key tables, sort columns must be defined before other columns. ##### Can the Prefix index be modified?[​](#can-the-prefix-index-be-modified "Direct link to Can the Prefix index be modified?") If the characteristics of queries in the business scenario evolve and columns other than the prefix fields are frequently used in the query filter conditions, the existing Prefix index can not filter data and the query performance may not be ideal. Since v3.0, the sort keys of Primary Key tables can be modified. And since v3.3, the sort keys of Duplicate Key tables, Aggregate tables, and Unique Key tables can be modified. The sort keys in Duplicate Key tables and Primary Key tables can be combination of any sort columns. The sort keys in Aggregate tables and Unique Key tables must include all key columns, but the order of these columns does not need to be consistent with the key columns. Alternatively, you can also create [synchronous materialized views](https://docs.starrocks.io/docs/using_starrocks/Materialized_view-single_table.md) based on this table and choose other columns commonly used as conditional columns to form the Prefix index, which can improve performance for these queries. But note that it will increase storage space. #### How to verify whether the Prefix index accelerates queries[​](#how-to-verify-whether-the-prefix-index-accelerates-queries "Direct link to How to verify whether the Prefix index accelerates queries") After executing a query, you can check whether the Prefix index takes effect and view its filtering effect from detailed metrics, such as `ShortKeyFilterRows`, in the scan node in the [Query Profile](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md). --- ### Vector Index Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) This topic introduces the vector index feature of StarRocks and how to perform an approximate nearest neighbor search (ANNS) with it. The vector index feature is only supported in shared-nothing clusters of v3.4 or later. #### Overview[​](#overview "Direct link to Overview") Currently, StarRocks supports vector indexing at the Segment file level. The index maps each search item to its row ID within Segment files, allowing fast data retrieval by directly locating the corresponding data rows without brute-force vector distance calculations. The system now provides two types of vector indexing: Inverted File with Product Quantization (IVFPQ) and Hierarchical Navigable Small World (HNSW), each with its own organizational structure. ##### Inverted File with Product Quantization (IVFPQ)[​](#inverted-file-with-product-quantization-ivfpq "Direct link to Inverted File with Product Quantization (IVFPQ)") Inverted File with Product Quantization (IVFPQ) is a method for approximate nearest neighbor search in large-scale high-dimensional vectors, commonly used in vector retrieval tasks within deep learning and machine learning. IVFPQ consists of two main components: the inverted file and product quantization. * **Inverted File**: This component is an indexing method. It divides the dataset into multiple clusters (or Voronoi cells), each with a centroid (seed point), and assigns each data point (vector) to its nearest cluster center. For a vector search, IVFPQ only needs to search the nearest cluster, significantly reducing the search scope and complexity. * **Product Quantization**: This component is a data compression technique. It splits high-dimensional vectors into subvectors and quantizes each subvector to map it to the nearest point in a predefined set, reducing storage and computation costs while retaining high precision. By combining inverted files and product quantization, IVFPQ enables efficient approximate nearest neighbor search in large, high-dimensional datasets. ##### Hierarchical Navigable Small World (HNSW)[​](#hierarchical-navigable-small-world-hnsw "Direct link to Hierarchical Navigable Small World (HNSW)") Hierarchical Navigable Small World (HNSW) is a graph-based algorithm for high-dimensional nearest neighbor search, also widely used in vector retrieval tasks. HNSW builds a hierarchical graph structure in which each layer is a navigable small world (NSW) graph. In the graph, each vertex represents a data point, and edges indicate similarity between vertices. The higher layers of the graph contain fewer vertices with sparser connections for fast global search, while the lower layers include all vertices with denser connections for precise local search. For a vector search, HNSW searches at the top layer first, quickly identifying an approximate nearest neighbor region, then moves down layer by layer to find the exact nearest neighbor at the bottom layer. HNSW offers both efficiency and precision, making it adaptable to various data and query distributions. ##### Comparison between IVFPQ and HNSW[​](#comparison-between-ivfpq-and-hnsw "Direct link to Comparison between IVFPQ and HNSW") * **Data Compression Ratio**: IVFPQ has a higher compression ratio (around 1:0.15). The indexing calculation only provides preliminary sorting results after rough ranking because PQ will compress the vectors. It requires additional fine ranking for final sorting results, leading to higher computation and latency. HNSW has a lower compression ratio (around 1:0.8) and provides precise ranking without extra processing, resulting in lower computation costs and latency and higher storage costs. * **Recall Adjustment**: Both indexes support recall rate tuning through parameter adjustments, but IVFPQ incurs higher computation costs at similar recall rates. * **Caching Strategy**: IVFPQ allows balancing memory cost and computation latency by adjusting the cache proportion of index blocks, while HNSW currently only supports full-file caching. #### Usage[​](#usage "Direct link to Usage") Each table supports only one vector index. ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before creating vector indexes, you must enable it by setting the FE configuration item `enable_experimental_vector` to `true`. Execute the following statement to enable it dynamically: ```sql ADMIN SET FRONTEND CONFIG ("enable_experimental_vector" = "true"); ``` To enable it permanently, you must add `enable_experimental_vector = true` to the FE configuration file `fe.conf` and restart FE. ##### Create vector index[​](#create-vector-index "Direct link to Create vector index") This tutorial creates vector indexes while creating tables. You can also append vector indexes to an existing table. See [Append vector index](#append-vector-index) for detailed instructions. * The following example creates an HNSW vector index `hnsw_vector` on column `vector` in table `hnsw`. ```sql CREATE TABLE hnsw ( id BIGINT(20) NOT NULL COMMENT "", vector ARRAY NOT NULL COMMENT "", INDEX hnsw_vector (vector) USING VECTOR ( "index_type" = "hnsw", "dim"="5", "metric_type" = "l2_distance", "is_vector_normed" = "false", "M" = "16", "efconstruction" = "40" ) ) ENGINE=OLAP DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1; ``` * The following example creates an IVFPQ vector index `ivfpq_vector` on column `vector` in table `ivfpq`. ```sql CREATE TABLE ivfpq ( id BIGINT(20) NOT NULL COMMENT "", vector ARRAY NOT NULL COMMENT "", INDEX ivfpq_vector (vector) USING VECTOR ( "index_type" = "ivfpq", "dim"="5", "metric_type" = "l2_distance", "is_vector_normed" = "false", "nbits" = "16", "nlist" = "40" ) ) ENGINE=OLAP DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1; ``` ###### Index construction parameters[​](#index-construction-parameters "Direct link to Index construction parameters") ###### USING VECTOR[​](#using-vector "Direct link to USING VECTOR") * **Default**: N/A * **Required**: Yes * **Description**: Creates a vector index. ###### index\_type[​](#index_type "Direct link to index_type") * **Default**: N/A * **Required**: Yes * **Description**: Vector index type. Valid Values: `hnsw` and `ivfpq`. ###### dim[​](#dim "Direct link to dim") * **Default**: N/A * **Required**: Yes * **Description**: Dimension of the index. After the index is built, vectors that do not meet the dimension requirements will be rejected for loading into the base column. It must be an integer greater than or equal to `1`. ###### metric\_type[​](#metric_type "Direct link to metric_type") * **Default**: N/A * **Required**: Yes * **Description**: Metric type (measurement function) of the vector index. Valid values: * `l2_distance`: Euclidean distance. The smaller the value, the higher the similarity. * `cosine_similarity`: Cosine similarity. The larger the value, the higher the similarity. ###### is\_vector\_normed[​](#is_vector_normed "Direct link to is_vector_normed") * **Default**: false * **Required**: No * **Description**: Whether the vectors are normalized. Valid values are `true` and `false`. It takes effect only when `metric_type` is `cosine_similarity`. If the vectors are normalized, the value of the calculated distances will be within \[-1, 1]. The vectors must satisfy that the sum of the squares is `1`, otherwise an error is returned. ###### M[​](#m "Direct link to M") * **Default**: 16 * **Required**: No * **Description**: HNSW-specific parameter. The number of bi-directional connections created for every new element during graph construction. It must be an integer greater than or equal to `2`. The value of `M` directly affects the efficiency and accuracy of graph construction and search. During graph construction, each vertex will attempt to establish connections with its nearest `M` vertices. If a vertex already has `M` connections but finds a closer vertex, the farthest connection will be deleted and a new connection will be established with the closer vertex. A vector search will start from an entry point and find the nearest neighbor along the vertices connected to it. Therefore, the larger the value of `M`, the larger the search scope for each vertex, the higher the search efficiency, but the higher the cost of graph construction and storage. ###### efconstruction[​](#efconstruction "Direct link to efconstruction") * **Default**: 40 * **Required**: No * **Description**: HNSW-specific parameter. The size of the candidate list containing the nearest neighbors. It must be an integer greater than or equal to `1`. It is used to control the search depth during the graph construction process. Specifically, `efconstruction` defines the size of the search list (also known as the candidate list) for each vertex during the graph construction process. This candidate list is used to store the neighbor candidates of the current vertex, and the size of the list is `efconstruction`. The larger the value of `efconstruction`, the more candidates is considered as the neighbors of the vertex during the graph construction process, and, as a result, the better quality (such as better connectivity) of the graph, but also the higher time consumption and computation complexity of graph construction. ###### nbits[​](#nbits "Direct link to nbits") * **Default**: 16 * **Required**: No * **Description**: IVFPQ-specific parameter. Precision of Product Quantization (PQ). It must be the multiplier of `8`. With IVFPQ, each vector is divided into multiple subvectors, and then each subvector is quantized. `Nbits` defines the precision of quantization, that is, how many binary digits each subvector is quantized into. The larger the value of `nbits`, the higher the quantization precision, but the higher storage and computation costs. ###### nlist[​](#nlist "Direct link to nlist") * **Default**: 16 * **Required**: No * **Description**: IVFPQ-specific parameter. Number of clusters, or inverted lists. It must be an integer greater than or equal to `1`. With IVFPQ, the dataset is divided into clusters, and the centroid of each cluster corresponds to an inverted list. A vector search will first find the cluster centroid closest to the data point, and then retrieve the nearest neibors in the corresponding inverted list. Therefore, the value of `nlist` will affect the accuracy and efficiency of the search. The larger the value of `nlist`, the finer the granularity of clustering, thus the higher the search accuracy, but also the higher the search complexity. ###### M\_IVFPQ[​](#m_ivfpq "Direct link to M_IVFPQ") * **Required**: Yes * **Description**: IVFPQ-specific parameter. Number of the subvectors the original vector will be split into. The IVFPQ index will divide a `dim`-dimensional vector into `M_IVFPQ` equal-length subvectors. Therefore, it must be a factor of the value of `dim`. ###### Append vector index[​](#append-vector-index "Direct link to Append vector index") You can also add vector indexes to an existing table using [CREATE INDEX](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_INDEX.md) or [ALTER TABLE ADD INDEX](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md). Example: ```sql CREATE INDEX ivfpq_vector ON ivfpq (vector) USING VECTOR ( "index_type" = "ivfpq", "metric_type" = "l2_distance", "is_vector_normed" = "false", "dim"="5", "nlist" = "256", "nbits"="10" ); ALTER TABLE ivfpq ADD INDEX ivfpq_vector (vector) USING VECTOR ( "index_type" = "ivfpq", "metric_type" = "l2_distance", "is_vector_normed" = "false", "dim"="5", "nlist" = "256", "nbits"="10" ); ``` ##### Manage vector indexes[​](#manage-vector-indexes "Direct link to Manage vector indexes") ###### View vector index[​](#view-vector-index "Direct link to View vector index") You can view the definition of a vector index using the [SHOW CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_CREATE_TABLE.md) statement: Example: ```sql mysql> SHOW CREATE TABLE hnsw \G *************************** 1. row *************************** Table: hnsw Create Table: CREATE TABLE hnsw ( id bigint(20) NOT NULL COMMENT "", vector array NOT NULL COMMENT "", INDEX index_vector (vector) USING VECTOR("dim" = "5", "efconstruction" = "40", "index_type" = "hnsw", "is_vector_normed" = "false", "M" = "512", "metric_type" = "l2_distance") COMMENT '' ) ENGINE=OLAP DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ( "compression" = "LZ4", "fast_schema_evolution" = "true", "replicated_storage" = "false", "replication_num" = "3" ); 1 row in set (0.00 sec) ``` ###### Drop vector index[​](#drop-vector-index "Direct link to Drop vector index") You can drop vector indexes using [DROP INDEX](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_INDEX.md) or [ALTER TABLE DROP INDEX](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md). ```sql DROP INDEX ivfpq_vector ON ivfpq; ALTER TABLE ivfpq DROP INDEX ivfpq_vector; ``` ##### Perform ANNS with vector indexes[​](#perform-anns-with-vector-indexes "Direct link to Perform ANNS with vector indexes") Before running a vector search, make sure the FE configuration item `enable_experimental_vector` is set to `true`. ###### Requirements for vector index-based queries[​](#requirements-for-vector-index-based-queries "Direct link to Requirements for vector index-based queries") ```sql SELECT *, (v1, [1,2,3]) as dis FROM table_name WHERE (v1, [1,2,3]) <= 10 ORDER BY (v1, [1,2,3]) LIMIT 10 ``` To use vector index in queries, all the following requirements must be met: * **ORDER BY requirements:** * ORDER BY clause format: The `ORDER BY` clause must follow the format `ORDER BY (vector_column, constant_array)` without including additional ORDER BY columns. * Function name requirements for ``: * If `metric_type` is `l2_distance`, the function name must be `approx_l2_distance`. * If `metric_type` is `cosine_similarity`, the function name must be `approx_cosine_similarity`. * Parameter requirements for ``: * One of the column `constant_array` must be a constant `ARRAY` with dimensions matching the vector index `dim`. * The other column `vector_column` must be the column corresponding to the vector index. * ORDER direction requirements: * If `metric_type` is `l2_distance`, the order must be `ASC`. * If `metric_type` is `cosine_similarity`, the order must be `DESC`. * A `LIMIT N` clause is required. * **Predicate Requirements:** * All predicates must be `` expressions, combined using `AND` and comparison operators (`>` or `<`). The comparison operator direction must align with the `ASC`/`DESC` order. Specifically: * Requirement 1: * If `metric_type` is `l2_distance`: `col_ref <= constant`. * If `metric_type` is `cosine_similarity`: `col_ref >= constant`. * Here, `col_ref` refers to the result of `(vector_column, constant_array)` and can be cast to `FLOAT` or `DOUBLE` types, for example: * `approx_l2_distance(v1, [1,2,3])` * `CAST(approx_l2_distance(v1, [1,2,3]) AS FLOAT)` * `CAST(approx_l2_distance(v1, [1,2,3]) AS DOUBLE)` * Requirement 2: * The predicate must use `AND`, with each child predicate meeting Requirement 1. ###### Preparation[​](#preparation "Direct link to Preparation") Create tables with vector indexes and insert vector data into them: ```sql CREATE TABLE test_hnsw ( id BIGINT(20) NOT NULL COMMENT "", vector ARRAY NOT NULL COMMENT "", INDEX index_vector (vector) USING VECTOR ( "index_type" = "hnsw", "metric_type" = "l2_distance", "is_vector_normed" = "false", "M" = "512", "dim"="5") ) ENGINE=OLAP DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1; INSERT INTO test_hnsw VALUES (1, [1,2,3,4,5]), (2, [4,5,6,7,8]); CREATE TABLE test_ivfpq ( id BIGINT(20) NOT NULL COMMENT "", vector ARRAY NOT NULL COMMENT "", INDEX index_vector (vector) USING VECTOR ( "index_type" = "ivfpq", "metric_type" = "l2_distance", "is_vector_normed" = "false", "nlist" = "256", "nbits"="8", "dim"="5", "M_IVFPQ"="1") ) ENGINE=OLAP DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1; INSERT INTO test_ivfpq VALUES (1, [1,2,3,4,5]), (2, [4,5,6,7,8]); ``` ###### Perform vector search[​](#perform-vector-search "Direct link to Perform vector search") ###### Approximate search[​](#approximate-search "Direct link to Approximate search") Approximate searches will hit the vector index and thereby accelerate the search process. The following example searches for the top 1 approximate nearest neighbor of the vector `[1,1,1,1,1]`. ```sql SELECT id, approx_l2_distance([1,1,1,1,1], vector) FROM test_hnsw ORDER BY approx_l2_distance([1,1,1,1,1], vector) LIMIT 1; ``` ###### Scalar-vector joint search[​](#scalar-vector-joint-search "Direct link to Scalar-vector joint search") You can combine scalar search with vector search. ```sql SELECT id, approx_l2_distance([1,1,1,1,1], vector) FROM test_hnsw WHERE id = 1 ORDER BY approx_l2_distance([1,1,1,1,1], vector) LIMIT 1; ``` ###### Range search[​](#range-search "Direct link to Range search") You can perform range search on vector data. The following example pushes the `score < 40` condition down to the index and filters vectors by `score` range. ```sql SELECT * FROM ( SELECT id, approx_l2_distance([1,1,1,1,1], vector) score FROM test_hnsw ) a WHERE score < 40 ORDER BY score LIMIT 1; ``` ###### Precise calculation[​](#precise-calculation "Direct link to Precise calculation") Precise calculations will ignore the vector index, and directly calculate distances between the vectors for precise results. ```sql SELECT id, l2_distance([1,1,1,1,1], vector) FROM test_hnsw WHERE id = 1 ORDER BY l2_distance([1,1,1,1,1], vector) LIMIT 1; ``` > **NOTE** > > Different distance metric functions define "similarity" differently. For `l2_distance`, smaller values indicate higher similarity; for `cosine_similarity`, larger values indicate higher similarity. Therefore, when calculating `topN`, the sorting (ORDER BY) direction should match the similarity direction of the metric. Use `ORDER BY ASC LIMIT x` for `l2_distance`, and `ORDER BY DESC LIMIT x` for `cosine_similarity`. ###### Fine tune index parameters for search[​](#fine-tune-index-parameters-for-search "Direct link to Fine tune index parameters for search") Parameter tuning is essential in vector search, as it affects both performance and accuracy. It is recommended to tune the search paramters on a small dataset and move to large datasets only when expected recall and latency are achieved. Search parameters are passed through hints in SQL statements. ###### For HNSW index[​](#for-hnsw-index "Direct link to For HNSW index") Before proceeding, make sure the vector column is built with the HNSW index. ```sql SELECT /*+ SET_VAR (ann_params='{efsearch=256}') */ id, approx_l2_distance([1,1,1,1,1], vector) FROM test_hnsw WHERE id = 1 ORDER BY approx_l2_distance([1,1,1,1,1], vector) LIMIT 1; ``` **Parameter**: ###### efsearch[​](#efsearch "Direct link to efsearch") * **Default**: 16 * **Required**: No * **Description**: Parameter that controls precision-speed tradeoff. During a hierarchical graph structure search, this parameter controls the size of the candidate list during the search. The larger the value of `efsearch`, the higher the accuracy, but the lower the speed. ###### For IVFPQ index[​](#for-ivfpq-index "Direct link to For IVFPQ index") Before proceeding, make sure the vector column is built with the IVFPQ index. ```sql SELECT /*+ SET_VAR (ann_params='{nprobe=256,max_codes=0,scan_table_threshold=0,polysemous_ht=0,range_search_confidence=0.1}') */ id, approx_l2_distance([1,1,1,1,1], vector) FROM test_ivfpq ORDER BY approx_l2_distance([1,1,1,1,1], vector) LIMIT 1; ``` **Parameter**: ###### nprobe[​](#nprobe "Direct link to nprobe") * **Default**: 1 * **Required**: No * **Description**: Number of inverted lists checked during the search. The larger the value of `nprobe`, the higher the accuracy, but the lower the speed. ###### max\_codes[​](#max_codes "Direct link to max_codes") * **Default**: 0 * **Required**: No * **Description**: Maximum codes checked per inverted list. This parameter will also affect accuracy and speed. ###### scan\_table\_threshold[​](#scan_table_threshold "Direct link to scan_table_threshold") * **Default**: 0 * **Required**: No * **Description**: Parameter that controls polysemous hashing. When the Hamming distance between an element's hash and the hash of the vector to be searched is below this threshold, the element is added to the candidate list. ###### polysemous\_ht[​](#polysemous_ht "Direct link to polysemous_ht") * **Default**: 0 * **Required**: No * **Description**: Parameter that controls polysemous hashing. When the Hamming distance between an element's hash and the hash of the vector to be searched is below this threshold, the element is directly added to the results. ###### range\_search\_confidence[​](#range_search_confidence "Direct link to range_search_confidence") * **Default**: 0.1 * **Required**: No * **Description**: Confidence of approximate range searches. Value range: \[0, 1]. Setting it to `1` produces the most accurate results. ###### Calculate approximate recall[​](#calculate-approximate-recall "Direct link to Calculate approximate recall") You can calculate approximate recall by intersecting the `topK` elements from brute-force retrieval with those from approximate retrieval: `Recall = TP / (TP + FN)`. ```sql -- Approximate retrieval SELECT id FROM test_hnsw ORDER BY approx_l2_distance([1,1,1,1,1], vector) LIMIT 5; 8 9 7 5 1 -- Brute-force retrieval SELECT id FROM test_hnsw ORDER BY l2_distance([1,1,1,1,1], vector) LIMIT 5; 8 9 5 7 10 ``` In the preceding example, approximate retrieval returns 8, 9, 7, and 5. However, the correct result is 8, 9, 5, 7, and 10. In this case, recall is 4/5=80%. ###### Check if vector indexes take effect[​](#check-if-vector-indexes-take-effect "Direct link to Check if vector indexes take effect") Execute [EXPLAIN](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/EXPLAIN.md) against the query statement. If `OlapScanNode` properties show `VECTORINDEX: ON`, it indicates that the vector index has been applied to approximate vector searches. Example: ```sql > EXPLAIN SELECT id FROM t_test_vector_table ORDER BY approx_l2_distance([1,1,1,1,1], vector) LIMIT 5; +-----------------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String | +-----------------------------------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS:1: id | | PARTITION: UNPARTITIONED | | | | RESULT SINK | | | | 4:Project | | | : 1: id | | | limit: 5 | | | | | 3:MERGING-EXCHANGE | | limit: 5 | | | | PLAN FRAGMENT 1 | | OUTPUT EXPRS: | | PARTITION: RANDOM | | | | STREAM DATA SINK | | EXCHANGE ID: 03 | | UNPARTITIONED | | | | 2:TOP-N | | | order by: 3: approx_l2_distance ASC | | | offset: 0 | | | limit: 5 | | | | | 1:Project | | | : 1: id | | | : 4: __vector_approx_l2_distance | | | | | 0:OlapScanNode | | TABLE: t_test_vector_table | | VECTORINDEX: ON | | IVFPQ: OFF, Distance Column: <4:__vector_approx_l2_distance>, LimitK: 5, Order: ASC, Query Vector: [1, 1, 1, 1, 1], Predicate Range: -1.0 | | PREAGGREGATION: ON | | partitions=1/1 | | rollup: t_test_vector_table | | tabletRatio=1/1 | | tabletList=11302 | | cardinality=2 | | avgRowSize=4.0 | +-----------------------------------------------------------------------------------------------------------------------------------------------------+ ``` #### Limitations[​](#limitations "Direct link to Limitations") * Each table supports only one vector index. --- ### Table overview Tables are units of data storage. Understanding the table structure in StarRocks and how to design an efficient table structure helps optimize data organization and enhance query efficiency. Also, compared to traditional databases, StarRocks can store complex semi-structured data such as JSON, ARRAY, in a columnar manner to improve query performance. This topic introduces the table structure in StarRocks from both basic and general perspectives. From v3.3.1 onwards, StarRocks supports creating temporary tables in the Default Catalog. #### Get started with basic table structure[​](#get-started-with-basic-table-structure "Direct link to Get started with basic table structure") Like in other relational databases, a table is logically composed of rows and columns: * Rows: Each row holds a record. Each row contains a set of related data values. * Columns: Columns define attributes for each record. Each column holds data of a specific attribute. For example, an employee table may include columns like name, employee ID, department, and salary, where each column stores corresponding data. Data in each column is of the same data type. All rows in a table have the same number of columns. It is simple to create a table in StarRocks. You just need to define columns and their data types in the CREATE TABLE statement to create a table. Example: ```sql CREATE DATABASE example_db; USE example_db; CREATE TABLE user_access ( uid int, name varchar(64), age int, phone varchar(16), last_access datetime, credits double ) ORDER BY (uid, name); ``` The above CREATE TABLE example creates a Duplicate Key table. No constraint is added to columns in this type of table, so duplicate data rows can exist in the table. The first two columns of the Duplicate Key table are specified as sort columns to form the sort key. Data is stored after being sorted based on the sort key, which can accelerate indexing during queries. Since v3.3.0, the Duplicate Key table supports specifying the sort key using `ORDER BY`. If both `ORDER BY` and `DUPLICATE KEY` are used, `DUPLICATE KEY` does not take effect. note If a StarRocks cluster in a staging environment contains only one BE, the number of replicas can be set to `1` in the `PROPERTIES` clause, such as `PROPERTIES( "replication_num" = "1" )`. The default number of replicas is 3, which is also the number recommended for production StarRocks clusters. If you want to use the default number, you do not need to configure the `replication_num` parameter. Execute [DESCRIBE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md) to view the table schema. ```sql MySQL [test]> DESCRIBE user_access; +-------------+-------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------------+-------------+------+-------+---------+-------+ | uid | int | YES | true | NULL | | | name | varchar(64) | YES | true | NULL | | | age | int | YES | false | NULL | | | phone | varchar(16) | YES | false | NULL | | | last_access | datetime | YES | false | NULL | | | credits | double | YES | false | NULL | | +-------------+-------------+------+-------+---------+-------+ 6 rows in set (0.00 sec) ``` Execute [SHOW CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_CREATE_TABLE.md) to view the CREATE TABLE statement. ```sql MySQL [example_db]> SHOW CREATE TABLE user_access\G *************************** 1. row *************************** Table: user_access Create Table: CREATE TABLE `user_access` ( `uid` int(11) NULL COMMENT "", `name` varchar(64) NULL COMMENT "", `age` int(11) NULL COMMENT "", `phone` varchar(16) NULL COMMENT "", `last_access` datetime NULL COMMENT "", `credits` double NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`uid`, `name`) DISTRIBUTED BY RANDOM ORDER BY(`uid`, `name`) PROPERTIES ( "bucket_size" = "4294967296", "compression" = "LZ4", "fast_schema_evolution" = "true", "replicated_storage" = "true", "replication_num" = "3" ); 1 row in set (0.01 sec) ``` #### Understand comprehensive table structure[​](#understand-comprehensive-table-structure "Direct link to Understand comprehensive table structure") A deep dive into StarRocks table structures helps you design efficient data management structure tailored to your business needs. ##### [Table types](https://docs.starrocks.io/docs/table_design/table_types.md)[​](#table-types "Direct link to table-types") StarRocks provides four types of tables which are Duplicate Key tables, Primary Key tables, Aggregate tables, and Unique Key tables, to store data for various business scenarios, such as raw data, frequently updated realtime data, and aggregated data. * Duplicate Key tables are simple and easy to use. No constraint is added to columns in this type of table, so duplicate data rows can exist in the table. Duplicate Key tables are suitable for storing raw data, such as logs, that does not need any constraints or pre-aggregation. * Primary Key tables are powerful. Both unique and non-null constraints are added to primary key columns. Primary Key tables support real-time frequent updates and partial column updates, while ensuring high query performance, and therefore are suitable for real-time queries scenarios. * Aggregate tables are suitable to store pre-aggregated data, helping reduce the amount of data scanned and calculated and improve efficiency for aggregation queries. * Unique tables are also suitable to store frequently updated realtime data. However this type of tables is being replaced by Primary Key tables, which are more powerful. ##### [Data distribution](https://docs.starrocks.io/docs/table_design/data_distribution.md)[​](#data-distribution "Direct link to data-distribution") StarRocks uses a partitioning+bucketing two-tier data distribution strategy, to evenly distribute data across BEs. A well-designed data distribution strategy can effectively reduce the amount of data scanned and maximize StarRocks' concurrent processing capabilities, thereby increasing query performance. ![img](/assets/images/table_overview-2bf1cedfc3be7f074b1ddc5d6831289f.png) ###### Partitioning[​](#partitioning "Direct link to Partitioning") The first level is partitioning: Data in tables can be divided into smaller data management units based on partitioning columns which are usually columns that hold dates and time. During queries, partition pruning can reduce the amount of data that needs to be scanned, effectively optimizing query performance. StarRocks provides an easy-to-use partitioning method, expression partitioning, and also offers more flexible methods like range and list partitioning. ###### Bucketing[​](#bucketing "Direct link to Bucketing") The second level is bucketing: Data within a partition is further divided into smaller data management units through bucketing. Replicas of each bucket are evenly distributed across BEs to ensure high data availability. StarRocks provides two bucketing methods: * Hash bucketing: Data is distributed into buckets based on the hash values of the bucketing key. You can select columns frequently used as condition columns in queries as bucketing columns, which helps improve query efficiency. * Random bucketing: Data is randomly distributed to buckets. This bucketing method is more simple and ease to use. ##### [Data types](https://docs.starrocks.io/docs/sql-reference/data-types)[​](#data-types "Direct link to data-types") In addition to basic data types such as NUMERIC, DATE, and STRING, StarRocks supports complex semi-structured data types, including ARRAY, JSON, MAP, and STRUCT. ##### [Index](https://docs.starrocks.io/docs/table_design/indexes.md)[​](#index "Direct link to index") An index is a special data structure and is used as a pointer to data in a table. When the conditional columns in queries are indexed columns, StarRocks can swiftly locate the data that meets the conditions. StarRocks provides built-in indexes: Prefix indexes, Ordinal indexes, and ZoneMap indexes. StarRocks also allows users to create indexes, that is, Bitmap indexes and Bloom Filter indexes, to further enhance query efficiency. ##### Constraints[​](#constraints "Direct link to Constraints") Constraints help ensure data integrity, consistency, and accuracy. The primary key columns in Primary Key tables must have unique and NOT NULL values. The aggregate key columns in Aggregate tables and the unique key columns in Unique Key tables must have unique values. ##### Temporary table[​](#temporary-table "Direct link to Temporary table") When processing data, you might need to save intermediate results for future reuse. In early versions, StarRocks only supports using CTE (Common Table Expressions) to define temporary results within a single query. However, CTEs are merely logical constructs, do not physically store the results, and cannot be used across different queries, which presents certain limitations. If you choose to create tables to save intermediate results, you will need to manage the lifecycle of these tables, which can be costly. To address this issue, StarRocks introduces temporary tables in v3.3.1. Temporary tables allow you to temporarily store data (such as intermediate results from ETL processes) in a table, with their lifecycle bound to the session and managed by StarRocks. When the session ends, the temporary tables are automatically cleared. Temporary tables are only visible within the current session, and different sessions can create temporary tables with the same name. ###### Usage[​](#usage "Direct link to Usage") You can use the `TEMPORARY` keyword in the following SQL statements to create and drop temporary tables: * [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) * [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) * [CREATE TABLE LIKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_LIKE.md) * [DROP TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md) note Similar to other types of native tables, temporary tables must be create under a database under the Default Catalog. However, because temporary tables are session-based, they are not subject to unique naming constraints. You can create temporary tables with the same name in different sessions, or even create temporary tables with the same names as other non-temporary, native tables. If there are temporary and non-temporary tables with the same name in a database, the temporary table takes precedence. Within the session, all queries and operations on the tables with the same name will only affect the temporary table. ###### Limitations[​](#limitations "Direct link to Limitations") While the usage of temporary tables is similar to that of native tables, there are some constraints and differences: * Temporary tables must be created in the Default Catalog. * Setting a colocate group is not supported. If the `colocate_with` property is explicitly specified during table creation, it will be ignored. * The `ENGINE` must be specified as `olap` during table creation. * ALTER TABLE statements are not supported. * Creating views and materialized views based on temporary tables is not supported. * EXPORT statements are not supported. * Creating temporary tables or loading data from/into them with asynchronous tasks using SUBMIT TASK is not supported. ##### More features[​](#more-features "Direct link to More features") Apart from the above features, you can adopt more features based on your business requirements to design a more robust table structure. For example, using Bitmap and HLL columns to accelerate distinct counting, specifying generated columns or auto-increment columns to speed up some queries, configuring flexible and automatic storage cooldown methods to reduce maintenance costs, and configuring Colocate Join to speed up multi-table JOIN queries. For more details, see [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md). --- ### Overview of table types You must specify a table type and define one or more columns as a sort key at table creation. This way, when data is initially loaded into the table that you created, StarRocks can sort, process, and store the data based on the sort key. This topic describes the table types that StarRocks provides to meet your varying business requirements. #### Basic concepts[​](#basic-concepts "Direct link to Basic concepts") ##### Table types[​](#table-types "Direct link to Table types") StarRocks provides four table types: Duplicate Key table, Aggregate table, Unique Key table, and Primary Key table. These four table types are well suited to a wide range of data analytics scenarios such as log analysis, data aggregation and analysis, and real-time data analysis. ##### Sort keys[​](#sort-keys "Direct link to Sort keys") When data is loaded into a table created by using a certain table type, data is sorted and stored according to one or more columns defined as the sort key when the table is created. The sort key is usually one or more columns that are frequently used as filter conditions in queries, thereby accelerating queries. Since v3.0, Primary Key tables support defining sort keys using `ORDER BY`. Since v3.3, Duplicate Key tables, Aggregate tables, and Unique Key tables support defining sort keys using `ORDER BY`. For more descriptions of sort keys, see [Sort keys and prefix indexes](https://docs.starrocks.io/docs/table_design/indexes/Prefix_index_sort_key.md). #### Precautions[​](#precautions "Direct link to Precautions") * After a table is created, you cannot modify its table types. For example, you cannot change a Duplicate Key table to a Primary Key table. You must create a new Primary Key table. * Sort key columns must be defined prior to the other columns in the statement for table creation. * The order of sort key columns in the statement for table creation specifies the order of the conditions based on which the rows in the table are sorted. * The length of the prefix index for a table is limited to 36 bytes. If the total length of the sort key columns exceeds 36 bytes, StarRocks stores only the first few sort key columns within the length limit for the prefix index. * If the records to be loaded into a table have the same primary key, StarRocks processes and stores the records based on the table type: * Duplicate Key table StarRocks loads each of the records as a separate row into the table. After the data load is complete, the table contains rows that have the same primary key, and the rows map the source records in a one-to-one relationship. You can recall all historical data that you loaded. * Aggregate table StarRocks aggregates the records into one record and loads the aggregated record as a row into the table. After the loading is complete, the table does not contain rows that have the same primary key. You can recall the aggregation results of all historical data that you loaded. However, you cannot recall all historical data. * Unique Key table and Primary Key table StarRocks replaces each previously loaded record with the newly loaded record and retains only the most recently loaded record as a row in the table. After the loading is complete, the table does not contain previous rows that have the same primary key. Only the latest record that contains the same unique key/primary key is retained. The Unique Key table and the Primary Key table can be considered a special Aggregate table in which the REPLACE aggregate function is specified for metric columns to return the most recent record among a group of records that have the same primary key. --- ### Aggregate table At table creation, you can define an aggregate key and specify an aggregate function for the value column. When multiple rows of data have the same aggregate key, the values in the value columns are aggregated. Additionally, you can define the sort key separately. If the filter conditions in queries include the sort key, StarRocks can quickly filter the data, improving query efficiency. In data analysis and aggregation scenarios, Aggregate tables can reduce the amount of data that needs to be processed, thereby enhancing query efficiency. #### Scenarios[​](#scenarios "Direct link to Scenarios") The Aggregate table is well suited to data statistics and analytics scenarios. A few examples are as follows: * Help website or app providers analyze the amount of traffic and time that their users spend on a specific website or app and the total number of visits to the website or app. * Help advertising agencies analyze the total clicks, total views, and consumption statistics of an advertisement that they provide for their customers. * Help e-commerce companies analyze their annual trading data to identify the geographic bestsellers within individual quarters or months. The data query and ingestion in the preceding scenarios have the following characteristics: * Most queries are aggregate queries, such as SUM, MAX, and MIN. * Raw detailed data does not need to be retrieved. * Historical data is not frequently updated. Only new data is appended. #### Principle[​](#principle "Direct link to Principle") From the phrase of data ingestion to data query, data in the Aggregate tables is aggregated multiple times as follows: 1. In the data ingestion phase, each batch of data forms a version when data is loaded into the Aggregate table in batches. In one version, data with the same aggregate key will be aggregated. 2. In the background compaction phase, when the files of multiple data versions that are generated at data ingestion are periodically compacted into a large file, StarRocks aggregates the data that has the same aggregate key in the large file. 3. In the data query phase, StarRocks aggregates the data that has the same aggregate key among all data versions before it returns the query result. The aggregate operations help reduce the amount of data that needs to be processed, thereby accelerating queries. Suppose that you have a table that uses the Aggregate table and want to load the following four raw records into the table. | Date | Country | PV | | ---------- | ------- | -- | | 2020.05.01 | CHN | 1 | | 2020.05.01 | CHN | 2 | | 2020.05.01 | USA | 3 | | 2020.05.01 | USA | 4 | StarRocks aggregates the four raw records into the following two records at data ingestion. | Date | Country | PV | | ---------- | ------- | -- | | 2020.05.01 | CHN | 3 | | 2020.05.01 | USA | 7 | #### Create a table[​](#create-a-table "Direct link to Create a table") Suppose that you want to analyze the numbers of visits by users from different cities to different web pages. In this example, create a table named `example_db.aggregate_tbl`, define `site_id`, `date`, and `city_code` as the aggregate key, define `pv` as a value column, and specify the SUM function for the `pv` column. The statement for creating the table is as follows: ```sql CREATE TABLE aggregate_tbl ( site_id LARGEINT NOT NULL COMMENT "id of site", date DATE NOT NULL COMMENT "time of event", city_code VARCHAR(20) COMMENT "city_code of user", pv BIGINT SUM DEFAULT "0" COMMENT "total page views" ) AGGREGATE KEY(site_id, date, city_code) DISTRIBUTED BY HASH(site_id); ``` > **NOTICE** > > * When you create a table, you must specify the bucketing column by using the `DISTRIBUTED BY HASH` clause. For detailed information, see [bucketing](https://docs.starrocks.io/docs/table_design/data_distribution.md#bucketing). > * Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). #### Generic aggregate function states[​](#generic-aggregate-function-states "Direct link to Generic aggregate function states") Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) StarRocks supports generic aggregate function states from v3.4.0. In data analysis and summaries, Aggregate tables reduce the data processed during queries, improving query performance. For large datasets, Aggregate tables are highly effective as they summarize data by dimensions before querying. They also serve as a vital method for incremental aggregate function computation in StarRocks. However, in earlier versions, support is limited to built-in functions such as `SUM`, `MAX`, `MIN`, `REPLACE`, `HLL_UNION`, `PERCENTILE_UNION`, and `BITMAP_UNION`, while in theory, all built-in aggregate functions can be used in aggregate tables. To address this limitation, generic aggregate states is introduced to support the storage of all built-in function states. ##### Store generic aggregate states[​](#store-generic-aggregate-states "Direct link to Store generic aggregate states") You can define generic aggregate states in Aggregate tables by specifying the function name and input parameter types to uniquely identify an aggregate function. The column type will automatically be inferred as the intermediate state type of the aggregate function. Definition: ```sql col_name agg_func_name(parameter1_type, [parameter2_type], ...) ``` * **col\_name**: Name of the column. * **agg\_func\_name**: Name of the aggregate function whose intermediate states needs to be stored. * **parameter\_type**: Input parameter type of the aggregate function. The function can be identified uniquely with the parameter type. note * Only StarRocks built-in functions with at least one parameter are supported. Java and Python UDAFs are not supported. * For stability and extensibility, the aggregate state column type is always Nullable (except for the count function) and cannot be modified. * Parameter values are unnecessary for defining multi-parameter functions because types can be inferred and parameter values are not involved in computation. * Complex parameters like ORDER BY and DISTINCT are not supported. * The support for specific built-in functions, such as `GROUP_CONCAT`, `WINDOW_FUNNEL`, and `APPROX_TOP_K`, is still under development. They will be supported in future releases. For detailed information, see [FunctionSet.java#UNSUPPORTED\_AGG\_STATE\_FUNCTIONS](https://github.com/StarRocks/starrocks/blob/main/fe/fe-core/src/main/java/com/starrocks/catalog/FunctionSet.java#L776). Example: ```sql CREATE TABLE test_create_agg_table ( dt VARCHAR(10), -- Define generic aggregate state storage. hll_sketch_agg ds_hll_count_distinct(varchar), avg_agg avg(bigint), array_agg_agg array_agg(int), min_by_agg min_by(varchar, bigint) ) AGGREGATE KEY(dt) PARTITION BY (dt) DISTRIBUTED BY HASH(dt) BUCKETS 4; ``` ##### Combinator functions[​](#combinator-functions "Direct link to Combinator functions") Generic aggregate states use combinator functions to encapsulate intermediate state computation and flow. ###### `_state` combinator function[​](#_state-combinator-function "Direct link to _state-combinator-function") The `_state` function converts input parameters into intermediate state types. Definition: ```sql agg_intermediate_type {agg_func_name}_state(input_col1, [input_col2], ...) ``` * **agg\_func\_name**: Name of the aggregate function that needs to transfer the input parameters into intermediate state types. * **input\_col1/col2**: Input columns of the aggregate function. * **agg\_intermediate\_type**: Return type of the `_state` function, that is, the intermediate state type of the aggregate function. note `_state` is a scalar function. You do not need to define the aggregate column for the computation of input parameter state. Example: ```sql CREATE TABLE t1 ( id BIGINT NOT NULL, province VARCHAR(64), age SMALLINT, dt VARCHAR(10) NOT NULL ) DUPLICATE KEY(id) PARTITION BY (dt) DISTRIBUTED BY HASH(id) BUCKETS 4; INSERT INTO t1 SELECT generate_series, generate_series, generate_series % 10, "2024-07-24" FROM table(generate_series(1, 100)); -- Transfer the data in t1 with _state combinator function, and insert it into the Aggregate table. INSERT INTO test_create_agg_table SELECT dt, ds_hll_count_distinct_state(id), avg_state(id), array_agg_state(id), min_by_state(province, id) FROM t1; ``` ###### `_union` combinator function[​](#_union-combinator-function "Direct link to _union-combinator-function") The `_union` function merges multiple intermediate state columns into a single state. Definition: ```sql -- Union multiple aggregate intermediate states. agg_intermediate_type {agg_func_name}_union(input_col) ``` * **agg\_func\_name**: Name of the aggregate function. * **input\_col**: Input columns of the aggregate function. The input column type is the intermediate state type of the aggregate function. You can obtain it using `_state` functions. * **agg\_intermediate\_type**: Return type of the `_union` function, that is, the intermediate state type of the aggregate function. note `_union` is an aggregate function. It returns the intermediate state type instead of the type of function's final result. Example: ```sql -- Case 1: Union the intermediate states of the Aggregate table. SELECT dt, ds_hll_count_distinct_union(hll_sketch_agg), avg_union(avg_agg), array_agg_union(array_agg_agg), min_by_union(min_by_agg) FROM test_create_agg_table GROUP BY dt LIMIT 1; -- Case 2: Union the intermediate states input by the _state combinator function. SELECT dt, ds_hll_count_distinct_union(ds_hll_count_distinct_state(id)), avg_union(avg_state(id)), array_agg_union(array_agg_state(id)), min_by_union(min_by_state(province, id)) FROM t1 GROUP BY dt LIMIT 1; ``` ###### `_merge` combinator function[​](#_merge-combinator-function "Direct link to _merge-combinator-function") The `_merge` combinator function encapsulates aggregate functions as a generic aggregate function to calculate the final aggregation result of multiple intermediate states. Definition: ```sql -- Merge multiple aggregate intermediate states. agg_result_type {agg_func_name}_merge(input_col) ``` * **agg\_func\_name**: Name of the aggregate function. * **input\_col**: Input columns of the aggregate function. The input column type is the intermediate state type of the aggregate function. You can obtain it using `_state` functions. * **agg\_intermediate\_type**: Return type of the `_merge` function, that is, the final aggregation result of the aggregate function. Example: ```sql -- Case 1: Merge the intermediate states of the Aggregate table to obtain the final aggregation result. SELECT dt, ds_hll_count_distinct_merge(hll_sketch_agg), avg_merge(avg_agg), array_agg_merge(array_agg_agg), min_by_merge(min_by_agg) FROM test_create_agg_table GROUP BY dt LIMIT 1; -- Case 2: Merge the intermediate states input by the _state combinator function to obtain the final aggregation result. SELECT dt, ds_hll_count_distinct_merge(ds_hll_count_distinct_state(id)), avg_merge(avg_state(id)), array_agg_merge(array_agg_state(id)), min_by_merge(min_by_state(province, id)) FROM t1 GROUP BY dt LIMIT 1; ``` ##### Use generic aggregate states in materialized views[​](#use-generic-aggregate-states-in-materialized-views "Direct link to Use generic aggregate states in materialized views") Generic aggregate states can be used in synchronous and asynchronous materialized views to accelerate query performance with rollup of aggregate states. ###### Generic aggregate states in synchronous materialized views[​](#generic-aggregate-states-in-synchronous-materialized-views "Direct link to Generic aggregate states in synchronous materialized views") Example: ```sql -- Create a synchronous materialized view test_mv1 to store aggregate states. CREATE MATERIALIZED VIEW test_mv1 AS SELECT dt, -- Original aggregate functions. min(id) AS min_id, max(id) AS max_id, sum(id) AS sum_id, bitmap_union(to_bitmap(id)) AS bitmap_union_id, hll_union(hll_hash(id)) AS hll_union_id, percentile_union(percentile_hash(id)) AS percentile_union_id, -- Generic aggregate state functions. ds_hll_count_distinct_union(ds_hll_count_distinct_state(id)) AS hll_id, avg_union(avg_state(id)) AS avg_id, array_agg_union(array_agg_state(id)) AS array_agg_id, min_by_union(min_by_state(province, id)) AS min_by_province_id FROM t1 GROUP BY dt; -- Wait until rollup creation finishes. show alter table rollup; -- Direct queries against the aggregate function will be transparently accelerated by test_mv1. SELECT dt, min(id), max(id), sum(id), bitmap_union_count(to_bitmap(id)), -- count(distinct id) hll_union_agg(hll_hash(id)), -- approx_count_distinct(id) percentile_approx(id, 0.5), ds_hll_count_distinct(id), avg(id), array_agg(id), min_by(province, id) FROM t1 WHERE dt >= '2024-01-01' GROUP BY dt; -- Direct queries against the aggregate function and the rollup will also be transparently accelerated by test_mv1. SELECT min(id), max(id), sum(id), bitmap_union_count(to_bitmap(id)), -- count(distinct id) hll_union_agg(hll_hash(id)), -- approx_count_distinct(id) percentile_approx(id, 0.5), ds_hll_count_distinct(id), avg(id), array_agg(id), min_by(province, id) FROM t1 WHERE dt >= '2024-01-01'; DROP MATERIALIZED VIEW test_mv1; ``` ###### Generic aggregate states in asynchronous materialized views[​](#generic-aggregate-states-in-asynchronous-materialized-views "Direct link to Generic aggregate states in asynchronous materialized views") Example: ```sql -- Create an asynchronous materialized view test_mv2 to store aggregate states. CREATE MATERIALIZED VIEW test_mv2 PARTITION BY (dt) DISTRIBUTED BY RANDOM AS SELECT dt, -- Original aggregate functions. min(id) AS min_id, max(id) AS max_id, sum(id) AS sum_id, bitmap_union(to_bitmap(id)) AS bitmap_union_id, hll_union(hll_hash(id)) AS hll_union_id, percentile_union(percentile_hash(id)) AS percentile_union_id, -- Generic aggregate state functions. ds_hll_count_distinct_union(ds_hll_count_distinct_state(id)) AS hll_id, avg_union(avg_state(id)) AS avg_id, array_agg_union(array_agg_state(id)) AS array_agg_id, min_by_union(min_by_state(province, id)) AS min_by_province_id FROM t1 GROUP BY dt; -- Refresh the materialized view. REFRESH MATERIALIZED VIEW test_mv2 WITH SYNC MODE; -- Direct queries against the aggregate function will be transparently accelerated by test_mv2. SELECT dt, min(id), max(id), sum(id), bitmap_union_count(to_bitmap(id)), -- count(distinct id) hll_union_agg(hll_hash(id)), -- approx_count_distinct(id) percentile_approx(id, 0.5), ds_hll_count_distinct(id), avg(id), array_agg(id), min_by(province, id) FROM t1 WHERE dt >= '2024-01-01' GROUP BY dt; SELECT min(id), max(id), sum(id), bitmap_union_count(to_bitmap(id)), -- count(distinct id) hll_union_agg(hll_hash(id)), -- approx_count_distinct(id) percentile_approx(id, 0.5), ds_hll_count_distinct(id), avg(id), array_agg(id), min_by(province, id) FROM t1 WHERE dt >= '2024-01-01'; ``` #### Usage notes[​](#usage-notes "Direct link to Usage notes") * **Aggregate key**: * In the CREATE TABLE statement, the aggregate key must be defined before other columns. * The aggregate key can be explicitly defined using `AGGREGATE KEY`. The `AGGREGATE KEY` must include all columns except the value columns, otherwise the table fails to be created. If the aggregate key is not explicitly defined using `AGGREGATE KEY`, all columns except the value columns are considered as the aggregate key by default. * The aggregate key has uniqueness constraint. * **Value column**: Define a column as the value column by specifying an aggregate function after the column name. This column generally holds data that needs to be aggregated. * **Aggregate function**: The aggregate function used for the value column. For supported aggregate functions for the Aggregate tables, see [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md). * **Sort key** * Since v3.3.0, the sort key is decoupled from the aggregate key in the Aggregate table. The Aggregate table supports specifying the sort key using `ORDER BY` and specifying the aggregate key using `AGGREGATE KEY`. The columns in the sort key and the aggregate key need to be the same, but the order of the columns does not need to be the same. * When queries are run, sort key columns are filtered before the aggregation of multiple data versions, whereas value columns are filtered after the aggregation of multiple data versions. Therefore, we recommend that you identify the columns that are frequently used as filter conditions and define these columns as the sort key. This way, data filtering can start before the aggregation of multiple data versions to improve query performance. * When you create a table, you can only create Bitmap indexes or Bloom Filter indexes on the key columns of the table. #### What to do next[​](#what-to-do-next "Direct link to What to do next") After a table is created, you can use various data ingestion methods to load data into StarRocks. For information about the data ingestion methods that are supported by StarRocks, see [Loading options](https://docs.starrocks.io/docs/loading/loading_introduction.md). > Note: When you load data into a table that uses the Aggregate table, you can only update all columns of the table. For example, when you update the preceding `example_db.aggregate_tbl` table, you must update all its columns, which are `site_id`, `date`, `city_code`, and `pv`. --- ### Duplicate Key table The Duplicate Key table is the default model in StarRocks. If you did not specify a model when you create a table, a Duplicate Key table is created by default. When you create a Duplicate Key table, you can define a sort key for that table. If the filter conditions contain the sort key columns, StarRocks can quickly filter data from the table to accelerate queries. The Duplicate Key table is suitable for scenarios, such as analyzing logs data. It supports appending new data, but does not support modifying historical data. #### Scenarios[​](#scenarios "Direct link to Scenarios") The Duplicate Key table is suitable for the following scenarios: * Analyze raw data, such as raw logs and raw operation records. * Query data by using a variety of methods without being limited by the pre-aggregation method. * Load log data or time-series data. New data is written in append-only mode, and existing data is not updated. #### Create a table[​](#create-a-table "Direct link to Create a table") Suppose that you want to analyze the event data over a specific time range. In this example, create a table named `detail` and define `event_time` and `event_type` as sort key columns. Statement for creating the table: ```sql CREATE TABLE detail ( event_time DATETIME NOT NULL COMMENT "datetime of event", event_type INT NOT NULL COMMENT "type of event", user_id INT COMMENT "id of user", device_code INT COMMENT "device code", channel INT COMMENT "") ORDER BY (event_time, event_type); ``` #### Usage notes[​](#usage-notes "Direct link to Usage notes") * **Sort key** Since v3.3.0, the Duplicate Key table supports specifying the sort key using `ORDER BY`, which can be combination of any columns. If both `ORDER BY` and `DUPLICATE KEY` are used, `DUPLICATE KEY` does not take effect. If neither `ORDER BY` nor `DUPLICATE KEY` is used, the first three columns of the table are used as the sort key by default. * **Bucketing** * **Bucketing method**: Since v3.1.0, StarRocks supports random bucketing for Duplicate Key tables (the default bucketing method). When creating tables or adding partitions, you do not need to set the hash bucketing key (that is the `DISTRIBUTED BY HASH` clause). Before v3.1.0, StarRocks only supports hash bucketing. You need to set the hash bucketing key (that is the `DISTRIBUTED BY HASH` clause) when creating tables or adding partitions, otherwise the table fails to be created. For more information on hash bucketing keys, see [Hash Bucketing](https://docs.starrocks.io/docs/table_design/data_distribution.md#hash-bucketing). * **Bucket number**: Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). * When you create a table, you can create Bitmap indexes or Bloom Filter indexes on the all columns of the table. #### What to do next[​](#what-to-do-next "Direct link to What to do next") After a table is created, you can use various data ingestion methods to load data into StarRocks. For information about the data ingestion methods that are supported by StarRocks, see [Loading options](https://docs.starrocks.io/docs/loading/loading_introduction.md). note * When you load data into a table that uses the Duplicate Key table, you can only append data to the table. You cannot modify the existing data in the table. * If two identical records are loaded, the Duplicate Key table retains them as two records, rather than one. --- ### Primary Key table The Primary Key table uses a new storage engine designed by StarRocks. Its main advantage lies in supporting real-time data updates while ensuring efficient performance for complex ad-hoc queries. In real-time business analytics, decision-making can benefit from Primary Key tables which use the newest data to analyze results in real-time, which can mitigate data latency in data analysis. The primary key of a Primary Key table has a UNIQUE constraint and NOT NULL constraint, and is used to uniquely identify each data row. If the primary key value of a new data row is same as that of an existing data row in the table, UNIQUE constraint violation occurs. Then the new data row will replace the existing data row. info * Since v3.0, the sort key of a Primary Key table is decoupled from the table's primary key, and the sort key can be specified separately. As such, table creation flexibility is improved. * Since v3.1, StarRocks shared-data clusters support creating Primary Key tables. * Since v3.1.4, persistent indexes can be created and stored in **local disks**. * Since v3.3.2, persistent indexes can be created and stored in **object storages**. #### Scenarios[​](#scenarios "Direct link to Scenarios") The Primary Key table can support real-time data updates while ensuring efficient query performance. It is suitable for the following scenarios: * **Stream data in real time from transaction processing systems into StarRocks.** In normal cases, transaction processing systems involve a large number of update and delete operations in addition to insert operations. If you need to synchronize data from a transaction processing system to StarRocks, we recommend that you create a Primary Key table. Then, you can use tools, such as CDC Connectors for Apache Flink®, to synchronize the binary logs of the transaction processing system to StarRocks. StarRocks uses the binary logs to add, delete, and update the data in the table in real time. This simplifies data synchronization and delivers 3 to 10 times higher query performance than when a Unique Key table that adopts the Merge-On-Read strategy is used. For more information, see [Realtime synchronization from MySQL](https://docs.starrocks.io/docs/loading/Flink_cdc_load.md). * **Join multiple streams by performing [partial updates on individual columns](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md#partial-updates)**. In business scenarios such as user profiling, flat tables are preferably used to improve multi-dimensional analysis performance and simplify the analytics model that is used by data analysts. Upstream data in these scenarios may come from various apps, such as shopping apps, delivery apps, and banking apps, or from systems, such as machine learning systems that perform computations to obtain the distinct tags and properties of users. The Primary Key table is well suited in these scenarios, because it supports updates to individual columns. Each app or system can update only the columns that hold the data within its own service scope while benefiting from real-time data additions, deletions, and updates at high query performance. #### How it works[​](#how-it-works "Direct link to How it works") The Unique Key table and Aggregate table adopt the Merge-On-Read strategy. This strategy makes data writing simple and efficient, but requires merging multiple versions of data files online during data reading. Moreover, because the Merge operator exists, the predicates and indexes cannot be pushed down to the underlying data, which severely impacts query performance. However, to balance the performance of real-time updates and query, the metadata structure and the read/write mechanism in the Primary Key table differ from those in the other types of tables. The Primary Key table uses the Delete+Insert strategy. This strategy is realized by using the primary key index and the DelVector. This strategy ensures that only the latest record among the records with the same primary key value needs to be read during queries, which eliminates the need to merge multiple versions of data files. Moreover, predicates and indexes can be pushed down to the underlying data, which greatly improves query performance. The overall process of writing and reading data within the Primary Key table is as follows: * The data writing is achieved through StarRocks's internal Loadjob that includes a batch of data change operations (Insert, Update, and Delete). StarRocks loads the primary key indexes of the corresponding tablets into memory. For Delete operations, StarRocks first uses the primary key index to find the original location (data file and row number) of each data row, marking the data row as deleted in the DelVector (which stores and manages delete markers generated during data loading). For Update operations, in addition to marking the original data row as deleted in the DelVector, StarRocks also writes the latest data row to a new data file, essentially transforming the Update into a Delete+Insert (as shown in the following figure). The primary key index is also updated to record the new location (data file and row number) of the changed data row. ![pk1](/assets/images/pk1-4bbd00fd75d01c71df5095902b203b37.png) * During data reading, because historical duplicate records in various data files have already been marked as deleted during data writing, only the latest data row with the the same primary key value needs to be read. Multiple versions of data files no longer need to be read online to deduplicate data and find the latest data. When the underlying data files are scanned, filter operators and various indexes help reduce scanning overhead (as shown in the following figure). Therefore, query performance can be significantly improved. Compared to the Merge-On-Read strategy of the Unique Key table, the Delete+Insert strategy of the Primary Key table can help improve query performance by 3 to 10 times. ![pk2](/assets/images/pk2-df87fbe1c427d469e672f983be58dda3.png) More details If you want a deeper understanding of how data is written into or read from Primary Key tables, you can explore the detailed data writing and reading processes as follows: StarRocks is an analytical database that uses columnar storage. Specifically, a tablet within a table often contains multiple rowset files, and the data of each rowset file is actually stored in segment files. Segment files organize data in columnar format (similar to Parquet) and are immutable. When the data to be written is distributed to the Executor BE nodes, each Executor BE node executes a Loadjob. A Loadjob includes a batch of data changes and can be considered as a transaction with ACID properties. A Loadjob can be divided into two stages: write and commit. 1. Write stage: Data is distributed to the corresponding tablet based on partition and bucket information. When a tablet receives data, data is stored in columnar format and then a new rowset is formed. 2. Commit stage: After all data is successfully written, the FE initiates the commits to all tablets that are involved. Each commit carries a version number representing the latest version of the tablet's data. The commit process mainly includes searching and updating the primary key index, marking all the changed data as deleted, creating a DelVector based on the data marked as deleted, and generating metadata for the new version. During data reading, the metadata is used to find which rowsets need to be read based on the latest tablet version. When a segment file in the rowset is being read, its latest version of DelVector is also being checked, which can ensure that only the latest data needs to be read and avoid reading old data with the same primary key value. Furthermore, filter operators pushed down to the Scan layer can directly utilize various indexes to reduce scanning overhead. * **Tablet**: A table is divided into multiple tablets based on partition and bucket mechanisms. It is the actual physical storage unit and is distributed as replicas across different BEs. ![pk3](/assets/images/pk3-177efb54fc21a04d72c6fc05b6e66d85.png) * **Metadata**: The metadata stores the version history of the tablet and information about each version (for example, which rowsets are included). The commit phase of each Loadjob or compaction generates a new version. ![pk4](/assets/images/pk4-0924355aff0db1fba6999e3924bdb808.png) * **Primary key index**: The primary key index stores the mapping between the data rows identified by those primary key values and the locations of those data rows. It is implemented as a HashMap, where the keys represent the encoded primary key values, and the values represent the locations of data rows（including `rowset_id`, `segment_id` and `rowid`）. Normarlly, the primary key index is only used during data writing to find the rowset and row in which each data row identified by a specific primary key value resides. * **DelVector**: The DelVector stores the delete markers for each segment file (columnar file) in every rowset. * **Rowset**: The rowset is a logical concept and stores the dataset from a batch of data changes in a tablet. * **Segment**: The data in the rowset is actually segmented and stored in one or more segment files (columnar files). Each segment file contains column values and indexes information related to the columns. #### Usage[​](#usage "Direct link to Usage") ##### Create Primary Key table[​](#create-primary-key-table "Direct link to Create Primary Key table") You just need to define the primary key in the `CREATE TABLE` statement to create a Primary Key table. Example: ```sql CREATE TABLE orders1 ( order_id bigint NOT NULL, dt date NOT NULL, user_id INT NOT NULL, good_id INT NOT NULL, cnt int NOT NULL, revenue int NOT NULL ) PRIMARY KEY (order_id) DISTRIBUTED BY HASH (order_id) ; ``` info Because the Primary Key table only supports hash bucketing as the bucketing strategy, you also need to define the hash bucketing key by using `DISTRIBUTED BY HASH ()`. However, in real business scenarios, when a Primary Key table is created, additional features such as data distribution and sort key are often used to accelerate queries and manage data more efficiently. For example, the `order_id` field in the order table can uniquely identify data rows, so the `order_id` field can be used as the primary key. Since v3.0, the sort key of a Primary Key table is decoupled from the table's primary key. Therefore, you can choose columns frequently used as query filter conditions to form the sort key. For example, if you frequently query product sales performance based on the combination of two dimensions, order date and merchant, you can specify the sort key as `dt` and `merchant_id` using the `ORDER BY (dt,merchant_id)` clause. Note that if you use [data distribution strategies](https://docs.starrocks.io/docs/table_design/data_distribution.md), the Primary Key table currently requires the primary key to include partitioning and bucketing columns. For example, the data distribution strategy uses `dt` as the partitioning column and `merchant_id` as the hash bucketing column. The primary key also needs to include `dt` and `merchant_id`. In summary, the CREATE TABLE statement for the above order table can be as follows: ```sql CREATE TABLE orders2 ( order_id bigint NOT NULL, dt date NOT NULL, merchant_id int NOT NULL, user_id int NOT NULL, good_id int NOT NULL, good_name string NOT NULL, price int NOT NULL, cnt int NOT NULL, revenue int NOT NULL, state tinyint NOT NULL ) PRIMARY KEY (order_id,dt,merchant_id) PARTITION BY date_trunc('day', dt) DISTRIBUTED BY HASH (merchant_id) ORDER BY (dt,merchant_id) PROPERTIES ( "enable_persistent_index" = "true" ); ``` ##### Primary key[​](#primary-key "Direct link to Primary key") The primary key of a table is used to uniquely identify each row in that table. The one or more columns comprising the primary key are defined in the `PRIMARY KEY`, and have the UNIQUE constraint and NOT NULL constraint. Take note of the following considerations about the primary key: * In the CREATE TABLE statement, the primary key columns must be defined before other columns. * The primary key columns must include partitioning and bucketing columns. * The primary key columns support the following data types: numeric (including integers and BOOLEAN), string, and date (DATE and DATETIME). * By default, the maximum length of an encoded primary key value is 128 bytes. * The primary key cannot be modified after table creation. * For data consistency purposes, the primary key values cannot be updated. ##### Primary key index[​](#primary-key-index "Direct link to Primary key index") The primary key index is used to store the mapping between the primary key values and the locations of the data rows identified by the primary key values. Typically, the primary key indexes of relevant tablets are loaded into memory only during data loading (which involves a batch of data changes). You can consider persisting the primary key indexes after comprehensively evaluating the performance requirement for queries and updates, as well as the memory and disk. * Persistent primary key index * Fully in-memory primary key index When `enable_persistent_index` is set to `true` (default), the primary key indexes can be persisted to the disk. During loading, a small portion of the primary key indexes is loaded in memory, while the majority is stored on disk to avoid taking up too much memory. In general, query and update performance of the table with the persistent primary key indexes is nearly equivalent to that of the table with the fully in-memory primary key indexes. If the disk is an SSD, it is recommended to set it to `true`. If the disk is an HDD and the load frequency is not high, you can also set it to `true`. Since v3.1.4, Primary Key tables created in StarRocks shared-data clusters support index persistence onto local disks. And from v3.3.2 onwards, StarRocks shared-data clusters further support index persistence onto object storage. You can enable this feature by setting the table property `persistent_index_type` to `CLOUD_NATIVE`. When `enable_persistent_index` is set to `false`, the primary key indexes are not persisted to the disk, that is, the primary key indexes are fully stored in memory. During loading, the primary key indexes of tablets related to the data loaded will be loaded into memory, which may result in higher memory consumption. (If a tablet has not had data loaded for a long time, its primary key index will be released from memory.) When using the fully in-memory primary key indexes, you are advised to follow the following guidelines when designing the Primary Key to control memory usage of the primary key indexes: * The number and total length of Primary Key columns must be properly designed. We recommend that you identify columns whose data types occupy less memory and define those columns as the primary key, such as INT and BIGINT, not VARCHAR. * Before you create the table, we recommend that you estimate the memory occupied by the primary key indexes based on the data types of the primary key columns and the number of rows in the table. This way, you can prevent running out of memory. The following example explains how to calculate the memory occupied by the primary key indexes: * Suppose that the `dt` column, which is of the DATE data type and occupies 4 bytes, and the `id` column, which is of the BIGINT data type and occupies 8 bytes, are defined as the Primary Key. In this case, the Primary Key is 12 bytes in length. * Suppose that the table contains 10,000,000 rows of hot data and is stored in three replicas. * Given the preceding information, the memory occupied by the primary key indexes is 945 MB based on the following formula: `(12 + 9) x 10,000,000 x 3 x 1.5 = 945 (MB)` In the preceding formula, `9` is the immutable overhead per row, and `1.5` is the average extra overhead per hash table. The Primary Key table with the fully in-memory primary key indexes is suitable for scenarios in which the memory occupied by the Primary Key is controllable. Example: * The table contains both fast-changing data and slow-changing data. Fast-changing data is frequently updated over the most recent days, whereas slow-changing data is rarely updated. Suppose that you need to synchronize a MySQL order table to StarRocks in real time for analytics and queries. In this example, the data of the table is partitioned by day, and most updates are performed on orders that are created within the most recent days. Historical orders are no longer updated after they are completed. When you run a data load job, the primary key indexes of historical orders are not loaded into the memory. Only the primary key indexes of the recently updated orders are loaded into the memory. As shown in the following figure, the data in the table is partitioned by day, and the data in the most recent two partitions is frequently updated. ![pk5](/assets/images/pk5-43ba2246bd30f75742e99085e21feed1.png) * The table is a flat table that is composed of hundreds or thousands of columns. The Primary Key comprises only a small portion of the table data and consumes only a small amount of memory. For example, a user status or profile table consists of a large number of columns but only tens to hundreds of millions of users. In this situation, the amount of memory consumed by the primary key is controllable. As shown in the following figure, the table contains only a few rows, and the Primary Key of the table comprises only a small portion of the table. ![pk6](/assets/images/pk6-5e26799f28b507f440de5346f5a3536c.png) ##### Sort key[​](#sort-key "Direct link to Sort key") From v3.0, the Primary Key table decouples the sort key from the Primary Key. The sort key is composed of the columns defined in `ORDER BY`, and can consist of any combination of columns, as long as the data type of the columns meets the requirement of the sort key. During data loading, the data is stored after being sorted according to the sort key. The sort key is also used to build the Prefix index to accelerate queries. It is recommended to [design the sort key appropriately to form the Prefix index that can accelerate queries](https://docs.starrocks.io/docs/table_design/indexes/Prefix_index_sort_key.md#how-to-design-the-sort-key-appropriately-to-form-the-prefix-index-that-can-accelerate-queries). info * If the sort key is specified, the Prefix index is built based on the sort key. If no sorti key is specified,the Prefix index are built based on the Primary Key. * After table creation, you can use `ALTER TABLE ... ORDER BY ...` to change the sort key. Deleting the sort key is not supported, and modifying the data types of sort columns is not supported. #### What's more[​](#whats-more "Direct link to What's more") * To load data into the table created, you can refer to [Loading overview](https://docs.starrocks.io/docs/loading/loading_introduction.md) to choose an appropriate load options. * If you need to change data in the Primary Key table, you can refer to [change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md) or use DML ([INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md), [UPDATE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/UPDATE.md), and [DELETE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DELETE.md)). * If you want to further accelerate queries, you can refer to [Query Acceleration](https://docs.starrocks.io/docs/using_starrocks/async_mv/Materialized_view.md). * If you need to modify the table schema, you can refer to [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/ALTER_RESOURCE.md). * An [AUTO\_INCREMENT](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns.md) column can be used as the Primary Key. --- ### Capabilities of different table types #### Key columns and sort key[​](#key-columns-and-sort-key "Direct link to Key columns and sort key") | | **Primary Key** **table** | **Duplicate Key table** | **Aggregate table** | **Unique Key table** | | ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Key columns and UNIQUE constraints** | The primary key has the UNIQUE constraint and NOT NULL constraint. | The duplicate key does not have the UNIQUE constraint. | The aggregate key has the UNIQUE constraint. | The unique key has the UNIQUE constraint. | | **Logical relationships between key columns and data changes**\*\*\*\* | If the primary key value of a new data row is the same as that of an existing data row in the table, unique constraint violation occurs. Then the new data row will replace the existing data row.
Compared to the Unique Key table, the Primary Key table has an enhanced underlying storage engine and therefore can replace the Unique Key table. | The duplicate key does not have the UNIQUE constraint. So, if the duplicate Key value of a new data row is the same as that of an existing data row in the table, both the new and old data rows are retained in the table. | If the aggregate Key value of a new data row is the same as that of an existing data row in the table, the new and old data rows will be aggregated based on the aggregate key and the aggregate functions of value columns. | If the unique key value of a new data row is the same as that of an existing data row in the table, the new data row will replace the existing data row.
The Unique Key table can be seen as an Aggregate table whose aggregate function is replace. | | **Relationship between key columns and sort key** | Since v3.0.0, the sort key is decoupled from the primary key in the Primary Key table. | Since v3.3.0, the Duplicate Key tables support specifying sort keys using `ORDER BY`. If both `ORDER BY` and `DUPLICATE KEY` are used, then `ORDER BY` takes precedence. | Since v3.3.0, the sort key is decoupled from the aggregate key in the Aggregate table. The Aggregate table supports specifying the sort key using `ORDER BY` and specifying the aggregate key using `AGGREGATE KEY`. The columns in the sort key and the aggregate key need to be the same, but the order of the columns does not need to be the same. | Since v3.3.0, the sort key is decoupled from the unique key in the Unique Key table. The Unique Key table supports specifying the sort key using `ORDER BY` and specifying the unique key using `UNIQUE KEY`. The columns in the sort key and the unique key need to be the same, but the order of the columns does not need to be the same. | | **Data types supported by key columns and sort key** | Numeric (including integers and BOOLEAN), string, and date (DATE and DATETIME). | Numeric (including integers, BOOLEAN, and DECIMAL), string, and date (DATE and DATETIME). | | | | **Relationship between key columns and partitioning/bucketing columns** | Partitioning columns and bucketing columns must be in the primary key. | None | Partitioning columns and bucketing columns must be in the aggregate key. | Partitioning columns and bucketing columns must be in the unique key. | #### Data types of key and value columns[​](#data-types-of-key-and-value-columns "Direct link to Data types of key and value columns") The key columns support the following data types: numeric (including integers, BOOLEAN, and DECIMAL), string, and date (DATE and DATETIME). note The key columns of a Primary Key table do not support the DECIMAL data type. On the other hand, the value columns support basic data types, including numeric, string, and date (DATE and DATETIME). The support for BITMAP, HLL, and semi-structured types varies for value columns of different types of tables, as detailed below: | | **Primary Key** **table** | **Duplicate Key table** | **Aggregate table** | **Unique Key table** | | ----------------------------------------------------- | ------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------- | -------------------- | | **BITMAP** | Supported | Not supported | Supported. The aggregate functions must be bitmap\_union, replace, or replace\_if\_not\_null. | Supported | | **HLL** | Supported | Not supported | Supported. The aggregate functions must be hll\_union, replace, or replace\_if\_not\_null. | Supported | | **PERCENTILE** | Supported | Not supported | Supported. The aggregate functions must be percentile\_union, replace, or replace\_if\_not\_null. | Supported | | **semi-structured data types：JSON/ARRAY/MAP/STRUCT** | Supported | Supported | Supported. The aggregate functions must be replace or replace\_if\_not\_null. | Supported | #### Data change[​](#data-change "Direct link to Data change") | | **Primary Key** **Table** | **Duplicate Key table** | **Aggregate table** | **Unique Key table** | | --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | **INSERT through data loading** | Supported. [Configure `__op=0` in the load job to achieve INSERT](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md).
In the internal implementation, StarRocks considers both INSERT and UPDATE operations as UPSERT operations. | Supported | Supported (Data rows with the same aggregate Key value will be aggregated.) | Supported (Data rows with the same unique key value will be updated.) | | **UPDATE through data loading** | | Not supported | Supported (This can be achieved by using replace as the aggregate function.) | Supported (The Unique Key table itself can be considered as an Aggregate table using the replace aggregate function.) | | **DELETE through data loading** | Supported. [Configure `__op=1` in the load job to achieve DELETE](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md). | Not supported | | | | **Integrity of data column values to be loaded** | By default, all column values need to be loaded. However, if partial column update (`partial_update`) is enabled, or a column has a default value, not all column values need to be loaded. | By default, all column values need to be loaded. However, if a column has a default value, not all column values need to be loaded. | By default, all column values need to be loaded. However, Aggregate tables can achieve partial column update by specifying the aggregate functions for value columns as REPLACE\_IF\_NOT\_NULL. For details, see [aggr\_type](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md#column_definition). Also, if a column has a default value, not all column values need to be loaded. | By default, all column values need to be loaded. However, if a column has a default value, not all column values need to be loaded. | | **[DML INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md)** | Supported | | | | | **[DML UPDATE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/UPDATE.md)** | - Key columns as filter conditions: Supported
- Value columns as filter conditions: Supported | Not supported | | | | **[DML DELETE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DELETE.md)** | - Key columns as filter conditions: Supported
- Value columns as filter conditions: Supported | - Key columns as filter conditions: Supported
- Value columns as filter conditions: SupportedNote that only simple filter conditions based on key or value columns themselves are supported, such as =, <, >. Complex filter conditions, such as functions or subqueries, are not supported. | - Key columns as filter conditions: Supported. Note that only simple filter conditions based on key columns themselves are supported, such as =, <, >. Complex filter conditions, such as functions or subqueries, are not supported.
- Value columns as filter conditions: Not supported. | | #### Compatibility with other features[​](#compatibility-with-other-features "Direct link to Compatibility with other features") | | | **Primary Key table** | **Duplicate Key table** | **Aggregate table** | **Unique Key table** | | ----------------------------------- | --------------------------------------------- | --------------------------- | --------------------------- | ------------------- | -------------------- | | **Bitmap index/bloom filter index** | **Build indexes on key columns** | Supported | | | | | | **Build indexes on value columns** | Supported | Supported | Not supported | Not supported | | **Partitioning/bucketing** | **Expression partitioning/list partitioning** | Supported | | | | | | **Random bucketing** | Not supported | Supported from v3.1 onwards | Not supported | Not supported | | **Materialized views** | **Asynchronous materialized views** | Supported | | | | | | **Synchronous materialized views** | Not supported | Supported | Supported | Supported | | **Other features** | **CTAS** | Supported | Supported | Not supported | Not supported | | | **Backup & restore** | Supported from v2.5 onwards | Supported | | | --- ### Unique Key table You need to define a unique key at table creation. When multiple rows of data have the same unique key, the values in the value columns are replaced. During queries, the latest data from a group of data with the same unique key is returned. Additionally, you can define the sort key separately. If the filter conditions in queries include the sort key, StarRocks can quickly filter the data, improving query efficiency. The Unique Key table can support real-time and frequent data updates. However, it is gradually replaced by the [Primary Key table](https://docs.starrocks.io/docs/table_design/table_types/primary_key_table.md). #### Scenarios[​](#scenarios "Direct link to Scenarios") The Unique Key table is suitable for business scenarios in which data needs to be frequently updated in real time. For example, in e-commerce scenarios, hundreds of millions of orders can be placed per day, and the statuses of the orders frequently change. #### Principle[​](#principle "Direct link to Principle") The Unique Key table can be considered a special Aggregate table in which the REPLACE aggregate function is specified for value columns to return the most recent record among a group of records that have the same unique key. When you load data into a Unique Key table, the data is split into multiple batches. Each batch is assigned a version number. Therefore, records with the same unique key may be included in multiple versions. Data in the most recent version (that is, the record with the largest version number) is returned for queries. As shown in the following table, `ID` is the unique key, `value` is a value column, and `_version` holds the data version numbers generated within StarRocks. In this example, the record with an `ID` of 1 is loaded by two batches whose version numbers are `1` and `2`, and the record with an `ID` of `2` is loaded by three batches whose version numbers are `3`, `4`, and `5`. | ID | value | \_version | | -- | ----- | --------- | | 1 | 100 | 1 | | 1 | 101 | 2 | | 2 | 100 | 3 | | 2 | 101 | 4 | | 2 | 102 | 5 | When you query the record with an `ID` of `1`, the most recent record with the largest version number, which is `2` in this case, is returned. When you query the record with an `ID` of `2`, the most recent record with the largest version number, which is `5` in this case, is returned. The following table shows the records returned by the two queries: | ID | value | | -- | ----- | | 1 | 101 | | 2 | 102 | #### Create a table[​](#create-a-table "Direct link to Create a table") In e-commerce scenarios, you often need to collect and analyze the statuses of orders by date. In this example, create a table named `orders` to hold the orders, define `create_time` and `order_id`, which are frequently used as conditions to filter the orders, as unique key columns, and define the other two columns, `order_state` and `total_price`, as value columns. This way, the orders can be updated in real time as their statuses change, and can be quickly filtered to accelerate queries. The statement for creating the table is as follows: ```sql CREATE TABLE orders ( create_time DATE NOT NULL COMMENT "create time of an order", order_id BIGINT NOT NULL COMMENT "id of an order", order_state INT COMMENT "state of an order", total_price BIGINT COMMENT "price of an order" ) UNIQUE KEY(create_time, order_id) DISTRIBUTED BY HASH(order_id); ``` > **NOTICE** > > * When you create a table, you must specify the bucketing column by using the `DISTRIBUTED BY HASH` clause. For detailed information, see [bucketing](https://docs.starrocks.io/docs/table_design/data_distribution.md#bucketing). > * Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). #### Usage notes[​](#usage-notes "Direct link to Usage notes") * **Unique Key**: * In the CREATE TABLE statement, the unique key must be defined before other columns. * The unique key needs to be explicitly defined using `UNIQUE KEY`. * The unique key has uniqueness constraint. * **Sort Key**: * Since v3.3.0, the sort key is decoupled from the unique key in the Unique Key table. The Unique Key table supports specifying the sort key using `ORDER BY` and specifying the unique key using `UNIQUE KEY`. The columns in the sort key and the unique key need to be the same, but the order of the columns does not need to be the same. * During queries, data can be filtered based on the sort keys before aggregation. However, data can be filtered based on the value columns after multi-version aggregation. Therefore, it is recommended to use frequently filtered fields as sort keys to filter data before aggregation and thereby improve query performance. * When you create a table, you can only create Bitmap indexes or Bloom Filter indexes on the key columns of the table. #### What to do next[​](#what-to-do-next "Direct link to What to do next") After a table is created, you can use various data ingestion methods to load data into StarRocks. For information about the data ingestion methods that are supported by StarRocks, see [Loading options](https://docs.starrocks.io/docs/loading/loading_introduction.md). note * When you load data into a table that uses the Unique Key table, you can only update all columns of the table. For example, when you update the preceding `orders` table, you must update all its columns, which are `create_time`, `order_id`, `order_state`, and `total_price`. * When you query data from a table that uses the Unique Key table, StarRocks needs to aggregate records of multiple data versions. In this situation, a large number of data versions decreases query performance. Therefore, we recommend that you specify a proper frequency at which data is loaded into the table to meet meet your requirements for real-time data analytics while preventing a large number of data versions. If you require minute-level data, you can specify a loading frequency of 1 minute instead of a loading frequency of 1 second. --- ## Unloading ### Interact with StarRocks via Arrow Flight SQL From v3.5.1 onwards, StarRocks supports connections via Apache Arrow Flight SQL protocol. #### Overview[​](#overview "Direct link to Overview") With Arrow Flight SQL protocol, you can execute normal DDL, DML, DQL statements, and use Python code or Java code to read large-scale data via Arrow Flight SQL ADBC or JDBC driver. This solution establishes a fully columnar data transfer pipeline from the StarRocks columnar execution engine to the client, eliminating the frequent row-column conversions and serialization overhead typically seen in traditional JDBC and ODBC interfaces. This enables StarRocks to transfer data with zero-copy, low latency, and high throughput. ##### Scenarios[​](#scenarios "Direct link to Scenarios") The Arrow Flight SQL integration makes StarRocks especially well-suited for: * Data science workflows, where tools like Pandas and Apache Arrow expect columnar data. * Data lake analytics, requiring high-throughput, low-latency access to massive datasets. * Machine learning, where fast iteration and processing speed are critical. * Real-time analytics platforms, that must deliver data with minimal delays. With Arrow Flight SQL, you can benefit from: * End-to-end columnar data transfer, eliminating costly conversions between columnar and row-based formats. * Zero-copy data movement, reducing CPU and memory overhead. * Low latency and extremely high throughput, accelerating analysis and responsiveness. ##### Technical approach[​](#technical-approach "Direct link to Technical approach") Traditionally, StarRocks organizes query results in a columnar Block structure internally. However, when using JDBC, ODBC, or MySQL protocols, data must be: 1. Serialized into row-based bytes on the server. 2. Transferred over the network. 3. Deserialized back into a target structure (often requiring re-conversion to columnar formats). This three-step process creates: * High serialization/deserialization overhead. * Complex data transformations. * Latency that grows with data volume. The integration with Arrow Flight SQL solves these issues by: * Retaining columnar format end-to-end, from the StarRocks execution engine directly to the client. * Leveraging Apache Arrow’s in-memory columnar representation, which is optimized for analytics workloads. * Using Arrow Flight’s protocol for high-speed transport, enabling efficient streaming without intermediate conversions. ![Arrow Flight](/assets/images/arrow_flight-49856c2356fc2c6332f48c8061f2e8db.png) This design provides true zero-copy transmission, which is both faster and more resource-efficient than traditional methods. Additionally, StarRocks offers a universal JDBC driver for Arrow Flight SQL, so applications can adopt this high-performance transfer path without sacrificing JDBC compatibility or interoperability with other Arrow Flight–enabled systems. For deployments where BE nodes are not directly accessible to clients (such as in private networks or Kubernetes clusters), StarRocks provides an Arrow Flight proxy feature. When enabled, the FE can act as a proxy to route Arrow data from BE nodes to clients, maintaining the columnar transfer benefits while accommodating network topology constraints. This proxy mode incurs a small performance overhead but enables Arrow Flight SQL access in environments where direct BE connectivity is not available. ##### Performance Comparison[​](#performance-comparison "Direct link to Performance Comparison") Comprehensive tests demonstrate significant improvements in data retrieval speed. Across various data types (integer, float, string, boolean, and mixed columns), Arrow Flight SQL consistently outperformed traditional PyMySQL and Pandas `read_sql` interfaces. Key results include: * For reading 10 million integer rows, execution time dropped from ~35 seconds to 0.4 seconds (~85× faster). * For a mixed-column table, performance improvements reached 160× acceleration. * Even in less complex queries (e.g., single string columns), performance gains exceeded 12×. On average, Arrow Flight SQL achieved: * 20× to 160× faster transfer times, depending on query complexity and data type. * A clear reduction in CPU and memory usage due to the elimination of redundant serialization steps. These performance gains translate directly into faster dashboards, more responsive data science workflows, and the ability to analyze much larger datasets in real time. For a detailed breakdown of how to reach these numbers in your client code — JDBC accessor methods, raw `VectorSchemaRoot` consumption, Parquet writers — and the measured speedup of each tuning step over MySQL JDBC, see [Arrow Flight SQL Best Practices](https://docs.starrocks.io/docs/unloading/arrow_flight_best_practices.md). #### Usage[​](#usage "Direct link to Usage") Follow these steps to connect to and interact with StarRocks using Python ADBC Driver via Arrow Flight SQL protocol. Refer to [Appendix](#appendix) for the complete code example. note Python 3.9 or later is a prerequisite. ##### Step 1. Install libraries[​](#step-1-install-libraries "Direct link to Step 1. Install libraries") Use `pip` to install `adbc_driver_manager` and `adbc_driver_flightsql` from PyPI: ```bash pip install adbc_driver_manager pip install adbc_driver_flightsql ``` Import the following modules or libraries into your code: * Required libraries: ```python import adbc_driver_manager import adbc_driver_flightsql.dbapi as flight_sql ``` * Optional modules for better usability and debugging: ```python import pandas as pd # Optional: for better result display using DataFrame import traceback # Optional: for detailed error traceback during SQL execution import time # Optional: for measuring SQL execution time ``` ##### Step 2. Connect to StarRocks[​](#step-2-connect-to-starrocks "Direct link to Step 2. Connect to StarRocks") note * If you want to start the FE service using command line, you can use the either of the following ways: * Specify the environment variable `JAVA_TOOL_OPTIONS`. ```bash export JAVA_TOOL_OPTIONS="--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED" ``` * Specify the FE configuration item `JAVA_OPTS` in **fe.conf**. This way, you can append other `JAVA_OPTS` values. ```bash JAVA_OPTS="--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED ..." ``` * If you want to run the service in IntelliJ IDEA, you must add the following option to `Build and run` in `Run/Debug Configurations`: ```bash --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED ``` ###### Configure StarRocks[​](#configure-starrocks "Direct link to Configure StarRocks") Before connecting to StarRocks via Arrow Flight SQL, you must first configure the FE and BE nodes to ensure that the Arrow Flight SQL service is enabled and listening on the specified ports. In both FE configuration file **fe.conf** and BE configuration file **be.conf**, set `arrow_flight_port` to an available port. After modifying the configuration files, restart FE and BE services to allow the modification to take effect. note You must set different `arrow_flight_port` for FE and BE. Example: ```properties // fe.conf arrow_flight_port = 9408 // be.conf arrow_flight_port = 9419 ``` ###### Configure Arrow Flight Proxy (Optional)[​](#configure-arrow-flight-proxy-optional "Direct link to Configure Arrow Flight Proxy (Optional)") If your BE nodes are not directly accessible from client applications (for example, when deployed in private networks or Kubernetes environments), you can enable the Arrow Flight proxy feature on the FE to route data from BE nodes through the FE. The proxy feature is controlled by two global variables: * `arrow_flight_proxy_enabled`: Controls whether proxy mode is enabled. Default is `true`. When enabled, there is a slight performance overhead. * `arrow_flight_proxy`: Specifies the proxy hostname. If empty (default), the current FE node acts as the proxy. You can set this to a specific hostname if using a different proxy endpoint. To configure these variables globally for all sessions: ```sql -- Enable or disable proxy mode (enabled by default) SET GLOBAL arrow_flight_proxy_enabled = true; -- Set a specific proxy hostname (optional, defaults to current FE) SET GLOBAL arrow_flight_proxy = 'your-proxy-hostname:Port'; ``` note * The proxy feature is enabled by default, which may result in 8-10% lower throughput compared to direct BE connections. If your clients have direct network access to BE nodes, or you have limited memory resources on the FE side, you can disable the proxy to achieve optimal performance: `SET GLOBAL arrow_flight_proxy_enabled = false;`. * When `arrow_flight_proxy` is empty, tickets will automatically route through the FE node that the client initially connected to. * **Important**: The `arrow_flight_proxy` and `arrow_flight_proxy_enabled` settings should be configured globally using `SET GLOBAL`. Session-level settings are not supported. * **Session restart required**: Changing the proxy settings only affects new sessions. Existing Arrow Flight SQL sessions will continue using their original settings until they reconnect. ###### Establish connection[​](#establish-connection "Direct link to Establish connection") On the client side, create an Arrow Flight SQL client using the following information: * Host address of the StarRocks FE * Port that Arrow Flight used for listening on the StarRocks FE * Username and password of the StarRocks user that has the necessary privileges Example: ```python FE_HOST = "127.0.0.1" FE_PORT = 9408 conn = flight_sql.connect( uri=f"grpc://{FE_HOST}:{FE_PORT}", db_kwargs={ adbc_driver_manager.DatabaseOptions.USERNAME.value: "root", adbc_driver_manager.DatabaseOptions.PASSWORD.value: "", } ) cursor = conn.cursor() ``` After the connection is established, you can interact with StarRocks by executing SQL statements through the returned Cursor. ##### Step 3. (Optional) Predefine utility functions[​](#step-3-optional-predefine-utility-functions "Direct link to Step 3. (Optional) Predefine utility functions") These functions are used to format the output, standardize the format, and simplify debugging. You can optionally define them in your code for testing. ```python # ============================================================================= # Utility functions for better output formatting and SQL execution # ============================================================================= # Print a section header def print_header(title: str): """ Print a section header for better readability. """ print("\n" + "=" * 80) print(f"🟢 {title}") print("=" * 80) # Print the SQL statement being executed def print_sql(sql: str): """ Print the SQL statement before execution. """ print(f"\n🟡 SQL:\n{sql.strip()}") # Print the result DataFrame def print_result(df: pd.DataFrame): """ Print the result DataFrame in a readable format. """ if df.empty: print("\n🟢 Result: (no rows returned)\n") else: print("\n🟢 Result:\n") print(df.to_string(index=False)) # Print the error traceback def print_error(e: Exception): """ Print the error traceback if SQL execution fails. """ print("\n🔴 Error occurred:") traceback.print_exc() # Execute a SQL statement and print the result def execute(sql: str): """ Execute a SQL statement and print the result and execution time. """ print_sql(sql) try: start = time.time() # Optional: start time for execution time measurement cursor.execute(sql) result = cursor.fetchallarrow() # Arrow Table df = result.to_pandas() # Optional: convert to DataFrame for better display print_result(df) print(f"\n⏱️ Execution time: {time.time() - start:.3f} seconds") except Exception as e: print_error(e) ``` ##### Step 4. Interact with StarRocks[​](#step-4-interact-with-starrocks "Direct link to Step 4. Interact with StarRocks") This section will guide you through some basic operations, such as creating a table, loading data, checking table metadata, setting variables, and running queries. note The examples of output listed below are implemented based on the optional modules and utility functions described in the preceding steps. 1. Create a database and a table where the data will be loaded, and check the table schema. ```python # Step 1: Drop and create database print_header("Step 1: Drop and Create Database") execute("DROP DATABASE IF EXISTS sr_arrow_flight_sql FORCE;") execute("SHOW DATABASES;") execute("CREATE DATABASE sr_arrow_flight_sql;") execute("SHOW DATABASES;") execute("USE sr_arrow_flight_sql;") # Step 2: Create table print_header("Step 2: Create Table") execute(""" CREATE TABLE sr_arrow_flight_sql_test ( k0 INT, k1 DOUBLE, k2 VARCHAR(32) NULL DEFAULT "" COMMENT "", k3 DECIMAL(27,9) DEFAULT "0", k4 BIGINT NULL DEFAULT '10', k5 DATE ) DISTRIBUTED BY HASH(k5) BUCKETS 5 PROPERTIES("replication_num" = "1"); """) execute("SHOW CREATE TABLE sr_arrow_flight_sql_test;") ``` Example output: ```sql ================================================================================ 🟢 Step 1: Drop and Create Database ================================================================================ 🟡 SQL: DROP DATABASE IF EXISTS sr_arrow_flight_sql FORCE; /Users/starrocks/test/venv/lib/python3.9/site-packages/adbc_driver_manager/dbapi.py:307: Warning: Cannot disable autocommit; conn will not be DB-API 2.0 compliant warnings.warn( 🟢 Result: StatusResult 0 ⏱️ Execution time: 0.025 seconds 🟡 SQL: SHOW DATABASES; 🟢 Result: Database _statistics_ hits information_schema sys ⏱️ Execution time: 0.014 seconds 🟡 SQL: CREATE DATABASE sr_arrow_flight_sql; 🟢 Result: StatusResult 0 ⏱️ Execution time: 0.012 seconds 🟡 SQL: SHOW DATABASES; 🟢 Result: Database _statistics_ hits information_schema sr_arrow_flight_sql sys ⏱️ Execution time: 0.005 seconds 🟡 SQL: USE sr_arrow_flight_sql; 🟢 Result: StatusResult 0 ⏱️ Execution time: 0.006 seconds ================================================================================ 🟢 Step 2: Create Table ================================================================================ 🟡 SQL: CREATE TABLE sr_arrow_flight_sql_test ( k0 INT, k1 DOUBLE, k2 VARCHAR(32) NULL DEFAULT "" COMMENT "", k3 DECIMAL(27,9) DEFAULT "0", k4 BIGINT NULL DEFAULT '10', k5 DATE ) DISTRIBUTED BY HASH(k5) BUCKETS 5 PROPERTIES("replication_num" = "1"); 🟢 Result: StatusResult 0 ⏱️ Execution time: 0.021 seconds 🟡 SQL: SHOW CREATE TABLE sr_arrow_flight_sql_test; 🟢 Result: Table Create Table sr_arrow_flight_sql_test CREATE TABLE `sr_arrow_flight_sql_test` (\n `k0` int(11) NULL COMMENT "",\n `k1` double NULL COMMENT "",\n `k2` varchar(32) NULL DEFAULT "" COMMENT "",\n `k3` decimal(27, 9) NULL DEFAULT "0" COMMENT "",\n `k4` bigint(20) NULL DEFAULT "10" COMMENT "",\n `k5` date NULL COMMENT ""\n) ENGINE=OLAP \nDUPLICATE KEY(`k0`)\nDISTRIBUTED BY HASH(`k5`) BUCKETS 5 \nPROPERTIES (\n"compression" = "LZ4",\n"fast_schema_evolution" = "true",\n"replicated_storage" = "true",\n"replication_num" = "1"\n); ⏱️ Execution time: 0.005 seconds ``` 2. Insert data, run some queries, and set variables. ```python # Step 3: Insert data print_header("Step 3: Insert Data") execute(""" INSERT INTO sr_arrow_flight_sql_test VALUES (0, 0.1, "ID", 0.0001, 1111111111, '2025-04-21'), (1, 0.20, "ID_1", 1.00000001, 0, '2025-04-21'), (2, 3.4, "ID_1", 3.1, 123456, '2025-04-22'), (3, 4, "ID", 4, 4, '2025-04-22'), (4, 122345.54321, "ID", 122345.54321, 5, '2025-04-22'); """) # Step 4: Query data print_header("Step 4: Query Data") execute("SELECT * FROM sr_arrow_flight_sql_test ORDER BY k0;") # Step 5: Session variables print_header("Step 5: Session Variables") execute("SHOW VARIABLES LIKE '%query_mem_limit%';") execute("SET query_mem_limit = 2147483648;") execute("SHOW VARIABLES LIKE '%query_mem_limit%';") # Step 6: Aggregation query print_header("Step 6: Aggregation Query") execute(""" SELECT k5, SUM(k1) AS total_k1, COUNT(1) AS row_count, AVG(k3) AS avg_k3 FROM sr_arrow_flight_sql_test GROUP BY k5 ORDER BY k5; """) ``` Example output: ```sql ================================================================================ 🟢 Step 3: Insert Data ================================================================================ 🟡 SQL: INSERT INTO sr_arrow_flight_sql_test VALUES (0, 0.1, "ID", 0.0001, 1111111111, '2025-04-21'), (1, 0.20, "ID_1", 1.00000001, 0, '2025-04-21'), (2, 3.4, "ID_1", 3.1, 123456, '2025-04-22'), (3, 4, "ID", 4, 4, '2025-04-22'), (4, 122345.54321, "ID", 122345.54321, 5, '2025-04-22'); 🟢 Result: StatusResult 0 ⏱️ Execution time: 0.149 seconds ================================================================================ 🟢 Step 4: Query Data ================================================================================ 🟡 SQL: SELECT * FROM sr_arrow_flight_sql_test ORDER BY k0; 🟢 Result: 0 0.10000 ID 0.000100000 1111111111 2025-04-21 1 0.20000 ID_1 1.000000010 0 2025-04-21 2 3.40000 ID_1 3.100000000 123456 2025-04-22 3 4.00000 ID 4.000000000 4 2025-04-22 4 122345.54321 ID 122345.543210000 5 2025-04-22 ⏱️ Execution time: 0.019 seconds ================================================================================ 🟢 Step 5: Session Variables ================================================================================ 🟡 SQL: SHOW VARIABLES LIKE '%query_mem_limit%'; 🟢 Result: Variable_name Value query_mem_limit 0 ⏱️ Execution time: 0.005 seconds 🟡 SQL: SET query_mem_limit = 2147483648; 🟢 Result: StatusResult 0 ⏱️ Execution time: 0.007 seconds 🟡 SQL: SHOW VARIABLES LIKE '%query_mem_limit%'; 🟢 Result: Variable_name Value query_mem_limit 2147483648 ⏱️ Execution time: 0.005 seconds ================================================================================ 🟢 Step 6: Aggregation Query ================================================================================ 🟡 SQL: SELECT k5, SUM(k1) AS total_k1, COUNT(1) AS row_count, AVG(k3) AS avg_k3 FROM sr_arrow_flight_sql_test GROUP BY k5 ORDER BY k5; 🟢 Result: 2025-04-21 0.30000 2 0.500050005000 2025-04-22 122352.94321 3 40784.214403333333 ⏱️ Execution time: 0.014 second ``` ##### Step 5. Close connection[​](#step-5-close-connection "Direct link to Step 5. Close connection") Include the following section in your code to close the connection. ```python # Step 7: Close print_header("Step 7: Close Connection") cursor.close() conn.close() print("✅ Test completed successfully.") ``` Example output: ```python ================================================================================ 🟢 Step 7: Close Connection ================================================================================ ✅ Test completed successfully. Process finished with exit code 0 ``` #### Use cases of large-scale data transfer[​](#use-cases-of-large-scale-data-transfer "Direct link to Use cases of large-scale data transfer") ##### Python[​](#python "Direct link to Python") After connecting to StarRocks (with Arrow Flight SQL support) via the ADBC Driver in Python, you can use various ADBC APIs to load the Clickbench dataset from StarRocks into Python. Code example: ```python #!/usr/bin/env python # -*- coding: utf-8 -*- import adbc_driver_manager import adbc_driver_flightsql.dbapi as flight_sql from datetime import datetime # ---------------------------------------- # StarRocks Flight SQL Connection Settings # ---------------------------------------- # Replace the URI and credentials as needed my_uri = "grpc://127.0.0.1:9408" # Default Flight SQL port for StarRocks my_db_kwargs = { adbc_driver_manager.DatabaseOptions.USERNAME.value: "root", adbc_driver_manager.DatabaseOptions.PASSWORD.value: "", } # ---------------------------------------- # SQL Query (ClickBench: hits table) # ---------------------------------------- # Replace with the actual table and dataset as needed sql = "SELECT * FROM clickbench.hits LIMIT 1000000;" # Read 1 million rows # ---------------------------------------- # Method 1: fetchallarrow + to_pandas # ---------------------------------------- def test_fetchallarrow(): conn = flight_sql.connect(uri=my_uri, db_kwargs=my_db_kwargs) cursor = conn.cursor() start = datetime.now() cursor.execute(sql) arrow_table = cursor.fetchallarrow() df = arrow_table.to_pandas() duration = datetime.now() - start print("\n[Method 1] fetchallarrow + to_pandas") print(f"Time taken: {duration}, Arrow table size: {arrow_table.nbytes / 1024 / 1024:.2f} MB, Rows: {len(df)}") print(df.info(memory_usage='deep')) # ---------------------------------------- # Method 2: fetch_df (recommended) # ---------------------------------------- def test_fetch_df(): conn = flight_sql.connect(uri=my_uri, db_kwargs=my_db_kwargs) cursor = conn.cursor() start = datetime.now() cursor.execute(sql) df = cursor.fetch_df() duration = datetime.now() - start print("\n[Method 2] fetch_df (recommended)") print(f"Time taken: {duration}, Rows: {len(df)}") print(df.info(memory_usage='deep')) # ---------------------------------------- # Method 3: adbc_execute_partitions (for parallel read) # ---------------------------------------- def test_execute_partitions(): conn = flight_sql.connect(uri=my_uri, db_kwargs=my_db_kwargs) cursor = conn.cursor() start = datetime.now() partitions, schema = cursor.adbc_execute_partitions(sql) # Read the first partition (for demo) cursor.adbc_read_partition(partitions[0]) arrow_table = cursor.fetchallarrow() df = arrow_table.to_pandas() duration = datetime.now() - start print("\n[Method 3] adbc_execute_partitions (parallel read)") print(f"Time taken: {duration}, Partitions: {len(partitions)}, Rows: {len(df)}") print(df.info(memory_usage='deep')) # ---------------------------------------- # Run All Tests # ---------------------------------------- if __name__ == "__main__": test_fetchallarrow() test_fetch_df() test_execute_partitions() ``` The results indicate that loading 1 million rows of Clickbench dataset (105 columns, 780 MB) from StarRocks took only 3 seconds. ```python [Method 1] fetchallarrow + to_pandas Time taken: 0:00:03.219575, Arrow table size: 717.42 MB, Rows: 1000000 RangeIndex: 1000000 entries, 0 to 999999 Columns: 105 entries, CounterID to CLID dtypes: int16(48), int32(19), int64(6), object(32) memory usage: 2.4 GB [Method 2] fetch_df (recommended) Time taken: 0:00:02.358840, Rows: 1000000 RangeIndex: 1000000 entries, 0 to 999999 Columns: 105 entries, CounterID to CLID dtypes: int16(48), int32(19), int64(6), object(32) memory usage: 2.4 GB [Method 3] adbc_execute_partitions (parallel read) Time taken: 0:00:02.231144, Partitions: 1, Rows: 1000000 RangeIndex: 1000000 entries, 0 to 999999 Columns: 105 entries, CounterID to CLID dtypes: int16(48), int32(19), int64(6), object(32) memory usage: 2.4 GB ``` ##### Arrow Flight SQL JDBC Driver[​](#arrow-flight-sql-jdbc-driver "Direct link to Arrow Flight SQL JDBC Driver") The Arrow Flight SQL protocol provides an open-source JDBC driver that is compatible with the standard JDBC interface. You can easily integrate it into various BI tools (such as Tableau, Power BI, DBeaver, etc.) to access the StarRocks database, just as you would with a traditional JDBC driver. A significant advantage of this driver is its support for high-speed data transfer based on Apache Arrow, which greatly improves the efficiency of query and data transmission. The usage is almost identical to that of a traditional MySQL JDBC driver. You only need to replace `jdbc:mysql` with `jdbc:arrow-flight-sql` in the connection URL to seamlessly switch. The query results are still returned in the standard `ResultSet` format, ensuring compatibility with existing JDBC processing logic. note Note that if you are using Java 9 or later, you must add `--add-opens=java.base/java.nio=ALL-UNNAMED` to your Java code to expose the internal structure of the JDK. Otherwise, you may encounter certain errors. * If you want to start the FE service using command line, you can use the either of the following ways: * Specify the environment variable `JAVA_TOOL_OPTIONS`. ```bash export JAVA_TOOL_OPTIONS="--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED" ``` * Specify the FE configuration item `JAVA_OPTS` in **fe.conf**. This way, you can append other `JAVA_OPTS` values. ```bash JAVA_OPTS="--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED ..." ``` * If you want to debug in IntelliJ IDEA, you must add the following option to `Build and run` in `Run/Debug Configurations`: ```bash --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED ``` ![Arrow Flight Example](/assets/images/arrow_flight_example-fcb0c6b6de54aca0ac39cd2a188221c0.png) **Click here to view the POM dependencies** ```xml 0.15.0 org.apache.arrow.adbc adbc-driver-jdbc ${adbc.version} org.apache.arrow.adbc adbc-core ${adbc.version} org.apache.arrow.adbc adbc-driver-manager ${adbc.version} org.apache.arrow.adbc adbc-sql ${adbc.version} org.apache.arrow.adbc adbc-driver-flight-sql ${adbc.version} ``` Code example: ```java import java.sql.Connection; import java.sql.DriverManager; import java.sql.ResultSet; import java.sql.Statement; public class ArrowFlightSqlIntegrationTest { private static final String JDBC_URL = "jdbc:arrow-flight-sql://127.0.0.1:9408" + "?useEncryption=false" + "&useServerPrepStmts=false" + "&useSSL=false" + "&useArrowFlightSql=true"; private static final String USER = "root"; private static final String PASSWORD = ""; private static int testCaseNum = 1; public static void main(String[] args) { try { // Load Arrow Flight SQL JDBC driver Class.forName("org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver"); try (Connection conn = DriverManager.getConnection(JDBC_URL, USER, PASSWORD); Statement stmt = conn.createStatement()) { testUpdate(stmt, "DROP DATABASE IF EXISTS sr_arrow_flight_sql FORCE;"); testQuery(stmt, "SHOW PROCESSLIST;"); testUpdate(stmt, "CREATE DATABASE sr_arrow_flight_sql;"); testQuery(stmt, "SHOW DATABASES;"); testUpdate(stmt, "USE sr_arrow_flight_sql;"); testUpdate(stmt, "CREATE TABLE sr_table_test (id INT, name STRING) ENGINE=OLAP PRIMARY KEY (id) " + "DISTRIBUTED BY HASH(id) BUCKETS 1 " + "PROPERTIES ('replication_num' = '1');"); testUpdate(stmt, "INSERT INTO sr_table_test VALUES (1, 'Alice'), (2, 'Bob');"); testQuery(stmt, "SELECT * FROM sr_arrow_flight_sql.sr_table_test;"); } } catch (Exception e) { e.printStackTrace(); } } /** * Executes a query and prints the result to the console. */ private static void testQuery(Statement stmt, String sql) throws Exception { System.out.println("Test Case: " + testCaseNum); System.out.println("▶ Executing query: " + sql); ResultSet rs = stmt.executeQuery(sql); try { System.out.println("Result:"); int columnCount = rs.getMetaData().getColumnCount(); while (rs.next()) { for (int i = 1; i <= columnCount; i++) { System.out.print(rs.getString(i) + "\t"); } System.out.println(); } } finally { rs.close(); } testCaseNum++; System.out.println(); } /** * Executes an update (DDL or DML) and prints the result to the console. */ private static void testUpdate(Statement stmt, String sql) throws Exception { System.out.println("Test Case: " + testCaseNum); System.out.println("▶ Executing update: " + sql); stmt.executeUpdate(sql); System.out.println("Result: ✅ Success"); testCaseNum++; System.out.println(); } } ``` Execution results: ```bash Test Case: 1 ▶ Executing update: DROP DATABASE IF EXISTS sr_arrow_flight_sql FORCE; Result: ✅ Success Test Case: 2 ▶ Executing query: SHOW PROCESSLIST; Result: 192.168.124.48_9010_1751449846872 16777217 root Query 2025-07-02 18:46:49 0 OK SHOW PROCESSLIST; false default_warehouse Test Case: 3 ▶ Executing update: CREATE DATABASE sr_arrow_flight_sql; Result: ✅ Success Test Case: 4 ▶ Executing query: SHOW DATABASES; Result: _statistics_ information_schema sr_arrow_flight_sql sys Test Case: 5 ▶ Executing update: USE sr_arrow_flight_sql; Result: ✅ Success Test Case: 6 ▶ Executing update: CREATE TABLE sr_table_test (id INT, name STRING) ENGINE=OLAP PRIMARY KEY (id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ('replication_num' = '1'); Result: ✅ Success Test Case: 7 ▶ Executing update: INSERT INTO sr_table_test VALUES (1, 'Alice'), (2, 'Bob'); Result: ✅ Success Test Case: 8 ▶ Executing query: SELECT * FROM sr_arrow_flight_sql.sr_table_test; Result: 1 Alice 2 Bob ``` ##### Java ADBC Driver[​](#java-adbc-driver "Direct link to Java ADBC Driver") The Arrow Flight SQL protocol provides an open-source JDBC driver that is compatible with the standard JDBC interface. You can easily integrate it into various BI tools (such as Tableau, Power BI, DBeaver, etc.) to access the StarRocks database, just as you would with a traditional JDBC driver. A significant advantage of this driver is its support for high-speed data transfer based on Apache Arrow, which greatly improves the efficiency of query and data transmission. The usage is almost identical to that of a traditional MySQL JDBC driver. note * If you want to start the FE service using command line, you can use the either of the following ways: * Specify the environment variable `JAVA_TOOL_OPTIONS`. ```bash export JAVA_TOOL_OPTIONS="--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED" ``` * Specify the FE configuration item `JAVA_OPTS` in **fe.conf**. This way, you can append other `JAVA_OPTS` values. ```bash JAVA_OPTS="--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED ..." ``` * If you want to debug in IntelliJ IDEA, you must add the following option to `Build and run` in `Run/Debug Configurations`: ```bash --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED ``` POM dependencies ```xml 0.15.0 org.apache.arrow.adbc adbc-driver-jdbc ${adbc.version} org.apache.arrow.adbc adbc-core ${adbc.version} org.apache.arrow.adbc adbc-driver-manager ${adbc.version} org.apache.arrow.adbc adbc-sql ${adbc.version} org.apache.arrow.adbc adbc-driver-flight-sql ${adbc.version} ``` Similar to that in Python, you can also directly create an ADBC client in Java to read data from StarRocks. In this process, you first need to obtain FlightInfo, and then connect to each Endpoint to fetch the data. Code example: ```java public static void main(String[] args) throws Exception { try (BufferAllocator allocator = new RootAllocator()) { FlightSqlDriver driver = new FlightSqlDriver(allocator); Map parameters = new HashMap<>(); String host = "localhost"; int port = 9408; String uri = Location.forGrpcInsecure(host, port).getUri().toString(); AdbcDriver.PARAM_URI.set(parameters, uri); AdbcDriver.PARAM_USERNAME.set(parameters, "root"); AdbcDriver.PARAM_PASSWORD.set(parameters, ""); try (AdbcDatabase database = driver.open(parameters); AdbcConnection connection = database.connect(); AdbcStatement statement = connection.createStatement()) { statement.setSqlQuery("SHOW DATABASES;"); try (AdbcStatement.QueryResult result = statement.executeQuery(); ArrowReader reader = result.getReader()) { int batchCount = 0; while (reader.loadNextBatch()) { batchCount++; VectorSchemaRoot root = reader.getVectorSchemaRoot(); System.out.println("Batch " + batchCount + ":"); System.out.println(root.contentToTSVString()); } System.out.println("Total batches: " + batchCount); } } } } ``` ###### Recommendations[​](#recommendations "Direct link to Recommendations") * Among the three Java Arrow Flight SQL connection methods mentioned above: * If subsequent data analysis relies on row-based data formats, it is recommended to use `jdbc:arrow-flight-sql`, which returns data in the JDBC ResultSet format. * If the analysis can directly process Arrow format or other columnar data formats, you can use the Flight AdbcDriver or Flight JdbcDriver. These options return Arrow-formatted data directly, avoiding row-column conversion and leveraging Arrow’s features to accelerate data parsing. * Regardless of whether you parse JDBC ResultSet or Arrow-formatted data, the parsing time is often longer than the time spent reading the data itself. If you find that Arrow Flight SQL does not deliver the expected performance improvement over `jdbc:mysql://`, consider investigating whether data parsing takes too long. * For all connection methods, data reading with JDK 17 is generally faster than with JDK 1.8. * When reading large-scale datasets, Arrow Flight SQL typically consumes less memory compared to `jdbc:mysql://`. Therefore, if you are experiencing memory constraints, it is also worth trying Arrow Flight SQL. * In addition to the three connection methods above, you can also use the native FlightClient to connect to the Arrow Flight Server, enabling more flexible parallel reading from multiple endpoints. The Java Flight AdbcDriver is built on top of FlightClient and provides a simpler interface compared to using FlightClient directly. ##### Spark[​](#spark "Direct link to Spark") Currently, the official Arrow Flight project has no plans to support Spark or Flink. In the future, support will be gradually added to allow [starrocks-spark-connector](https://github.com/qwshen/spark-flight-connector) to access StarRocks via Arrow Flight SQL, with expected read performance improvements several times. When accessing StarRocks with Spark, in addition to the traditional JDBC or Java client methods, you can also use the open-source Spark-Flight-Connector component to directly read from and write to the StarRocks Flight SQL Server as a Spark DataSource. This approach, based on the Apache Arrow Flight protocol, offers the following significant advantages: * **High-performance data transfer** Spark-Flight-Connector uses Apache Arrow as the data transfer format, enabling zero-copy, highly efficient data exchange. The conversion between StarRocks' `internal Block` data format and Arrow is highly efficient, achieving performance improvements of up to 10 times compared to traditional `CSV` or `JDBC` methods, and significantly reducing data transfer overhead. * **Native support for complex data types** The Arrow data format natively supports complex types (such as `Map`, `Array`, `Struct`, etc.), enabling better adaptation to StarRocks' complex data models compared to traditional JDBC methods, and enhancing data expressiveness and compatibility. * **Support for read, write, and streaming write** The component supports Spark as a Flight SQL client for efficient read and write operations, including `insert`, `merge`, `update`, and `delete` DML statements, and even supports streaming write, making it suitable for real-time data processing scenarios. * **Support for predicate pushdown and column pruning** When reading data, Spark-Flight-Connector supports predicate pushdown and column pruning, enabling data filtering and column selection on the StarRocks side, significantly reducing the amount of data transferred and improving query performance. * **Support for aggregation pushdown and parallel read** Aggregation operations (such as `sum`, `count`, `max`, `min`, etc.) can be pushed down to StarRocks for execution, reducing the computational load on Spark. Parallel reading based on partitioning is also supported, improving read efficiency in large data scenarios. * **Better for big data scenarios** Compared to traditional JDBC methods, the Flight SQL protocol is better suited for large-scale, high concurrency access scenarios, enabling StarRocks to fully leverage its high-performance analytical capabilities. #### Appendix[​](#appendix "Direct link to Appendix") The following is the complete code example in the usage tutorial. ```python # ============================================================================= # StarRocks Arrow Flight SQL Test Script # ============================================================================= # pip install adbc_driver_manager adbc_driver_flightsql pandas # ============================================================================= # ============================================================================= # Required core modules for connecting to StarRocks via Arrow Flight SQL # ============================================================================= import adbc_driver_manager import adbc_driver_flightsql.dbapi as flight_sql # ============================================================================= # Optional modules for better usability and debugging # ============================================================================= import pandas as pd # Optional: for better result display using DataFrame import traceback # Optional: for detailed error traceback during SQL execution import time # Optional: for measuring SQL execution time # ============================================================================= # StarRocks Flight SQL Configuration # ============================================================================= FE_HOST = "127.0.0.1" FE_PORT = 9408 # ============================================================================= # Connect to StarRocks # ============================================================================= conn = flight_sql.connect( uri=f"grpc://{FE_HOST}:{FE_PORT}", db_kwargs={ adbc_driver_manager.DatabaseOptions.USERNAME.value: "root", adbc_driver_manager.DatabaseOptions.PASSWORD.value: "", } ) cursor = conn.cursor() # ============================================================================= # Utility functions for better output formatting and SQL execution # ============================================================================= def print_header(title: str): """ Print a section header for better readability. """ print("\n" + "=" * 80) print(f"🟢 {title}") print("=" * 80) def print_sql(sql: str): """ Print the SQL statement before execution. """ print(f"\n🟡 SQL:\n{sql.strip()}") def print_result(df: pd.DataFrame): """ Print the result DataFrame in a readable format. """ if df.empty: print("\n🟢 Result: (no rows returned)\n") else: print("\n🟢 Result:\n") print(df.to_string(index=False)) def print_error(e: Exception): """ Print the error traceback if SQL execution fails. """ print("\n🔴 Error occurred:") traceback.print_exc() def execute(sql: str): """ Execute a SQL statement and print the result and execution time. """ print_sql(sql) try: start = time.time() # Start time for execution time measurement cursor.execute(sql) result = cursor.fetchallarrow() # Arrow Table df = result.to_pandas() # Convert to DataFrame for better display print_result(df) print(f"\n⏱️ Execution time: {time.time() - start:.3f} seconds") except Exception as e: print_error(e) # ============================================================================= # Step 1: Drop and Create Database # ============================================================================= print_header("Step 1: Drop and Create Database") execute("DROP DATABASE IF EXISTS sr_arrow_flight_sql FORCE;") execute("SHOW DATABASES;") execute("CREATE DATABASE sr_arrow_flight_sql;") execute("SHOW DATABASES;") execute("USE sr_arrow_flight_sql;") # ============================================================================= # Step 2: Create Table # ============================================================================= print_header("Step 2: Create Table") execute(""" CREATE TABLE sr_arrow_flight_sql_test ( k0 INT, k1 DOUBLE, k2 VARCHAR(32) NULL DEFAULT "" COMMENT "", k3 DECIMAL(27,9) DEFAULT "0", k4 BIGINT NULL DEFAULT '10', k5 DATE ) DISTRIBUTED BY HASH(k5) BUCKETS 5 PROPERTIES("replication_num" = "1"); """) execute("SHOW CREATE TABLE sr_arrow_flight_sql_test;") # ============================================================================= # Step 3: Insert Data # ============================================================================= print_header("Step 3: Insert Data") execute(""" INSERT INTO sr_arrow_flight_sql_test VALUES (0, 0.1, "ID", 0.0001, 1111111111, '2025-04-21'), (1, 0.20, "ID_1", 1.00000001, 0, '2025-04-21'), (2, 3.4, "ID_1", 3.1, 123456, '2025-04-22'), (3, 4, "ID", 4, 4, '2025-04-22'), (4, 122345.54321, "ID", 122345.54321, 5, '2025-04-22'); """) # ============================================================================= # Step 4: Query Data # ============================================================================= print_header("Step 4: Query Data") execute("SELECT * FROM sr_arrow_flight_sql_test ORDER BY k0;") # ============================================================================= # Step 5: Session Variables # ============================================================================= print_header("Step 5: Session Variables") execute("SHOW VARIABLES LIKE '%query_mem_limit%';") execute("SET query_mem_limit = 2147483648;") execute("SHOW VARIABLES LIKE '%query_mem_limit%';") # ============================================================================= # Step 6: Aggregation Query # ============================================================================= print_header("Step 6: Aggregation Query") execute(""" SELECT k5, SUM(k1) AS total_k1, COUNT(1) AS row_count, AVG(k3) AS avg_k3 FROM sr_arrow_flight_sql_test GROUP BY k5 ORDER BY k5; """) # ============================================================================= # Step 7: Close Connection # ============================================================================= print_header("Step 7: Close Connection") cursor.close() conn.close() print("✅ Test completed successfully.") ``` --- ### Arrow Flight SQL Best Practices Arrow Flight SQL is the fastest way to pull large result sets out of StarRocks. Against the MySQL protocol, on the same hardware and against the same cluster, Arrow Flight is consistently faster: **3×–9×** faster at the raw protocol fetch, and **19×–97×** faster end-to-end to a pandas DataFrame. The exact factor depends on row count, column shape, and which MySQL client you compare against. But the speedup is not automatic: how the client code reads the result has a large effect on the end-to-end time, and a few simple mistakes can give back most of it. This page shows the overall numbers you can expect, summarises the aspects that affect them, and then describes each aspect with the code change and the measured impact. #### Overall Performance[​](#overall-performance "Direct link to Overall Performance") Two comparisons follow. The first measures only the **protocol fetch** — how long it takes for the bytes to arrive and be parsed, with no language-level object conversion. The second measures a real-world Python application where data is read into a `pandas` DataFrame. See [Test Environment](#test-environment) for the hardware. ##### Protocol-level fetch (Arrow Flight ADBC vs `mysql --quick`)[​](#protocol-level-fetch-arrow-flight-adbc-vs-mysql---quick "Direct link to protocol-level-fetch-arrow-flight-adbc-vs-mysql---quick") `fetch_arrow_table()` drains the network into Arrow buffers without converting cells into Python objects. `mysql --quick` drains the MySQL wire protocol with a streaming C client that parses rows. Both are protocol-only — neither pays for language-native object materialization. | Workload | Rows | MySQL protocol
(`mysql --quick`) | Arrow Flight
(`fetch_arrow_table`) | Speedup | | ----------------------------------- | ----- | ------------------------------------- | --------------------------------------- | -------- | | Single numeric column (`SELECT id`) | 1 M | 831 ms | 215 ms | **3.9×** | | Single numeric column (`SELECT id`) | 5 M | 2,216 ms | 456 ms | **4.9×** | | Single numeric column (`SELECT id`) | 10 M | 4,166 ms | 1,163 ms | **3.6×** | | Single numeric column (`SELECT id`) | 100 M | 35,629 ms | 6,737 ms | **5.3×** | | 20 numeric columns (`SELECT *`) | 1 M | 1,994 ms | 370 ms | **5.4×** | | 20 numeric columns (`SELECT *`) | 5 M | 9,665 ms | 1,251 ms | **7.7×** | | 20 numeric columns (`SELECT *`) | 10 M | 18,461 ms | 2,577 ms | **7.2×** | | 20 numeric columns (`SELECT *`) | 100 M | 178,416 ms | 19,047 ms | **9.4×** | | 20 VARCHAR columns (`SELECT *`) | 1 M | 4,549 ms | 1,294 ms | **3.5×** | | 20 VARCHAR columns (`SELECT *`) | 5 M | 19,077 ms | 5,959 ms | **3.2×** | | 20 VARCHAR columns (`SELECT *`) | 10 M | 36,079 ms | 11,499 ms | **3.1×** | | 20 VARCHAR columns (`SELECT *`) | 100 M | 370,858 ms | 164,508 ms (chunked) | **2.3×** | ##### Real-world Python application — `pd.read_sql` with ADBC vs PyMySQL[​](#real-world-python-application--pdread_sql-with-adbc-vs-pymysql "Direct link to real-world-python-application--pdread_sql-with-adbc-vs-pymysql") The canonical Python pipeline is `pd.read_sql(sql, conn) → pandas.DataFrame`. The connection object you hand it is the entire migration: pass a PyMySQL `Connection` and pandas calls `cursor.fetchall()` + `pd.DataFrame(rows)`, walking every row to build the DataFrame. Pass an ADBC Flight SQL connection and pandas uses ADBC's native Arrow fetch + near-zero-copy DataFrame conversion. | Workload | Rows | `pd.read_sql(sql,`
`adbc_conn)` | `pd.read_sql(sql,`
`pymysql_conn)` | Speedup | | ----------------------------------- | ----- | ------------------------------------ | --------------------------------------- | --------- | | Single numeric column (`SELECT id`) | 1 M | 320 ms | 6,185 ms | **19.3×** | | Single numeric column (`SELECT id`) | 5 M | 421 ms | 30,751 ms | **73.0×** | | Single numeric column (`SELECT id`) | 10 M | 970 ms | 61,524 ms | **63.4×** | | Single numeric column (`SELECT id`) | 100 M | 6,024 ms | 585,556 ms | **97.2×** | | 20 numeric columns (`SELECT *`) | 1 M | 522 ms | 27,521 ms | **52.7×** | | 20 numeric columns (`SELECT *`) | 5 M | 1,530 ms | 141,500 ms | **92.5×** | | 20 numeric columns (`SELECT *`) | 10 M | 2,689 ms | 255,408 ms | **95.0×** | | 20 numeric columns (`SELECT *`) | 100 M | 24,568 ms | OOM | — | | 20 VARCHAR columns (`SELECT *`) | 1 M | 1,560 ms | 31,407 ms | **20.1×** | | 20 VARCHAR columns (`SELECT *`) | 5 M | 6,937 ms | 154,560 ms | **22.3×** | | 20 VARCHAR columns (`SELECT *`) | 10 M | 13,260 ms | 304,647 ms | **23.0×** | Each cell is *fetch* + *convert* = *total*; the speedup is total vs total. Narrow numeric queries hit the largest ratio because the PyMySQL side allocates a Python `int` per cell during fetch and pandas then walks the tuple list during conversion — the ADBC side skips both costs. Arrow's columnar memory format wins twice: it skips per-cell Python object allocation during fetch, and makes the DataFrame conversion almost free afterwards. If your existing code already uses `pd.read_sql`, the migration is one line: ```python import adbc_driver_manager import adbc_driver_flightsql.dbapi as fl import pandas as pd with fl.connect( uri="grpcs://host:443", db_kwargs={ adbc_driver_manager.DatabaseOptions.USERNAME.value: "admin", adbc_driver_manager.DatabaseOptions.PASSWORD.value: "...", }) as conn: df = pd.read_sql("SELECT * FROM my_table LIMIT 5000000", conn) ``` #### Test Environment[​](#test-environment "Direct link to Test Environment") | Component | Details | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | Client host | AWS EC2 `t3.2xlarge`, same VPC subnet as the cluster | | Cluster | 3 FE + 2 BE on `m6g.xlarge`; Arrow Flight on `grpcs://…:443`, MySQL on `:9030` | | Java stack | OpenJDK 17, `jdbc:arrow-flight-sql`, `arrow-jdbc`, `parquet-hadoop` | | Python stack | `python` 3.12, `pyarrow` 24.0, `adbc-driver-flightsql` 1.11, `PyMySQL` 1.2 | | Workload | Two 20-column tables — one VARCHAR-heavy, one all-integer — plus single-column projections; row counts of 1 M, 5 M, 10 M, and 100 M via `SELECT … LIMIT N` | | MySQL drain mode | `cursor.fetchall()` buffered for all measurements | #### Choosing a Client[​](#choosing-a-client "Direct link to Choosing a Client") Before any code-level tuning, the biggest single decision is which client API you read results through. [Interact with StarRocks via Arrow Flight SQL](https://docs.starrocks.io/docs/unloading/arrow_flight.md) covers the full setup for Python ADBC, the Arrow Flight JDBC driver, the Java ADBC driver, and the native `FlightClient`. For performance, those collapse into two paths: * **Raw Arrow batches via `FlightSqlClient` or ADBC (recommended).** This is the columnar end-to-end path the Flight SQL protocol is designed for: your code receives `VectorSchemaRoot` batches and reads them with primitive-returning vector accessors, with no per-row object allocation. End-to-end (drain + typed conversion), this path is about **10×** faster than Java MySQL JDBC on 10 M numeric rows, and **up to 97×** faster than PyMySQL on 100 M narrow numeric queries delivered as a pandas DataFrame. Use it whenever your downstream code can consume columnar data (Pandas, Arrow, ML pipelines, Parquet writers, custom analytics). * **Arrow Flight JDBC driver (`jdbc:arrow-flight-sql`).** Use this when you need a drop-in `ResultSet` for an existing JDBC code path, or for BI tools like Tableau, Power BI, and DBeaver where the JDBC interface is required. JDBC's API forces the driver to return a boxed `Object` for every cell, so this path cannot reach the performance of raw Arrow batches. The JDBC driver is still substantially faster than MySQL JDBC; it is the right tool when JDBC compatibility is the requirement. The per-aspect tables further down switch baselines: they compare the Java Arrow Flight JDBC driver against Java MySQL JDBC, not against PyMySQL. The Java MySQL JDBC connector is much faster at row materialization than PyMySQL — for example the same 5 M VARCHAR `SELECT *` takes ~22 s through Java MySQL JDBC versus ~105 s through PyMySQL — so the Java ratios you'll see are smaller than the Python numbers in Overall Performance. Java MySQL JDBC is the right baseline when you are choosing between Java drivers. The four aspects below apply within whichever client you choose: Aspect 1 is for JDBC consumers, Aspects 2–3 are for raw-batch consumers, and Aspect 4 covers Parquet output from either. #### What Affects Performance[​](#what-affects-performance "Direct link to What Affects Performance") The speedups above assume the client code is written for Arrow. The following four aspects each move the needle by 2× or more on the right workload. Getting them right is the difference between the "tuned" column in the table above and a fetch that looks no faster than MySQL. 1. **JDBC accessor method.** Use `rs.getObject(i)` with a typed cast for numeric columns. `rs.getString(i)` forces the driver to format every value as a string. 2. **Vector resolution scope.** When consuming raw `VectorSchemaRoot` batches, resolve each `FieldVector` once per batch outside the row loop, not once per row. 3. **Typed `.get(i)` for numerics.** On numeric vectors, the typed `.get(i)` returns a primitive with no allocation. The generic accessors box every value. 4. **Parquet writer choice.** PyArrow writes Parquet directly from the Arrow stream with no row-by-row code. Java has no pre-built library for this — every Java path requires a hand-written `WriteSupport` on top of `parquet-hadoop`. ##### Aspect 1 — JDBC: Use Typed Column Access[​](#aspect-1--jdbc-use-typed-column-access "Direct link to Aspect 1 — JDBC: Use Typed Column Access") When using the Arrow Flight JDBC driver, use `rs.getObject(i)` and cast to the expected Java type. This lets the driver return the native Java type directly without an extra conversion step, which matters most for numeric columns. ```java while (rs.next()) { Integer id = (Integer) rs.getObject(1); String name = (String) rs.getObject(2); Long ts = (Long) rs.getObject(3); } ``` ##### Benchmark: JDBC Accessor Methods (includes network)[​](#benchmark-jdbc-accessor-methods-includes-network "Direct link to Benchmark: JDBC Accessor Methods (includes network)") | Workload | MySQL JDBC | Arrow Flight JDBC, typed `getObject()` | Speedup | | -------------- | ---------- | -------------------------------------- | --------- | | VARCHAR — 5 M | 22,651 ms | 12,660 ms | **1.79×** | | VARCHAR — 10 M | 49,216 ms | 27,646 ms | **1.78×** | | Numeric — 5 M | 16,043 ms | 3,123 ms | **5.14×** | | Numeric — 10 M | 38,134 ms | 9,123 ms | **4.18×** | ##### Aspect 2 — Raw Arrow Batches: Pre-Resolve Vectors and Use Typed Access[​](#aspect-2--raw-arrow-batches-pre-resolve-vectors-and-use-typed-access "Direct link to Aspect 2 — Raw Arrow Batches: Pre-Resolve Vectors and Use Typed Access") When consuming raw Arrow batches via the native `FlightSqlClient` (i.e., iterating over `VectorSchemaRoot` objects), follow two rules. **Resolve vectors once per batch, not once per row.** Call `root.getVector("column_name")` before the row loop so the lookup cost is paid once per batch rather than once per row. **Use typed `.get(i)` for numeric vectors.** This returns a primitive value directly — no heap allocation, no GC pressure. ```java IntVector idVec = (IntVector) root.getVector("id"); SmallIntVector yearVec = (SmallIntVector) root.getVector("birth_year"); TinyIntVector monthVec = (TinyIntVector) root.getVector("birth_month"); for (int i = 0; i < rowCount; i++) { record.id = idVec.get(i); // int — no allocation record.birthYear = yearVec.get(i); // short — no allocation record.birthMonth = monthVec.get(i); // byte — no allocation } ``` ##### Benchmark: Arrow Batch Conversion Cost (pre-fetched)[​](#benchmark-arrow-batch-conversion-cost-pre-fetched "Direct link to Benchmark: Arrow Batch Conversion Cost (pre-fetched)") The Arrow Flight numbers below isolate the conversion cost: batches are drained from the cluster into memory first, then timed. | Workload | MySQL JDBC | Typed `.get*()`, vectors resolved once per batch | Speedup | | -------------- | ---------- | ------------------------------------------------ | --------- | | VARCHAR — 5 M | 22,651 ms | 11,921 ms | **1.90×** | | VARCHAR — 10 M | 49,216 ms | 24,686 ms | **1.99×** | | Numeric — 5 M | 16,043 ms | 1,141 ms | **14.1×** | | Numeric — 10 M | 38,134 ms | 2,092 ms | **18.2×** | ##### Aspect 3 — Writing Results to Parquet[​](#aspect-3--writing-results-to-parquet "Direct link to Aspect 3 — Writing Results to Parquet") Apache Arrow does not include a pre-built Parquet writer for `VectorSchemaRoot`. If your goal is simply to export query results to Parquet files, [INSERT INTO FILES](https://docs.starrocks.io/docs/unloading/unload_using_insert_into_files.md) lets StarRocks write the files server-side without any client-side conversion code. The options below apply when you need client-side Parquet output via Arrow Flight. ##### Option 1: Python with PyArrow (Recommended)[​](#option-1-python-with-pyarrow-recommended "Direct link to Option 1: Python with PyArrow (Recommended)") PyArrow handles the Arrow → Parquet conversion with no custom write logic. It preserves column types natively (INT32, INT64, TIMESTAMP\_MICROS, etc.). **Streaming batch-by-batch from Arrow Flight:** ```python import pyarrow.flight as fl import pyarrow.parquet as pq client = fl.connect("grpc+tls://host:443") info = client.get_flight_info( fl.FlightDescriptor.for_command(b"SELECT ...")) reader = client.do_get(info.endpoints[0].ticket) with pq.ParquetWriter("output.parquet", reader.schema_arrow, compression="snappy") as writer: for batch in reader: writer.write_batch(batch) ``` **If the full result fits in memory:** ```python table = reader.read_all() pq.write_table(table, "output.parquet", compression="snappy") ``` **Via ADBC (the recommended Python Flight SQL client):** ```python import adbc_driver_flightsql.dbapi as fl_sql import pyarrow.parquet as pq with fl_sql.connect("grpcs://host:443", db_kwargs={"username": "admin", "password": "..."}) as conn: with conn.cursor() as cur: cur.execute("SELECT * FROM my_table LIMIT 5000000") pq.write_table(cur.fetch_arrow_table(), "output.parquet", compression="snappy") ``` ##### Option 2: Java WriteSupport[​](#option-2-java-writesupport "Direct link to Option 2: Java WriteSupport") For Java, build a custom `WriteSupport` on top of `org.apache.parquet:parquet-hadoop`. Build the schema and writer once per job, then use typed vector reads inside `WriteSupport.write()`. **Build schema and writer once:** ```java MessageType parquetSchema = new SchemaConverter().fromArrow(arrowSchema).getParquetSchema(); ParquetWriter writer = /* build once per job */; // Per batch: writer.write(batch); ``` **Use typed reads inside `WriteSupport.write()`:** ```java class ArrowWriteSupport extends WriteSupport { private RecordConsumer recordConsumer; @Override public void prepareForWrite(RecordConsumer consumer) { this.recordConsumer = consumer; } @Override public void write(VectorSchemaRoot root) { int rowCount = root.getRowCount(); for (FieldVector vec : root.getFieldVectors()) { if (vec instanceof IntVector) { IntVector iv = (IntVector) vec; for (int row = 0; row < rowCount; row++) { recordConsumer.addInteger(iv.get(row)); } } // else if (vec instanceof SmallIntVector) ... BigIntVector ... VarCharVector ... } } } ``` ##### Parquet Benchmark[​](#parquet-benchmark "Direct link to Parquet Benchmark") Numbers include both Parquet encoding and file I/O cost (see [Test Environment](#test-environment)). VARCHAR and numeric tables are benchmarked separately because they stress different parts of the Arrow encoding path: VARCHAR columns require offset-buffer arithmetic on variable-length data, while numeric columns use fixed-width typed vectors where the gains from typed access are much larger. ###### Java (5 M and 10 M rows)[​](#java-5-m-and-10-m-rows "Direct link to Java (5 M and 10 M rows)") Both rows use the same `parquet-hadoop` write path (`MySqlParquetConverter` + `arrow-jdbc` adapter, batch size 65,536) so the only variable is the inbound JDBC driver. | Approach | Rows | VARCHAR UNCOMP | vs MySQL | VARCHAR Snappy | vs MySQL | Numeric UNCOMP | vs MySQL | Numeric Snappy | vs MySQL | | --------------------------- | ---- | -------------- | --------- | -------------- | --------- | -------------- | --------- | -------------- | --------- | | MySQL JDBC → Parquet | 5 M | 55,477 ms | 1.0× | 54,888 ms | 1.0× | 24,006 ms | 1.0× | 25,289 ms | 1.0× | | Arrow Flight JDBC → Parquet | 5 M | **46,341 ms** | **1.20×** | **47,881 ms** | **1.15×** | **13,978 ms** | **1.72×** | **14,297 ms** | **1.77×** | | MySQL JDBC → Parquet | 10 M | 110,229 ms | 1.0× | 116,999 ms | 1.0× | 50,509 ms | 1.0× | 49,126 ms | 1.0× | | Arrow Flight JDBC → Parquet | 10 M | **91,386 ms** | **1.21×** | **102,534 ms** | **1.14×** | **29,739 ms** | **1.70×** | **30,102 ms** | **1.63×** | ###### Python (PyArrow 24.0.0 / ADBC 1.11.0)[​](#python-pyarrow-2400--adbc-1110 "Direct link to Python (PyArrow 24.0.0 / ADBC 1.11.0)") The MySQL baseline is the same Java MySQL JDBC → Parquet number from the table above; "MySQL → PyArrow" is not a real path because there is no MySQL → Arrow adapter outside of `arrow-jdbc`. Python numbers were collected at 5 M only. | Approach | VARCHAR UNCOMP | vs MySQL | VARCHAR Snappy | vs MySQL | Numeric UNCOMP | vs MySQL | Numeric Snappy | vs MySQL | | ----------------------------------------- | -------------- | --------- | -------------- | --------- | -------------- | --------- | -------------- | --------- | | MySQL JDBC → Parquet (Java baseline, 5 M) | 55,477 ms | 1.0× | 54,888 ms | 1.0× | 24,006 ms | 1.0× | 25,289 ms | 1.0× | | Arrow Flight + PyArrow (5 M) | **10,675 ms** | **5.20×** | **14,128 ms** | **3.89×** | **3,953 ms** | **6.07×** | **3,848 ms** | **6.57×** | PyArrow adds almost no overhead on top of the raw network fetch and requires far less code than the Java path. Use PyArrow unless Java is a hard requirement. ##### Summary[​](#summary "Direct link to Summary") | Use case | Recommendation | | ------------------------------ | ----------------------------------------------------------------------- | | Arrow Flight JDBC | Use `getObject()` with typed cast | | Raw `VectorSchemaRoot` batches | Resolve vectors once per batch; use typed `.get(i)` for numeric columns | | Arrow → Parquet in Python | `pyarrow.parquet` via ADBC — single function call, no custom code | | Arrow → Parquet in Java | Hand-written `WriteSupport` with typed vector reads | --- ### Export data using EXPORT This topic describes how to export data from specified tables or partitions in your StarRocks cluster as CSV data files to an external storage system, which can be a distributed file system HDFS or a cloud storage system such as AWS S3. > **NOTICE** > > You can export data out of StarRocks tables only as a user who has the EXPORT privilege on those StarRocks tables. If you do not have the EXPORT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the EXPORT privilege to the user that you use to connect to your StarRocks cluster. #### Background information[​](#background-information "Direct link to Background information") In v2.4 and earlier, StarRocks depends on brokers to set up connections between your StarRocks cluster and your external storage system when it uses the EXPORT statement to export data. Therefore, you need to input `WITH BROKER ""` to specify the broker you want to use in the EXPORT statement. This is called "broker-based unloading." A broker is an independent, stateless service that is integrated with a file-system interface, helping StarRocks export data to your external storage system. From v2.5 onwards, StarRocks no longer depends on brokers to set up connections between your StarRocks cluster and your external storage system when it uses the EXPORT statement to export data. Therefore, you no longer need to specify a broker in the EXPORT statement, but you still need to retain the `WITH BROKER` keyword. This is called "broker-free unloading." When your data is stored in HDFS, however, broker-free unloading may not work and you can resort to broker-based unloading: * If you export data to multiple HDFS clusters, you need to deploy and configure an independent broker for each of these HDFS clusters. * If you export data to a single HDFS cluster and you have configured multiple Kerberos users, you need to deploy one independent broker. #### Supported storage systems[​](#supported-storage-systems "Direct link to Supported storage systems") * Distributed file system HDFS * Cloud storage system such as AWS S3 #### Precautions[​](#precautions "Direct link to Precautions") * We recommend that the data you export at a time do not exceed a few dozen GB. If you export an exceedingly large amount of data at a time, the export may fail and the cost of retrying the export increases. * If the source StarRocks table contains a large amount of data, we recommend that you export data from only a few partitions of the table each time until all data from the table is exported. * If the FEs in your StarRocks cluster restart or a new leader FE is elected when an export job is running, the export job fails. In this situation, you must submit the export job again. * If the FEs in your StarRocks cluster restart or a new leader FE is elected after an export job finishes, some of the job information returned by the [SHOW EXPORT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/unloading/SHOW_EXPORT.md) statement may be lost. * StarRocks exports only the data of base tables. StarRocks does not export the data of materialized views created on base tables. * Export jobs require data scans, which occupy I/O resources and consequently increase query latency. #### Workflow[​](#workflow "Direct link to Workflow") After you submit an export job, StarRocks identifies all tablets involved in the export job. Then, StarRocks divides the involved tablets into groups and generates query plans. The query plans are used to read data from the involved tablets and to write the data to a specified path of the destination storage system. The following figure shows the general workflow. ![img](/assets/images/5.3.1-1-2fbbb20fb5b4f822beccc1b8d08bace4.png) The general workflow consists of the following three steps: 1. The user submits an export job to the leader FE. 2. The leader FE issues `snapshot` instructions to all BEs or CNs in the StarRocks cluster, so the BEs or CNs can take snapshots of the involved tablets to ensure the consistency of the data to be exported. The leader FE also generates multiple export tasks. Each export task is a query plan, and each query plan is used to process a portion of the involved tablets. 3. The leader FE distributes the export tasks to the BEs or CNs. #### Principles[​](#principles "Direct link to Principles") When StarRocks execute query plans, it first creates a temporary folder named `__starrocks_export_tmp_xxx` in the specified path of the destination storage system. In the name of the temporary folder, `xxx` represents the ID of the export job. An example name for the temporary folder is `__starrocks_export_tmp_921d8f80-7c9d-11eb-9342-acde48001122`. After StarRocks successfully executes a query plan, it generates a temporary file in the temporary folder and writes the exported data into the generated temporary file. After all data is exported, StarRocks uses the RENAME statement to save the generated temporary files to a specified path. #### Related parameters[​](#related-parameters "Direct link to Related parameters") This section describes some export-related parameters that you can configure in the FEs of your StarRocks cluster. * `export_checker_interval_second`: the interval at which export jobs are scheduled. The default interval is 5 seconds. After you reconfigure this parameter for an FE, you need to restart the FE to make the new parameter setting take effect. * `export_running_job_num_limit`: the maximum number of running export jobs that are allowed. If the number of running export jobs exceeds this limit, excessive export jobs enter a waiting state after `snapshot` is run. The default maximum number is 5. You can reconfigure this parameter when export jobs are running. * `export_task_default_timeout_second`: the timeout period for export jobs. The default timeout period is 2 hours. You can reconfigure this parameter when export jobs are running. * `export_max_bytes_per_be_per_task`: the maximum amount of data as compressed that can be exported per export task from each BE or CN. This parameter provides a policy based on which StarRocks splits export jobs into export tasks that can be concurrently run. The default maximum amount is 256 MB. * `export_task_pool_size`: the maximum number of export tasks that can be concurrently run by the thread pool. The default maximum number is 5. #### Basic operations[​](#basic-operations "Direct link to Basic operations") ##### Submit an export job[​](#submit-an-export-job "Direct link to Submit an export job") Suppose that your StarRocks database `db1` contains a table named `tbl1`. To export the data of columns `col1` and `col3` from partitions `p1` and `p2` of `tbl1` to the `export` path of your HDFS cluster, run the following command: ```sql EXPORT TABLE db1.tbl1 PARTITION (p1,p2) (col1, col3) TO "hdfs://HDFS_IP:HDFS_Port/export/lineorder_" PROPERTIES ( "column_separator"=",", "load_mem_limit"="2147483648", "timeout" = "3600" ) WITH BROKER ( "username" = "user", "password" = "passwd" ); ``` For detailed syntax and parameter descriptions as well as the command examples of exporting data to AWS S3, see [EXPORT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/unloading/EXPORT.md). ##### Obtain the query ID of an export job[​](#obtain-the-query-id-of-an-export-job "Direct link to Obtain the query ID of an export job") After you submit an export job, you can use the SELECT LAST\_QUERY\_ID() statement to query the query ID of the export job. With the query ID, you can view or cancel the export job. For the detailed syntax and parameter descriptions, see [last\_query\_id](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/last_query_id.md). ##### View the status of an export job[​](#view-the-status-of-an-export-job "Direct link to View the status of an export job") After you submit an export job, you can use the SHOW EXPORT statement to view the status of the export job. Example: ```sql SHOW EXPORT WHERE queryid = "edee47f0-abe1-11ec-b9d1-00163e1e238f"; ``` > **NOTE** > > In the preceding example, `queryid` is the query ID of the export job. Information similar to the following output is returned: ```plain JobId: 14008 State: FINISHED Progress: 100% TaskInfo: {"partitions":["*"],"mem limit":2147483648,"column separator":",","line delimiter":"\n","tablet num":1,"broker":"hdfs","coord num":1,"db":"default_cluster:db1","tbl":"tbl3",columns:["col1", "col3"]} Path: oss://bj-test/export/ CreateTime: 2019-06-25 17:08:24 StartTime: 2019-06-25 17:08:28 FinishTime: 2019-06-25 17:08:34 Timeout: 3600 ErrorMsg: N/A ``` For detailed syntax and parameter descriptions, see [SHOW EXPORT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/unloading/SHOW_EXPORT.md). ##### Cancel an export job[​](#cancel-an-export-job "Direct link to Cancel an export job") You can use the CANCEL EXPORT statement to cancel an export job that you have submitted. Example: ```sql CANCEL EXPORT WHERE queryid = "921d8f80-7c9d-11eb-9342-acde48001122"; ``` > **NOTE** > > In the preceding example, `queryid` is the query ID of the export job. For detailed syntax and parameter descriptions, see [CANCEL EXPORT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/unloading/CANCEL_EXPORT.md). #### Best practices[​](#best-practices "Direct link to Best practices") ##### Query plan splitting[​](#query-plan-splitting "Direct link to Query plan splitting") The number of query plans into which an export job is split varies depending on the number of tablets involved in the export job and on the maximum amount of data that can be processed per query plan. Export jobs are retried as query plans. If the amount of data processed by a query plan exceeds the maximum amount allowed, the query plan encounters errors such as jitters in remote storage. As a result, the cost of retrying the query plan increases. The maximum amount of data that can be processed per query plan by each BE or CN is specified by the `export_max_bytes_per_be_per_task` parameter, which defaults to 256 MB. In a query plan, each BE or CN is allocated at least one tablet and can export a data amount that does not exceed the limit specified by the `export_max_bytes_per_be_per_task` parameter. The multiple query plans of an export job are concurrently executed. You can use the FE parameter `export_task_pool_size` to specify the maximum number of export tasks that are allowed to concurrently run by the thread pool. This parameter defaults to `5`. In normal cases, each query plan of an export job consists of only two parts: scanning and exporting. The logic for performing computations required by query plans does not consume much memory. Therefore, the default memory limit of 2 GB can meet most of your business requirements. However, in certain circumstances, such as when a query plan requires scanning many tablets on a BE or CN or a tablet has many versions, the 2-GB memory capacity may be insufficient. In these circumstances, you need to use the `load_mem_limit` parameter to specify a higher memory capacity limit, such as 4 GB or 8 GB. --- ### Feature Support: Data Loading and Unloading ### Feature Support: Data Loading and Unloading This document outlines the features of various data loading and unloading methods supported by StarRocks. #### File format[​](#file-format "Direct link to File format") ##### Loading file formats[​](#loading-file-formats "Direct link to Loading file formats") | | Data Source | File Format | | | | | | | | -------------------- | -------------------------------------------- | ----------- | --------------- | --------------- | --------------- | ---------------- | --------------- | --------------- | | | | CSV | JSON \[3] | Parquet | ORC | Avro | ProtoBuf | Thrift | | Stream Load | Local file systems, applications, connectors | Yes | Yes | To be supported | To be supported | To be supported | | | | INSERT from FILES | HDFS, S3, OSS, Azure, GCS, NFS(NAS) \[5] | Yes (v3.3+) | To be supported | Yes (v3.1+) | Yes (v3.1+) | Yes (v3.4.4+) | To be supported | | | Broker Load | | Yes | Yes (v3.2.3+) | Yes | Yes | To be supported | | | | Routine Load | Kafka | Yes | Yes | To be supported | To be supported | Yes (v3.0+) \[1] | To be supported | To be supported | | Spark Load | | Yes | To be supported | Yes | Yes | To be supported | | | | Connectors | Flink, Spark | Yes | Yes | To be supported | To be supported | To be supported | | | | Kafka Connector \[2] | Kafka | Yes (v3.0+) | | To be supported | To be supported | Yes (v3.0+) | | To be supported | | PIPE \[4] | Consistent with INSERT from FILES | | | | | | | | note \[1], \[2]: Schema Registry is required. \[3]: JSON supports a variety of CDC formats. For details about the JSON CDC formats supported by StarRocks, see [JSON CDC format](#json-cdc-formats). \[4]: Currently, only INSERT from FILES is supported for loading with PIPE. \[5]: You need to mount a NAS device as NFS under the same directory of each BE or CN node to access the files in NFS via the `file://` protocol. ###### JSON CDC formats[​](#json-cdc-formats "Direct link to JSON CDC formats") | | Stream Load | Routine Load | Broker Load | INSERT from FILES | Kafka Connector \[1] | | -------- | --------------- | --------------- | --------------- | ----------------- | -------------------- | | Debezium | To be supported | To be supported | To be supported | To be supported | Yes (v3.0+) | | Canal | To be supported | | | | | | Maxwell | | | | | | note \[1]: You must configure the `transforms` parameter while loading Debezium CDC format data into Primary Key tables in StarRocks. ##### Unloading file formats[​](#unloading-file-formats "Direct link to Unloading file formats") | | Target | | File format | | | | | ------------------- | -------------------- | ---------------------------------------- | --------------- | --------------- | --------------- | --------------- | | | Table format | Remote storage | CSV | JSON | Parquet | ORC | | INSERT INTO FILES | N/A | HDFS, S3, OSS, Azure, GCS, NFS(NAS) \[3] | Yes (v3.3+) | To be supported | Yes (v3.2+) | Yes (v3.3+) | | INSERT INTO Catalog | Hive | HDFS, S3, OSS, Azure, GCS | Yes (v3.3+) | To be supported | Yes (v3.2+) | Yes (v3.3+) | | | Iceberg | HDFS, S3, OSS, Azure, GCS | To be supported | To be supported | Yes (v3.2+) | To be supported | | | Hudi/Delta | | To be supported | | | | | EXPORT | N/A | HDFS, S3, OSS, Azure, GCS | Yes \[1] | To be supported | To be supported | To be supported | | PIPE | To be supported \[2] | | | | | | note \[1]: Configuring Broker process is supported. \[2]: Currently, unloading data using PIPE is not supported. \[3]: You need to mount a NAS device as NFS under the same directory of each BE or CN node to access the files in NFS via the `file://` protocol. #### File format-related parameters[​](#file-format-related-parameters "Direct link to File format-related parameters") ##### Loading file format-related parameters[​](#loading-file-format-related-parameters "Direct link to Loading file format-related parameters") | File format | Parameter | Loading method | | | | | | ----------- | ------------------- | -------------- | ----------------- | ---------------- | ---------------- | --------------- | | | | Stream Load | INSERT from FILES | Broker Load | Routine Load | Spark Load | | CSV | column\_separator | Yes | Yes (v3.3+) | Yes \[1] | | | | | row\_delimiter | Yes | | Yes \[2] (v3.1+) | Yes \[3] (v2.2+) | To be supported | | | enclose | Yes (v3.0+) | | Yes (v3.0+) | Yes (v3.0+) | To be supported | | | escape | | | | | | | | skip\_header | | To be supported | | | | | | trim\_space | | Yes (v3.0+) | | | | | JSON | jsonpaths | Yes | To be supported | Yes (v3.2.3+) | Yes | To be supported | | | strip\_outer\_array | | | | | | | | json\_root | | | | | | | | ignore\_json\_size | | | | To be supported | | note \[1]: The corresponding parameter is `COLUMNS TERMINATED BY`. \[2]: The corresponding parameter is `ROWS TERMINATED BY`. \[3]: The corresponding parameter is `ROWS TERMINATED BY`. ##### Unloading file format-related parameters[​](#unloading-file-format-related-parameters "Direct link to Unloading file format-related parameters") | File format | Parameter | Unloading method | | | ----------- | -------------------- | ----------------- | ------ | | | | INSERT INTO FILES | EXPORT | | CSV | column\_separator | Yes (v3.3+) | Yes | | | line\_delimiter \[1] | | | | | enclose | Yes | No | | | escape | Yes | No | | | include\_header | Yes | No | note \[1]: The corresponding parameter in data loading is `row_delimiter`. #### Compression formats[​](#compression-formats "Direct link to Compression formats") ##### Loading compression formats[​](#loading-compression-formats "Direct link to Loading compression formats") | File format | Compression format | Loading method | | | | | | ----------- | ---------------------------------------------------------------- | ------------------ | --------------- | ----------------- | --------------- | --------------- | | | | Stream Load | Broker Load | INSERT from FILES | Routine Load | Spark Load | | CSV | - deflate
- bzip2
- gzip
- lz4\_frame
- zstd | Yes \[1] | Yes \[2] | To be supported | To be supported | To be supported | | JSON | | Yes (v3.2.7+) \[3] | To be supported | N/A | To be supported | N/A | | Parquet | - gzip
- lz4
- snappy
- zlib
- zstd | N/A | Yes \[4] | | To be supported | Yes \[4] | | ORC | | | | | | | note \[1]: Currently, only when loading CSV files with Stream Load can you specify the compression format by using `format=gzip`, indicating gzip-compressed CSV files. `deflate` and `bzip2` formats are also supported. \[2]: Broker Load does not support specifying the compression format of CSV files by using the parameter `format`. Broker Load identifies the compression format by using the suffix of the file. The suffix of gzip-compressed files is `.gz`, and that of the zstd-compressed files is `.zst`. Besides, other `format`-related parameters, such as `trim_space` and `enclose`, are also not supported. \[3]: Supports specifying the compression format by using `compression = gzip`. \[4]: Supported by Arrow Library. You do not need to configure the `compression` parameter. ##### Unloading compression formats[​](#unloading-compression-formats "Direct link to Unloading compression formats") | File format | Compression format | Unloading method | | | | | | ----------- | ---------------------------------------------------------------- | ----------------- | ------------------- | --------------- | --------------- | --------------- | | | | INSERT INTO FILES | INSERT INTO Catalog | | | EXPORT | | | | | Hive | Iceberg | Hudi/Delta | | | CSV | - deflate
- bzip2
- gzip
- lz4\_frame
- zstd | To be supported | To be supported | To be supported | To be supported | To be supported | | JSON | N/A | N/A | N/A | N/A | N/A | N/A | | Parquet | - gzip
- lz4
- snappy
- zstd | Yes (v3.2+) | Yes (v3.2+) | Yes (v3.2+) | To be supported | N/A | | ORC | | | | | | | #### Credentials[​](#credentials "Direct link to Credentials") ##### Loading - Authentication[​](#loading---authentication "Direct link to Loading - Authentication") | Authentication | Loading method | | | | | | -------------------------------------------------------- | -------------- | ------------------------------------------- | ------------------------------------- | ------------------ | ---------------------- | | | Stream Load | INSERT from FILES | Broker Load | Routine Load | External Catalog | | Single Kerberos | N/A | Yes (v3.1+) | Yes \[1] (versions earlier than v2.5) | Yes \[2] (v3.1.4+) | Yes | | Kerberos Ticket Granting Ticket (TGT) | N/A | To be supported | | | Yes (v3.1.10+/v3.2.1+) | | Single KDC Multiple Kerberos | N/A | | | | | | Basic access authentications (Access Key pair, IAM Role) | N/A | Yes (HDFS and S3-compatible object storage) | | Yes \[3] | Yes | note \[1]: For HDFS, StarRocks supports both simple authentication and Kerberos authentication. \[2]: When the security protocol is set to `sasl_plaintext` or `sasl_ssl`, both SASL and GSSAPI (Kerberos) authentications are supported. \[3]: When the security protocol is set to `sasl_plaintext` or `sasl_ssl`, both SASL and PLAIN authentications are supported. ##### Unloading - Authentication[​](#unloading---authentication "Direct link to Unloading - Authentication") | | INSERT INTO FILES | EXPORT | | --------------- | ----------------- | --------------- | | Single Kerberos | To be supported | To be supported | #### Loading - Other parameters and features[​](#loading---other-parameters-and-features "Direct link to Loading - Other parameters and features") | Parameter and feature | Loading method | | | | | | | | | -------------------------------------------- | -------------- | ----------------- | ------------------------- | ---------------------- | --------------- | --------------- | --------------- | - | | | Stream Load | INSERT from FILES | INSERT from SELECT/VALUES | Broker Load | PIPE | Routine Load | Spark Load | | | partial\_update | Yes (v3.0+) | Yes \[1] (v3.3+) | | Yes (v3.0+) | N/A | Yes (v3.0+) | To be supported | | | partial\_update\_mode | Yes (v3.1+) | To be supported | | Yes (v3.1+) | N/A | To be supported | To be supported | | | COLUMNS FROM PATH | N/A | Yes (v3.2+) | N/A | Yes | N/A | N/A | Yes | | | timezone or session variable time\_zone \[2] | Yes \[3] | Yes \[4] | Yes \[4] | Yes \[4] | To be supported | Yes \[4] | To be supported | | | Time accuracy - Microsecond | Yes | Yes | Yes | Yes (v3.1.11+/v3.2.6+) | To be supported | Yes | Yes | | note \[1]: From v3.3 onwards, StarRocks supports Partial Updates in Row mode for INSERT INTO by specifying the column list. \[2]: Setting the time zone by the parameter or the session variable will affect the results returned by functions such as strftime(), alignment\_timestamp(), and from\_unixtime(). \[3]: Only the parameter `timezone` is supported. \[4]: Only the session variable `time_zone` is supported. #### Unloading - Other parameters and features[​](#unloading---other-parameters-and-features "Direct link to Unloading - Other parameters and features") | Parameter and feature | INSERT INTO FILES | EXPORT | | --------------------------- | ----------------- | --------------- | | target\_max\_file\_size | Yes (v3.2+) | To be supported | | single | | | | Partitioned\_by | | | | Session variable time\_zone | To be supported | | | Time accuracy - Microsecond | To be supported | To be supported | --- ### Read data from StarRocks using Flink connector StarRocks provides a self-developed connector named StarRocks Connector for Apache Flink® (Flink connector for short) to help you read data in bulk from a StarRocks cluster by using Flink. The Flink connector supports two reading methods: Flink SQL and Flink DataStream. Flink SQL is recommended. > **NOTE** > > The Flink connector also supports writing the data read by Flink to another StarRocks cluster or storage system. See [Continuously load data from Apache Flink®](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks.md). #### Background information[​](#background-information "Direct link to Background information") Unlike the JDBC connector provided by Flink, the Flink connector of StarRocks supports reading data from multiple BEs of your StarRocks cluster in parallel, greatly accelerating read tasks. The following comparison shows the difference in implementation between the two connectors. * Flink connector of StarRocks With the Flink connector of StarRocks, Flink can first obtain the query plan from the responsible FE, then distribute the obtained query plan as parameters to all the involved BEs, and finally obtain the data returned by the BEs. ![- Flink connector of StarRocks](/assets/images/5.3.2-1-2720516ba00b6b5b889d0dc012d53cfa.png) * JDBC connector of Flink With the JDBC connector of Flink, Flink can only read data from individual FEs, one at a time. Data reads are slow. ![JDBC connector of Flink](/assets/images/5.3.2-2-9404d95ac1586562c16f7559722fd1ba.png) #### Version requirements[​](#version-requirements "Direct link to Version requirements") | Connector | Flink | StarRocks | Java | Scala | | --------- | ----------------------------- | ------------- | ---- | --------- | | 1.2.16 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.15 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.14 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.12 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.11 | 1.15,1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Flink has been deployed. If Flink has not been deployed, follow these steps to deploy it: 1. Install Java 8 or Java 11 in your operating system to ensure Flink can run properly. You can use the following command to check the version of your Java installation: ```sql java -version ``` For example, if the following information is returned, Java 8 has been installed: ```sql openjdk version "1.8.0_322" OpenJDK Runtime Environment (Temurin)(build 1.8.0_322-b06) OpenJDK 64-Bit Server VM (Temurin)(build 25.322-b06, mixed mode) ``` 2. Download and unzip the [Flink package](https://flink.apache.org/downloads/) of your choice. > **NOTE** > > We recommend that you use Flink v1.14 or later. The minimum Flink version supported is v1.11. ```sql # Download the Flink package. wget https://dlcdn.apache.org/flink/flink-1.14.5/flink-1.14.5-bin-scala_2.11.tgz # Unzip the Flink package. tar -xzf flink-1.14.5-bin-scala_2.11.tgz # Go to the Flink directory. cd flink-1.14.5 ``` 3. Start your Flink cluster. ```sql # Start your Flink cluster. ./bin/start-cluster.sh # When the following information is displayed, your Flink cluster has successfully started: Starting cluster. Starting standalonesession daemon on host. Starting taskexecutor daemon on host. ``` You can also deploy Flink by following the instructions provided in [Flink documentation](https://nightlies.apache.org/flink/flink-docs-release-2.3/docs/getting-started/local_installation/#option-b-local-installation). #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Deploy the Flink connector[​](#deploy-the-flink-connector "Direct link to Deploy the Flink connector") Follow these steps to deploy the Flink connector: 1. Select and download the [flink-connector-starrocks](https://github.com/StarRocks/flink-connector-starrocks/releases) JAR package matching the Flink version that you are using. If code debugging is needed, compile the Flink connector package to suit your business requirements. > **NOTICE** > > We recommend that you download the Flink connector package whose version is 1.2.x or later and whose matching Flink version has the same first two digits as the Flink version that you are using. For example, if you use Flink v1.14.x, you can download `flink-connector-starrocks-1.2.4_flink-1.14_x.yy.jar`. 2. Place the Flink connector package you downloaded or compiled into the `lib` directory of Flink. 3. Restart your Flink cluster. ##### Network configuration[​](#network-configuration "Direct link to Network configuration") Ensure that the machine where Flink is located can access the FE nodes of the StarRocks cluster via the [`http_port`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#http_port) (default: `8030`) and [`query_port`](https://docs.starrocks.io/docs/administration/configuration/FE_parameters.md#query_port) (default: `9030`), and the BE nodes via the [`be_port`](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#be_port) (default: `9060`). #### Parameters[​](#parameters "Direct link to Parameters") ##### Common parameters[​](#common-parameters "Direct link to Common parameters") The following parameters apply to both the Flink SQL and Flink DataStream reading methods. | Parameter | Required | Data type | Description | | --------------------------- | -------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | connector | Yes | STRING | The type of connector that you want to use to read data. Set the value to `starrocks`. | | scan-url | Yes | STRING | The address that is used to connect the FE from the web server. Format: `:`. The default port is `8030`. You can specify multiple addresses, which must be separated with a comma (,). Example: `192.168.xxx.xxx:8030,192.168.xxx.xxx:8030`. | | jdbc-url | Yes | STRING | The address that is used to connect the MySQL client of the FE. Format: `jdbc:mysql://:`. The default port number is `9030`. | | username | Yes | STRING | The username of your StarRocks cluster account. The account must have read permissions on the StarRocks table you want to read. See [User privileges](https://docs.starrocks.io/docs/administration/user_privs/authorization/User_privilege.md). | | password | Yes | STRING | The password of your StarRocks cluster account. | | database-name | Yes | STRING | The name of the StarRocks database to which the StarRocks table you want to read belongs. | | table-name | Yes | STRING | The name of the StarRocks table you want to read. | | scan.connect.timeout-ms | No | STRING | The maximum amount of time after which the connection from the Flink connector to your StarRocks cluster times out. Unit: milliseconds. Default value: `1000`. If the amount of time taken to establish the connection exceeds this limit, the read task fails. | | scan.params.keep-alive-min | No | STRING | The maximum amount of time during which the read task keeps alive. The keep-alive time is checked on a regular basis by using a polling mechanism. Unit: minutes. Default value: `10`. We recommend that you set this parameter to a value that is greater than or equal to `5`. | | scan.params.query-timeout-s | No | STRING | The maximum amount of time after which the read task times out. The timeout duration is checked during task execution. Unit: seconds. Default value: `600`. If no read result is returned after the time duration elapses, the read task stops. | | scan.params.mem-limit-byte | No | STRING | The maximum amount of memory allowed per query on each BE. Unit: bytes. Default value: `1073741824`, equal to 1 GB. | | scan.max-retries | No | STRING | The maximum number of times that the read task can be retried upon failures. Default value: `1`. If the number of times that the read task is retried exceeds this limit, the read task returns errors. | ##### Parameters for Flink DataStream[​](#parameters-for-flink-datastream "Direct link to Parameters for Flink DataStream") The following parameters apply only to the Flink DataStream reading method. | Parameter | Required | Data type | Description | | ------------ | -------- | --------- | ----------------------------------------------------------------------------------------------------------- | | scan.columns | No | STRING | The column that you want to read. You can specify multiple columns, which must be separated by a comma (,). | | scan.filter | No | STRING | The filter condition based on which you want to filter data. | Assume that in Flink you create a table that consists of three columns, which are `c1`, `c2`, `c3`. To read the rows whose values in the `c1` column of this Flink table are equal to `100`, you can specify two filter conditions `"scan.columns, "c1"` and `"scan.filter, "c1 = 100"`. #### Data type mapping between StarRocks and Flink[​](#data-type-mapping-between-starrocks-and-flink "Direct link to Data type mapping between StarRocks and Flink") The following data type mapping is valid only for Flink reading data from StarRocks. For the data type mapping used for Flink writing data into StarRocks, see [Continuously load data from Apache Flink®](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks.md). | StarRocks | Flink | | ---------- | ---------------------------------------------------------------------------------------------------------------- | | NULL | NULL | | BOOLEAN | BOOLEAN | | TINYINT | TINYINT | | SMALLINT | SMALLINT | | INT | INT | | BIGINT | BIGINT | | LARGEINT | STRING | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | DATE | DATE | | DATETIME | TIMESTAMP | | DECIMAL | DECIMAL | | DECIMALV2 | DECIMAL | | DECIMAL32 | DECIMAL | | DECIMAL64 | DECIMAL | | DECIMAL128 | DECIMAL | | CHAR | CHAR | | VARCHAR | STRING | | JSON | STRING
**NOTE:**
**Supported since version 1.2.10** | | ARRAY | ARRAY
**NOTE:**
**Supported since version 1.2.10, and StarRocks v3.1.12/v3.2.5 or later is required.** | | STRUCT | ROW
**NOTE:**
**Supported since version 1.2.10, and StarRocks v3.1.12/v3.2.5 or later is required.** | | MAP | MAP
**NOTE:**
**Supported since version 1.2.10, and StarRocks v3.1.12/v3.2.5 or later is required.** | #### Examples[​](#examples "Direct link to Examples") The following examples assume you have created a database named `test` in your StarRocks cluster and you have the permissions of user `root`. > **NOTE** > > If a read task fails, you must re-create it. ##### Data example[​](#data-example "Direct link to Data example") 1. Go to the `test` database and create a table named `score_board`. ```sql MySQL [test]> CREATE TABLE `score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`) PROPERTIES ( "replication_num" = "3" ); ``` 2. Insert data into the `score_board` table. ```sql MySQL [test]> INSERT INTO score_board VALUES (1, 'Bob', 21), (2, 'Stan', 21), (3, 'Sam', 22), (4, 'Tony', 22), (5, 'Alice', 22), (6, 'Lucy', 23), (7, 'Polly', 23), (8, 'Tom', 23), (9, 'Rose', 24), (10, 'Jerry', 24), (11, 'Jason', 24), (12, 'Lily', 25), (13, 'Stephen', 25), (14, 'David', 25), (15, 'Eddie', 26), (16, 'Kate', 27), (17, 'Cathy', 27), (18, 'Judy', 27), (19, 'Julia', 28), (20, 'Robert', 28), (21, 'Jack', 29); ``` 3. Query the `score_board` table. ```sql MySQL [test]> SELECT * FROM score_board; +------+---------+-------+ | id | name | score | +------+---------+-------+ | 1 | Bob | 21 | | 2 | Stan | 21 | | 3 | Sam | 22 | | 4 | Tony | 22 | | 5 | Alice | 22 | | 6 | Lucy | 23 | | 7 | Polly | 23 | | 8 | Tom | 23 | | 9 | Rose | 24 | | 10 | Jerry | 24 | | 11 | Jason | 24 | | 12 | Lily | 25 | | 13 | Stephen | 25 | | 14 | David | 25 | | 15 | Eddie | 26 | | 16 | Kate | 27 | | 17 | Cathy | 27 | | 18 | Judy | 27 | | 19 | Julia | 28 | | 20 | Robert | 28 | | 21 | Jack | 29 | +------+---------+-------+ 21 rows in set (0.00 sec) ``` ##### Read data using Flink SQL[​](#read-data-using-flink-sql "Direct link to Read data using Flink SQL") 1. In your Flink cluster, create a table named `flink_test` based on the schema of the source StarRocks table (which is `score_board` in this example). In the table creation command, you must configure the read task properties, including the information about the Flink connector, the source StarRock database, and the source StarRocks table. ```sql CREATE TABLE flink_test ( `id` INT, `name` STRING, `score` INT ) WITH ( 'connector'='starrocks', 'scan-url'='192.168.xxx.xxx:8030', 'jdbc-url'='jdbc:mysql://192.168.xxx.xxx:9030', 'username'='xxxxxx', 'password'='xxxxxx', 'database-name'='test', 'table-name'='score_board' ); ``` 2. Use SELECT to read data from StarRocks. ```sql SELECT id, name FROM flink_test WHERE score > 20; ``` When you read data by using Flink SQL, take note of the following points: * You can use only SQL statements like `SELECT ... FROM WHERE ...` to read data from StarRocks. Of all aggregate functions, only `count` is supported. * Predicate pushdown is supported. For example, if your query contains a filter condition `char_1 <> 'A' and int_1 = -126`, the filter condition will be pushed down to the Flink connector and transformed into a statement that can be executed by StarRocks before the query is run. You do not need to perform extra configurations. * The LIMIT statement is not supported. * StarRocks does not support the checkpointing mechanism. As a result, data consistency cannot be guaranteed if the read task fails. * The order of the fields in the created table must be the same as in the StarRocks table. ##### Read data using Flink DataStream[​](#read-data-using-flink-datastream "Direct link to Read data using Flink DataStream") 1. Add the following dependencies to the `pom.xml` file: ```sql com.starrocks flink-connector-starrocks x.x.x_flink-1.15 x.x.x_flink-1.14_2.11 x.x.x_flink-1.14_2.12 x.x.x_flink-1.13_2.11 x.x.x_flink-1.13_2.12 x.x.x_flink-1.12_2.11 x.x.x_flink-1.12_2.12 x.x.x_flink-1.11_2.11 x.x.x_flink-1.11_2.12 ``` You must replace `x.x.x` in the preceding code example with the latest Flink connector version that you are using. See [Version information](https://search.maven.org/search?q=g:com.starrocks). 2. Call the Flink connector to read data from StarRocks: ```java import com.starrocks.connector.flink.StarRocksSource; import com.starrocks.connector.flink.table.source.StarRocksSourceOptions; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.table.api.DataTypes; import org.apache.flink.table.api.TableSchema; public class StarRocksSourceApp { public static void main(String[] args) throws Exception { StarRocksSourceOptions options = StarRocksSourceOptions.builder() .withProperty("scan-url", "192.168.xxx.xxx:8030") .withProperty("jdbc-url", "jdbc:mysql://192.168.xxx.xxx:9030") .withProperty("username", "root") .withProperty("password", "") .withProperty("table-name", "score_board") .withProperty("database-name", "test") .build(); TableSchema tableSchema = TableSchema.builder() .field("id", DataTypes.INT()) .field("name", DataTypes.STRING()) .field("score", DataTypes.INT()) .build(); StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.addSource(StarRocksSource.source(tableSchema, options)).setParallelism(5).print(); env.execute("StarRocks flink source"); } } ``` #### FAQ[​](#faq "Direct link to FAQ") ###### I got an error "Failed to get next from be" while exporting data with Flink Connector. What should I do?[​](#i-got-an-error-failed-to-get-next-from-be-while-exporting-data-with-flink-connector-what-should-i-do "Direct link to I got an error \"Failed to get next from be\" while exporting data with Flink Connector. What should I do?") You can set the BE configuration `scan_context_gc_interval_min` (Default: 5, Unit: Minutes) to a larger value to increase the time interval at which the Scan Context is cleaned. #### What's next[​](#whats-next "Direct link to What's next") After Flink successfully reads data from StarRocks, you can use the Flink WebUI to monitor the read task. For example, you can view the `totalScannedRows` metric on the **Metrics** page of the WebUI to obtain the number of rows that are successfully read. You can also use Flink SQL to perform calculations such as joins on the data you have read. --- ### Read data from StarRocks using Spark connector StarRocks provides a self-developed connector named StarRocks Connector for Apache Spark™ (Spark connector for short) to help you read data from a StarRocks table by using Spark. You can use Spark for complex processing and machine learning on the data you have read from StarRocks. The Spark connector supports three reading methods: Spark SQL, Spark DataFrame, and Spark RDD. You can use Spark SQL to create a temporary view on the StarRocks table, and then directly read data from the StarRocks table by using that temporary view. You can also map the StarRocks table to a Spark DataFrame or a Spark RDD, and then read data from the Spark DataFrame or Spark RDD. We recommend the use of a Spark DataFrame. > **NOTICE** > > Only users with the SELECT privilege on a StarRocks table can read data from this table. You can follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the privilege to a user. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * You can filter data on StarRocks before you read the data, thereby reducing the amount of data transferred. * If the overhead of reading data is substantial, you can employ appropriate table design and filter conditions to prevent Spark from reading an excessive amount of data at a time. As such, you can reduce I/O pressure on your disk and network connection, thereby ensuring routine queries can be run properly. #### Version requirements[​](#version-requirements "Direct link to Version requirements") | Spark connector | Spark | StarRocks | Java | Scala | | --------------- | ------------------ | -------------- | ---- | ----- | | 1.1.4 | 4.0, 4.1 | 2.5 and later | 17 | 2.13 | | 1.1.4 | 3.3, 3.4, 3.5 | 2.5 and later | 8 | 2.12 | | 1.1.3 | 3.2, 3.3, 3.4, 3.5 | 2.5 and later | 8 | 2.12 | | 1.1.2 | 3.2, 3.3, 3.4, 3.5 | 2.5 and later | 8 | 2.12 | | 1.1.1 | 3.2, 3.3, or 3.4 | 2.5 and later | 8 | 2.12 | | 1.1.0 | 3.2, 3.3, or 3.4 | 2.5 and later | 8 | 2.12 | | 1.0.0 | 3.x | 1.18 and later | 8 | 2.12 | | 1.0.0 | 2.x | 1.18 and later | 8 | 2.11 | > **NOTICE** > > * Please see [Upgrade Spark connector](#upgrade-spark-connector) for behavior changes among different connector versions. > * The connector does not provide MySQL JDBC driver since version 1.1.1, and you need import the driver to the Spark classpath manually. You can find the driver on [Maven Central](https://repo1.maven.org/maven2/mysql/mysql-connector-java/). > * In version 1.0.0, the Spark connector only supports reading data from StarRocks. From version 1.1.0 onwards, the Spark connector supports both reading data from and writing data to StarRocks. > * Version 1.0.0 differs from version 1.1.0 in terms of parameters and data type mappings. See [Upgrade Spark connector](#upgrade-spark-connector). > * In general cases, no new features will be added to version 1.0.0. We recommend that you upgrade your Spark connector at your earliest opportunity. #### Obtain Spark connector[​](#obtain-spark-connector "Direct link to Obtain Spark connector") Use one of the following methods to obtain the Spark connector **.jar** package that suits your business needs: * Download a compiled package. * Use Maven to add the dependencies required by the Spark connector. (This method is supported only for Spark connector 1.1.0 and later.) * Manually compile a package. ##### Spark connector 1.1.0 and later[​](#spark-connector-110-and-later "Direct link to Spark connector 1.1.0 and later") Spark connector **.jar** packages are named in the following format: `starrocks-spark-connector-${spark_version}_${scala_version}-${connector_version}.jar` For example, if you want to use Spark connector 1.1.0 with Spark 3.2 and Scala 2.12, you can choose `starrocks-spark-connector-3.2_2.12-1.1.0.jar`. > **NOTICE** > > In normal cases, the latest Spark connector version can be used with the most recent three Spark versions. ###### Download a compiled package[​](#download-a-compiled-package "Direct link to Download a compiled package") You can obtain Spark connector **.jar** packages of various versions at [Maven Central Repository](https://repo1.maven.org/maven2/com/starrocks). ###### Add Maven dependencies[​](#add-maven-dependencies "Direct link to Add Maven dependencies") Configure the dependencies required by the Spark connector as follows: > **NOTICE** > > You must replace `spark_version`, `scala_version`, and `connector_version` with the Spark version, Scala version, and Spark connector version you use. ```xml com.starrocks starrocks-spark-connector-${spark_version}_${scala_version} ${connector_version} ``` For example, if you want to use Spark connector 1.1.0 with Spark 3.2 and Scala 2.12, configure the dependencies as follows: ```xml com.starrocks starrocks-spark-connector-3.2_2.12 1.1.0 ``` ###### Manually compile a package[​](#manually-compile-a-package "Direct link to Manually compile a package") 1. Download [the Spark connector code](https://github.com/StarRocks/starrocks-connector-for-apache-spark). 2. Use the following command to compile the Spark connector: > **NOTICE** > > You must replace `spark_version` with the Spark version you use. ```shell sh build.sh ``` For example, if you want to use the Spark connector with Spark 3.2, compile the Spark connector as follows: ```shell sh build.sh 3.2 ``` 3. Go to the `target/` path, in which a Spark connector **.jar** package like `starrocks-spark-connector-3.2_2.12-1.1.0-SNAPSHOT.jar` is generated upon compilation. > **NOTICE** > > If you are using a Spark connector version that is not officially released, the name of the generated Spark connector **.jar** package contains `SNAPSHOT` as a suffix. ##### Spark connector 1.0.0[​](#spark-connector-100 "Direct link to Spark connector 1.0.0") ###### Download a compiled package[​](#download-a-compiled-package-1 "Direct link to Download a compiled package") * [Spark 2.x](https://cdn-thirdparty.starrocks.com/spark/starrocks-spark2_2.11-1.0.0.jar) * [Spark 3.x](https://cdn-thirdparty.starrocks.com/spark/starrocks-spark3_2.12-1.0.0.jar) ###### Manually compile a package[​](#manually-compile-a-package-1 "Direct link to Manually compile a package") 1. Download [the Spark connector code](https://github.com/StarRocks/starrocks-connector-for-apache-spark/tree/spark-1.0). > **NOTICE** > > You must switch to `spark-1.0`. 2. Take one of the following actions to compile the Spark connector: * If you are using Spark 2.x, run the following command, which compiles the Spark connector to suit Spark 2.3.4 by default: ```plain sh build.sh 2 ``` * If you are using Spark 3.x, run the following command, which compiles the Spark connector to suit Spark 3.1.2 by default: ```plain sh build.sh 3 ``` 3. Go to the `output/` path, in which the `starrocks-spark2_2.11-1.0.0.jar` file is generated upon compilation. Then, copy the file to the classpath of Spark: * If your Spark cluster runs in `Local` mode, place the file into the `jars/` path. * If your Spark cluster runs in `Yarn` mode, place the file into the pre-deployment package. You can use the Spark connector to read data from StarRocks only after you place the file into the specified location. #### Parameters[​](#parameters "Direct link to Parameters") This section describes the parameters you need to configure when you use the Spark connector to read data from StarRocks. ##### Common parameters[​](#common-parameters "Direct link to Common parameters") The following parameters apply to all three reading methods: Spark SQL, Spark DataFrame, and Spark RDD. | Parameter | Default value | Description | | ------------------------------------ | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | starrocks.fenodes | None | The HTTP URL of the FE in your StarRocks cluster. Format `:`. You can specify multiple URLs, which must be separated by a comma (,). | | starrocks.table.identifier | None | The name of the StarRocks table. Format: `.`. | | starrocks.request.retries | 3 | The maximum number of times that Spark can retry to send a read request o StarRocks. | | starrocks.request.connect.timeout.ms | 30000 | The maximum amount of time after which a read request sent to StarRocks times out. | | starrocks.request.read.timeout.ms | 30000 | The maximum amount of time after which the reading for a request sent to StarRocks times out. | | starrocks.request.query.timeout.s | 3600 | The maximum amount of time after which a query of data from StarRocks times out. The default timeout period is 1 hour. `-1` means that no timeout period is specified. | | starrocks.request.tablet.size | Integer.MAX\_VALUE | The number of StarRocks tablets grouped into each Spark RDD partition. A smaller value of this parameter indicates that a larger number of Spark RDD partitions will be generated. A larger number of Spark RDD partitions means higher parallelism on Spark but greater pressure on StarRocks. | | starrocks.batch.size | 4096 | The maximum number of rows that can be read from BEs at a time. Increasing the value of this parameter can reduce the number of connections established between Spark and StarRocks, thereby mitigating extra time overheads caused by network latency. | | starrocks.exec.mem.limit | 2147483648 | The maximum amount of memory allowed per query. Unit: bytes. The default memory limit is 2 GB. | | starrocks.deserialize.arrow.async | false | Specifies whether to support asynchronously converting the Arrow memory format to RowBatches required for the iteration of the Spark connector. | | starrocks.deserialize.queue.size | 64 | The size of the internal queue that holds tasks for asynchronously converting the Arrow memory format to RowBatches. This parameter is valid when `starrocks.deserialize.arrow.async` is set to `true`. | | starrocks.filter.query | None | The condition based on which you want to filter data on StarRocks. You can specify multiple filter conditions, which must be joined by `and`. StarRocks filters the data from the StarRocks table based on the specified filter conditions before the data is read by Spark. | | starrocks.timezone | Default timezone of JVM | Supported since 1.1.1. The timezone used to convert StarRocks `DATETIME` to Spark `TimestampType`. The default is the timezone of JVM returned by `ZoneId#systemDefault()`. The format could be a timezone name such as `Asia/Shanghai`, or a zone offset such as `+08:00`. | ##### Parameters for Spark SQL and Spark DataFrame[​](#parameters-for-spark-sql-and-spark-dataframe "Direct link to Parameters for Spark SQL and Spark DataFrame") The following parameters apply only to the Spark SQL and Spark DataFrame reading methods. | Parameter | Default value | Description | | ----------------------------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | starrocks.fe.http.url | None | The HTTP IP address of the FE. This parameter is supported from Spark connector 1.1.0 onwards. This parameter is equivalent to `starrocks.fenodes`. You only need to configure one of them. In Spark connector 1.1.0 and later, we recommend that you use `starrocks.fe.http.url` because `starrocks.fenodes` may be deprecated. | | starrocks.fe.jdbc.url | None | The address that is used to connect to the MySQL server of the FE. Format: `jdbc:mysql://:`.
**NOTICE**
In Spark connector 1.1.0 and later, this parameter is mandatory. | | user | None | The username of your StarRocks cluster account. The user needs the [SELECT privilege](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) on the StarRocks table. | | starrocks.user | None | The username of your StarRocks cluster account. This parameter is supported from Spark connector 1.1.0 onwards. This parameter is equivalent to `user`. You only need to configure one of them. In Spark connector 1.1.0 and later, we recommend that you use `starrocks.user` because `user` may be deprecated. | | password | None | The password of your StarRocks cluster account. | | starrocks.password | None | The password of your StarRocks cluster account. This parameter is supported from Spark connector 1.1.0 onwards. This parameter is equivalent to `password`. You only need to configure one of them. In Spark connector 1.1.0 and later, we recommend that you use `starrocks.password` because `password` may be deprecated. | | starrocks.filter.query.in.max.count | 100 | The maximum number of values supported by the IN expression during predicate pushdown. If the number of values specified in the IN expression exceeds this limit, the filter conditions specified in the IN expression are processed on Spark. | ##### Parameters for Spark RDD[​](#parameters-for-spark-rdd "Direct link to Parameters for Spark RDD") The following parameters apply only to the Spark RDD reading method. | Parameter | Default value | Description | | ------------------------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | starrocks.request.auth.user | None | The username of your StarRocks cluster account. | | starrocks.request.auth.password | None | The password of your StarRocks cluster account. | | starrocks.read.field | None | The StarRocks table column from which you want to read data. You can specify multiple columns, which must be separated by a comma (,). | #### Data type mapping between StarRocks and Spark[​](#data-type-mapping-between-starrocks-and-spark "Direct link to Data type mapping between StarRocks and Spark") ##### Spark connector 1.1.0 and later[​](#spark-connector-110-and-later-1 "Direct link to Spark connector 1.1.0 and later") | StarRocks data type | Spark data type | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | BOOLEAN | DataTypes.BooleanType | | TINYINT | DataTypes.ByteType | | SMALLINT | DataTypes.ShortType | | INT | DataTypes.IntegerType | | BIGINT | DataTypes.LongType | | LARGEINT | DataTypes.StringType | | FLOAT | DataTypes.FloatType | | DOUBLE | DataTypes.DoubleType | | DECIMAL | DecimalType | | CHAR | DataTypes.StringType | | VARCHAR | DataTypes.StringType | | STRING | DataTypes.StringType | | DATE | DataTypes.DateType | | DATETIME | DataTypes.TimestampType | | JSON | DataTypes.StringType
**NOTE:**
This data type mapping is supported since Spark connector v1.1.2, and requires a StarRocks version of at least 2.5.13, 3.0.3, 3.1.0 or later. | | ARRAY | ArrayType
**NOTE:**
This data type mapping is supported since Spark connector v1.1.3. Nested types must be declared via `starrocks.column.types`. See [Read nested columns](#read-nested-columns-struct-array-and-map). | | HLL | Unsupported datatype | | BITMAP | Unsupported datatype | | MAP | MapType
**NOTE:**
This data type mapping is supported since Spark connector v1.1.3. Nested types must be declared via `starrocks.column.types`. See [Read nested columns](#read-nested-columns-struct-array-and-map). | | STRUCT | StructType
**NOTE:**
This data type mapping is supported since Spark connector v1.1.3. Nested types must be declared via `starrocks.column.types`. See [Read nested columns](#read-nested-columns-struct-array-and-map). | ##### Spark connector 1.0.0[​](#spark-connector-100-1 "Direct link to Spark connector 1.0.0") | StarRocks data type | Spark data type | | ------------------- | --------------------- | | BOOLEAN | DataTypes.BooleanType | | TINYINT | DataTypes.ByteType | | SMALLINT | DataTypes.ShortType | | INT | DataTypes.IntegerType | | BIGINT | DataTypes.LongType | | LARGEINT | DataTypes.StringType | | FLOAT | DataTypes.FloatType | | DOUBLE | DataTypes.DoubleType | | DECIMAL | DecimalType | | CHAR | DataTypes.StringType | | VARCHAR | DataTypes.StringType | | DATE | DataTypes.StringType | | DATETIME | DataTypes.StringType | | ARRAY | Unsupported datatype | | HLL | Unsupported datatype | | BITMAP | Unsupported datatype | The processing logic of the underlying storage engine used by StarRocks cannot cover an expected time range when DATE and DATETIME data types are directly used. Therefore, the Spark connector maps the DATE and DATETIME data types from StarRocks to the STRING data type from Spark, and generates readable string texts matching the date and time data read from StarRocks. #### Upgrade Spark connector[​](#upgrade-spark-connector "Direct link to Upgrade Spark connector") ##### Upgrade from version 1.0.0 to version 1.1.0[​](#upgrade-from-version-100-to-version-110 "Direct link to Upgrade from version 1.0.0 to version 1.1.0") * Since 1.1.1, the Spark connector does not provide `mysql-connector-java` which is the official JDBC driver for MySQL, because of the limitations of the GPL license used by `mysql-connector-java`. However, the Spark connector still needs the `mysql-connector-java` to connect to StarRocks for the table metadata, so you need to add the driver to the Spark classpath manually. You can find the driver on [MySQL site](https://dev.mysql.com/downloads/connector/j/) or [Maven Central](https://repo1.maven.org/maven2/mysql/mysql-connector-java/). * In version 1.1.0, the Spark connector uses JDBC to access StarRocks to obtain more detailed table information. Therefore, you must configure `starrocks.fe.jdbc.url`. * In version 1.1.0, some parameters are renamed. Both the old and new parameters are retained for now. For each pair of equivalent parameters, you only need to configure one of them, but we recommend that you use the new one because the old one may be deprecated. * `starrocks.fenodes` is renamed as `starrocks.fe.http.url`. * `user` is renamed as `starrocks.user`. * `password` is renamed as `starrocks.password`. * In version 1.1.0, the mappings of some data types are adjusted based on Spark 3.x: * `DATE` in StarRocks is mapped to `DataTypes.DateType` (originally `DataTypes.StringType`) in Spark. * `DATETIME` in StarRocks is mapped to `DataTypes.TimestampType` (originally `DataTypes.StringType`) in Spark. #### Examples[​](#examples "Direct link to Examples") The following examples assume you have created a database named `test` in your StarRocks cluster and you have the permissions of user `root`. The parameter settings in the examples are based on Spark Connector 1.1.0. ##### Network configuration[​](#network-configuration "Direct link to Network configuration") Ensure that the machine where Spark is located can access the FE nodes of the StarRocks cluster via the `http_port` (default: `8030`) and `query_port` (default: `9030`), and the BE nodes via the `be_port` (default: `9060`). ##### Data example[​](#data-example "Direct link to Data example") Do as follows to prepare a sample table: 1. Go to the `test` database and create a table named `score_board`. ```sql MySQL [test]> CREATE TABLE `score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`) PROPERTIES ( "replication_num" = "3" ); ``` 2. Insert data into the `score_board` table. ```sql MySQL [test]> INSERT INTO score_board VALUES (1, 'Bob', 21), (2, 'Stan', 21), (3, 'Sam', 22), (4, 'Tony', 22), (5, 'Alice', 22), (6, 'Lucy', 23), (7, 'Polly', 23), (8, 'Tom', 23), (9, 'Rose', 24), (10, 'Jerry', 24), (11, 'Jason', 24), (12, 'Lily', 25), (13, 'Stephen', 25), (14, 'David', 25), (15, 'Eddie', 26), (16, 'Kate', 27), (17, 'Cathy', 27), (18, 'Judy', 27), (19, 'Julia', 28), (20, 'Robert', 28), (21, 'Jack', 29); ``` 3. Query the `score_board` table. ```sql MySQL [test]> SELECT * FROM score_board; +------+---------+-------+ | id | name | score | +------+---------+-------+ | 1 | Bob | 21 | | 2 | Stan | 21 | | 3 | Sam | 22 | | 4 | Tony | 22 | | 5 | Alice | 22 | | 6 | Lucy | 23 | | 7 | Polly | 23 | | 8 | Tom | 23 | | 9 | Rose | 24 | | 10 | Jerry | 24 | | 11 | Jason | 24 | | 12 | Lily | 25 | | 13 | Stephen | 25 | | 14 | David | 25 | | 15 | Eddie | 26 | | 16 | Kate | 27 | | 17 | Cathy | 27 | | 18 | Judy | 27 | | 19 | Julia | 28 | | 20 | Robert | 28 | | 21 | Jack | 29 | +------+---------+-------+ 21 rows in set (0.01 sec) ``` ##### Read data using Spark SQL[​](#read-data-using-spark-sql "Direct link to Read data using Spark SQL") 1. Run the following command in the Spark directory to start Spark SQL: ```plain sh spark-sql ``` 2. Run the following command to create a temporary view named `spark_starrocks` on the `score_board` table which belongs to the `test` database: ```sql spark-sql> CREATE TEMPORARY VIEW spark_starrocks USING starrocks OPTIONS ( "starrocks.table.identifier" = "test.score_board", "starrocks.fe.http.url" = ":", "starrocks.fe.jdbc.url" = "jdbc:mysql://:", "starrocks.user" = "root", "starrocks.password" = "" ); ``` 3. Run the following command to read data from the temporary view: ```sql spark-sql> SELECT * FROM spark_starrocks; ``` Spark returns the following data: ```sql 1 Bob 21 2 Stan 21 3 Sam 22 4 Tony 22 5 Alice 22 6 Lucy 23 7 Polly 23 8 Tom 23 9 Rose 24 10 Jerry 24 11 Jason 24 12 Lily 25 13 Stephen 25 14 David 25 15 Eddie 26 16 Kate 27 17 Cathy 27 18 Judy 27 19 Julia 28 20 Robert 28 21 Jack 29 Time taken: 1.883 seconds, Fetched 21 row(s) 22/08/09 15:29:36 INFO thriftserver.SparkSQLCLIDriver: Time taken: 1.883 seconds, Fetched 21 row(s) ``` ##### Read data using Spark DataFrame[​](#read-data-using-spark-dataframe "Direct link to Read data using Spark DataFrame") 1. Run the following command in the Spark directory to start Spark Shell: ```plain sh spark-shell ``` 2. Run the following command to create a DataFrame named `starrocksSparkDF` on the `score_board` table which belongs to the `test` database: ```scala scala> val starrocksSparkDF = spark.read.format("starrocks") .option("starrocks.table.identifier", s"test.score_board") .option("starrocks.fe.http.url", s":") .option("starrocks.fe.jdbc.url", s"jdbc:mysql://:") .option("starrocks.user", s"root") .option("starrocks.password", s"") .load() ``` 3. Read data from the DataFrame. For example, if you want to read the first 10 rows, run the following command: ```scala scala> starrocksSparkDF.show(10) ``` Spark returns the following data: ```scala +---+-----+-----+ | id| name|score| +---+-----+-----+ | 1| Bob| 21| | 2| Stan| 21| | 3| Sam| 22| | 4| Tony| 22| | 5|Alice| 22| | 6| Lucy| 23| | 7|Polly| 23| | 8| Tom| 23| | 9| Rose| 24| | 10|Jerry| 24| +---+-----+-----+ only showing top 10 rows ``` > **NOTE** > > By default, if you do not specify the number of rows you want to read, Spark returns the first 20 rows. ##### Read data using Spark RDD[​](#read-data-using-spark-rdd "Direct link to Read data using Spark RDD") 1. Run the following command in the Spark directory to start Spark Shell: ```plain sh spark-shell ``` 2. Run the following command to create an RDD named `starrocksSparkRDD` on the `score_board` table which belongs to the `test` database. ```scala scala> import com.starrocks.connector.spark._ scala> val starrocksSparkRDD = sc.starrocksRDD ( tableIdentifier = Some("test.score_board"), cfg = Some(Map( "starrocks.fenodes" -> ":", "starrocks.request.auth.user" -> "root", "starrocks.request.auth.password" -> "" )) ) ``` 3. Read data from the RDD. For example, if you want to read the first 10 elements, run the following command: ```scala scala> starrocksSparkRDD.take(10) ``` Spark returns the following data: ```scala res0: Array[AnyRef] = Array([1, Bob, 21], [2, Stan, 21], [3, Sam, 22], [4, Tony, 22], [5, Alice, 22], [6, Lucy, 23], [7, Polly, 23], [8, Tom, 23], [9, Rose, 24], [10, Jerry, 24]) ``` To read the entire RDD, run the following command: ```scala scala> starrocksSparkRDD.collect() ``` Spark returns the following data: ```scala res1: Array[AnyRef] = Array([1, Bob, 21], [2, Stan, 21], [3, Sam, 22], [4, Tony, 22], [5, Alice, 22], [6, Lucy, 23], [7, Polly, 23], [8, Tom, 23], [9, Rose, 24], [10, Jerry, 24], [11, Jason, 24], [12, Lily, 25], [13, Stephen, 25], [14, David, 25], [15, Eddie, 26], [16, Kate, 27], [17, Cathy, 27], [18, Judy, 27], [19, Julia, 28], [20, Robert, 28], [21, Jack, 29]) ``` #### Best practices[​](#best-practices "Direct link to Best practices") When you read data from StarRocks using the Spark connector, you can use the `starrocks.filter.query` parameter to specify filter conditions based on which Spark prunes partitions, buckets, and prefix indexes to reduce the cost of data pulling. This section uses Spark DataFrame as an example to show how this is achieved. ##### Environment setup[​](#environment-setup "Direct link to Environment setup") | Component | Version | | --------------- | ------------------------------------------------------------------------- | | Spark | Spark 2.4.4 and Scala 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0\_302) | | StarRocks | 2.2.0 | | Spark connector | starrocks-spark2\_2.11-1.0.0.jar | ##### Data example[​](#data-example-1 "Direct link to Data example") Do as follows to prepare a sample table: 1. Go to the `test` database and create a table named `mytable`. ```sql MySQL [test]> CREATE TABLE `mytable` ( `k` int(11) NULL COMMENT "bucket", `b` int(11) NULL COMMENT "", `dt` datetime NULL COMMENT "", `v` int(11) NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`k`,`b`, `dt`) COMMENT "OLAP" PARTITION BY RANGE(`dt`) ( PARTITION p202201 VALUES [('2022-01-01 00:00:00'), ('2022-02-01 00:00:00')), PARTITION p202202 VALUES [('2022-02-01 00:00:00'), ('2022-03-01 00:00:00')), PARTITION p202203 VALUES [('2022-03-01 00:00:00'), ('2022-04-01 00:00:00')) ) DISTRIBUTED BY HASH(`k`) PROPERTIES ( "replication_num" = "3" ); ``` 2. Insert data into `mytable`. ```sql MySQL [test]> INSERT INTO mytable VALUES (1, 11, '2022-01-02 08:00:00', 111), (2, 22, '2022-02-02 08:00:00', 222), (3, 33, '2022-03-02 08:00:00', 333); ``` 3. Query the `mytable` table. ```sql MySQL [test]> select * from mytable; +------+------+---------------------+------+ | k | b | dt | v | +------+------+---------------------+------+ | 1 | 11 | 2022-01-02 08:00:00 | 111 | | 2 | 22 | 2022-02-02 08:00:00 | 222 | | 3 | 33 | 2022-03-02 08:00:00 | 333 | +------+------+---------------------+------+ 3 rows in set (0.01 sec) ``` ##### Full table scan[​](#full-table-scan "Direct link to Full table scan") 1. Run the following command in the Spark directory to create a DataFrame named `df` on `mytable` table which belongs to the `test` database: ```scala scala> val df = spark.read.format("starrocks") .option("starrocks.table.identifier", s"test.mytable") .option("starrocks.fenodes", s":") .option("user", s"root") .option("password", s"") .load() ``` 2. View the FE log file **fe.log** of your StarRocks cluster, and find the SQL statement executed to read data. Example: ```sql 2022-08-09 18:57:38,091 INFO (nioEventLoopGroup-3-10|196) [TableQueryPlanAction.executeWithoutPassword():126] receive SQL statement [select `k`,`b`,`dt`,`v` from `test`.`mytable`] from external service [ user ['root'@'%']] for database [test] table [mytable] ``` 3. In the `test` database, use EXPLAIN to obtain the execution plan of the SELECT `k`,`b`,`dt`,`v` from `test`.`mytable` statement: ```scala MySQL [test]> EXPLAIN select `k`,`b`,`dt`,`v` from `test`.`mytable`; +-----------------------------------------------------------------------+ | Explain String | +-----------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS:1: k | 2: b | 3: dt | 4: v | | PARTITION: UNPARTITIONED | | | | RESULT SINK | | | | 1:EXCHANGE | | | | PLAN FRAGMENT 1 | | OUTPUT EXPRS: | | PARTITION: RANDOM | | | | STREAM DATA SINK | | EXCHANGE ID: 01 | | UNPARTITIONED | | | | 0:OlapScanNode | | TABLE: mytable | | PREAGGREGATION: ON | | partitions=3/3 | | rollup: mytable | | tabletRatio=9/9 | | tabletList=41297,41299,41301,41303,41305,41307,41309,41311,41313 | | cardinality=3 | | avgRowSize=4.0 | | numNodes=0 | +-----------------------------------------------------------------------+ 26 rows in set (0.00 sec) ``` In this example, no pruning is performed. Therefore, Spark scans all of the three partitions (as suggested by `partitions=3/3`) that hold data, and scans all of the 9 tablets (as suggested by `tabletRatio=9/9`) in those three partitions. ##### Partition pruning[​](#partition-pruning "Direct link to Partition pruning") 1. Run the following command, in which you use the `starrocks.filter.query` parameter to specify a filter condition `dt='2022-01-02 08:00:00` for partition pruning, in the Spark directory to create a DataFrame named `df` on the `mytable` table which belongs to the `test` database: ```scala scala> val df = spark.read.format("starrocks") .option("starrocks.table.identifier", s"test.mytable") .option("starrocks.fenodes", s":") .option("user", s"root") .option("password", s"") .option("starrocks.filter.query", "dt='2022-01-02 08:00:00'") .load() ``` 2. View the FE log file **fe.log** of your StarRocks cluster, and find the SQL statement executed to read data. Example: ```sql 2022-08-09 19:02:31,253 INFO (nioEventLoopGroup-3-14|204) [TableQueryPlanAction.executeWithoutPassword():126] receive SQL statement [select `k`,`b`,`dt`,`v` from `test`.`mytable` where dt='2022-01-02 08:00:00'] from external service [ user ['root'@'%']] for database [test] table [mytable] ``` 3. In the `test` database, use EXPLAIN to obtain the execution plan of the SELECT `k`,`b`,`dt`,`v` from `test`.`mytable` where dt='2022-01-02 08:00:00' statement: ```scala MySQL [test]> EXPLAIN select `k`,`b`,`dt`,`v` from `test`.`mytable` where dt='2022-01-02 08:00:00'; +------------------------------------------------+ | Explain String | +------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS:1: k | 2: b | 3: dt | 4: v | | PARTITION: UNPARTITIONED | | | | RESULT SINK | | | | 1:EXCHANGE | | | | PLAN FRAGMENT 1 | | OUTPUT EXPRS: | | PARTITION: RANDOM | | | | STREAM DATA SINK | | EXCHANGE ID: 01 | | UNPARTITIONED | | | | 0:OlapScanNode | | TABLE: mytable | | PREAGGREGATION: ON | | PREDICATES: 3: dt = '2022-01-02 08:00:00' | | partitions=1/3 | | rollup: mytable | | tabletRatio=3/3 | | tabletList=41297,41299,41301 | | cardinality=1 | | avgRowSize=20.0 | | numNodes=0 | +------------------------------------------------+ 27 rows in set (0.01 sec) ``` In this example, only partition pruning is performed, whereas bucket pruning is not. Therefore, Spark scans one of the three partitions (as suggested by `partitions=1/3`) and all of the tablets (as suggested by `tabletRatio=3/3`) in that partition. ##### Bucket pruning[​](#bucket-pruning "Direct link to Bucket pruning") 1. Run the following command, in which you use the `starrocks.filter.query` parameter to specify a filter condition `k=1` for bucket pruning, in the Spark directory to create a DataFrame named `df` on the `mytable` table which belongs to the `test` database: ```scala scala> val df = spark.read.format("starrocks") .option("starrocks.table.identifier", s"test.mytable") .option("starrocks.fenodes", s":") .option("user", s"root") .option("password", s"") .option("starrocks.filter.query", "k=1") .load() ``` 2. View the FE log file **fe.log** of your StarRocks cluster, and find the SQL statement executed to read data. Example: ```sql 2022-08-09 19:04:44,479 INFO (nioEventLoopGroup-3-16|208) [TableQueryPlanAction.executeWithoutPassword():126] receive SQL statement [select `k`,`b`,`dt`,`v` from `test`.`mytable` where k=1] from external service [ user ['root'@'%']] for database [test] table [mytable] ``` 3. In the `test` database, use EXPLAIN to obtain the execution plan of the SELECT `k`,`b`,`dt`,`v` from `test`.`mytable` where k=1 statement: ```scala MySQL [test]> EXPLAIN select `k`,`b`,`dt`,`v` from `test`.`mytable` where k=1; +------------------------------------------+ | Explain String | +------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS:1: k | 2: b | 3: dt | 4: v | | PARTITION: UNPARTITIONED | | | | RESULT SINK | | | | 1:EXCHANGE | | | | PLAN FRAGMENT 1 | | OUTPUT EXPRS: | | PARTITION: RANDOM | | | | STREAM DATA SINK | | EXCHANGE ID: 01 | | UNPARTITIONED | | | | 0:OlapScanNode | | TABLE: mytable | | PREAGGREGATION: ON | | PREDICATES: 1: k = 1 | | partitions=3/3 | | rollup: mytable | | tabletRatio=3/9 | | tabletList=41299,41305,41311 | | cardinality=1 | | avgRowSize=20.0 | | numNodes=0 | +------------------------------------------+ 27 rows in set (0.01 sec) ``` In this example, only bucket pruning is performed, whereas partition pruning is not. Therefore, Spark scans all of the three partitions (as suggested by `partitions=3/3`) that hold data, and scans all of the three tablets (as suggested by `tabletRatio=3/9`) to retrieve Hash values that meet the `k = 1` filter condition within those three partitions. ##### Partition pruning and bucket pruning[​](#partition-pruning-and-bucket-pruning "Direct link to Partition pruning and bucket pruning") 1. Run the following command, in which you use the `starrocks.filter.query` parameter to specify two filter conditions `k=7` and `dt='2022-01-02 08:00:00'` for bucket pruning and partition pruning, in the Spark directory to create a DataFrame named `df` on the `mytable` table on the `test` database: ```scala scala> val df = spark.read.format("starrocks") .option("starrocks.table.identifier", s"test.mytable") .option("starrocks.fenodes", s":") .option("user", s"") .option("password", s"") .option("starrocks.filter.query", "k=7 and dt='2022-01-02 08:00:00'") .load() ``` 2. View the FE log file **fe.log** of your StarRocks cluster, and find the SQL statement executed to read data. Example: ```sql 2022-08-09 19:06:34,939 INFO (nioEventLoopGroup-3-18|212) [TableQueryPlanAction.executeWithoutPassword():126] receive SQL statement [select `k`,`b`,`dt`,`v` from `test`.`mytable` where k=7 and dt='2022-01-02 08:00:00'] from external service [ user ['root'@'%']] for database [test] t able [mytable] ``` 3. In the `test` database, use EXPLAIN to obtain the execution plan of the SELECT `k`,`b`,`dt`,`v` from `test`.`mytable` where k=7 and dt='2022-01-02 08:00:00' statement: ```scala MySQL [test]> EXPLAIN select `k`,`b`,`dt`,`v` from `test`.`mytable` where k=7 and dt='2022-01-02 08:00:00'; +----------------------------------------------------------+ | Explain String | +----------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS:1: k | 2: b | 3: dt | 4: v | | PARTITION: RANDOM | | | | RESULT SINK | | | | 0:OlapScanNode | | TABLE: mytable | | PREAGGREGATION: ON | | PREDICATES: 1: k = 7, 3: dt = '2022-01-02 08:00:00' | | partitions=1/3 | | rollup: mytable | | tabletRatio=1/3 | | tabletList=41301 | | cardinality=1 | | avgRowSize=20.0 | | numNodes=0 | +----------------------------------------------------------+ 17 rows in set (0.00 sec) ``` In this example, both partition pruning and bucket pruning are performed. Therefore, Spark scans only one of the three partitions (as suggested by `partitions=1/3`) and only one tablet (as suggested by `tabletRatio=1/3`) in that partition. ##### Prefix index filtering[​](#prefix-index-filtering "Direct link to Prefix index filtering") 1. Insert more data records into a partition of the `mytable` table which belongs to the `test` database: ```scala MySQL [test]> INSERT INTO mytable VALUES (1, 11, "2022-01-02 08:00:00", 111), (3, 33, "2022-01-02 08:00:00", 333), (3, 33, "2022-01-02 08:00:00", 333), (3, 33, "2022-01-02 08:00:00", 333); ``` 2. Query the `mytable` table: ```scala MySQL [test]> SELECT * FROM mytable; +------+------+---------------------+------+ | k | b | dt | v | +------+------+---------------------+------+ | 1 | 11 | 2022-01-02 08:00:00 | 111 | | 3 | 33 | 2022-01-02 08:00:00 | 333 | | 3 | 33 | 2022-01-02 08:00:00 | 333 | | 3 | 33 | 2022-01-02 08:00:00 | 333 | +------+------+---------------------+------+ 4 rows in set (0.01 sec) ``` 3. Run the following command, in which you use the `starrocks.filter.query` parameter to specify a filter condition `k=1` for prefix index filtering, in the Spark directory to create a DataFrame named `df` on the `mytable` table which belongs to the `test` database: ```scala scala> val df = spark.read.format("starrocks") .option("starrocks.table.identifier", s"test.mytable") .option("starrocks.fenodes", s":") .option("user", s"root") .option("password", s"") .option("starrocks.filter.query", "k=1") .load() ``` 4. In the `test` database, set `is_report_success` to `true` to enable profile reporting: ```sql MySQL [test]> SET is_report_success = true; Query OK, 0 rows affected (0.00 sec) ``` 5. Use a browser to open the `http://:/query` page, and view the profile of the SELECT \* FROM mytable where k=1 statement. Example: ```sql OLAP_SCAN (plan_node_id=0): CommonMetrics: - CloseTime: 1.255ms - OperatorTotalTime: 1.404ms - PeakMemoryUsage: 0.00 - PullChunkNum: 8 - PullRowNum: 2 - __MAX_OF_PullRowNum: 2 - __MIN_OF_PullRowNum: 0 - PullTotalTime: 148.60us - PushChunkNum: 0 - PushRowNum: 0 - PushTotalTime: 0ns - SetFinishedTime: 136ns - SetFinishingTime: 129ns UniqueMetrics: - Predicates: 1: k = 1 - Rollup: mytable - Table: mytable - BytesRead: 88.00 B - __MAX_OF_BytesRead: 88.00 B - __MIN_OF_BytesRead: 0.00 - CachedPagesNum: 0 - CompressedBytesRead: 844.00 B - __MAX_OF_CompressedBytesRead: 844.00 B - __MIN_OF_CompressedBytesRead: 0.00 - CreateSegmentIter: 18.582us - IOTime: 4.425us - LateMaterialize: 17.385us - PushdownPredicates: 3 - RawRowsRead: 2 - __MAX_OF_RawRowsRead: 2 - __MIN_OF_RawRowsRead: 0 - ReadPagesNum: 12 - __MAX_OF_ReadPagesNum: 12 - __MIN_OF_ReadPagesNum: 0 - RowsRead: 2 - __MAX_OF_RowsRead: 2 - __MIN_OF_RowsRead: 0 - ScanTime: 154.367us - SegmentInit: 95.903us - BitmapIndexFilter: 0ns - BitmapIndexFilterRows: 0 - BloomFilterFilterRows: 0 - ShortKeyFilterRows: 3 - __MAX_OF_ShortKeyFilterRows: 3 - __MIN_OF_ShortKeyFilterRows: 0 - ZoneMapIndexFilterRows: 0 - SegmentRead: 2.559us - BlockFetch: 2.187us - BlockFetchCount: 2 - __MAX_OF_BlockFetchCount: 2 - __MIN_OF_BlockFetchCount: 0 - BlockSeek: 7.789us - BlockSeekCount: 2 - __MAX_OF_BlockSeekCount: 2 - __MIN_OF_BlockSeekCount: 0 - ChunkCopy: 25ns - DecompressT: 0ns - DelVecFilterRows: 0 - IndexLoad: 0ns - PredFilter: 353ns - PredFilterRows: 0 - RowsetsReadCount: 7 - SegmentsReadCount: 3 - __MAX_OF_SegmentsReadCount: 2 - __MIN_OF_SegmentsReadCount: 0 - TotalColumnsDataPageCount: 8 - __MAX_OF_TotalColumnsDataPageCount: 8 - __MIN_OF_TotalColumnsDataPageCount: 0 - UncompressedBytesRead: 508.00 B - __MAX_OF_UncompressedBytesRead: 508.00 B - __MIN_OF_UncompressedBytesRead: 0.00 ``` In this example, the filter condition `k = 1` can hit the prefix index. Therefore, Spark can filter out three rows (as suggested by `ShortKeyFilterRows: 3`). #### Read nested columns (STRUCT, ARRAY, and MAP)[​](#read-nested-columns-struct-array-and-map "Direct link to Read nested columns (STRUCT, ARRAY, and MAP)") Reading nested columns is supported since version 1.1.3. The Spark connector supports reading StarRocks columns of type `STRUCT`, `ARRAY`, and `MAP`. Because the connector cannot automatically infer the full nested type from the StarRocks schema, you must declare the column types explicitly using the `starrocks.column.types` option for every nested column. ##### Nested data type mapping[​](#nested-data-type-mapping "Direct link to Nested data type mapping") | StarRocks data type | Spark data type | | ----------------------------------------- | --------------- | | `STRUCT` | `StructType` | | `ARRAY` | `ArrayType` | | `MAP` | `MapType` | Nested types can be composed arbitrarily (for example, `STRUCT, b MAP>`). ##### Type-mapping caveats inside nested types[​](#type-mapping-caveats-inside-nested-types "Direct link to Type-mapping caveats inside nested types") Logical StarRocks types that share an Arrow wire representation (notably `DATE`, `DATETIME`, and some `DECIMAL` encodings) are decoded correctly only when the declared type is provided via `starrocks.column.types`. Without it, those fields fall back to plain `STRING`. ##### Example[​](#example "Direct link to Example") Given the following StarRocks table: ```sql CREATE TABLE nested_tbl ( id INT, info STRUCT, tags ARRAY, metadata MAP> ) ENGINE=OLAP DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 4; ``` Read the table with Spark: ```scala val columnTypes = "info STRUCT, " + "tags ARRAY, " + "metadata MAP>" val df = spark.read.format("starrocks") .option("starrocks.fenodes", "127.0.0.1:8030") .option("starrocks.table.identifier", "test.nested_tbl") .option("starrocks.user", "root") .option("starrocks.password", "") .option("starrocks.column.types", columnTypes) .load() df.printSchema() // root // |-- id: integer // |-- info: struct // |-- tags: array // |-- metadata: map> ``` ##### Current limitations[​](#current-limitations "Direct link to Current limitations") * Nested type inference is **not** automatic. You must supply `starrocks.column.types` for every nested column. * `DATE` and `DATETIME` fields inside nested types are returned as Spark `DateType` / `TimestampType` only when the declared type is provided. Without the declaration they fall back to `StringType`. --- ### Unload data using INSERT INTO FILES This topic describes how to unload data from StarRocks into remote storage by using INSERT INTO FILES. From v3.2 onwards, StarRocks supports using the table function FILES() to define a writable file in remote storage. You can then combine FILES() with the INSERT statement to unload data from StarRocks to your remote storage. Compared to other data export methods supported by StarRocks, unloading data with INSERT INTO FILES provides a more unified, easy-to-use interface. You can unload the data directly into your remote storage using the same syntax that you used to load data from it. Moreover, this method supports storing the data files in different storage paths by extracting the values of a specified column, allowing you to manage the exported data in a partitioned layout. > **NOTE** > > Please note that unloading data with INSERT INTO FILES does not support directly exporting data into local file systems. However, you can export the data into local files by using NFS. See [Unload to local files using NFS](#unload-to-local-files-using-nfs). #### Preparation[​](#preparation "Direct link to Preparation") The following example creates a database `unload` and a table `sales_records` as the data objects that can be used in the following tutorial. You can use your own data instead. ```sql CREATE DATABASE unload; USE unload; CREATE TABLE sales_records( record_id BIGINT, seller STRING, store_id INT, sales_time DATETIME, sales_amt DOUBLE ) DUPLICATE KEY(record_id) PARTITION BY date_trunc('day', sales_time) DISTRIBUTED BY HASH(record_id); INSERT INTO sales_records VALUES (220313001,"Amy",1,"2022-03-13 12:00:00",8573.25), (220314002,"Bob",2,"2022-03-14 12:00:00",6948.99), (220314003,"Amy",1,"2022-03-14 12:00:00",4319.01), (220315004,"Carl",3,"2022-03-15 12:00:00",8734.26), (220316005,"Carl",3,"2022-03-16 12:00:00",4212.69), (220317006,"Bob",2,"2022-03-17 12:00:00",9515.88); ``` The table `sales_records` contains the transaction ID `record_id`, salesperson `seller`, store ID `store_id`, time `sales_time`, and sales amount `sales_amt` for each transaction. It is partitioned on a daily basis in accordance with `sales_time`. You also need to prepare a remote storage system with write access. The following examples export data to the following remote storages: * An HDFS cluster with the simple authentication method enabled. * An AWS S3 bucket using the IAM User credential. For more about the remote storage systems and credential methods supported by FILES(), see [SQL reference - FILES()](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md). #### Unload data[​](#unload-data "Direct link to Unload data") INSERT INTO FILES supports unloading data into a single file or multiple files. You can further partition these data files by specifying separate storage paths for them. When unloading data using INSERT INTO FILES, you must manually set the compression algorithm using the property `compression`. For more information on the data compression algorithm supported by FILES, see [unload\_data\_param](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md#unload_data_param). ##### Unload data into multiple files[​](#unload-data-into-multiple-files "Direct link to Unload data into multiple files") By default, INSERT INTO FILES unloads data into multiple data files, each with a size of 1 GB. You can configure the file size using the property `target_max_file_size` (Unit: Bytes). The following example unloads all data rows in `sales_records` as multiple Parquet files prefixed by `data1`. The size of each file is 1 KB. note Here, setting `target_max_file_size` to 1 KB is to demonstrate unloading into multiple files with a small dataset. In production environment, you are strongly advised to set this value within the range of hundreds of MB to multiple GB. * **To S3**: ```sql INSERT INTO FILES( "path" = "s3://mybucket/unload/data1", "format" = "parquet", "compression" = "uncompressed", "target_max_file_size" = "1024", -- 1KB "aws.s3.access_key" = "xxxxxxxxxx", "aws.s3.secret_key" = "yyyyyyyyyy", "aws.s3.region" = "us-west-2" ) SELECT * FROM sales_records; ``` * **To HDFS**: ```sql INSERT INTO FILES( "path" = "hdfs://xxx.xx.xxx.xx:9000/unload/data1", "format" = "parquet", "compression" = "uncompressed", "target_max_file_size" = "1024", -- 1KB "hadoop.security.authentication" = "simple", "username" = "xxxxx", "password" = "xxxxx" ) SELECT * FROM sales_records; ``` ##### Unload data into multiple files under different paths[​](#unload-data-into-multiple-files-under-different-paths "Direct link to Unload data into multiple files under different paths") You can also partition the data files in different storage paths by extracting the values of the specified column using the property `partition_by`. The following example unloads all data rows in `sales_records` as multiple Parquet files under the path **/unload/partitioned/**. These files are stored in different subpaths distinguished by the values in the column `sales_time`. * **To S3**: ```sql INSERT INTO FILES( "path" = "s3://mybucket/unload/partitioned/", "format" = "parquet", "compression" = "lz4", "partition_by" = "sales_time", "aws.s3.access_key" = "xxxxxxxxxx", "aws.s3.secret_key" = "yyyyyyyyyy", "aws.s3.region" = "us-west-2" ) SELECT * FROM sales_records; ``` * **To HDFS**: ```sql INSERT INTO FILES( "path" = "hdfs://xxx.xx.xxx.xx:9000/unload/partitioned/", "format" = "parquet", "compression" = "lz4", "partition_by" = "sales_time", "hadoop.security.authentication" = "simple", "username" = "xxxxx", "password" = "xxxxx" ) SELECT * FROM sales_records; ``` ##### Unload data into a single file[​](#unload-data-into-a-single-file "Direct link to Unload data into a single file") To unload data into a single data file, you must specify the property `single` as `true`. The following example unloads all data rows in `sales_records` as a single Parquet file prefixed by `data2`. * **To S3**: ```sql INSERT INTO FILES( "path" = "s3://mybucket/unload/data2", "format" = "parquet", "compression" = "lz4", "single" = "true", "aws.s3.access_key" = "xxxxxxxxxx", "aws.s3.secret_key" = "yyyyyyyyyy", "aws.s3.region" = "us-west-2" ) SELECT * FROM sales_records; ``` * **To HDFS**: ```sql INSERT INTO FILES( "path" = "hdfs://xxx.xx.xxx.xx:9000/unload/data2", "format" = "parquet", "compression" = "lz4", "single" = "true", "hadoop.security.authentication" = "simple", "username" = "xxxxx", "password" = "xxxxx" ) SELECT * FROM sales_records; ``` ##### Unload to MinIO[​](#unload-to-minio "Direct link to Unload to MinIO") The parameters used for MinIO are different from those used for AWS S3. Example: ```sql INSERT INTO FILES( "path" = "s3://huditest/unload/data3", "format" = "parquet", "compression" = "zstd", "single" = "true", "aws.s3.access_key" = "xxxxxxxxxx", "aws.s3.secret_key" = "yyyyyyyyyy", "aws.s3.region" = "us-west-2", "aws.s3.use_instance_profile" = "false", "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "http://minio:9000" ) SELECT * FROM sales_records; ``` ##### Unload to local files using NFS[​](#unload-to-local-files-using-nfs "Direct link to Unload to local files using NFS") To access the files in NFS via the `file://` protocol, you need to mount a NAS device as NFS under the same directory of each BE or CN node. Example: ```sql -- Unload data into CSV files. INSERT INTO FILES( 'path' = 'file:///home/ubuntu/csvfile/', 'format' = 'csv', 'csv.column_separator' = ',', 'csv.row_delimitor' = '\n' ) SELECT * FROM sales_records; -- Unload data into CSV files with field enclosing (RFC 4180 style). -- Field values containing commas or quotes are wrapped with the enclose -- character, and internal quotes are escaped by doubling (because -- csv.escape is the same as csv.enclose). NULL values are output as \N -- without enclosing. INSERT INTO FILES( 'path' = 'file:///home/ubuntu/csvfile_enclosed/', 'format' = 'csv', 'csv.column_separator' = ',', 'csv.row_delimiter' = '\n', 'csv.enclose' = '"', 'csv.escape' = '"', 'csv.include_header' = 'true' ) SELECT * FROM sales_records; -- Unload data into Parquet files, and set Parquet version to 1.0. INSERT INTO FILES( 'path' = 'file:///home/ubuntu/parquetfile/', 'format' = 'parquet', 'parquet.version' = '1.0' ) SELECT * FROM sales_records; ``` note When `csv.enclose` is set, every non-NULL field value is wrapped with the enclose character, and occurrences of the enclose character (and the escape character itself, when different) inside field values are escaped with the `csv.escape` character. NULL values are output as `\N` without enclosing. If you need to re-import RFC 4180-style doubled-quote output (where `csv.escape` equals `csv.enclose`) back into StarRocks, set only `csv.enclose` on the read side and do NOT set `csv.escape`. StarRocks' CSV reader natively handles doubled quotes via its ENCLOSE state. Also note that NULL values round-trip cleanly **only** when `csv.escape` is not backslash (`\`). The NULL marker `\N` is fixed, and when `csv.escape = '\\'` the reader's ESCAPE state strips the leading backslash and reads `\N` as the literal character `N` instead of NULL. Use `csv.escape = '"'` (RFC 4180 doubling) for datasets that need NULL roundtrip. #### See also[​](#see-also "Direct link to See also") * For more instructions on the usage of INSERT, see [SQL reference - INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). * For more instructions on the usage of FILES(), see [SQL reference - FILES()](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md). --- ## Using_starrocks ### Query Acceleration #### [📄️ View Running Queries](https://docs.starrocks.io/docs/using_starrocks/running_queries.md) [How to view and monitor running queries in StarRocks using current\_queries and related views, analyze their resource consumption, and cancel long-running...](https://docs.starrocks.io/docs/using_starrocks/running_queries.md) #### [📄️ Gather statistics for CBO](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md) [How to configure StarRocks' cost-based optimizer (CBO) by collecting full, sampled, and histogram statistics for internal and external tables to improve...](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md) #### [📄️ SQL Plan Manager](https://docs.starrocks.io/docs/using_starrocks/SQL_plan_manager.md) [How to use SQL Plan Manager in StarRocks to bind a fixed query plan to a SQL statement, preventing plan changes from statistics or data updates and...](https://docs.starrocks.io/docs/using_starrocks/SQL_plan_manager.md) #### [📄️ Query Feedback](https://docs.starrocks.io/docs/using_starrocks/query_feedback.md) [How to use Query Feedback in StarRocks to automatically analyze slow query plans, generate optimization guides via Query Plan Advisor, and improve...](https://docs.starrocks.io/docs/using_starrocks/query_feedback.md) #### [📄️ Synchronous materialized view](https://docs.starrocks.io/docs/using_starrocks/Materialized_view-single_table.md) [How to create, use, and manage synchronous materialized views (Rollups) in StarRocks for transparent acceleration of single-table aggregate queries with...](https://docs.starrocks.io/docs/using_starrocks/Materialized_view-single_table.md) #### [🗃️ Asynchronous materialized views](https://docs.starrocks.io/docs/using_starrocks/async_mv.md) [6 items](https://docs.starrocks.io/docs/using_starrocks/async_mv.md) #### [📄️ Colocate Join](https://docs.starrocks.io/docs/using_starrocks/Colocate_join.md) [How to use Colocate Join in StarRocks to perform joins locally without network data transmission by co-locating data from joined tables in the same...](https://docs.starrocks.io/docs/using_starrocks/Colocate_join.md) #### [📄️ Use Lateral Join for column-to-row conversion](https://docs.starrocks.io/docs/using_starrocks/Lateral_join.md) [How to use Lateral Join with unnest() in StarRocks to convert array columns to rows (explode), commonly used in ETL pipelines for column-to-row transformation.](https://docs.starrocks.io/docs/using_starrocks/Lateral_join.md) #### [🗃️ Caching](https://docs.starrocks.io/docs/using_starrocks/caching.md) [2 items](https://docs.starrocks.io/docs/using_starrocks/caching.md) #### [🗃️ Computing distinct value count](https://docs.starrocks.io/docs/using_starrocks/distinct_values.md) [2 items](https://docs.starrocks.io/docs/using_starrocks/distinct_values.md) #### [📄️ Sorted streaming aggregate](https://docs.starrocks.io/docs/using_starrocks/sorted_aggregate.md) [How to use sorted streaming aggregate in StarRocks (available from v2.5) to optimize GROUP BY queries on pre-sorted data, reducing memory overhead compared...](https://docs.starrocks.io/docs/using_starrocks/sorted_aggregate.md) #### [📄️ Accelerate COUNT(DISTINCT) and Joins with AUTO INCREMENT and Global Dictionary](https://docs.starrocks.io/docs/using_starrocks/query_acceleration_with_auto_increment.md) [How to use AUTO INCREMENT columns and Global Dictionary in StarRocks to replace string keys with integer IDs, accelerating COUNT(DISTINCT) and join operations.](https://docs.starrocks.io/docs/using_starrocks/query_acceleration_with_auto_increment.md) #### [📄️ Flat JSON](https://docs.starrocks.io/docs/using_starrocks/Flat_json.md) [How to use Flat JSON in StarRocks to accelerate JSON queries by extracting common fields from JSON columns into flat columnar storage, reducing parsing...](https://docs.starrocks.io/docs/using_starrocks/Flat_json.md) #### [📄️ Accelerate Cardinality-Preserving Joins](https://docs.starrocks.io/docs/using_starrocks/accelerate-cardinality-preserving-joins.md) [How to use table pruning to accelerate cardinality-preserving joins in StarRocks by removing redundant joined tables that do not affect query results.](https://docs.starrocks.io/docs/using_starrocks/accelerate-cardinality-preserving-joins.md) #### [📄️ JIT Compilation for Expressions](https://docs.starrocks.io/docs/using_starrocks/jit.md) [How to enable and configure JIT (just-in-time) compilation for expressions in StarRocks to accelerate expression evaluation in query execution.](https://docs.starrocks.io/docs/using_starrocks/jit.md) #### [📄️ Skew Join V2](https://docs.starrocks.io/docs/using_starrocks/skew_join_v2.md) [How to use Skew Join V2 in StarRocks to optimize JOIN queries with data skew by broadcasting skew values, improving performance for heavily skewed data...](https://docs.starrocks.io/docs/using_starrocks/skew_join_v2.md) --- ### Accelerate Cardinality-Preserving Joins This topic describes how to accelerate cardinality-preserving joins with table pruning. This feature is supported from v3.1 onwards. #### Overview[​](#overview "Direct link to Overview") A cardinality-preserving join ensures that the cardinality and duplication factor of the output rows remain the same as those of one of the input tables in the join. Consider the following examples: * **Inner Join:** ```sql SELECT A.* FROM A INNER JOIN B ON A.fk = B.pk; ``` In this case, `A.fk` (Foreign Key) is **NOT NULL** and references `B.pk` (Primary Key). Each row in `A` matches exactly one row in `B`, so the cardinality and duplication factor of the output match those of `A`. * **Left Join:** ```sql SELECT A.* FROM A LEFT JOIN B ON A.fk = B.pk; ``` Here, `A.fk` references `B.pk`, but `A.fk` can contain NULL values. Each row in `A` matches at most one row in `B`. As a result, the cardinality and duplication factor of the output remain aligned with `A`.
In these types of joins, if the final output columns only depend on columns from table `A`, and the columns from table `B` are not used, table `B` can be pruned from the join. Since v3.1, StarRocks supports **table pruning** in cardinality-preserving joins, which can occur in Common Table Expressions (CTEs), logical views, and subqueries. ##### Use case: Real-time feature selection in risk control[​](#use-case-real-time-feature-selection-in-risk-control "Direct link to Use case: Real-time feature selection in risk control") The table pruning feature for cardinality-preserving join is particularly useful in scenarios like **real-time feature selection** for risk control. In this context, users need to select data from a large number of tables, often dealing with a combinatorial explosion of columns and tables. The following characteristics are common in the risk control domain: * Numerous features are spread across many independently updated tables. * Fresh data must be visible and queryable in real time. * A **flat logical view** is used to simplify the data model, making SQL for column extraction more succinct and productive. Using a flat logical view, rather than other accelerated data layers, helps users access real-time data efficiently. In each column extraction query, only a few tables (not all tables in the logical view) are required to be joined. By pruning the unused tables from these queries, you can reduce the number of joins and improve performance. ##### Feature support[​](#feature-support "Direct link to Feature support") The table pruning feature supports multi-table joins in both **star schema** and **snowflake schema**. Multi-table joins can appear in CTEs, logical views, and subqueries, allowing for more efficient query execution. Currently, the table pruning feature is only supported on OLAP tables and Cloud-native tables. External tables in multi-join can not be pruned. #### Usage[​](#usage "Direct link to Usage") The following examples use the TPC-H dataset. ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") To use the table pruning feature, the following conditions must be met: 1. Enable table pruning 2. Set key constraints ###### Enable table pruning[​](#enable-table-pruning "Direct link to Enable table pruning") By default, table pruning is disabled. You need to enable the feature by configuring the following session variables: ```sql -- Enable RBO-phase table pruning. SET enable_rbo_table_prune=true; -- Enable CBO-phase table pruning. SET enable_cbo_table_prune=true; -- Enable RBO-phase table pruning for the UPDATE statement on the Primary Key tables. SET enable_table_prune_on_update = true; ``` ###### Set key constraints[​](#set-key-constraints "Direct link to Set key constraints") Tables to be pruned must have Unique Key or Primary Key constraints at least in LEFT or RIGHT Joins. To prune tables in INNER JOIN, you must define Foreign Key constraints in addition to Unique Key or Primary Key constraints. Primary Key tables and Unique Key tables have their implicit Primary Key or Unique Key constraints naturally built into them. However, for Duplicate Key tables, you must manually define Unique Key constraints, and ensure that no duplicate rows exist. Note that StarRocks does not enforce Unique Key constraints on Duplicate Key tables. Instead, it treats them as optimization hints for more aggressive query planning. Example: ```sql -- Define the Unique Key constraint during table creation. CREATE TABLE `lineitem` ( `l_orderkey` int(11) NOT NULL COMMENT "", `l_partkey` int(11) NOT NULL COMMENT "", `l_suppkey` int(11) NOT NULL COMMENT "", `l_linenumber` int(11) NOT NULL COMMENT "", `l_quantity` decimal64(15, 2) NOT NULL COMMENT "", `l_extendedprice` decimal64(15, 2) NOT NULL COMMENT "", `l_discount` decimal64(15, 2) NOT NULL COMMENT "", `l_tax` decimal64(15, 2) NOT NULL COMMENT "", `l_returnflag` varchar(1) NOT NULL COMMENT "", `l_linestatus` varchar(1) NOT NULL COMMENT "", `l_shipdate` date NOT NULL COMMENT "", `l_commitdate` date NOT NULL COMMENT "", `l_receiptdate` date NOT NULL COMMENT "", `l_shipinstruct` varchar(25) NOT NULL COMMENT "", `l_shipmode` varchar(10) NOT NULL COMMENT "", `l_comment` varchar(44) NOT NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`l_orderkey`,`l_partkey`, `l_suppkey`) DISTRIBUTED BY HASH(`l_orderkey`) BUCKETS 96 PROPERTIES ( "unique_constraints" = "l_orderkey,l_linenumber" ); -- Or you can define the Unique Key constraint after table creation. ALTER TABLE lineitem SET ("unique_constraints" = "l_orderkey,l_linenumber"); ``` Foreign Key constraints, on the other hand, must be defined explicitly. Similar to Unique Key constraints on Duplicate Key tables, Foreign Key constraints act as hints for the optimizer. StarRocks does not enforce Foreign Key constraint consistency. You must ensure data integrity when producing and ingesting data into StarRocks. Example: ```sql -- Create the table to be referenced in the Foreign Key constraint. -- Note the column to be referenced must have Unique Key or Primary Key constraints. -- In this example, `p_partkey` is the Primary Key of the table `part`. CREATE TABLE part ( p_partkey int(11) NOT NULL, p_name VARCHAR(55) NOT NULL, p_mfgr CHAR(25) NOT NULL, p_brand CHAR(10) NOT NULL, p_type VARCHAR(25) NOT NULL, p_size INT NOT NULL, p_container CHAR(10) NOT NULL, p_retailprice DOUBLE NOT NULL, p_comment VARCHAR(23) NOT NULL ) ENGINE=OLAP PRIMARY KEY(`p_partkey`) DISTRIBUTED BY HASH(`p_partkey`) BUCKETS 12; -- Define the Foreign Key constraint during table creation. CREATE TABLE `lineitem` ( `l_orderkey` int(11) NOT NULL COMMENT "", `l_partkey` int(11) NOT NULL COMMENT "", `l_suppkey` int(11) NOT NULL COMMENT "", `l_linenumber` int(11) NOT NULL COMMENT "", `l_quantity` decimal64(15, 2) NOT NULL COMMENT "", `l_extendedprice` decimal64(15, 2) NOT NULL COMMENT "", `l_discount` decimal64(15, 2) NOT NULL COMMENT "", `l_tax` decimal64(15, 2) NOT NULL COMMENT "", `l_returnflag` varchar(1) NOT NULL COMMENT "", `l_linestatus` varchar(1) NOT NULL COMMENT "", `l_shipdate` date NOT NULL COMMENT "", `l_commitdate` date NOT NULL COMMENT "", `l_receiptdate` date NOT NULL COMMENT "", `l_shipinstruct` varchar(25) NOT NULL COMMENT "", `l_shipmode` varchar(10) NOT NULL COMMENT "", `l_comment` varchar(44) NOT NULL COMMENT "" ) ENGINE=OLAP DUPLICATEK KEY(`l_orderkey`,`l_partkey`, `l_suppkey`) DISTRIBUTED BY HASH(`l_orderkey`) BUCKETS 96 PROPERTIES ( "foreign_key_constraints" = "(l_partkey) REFERENCES part(p_partkey)" ); -- Or you can define the Foreign Key constraint after table creation. ALTER TABLE lineitem SET ("foreign_key_constraints" = "(l_partkey) REFERENCES part(p_partkey)"); ``` ##### Table pruning in LEFT/RIGHT JOINs based on Unique or Primary Keys[​](#table-pruning-in-leftright-joins-based-on-unique-or-primary-keys "Direct link to Table pruning in LEFT/RIGHT JOINs based on Unique or Primary Keys") Table pruning in LEFT or RIGHT JOINs does not require that the retained side of the join has a Foreign Key referencing the pruned side. This makes pruning more flexible and robust, even if reference integrity cannot be guaranteed. Pruning in LEFT/RIGHT JOINs based on Unique or Primary keys has **less strict requirements** compared to INNER JOIN pruning based on Foreign Keys. The conditions for pruning are: * **Pruned Side** The pruned table must be the **right side** in a LEFT JOIN or the **left side** in a RIGHT JOIN. * **Join Conditions** The join must use equality conditions (`=`) only, and the joined columns of the pruned side must be a **superset** of the Unique or Primary keys. * **Output Columns** Only the columns from the retained side should be output, and the result should maintain the **same cardinality and duplication factor** as the retained side. * **NULL/Default Values** The joined columns in the retained side **can** contain NULL or other default values that do not match the pruned side. Example: 1. Create tables and insert data. ```sql -- The table `depts` has its Primary Key constraint on the column `deptno`. CREATE TABLE `depts` ( `deptno` int(11) NOT NULL COMMENT "", `name` varchar(25) NOT NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`deptno`) DISTRIBUTED BY HASH(`deptno`) BUCKETS 10; CREATE TABLE `emps` ( `empid` int(11) NOT NULL COMMENT "", `deptno` int(11) NOT NULL COMMENT "", `name` varchar(25) NOT NULL COMMENT "", `salary` double NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`empid`) DISTRIBUTED BY HASH(`empid`) BUCKETS 10; INSERT INTO depts VALUES (1, "R&D"), (2, "Marketing"), (3, "Community"), (4, "DBA"),(5, "POC"); INSERT INTO emps VALUES (1, 1, "Alice", "6000"), (2, 1, "Bob", "6100"), (3, 2, "Candy", "10000"), (4, 2, "Dave", "20000"), (5, 3, "Evan","18000"), (6, 3, "Freman","1000"), (7, 4, "George","1800"), (8, 4, "Harry","2000"), (9, 5, "Ivan", "15000"), (10, 5, "Jim","20000"), (11, -1, "Kevin","1500"), (12, -1, "Lily","2500"); ``` 2. View the logical execution plans of the queries. ```sql -- Q1: Query all columns of `emps` columns. Not all columns of `depts` are queried. EXPLAIN LOGICAL SELECT emps.* FROM emps LEFT JOIN depts ON emps.deptno = depts.deptno; +-----------------------------------------------------------------------------+ | Explain String | +-----------------------------------------------------------------------------+ | - Output => [1:empid, 2:deptno, 3:name, 4:salary] | | - SCAN [emps] => [1:empid, 2:deptno, 3:name, 4:salary] | | Estimates: {row: 10, cpu: ?, memory: ?, network: ?, cost: 20.0} | | partitionRatio: 1/1, tabletRatio: 10/10 | +-----------------------------------------------------------------------------+ -- Q2: Query only `deptno` and `salary` in `emps`. No column in `depts` is queried. EXPLAIN LOGICAL SELECT emps.deptno, avg(salary) AS mean_salary FROM emps LEFT JOIN depts ON emps.deptno = depts.deptno GROUP BY emps.deptno ORDER BY mean_salary DESC LIMIT 5; +-------------------------------------------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------------------------------------------+ | - Output => [2:deptno, 7:avg] | | - TOP-5(FINAL)[7: avg DESC NULLS LAST] | | Estimates: {row: 3, cpu: ?, memory: ?, network: ?, cost: 138.0} | | - TOP-5(PARTIAL)[7: avg DESC NULLS LAST] | | Estimates: {row: 3, cpu: ?, memory: ?, network: ?, cost: 114.0} | | - AGGREGATE(GLOBAL) [2:deptno] | | Estimates: {row: 3, cpu: ?, memory: ?, network: ?, cost: 90.0} | | 7:avg := avg(7:avg) | | - EXCHANGE(SHUFFLE) [2] | | Estimates: {row: 6, cpu: ?, memory: ?, network: ?, cost: 72.0} | | - AGGREGATE(LOCAL) [2:deptno] | | Estimates: {row: 6, cpu: ?, memory: ?, network: ?, cost: 48.0} | | 7:avg := avg(4:salary) | | - SCAN [emps] => [2:deptno, 4:salary] | | Estimates: {row: 12, cpu: ?, memory: ?, network: ?, cost: 12.0} | | partitionRatio: 1/1, tabletRatio: 10/10 | +-------------------------------------------------------------------------------------------------+ -- Q3: Only columns in `emps` are queried. Although predicate `name = "R&D"` only -- selects certain rows of `depts`, the final results only depends on `emps`. EXPLAIN LOGICAL SELECT emps.deptno, avg(salary) AS mean_salary FROM emps LEFT JOIN (SELECT deptno FROM depts WHERE name="R&D") t ON emps.deptno = t.deptno GROUP BY emps.deptno ORDER BY mean_salary DESC LIMIT 5; +-------------------------------------------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------------------------------------------+ | - Output => [2:deptno, 7:avg] | | - TOP-5(FINAL)[7: avg DESC NULLS LAST] | | Estimates: {row: 3, cpu: ?, memory: ?, network: ?, cost: 138.0} | | - TOP-5(PARTIAL)[7: avg DESC NULLS LAST] | | Estimates: {row: 3, cpu: ?, memory: ?, network: ?, cost: 114.0} | | - AGGREGATE(GLOBAL) [2:deptno] | | Estimates: {row: 3, cpu: ?, memory: ?, network: ?, cost: 90.0} | | 7:avg := avg(7:avg) | | - EXCHANGE(SHUFFLE) [2] | | Estimates: {row: 6, cpu: ?, memory: ?, network: ?, cost: 72.0} | | - AGGREGATE(LOCAL) [2:deptno] | | Estimates: {row: 6, cpu: ?, memory: ?, network: ?, cost: 48.0} | | 7:avg := avg(4:salary) | | - SCAN [emps] => [2:deptno, 4:salary] | | Estimates: {row: 12, cpu: ?, memory: ?, network: ?, cost: 12.0} | | partitionRatio: 1/1, tabletRatio: 10/10 | +-------------------------------------------------------------------------------------------------+ -- Q4: The predicate `depts.name="R&D"` in the WHERE clause breaches the -- cardinality preserving conditions, so `depts` can not be pruned. EXPLAIN LOGICAL SELECT emps.deptno, avg(salary) AS mean_salary FROM emps LEFT JOIN depts ON emps.deptno = depts.deptno WHERE depts.name="R&D"; +-----------------------------------------------------------------------------------------------+ | Explain String | +-----------------------------------------------------------------------------------------------+ | - Output => [8:any_value, 7:avg] | | - AGGREGATE(GLOBAL) [] | | Estimates: {row: 1, cpu: ?, memory: ?, network: ?, cost: 110.5} | | 7:avg := avg(7:avg) | | 8:any_value := any_value(8:any_value) | | - EXCHANGE(GATHER) | | Estimates: {row: 1, cpu: ?, memory: ?, network: ?, cost: 105.5} | | - AGGREGATE(LOCAL) [] | | Estimates: {row: 1, cpu: ?, memory: ?, network: ?, cost: 101.5} | | 7:avg := avg(4:salary) | | 8:any_value := any_value(2:deptno) | | - HASH/INNER JOIN [2:deptno = 5:deptno] => [2:deptno, 4:salary] | | Estimates: {row: 12, cpu: ?, memory: ?, network: ?, cost: 79.5} | | - SCAN [emps] => [2:deptno, 4:salary] | | Estimates: {row: 12, cpu: ?, memory: ?, network: ?, cost: 12.0} | | partitionRatio: 1/1, tabletRatio: 10/10 | | - EXCHANGE(BROADCAST) | | Estimates: {row: 5, cpu: ?, memory: ?, network: ?, cost: 25.0} | | - SCAN [depts] => [5:deptno] | | Estimates: {row: 5, cpu: ?, memory: ?, network: ?, cost: 5.0} | | partitionRatio: 1/1, tabletRatio: 10/10 | | predicate: 6:name = 'R&D' | +-----------------------------------------------------------------------------------------------+ ``` In the above example, table pruning is allowed in Q1, Q2, and Q3 as shown in the execution plan, while Q4, which breaches the cardinality preserving conditions, fails to perform table pruning. ##### Table Pruning in INNER/LEFT/RIGHT JOINs Based on Foreign Keys[​](#table-pruning-in-innerleftright-joins-based-on-foreign-keys "Direct link to Table Pruning in INNER/LEFT/RIGHT JOINs Based on Foreign Keys") Table pruning in **INNER JOINs** is more restrictive because the retained side must have a **Foreign Key** that references the pruned side, and reference integrity must be ensured. Currently, StarRocks does not enforce consistency checks on Foreign Key constraints. Although table pruning based on Foreign Keys is more strict, it is also more powerful. It allows the optimizer to leverage column equivalence inference in INNER JOINs, making pruning possible in more complex scenarios. The conditions for pruning are as follows: * **Pruned Side** * In a LEFT JOIN, the pruned table must be on the right side. In a RIGHT JOIN, it must be on the left side. * In an INNER JOIN, the pruned table must have a Unique Key constraint on the joined columns, and each row in the retained side must match exactly one row in the pruned side. If unmatched rows exist in the retained side, pruning cannot occur. * **Join Conditions** The join must use equality conditions (`=`) only. The joined columns in the retained side must be **Foreign Keys**, while the joined columns in the pruned side must be **Primary Keys** or **Unique Keys**. These columns must align according to the Foreign Key constraints. * **Output Columns** * Only the columns from the retained side should be output, and the result should maintain the **same cardinality and duplication factor** as the retained side. * For INNER JOINs, joined columns from the pruned side can be substituted by their equivalent columns in the retained side. If all output columns from the pruned side are joined columns, pruning can also occur. * **NULL/Default Values:** Joined columns in the retained side **cannot** contain NULL or other default values that do not match the pruned side. Example: 1. Create tables, define the Foreign Key, and insert data. ```sql CREATE TABLE `depts` ( `deptno` int(11) NOT NULL COMMENT "", `name` varchar(25) NOT NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`deptno`) DISTRIBUTED BY HASH(`deptno`) BUCKETS 10; CREATE TABLE `emps` ( `empid` int(11) NOT NULL COMMENT "", `deptno` int(11) NOT NULL COMMENT "", `name` varchar(25) NOT NULL COMMENT "", `salary` double NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`empid`) DISTRIBUTED BY HASH(`empid`) BUCKETS 10; ALTER TABLE emps SET ("foreign_key_constraints" = "(deptno) REFERENCES depts(deptno)"); INSERT INTO depts VALUES (1, "R&D"), (2, "Marketing"), (3, "Community"), (4, "DBA"),(5, "POC"); INSERT INTO emps VALUES (1, 1, "Alice", "6000"), (2, 1, "Bob", "6100"), (3, 2, "Candy", "10000"), (4, 2, "Dave", "20000"), (5, 3, "Evan","18000"), (6, 3, "Freman","1000"), (7, 4, "George","1800"), (8, 4, "Harry","2000"), (9, 5, "Ivan", "15000"), (10, 5, "Jim","20000"); ``` 2. View the logical execution plans of the queries. ```sql -- Q1: Query `empid` and `name` in `emps`, and `deptno` in `depts`. -- However, as shown in the Join condition, `emps.deptno` is equivalent to `depts.deptno`, -- so `emps.deptno` can substitute `depts.deptno`. EXPLAIN LOGICAL WITH t0 AS ( SELECT empid, depts.deptno, emps.name, emps.salary, depts.name AS dept_name FROM emps INNER JOIN depts ON emps.deptno = depts.deptno ) SELECT empid, deptno, name FROM t0; +-----------------------------------------------------------------------------+ | Explain String | +-----------------------------------------------------------------------------+ | - Output => [7:empid, 8:deptno, 9:name] | | - SCAN [emps] => [7:empid, 8:deptno, 9:name] | | Estimates: {row: 12, cpu: ?, memory: ?, network: ?, cost: 18.0} | | partitionRatio: 1/1, tabletRatio: 10/10 | +-----------------------------------------------------------------------------+ -- Q2: Only query `salary` in `emps`. EXPLAIN LOGICAL SELECT avg(salary) FROM emps INNER JOIN depts ON emps.deptno = depts.deptno; +----------------------------------------------------------------------------------------+ | Explain String | +----------------------------------------------------------------------------------------+ | - Output => [7:avg] | | - AGGREGATE(GLOBAL) [] | | Estimates: {row: 1, cpu: ?, memory: ?, network: ?, cost: 16.5} | | 7:avg := avg(7:avg) | | - EXCHANGE(GATHER) | | Estimates: {row: 1, cpu: ?, memory: ?, network: ?, cost: 14.0} | | - AGGREGATE(LOCAL) [] | | Estimates: {row: 1, cpu: ?, memory: ?, network: ?, cost: 12.0} | | 7:avg := avg(4:salary) | | - SCAN [emps] => [4:salary] | | Estimates: {row: 10, cpu: ?, memory: ?, network: ?, cost: 5.0} | | partitionRatio: 1/1, tabletRatio: 10/10 | +----------------------------------------------------------------------------------------+ -- Q3: The predicate `name="R&D"` affects the cardinality of final results. EXPLAIN LOGICAL SELECT emps.deptno, avg(salary) AS mean_salary FROM emps INNER JOIN (SELECT deptno FROM depts WHERE name="R&D") t ON emps.deptno = t.deptno GROUP BY emps.deptno ORDER BY mean_salary DESC LIMIT 5; +-------------------------------------------------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------------------------------------------------+ | - Output => [2:deptno, 7:avg] | | - TOP-5(FINAL)[7: avg DESC NULLS LAST] | | Estimates: {row: 2, cpu: ?, memory: ?, network: ?, cost: 165.0480769230769} | | - TOP-5(PARTIAL)[7: avg DESC NULLS LAST] | | Estimates: {row: 2, cpu: ?, memory: ?, network: ?, cost: 145.0480769230769} | | - AGGREGATE(GLOBAL) [2:deptno] | | Estimates: {row: 2, cpu: ?, memory: ?, network: ?, cost: 125.04807692307692} | | 7:avg := avg(4:salary) | | - HASH/INNER JOIN [2:deptno = 5:deptno] => [2:deptno, 4:salary] | | Estimates: {row: 10, cpu: ?, memory: ?, network: ?, cost: 100.04807692307692} | | - EXCHANGE(SHUFFLE) [2] | | Estimates: {row: 10, cpu: ?, memory: ?, network: ?, cost: 50.0} | | - SCAN [emps] => [2:deptno, 4:salary] | | Estimates: {row: 10, cpu: ?, memory: ?, network: ?, cost: 10.0} | | partitionRatio: 1/1, tabletRatio: 10/10 | | - EXCHANGE(SHUFFLE) [5] | | Estimates: {row: 5, cpu: ?, memory: ?, network: ?, cost: 15.0} | | - SCAN [depts] => [5:deptno] | | Estimates: {row: 5, cpu: ?, memory: ?, network: ?, cost: 5.0} | | partitionRatio: 1/1, tabletRatio: 10/10 | | predicate: 6:name = 'R&D' | +-------------------------------------------------------------------------------------------------------+ ``` In the above example, table pruning is allowed in Q1, and Q2 as shown in the execution plan, while Q3, whose predicate affects the cardinality of final results, fails to perform table pruning. --- ### Asynchronous materialized views #### [📄️ Asynchronous materialized views](https://docs.starrocks.io/docs/using_starrocks/async_mv/Materialized_view.md) [How to create, use, refresh, and manage asynchronous materialized views in StarRocks for accelerating multi-table join and aggregation queries with...](https://docs.starrocks.io/docs/using_starrocks/async_mv/Materialized_view.md) #### [📄️ Related References](https://docs.starrocks.io/docs/using_starrocks/async_mv/mv_ref.md) [Reference index for asynchronous materialized view SQL statements (CREATE, ALTER, REFRESH, DROP) and system views for monitoring in StarRocks.](https://docs.starrocks.io/docs/using_starrocks/async_mv/mv_ref.md) #### [🗃️ Use cases](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases.md) [4 items](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases.md) #### [📄️ Troubleshooting asynchronous materialized views](https://docs.starrocks.io/docs/using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md) [Troubleshooting guide for asynchronous materialized views in StarRocks: how to diagnose and fix refresh failures, query rewrite issues, and data consistency...](https://docs.starrocks.io/docs/using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md) #### [📄️ Feature Support](https://docs.starrocks.io/docs/using_starrocks/async_mv/feature-support-asynchronous-materialized-views.md) [Reference for feature support boundaries of asynchronous materialized views in StarRocks: supported join types, aggregate functions, query rewrite...](https://docs.starrocks.io/docs/using_starrocks/async_mv/feature-support-asynchronous-materialized-views.md) #### [📄️ Understand Materialized View Task Runs](https://docs.starrocks.io/docs/using_starrocks/async_mv/materialized_view_task_run_details.md) [How to view and interpret materialized view refresh task run details in StarRocks to understand refresh behavior, troubleshoot failures, and monitor refresh...](https://docs.starrocks.io/docs/using_starrocks/async_mv/materialized_view_task_run_details.md) --- ### Feature Support: Asynchronous Materialized Views Asynchronous materialized views are supported from StarRocks v2.4 onwards. Asynchronous materialized views are designed to accelerate complex queries with joins or aggregations on large tables within StarRocks or in data lakes. The performance difference can be significant when a query is run frequently or is sufficiently complex. Additionally, asynchronous materialized views are especially useful for building mathematical models upon your data warehouse. This document outlines the boundaries of competence for asynchronous materialized views and the supported version of the features involved. #### DDL Features[​](#ddl-features "Direct link to DDL Features") | Feature | Description | Supported Version(s) | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | | Auto Analyze | Automatically collects statistics after the materialized view is created to avoid rewrite failures. | v3.0+ | | Random Bucketing | Enables random bucketing strategy for materialized views by default. | v3.1+ | | Deferred Refresh | Supports specifying whether to refresh the materialized view immediately after being created by using DEFERRED or IMMEDIATE in CREATE MATERIALIZED VIEW. | v3.0+ | | Order By | Supports specifying the sort key for materialized views using ORDER BY. | v3.1+ | | Window/CTE/Union/Subquery | Supports using window functions, CTEs, Unions, and subqueries in materialized views. | v2.5+ | | ALTER ACTIVE | Activates invalid materialized views after Schema Changes in base tables using the ACTIVE keyword in ALTER MATERIALIZED VIEW. | v2.5.7+
v3.0.1+
v3.1+ | | REFRESH SYNC MODE | Supports synchronous execution for materialized view refresh tasks by using WITH SYNC MODE keywords in REFRESH MATERIALIZED VIEW. | v2.5.8+
v3.0.4+
v3.1+ | | Intermediate Result Spilling | Supports enabling Intermediate Result Spilling using the `enable_spill` property to avoid OOM during materialized view construction. | v3.1+ | | Resource Group | Supports specifying resource groups using the `resource_group` property for materialized view construction to achieve resource isolation. | v3.1+ | | Materialized View on View | Supports creating materialized views based on logical views. | v3.1+ | | Swap Materialized View | Supports atomically replacing materialized view using the SWAP WITH keywords in ALTER MATERIALIZED VIEW. | v3.1+ | | CREATE INDEX ON Materialized View | Supports creating indexes on materialized views to accelerate point queries. | v3.0.7+
v3.1.4+
v3.2+ | | AUTO ACTIVE | Automatically activates invalid materialized views in the background with exponential backoff, stopping after the interval reaches 60 minutes. | v3.1.4+
v3.2+ | | Backup and Restore | Supports Backup and Restore for materialized views. | v3.2+ | | Object Dependencies | Provides a system-defined view `sys.object_dependencies` to clarify the dependency relationship between materialized views and base tables. | v3.2+ | #### Variables[​](#variables "Direct link to Variables") | Variable | Description | Default | Supported Version(s) | | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------ | | enable\_materialized\_view\_rewrite | Whether to enable materialized view query rewrite. | true | v2.5+ | | enable\_materialized\_view\_for\_insert | Whether to enable materialized view query rewrite for INSERT statements. | false | v2.5.18+
v3.0.9+
v3.1.7+
v3.2.2+ | | materialized\_view\_rewrite\_mode | Mode of materialized view query rewrite. | DEFAULT | v3.2+ | | optimizer\_materialized\_view\_timelimit | Maximum time can be used for materialized view query rewrite, after which query rewrite is abandoned and the Optimizer process continues. | 1000 | v3.1.9+
v3.2.5+ | | analyze\_mv | Method of collecting statistics after the materialized view is refreshed. | SAMPLE | v3.0+ | | enable\_materialized\_view\_plan\_cache | Whether to enable plan cache for materialized views. By default, 1000 materialized view plans are cached. | TRUE | v2.5.13+
v3.0.7+
v3.1.4+
v3.2.0+
v3.3.0+ | | query\_including\_mv\_names | Whitelist of the materialized views that can be used for query rewrite. | | v3.1.11+
v3.2.5+ | | query\_excluding\_mv\_names | Blacklist of the materialized views that can be used for query rewrite. | | v3.1.11+
v3.2.5+ | | cbo\_materialized\_view\_rewrite\_related\_mvs\_limit | Maximum number of candidate materialized views in the Plan stage. | 64 | v3.1.9+
v3.2.5+ | #### Properties[​](#properties "Direct link to Properties") | Property | Description | Supported Version(s) | | -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | | `session.` | Prefix of session variables used for materialized view construction, for example, `session.insert_timeout` and `session.query_mem_limit`. | v3.4+ | | auto\_refresh\_partitions\_limit | Maximum number of materialized view partitions to be refreshed each time an automatic refresh is triggered. | v2.5+ | | excluded\_trigger\_tables | Base tables whose updates will not trigger the materialized view automatic refresh. | v2.5+ | | partition\_refresh\_number | Number of partitions to be refreshed in each batch when the refresh task is executed in batches. | v2.5+ | | partition\_ttl\_number | The number of most recent materialized view partitions to retain. | v2.5+ | | partition\_ttl | The time-to-live (TTL) for materialized view partitions. This property is recommended over `partition_ttl_number`. | v3.1.4+
v3.2+ | | force\_external\_table\_query\_rewrite | Whether to enable query rewrite for external catalog-based materialized views. | v2.5+ | | query\_rewrite\_consistency | The query rewrite rule for materialized views built on internal tables. | v3.0.5+
v3.1+ | | resource\_group | The resource group to which the refresh tasks of the materialized view belong. | v3.1+ | | colocate\_with | The colocation group of the materialized view. | v3.1+ | | foreign\_key\_constraints | The Foreign Key constraints when you create a materialized view for query rewrite in the View Delta Join scenario. | v2.5.4+
v3.0+ | | unique\_constraints | The Unique Key constraints when you create a materialized view for query rewrite in the View Delta Join scenario. | v2.5.4+
v3.0+ | | mv\_rewrite\_staleness\_second | Staleness tolerance for materialized view data during query rewrite. | v3.1+ | | enable\_query\_rewrite | Whether the materialized view can be used for query rewrite. | v3.3+ | | excluded\_refresh\_tables | Base tables that do not trigger data synchronization during materialized view refresh. | v3.3+ | #### Partitioning[​](#partitioning "Direct link to Partitioning") | Alignment | Use Case | Supported Version(s) | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | | [Align partitions one-to-one (Date types)](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md#align-partitions-one-to-one) | Create a materialized view whose partitions correspond to the partitions of the base table one-to-one by using the same Partitioning Key. The Partitioning Key must be the DATE or DATETIME type. | v2.5+ | | [Align partitions one-to-one (STRING type)](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md#align-partitions-one-to-one) | Create a materialized view whose partitions correspond to the partitions of the base table one-to-one by using the same Partitioning Key. The Partitioning Key must be the STRING type. | v3.1.4+
v3.2+ | | [Align partitions with time granularity rollup (Date types)](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md#align-partitions-with-time-granularity-rollup) | Create a materialized view whose partitioning granularity is larger than that of the base table by using the `date_trunc` function on the Partitioning Key. The Partitioning Key must be the DATE or DATETIME type. | v2.5+ | | [Align partitions with time granularity rollup (STRING type)](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md#align-partitions-with-time-granularity-rollup) | Create a materialized view whose partitioning granularity is larger than that of the base table by using the `date_trunc` function on the Partitioning Key. The Partitioning Key must be the STRING type. | v3.1.4+
v3.2+ | | [Align partitions at a customized time granularity](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md#align-partitions-at-a-customized-time-granularity) | Create a materialized view and customize the time granularity for its partitions by using the `date_trunc` function with the `time_slice` or `date_slice` function. | v3.2+ | | [Align partitions with multiple base tables](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md#align-partitions-with-multiple-base-tables) | Create a materialized view whose partitions are aligned with those of multiple base tables, as long as the base tables use the same type of Partitioning Key. | v3.3+ | | [Align multiple partition columns](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md#align-multiple-partition-columns) | Specify multiple partition columns for the materialized view, and one-to-one map them to the partition columns of the base tables. | v3.5+ | **Different Join Methods** * **Single Fact Table (v2.4+)**: Establishing a partition mapping between the materialized view and the fact table ensures the automatic refresh of materialized view partitions when the fact table is updated. * **Multiple Fact Tables (v3.3+)**: Establishing a partition mapping between the materialized view and multiple fact tables that are joined/unioned at the same time granularity ensures the automatic refresh of materialized view partitions when any of the fact tables are updated. * **Temporal Dimension Table (v3.3+)**: Suppose the dimension table stores historical version data and is partitioned at a specific time granularity, and a fact table joins the dimension table at the same time granularity. Establishing a partition mapping between the materialized view and both the fact table and the dimension table ensures the automatic refresh of materialized view partitions when either table is updated. #### Materialized views on external catalogs[​](#materialized-views-on-external-catalogs "Direct link to Materialized views on external catalogs") | External Data Source | Supported Scenario and Version(s) | Stable Version(s) | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------- | | Hive | - Non-partitioned table: v2.5.4 & v3.0+
- DATE and DATETIME-type partition: v2.5.4 & v3.0+
- Transforming STRING-type Partition Key to DATE-type: v3.1.4 & v3.2+
- Materialized views on Hive View: To be supported
- Multi-level partitioning: To be supported | v2.5.13+
v3.0.6+
v3.1.5+
v3.2+ | | Iceberg | - Non-partitioned table: v3.0+
- DATE and DATETIME-type partition: v3.1.4 & v3.2+
- Transforming STRING-type Partition Key to DATE-type: v3.1.4 & v3.2+
- Materialized views on Iceberg View: To be supported
- Partition Transform: v3.2.3
- Partition-level refresh: v3.1.7 & v3.2.3
- Multi-level partitioning: v3.5+ | v3.1.5+
v3.2+ | | Hudi | - Non-partitioned table: v3.2+
- DATE and DATETIME-type partition: v3.2+
- Multi-level partitioning: To be supported | Not Stable | | Paimon | - Non-partitioned table: v2.5.4 & v3.0+
- DATE and DATETIME-type partition: To be supported
- Multi-level partitioning: To be supported | Not Stable | | DeltaLake | - Non-partitioned table: v3.2+
- Partitioned table: To be supported
- Multi-level partitioning: To be supported | Not Stable | | JDBC | - Non-partitioned table: v3.0+
- Partitioned table: MySQL RangeColumn Partition v3.1.4 | Not Stable | note External table materialized views do not support automatic refresh **triggered by base table data changes**. They only support asynchronous **fixed-interval** refresh and manual refresh. #### Query Rewrite[​](#query-rewrite "Direct link to Query Rewrite") | Feature | Description | Supported Version(s) | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------- | | Single Table Rewrite | Query rewrite with materialized views built on a single internal table. | v2.5+ | | Inner Join Rewrite | Query rewrite for INNER/CROSS JOIN on internal tables. | v2.5+ | | Aggregate Rewrite | Query rewrite for Joins with basic aggregations. | v2.5+ | | UNION Rewrite | Predicate UNION compensation rewrite and partition UNION compensation rewrite on internal tables. | v2.5+ | | Nested Materialized View Rewrite | Query rewrite with nested materialized views on internal tables. | v2.5+ | | Count Distinct Rewrite (bitmap/hll) | Query rewrite for COUNT DISTINCT calculations into bitmap- or HLL-based calculations. | v2.5.6+
v3.0+ | | View Delta Join Rewrite | Rewrite queries that join the tables that are the subset of the tables the materialized view joins. | v2.5.4+
v3.0+ | | Join Derivability Rewrite | Query rewrite between different join types. | v2.5.8+
v3.0.4+
v3.1+ | | Full Outer Join and Other Joins | Query rewrite for Full Outer Join, Semi Join, and Anti Join. | v3.1+ | | Avg to Sum/Count Rewrite | Query rewrite for avg() to sum() / count() | v3.1+ | | View-based Rewrite | Query rewrite with materialized views built upon views without transcribing queries against the view into queries against the base tables of the view. | v3.2.2+ | | Count Distinct Rewrite (ArrayAgg) | Query rewrite for COUNT DISTINCT calculations into calculations with the `array_agg_distinct` function. | v3.2.5+
v3.3+ | | Text-based Query Rewrite | Rewrite the query that has the identical abstract syntax tree with that of the materialized view's definition. | v3.3+ | #### Diagnostic Features[​](#diagnostic-features "Direct link to Diagnostic Features") | Feature | Usage Scenario | Supported Version(s) | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------- | | TRACE REWRITE | Use the TRACE REWRITE statement to diagnose rewrite issues. | v2.5.10+
v3.0.5+
v3.1+ | | Query Dump | Dump the information of a materialized view once it is queried. | v3.1+ | | Refresh Audit Log | Record the SQL executed in the Audit Log when a materialized view is refreshed. | v2.5.8+
v3.0.3+
v3.1+ | | Hit Audit Log | Record the hit materialized view and candidate materialized views in the Audit Log when a query is rewritten to a materialized view. | v3.1.4+
v3.2+ | | Monitoring Metrics | Dedicated monitoring metrics for materialized views. | v3.1.4+
v3.2+ | --- ### Asynchronous materialized views This topic describes how to understand, create, use, and manage an asynchronous materialized view. Asynchronous materialized views are supported from StarRocks v2.4 onwards. Compared with synchronous materialized views, asynchronous materialized views support multi-table join and more aggregate functions. The refresh of asynchronous materialized views can be triggered manually, regualrly at pre-specified intervals, or automatically at data changes in the base table. You can also refresh some of the partitions instead of the whole materialized view, greatly reducing the cost of refresh. In addition, asynchronous materialized views support a variety of query rewrite scenarios, allowing automatic, transparent query acceleration. For the scenario and usage of the synchronous materialized views (Rollup), see [Synchronous materialized view (Rollup)](https://docs.starrocks.io/docs/using_starrocks/Materialized_view-single_table.md). #### Overview[​](#overview "Direct link to Overview") Applications in databases often perform complex queries on large tables. Such queries involve multi-table joins and aggregations on tables that contain billions of rows. Processing these queries can be expensive, in terms of system resources and the time it takes to compute the results. Asynchronous materialized views in StarRocks are designed to tackle these issues. An asynchronous materialized view is a special physical table that holds pre-computed query results from one or more base tables. When you perform complex queries on the base table, StarRocks returns the pre-computed results from the relevant materialized views to process these queries. This way, query performance can be improved because repetitive complex calculations are avoided. This performance difference can be significant when a query is run frequently or is sufficiently complex. Additionally, asynchronous materialized views are especially useful for building mathematical models upon your data warehouse. By doing so, you can provide a unified data specification to upper-layer applications, shield the underlying implementation, or protect the raw data security of the base tables. ##### Understand materialized views in StarRocks[​](#understand-materialized-views-in-starrocks "Direct link to Understand materialized views in StarRocks") StarRocks v2.3 and earlier versions provided a synchronous materialized view that can be built only on a single table. Synchronous materialized views, or the Rollup, retain higher data freshness and lower refreshing costs. However, compared to asynchronous materialized views supported from v2.4 onwards, synchronous materialized views have many limitations. You have limited choices of aggregation operators when you want to build a synchronous materialized view to accelerate or rewrite your queries. The following table compares the asynchronous materialized views (ASYNC MV) and the synchronous materialized view (SYNC MV) in StarRocks in the perspective of features that they support: | | **Single-table aggregation** | **Multi-table join** | **Query rewrite** | **Refresh strategy** | **Base table** | | -------------------- | -------------------------------------- | -------------------- | ----------------- | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | **ASYNC MV** | Yes | Yes | Yes | - Asynchronous refresh
- Manual refresh | Multiple tables from:- Default catalog
- External catalogs (v2.5)
- Existing materialized views (v2.5)
- Existing views (v3.1) | | **SYNC MV (Rollup)** | Limited choices of aggregate functions | No | Yes | Synchronous refresh during data loading | Single table in the default catalog | ##### Basic concepts[​](#basic-concepts "Direct link to Basic concepts") * **Base table** Base tables are the driving tables of a materialized view. For StarRocks' asynchronous materialized views, base tables can be StarRocks native tables in the [default catalog](https://docs.starrocks.io/docs/data_source/catalog/default_catalog.md), tables in external catalogs (supported from v2.5), or even existing asynchronous materialized views (supported from v2.5) and views (supported from v3.1). StarRocks supports creating asynchronous materialized views on all [types of StarRocks tables](https://docs.starrocks.io/docs/table_design/table_types.md). * **Refresh** When you create an asynchronous materialized view, its data reflects only the state of the base tables at that time. When the data in the base tables change, you need to refresh the materialized view to keep the changes synchronized. Currently, StarRocks supports two generic refreshing strategies: * ASYNC: Asynchronous refresh mode. Materialized views can be refreshed automatically when the data in the base table changes, or at regular intervals based on specified intervals. * MANUAL: Manual refresh mode. The materialized view will not be automatically refreshed. The refresh tasks can only be triggered manually by users. * **Query rewrite** Query rewrite means that when executing a query on base tables with materialized views built on, the system automatically judges whether the pre-computed results in the materialized view can be reused for the query. If they can be reused, the system will load the data directly from the relevant materialized view to avoid the time- and resource-consuming computations or joins. From v2.5, StarRocks supports automatic, transparent query rewrite based on the SPJG-type asynchronous materialized views. The SPJG-type materialized views refer to materialized views whose plan only includes Scan, Filter, Project, and Aggregate types of operators. > **NOTE** > > Asynchronous materialized views created on base tables in a JDBC catalog or a Hudi catalog do not support query rewrite. #### Decide when to create a materialized view[​](#decide-when-to-create-a-materialized-view "Direct link to Decide when to create a materialized view") You can create an asynchronous materialized view if you have the following demands in your data warehouse environment: * **Accelerating queries with repetitive aggregate functions** Suppose that most queries in your data warehouse include the same sub-query with an aggregate function, and these queries have consumed a huge proportion of your computing resources. Based on this sub-query, you can create an asynchronous materialized view, which will compute and store all results of the sub-query. After the materialized view is built, StarRocks rewrites all queries that contain the sub-query, loads the intermediate results stored in the materialized view, and thus accelerates these queries. * **Regular JOIN of multiple tables** Suppose that you need to regularly join multiple tables in your data warehouse to make a new wide table. You can build an asynchronous materialized view for these tables, and set the ASYNC refreshing strategy that triggers refreshing tasks at a fixed time interval. After the materialized view is built, query results are returned directly from the materialized view, and thus the latency caused by JOIN operations is avoided. * **Data warehouse layering** Suppose that your data warehouse contains a mass of raw data, and queries in it require a complex set of ETL operations. You can build multiple layers of asynchronous materialized views to stratify the data in your data warehouse, and thus decompose the query into a series of simple sub-queries. It can significantly reduce repetitive computation, and, more importantly, help your DBA identify the problem with ease and efficiency. Beyond that, data warehouse layering helps decouple raw data and statistical data, protecting the security of sensitive raw data. * **Accelerating queries in data lakes** Querying a data lake can be slow due to network latency and object storage throughput. You can enhance the query performance by building an asynchronous materialized view on top of the data lake. Moreover, StarRocks can intelligently rewrite queries to use the existing materialized views, saving you the trouble of modifying your queries manually. For specific use cases of asynchronous materialized views, refer to the following content: * [Data Modeling](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/data_modeling_with_materialized_views.md) * [Query Rewrite](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md) * [Data Lake Query Acceleration](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md) #### Create an asynchronous materialized view[​](#create-an-asynchronous-materialized-view "Direct link to Create an asynchronous materialized view") StarRocks' asynchronous materialized views can be created on the following base tables: * StarRocks' native tables (all StarRocks table types are supported) * Tables in [external catalogs](https://docs.starrocks.io/docs/using_starrocks/async_mv/feature-support-asynchronous-materialized-views.md#materialized-views-on-external-catalogs) * Hive Catalog * Hudi Catalog * Iceberg Catalog * JDBC Catalog * Paimon Catalog * DeltaLake Catalog * Existing asynchronous materialized views * Existing views ##### Before you begin[​](#before-you-begin "Direct link to Before you begin") The following examples involve two base tables in the default catalog: * The table `goods` records the item ID `item_id1`, the item name `item_name`, and the item price `price`. * The table `order_list` records the order ID `order_id`, client ID `client_id`, item ID `item_id2`, and order date `order_date`. The column `goods.item_id1` is equivalent to the column `order_list.item_id2`. Execute the following statements to create the tables and insert data into them: ```sql CREATE TABLE goods( item_id1 INT, item_name STRING, price FLOAT ) DISTRIBUTED BY HASH(item_id1); INSERT INTO goods VALUES (1001,"apple",6.5), (1002,"pear",8.0), (1003,"potato",2.2); CREATE TABLE order_list( order_id INT, client_id INT, item_id2 INT, order_date DATE ) DISTRIBUTED BY HASH(order_id); INSERT INTO order_list VALUES (10001,101,1001,"2022-03-13"), (10001,101,1002,"2022-03-13"), (10002,103,1002,"2022-03-13"), (10002,103,1003,"2022-03-14"), (10003,102,1003,"2022-03-14"), (10003,102,1001,"2022-03-14"); ``` The scenario in the following example demands frequent calculations of the total of each order. It requires frequent joins of the two base tables and intensive usage of the aggregate function `sum()`. Besides, the business scenario demands the data refresh at an interval of one day. The query statement is as follows: ```sql SELECT order_id, sum(goods.price) as total FROM order_list INNER JOIN goods ON goods.item_id1 = order_list.item_id2 GROUP BY order_id; ``` ##### Create the materialized view[​](#create-the-materialized-view "Direct link to Create the materialized view") You can create a materialized view based on a specific query statement using [CREATE MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md). Based on the table `goods`, `order_list`, and the query statement mentioned above, the following example creates the materialized view `order_mv` to analyze the total of each order. The materialized view is set to refresh itself at an interval of one day. ```sql CREATE MATERIALIZED VIEW order_mv DISTRIBUTED BY HASH(`order_id`) REFRESH ASYNC START('2022-09-01 10:00:00') EVERY (interval 1 day) AS SELECT order_list.order_id, sum(goods.price) as total FROM order_list INNER JOIN goods ON goods.item_id1 = order_list.item_id2 GROUP BY order_id; ``` > **NOTE** > > * While creating an asynchronous materialized view, you must specify either the data distribution strategy or the refresh strategy of the materialized view, or both. > * You can set different partitioning and bucketing strategies for an asynchronous materialized view from those of its base tables, but you must include the partition keys and bucket keys of the materialized views in the query statement used to create the materialized view. > * Asynchronous materialized views support a dynamic partitioning strategy in a longer span. For example, if the base table is partitioned at an interval of one day, you can set the materialized view to be partitioned at an interval of one month. > * The query statement used to create a materialized view does not support random functions, including rand(), random(), uuid(), and sleep(). > * Asynchronous materialized views support a variety of data types. For more information, see [CREATE MATERIALIZED VIEW - Supported data types](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md#supported-data-types). > * By default, executing a CREATE MATERIALIZED VIEW statement immediately triggers the refresh task, which can consume a certain proportion of the system resources. If you want to defer the refresh task, you can add the REFRESH DEFERRED parameter to your CREATE MATERIALIZED VIEW statement. > * Materialized view can only be created in default\_catalog. You could either create it with default\_catalog.database.mv, or switch to default\_catalog through `set catalog ` statement. * **About refresh mechanisms of asynchronous materialized views** Currently, StarRocks supports two ON DEMAND refresh strategies: MANUAL refresh and ASYNC refresh. In StarRocks v2.5, asynchronous materialized views further support a variety of asynchronous refreshing mechanisms to control the cost of refresh and increase the success rate: * If an MV has many large partitions, each refresh can consume a large amount of resources. In v2.5, StarRocks supports splitting refresh tasks. You can specify the maximum number of partitions to be refreshed, and StarRocks performs refresh in batches, with a batch size smaller or equal to the specified maximum number of partitions. This feature ensures large asynchronous materialized views are stably refreshed, enhancing the stability and robustness of data modeling. * You can specify the time to live (TTL) for partitions of an asynchronous materialized view, reducing the storage size taken by the materialized view. * You can specify the refresh range to refresh only the latest few partitions, reducing the refresh overhead. * You can specify the base tables where data changes will not automatically trigger a refresh of the corresponding materialized view. * You can assign a resource group to the refresh task. For more information, see the **PROPERTIES** section in [CREATE MATERIALIZED VIEW - Parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md#parameters). You can also modify the mechanisms of an existing asynchronous materialized view using [ALTER MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/ALTER_MATERIALIZED_VIEW.md). > **CAUTION** > > To prevent full refresh operations from exhausting system resources and causing task failures, it is recommended to create partitioned materialized views based on partitioned base tables. This ensures that when data updates occur within a base table partition, only the corresponding partition of the materialized view are refreshed, rather than refreshing the entire materialized view. For more information, please refer to [Data Modeling with Materialized Views - Partitioned Modeling](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/data_modeling_with_materialized_views.md#partitioned-modeling). * **About nested materialized views** StarRocks v2.5 supports creating nested asynchronous materialized views. You can build asynchronous materialized views based on existing asynchronous materialized views. The refreshing strategy for each materialized view does not affect the materialized views on the upper or lower layers. Currently, StarRocks does not limit the number of nesting levels. In a production environment, we recommend that the number of nesting layers does not exceed THREE. * **About external catalog materialized views** StarRocks supports building asynchronous materialized views based on Hive Catalog (since v2.5), Hudi Catalog (since v2.5), Iceberg Catalog (since v2.5), and JDBC Catalog (since v3.0). Creating a materialized view on external catalogs is similar to creating an asynchronous materialized view on the default catalog, but with some usage restrictions. For more information, please refer to [Data lake query acceleration with materialized views](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md). #### Manually refresh an asynchronous materialized view[​](#manually-refresh-an-asynchronous-materialized-view "Direct link to Manually refresh an asynchronous materialized view") You can refresh an asynchronous materialized view regardless of its refreshing strategy via [REFRESH MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/REFRESH_MATERIALIZED_VIEW.md). StarRocks v2.5 supports refreshing specific partitions of an asynchronous materialized view by specifying partition names. StarRocks v3.1 supports making a synchronous call of the refresh task, and the SQL statement is returned only when the task succeeds or fails. ```sql -- Refresh the materialized view via an asynchronous call (default). REFRESH MATERIALIZED VIEW order_mv; -- Refresh the materialized view via a synchronous call. REFRESH MATERIALIZED VIEW order_mv WITH SYNC MODE; ``` You can cancel a refresh task submitted via an asynchronous call using [CANCEL REFRESH MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/CANCEL_REFRESH_MATERIALIZED_VIEW.md). #### Query the asynchronous materialized view directly[​](#query-the-asynchronous-materialized-view-directly "Direct link to Query the asynchronous materialized view directly") The asynchronous materialized view you created is essentially a physical table that contains the complete set of pre-computed results in accordance with the query statement. Therefore, you can directly query the materialized view after the materialized view is refreshed for the first time. ```plain MySQL > SELECT * FROM order_mv; +----------+--------------------+ | order_id | total | +----------+--------------------+ | 10001 | 14.5 | | 10002 | 10.200000047683716 | | 10003 | 8.700000047683716 | +----------+--------------------+ 3 rows in set (0.01 sec) ``` > **NOTE** > > You can directly query an asynchronous materialized view, but the results may be inconsistent with what you get from the query on its base tables. #### Rewrite and accelerate queries with the asynchronous materialized view[​](#rewrite-and-accelerate-queries-with-the-asynchronous-materialized-view "Direct link to Rewrite and accelerate queries with the asynchronous materialized view") StarRocks v2.5 supports automatic and transparent query rewrite based on the SPJG-type asynchronous materialized views. The SPJG-type materialized views query rewrite includes single table query rewrite, Join query rewrite, aggregation query rewrite, Union query rewrite and query rewrite based on nested materialized views. From v3.3.0 onwards, StarRocks further supports text-based materialized view rewrite. For more information, please refer to [Query Rewrite with Materialized Views](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md). Currently, StarRocks supports rewriting queries on asynchronous materialized views that are created on the default catalog or an external catalog such as a Hive catalog, Hudi catalog, or Iceberg catalog. When querying data in the default catalog, StarRocks ensures strong consistency of results between the rewritten query and the original query by excluding materialized views whose data is inconsistent with the base table. When the data in a materialized view expires, the materialized view will not be used as a candidate materialized view. When querying data in external catalogs, StarRocks does not ensure a strong consistency of the results because StarRocks cannot perceive the data changes in external catalogs. For more about asynchronous materialized views that are created based on an external catalog, please refer to [Data lake query acceleration with materialized views](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md). > **NOTE** > > Asynchronous materialized views created on base tables in a JDBC catalog do not support query rewrite. #### Manage an asynchronous materialized view[​](#manage-an-asynchronous-materialized-view "Direct link to Manage an asynchronous materialized view") ##### Alter an asynchronous materialized view[​](#alter-an-asynchronous-materialized-view "Direct link to Alter an asynchronous materialized view") You can alter the property of an asynchronous materialized view using [ALTER MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/ALTER_MATERIALIZED_VIEW.md). * Enable an inactive materialized view. ```sql ALTER MATERIALIZED VIEW order_mv ACTIVE; ``` * Rename an asynchronous materialized view. ```sql ALTER MATERIALIZED VIEW order_mv RENAME order_total; ``` * Alter the refreshing interval of an asynchronous materialized view to 2 days. ```sql ALTER MATERIALIZED VIEW order_mv REFRESH ASYNC EVERY(INTERVAL 2 DAY); ``` ##### Show asynchronous materialized views[​](#show-asynchronous-materialized-views "Direct link to Show asynchronous materialized views") You can view the asynchronous materialized views in your database by using [SHOW MATERIALIZED VIEWS](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/SHOW_MATERIALIZED_VIEW.md) or querying the system metadata view in Information Schema. * Check all asynchronous materialized views in your database. ```sql SHOW MATERIALIZED VIEWS; ``` * Check a specific asynchronous materialized view. ```sql SHOW MATERIALIZED VIEWS WHERE NAME = "order_mv"; ``` * Check specific asynchronous materialized views by matching the name. ```sql SHOW MATERIALIZED VIEWS WHERE NAME LIKE "order%"; ``` * Check all asynchronous materialized views by querying the metadata view `materialized_views` in Information Schema. For more information, please refer to [information\_schema.materialized\_views](https://docs.starrocks.io/docs/sql-reference/information_schema/materialized_views.md). ```sql SELECT * FROM information_schema.materialized_views; ``` ##### Check the definition of asynchronous materialized view[​](#check-the-definition-of-asynchronous-materialized-view "Direct link to Check the definition of asynchronous materialized view") You can check the query used to create an asynchronous materialized view via [SHOW CREATE MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/SHOW_CREATE_MATERIALIZED_VIEW.md). ```sql SHOW CREATE MATERIALIZED VIEW order_mv; ``` ##### Check the execution status of asynchronous materialized view[​](#check-the-execution-status-of-asynchronous-materialized-view "Direct link to Check the execution status of asynchronous materialized view") You can check the execution (building or refreshing) status of an asynchronous materialized view by querying the [`tasks`](https://docs.starrocks.io/docs/sql-reference/information_schema/tasks.md) and [`task_runs`](https://docs.starrocks.io/docs/sql-reference/information_schema/task_runs.md) in [Information Schema](https://docs.starrocks.io/docs/sql-reference/information_schema.md). The following example checks the execution status of the materialized view that was created most recently: 1. Check the `TASK_NAME` of the most recent task in the table `tasks`. ```plain mysql> select * from information_schema.tasks order by CREATE_TIME desc limit 1\G; *************************** 1. row *************************** TASK_NAME: mv-59299 CREATE_TIME: 2022-12-12 17:33:51 SCHEDULE: MANUAL DATABASE: ssb_1 DEFINITION: insert overwrite hive_mv_lineorder_flat_1 SELECT `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_orderkey`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_linenumber`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_custkey`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_partkey`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_orderpriority`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_ordtotalprice`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_revenue`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`p_mfgr`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`s_nation`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`c_city`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`c_nation`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_orderdate` FROM `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc` WHERE `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_orderdate` = '1997-01-01' EXPIRE_TIME: NULL 1 row in set (0.02 sec) ``` 2. Check the execution status in the table `task_runs` using the `TASK_NAME` you have found. ```plain mysql> select * from information_schema.task_runs where task_name='mv-59299' order by CREATE_TIME\G *************************** 1. row *************************** QUERY_ID: d9cef11f-7a00-11ed-bd90-00163e14767f TASK_NAME: mv-59299 CREATE_TIME: 2022-12-12 17:39:19 FINISH_TIME: 2022-12-12 17:39:22 STATE: SUCCESS DATABASE: ssb_1 DEFINITION: insert overwrite hive_mv_lineorder_flat_1 SELECT `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_orderkey`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_linenumber`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_custkey`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_partkey`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_orderpriority`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_ordtotalprice`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_revenue`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`p_mfgr`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`s_nation`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`c_city`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`c_nation`, `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_orderdate` FROM `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc` WHERE `hive_ci`.`dla_scan`.`lineorder_flat_1000_1000_orc`.`lo_orderdate` = '1997-01-01' EXPIRE_TIME: 2022-12-15 17:39:19 ERROR_CODE: 0 ERROR_MESSAGE: NULL PROGRESS: 100% 2 rows in set (0.02 sec) ``` ##### Drop an asynchronous materialized view[​](#drop-an-asynchronous-materialized-view "Direct link to Drop an asynchronous materialized view") You can drop an asynchronous materialized view via [DROP MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/DROP_MATERIALIZED_VIEW.md). ```plain DROP MATERIALIZED VIEW order_mv; ``` ##### Relevant session variables[​](#relevant-session-variables "Direct link to Relevant session variables") The following variables control the behavior of an asynchronous materialized view: * `analyze_mv`: Whether and how to analyze the materialized view after refresh. Valid values are an empty string (Do not analyze), `sample` (Sampled statistics collection), and `full` (Full statistics collection). Default is `sample`. * `enable_materialized_view_rewrite`: Whether to enable the automatic rewrite for materialized view. Valid values are `true` (Default since v2.5) and `false`. --- ### Understand Materialized View Task Runs The information of materialized view refresh task runs can help understand materialized view refresh behavior, troubleshoot issues, and monitor performance. #### Overview[​](#overview "Direct link to Overview") When the system refreshes a materialized view, it creates a task run that contains detailed information about the refresh operation. This information is stored in the `MVTaskRunExtraMessage` object and can be obtained by querying the `EXTRA_MESSAGE` field in the system view [`information_schema.task_runs`](https://docs.starrocks.io/docs/sql-reference/information_schema/task_runs.md). #### Extra message of task runs[​](#extra-message-of-task-runs "Direct link to Extra message of task runs") This section describes the fields provided in `MVTaskRunExtraMessage`. ###### `forceRefresh`[​](#forcerefresh "Direct link to forcerefresh") * **Type**: Boolean * **Description**: Indicates whether this is a forced refresh that bypasses normal refresh conditions. `true` is returned when a forced full refresh is manually triggered by running `REFRESH MATERIALIZED VIEW ... FORCE`. ###### `partitionStart`[​](#partitionstart "Direct link to partitionstart") * **Type**: String * **Description**: The starting partition boundary for this refresh operation. It defines the lower bound of the partition range to refresh. * **Format**: Partition key value (for example, `"2024-01-01"` for date-based partitions) * **Example**: When the data from January 2024 onwards is refreshed, this field would be `"2024-01-01"`. ###### `partitionEnd`[​](#partitionend "Direct link to partitionend") * **Type**: String * **Description**: The ending partition boundary for this refresh operation. It defines the upper bound of the partition range to refresh. * **Format**: Partition key value (for example, `"2024-01-31"` for date-based partitions) * **Example**: When the data up to January 2024 is refresh, this field would be `"2024-01-31"`. ###### `mvPartitionsToRefresh`[​](#mvpartitionstorefresh "Direct link to mvpartitionstorefresh") * **Type**: Set of Strings * **Description**: The list of materialized view partitions that are scheduled to be refreshed in the task run. This item helps you track which materialized view partitions will be updated. * **Size Limit**: The system will automatically truncated the number of partitions to `max_mv_task_run_meta_message_values_length` (Default: 8, or 16 in v4.0) to prevent excessive metadata storage if the actual number exceeds the value. * **Example**: `["p20240101", "p20240102", "p20240103"]` * **Note**: This represents the materialized view's own partitions, not the base table partitions. ###### `refBasePartitionsToRefreshMap`[​](#refbasepartitionstorefreshmap "Direct link to refbasepartitionstorefreshmap") * **Type**: Map of String to Set of Strings * **Description**: The mapping between reference base tables to the set of their partitions that should be refreshed. * **Usage**: The value is set during the materialized view plan scheduler stage. It can be used to track which base table partitions need to be scanned. * **Format**: `{tableName -> Set}` * **Size Limit**: The system will automatically truncated the number of partitions to `max_mv_task_run_meta_message_values_length` (Default: 8, or 16 in v4.0) to prevent excessive metadata storage if the actual number exceeds the value. * **Example**: ```json { "orders": ["p20240101", "p20240102"], "customers": ["p202401"] } ``` * **Note**: This is the **planned** set of partitions before optimization. ###### `basePartitionsToRefreshMap`[​](#basepartitionstorefreshmap "Direct link to basepartitionstorefreshmap") * **Type**: Map of String to Set of Strings * **Description**: The mapping between reference base tables to the set of their partitions that were refreshed during execution. * **Usage**: The value is set after the materialized view version map is committed. It reflects the real partitions used by the optimizer. * **Format**: `{tableName -> Set}` * **Size Limit**: The system will automatically truncated the number of partitions to `max_mv_task_run_meta_message_values_length` (Default: 8, or 16 in v4.0) to prevent excessive metadata storage if the actual number exceeds the value. * **Example**: ```json { "orders": ["p20240101", "p20240102"], "line_items": ["p20240101_batch1", "p20240101_batch2"] } ``` * **Note**: This is the **actual** set of partitions after query optimization and execution. note **Difference between refBasePartitionsToRefreshMap and basePartitionsToRefreshMap:** * `refBasePartitionsToRefreshMap`: Planned partitions before optimization (usually for the main reference table) * `basePartitionsToRefreshMap`: Actual partitions after optimization (includes all tables and optimized partition sets) ###### `nextPartitionStart`[​](#nextpartitionstart "Direct link to nextpartitionstart") * **Type**: String * **Description**: The starting partition boundary for the next incremental refresh. This value defines the lower bound of the partition range to be refreshed in the next task run when a refresh operation is split across multiple task runs due to resource limits or large data volumes. * **Example**: If the current task run refreshes the data up to `"2024-01-15"`, the value of this field might be `"2024-01-16"`. ###### `nextPartitionEnd`[​](#nextpartitionend "Direct link to nextpartitionend") * **Type**: String * **Description**: The ending partition boundary for the next incremental refresh. This value defines the upper bound of the partition range to be refreshed in the next task run when a refresh operation is split across multiple task runs due to resource limits or large data volumes. * **Example**: `"2024-01-31"` for the next batch of partitions to process. ###### `nextPartitionValues`[​](#nextpartitionvalues "Direct link to nextpartitionvalues") * **Type**: String * **Description**: Serialized partition values for the next refresh (used for list partitioning or complex partition schemes). It stores specific partition values when simple start/end ranges are insufficient. * **Example**: `{"data":[{"partitionItems":[["US","ACTIVE"]]},{"partitionItems":[["UK","ACTIVE"]]}]}` for multi-column list partitions. * **Archived task runs**: When a finished task run is archived to the task run history, this field and the `PARTITION_VALUES` property of the task run keep only the first `max_mv_task_run_meta_message_values_length` partition values. A `total` field records how many values there were, for example `{"data":[...],"total":998}`. The archived value is for display only. The next task run of the refresh is created with the full list. ###### `processStartTime`[​](#processstarttime "Direct link to processstarttime") * **Type**: Integer (timestamp in milliseconds) * **Description**: The timestamp when the task run actually started processing (pending time is excluded). It can be used to calculate the actual processing time using the formula `Processing Time = Finish Time - Process Start Time` (queue waiting time is excluded). * **Example**: `1704067200000` (2024-01-01 00:00:00 UTC) ###### `executeOption`[​](#executeoption "Direct link to executeoption") * **Type**: ExecuteOption object * **Description**: Configuration options for the task execution. * **Default**: `Priority` = `LOWEST`, `isMergeRedundant` = `false` * **Fields**: * `priority`: Task execution priority (values from `Constants.TaskRunPriority`) * `HIGHEST`: 0 * `HIGH`: 32 * `NORMAL`: 64 * `LOW`: 96 * `LOWEST`: 127 * `isMergeRedundant`: Whether to merge redundant refresh operations. * `properties`: Additional execution properties in the format `Map`. ###### `planBuilderMessage`[​](#planbuildermessage "Direct link to planbuildermessage") * **Type**: Map of String to String * **Description**: Diagnostic messages and metadata from the query plan builder. It contains information about query planning, optimization decisions, and potential issues. * **Size Limit**: The system will automatically truncated the number of partitions to `max_mv_task_run_meta_message_values_length` (Default: 8, or 16 in v4.0) to prevent excessive metadata storage if the actual number exceeds the value. ###### `refreshMode`[​](#refreshmode "Direct link to refreshmode") * **Type**: String * **Description**: The refresh mode of this task run. It indicates how the materialized view was refreshed. * **Valid Values**: * `"COMPLETE"`: Full refresh of all partitions * `"PARTIAL"`: Incremental refresh of specific partitions * `"FORCE"`: Forced refresh bypassing staleness checks * `""` (empty): Default or unspecified ###### `adaptivePartitionRefreshNumber`[​](#adaptivepartitionrefreshnumber "Direct link to adaptivepartitionrefreshnumber") * **Type**: Integer * **Description**: The number of partitions that should be refreshed in each iteration when adaptive partition refresh is used. This value is automatically determined based on system resources and data volume to optimize refresh performance. * **Default**: `-1` (indicates that adaptive refresh is not set or used) * **Example**: `10` (indicates to refresh 10 partitions at a time) #### Query Task Run Details[​](#query-task-run-details "Direct link to Query Task Run Details") You can query materialized view task run information through the system view `information_schema.task_runs`. ```sql SELECT TASK_NAME, CREATE_TIME, FINISH_TIME, STATE, EXTRA_MESSAGE FROM information_schema.task_runs WHERE TASK_NAME LIKE 'mv-%' ORDER BY CREATE_TIME DESC LIMIT 10; ``` The `EXTRA_MESSAGE` column contains the JSON representation of `MVTaskRunExtraMessage`. You can further parse the JSON string `EXTRA_MESSAGE` for better readability. ```sql SELECT TASK_NAME, CREATE_TIME, get_json_string(EXTRA_MESSAGE, '$.refreshMode') AS refresh_mode, get_json_string(EXTRA_MESSAGE, '$.forceRefresh') AS force_refresh, get_json_string(EXTRA_MESSAGE, '$.mvPartitionsToRefresh') AS mv_partitions, get_json_int(EXTRA_MESSAGE, '$.processStartTime') AS process_start_ms, get_json_int(EXTRA_MESSAGE, '$.adaptivePartitionRefreshNumber') AS adaptive_batch_size FROM information_schema.task_runs WHERE TASK_NAME = 'mv-12345' ORDER BY CREATE_TIME DESC; ``` #### Understand Refresh Performance[​](#understand-refresh-performance "Direct link to Understand Refresh Performance") ##### Calculate Processing Time[​](#calculate-processing-time "Direct link to Calculate Processing Time") ```sql SELECT TASK_NAME, FINISH_TIME, get_json_bigint(EXTRA_MESSAGE, '$.processStartTime') AS process_start_time, (unix_timestamp(FINISH_TIME) * 1000 - get_json_bigint(EXTRA_MESSAGE, '$.processStartTime')) / 1000 AS processing_seconds FROM information_schema.task_runs WHERE TASK_NAME LIKE 'mv-%' AND STATE = 'SUCCESS'; ``` ##### Analyze Partition Refresh Patterns[​](#analyze-partition-refresh-patterns "Direct link to Analyze Partition Refresh Patterns") ```sql SELECT TASK_NAME, CREATE_TIME, get_json_string(EXTRA_MESSAGE, '$.partitionStart') AS start_partition, get_json_string(EXTRA_MESSAGE, '$.partitionEnd') AS end_partition, get_json_string(EXTRA_MESSAGE, '$.nextPartitionStart') AS next_start, get_json_string(EXTRA_MESSAGE, '$.nextPartitionEnd') AS next_end FROM information_schema.task_runs WHERE TASK_NAME = 'mv-12345' ORDER BY CREATE_TIME DESC; ``` #### Configuration Items[​](#configuration-items "Direct link to Configuration Items") ##### max\_mv\_task\_run\_meta\_message\_values\_length[​](#max_mv_task_run_meta_message_values_length "Direct link to max_mv_task_run_meta_message_values_length") * **Type**: Integer * **Default**: 8 (16 in v4.0) * **Scope**: FE configuration * **Description**: Maximum number of items to store in the set or the MAP fields to prevent excessive metadata growth. It limits the size of `mvPartitionsToRefresh`, `refBasePartitionsToRefreshMap`, `basePartitionsToRefreshMap`, and `planBuilderMessage`. For archived task runs, it also limits the list partition values kept in `nextPartitionValues` and in the `PARTITION_VALUES` property. #### Best Practices[​](#best-practices "Direct link to Best Practices") ##### Monitor Refresh Performance[​](#monitor-refresh-performance "Direct link to Monitor Refresh Performance") * Compare `processStartTime` with the actual finish time to identify queueing issues. * Use `adaptivePartitionRefreshNumber` to optimize batch sizes. ##### Debug Failed Refreshes[​](#debug-failed-refreshes "Direct link to Debug Failed Refreshes") * Check `planBuilderMessage` for optimizer issues. * Compare `refBasePartitionsToRefreshMap` with `basePartitionsToRefreshMap` for partition pruning problems. ##### Optimize Incremental Refreshes[​](#optimize-incremental-refreshes "Direct link to Optimize Incremental Refreshes") * Monitor `nextPartitionStart` and `nextPartitionEnd` to understand multi-iteration refresh patterns. * Adjust partition granularity if refreshes frequently span multiple runs. ##### Understand Partition Coverage[​](#understand-partition-coverage "Direct link to Understand Partition Coverage") * Compare `mvPartitionsToRefresh` with `basePartitionsToRefreshMap` to check the materialized view-to-base-table partition mapping. * Verify whether base table partitions align with expected refresh ranges. #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") ##### Issue: Refresh Takes Too Long[​](#issue-refresh-takes-too-long "Direct link to Issue: Refresh Takes Too Long") **Check:** 1. `processStartTime` - A significant difference from the create time indicates the task run were in the queue. 2. `basePartitionsToRefreshMap` - A significant value indicates too many partitions being scanned. 3. `adaptivePartitionRefreshNumber` - You may need to tune the workload. ##### Issue: Unexpected Partitions Refreshed[​](#issue-unexpected-partitions-refreshed "Direct link to Issue: Unexpected Partitions Refreshed") **Check:** 1. `forceRefresh` - If `true` is returned, it indicates that a forced full refresh is performed. 2. `refBasePartitionsToRefreshMap` - This field shows the planned partitions. 3. `basePartitionsToRefreshMap` - This field shows the actual partitions after optimization. 4. Compare the above two maps to see if optimizer changed the plan. ##### Issue: Refresh Stuck in Multiple Iterations[​](#issue-refresh-stuck-in-multiple-iterations "Direct link to Issue: Refresh Stuck in Multiple Iterations") **Check:** 1. `nextPartitionStart` and `nextPartitionEnd` - This field shows the incomplete refresh state. 2. `adaptivePartitionRefreshNumber` - You may need to tune the workload. 3. Consider to increase batch size or reducing partition granularity. #### See Also[​](#see-also "Direct link to See Also") * [CREATE MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md) * [REFRESH MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/REFRESH_MATERIALIZED_VIEW.md) * [Information Schema.task\_runs](https://docs.starrocks.io/docs/sql-reference/information_schema/task_runs.md) * [Materialized View Management](https://docs.starrocks.io/docs/using_starrocks/async_mv/Materialized_view.md) --- ### Asynchronous Materialized View References * [CREATE MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md) * [ALTER MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/ALTER_MATERIALIZED_VIEW.md) * [DROP MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/DROP_MATERIALIZED_VIEW.md) * [SHOW MATERIALIZED VIEWS](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/SHOW_MATERIALIZED_VIEW.md) * [Information\_schema.materialized\_views](https://docs.starrocks.io/docs/sql-reference/information_schema/materialized_views.md) --- ### Troubleshooting asynchronous materialized views This topic describes how to examine your asynchronous materialized views and solve the problems you have encountered while working with them. > **CAUTION** > > Some of the features shown below are only supported from StarRocks v3.1 onwards. #### Examine an asynchronous materialized view[​](#examine-an-asynchronous-materialized-view "Direct link to Examine an asynchronous materialized view") To get a whole picture of asynchronous materialized views that you are working with, you can first check their working state, refresh history, and resource consumption. ##### Check the working state of an asynchronous materialized view[​](#check-the-working-state-of-an-asynchronous-materialized-view "Direct link to Check the working state of an asynchronous materialized view") You can check the working state of an asynchronous materialized view using [SHOW MATERIALIZED VIEWS](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/SHOW_MATERIALIZED_VIEW.md). Among all the information returned, you can focus on the following fields: * `is_active`: Whether the state of the materialized view is active. Only an active materialized view can be used for query acceleration and rewrite. * `last_refresh_state`: The state of the last refresh, including PENDING, RUNNING, FAILED, and SUCCESS. * `last_refresh_error_message`: The reason why the last refresh failed (if the materialized view state is not active). * `rows`: The number of data rows in the materialized view. Please note that this value can be different from the actual row count of the materialized view because the updates can be deferred. For detailed information on other fields returned, see [SHOW MATERIALIZED VIEWS - Returns](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/SHOW_MATERIALIZED_VIEW.md#returns). Example: ```plain MySQL > SHOW MATERIALIZED VIEWS LIKE 'mv_pred_2'\G ***************************[ 1. row ]*************************** id | 112517 database_name | ssb_1g name | mv_pred_2 refresh_type | ASYNC is_active | true inactive_reason | partition_type | UNPARTITIONED task_id | 457930 task_name | mv-112517 last_refresh_start_time | 2023-08-04 16:46:50 last_refresh_finished_time | 2023-08-04 16:46:54 last_refresh_duration | 3.996 last_refresh_state | SUCCESS last_refresh_force_refresh | false last_refresh_start_partition | last_refresh_end_partition | last_refresh_base_refresh_partitions | {} last_refresh_mv_refresh_partitions | last_refresh_error_code | 0 last_refresh_error_message | rows | 0 text | CREATE MATERIALIZED VIEW `mv_pred_2` (`lo_quantity`, `lo_revenue`, `sum`) DISTRIBUTED BY HASH(`lo_quantity`, `lo_revenue`) BUCKETS 2 REFRESH ASYNC PROPERTIES ( "replication_num" = "3", "storage_medium" = "HDD" ) AS SELECT `lineorder`.`lo_quantity`, `lineorder`.`lo_revenue`, sum(`lineorder`.`lo_tax`) AS `sum` FROM `ssb_1g`.`lineorder` WHERE `lineorder`.`lo_linenumber` = 1 GROUP BY 1, 2; 1 row in set Time: 0.003s ``` ##### View the refresh history of an asynchronous materialized view[​](#view-the-refresh-history-of-an-asynchronous-materialized-view "Direct link to View the refresh history of an asynchronous materialized view") You can view the refresh history of an asynchronous materialized view by querying the table `task_runs` in the database `information_schema`. Among all the information returned, you can focus on the following fields: * `CREATE_TIME` and `FINISH_TIME`: The start and end time of the refresh task. * `STATE`: The state of the refresh task, including PENDING, RUNNING, FAILED, and SUCCESS. * `ERROR_MESSAGE`: The reason why the refresh task failed. Example: ```plain MySQL > SELECT * FROM information_schema.task_runs WHERE task_name ='mv-112517' \G ***************************[ 1. row ]*************************** QUERY_ID | 7434cee5-32a3-11ee-b73a-8e20563011de TASK_NAME | mv-112517 CREATE_TIME | 2023-08-04 16:46:50 FINISH_TIME | 2023-08-04 16:46:54 STATE | SUCCESS DATABASE | ssb_1g EXPIRE_TIME | 2023-08-05 16:46:50 ERROR_CODE | 0 ERROR_MESSAGE | PROGRESS | 100% EXTRA_MESSAGE | {"forceRefresh":false,"mvPartitionsToRefresh":[],"refBasePartitionsToRefreshMap":{},"basePartitionsToRefreshMap":{}} PROPERTIES | {"FORCE":"false"} ***************************[ 2. row ]*************************** QUERY_ID | 72dd2f16-32a3-11ee-b73a-8e20563011de TASK_NAME | mv-112517 CREATE_TIME | 2023-08-04 16:46:48 FINISH_TIME | 2023-08-04 16:46:53 STATE | SUCCESS DATABASE | ssb_1g EXPIRE_TIME | 2023-08-05 16:46:48 ERROR_CODE | 0 ERROR_MESSAGE | PROGRESS | 100% EXTRA_MESSAGE | {"forceRefresh":true,"mvPartitionsToRefresh":["mv_pred_2"],"refBasePartitionsToRefreshMap":{},"basePartitionsToRefreshMap":{"lineorder":["lineorder"]}} PROPERTIES | {"FORCE":"true"} ``` ##### Monitor the resource consumption of an asynchronous materialized view[​](#monitor-the-resource-consumption-of-an-asynchronous-materialized-view "Direct link to Monitor the resource consumption of an asynchronous materialized view") You can monitor and analyze the resource consumed by an asynchronous materialized view during and after the refresh. ###### Monitor resource consumption during refresh[​](#monitor-resource-consumption-during-refresh "Direct link to Monitor resource consumption during refresh") During a refresh task, you can monitor its real-time resource consumption using [SHOW PROC '/current\_queries'](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md). Among all the information returned, you can focus on the following fields: * `ScanBytes`: The size of data that is scanned. * `ScanRows`: The number of data rows that are scanned. * `MemoryUsage`: The size of memory that is used. * `CPUTime`: CPU time cost. * `ExecTime`: The execution time of the query. Example: ```plain MySQL > SHOW PROC '/current_queries'\G ***************************[ 1. row ]*************************** StartTime | 2023-08-04 17:01:30 QueryId | 806eed7d-32a5-11ee-b73a-8e20563011de ConnectionId | 0 Database | ssb_1g User | root ScanBytes | 70.981 MB ScanRows | 6001215 rows MemoryUsage | 73.748 MB DiskSpillSize | 0.000 CPUTime | 2.515 s ExecTime | 2.583 s ``` ###### Analyze resource consumption after refresh[​](#analyze-resource-consumption-after-refresh "Direct link to Analyze resource consumption after refresh") After a refresh task, you can analyze its resource consumption via query profiles. You can view the profile of the materialized view refresh task through the Web UI of the cluster Leader FE node. While an asynchronous materialized view refreshes itself, an INSERT OVERWRITE statement is executed. You can check the corresponding query profile to analyze the time and resources consumed by the refresh task. Among all the information returned, you can focus on the following metrics: * `Total`: Total time consumed by the query. * `QueryCpuCost`: Total CPU time cost of the query. CPU time costs are aggregated for concurrent processes. As a result, the value of this metric may be greater than the actual execution time of the query. * `QueryMemCost`: Total memory cost of the query. * Other metrics for individual operators, such as join operators and aggregate operators. For detailed information on how to check the query profile and understand other metrics, see [Analyze query profile](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md). ##### Verify whether queries are rewritten by an asynchronous materialized view[​](#verify-whether-queries-are-rewritten-by-an-asynchronous-materialized-view "Direct link to Verify whether queries are rewritten by an asynchronous materialized view") You can check whether a query can be rewritten with an asynchronous materialized view from its query plan using [EXPLAIN](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/EXPLAIN.md). If the metric `SCAN` in the query plan shows the name of the corresponding materialized view, the query has been rewritten by the materialized view. Example 1: ```plain MySQL > SHOW CREATE TABLE mv_agg\G ***************************[ 1. row ]*************************** Materialized View | mv_agg Create Materialized View | CREATE MATERIALIZED VIEW `mv_agg` (`c_custkey`) DISTRIBUTED BY RANDOM REFRESH ASYNC PROPERTIES ( "replication_num" = "3", "replicated_storage" = "true", "storage_medium" = "HDD" ) AS SELECT `customer`.`c_custkey` FROM `ssb_1g`.`customer` GROUP BY `customer`.`c_custkey`; MySQL > EXPLAIN LOGICAL SELECT `customer`.`c_custkey` -> FROM `ssb_1g`.`customer` -> GROUP BY `customer`.`c_custkey`; +-----------------------------------------------------------------------------------+ | Explain String | +-----------------------------------------------------------------------------------+ | - Output => [1:c_custkey] | | - SCAN [mv_agg] => [1:c_custkey] | | Estimates: {row: 30000, cpu: ?, memory: ?, network: ?, cost: 15000.0} | | partitionRatio: 1/1, tabletRatio: 12/12 | | 1:c_custkey := 10:c_custkey | +-----------------------------------------------------------------------------------+ ``` If you disable the query rewrite feature, StarRocks adopts the regular query plan. Example 2: ```plain MySQL > SET enable_materialized_view_rewrite = false; MySQL > EXPLAIN LOGICAL SELECT `customer`.`c_custkey` -> FROM `ssb_1g`.`customer` -> GROUP BY `customer`.`c_custkey`; +---------------------------------------------------------------------------------------+ | Explain String | +---------------------------------------------------------------------------------------+ | - Output => [1:c_custkey] | | - AGGREGATE(GLOBAL) [1:c_custkey] | | Estimates: {row: 15000, cpu: ?, memory: ?, network: ?, cost: 120000.0} | | - SCAN [mv_bitmap] => [1:c_custkey] | | Estimates: {row: 60000, cpu: ?, memory: ?, network: ?, cost: 30000.0} | | partitionRatio: 1/1, tabletRatio: 12/12 | +---------------------------------------------------------------------------------------+ ``` #### Diagnose and solve problems[​](#diagnose-and-solve-problems "Direct link to Diagnose and solve problems") Here we list some common problems you might encounter while working with an asynchronous materialized view, and the corresponding solutions. ##### Materialized view building failure[​](#materialized-view-building-failure "Direct link to Materialized view building failure") If you failed to create an asynchronous materialized view, that is, the CREATE MATERIALIZED VIEW statement cannot be executed, you can look into the following aspects: * **Check whether you have mistakenly used the SQL statement for synchronous materialized views.** StarRocks provides two different materialized views: the synchronous materialized view and the asynchronous materialized view. The basic SQL statement used to create a synchronous materialized view is as follows: ```sql CREATE MATERIALIZED VIEW AS ``` However, the SQL statement used to create an asynchronous materialized view contains more parameters: ```sql CREATE MATERIALIZED VIEW REFRESH ASYNC -- The refresh strategy of the asynchronous materialized view. DISTRIBUTED BY HASH() -- The data distribution strategy of the asynchronous materialized view. AS ``` In addition to the SQL statement, the main difference between the two materialized views is that asynchronous materialized views support all query syntax that StarRocks provides, but synchronous materialized views only support limited choices of aggregate functions. * **Check whether you have specified a correct partition column.** When creating an asynchronous materialized view, you can specify a partitioning strategy, which allows you to refresh the materialized view on a finer granularity level. Currently, StarRocks only supports range partitioning, and only supports referencing a single column from the SELECT expression in the query statement that is used to build the materialized view. You can use the date\_trunc() function to truncate the column to change the granularity level of the partitioning strategy. Please note that any other expressions are not supported. * **Check whether you have the necessary privileges to create the materialized view.** When creating an asynchronous materialized view, you need the SELECT privileges of all objects (tables, views, materialized views) that are queried. When UDFs are used in the query, you also need the USAGE privileges of the functions. ##### Materialized view refresh failure[​](#materialized-view-refresh-failure "Direct link to Materialized view refresh failure") If the materialized view fails to refresh, that is, the state of the refresh task is not SUCCESS, you can look into the following aspects: * **Check whether you have adopted an inappropriate refresh strategy.** By default, the materialized view refreshes immediately after it is created. However, in v2.5 and early versions, materialized views that adopt the MANUAL refresh strategy do not refresh after being created. You must refresh it manually using REFRESH MATERIALIZED VIEW. * **Check whether the refresh task exceeds the memory limit.** Usually, when an asynchronous materialized view involves large-scale aggregation or join calculations that exhaust the memory resources. To solve this problem, you can: * Specify a partitioning strategy for the materialized view to refresh one partition each time. * Enable the Spill to Disk feature for the refresh task. From v3.1 onwards, StarRocks supports spilling the intermediate results to disks when refreshing a materialized view. Execute the following statement to enable Spill to Disk: ```sql -- Define the properties when creating the materialized view. CREATE MATERIALIZED VIEW mv1 REFRESH ASYNC PROPERTIES ( 'session.enable_spill'='true' ) AS ; -- Add the properties to an existing materialized view. ALTER MATERIALIZED VIEW mv2 SET ('session.enable_spill' = 'true'); ``` ##### Materialized view refresh timeout[​](#materialized-view-refresh-timeout "Direct link to Materialized view refresh timeout") Large materialized views may fail to refresh because the refresh task exceeds the timeout period. You can consider the following solutions to solve this problem: * **Specify a partitioning strategy for the materialized view to achieve fine-grained refresh** As described in [Create partitioned materialized views](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md), by partitioning the materialized view you can achieve incremental building and refresh, thus avoiding the issue of excessive resource consumption during the initial refresh. * **Set a longer timeout period** The default timeout for materialized view refresh tasks is 5 minutes in versions earlier than v3.2 and 1 hour in v3.2 and later. If you encounter timeout exceptions, you can adjust the timeout period by using the following statement: ```sql ALTER MATERIALIZED VIEW mv2 SET ('session.insert_timeout' = '4000'); ``` * **Analyze performance bottlenecks of the materialized view refresh** Refreshing materialized views with complex computation is time-consuming. You can analyze its performance bottlenecks by analyzing the query profile of the refresh task: * Obtain the `query_id` corresponding to the refresh task by querying `information_schema.task_runs`. * Analyze the query profile of the refresh task using the following statements: * [GET\_QUERY\_PROFILE](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/get_query_profile.md): Retrive the original query profile based on `query_id`. * [ANALYZE PROFILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/ANALYZE_PROFILE.md): Analyze the query profile on a per-fragment basis, and display it in a tree structure. ##### Materialized view state is not active[​](#materialized-view-state-is-not-active "Direct link to Materialized view state is not active") If the materialized view cannot rewrite queries or refresh, and the state of the materialized view `is_active` is `false`, it could be a result of the schema change on the base tables. To solve this problem, you can manually set the materialized view state to active by executing the following statement: ```sql ALTER MATERIALIZED VIEW mv1 ACTIVE; ``` If setting the materialized view state to active does not take effect, you need to drop the materialized view and create it again. ##### Materialized view refresh task uses excessive resources[​](#materialized-view-refresh-task-uses-excessive-resources "Direct link to Materialized view refresh task uses excessive resources") If you find that the refresh tasks are using excessive system resources, you can look into the following aspects: * **Check whether you have created a materialized view too large.** If you have joined too many tables that cause a significant amount of calculation, the refresh task will occupy many resources. To solve this problem, you need to assess the size of the materialized view and re-plan it. * **Check whether you have set an unnecessarily frequent refresh interval.** If you adopt the fixed-interval refresh strategy, you can set a lower refresh frequency to solve the problem. If the refresh tasks are triggered by data changes in the base tables, loading data too frequently can also cause this problem. To solve this problem, you need to define a proper refresh strategy for your materialized view. * **Check whether the materialized view is partitioned.** An unpartitioned materialized view can be costly to refresh because StarRocks refreshes the whole materialized view each time. To solve this problem, you need to specify a partitioning strategy for the materialized view to refresh one partition each time. To stop a refresh task that occupies too many resources, you can: * Set the materialized view state to inactive so that all of its refresh tasks are stopped: ```sql ALTER MATERIALIZED VIEW mv1 INACTIVE; ``` * Terminate the running refresh task using [CANCEL REFRESH MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/CANCEL_REFRESH_MATERIALIZED_VIEW.md): ```sql CANCEL REFRESH MATERIALIZED VIEW mv1; ``` ##### Materialized view query rewrite failure[​](#materialized-view-query-rewrite-failure "Direct link to Materialized view query rewrite failure") If your materialized view fails to rewrite relevant queries, you can look into the following aspects: * **Diagnose rewrite failure using TRACE** StarRocks provides the TRACE statement for you to diagnose materialized view rewrite failure: * `TRACE LOGS MV `: Available in v3.2 and later, this command analyzes the detailed rewrite process and the reasons of the failure. * `TRACE REASON MV `: Available in v3.2.8 and later, this command provides concise reasons for rewrite failure. ```sql MySQL > TRACE REASON MV SELECT sum(c1) FROM `glue_ice`.`iceberg_test`.`ice_test3`; +----------------------------------------------------------------------------------------------------------------------+ | Explain String | +----------------------------------------------------------------------------------------------------------------------+ | MV rewrite fail for mv1: Rewrite aggregate rollup sum(1: c1) failed: only column-ref is supported after rewrite | | MV rewrite fail for mv1: Rewrite aggregate function failed, cannot get rollup function: sum(1: c1) | | MV rewrite fail for mv1: Rewrite rollup aggregate failed: cannot rewrite aggregate functions | +----------------------------------------------------------------------------------------------------------------------+ ``` * **Check whether the materialized view and the query match.** * StarRocks matches the materialized view and the query with a structure-based matching technique rather than text-based matching. Therefore, it's not guaranteed that a query can be rewritten just because it appears similar to that of a materialized view. * Materialized views can only rewrite SPJG (Select/Projection/Join/Aggregation) type of queries. Queries involving window functions, nested aggregation, or join plus aggregation are not supported. * Materialized views cannot rewrite queries that involve complex Join predicates in Outer Joins. For example, in cases like `A LEFT JOIN B ON A.dt > '2023-01-01' AND A.id = B.id`, we recommend you specify the predicate from the `JOIN ON` clause in a `WHERE` clause. For more information on the limitations of the materialized view query rewrite, see [Query rewrite with materialized views - Limitations](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md#limitations). * **Check whether the materialized view state is active.** StarRocks checks the status of the materialized view before rewriting queries. Queries can be rewritten only when the materialized view state is active. To solve this problem, you can manually set the materialized view state to active by executing the following statement: ```sql ALTER MATERIALIZED VIEW mv1 ACTIVE; ``` * **Check whether the materialized view meets the data consistency requirements.** StarRocks checks the consistency of data in the materialized view and in the base table data. By default, queries can be rewritten only when the data in the materialized view is up-to-date. To solve this problem, you can: * Add `PROPERTIES('query_rewrite_consistency'='LOOSE')` to the materialized view to disable consistency checks. * Add `PROPERTIES('mv_rewrite_staleness_second'='5')` to tolerate a certain degree of data inconsistency. Queries can be rewritten if the materialized view's last confirmed complete refresh is within this time interval, regardless of whether the data in the base tables changes. A refresh covering only part of the partitions does not renew this baseline. * **Check whether the query statement of the materialized view lacks output columns.** To rewrite range and point queries, you must specify the columns used as the filtering predicates in the SELECT expression of the materialized view's query statement. You need to check the SELECT statement of the materialized view to ensure it includes columns referenced in the WHERE and ORDER BY clauses of the query. Example 1: The materialized view `mv1` uses nested aggregation. Thus, it cannot be used to rewrite queries. ```sql CREATE MATERIALIZED VIEW mv1 REFRESH ASYNC AS select count(distinct cnt) from ( select c_city, count(*) cnt from customer group by c_city ) t; ``` Example 2: The materialized view `mv2` uses join plus aggregation. Thus, it cannot be used to rewrite queries. To solve this problem, you can create a materialized view with aggregation, and then a nested materialized view with join based on the previous one. ```sql CREATE MATERIALIZED VIEW mv2 REFRESH ASYNC AS select * from ( select lo_orderkey, lo_custkey, p_partkey, p_name from lineorder join part on lo_partkey = p_partkey ) lo join ( select c_custkey from customer group by c_custkey ) cust on lo.lo_custkey = cust.c_custkey; ``` Example 3: The materialized view `mv3` cannot rewrite queries in the pattern of `SELECT c_city, sum(tax) FROM tbl WHERE dt='2023-01-01' AND c_city = 'xxx'` because the column that the predicate referenced is not in the SELECT expression. ```sql CREATE MATERIALIZED VIEW mv3 REFRESH ASYNC AS SELECT c_city, sum(tax) FROM tbl GROUP BY c_city; ``` To solve this problem, you can create the materialized view as follows: ```sql CREATE MATERIALIZED VIEW mv3 REFRESH ASYNC AS SELECT dt, c_city, sum(tax) FROM tbl GROUP BY dt, c_city; ``` #### FAQ[​](#faq "Direct link to FAQ") ###### Which resource group controls the resources for asynchronous materialized views? How can I adjust the configuration of it?[​](#which-resource-group-controls-the-resources-for-asynchronous-materialized-views-how-can-i-adjust-the-configuration-of-it "Direct link to Which resource group controls the resources for asynchronous materialized views? How can I adjust the configuration of it?") If the `resource_group` property is not specified when creating an asynchronous materialized view, the system assigns it to the default resource group `default_mv_wg`. Its default configuration is: * `cpu_core_limit`: 1 * `mem_limit`: 80% * `concurrency_limit`: 0 * `spill_mem_limit_threshold`: 80% You can adjust its CPU limit, memory limit, concurrency limit, and spill threshold through the following BE configuration items: * `default_mv_resource_group_cpu_limit` * `default_mv_resource_group_memory_limit` * `default_mv_resource_group_concurrency_limit` * `default_mv_resource_group_spill_mem_limit_threshold` You can also assign a dedicated resource group to a materialized view by specifying the `resource_group` property. Example: ```sql CREATE MATERIALIZED VIEW IF NOT EXISTS mv_test.test_1 PARTITION BY `metric_date` REFRESH MANUAL PROPERTIES ( "replicated_storage" = "true", "partition_refresh_number" = "1", "force_external_table_query_rewrite" = "CHECKED", "query_rewrite_consistency" = "LOOSE", "replication_num" = "1", "storage_medium" = "HDD", # highlight-start "resource_group" = "rg_mv" # highlight-end ) AS ... ``` ###### If an asynchronous materialized view is scheduled to refresh every 1 minute, what happens if the refresh takes longer than 1 minute?[​](#if-an-asynchronous-materialized-view-is-scheduled-to-refresh-every-1-minute-what-happens-if-the-refresh-takes-longer-than-1-minute "Direct link to If an asynchronous materialized view is scheduled to refresh every 1 minute, what happens if the refresh takes longer than 1 minute?") The system handles this automatically: * If there are pending tasks in the queue, the newly triggered task is merged (status `MERGED`). * If there are no pending tasks (only running or succeeded/failed tasks), the newly triggered task waits until the current running task finishes before executing. Limitation: Only one refresh task per materialized view can run at a time. ###### After setting auto\_refresh\_partitions\_limit, will the system always refresh exactly that number of partitions?[​](#after-setting-auto_refresh_partitions_limit-will-the-system-always-refresh-exactly-that-number-of-partitions "Direct link to After setting auto_refresh_partitions_limit, will the system always refresh exactly that number of partitions?") No. This parameter represents an upper limit. If the system detects that fewer base-table partitions have changed, only those actual partitions will be refreshed. ###### A materialized view is configured to refresh every 5 minutes, but in `information_schema.task_runs` it appears to refresh every 5 seconds. Why?[​](#a-materialized-view-is-configured-to-refresh-every-5-minutes-but-in-information_schematask_runs-it-appears-to-refresh-every-5-seconds-why "Direct link to a-materialized-view-is-configured-to-refresh-every-5-minutes-but-in-information_schematask_runs-it-appears-to-refresh-every-5-seconds-why") From v3.3 onwards, the default value of `partition_refresh_number` changed from -1 to 1, meaning each refresh operation only processes one partition at a time. If many partitions need to be refreshed, the system splits the work into multiple subtasks, which results in more frequent scheduling entries in `task_runs` than the user-defined refresh interval. ###### Can multiple asynchronous materialized views be nested? Any recommendations?[​](#can-multiple-asynchronous-materialized-views-be-nested-any-recommendations "Direct link to Can multiple asynchronous materialized views be nested? Any recommendations?") Nesting is supported, but it is recommended to minimize nesting depth. Excessive nesting significantly increases troubleshooting complexity. --- ### Use cases #### [📄️ Data modeling with materialized views](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/data_modeling_with_materialized_views.md) [How to use asynchronous materialized views in StarRocks to simplify ETL pipelines, build reusable data models, and improve query performance through layered...](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/data_modeling_with_materialized_views.md) #### [📄️ Query rewrite with materialized views](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md) [How StarRocks automatically rewrites queries to use asynchronous materialized views for transparent query acceleration, including supported rewrite...](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md) #### [📄️ Data lake query acceleration with materialized views](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md) [How to use asynchronous materialized views in StarRocks to accelerate queries against data lake tables (Iceberg, Hive, Hudi, Delta Lake) through transparent...](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md) #### [📄️ Create a partitioned materialized view](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md) [How to create a partitioned materialized view in StarRocks with time-range or list partitioning for incremental refresh and partition-level query rewrite.](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md) --- ### Create a partitioned materialized view This topic introduces how to create a partitioned materialized view to accommodate different use cases. #### Overview[​](#overview "Direct link to Overview") StarRocks' asynchronous materialized views support a variety of partitioning strategies and functions that allow you to achieve the following effects: * **Incremental construction** When creating a partitioned materialized view, you can set the creation task to refresh partitions in batches to avoid excessive resource consumption. * **Incremental refresh** You can set the refresh task to update only the corresponding partitions of the materialized view when it detects data changes in certain partitions of the base table. Partition-level refresh can significantly prevent the waste of resources used to refresh the entire materialized view. * **Partial materialization** You can set TTL for materialized view partitions, allowing for partial materialization of the data. * **Transparent query rewrite** Queries can be rewritten transparently based on only those updated materialized view partitions. Partitions that are deemed outdated will not be involved in the query plan, and the query will be executed on the base tables to guarantee the consistency of data. #### Limitations[​](#limitations "Direct link to Limitations") A partitioned materialized view can be created only on a partitioned base table (usually a fact table). Only by mapping the partition relationship between the base table and the materialized view can you build the synergy between them. Currently, StarRocks supports building partitioned materialized views on tables from the following data sources: * **StarRocks OLAP tables in the default catalog** * Supported partitioning strategy: Range partitioning, List Partitioning, and Expression Partitioning * Supported data types for Partitioning Key: INT, DATE, DATETIME, and STRING * Supported table types: Primary Key, Duplicate Key, Aggregate Key, and Unique Key * Supported both in shared-nothing cluster and shared-data cluster * **Tables in Hive Catalog, Hudi Catalog, Iceberg Catalog, and Paimon Catalog** * Supported partitioning level: Primary level * Supported data types for Partitioning Key: INT, DATE, DATETIME, and STRING note * You cannot create a partitioned materialized view based on a non-partitioned base (fact) table. * For StarRocks OLAP tables, the two adjacent partitions of the base table must have consecutive ranges. * For multi-level partitioned base tables in external catalogs, only the primary level partitioning path can be used to create a partitioned materialized view. For example, for a table partitioned in the `yyyyMMdd/hour` format, you can only build the materialized view partitioned by `yyyyMMdd`. * From v3.2.3, StarRocks supports creating partitioned materialized views upon Iceberg tables with [Partition Transforms](https://iceberg.apache.org/spec/#partition-transforms), and the materialized views are partitioned by the column after the transformation. For more information, see [Data lake query acceleration with materialized views - Choose a suitable refresh strategy](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md#choose-a-suitable-refresh-strategy). #### Use cases[​](#use-cases "Direct link to Use cases") Suppose there are base tables as follows: ```sql CREATE TABLE IF NOT EXISTS par_tbl1 ( datekey DATE, -- DATE type date column used as the Partitioning Key. k1 STRING, v1 INT, v2 INT ) ENGINE=olap PARTITION BY RANGE (datekey) ( START ("2021-01-01") END ("2021-01-04") EVERY (INTERVAL 1 DAY) ) DISTRIBUTED BY HASH(k1); CREATE TABLE IF NOT EXISTS par_tbl2 ( datekey STRING, -- STRING type date column used as the Partitioning Key. k1 STRING, v1 INT, v2 INT ) ENGINE=olap PARTITION BY RANGE (str2date(datekey, '%Y-%m-%d')) ( START ("2021-01-01") END ("2021-01-04") EVERY (INTERVAL 1 DAY) ) DISTRIBUTED BY HASH(k1); CREATE TABLE IF NOT EXISTS par_tbl3 ( datekey_new DATE, -- Equivalent column with par_tbl1.datekey. k1 STRING, v1 INT, v2 INT ) ENGINE=olap PARTITION BY RANGE (datekey_new) ( START ("2021-01-01") END ("2021-01-04") EVERY (INTERVAL 1 DAY) ) DISTRIBUTED BY HASH(k1); ``` ##### Align partitions one-to-one[​](#align-partitions-one-to-one "Direct link to Align partitions one-to-one") You can create a materialized view whose partitions correspond to the partitions of the base table one-to-one by using the same Partitioning Key. ![Partitioned Materialized View-1](/assets/images/partitioned_mv-1-81675a206fb84095ac0016936f056ede.png) * If the Partitioning Key of the base table is the DATE or DATETIME type, you can directly specify the same Partitioning Key for the materialized view. ```sql PARTITION BY ``` Example: ```sql CREATE MATERIALIZED VIEW par_mv1 REFRESH ASYNC PARTITION BY datekey AS SELECT k1, sum(v1) AS SUM, datekey FROM par_tbl1 GROUP BY datekey, k1; ``` * If the Partitioning Key of the base table is the STRING type, you can use the [str2date](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/str2date.md) function to convert the date string into the DATE or DATETIME type. ```sql PARTITION BY str2date(, ) ``` Example: ```sql CREATE MATERIALIZED VIEW par_mv2 REFRESH ASYNC PARTITION BY str2date(datekey, '%Y-%m-%d') AS SELECT k1, sum(v1) AS SUM, datekey FROM par_tbl2 GROUP BY datekey, k1; ``` ##### Align partitions with time granularity rollup[​](#align-partitions-with-time-granularity-rollup "Direct link to Align partitions with time granularity rollup") You can create a materialized view whose partitioning granularity is larger than that of the base table by using the [date\_trunc](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_trunc.md) function on the Partitioning Key. When data changes are detected in the partitions of the base table, StarRocks refreshes the corresponding rollup partitions in the materialized view. ![Partitioned Materialized View-2](/assets/images/partitioned_mv-2-798cd8c1ea00cdb3b318d315ceed283a.png) * If the Partitioning Key of the base table is the DATE or DATETIME type, you can directly use the date\_trunc function on the Partitioning Key of the base table. ```sql PARTITION BY date_trunc(, ) ``` Example: ```sql CREATE MATERIALIZED VIEW par_mv3 REFRESH ASYNC PARTITION BY date_trunc('month', datekey) AS SELECT k1, sum(v1) AS SUM, datekey FROM par_tbl1 GROUP BY datekey, k1; ``` * If the Partitioning Key of the base table is the STRING type, you must convert the Partitioning Key of the base table into the DATE or DATETIME type in the SELECT list, set an alias for it, and use it in the date\_trunc function to specify the Partitioning Key of the materialized view. ```sql PARTITION BY date_trunc(, ) AS SELECT str2date(, ) AS ``` Example: ```sql CREATE MATERIALIZED VIEW par_mv4 REFRESH ASYNC PARTITION BY date_trunc('month', mv_datekey) AS SELECT datekey, k1, sum(v1) AS SUM, str2date(datekey, '%Y-%m-%d') AS mv_datekey FROM par_tbl2 GROUP BY datekey, k1; ``` ##### Align partitions at a customized time granularity[​](#align-partitions-at-a-customized-time-granularity "Direct link to Align partitions at a customized time granularity") The partition rollup method mentioned above only allows partitioning the materialized view based on specific time granularities and does not permit customizing the partition time range. If your business scenario requires partitioning using a customized time granularity, you can create a materialized view and define the time granularity for its partitions by using the date\_trunc function with the [time\_slice](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/time_slice.md) function, which can convert a given time into the beginning or end of a time interval based on the specified time granularity. You need to define the new time granularity (interval) by using the time\_slice function on the Partitioning Key of the base table in the SELECT list, set an alias for it, and use it in the date\_trunc function to specify the Partitioning Key of the materialized view. ```sql PARTITION BY date_trunc(, ) AS SELECT -- You can use time_slice. time_slice(, ) AS ``` Example: ```sql CREATE MATERIALIZED VIEW par_mv5 REFRESH ASYNC PARTITION BY date_trunc('day', mv_datekey) AS SELECT k1, sum(v1) AS SUM, time_slice(datekey, INTERVAL 5 MINUTE) AS mv_datekey FROM par_tbl1 GROUP BY datekey, k1; ``` ##### Align partitions with multiple base tables[​](#align-partitions-with-multiple-base-tables "Direct link to Align partitions with multiple base tables") ![Partitioned Materialized View-3](/assets/images/partitioned_mv-3-af28d5f86052c9ff0d881bfb14381ae1.png) You can create a materialized view whose partitions are aligned with those of multiple base tables, as long as the partitions of the base tables can align with each other, that is, the base tables use the same type of Partitioning Key. You can use JOIN to connect the base tables, and set the Partition Key as the common column. Alternatively, you can connect them with UNION. The base tables with aligned partitions are called reference tables. Data changes in any of the reference tables will trigger the refresh task on the corresponding partitions of the materialized view. This feature is supported from v3.3 onwards. ```sql -- Connect tables with JOIN. CREATE MATERIALIZED VIEW par_mv6 REFRESH ASYNC PARTITION BY datekey AS SELECT par_tbl1.datekey, par_tbl1.k1 AS t1k1, par_tbl3.k1 AS t2k1, sum(par_tbl1.v1) AS SUM1, sum(par_tbl3.v1) AS SUM2 FROM par_tbl1 JOIN par_tbl3 ON par_tbl1.datekey = par_tbl3.datekey_new GROUP BY par_tbl1.datekey, t1k1, t2k1; -- Connect tables with UNION. CREATE MATERIALIZED VIEW par_mv7 REFRESH ASYNC PARTITION BY datekey AS SELECT par_tbl1.datekey, par_tbl1.k1 AS t1k1, sum(par_tbl1.v1) AS SUM1 FROM par_tbl1 GROUP BY par_tbl1.datekey, par_tbl1.k1 UNION ALL SELECT par_tbl3.datekey_new, par_tbl3.k1 AS t2k1, sum(par_tbl3.v1) AS SUM2 FROM par_tbl3 GROUP BY par_tbl3.datekey_new, par_tbl3.k1; ``` ##### Align multiple partition columns[​](#align-multiple-partition-columns "Direct link to Align multiple partition columns") From v3.5.0 onwards, asynchronous materialized views support multi-column partition expressions. You can specify multiple partition columns for the materialized view, and one-to-one map them to the partition columns of the base tables. This enables Lakehouse integration and enhances the reuse of existing StarRocks internal table capabilities, providing a better Lakehouse solution: * **Simplified user experience**: Simplifies the process of transferring data from Data Lake to StarRocks. * **Accelerated query performance**: Reuses native table capabilities such as Colocate Group, Bitmap Index, Bloom Filter Index, Sort Key, and global dictionaries when creating materialized views. Queries directly referencing the base table will benefit from transparent query acceleration through automatic rewrite. ###### Notes for multi-column partition expressions[​](#notes-for-multi-column-partition-expressions "Direct link to Notes for multi-column partition expressions") * Currently, multi-column partitions in materialized views can only be mapped one-to-one or in an N:1 relationship with the base table's partitions, not an M:N relationship. For example, if the base table has partition columns `(col1, col2, ..., coln)`, the materialized view partition expression can only be a single column partition, such as `col1`, `col2`, or `coln`, or a one-to-one mapping to the base table’s partition columns, that is, `(col1, col2, ..., coln)`. This limitation is designed to simplify the partition mapping logic between the base table and the materialized view, avoiding the complexity introduced by M:N relationships. * Because Iceberg partition expressions support the `transform` function, additional handling is required when mapping Iceberg partition expressions to StarRocks materialized view partition expressions. The mapping relationship is as follows: | Iceberg Transform | Iceberg partition expression | Materialized view partition expression | | ----------------- | ---------------------------- | -------------------------------------- | | Identity | `` | `` | | hour | `hour()` | `date_trunc('hour', )` | | day | `day()` | `date_trunc('day', )` | | month | `month()` | `date_trunc('month', )` | | year | `year()` | `date_trunc('year', )` | | bucket | `bucket(, )` | Not supported | | truncate | `truncate()` | Not supported | * For non-Iceberg partition columns, where partition expression computation is not involved, additional partition expression handling is not required. You can map them directly. The following example creates a partitioned materialized view with multi-column partition expression upon the base table from Iceberg Catalog (Spark). The definition of the base table in Spark: ```sql -- The partition expression of the base table contains multiple columns and a `days` transform. CREATE TABLE lineitem_days ( l_orderkey BIGINT, l_partkey INT, l_suppkey INT, l_linenumber INT, l_quantity DECIMAL(15, 2), l_extendedprice DECIMAL(15, 2), l_discount DECIMAL(15, 2), l_tax DECIMAL(15, 2), l_returnflag VARCHAR(1), l_linestatus VARCHAR(1), l_shipdate TIMESTAMP, l_commitdate TIMESTAMP, l_receiptdate TIMESTAMP, l_shipinstruct VARCHAR(25), l_shipmode VARCHAR(10), l_comment VARCHAR(44) ) USING ICEBERG PARTITIONED BY (l_returnflag, l_linestatus, days(l_shipdate)); ``` Create a materialized view with its partition columns mapped one-to-one with those of the base table: ```sql CREATE MATERIALIZED VIEW test_days PARTITION BY (l_returnflag, l_linestatus, date_trunc('day', l_shipdate)) REFRESH DEFERRED MANUAL AS SELECT * FROM iceberg_catalog.test_db.lineitem_days; ``` ##### Achieve incremental refresh and transparent rewrite[​](#achieve-incremental-refresh-and-transparent-rewrite "Direct link to Achieve incremental refresh and transparent rewrite") You can create a partitioned materialized view that refreshes by partitions to achieve incremental updates of the materialized view and transparent rewrite of queries with partial data materialization. To achieve these goals, you must consider the following aspects when creating a materialized view: * **Refresh granularity** You can use the property `partition_refresh_number` to specify the granularity of each refresh operation. `partition_refresh_number` controls the maximum number of partitions to be refreshed in a refresh task when a refresh is triggered. If the number of partitions to be refreshed exceeds this value, StarRocks will split the refresh task and complete it in batches. The partitions are refreshed in chronological order from the least recent partition to the most recent partition (excluding partitions created dynamically for the future). When the value is `-1`, the refresh task will not be split. The default value is changed from `-1` to `1` since v3.3, meaning StarRocks refeshes partitions one by one. You can also use the `partition_refresh_strategy` property to control the adaptability of the refresh strategy. This parameter supports the following two values: * "strict" (default): Partition refresh strictly follows the partition\_refresh\_number setting to control the refresh granularity. * "adaptive": Adjusts the number of partitions refreshed per batch adaptively based on the amount of data in the base table partitions, improving refresh efficiency. * **Materialization scope** The scope of the materialized data is controlled by the properties `partition_ttl_number` (for versions earlier than v3.1.5) or `partition_ttl` (recommended for v3.1.5 and later). `partition_ttl_number` specifies the number of the most recent partitions to retain, and `partition_ttl` specifies the time range of the materialized view data to retain. During each refresh, StarRocks arranges the partitions in chronological order, and retains only those who satisfy the TTL requirements. * **Refresh strategy** * Materialized views with automatic refresh strategies (`REFRESH ASYNC`) are automatically refreshed each time the base table data changes. * Materialized views with regular refresh strategies (`REFRESH ASYNC [START ()] EVERY (INTERVAL )`) are refreshed regularly at the interval defined. note Materialized views with automatic refresh strategies and regular refresh strategies are refreshed automatically once the refresh tasks are triggered. StarRocks records and compares the data versions of each partition of the base table. A change in the data version indicates a data change in the partition. Once StarRocks detects a data change in the partition of the base table, it refreshes the corresponding partition of the materialized view. When no data changes are detected on the base table partition, the refresh for the corresponding materialized view partition is skipped. * Materialized views with manual refresh strategies (`REFRESH MANUAL`) can be refreshed only by manually executing the REFRESH MATERIALIZED VIEW statement. You can specify the time range of the partitions to be refreshed to avoid refreshing the whole materialized view. If you specify `FORCE` in the statement, StarRocks forcibly refreshes the corresponding materialized view or partitions regardless of whether the data in the base table is changed. By adding `WITH SYNC MODE` to the statement, you can make a synchronous call of the refresh task, and StarRocks returns the task result only when the task succeeds or fails. The following example creates a partitioned materialized view `par_mv8`. If StarRocks detects data changes in a partition of the base table, it refreshes the corresponding partition in the materialized view. A refresh task is split into batches, each of which only refreshes one partition (`"partition_refresh_number" = "1"`). Only two most recent partitions are retained (`"partition_ttl_number" = "2"`), the others are deleted during the refresh. ```sql CREATE MATERIALIZED VIEW par_mv8 REFRESH ASYNC PARTITION BY datekey PROPERTIES( "partition_ttl_number" = "2", "partition_refresh_number" = "1" ) AS SELECT k1, sum(v1) AS SUM, datekey FROM par_tbl1 GROUP BY datekey, k1; ``` You can use the REFRESH MATERIALIZED VIEW statement to refresh this materialized view. The following example makes a synchronous call to forcibly refresh some partitions of `par_mv8` within a certain time range. ```sql REFRESH MATERIALIZED VIEW par_mv8 PARTITION START ("2021-01-03") END ("2021-01-04") FORCE WITH SYNC MODE; ``` Output: ```plain +--------------------------------------+ | QUERY_ID | +--------------------------------------+ | 1d1c24b8-bf4b-11ee-a3cf-00163e0e23c9 | +--------------------------------------+ 1 row in set (1.12 sec) ``` With the TTL feature, only some of the partitions are retained in `par_mv8`. You have thus achieved the materialization of partial data, which is important in scenarios where most queries are against the recent data. The TTL feature allows you to transparently accelerate queries on new data (for example, within a week or month) with the materialized view while significantly saving storage costs. Queries that do not fall into this time range are routed to the base table. In the following example, Query 1 will be accelerated by the materialized view because it hits the partition that is retained in `par_mv8`, while Query 2 will be routed to the base table because it does not fall into the time range where the partitions are retained. ```sql -- Query 1 SELECT k1, sum(v1) AS SUM, datekey FROM par_tbl1 WHERE datekey='2021-01-04' GROUP BY datekey, k1; -- Query 2 SELECT k1, sum(v1) AS SUM, datekey FROM par_tbl1 WHERE datekey='2021-01-01' GROUP BY datekey, k1; ``` ![Partitioned Materialized View-4](/assets/images/partitioned_mv-4-27d290ea0587907f34f48b8d34e48a84.png) --- ### Data lake query acceleration with materialized views This topic describes how to optimize query performance in your data lake using StarRocks' asynchronous materialized views. StarRocks offers out-of-the-box data lake query capabilities, which are highly effective for exploratory queries and analysis of data in the lake. In most scenarios, [Data Cache](https://docs.starrocks.io/docs/data_source/data_cache.md) can provide block-level file caching, avoiding performance degradation caused by remote storage jitter and a large number of I/O operations. However, when it comes to building complex and efficient reports using data from the lake or further accelerating these queries, you may still encounter performance challenges. With asynchronous materialized views, you can achieve higher concurrency and better query performance for reports and data applications on the lake. #### Overview[​](#overview "Direct link to Overview") StarRocks supports building asynchronous materialized views based on external catalogs such as Hive catalog, Iceberg catalog, Hudi catalog, JDBC catalog, and Paimon catalog. External catalog-based materialized views are particularly useful in the following scenarios: * **Transparent Acceleration of data lake reports** To ensure the query performance of data lake reports, data engineers typically need to work closely with data analysts to probe into the construction logic of the acceleration layer for reports. If the acceleration layer requires further updates, they must update the construction logic, processing schedules, and query statements accordingly. Through the query rewrite capability of materialized views, report acceleration can be made transparent and imperceptible to users. When slow queries are identified, data engineers can analyze the pattern of slow queries and create materialized views on demand. Application-side queries are then intelligently rewritten and transparently accelerated by the materialized view, allowing for rapid improvement in query performance without modifying the logic of the business application or the query statement. * **Incremental calculation of real-time data associated with historical data** Suppose your business application requires the association of real-time data in StarRocks native tables and historical data in the data lake for incremental calculations. In this situation, materialized views can provide a straightforward solution. For example, if the real-time fact table is a native table in StarRocks and the dimension table is stored in the data lake, you can easily perform incremental calculations by constructing materialized views that associate the native table with the table in the external data sources. * **Rapid construction of metric layers** Calculating and processing metrics may encounter challenges when dealing with a high dimensionality of data. You can use materialized views, which allows you to perform data pre-aggregation and rollup, to create a relatively lightweight metric layer. Moreover, materialized views can be refreshed automatically, further reducing the complexity of metric calculations. Materialized views, Data Cache, and native tables in StarRocks are all effective methods to achieve significant boosts in query performance. The following table compares their major differences: |   | Data Cache | Materialized view | Native table | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | **Data loading and updates** | Queries automatically trigger data caching. | Refresh tasks are triggered automatically. | Supports various import methods but requires manual maintenance of import tasks | | **Data caching granularity** | - Supports block-level data caching
- Follows LRU cache eviction mechanism
- No computation results are cached | Stores the precomputed query results | Stores data based on table schema | | **Query performance** | Data Cache ≤ Materialized view = Native table | | | | **Query statement** | - No need to modify query statements against the data lake
- Once queries hit the cache, computation occurs. | - No need to modify query statements against the data lake
- Leverages query rewrite to reuse precomputed results | Requires modification of query statements to query the native table |
Compared to directly querying lake data or loading data into native tables, materialized views offer several unique advantages: * **Local storage acceleration**: Materialized views can leverage StarRocks' acceleration advantages with local storage, such as indexes, partitioning, bucketing, and collocate groups, resulting in better query performance compared to querying data from the data lake directly. * **Zero maintenance for loading tasks**: Materialized views update data transparently via automatic refresh tasks. There's no need to maintain loading tasks to perform scheduled data updates. Additionally, Hive, Iceberg, and Paimon catalog-based materialized views can detect data changes and perform incremental refreshes at the partition level. * **Intelligent query rewrite**: Queries can be transparently rewritten to use materialized views. You can benefit from acceleration instantly without the need to modify the query statements your application uses. Therefore, we recommend using materialized views in the following scenarios: * Even when Data Cache is enabled, query performance does not meet your requirements for query latency and concurrency. * Queries involve reusable components, such as fixed aggregation functions or join patterns. * Data is organized in partitions, while queries involve aggregation on a relatively high level (e.g., aggregating by day). In the following scenarios, we recommend prioritizing acceleration through Data Cache: * Queries do not have many reusable components and may scan any data from the data lake. * Remote storage has significant fluctuations or instability, which could potentially impact access. #### Create external catalog-based materialized views[​](#create-external-catalog-based-materialized-views "Direct link to Create external catalog-based materialized views") Creating a materialized view on tables in external catalogs is similar to creating a materialized view on StarRocks native tables. You only need to set a suitable refresh strategy in accordance with the data source you are using, and manually enable query rewrite for external catalog-based materialized views. note External table materialized views do not support automatic refresh **triggered by base table data changes**. They only support asynchronous **fixed-interval** refresh and manual refresh. ##### Choose a suitable refresh strategy[​](#choose-a-suitable-refresh-strategy "Direct link to Choose a suitable refresh strategy") Currently, StarRocks cannot detect partition-level data changes in Hudi catalogs. Therefore, a full-size refresh is performed once the task is triggered. For Hive Catalog, Iceberg Catalog (starting from v3.1.4), JDBC catalog (starting from v3.1.4, only for MySQL range-partitioned tables), and Paimon Catalog (starting from v3.2.1), StarRocks supports detecting data changes at the partition level. As a result, StarRocks can: * Refresh only the partitions with data changes to avoid full-size refresh, reducing resource consumption caused by refresh. * Ensure data consistency to some extent during query rewrite. If there are data changes in the base table in the data lake, the query will not be rewritten to use the materialized view. tip You can still choose to tolerate a certain level of data inconsistency by setting the property `mv_rewrite_staleness_second` when creating the materialized view. For more information, see [CREATE MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md). Please note that if you need to refresh by partition, the partitioning keys of the materialized view must be included in that of the base table. From v3.2.3, StarRocks supports creating partitioned materialized views upon Iceberg tables with [Partition Transforms](https://iceberg.apache.org/spec/#partition-transforms), and the materialized views are partitioned by the column after the transformation. Currently, only Iceberg tables with the `identity`, `year`, `month`, `day`, or `hour` transforms are supported. The following example shows the definition of an Iceberg table with the `day` partition transform and creates a materialized view with aligned partitions upon it: ```sql -- Iceberg table definition. CREATE TABLE spark_catalog.test.iceberg_sample_datetime_day ( id BIGINT, data STRING, category STRING, ts TIMESTAMP) USING iceberg PARTITIONED BY (days(ts)) -- Create a materialized view upon the Iceberg table. CREATE MATERIALIZED VIEW `test_iceberg_datetime_day_mv` (`id`, `data`, `category`, `ts`) PARTITION BY (`ts`) DISTRIBUTED BY HASH(`id`) REFRESH MANUAL AS SELECT `iceberg_sample_datetime_day`.`id`, `iceberg_sample_datetime_day`.`data`, `iceberg_sample_datetime_day`.`category`, `iceberg_sample_datetime_day`.`ts` FROM `iceberg`.`test`.`iceberg_sample_datetime_day`; ``` For Hive catalogs, you can enable the Hive metadata cache refresh feature to allow StarRocks to detect data changes at the partition level. When this feature is enabled, StarRocks periodically accesses the Hive Metastore Service (HMS) or AWS Glue to check the metadata information of recently queried hot data. To enable the Hive metadata cache refresh feature, you can set the following FE dynamic configuration item using [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md): ##### Configuration items[​](#configuration-items "Direct link to Configuration items") ###### enable\_background\_refresh\_connector\_metadata[​](#enable_background_refresh_connector_metadata "Direct link to enable_background_refresh_connector_metadata") **Default**: true in v3.0 false in v2.5
**Description**: Whether to enable the periodic Hive metadata cache refresh. After it is enabled, StarRocks polls the metastore (Hive Metastore or AWS Glue) of your Hive cluster, and refreshes the cached metadata of the frequently accessed Hive catalogs to perceive data changes. True indicates to enable the Hive metadata cache refresh, and false indicates to disable it.
###### background\_refresh\_metadata\_interval\_millis[​](#background_refresh_metadata_interval_millis "Direct link to background_refresh_metadata_interval_millis") **Default**: 600000 (10 minutes)
**Description**: The interval between two consecutive Hive metadata cache refreshes. Unit: millisecond.
###### background\_refresh\_metadata\_time\_secs\_since\_last\_access\_secs[​](#background_refresh_metadata_time_secs_since_last_access_secs "Direct link to background_refresh_metadata_time_secs_since_last_access_secs") **Default**: 86400 (24 hours)
**Description**: The expiration time of a Hive metadata cache refresh task. For the Hive catalog that has been accessed, if it has not been accessed for more than the specified time, StarRocks stops refreshing its cached metadata. For the Hive catalog that has not been accessed, StarRocks will not refresh its cached metadata. Unit: second. From v3.1.4, StarRocks supports detecting data changes for Iceberg Catalog at the partition level. Currently, only Iceberg V1 tables are supported. ##### Enable query rewrite for external catalog-based materialized views[​](#enable-query-rewrite-for-external-catalog-based-materialized-views "Direct link to Enable query rewrite for external catalog-based materialized views") By default, StarRocks does not support query rewrite for materialized views built on Hudi and JDBC catalogs because query rewrite in this scenario cannot ensure a strong consistency of results. You can enable this feature by setting the property `force_external_table_query_rewrite` to `true` when creating the materialized view. For materialized views built on tables in Hive catalogs, the query rewrite is enabled by default. Example: ```sql CREATE MATERIALIZED VIEW ex_mv_par_tbl PARTITION BY emp_date DISTRIBUTED BY hash(empid) PROPERTIES ( "force_external_table_query_rewrite" = "true" ) AS select empid, deptno, emp_date from `hive_catalog`.`emp_db`.`emps_par_tbl` where empid < 5; ``` In scenarios involving query rewriting, if you use a very complex query statement to build a materialized view, we recommend that you split the query statement and construct multiple simple materialized views in a nested manner. Nested materialized views are more versatile and can accommodate a broader range of query patterns. #### Best practices[​](#best-practices "Direct link to Best practices") In real-world business scenarios, you can identify queries with high execution latency and resource consumption by analyzing audit logs or big query logs. You can further use [query profiles](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md) to pinpoint the specific stages where the query is slow. The following sections provide instructions and examples on how to boost data lake query performance with materialized views. ##### Case One: Accelerate join calculation in data lake[​](#case-one-accelerate-join-calculation-in-data-lake "Direct link to Case One: Accelerate join calculation in data lake") You can use materialized views to accelerate join queries in the data lake. Suppose the following queries on your Hive catalog are particularly slow: ```sql --Q1 SELECT SUM(lo_extendedprice * lo_discount) AS REVENUE FROM hive.ssb_1g_csv.lineorder, hive.ssb_1g_csv.dates WHERE lo_orderdate = d_datekey AND d_year = 1993 AND lo_discount BETWEEN 1 AND 3 AND lo_quantity < 25; --Q2 SELECT SUM(lo_extendedprice * lo_discount) AS REVENUE FROM hive.ssb_1g_csv.lineorder, hive.ssb_1g_csv.dates WHERE lo_orderdate = d_datekey AND d_yearmonth = 'Jan1994' AND lo_discount BETWEEN 4 AND 6 AND lo_quantity BETWEEN 26 AND 35; --Q3 SELECT SUM(lo_revenue), d_year, p_brand FROM hive.ssb_1g_csv.lineorder, hive.ssb_1g_csv.dates, hive.ssb_1g_csv.part, hive.ssb_1g_csv.supplier WHERE lo_orderdate = d_datekey AND lo_partkey = p_partkey AND lo_suppkey = s_suppkey AND p_brand BETWEEN 'MFGR#2221' AND 'MFGR#2228' AND s_region = 'ASIA' GROUP BY d_year, p_brand ORDER BY d_year, p_brand; ``` By analyzing their query profiles, you may notice that the query execution time is mostly spent on the hash join between the table `lineorder` and the other dimension tables on the column `lo_orderdate`. Here, Q1 and Q2 perform aggregation after joining `lineorder` and `dates`, while Q3 performs aggregation after joining `lineorder`, `dates`, `part`, and `supplier`. Therefore, you can utilize the [View Delta Join rewrite](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md#view-delta-join-rewrite) capability of StarRocks to build a materialized view that joins `lineorder`, `dates`, `part`, and `supplier`. ```sql CREATE MATERIALIZED VIEW lineorder_flat_mv DISTRIBUTED BY HASH(LO_ORDERDATE, LO_ORDERKEY) BUCKETS 48 PARTITION BY LO_ORDERDATE REFRESH ASYNC EVERY(INTERVAL 1 DAY) PROPERTIES ( -- Specify the unique constraints. "unique_constraints" = " hive.ssb_1g_csv.supplier.s_suppkey; hive.ssb_1g_csv.part.p_partkey; hive.ssb_1g_csv.dates.d_datekey", -- Specify the Foreign Keys. "foreign_key_constraints" = " hive.ssb_1g_csv.lineorder(lo_partkey) REFERENCES hive.ssb_1g_csv.part(p_partkey); hive.ssb_1g_csv.lineorder(lo_suppkey) REFERENCES hive.ssb_1g_csv.supplier(s_suppkey); hive.ssb_1g_csv.lineorder(lo_orderdate) REFERENCES hive.ssb_1g_csv.dates(d_datekey)", -- Enable query rewrite for the external catalog-based materialized view. "force_external_table_query_rewrite" = "TRUE" ) AS SELECT l.LO_ORDERDATE AS LO_ORDERDATE, l.LO_ORDERKEY AS LO_ORDERKEY, l.LO_PARTKEY AS LO_PARTKEY, l.LO_SUPPKEY AS LO_SUPPKEY, l.LO_QUANTITY AS LO_QUANTITY, l.LO_EXTENDEDPRICE AS LO_EXTENDEDPRICE, l.LO_DISCOUNT AS LO_DISCOUNT, l.LO_REVENUE AS LO_REVENUE, s.S_REGION AS S_REGION, p.P_BRAND AS P_BRAND, d.D_YEAR AS D_YEAR, d.D_YEARMONTH AS D_YEARMONTH FROM hive.ssb_1g_csv.lineorder AS l INNER JOIN hive.ssb_1g_csv.supplier AS s ON s.S_SUPPKEY = l.LO_SUPPKEY INNER JOIN hive.ssb_1g_csv.part AS p ON p.P_PARTKEY = l.LO_PARTKEY INNER JOIN hive.ssb_1g_csv.dates AS d ON l.LO_ORDERDATE = d.D_DATEKEY; ``` ##### Case Two: Accelerate aggregations and aggregations on joins in data lake[​](#case-two-accelerate-aggregations-and-aggregations-on-joins-in-data-lake "Direct link to Case Two: Accelerate aggregations and aggregations on joins in data lake") Materialized views can be used to accelerate aggregation queries, whether they are on a single table or involve multiple tables. * Single-table aggregation query For typical queries on a single table, their query profile will reveal that the AGGREGATE node consumes a significant amount of time. You can use common aggregation operators to construct materialized views. Suppose the following is a slow query: ```sql --Q4 SELECT lo_orderdate, count(distinct lo_orderkey) FROM hive.ssb_1g_csv.lineorder GROUP BY lo_orderdate ORDER BY lo_orderdate limit 100; ``` Q4 calculates the daily count of unique orders. Because the count distinct calculation can be computationally expensive, you can create the following two types of materialized views to accelerate: ```sql CREATE MATERIALIZED VIEW mv_2_1 DISTRIBUTED BY HASH(lo_orderdate) PARTITION BY LO_ORDERDATE REFRESH ASYNC EVERY(INTERVAL 1 DAY) AS SELECT lo_orderdate, count(distinct lo_orderkey) FROM hive.ssb_1g_csv.lineorder GROUP BY lo_orderdate; CREATE MATERIALIZED VIEW mv_2_2 DISTRIBUTED BY HASH(lo_orderdate) PARTITION BY LO_ORDERDATE REFRESH ASYNC EVERY(INTERVAL 1 DAY) AS SELECT -- lo_orderkey must be the BIGINT type so that it can be used for query rewrite. lo_orderdate, bitmap_union(to_bitmap(lo_orderkey)) FROM hive.ssb_1g_csv.lineorder GROUP BY lo_orderdate; ``` Please note that, in this context, do not create materialized views with LIMIT and ORDER BY clauses to avoid rewrite failures. For more information on the query rewrite limitations, see [Query rewrite with materialized views - Limitations](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md#limitations). * Multi-table aggregation query In scenarios involving aggregations of join results, you can create nested materialized views on existing materialized views that join the tables to further aggregate the join results. For example, based on the example in Case One, you can create the following materialized view to accelerate Q1 and Q2 because their aggregation patterns are similar: ```sql CREATE MATERIALIZED VIEW mv_2_3 DISTRIBUTED BY HASH(lo_orderdate) PARTITION BY LO_ORDERDATE REFRESH ASYNC EVERY(INTERVAL 1 DAY) AS SELECT lo_orderdate, lo_discount, lo_quantity, d_year, d_yearmonth, SUM(lo_extendedprice * lo_discount) AS REVENUE FROM lineorder_flat_mv GROUP BY lo_orderdate, lo_discount, lo_quantity, d_year, d_yearmonth; ``` Of course, it is possible to perform both join and aggregation calculations within a single materialized view. While these types of materialized views may have fewer opportunities for query rewrite (due to their specific calculations), they typically occupy less storage space after aggregation. Your choice can be based on your specific use case. ```sql CREATE MATERIALIZED VIEW mv_2_4 DISTRIBUTED BY HASH(lo_orderdate) PARTITION BY LO_ORDERDATE REFRESH ASYNC EVERY(INTERVAL 1 DAY) PROPERTIES ( "force_external_table_query_rewrite" = "TRUE" ) AS SELECT lo_orderdate, lo_discount, lo_quantity, d_year, d_yearmonth, SUM(lo_extendedprice * lo_discount) AS REVENUE FROM hive.ssb_1g_csv.lineorder, hive.ssb_1g_csv.dates WHERE lo_orderdate = d_datekey GROUP BY lo_orderdate, lo_discount, lo_quantity, d_year, d_yearmonth; ``` ##### Case Three: Accelerate joins on aggregations in data lake[​](#case-three-accelerate-joins-on-aggregations-in-data-lake "Direct link to Case Three: Accelerate joins on aggregations in data lake") In some scenarios, it may be necessary to perform aggregation calculations on one table first and then perform join queries with other tables. To fully use StarRocks' query rewrite capabilities, we recommend constructing nested materialized views. For example: ```sql --Q5 SELECT * FROM ( SELECT l.lo_orderkey, l.lo_orderdate, c.c_custkey, c_region, sum(l.lo_revenue) FROM hive.ssb_1g_csv.lineorder l INNER JOIN ( SELECT distinct c_custkey, c_region from hive.ssb_1g_csv.customer WHERE c_region IN ('ASIA', 'AMERICA') ) c ON l.lo_custkey = c.c_custkey GROUP BY l.lo_orderkey, l.lo_orderdate, c.c_custkey, c_region ) c1 WHERE lo_orderdate = '19970503'; ``` Q5 first performs an aggregation query on the `customer` table and then performs a join and aggregation with the `lineorder` table. Similar queries might involve different filters on `c_region` and `lo_orderdate`. To leverage query rewrite capabilities, you can create two materialized views, one for aggregation and another for the join. ```sql --mv_3_1 CREATE MATERIALIZED VIEW mv_3_1 DISTRIBUTED BY HASH(c_custkey) REFRESH ASYNC EVERY(INTERVAL 1 DAY) PROPERTIES ( "force_external_table_query_rewrite" = "TRUE" ) AS SELECT distinct c_custkey, c_region from hive.ssb_1g_csv.customer; --mv_3_2 CREATE MATERIALIZED VIEW mv_3_2 DISTRIBUTED BY HASH(lo_orderdate) PARTITION BY LO_ORDERDATE REFRESH ASYNC EVERY(INTERVAL 1 DAY) PROPERTIES ( "force_external_table_query_rewrite" = "TRUE" ) AS SELECT l.lo_orderdate, l.lo_orderkey, mv.c_custkey, mv.c_region, sum(l.lo_revenue) FROM hive.ssb_1g_csv.lineorder l INNER JOIN mv_3_1 mv ON l.lo_custkey = mv.c_custkey GROUP BY l.lo_orderkey, l.lo_orderdate, mv.c_custkey, mv.c_region; ``` ##### Case Four: Separate hot and cold data for real-time data and historical data in data lake[​](#case-four-separate-hot-and-cold-data-for-real-time-data-and-historical-data-in-data-lake "Direct link to Case Four: Separate hot and cold data for real-time data and historical data in data lake") Consider the following scenario: data updated within the past three days is directly written into StarRocks, while less recent data is checked and batch-written into Hive. However, there are still queries that may involve data from the past seven days. In this case, you can create a simple model with materialized views to expire the data automatically. ```sql CREATE MATERIALIZED VIEW mv_4_1 DISTRIBUTED BY HASH(lo_orderdate) PARTITION BY LO_ORDERDATE REFRESH ASYNC EVERY(INTERVAL 1 DAY) AS SELECT lo_orderkey, lo_orderdate, lo_revenue FROM hive.ssb_1g_csv.lineorder WHERE lo_orderdate<=current_date() AND lo_orderdate>=date_add(current_date(), INTERVAL -4 DAY); ``` You can further build views or materialized views upon it based on the logic of the upper-layer applications. --- ### Data modeling with materialized views This topic describes how to perform data modeling with the help of StarRocks' asynchronous materialized views. By doing so, you can greatly simplify the ETL pipeline of your data warehouses, and significantly improve data quality and query performance. #### Overview[​](#overview "Direct link to Overview") Data modeling is the process of cleaning, layering, aggregating, and associating data with rational methodologies. It can create a comprehensible representation of the raw data that is too rough, too complicated, or too costly to analyze directly, and provide actionable insights into the data. However, the common challenge in real-world data modeling is that the modeling process struggles to keep up with the pace of business development, and it is difficult to measure the return on investment for data modeling efforts. Despite the fact that the modeling methodologies are simple, business experts are required to have a solid background in data organization and governance, which is a complex process. In the early stages of business, decision-makers rarely spare sufficient resources for data modeling, and it is challenging to see the value that data modeling can bring. Moreover, business models can change rapidly, and the modeling methodologies themselves need iteration and evolution. Therefore, many data analysts tend to avoid modeling and use raw data directly, thus inevitably leading to issues of data quality and query performance. When the need for modeling arises, it becomes difficult to restructure data analytics patterns that have already been established to match those of the data models. Using materialized views for data modeling can effectively address these problems. StarRocks' asynchronous materialized views can: * **Simplify the data warehouse architecture**: Because StarRocks can provide a one-stop data governance experience, you do not need to maintain other data processing systems, saving the human and system resources spent on them. * **Ease the data modeling experience**: Any data analyst with only basic SQL knowledge is capable of data modeling with StarRocks. Data modeling is no longer the exclusive province of experienced data engineers. * **Reduce maintenance complexity**: StarRocks' asynchronous materialized views can automatically manage the lineage relationships and dependencies across data layers, eliminating the need for an entire data platform to handle this task. ![Modeling-1](/assets/images/Modeling-1-3089fe70e1f2202cc1c5d36e1595c6e7.png) In real-world situations, you can perform data modeling by combining the usage of StarRocks' views (logical views) and asynchronous materialized views as follows: 1. Use views to associate real-time data with dimension data, and use materialized views to associate historical data from the data lake with dimension data. Perform necessary data cleaning and semantic mapping to obtain the detail data for the Intermediate Layer, which reflects the semantics required in your business scenarios. 2. In the Application Layer, perform data Join, Aggregation, Union, and Window calculations tailored to different business scenarios. This will yield views for real-time pipelines and materialized views for near-real-time pipelines. 3. On the Application side, choose an appropriate Analytical Data Store (ADS) for query analysis based on your timeliness and performance requirements. These ADS can serve real-time dashboards, near-real-time BI, ad-hoc queries, and scheduled reports. During this process, you will leverage several built-in capabilities of StarRocks, which will be elaborated in the following section. #### Capabilities of asynchronous materialized view[​](#capabilities-of-asynchronous-materialized-view "Direct link to Capabilities of asynchronous materialized view") StarRocks' asynchronous materialized view possesses the following atomic functions that can aid in data modeling: * **Automatic refresh**: After data is loaded into base tables, materialized views can be automatically refreshed. You do not need to maintain a scheduling task externally. * **Partitioned refresh**: Near-real-time calculations can be achieved through the partitioned refresh of the materialized views built on tables featuring time series. * **Synergy with views**: You can achieve multi-layer modeling by using materialized views and logical views, thus enabling the reuse of the Intermediate Layer and simplification of data models. * **Schema change**: You can alter the calculation results through simple SQL statements, without the need to modify the complex data pipelines. With these features, you can design comprehensive and adaptable data models to meet various business needs and scenarios. ##### Automatic refresh[​](#automatic-refresh "Direct link to Automatic refresh") When creating an asynchronous materialized view, you can specify the refresh strategy using the REFRESH clause. Currently, StarRocks supports the following refresh strategies of asynchronous materialized views: * **Automatic Refresh** (`REFRESH ASYNC`): Refresh tasks are triggered each time data in the base tables changes. Data dependencies are automatically managed by the materialized view. * **Scheduled Refresh** (`REFRESH ASYNC EVERY (INTERVAL )`): Refresh tasks are triggered at regular intervals, for example, every minute, day, or month. If there are no data changes in the base tables, the refresh task will not be triggered. * **Manual Refresh** (`REFRESH MANUAL`): Refresh tasks are triggered only by executing REFRESH MATERIALIZED VIEW manually. This refresh strategy can be used when you maintain an external scheduling framework to trigger refresh tasks. Syntax: ```sql CREATE MATERIALIZED VIEW REFRESH [ ASYNC | ASYNC [START
WHERE [and ] GROUP BY [HAVING ] ``` > **NOTE** > > In the preceding example, the HAVING clause is optional. * GROUP BY DISTINCT aggregations ```sql SELECT DISTINCT , FROM
WHERE [and ] GROUP BY HAVING ``` > **NOTE** > > In the preceding example, the HAVING clause is optional. * Non-GROUP BY aggregations ```sql SELECT FROM
WHERE [and ] ``` * Non-GROUP BY DISTINCT aggregations ```sql SELECT DISTINCT FROM
WHERE [and ] ``` * If either query includes `PartitionColumnRangePredicate`, `PartitionColumnRangePredicate` is removed before the two queries are evaluated for semantic equivalence. `PartitionColumnRangePredicate` specifies one of the following types of predicates that reference a partitioning column: * `col between v1 and v2`: The values of the partitioning column fall within the \[v1, v2] range, in which `v1` and `v2` are constant expressions. * `v1 < col and col < v2`: The values of the partitioning column fall within the (v1, v2) range, in which `v1` and `v2` are constant expressions. * `v1 < col and col <= v2`: The values of the partitioning column fall within the (v1, v2] range, in which `v1` and `v2` are constant expressions. * `v1 <= col and col < v2`: The values of the partitioning column fall within the \[v1, v2) range, in which `v1` and `v2` are constant expressions. * `v1 <= col and col <= v2`: The values of the partitioning column fall within the \[v1, v2] range, in which `v1` and `v2` are constant expressions. * If the output columns of the SELECT clauses of the two queries are the same after they are rearranged, the two queries are evaluated as semantically equivalent. * If the output columns of the GROUP BY clauses of the two queries are the same after they are rearranged, the two queries are evaluated as semantically equivalent. * If the remaining predicates of the WHERE clauses of the two queries are semantically equivalent after `PartitionColumnRangePredicate` is removed, the two queries are evaluated as semantically equivalent. * If the predicates in the HAVING clauses of the two queries are semantically equivalent, the two queries are evaluated as semantically equivalent. Use the following table `lineorder_flat` as an example: ```sql CREATE TABLE `lineorder_flat` ( `lo_orderdate` date NOT NULL COMMENT "", `lo_orderkey` int(11) NOT NULL COMMENT "", `lo_linenumber` tinyint(4) NOT NULL COMMENT "", `lo_custkey` int(11) NOT NULL COMMENT "", `lo_partkey` int(11) NOT NULL COMMENT "", `lo_suppkey` int(11) NOT NULL COMMENT "", `lo_orderpriority` varchar(100) NOT NULL COMMENT "", `lo_shippriority` tinyint(4) NOT NULL COMMENT "", `lo_quantity` tinyint(4) NOT NULL COMMENT "", `lo_extendedprice` int(11) NOT NULL COMMENT "", `lo_ordtotalprice` int(11) NOT NULL COMMENT "", `lo_discount` tinyint(4) NOT NULL COMMENT "", `lo_revenue` int(11) NOT NULL COMMENT "", `lo_supplycost` int(11) NOT NULL COMMENT "", `lo_tax` tinyint(4) NOT NULL COMMENT "", `lo_commitdate` date NOT NULL COMMENT "", `lo_shipmode` varchar(100) NOT NULL COMMENT "", `c_name` varchar(100) NOT NULL COMMENT "", `c_address` varchar(100) NOT NULL COMMENT "", `c_city` varchar(100) NOT NULL COMMENT "", `c_nation` varchar(100) NOT NULL COMMENT "", `c_region` varchar(100) NOT NULL COMMENT "", `c_phone` varchar(100) NOT NULL COMMENT "", `c_mktsegment` varchar(100) NOT NULL COMMENT "", `s_name` varchar(100) NOT NULL COMMENT "", `s_address` varchar(100) NOT NULL COMMENT "", `s_city` varchar(100) NOT NULL COMMENT "", `s_nation` varchar(100) NOT NULL COMMENT "", `s_region` varchar(100) NOT NULL COMMENT "", `s_phone` varchar(100) NOT NULL COMMENT "", `p_name` varchar(100) NOT NULL COMMENT "", `p_mfgr` varchar(100) NOT NULL COMMENT "", `p_category` varchar(100) NOT NULL COMMENT "", `p_brand` varchar(100) NOT NULL COMMENT "", `p_color` varchar(100) NOT NULL COMMENT "", `p_type` varchar(100) NOT NULL COMMENT "", `p_size` tinyint(4) NOT NULL COMMENT "", `p_container` varchar(100) NOT NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`lo_orderdate`, `lo_orderkey`) COMMENT "olap" PARTITION BY RANGE(`lo_orderdate`) (PARTITION p1 VALUES [('0000-01-01'), ('1993-01-01')), PARTITION p2 VALUES [('1993-01-01'), ('1994-01-01')), PARTITION p3 VALUES [('1994-01-01'), ('1995-01-01')), PARTITION p4 VALUES [('1995-01-01'), ('1996-01-01')), PARTITION p5 VALUES [('1996-01-01'), ('1997-01-01')), PARTITION p6 VALUES [('1997-01-01'), ('1998-01-01')), PARTITION p7 VALUES [('1998-01-01'), ('1999-01-01'))) DISTRIBUTED BY HASH(`lo_orderkey`) PROPERTIES ( "replication_num" = "3", "colocate_with" = "groupxx1", "storage_format" = "DEFAULT", "enable_persistent_index" = "true", "compression" = "LZ4" ); ``` The following two queries, Q1 and Q2, on the table `lineorder_flat` are semantically equivalent after they are processed as follows: 1. Rearrange the output columns of the SELECT statement. 2. Rearrange the output columns of the GROUP BY clause. 3. Remove the output columns of the ORDER BY clause. 4. Rearrange the predicates in the WHERE clause. 5. Add `PartitionColumnRangePredicate`. * Q1 ```sql SELECT sum(lo_revenue), year(lo_orderdate) AS year,p_brand FROM lineorder_flat WHERE p_category = 'MFGR#12' AND s_region = 'AMERICA' GROUP BY year,p_brand ORDER BY year,p_brand; ``` * Q2 ```sql SELECT year(lo_orderdate) AS year, p_brand, sum(lo_revenue) FROM lineorder_flat WHERE s_region = 'AMERICA' AND p_category = 'MFGR#12' AND lo_orderdate >= '1993-01-01' AND lo_orderdate <= '1993-12-31' GROUP BY p_brand, year(lo_orderdate) ``` Semantic equivalence is evaluated based on the physical plans of queries. Therefore, literal differences in queries do not impact the evaluation for semantic equivalence. Additionally, constant expressions are removed from queries, and `cast` expressions are removed during query optimizations. Therefore, these expressions do not impact the evaluation for semantic equivalence. Thirdly, the aliases of columns and relations do not impact the evaluation for semantic equivalence either. ##### Queries with overlapping scanned partitions[​](#queries-with-overlapping-scanned-partitions "Direct link to Queries with overlapping scanned partitions") Query Cache supports predicate-based query splitting. Splitting queries based on predicate semantics help implement reuse of partial computation results. When a query contains a predicate that references the partitioning column of a table and the predicate specifies a value range, StarRocks can split the range into multiple intervals based on table partitioning. The computation results from each individual interval can be separately reused by other queries. Use the following table `t0` as an example: ```sql CREATE TABLE if not exists t0 ( ts DATETIME NOT NULL, k0 VARCHAR(10) NOT NULL, k1 BIGINT NOT NULL, v1 DECIMAL64(7, 2) NOT NULL ) ENGINE=OLAP DUPLICATE KEY(`ts`, `k0`, `k1`) COMMENT "OLAP" PARTITION BY RANGE(ts) ( START ("2022-01-01 00:00:00") END ("2022-02-01 00:00:00") EVERY (INTERVAL 1 day) ) DISTRIBUTED BY HASH(`ts`, `k0`, `k1`) PROPERTIES ( "replication_num" = "3", "storage_format" = "default" ); ``` The table `t0` is partitioned by day, and the column `ts` is the table's partitioning column. Among the following four queries, Q2, Q3, and Q4 can reuse portions of the computation results cached for Q1: * Q1 ```sql SELECT date_trunc('day', ts) as day, sum(v0) FROM t0 WHERE ts BETWEEN '2022-01-02 12:30:00' AND '2022-01-14 23:59:59' GROUP BY day; ``` The value range specified by the predicate `ts between '2022-01-02 12:30:00' and '2022-01-14 23:59:59'` of Q1 can be split into the following intervals: ```sql 1. [2022-01-02 12:30:00, 2022-01-03 00:00:00), 2. [2022-01-03 00:00:00, 2022-01-04 00:00:00), 3. [2022-01-04 00:00:00, 2022-01-05 00:00:00), ... 12. [2022-01-13 00:00:00, 2022-01-14 00:00:00), 13. [2022-01-14 00:00:00, 2022-01-15 00:00:00), ``` * Q2 ```sql SELECT date_trunc('day', ts) as day, sum(v0) FROM t0 WHERE ts >= '2022-01-02 12:30:00' AND ts < '2022-01-05 00:00:00' GROUP BY day; ``` Q2 can reuse the computation results within the following intervals of Q1: ```sql 1. [2022-01-02 12:30:00, 2022-01-03 00:00:00), 2. [2022-01-03 00:00:00, 2022-01-04 00:00:00), 3. [2022-01-04 00:00:00, 2022-01-05 00:00:00), ``` * Q3 ```sql SELECT date_trunc('day', ts) as day, sum(v0) FROM t0 WHERE ts >= '2022-01-01 12:30:00' AND ts <= '2022-01-10 12:00:00' GROUP BY day; ``` Q3 can reuse the computation results within the following intervals of Q1: ```sql 2. [2022-01-03 00:00:00, 2022-01-04 00:00:00), 3. [2022-01-04 00:00:00, 2022-01-05 00:00:00), ... 8. [2022-01-09 00:00:00, 2022-01-10 00:00:00), ``` * Q4 ```sql SELECT date_trunc('day', ts) as day, sum(v0) FROM t0 WHERE ts BETWEEN '2022-01-02 12:30:00' and '2022-01-02 23:59:59' GROUP BY day; ``` Q4 can reuse the computation results within the following intervals of Q1: ```sql 1. [2022-01-02 12:30:00, 2022-01-03 00:00:00), ``` The support for reuse of partial computation results varies depending on the partitioning policy used, as described in the following table. | **Partitioning policy** | **Support for reuse of partial computation results** | | ------------------------- | ----------------------------------------------------------------------------- | | Unpartitioned | Not supported | | Multi-Column Partitioned | Not supported
**NOTE**
This feature may be supported in the future. | | Single-Column Partitioned | Supported | ##### Queries against data with append-only data changes[​](#queries-against-data-with-append-only-data-changes "Direct link to Queries against data with append-only data changes") Query Cache supports multi-version caching. As data loads are made, new versions of tablets are generated. Consequently, the cached computation results that are generated from the previous versions of the tablets become stale and lag behind the latest tablet versions. In this situation, the multi-version caching mechanism tries to merge the stale results saved in the query cache and the incremental versions of the tablets stored on disk into the final results of the tablets so that new queries can carry the latest tablet versions. Multi-version caching is constrained by table types, query types, and data update types. The support for multi-version caching varies depending on table types and query types, as described in the following table. | **Table type** | **Query** **type** | **Support for multi-version caching** | | ------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Duplicate Key table | - Queries on base tables
- Queries on synchronous materialized views | - Queries on base tables: supported in all situations except when incremental tablet versions contain data deletion records.
- Queries on synchronous materialized views: supported in all situations except when the GROUP BY, HAVING, or WHERE clauses of queries reference aggregation columns. | | Aggregate table | Queries on base tables or queries on synchronous materialized views | Supported in all situations except the following:The schemas of base tables contain the aggregate function `replace`.The GROUP BY, HAVING, or WHERE clauses of queries reference aggregation columns.Incremental tablet versions contain data deletion records. | | Unique Key table | N/A | Not supported. However, the query cache is supported. | | Primary Key table | N/A | Not supported. However, the query cache is supported. | The impact of data update types on multi-version caching is as follows: * Data deletions Multi-version caching cannot work if incremental versions of tablets contain delete operations. * Data insertions * If an empty version is generated for a tablet, the existing data of the tablet in the query cache remains valid and can still be retrieved. * If a non-empty version is generated for a tablet, the existing data of the tablet in the query cache remains valid, but its version lags behind the latest version of the tablet. In this situation, StarRocks reads the incremental data generated from the version of the existing data to the latest version of the tablet, merges the existing data with the incremental data, and populates the merged data into the query cache. * Schema changes and tablet truncations If the schema of a table is changed or specific tablets of the table are truncated, new tablets are generated for the table. As a result, the existing data of the tablets of the table in the query cache becomes invalid. #### Metrics[​](#metrics "Direct link to Metrics") The profiles of queries for which the query cache works contain `CacheOperator` statistics. In the source plan of a query, if the pipeline contains `OlapScanOperator`, the names of `OlapScanOperator` and aggregate operators are prefixed with `ML_` to denote that the pipeline uses `MultilaneOperator` to perform per-tablet computations. `CacheOperator` is inserted preceding `ML_CONJUGATE_AGGREGATE` to process the logic that controls how the query cache runs in Passthrough, Populate, and Probe modes. The profile of the query contains the following `CacheOperator` metrics that help you understand the query cache usage. | **Metric** | **Description** | | ------------------------- | ------------------------------------------------------------- | | CachePassthroughBytes | The number of bytes generated in Passthrough mode. | | CachePassthroughChunkNum | The number of chunks generated in Passthrough mode. | | CachePassthroughRowNum | The number of rows generated in Passthrough mode. | | CachePassthroughTabletNum | The number of tablets generated in Passthrough mode. | | CachePassthroughTime: | The amount of computation time taken in Passthrough mode. | | CachePopulateBytes | The number of bytes generated in Populate mode. | | CachePopulateChunkNum | The number of chunks generated in Populate mode. | | CachePopulateRowNum | The number of rows generated in Populate mode. | | CachePopulateTabletNum | The number of tablets generated in Populate mode. | | CachePopulateTime | The amount of computation time taken in Populate mode. | | CacheProbeBytes | The number of bytes generated for cache hits in Probe mode. | | CacheProbeChunkNum | The number of chunks generated for cache hits in Probe mode. | | CacheProbeRowNum | The number of rows generated for cache hits in Probe mode. | | CacheProbeTabletNum | The number of tablets generated for cache hits in Probe mode. | | CacheProbeTime | The amount of computation time taken in Probe mode. | `CachePopulate`*`XXX`* metrics provide statistics about query cache misses for which the query cache is updated. `CachePassthrough`*`XXX`* metrics provide statistics about query cache misses for which the query cache is not updated because the size of per-tablet computation results generated is large. `CacheProbe`*`XXX`* metrics provide statistics about query cache hits. In the multi-version caching mechanism, `CachePopulate` metrics and `CacheProbe` metrics may contain the same tablet statistics, and `CachePassthrough` metrics and `CacheProbe` metrics may also contain the same tablet statistics. For example, when StarRocks computes the data of each tablet, it hits the computation results generated on the historical version of the tablet. In this situation, StarRocks reads the incremental data generated from the historical version to the latest version of the tablet, computes the data, and merges the incremental data with the cached data. If the size of the computation results generated after the merging does not exceed the threshold specified by the `query_cache_entry_max_bytes` or `query_cache_entry_max_rows` parameter, the statistics of the tablet are collected into `CachePopulate` metrics. Otherwise, the statistics of the tablet are collected into `CachePassthrough` metrics. #### RESTful API operations[​](#restful-api-operations "Direct link to RESTful API operations") * `metrics |grep query_cache` This API operation is used to query the metrics related to the query cache. ```shell curl -s http://:/metrics |grep query_cache # TYPE starrocks_be_query_cache_capacity gauge starrocks_be_query_cache_capacity 536870912 # TYPE starrocks_be_query_cache_hit_count gauge starrocks_be_query_cache_hit_count 5084393 # TYPE starrocks_be_query_cache_hit_ratio gauge starrocks_be_query_cache_hit_ratio 0.984098 # TYPE starrocks_be_query_cache_lookup_count gauge starrocks_be_query_cache_lookup_count 5166553 # TYPE starrocks_be_query_cache_usage gauge starrocks_be_query_cache_usage 0 # TYPE starrocks_be_query_cache_usage_ratio gauge starrocks_be_query_cache_usage_ratio 0.000000 ``` * `api/query_cache/stat` This API operation is used to query the usage of the query cache. ```shell curl http://:/api/query_cache/stat { "capacity": 536870912, "usage": 0, "usage_ratio": 0.0, "lookup_count": 5025124, "hit_count": 4943720, "hit_ratio": 0.983800598751394 } ``` * `api/query_cache/invalidate_all` This API operation is used to clear the query cache. ```shell curl -XPUT http://:/api/query_cache/invalidate_all { "status": "OK" } ``` The parameters in the preceding API operations are as follows: * `be_host`: the IP address of the node on which the BE resides. * `be_http_port`: the HTTP port number of the node on which the BE resides. #### Precautions[​](#precautions "Direct link to Precautions") * StarRocks needs to populate the query cache with the computation results of queries that are initiated for the first time. As a result, the query performance may be slightly lower than expected, and the query latency is increased. * If you configure a large query cache size, the amount of memory that can be provisioned to query evaluation on the BE is decreased. We recommend that the query cache size do not exceed 1/6 of the memory capacity provisioned to query evaluation. * If the number of tablets that need to be processed is smaller than the value of `pipeline_dop`, the query cache does not work. To enable the query cache to work, you can set `pipeline_dop` to a smaller value such as `1`. From v3.0 onwards, StarRocks adaptively adjusts this parameter based on query parallelism. #### Examples[​](#examples "Direct link to Examples") ##### Dataset[​](#dataset "Direct link to Dataset") 1. Log in to your StarRocks cluster, go to the destination database, and run the following command to create a table named `t0`: ```sql CREATE TABLE t0 ( `ts` datetime NOT NULL COMMENT "", `k0` varchar(10) NOT NULL COMMENT "", `k1` char(6) NOT NULL COMMENT "", `v0` bigint(20) NOT NULL COMMENT "", `v1` decimal64(7, 2) NOT NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`ts`, `k0`, `k1`) COMMENT "OLAP" PARTITION BY RANGE(`ts`) ( START ("2022-01-01 00:00:00") END ("2022-02-01 00:00:00") EVERY (INTERVAL 1 DAY) ) DISTRIBUTED BY HASH(`ts`, `k0`, `k1`) PROPERTIES ( "replication_num" = "3", "storage_format" = "DEFAULT", "enable_persistent_index" = "true" ); ``` 2. Insert the following data records into `t0`: ```sql INSERT INTO t0 VALUES ('2022-01-11 20:42:26', 'n4AGcEqYp', 'hhbawx', '799393174109549', '8029.42'), ('2022-01-27 18:17:59', 'i66lt', 'mtrtzf', '100400167', '10000.88'), ('2022-01-28 20:10:44', 'z6', 'oqkeun', '-58681382337', '59881.87'), ('2022-01-29 14:54:31', 'qQ', 'dzytua', '-19682006834', '43807.02'), ('2022-01-31 08:08:11', 'qQ', 'dzytua', '7970665929984223925', '-8947.74'), ('2022-01-15 00:40:58', '65', 'hhbawx', '4054945', '156.56'), ('2022-01-24 16:17:51', 'onqR3JsK1', 'udtmfp', '-12962', '-72127.53'), ('2022-01-01 22:36:24', 'n4AGcEqYp', 'fabnct', '-50999821', '17349.85'), ('2022-01-21 08:41:50', 'Nlpz1j3h', 'dzytua', '-60162', '287.06'), ('2022-01-30 23:44:55', '', 'slfght', '62891747919627339', '8014.98'), ('2022-01-18 19:14:28', 'z6', 'dzytua', '-1113001726', '73258.24'), ('2022-01-30 14:54:59', 'z6', 'udtmfp', '111175577438857975', '-15280.41'), ('2022-01-08 22:08:26', 'z6', 'ympyls', '3', '2.07'), ('2022-01-03 08:17:29', 'Nlpz1j3h', 'udtmfp', '-234492', '217.58'), ('2022-01-27 07:28:47', 'Pc', 'cawanm', '-1015', '-20631.50'), ('2022-01-17 14:07:47', 'Nlpz1j3h', 'lbsvqu', '2295574006197343179', '93768.75'), ('2022-01-31 14:00:12', 'onqR3JsK1', 'umlkpo', '-227', '-66199.05'), ('2022-01-05 20:31:26', '65', 'lbsvqu', '684307', '36412.49'), ('2022-01-06 00:51:34', 'z6', 'dzytua', '11700309310', '-26064.10'), ('2022-01-26 02:59:00', 'n4AGcEqYp', 'slfght', '-15320250288446', '-58003.69'), ('2022-01-05 03:26:26', 'z6', 'cawanm', '19841055192960542', '-5634.36'), ('2022-01-17 08:51:23', 'Pc', 'ghftus', '35476438804110', '13625.99'), ('2022-01-30 18:56:03', 'n4AGcEqYp', 'lbsvqu', '3303892099598', '8.37'), ('2022-01-22 14:17:18', 'i66lt', 'umlkpo', '-27653110', '-82306.25'), ('2022-01-02 10:25:01', 'qQ', 'ghftus', '-188567166', '71442.87'), ('2022-01-30 04:58:14', 'Pc', 'ympyls', '-9983', '-82071.59'), ('2022-01-05 00:16:56', '7Bh', 'hhbawx', '43712', '84762.97'), ('2022-01-25 03:25:53', '65', 'mtrtzf', '4604107', '-2434.69'), ('2022-01-27 21:09:10', '65', 'udtmfp', '476134823953365199', '38736.04'), ('2022-01-11 13:35:44', '65', 'qmwhvr', '1', '0.28'), ('2022-01-03 19:13:07', '', 'lbsvqu', '11', '-53084.04'), ('2022-01-20 02:27:25', 'i66lt', 'umlkpo', '3218824416', '-71393.20'), ('2022-01-04 04:52:36', '7Bh', 'ghftus', '-112543071', '-78377.93'), ('2022-01-27 18:27:06', 'Pc', 'umlkpo', '477', '-98060.13'), ('2022-01-04 19:40:36', '', 'udtmfp', '433677211', '-99829.94'), ('2022-01-20 23:19:58', 'Nlpz1j3h', 'udtmfp', '361394977', '-19284.18'), ('2022-01-05 02:17:56', 'Pc', 'oqkeun', '-552390906075744662', '-25267.92'), ('2022-01-02 16:14:07', '65', 'dzytua', '132', '2393.77'), ('2022-01-28 23:17:14', 'z6', 'umlkpo', '61', '-52028.57'), ('2022-01-12 08:05:44', 'qQ', 'hhbawx', '-9579605666539132', '-87801.81'), ('2022-01-31 19:48:22', 'z6', 'lbsvqu', '9883530877822', '34006.42'), ('2022-01-11 20:38:41', '', 'piszhr', '56108215256366', '-74059.80'), ('2022-01-01 04:15:17', '65', 'cawanm', '-440061829443010909', '88960.51'), ('2022-01-05 07:26:09', 'qQ', 'umlkpo', '-24889917494681901', '-23372.12'), ('2022-01-29 18:13:55', 'Nlpz1j3h', 'cawanm', '-233', '-24294.42'), ('2022-01-10 00:49:45', 'Nlpz1j3h', 'ympyls', '-2396341', '77723.88'), ('2022-01-29 08:02:58', 'n4AGcEqYp', 'slfght', '45212', '93099.78'), ('2022-01-28 08:59:21', 'onqR3JsK1', 'oqkeun', '76', '-78641.65'), ('2022-01-26 14:29:39', '7Bh', 'umlkpo', '176003552517', '-99999.96'), ('2022-01-03 18:53:37', '7Bh', 'piszhr', '3906151622605106', '55723.01'), ('2022-01-04 07:08:19', 'i66lt', 'ympyls', '-240097380835621', '-81800.87'), ('2022-01-28 14:54:17', 'Nlpz1j3h', 'slfght', '-69018069110121', '90533.64'), ('2022-01-22 07:48:53', 'Pc', 'ympyls', '22396835447981344', '-12583.39'), ('2022-01-22 07:39:29', 'Pc', 'uqkghp', '10551305', '52163.82'), ('2022-01-08 22:39:47', 'Nlpz1j3h', 'cawanm', '67905472699', '87831.30'), ('2022-01-05 14:53:34', '7Bh', 'dzytua', '-779598598706906834', '-38780.41'), ('2022-01-30 17:34:41', 'onqR3JsK1', 'oqkeun', '346687625005524', '-62475.31'), ('2022-01-29 12:14:06', '', 'qmwhvr', '3315', '22076.88'), ('2022-01-05 06:47:04', 'Nlpz1j3h', 'udtmfp', '-469', '42747.17'), ('2022-01-19 15:20:20', '7Bh', 'lbsvqu', '347317095885', '-76393.49'), ('2022-01-08 16:18:22', 'z6', 'fghmcd', '2', '90315.60'), ('2022-01-02 00:23:06', 'Pc', 'piszhr', '-3651517384168400', '58220.34'), ('2022-01-12 08:23:31', 'onqR3JsK1', 'udtmfp', '5636394870355729225', '33224.25'), ('2022-01-28 10:46:44', 'onqR3JsK1', 'oqkeun', '-28102078612755', '6469.53'), ('2022-01-23 23:16:11', 'onqR3JsK1', 'ghftus', '-707475035515433949', '63422.66'), ('2022-01-03 05:32:31', 'z6', 'hhbawx', '-45', '-49680.52'), ('2022-01-27 03:24:33', 'qQ', 'qmwhvr', '375943906057539870', '-66092.96'), ('2022-01-25 20:07:22', '7Bh', 'slfght', '1', '72440.21'), ('2022-01-04 16:07:24', 'qQ', 'uqkghp', '751213107482249', '16417.31'), ('2022-01-23 19:22:00', 'Pc', 'hhbawx', '-740731249600493', '88439.40'), ('2022-01-05 09:04:20', '7Bh', 'cawanm', '23602', '302.44'); ``` ##### Query examples[​](#query-examples "Direct link to Query examples") The statistics of query cache-related metrics in this section are examples and are for reference only. ###### Query cache works for local aggregations at stage 1[​](#query-cache-works-for-local-aggregations-at-stage-1 "Direct link to Query cache works for local aggregations at stage 1") This includes three situations: * The query accesses only a single tablet. * The query accesses multiple tablets from multiple partitions of a table that itself comprises a colocated group, and data does not need to be shuffled for aggregations. * The query accesses multiple tablets from the same partition of a table, and data does not need to be shuffled for aggregations. Query example: ```sql SELECT date_trunc('hour', ts) AS hour, k0, sum(v1) AS __c_0 FROM t0 WHERE ts between '2022-01-03 00:00:00' and '2022-01-03 23:59:59' GROUP BY date_trunc('hour', ts), k0 ``` The following figure shows the query cache-related metrics in the query profile. ![Query Cache - Stage 1 - Metrics](/assets/images/query_cache_stage1_agg_with_cache_en-9eb1b64beb2c0681cbd7a6438972e727.png) ###### Query cache does not work for remote aggregations at stage 1[​](#query-cache-does-not-work-for-remote-aggregations-at-stage-1 "Direct link to Query cache does not work for remote aggregations at stage 1") When aggregations on multiple tablets are forcibly performed at stage 1, data is first shuffled and then aggregated. Query example: ```sql SET new_planner_agg_stage = 1; SELECT date_trunc('hour', ts) AS hour, v0 % 2 AS is_odd, sum(v1) AS __c_0 FROM t0 WHERE ts between '2022-01-03 00:00:00' and '2022-01-03 23:59:59' GROUP BY date_trunc('hour', ts), is_odd ``` ###### Query cache works for local aggregations at stage 2[​](#query-cache-works-for-local-aggregations-at-stage-2 "Direct link to Query cache works for local aggregations at stage 2") This includes three situations: * The aggregations at stage 2 of the query are compiled to compare the same type of data. The first aggregation is a local aggregation. After the first aggregation is complete, the results generated from the first aggregation are computed to perform a second aggregation, which is a global aggregation. * The query is a SELECT DISTINCT query. * The query includes one of the following DISTINCT aggregate functions: `sum(distinct)`, `count(distinct)`, and `avg(distinct)`. In most cases, aggregations are performed at stage 3 or 4 for such a query. However, you can run `set new_planner_agg_stage = 1` to forcibly perform aggregations at stage 2 for the query. If the query contains `avg(distinct)` and you want to perform aggregations at stage, you also need to run `set cbo_cte_reuse = false` to disable CTE optimizations. Query example: ```sql SELECT date_trunc('hour', ts) AS hour, v0 % 2 AS is_odd, sum(v1) AS __c_0 FROM t0 WHERE ts between '2022-01-03 00:00:00' and '2022-01-03 23:59:59' GROUP BY date_trunc('hour', ts), is_odd ``` The following figure shows the query cache-related metrics in the query profile. ![Query Cache - Stage 2 - Metrics](/assets/images/query_cache_stage2_agg_with_cache_en-cea3ccce1963fcdc4eeb3a86af61ed2e.png) ###### Query cache works for local aggregations at stage 3[​](#query-cache-works-for-local-aggregations-at-stage-3 "Direct link to Query cache works for local aggregations at stage 3") The query is a GROUP BY aggregate query that includes a single DISTINCT aggregate function. The supported DISTINCT aggregate functions are `sum(distinct)`, `count(distinct)`, and `avg(distinct)`. > **NOTICE** > > If the query includes `avg(distinct)`, you also need to run `set cbo_cte_reuse = false` to disable CTE optimizations. Query example: ```sql SELECT date_trunc('hour', ts) AS hour, v0 % 2 AS is_odd, sum(distinct v1) AS __c_0 FROM t0 WHERE ts between '2022-01-03 00:00:00' and '2022-01-03 23:59:59' GROUP BY date_trunc('hour', ts), is_odd; ``` The following figure shows the query cache-related metrics in the query profile. ![Query Cache - Stage 3 - Metrics](/assets/images/query_cache_stage3_agg_with_cache_en-85ffd774bb5a811f21585e207f0654d7.png) ###### Query cache works for local aggregations at stage 4[​](#query-cache-works-for-local-aggregations-at-stage-4 "Direct link to Query cache works for local aggregations at stage 4") The query is a non-GROUP BY aggregate query that includes a single DISTINCT aggregate function. Such queries include classical queries that remove deduplicate data. Query example: ```sql SELECT count(distinct v1) AS __c_0 FROM t0 WHERE ts between '2022-01-03 00:00:00' and '2022-01-03 23:59:59' ``` The following figure shows the query cache-related metrics in the query profile. ![Query Cache - Stage 4 - Metrics](/assets/images/query_cache_stage4_agg_with_cache_en-b930e2ae7360201a037c79a27eac5f22.png) ###### Cached results are reused for two queries whose first aggregations are semantically equivalent[​](#cached-results-are-reused-for-two-queries-whose-first-aggregations-are-semantically-equivalent "Direct link to Cached results are reused for two queries whose first aggregations are semantically equivalent") Use the following two queries, Q1 and Q2, as an example. Q1 and Q2 both include multiple aggregations, but their first aggregations are semantically equivalent. Therefore, Q1 and Q2 are evaluated as semantically equivalent, and can reuse each other's computation results saved in the query cache. * Q1 ```sql SELECT ( ifnull(sum(murmur_hash3_32(hour)), 0) + ifnull(sum(murmur_hash3_32(k0)), 0) + ifnull(sum(murmur_hash3_32(__c_0)), 0) ) AS fingerprint FROM ( SELECT date_trunc('hour', ts) AS hour, k0, sum(v1) AS __c_0 FROM t0 WHERE ts between '2022-01-03 00:00:00' and '2022-01-03 23:59:59' GROUP BY date_trunc('hour', ts), k0 ) AS t; ``` * Q2 ```sql SELECT date_trunc('hour', ts) AS hour, k0, sum(v1) AS __c_0 FROM t0 WHERE ts between '2022-01-03 00:00:00' and '2022-01-03 23:59:59' GROUP BY date_trunc('hour', ts), k0 ``` The following figure shows the `CachePopulate` metrics for Q1. ![Query Cache - Q1 - Metrics](/assets/images/query_cache_reuse_Q1_en-e7d3ec4680a192e1956c1211cd11c30e.png) The following figure shows the `CacheProbe` metrics for Q2. ![Query Cache - Q2 - Metrics](/assets/images/query_cache_reuse_Q2_en-181dfb454bbddcc93f88c8aede92558d.png) ###### Query cache does not work for DISTINCT queries for which CTE optimizations are enabled[​](#query-cache-does-not-work-for-distinct-queries-for-which-cte-optimizations-are-enabled "Direct link to Query cache does not work for DISTINCT queries for which CTE optimizations are enabled") After you run `set cbo_cte_reuse = true` to enable CTE optimizations, the computation results for specific queries that include DISTINCT aggregate functions cannot be cached. A few examples are as follows: * The query contains a single DISTINCT aggregate function `avg(distinct)`: ```sql SELECT avg(distinct v1) AS __c_0 FROM t0 WHERE ts between '2022-01-03 00:00:00' and '2022-01-03 23:59:59'; ``` ![Query Cache - CTE - 1](/assets/images/query_cache_distinct_with_cte_Q1_en-78baec68173add1602ef9b87daeb18ec.png) * The query contains multiple DISTINCT aggregate functions that reference the same column: ```sql SELECT avg(distinct v1) AS __c_0, sum(distinct v1) AS __c_1, count(distinct v1) AS __c_2 FROM t0 WHERE ts between '2022-01-03 00:00:00' and '2022-01-03 23:59:59'; ``` ![Query Cache - CTE - 2](/assets/images/query_cache_distinct_with_cte_Q2_en-eb3a31a195cdd8ac67d74ed6dac0a98d.png) * The query contains multiple DISTINCT aggregate functions that each reference a different column: ```sql SELECT sum(distinct v1) AS __c_1, count(distinct v0) AS __c_2 FROM t0 WHERE ts between '2022-01-03 00:00:00' and '2022-01-03 23:59:59'; ``` ![Query Cache - CTE - 3](/assets/images/query_cache_distinct_with_cte_Q3_en-a88efd0f3f2e469cfa5806fd46afc966.png) #### Best practices[​](#best-practices "Direct link to Best practices") When you create your table, specify a reasonable partition description and a reasonable distribution method, including: * Choose a single DATE-type column as the partition column. If the table contains more than one DATE-type column, choose the column whose values roll forward as data is incrementally ingested and that is used to define your interesting time ranges of queries. * Choose a proper partition width. The data ingested most recently may modify the latest partitions of the table. Therefore, the cache entries involving the latest partitions are unstable and are apt to be invalidated. * Specify a bucket number that is several dozen in the distribution description of the table creation statement. If the bucket number is exceedingly small, the query cache cannot take effect when the number of tablets that need to be processed by the BE is less than the value of `pipeline_dop`. --- ### Colocate Join For shuffle join and broadcast join, if the join condition is met, the data rows of the two joining tables are merged into a single node to complete the join. Neither of these two join methods can avoid latency or overhead caused by data network transmission between nodes. The core idea is to keep bucketing key, number of copies, and copy placement consistent for tables in the same Colocation Group. If the join column is a bucketing key, the computing node only needs to do local join without getting data from other nodes. Colocate Join supports equi joins. This document introduces the principle, implementation, usage, and considerations of Colocate Join. #### Terminology[​](#terminology "Direct link to Terminology") * **Colocation Group (CG)**: A CG will contain one or more Tables. The Tables within a CG have the same bucketing and replica placement, and are described using the Colocation Group Schema. * **Colocation Group Schema (CGS)**: A CGS contains the bucketing key, number of buckets, and number of replicas of a CG. #### Principle[​](#principle "Direct link to Principle") Colocate Join is to form a CG with a set of Tables having the same CGS, and ensure that the corresponding bucket copies of these Tables will fall on the same set of BE nodes. When the tables in the CG perform Join operations on the bucketed columns, the local data can be joined directly, saving time from transferring data between nodes. Bucket Seq is obtained by `hash(key) mod buckets`. Suppose a Table has 8 buckets, then there are \[0, 1, 2, 3, 4, 5, 6, 7] 8 buckets, and each Bucket has one or more sub-tables, the number of sub-tables depends on the number of partitions. If it is a multi-partitioned table, there will be multiple tablets. In order to have the same data distribution, tables within the same CG must comply with the following. 1. Tables within the same CG must have the identical bucketing key (type, number, order) and the same number of buckets so that the data slices of multiple tables can be distributed and controlled one by one. The bucketing key is the columns specified in the table creation statement `DISTRIBUTED BY HASH(col1, col2, ...)`. The bucketing key determines which columns of data are Hashed into different Bucket Seqs. The name of the bucketing key can vary for tables within the same CG.The bucketing columns can be different in the creation statement, but the order of the corresponding data types in `DISTRIBUTED BY HASH(col1, col2, ...)` should be exactly the same . 2. Tables within the same CG may have different numbers of partitions and different partition keys. When creating a table, the CG is specified by the attribute `"colocate_with" = "group_name"` in the table PROPERTIES. If the CG does not exist, it means the table is the first table of the CG and called Parent Table. The data distribution of the Parent Table (type, number and order of split bucket keys, number of copies and number of split buckets) determines the CGS. If the CG exists, check whether the data distribution of the table is consistent with the CGS. The copy placement of tables within the same CG satisfies: 1. The mapping between the Bucket Seq and BE nodes of all the Tables is the same as that of the Parent Table. 2. The mapping between the Bucket Seq and BE nodes of all the Partitions in the Parent Table is the same as that of the first Partition. 3. The mapping between the Bucket Seq and BE nodes of the first Partition of the Parent Table is determined using the native Round Robin algorithm. The consistent data distribution and mapping guarantee that the data rows with the same value taken by bucketing key fall on the same BE. Therefore, when using the bucketing key to join columns, only local joins are required. #### Usage[​](#usage "Direct link to Usage") ##### Table creation[​](#table-creation "Direct link to Table creation") When creating a table, you can specify the attribute `"colocate_with" = "group_name"` in PROPERTIES to indicate that the table is a Colocate Join table and belongs to a specified Colocation Group. > **NOTE** > > From version 2.5.4, Colocate Join can be performed on tables from different databases. You only need to specify the same `colocate_with` property when you create tables. For example: ```sql CREATE TABLE tbl (k1 int, v1 int sum) DISTRIBUTED BY HASH(k1) BUCKETS 8 PROPERTIES( "colocate_with" = "group1" ); ``` If the specified Group does not exist, StarRocks automatically creates a Group that only contains the current table. If the Group exists, StarRocks checks to see if the current table meets the Colocation Group Schema. If so, it creates the table and adds it to the Group. At the same time, the table creates a partition and a tablet based on the data distribution rules of the existing Group. A Colocation Group belongs to a database. The name of a Colocation Group is unique within a database. In the internal storage, the full name of the Colocation Group is `dbId_groupName`, but you only perceive `groupName`. > **NOTE** > > If you specify the same Colocation Group to associate tables from different databases for Colocate Join, the Colocation Group exists in each of these databases. You can run `show proc "/colocation_group"` to check the Colocation Groups in different databases. ##### Delete[​](#delete "Direct link to Delete") A complete deletion is a deletion from the Recycle Bin. Normally, after a table is deleted with the `DROP TABLE` command, by default it will stay in the recycle bin for a day before being deleted. When the last table in a Group is completely deleted, the Group will also be deleted automatically. ##### View group information[​](#view-group-information "Direct link to View group information") The following command allows you to view the Group information that already exists in the cluster. ```plain SHOW PROC '/colocation_group'; +-------------+--------------+--------------+------------+----------------+----------+----------+ | GroupId | GroupName | TableIds | BucketsNum | ReplicationNum | DistCols | IsStable | +-------------+--------------+--------------+------------+----------------+----------+----------+ | 10005.10008 | 10005_group1 | 10007, 10040 | 10 | 3 | int(11) | true | +-------------+--------------+--------------+------------+----------------+----------+----------+ ``` * **GroupId**: The cluster-wide unique identifier of a Group, with the first half being the db id and the second half being the group id. * **GroupName**: The full name of the Group. * **TabletIds**: The list of ids of the Tables in the Group. * **BucketsNum**: The number of buckets. * **ReplicationNum**: The number of replicas. * **DistCols**: Distribution columns, i.e. bucketing column type. * **IsStable**: Whether the Group is stable (for the definition of stability, see the section of Colocation Replica Balancing and Repair). You can further view the data distribution of a Group with the following command. ```plain SHOW PROC '/colocation_group/10005.10008'; +-------------+---------------------+ | BucketIndex | BackendIds | +-------------+---------------------+ | 0 | 10004, 10002, 10001 | | 1 | 10003, 10002, 10004 | | 2 | 10002, 10004, 10001 | | 3 | 10003, 10002, 10004 | | 4 | 10002, 10004, 10003 | | 5 | 10003, 10002, 10001 | | 6 | 10003, 10004, 10001 | | 7 | 10003, 10004, 10002 | +-------------+---------------------+ ``` * **BucketIndex**: Subscript of the sequence of buckets. * **BackendIds**: The ids of BE nodes where the bucketing data slices t are located. > Note: The above command requires the NODE privilege or the `cluster_admin` role. Regular users cannot access it. ##### Modifying Table Group Properties[​](#modifying-table-group-properties "Direct link to Modifying Table Group Properties") You can modify the Colocation Group properties of a table. For example: ```sql ALTER TABLE tbl SET ("colocate_with" = "group2"); ``` If the table has not been assigned to a Group before, the command will check the Schema and add the table to the Group (the Group will be created first if it does not exist). If the table has been previously assigned to another Group, the command will remove the table from the original Group and add it to the new Group (the Group will be created first if it does not exist). You can also remove the Colocation properties of a table with the following command. ```sql ALTER TABLE tbl SET ("colocate_with" = ""); ``` ##### Other related operations[​](#other-related-operations "Direct link to Other related operations") When adding a partition using `ADD PARTITION` or modifying the number of copies to a table with the Colocation attribute, StarRocks checks if the operation will violate the Colocation Group Schema and rejects it if it does. #### Colocation replica balancing and repair[​](#colocation-replica-balancing-and-repair "Direct link to Colocation replica balancing and repair") The replicas distribution of a Colocation table needs to follow the distribution rules specified in the Group schema, so it differs from normal sharding in terms of replica repair and balancing. The Group itself has a `stable` property. When `stable` is `true`, it means that no changes are being made to table slices in the Group and the Colocation feature is working properly. When `stable` is `false`, it means that some table slices in the current Group are being repaired or migrated, and the Colocate Join of the affected tables will degrade to a normal Join. ##### Replica repair[​](#replica-repair "Direct link to Replica repair") Replicas can only be stored on the specified BE node. StarRocks will look for the least loaded BE to replace an unavailable BE (e.g.,down, decommission),. After the replacement, all the bucketing data slices on the old BE are repaired. During migration, the Group is marked as **Unstable**. ##### Replica balancing[​](#replica-balancing "Direct link to Replica balancing") StarRocks tries to distribute the Colocation table slices evenly across all BE nodes. Balancing for normal tables is at the replica level, that is, each replica individually finds a lower-load BE node. Balancing for Colocation tables is at the Bucket level, that is, all replicas within a Bucket are migrated together. We use a simple balancing algorithm that distributes `BucketsSequnce` evenly across all BE nodes without considering the actual size of the replicas but only the number of replicas. The exact algorithm can be found in the code comments in `ColocateTableBalancer.java`. > Note 1: The current Colocation replica balancing and repair algorithm may not work well for StarRocks clusters with heterogeneous deployment. The so-called heterogeneous deployment means that the disk capacity, number of disk, and disk type (SSD and HDD) of BE nodes are not consistent. In the case of heterogeneous deployment, it may happen that a small-capacity BE node stores the same number of replicas as a large-capacity BE node. > > Note 2: When a Group is in the Unstable state, the Join of its tables will degrade to a normal Join, which may significantly degrade the query performance of the cluster. If you do not want the system to be automatically balanced, set the FE configuration `disable_colocate_balance` to disable automatic balancing and enable it back at the appropriate time. (See the section Advanced Operations (#Advanced Operations) for details) #### Query[​](#query "Direct link to Query") The Colocation table is queried in the same way as a normal table. If the Group where the Colocation table is located is in Unstable state, it will automatically degrade to a normal Join, as illustrated by the following example. Table 1: ```sql CREATE TABLE `tbl1` ( `k1` date NOT NULL COMMENT "", `k2` int(11) NOT NULL COMMENT "", `v1` int(11) SUM NOT NULL COMMENT "" ) ENGINE=OLAP AGGREGATE KEY(`k1`, `k2`) PARTITION BY RANGE(`k1`) ( PARTITION p1 VALUES LESS THAN ('2019-05-31'), PARTITION p2 VALUES LESS THAN ('2019-06-30') ) DISTRIBUTED BY HASH(`k2`) BUCKETS 6 PROPERTIES ( "colocate_with" = "group1" ); INSERT INTO tbl1 VALUES ("2015-09-12",1000,1), ("2015-09-13",2000,2); ``` Table 2: ```sql CREATE TABLE `tbl2` ( `k1` datetime NOT NULL COMMENT "", `k2` int(11) NOT NULL COMMENT "", `v1` double SUM NOT NULL COMMENT "" ) ENGINE=OLAP AGGREGATE KEY(`k1`, `k2`) DISTRIBUTED BY HASH(`k2`) BUCKETS 6 PROPERTIES ( "colocate_with" = "group1" ); INSERT INTO tbl2 VALUES ("2015-09-12 00:00:00",3000,3), ("2015-09-12 00:00:00",4000,4); ``` View query plan: ```plain EXPLAIN SELECT * FROM tbl1 INNER JOIN tbl2 ON (tbl1.k2 = tbl2.k2); +-------------------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS:1: k1 | 2: k2 | 3: v1 | 4: k1 | 5: k2 | 6: v1 | | PARTITION: UNPARTITIONED | | | | RESULT SINK | | | | 3:EXCHANGE | | | | PLAN FRAGMENT 1 | | OUTPUT EXPRS: | | PARTITION: RANDOM | | | | STREAM DATA SINK | | EXCHANGE ID: 03 | | UNPARTITIONED | | | | 2:HASH JOIN | | | join op: INNER JOIN (COLOCATE) | | | colocate: true | | | equal join conjunct: 5: k2 = 2: k2 | | | | | |----1:OlapScanNode | | | TABLE: tbl1 | | | PREAGGREGATION: OFF. Reason: Has can not pre-aggregation Join | | | partitions=1/2 | | | rollup: tbl1 | | | tabletRatio=6/6 | | | tabletList=15344,15346,15348,15350,15352,15354 | | | cardinality=1 | | | avgRowSize=3.0 | | | | | 0:OlapScanNode | | TABLE: tbl2 | | PREAGGREGATION: OFF. Reason: None aggregate function | | partitions=1/1 | | rollup: tbl2 | | tabletRatio=6/6 | | tabletList=15373,15375,15377,15379,15381,15383 | | cardinality=1 | | avgRowSize=3.0 | +-------------------------------------------------------------------------+ 40 rows in set (0.03 sec) ``` If a Colocate Join takes effect, the Hash Join node displays `colocate: true`. If it doesn’t take effect, the query plan is as follows: ```plain +----------------------------------------------------+ | Explain String | +----------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS:`tbl1`.`k1` | | | PARTITION: RANDOM | | | | RESULT SINK | | | | 2:HASH JOIN | | | join op: INNER JOIN (BROADCAST) | | | hash predicates: | | | colocate: false, reason: group is not stable | | | `tbl1`.`k2` = `tbl2`.`k2` | | | tuple ids: 0 1 | | | | | |----3:EXCHANGE | | | tuple ids: 1 | | | | | 0:OlapScanNode | | TABLE: tbl1 | | PREAGGREGATION: OFF. Reason: No AggregateInfo | | partitions=0/2 | | rollup: null | | buckets=0/0 | | cardinality=-1 | | avgRowSize=0.0 | | numNodes=0 | | tuple ids: 0 | | | | PLAN FRAGMENT 1 | | OUTPUT EXPRS: | | PARTITION: RANDOM | | | | STREAM DATA SINK | | EXCHANGE ID: 03 | | UNPARTITIONED | | | | 1:OlapScanNode | | TABLE: tbl2 | | PREAGGREGATION: OFF. Reason: null | | partitions=0/1 | | rollup: null | | buckets=0/0 | | cardinality=-1 | | avgRowSize=0.0 | | numNodes=0 | | tuple ids: 1 | +----------------------------------------------------+ ``` The HASH JOIN node will show the corresponding reason: `colocate: false, reason: group is not stable`. An EXCHANGE node will be generated at the same time. #### Advanced operations[​](#advanced-operations "Direct link to Advanced operations") ##### FE configuration items[​](#fe-configuration-items "Direct link to FE configuration items") * **disable\_colocate\_relocate** Whether to disable automatic Colocation replica repair for StarRocks. The default is false, which means it is turned on. This parameter only affects replica repair for Colocation tables, not for normal tables. * **disable\_colocate\_balance** Whether to disable automatic Colocation replica balancing for StarRocks. The default is false, which means it is turned on. This parameter only affects the replica balancing of Colocation tables, not for normal tables. * **disable\_colocate\_join** You can disable Colocate join at session granularity by changing this variable. * **disable\_colocate\_join** The Colocate join function can be disabled by changing this variable. ##### HTTP Restful API[​](#http-restful-api "Direct link to HTTP Restful API") StarRocks provides several HTTP Restful APIs related to Colocate Join for viewing and modifying Colocation Groups. This API is implemented on the FE and can be accessed using `fe_host:fe_http_port` with `db_admin` and `user_admin` permissions. 1. View all Colocation information of a cluster ```bash curl --location-trusted -u: 'http://:/api/colocate' ``` ```json // Returns the internal Colocation information in Json format. { "colocate_meta": { "groupName2Id": { "g1": { "dbId": 10005, "grpId": 10008 } }, "group2Tables": {}, "table2Group": { "10007": { "dbId": 10005, "grpId": 10008 }, "10040": { "dbId": 10005, "grpId": 10008 } }, "group2Schema": { "10005.10008": { "groupId": { "dbId": 10005, "grpId": 10008 }, "distributionColTypes": [{ "type": "INT", "len": -1, "isAssignedStrLenInColDefinition": false, "precision": 0, "scale": 0 }], "bucketsNum": 10, "replicationNum": 2 } }, "group2BackendsPerBucketSeq": { "10005.10008": [ [10004, 10002], [10003, 10002], [10002, 10004], [10003, 10002], [10002, 10004], [10003, 10002], [10003, 10004], [10003, 10004], [10003, 10004], [10002, 10004] ] }, "unstableGroups": [] }, "status": "OK" } ``` 2. Mark the Group as Stable or Unstable ```bash # Mark as Stable curl -XPOST --location-trusted -u: ​'http://:/api/colocate/group_stable?db_id=&group_id=​' # Mark as Unstable curl -XPOST --location-trusted -u: ​'http://:/api/colocate/group_unstable?db_id=&group_id=​' ``` If the returned result is `200`, the Group is successfully marked as Stable or Unstable. 3. Set the data distribution of a Group This interface allows you to force the number distribution of a Group. `POST /api/colocate/bucketseq?db_id=10005&group_id= 10008` `Body:` `[[10004,10002],[10003,10002],[10002,10004],[10003,10002],[10002,10004],[10003,10002],[10003,10004],[10003,10004],[10003,10004],[10002,10004]]` `returns: 200` Where `Body` is `BucketsSequence` represented as a nested array and the ids of the BEs in which bucketing slices are located. > Note that to use this command, you may need to set the FE configuration `disable_colocate_relocate` and `disable_colocate_balance` to true, that is, to disable the system to perform automatic Colocation replica repair and balancing. Otherwise, it may be automatically reset by the system after modification. --- ### Gather statistics for CBO This topic describes the basic concept of StarRocks cost-based optimizer (CBO) and how to collect statistics for the CBO to select an optimal query plan. StarRocks 2.4 introduces histograms to gather accurate data distribution statistics. Since v3.2.0, StarRocks supports collecting statistics from Hive, Iceberg, and Hudi tables, reducing the dependency on other metastore systems. The syntax is similar to collecting StarRocks internal tables. #### What is CBO[​](#what-is-cbo "Direct link to What is CBO") CBO is critical to query optimization. After an SQL query arrives at StarRocks, it is parsed into a logical execution plan. The CBO rewrites and transforms the logical plan into multiple physical execution plans. The CBO then estimates the execution cost of each operator in the plan (such as CPU, memory, network, and I/O) and chooses a query path with the lowest cost as the final physical plan. StarRocks CBO was launched in StarRocks 1.16.0 and is enabled by default from 1.19 onwards. Developed upon the Cascades framework, StarRocks CBO estimates cost based on various statistical information. It is capable of choosing an execution plan with the lowest cost among tens of thousands of execution plans, significantly improving the efficiency and performance of complex queries. Statistics are important for the CBO. They determine whether cost estimation is accurate and useful. The following sections detail the types of statistical information, the collection policy, and how to collect statistics and view statistical information. #### Types of statistical information[​](#types-of-statistical-information "Direct link to Types of statistical information") StarRocks collects a variety of statistics as the input for cost estimation. ##### Basic statistics[​](#basic-statistics "Direct link to Basic statistics") By default, StarRocks periodically collects the following basic statistics of tables and columns: * row\_count: total number of rows in a table * data\_size: data size of a column * ndv: column cardinality, which is the number of distinct values in a column * null\_count: the amount of data with NULL values in a column * min: minimum value in a column * max: maximum value in a column Full statistics are stored in the `column_statistics` of the `_statistics_` database. You can view this table to query table statistics. Following is an example of querying statistics data from this table. ```sql SELECT * FROM _statistics_.column_statistics\G *************************** 1. row *************************** table_id: 10174 partition_id: 10170 column_name: is_minor db_id: 10169 table_name: zj_test.source_wiki_edit partition_name: p06 row_count: 2 data_size: 2 ndv: NULL null_count: 0 max: 1 min: 0 ``` ##### Histogram[​](#histogram "Direct link to Histogram") StarRocks 2.4 introduces histograms to complement basic statistics. Histogram is considered an effective way of data representation. For tables with skewed data, histograms can accurately reflect data distribution. StarRocks uses equi-height histograms, which are constructed on several buckets. Each bucket contains an equal amount of data. For data values that are frequently queried and have a major impact on selectivity, StarRocks allocates separate buckets for them. More buckets mean more accurate estimation, but may also cause a slight increase in memory usage. You can adjust the number of buckets and most common values (MCVs) for a histogram collection task. **Histograms are applicable to columns with highly skewed data and frequent queries.If your table data is uniformly distributed, you do not need to create histograms. Histograms can be created only on columns of numeric, DATE, DATETIME, or string types. For string types, StarRocks collects only the most common values (MCVs) and does not collect histogram buckets.** Histograms are stored in the `histogram_statistics` table of the `_statistics_` database. Following is an example of querying statistics data from this table: ```sql SELECT * FROM _statistics_.histogram_statistics\G *************************** 1. row *************************** table_id: 10174 column_name: added db_id: 10169 table_name: zj_test.source_wiki_edit buckets: NULL mcv: [["23","1"],["5","1"]] update_time: 2023-12-01 15:17:10.274000 ``` ##### Multi-column joint statistics[​](#multi-column-joint-statistics "Direct link to Multi-column joint statistics") Since v3.5.0, StarRocks supports multi-column joint statistics collection. Currently, when StarRocks performs cardinality estimation, the optimizer in most scenarios assumes that multiple columns are completely independent of each other, that is, there is no correlation among them. However, if there is a correlation among columns, the current estimation method may lead to an incorrect result. This causes the optimizer to generate an incorrect execution plan. Currently, only multi-column joint NDV is supported, and it is mainly used in the following scenarios for cardinality estimation. * Evaluating multiple AND-joined equivalent predicates. * Evaluating Agg nodes. * Apply to the aggregate push-down strategy. Currently, multi-column joint statistics collection only supports manual collection. The default type is sampled collection. Multi-column statistics are stored in the `multi_column_statistics` table of the `_statistics_` database in each StarRocks cluster. A query will return information similar to the following: ```sql mysql> select * from _statistics_.multi_column_statistics \G *************************** 1. row *************************** table_id: 1695021 column_ids: 0#1 db_id: 110099 table_name: db.test_multi_col_stats column_names: id,name ndv: 11 update_time: 2025-04-11 15:09:50 ``` #### Collection types and methods[​](#collection-types-and-methods "Direct link to Collection types and methods") Data size and data distribution constantly change in a table. Statistics must be updated regularly to represent that data change. Before creating a statistics collection task, you must choose a collection type and method that best suit your business requirements. StarRocks supports full and sampled collection, both can be performed automatically and manually. By default, StarRocks automatically collects full statistics of a table. It checks for any data updates every 5 minutes. If data change is detected, data collection will be automatically triggered. If you do not want to use automatic full collection, you can customize a collection task. | **Collection type** | **Collection method** | **Description** | **Advantage and disadvantage** | | ------------------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Full collection | Automatic/manual | Scans the full table to gather statistics.Statistics are collected partition. If a partition has no data change, data will not be collected from this partition, reducing resource consumption.Full statistics are stored in the `_statistics_.column_statistics` table. | Advantage: The statistics are accurate, which helps the CBO make accurate estimation.Disadvantage: It consumes system resources and is slow. From 2.5 onwards, StarRocks allows you to specify an automatic collection period, which reduces resource consumption. | | Sampled collection | Automatic/manual | Evenly extracts `N` rows of data from each partition of a table.Statistics are collected by table. Basic statistics of each column are stored as one record. Cardinality information (ndv) of a column is estimated based on the sampled data, which is not accurate. Sampled statistics are stored in the `_statistics_.table_statistic_v1` table. | Advantage: It consumes less system resources and is fast.Disadvantage: The statistics are not complete, which may affect the accuracy of cost estimation. | Since v3.5.0, both sampled and full collection statistics are stored in the `_statistics_.column_statistics` table. This is because the current optimizer will focus on the most recently collected statistics in the cardinality estimation, while each time the background automatically collects statistics, it may use a different collection method due to the different health status of the table. If there is data skew, the error rate of the sampled statistical information of the full table may be higher, and the same query may use different statistical information due to the different collection methods, which may cause the optimizer to generate an incorrect execution plan. Therefore, both sampled collection and full collection assume that the statistical information is collected at the partition level. You can adjust to the previous collection and storage method by modifying the FE configuration item `statistic_use_meta_statistics` to `false`. #### Predicate Column[​](#predicate-column "Direct link to Predicate Column") Since v3.5.0, StarRocks supports collecting Predicate Column statistics. A Predicate Column is a column that is often used as a filter condition (WHERE clause, JOIN condition, GROUP BY column, DISTINCT column) in a query. StarRocks automatically records each Predicate Column for the table involved in a query and stores it in `_statistics_. predicate_columns` table. The query will return the following: ```sql select * from _statistics_.predicate_columns \G *************************** 1. row *************************** fe_id: 127.0.0.1_9011_1735874871718 db_id: 1684773 table_id: 1685786 column_id: 1 usage: normal,predicate,join,group_by last_used: 2025-04-11 20:39:32 created: 2025-04-11 20:39:53 ``` You can also query the `information_schema.column_stats_usage` view for more observable information. In extreme scenarios (with a large number of columns), there is a large overhead for full collection, whereas in reality, for a stable workload, you often do not need statistics for all columns, but only the columns involved in some key Filter, Join, and Aggregation. Therefore, in order to weigh the cost and accuracy, StarRocks supports collecting Predicate Column statistics manually, and collecting Predicate Column statistics according to the policy when collecting statistics automatically, avoiding collecting all column statistics in the table. Each FE node in the cluster synchronously updates the Predicate Column information to the FE cache at regular intervals to accelerate the usage. #### Collect statistics[​](#collect-statistics "Direct link to Collect statistics") StarRocks offers flexible statistics collection methods. You can choose automatic, manual, or custom collection, whichever suits your business scenarios. By default, StarRocks automatically collects the full amount of statistics for a table on a periodic basis. The default interval is 10 minutes. If the system finds that the update ratio of the data meets the condition, the collection will be triggered automatically. **Collecting full statistics may consume a lot of system resources**. If you do not want to use automatic full statistics collection, you can set the FE configuration item `enable_collect_full_statistic` to `false`, and the periodic collection task will be changed from full statistics collection to sampled collection. Since v3.5.0, StarRocks will automatically change the full collection to sampled collection if it finds that the data of a table has changed a lot since the last collection to the current table during automatic collection. In the case of sampled collection is used, if there is a Predicate Column (PREDICATE/JOIN/GROUP BY/DISTINCT) in the table, StarRocks will convert the sampling task to full collection and collect the statistics of the Predicate Column to ensure the accuracy of the statistics, and will not collect the statistics of all columns in the table. It will not collect statistics from all columns in the table. You can configure this with the FE configuration item `statistic_auto_collect_use_full_predicate_column_for_sample`. In addition, StarRocks also switches from full collection of all columns to full collection of Predicate Columns if the number of columns in the table exceeds the FE configuration item `statistic_auto_collect_predicate_columns_threshold` when full collection is used. ##### Automatic collection[​](#automatic-collection "Direct link to Automatic collection") For basic statistics, StarRocks automatically collects full statistics of a table by default, without requiring manual operations. For tables on which no statistics have been collected, StarRocks automatically collects statistics within the scheduling period. For tables on which statistics have been collected, StarRocks updates the total number of rows and modified rows in the tables, and persists this information regularly for judging whether to trigger automatic collection. Histograms and multi-column joint statistics are not captured by the current automatic collection task. From 2.4.5 onwards, StarRocks allows you to specify a collection period for automatic full collection, which prevents cluster performance jitter caused by automatic full collection. This period is specified by FE parameters `statistic_auto_analyze_start_time` and `statistic_auto_analyze_end_time`. Conditions that will trigger automatic collection: * Table data has changed since previous statistics collection. * The collection time falls within the range of the configured collection period. (The default collection period is all day.) * The update time of the previous collecting job is earlier than the latest update time of partitions. * The health of table statistics is below the specified threshold (`statistic_auto_collect_ratio`). > Formula for calculating statistics health: > > 1. If the number of partitions with data updated is less than 10, the formula is `1 - (Number of updated rows since previous collection/Total number of rows)`. > 2. If the number of partitions with data updated is greater than or equal to 10, the formula is `1 - MIN(Number of updated rows since previous collection/Total number of rows, Number of updated partitions since previous collection/Total number of partitions)`. > 3. Since v3.5.0, in order to determine whether a partition is healthy or not, StarRocks no longer compares the statistic information collection time with the data update time, but rather the ratio of partition update rows. You can configure this via the FE configuration item `statistic_partition_health_v2_threshold`. Also, you can set the FE configuration item `statistic_partition_healthy_v2` to `false` to use to the previous health check behavior. In addition, StarRocks allows you to configure collection policies based on table size and table update frequency: * For tables with small data volume, \*\*statistics are collected in real time with no restrictions, even though the table data is frequently updated. The `statistic_auto_collect_small_table_size` parameter can be used to determine whether a table is a small table or a large table. You can also use `statistic_auto_collect_small_table_interval` to configure collection intervals for small tables. * For tables with large data volume, the following restrictions apply: * The default collection interval is not less than 12 hours, which can be configured using `statistic_auto_collect_large_table_interval`. * When the collection interval is met and the statistics health is lower than the threshold for automatic sampled collection, sampled collection is triggered. You can configure this behavior using the FE configurartion item `statistic_auto_collect_sample_threshold`. Since v3.5.0, sampled collection is converted to full collection of Predicate Columns if all of the following conditions are met: * Predicate Column exists in the table, and the number of Predicate Columns is less than `statistic_auto_collect_max_predicate_column_size_on_sample_strategy`. * The FE configuration `statistic_auto_collect_use_full_predicate_column_for_sample` is set to `true`. * When the collection interval is met and the statistics health is higher than the threshold for automatic sampled collection (`statistic_auto_collect_sample_threshold`) and lower than the automatic collection threshold (`statistic_auto_collect_ratio`), full collection is triggered. * When the size of partitions to collect data (`statistic_max_full_collect_data_size`) is greater than 100 GB, sampled collection is triggered. * Only statistics of partitions whose update time is later than the time of the previous collection task are collected. Statistics of partitions with no data change are not collected. * For tables that have a Predicate Column and the total number of columns exceeds the `statistic_auto_collect_predicate_columns_threshold`, statistics are only collected for the Predicate Column in the table. tip After the data of a table is changed, manually triggering a sampled collection task for this table will make the update time of the sampled collection task later than the data update time, which will not trigger automatic full collection for this table in this scheduling period. Automatic full collection is enabled by default and run by the system using the default settings. The following table describes the default settings. If you need to modify them, run the **ADMIN SET CONFIG** command. | **FE** **configuration item** | **Type** | **Default value** | **Description** | | ---------------------------------------------------------------------------- | -------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | enable\_statistic\_collect | BOOLEAN | TRUE | Whether to turn on the default and user-defined auto collection tasks. This switch is turned on by default. | | enable\_collect\_full\_statistic | BOOLEAN | TRUE | Whether to enable the default auto collection. This switch is turned on by default. | | statistic\_collect\_interval\_sec | LONG | 600 | The interval for checking data updates during automatic collection. Unit: seconds. | | statistic\_auto\_analyze\_start\_time | STRING | 00:00:00 | The start time of automatic collection. Value range: `00:00:00` - `23:59:59`. | | statistic\_auto\_analyze\_end\_time | STRING | 23:59:59 | The end time of automatic collection. Value range: `00:00:00` - `23:59:59`. | | statistic\_auto\_collect\_small\_table\_size | LONG | 5368709120 | The threshold for determining whether a table is a small table for automatic full collection. A table whose size is greater than this value is considered a large table, whereas a table whose size is less than or equal to this value is considered a small table. Unit: Byte. Default value: 5368709120 (5 GB). | | statistic\_auto\_collect\_small\_table\_interval | LONG | 0 | The interval for automatically collecting full statistics of small tables. Unit: seconds. | | statistic\_auto\_collect\_large\_table\_interval | LONG | 43200 | The interval for automatically collecting full statistics of large tables. Unit: seconds. Default value: 43200 (12 hours). | | statistic\_auto\_collect\_ratio | FLOAT | 0.8 | The threshold for determining whether the statistics for automatic collection are healthy. If statistics health is below this threshold, automatic collection is triggered. | | statistic\_auto\_collect\_sample\_threshold | DOUBLE | 0.3 | The statistics health threshold for triggering automatic sampled collection. If the health value of statistics is lower than this threshold, automatic sampled collection is triggered. | | statistic\_max\_full\_collect\_data\_size | LONG | 107374182400 | The data size of the partitions for automatic collection to collect data. Unit: Byte. Default value: 107374182400 (100 GB). If the data size exceeds this value, full collection is discarded and sampled collection is performed instead. | | statistic\_full\_collect\_buffer | LONG | 20971520 | The maximum buffer size taken by automatic collection tasks. Unit: Byte. Default value: 20971520 (20 MB). | | statistic\_collect\_max\_row\_count\_per\_query | INT | 5000000000 | The maximum number of rows to query for a single analyze task. An analyze task will be split into multiple queries if this value is exceeded. | | statistic\_collect\_too\_many\_version\_sleep | LONG | 600000 | The sleep time of automatic collection tasks if the table on which the collection task runs has too many data versions. Unit: ms. Default value: 600000 (10 minutes). | | statistic\_auto\_collect\_use\_full\_predicate\_column\_for\_sample | BOOLEAN | TRUE | Whether to convert an automatic full collection task to full collection of a Predicate Column when it hits the sampled collection policy. | | statistic\_auto\_collect\_max\_predicate\_column\_size\_on\_sample\_strategy | INT | 16 | When the Auto Full Collection task hits the sampled collection policy, if the table has an unusually large number of Predicate Columns and exceeds this configuration item, the task will not switch to full collection of the Predicate Columns, but maintains to be the sampled collection of all columns. This configuration item controls the maximum value of Predicate Column for this behavior. | | statistic\_auto\_collect\_predicate\_columns\_threshold | INT | 32 | If the number of columns in the table exceeds this configuration during automatic collection, only the column statistics for the Predicate Column will be collected. | | statistic\_predicate\_columns\_persist\_interval\_sec | LONG | 60 | The interval at which FE synchronize and persists statistics of Predicate Column. | | statistic\_predicate\_columns\_ttl\_hours | LONG | 24 | The TTL in hours for Predicate Column entries in FE; vacuum removes older data. A negative value disables vacuum. | | enable\_predicate\_columns\_collection | BOOLEAN | TRUE | Whether to enable predicate columns collection. If disabled, predicate columns will not be recorded during query optimization. | | enable\_manual\_collect\_array\_ndv | BOOLEAN | FALSE | Whether to enable manual collection for the NDV information of the ARRAY type. | | enable\_auto\_collect\_array\_ndv | BOOLEAN | FALSE | Whether to enable automatic collection for the NDV information of the ARRAY type. | You can rely on automatic jobs for a majority of statistics collection, but if you have specific requirements, you can manually create a task by executing the ANALYZE TABLE statement or customize an automatic task by executing the CREATE ANALYZE statement. ##### Manual collection[​](#manual-collection "Direct link to Manual collection") You can use ANALYZE TABLE to create a manual collection task. By default, manual collection is a synchronous operation. You can also set it to an asynchronous operation. In asynchronous mode, after you run ANALYZE TABLE, the system immediately returns whether this statement is successful. However, the collection task will be running in the background and you do not have to wait for the result. You can check the status of the task by running SHOW ANALYZE STATUS. Asynchronous collection is suitable for tables with large data volume, whereas synchronous collection is suitable for tables with small data volume. **Manual collection tasks are run only once after creation. You do not need to delete manual collection tasks.** You must have the INSERT and SELECT privileges on the coreesponding table to perform the ANALYZE TABLE operation. ###### Manually collect basic statistics[​](#manually-collect-basic-statistics "Direct link to Manually collect basic statistics") ```sql ANALYZE [FULL|SAMPLE] TABLE tbl_name [( col_name [, col_name]... ) | col_name [, col_name]... | ALL COLUMNS | PREDICATE COLUMNS | MULTIPLE COLUMNS ( col_name [, col_name]... )] [PARTITION (partition_name [, partition_name]...)] [WITH [SYNC | ASYNC] MODE] [PROPERTIES (property [, property]...)] ``` Parameter description: * Collection type * FULL: indicates full collection. * SAMPLE: indicates sampled collection. * If no collection type is specified, full collection is used by default. * The type of the column from which to collect statistics: * `col_name`: Columns from which to collect statistics. Separate multiple columns with commas (`,`). If this parameter is not specified, the entire table is collected. * `ALL COLUMNS`: Collect statistics from all columns. Supported since v3.5.0. * `PREDICATE COLUMNS`: Collect statistics from only Predicate Columns. Supported for native tables since v3.5.0 and for analyzable external tables, including Hive, Iceberg, Hudi, ODPS, Delta Lake, and Paimon, since v4.1.4. * `MULTIPLE COLUMNS`: Collects joint statistics from the specified multiple columns. Currently, only manual synchronous collection of multiple columns is supported. The number of columns for manual statistics collection cannot exceed `statistics_max_multi_column_combined_num`, the default value is `10`. Supported since v3.5.0. * `WITH SYNC | ASYNC MODE`: whether to run the manual collection task in synchronous or asynchronous mode. Synchronous collection is used by default if you do not specify this parameter. * `PROPERTIES`: custom parameters. If `PROPERTIES` is not specified, the default settings in the `fe.conf` file are used. The properties that are actually used can be viewed via the `Properties` column in the output of SHOW ANALYZE STATUS. | **PROPERTIES** | **Type** | **Default value** | **Description** | | -------------------------------- | -------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | statistic\_sample\_collect\_rows | INT | 200000 | The minimum number of rows to collect for sampled collection.If the parameter value exceeds the actual number of rows in your table, full collection is performed. | Examples Manual full collection ```sql -- Manually collect full stats of a table using default settings. ANALYZE TABLE tbl_name; -- Manually collect full stats of a table using default settings. ANALYZE FULL TABLE tbl_name; -- Manually collect stats of specified columns in a table using default settings. ANALYZE TABLE tbl_name(c1, c2, c3); ``` Manual sampled collection ```sql -- Manually collect partial stats of a table using default settings. ANALYZE SAMPLE TABLE tbl_name; -- Manually collect stats of specified columns in a table, with the number of rows to collect specified. ANALYZE SAMPLE TABLE tbl_name (v1, v2, v3) PROPERTIES( "statistic_sample_collect_rows" = "1000000" ); ``` * Manual collection of multi-column joint statistics ```sql -- Manual sampled collection of multi-column joint statistics ANALYZE SAMPLE TABLE tbl_name MULTIPLE COLUMNS (v1, v2); -- Manual full collection of multi-column joint statistics ANALYZE FULL TABLE tbl_name MULTIPLE COLUMNS (v1, v2); ``` * Manual collection of Predicate Column ```sql -- Manual sampled collection of Predicate Column ANALYZE SAMPLE TABLE tbl_name PREDICATE COLUMNS -- Manual full collection of Predicate Column ANALYZE FULL TABLE tbl_name PREDICATE COLUMNS ``` ###### Manually collect histograms[​](#manually-collect-histograms "Direct link to Manually collect histograms") ```sql ANALYZE TABLE tbl_name UPDATE HISTOGRAM ON col_name [, col_name] [WITH SYNC | ASYNC MODE] [WITH N BUCKETS] [PROPERTIES (property [,property])] ``` Parameter description: * `col_name`: columns from which to collect statistics. Separate multiple columns with commas (`,`). If this parameter is not specified, the entire table is collected. This parameter is required for histograms. * `WITH SYNC | ASYNC MODE`: whether to run the manual collection task in synchronous or asynchronous mode. Synchronous collection is used by default if you not specify this parameter. * `WITH N BUCKETS`: `N` is the number of buckets for histogram collection. If not specified, the default value in `fe.conf` is used. * PROPERTIES: custom parameters. If `PROPERTIES` is not specified, the default settings in `fe.conf` are used. | **PROPERTIES** | **Type** | **Default value** | **Description** | | ------------------------------------- | -------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | statistic\_sample\_collect\_rows | INT | 200000 | The minimum number of rows to collect. If the parameter value exceeds the actual number of rows in your table, full collection is performed. | | histogram\_buckets\_size | LONG | 64 | The default bucket number for a histogram. | | histogram\_mcv\_size | INT | 100 | The number of most common values (MCV) for a histogram. | | histogram\_sample\_ratio | FLOAT | 0.1 | The sampling ratio for a histogram. | | histogram\_max\_sample\_row\_count | LONG | 10000000 | The maximum number of rows to collect for a histogram. | | histogram\_collect\_bucket\_ndv\_mode | STRING | none | The mode for estimating the number of distinct values (NDV) per histogram bucket. `none` (default, no distinct count collected), `hll` (uses HyperLogLog for accurate estimation), or `sample` (uses a low overhead sample-based estimator). | The number of rows to collect for a histogram is controlled by multiple parameters. It is the larger value between `statistic_sample_collect_rows` and table row count \* `histogram_sample_ratio`. The number cannot exceed the value specified by `histogram_max_sample_row_count`. If the value is exceeded, `histogram_max_sample_row_count` takes precedence. The properties that are actually used can be viewed via the `Properties` column in the output of SHOW ANALYZE STATUS. Examples ```sql -- Manually collect histograms on v1 using the default settings. ANALYZE TABLE tbl_name UPDATE HISTOGRAM ON v1; -- Manually collect histograms on v1 and v2, with 32 buckets, 32 MCVs, and 50% sampling ratio. ANALYZE TABLE tbl_name UPDATE HISTOGRAM ON v1,v2 WITH 32 BUCKETS PROPERTIES( "histogram_mcv_size" = "32", "histogram_sample_ratio" = "0.5" ); -- Collect histograms on v3 using the 'hll' mode for accurate distinct count estimation per bucket. ANALYZE TABLE tbl_name UPDATE HISTOGRAM ON v3 PROPERTIES( "histogram_collect_bucket_ndv_mode" = "hll" ); ``` ##### Custom collection[​](#custom-collection "Direct link to Custom collection") ###### Customize an automatic collection task[​](#customize-an-automatic-collection-task "Direct link to Customize an automatic collection task") The default collection tasks provided by StarRocks automatically collect statistics for all databases and all tables according to the policy, and by default, you do not need to create custom collection tasks. If you want to customize an automatic collection task, you need to create it through the CREATE ANALYZE statement. To create a collection task, you need the INSERT and SELECT permissions for the table being collected. ```sql -- Automatically collect stats of all databases. CREATE ANALYZE [FULL|SAMPLE] ALL [PROPERTIES (property [,property])] -- Automatically collect stats of all tables in a database. CREATE ANALYZE [FULL|SAMPLE] DATABASE db_name [PROPERTIES (property [,property])] -- Automatically collect stats of specified columns in a table. CREATE ANALYZE [FULL|SAMPLE] TABLE tbl_name (col_name [,col_name]) [PROPERTIES (property [,property])] -- Automatically collect histograms of specified columns in a table. CREATE ANALYZE TABLE tbl_name UPDATE HISTOGRAM ON col_name [, col_name] [WITH SYNC | ASYNC MODE] [WITH N BUCKETS] [PROPERTIES (property [,property])] ``` Parameter description: * Collection type * FULL: indicates full collection. * SAMPLE: indicates sampled collection. * If no collection type is specified, sample collection is used by default. * `col_name`: columns from which to collect statistics. Separate multiple columns with commas (`,`). If this parameter is not specified, the entire table is collected. * `PROPERTIES`: custom parameters. If `PROPERTIES` is not specified, the default settings in `fe.conf` are used. | **PROPERTIES** | **Type** | **Default value** | **Description** | | ---------------------------------- | -------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | statistic\_auto\_collect\_ratio | FLOAT | 0.8 | The threshold for determining whether the statistics for automatic collection are healthy. If the statistics health is below this threshold, automatic collection is triggered. | | statistic\_sample\_collect\_rows | INT | 200000 | The minimum number of rows to collect.If the parameter value exceeds the actual number of rows in your table, full collection is performed. | | statistic\_exclude\_pattern | String | null | The name of the database or table that needs to be excluded in the job. You can specify the database and table that do not collect statistics in the job. Note that this is a regular expression pattern, and the match content is `database.table`. | | statistic\_auto\_collect\_interval | LONG | 0 | The interval for automatic collection. Unit: seconds. By default, StarRocks chooses `statistic_auto_collect_small_table_interval` or `statistic_auto_collect_large_table_interval` as the collection interval based on the table size. If you specified the `statistic_auto_collect_interval` property when you create an analyze job, this setting takes precedence over `statistic_auto_collect_small_table_interval` and `statistic_auto_collect_large_table_interval`. | Examples Automatic full collection ```sql -- Automatically collect full stats of all databases. CREATE ANALYZE ALL; -- Automatically collect full stats of a database. CREATE ANALYZE DATABASE db_name; -- Automatically collect full stats of all tables in a database. CREATE ANALYZE FULL DATABASE db_name; -- Automatically collect full stats of specified columns in a table. CREATE ANALYZE TABLE tbl_name(c1, c2, c3); -- Automatically collect stats of all databases, excluding specified database 'db_name'. CREATE ANALYZE ALL PROPERTIES ( "statistic_exclude_pattern" = "db_name\." ); -- Automatically collect histograms of specified database, table, or columns. CREATE ANALYZE TABLE tbl_name UPDATE HISTOGRAM ON c1,c2; ``` Automatic sampled collection ```sql -- Automatically collect stats of all tables in a database with default settings. CREATE ANALYZE SAMPLE DATABASE db_name; -- Automatically collect stats of all tables in a database, excluding specified table 'db_name.tbl_name'. CREATE ANALYZE SAMPLE DATABASE db_name PROPERTIES ( "statistic_exclude_pattern" = "db_name.tbl_name" ); -- Automatically collect stats of specified columns in a table, with statistics health and the number of rows to collect specified. CREATE ANALYZE SAMPLE TABLE tbl_name(c1, c2, c3) PROPERTIES ( "statistic_auto_collect_ratio" = "0.5", "statistic_sample_collect_rows" = "1000000" ); ``` **Instead of the auto collection task provided by StarRocks, use a user-defined collection task that does not collect the `db_name.tbl_name` table.** ```sql ADMIN SET FRONTEND CONFIG("enable_auto_collect_statistics"="false"); DROP ALL ANALYZE JOB; CREATE ANALYZE FULL ALL db_name PROPERTIES ( "statistic_exclude_pattern" = "db_name.tbl_name" ); ``` **Keep the auto collection task provided by StarRocks, but skip the `db_name.tbl_name` table.** ```sql ALTER TABLE db_name.tbl_name SET ("enable_statistic_auto_collect" = "false"); ``` ##### Collect statistics during data loading[​](#collect-statistics-during-data-loading "Direct link to Collect statistics during data loading") To ensure a good execution plan for queries immediately after data loading, StarRocks triggers an asynchronous statistic collection task at the end of the INSERT INTO/OVERWRITE DML statements, which by default waits for 30 seconds after the DML finishes. If the statistics collection task does not finish in 30 seconds, the DML execution result is returned. ###### INSERT INTO[​](#insert-into "Direct link to INSERT INTO") * Statistics is collected only for the first import of data into the partition. * If the number of rows in this import is greater than `statistic_sample_collect_rows`, the sampled collection task is triggered, otherwise full collection is used. ###### INSERT OVERWRITE[​](#insert-overwrite "Direct link to INSERT OVERWRITE") * If the ratio of the change in the number of rows before and after the OVERWRITE is less than `statistic_sample_collect_ratio_threshold_of_first_load`, the statistics collection task is not triggered. * If the number of rows in this OVERWRITE operation is greater than `statistic_sample_collect_rows`, then a sampled collection task is triggered, otherwise full collection is used. The following properties (PROPERTIES) are used to create customized collection tasks for data loading. If not configured, the value of the corresponding FE configuration item is used. | **PROPERTIES** | **Type** | **Dedault** | **Description** | | ------------------------------------------------------------- | -------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------- | | enable\_statistic\_collect\_on\_first\_load | BOOLEAN | TRUE | Whether to trigger the statistics collection task after executing INSERT INTO/OVERWRITE. | | semi\_sync\_collect\_statistic\_await\_seconds | LONG | 30 | Maximum time to wait for statistics to be collected before returning results. | | statistic\_sample\_collect\_ratio\_threshold\_of\_first\_load | DOUBLE | 0.1 | The ratio of data changes in OVERWRITE operations that will not trigger statistics collection tasks. | | statistic\_sample\_collect\_rows | LONG | 200000 | When the total data rows loaded via DML statements exceeds this value, sampled collection is used for statistics collection. | ###### View custom collection tasks[​](#view-custom-collection-tasks "Direct link to View custom collection tasks") ```sql SHOW ANALYZE JOB [WHERE predicate][ORDER BY columns][LIMIT num] ``` You can filter results by using the WHERE clause. The statement returns the following columns. | **Column** | **Description** | | ------------ | ---------------------------------------------------------------------------------- | | Id | The ID of the collection task. | | Database | The database name. | | Table | The table name. | | Columns | The column names. | | Type | The type of statistics, including `FULL` and `SAMPLE`. | | Schedule | The type of scheduling. The type is `SCHEDULE` for an automatic task. | | Properties | Custom parameters. | | Status | The task status, including PENDING, RUNNING, SUCCESS, and FAILED. | | LastWorkTime | The time of the last collection. | | Reason | The reason why the task failed. NULL is returned if task execution was successful. | Examples ```sql -- View all the custom collection tasks. SHOW ANALYZE JOB -- View custom collection tasks of database `test`. SHOW ANALYZE JOB where `database` = 'test'; ``` ###### Delete a custom collection task[​](#delete-a-custom-collection-task "Direct link to Delete a custom collection task") ```sql DROP ANALYZE | DROP ALL ANALYZE JOB ``` The task ID can be obtained by using the SHOW ANALYZE JOB statement. Examples ```sql DROP ANALYZE 266030; ``` ```sql DROP ALL ANALYZE JOB; ``` #### View status of collection tasks[​](#view-status-of-collection-tasks "Direct link to View status of collection tasks") You can view the status of all the current tasks by running the SHOW ANALYZE STATUS statement. This statement cannot be used to view the status of custom collection tasks. To view the status of custom collection tasks, use SHOW ANALYZE JOB. ```sql SHOW ANALYZE STATUS [WHERE predicate]; ``` You can use `LIKE or WHERE` to filter the information to return. This statement returns the following columns. | **List name** | **Description** | | ------------- | --------------------------------------------------------------------------------- | | Id | The ID of the collection task. | | Database | The database name. | | Table | The table name. | | Columns | The column names. | | Type | The type of statistics, including FULL, SAMPLE, and HISTOGRAM. | | Schedule | The type of scheduling. `ONCE` means manual and `SCHEDULE` means automatic. | | Status | The status of the task. | | StartTime | The time when the task starts executing. | | EndTime | The time when the task execution ends. | | Properties | Custom parameters. | | Reason | The reason why the task failed. NULL is returned if the execution was successful. | #### View statistics[​](#view-statistics "Direct link to View statistics") ##### View metadata of basic statistics[​](#view-metadata-of-basic-statistics "Direct link to View metadata of basic statistics") ```sql SHOW STATS META [WHERE predicate][ORDER BY columns][LIMIT num] ``` This statement returns the following columns. | **Column** | **Description** | | --------------------- | ------------------------------------------------------------------------------------------- | | Database | The database name. | | Table | The table name. | | Columns | The column names. | | Type | The type of statistics. `FULL` means full collection and `SAMPLE` means sampled collection. | | UpdateTime | The latest statistics update time for the current table. | | Properties | Custom parameters. | | Healthy | The health of statistical information. | | ColumnStats | Column ANALYZE type. | | TabletStatsReportTime | The time when the Tablet metadata for the table was updated in the FE. | | TableHealthyMetrics | Metric of health in statistical information. | | TableUpdateTime | The time when the table was updated. | ##### View metadata of histograms[​](#view-metadata-of-histograms "Direct link to View metadata of histograms") ```sql SHOW HISTOGRAM META [WHERE predicate] ``` This statement returns the following columns. | **Column** | **Description** | | ---------- | ------------------------------------------------------------ | | Database | The database name. | | Table | The table name. | | Column | The columns. | | Type | Type of statistics. The value is `HISTOGRAM` for histograms. | | UpdateTime | The latest statistics update time for the current table. | | Properties | Custom parameters. | #### Delete statistics[​](#delete-statistics "Direct link to Delete statistics") You can delete statistical information you do not need. When you delete statistics, both the data and metadata of the statistics are deleted, as well as the statistics in expired cache. Note that if an automatic collection task is ongoing, previously deleted statistics may be collected again. You can use `SHOW ANALYZE STATUS` to view the history of collection tasks. ##### Delete basic statistics[​](#delete-basic-statistics "Direct link to Delete basic statistics") The following statement deletes the statistics stored in the `default_catalog._statistics_.column_statistics` table, and the corresponding table statistics cached by FE will also be invalidated. Since v3.5.0, this statement also deletes multi-column joint statistics for this table. ```sql DROP STATS tbl_name ``` The following statement deletes the multi-column joint statistics stored in the `default_catalog._statistics_.multi_column_statistics` table, and the multi-column joint statistics in the corresponding table cached by FE will also be invalidated. This statement does not delete the basic statistics of the table. ```sql DROP MULTIPLE COLUMNS STATS tbl_name ``` ##### Delete histograms[​](#delete-histograms "Direct link to Delete histograms") ```sql ANALYZE TABLE tbl_name DROP HISTOGRAM ON col_name [, col_name] ``` #### Cancel a collection task[​](#cancel-a-collection-task "Direct link to Cancel a collection task") You can use the KILL ANALYZE statement to cancel a **running** collection task, including manual and custom tasks. ```sql KILL ANALYZE ``` The task ID for a manual collection task can be obtained from SHOW ANALYZE STATUS. The task ID for a custom collection task can be obtained from SHOW ANALYZE SHOW ANALYZE JOB. #### Other FE configuration items[​](#other-fe-configuration-items "Direct link to Other FE configuration items") | **FE configuration item** | **Type** | **Default value** | **Description** | | ---------------------------------------- | -------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | statistic\_manager\_sleep\_time\_sec | LONG | 60 | The interval at which metadata is scheduled. Unit: seconds. The system performs the following operations based on this interval:Create tables for storing statistics.Delete statistics that have been deleted.Delete expired statistics. | | statistic\_analyze\_status\_keep\_second | LONG | 259200 | The duration to retain the history of collection tasks. Unit: seconds. Default value: 259200 (3 days). | #### Session variable[​](#session-variable "Direct link to Session variable") `statistic_collect_parallel`: Used to adjust the parallelism of statistics collection tasks that can run on BEs. Default value: 1. You can increase this value to speed up collection tasks. #### Collect statistics of external tables[​](#collect-statistics-of-external-tables "Direct link to Collect statistics of external tables") Since v3.2.0, StarRocks supports collecting statistics of Hive, Iceberg, and Hudi tables. The syntax is similar to collecting StarRocks internal tables. **However, only manual full collection, manual histogram collection (since v3.2.7), and automatic full collection are supported.** Since v3.3.0, StarRocks supports collecting statistics of Delta Lake tables and statistics of sub-fields in STRUCT. Since v3.4.0, StarRocks supports automatic statistics collection via query-triggered ANALYZE tasks. Since v3.4.0, StarRocks also supports manual sampled collection (`ANALYZE TABLE ... SAMPLE`) for external tables. Histogram collection is currently only supported on Hive external tables. The collected statistics are stored in the `external_column_statistics` table of the `_statistics_` in the `default_catalog`. They are not stored in Hive Metastore and cannot be shared by other search engines. You can query data from the `default_catalog._statistics_.external_column_statistics` table to verify whether statistics are collected for a Hive/Iceberg/Hudi table. Following is an example of querying statistics data from `external_column_statistics`. ```sql SELECT * FROM _statistics_.external_column_statistics\G *************************** 1. row *************************** table_uuid: hive_catalog.tn_test.ex_hive_tbl.1673596430 partition_name: column_name: k1 catalog_name: hive_catalog db_name: tn_test table_name: ex_hive_tbl row_count: 3 data_size: 12 ndv: NULL null_count: 0 max: 3 min: 2 update_time: 2023-12-01 14:57:27.137000 ``` ##### Limits[​](#limits "Direct link to Limits") The following limits apply when you collect statistics for external tables: * You can collect statistics of only Hive, Iceberg, Hudi, and Delta Lake (Since v3.3.0) tables. * Only manual full collection, manual histogram collection (since v3.2.7), automatic full collection, query-triggered collection (since v3.4.0), and manual sampled collection (since v3.4.0) are supported. Histogram collection is only supported on Hive external tables. * For the system to automatically collect full statistics, you must create an Analyze job, which is different from collecting statistics of StarRocks internal tables where the system does this in the background by default. * For automatic collection tasks: * You can only collect statistics of a specific table. You cannot collect statistics of all tables in a database or statistics of all databases in an external catalog. * StarRocks can detect whether data in Hive and Iceberg tables are updated and if so, collect statistics of only partitions whose data is updated. StarRocks cannot perceive whether data in Hudi tables are updated and can only perform periodic full collection. * For query-triggered collection tasks: * Currently, only Leader FE node can trigger ANALYZE tasks. * The system only supports checking for partition changes on Hive and Iceberg tables, and only collects statistics on the partitions where the data has changed. For the Delta Lake/Hudi tables, the system collects statistics of the entire table. * If Partition Transforms are applied to Iceberg tables, the collection of statistics is supported only for `identity`, `year`, `month`, `day`, `hour` type Transforms. * Collecting statistics for Partition Evolution for Iceberg tables is supported in version 4.0 and above. The following examples happen in a database under the Hive external catalog. If you want to collect statistics of a Hive table from the `default_catalog`, reference the table in the `[catalog_name.][database_name.]` format. ##### Query-triggered collection[​](#query-triggered-collection "Direct link to Query-triggered collection") Since v3.4.0, the system supports automatic statistics collection of external tables by query-triggered ANALYZE tasks. When querying Hive, Iceberg, Hudi, or Delta Lake tables, the system will automatically trigger ANALYZE tasks in the background to collect statistics of the corresponding tables and columns, and use it for subsequent query plan optimization. Workflow: 1. When the optimizer queries the cached statistics in FE, it will determine the object of the ANALYZE task based on the queried tables and columns (the ANALYZE task will only collect the statistics of the columns included in the query). 2. The system will encapsulate the task object as an ANALYZE task and add it to the PendingTaskQueue. 3. The Scheduler thread periodically fetches tasks from the PendingTaskQueue and add them to the RunningTasksQueue. 4. During the execution of the ANALYZE task, it collects statistics and writes them to the BE and clears the expired statistics cached in the FE. This feature is enabled by default. You can control the above process with the following system variables and configuration items. ###### System variable[​](#system-variable "Direct link to System variable") ###### enable\_query\_trigger\_analyze[​](#enable_query_trigger_analyze "Direct link to enable_query_trigger_analyze") * Default: true * Type: Boolean * Description: Whether to enable query-trigger ANALYZE tasks. * Introduced in: v3.4.0 ###### FE configuration[​](#fe-configuration "Direct link to FE configuration") ###### connector\_table\_query\_trigger\_analyze\_small\_table\_rows[​](#connector_table_query_trigger_analyze_small_table_rows "Direct link to connector_table_query_trigger_analyze_small_table_rows") * Default: 10000000 * Type: Int * Unit: - * Is mutable: Yes * Description: The threshold for determining whether a table is a small table for query-trigger ANALYZE tasks. * Introduced in: v3.4.0 ###### connector\_table\_query\_trigger\_analyze\_small\_table\_interval[​](#connector_table_query_trigger_analyze_small_table_interval "Direct link to connector_table_query_trigger_analyze_small_table_interval") * Default: 2 \* 3600 * Type: Int * Unit: Second * Is mutable: Yes * Description: The interval for query-trigger ANALYZE tasks of small tables. * Introduced in: v3.4.0 ###### connector\_table\_query\_trigger\_analyze\_large\_table\_interval[​](#connector_table_query_trigger_analyze_large_table_interval "Direct link to connector_table_query_trigger_analyze_large_table_interval") * Default: 12 \* 3600 * Type: Int * Unit: Second * Is mutable: Yes * Description: The interval for query-trigger ANALYZE tasks of large tables. * Introduced in: v3.4.0 ###### connector\_table\_query\_trigger\_analyze\_max\_pending\_task\_num[​](#connector_table_query_trigger_analyze_max_pending_task_num "Direct link to connector_table_query_trigger_analyze_max_pending_task_num") * Default: 100 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of query-trigger ANALYZE tasks that are in Pending state on the FE. * Introduced in: v3.4.0 ###### connector\_table\_query\_trigger\_analyze\_schedule\_interval[​](#connector_table_query_trigger_analyze_schedule_interval "Direct link to connector_table_query_trigger_analyze_schedule_interval") * Default: 30 * Type: Int * Unit: Second * Is mutable: Yes * Description: The interval at which the Scheduler thread schedules to query-trigger ANALYZE tasks. * Introduced in: v3.4.0 ###### connector\_table\_query\_trigger\_analyze\_max\_running\_task\_num[​](#connector_table_query_trigger_analyze_max_running_task_num "Direct link to connector_table_query_trigger_analyze_max_running_task_num") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of query-trigger ANALYZE tasks that are in Running state on the FE. * Introduced in: v3.4.0 ##### Manual collection[​](#manual-collection-1 "Direct link to Manual collection") You can create an Analyze job on demand and the job runs immediately after you create it. ###### Create a manual collection task[​](#create-a-manual-collection-task "Direct link to Create a manual collection task") Syntax: ```sql -- Manual full collection ANALYZE [FULL] TABLE tbl_name (col_name [,col_name]) [WITH SYNC | ASYNC MODE] [PROPERTIES(property [,property])] -- Manual histogram collection (since v3.3.0) ANALYZE TABLE tbl_name UPDATE HISTOGRAM ON col_name [, col_name] [WITH SYNC | ASYNC MODE] [WITH N BUCKETS] [PROPERTIES (property [,property])] ``` Following is an example of manual full collection: ```sql ANALYZE TABLE ex_hive_tbl(k1); +----------------------------------+---------+----------+----------+ | Table | Op | Msg_type | Msg_text | +----------------------------------+---------+----------+----------+ | hive_catalog.tn_test.ex_hive_tbl | analyze | status | OK | +----------------------------------+---------+----------+----------+ ``` ###### View the task status[​](#view-the-task-status "Direct link to View the task status") Syntax: ```sql SHOW ANALYZE STATUS [LIKE | WHERE predicate] ``` Example: ```sql SHOW ANALYZE STATUS where `table` = 'ex_hive_tbl'; +-------+----------------------+-------------+---------+------+----------+---------+---------------------+---------------------+------------+--------+ | Id | Database | Table | Columns | Type | Schedule | Status | StartTime | EndTime | Properties | Reason | +-------+----------------------+-------------+---------+------+----------+---------+---------------------+---------------------+------------+--------+ | 16400 | hive_catalog.tn_test | ex_hive_tbl | k1 | FULL | ONCE | SUCCESS | 2023-12-04 16:31:42 | 2023-12-04 16:31:42 | {} | | | 16465 | hive_catalog.tn_test | ex_hive_tbl | k1 | FULL | ONCE | SUCCESS | 2023-12-04 16:37:35 | 2023-12-04 16:37:35 | {} | | | 16467 | hive_catalog.tn_test | ex_hive_tbl | k1 | FULL | ONCE | SUCCESS | 2023-12-04 16:37:46 | 2023-12-04 16:37:46 | {} | | +-------+----------------------+-------------+---------+------+----------+---------+---------------------+---------------------+------------+--------+ ``` ###### View metadata of statistics[​](#view-metadata-of-statistics "Direct link to View metadata of statistics") Syntax: ```sql SHOW STATS META [WHERE predicate] ``` Example: ```sql SHOW STATS META where `table` = 'ex_hive_tbl'; +----------------------+-------------+---------+------+---------------------+------------+---------+ | Database | Table | Columns | Type | UpdateTime | Properties | Healthy | +----------------------+-------------+---------+------+---------------------+------------+---------+ | hive_catalog.tn_test | ex_hive_tbl | k1 | FULL | 2023-12-04 16:37:46 | {} | | +----------------------+-------------+---------+------+---------------------+------------+---------+ ``` ###### Cancel a collection task[​](#cancel-a-collection-task-1 "Direct link to Cancel a collection task") Cancels a running collection task. Syntax: ```sql KILL ANALYZE ``` You can view the task ID in the output of SHOW ANALYZE STATUS. ##### Automatic collection[​](#automatic-collection-1 "Direct link to Automatic collection") For the system to automatically collect statistics of tables in an external data source, you can create an Analyze job. StarRocks automatically checks whether to run the task at the default check interval of 5 minutes. For Hive and Iceberg tables, StarRocks runs a collection task only when data in the tables are updated. However, data changes in Hudi tables cannot be perceived and StarRocks periodically collects statistics based on the check interval and collection interval you specified. You can specify the following FE configuration items to control the collection behaviors: * statistic\_collect\_interval\_sec The interval for checking data updates during automatic collection. Unit: seconds. Default: 5 minutes. * statistic\_auto\_collect\_small\_table\_rows (v3.2 and later) Threshold to determine whether a table in an external data source (Hive, Iceberg, Hudi) is a small table during automatic collection. Default: 10000000. * statistic\_auto\_collect\_small\_table\_interval The interval for collecting statistics of small tables. Unit: seconds. Default: 0. * statistic\_auto\_collect\_large\_table\_interval The interval for collecting statistics of large tables. Unit: seconds. Default: 43200 (12 hours). Automatic collection threads checks data updates at the interval specified by `statistic_collect_interval_sec`. If the number of rows in a table is less than `statistic_auto_collect_small_table_rows`, it collects statistics of such tables based on `statistic_auto_collect_small_table_interval`. If the number of rows in a table exceeds `statistic_auto_collect_small_table_rows`, it collects statistics of such tables based on `statistic_auto_collect_large_table_interval`. It updates statistics of large tables only when `Last table update time + Collection interval > Current time`. This prevents frequent analyze tasks for large tables. ###### Create an automatic collection task[​](#create-an-automatic-collection-task "Direct link to Create an automatic collection task") Syntax: ```sql CREATE ANALYZE TABLE tbl_name (col_name [,col_name]) [PROPERTIES (property [,property])] ``` You can specify the property `statistic_auto_collect_interval` to set the collection interval exclusively for the automatic collection task. The FE configuration items `statistic_auto_collect_small_table_interval` and `statistic_auto_collect_large_table_interval` will not take effect on this task. Example: ```sql CREATE ANALYZE TABLE ex_hive_tbl (k1) PROPERTIES ("statistic_auto_collect_interval" = "5"); Query OK, 0 rows affected (0.01 sec) ``` ###### View the status of an automatic collection task[​](#view-the-status-of-an-automatic-collection-task "Direct link to View the status of an automatic collection task") Same as manual collection. ###### View metadata of statistics[​](#view-metadata-of-statistics-1 "Direct link to View metadata of statistics") Same as manual collection. ###### View automatic collection tasks[​](#view-automatic-collection-tasks "Direct link to View automatic collection tasks") Syntax: ```sql SHOW ANALYZE JOB [WHERE predicate] ``` Example: ```sql SHOW ANALYZE JOB WHERE `id` = '17204'; Empty set (0.00 sec) ``` ###### Cancel a collection task[​](#cancel-a-collection-task-2 "Direct link to Cancel a collection task") Same as manual collection. ###### Delete statistics[​](#delete-statistics-1 "Direct link to Delete statistics") ```sql DROP STATS tbl_name ``` #### References[​](#references "Direct link to References") * To query FE configuration items, run [ADMIN SHOW CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). * To modify FE configuration items, run [ADMIN SET CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md). --- ### Computing distinct value count #### [📄️ Bitmap for Count Distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_bitmap.md) [How to use bitmaps in StarRocks for efficient COUNT(DISTINCT) calculations using the BITMAP type and bitmap functions, reducing storage space and...](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_bitmap.md) #### [📄️ Use HLL for approximate count distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_HLL.md) [How to use HyperLogLog (HLL) in StarRocks for approximate COUNT(DISTINCT) with ~1-10% error rate, significantly reducing memory usage for large-scale...](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_HLL.md) --- ### Bitmap for Count Distinct Use bitmaps to compute the number of distinct values. Bitmaps are a useful tool for computing the number of distinct values in an array. This method takes up less storage space and can accelerate computation when compared to traditional Count Distinct. Assume there is an array named A with a value range of \[0, n). By using a bitmap of (n+7)/8 bytes, you can compute the number of distinct elements in the array. To do this, initialize all bits to 0, set the values of the elements as the subscripts of bits, and then set all bits to 1. The number of 1s in the bitmap is the number of distinct elements in the array. #### Traditional Count Distinct[​](#traditional-count-distinct "Direct link to Traditional Count Distinct") The database MPP architecture can retain detailed data when using Count Distinct. However, the Count Distinct feature requires multiple data shuffles during query processing, which consumes more resources and results in a linear decrease in performance as the data volume increases. The following scenario calculates UVs based on detailed data in table (`dt`, `page`, `user_id`). | dt | page | user\_id | | -------- | -------- | -------- | | 20191206 | game | 101 | | 20191206 | shopping | 102 | | 20191206 | game | 101 | | 20191206 | shopping | 101 | | 20191206 | game | 101 | | 20191206 | shopping | 101 | The data is computed according to the following figure. It first groups data by the `page` and `user_id` columns, and then counts the processed result. ![alter](/assets/images/6.1.2-2-6a82bd242b24c4e2c7a406aa3c1cfa89.png) note The figure shows a schematic of 6 rows of data computed on two BE nodes. When dealing with large volumes of data that require multiple shuffle operations, the computational resources needed can increase significantly. This slows queries. However, using the Bitmap technology can help address this issue and improve query performance in such scenarios. Count `uv` grouping by `page`: ```sql select page, count(distinct user_id) as uv from table group by page; ``` | page | uv | | -------- | -- | | game | 1 | | shopping | 2 | #### Benefits of Count Distinct with Bitmap[​](#benefits-of-count-distinct-with-bitmap "Direct link to Benefits of Count Distinct with Bitmap") You can benefit from bitmaps in the following aspects compared with `COUNT(DISTINCT expr)`: * Less storage space: If you use bitmap to compute the number of distinct values for `INT32` data, the required storage space is only 1/32 of `COUNT(DISTINCT expr)`. Compressed roaring bitmaps are used to execute computations, further reducing storage space usage compared to traditional bitmaps. * Faster computation: Bitmaps use bitwise operations, resulting in faster computation compared to `COUNT(DISTINCT expr)`. Computation of the number of distinct values can be processed in parallel, leading to further improvements in query performance. For the implementation of Roaring Bitmap, see [specific paper and implementation](https://github.com/RoaringBitmap/RoaringBitmap). #### Usage notes[​](#usage-notes "Direct link to Usage notes") * Both bitmap indexing and bitmap Count Distinct use the bitmap technique. However, the purpose for introducing them and the problem they solve are completely different. The former is used to filter enumerated columns with a low cardinality, while the latter is used to calculate the number of distinct elements in the value columns of a data row. * You can define a value column as `BITMAP` regardless of the table types (Aggregate table, Duplicate Key table, Primary Key table, or Unique Key table). However, the [sort key](https://docs.starrocks.io/docs/table_design/indexes/Prefix_index_sort_key.md) of a table cannot be of the `BITMAP` type. * When creating a table, you can define the value column as `BITMAP` and the aggregate function as [`BITMAP_UNION`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_union.md). * You can only use roaring bitmaps to compute the number of distinct values for data of the following types: `TINYINT`, `SMALLINT`, `INT`, and `BIGINT`. For data of other types, you need to [build global dictionaries](#global-dictionary). #### Count Distinct with Bitmap[​](#count-distinct-with-bitmap "Direct link to Count Distinct with Bitmap") Take the calculation of page UVs as an example. 1. Create an Aggregate table with a `BITMAP` column `visit_users`, which uses the aggregate function `BITMAP_UNION`. ```sql CREATE TABLE `page_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` BITMAP BITMAP_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`) PROPERTIES ( "replication_num" = "3", "storage_format" = "DEFAULT" ); ``` 2. Load data into this table. Use `INSERT INTO` to load data: ```sql INSERT INTO page_uv VALUES (1, '2020-06-23 01:30:30', to_bitmap(13)), (1, '2020-06-23 01:30:30', to_bitmap(23)), (1, '2020-06-23 01:30:30', to_bitmap(33)), (1, '2020-06-23 02:30:30', to_bitmap(13)), (2, '2020-06-23 01:30:30', to_bitmap(23)); ``` After data is loaded: * In the row `page_id = 1, visit_date = '2020-06-23 01:30:30'`, the `visit_users` field contains three bitmap elements (13, 23, 33). * In the row `page_id = 1, visit_date = '2020-06-23 02:30:30'`, the `visit_users` field contains one bitmap element (13). * In the row `page_id = 2, visit_date = '2020-06-23 01:30:30'`, the `visit_users` field contains one bitmap element (23). 3. Calculate page UVs. ```sql SELECT page_id, count(distinct visit_users) FROM page_uv GROUP BY page_id; ``` ```sql +-----------+------------------------------+ | page_id | count(DISTINCT `visit_users`)| +-----------+------------------------------+ | 1 | 3 | | 2 | 1 | +-----------+------------------------------+ 2 row in set (0.00 sec) ``` #### Global Dictionary[​](#global-dictionary "Direct link to Global Dictionary") Currently, the Bitmap-based Count Distinct mechanism requires the input to be an integer. If the user needs to use other data types as input to the Bitmap, then the user needs to build their own global dictionary to map other types of data (such as string types) to integer types. There are several ideas for building a global dictionary: ##### Hive table-based Global Dictionary[​](#hive-table-based-global-dictionary "Direct link to Hive table-based Global Dictionary") The global dictionary itself in this scheme is a Hive table, which has two columns, one for raw values and one for encoded Int values. The steps to generate the global dictionary are as follows: 1. De-duplicate the dictionary columns of the fact table to generate a temporary table 2. Left join the temporary table and the global dictionary, add `new value` to the temporary table. 3. Encode the `new value` and insert it into the global dictionary. 4. Left join the fact table and the updated global dictionary, replace the dictionary items with IDs. In this way, the global dictionary can be updated and the value columns in the fact table can be replaced using Spark or MR. Compared with the trie tree-based global dictionary, this approach can be distributed and the global dictionary can be reused. However, there are a few things to note: the original fact table is read multiple times, and there are two joins that consume a lot of extra resources during the calculation of the global dictionary. ##### Build a global dictionary based on a trie tree[​](#build-a-global-dictionary-based-on-a-trie-tree "Direct link to Build a global dictionary based on a trie tree") Users can also build their own global dictionaries using trie trees (also known as prefix trees or dictionary trees). The trie tree has common prefixes for the descendants of nodes, which can be used to reduce query time and minimize string comparisons, and therefore is well suited for implementing dictionary encoding. However, the implementation of trie tree is not easy to distribute and can create performance bottlenecks when the data volume is relatively large. By building a global dictionary and converting other types of data to integer data, you can use Bitmap to perform accurate Count Distinct analysis of non-integer data columns. --- ### Use HLL for approximate count distinct #### Background[​](#background "Direct link to Background") In a real-world scenario, the pressure to de-duplicate the data increases as the volume of data increases. When the size of data reaches a certain level, the cost of accurate de-duplication is relatively high. In this case, users usually use approximate algorithms to reduce the computational pressure. HyperLogLog (HLL), which will be introduced in this section, is an approximate de-duplication algorithm that has excellent space complexity O(mloglogn) and time complexity O(n). What’s more, the error rate of the computation result can be controlled to about 1%-10%, depending on the size of the data set and the hash function used. #### What is HyperLogLog[​](#what-is-hyperloglog "Direct link to What is HyperLogLog") HyperLogLog is an approximate de-duplication algorithm that consumes very little storage space. The **HLL type** is used for implementing the HyperLogLog algorithm. It holds the intermediate results of the HyperLogLog calculation and can only be used as an indicator column type for data tables. Since the HLL algorithm involves a lot of mathematical knowledge, we will use a practical example to illustrate it. Suppose we design a randomized experiment A, that is to do independent repetitions of coin flips until the first head; record the number of coin flips for the first head as a random variable X, then: * X=1, P(X=1)=1/2 * X=2, P(X=2)=1/4 * ... * X=n, P(X=n)=(1/2)n We use test A to construct randomized test B which is to do N independent repetitions of test A, generating N independent identically distributed random variables X1, X2, X3, ..., XN.Take the maximum value of the random variables as Xmax. Leveraging the great likelihood estimation, the estimated value of N is 2Xmax.
Now, we simulate the above experiment using the hash function on the given dataset: * Test A: Calculate the hash value of dataset elements and convert the hash value to binary representation. Record the occurrence of bit=1, starting from the lowest bit of the binary. * Test B: Repeat the Test A process for dataset elements of Test B. Update the maximum position "m" of the first bit 1 occurrence for each test; * Estimate the number of non-repeating elements in the dataset as m2. In fact, the HLL algorithm divides the elements into K=2k buckets based on the lower k bits of the element hash. Count the maximum value of the first bit 1 occurrence from the k+1st bit as m1, m2,..., mk, and estimate the number of non-repeating elements in the bucket as 2m1, 2m2,..., 2mk. The number of non-repeating elements in the data set is the summed average of the number of buckets multiplied by the number of non-repeating elements in the buckets: N = K(K/(2-m1+2-m2,..., 2-mK)).
HLL multiplies the correction factor with the estimation result to make the result more accurate. Refer to the article on implementing HLL de-duplication algorithm with StarRocks SQL statements: ```sql SELECT floor((0.721 * 1024 * 1024) / (sum(pow(2, m * -1)) + 1024 - count(*))) AS estimate FROM(select(murmur_hash3_32(c2) & 1023) AS bucket, max((31 - CAST(log2(murmur_hash3_32(c2) & 2147483647) AS INT))) AS m FROM db0.table0 GROUP BY bucket) bucket_values ``` This algorithm de-duplicates col2 of db0.table0. * Use the hash function `murmur_hash3_32` to calculate the hash value of col2 as a 32-signed integer. * Use 1024 buckets, the correction factor is 0.721, and take the lower 10 bits of the hash value as the subscript of the bucket. * Ignore the sign bit of the hash value, start from the next highest bit to the lower bit, and determine the position of the first bit 1 occurrence. * Group the calculated hash values by bucket, and use the `MAX` aggregation to find the maximum position of the first bit 1 occurrence in the bucket. * The aggregation result is used as a subquery, and the summed average of all bucket estimates is multiplied by the number of buckets and the correction factor. * Note that the empty bucket count is 1. The above algorithm has a very low error rate when the data volume is large. This is the core idea of the HLL algorithm. Please refer to the [HyperLogLog paper](https://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf) if you are interested. ##### How to use HyperLogLog[​](#how-to-use-hyperloglog "Direct link to How to use HyperLogLog") 1. To use HyperLogLog de-duplication, you need to set the target indicator column type to `HLL` and the aggregation function to `HLL_UNION` in the table creation statement. 2. Currently, only the Aggregate table supports HLL as indicator column type. 3. When using `count distinct` on columns of the HLL type, StarRocks will automatically convert it to the `HLL_UNION_AGG` calculation. ###### Example[​](#example "Direct link to Example") First, create a table with **HLL** columns, where uv is an aggregated column, the column type is `HLL`, and the aggregation function is [HLL\_UNION](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/hll_union.md). ```sql CREATE TABLE test( dt DATE, id INT, uv HLL HLL_UNION ) DISTRIBUTED BY HASH(ID); ``` > * Note: When the data volume is large, it is better to create a corresponding rollup table for high frequency HLL queries Load data using [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md): ```bash curl --location-trusted -u : -H "label:label_1600997542287" \ -H "column_separator:," \ -H "columns:dt,id,user_id, uv=hll_hash(user_id)" -T /root/test.csv http://starrocks_be0:8040/api/db0/test/_stream_load { "TxnId": 2504748, "Label": "label_1600997542287", "Status": "Success", "Message": "OK", "NumberTotalRows": 5, "NumberLoadedRows": 5, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 120, "LoadTimeMs": 46, "BeginTxnTimeMs": 0, "StreamLoadPutTimeMs": 1, "ReadDataTimeMs": 0, "WriteDataTimeMs": 29, "CommitAndPublishTimeMs": 14 } ``` Broker Load mode: ```sql LOAD LABEL test_db.label ( DATA INFILE("hdfs://:/user/starrocks/data/input/file") INTO TABLE `test` COLUMNS TERMINATED BY "," (dt, id, user_id) SET ( uv = HLL_HASH(user_id) ) ); ``` Querying data * The HLL column does not allow direct query of its original value, use the function [HLL\_UNION\_AGG](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/hll_union_agg.md) to query. * To find the total uv, `SELECT HLL_UNION_AGG(uv) FROM test;` This statement is equivalent to `SELECT COUNT(DISTINCT uv) FROM test;` * Query uv of everyday `SELECT COUNT(DISTINCT uv) FROM test GROUP BY ID;` ##### Cautions[​](#cautions "Direct link to Cautions") How should I choose between Bitmap and HLL? If the base of the dataset is in the millions or tens of millions, and you have a few dozen machines, use `count distinct`. If the base is in the hundreds of millions and needs to be accurately de-duplicated, use`Bitmap`; if approximate de-duplication is acceptable, use the `HLL` type. Bitmap only supports TINYINT, SMALLINT, INT, and BIGINT. Note that LARGEINT is not supported. For other types of data sets to be de-duplicated, a dictionary needs to be built to map the original type to an integer type. Building a dictionary is complex, and requires a trade-off between data volume, update frequency, query efficiency, storage, and other issues. HLL does not need a dictionary, but it needs the corresponding data type to support the hash function. Even in an analytical system that does not support HLL internally, it is still possible to use the hash function and SQL to implement HLL de-duplication. For common columns, users can use the NDV function for approximate de-duplication. This function returns an approximate aggregation of COUNT(DISTINCT col) results, and the underlying implementation converts the data storage type to the HyperLogLog type for calculation. The NDV function consumes a lot of resources when calculating and is therefore not well suited for high concurrency scenarios. If you want to perform user behavior analysis, you may consider IntersectCount or custom UDAF. --- ### Flat JSON Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) This article introduces the basic concept of Flat JSON and how to use this feature. Starting from version 2.2.0, StarRocks supports the JSON data type to allow more flexible data storage. However, when querying JSON, most scenarios do not involve directly reading the entire JSON data but rather accessing data at specified paths. For example: ```sql -- Store required fields in logs as fixed fields, and package other fields that frequently change with business as JSON. SELECT time, event, user, get_json_string(remain_json, "$.from_system"), get_json_string(remain_json, "$.tag") FROM logs; ``` Due to the special nature of the JSON type, its performance in queries is not as good as standard types (INT, STRING, etc.). The reasons include: * Storage overhead: JSON is a semi-structured type that requires storing the structure information of each row, leading to high storage usage and low compression efficiency. * Query complexity: Queries need to detect data structures based on runtime data, making it difficult to achieve vectorized execution optimization. * Redundant data: Queries need to read the entire JSON data, which includes many redundant fields. StarRocks introduces the Flat JSON feature to improve JSON data query efficiency and reduce the complexity of using JSON. * This feature is available from version 3.3.0. It is disabled by default before v4.0 and needs to be enabled manually. From v4.0 onwards, it is enabled by default. #### What is Flat JSON[​](#what-is-flat-json "Direct link to What is Flat JSON") The core principle of Flat JSON is to detect JSON data during load and extract common fields from JSON data for storage as standard type data. When querying JSON, these common fields optimize the query speed of JSON. Example data: ```plaintext 1, {"a": 1, "b": 21, "c": 3, "d": 4} 2, {"a": 2, "b": 22, "d": 4} 3, {"a": 3, "b": 23, "d": [1, 2, 3, 4]} 4, {"a": 4, "b": 24, "d": null} 5, {"a": 5, "b": 25, "d": null} 6, {"c": 6, "d": 1} ``` When loading the above JSON data, fields `a` and `b` are present in most JSON data and have similar data types (both INT). Therefore, the data of fields `a` and `b` can be extracted from JSON and stored separately as two INT columns. When these two columns are used in queries, their data can be directly read without needing to process additional JSON fields, reducing the computational overhead of handling JSON structures. #### Enable Flat JSON[​](#enable-flat-json "Direct link to Enable Flat JSON") Flat JSON is enabled globally by default from v4.0 onwards. For versions earlier than v4.0, you must manually enable it. From v4.0, this feature can be configured on the table level. ##### Enable for versions earlier than v4.0[​](#enable-for-versions-earlier-than-v40 "Direct link to Enable for versions earlier than v4.0") 1. Modify BE configuration: `enable_json_flat`, which defaults to `false` before v4.0. For modification methods, refer to [Configure BE parameters](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#configure-be-parameters). 2. Enable FE pruning feature: ```sql SET GLOBAL cbo_prune_json_subfield = true; ``` ##### Enable Flat JSON Feature on table level[​](#enable-flat-json-feature-on-table-level "Direct link to Enable Flat JSON Feature on table level") Setting Flat JSON-related properties on table level is supported from v4.0 onwards. 1. When creating the table, you can set `flat_json.enable` and other Flat JSON-related properties. For detailed instructions, see [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md#set-flat-json-properties-on-table-level). Alternatively, you can set these properties using [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md). Example: ```sql ALTER TABLE t1 SET ("flat_json.enable" = "true"); ALTER TABLE t1 SET ("flat_json.null.factor" = "0.1"); ALTER TABLE t1 SET ("flat_json.sparsity.factor" = "0.8"); ALTER TABLE t1 SET ("flat_json.column.max" = "90"); ``` 2. Enable FE pruning feature: ```sql SET GLOBAL cbo_prune_json_subfield = true; ``` #### Verify if Flat JSON is Effective[​](#verify-if-flat-json-is-effective "Direct link to Verify if Flat JSON is Effective") After loading data, you can query the extracted sub-columns of the corresponding column: ```sql SELECT flat_json_meta() FROM [_META_]; ``` You can verify whether the executed query benefits from Flat JSON optimization through the [Query Profile](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md) by observing the following metrics: * `PushdownAccessPaths`: The number of sub-field paths pushed down to storage. * `AccessPathHits`: The number of times Flat JSON sub-fields are hit, with detailed information on the specific JSON hit. * `AccessPathUnhits`: The number of times Flat JSON sub-fields are not hit, with detailed information on the specific JSON not hit. * `JsonFlattern`: The time taken to extract sub-columns on-site when Flat JSON is not hit. #### Usage Example[​](#usage-example "Direct link to Usage Example") * Method 1: Configure Flat JSON properties when creating a table with JSON columns. Supported from v4.0 onwards. ```sql CREATE TABLE `t1` ( `k1` int, `k2` JSON, `k3` VARCHAR(20), `k4` JSON ) DUPLICATE KEY(`k1`) COMMENT "OLAP" DISTRIBUTED BY HASH(`k1`) BUCKETS 2 PROPERTIES ( "replication_num" = "3", "flat_json.enable" = "true", "flat_json.null.factor" = "0.5", "flat_json.sparsity.factor" = "0.5", "flat_json.column.max" = "50"); INSERT INTO t1 (k1,k2) VALUES (11,parse_json('{"str":"test_flat_json","Integer":123456,"Double":3.14158,"Object":{"c":"d"},"arr":[10,20,30],"Bool":false,"null":null}')), (15,parse_json('{"str":"test_str0","Integer":11,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')), (15,parse_json('{"str":"test_str1","Integer":111,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')), (15,parse_json('{"str":"test_str2","Integer":222,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')), (15,parse_json('{"str":"test_str2","Integer":222,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')), (16,parse_json('{"str":"test_str3","Integer":333,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')), (17,parse_json('{"str":"test_str3","Integer":333,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')), (18,parse_json('{"str":"test_str5","Integer":444,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')), (19,parse_json('{"str":"test_str6","Integer":444,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')), (20,parse_json('{"str":"test_str6","Integer":444,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')); ``` * Method 2: Enable Flat JSON before creating the table. 1. Enable the feature (refer to other sections) 2. Create a table with JSON columns. In this example, use INSERT INTO to load JSON data into the table. ```sql CREATE TABLE `t1` ( `k1` int, `k2` JSON, `k3` VARCHAR(20), `k4` JSON ) DUPLICATE KEY(`k1`) COMMENT "OLAP" DISTRIBUTED BY HASH(`k1`) BUCKETS 2 PROPERTIES ("replication_num" = "3"); INSERT INTO t1 (k1,k2) VALUES (11,parse_json('{"str":"test_flat_json","Integer":123456,"Double":3.14158,"Object":{"c":"d"},"arr":[10,20,30],"Bool":false,"null":null}')), (15,parse_json('{"str":"test_str0","Integer":11,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')), (15,parse_json('{"str":"test_str1","Integer":111,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')), (15,parse_json('{"str":"test_str2","Integer":222,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')), (15,parse_json('{"str":"test_str2","Integer":222,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')), (16,parse_json('{"str":"test_str3","Integer":333,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')), (17,parse_json('{"str":"test_str3","Integer":333,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')), (18,parse_json('{"str":"test_str5","Integer":444,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')), (19,parse_json('{"str":"test_str6","Integer":444,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')), (20,parse_json('{"str":"test_str6","Integer":444,"Double":3.14,"Object":{"a":"b"},"arr":[1,2,3],"Bool":true,"null":null}')); ``` 3. View the extracted sub-columns for the `k2` column. ```plaintext SELECT flat_json_meta(k2) FROM t1[_META_]; +---------------------------------------------------------------------------------------------------------------------------+ | flat_json_meta(k2) | +---------------------------------------------------------------------------------------------------------------------------+ | ["nulls(TINYINT)","Integer(BIGINT)","Double(DOUBLE)","str(VARCHAR)","Bool(JSON)","Object(JSON)","arr(JSON)","null(JSON)"] | +---------------------------------------------------------------------------------------------------------------------------+ ``` 4. Execute data queries. ```sql SELECT * FROM t1; SELECT get_json_string(k2,'\$.Integer') FROM t1 WHERE k2->'str' = 'test_flat_json'; SELECT get_json_string(k2,'\$.Double') FROM t1 WHERE k2->'Integer' = 123456; SELECT get_json_string(k2,'\$.Object') FROM t1 WHERE k2->'Double' = 3.14158; SELECT get_json_string(k2,'\$.arr') FROM t1 WHERE k2->'Object' = to_json(map{'c':'d'}); SELECT get_json_string(k2,'\$.Bool') FROM t1 WHERE k2->'arr' = '[10,20,30]'; ``` 5. View Flat JSON-related metrics in the [Query Profile](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md) ```yaml PushdownAccessPaths: 2 - Table: t1 - AccessPathHits: 2 - __MAX_OF_AccessPathHits: 1 - __MIN_OF_AccessPathHits: 1 - /k2: 2 - __MAX_OF_/k2: 1 - __MIN_OF_/k2: 1 - AccessPathUnhits: 0 - JsonFlattern: 0ns ``` #### Related variables and configurations[​](#related-variables-and-configurations "Direct link to Related variables and configurations") ##### Session variables[​](#session-variables "Direct link to Session variables") * `cbo_json_v2_rewrite` (default: true): Enables JSON v2 path rewrite so that expressions like `get_json_*` can be rewritten to direct access of Flat JSON subcolumns, unlocking predicate pushdown and column pruning. * `cbo_json_v2_dict_opt` (default: true): Enables low-cardinality dictionary optimization for Flat JSON extended string subcolumns produced by the rewrite, which can speed up string expressions, GROUP BY, and JOIN. Example: ```sql SET cbo_json_v2_rewrite = true; SET cbo_json_v2_dict_opt = true; ``` ##### BE Configurations[​](#be-configurations "Direct link to BE Configurations") * [json\_flat\_null\_factor](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#json_flat_null_factor) * [json\_flat\_column\_max](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#json_flat_column_max) * [json\_flat\_sparsity\_factor](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#json_flat_sparsity_factor) * [enable\_compaction\_flat\_json](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#enable_compaction_flat_json) * [enable\_lazy\_dynamic\_flat\_json](https://docs.starrocks.io/docs/administration/configuration/BE_parameters.md#enable_lazy_dynamic_flat_json) #### Feature Limitations[​](#feature-limitations "Direct link to Feature Limitations") * All table types in StarRocks support Flat JSON. * Compatible with historical data without needing re-import. Historical data will coexist with data flattened by Flat JSON. * Historical data will not automatically apply Flat JSON optimization unless new data is loaded or Compaction occurs. * Enabling Flat JSON will increase the time taken to load JSON. The more JSON extracted, the longer it takes. * Flat JSON can only support materializing common keys in JSON Objects, not keys in JSON Arrays. * Flat JSON does not change the data sorting method, so query performance and data compression rate will still be affected by data sorting. To achieve optimal performance, further adjustments to data sorting may be necessary. #### Version Notes[​](#version-notes "Direct link to Version Notes") StarRocks shared-nothing clusters support Flat JSON starting from v3.3.0, and shared-data clusters support it from v3.3.3. In versions v3.3.0, v3.3.1, and v3.3.2: * When loading data, it supports extracting common fields and storing them separately as JSON types, without type inference. * Both the extracted columns and the original JSON data are stored. The extracted data will be deleted along with the original data. Starting from version v3.3.3: * The results extracted by Flat JSON are divided into common columns and reserved field columns. When all JSON Schemas are consistent, no reserved field columns are generated. * Flat JSON only stores common field columns and reserved field columns, without additionally storing the original JSON data. * When loading data, common fields will automatically infer types as BIGINT/LARGEINT/DOUBLE/STRING. Unrecognized types will be inferred as JSON types, and reserved field columns will be stored as JSON types. --- ### JIT Compilation for Expressions This topic describes how to enable and configure JIT compilation for expressions in StarRocks. #### Overview[​](#overview "Direct link to Overview") Just-in-time compilation (JIT) is to generate the machine code and execute it in runtime. Compared to Interpreter, JIT compiler can significantly improve the execution efficiency of the most used code parts. StarRocks supports JIT compilation for certain complex expressions, allowing a multiplied performance improvement. #### Usage[​](#usage "Direct link to Usage") As of v3.3.0, StarRocks enables JIT compilation by default for BE nodes with a memory limit (configured by BE configuration item `mem_limit`) greater or equal to 16 GB. Because JIT compilation will consume a certain amount of memory resources, it is disabled by default for BE nodes with less than 16 GB of memory. You can enable and configure JIT compilation for expressions using the following parameters. ##### jit\_lru\_cache\_size (BE configuration)[​](#jit_lru_cache_size-be-configuration "Direct link to jit_lru_cache_size (BE configuration)") * Default: 0 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The LRU cache size for JIT compilation. It represents the actual size of the cache if it is set to greater than 0. If it is set to less than or equal to 0, the system will adaptively set the cache using the formula `jit_lru_cache_size = min(mem_limit*0.01, 1GB)` (while `mem_limit` of the node must be greater or equal to 16 GB). * Introduced in: - ##### jit\_level (System variable)[​](#jit_level--system-variable "Direct link to jit_level (System variable)") * Description: The level at which JIT compilation for expressions is enabled. Valid values: * `1`: The system adaptively enables JIT compilation for compilable expressions. * `-1`: JIT compilation is enabled for all compilable, non-constant expressions. * `0`: JIT compilation is disabled. You can disable it manually if any error is returned for this feature. * Default: 1 * Data type: Int * Introduced in: - #### Feature support[​](#feature-support "Direct link to Feature support") ##### Supported expressions[​](#supported-expressions "Direct link to Supported expressions") * `+`, `-`, `*`, `/`, `%`, `&`, `|`, `^`, `>>`, `<<` * Data type conversion with CAST * CASE WHEN * `=`, `!=`, `>`, `>=`, `<`, `<=`, `<=>` * `AND`, `OR`, `NOT` ##### Supported operators[​](#supported-operators "Direct link to Supported operators") * OLAP Scan Operator for filter * Projection Operator * Aggregate Operator for expression * HAVING * Sort Operator for expression ##### Supported data types[​](#supported-data-types "Direct link to Supported data types") * BOOLEAN * TINYINT * SMALLINT * INT * BIGINT * LARGEINT * FLOAT * DOUBLE --- ### Use Lateral Join for column-to-row conversion Column-to-row conversion is a common operation in ETL processing. Lateral is a special Join keyword that can associate a row with an internal subquery or table function. By using Lateral in conjunction with unnest(), you can expand one row into multiple rows. For more information, see [unnest](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/unnest.md). #### Limits[​](#limits "Direct link to Limits") * Currently, Lateral Join is only used with unnest() to achieve column-to-row conversion. Other table functions and UDTFs will be supported later. * Currently, Lateral Join does not support subqueries. #### Use Lateral Join[​](#use-lateral-join "Direct link to Use Lateral Join") Syntax: ```sql from table_reference join [lateral] table_reference; ``` Examples: ```sql SELECT student, score FROM tests CROSS JOIN LATERAL UNNEST(scores) AS t (score); SELECT student, score FROM tests, UNNEST(scores) AS t (score); ``` The second syntax here is a shortened version of the first one, where the Lateral keyword can be omitted using the UNNEST keyword. The UNNEST keyword is a table function that converts an array into multiple rows. Together with Lateral Join, it can implement common row expansion logic. > **NOTE** > > If you want to perform unnest on multiple columns, you must specify an alias for each column, for example, `select v1, t1.unnest as v2, t2.unnest as v3 from lateral_test, unnest(v2) t1, unnest(v3) t2;`. StarRocks supports type conversion among BITMAP, STRING, ARRAY, and Column. ![Some type conversions in Lateral Join](/assets/images/lateral_join_type_conversion-9a1080d8bfac91c8c2d09b8957fad97a.png) #### Usage examples[​](#usage-examples "Direct link to Usage examples") Together with unnest(), you can achieve the following column-to-row conversion features: ##### Expand a string into multiple rows[​](#expand-a-string-into-multiple-rows "Direct link to Expand a string into multiple rows") 1. Create a table and insert data into this table. ```sql CREATE TABLE lateral_test2 ( `v1` bigint(20) NULL COMMENT "", `v2` string NULL COMMENT "" ) DUPLICATE KEY(v1) DISTRIBUTED BY HASH(`v1`) PROPERTIES ( "replication_num" = "3", "storage_format" = "DEFAULT" ); INSERT INTO lateral_test2 VALUES (1, "1,2,3"), (2, "1,3"); ``` 2. Query data before expansion. ```plain select * from lateral_test2; +------+-------+ | v1 | v2 | +------+-------+ | 1 | 1,2,3 | | 2 | 1,3 | +------+-------+ ``` 3. Expand `v2` into multiple rows. ```plain -- Perform unnest on a single column. select v1,unnest from lateral_test2, unnest(split(v2, ",")) as unnest; +------+--------+ | v1 | unnest | +------+--------+ | 1 | 1 | | 1 | 2 | | 1 | 3 | | 2 | 1 | | 2 | 3 | +------+--------+ -- Perform unnest on multiple columns. You must specify an alias for each operation. select v1, t1.unnest as v2, t2.unnest as v3 from lateral_test2, unnest(split(v2, ",")) t1, unnest(split(v3, ",")) t2; +------+------+------+ | v1 | v2 | v3 | +------+------+------+ | 1 | 1 | 1 | | 1 | 1 | 2 | | 1 | 2 | 1 | | 1 | 2 | 2 | | 1 | 3 | 1 | | 1 | 3 | 2 | | 2 | 1 | 1 | | 2 | 1 | 3 | | 2 | 3 | 1 | | 2 | 3 | 3 | +------+------+------+ ``` ##### Expand an array into multiple rows[​](#expand-an-array-into-multiple-rows "Direct link to Expand an array into multiple rows") **From v2.5, unnest() can take multiple arrays of different types and lengths.** For more information, see [unnest()](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/unnest.md). 1. Create a table and insert data into this table. ```sql CREATE TABLE lateral_test ( `v1` bigint(20) NULL COMMENT "", `v2` ARRAY NULL COMMENT "" ) DUPLICATE KEY(v1) DISTRIBUTED BY HASH(`v1`) PROPERTIES ( "replication_num" = "3", "storage_format" = "DEFAULT" ); INSERT INTO lateral_test VALUES (1, [1,2]), (2, [1, null, 3]), (3, null); ``` 2. Query data before expansion. ```plain select * from lateral_test; +------+------------+ | v1 | v2 | +------+------------+ | 1 | [1,2] | | 2 | [1,null,3] | | 3 | NULL | +------+------------+ ``` 3. Expand `v2` into multiple rows. ```plain select v1,v2,unnest from lateral_test , unnest(v2) as unnest; +------+------------+--------+ | v1 | v2 | unnest | +------+------------+--------+ | 1 | [1,2] | 1 | | 1 | [1,2] | 2 | | 2 | [1,null,3] | 1 | | 2 | [1,null,3] | NULL | | 2 | [1,null,3] | 3 | +------+------------+--------+ ``` ##### Expand Bitmap data[​](#expand-bitmap-data "Direct link to Expand Bitmap data") You can use Lateral Join with the unnest\_bitmap function to expand Bitmap data. 1. Create a table and insert data into this table. ```sql CREATE TABLE lateral_test3 ( `v1` bigint(20) NULL COMMENT "", `v2` Bitmap BITMAP_UNION COMMENT "" ) AGGREGATE KEY(v1) DISTRIBUTED BY HASH(`v1`); INSERT INTO lateral_test3 VALUES (1, bitmap_from_string('1, 2')), (2, to_bitmap(3)); ``` 2. Query data before expansion. ```plain select v1, bitmap_to_string(v2) from lateral_test3; +------+------------------------+ | v1 | bitmap_to_string(`v2`) | +------+------------------------+ | 1 | 1,2 | | 2 | 3 | +------+------------------------+ ``` 3. Insert a new row. ```plain insert into lateral_test3 values (1, to_bitmap(3)); select v1, bitmap_to_string(v2) from lateral_test3; +------+------------------------+ | v1 | bitmap_to_string(`v2`) | +------+------------------------+ | 1 | 1,2,3 | | 2 | 3 | +------+------------------------+ ``` 4. Expand data in `v2` into multiple rows. ```plain mysql> select v1, unnest_bitmap from lateral_test3, unnest_bitmap(v2) as unnest_bitmap; +------+---------------+ | v1 | unnest_bitmap | +------+---------------+ | 1 | 1 | | 1 | 2 | | 1 | 3 | | 2 | 3 | +------+---------------+ ``` --- ### Synchronous materialized view This topic describes how to create, use, and manage a **synchronous materialized view (Rollup)**. For a synchronous materialized view, all changes in the base table are simultaneously updated to the corresponding synchronous materialized views. The refresh of a synchronous materialized view is triggered automatically. Synchronous materialized views are significantly inexpensive to maintain and update, making them suitable for transparent acceleration of real-time, single-table aggregate queries. Synchronous materialized views in StarRocks can be created only on a single base table from [the default catalog](https://docs.starrocks.io/docs/data_source/catalog/default_catalog.md). They are essentially a special index for query acceleration rather than a physical table like asynchronous materialized views. From v2.4 onwards, StarRocks provides asynchronous materialized views, which supports creation on multiple tables and more aggregation operators. For the usage of **asynchronous materialized views**, see [Asynchronous materialized view](https://docs.starrocks.io/docs/using_starrocks/async_mv/Materialized_view.md). note * Synchronous materialized views support WHERE clauses from v3.1.8 onwards. * Synchronous materialized views are supported in the shared-data clusters from v3.4.0 onwards. The following table compares the asynchronous materialized views (ASYNC MVs) in StarRocks v2.5, v2.4, and the synchronous materialized view (SYNC MV) in the perspective of features that they support: | | **Single-table aggregation** | **Multi-table join** | **Query rewrite** | **Refresh strategy** | **Base table** | | -------------------- | -------------------------------------------------------------------------------- | -------------------- | ----------------- | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | **ASYNC MV** | Yes | Yes | Yes | - Asynchronous refresh
- Manual refresh | Multiple tables from:- Default catalog
- External catalogs (v2.5)
- Existing materialized views (v2.5)
- Existing views (v3.1) | | **SYNC MV (Rollup)** | Limited choices of [aggregate functions](#correspondence-of-aggregate-functions) | No | Yes | Synchronous refresh during data loading | Single table in the default catalog | #### Basic concepts[​](#basic-concepts "Direct link to Basic concepts") * **Base table** Base tables are the driving tables of a materialized view. For StarRocks' synchronous materialized views, base tables must be a single native table from the [default catalog](https://docs.starrocks.io/docs/data_source/catalog/default_catalog.md). StarRocks supports creating synchronous materialized views on Duplicate Key tables and Aggregate tables. * **Refresh** A synchronous materialized view updates itself every time the data in the base table changes. You do not need to trigger the refresh manually. * **Query rewrite** Query rewrite means that when executing a query on base tables with materialized views built on, the system automatically judges whether the pre-computed results in the materialized view can be reused for the query. If they can be reused, the system will load the data directly from the relevant materialized view to avoid the time- and resource-consuming computations or joins. Synchronous materialized views support query rewrite based on some of the aggregate operators. For more information, see [Correspondence of aggregate functions](#correspondence-of-aggregate-functions). #### Preparation[​](#preparation "Direct link to Preparation") Before creating a synchronous materialized view, check if your data warehouse is eligible for query acceleration through synchronous materialized views. For example, check if the queries reuse certain sub-query statements. The following example is based on the table `sales_records`, which contains the transaction ID `record_id`, salesperson ID `seller_id`, store ID `store_id`, date `sale_date`, and sales amount `sale_amt` for each transaction. Follow these steps to create the table and insert data into it: ```sql CREATE TABLE sales_records( record_id INT, seller_id INT, store_id INT, sale_date DATE, sale_amt BIGINT ) DISTRIBUTED BY HASH(record_id); INSERT INTO sales_records VALUES (001,01,1,"2022-03-13",8573), (002,02,2,"2022-03-14",6948), (003,01,1,"2022-03-14",4319), (004,03,3,"2022-03-15",8734), (005,03,3,"2022-03-16",4212), (006,02,2,"2022-03-17",9515); ``` The business scenario of this example demands frequent analyses on the sales amounts of different stores. As a result, the `sum()` function is used on each query, consuming a massive amount of compute resources. You can run the query to record its time, and view its query profile by using EXPLAIN command. ```plain MySQL > SELECT store_id, SUM(sale_amt) FROM sales_records GROUP BY store_id; +----------+-----------------+ | store_id | sum(`sale_amt`) | +----------+-----------------+ | 2 | 16463 | | 3 | 12946 | | 1 | 12892 | +----------+-----------------+ 3 rows in set (0.02 sec) MySQL > EXPLAIN SELECT store_id, SUM(sale_amt) FROM sales_records GROUP BY store_id; +-----------------------------------------------------------------------------+ | Explain String | +-----------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS:3: store_id | 6: sum | | PARTITION: UNPARTITIONED | | | | RESULT SINK | | | | 4:EXCHANGE | | | | PLAN FRAGMENT 1 | | OUTPUT EXPRS: | | PARTITION: HASH_PARTITIONED: 3: store_id | | | | STREAM DATA SINK | | EXCHANGE ID: 04 | | UNPARTITIONED | | | | 3:AGGREGATE (merge finalize) | | | output: sum(6: sum) | | | group by: 3: store_id | | | | | 2:EXCHANGE | | | | PLAN FRAGMENT 2 | | OUTPUT EXPRS: | | PARTITION: RANDOM | | | | STREAM DATA SINK | | EXCHANGE ID: 02 | | HASH_PARTITIONED: 3: store_id | | | | 1:AGGREGATE (update serialize) | | | STREAMING | | | output: sum(5: sale_amt) | | | group by: 3: store_id | | | | | 0:OlapScanNode | | TABLE: sales_records | | PREAGGREGATION: ON | | partitions=1/1 | | rollup: sales_records | | tabletRatio=10/10 | | tabletList=12049,12053,12057,12061,12065,12069,12073,12077,12081,12085 | | cardinality=1 | | avgRowSize=2.0 | | numNodes=0 | +-----------------------------------------------------------------------------+ 45 rows in set (0.00 sec) ``` It can be observed that the query takes about 0.02 seconds, and no synchronous materialized view is used to accelerate the query because the value of `rollup` field in the query profile is `sales_records`, which is the base table. #### Create a synchronous materialized view[​](#create-a-synchronous-materialized-view "Direct link to Create a synchronous materialized view") You can create a synchronous materialized view based on a specific query statement using [CREATE MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md). Based on the table `sales_records` and the query statement mentioned above, the following example creates the synchronous materialized view `store_amt` to analyze the sum of sales amount in each store. ```sql CREATE MATERIALIZED VIEW store_amt AS SELECT store_id, SUM(sale_amt) FROM sales_records GROUP BY store_id; ``` > **CAUTION** > > * When using aggregate functions in synchronous materialized views, you must use the GROUP BY clause and specify at least one GROUP BY column in your SELECT list. > * Synchronous materialized views do not support using one aggregate function on multiple columns. Query statements in the form of `sum(a+b)` are not supported. > * Synchronous materialized views do not support using multiple aggregate functions on one column. Query statements in the form of `select sum(a), min(a) from table` are not supported. > * JOIN is not supported when creating a synchronous materialized view. > * When using ALTER TABLE DROP COLUMN to drop a specific column in a base table, you need to ensure that all synchronous materialized views of the base table do not contain the dropped column, otherwise the drop operation cannot be performed. To drop a column that used in synchronous materialized views, you need to first drop all synchronous materialized views that contain the column, and then drop the column. > * Creating too many synchronous materialized views for a table will affect the data loading efficiency. When data is being loaded to the base table, the data in synchronous materialized views and base table are updated synchronously. If the base table contains `n` synchronous materialized views, the efficiency of loading data into the base table is about the same as that of loading data into `n` tables. > * Currently, StarRocks does not support creating multiple synchronous materialized views at the same time. A new synchronous materialized view can only be created when the previous one is completed. > * Materialized view can only be created in default\_catalog. You could either create it with default\_catalog.database.mv, or switch to default\_catalog through `set catalog ` statement. #### Check the building status of a synchronous materialized view[​](#check-the-building-status-of-a-synchronous-materialized-view "Direct link to Check the building status of a synchronous materialized view") Creating a synchronous materialized view is an asynchronous operation. Executing CREATE MATERIALIZED VIEW successfully indicates that the task of creating the materialized view is submitted successfully. You can view the building status of the synchronous materialized view in a database via [SHOW ALTER MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/SHOW_ALTER_MATERIALIZED_VIEW.md). ```plain MySQL > SHOW ALTER MATERIALIZED VIEW\G *************************** 1. row *************************** JobId: 12090 TableName: sales_records CreateTime: 2022-08-25 19:41:10 FinishedTime: 2022-08-25 19:41:39 BaseIndexName: sales_records RollupIndexName: store_amt RollupId: 12091 TransactionId: 10 State: FINISHED Msg: Progress: NULL Timeout: 86400 1 row in set (0.00 sec) ``` The `RollupIndexName` section indicates the name of the synchronous materialized view, and `State` section indicates if the building is completed. #### Query a synchronous materialized view directly[​](#query-a-synchronous-materialized-view-directly "Direct link to Query a synchronous materialized view directly") Because a synchronous materialized view is essentially an index of the base table rather than a physical table, you can only query a synchronous materialized view using the hint `[_SYNC_MV_]`: ```sql -- Do not omit the brackets [] in the hint. MySQL > SELECT * FROM store_amt [_SYNC_MV_]; +----------+----------+ | store_id | sale_amt | +----------+----------+ | 2 | 6948 | | 3 | 8734 | | 1 | 4319 | | 2 | 9515 | | 3 | 4212 | | 1 | 8573 | +----------+----------+ ``` > **CAUTION** > > Currently, StarRocks automatically generates names for columns in a synchronous materialized view even if you have specified aliases for them. #### Rewrite and accelerate queries with the synchronous materialized view[​](#rewrite-and-accelerate-queries-with-the-synchronous-materialized-view "Direct link to Rewrite and accelerate queries with the synchronous materialized view") The synchronous materialized view you created contains the complete set of pre-computed results in accordance with the query statement. Subsequent queries use the data within it. You can run the same query to test the query time as you did in the preparation. ```plain MySQL > SELECT store_id, SUM(sale_amt) FROM sales_records GROUP BY store_id; +----------+-----------------+ | store_id | sum(`sale_amt`) | +----------+-----------------+ | 2 | 16463 | | 3 | 12946 | | 1 | 12892 | +----------+-----------------+ 3 rows in set (0.01 sec) ``` It can be observed that the query time is reduced to 0.01 seconds. #### Check if a query hits the synchronous materialized view[​](#check-if-a-query-hits-the-synchronous-materialized-view "Direct link to Check if a query hits the synchronous materialized view") Execute EXPLAIN command again to check if the query hits the synchronous materialized view. ```plain MySQL > EXPLAIN SELECT store_id, SUM(sale_amt) FROM sales_records GROUP BY store_id; +-----------------------------------------------------------------------------+ | Explain String | +-----------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS:3: store_id | 6: sum | | PARTITION: UNPARTITIONED | | | | RESULT SINK | | | | 4:EXCHANGE | | | | PLAN FRAGMENT 1 | | OUTPUT EXPRS: | | PARTITION: HASH_PARTITIONED: 3: store_id | | | | STREAM DATA SINK | | EXCHANGE ID: 04 | | UNPARTITIONED | | | | 3:AGGREGATE (merge finalize) | | | output: sum(6: sum) | | | group by: 3: store_id | | | | | 2:EXCHANGE | | | | PLAN FRAGMENT 2 | | OUTPUT EXPRS: | | PARTITION: RANDOM | | | | STREAM DATA SINK | | EXCHANGE ID: 02 | | HASH_PARTITIONED: 3: store_id | | | | 1:AGGREGATE (update serialize) | | | STREAMING | | | output: sum(5: sale_amt) | | | group by: 3: store_id | | | | | 0:OlapScanNode | | TABLE: sales_records | | PREAGGREGATION: ON | | partitions=1/1 | | rollup: store_amt | | tabletRatio=10/10 | | tabletList=12092,12096,12100,12104,12108,12112,12116,12120,12124,12128 | | cardinality=6 | | avgRowSize=2.0 | | numNodes=0 | +-----------------------------------------------------------------------------+ 45 rows in set (0.00 sec) ``` It can be observed that the value of `rollup` section in the query profile is now `store_amt`, which is the synchronous materialized view you have built. That means this query has hit the synchronous materialized view. #### Show synchronous materialized views[​](#show-synchronous-materialized-views "Direct link to Show synchronous materialized views") You can execute DESC \ ALL to check the schema of a table and its subordinate synchronous materialized views. ```plain MySQL > DESC sales_records ALL; +---------------+---------------+-----------+--------+------+-------+---------+-------+ | IndexName | IndexKeysType | Field | Type | Null | Key | Default | Extra | +---------------+---------------+-----------+--------+------+-------+---------+-------+ | sales_records | DUP_KEYS | record_id | INT | Yes | true | NULL | | | | | seller_id | INT | Yes | true | NULL | | | | | store_id | INT | Yes | true | NULL | | | | | sale_date | DATE | Yes | false | NULL | NONE | | | | sale_amt | BIGINT | Yes | false | NULL | NONE | | | | | | | | | | | store_amt | AGG_KEYS | store_id | INT | Yes | true | NULL | | | | | sale_amt | BIGINT | Yes | false | NULL | SUM | +---------------+---------------+-----------+--------+------+-------+---------+-------+ 8 rows in set (0.00 sec) ``` #### Drop a synchronous materialized view[​](#drop-a-synchronous-materialized-view "Direct link to Drop a synchronous materialized view") Under the following circumstances, you need to drop the synchronous materialized view: * You have created a wrong materialized view and you need to drop it before the building completed. * You have created too many materialized views, which results in a huge drop in load performance, and some of the materialized views are duplicate. * The frequency of the involved queries is low, and you can tolerate a relatively high query latency. ##### Drop an unfinished synchronous materialized view[​](#drop-an-unfinished-synchronous-materialized-view "Direct link to Drop an unfinished synchronous materialized view") You can drop a synchronous materialized view that is being created by canceling the in-progress creation task. First, you need to get the job ID `JobID` of the materialized view creation task by [checking the building status of the materialized view](#check-the-building-status-of-a-synchronous-materialized-view). After getting the job ID, you need to cancel the creation task with the CANCEL ALTER command. ```plain CANCEL ALTER TABLE ROLLUP FROM sales_records (12090); ``` ##### Drop an existing synchronous materialized view[​](#drop-an-existing-synchronous-materialized-view "Direct link to Drop an existing synchronous materialized view") You can drop an existing synchronous materialized view with the [DROP MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/DROP_MATERIALIZED_VIEW.md) command. ```sql DROP MATERIALIZED VIEW store_amt; ``` #### Best practices[​](#best-practices "Direct link to Best practices") ##### Exact count distinct[​](#exact-count-distinct "Direct link to Exact count distinct") The following example is based on an advertisement business analysis table `advertiser_view_record`, which records the date that the ad is viewed `click_time`, the name of the ad `advertiser`, the channel of the ad `channel`, and the ID of the user who viewed the ID `user_id`. ```sql CREATE TABLE advertiser_view_record( click_time DATE, advertiser VARCHAR(10), channel VARCHAR(10), user_id INT ) distributed BY hash(click_time); ``` Analysis is mainly focused on the UV of the ads. ```sql SELECT advertiser, channel, count(distinct user_id) FROM advertiser_view_record GROUP BY advertiser, channel; ``` To accelerate exact count distinct, you can create a synchronous materialized view based on this table and use the bitmap\_union function to pre-aggregate the data. ```sql CREATE MATERIALIZED VIEW advertiser_uv AS SELECT advertiser, channel, bitmap_union(to_bitmap(user_id)) FROM advertiser_view_record GROUP BY advertiser, channel; ``` After the synchronous materialized view is created, the sub-query `count(distinct user_id)` in the subsequent queries will be automatically rewritten as `bitmap_union_count (to_bitmap(user_id))` so that they can hit the synchronous materialized view. ##### Approximate count distinct[​](#approximate-count-distinct "Direct link to Approximate count distinct") Use the table `advertiser_view_record` above as an example again. To accelerate approximate count distinct, you can create a synchronous materialized view based on this table and use the [hll\_union()](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/hll_union.md) function to pre-aggregate the data. ```sql CREATE MATERIALIZED VIEW advertiser_uv2 AS SELECT advertiser, channel, hll_union(hll_hash(user_id)) FROM advertiser_view_record GROUP BY advertiser, channel; ``` ##### Set extra sort keys[​](#set-extra-sort-keys "Direct link to Set extra sort keys") Suppose that the base table `tableA` contains columns `k1`, `k2` and `k3`, where only `k1` and `k2` are sort keys. If the query including the sub-query `where k3=x` must be accelerated, you can create a synchronous materialized view with `k3` as the first column. ```sql CREATE MATERIALIZED VIEW k3_as_key AS SELECT k3, k2, k1 FROM tableA ``` #### Correspondence of aggregate functions[​](#correspondence-of-aggregate-functions "Direct link to Correspondence of aggregate functions") When a query is executed with a synchronous materialized view, the original query statement will be automatically rewritten and used to query the intermediate results stored in the synchronous materialized view. The following table shows the correspondence between the aggregate function in the original query and the aggregate function used to construct the synchronous materialized view. You can select the corresponding aggregate function to build a synchronous materialized view according to your business scenario. | **aggregate function in the original query** | **aggregate function of the materialized view** | | ------------------------------------------------------------ | ----------------------------------------------- | | sum | sum | | min | min | | max | max | | count | count | | bitmap\_union, bitmap\_union\_count, count(distinct) | bitmap\_union | | hll\_raw\_agg, hll\_union\_agg, ndv, approx\_count\_distinct | hll\_union | --- ### Accelerate COUNT(DISTINCT) and Joins with AUTO INCREMENT and Global Dictionary This topic describes how to accelerate COUNT(DISTINCT) calculation and Joins using AUTO INCREMENT columns and Global Dictionary. #### Use Cases[​](#use-cases "Direct link to Use Cases") * **Scenario one**: Suppose you need to perform exact deduplication on massive data (such as retail or delivery orders). However, the column for deduplication is of STRING type, which may lead to suboptimal performance during counting. For instance, in the `orders` table, the `order_uuid` column representing order ID is of STRING type, typically 32 to 36 bytes in size, generated by `UUID()` or similar functions. In this case, COUNT(DISTINCT) on the STRING column `order_uuid` with a query like `SELECT count(DISTINCT order_uuid) FROM orders WHERE create_date >= CURDATE();` might not deliver satisfactory performance. Using an INTEGER column for exact deduplication would significantly enhance performance. * **Scenario two**: Suppose you want to [accelerate exact deduplication in multi-dimensional analysis using bitmap functions](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_bitmap.md). The `bitmap_count()` function requires INTEGER input, but if the column to be deduplicated is of STRING type, you need to use the `bitmap_hash()` function. This may result in approximate, slightly lower deduplication counts and could also reduce query performance and increase storage requirements, as the INTEGER values generated by bitmap\_hash() are more dispersed compared to sequentially assigned INTEGER values. * **Scenario three**: Suppose you need to query the number of orders with a short time span between placing and paying for the order, where the order placement and payment times might be stored in different tables maintained by different business teams. You might need to join these tables based on the order ID and then deduplicate the orders. For example: ```sql SELECT count(distinct order_uuid) FROM orders_t1 as t1 JOIN orders_t2 as t2 ON t1.order_uuid = t2.order_uuid WHERE t2.payment_time - t1.create_time <= 3600 AND create_date >= CURDATE(); ``` However, using a STRING type `order_uuid` column for joins is less efficient than using an INTEGER column. #### Optimization Approach[​](#optimization-approach "Direct link to Optimization Approach") To address the the issues in the above scenarios, the optimization approach involves loading order data into a target table and establishing a mapping between STRING and INTEGER values. Subsequent query analysis will be based on the INTEGER column. This approach can be divided into the following stages: 1. Stage 1: Create a global dictionary and establish a mapping between STRING and INTEGER values. In this dictionary, the key column is of STRING type, and the value column is an AUTO INCREMENT INTEGER type. When data is loaded, the system automatically generates a unique ID for each STRING value, creating a mapping between STRING and INTEGER values. 2. Stage 2: Load the mapping relationship between order data and the global dictionary into the target table. 3. Stage 3: Use the INTEGER column from the target table for exact deduplication or joins during subsequent query analysis, which can significantly improve performance. 4. Stage 4: For further performance optimization, you can use bitmap functions on the INTEGER column to accelerate exact deduplication. #### Solution[​](#solution "Direct link to Solution") Before v3.2.5, Stage 2 could be implemented through two methods: * Using an external table or an internal table as intermediate table to join with the dictionary table to obtain the corresponding dictionary ID before loading. * Using a Primary Key table for data loading, and then using an UPDATE statement with a JOIN operation to update the dictionary ID. However, this data loading process can be inconvenient and comes with many constraints. From v3.2.5 onwards, StarRocks introduced the `dict_mapping()` function, allowing you to define the dictionary ID column in the target table as a generated column using the `dict_mapping()` expression. Subsequent data loading tasks are handled like regular data loading, without needing UPDATE statements with JOIN operations to write dictionary IDs. During data loading, the system automatically associates the original table with the dictionary table and inserts the corresponding dictionary ID, greatly simplifying the data loading process with a global dictionary table, regardless of table types and supporting various loading methods. ##### Business Scenario[​](#business-scenario "Direct link to Business Scenario") The following example uses two example CSV files, `batch1.csv` and `batch2.csv`, each containing two columns: `id` and `order_uuid`. * `batch1.csv` ```csv 1, a1 2, a2 3, a3 11, a1 11, a2 12, a1 ``` * `batch2.csv` ```csv 1, a2 2, a2 3, a2 11, a2 12, a101 12, a102 13, a102 ``` ##### Process[​](#process "Direct link to Process") ###### Stage one[​](#stage-one "Direct link to Stage one") Create a global dictionary table and load the order ID column values from the CSV files to establish a mapping between STRING and INTEGER values. 1. Create a Primary Key table to serve as the global dictionary. Define the Primary Key, `order_uuid` (of STRING type), and the value column, `order_id_int` (of AUTO INCREMENT INTEGER type). info The `dict_mapping` function requires the global dictionary table to be a Primary Key table. ```sql CREATE TABLE dict ( order_uuid STRING, order_id_int BIGINT AUTO_INCREMENT -- Automatically assign an ID to each order_uuid value. ) PRIMARY KEY (order_uuid) DISTRIBUTED BY HASH (order_uuid) PROPERTIES("replicated_storage" = "true"); ``` 2. Use Stream Load to batch load the `order_uuid` column from the two CSV files into the `order_uuid` column of the dictionary table `dict`. Make sure you have used the Partial Update in column mode. ```bash curl --location-trusted -u root: \ -H "partial_update: true" \ -H "format: CSV" -H "column_separator:," -H "columns: id, order_uuid" \ -T batch1.csv \ -XPUT http://:/api/example_db/dict/_stream_load curl --location-trusted -u root: \ -H "partial_update: true" \ -H "format: CSV" -H "column_separator:," -H "columns: id, order_uuid" \ -T batch2.csv \ -XPUT http://:/api/example_db/dict/_stream_load ``` > **NOTE** > > If new data is added to the data source before proceeding to the next stage, all new data must be loaded into the dictionary table to ensure the mapping exists. ###### Stage two[​](#stage-two "Direct link to Stage two") Create a target table that includes a dictionary ID column with the `dict_mapping` attribute. When order data is loaded into the target table, the system will automatically associate it with the dictionary table and insert the corresponding dictionary ID. 1. Create a table `dest_table` that includes all columns from the CSV files. You also need to define an INTEGER column `order_id_int` (typically BIGINT) to map with the STRING type `order_uuid` column and has the `dict_mapping` column attribute. Future query analysis will be based on this `order_id_int` column. ```sql -- In the target table, define a BIGINT dict_mapping column `order_id_int` to map with the STRING-type column `order_uuid`. CREATE TABLE dest_table ( id BIGINT, order_uuid STRING, -- This column records STRING-type Order ID. batch INT comment 'Used to distinguish different batch loading', order_id_int BIGINT AS dict_mapping('dict', order_uuid) -- Dictionary ID dict_mapping column corresponds to `order_uuid`. ) DUPLICATE KEY (id, order_uuid) DISTRIBUTED BY HASH(id); ``` 2. Load data into the target table with Stream Load or any other available methods. Since the `order_id_int` column has the `dict_mapping` attribute, the system will automatically fetch the dictionary ID from `dict` during loading. ```bash curl --location-trusted -u root: \ -H "format: CSV" -H "column_separator:," -H "columns: id, order_uuid, batch=1" \ -T batch1.csv \ -XPUT http://:/api/example_db/dest_table/_stream_load curl --location-trusted -u root: \ -H "format: CSV" -H "column_separator:," -H "columns: id, order_uuid, batch=2" \ -T batch2.csv \ -XPUT http://:/api/example_db/dest_table/_stream_load ``` ###### Stage three[​](#stage-three "Direct link to Stage three") During query analysis, you can perform exact deduplication or joins on the INTEGER column `order_id_int`, significantly enhancing performance compared to using the STRING column `order_uuid`. ```sql -- Exact deduplication based on BIGINT-type order_id_int. SELECT id, COUNT(DISTINCT order_id_int) FROM dest_table GROUP BY id ORDER BY id; -- Exact deduplication based on STRING-type order_uuid . SELECT id, COUNT(DISTINCT order_uuid) FROM dest_table GROUP BY id ORDER BY id; ``` You can also [use bitmap functions to accelerate exact deduplication](#use-bitmap-functions-to-accelerate-exact-deduplication). ##### Use bitmap functions to accelerate exact deduplication[​](#use-bitmap-functions-to-accelerate-exact-deduplication "Direct link to Use bitmap functions to accelerate exact deduplication") To further accelerate the calculation, you can insert the INTEGER column values of the dictionary table directly into a bitmap column after creating a global dictionary. Subsequently, you can use bitmap functions on this bitmap column for exact deduplication. ###### Method one[​](#method-one "Direct link to Method one") If you have built a global dictionary and already imported the order data into `dest_table`, follow these steps: 1. Create an Aggregate table `dest_table_bitmap` with two columns: a BITMAP type `order_id_bitmap` column for aggregation using the `bitmap_union()` function, and an INTEGER type column `id`. This table does not include the original STRING column, otherwise, each bitmap would contain only one value, negating the acceleration benefit. ```sql CREATE TABLE dest_table_bitmap ( id BIGINT, order_id_bitmap BITMAP BITMAP_UNION ) AGGREGATE KEY (id) DISTRIBUTED BY HASH(id) BUCKETS 6; ``` 2. Insert data into `dest_table_bitmap`. Insert the data from the `id` column of `dest_table` into the `id` column, and insert the INTEGER column `order_id_int` data from the dictionary table `dict` (processed with the `to_bitmap()` function) into the `order_id_bitmap` column. ```sql INSERT INTO dest_table_bitmap (id, order_id_bitmap) SELECT id, to_bitmap(dict_mapping('dict', order_uuid)) FROM dest_table WHERE dest_table.batch = 1; -- Indicates different batches. INSERT INTO dest_table_bitmap (id, order_id_bitmap) SELECT id, to_bitmap(dict_mapping('dict', order_uuid)) FROM dest_table WHERE dest_table.batch = 2; ``` 3. Use the `BITMAP_UNION_COUNT()` function on the BITMAP column for exact deduplication. ```sql SELECT id, BITMAP_UNION_COUNT(order_id_bitmap) FROM dest_table_bitmap GROUP BY id ORDER BY id; ``` ###### Method two[​](#method-two "Direct link to Method two") If you do not need to retain specific order data after creating a global dictionary, and you want to load data directly into the `dest_table_bitmap` table, follow these steps: 1. Create an Aggregate table `dest_table_bitmap` with two columns: a BITMAP type `order_id_bitmap` column for aggregation using the `bitmap_union()` function, and an INTEGER type column `id`. This table does not include the original STRING column, otherwise, each bitmap would contain only one value, negating the acceleration benefit. ```sql CREATE TABLE dest_table_bitmap ( id BIGINT, order_id_bitmap BITMAP BITMAP_UNION ) AGGREGATE KEY (id) DISTRIBUTED BY HASH(id) BUCKETS 6; ``` 2. Insert data into the Aggregate table. Insert the data from the `id` column of the CSV files into the `id` column and the INTEGER column `order_id_int` data from the dictionary table `dict` (processed with the `to_bitmap()` function) into the `order_id_bitmap` column. ```bash curl --location-trusted -u root: \ -H "format: CSV" -H "column_separator:," \ -H "columns: id, order_uuid, order_id_bitmap=to_bitmap(dict_mapping('dict', order_uuid))" \ -T batch1.csv \ -XPUT http://:/api/example_db/dest_table_bitmap/_stream_load curl --location-trusted -u root: \ -H "format: CSV" -H "column_separator:," \ -H "columns: id, order_uuid, order_id_bitmap=to_bitmap(dict_mapping('dict', order_uuid))" \ -T batch2.csv \ -XPUT http:///:/api/example_db/dest_table_bitmap/_stream_load ``` 3. Use the `BITMAP_UNION_COUNT()` function on the BITMAP column for exact deduplication. ```sql SELECT id, BITMAP_UNION_COUNT(order_id_bitmap) FROM dest_table_bitmap GROUP BY id ORDER BY id; ``` --- ### Query Feedback Experimental feature [Advice on use of experimental features](https://docs.starrocks.io/docs/maturity/features.md) This topic introduces the Query Feedback feature, its application scenarios, and how to optimize query plans based on feedback from execution statistics using Query Plan Advisor. StarRocks supports the Query Feedback feature from v3.4.0 onwards. #### Overview[​](#overview "Direct link to Overview") Query Feedback is a framework and a critical component of Cost-based Optimizer (CBO). It records the execution statistics during query execution, and reuses it in subsequent queries with similar query plans to help CBO generate optimized query plans. CBO optimizes query plans based on estimated statistics, so when statistical information is outdated or inaccurate, it may select inefficient query plans (bad plans), such as broadcasting a large table or misordering the left and right tables. These bad plans can cause query execution timeouts, excessive resource consumption, or even system crashes. #### Workflow[​](#workflow "Direct link to Workflow") The workflow of Query Feedback-based plan optimization consists of three stages: 1. **Observation**: BE or CN records the major metrics (including `InputRows` and `OutputRows`) of PlanNode in each query plan. 2. **Analysis**: For slow queries that exceed the configured threshold and queries that are manually marked to be analyzed, the system will analyze execution details at critical nodes to identify optimization opportunities in the current query plan after the query finishes and before the result is returned. FE compares the query plan with the execution statistics, and checks if the query is a slow query caused by an abnormal query plan. While FE analyzes the inaccurate statistics, it will generate a SQL tuning guide for each query, instruct CBO to optimize the query dynamically, and recommend strategies to improve the performance. 3. **Optimization**: After CBO generates a physical plan, it will search for any existing tuning guide that applies to the plan. If there is one, CBO will dynamically optimize the plan according to the guide and strategies, correcting problematic sections, and thereby eliminating the impact on the query performance due to the repetitive usage of bad query plans. The execution time of the optimized plan is compared to that of the original plan to evaluate the effectiveness of the tuning. #### Usage[​](#usage "Direct link to Usage") Controlled by system variable `enable_plan_advisor` (Default: `true`), Query Plan Advisor is enabled by default for slow queries, that is, queries with execution time exceeding the threshold defined in the FE configuration item `slow_query_analyze_threshold` (Default: `5` seconds). In addition, you can manually analyze a specific query or enable automatic analysis for all queries executed. ##### Manually analyze a specific query[​](#manually-analyze-a-specific-query "Direct link to Manually analyze a specific query") You can manually analyze a specific query statement even if its execution time does not exceed `slow_query_analyze_threshold`. ```sql ALTER PLAN ADVISOR ADD ``` Example: ```sql ALTER PLAN ADVISOR ADD SELECT COUNT(*) FROM ( SLECT * FROM c1_skew_left_over t1 JOIN (SELECT * FROM c1_skew_left_over WHERE c1 = 'c') t2 ON t1.c2 = t2.c2 WHERE t1.c1 > 'c' ) t; ``` ##### Enable automatic analysis for all queries[​](#enable-automatic-analysis-for-all-queries "Direct link to Enable automatic analysis for all queries") To enable automatic analysis for all queries, you need to set the system variable `enable_plan_analyzer` (Default: `false) to `true\`. ```sql SET enable_plan_analyzer = true; ``` ##### Show the tuning guides on the current FE[​](#show-the-tuning-guides-on-the-current-fe "Direct link to Show the tuning guides on the current FE") Each FE maintains its own record of tuning guides. You can use the following statement to view the tuning guides generated for the entitled queries on the current FE: ```sql SHOW PLAN ADVISOR ``` ##### Check if Tuning Guide takes effect[​](#check-if-tuning-guide-takes-effect "Direct link to Check if Tuning Guide takes effect") Execute [EXPLAIN](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/EXPLAIN.md) against the query statement. In the EXPLAIN string, the message `Plan had been tuned by Plan Advisor` indicates that a tuning guide has been applied to the corresponding query. Example: ```sql EXPLAIN SELECT COUNT(*) FROM ( SLECT * FROM c1_skew_left_over t1 JOIN (SELECT * FROM c1_skew_left_over WHERE c1 = 'c') t2 ON t1.c2 = t2.c2 WHERE t1.c1 > 'c' ) t; +-----------------------------------------------------------------------------------------------+ | Explain String | +-----------------------------------------------------------------------------------------------+ | Plan had been tuned by Plan Advisor. | | Original query id:8e010cf4-b178-11ef-8aa4-8a5075cec65e | | Original time cost: 148 ms | | 1: LeftChildEstimationErrorTuningGuide | | Reason: left child statistics of JoinNode 5 had been overestimated. | | Advice: Adjust the distribution join execution type and join plan to improve the performance. | | | | PLAN FRAGMENT 0 | | OUTPUT EXPRS:9: count | | PARTITION: UNPARTITIONED ``` ##### Delete the tuning guide of a specific query[​](#delete-the-tuning-guide-of-a-specific-query "Direct link to Delete the tuning guide of a specific query") You can delete the tuning guide of a specific query based on the query ID returned from `SHOW PLAN ADVISOR`. ```sql ALTER PLAN ADVISOR DROP ``` Example: ```sql ALTER PLAN ADVISOR DROP "8e010cf4-b178-11ef-8aa4-8a5075cec65e"; ``` ##### Clear all tuning guides on the current FE[​](#clear-all-tuning-guides-on-the-current-fe "Direct link to Clear all tuning guides on the current FE") To clear all tuning guides on the current FE, execute the following statement: ```sql TRUNCATE PLAN ADVISOR ``` #### Use Cases[​](#use-cases "Direct link to Use Cases") Currently, Query Feedback is primarily used to optimize the following scenarios: * Optimizing the order of left and right sides in local Join nodes * Optimizing the execution method of local Join nodes (e.g., switching from Broadcast to Shuffle) * For cases with significant aggregation potential, enforcing `pre_aggregation` mode to maximize data aggregation in the first phase The tuning guides are mainly based on metrics `Runtime Exec Node Input/Output Rows` and FE `statistics estimated rows`. Since the current tuning thresholds are relatively conservative, you are encouraged to leverage Query Feedback to check for potential performance improvements if issues are observed in the Query Profile or EXPLAIN string. Below are three common user cases. ##### Case 1: Incorrect Join Order[​](#case-1-incorrect-join-order "Direct link to Case 1: Incorrect Join Order") Original Bad Plan: ```sql small left table inner join large table (broadcast) ``` Optimized Plan: ```sql large left table inner join small right table (broadcast) ``` **Cause** The issue might be caused by outdated or missing statistics, which leads the Cost-Based Optimizer (CBO) to generate an incorrect plan based on inaccurate data. **Solution** During query execution, the system compares the `input/output rows` and `statistics estimated rows` of Left Child and Right Child, generating tuning guides. Upon re-execution, the system automatically adjusts the Join order. ##### Case 2: Incorrect Join Execution Method[​](#case-2-incorrect-join-execution-method "Direct link to Case 2: Incorrect Join Execution Method") Original Bad Plan: ```sql large left table1 inner join large right table2 (broadcast) ``` Optimized Plan: ```sql large left table1 (shuffle) inner join large right table2 (shuffle) ``` **Cause** The issue may result from data skew. When the right table has many partitions and one of them contains disproportionately large amounts of data, the system may incorrectly estimate the row count of right table after predicates are applied. **Solution** During query execution, the system compares the `input/output rows` and `statistics estimated rows` of Left Child and Right Child, generating tuning guides. After optimization, the Join method is adjusted from Broadcast Join to Shuffle Join. ##### Case 3: Inefficient First-Phase Pre-aggregation Mode[​](#case-3-inefficient-first-phase-pre-aggregation-mode "Direct link to Case 3: Inefficient First-Phase Pre-aggregation Mode") **Symptom** For data with good aggregation potential, the `auto` mode of First-phase aggregation may only aggregate a small amount of local data, missing the opportunity for performance gains. **Solution** During query execution, the system collects `Input/Output Rows` for both local and global aggregations. Based on historical data, it evaluates the potential of the aggregation columns. If the potential is significant, the system enforces the use of `pre_aggregation` mode in local aggregations, maximizing data aggregation in the first phase and improving overall query performance. #### Limitations[​](#limitations "Direct link to Limitations") * A tuning guide can only be used for the exact same query for which it was generated. It is not applicable to queries with the same pattern but different parameters. * Each FE manages its Query Plan Advisor independently, and synchronization across FE nodes is not supported. If the same query is submitted to different FE nodes, the tuning results may vary. * Query Plan Advisor uses an in-memory cache structure: * When the number of tuning guides exceeds the limit, expired tuning guides are automatically evicted. * The default limit of tuning guides is 300 and the persistence of historical tuning guides is not supported. --- ### View Running Queries This article describes how to view running queries in StarRocks and analyze their resource consumption. #### current\_queries[​](#current_queries "Direct link to current_queries") `show proc '/current_queries'` allows you to view the queries running on the current FE node, including the following information: | **Column Name** | **Description** | | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | StartTime | Query start time | | feIp | IP address of the FE node executing the query | | QueryId | Query ID | | ConnectionId | Connection ID | | Database | Database where the query is running | | User | User executing the query | | ScanBytes | Amount of data scanned | | ScanRows | Number of rows scanned | | MemoryUsage | Amount of memory used by the query | | DiskSpillSize | Amount of data spilled to disk | | CPUTime | CPU time consumed by the query | | ExecTime | Query execution time | | Warehouse | Warehouse used by the query | | CustomQueryId | User-defined query ID | | ResourceGroup | Resource group used by the query | | QueryType | Type of the query: `Query` (user query), `Statistics` (statistics collection, such as ANALYZE), `Task` (task run), `MV` (materialized view refresh), or `Internal` (other internal query) | **Example**: ```sql MySQL > show proc '/current_queries'\G ***************************[ 1. row ]*************************** StartTime | 2025-03-07 02:16:04 feIp | 172.26.92.227 QueryId | 10db481c-fab7-11ef-8063-461f20abc3f0 ConnectionId | 13 Database | tpcds_2 User | root ScanBytes | 120.573 MB ScanRows | 5859503 rows MemoryUsage | 225.893 MB DiskSpillSize | 0.000 B CPUTime | 47.878 s ExecTime | 4.077 s Warehouse | default_warehouse CustomQueryId | ResourceGroup | rg1 QueryType | Query ``` #### global\_current\_queries[​](#global_current_queries "Direct link to global_current_queries") Similar to `current_queries`, `show proc '/global_current_queries'` shows information about queries running on all FE nodes. This command is supported since version 3.4. **Example**: ```sql MySQL root@127.1:(none)> show proc '/global_current_queries'\G ***************************[ 1. row ]*************************** StartTime | 2025-03-07 02:21:48 feIp | 172.26.92.227 QueryId | de516505-fab7-11ef-8063-461f20abc3f0 ConnectionId | 14 Database | tpcds_2 User | root ScanBytes | 120.573 MB ScanRows | 5859503 rows MemoryUsage | 346.915 MB DiskSpillSize | 0.000 B CPUTime | 33.265 s ExecTime | 3.032 s Warehouse | default_warehouse CustomQueryId | ResourceGroup | rg1 QueryType | Query ``` #### running queries[​](#running-queries "Direct link to running queries") `SHOW RUNNING QUERIES` is mainly used to view the Query Queue status. If a query is in the queue, it will be in a `PENDING` state. | Field Name | Description | | --------------- | ------------------------------------------ | | QueryId | Query ID | | ResourceGroupId | Resource group ID | | StartTime | Query start time | | PendingTimeout | Waiting timeout | | QueryTimeout | Query timeout | | State | Query state, including PENDING and RUNNING | | Slots | Number of slots occupied | | Fragments | Number of fragments in the query plan | | DOP | Degree of parallelism | | Frontend | FE node information | | FeStartTime | FE start time | **Example**: ```sql MySQL root@127.1:(none)> show running queries\G ***************************[ 1. row ]*************************** QueryId | 50029ec1-fab8-11ef-8063-461f20abc3f0 ResourceGroupId | 562275 StartTime | 2025-03-07 02:24:59 PendingTimeout | 2025-03-07 02:27:29 QueryTimeout | 2025-03-07 02:27:29 State | RUNNING Slots | 1 Fragments | 11 DOP | 0 Frontend | 172.26.92.227_8034_1709578860161 FeStartTime | 2025-03-06 23:39:00 ``` --- ### Skew Join V2 Skew Join V2 is an advanced optimization feature in StarRocks that addresses data skew issues in JOIN operations by broadcasting skew values. This feature significantly improves query performance when dealing with heavily skewed data distributions. #### Overview[​](#overview "Direct link to Overview") Data skew occurs when certain values in join columns appear much more frequently than others, leading to uneven data distribution across nodes and causing performance bottlenecks. Skew Join V2 solves this problem by: 1. **Identifying skew values**: Manually specifying values that cause data skew 2. **Broadcasting skew values**: Broadcasting these specific values to all nodes to ensure even data distribution 3. **Hybrid execution**: Using a combination of shuffle and broadcast joins for optimal performance Skew Join V2 creates a hybrid execution plan that combines: 1. **Shuffle Join**: For non-skew data, uses the standard shuffle-based join 2. **Broadcast Join**: For skew values, broadcasts the right table data to all nodes #### Enabling Skew Join V2[​](#enabling-skew-join-v2 "Direct link to Enabling Skew Join V2") By default, the Skew Join V2 optimization is disabled, and its predecessor, Skew Join V1, is enabled. If you find that the performance of Skew Join V1 is not satisfactory, you must disable Skew Join V1 before enabling Skew Join V2. ```sql -- Disable Skew Join V1 SET enable_optimize_skew_join_v1 = false; -- Enable Skew Join V2 SET enable_optimize_skew_join_v2 = true; ``` After enabling Skew Join V2, you can then update your queries based on the [syntax](#usage) with explicitly specified skew values. note Currently, Skew Join V2 does not support automatic plan rewrite based on statistics. Only hint-based manual SQL rewrite is supported. #### Usage[​](#usage "Direct link to Usage") Compared to the query rewrite method of Skew Join V1, Skew Join V2 offers a new syntax that allows you to explicitly specify the skew values with Broadcast. Syntax: ```sql SELECT select_list FROM table1 JOIN [skew|table1.column(skew_value1, skew_value2, ...)] table2 ON join_condition [WHERE where_clause] ``` Parameters: `[skew|table1.column(skew_value1, skew_value2, ...)]`: The skew hint. It includes: * `table1.column`: The column from the left table that contains skew values. * `skew_value1, skew_value2, ...`: A comma-separated list of values that cause data skew. note Do not omit the brackets in the skew hint. #### Examples[​](#examples "Direct link to Examples") ##### Basic Usage[​](#basic-usage "Direct link to Basic Usage") 1. Create test tables. ```sql CREATE TABLE orders ( order_id INT, customer_id INT, order_date DATE, amount DECIMAL(10,2) ) DUPLICATE KEY(order_id) DISTRIBUTED BY HASH(order_id) BUCKETS 8; CREATE TABLE customers ( customer_id INT, customer_name VARCHAR(100), city VARCHAR(50) ) DUPLICATE KEY(customer_id) DISTRIBUTED BY HASH(customer_id) BUCKETS 8; ``` 2. Insert sample data with skew. ```sql INSERT INTO orders VALUES (1, 1001, '2024-01-01', 100.00), (2, 1001, '2024-01-02', 200.00), (3, 1001, '2024-01-03', 150.00), (4, 1002, '2024-01-01', 300.00), (5, 1003, '2024-01-01', 250.00); INSERT INTO customers VALUES (1001, 'John Doe', 'New York'), (1002, 'Jane Smith', 'Los Angeles'), (1003, 'Bob Johnson', 'Chicago'); ``` 3. Query data with Skew Join V2. ```sql SELECT o.order_id, c.customer_name, o.amount FROM orders o JOIN [skew|o.customer_id(1001)] customers c ON o.customer_id = c.customer_id; ``` ##### Multiple Skew Values[​](#multiple-skew-values "Direct link to Multiple Skew Values") Skew Join V2 supports multiple skew values. ```sql SELECT o.order_id, c.customer_name, o.amount FROM orders o JOIN [skew|o.customer_id(1001, 1002, 1003)] c ON o.customer_id = c.customer_id; ``` ##### Different Data Types[​](#different-data-types "Direct link to Different Data Types") Skew Join V2 supports various data types for skew values. ```sql -- String values SELECT t1.id, t2.name FROM table1 t1 JOIN [skew|t1.category('electronics', 'clothing', 'books')] t2 ON t1.category = t2.category; -- Date values SELECT t1.id, t2.event_name FROM events t1 JOIN [skew|t1.event_date('2024-01-01', '2024-01-02')] t2 ON t1.event_date = t2.event_date; -- Numeric values SELECT t1.id, t2.region FROM sales t1 JOIN [skew|t1.region_id(1, 2, 3)] t2 ON t1.region_id = t2.region_id; ``` ##### Complex Join Conditions[​](#complex-join-conditions "Direct link to Complex Join Conditions") Skew Join V2 supports complex Join conditions. ```sql -- Join condition with complex expressions SELECT t1.id, t2.value FROM table1 t1 JOIN [skew|t1.key(abs(t1.key))] t2 ON abs(t1.key) = abs(t2.key); -- Multiple Join conditions SELECT t1.id, t2.name FROM table1 t1 JOIN [skew|t1.category('electronics')] t2 ON t1.category = t2.category AND t1.region = t2.region; ``` ##### Different Join Types[​](#different-join-types "Direct link to Different Join Types") Skew Join V2 supports various Join types. ```sql -- Left Join SELECT t1.id, t2.name FROM table1 t1 LEFT JOIN [skew|t1.category('electronics')] t2 ON t1.category = t2.category; -- Left Semi Join SELECT t1.id FROM table1 t1 LEFT SEMI JOIN [skew|t1.category('electronics')] t2 ON t1.category = t2.category; -- Left Anti Join SELECT t1.id FROM table1 t1 LEFT ANTI JOIN [skew|t1.category('electronics')] t2 ON t1.category = t2.category; ``` #### Best Practices[​](#best-practices "Direct link to Best Practices") ##### Step 1. Identify Skew Values[​](#step-1-identify-skew-values "Direct link to Step 1. Identify Skew Values") Before using Skew Join V2, identify the values that cause data skew: ```sql -- Analyze data distribution SELECT customer_id, COUNT(*) as cnt FROM orders GROUP BY customer_id ORDER BY cnt DESC LIMIT 10; ``` ##### Step 2. Choose Appropriate Skew Values[​](#step-2-choose-appropriate-skew-values "Direct link to Step 2. Choose Appropriate Skew Values") Follow these rules to choose the appropriate skew values: * Include only the most frequent values that cause significant skew. * Avoid including too many values as it may increase broadcast overhead. * Consider the trade-off between skew optimization and broadcast cost. #### Limitations[​](#limitations "Direct link to Limitations") * **Supported Join Types** Currently, only INNER JOIN, LEFT JOIN, LEFT SEMI JOIN, and LEFT ANTI JOIN are supported. * **Data Types** Currently, only basic data types (INT, BIGINT, STRING, DATE, DATETIME, and string types) are supported. * **Complex Expressions** The support for complex expressions in Join conditions is limited. * **Skew Table** Skew Join V2 can only handle scenarios where the large table in the JOIN operation is skewed, and the large table must be used as the left table. * **Join Reordering** Using the `skew` hint will prevent the optimizer from reordering the Join. The Join will be executed in the order specified in the SQL, and the optimizer will not attempt to change the Join order or swap the left and right tables of the Join node containing the hint. #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") ##### Debug Information[​](#debug-information "Direct link to Debug Information") Execute EXPLAIN VERBOSE against the query to collect its execution plan: ```sql EXPLAIN VERBOSE SELECT ... FROM ... JOIN [skew|...] ...; ``` Check the following fields in the plan: * `SplitCastDataSink`: Data splitting. * `BROADCAST` and `SHUFFLE`: Distribution type. * `UNION`: Union results of both Join types. #### Related Topics[​](#related-topics "Direct link to Related Topics") * [Query Planning](https://docs.starrocks.io/docs/best_practices/query_tuning/query_planning.md) * [Query Profile Tuning](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_tuning_recipes.md) * [System Variables](https://docs.starrocks.io/docs/sql-reference/System_variable.md) * [JOIN Operations](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md#join) --- ### Sorted streaming aggregate Common aggregation methods in database systems include hash aggregate and sort aggregate. From v2.5 onwards, StarRocks supports **sorted streaming aggregate**. #### Working principles[​](#working-principles "Direct link to Working principles") Aggregate nodes (AGG) are mainly responsible for handling GROUP BY and aggregate functions. Sorted streaming aggregate can group data by comparing GROUP BY keys according to the sequence of the keys, without the need to create a hash table. This effectively reduces memory resources consumed by aggregation. For queries with a high aggregation cardinality, sorted streaming aggregate improves aggregation performance and reduces memory usage. You can enable sorted streaming aggregate by setting the following variable: ```sql set enable_sort_aggregate=true; ``` #### Limits[​](#limits "Direct link to Limits") * StarRocks shared-data clusters do not support sorted streaming aggregate. * The keys in GROUP BY must have a sequence. For example, if the sort key is `k1, k2, k3`, then: * `GROUP BY k1` and `GROUP BY k1, k2` are allowed. * `GROUP BY k1, k3` does not follow the sort key sequence. Therefore, sorted streaming aggregate cannot take effect on such a clause. * The selected partition must be a single partition (because the same key may be distributed on different machines in different partitions). * The GROUP BY keys must have the same distribution as the bucket keys specified when you create a table. For example, if a table has three columns `k1, k2, k3`, the bucket key can be `k1` or `k1, k2`. * If the bucket key is `k1`, the `GROUP BY` key can be `k1`, `k1, k2`, or `k1, k2, k3`. * If the bucket key is `k1, k2`, the GROUP BY key can be `k1, k2` or `k1, k2, k3`. * If the query plans do not meet this requirement, the sorted streaming aggregate feature cannot take effect even if this feature is enabled. * Sorted streaming aggregate works only for first-stage aggregation (that is, there is only one Scan node under the AGG node). #### Examples[​](#examples "Direct link to Examples") 1. Create a table and insert data. ```sql CREATE TABLE `test_sorted_streaming_agg_basic` ( `id_int` int(11) NOT NULL COMMENT "", `id_string` varchar(100) NOT NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`id_int`)COMMENT "OLAP" DISTRIBUTED BY HASH(`id_int`) PROPERTIES ("replication_num" = "3"); INSERT INTO test_sorted_streaming_agg_basic VALUES (1, 'v1'), (2, 'v2'), (3, 'v3'), (1, 'v4'); ``` 2. Enable sorted streaming aggregate and use EXPLAIN to query the SQL profile. ```sql set enable_sort_aggregate = true; explain costs select id_int, max(id_string) from test_sorted_streaming_agg_basic group by id_int; ``` #### Check whether sorted streaming aggregate is enabled[​](#check-whether-sorted-streaming-aggregate-is-enabled "Direct link to Check whether sorted streaming aggregate is enabled") View the results of `EXPLAIN costs`. If the `sorted streaming` field is `true` in the AGG node, this feature is enabled. ```plain | | | 1:AGGREGATE (update finalize) | | | aggregate: max[([2: id_string, VARCHAR, false]); args: VARCHAR; result: VARCHAR; args nullable: false; result nullable: true] | | | group by: [1: id_int, INT, false] | | | sorted streaming: true | | | cardinality: 1 | | | column statistics: | | | * id_int-->[-Infinity, Infinity, 0.0, 1.0, 1.0] UNKNOWN | | | * max-->[-Infinity, Infinity, 0.0, 1.0, 1.0] UNKNOWN | | | | | 0:OlapScanNode | | table: test_sorted_streaming_agg_basic, rollup: test_sorted_streaming_agg_basic | | preAggregation: on | | partitionsRatio=1/1, tabletsRatio=10/10 | | tabletList=30672,30674,30676,30678,30680,30682,30684,30686,30688,30690 | | actualRows=0, avgRowSize=2.0 | | cardinality: 1 | | column statistics: | | * id_int-->[-Infinity, Infinity, 0.0, 1.0, 1.0] UNKNOWN | | * id_string-->[-Infinity, Infinity, 0.0, 1.0, 1.0] UNKNOWN | ``` --- ### SQL Plan Manager Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/maturity/features.md) This topic introduces the basic concepts and usage scenarios of the SQL Plan Manager feature, as well as how to use SQL Plan Manager to regularize query plans. Starting from v3.5.0, StarRocks supports the SQL Plan Manager feature. #### Overview[​](#overview "Direct link to Overview") SQL Plan Manager allows users to bind a query plan to a query, thereby preventing the query plan from changing due to system state changes (mainly data updates and statistics updates), thus stabilizing query performance. ##### Workflow[​](#workflow "Direct link to Workflow") SQL Plan Manager requires pre-specifying the bound query SQL and the query plan (Baseline) to be used. Here, the query refers to the actual query SQL executed by the user, while the query plan is the query SQL optimized manually or with hints added. The workflow of SQL Plan Manager is as follows: 1. **Create Baseline**: Use the `CREATE BASELINE` command to bind a query plan to the specified query SQL. 2. **Query Rewrite**: Queries submitted to StarRocks are automatically matched against the Baselines stored in SQL Plan Manager. If they match successfully, the query plan of the Baseline is used on the query. Notes on Baseline creation: * You must ensure the logical consistency between the bound SQL and the execution plan SQL in the Baseline. StarRocks performs basic table and parameter checks but cannot guarantee full logical consistency checks. Ensuring the logical correctness of the execution plan is users' own responsibility. * By default, the bound SQL in the Baseline stores the its own SQL fingerprint. By default, the constant values in the SQL will be replaced with variable parameters (fow example, changing `t1.v1 > 1000` to `t1.v1 > ?`) to improve SQL matching. * The execution plan bound in the Baseline can be customized by modifying the SQL logic or by adding Hints (Join Hints or `Set_Var`) to ensure the desired execution plan is generated. * For complex SQL, StarRocks may not automatically bind the SQL and execution plan in the Baseline. In such cases, manual binding can be used, as detailed in the [Advanced Usage](#advanced-usage) section. Notes on query rewrite: * SQL Plan Manager primarily relies on SQL fingerprint matching. It checks whether the SQL fingerprint of the query matches a that of a Baseline. If a query matches a Baseline, the parameters in the query are automatically substituted into the Baseline’s execution plan. * During the matching process, if a query matches multiple Baselines with status `enable`, the optimizer evaluates and selects the optimal Baseline. * During the matching process, SQL Plan Manager validates whether the Baseline and the query match. If the match fails, the Baseline’s query plan will not be used. * For execution plans rewritten by SQL Plan Manager, the `EXPLAIN` statement will return `Using baseline plan[id]`. #### Manage Baseline[​](#manage-baseline "Direct link to Manage Baseline") ##### Create Baseline[​](#create-baseline "Direct link to Create Baseline") **Syntax**: ```sql CREATE [GLOBAL] BASELINE [ON ] USING [PROPERTIES ("key" = "value"[, ...])] ``` **Parameters**: * `GLOBAL`: (Optional) Creates a global-level Baseline. * `BindSQL`: (Optional) The specific query to be bound to the Baseline (execution plan) query. If this parameter is not specified, the Baseline query is bound to itself and use its own query plan. * `PlanSQL`: The query used to define the execution plan. **Examples**: ```sql -- Create a session-level BASELINE, directly binding the Baseline SQL to itself and using its own query plan. CREATE BASELINE USING SELECT t1.v2, t2.v3 FROM t1 JOIN t2 ON t1.v2 = t2.v2 WHERE t1.v2 > 100; -- Create a global-level BASELINE, directly binding the Baseline SQL to itself and using its own query plan with specified Join Hints. CREATE GLOBAL BASELINE USING SELECT t1.v2, t2.v3 FROM t1 JOIN[BROADCAST] t2 ON t1.v2 = t2.v2 WHERE t1.v2 > 100; -- Create a session-level BASELINE, binding a query to the Baseline SQL and using the Baseline SQL query plan with specified Join Hints. CREATE BASELINE ON SELECT t1.v2, t2.v3 FROM t1, t2 WHERE t1.v2 = t2.v2 AND t1.v2 > 100 USING SELECT t1.v2, t2.v3 FROM t1 JOIN[BROADCAST] t2 on t1.v2 = t2.v2 where t1.v2 > 100; ``` ##### Show Baseline[​](#show-baseline "Direct link to Show Baseline") **Syntax**: ```sql SHOW BASELINE [WHERE ] SHOW BASELINE [ON ] ``` **Example**: ```plain MySQL > show baseline\G; ***************************[ 1. row ]*************************** Id | 646125 global | N enable | N bindSQLDigest | SELECT * FROM `td`.`t2` INNER JOIN `td`.`t1` ON `td`.`t2`.`v2` = `td`.`t1`.`v2` LIMIT 2 bindSQLHash | 1085294 bindSQL | SELECT * FROM `td`.`t2` INNER JOIN `td`.`t1` ON `td`.`t2`.`v2` = `td`.`t1`.`v2` LIMIT 2 planSQL | SELECT t2.v1 AS c_1, t2.v2 AS c_2, t2.v3 AS c_3, t1.v1 AS c_4, t1.v2 AS c_5 FROM t2 INNER JOIN[SHUFFLE] t1 ON t2.v2 = t1.v2 LIMIT 2 costs | 582.0 queryMs | -1.0 source | USER updateTime | 2025-05-16 14:50:45 ***************************[ 2. row ]*************************** Id | 636134 global | Y enable | Y bindSQLDigest | SELECT * FROM `td`.`t2` INNER JOIN `td`.`t1` ON `td`.`t2`.`v2` = `td`.`t1`.`v2` WHERE `td`.`t2`.`v3` = ? bindSQLHash | 1085294 bindSQL | SELECT * FROM `td`.`t2` INNER JOIN `td`.`t1` ON `td`.`t2`.`v2` = `td`.`t1`.`v2` WHERE `td`.`t2`.`v3` = _spm_const_range(1, 10, 20) planSQL | SELECT t_0.v1 AS c_1, t_0.v2 AS c_2, t_0.v3 AS c_3, t1.v1 AS c_4, t1.v2 AS c_5 FROM (SELECT * FROM t2 WHERE v3 = _spm_const_range(1, 10, 20)) t_0 INNER JOIN[SHUFFLE] t1 ON t_0.v2 = t1.v2 costs | 551.0204081632653 queryMs | -1.0 source | USER updateTime | 2025-05-13 15:29:04 2 rows in set Time: 0.019s MySQL > show baseline where global = true\G; ***************************[ 1. row ]*************************** Id | 636134 global | Y enable | Y bindSQLDigest | SELECT * FROM `td`.`t2` INNER JOIN `td`.`t1` ON `td`.`t2`.`v2` = `td`.`t1`.`v2` WHERE `td`.`t2`.`v3` = ? bindSQLHash | 1085294 bindSQL | SELECT * FROM `td`.`t2` INNER JOIN `td`.`t1` ON `td`.`t2`.`v2` = `td`.`t1`.`v2` WHERE `td`.`t2`.`v3` = _spm_const_range(1, 10, 20) planSQL | SELECT t_0.v1 AS c_1, t_0.v2 AS c_2, t_0.v3 AS c_3, t1.v1 AS c_4, t1.v2 AS c_5 FROM (SELECT * FROM t2 WHERE v3 = _spm_const_range(1, 10, 20)) t_0 INNER JOIN[SHUFFLE] t1 ON t_0.v2 = t1.v2 costs | 551.0204081632653 queryMs | -1.0 source | USER updateTime | 2025-05-13 15:29:04 1 row in set Time: 0.013s MySQL > show baseline on SELECT count(1) AS `count(1)` FROM `old`.`t1` INNER JOIN `old`.`t2` ON `old`.`t1`.`k2` = `old`.`t2`.`k2` LIMIT 10\G; ***************************[ 1. row ]*************************** Id | 679817 global | Y enable | Y bindSQLDigest | SELECT count(?) AS `count(1)` FROM `old`.`t1` INNER JOIN `old`.`t2` ON `old`.`t1`.`k2` = `old`.`t2`.`k2` LIMIT 10 bindSQLHash | 1085927 bindSQL | SELECT count(_spm_const_var(1)) AS `count(1)` FROM `old`.`t1` INNER JOIN `old`.`t2` ON `old`.`t1`.`k2` = `old`.`t2`.`k2` LIMIT 10 planSQL | SELECT count(_spm_const_var(1)) AS c_7 FROM (SELECT 1 AS c_9 FROM t1 INNER JOIN[SHUFFLE] t2 ON t1.k2 = t2.k2) t_0 LIMIT 10 costs | 2532.6 queryMs | 35.0 source | CAPTURE updateTime | 2025-05-27 11:17:48 1 row in set Time: 0.026s ``` ##### Drop Baseline[​](#drop-baseline "Direct link to Drop Baseline") **Syntax**: ```sql DROP BASELINE ,... ``` **Parameter**: `id`: The ID of the Baseline. You can obtain the Baseline ID by executing `SHOW BASELINE`. **Example**: ```sql -- Drop tge Baseline with ID being 140035. DROP BASELINE 140035; ``` ##### Enable/Disable Baseline[​](#enabledisable-baseline "Direct link to Enable/Disable Baseline") **Syntax**: ```sql ENABLE BASELINE ,... DISABLE BASELINE ,... ``` **Parameter**: `id`: The ID of the Baseline. You can obtain the Baseline ID by executing `SHOW BASELINE`. **Example**: ```sql -- enable the baseline with id 140035 ENABLE BASELINE 140035; -- disable the baseline with id 140035 140037 DISABLE BASELINE 140035, 140037; ``` #### Query Rewrite[​](#query-rewrite "Direct link to Query Rewrite") You can enable the SQL Plan Manager query rewrite feature by setting the variable `enable_spm_rewrite` to `true`. ```sql SET enable_spm_rewrite = true; ``` After binding the execution plan, StarRocks will automatically rewrite the corresponding query into the corresponding query plan. **Example**: Check the original SQL and execution plan: ```plain mysql> EXPLAIN SELECT t1.v2, t2.v3 FROM t1, t2 WHERE t1.v2 = t2.v2 AND t1.v1 > 20; +-----------------------------------------+ | Explain String | +-----------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS:2: v2 | 5: v3 | | PARTITION: UNPARTITIONED | ...... | 5:HASH JOIN | | | join op: INNER JOIN (PARTITIONED) | | | colocate: false, reason: | | | equal join conjunct: 4: v2 = 2: v2 | | | | | |----4:EXCHANGE | | | | | 1:EXCHANGE | ...... ``` Create a Baseline to bind the original SQL to a SQL execution plan with Join Hints: ```sql MySQL td> create global baseline on select t1.v2, t2.v3 from t1, t2 where t1.v2 = t2.v2 and t1.v1 > 1000 using select t1.v2, t2.v3 from t1 join[broadcast] t2 on t1.v2 = t2.v2 where t1.v1 > 1000; Query OK, 0 rows affected Time: 0.074s MySQL td> show baseline\G; ***************************[ 1. row ]*************************** Id | 647139 global | Y enable | Y bindSQLDigest | SELECT `td`.`t1`.`v2`, `td`.`t2`.`v3` FROM `td`.`t1` , `td`.`t2` WHERE (`td`.`t1`.`v2` = `td`.`t2`.`v2`) AND (`td`.`t1`.`v1` > ?) bindSQLHash | 1085294 bindSQL | SELECT `td`.`t1`.`v2`, `td`.`t2`.`v3` FROM `td`.`t1` , `td`.`t2` WHERE (`td`.`t1`.`v2` = `td`.`t2`.`v2`) AND (`td`.`t1`.`v1` > _spm_const_var(1)) planSQL | SELECT t_0.v2 AS c_2, t2.v3 AS c_5 FROM (SELECT v2 FROM t1 WHERE v1 > _spm_const_var(1)) t_0 INNER JOIN[BROADCAST] t2 ON t_0.v2 = t2.v2 costs | 1193.0 queryMs | -1.0 source | USER updateTime | 2025-05-16 15:51:36 1 rows in set Time: 0.016s ``` Enable SQL Plan Manager query rewrite and check if the original query is rewritten by the Baseline: ```plain MySQL td> show variables like '%enable_spm_re%' +--------------------+-------+ | Variable_name | Value | +--------------------+-------+ | enable_spm_rewrite | false | +--------------------+-------+ 1 row in set Time: 0.007s MySQL td> set enable_spm_rewrite=true Query OK, 0 rows affected Time: 0.001s MySQL td> explain select t1.v2, t2.v3 from t1, t2 where t1.v2 = t2.v2 and t1.v1 > 20; +-----------------------------------------+ | Explain String | +-----------------------------------------+ | Using baseline plan[647139] | | | | PLAN FRAGMENT 0 | | OUTPUT EXPRS:2: v2 | 5: v3 | ............. | 4:HASH JOIN | | | join op: INNER JOIN (BROADCAST) | | | colocate: false, reason: | | | equal join conjunct: 2: v2 = 4: v2 | | | | | |----3:EXCHANGE | | | | | 1:Project | | | : 2: v2 | ............. ``` #### Advanced usage[​](#advanced-usage "Direct link to Advanced usage") For the following scenarios, you can try using manual query plan binding: * For complex SQL, SQL Plan Manager can't automatically bind the SQL and query plan. * For specific scenarios (for example, fixed parameters or conditional parameters), automatic binding cannot meet the requirements. Compared to automatic binding, manual binding offers greater flexibility but requires understanding some of the execution mechanisms of SQL Plan Manager. ##### Execution Logic[​](#execution-logic "Direct link to Execution Logic") ###### SQL Plan Manager functions[​](#sql-plan-manager-functions "Direct link to SQL Plan Manager functions") SQL Plan Manager function are placeholder functions in SQL Plan Manager with two main purposes: * Mark expressions in parameterized SQL for subsequent parameter extraction and replacement in the process. * Check parameter conditions, mapping SQL with different parameters to different query plans through parameter conditions. Currently, StarRocks supports the following SQL Plan Manager functions: * `_spm_const_var(placeholdID)`: Used to mark a single constant value. * `_spm_const_list(placeholdID)`: Used to mark multiple constant values, typically used to mark multiple constant values in an IN condition. * `_spm_const_range(placeholdID, min, max)`：Used to mark a single constant value, but requires the constant value to be within the specified range '\[min, max]'. * `_spm_const_num(placeholdID, value...)`：Used to mark a single constant value, but requires the constant value to be a value in the specified enum 'value...'. The 'placeholdID' is an integer, which is used as a unique identifier for the parameter, which is used when binding Baseline and generating the Plan. ###### Baseline Creation Process[​](#baseline-creation-process "Direct link to Baseline Creation Process") 1. Execute `CREATE BASELINE` to obtain the `BindSQL` to be bound and the execution plan query `PlanSQL`. 2. Parameterize `BindSQL`: Replace literal values or expressions with SQL Plan Manager functions. For example, replace `id > 200` with `id > _spm_const_var(0)`, where parameter `0` is a placeholder ID used to confirm the position of the expression in `BindSQL` and `PlanSQL`. For more SQL Plan Manager functions, see [SQL Plan Manager Functions](#sql-plan-manager-functions). 3. Bind placeholders in `PlanSQL`: Locate the position of placeholders in `PlanSQL` and replace them with the original expressions. 4. Use the optimizer to optimize `PlanSQL` and obtain the query plan. 5. Serialize the query plan into SQL with Hints. 6. Save the Baseline (SQL fingerprint of BindSQL, optimized execution plan SQL). ###### Query Rewrite Process[​](#query-rewrite-process "Direct link to Query Rewrite Process") The query rewrite logic is similar to the `PREPARE` statement. 1. Execute the query. 2. Normalize the query into an SQL fingerprint. 3. Use the SQL fingerprint to find the Baseline (match against the `BindSQL` of the Baseline). 4. Bind the query to the Baseline, check if the query matches the Baseline’s `BindSQL`, and extract the corresponding parameter values from the query using the SQL Plan Manager functions in `BindSQL`. For example, `id > 1000` in the query is bound to `id > _spm_const_var(0)` in BindSQL, extracting `_spm_const_var(0) = 1000`. 5. Replace the SQL Plan Manager parameters in the Baseline’s `PlanSQL`. 6. Return `PlanSQL` to replace the original query. ##### Bind query manually[​](#bind-query-manually "Direct link to Bind query manually") You can use SQL Plan Manager functions to bind more complex SQL to Baselines. ###### Example 1[​](#example-1 "Direct link to Example 1") For example, the SQL to be bound is as follows: ```sql create global baseline using with ss as ( select i_item_id, sum(ss_ext_sales_price) total_sales from store_sales, item where i_color in ('slate', 'blanched', 'burnished') and ss_item_sk = i_item_sk group by i_item_id ), cs as ( select i_item_id, sum(cs_ext_sales_price) total_sales from catalog_sales, item where i_color in ('slate', 'blanched', 'burnished') and cs_item_sk = i_item_sk group by i_item_id ) select i_item_id, sum(total_sales) total_sales from ( select * from ss union all select * from cs) tmp1 group by i_item_id; ``` Since the constant values in `i_color in ('slate', 'blanched', 'burnished')` are the same, the SQL will be recognized with SQL Plan Manager as: ```sql with ss as ( select i_item_id, sum(ss_ext_sales_price) total_sales from store_sales, item where i_color IN (_spm_const_list(1)) and ss_item_sk = i_item_sk group by i_item_id ), cs as ( select i_item_id, sum(cs_ext_sales_price) total_sales from catalog_sales, item where i_color IN (_spm_const_list(1)) and cs_item_sk = i_item_sk group by i_item_id ) select i_item_id, sum(total_sales) total_sales from ( select * from ss union all select * from cs) tmp1 group by i_item_id; ``` That means both `i_color in ('xxx', 'xxx')` instances are recognized as the same parameter, making it impossible for SQL Plan Manager to distinguish them when different parameters are used in the SQL. ```sql -- can be bind baseline MySQL tpcds> explain with ss as ( select i_item_id, sum(ss_ext_sales_price) total_sales from store_sales, item where i_color in ('A', 'B', 'C') and ss_item_sk = i_item_sk group by i_item_id ), cs as ( select i_item_id, sum(cs_ext_sales_price) total_sales from catalog_sales, item where i_color in ('A', 'B', 'C') and cs_item_sk = i_item_sk group by i_item_id ) select i_item_id, sum(total_sales) total_sales from ( select * from ss union all select * from cs) tmp1 group by i_item_id; +-------------------------------------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------------------------------------+ | Using baseline plan[646215] | | | | PLAN FRAGMENT 0 | | OUTPUT EXPRS:104: i_item_id | 106: sum | | PARTITION: UNPARTITIONED | ................................ | | avgRowSize=3.0 | +-------------------------------------------------------------------------------------------+ 184 rows in set Time: 0.095s -- can't bind basline MySQL tpcds> explain with ss as ( select i_item_id, sum(ss_ext_sales_price) total_sales from store_sales, item where i_color in ('A', 'B', 'C') and ss_item_sk = i_item_sk group by i_item_id ), cs as ( select i_item_id, sum(cs_ext_sales_price) total_sales from catalog_sales, item where i_color in ('E', 'F', 'G') and cs_item_sk = i_item_sk group by i_item_id ) select i_item_id, sum(total_sales) total_sales from ( select * from ss union all select * from cs) tmp1 group by i_item_id; +-------------------------------------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS:104: i_item_id | 106: sum | | PARTITION: UNPARTITIONED | ................................ | | avgRowSize=3.0 | +-------------------------------------------------------------------------------------------+ 182 rows in set Time: 0.040s ``` In such cases, you can manually specify the parameters in `BindSQL` and `PlanSQL`: ```sql create global baseline using with ss as ( select i_item_id, sum(ss_ext_sales_price) total_sales from store_sales, item where i_color IN (_spm_const_list(1)) and ss_item_sk = i_item_sk group by i_item_id ), cs as ( select i_item_id, sum(cs_ext_sales_price) total_sales from catalog_sales, item where i_color IN (_spm_const_list(2)) and cs_item_sk = i_item_sk group by i_item_id ) select i_item_id, sum(total_sales) total_sales from ( select * from ss union all select * from cs) tmp1 group by i_item_id; ``` Check the query is rewritten by the Baseline: ```plain MySQL td> show baseline\G; ***************************[ 1. row ]*************************** Id | 646215 global | Y enable | Y bindSQLDigest | WITH `ss` (`i_item_id`, `total_sales`) AS (SELECT `tpcds`.`item`.`i_item_id`, sum(`tpcds`.`store_sales`.`ss_ext_sales_price`) AS `total_sales` FROM `tpcds`.`store_sales` , `tpcds`.`item` WHERE (`tpcds`.`item`.`i_color` IN (?)) AND (`tpcds`.`store_sales`.`ss_item_sk` = `tpcds`.`item`.`i_item_sk`) GROUP BY `tpcds`.`item`.`i_item_id`) , `cs` (`i_item_id`, `total_sales`) AS (SELECT `tpcds`.`item`.`i_item_id`, sum(`tpcds`.`catalog_sales`.`cs_ext_sales_price`) AS `total_sales` FROM `tpcds`.`catalog_sales` , `tpcds`.`item` WHERE (`tpcds`.`item`.`i_color` IN (?)) AND (`tpcds`.`catalog_sales`.`cs_item_sk` = `tpcds`.`item`.`i_item_sk`) GROUP BY `tpcds`.`item`.`i_item_id`) SELECT `tmp1`.`i_item_id`, sum(`tmp1`.`total_sales`) AS `total_sales` FROM (SELECT * FROM `ss` UNION ALL SELECT * FROM `cs`) `tmp1` GROUP BY `tmp1`.`i_item_id` bindSQLHash | 203487418 bindSQL | WITH `ss` (`i_item_id`, `total_sales`) AS (SELECT `tpcds`.`item`.`i_item_id`, sum(`tpcds`.`store_sales`.`ss_ext_sales_price`) AS `total_sales` FROM `tpcds`.`store_sales` , `tpcds`.`item` WHERE (`tpcds`.`item`.`i_color` IN (_spm_const_list(1))) AND (`tpcds`.`store_sales`.`ss_item_sk` = `tpcds`.`item`.`i_item_sk`) GROUP BY `tpcds`.`item`.`i_item_id`) , `cs` (`i_item_id`, `total_sales`) AS (SELECT `tpcds`.`item`.`i_item_id`, sum(`tpcds`.`catalog_sales`.`cs_ext_sales_price`) AS `total_sales` FROM `tpcds`.`catalog_sales` , `tpcds`.`item` WHERE (`tpcds`.`item`.`i_color` IN (_spm_const_list(1))) AND (`tpcds`.`catalog_sales`.`cs_item_sk` = `tpcds`.`item`.`i_item_sk`) GROUP BY `tpcds`.`item`.`i_item_id`) SELECT `tmp1`.`i_item_id`, sum(`tmp1`.`total_sales`) AS `total_sales` FROM (SELECT * FROM `ss` UNION ALL SELECT * FROM `cs`) `tmp1` GROUP BY `tmp1`.`i_item_id` planSQL | SELECT c_104, sum(c_105) AS c_106 FROM (SELECT * FROM (SELECT i_item_id AS c_104, c_46 AS c_105 FROM (SELECT i_item_id, sum(ss_ext_sales_price) AS c_46 FROM (SELECT i_item_id, ss_ext_sales_price FROM store_sales INNER JOIN[BROADCAST] (SELECT i_item_sk, i_item_id FROM item WHERE i_color IN (_spm_const_list(1))) t_0 ON ss_item_sk = i_item_sk) t_1 GROUP BY i_item_id) t_2 UNION ALL SELECT i_item_id AS c_104, c_103 AS c_105 FROM (SELECT i_item_id, sum(cs_ext_sales_price) AS c_103 FROM (SELECT i_item_id, cs_ext_sales_price FROM catalog_sales INNER JOIN[BROADCAST] (SELECT i_item_sk, i_item_id FROM item WHERE i_color IN (_spm_const_list(1))) t_3 ON cs_item_sk = i_item_sk) t_4 GROUP BY i_item_id) t_5) t_6) t_7 GROUP BY c_104 costs | 2.608997082E8 queryMs | -1.0 source | USER updateTime | 2025-05-16 15:30:29 ***************************[ 2. row ]*************************** Id | 646237 global | Y enable | Y bindSQLDigest | WITH `ss` (`i_item_id`, `total_sales`) AS (SELECT `tpcds`.`item`.`i_item_id`, sum(`tpcds`.`store_sales`.`ss_ext_sales_price`) AS `total_sales` FROM `tpcds`.`store_sales` , `tpcds`.`item` WHERE (`tpcds`.`item`.`i_color` IN (?)) AND (`tpcds`.`store_sales`.`ss_item_sk` = `tpcds`.`item`.`i_item_sk`) GROUP BY `tpcds`.`item`.`i_item_id`) , `cs` (`i_item_id`, `total_sales`) AS (SELECT `tpcds`.`item`.`i_item_id`, sum(`tpcds`.`catalog_sales`.`cs_ext_sales_price`) AS `total_sales` FROM `tpcds`.`catalog_sales` , `tpcds`.`item` WHERE (`tpcds`.`item`.`i_color` IN (?)) AND (`tpcds`.`catalog_sales`.`cs_item_sk` = `tpcds`.`item`.`i_item_sk`) GROUP BY `tpcds`.`item`.`i_item_id`) SELECT `tmp1`.`i_item_id`, sum(`tmp1`.`total_sales`) AS `total_sales` FROM (SELECT * FROM `ss` UNION ALL SELECT * FROM `cs`) `tmp1` GROUP BY `tmp1`.`i_item_id` bindSQLHash | 203487418 bindSQL | WITH `ss` (`i_item_id`, `total_sales`) AS (SELECT `tpcds`.`item`.`i_item_id`, sum(`tpcds`.`store_sales`.`ss_ext_sales_price`) AS `total_sales` FROM `tpcds`.`store_sales` , `tpcds`.`item` WHERE (`tpcds`.`item`.`i_color` IN (_spm_const_list(1))) AND (`tpcds`.`store_sales`.`ss_item_sk` = `tpcds`.`item`.`i_item_sk`) GROUP BY `tpcds`.`item`.`i_item_id`) , `cs` (`i_item_id`, `total_sales`) AS (SELECT `tpcds`.`item`.`i_item_id`, sum(`tpcds`.`catalog_sales`.`cs_ext_sales_price`) AS `total_sales` FROM `tpcds`.`catalog_sales` , `tpcds`.`item` WHERE (`tpcds`.`item`.`i_color` IN (_spm_const_list(2))) AND (`tpcds`.`catalog_sales`.`cs_item_sk` = `tpcds`.`item`.`i_item_sk`) GROUP BY `tpcds`.`item`.`i_item_id`) SELECT `tmp1`.`i_item_id`, sum(`tmp1`.`total_sales`) AS `total_sales` FROM (SELECT * FROM `ss` UNION ALL SELECT * FROM `cs`) `tmp1` GROUP BY `tmp1`.`i_item_id` planSQL | SELECT c_104, sum(c_105) AS c_106 FROM (SELECT * FROM (SELECT i_item_id AS c_104, c_46 AS c_105 FROM (SELECT i_item_id, sum(ss_ext_sales_price) AS c_46 FROM (SELECT i_item_id, ss_ext_sales_price FROM store_sales INNER JOIN[BROADCAST] (SELECT i_item_sk, i_item_id FROM item WHERE i_color IN (_spm_const_list(1))) t_0 ON ss_item_sk = i_item_sk) t_1 GROUP BY i_item_id) t_2 UNION ALL SELECT i_item_id AS c_104, c_103 AS c_105 FROM (SELECT i_item_id, sum(cs_ext_sales_price) AS c_103 FROM (SELECT i_item_id, cs_ext_sales_price FROM catalog_sales INNER JOIN[BROADCAST] (SELECT i_item_sk, i_item_id FROM item WHERE i_color IN (_spm_const_list(2))) t_3 ON cs_item_sk = i_item_sk) t_4 GROUP BY i_item_id) t_5) t_6) t_7 GROUP BY c_104 costs | 2.635637082E8 queryMs | -1.0 source | USER updateTime | 2025-05-16 15:37:35 2 rows in set Time: 0.013s MySQL td> explain with ss as ( select i_item_id, sum(ss_ext_sales_price) total_sales from store_sales, item where i_color in ('A', 'B', 'C') and ss_item_sk = i_item_sk group by i_item_id ), cs as ( select i_item_id, sum(cs_ext_sales_price) total_sales from catalog_sales, item where i_color in ('E', 'F', 'G') and cs_item_sk = i_item_sk group by i_item_id ) select i_item_id, sum(total_sales) total_sales from ( select * from ss union all select * from cs) tmp1 group by i_item_id; +-------------------------------------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------------------------------------+ | Using baseline plan[646237] | | | | PLAN FRAGMENT 0 | | OUTPUT EXPRS:104: i_item_id | 106: sum | | PARTITION: UNPARTITIONED | ...... ``` From the output above, you can see the query is rewritten by the Baseline with distinguished parameters in SQL Plan Manager functions. ###### Example 2[​](#example-2 "Direct link to Example 2") For the following query, if you want to use a different baseline for different `i_color` ```sql select i_item_id, sum(ss_ext_sales_price) total_sales from store_sales join item on ss_item_sk = i_item_sk where i_color = 25 group by i_item_id ``` You can use '\_spm\_const\_range' to: ```sql -- 10 <= i_color <= 50 时，使用 SHUFFLE JOIN MySQL tpcds> create baseline using select i_item_id, sum(ss_ext_sales_price) total_sales from store_sales join[SHUFFLE] item on ss_item_sk = i_item_sk where i_color = _spm_const_range(1, 10, 50) group by i_item_id Query OK, 0 rows affected Time: 0.017s -- i_color 为 60，70，80 时，使用 BROADCAST JOIN MySQL tpcds> create baseline using select i_item_id, sum(ss_ext_sales_price) total_sales from store_sales join[BROADCAST] item on ss_item_sk = i_item_sk where i_color = _spm_const_enum(1, 60, 70, 80) group by i_item_id Query OK, 0 rows affected Time: 0.009s MySQL tpcds> show baseline\G; ***************************[ 1. row ]*************************** Id | 647167 global | N enable | Y bindSQLDigest | SELECT `tpcds`.`item`.`i_item_id`, sum(`tpcds`.`store_sales`.`ss_ext_sales_price`) AS `total_sales` FROM `tpcds`.`store_sales` INNER JOIN `tpcds`.`item` ON `tpcds`.`store_sales`.`ss_item_sk` = `tpcds`.`item`.`i_item_sk` WHERE `tpcds`.`item`.`i_color` = ? GROUP BY `tpcds`.`item`.`i_item_id` bindSQLHash | 68196091 bindSQL | SELECT `tpcds`.`item`.`i_item_id`, sum(`tpcds`.`store_sales`.`ss_ext_sales_price`) AS `total_sales` FROM `tpcds`.`store_sales` INNER JOIN `tpcds`.`item` ON `tpcds`.`store_sales`.`ss_item_sk` = `tpcds`.`item`.`i_item_sk` WHERE `tpcds`.`item`.`i_color` = _spm_const_range(1, 10, 50) GROUP BY `tpcds`.`item`.`i_item_id` planSQL | SELECT i_item_id, sum(ss_ext_sales_price) AS c_46 FROM (SELECT i_item_id, ss_ext_sales_price FROM store_sales INNER JOIN[SHUFFLE] (SELECT i_item_sk, i_item_id FROM item WHERE i_color = _spm_const_range(1, '10', '50')) t_0 ON ss_item_sk = i_item_sk) t_1 GROUP BY i_item_id costs | 1.612502146E8 queryMs | -1.0 source | USER updateTime | 2025-05-16 16:02:46 ***************************[ 2. row ]*************************** Id | 647171 global | N enable | Y bindSQLDigest | SELECT `tpcds`.`item`.`i_item_id`, sum(`tpcds`.`store_sales`.`ss_ext_sales_price`) AS `total_sales` FROM `tpcds`.`store_sales` INNER JOIN `tpcds`.`item` ON `tpcds`.`store_sales`.`ss_item_sk` = `tpcds`.`item`.`i_item_sk` WHERE `tpcds`.`item`.`i_color` = ? GROUP BY `tpcds`.`item`.`i_item_id` bindSQLHash | 68196091 bindSQL | SELECT `tpcds`.`item`.`i_item_id`, sum(`tpcds`.`store_sales`.`ss_ext_sales_price`) AS `total_sales` FROM `tpcds`.`store_sales` INNER JOIN `tpcds`.`item` ON `tpcds`.`store_sales`.`ss_item_sk` = `tpcds`.`item`.`i_item_sk` WHERE `tpcds`.`item`.`i_color` = _spm_const_enum(1, 60, 70, 80) GROUP BY `tpcds`.`item`.`i_item_id` planSQL | SELECT i_item_id, sum(ss_ext_sales_price) AS c_46 FROM (SELECT i_item_id, ss_ext_sales_price FROM store_sales INNER JOIN[BROADCAST] (SELECT i_item_sk, i_item_id FROM item WHERE i_color = _spm_const_enum(1, '60', '70', '80')) t_0 ON ss_item_sk = i_item_sk) t_1 GROUP BY i_item_id costs | 1.457490986E8 queryMs | -1.0 source | USER updateTime | 2025-05-16 16:03:23 2 rows in set Time: 0.011s MySQL tpcds> MySQL tpcds> explain select i_item_id, sum(ss_ext_sales_price) total_sales from store_sales join item on ss_item_sk = i_item_sk where i_color = 40 -- hit SHUFFLE JOIN group by i_item_id +-------------------------------------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------------------------------------+ | Using baseline plan[647167] | | | | PLAN FRAGMENT 0 | ................. | | | | 5:HASH JOIN | | | join op: INNER JOIN (PARTITIONED) | | | colocate: false, reason: | | | equal join conjunct: 2: ss_item_sk = 24: i_item_sk | | | | | |----4:EXCHANGE | | | | | 1:EXCHANGE | ................. MySQL tpcds> explain select i_item_id, sum(ss_ext_sales_price) total_sales from store_sales join item on ss_item_sk = i_item_sk where i_color = 70 -- hit BROADCAST JOIN group by i_item_id +-------------------------------------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------------------------------------+ | Using baseline plan[647171] | | | | PLAN FRAGMENT 0 | ................. | 4:HASH JOIN | | | join op: INNER JOIN (BROADCAST) | | | colocate: false, reason: | | | equal join conjunct: 2: ss_item_sk = 24: i_item_sk | | | | | |----3:EXCHANGE | | | | | 0:OlapScanNode | | TABLE: store_sales | ................. MySQL tpcds> explain select i_item_id, sum(ss_ext_sales_price) total_sales from store_sales join item on ss_item_sk = i_item_sk where i_color = 100 -- don't use any Baseline group by i_item_id +-------------------------------------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS:25: i_item_id | 46: sum | | PARTITION: UNPARTITIONED | | | | RESULT SINK | ................. ``` ##### Auto-Capture[​](#auto-capture "Direct link to Auto-Capture") Auto-Capture queries the query SQL statements in the past period of time (default 3 hours), and generates and saves the baseline based on these queries, and the generated baseline is in the 'disable' state by default and doesn't take effect immediately. In the following scenarios: * After the upgrade, the execution plan changes, resulting in a higher query time * After the data changed, and the statistics are changed, resulting in a change in the execution plan You can find the historical baseline with `show baseline` and manually roll back the plan with `enable baseline`. The Auto-Capture feature depends on the save query history feature and requires the following settings: ```sql set global enable_query_history=true; ``` The query history is stored in the '*statistics*.query\_history' table. To enable automatic capture: ```sql set global enable_plan_capture=true; ``` Other configurations： ```sql -- The historical duration of the storage query history, unit: seconds, defaults to 3 days set global query_history_keep_seconds = 259200; -- The work interval of Auto-Capture, unit: seconds, defaults to 3 hours set global plan_capture_interval=10800; -- Captures regular checks for SQL tables, only captures SQL when table names(db.table) can match plan_capture_include_pattern, defaults .*, which represents all tables set global plan_capture_include_pattern=".*"; ``` note 1. Save query history and Auto-Capture will cost some storage and computing resources, so please set it reasonably according to your own scenarios. 2. After binding baselines, newly added optimizations after upgrades may become ineffective, so auto-captured baselines are disabled by default. ###### Usage Example[​](#usage-example "Direct link to Usage Example") Use Auto-Capture feature during upgrades to avoid plan regression issues after upgrade: 1. Enable Auto-Capture functionality 1-2 days before upgrade: ```sql set global enable_query_history=true; set global enable_plan_capture=true; ``` 2. StarRocks starts to periodically record query plans, which can be viewed with `show baseline` 3. Upgrade StarRocks 4. After upgrade, check query execution time, or use the following SQL to identify queries with plan changes: ```sql WITH recent_queries AS ( -- Use query execution time within 3 days as average execution time SELECT dt, -- Query execution time sql_digest, -- SQL fingerprint of the query `sql`, -- Query SQL query_ms, -- Execution time plan, -- Query plan used AVG(query_ms) OVER (PARTITION BY sql_digest) AS avg_ms, -- Average execution time within SQL fingerprint group RANK() OVER (PARTITION BY sql_digest ORDER BY plan) != 1 AS is_changed -- Count different plan formats as change indicator FROM _statistics_.query_history WHERE dt >= NOW() - INTERVAL 3 DAY ) -- Queries with execution time higher than 1.5 times the average in the last 12 hours SELECT *, RANK() OVER (PARTITION BY sql_digest ORDER BY query_ms DESC) AS rnk FROM recent_queries WHERE query_ms > avg_ms * 1.5 and dt >= now() - INTERVAL 12 HOUR ``` 5. Based on plan change information or query execution time information, determine if plan rollback is needed 6. Find the corresponding baseline for the SQL and time point: ```sql show baseline on ``` 7. Use Enable baseline to roll back: ```sql enable baseline ``` 8. Turn on baseline rewrite switch: ```sql set enable_spm_rewrite = true; ``` #### Future Plans[​](#future-plans "Direct link to Future Plans") In the future, StarRocks will provide more advanced features based on SQL Plan Manager, including: * Enhanced stability checks for SQL Plans. * Automatic optimization of fixed query plans. * Support for more conditional parameter binding methods. ---